Robotics and Deep Reinforcement Learning at Neural Information Processing Systems (NeurIPS) 2020
By: Nathan Lambert
Tags: Artificial Intelligence, Reinforcement Learning, Robotics, Machine Learning

Other Talks

Otherwise, there were some other talks that were interesting, but not “must watch” territory.

Besmira Nushi from MSR on human-centered AI

This talk was focusing on engineering tools to help minimize data and algorithmic bias. I liked parts of it because they actually detailed different design decisions that could be made in realistic scenarios, rather than broadly discussing the problems of AI Bias. They contacted ML engineers rather than just researchers, as they implement everything discussed. The relevant paper and code

Martha White’s on sources of uncertainty in policy-gradient methods

An interesting talk discussing the three sources of variance in policy-gradient methods of RL (state sampling, action sampling, and reward sampling). It is a very good review of policy gradients, and a lesson for how to investigate a problem space. For state-sampling, people use mini-batches when computing gradients. The other two sources, action and return, involve more detailed solutions. Reducing for actions is by looking at all possible actions (simplest). In practice this doesn’t work because it can be expensive and we don’t know Q^pi. The solution is to use a baseline. The key is that the control variate z is uncorrelated across the whole expectation of actions.

Reducing reward variance via an estimator, which doesn’t seem like solving the whole problem (estimators aren’t perfect).

Why did I include this talk: it is important to be able to reason about your problem space and go into abundant detail about where your implementation may be imperfect. Variance comes up in every numerical/data-driven method because we do not have infinite data. I was making connections during this talk to the problems of model-based learning and how it is very hard to disambiguate uncertainty introduced by the model. In a way, MBRL could be diagnosed like this talk with a fourth source of uncertainty: structural — ie variance from model and controller formulation decisions (this talk focuses on policy gradient specifically).

America’s Cup boat design via RL

This expo talk from QuantumBlack caught my attention because it combined theAmerica’s cup + RL. I was impressed by the level of implementation details they discussed. There were a lot of items such as starting and stopping a simulation (where PPO is easier to work with than SAC). Also how to deploy many elements and integrate high-fidelity simulations with RL. The crucial question is if this RL actually helps, or if the design is so under-optimized that any offline method would likely give substantial gains.

The new boats are insane. Source.

I had some more thoughts on twitter.

Jeff Shamma: Perspectives from Feedback Control on ML talk

This talk I included because I think it is an area a lot of people could benefit from, but it requires a lot of skills (machine learning and control theory expertise) to benefit from. It goes through a running example of attitude control of an aircraft and what takeaways people can take from applying learning to classic controls problems.