The best of CoRL 2017
I was fortunate enough to attend the very first conference on robot learning (CoRL) held recently in Mountain View. The conference gathered…
Category: Machine Learning
Author: Shay Zweig

The best of CoRL 2017
I was fortunate enough to attend the very first conference on robot learning (CoRL) held recently in Mountain View. The conference gathered the top researchers in the field and presented the state of the art in terms of applying machine learning to robotic systems. In this post I’ll give a general overview of the works presented as well as my opinions regarding the field and the challenges we will be facing in the near future. You can find videos of the full conference here: Day 1, Day 2, Day 3

The Good News
If you ask most people about AI, one of the first things that will rise in the conversation are robots. However, somewhat surprisingly, most of the work being done in robotics, does not involve machine learning, which became almost a synonym for AI in the past few years. There is a growing gap between robotics and machine learning, which must be breached if our goal is to create autonomous systems with some physical existence in this world. Therefore, the very fact that this conference was established this year is a very good sign and emphasizes that both in the academy and the industry, there is a focus to develop learning capabilities for robots. Several of the works presented described some breakthroughs that can benefit the general ML community. The keynotes were inspirational, giving a fascinating overview of topics such as Human-Robotic Interaction, Active Learning and others. To conclude, the conference presented high quality research, and extensive progress in the field is only a matter of time.
The Not So Good News
One general notion that bothered me was that despite the latest proliferation in robotic platforms it seems that most of the work is still done using well known robotic arm platforms. This inevitably resulted in the fact that governing tasks are still the classic grasping, picking, pushing, reaching etc. Social robots for instance, almost did not get any representation in the works presented. The reasoning behind this is clear and understandable. Years of research around this area resulted in many tools to support using these research platforms from simulators to benchmarks. Yet, I personally feel (and predict) that we must extend our tools to new platforms. Some comfort can be gained by noting that despite the homogeneity in the tasks and platforms, a lot of the work can be generalized to other domains.
The talks
And now to the good stuff. The conference was a one track conference so I got to attend all talks. I cannot of course cover all the papers in this post but I will cover some of the topics talked about in the conference with references to works I found interesting. The papers I leave out are not in any sense inferior to the work that I do reference, but one must make hard choices.
Keynotes
The two talks I found the most interesting were the keynotes by Stefanie Tellex from Brown University and Anca Dragan from Berkley. Both talks discussed human robotics interactions, each from a different perspective. I will only give a brief description of each of their talks but if you find these topics interesting, I strongly encourage you to watch the videos of their talks and of course, take a look at some of their papers.
Anca Dragan’s talk discussed two main issues. The first one is what she called the problem of coordination: how to make a robot actively coordinate with humans rather than giving the humans constant priority. The toy example was cutting in front of a car in order to switch lanes. Classic algorithms will never try to switch lanes if they don’t have room however humans will try this a lot. The general solution is for the robot to account for the utility function of the human it interacts with, within its own utility function (Sadigh et al. 2016). The second part of her talk was devoted to the learning algorithms designers and specifically reward engineering. Here she presented the “inverse reward design” where the designed reward is considered as only an observation over the actual reward taking the training environment into consideration (Hadfield-Menell et al. 2017) .
Stefanie Tellex discussed the right way to model a robotic learning system and how it can incorporate interactions with humans. She described three approaches for robots collaborating with people: 
1. Communication for collaboration: allowing robots to better understand and express language(Whitney et al. 2017, Tellex et al. 2014).
2. Action for collaboration: how robots can plan under a very large, and uncertain state-action space(Gopalan et al. 2017, Arumugam et al. 2017).
3. Perception for collaboration: how to improve the perception (Oberlin and Tellex 2017), all in the purpose of better complying with the robot’s human collaborator needs.
Other keynotes were given by Rodney Brooks, talking about what he considers the seven sins of AI (you can read all about it in his blog post here). Yann Lecun gave a great brief of the “CNN for computer vision” world, emphasizing the importance of unsupervised learning. He weaved it into the conference main theme by presenting some nice robotics applications. I found the final section of his talk, about predictive models, fascinating. He mentioned some video prediction works, memory networks, and also a new work just released to arXiv called error encoding networks, where the network predicts her own errors and feeds them back as a signal for the next prediction. Andrew Bagnell from MIT was the final speaker, discussing RL as an adversarial game and designing methods that tries to find an equilibrium in the game rather than an optimum for the system.
Paper Talks
The paper talks can be roughly divided into 6 topics. I will briefly review each topic presenting a paper or two which I personally liked.
Transferring Knowledge from Simulations to the Real World
An ever-growing concern in the world of robotics is how to generalize models learned in the simulator to the real world. While simulators can capture a lot of the variability in the robotic environment, it can never completely model the real world and therefore models learned in the simulator often fails in real world scenarios. This concern was only amplified with the latest proliferation of deep learning that usually demand massive amounts of data for training. It is generally unrealistic to gather these amounts outside of the simulator. Several papers presented at the conference tackled this issue. One paper that I liked, from DeepMind, called Sim-to-Real Robot Learning from Pixels with Progressive Nets, used a framework called progressive nets. This is an interesting architecture used mainly for continuous learning, i.e. learning tasks in which new data is continuously introduced. The basic architecture is composed of a first network which is being trained on the first (and usually biggest) chunk of data. After training, all the weights in this network are frozen. When a new batch of data arrives, instead of fine tuning the first network, a second network is built, usually with a thinner architecture, and an adapter function is built from all the layers in the first network to the layers in the second network. This adapter function can be as simple as the identity function or it could have learned parameters. Then the second network is trained. This is an iterative process in which technically with each batch of new data, new networks can be built with an adapter function to all previous networks.
Depiction of a progressive network . recreated from figure 1 in the paper
In this paper, the progressive nets architecture was used to adapt a model trained on data from the simulator to the real world. The first network was trained on the simulated data (which was abundant) and it was than adapted using the second network to a small amount of real-world data. This approach gained better results then simply fine tuning the second network. Interestingly, the learned concepts could be adapted also for a different task using a third network trained on this new task.
Active Learning
Active learning is a fascinating field which I am personally very fond of. In my opinion this might be the single most important building block for HRI (human-robotics interaction) to be successful. The basic setup in active learning is of an agent that tries to learn a model of the world. While he is learning, he develops a belief regarding the state of the world with some uncertainty. The unique thing here is that the agent can interact with the world and “ask” for information that might help him. This is the key point — the ability to perform learning by actively gaining information. Robots fit very well with this framework as they are agents that can interact with the world and sometimes are able to ask questions using some modality. Research regarding active learning usually involves optimizing the decisions regarding what to ask and when to ask it. One interesting paper was Opportunistic Active Learning for Grounding Natural Language Descriptions. In this paper, the authors referred to a task where a robot tries to infer the intention of a user based on its natural language description. Specifically, the robot tries to understand which object the user wants it to pick, using only the human description of the object. For example, the user can say “Bring me that blue and narrow bottle”. The authors introduced a new policy, that does not limit the agent to asking questions regarding the task at hand, but allows it to explore using off-topic questions. They called the agent using this policy, the inquisitive agent. They claim that the inquisitive agent gained better results without annoying the users too much (this was measured by a set of questions regarding the interactions).
A description of the decision tree of the robot in the research. Recreated from figure 2 in the paper.
While I find the results and analysis somewhat lacking, I think that the approach is quite interesting and should be further explored.
Imitation Learning
This is a very active field where human knowledge is harnessed into the learning process. The basic idea is to teach robots to learn from human demonstrations. Usually a human will demonstrate how to perform a specific task and the robot is expected to learn and execute the task in a similar manner. The demonstration can include a variety of modalities, from visual (i.e. the robot looks at the demonstrator performing the task) to physical guidance (e.g. The robotic arm is being guided to pick and drop). This is a very powerful tool that could enhance transfer learning to new tasks and allow smart exploration of the state space. Many interesting papers were presented around this topic. IntentionGAN: Multi-Task Imitation Learning from Unstructured Demonstrations was one of my favorites. The authors basically adopted the concepts used in GAN into the imitation learning domain — which is classic in my opinion. A robot tries to infer the policy that imitates the action it sees and a discriminator network tries to discriminate between the robot’s action and a human action. The target of the robot is to minimize the discriminator success. Using this approach they manage to learn different skills from imitations from unlabeled data (demonstrations without the skill label). They experimented their methods on different simulated (only…) tasks yielding good results. Another very impressive paper was: One-Shot Visual Imitation Learning via Meta-Learning. Here the authors aimed for one shot demonstration learning from raw pixels (i.e. learning from one example fed as a video to the model), which is quite ambitious. To achieve this, they relied on a meta learning architecture. The idea is to use data collected from demonstrations of other tasks to learn the parameters of a meta model from which using one gradient step over one demonstration can reach a good performance on a new task. The meta training process includes sampling a batch of tasks and then sampling two demonstrations for each task. For each task: use the first demonstration to perform a GD step and create a set of specialized parameters. Use the specialized parameters from each task to try and imitate the second demonstration. And finally use the second demonstration to evaluate the performance and perform a GD step on the meta parameters. They also introduced some interesting architectural innovations such as a two-headed network that incorporates the meta learning and learning steps together as well as reparameterization of the bias they called “bias transformation” that allows some control over the bias in the shape of some more parameters.
The meta learning network architecture. Recreated from figure 2 in the paper
Their results on both simulations and real robots are quite impressive though one can argue that they did not introduce new tasks but rather used the same tasks with new objects.
Autonomous Driving
There is no need to introduce this topic. Autonomous cars are basically big robots and therefore can be treated as such. The papers presented were mainly focused on the planning and decision making components within the autonomous car algorithmic stack, and less on the perception aspects. Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation is a nice paper that discusses robust sensor fusion. The authors introduce two innovations in their policy network. First is a sensor dropout layer — which is basically dropout in the sensor level, i.e. randomly dropping sensors as part of the training process. This results in a network that is robust facing the loss of sensory input which is essential in an autonomous car setting.
The sensor dropout layer. Recreated from figure 1 in the paper
The other innovation is an auxiliary loss aiming at reducing the variance between policies inferred by different subsets of the same sensory input. So instead of training the network to infer the ground truth policy for each subset of sensors, also train it to produce similar outputs regardless of it being close or far from the GT.
One cool paper was: Emergent Behaviors in Mixed-Autonomy Traffic. It discusses the “Mixed autonomy traffic problem” in which both autonomous and human agents are taking part of the same system. The autonomous agents’ objective is to optimize traffic given the presence of the human agents. They showed some nice demos in which introducing only one autonomous car (!) into a system results in completely abolishing traffic jams.
Reinforcement Learning
As expected, the most prevalent learning framework in the conference was reinforcement learning. Many of the papers introduced innovations in RL. In Reverse Curriculum Generation for Reinforcement Learning the authors aim to deal with the problem of learning under a sparse reward function. In many tasks, there is a goal to be reached without getting any reward signal before reaching the goal. For instance, games can have only a win/loss signal at the end of the game without any intermediate reward. Several robotic tasks suffer from the same problem where reward is only given at the end of the task. The reward in these goal-oriented RL problems is binary and sparse over the state space. The problem with sparse reward is that most samples will have no signal and therefore will not be useful in learning, the only useful samples are those that reach the target and they are too few. The signal propagates slowly and learning is hard. The common solution to this problem is engineering the reward function to be denser. The problem with this approach is that when the reward does not actually represent the target, it could easily lead to an undesired outcome. Instead of reward engineering, the authors of this paper suggest changing the sampling regime. It requires the knowledge of one target-point, the algorithm samples starting points close to this target-point and uses them in its learning. Under the assumption that close to the target it is easily reachable, the local sampling should lead to several positive trajectories. After each iteration, the starting points that turned out to be “successful” are concatenated to the target point and the sampling process repeats itself, expanding the circle of possible starting points. The signal propagates in reverse and learning is fast and feasible. There are a few delicate points here worth mentioning. First is the notion of proximity: which states can be considered close to the target point? The approach taken in the paper is noise sampling in action space around the known “good points” (i.e. the target points, and all the starting points concatenated to it in following iterations). Another important issue is balancing between old and new starting points, in each iteration the authors discover “new” starting points from the sampling procedure, they then concatenate a group of “old” starting points from previous iteration to keep refining the estimation. The final starting point for each training iteration is sampled from the group of old and new starting points. The method is tested on four different challenging tasks, including trying to place a ring on a peg and key insertion, both usually fail under reward engineering. Good results are obtained on all tasks, some of them are only able to gain actual success by using this method.
Other interesting papers which I couldn’t fit into a specific category:
Self-Supervised Visual Planning with Temporal Skip Connections 
Using a CNN based Video prediction model to predict how an action will affect the state. Using an RNN that gets a frame and action and outputs the transformation matrices to the next frame. They introduce skip connections to enable looking to the past and deal with occlusions.
How Robots Learn to Classify New Objects Trained from Small Data Sets
Using Progressive Neural Networks for transfer learning from one task to another.
Learning to Fly by Crashing
Using standard RL but letting the drone explore and crush if needed. They had really funny videos and gained great results.
Harvesting Common-sense Navigational Knowledge for Robotics from Uncurated Text Corpora
Tries to apply the word2vec arithmetic trick (ie.g. king-man+woman=queen) in a better manner introducing a new operator that gives more weight to the direction of the difference vector on the expense of its length.
Datasets
Finally, there were some new datasets introduced in the conference, the most noteworthy is CARLA: An Open Urban Driving Simulator which looks like the next generation of simulators for autonomous car. It includes many environments, weathers, and even sensor inputs from the car, and it is completely open sourced, looks quite impressive. Other datasets were presented in the papers:
image2mass: Estimating the Mass of an Object from Its Image: Mapping between object from amazon and their mass.
Semi-Supervised Haptic Material Recognition for Robots using Generative Adversarial Networks: includes a dataset taken from touch sensors applied to many objects.
CORe50: a New Dataset and Benchmark for Continuous Object Recognition: Dataset composed of 50 temporally changing objects. Includes three benchmarks: New instances, new classes and new instances and classes.
If you survived so far, I guess you are really interested in these stuff and you should probably attend the conference next year :) My personal experience was great, I suspect we will see proliferation in this area in the upcoming years. With new platforms that are being introduced, new challenges arises, and with them, the interest of the community. I am really looking forward to seeing the progress in this fascinating field.