Welcome to the Meta-World
By: Ronak Mehta
Tags: Robotics, Deep Learning, General Ai, Artificial Intelligence, Reinforcement Learning

Welcome to the Meta-World

Improvements in compute have been a key component in AI progress, thanks to Moore’s law or rather its generalization of continued exponential falling cost per unit of computation. Computational scaling can lead to better performance which is often complementary to the algorithm advances. To seek an improvement that makes a difference in the short term, AI researchers tend to utilize the human understanding of the domain, but it is often seen that deep learning methods rely less on human knowledge to produce dramatically more performance. Based on historical observations, building human-based knowledge into the agents didn’t help in the long run since objects, multiple agents, and symmetries are part of an arbitrary, intrinsically complex, outside world. Since the complexity is endless, meta-methods can be used to find and capture the arbitrary complexity.

Why is generalization required in Meta-reinforcement learning?

Meta-reinforcement learning (Meta-RL) is meta-learning on reinforcement learning tasks with the ability to adapt efficiently to unseen test tasks. Existing Meta-RL algorithms are sensitive to the task distribution shift. When policies are meta-trained on narrow task distributions, the meta-learning model cannot possibly generalize to quickly acquire entirely new tasks. A good meta-learning model should generalize to new tasks or new environments that have never been encountered before. Even without any explicit fine-tuning, the meta-learning model should adjust internal hidden states autonomously to learn. Hence, to enable faster acquisition of entirely new behaviors, the model should be evaluated based on task distributions that are sufficiently broad to enable general-purpose methods which would solve all RL problems without much human interference or manual feature engineering.

Background

The goal of Multi-task Reinforcement Learning (Multi-task RL) is to learn a single policy that can solve a fixed set of skills more efficiently with minimal data rather than learning the skills individually. But in the Meta-RL, prior experience is used to learn and optimize for faster adaption to new skills.

Why is there a lack of established benchmarks and evaluation protocols for Multi-task RL and Meta-RL?

The evaluation of Multi-task RL methods has been done on disjoint and excessively diverse tasks such as the Atari suite, where there is little efficiency to be gained by learning across games and the evaluation of meta-RL methods have been done on very narrow task distributions.

For example, as shown above, the evaluation of meta-learning involves choosing different running directions for simulated-legged robots. This enables faster adaption to new directions. Even though they are technically different tasks, but meta-learned models are capable of much more than this and can adapt to any new task within some domain. Hence, to make it feasible to design new algorithms that can generalize and adapt quickly to meaningful distinct tasks, we need a benchmark, evaluation protocols, and task suites that are broad enough to enable more generalization with sufficient shared structure in the distribution of tasks.

Introducing Meta-World

Meta-World is an open-source simulated benchmark for Meta-RL and Multi-task RL. This benchmark contains 50 distinct simulated robotic manipulation tasks with everyday objects, all of which are contained in a shared environment with a simulated Sawyer arm. This section describes the design decisions behind the range of tasks in the Meta-World benchmark, describes the representation of the actions, observations, and rewards, and presents a set of evaluation protocols of varying difficulty for both meta-RL and multi-task RL.

The Space of Manipulation Tasks

Two critical assumptions of Meta-learning are:

The meta-training and meta-test tasks are drawn from the same distribution

The task distribution exhibits a shared structure that can be used to learn new tasks efficiently.

Generalization cannot be hoped when task distributions are within particular control tasks. For example, an agent won’t be able to quickly learn to open a door without experiencing doors before. Since the agent was trained on meta-training tasks that are homogeneous and narrow. Hence, in new manipulation tasks continuous parametric variation cannot be used to differentiate between tasks. But with non-parametric variation, there is a likelihood that the tasks will not allow having a common shared structure.

Without parametric variation, the model would memorize a particular door at a particular location and any other object at another location could be let’s say drawer. So if the positions are not fixed, then the model is forced to generalize broadly. Hence, each task must include parametric variation in object and goal positions as shown above. This parametric variability would allow a variety of tasks that could be used with a meta-trained model to generalize and acquire entirely new tasks efficiently. So with enough tasks and variation within tasks, pairs of distinct tasks might overlap and help in generalizing. For example, pushing a block and closing a drawer can appear as the same task for some initial and goal positions of the object.

The 50 Meta-World Tasks

The 50 environments include parametric variability for each of the tasks and require the same robotic arm to interact with different objects, shapes, joints, and connectivity. The robotic arm performs a range of actions like reaching, pushing, and grasping depending on the task. These Meta-World tasks are implemented in the MuJoCo physics engine and are interfaced with the OpenAI Gym environment interface. These tasks can be visualized as shown below.