Model-based micro-data reinforcement learning
By: Balázs Kégl
Tags: Heteroscedasticity, Dynamic Systems, Autopilot, Model Based Rl, Reinforcement Learning

Model-based micro-data reinforcement learning

Joint work with Gabriel Hurtado and Albert Thomas. This research is part of a broader theme to build AI autopilots for engineering systems.

The corresponding research paper.

Open source code to reproduce all our results.

Video teaser.

Context

One of our main research objectives at the Noah’s Ark Lab in Huawei France is to build AI autopilots for engineering systems.

We can improve various metrics: making the systems better, cheaper, more reliable, safer, or more energy efficient. Typical systems we are working on include the cooling system of data centers, the system that manages the connections in the wireless antenna, or your local Wi-Fi network. The applications of the technology we are developing are countless: given that engineering systems are the backbone of most industry and transportation, making AI useful in this domain is arguably a multi-trillion dollar endeavor.

What makes engineering systems special: they are physical, not getting faster with time, unlike computers, and they are tightly controlled by system engineers. This means that we need to learn a good control policy based on small offline data and iterate with systems engineers slowly: a typical iteration of policy validation and data taking may take several weeks even months. Model-based RL (MBRL) is considered the best approach for this micro-data regime, and it fits well the human work process we can hope to implement within the constraints of a real engineering system. In addition, systems engineers can validate the learned simulator on its own, increasing their trust in the process.

A typical controlled engineering system. System = airplane, engineer = pilot. (All images by author.)

Where to start?

Scanning the literature, we found excellent model-based techniques and this immensely useful horizontal study on the best known models compared on many systems. The MBRL research field seems to operate in an engineering paradigm where the main objective is to design end-to-end systems that beat the state of the art on some benchmark systems. Our goal is to learn generalizable principles and state and rigorously test hypotheses, while also attempting to obtain good performance on benchmark systems. More pragmatically, we would like to develop

a toolkit (a model and agent zoo) and well-established guidelines

so that the data scientist (and eventually the systems engineer) can make informed decisions given his or her system to automate. For this, we started with a systematic study on models and proper ways to compare and tune them. We defined a set of criteria to compare models, not only on their performance but also on their practical usability. We designed rigorous and fair experiments, tested and (in)validated hypotheses, and found some curious experimental facts that we cannot yet explain.

We made a choice to do a vertical study: instead of running the methods on several environments, we chose one, the relatively simple yet challenging Acrobot system. Unlike the name suggests, Acrobot mimics a gymnast on the horizontal bar rather than an acrobat. It is an underactuated double pendulum (the only action is a discrete torque at the second joint: the gymnast can kick his legs forward, backward, or no action). It is noiseless but chaotic at the top equilibrium. We use Acrobot with continuous (dense) reward (height of the tip) since it is closer to what we have in engineering systems than Acrobot with the (sparse) 0/1 reward (height > 3 on our scale, the horizontal line in the video below).

The Acrobot system: underactuated double pendulum, mimicking a gymnast.

The Acrobot system with a random shooting agent and a trained DARMDN model.

We used two setups to test the different aspects of the models. The “sincos” system predicts the sine and cosine of the two angles, eliminating the discontinuities at ±π, while introducing strong (functional) y-interdependence between the observables. This is the usual setup in the literature.

How different models handle y-interdependence (dependence of observables even given the history). Multivariate Gaussian models (like Gaussian processes) “spread” the uncertainty in all directions. Mixtures of multivariate Gaussians (DMDN) may “tile” the nonlinear y-interdependence by smaller Gaussians but the match is not perfect, unless we use a lot of components. Autoregressive mixtures (DARMDN) may put the right amount of uncertainty on y2|y1, learning for example the noiseless relationship of cosθ and sinθ.

The “raw angles” system keeps the angles as the prediction targets, challenging the models at ±π where the angle trajectories jump and Acrobot behaves chaotically (small differences in the initial conditions lead to qualitatively different futures).