Project Proposal : Transfer of generative models in reinforcement learning

Reinforcement learning (RL) is the area of machine learning that studies sequential decision-making problems, in which an agent acts in an environment with the aim of maximizing a cumulative reward. Training an agent to learn a policy to choose actions optimally in every state of the environment is often quite expensive, slow or dangerous. Furthermore, many RL approaches struggle to transfer knowledge across similar environments or tasks. The application to RL of generative modeling, the area of machine learning that addresses the problem of estimating the generating distribution for a given set of data, has shown to be able to improve both the efficiency of learning, drastically reducing the need for difficult interactions with the environment, and the generalization of policies among related tasks. A learning agent can leverage generative models in various ways, but often they are used in estimating environmental dynamics, either by modeling the distribution of entire state-action trajectories or single-step state transitions. This class of approaches goes under the name of model-based reinforcement learning. Model-based RL is a powerful paradigm: if an agent is able to predict the effect that actions will have on the environment, it can use this knowledge to plan or learn faster when presented with new problems. In several scenarios, it is possible to collect interactions either from more than one agent acting in the environment according to its own policy, or from multiple related Markov Decision Processes (MDPs), which formalize reinforcement learning problems [13]. In both cases, a model-based agent could take advantage of these different experiences and use the additional information they provide about the environment during its learning process, approximating its dynamics. In fact, with this amount of information available, training a generative model for the sole estimation of trajectories generated by a single target policy would be a waste of precious information; nonetheless, a generative model that is trained to approximate the environmental dynamics for all available policies at the same time struggles to learn the complex, inherently multimodal distribution that fits all the differences in policies. Similarly, a single, monolithic model would be not able to estimate plausible trajectories for a whole family of related MDPs, but still knowledge of the dynamics of different MDPs can be exploited for better modeling in a related setting, with the ultimate goal of improving the performance of an agent. An example scenario is that of autonomous driving, where an agent is asked to drive a vehicle, receiving input signals from its sensors and controlling its actuators. It is easy to imagine that there could be multiple and diverse sources of driving experience, such as different drivers, different vehicles, different driving conditions. Moreover, you could use simulations to obtain transitions that are not very likely in the real world (e.g., car crashes). A learning agent that uses a generative model for estimating the environment dynamics would waste all this experience if it only learns from its own interactions. Being data collection for autonomous driving expensive and time-consuming, a dynamic model able to understand how to reuse the trajectories generated by other policies or in other conditions to estimate the ones of a target policy or condition would be extremely beneficial. Other application settings that feature great variability are related to the medical domain, in which RL methods can be used to help in controlling medical equipment. There can be significant variation of conditions among different patients or even for the same individual, due either to patient’s state or practical issues. For instance, in functional electrical stimulation (FES) [9], a medical technology used for rehabilitation of individuals and to address problematic neuromuscular conditions, there is uncertainty about patients and equipment at the start of each session. Nonetheless, an agent must be able to learn the optimal behavior quickly, given a very short duration for the session itself, and flexible generative models can offer an effective performance boost.