Project Proposal : Exploiting Environment Configurability for Policy Space Identification

Reinforcement Learning (RL) [10] is the branch of Machine Learning focused on decision making. The theoretical framework used to represent the typical learning scenario is the Markov Decision Process (MDP) [8]. An MDP is a stochastic control process that allows to model situations in which an agent has access to a set of observable features, can choose what to do from a set of actions, and receives rewards in the measure of how well it is doing a certain task. The agent acts following a policy π(·|s) that specifies the probability distribution over the possible actions given that the agent is in state s. In the classical application, the goal is to find the optimal policy π∗, i.e. the one that gives the agent the greatest expected sum of future rewards. To find this policy, the agent can search inside a space of functions called policy space and indicated with ΠΘ. The policy space is defined by the set of observable features the agent has access to, that represent its perception of the environment and of itself. The size of this space is an index of the learning capabilities of the agent: the greater it is, the more complex behaviours it will be able to show. In fact a particular policy depends on a linear or nonlinear combination of these features specified by a vector of parameters θ ∈ Θ. In general the policy space of an agent may not be known, and its inference can be interesting in many situations. A common problem in the RL literature is Imitation Learning, the task of learning through the observation of the behaviour of an expert. This method allows a faster convergence to an optimal solution and can be tackled with two different methods: Behavioural Cloning (BC) and Inverse Reinforcement Learning (IRL). The first tries to directly reproduce the policy used in the observed interactions with the environment, while the latter aims at understanding the reward function used by the expert. Both these approaches however usually require a representation of the policy space of the observed agent to work properly, hence the importance of its identification. Another motivation to support the importance of this topic is related to the choice of the environment in which to put a learning agent. In fact, it is often possible to choose a specific environment in a learning experiment, and knowing the policy space of the agent offers the possibility of choosing the right level of difficulty for its specific capabilities. The problem of identifying the policy space of an agent is not trivial. Since this space depends on the features the agent has access to, a possible approach could consist in their detection. Having access to a greater set of features, we may find out which ones are necessary to explain the behaviour of the agent. However in the classical framework with a fixed environment, many observable state features might be useless to the agent in the definition of the optimal policy, and hence undistinguishable from the ones that are not actually perceived. To solve this issue we would need to put the agent into different environments, so to solicit the use of the features which we are uncertain about. Configurable Markov Decision Processes (Conf-MDPs) [3] are an extension of MDPs that add the possibility to configure the environment, and hence can be used to distinguish observable features from useless ones. For example, in the task of driving a car it is possible to change how the car reacts to the agent’s actions, such as increasing the maximum speed, e.g. to discover if an agent can perceive obstacles at higher speeds. We can imagine the presence of a supervisor modifying the probability distributions that define the model’s dynamics or