Learning to Observe with Reinforcement Learning

We consider a decision making problem where an autonomous agent decides on which actions to take based on the observations it collects from the environment . We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important ( such as position versus velocity ) and the dependence of this on the state of agent ( such as at the bottom versus top of a hill ) . We approach this problem by associating a cost with collecting observations which increases with the accuracy . We adopt a reinforcement learning ( RL ) framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task . We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels , such as taking a sample perfectly or not taking a sample at all . In contrast to the existing work that mostly focuses on sample efficiency during training , our focus is on the behaviour during the actual task . Our results illustrate that the RL agent can learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data . By uncovering the relative usefulness of different types of observations and trade-offs within , these results also provide insights for further design of active data acquisition schemes . 1 INTRODUCTION . Autonomous decision making relies on collecting data , i.e . observations , from the environment where the actions are decided based on the observations . We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important ( such as position versus velocity ) . Revealing this structure is challenging since the usefulness of the information that an observation can bring is a priori unknown and depends on the environment as well as the current knowledge state of the decision-maker , for instance , whether the agent is at the bottom versus the top of a hill and how sure the agent is about its position . Hence , we ’ re interested in questions such as “ Instead of collecting all available observations , is it possible to skip some observations and obtain satisfactory performance ? ” , “ Which observation components ( such as the position or the velocity ) are the most useful when the object is far away from ( or close to ) the target state ? ” . The primary aim of this work is to reveal this information structure of the observation space within a systematic framework . We approach this problem by associating a cost with collecting observations which increases with the accuracy . The agent can choose the accuracy level of its observations . Since cost increases with the accuracy , we expect that the agent will choose to collect only the observations which are most likely to be informative and worth the cost . We adopt a reinforcement learning ( RL ) framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task . We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels , such as taking a sample perfectly or not taking a sample at all . In contrast to the existing work that mostly focuses on sample efficiency during training , our focus is on the behaviour during the actual task . Our results illustrate that the RL agent can learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data . 2 RELATED WORK . A related setting is active learning ( Settles , 2010 ; Donmez et al. , 2010 ) where an agent decides which queries to perform , i.e. , which samples to take , during training . For instance , in an active learning set-up , an agent learning to classify images can decide which images from a large dataset it would like to have labels for in order to have improved classification performance . In a standard active learning approach ( Settles , 2010 ; Donmez et al. , 2010 ) as well as its extensions in RL ( Lopes et al. , 2009 ) , the main aim is to reduce the size of the training set , hence the agent tries to determine informative queries during training so that the performance during the test phase is optimal . In the test phase , the agent can not ask any questions ; instead , it will answer questions , for instance , it will be given images to label . In contrast , in our setting the agent continues to perform queries during the test phase , since it still needs to collect observations during the test phase , for instance as in the case of collecting camera images for an autonomous driving application . From this perspective , one of our main aims is to reduce the number of queries the agent performs during this actual operation as opposed to number of queries in its training phase . Another related line of work consists of the RL approaches that facilitate efficient exploration of state space , such as curiosity-driven RL and intrinsic motivation ( Pathak et al. , 2017 ; Bellemare et al. , 2016 ; Mohamed & Rezende , 2015 ; Still & Precup , 2012 ) or active-inference based methods utilizing free-energy ( Ueltzhöffer , 2018 ; Schwöbel et al. , 2018 ) ; and the works that focus on operation with limited data using a model ( Chua et al. , 2018 ; Deisenroth & Rasmussen , 2011 ; Henaff et al. , 2018 ; Gal et al. , 2016 ) . In these works , the focus is either finding informative samples ( Pathak et al. , 2017 ) or using a limited number of samples/trials as much as possible by making use of a forward dynamics model ( Boedecker et al. , 2014 ; Chua et al. , 2018 ; Deisenroth & Rasmussen , 2011 ; Henaff et al. , 2018 ; Gal et al. , 2016 ) during the agent ’ s training . In contrast to these approaches , we would like to decrease the effective size of the data or the number of samples taken during the test phase , i.e . operation of the agent after the training phase is over . Representation learning for control and RL constitutes another line of related work ( Watter et al. , 2015 ; Hafner et al. , 2019 ; Banijamali et al. , 2018 ) . In these works , the transformation of the observation space to a low-dimensional space is investigated so that action selection can be performed using this low-dimensional space . Similar to these works , our framework can be also interpreted as a transformation of the original observation space where an effectively low-dimensional space is sought after . Instead of allowing a general class of transformations on the observations , here we consider a constrained setting so that only specific operations are allowed , for instance , we allow dropping some of the samples but we do not allow collecting observations and then applying arbitrary transformations on them . Our work associates a cost with obtaining observations . Cost of data acquisition in the context of Markov decision processes ( MDPs ) has been considered in a number of works , both as a direct cost on the observations ( Hansen , 1997 ; Zubek & Dietterich , 2000 ; 2002 ) or as an indirect cost of information sharing in multiple agent settings ( Melo & Veloso , 2009 ; De Hauwere et al. , 2010 ) . Another related line of work is performed under the umbrella of configurable MDPs ( Metelli et al. , 2018 ; Silva et al. , 2019 ) where the agent can modify the dynamics of the environment . Although in our setting , it is the accuracy of the observations rather than the dynamics of the environment that the agent can modify , in some settings our work can be also interpreted as a configurable MDP . We further discuss this point in Section 4.2 . 3 PROPOSED FRAMEWORK AND THE SOLUTION APPROACH . 3.1 PRELIMINARIES . Consider a Markov decision process given by 〈S , A , P , R , Ps0 , γ〉 where S is the state space , A is the set of actions , P : S ×A× S → R denotes the transition probabilities , R : S ×A → R denotes the bounded reward function , Ps0 : S → R denotes the probability distribution over the initial state and γ ∈ ( 0 , 1 ] is the discount factor . The agent , i.e . the decision maker , observes the state of the system st at time t and decides on its action at based on its policy π ( s , a ) . The policy mapping of the agent π ( s , a ) : S × A → [ 0 , 1 ] is possibly stochastic and gives the probability of taking the action a at the state s. After the agent implements the action at , it receives a reward r ( st , at ) and the environment moves to the next state st+1 which is governed by P and depends on at and st . The aim of the RL agent is to learn an optimal policy mapping π ( s , a ) so that the expected return , i.e . expected cumulative discounted reward , J ( π ) = Eat∼π , st∼P [ ∑ t γ tr ( st , at ) ] is maximized . 3.2 PARTIAL OBSERVABILITY . Although most RL algorithms are typically expressed in terms of MDPs , in typical real-life applications the states are not directly observable , i.e. , the observations only provide partial , possibly inaccurate information . For instance , consider a vehicle which uses the noisy images with limited angle-of-view obtained from cameras mounted on the vehicle for autonomous-driving decisions . In such scenarios , the data used by the agent to make decisions is not a direct representation of the state of the world . Hence , we consider a partially observable Markov decision process ( POMDP ) where the above MDP is augmented by O and Po where O represents the set of observations and Po : S → O represents the observation probabilities . Accordingly , the policy mapping is now expressed as π ( o , a ) : O ×A → [ 0 , 1 ] . The observation vector at time t is given by ot = [ o1t ; . . . ; o n t ] ∈ Rn , where n is the dimension of the observation vector . The observations are governed by ot ∼ po ( ot|st ; βt ) ( 1 ) where po ( ot|st ; βt ) denotes the conditional probability distribution function ( pdf ) of ot given st and is parametrized by the accuracy vector βt = [ β 1 t ; . . . ; β n t ] ∈ Rn ( 2 ) The parameter βit ≥ 0 represents the average accuracy of the observation component i at time step t , i.e . oit . For instance , say we have two observations , position o 1 and velocity o2 . Then , β1t denotes the accuracy of the position and β2t denotes the accuracy of the velocity . As β i t increases , the accuracy of the observation oit decreases . Given st and βt , the observations are statistically independent , i.e . we have the factorization po ( ot|st ; βt ) = ∏ i=1 , ... , n poi ( o i t|st ; βit ) ( 3 ) where poi ( oit|st ; βit ) denotes the conditional pdf of oit given st and βit . Note that βit determines the average accuracy , i.e . the accuracy in the statistical sense . We provide an example below : Example : Consider the common Gaussian additive noise model with oit = s i t + v i t , i = 1 , . . . , n , ( 4 ) where st = [ s1t ; . . . ; s n t ] ∈ Rn is the state vector and vt = [ v1t ; . . . ; vnt ] ∈ Rn is the Gaussian noise vector with N ( 0 , diag ( σ2 vit ) ) . Here , vt and vt′ are statistically independent ( stat . ind . ) for all t 6= t′ and also vt and st′ are stat . ind . for all t , t′ . Under this observation model , a reasonable choice for βit is β i t = σ 2 vit . Hence , we parametrize pio ( . ) as p i o ( o i t|sit ; βit ) = N ( sit , βit = σ2vit ) . Note that the parametrization in terms of βit can be done in multiple ways , for instance , one may also adopt βit = σvit .

Summary: In contrast to standard reinforcement learning (RL), the paper investigates the variant where the observation made by the agent about its state has a cost. The authors propose to model the problem as a POMDP with an augmented action space (normal action + observation accuracy) and a new reward function that is defined as the original one penalized by the observation cost. They solve the problem with TRPO in three control domains: mountain car, pendulum, and cart pole.