Learning to Observe with Reinforcement Learning

We consider a decision making problem where an autonomous agent decides on which actions to take based on the observations it collects from the environment . We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important ( such as position versus velocity ) and the dependence of this on the state of agent ( such as at the bottom versus top of a hill ) . We approach this problem by associating a cost with collecting observations which increases with the accuracy . We adopt a reinforcement learning ( RL ) framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task . We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels , such as taking a sample perfectly or not taking a sample at all . In contrast to the existing work that mostly focuses on sample efficiency during training , our focus is on the behaviour during the actual task . Our results illustrate that the RL agent can learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data . By uncovering the relative usefulness of different types of observations and trade-offs within , these results also provide insights for further design of active data acquisition schemes . 1 INTRODUCTION . Autonomous decision making relies on collecting data , i.e . observations , from the environment where the actions are decided based on the observations . We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important ( such as position versus velocity ) . Revealing this structure is challenging since the usefulness of the information that an observation can bring is a priori unknown and depends on the environment as well as the current knowledge state of the decision-maker , for instance , whether the agent is at the bottom versus the top of a hill and how sure the agent is about its position . Hence , we ’ re interested in questions such as “ Instead of collecting all available observations , is it possible to skip some observations and obtain satisfactory performance ? ” , “ Which observation components ( such as the position or the velocity ) are the most useful when the object is far away from ( or close to ) the target state ? ” . The primary aim of this work is to reveal this information structure of the observation space within a systematic framework . We approach this problem by associating a cost with collecting observations which increases with the accuracy . The agent can choose the accuracy level of its observations . Since cost increases with the accuracy , we expect that the agent will choose to collect only the observations which are most likely to be informative and worth the cost . We adopt a reinforcement learning ( RL ) framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task . We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels , such as taking a sample perfectly or not taking a sample at all . In contrast to the existing work that mostly focuses on sample efficiency during training , our focus is on the behaviour during the actual task . Our results illustrate that the RL agent can learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data . 2 RELATED WORK . A related setting is active learning ( Settles , 2010 ; Donmez et al. , 2010 ) where an agent decides which queries to perform , i.e. , which samples to take , during training . For instance , in an active learning set-up , an agent learning to classify images can decide which images from a large dataset it would like to have labels for in order to have improved classification performance . In a standard active learning approach ( Settles , 2010 ; Donmez et al. , 2010 ) as well as its extensions in RL ( Lopes et al. , 2009 ) , the main aim is to reduce the size of the training set , hence the agent tries to determine informative queries during training so that the performance during the test phase is optimal . In the test phase , the agent can not ask any questions ; instead , it will answer questions , for instance , it will be given images to label . In contrast , in our setting the agent continues to perform queries during the test phase , since it still needs to collect observations during the test phase , for instance as in the case of collecting camera images for an autonomous driving application . From this perspective , one of our main aims is to reduce the number of queries the agent performs during this actual operation as opposed to number of queries in its training phase . Another related line of work consists of the RL approaches that facilitate efficient exploration of state space , such as curiosity-driven RL and intrinsic motivation ( Pathak et al. , 2017 ; Bellemare et al. , 2016 ; Mohamed & Rezende , 2015 ; Still & Precup , 2012 ) or active-inference based methods utilizing free-energy ( Ueltzhöffer , 2018 ; Schwöbel et al. , 2018 ) ; and the works that focus on operation with limited data using a model ( Chua et al. , 2018 ; Deisenroth & Rasmussen , 2011 ; Henaff et al. , 2018 ; Gal et al. , 2016 ) . In these works , the focus is either finding informative samples ( Pathak et al. , 2017 ) or using a limited number of samples/trials as much as possible by making use of a forward dynamics model ( Boedecker et al. , 2014 ; Chua et al. , 2018 ; Deisenroth & Rasmussen , 2011 ; Henaff et al. , 2018 ; Gal et al. , 2016 ) during the agent ’ s training . In contrast to these approaches , we would like to decrease the effective size of the data or the number of samples taken during the test phase , i.e . operation of the agent after the training phase is over . Representation learning for control and RL constitutes another line of related work ( Watter et al. , 2015 ; Hafner et al. , 2019 ; Banijamali et al. , 2018 ) . In these works , the transformation of the observation space to a low-dimensional space is investigated so that action selection can be performed using this low-dimensional space . Similar to these works , our framework can be also interpreted as a transformation of the original observation space where an effectively low-dimensional space is sought after . Instead of allowing a general class of transformations on the observations , here we consider a constrained setting so that only specific operations are allowed , for instance , we allow dropping some of the samples but we do not allow collecting observations and then applying arbitrary transformations on them . Our work associates a cost with obtaining observations . Cost of data acquisition in the context of Markov decision processes ( MDPs ) has been considered in a number of works , both as a direct cost on the observations ( Hansen , 1997 ; Zubek & Dietterich , 2000 ; 2002 ) or as an indirect cost of information sharing in multiple agent settings ( Melo & Veloso , 2009 ; De Hauwere et al. , 2010 ) . Another related line of work is performed under the umbrella of configurable MDPs ( Metelli et al. , 2018 ; Silva et al. , 2019 ) where the agent can modify the dynamics of the environment . Although in our setting , it is the accuracy of the observations rather than the dynamics of the environment that the agent can modify , in some settings our work can be also interpreted as a configurable MDP . We further discuss this point in Section 4.2 . 3 PROPOSED FRAMEWORK AND THE SOLUTION APPROACH . 3.1 PRELIMINARIES . Consider a Markov decision process given by 〈S , A , P , R , Ps0 , γ〉 where S is the state space , A is the set of actions , P : S ×A× S → R denotes the transition probabilities , R : S ×A → R denotes the bounded reward function , Ps0 : S → R denotes the probability distribution over the initial state and γ ∈ ( 0 , 1 ] is the discount factor . The agent , i.e . the decision maker , observes the state of the system st at time t and decides on its action at based on its policy π ( s , a ) . The policy mapping of the agent π ( s , a ) : S × A → [ 0 , 1 ] is possibly stochastic and gives the probability of taking the action a at the state s. After the agent implements the action at , it receives a reward r ( st , at ) and the environment moves to the next state st+1 which is governed by P and depends on at and st . The aim of the RL agent is to learn an optimal policy mapping π ( s , a ) so that the expected return , i.e . expected cumulative discounted reward , J ( π ) = Eat∼π , st∼P [ ∑ t γ tr ( st , at ) ] is maximized . 3.2 PARTIAL OBSERVABILITY . Although most RL algorithms are typically expressed in terms of MDPs , in typical real-life applications the states are not directly observable , i.e. , the observations only provide partial , possibly inaccurate information . For instance , consider a vehicle which uses the noisy images with limited angle-of-view obtained from cameras mounted on the vehicle for autonomous-driving decisions . In such scenarios , the data used by the agent to make decisions is not a direct representation of the state of the world . Hence , we consider a partially observable Markov decision process ( POMDP ) where the above MDP is augmented by O and Po where O represents the set of observations and Po : S → O represents the observation probabilities . Accordingly , the policy mapping is now expressed as π ( o , a ) : O ×A → [ 0 , 1 ] . The observation vector at time t is given by ot = [ o1t ; . . . ; o n t ] ∈ Rn , where n is the dimension of the observation vector . The observations are governed by ot ∼ po ( ot|st ; βt ) ( 1 ) where po ( ot|st ; βt ) denotes the conditional probability distribution function ( pdf ) of ot given st and is parametrized by the accuracy vector βt = [ β 1 t ; . . . ; β n t ] ∈ Rn ( 2 ) The parameter βit ≥ 0 represents the average accuracy of the observation component i at time step t , i.e . oit . For instance , say we have two observations , position o 1 and velocity o2 . Then , β1t denotes the accuracy of the position and β2t denotes the accuracy of the velocity . As β i t increases , the accuracy of the observation oit decreases . Given st and βt , the observations are statistically independent , i.e . we have the factorization po ( ot|st ; βt ) = ∏ i=1 , ... , n poi ( o i t|st ; βit ) ( 3 ) where poi ( oit|st ; βit ) denotes the conditional pdf of oit given st and βit . Note that βit determines the average accuracy , i.e . the accuracy in the statistical sense . We provide an example below : Example : Consider the common Gaussian additive noise model with oit = s i t + v i t , i = 1 , . . . , n , ( 4 ) where st = [ s1t ; . . . ; s n t ] ∈ Rn is the state vector and vt = [ v1t ; . . . ; vnt ] ∈ Rn is the Gaussian noise vector with N ( 0 , diag ( σ2 vit ) ) . Here , vt and vt′ are statistically independent ( stat . ind . ) for all t 6= t′ and also vt and st′ are stat . ind . for all t , t′ . Under this observation model , a reasonable choice for βit is β i t = σ 2 vit . Hence , we parametrize pio ( . ) as p i o ( o i t|sit ; βit ) = N ( sit , βit = σ2vit ) . Note that the parametrization in terms of βit can be done in multiple ways , for instance , one may also adopt βit = σvit .

Summary: The paper proposes a reinforcement learning algorithm that enables an agent to "fine tune" the quality/accuracy of its sensors to its current task. The paper considers a partially observable MDP setting where the agent, besides the control actions, is endowed with a set of "tuning actions" that control the noise in the perception of the different components of the state. Additional reward terms are introduced that discourage the use of "tuning". By enabling the agent to fine tune its perception to the current task, the paper seeks to also investigate the relative importance of different state features in terms of the task.