Learning to Observe with Reinforcement Learning
Authors: Mehmet Koseoglu
Ece Kunduracioglu
Ayca Ozcelikkale
Venue:  (2021)
Keywords: Reinforcement learning
observation strategies
active data collection

Abstract: We consider a decision making problem where an autonomous agent decides on which actions to take based on the observations it collects from the environment. We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important (such as position versus velocity) and the dependence of this on the state of agent (such as at the bottom versus top of a hill). We approach this problem by associating a cost with collecting observations which increases with the accuracy. We adopt a reinforcement learning (RL) framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task. We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels, such as taking a sample perfectly or not taking a sample at all.   In contrast to the existing work that mostly focuses on sample efficiency during training, our focus is on the behaviour during the actual task. Our results illustrate that the RL agent can  learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data. By uncovering the relative usefulness of different types of observations and trade-offs within, these results also provide insights for further design of active data acquisition schemes. 

Under review as a conference paper at ICLR 2021
LEARNING TO OBSERVE WITH REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We consider decision making problem where autonomous agent decides which actions to take based on the observations it collects from the environment. We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important (such as position versus velocity) and the dependence of this on the state of agent (such as the bottom versus top of hill). We approach this problem by associating a cost with collecting observations which increases with the accuracy. We adopt a reinforcement learning (RL) framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task, We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels, such as taking a sample perfectly or not taking sample at all. In contrast to the existing work that mostly focuses on sample efficiency during training, our focus is on the behaviour during the actual task. Our results illustrate that the RL agent can learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data. By uncovering the relative usefulness of different types of observations and trade-off' within, these results also provide insights for further design of active data acquisition schemes.
INTRODUCTION
Autonomous decision making relies on collecting data, i.e. observations, from the environment where the actions are decided based on the observations. We are interested in revealing the information structure of the observation space illustrating which type of observations are the most important (such position versus velocity). Revealing this structure is challenging since the usefulness of the information that an observation can bring is priori unknown and depends on the environment as well as the current knowledge state of the decision-man maker, for instance, whether the agent is at the bottom versus the top of hill and how sure the agent is about its position. Hence, we'r re interested in questions such as "Instead of collecting all available observations, is it possible observations and obtain satisfactory performance?" "Which observation components (such as the position or the velocity) are the most useful when the object is far away from (or close to) the target state?". The primary aim of this work is to reveal this information structure of the observation space within a systematic framework.
skip some
We approach this problem by associating a cost with collecting observations which increases with the accuracy. The agent can choose the accuracy level of its observations. Since cost increases with the accuracy, we expect that the agent will choose to collect only the observations which are most likely to be informative and worth the cost. We adopt reinforcement learning (RL. framework where the RL agent learns to adjust the accuracy of the observations alongside learning to perform the original task. We consider both the scenario where the accuracy can be adjusted continuously and also the scenario where the agent has to choose between given preset levels, such taking a sample perfectly or not taking a sample at all. In contrast to the existing work that mostly focuses on sample efficiency during training, our focus is on the behaviour during the actual task. Our results illustrate that the RL agent can learn to use the observation space efficiently and obtain satisfactory performance in the original task while collecting effectively smaller amount of data.

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
A related setting is active learning (Settles 2010 Donmez et al. 2010) where an agent decides which queries to perform, i.e., which samples take, during training. For instance, in an active learning set-up, an agent learning to classify images can decide which images from a large dataset it would like to have labels for order to have improved classification performance.
standard active learning approach (Settles) 2010 Donmez et al.) 2010) as well as its extensions in RL (Lopes et al., 2009), the main aim is to reduce the size of the training set, hence the agent tries to determine informative queries during training that the performance during the test phase is optimal. In the test phase, the agent cannot ask any questions; instead, it will answer questions, for instance, it will be given images to label. In contrast, in our setting the agent continues to perform queries during the test phase, since it still needs to collect observations during the test phase, for instance as in the case of collecting camera images for an autonomous driving application. From this perspective, one of our main aims is to reduce the number of queries the agent performs during this actual operation as opposed to number of queries in its training phase.
Another related line of work consists of the RL approaches that facilitate efficient exploration of state space, such as curiosity-driver RL and intrinsic motivation (Pathak et al., 2017 Bellemare et al. Mohamed & Rezende, 2015 Still& & Precup 2012) or active-inference based methods utilizing
free-energy (Ueltzhöffer 2018: Schwöbel et al. 2018); and the works that focus on operation with
limited data using model (Chua et al. 201 Deisenroth & Rasmussen] 201 1 Henaff al. 2018
Gal et al. 2016). In these works, the focus is either finding informative samples (Pathak et al., 2017)
or using limited number of samples/trials much possible by making use of a forward dynamics
model (Boedecker et al., 2014] Chua et al., 2018 Deisenroth & Rasmussen, 2011]] Henaff et al.,
2018 Gal et al.] (016) during the agent's training. In contrast these approaches, we would like to decrease the effective size of the data or the number of samples taken during the test phase, i.e.
operation of the agent after the training phase is over.
Representation learning for control and RL constitutes another line of related work (Watter et al. 2015 Hafner al. 2019 Banijamali et al., 2018). In these works, the transformation of the observation space to a low-dimensional space is investigated so that action selection can be performed using this low-dimensional space. Similar to these works, our framework can be also interpreted as a transformation of the original observation space where an effectively lww-dimesiional space is sought after. Instead of allowing general class of transformations on the observations, here we consider a constrained setting so that only specific operations are allowed, for instance, we allow dropping some of the samples but we do not allow collecting observations and then applying arbitrary transformations on them.
Our work associates a cost with obtaining observations. Cost of data acquisition in the context of Markov decision processes (MDPs) has been considered in number of works, both as a direct cost on the observations (Hansen 1997 Zubee & Dietterich] 2000; 2002) or as an indirect cost of information sharing in multiple agent settings (Melo & Veloso 2009 De Hauwere et al., 2010). Another related line of work is performed under the umbrella of configurable MDPs (Metelli et al. 2018 Silva et al.) 2019) where the agent can modify the dynamics of the environment. Although in our setting, it is the accuracy of the observations rather than the dynamics of the environment that the agent can modify, in some settings our work can be also interpreted configurable MDP. We further discuss this point in Section 4.2
3 PROPOSED FRAMEWORK AND THE SOLUTION APPROACH
3.1 PRELIMINARIES
Consider a Markov decision process given by (S,A, P..
Pao: 7) where S is the state space, A is the set of actions, P S x A x S + R denotes the transition probabilities, R S x A + R denotes the bounded reward function, P., S -> IR denotes the probability distribution over the initial state and y E (0, 1] is the discount factor.
The agent, i.e. the decision maker, observes the state of the system st at time and decides on its action a, based on its policy T(s, a). The policy mapping of the agent T(s, a) : S x A + [0, 1] is possibly stochastic and gives the probability of taking the action at the state s. After the agent
2

---

Under review as a conference paper at ICLR 2021
implements the action at, it receives reward r(st; 0e) and the environment moves the next state S1+1 which is governed by P and depends on a, and St. The aim of the RL agent is to learn an optimal policy mapping (Ts, a) that the expected return, Le. expected cumulative discounted reward, J(T) = Bai E 7'r(st, 4z)] is maximized.
3.2 PARTIAL OBSERVABILITY
Although most RL algorithms are typically expressed in terms of MDPs, in typical real-life appli- cations the states are not directly observable, i.e., the observations only provide partial, possibly inacourate information. For instance, consider vehicle which uses the noisy images with limited angle-of-view obtained from cameras mounted on the vehicle for autonomous-driving decisions. In such scenarios, the data used by the agent make decisions is not a direct representation of the state of the world. Hence, consider a partially observable Markov decision process (POMDP) where the above MDP is augmented by and Po where O represents the set of observations and Po S O represents the observation probabilities. Accordingly, the policy mapping is now expressed as π a). x A [0, 1].
The observation vector time is given by iol: R", where is the dimension of the observation vector. The observations are governed by
(o s(; 3e)
(1)
where ss Bt) denotes the conditional probability distribution function (pdf) of or given 8 and is parametrized by the accuracy vector
3, = [8] [7r
E R
The parameter 8 2 O represents the average accuracy of the observation component i at time step t, i.c of. For instance, say we have two observations, position of and velocity o². Then, 88 denotes the accuracy of the position and 3f denotes the accuracy of the velocity. As 3i increases, the accuracy of the observation of decreases. Given St and Be, the observations are statistically independent, i.e. we have the factorization
po(or(o15,8)
II Po (oilss
(3)
where Po (151ss))))) denotes the conditional pdf of of given 8, and 8
Note that 8 determines the average accuracy, i.e. the accuracy in the statistical sense. We provide an example below:
Example: Consider the common Gaussian additive noise model with
= r 1= I,...,
where = [s];..... : E IR" is the state vector and [u]:... ofll E R" is the Gaussian noise vector with A (O, diag(o"))) Here, e, and oy are statistically independent (stat. ind.) for + l' and also 2/ and Sy stat. ind. for all t, Under this observation model, reasonable choice for 3 is 5 = o² Hence, we parametrize pi(() as (' 8)) = Nss), St = o²,). Note that the parametrization in terms of 83 can be done multiple ways, for instance, one may also adopt 3 o
3.3 DECISION MAKER CHOOSES THE ACCURACY THE OBSERVATIONS
The agent can choose 83, hence 88; is decision variable. Observations have cost which increases with increasing accuracy, i.e. the cost increases with decreasing B
In Scenario A, the agent can vary 8f on continuous scale, i.e. 38 E co].
In Scenario B, the agent chooses between i) collecting all the observations with fixed level of accuracy or ii) not getting any of them at all. This setting corresponds to the case with 3 3,1, B, E {8r. 20), where 1 R² denotes the vector of ones. Here 3p 0 represents a fixed accuracy level. Note that 3p can be zero, corresponding to the case 0r St-
3

---

Under review as a conference paper at ICLR 2021
Remark 3.1 Our proposed setting can be interpreted as a constrained representation learning problem for RL. In particular, consider the problem of learning the best mapping h-)) with
h((ee)
(5)
from the high-dimensional original observations o some new possibly low-dimensional variables 2, so that control can be performed reliably on ž instead of o, Such settings have been utilized in various influential work, see for instance E2C approach of Watter al. 2015))
The proposed approach can be also formulated in a representation framework. In particular, we interpret the possibly noisy observations Ot the effectively low-dimensional representation b used in Hence, consider the mapping h(.)
O h())),
(6)
where Or and On denote the noisy and the original measurements, respectively. Compared to ( the family of the mappings allowed in (6) is constrained, i.e. one can only adjust the accuracy parameter instead of using arbitrary transformations from o, to Here, on is effectively low- dimensional compared o because i) noise decreases the dynamic range and allows effectively higher compression rates for the data (Scenario A); or ii) the total number of observations acquired is smaller (Scenario B). Note that not all transformations from 8 to oe can be written using (6) as an intermediate step. From this perspective, the formulation in [ can be said to be more general than (6).
3.3.1 MOTIVATION
The primary motivation behind the proposed framework is to reveal the inherent nature of the observation space in terms of usefulness of information the observations provide with respect the task at hand. The secondary motivation is to provide a RL. framework for solving decision making problems when the observations have cost associated with them.
In regard to the first task, we note the following: To reveal this information structure, we associate an artificial cost with the observations that increase with the accuracy. Hence, only the observation components (or the observation vectors) which are mostly likely to be informative and worth the cost will be collected. This decision heavily depends on the state that the agent believes itself to be in. For instance, in the case of balancing object at unstable state (such pendulum in OpenAi Gym (Brockman et al. 2016)), we intuitively expect that the agent does not need accurate measurements when it is far away from the target state. Hence, we're interested in questions such as "Is it possible to skip some observations and obtain satisfactory performance?". "Which observation components (such as the position or the velocity) are most useful when the object is far away from (or close to) the target state?", "How are these results affected by the possible discrepancy between the true state the agent is in and the one that it believes it to be in due to noisy skipped observations?". The proposed framework reveals this information structure within a systematic setting.
In regard the second task, note that there are many practical problems where there is cost associated with acquiring observations (Hansen 1997] Zubek & Dietterich] 2000 2002), for instance consider the expensive medical tests (i.e. observations) that have performed to diagnose a certain disease (Zubek& & Dietterich] 2002) and wireless communications where there is a cost associated with channel usage (i.e. the right to use communication channel) and power cost that increases with the reliability of communications (Goldsmith, 2005: Cover & Thomas 1991). see also Section A. The proposed framework can be used find efficient observation strategies in such problems and to quantify the possible performance degradation due to the observation cost.
The proposed scenarios and B also correspond to practical data acquisition schemes. We now give some examples: An example for Scenario A is the case where the observations are obtained using different sensors on the device where the accuracy of each sensor can be individually adjusted. Another example is the case where the sensors are distributed over the environment and the readings of the sensors has to be relayed to central decision unit using individual compression of each observation type and wireless communications. Here, the compression and the wireless communication introduces an accuracy-cos trade-off where the agent can choose to operate at different points of. Please see Section A example illustrating the accuracy-cost trade-off in
Examples:
4

---

Under review as a conference paper at ICLR 2021
wireless communications. An example for Scenario is the remote control of device, such drone, where all sensor readings of the device are compressed together and then sent to a decision unit. Since all readings compressed and transmitted together. decision of whether to transmit the whole observation vector or not has to be made, for instance due the limited power or wireless channel occupancy constraints.
3.4 REWARD SHAPING
Reward shaping is popular approach to direct RL agents towards desired goal. Here, we want the agent only move towards the original goal (which encouraged by the original reward r), we also want it to learn to control 3.. Hence, we propose reward shaping in the following form:
ir = Bt)
(7)
where r, is the original reward, F; is the new modified reward and f(rr; (e) is a monotonically non-decreasing function of r, and 8t, Vi. Hence, the agent not only tries maximize the average of the original reward but it also tries to maximize the "inaccuracy" of the measurements. This can be equivalently interpreted as minimizing the cost due accurate measurements. In the case where there is direct cost function c'(.) that increases with the accuracy of the observation o² (see, for instance, the example in Section A.I where transmission power can be interpreted as the direct cost), the following additive form can be used
F: -
(B)),
(8)
where ((()) is non-increasing funetion of 3 and À 2 is weighting parameter. Hence, the agent's 's aim is to maximize the original reward as well as minimize the cost of the observations.
EXPERIMENTS
4.1 SETTING
Observation Models: We consider the following environments from the OpenAl Gym (Brockman ct al. 2016):: MountanCCarCnntimuuus-00 Pendulum-vO,, CartPole-vl. In this section, we illustrate how the modified environment with noisy observations is obtained for MountainCarContinuuus The details and the parameter values for the other environments can be found in the Appendix AA. We also consider version of MountainCarContinoon vO with observations of the vertical position, which is presented in Section A.4
We first explain Scenario A, and then Scenario B. The original observations of the mountain environment are the position 2, and the velocity ă;. In our framework, the agent has access to noisy versions of these original observations
f. 24 Q₂ Ar (B)),
(9a)
3 = b4 + Q2 A+r
(9b)
where (z( ~ (b)) ~ u(-33 37 B?? and ( 3) denotes the uniform distri- bution over 1-3, 3). The noise variables are stat. ind., in particular (z, and (,) are stat. ind. from each other and also stat. ind. over time. Here, Qn and Q2 determine the ranges of the noise levels and they are set as the 0.1 times of the full range of the corresponding observation, i.e..
Qn 0.18 and Qa 0.014.
Our agent chooses 3 E [0, 1] in addition to the original action of the environment, i.e. the force a, that would be exerted on the car. The original reward of the environment per step is given by r. = 0.1 x ag. The reward shaped using an additive model
F = rr KA X
(10)
where n 2 and K.A 0 is chosen as 5 10 The original environment has also termination reward which the agent gets when the car passes the target position at 0.45, which is also provided to agent upon successful termination.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison of the average returns
ENVIRONMENT
ORIGINAL
A
MOUNTTIINCARCONTTIUOUSSSS
94
94
PENDULUM-VO
-152
-158
-170
CARTPOLE-VI
494
482
472
In Scenario B, at each time instant we either have no observation or we obtain the original observation vector, i.c. H.. = and 2; 2,. These cases correspond to 3 reward function is given as f. = rr + KB x ((a) where KB = 0.5; and g(3) = 1 for 3. 0. and O otherwise. In the implementation, we have mapped oc to 1, i.e. the decision variable is 3e E (0, 1}. hence 3. = corresponds to not obtaining a sample Scenario B.
and 3, = 0, respectively. The
RL algorithm: We adopt deep RL setting, combining reinforcement learning with deep learn- ing using the policy-based approach Trust Region Policy Optimization (TRPO) (Schulman et al.) 2015; Hill et al. 2018). The parameters are kept constant for all experiments and are provided in Appendix|A.. A.3] For Scenario A, at each time step, noisy observations obtained at that time step are fed to the algorithm the observations. For Scenario B, the last acquired observation is fed to the algorithm as the observation at that time step.
Plots: Unless otherwise stated, all results reported averages (such average cumulative rewards and average 83) using 1000 episodes. For the plots, observation space is mapped to grid with uniform intervals. Averages are taken with respect the number of visits each given range of the observation state. For example, for Scenario A the average of 8 when T. E [-0.1, --, +0.1 is shown as one average value at the center 0. For Scenario B. we report the sample skip frequency, i.e. the number of times the agent decided not to acquire new observation when the last observed state of the agent falls into given interval, such as the average sample skip frequency for ă E [-0.1, +0.1] is reported as one value at O. In all 2-D plots, the color pink indicates there was no visit to that observation state.
4.2 OVERVIEW
We benchmark our results against the performance of the agent that use the original observations, and trained using the same RL algorithm. The resulting average cumulative rewards in terms of r. are presented in Table We present the reward corresponding only to the original task so that we can evaluate the success of the agent in this task. These results illustrate that the agent learn to adjust the accuracy level and still obtain successful performance. For the Mountain car environment, all agents have the same average return and for the others, the agents working with the noisy/skipped observations have slightly weaker performance but still achieve the task of bringing/keeping the pendulum/pole in vertical position in a reasonable number of time steps.
At first sight, it may be surprising that the agent can learn to perform these tasks satisfactorily even if have not injected any memory to our algorithm, for instance when we only use the current noisy observations for Scenario A. On the other hand, note that in these environments the observations are either noisy versions of hidden states which govern the dynamics or they are closely related to them. From the point of the agent that treats the noisy observations as state this can be interpreted as a configurable MDP (Meellli et al. 2018 8 Silva et al. 2019) where the agent controls the noise the dynamics. Hence, the task of the agent can be interpreted as adjusting the noise level in the dynamics which does not necessarily require usage of memory in the decision maker.
We now focus the data collection strategies chosen by the agent for the mountain car and pendulum environments. The results for the other environments are provided in the appendix.
4.3 MOUNTAIN CAR
The chosen noise levels and the sample skip frequencies for the mountain car environment are presented in Figure [ Note that in Figure Ic we present the sample skip frequency with respect to the velocity and the position on the same plot, where the legend also gives the corresponding a-axis label. In the mountain car environment, the car starts randomly around position -0.5 and it has to

---

Under review as a conference paper at ICLR 2021
Carvalee
-50
042
Skip Seeewenc
No0ce (M00)
0.2
122
1200 8:00 (b) Scenario A, noise levels vs 2ec
-0.0 +20
(a) Scenario A, noise levels vs Ft
(c) Scenario B
Figure I: Mountain car, noise levels or sample skip frequency versus one observation type
Noise Level (Car Position)
Noise Level (Car Velocity)
Sample Skip Frequency
CerWiieetyy
0.40
2.40
4002
-000
-0/4
-054
-054
-05
-.49
125
126 376
131
4.58
125
100
231
400
(a) Scenario A, position noise
(b) Scenario A, velacity noise
(c) Scenario B
Figure 2: Mountain car, noise levels skip frequencies over the whole observation space
first go in the reverse direction (corresponding to negative velocity) to climb the hill located around position -1.25 in order to gain momentum and climb to hill at the right (corresponding to a positive velocity) and reach the target location 0.45 which is at the top of this hill. The results reflect some of the trade-off's this strategy:
Figure la shows that most noisy observations in position and velocity (Scenario A) are preferred around 0.5 (where the car position is initialized), and the most accurate samples are taken when the is around position- - 1.2. This is the position where the has make sure that it has reached to the top of the left hill so that it has enough momentum to climb the right hill. In the case of the dependence of the noise level the velocity, Figure Tbl shows that accurate samples when the velocity has high positive values. We note that this is not the only viable observation strategy and there are multiple observation strategies that give approximately the same average return in the original task. These can be explored using different Q and k values in our framework.
preferred
Figure Tc shows that approximately half of the samples are dropped in Scenario B regardless of the observation state, suggesting a high inherent sampling rate in the environment. This difference in the behaviour with the noisy and skipped observations illustrates the fundamental difference in these frameworks. In the case of noisy observations, the agent has to discover that the observations are uncertain and counteract this uncertainty. On the other hand, when taking perfect observations are possible, in the case of Scenario B, the agent (since mountain car environment has no inherent noise in its observations) and determine its exact state using the previous observed state and its action.
internalize the exact environment dynamics
we observe that in the case of noisy observations a larger part of observation space is visited, which is partly due the fact that the plots drawn according to the observations acquired by the agent and not the true states. Note that this does not affect the performance in the original task, illustrated in Table
Comparing Figure 2a 2b with Figure 2c

---

Under review as a conference paper at ICLR 2021
Anpuir Veloely
Somple Skip RecoMeTTO
Angle
(a) Scenario A, noise levels vs
(b) Scenario A, noise levels vs
(c) Sccnario B
Figure 3: Pendulum, noise levels or sample skip frequency versus one observation type
Noise Level (Angle)
Noise Level (Angular Velocity)
Sample Skip Frequency
Aro Wodhy
10
(a) Scenario A, position noise
(b) Scenario A, velocity noise
(c) Scenario B
Figure 4: Pendulum, noise levels skip frequencies over the whole observation space
4.4 PENDULUM
The results for the pendulum are presented in Figure 34 Here, the task is to keep the pendulum at vertical position, corresponding to an angle of Figure Baland and 4alureeaashhw that observations with low position (i.e. angle) noise (Scenario A) are preferred when the pendulum is close to the vertical position and has relatively small angular velocity. On the other hand, when the samples can be completely skipped (Scenario B), the agent skips large ratio of the samples in this region, as shown Figure B3cladd Figure[4c] Note that the agent spends most of the episode in this target region in the vertical position. Here, the agent prefers noiseless samples since noisy sample may cause the control policy to choose wild movement which might destabilize the pendulum. On the other hand, the agent may safely skip some of the samples the upright position the last sample is very close to current one because the angular velocity typically low.
5 DISCUSSION AND CONCLUSIONS
We have proposed framework for revealing the information structure of the observation space in systematic manner. We have adopted reinforcement learning approach which utilizes cost function which increases with the accuracy of the observations. Our results uncover the relative usefulness of different types observations and the trade-off's within; and provide insights for further design of active data acquisition schemes for autonomous decision making. Further discussion of our results and some research directions are as follows:
Our results illustrate that settings with the inaccurate observations and skipped observations should be treated differently since the type of uncertainty that the agent has to counteract in these settings are inherently different.
Strategies for processing of the noisy/skipped observations should be investigated. Questions such as the following arise: Should all the processing be off-ee the RI. agent
8

---

Under review as a conference paper at ICLR 2021
should the pre- pee-proesssing of observations be performed. similar to Kalman filtering in the case linear control under linear state space models (Ljung 1999)?", "How does the answer to the former question depend on the RL approach, the environment and the observation models?"
Our results suggest that inherent sampling rate of some of the standard RI. environments may be higher than needed (for instance, the Mountain Car environment where average one can skip one out of every two samples without affecting the performance), indicating yet-another reason why some of these environments are the state-of-art RL algorithms.
unchallenging for most of
We have provided quantification of the sensitivity of the agent's performance to noisy/skipped observations at different observation regions illustrating that this sensitivity can be quite different based on the observation region. Utilizing this information for sup- porting robust designs well preparing adversarial examples is interesting line of future research.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ershad Banijamali, Rui Shu, Mohammad Ghavamzadch, Hung Hai Bui, and Ali Ghodsi. Robust locally-linear controllable embedding. Inter. Conf. on Artificial Intelligence and Statistics, AISTATS, 84:1751-1759, 2018.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, pp. 1471-1477 2016.
J. Boedecker, J. T. Springenberg, J. Wülfing, and M. Riedmiller. Approximate real-time optimal control based on sparse Gaussian process models. In 2014 IEEE Symp. on Adaptive Dynamic Program. and Reinforcement Learning (ADPRL), 2014.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Adv. in Neural Information Processing Systems 31, pp. 4754 4765. 2018.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley, 1991.
Yann-Michaél De Hauwere, Peter Vranex, and Ann Nowé. Learning multi-agent state space rep- resentations. In Proc.of the 9th Inter. Conf. on Autonomous Agents and Multiagent Systems, pp- 715-722, 2010.
Mare Peter Deisenroth and Carl Edward Rasmussen. PILCO: Model-Based and Data-Efficient Approach to Policy Search. Proc. of the International Conference on Machine Learning. 201 1.
Pinar Donmez, Jaime G. Carbonell, and Jeff G. Schneider. A probabilistic framework to learn from multiple annotators with time-varying accuracy. In Proc. of the SIAM International Conference on Data Mining, pp. 826-80 837, 2010.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving PILCO with Bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop. ICML, 2016.
Andrea Goldsmith. Wireless Communications Cambridge University Press, 2005.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. Proceedings of Machine Learning Research, 97:2555-2556 2019.
E. Hansen. Markov decision processes with observation costs. Technical Report 97-01, University of Massachusetts at Amherst, 1997.
Mikael Henaff, William Whitney, and Yann LeCun. Model- Model-based based planning with discrete and continuous actions, 2018.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore,
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https:/ /tte:://gtttut githubt.hh.... com/
hi 1 a welbbeebbbsseee nes 2018.
Lennart Ljung. System Identification. Prentice-Has Hall, 1999.
Manuel Lopes, Francisco S. Melo, and Luis Montesano. Active learning for reward estimation in inverse reinforcement learning. In European Conference Machine Learning and Knowledge Discovery in Databases, ECML, pp. 31-46, 2009.
Francisco S. Melo and Manuela Veloso. Learning of coordination: Exploiting sparse interactions in multiagent systems. Proceedings of The 8th Inter. Conf. on Autonomous Agents and Multiagent Systems, pp. 773-780, 2009.
10

---

Under review as a conference paper at ICLR 2021
Alberto Maria Metelli, Mirco Mutti, and Marcello Restelli. Configurable Markov decision processes. Proc. of Machine Learning Research (PMLR), pp. 3491-3500, 2018.
Shakir Mohamed and Danilo J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. pp. 2125-2133, 2015.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proc. of the International Conference on Machine Learning, pp. 2778-2787, 2017.
John Schulman, Sergey Levine, Picter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proc. of the 32nd International Conference Machine Learning, pp- 1889-1897, 2015.
Sarah Schwöbel, Stefan Kiebel, and Dimitrije Markovic. Active Inference, Belief Propagation, and the Bethe Approximation. Neural Computation, 30(9):2530-256 September 2018.
Burr Settles. From theories to queries: Active learning in practice. Active Learning and Experi- mental Design workshop AISTATS, 2010.
Rui Silva, Gabriele Farina, Francisco S. Melo, and Manuela Veloso. A theoretical and algorithmic analysis of configurable mdps. Proc. of the Inter. Conf. on Automated Planning and Scheduling. 29:455-463 Jul. 2019.
Susanne Still and Doina Precup. An information-thhoreii approach to curiosity-driven reinforcement learning. Theory of Bioscience, 131(3):139-148, 2012.
Kai Ueltzhöffer. Deep Active Inference. Biological Cyhernetics, 112(6))547-573, December 2018.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, pp. 2746-2754, 2015.
Valentina Bayer Zubek and Thomas G. Dietterich. A POMDP approximation algorithm that antic- ipates the need to observe. In PRICAI 6th Pacific Rim Inter. Conf. on Artificial Intelligence, pp. 521-532, 2000.
Valentina Bayer Zubek and Thomas G. Dietterich. Pruning improves heuristic search for cost-sensitive learning. In Proc. of the 32nd Inter. Conf. on Machine Learning, pp. 19-26, 2002.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A. EXAMPLE: WIRELESS COMMUNICATIONS
We now provide motivating example to illustrate how observations can have a cost that is increasing with the accuracy and the decision maker can choose this accuracy level.
standard model for single terminal wireless communications is the additive white Gaussian noise (AWGN) channel (Goldsmith 2005; Cover & Thomas 1991)
3/t 2i + Ut
(11)
where xe represents the channel input (i.e. message at the transmitter at time t, yt represents the corresponding channel output (i.c. the observation at the receiver) and the white Gaussian random process V/ represents the channel noise. The capacity of this channel, i.c. the maximum number of information bits that can be sent, is determined by the signal-to- to-ise ratio (SNR), i.e. the average power in za divided by the average power in tt. In particular, the capacity is given by (Goldsmith 2005
Cover & Thomas 1991)
Ps P.
C= == (1 +
(12)
where Ps and P. are the average power levels of x2 and Ut, respectively. Hence, the capacity increases with Pz. On the other hand, one cannot use a very high value of Pa since broadcasting at high power levels is costly. In particular, Ps directly contributes to the actual power required by the transmitter. Note that Pa controls the accuracy of the observations. In particular, by dividing both sides by vPz. can be equivalently represented as
31 = 2z + Te
(13)
where Vr JP. yt. Ft T1 and 8e Vr. The average power of T, is and average
power of 02 Pa/Pz. The SNR, and hence, the channel capacity are the same in (T] and (33) and hence these representations directly determines the effective noise level. With Ue Gaussian, we have U/ ~ (0, Po). Hence, the conditional distribution of the observations ÿt is given by p (7 [.,) = (3,,, Pa/Px) where P./Pz can be chosen as St. Hence, as the accuracy of the observations increases (Pu/Pz decreases ), the cost of the observations (P₂) increases. In this context, several interesting questions that relates the accuracy of the observations and the power cost can be posed, for instance how to distribute a certain total power budget Ptotal over channels vt =t + rt with different intrinsic power levels Pr.-
equivalent for all relevant purposes. In particular, determining Ps
This example illustrates the basic premise of our problem setting in a practical scenario; a decision maker who can adjust the noise levels of the observations which has a cost associated with them. It also suggests that the constraints on the wireless communications constitute a general and potential hindrance in remote control applications. Consider device that makes the observations and takes actions but gets its commands (i.e. decisions about which actions to take) from another decision unit, such as the control of a robot or a drone by a remotely run RL algorithm which is controlling a large number of such units. Here, it is beneficial to consider policies that can work with inaccurate observations since sending accurate measurements are costly from power perspective, which will be particularly important for device with limited battery, such as a drone flying at remote location. Similarly, if the wireless communication channel cannot be used at all times, for instance, due to the limited bandwidth available, RL methods that can utilize the limited communication resources efficiently and optimize performance under such conditions are needed.
A.2 ENVIRONMENT PARAMETERS
In this section, we provide the parameters for all the environments in the experiments that are used directly from OpenAl Gym. We also consider a vertical position version of MountainCarContinuon v0, which is explained in Section A.4
Consider a generic environment with the observation variables of. where of denotes the ith observation variable time t. The ummited-accural observations of are obtained using
o, of + Q' (z))
(14)
12

---

Under review as a conference paper at ICLR 2021
Table 2: Environment parameters, reward weighting for different scenarios
ENVIRONMENT
RA, RB
MOUNTAINCARRCNNTNEOISSS
5 x 10 0.5
PENDULUM-VO
0.2.2 0.2, 0.04
CARTPOLE-VI
Table 3: Hyperparameters of the TRPO algorithm
PARAMETER
VALUE
GRADIENT DAMPENING FACTOR
2.35E-05
WEIGHT FOR ENTROPY
0.01118
GAMMA
0.98
GAE FACTOR
0.9
KULLBACK-LEIBLER LOSS THRESHOLD
0.000193
No. OF TIMESTEPS RUN BATCH
1024
No. ITERS. FOR LEARNING FOR VALUE FUNC STEPSIZE OF VALUE FUNC.
0.00428
where Ao; ~ ~
3t). We choose Q' 0.1 and Q² 0.2 for the Pendulum-vO. Q' 0.2 for the CartPole-vI, and Q* = 0.1 for the CoommainCarCooiiuuuuus the same with the ones provided in OpenAl Gym (Brockman The ordering of the observations al., (016). For instance, for MountainCarrConinuuuu v0, position and velocity correspond to of and of, respectively. Note that indices start with in OpenAl Gym whereas here we start with =
The reward function under Scenario A is given by
1 -
i r2. + TA
(15)
where r, is the original reward and KA O. For Scenario B, is given by Fr - r KB x (3,) where (B,)) = 1 for 3, = 0, and 0 otherwise. The associated & values for different environments
are presented Table
The scaling factor Q's for the noise levels and к values for the reward function are determined empirically by first fixing Q (as a percentage of the full range of the associated observation) and searching for values that provide satisfactory performance in the original task. Note that the rest of the values are determined by the specifications of the environments OpenAl Gym. The results depend on the values of Q and For instance, using larger к puts larger weight on the reward due to noise. Hence, the agent prioritizes the reward due to noise instead of the reward from the original environment and, for large enough values, the agent cannot learn perform the original task.
A.3 TRPO PARAMETERS
The same TRPO parameters are used in all experiments. These are provided in Table 3
A.4
MOUNTAIN CAR WITH OBSERVATIONS OF THE VERTICAL POSITION
To have better understanding of the effect partial observability, we have investigated the following modification MountainCarContiuuuu vertical position as the observation. Hence, the observations are given by
Instead of the horizontal position, the agent uses the
Tt = y/ + Qu x v/ (8!),
(16a)
J = i Q2 x A (B,))
(16b)
where the vertical position 3/. E [0.1, 1] is given by 8/+ = 0.45 sin (3.,,) + 0.55 (Brockman et al. 2016) and (3)) 4(-3)
3/ ( and ( -4(-1k -33 (82) Note that due sin(.) function,
13

---

Under review as a conference paper at ICLR 2021
& Ser
Notee
-341
-208
Car Ver Neciiall Postion
(a) Scenario A, noise levels VS Z1
(b) Scenario A, noise levels 3
Figure 5: Mountain car with vertical position observation, noise levels versus observation type
Noise Level (Car Vertical Posttion)
Nolse Velocity)
648
0.4
0.44
0.43
440
0.99
640
0442
-440
0.4
-444
6.6
+000
Poution
Vertical
(a) vertical position noise
(b) velocity noise
Figure 6: Mountain car with vertical position observation, noise levels over the observation space
for most the Ye values in the range [0.1, 1], there are two possible horizontal position (x²) values. Hence, this environment constitutes POMDP even without any observation noise. Similar to experiments with the original environment, Qy and Q are set as the 0.1 times of the full range of the corresponding observation, i.e., Q. 0.09 and Q 0.014. As before, the reward is calculated with (TO) with AA = 5 x 10-6
The average return due the original task is 93, hence the agent again learns to perform the original task successfully, see Table for comparison. The chosen noise levels are presented in Figure 6 Comparing these results with Figure 2 where the agent takes the horizontal position observation, we observe that the general trend of the velocity noise with respect to the velocity are the same in both settings, i.e. decreasing as the agent moves from the negative velocities positive velocities. Comparing Figure [ with Figure []] we observe that lower relative noise levels are preferred for the setting with the vertical location observations.
A.5 ADDITIONAL RESULTS -CART POLE
We now provide the results for the cart pole environment in Figure 10 which were not included in the main text due to page limitations. For the sake of brevity, the noise levels over observations pairs only provided for the position noise levels whereas averages provided for all observation types.
14

---

Under review as a conference paper at ICLR 2021
Boxee (eve)
Noce teve!
Nosse Level
042
(a) Scenario A, noise vs position
(b) Sccnario A, noise vs velocity
(c) Scenario A, noise vs angle
Figure 7: Cart pole, noise levels versus observation type
Calt Valelly
Polc Velecity
Sample Sulp freqgatcc
Pole Angubrrrrreoo Veldity
Cart Fostoon
Pal: Angle
(a) Scenario A, noise angular ve-(b) Scenario B, sampling frequency (c) Scenario B, sampling frequency vs angle and angular frequency
lncity
vs position and velocity
Figure 8: Cart pole, noise levels or sample skip frequency versus one observation type
Nolse Level (Cart Position)
Noise Level (Cart Position)
Noise Level (Cart Position)
0.0
-1.3
Cart
Cart
Can
Noise Level (Cart Position)
Noise Level (Cart Position)
Noise Level (Cart Position)
0.1
Polo www..aaauaooooooooooo Matoolt
0.0
2000
5.55
Vilccty
Cart
Angle
Figure 9: Cart Pole, Scenario A, noise levels for the position over the pairs of the observation variables
15

---

Under review as a conference paper at ICLR 2021
Sample Skip Frequency
Sample Skip Frequency
Sample Slip Frequency
4000
-.19
40.1
-037
Carl Poxtoon
Poution
PastiPotiion
Sample Skip Frequency
Sample Skip Frequency
Sample Skip Frequency
40
Pole Anrp WWcCET
0.06
-0.03
-0.1
4.15
4037
Wilsity
Wilccly
Angle
Figure 10: Cart Pole, Scenario B, sample skip frequencies over the pairs of the observation variables
16


Reviews:
[{"review_id": "pJorKl0ypul", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Overview = \n\nThe paper proposes a reinforcement learning algorithm that enables an agent to \"fine tune\" the quality/accuracy of its sensors to its current task. The paper considers a partially observable MDP setting where the agent, besides the control actions, is endowed with a set of \"tuning actions\" that control the noise in the perception of the different components of the state. Additional reward terms are introduced that discourage the use of \"tuning\". By enabling the agent to fine tune its perception to the current task, the paper seeks to also investigate the relative importance of different state features in terms of the task.\n\n= Positive points =\n\nThe paper is well written and the ideas clearly presented. The ideas seem vaguely related with recent work on \"tuning\" MDPs [a] and some older work on learning state representations in multiagent settings [b,c], where the agents are allowed to \"pay\" to have better models or perceptions. The paper proposes the use of similar ideas in a completely different context - to identify relevant information state information in POMDP settings. \n\n= Negative points =\n\nMy main criticism is concerned with the particular domains considered, which I believe are too structured to provide a clear understanding of the potential impact of the proposed approach.\n\n= Comments = \n\nI believe that the problem considered in the paper is interesting and follows some recent work on \"tuning\" MDPs (see ref[a] below). The approach explored is quite simple but that is not an inconvenient per se. My main criticism lies in the fact that -- in my understanding -- the domains selected are too structured to provide really interesting insights. \n\nIn particular, all domains considered are classical control problems with essentially deterministic dynamics and full observability. The approach in the paper injects artificial additive noise in the state as perceived by the agent (the paper only provides explicit information regarding the noise in the Mountain Car domain, but I'm assuming that is similar in the other domains). \n\nNow I may be missing something, but it seems to me that, from the agent's perspective, this is equivalent to adding noise to the dynamics of the environment, since the agent treats the observations as state. Therefore, from the agent's perspective, the practical effect of the \"sensor tuning\" is to actually attenuate the noise in the dynamics, which partly explains the results provided. This renders this work particularly close to those on MDP tuning referred above, and more discussion in this direction would be appreciated.  \n\nI think that the paper would greatly benefit from considering richer domains, either where partial observability is a central issue -- such as those from the POMDP literature -- or with richer perceptual inputs --- such as those from game domains.\n\n= References = \n\n[a] A. Metelli, M. Mutti, M. Restelli. \"Configurable Markov Decision Processes.\" Proc. 35th Int. Conf. Machine Learning, pp. 3491-3500, 2018.\n\n[b] F. Melo, M. Veloso. \"Learning of coordination: Exploiting sparse interactions in multiagent systems.\" Proc. 8th Int. Conf. Autonomous Agents and Multiagent Systems, pp. 773-780, 2009.\n\n[c] Y. De Hauwere, P. Vrancx, A. Nowé. \"Learning multi-agent state space representations.\" Proc. 9th Int. Conf. Autonomous Agents and Multiagent Systems, pp. 715-722, 2010.", "extra_scores": {}, "extra_text": {}}, {"review_id": "mWjPey7DLau", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn contrast to standard reinforcement learning (RL), the paper investigates the variant where the observation made by the agent about its state has a cost. The authors propose to model the problem as a POMDP with an augmented action space (normal action + observation accuracy) and a new reward function that is defined as the original one penalized by the observation cost. They solve the problem with TRPO in three control domains: mountain car, pendulum, and cart pole.\n\nPROS\n\nI find the research questions asked in the paper interesting. The proposed variation seems to be novel as far as I know. Besides, two scenarios are studied in the paper, which correspond to two extreme cases: continuous vs discrete accuracy.\n\n\nCONS\n\nThe conclusions of the experiments seems to depend on the specific values set notably in Equations (9a-b) and (10). I think a discussion is warranted about how they were chosen. Notably, can the same conclusions be drawn if those values are changed? \n\nI didn't find the information about the policy used in TRPO. Notably, how does it deal with the partial observability?\n\nThe paper should be proof-read.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ORSDnE6UQjv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at studying an optimized way of collecting samples from an environment, discarding the ones for which the accuracy of the observation is high. This way the agent focuses on collecting only the samples that improve the knowledge of the state space.\n\nThis paper could be presented better, as the motivations of the work and the description of the method lack clarity and effectiveness. First, the title is somewhat misleading, as we cannot say that the agent is \"learning to observe\", which can remind something more related to feature extraction in representation learning. Indeed, the agent is learning to explore states under a certain criterion, i.e. minimizing the accuracy of the observation, closely reminding all the literature about intrinsically motivated exploration, that in this paper is only cited in the related works. After all, it looks to me that this paper is exclusively proposing a form of intrinsic reward, but it fails to explain it thoroughly. In particular, only a small subsection, namely 3.4, is dedicated to this description, moreover referring to \"reward shaping\", which is not the same concept as intrinsic motivation. The experimental section is weak, as it only analyses two simple RL problems, more problematically not comparing with any method in the literature.\n\nPros\n------\n* The paper addresses an interesting problem that can potentially improve sample-efficiency in deep RL problems.\n\nCons\n-------\n* Poor description of the methodology, in particular explaining its connection with intrinsic motivation;\n* No deep RL problems considered;\n* No comparisons with methods in literature.\n\nI recommend the authors to substantially restructure the paper to include a better analysis of how their method compares with intrinsic motivation, include deep RL problems where the problem of exploration and accuracy of observations is more accentuated, and add comparisons with representative baselines, e.g. Pathak et al (2017), Bellemare et al (2016), etc..\n\nPost-rebuttal feedback\n-------------------------------\nI thank the authors for their reply.\n\n> In contrast, our paper focuses on the following question: “how can we reduce the number/accuracy of the samples the agent \ntakes during the test phase”? (Here, the test phase corresponds to the agent’s behaviour after the training is completed.)\n\nI agree with the authors that intrinsic motivation is different, and perhaps in my review I expressed this concern too strongly. So I thank the authors for their long and informative answer.\n\n> We believe that the reviewer refers to the problems where a possibly large multi-dimensional data such as images in games are used as input to the RL algorithm.\n\nExactly. Experiments on high-dimensional problems would make the contribution of this paper stronger, considering the rather limited  theoretical/methodological impact that it has now. I strongly suggest the authors to work in this direction, perhaps on robotic application if possible.\n\nAfter the rebuttal, I still argue for rejection, although I increase my score from 3 to 4.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1lnU4QKsol", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows how to incorporate an observation cost into RL control problems to assess the inherent value of information in different domains.\n\nI found the paper fun, and well written/edited.\n\nHowever, I don't see much of a scientific contribution here. The paper says its aim work is to reveal the information structure in the observation space within a systematic framework. So, it's essentially a kind of \"ML for scientific visualization\" paper. The ML novelty appears small---standard algorithms and test problems are used. The paper isn't really evaluated from a scientific visualization perspective, so it's not clear that it is over the bar from that perspective.\n\nThe light shed on some standard test problems (\"decisions aren't that impactful when the pole is almost balanced\", etc.) are nice, but not really impactful.\n\nDetailed comments:\n\nRelated work: I think it would be appropriate to cite Valentina Bayer's \"cost sensitive learning\" work. I think there's also a \"cost observable MDP\" model that is very related. The earlier work isn't able to solve these problems as well as the current paper, but the model is very related.\n\n\"towards to\" -> \"towards\"\n\n\"maximize average\" -> \"maximize the average\"?\n\nTable 1: Use right justification for easier visual comparison.\n\nI'm confused about the state used in the experiments. It's a POMDP, so was there a recurrent network used? Were multiple steps available in the state representation? How were the RL algorithms able to represent and learn the strategy?\n\n\"For Mountain car environment all\" -> \"For the mountain car environment, all\"\n\n\"following rise\" -> \"following arise\"\n\n\"the agents performance\" -> \"the agents' performance\"?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The setting and the problem addressed by this paper has been considered as important and interesting to tackle with reinforcement learning. Yet, the reviewers expressed several concerns about this paper. Especially, the lack of comparison to state-of-the-art methods and to the standard visualization methods was a shared concern. The empirical validation also appeared as not ambitious enough. Finally, the novelty in the field of machine learning was also questioned since the paper is mainly about applying existing algorithms to a known problem. 
Decision: Reject