Active Vision Reinforcement Learning under Limited Visual Observability
Authors: Jinghuan Shang
Michael S Ryoo
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Active Reinforcement Learning
Visual Reinforcement Learning
Active Vision
Active Perception
Partial Observability
Sensorimotor
TL;DR: Learning how to choose useful visual observations in Active-RL task by sensorimotor reward.

Abstract: In this work, we investigate Active Vision Reinforcement Learning (ActiveVision-RL), where an embodied agent simultaneously learns action policy for the task while also controlling its visual observations in partially observable environments. We denote the former as motor policy and the latter as sensory policy. For example, humans solve real world tasks by hand manipulation (motor policy) together with eye movements (sensory policy). ActiveVision-RL poses challenges on coordinating two policies given their mutual influence. We propose SUGARL, Sensorimotor Understanding Guided Active Reinforcement Learning, a framework that models motor and sensory policies separately, but jointly learns them using with an intrinsic sensorimotor reward. This learnable reward is assigned by sensorimotor reward module, incentivizes the sensory policy to select observations that are optimal to infer its own motor action, inspired by the sensorimotor stage of humans. Through a series of experiments, we show the effectiveness of our method across a range of observability conditions and its adaptability to existed RL algorithms. The sensory policies learned through our method are observed to exhibit effective active vision strategies.

Active Vision Reinforcement Learning under Limited Visual Observability
Jinghuan Shang Michael Ryoo
Department of Computer Science, Stony Brook University [jishang, mryoo) wtw.oo/bo.stttoooooooooo stonybrook.edu
Abstract
In this work, we investigate Active Vision Reinforcement Learning (AstiveVision- RL), where an embodied agent simultaneously learns action policy for the task while also controlling its visual observations in partially observable environments. We denote the former as motor policy and the latter as sensory policy. For example, humans solve real world tasks by hand manipulation (motor policy) together with eye movements (sensory policy). coordinating two policies given their mutual influence. We propose SUGARL, Sensorimotor Understanding Guided Active Reinforcement Learning, framework that models motor and sensory policies separately. but jointly learns them using with an intrinsic sensorimotor reward. This learnable reward is assigned by sensorimotor reward module, incentivizes the sensory policy to select observations that are optimal to infer its own motor action, inspired by the sensorimotor stage of humans. Through series of experiments, we show the effectiveness of our method across range of observability conditions and its adaptability to existed RL algorithms. The sensory policies learned through our method are observed to exhibit effective active vision strategies.
poses challenges on
Introduction
Although Reinforcement Learning (RL) has demonstrated success across challenging tasks and games in both simulated and real environments 2 I 89 97]. the observation spaces for visual RL tasks are typically predefined to offer the most advantageous views based on prior knowledge and can not be actively adjusted by the agent itself. For instance, table-top robot manipulators often utilize fixed overhead camera view [ST]. While such fixed viewpoints can potentially stabilize the training of an image feature encoder [14), this form of perception is different from humans who actively adjust their perception system to finish the task, c.g. eye movements (24). The absence of active visual perception poses challenges learning in highly dynamic environments (59) 61], open- world tasks (30] and partially observable environments with occlusions, limited field-of-views, multiple view angles
83
We study Active Reinforcement Learning (Active-RL) 102 ] the RL process that allows the embodied
agent actively acquire new perceptual information in contrast the standard RL, where the new
information could be reward signals [3] 23 forms. Specifically, controls its own views of visual observation, in an environment with limited visual observability (33). Therefore, the goal of Active Vision-RL is to learn two policies that still maximize the task return: the motor policy to finish the task and the sensory policy to control the observation.
27 50
56 62.. visual observations 332 33
and other
are interested in visual Active-RL. RL tasks, i.e. Active Vision-RI. that an agent
Our project page, code, library are available at this link
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

S2
81+1
St
a
a+
O++
a
a
Standard RI
ActiveVision-Be
ActiveVision-RI Example on Atari-Boxing
Figure I: Active with limited visual observability in comparison with standard RL, with exemplary process in Atari game Boxing. Red arrows stand for additional relationships considered in Actíve Vision-RL. In the example on the right, the highlighted regions are the actual observations visible to the agent at each step. The rest of the pixels are not visible the agent.
Active VisiveVisiio-RU tasks present a considerable challenge due to the coordination between motor and sensory policies, given their mutual influence [102]. The motor policy requires clear visual observation for decision-making, while the sensory policy should adapt accordingly to the motor action. Depending on the sensory policy, transitioning to a new view could either aid or hinder the motor policy learning [1] 40 [02). One notable impediment is the perceptual aliasing mentioned by Whitehead and Ballard [102]. An optimal strategy for sensory policy should be incorporating crucial visual information while eliminating any distractions. In the real world, humans disentangle their sensory actions, such as eye movements, from their motor actions, such as manipulation, and subsequently learn to coordinate them [59] [1]. Despite being modeled separately, these two action policies and the coordination can be learned jointly through the interaction during sensorimotor [03].
stage 25
Taking inspiration from human capabilities, we propose SUGARL: Sensorimotor Understanding Guided Active Reinforcement Learning, an Active-RI. framework designed to jointly learn sensory and motor policies by maximizing extra intrinsic sensorimotor reward together with environmental reward. We model the ActiveVision agent with separate sensory and motor policies by extending existing RL algorithms with two policy/value branches. Inspired by sensorimotor stage 77 72 [03].
we use the intrinsic sensorimotor reward to guide the joint learning of two policies, imposing penalties on the agent for selecting sub-optimal observations. We use a learned sensorimotor reward module to assign the intrinsic reward. The module is trained using inverse dynamics prediction task [52 (5]. with the same experiences as policy learning without additional data or pre- pre-training.
In our experiments, we use modified Atari [ and DeepMind Control suite (DMC) 197 with limited observability to comprehensively evaluate our proposed method. We also test on Robosuite tasks to demonstrate the effectiveness of active agent in 3D manipulation. Through the challenging benchmarks, we experimentally show that SUGARL an effective and generic approach for Active- RL with minimum modification on top of existed RL algorithms. The learned sensory policy also exhibit active vision skills by analogy with humans' fixation and tracking.
2 Active Vision Reinforcement Learning Settings
Consider vanilla RL setting based Markov Decision Process (MDP) described by (8, A,,, P, where S is the state space, A is the action space, is the reward function, P describes state transition which is unknown and is the discount factor. In this work, we study Active visual observability, described by (S, A,A", , r, P,7), as shown in Figure [ O is the actual partial observation space the agent perceives. In particular, observation is an image contains partial information of an environmental state like an image crop in 2D space are two action spaces for the agent in Active-RL formulation. AA' is the motor action space that causes state change p(s' uss|ss,a a²). AA" the sensory action space that only changes the observation of an environmental state pl p(o|s, a²). ). In this setting, the agent needs to take (a" a") for each step, based on observation(s) only. An example is shown Figure
under limited
are interested in visual tasks, so each
photo from a viewpoint in 3D space. To emulate the human ability, there
Our goal is to learn the motor and sensory action policies (TT", T²) that still maximize the return Σn. Note that the agent is never exposed the full environmental state s. Both policies completely based on the partial observations: a (+ a = (-lo). Therefore the overall policy learning challenging due to the limited information per step and the non-stationary observations.

---

Policy
Sensory
Sensorimstor Reward Module
at
Q
-a,
Ot
Oz
O
U
a
sugarl
Chit
0
ag
a
-
O++-
Learaing eRV p
Policy Learning
Senserimotor Reward Moduul Learning
Policy
Env
Policies
cnV
sugarl
Env
0,
U
Loss
Oc+1
Original Algorithm
SUGARL
Figure 2: Overview of SUGARL and the comparison with original RL algorithm formulation. SUGARL introduces an extra sensory policy head, and jointly learns two policies together with the extra sensorimotor reward. We use the formulation of SAC [36) as an example. We introduce sensorimotor reward module to assign the reward. The reward indicates the quality of the sensory policy through the prediction task. The sensorimotor reward module is trained independently to the policies by action prediction error.
3
SUGARL: Sensorimotor Understanding Guided Active-RL
3.1 Active-RL Algorithm with Sensorimotor Understanding
We implement Active-RL algorithms based on the normal vision-based RL algorithms with simple modifications regarding separated motor and sensory policies (T* =²), and the sensorimotor reward pugar We use DQN [65] and SAC (36] examples to show the modifications are generally applicable. The example diagram of SAC is shown in Figure 2 describe the sensorimotor reward in Section
We first introduce the policy then
Network Architecture The architectural modification is branching an additional head for sensory policy, and both policies share visual encoder stem. For DQN (5). two heads output Q" Q" for each policy respectively. This allows the algorithm to select a = arg max.. Q" and a = arg max.. Qo for each step. Similarly for SAC (36), the value and actor networks also have two separate heads for motor and sensory policies. The example in the form of SAC is in Figure
Joint Learning of Motor and Sensory Policies Though two types of actions are individually selected or sampled from network outputs, we find that the joint learning of two policies benefits the whole learning [II. The joint learning here means both policies trained using shared reward function. Otherwise, the sensory policy usually fails to learn with intrinsic reward signal only. Below give the formal losses for DQN and SAC.
For DQN, we take the sum of Q-values from two policies Q = Q" + Q" and train both heads jointly. The loss is the following where we indicate our modifications by blue:
2 (0,)
(u=- - (Qē, (o,, af) + Qù, (o,, (o,,,a)))
a(41) + max Q8,_, (o af ). a
y: Eo..1
ri + Br sugart
+ max ail1
where - is the loss for Q-networks, 0 stands for the parameters of both heads and the encoder stem, i is the iteration, and D is the replay buffer. 3r.uuganl is the extra sensorimotor reward with balancing scale which will be described in Section 3.2
For SAC, we do the joint learning similarly. The soft-Q loss is in the similar form of above DQN which is omitted for simplicity. The soft-value loss LV and is
(1)) = E.,DD
2 (V)(o)) + Vo((o)
Bapragaapmrg [Q a;) + (% a²) - log (af(oo)) - log 7g (3 27
and the actor loss _
(T) = Eo,~D [log (a]lo)))) + log (a) (2.
(lor.a)) a;) a?)]
3

---

where sb, & are parameters for the critic and the actor respectively, and reparameterization is omitted for clearness.
3.2 Sensorimotor Reward
The motor and sensory policies are jointly trained using a shared reward function, which is the combination of environmental reward and our sensorimotor reward. We first introduce the assignment of sensorimotor reward and then describe the combination two rewards.
The sensorimotor reward is assigned by the sensorimotor reward module wg(.). The module is trained to have the sensorimotor understanding, and is used to indicate the goodness of the sensory policy, tacking inspiration of human sensorimotor learning (72). The way we obtain such reward module is similar to learning an inverse dynamics model [95]. Given a transition (O,, a Oc+1), the module predicts the motor action a only, based on observation transition tuple (0,, Oz+1). When the module is (nearly) fully trained, the higher prediction error indicates the worse quality of visual observations. For example, if the agent is absent from observations, it is hard to infer the motor action. Such sub-optimal observations also confuse agent's 's motor policy, Since the sensory policy selects those visual observations, the quality of visual observations is tied the sensory policy. As result, we can employ the negative error of action prediction as the sensorimotor reward:
T'
sugarl
(1-p(aj" or, 01+1; u²))
p(a"
(1)
This non- possitsie intrinsic reward penalizes the sensory policy for selecting sub-optimal views that do contribute the accuracy of action prediction and confuse motor policy learning, In Section 5.4 we show that naive positive rewarding does not guide the policy well. Note that the reward is less noisy when the module is fully trained. However, it harmful for being noisy at the early stage of training, as the noisy signal may encourage the exploration of policies. We use the sensorimotor reward though the whole learning.
The sensorimotor reward module is implemented by an independent neural network. The loss is a simple prediction error:
( Ea, di 1--] [Error (a ug(on, (O,, 0z+1))]
where the Error(-) can be cross-entropy loss for discrete action space or L2 loss for continuous action space. Though being implemented and optimized separately, the sensorimotor reward module uses the same experience data as policy learning, with extra data prior knowledge introduced.
Combining Sensorimotor Reward and Balancing The sensorimotor reward yuugarl is added densely on per-step basis, on top of the environmental reward eny in balanced form:
Sugarl
It r
where 3 is the balance parameter varies across environments. The reward balance is very important to make both motor and sensory policies work as expected, without heavy bias towards one side (28), which will be discussed in our ablation study in Section 5.4 Following the studies in learning
with intrinsic rewards and rewards of many magnitudes [20] 38
75
96 99], we empirically set
t-1 r:th [T]] which is the average environmental return normalized by the length of the trajectory. We get these referenced return data from the baseline agents trained on normal, fully observable environments, or from the maximum possible environmental return of one episode.
3= E. E
3.3 Persistence-of- -fiiiio Memory
To address Active Vision-RL more effectively, we introduce a Persistence-of -esistencc-oo--iiionn Memory (PVM) to spatio-temporally combine multiple recent partial observations into one, mimicking the nature
Full
Partial
CNN
UNS
CNN
CNN
Partial
LSTM
PVM
Obs.
Stltching PVM
Neural Representation
Figure 3: Examples for different instantiations of PVM with B = 3.

---

Diaa
Foe New
the
Troichol Vee
Trem
Wet Vie
Pyoie Miee
Agon Vev
Tyn
Slock
Frans Mape
Alc Voce
Dy-iilard Vipe
Fost View
Agai
Eypindral Vay
Figure 4: Five selected Robosuite tasks with examples on four hand-coded views.
of human eyes and the memory. PVM aims to expand the effective observable area, even though some visual observations may become outdated or be superseded by more recent observations. PVM stores the observations from B past steps in buffer and combines them into a single PVM observation according to there spatial positions. This PVM observation subsequently replaces the original observation at each step:
PVM(o.) f(o:-Bth....
where f() is combination operation. In the context of ActiveVision-RL, it is reasonable to assume that the agent possesses knowledge its focus point, as it maintains the control over the view. Therefore, the viewpoint or position information can be used. In our implementations of f(-) shown Figure we show a 2D case PVM using stitching, and a PVM using LSTM which can be used in both 2D and 3D environments. In stitching PVM, the partial observations are combined like Jiasaw puzzle. It is worth noting that combination function can be implemented by other pooling operations, sequence representations [15 44 885 86). neural memories [37 39 76 (8), neural 3D representations 74 [07], depending on the input modalities, tasks, and approaches.
4
Environments and Settings
4.1 Active-Gym: An Environment for Visionnis
We present Active- -Gmm, an open-sourced customized environment wrapper designed transform RL environments into ActiveVision-Ril constructs. Our library currently supports active vision agent Robosuite 114). robot manipulation environment in 3D, well Atari games [9] and the DeepMind Control Suite (DMC) (97] offering 2D) active vision cases. These 2D environments were chosen due to the availability of full observations for establishing baselines and upper bounds, and the availability to manipulate observability for systematic study.
4.2 Robosuite Settings
Observation Space In the Robosuite environment, the robot controls a movable camera. The image captured by that camera is partial observation the 3D space.
Action Spaces Each step in Active-Gym requires motor and sensory actions (a", a²). The motor action space A is the same as the base environment. The sensory action space A is 5-DoF control: relative (x, y, z, yaw, pitch). The maximum linear and angular velocities are constrained to 0.01/step and degrees/step. respectively.
4.3 2D Benchmark Settings
the 2D cases Active-Gym, given a full observation with dimensions (H, W), only crop of it is given to the agent's input. Examples highlighted in red boxes in Figure The sensory action decides an observable area by location (x. y), corresponding to the lep-left corner of the bounding box, and the size of the bounding box (h, w). The pixels within the observable area becomes the foveal observation, defined as of = of Ofz x + h, Optionally, the foveal observation can be interpolated to other resolutions of = Interp(o"; (h, w)
Observation Space
w].
(Ths r(f)), where (r%, r(.) is the foveal resolution. This design allows for flexibility in altering the observable area size while keeping the effective foveal resolution constant. Typically we (r%, rl/) = (h, w) and fixed them during task. The peripheral observation can optionally provided as well, obtained by interpolating the non-foveal part o? Interp O o (H, W) - (rf,r() where (ry, (()) the peripheral resolution. The examples are the even columns of Figure the peripheral observation is not provided, o" 0.
5

---

12.11
Fdll
20c20
20x20
F: F.30x50
30.300
E:50.50
F:50x50
464 didirittiio accaery actions
P: 1:20c20
wite
wi
wis
w
we
wil
(Se Soveal obsercation
Figure 5: Left: observations from Active- with different foveal resolutions (F) and peripheral settings (P) in Atari and DMC. The red bounding boxes show the observable area (foveal) for clarity, Right: 4x4 discrete sensory action options in our experiments.
Table 1:
Results on Robosuite. We report the IQM of raw rewards from 30 evaluations. Highlighted task names are harder tasks. Bold numbers are the best scores of each task and underscored numbers are the second best.
Approach
Wipe
Door
NutAssemblyySuare
Lift
Stack
SUGARL-DDiC (Stacking PVM)
56.0
274.8
78.0
79.2
12.7
SUGARL-DrQ (LSTM PVM)
58.5
266.9
108.6
88.8
31.5
SUGARL-DiQ (3D Transforattion+LSST PVM)
74.1
291.0
65.2
87.5
32.4
SUGARL.-DrQ w/o Joint Learning
43.6
175.4
58.0
107.2
12.0
SUGARL-DiQ w/o PVM
52.8
243.3
37.9
55.6
7.7
Single Policy
12.4
22.8
8.42
10.7
0.53
DrQ w/ Object Detection (DETR)
15.2
43.1
54.8
15.4
DrQ w/ End-to-End Attention
14.2
141.4
28.5
33.0
13.6
Eye-in-hand View (hand-coded, moving camera)
16.1
114.6
102,9
233.9
73.0
Front View (hand-coded, fixed camera)
49.4
240.6
39.6
69.0
13.8
Agent View (hand-coded, fixed camera)
12.7
190.3
49.9
122.6
14.7
Side View (hand-coded, fixed camera)
25.9
136.2
34,5
56.6
12.8
Action Spaces The sensory action space A includes all the possible (pixel) locations on the full observation, but can be further formulated to either continuous or discrete spaces according to specific task designs. In our experiments, we simplify the space by 4x4 discrete grid-like anchors for (2; y) (Figure5]right right). Each anchor corresponds the top-left corner of the observable area. The sensory policy chooses to place the observable area among one 16 anchors (absolute control), or moves from one to the four neighbor locations (relative control).
4.4 Learning Settings
In our study, we primarily use DQN 665 and SAC [36] as the backbone algorithms of SUGARL to address tasks with discrete action spaces (Atari), and use DrQv2 ([08) for continuous action spaces (DMC and Robosuite). All the visual encoders are standardized as the convolutional networks utilized in DQN [55] To keep the network same, we resize all inputs to 84x84. For the sensorimotor understanding model, we employ the similar visual encoder architecture with a linear head predict at. Each agent is trained with one million transitions for each of the 26 Atari games, or trained with 0.1 million transitions for each the 6 DMC tasks and 5 Robosuite tasks. The 26 games are selected following Atari- benchmark [47]. We report the results using Interquartile Mean (IQM), with the scores normalized by the IQM of the base DQN agent under full observation (except Robosuite),
averaged across five seeds (three for Robosuite) and all games/tasks per benchmark. Details architectures and hyperparameters can be found in the Appendix.
Results
5.1 Robosuite Results
We selected five of available tasks in Robosuite [14], namely block lifting (Lift), block stacking (Stack), nut assembling (NuAAssemleSSuurre), door opening (Door), wiping the table (Wipe). The first two are easier compared to the later three. Example observations are available in Figure 4 We compare against a straightforward baseline that single policy learned to govern both motor and sensory actions. We also compare to baselines including RL with object detection (a replication of Cheng al. [17),, learned attention [93], and standard RI. with hand-coded views. Results are In[ We confirm that our SUGARL works outperforms baselines all the time, and also outperforms

---

14
2004
18-30
2001
20:44
20:30
Randon
facer Sonsiing
SEGAAL
Falow Sounting
Upp Agge
(a) Without peripheral observation
(b) With peripheral observation
(c) Static policies comparison
Figure 6: Results with different observation size and peripheral observation settings. The green bars are results from SUGARL. The red dashed lines stand for the DQN baseline trained on full observations using the same amount of data. We compare SUGARL against two dynamic views: Random View and Raster Scanning, and three static view baselines. In (b) with peripheral observation, we compare baseline using peripheral observation only indicated by the cyan line.
the hand-coded views most of the time. Specifically, for the harder tasks including Wipe, Door, and NutAssemblySquare, SUGARL gets the best scores.
Designs of PVM We compare different instantiations of proposed PVMs including: Stacking: naively stacking multiple frames; LSTM: Each image is first encoded by CNN and fed into LSTM; 3D Transformation LSTM: we use camera parameters align pixels from different images the current camera frame. Then an LSTM encodes the images after going through CNN. We find that 3D Transformation LSTM works the best, because it tackles spatial aligning and temporal merging together. LSTM also works well in general.
5.2 2D Benchmark Results
We evaluate the policy learning on Atari under two primary visual settings: with and without peripheral observation. In each visual setting we explore three sizes of observable area (set equivalent to foveal resolution): 20x20, 30x30, and 50x50. In with peripheral observation setting, the peripheral resolution is set 20x20 for all tasks. We use DQN -b5ee SUGARL (SUGARL- DQN) and compare it against two variants by replacing the learnable sensory policy in SUGARL with: Random View and Raster Scanning. Random View always uniformly samples from all possible crops. Raster Scanning uses pre-designed sensory policy that chooses observable areas from left to right and top to down sequentially. Raster Scanning yields relatively stable observation patterns and provides maximum information under PVM. We also provide DQN baseline trained on full observations (84x84) as a soft oracle. In with peripheral observation settings, another baseline trained on peripheral observation only (20x20) is compared
soft lower bound.
In Figure 6a and 6b. we find that SUGARL performs the best in all settings, showing that SUG- ARL learns effective sensory and motor policies jointly. More importantly, SUGARL with peripheral observation achieves higher overall scores (+0.01~-0.2) than the full observation baselines. details, SUGARL gains higher scores than the full observation bascline in 13 out of 26 games with 50x50 foveal resolution (details are available the Appendix). This finding suggests the untapped potential of Active Visivveisin-RR.... agents which leverage partial observations better than full observations. By ac- tively selecting views, the agent can filter out extraneous information and concentrate on task-centric information.
Compare against Static Sensory Policies We also examine baselines with static sensory policies, which consistently select one region to observe throughout all steps. The advantage of this type baseline lies the stability of observation. We select 3 regions: Center, Upper Left. and Bottom Right, and compare them against SUGARL in environment w/o peripheral observation. As shown in Figure 6c| we observe that SUGARL still surpasses all three static policies. The performance gaps between SUGARL and Center and Bottom Right are relatively small when the observation size larger (50x50), as the most valuable information is typically found at these locations in Atari
Table 2: Evaluation results different conditions and algorithm backbones
(a) Action modeling
(b) Train more steps
(c) SUGARL with SAC
(d) Different PVMs
Model
20
30
50
Steps
30
Madell
SUGARL (rel) 0.367
Single Policy
SUGARL (ahs) 0.475 0.810
0.745
0.171
0.805
0.945
SM
Sugle
Single halley 0.600
SUGARL asse 6.05
SUGARI 1.17 L121 1.583
222
SAC-umkom view 0.155 0.104
SUGARL
SAC-nater womning 0.117 0.195
SUGARI
Model
LSTM PVM 0.397 0.448 0.170
Stitching PVM 0.475 0.815 0.810
20
50

---

bookg
"Fixation"
"Movement"
Figure 7: Examples of learned sensory policies from SUGARL-DON Top: frequency heat maps of
pixels being observed. Bottom: sequences of observations showing fixation-like and tracking-like behaviors. More videos are available at this link
environments, Static observation is therefore quite beneficial for decision-making However, as the observation size decreases, an active policy becomes essential for identifying optimal observations, leading SUGARL to outperform others by significant margin.
Sensory-Motor Action Spaces Modeling SUGARL models the sensory and motor action spaces separately inspired by the human abilities. An alternative approach is jointly modeling two action spaces A AA x A and learning one single action policy, with environmental reward only (referred as Single Policy). We compare Single Policy and display the results in Table 2a which reveal that modeling one large action space fails learn the policy properly due the expanded action space. Additionally, we examine two types of sensory action spaces: absolute (abs) and relative (rel). Absolute modeling allows the view to jump across the entire space while relative modeling restricts the view to moving only to the neighboring options at each step. Results in Table 2a indicate that the absolute modeling performs better in smaller observable regions, as can select the desired view more quickly than relative modeling. In contrast, relative modeling demonstrates better performance in larger observable region setting as produces more stable observations across steps.
Training More Steps We subsequently explore SUGARL performance when trained with more environmental steps, comparing outcomes between 1M and 5M steps. Results in Table confirm that SUGARL continuous to improve with more steps and consistently outperforms the single policy baseline, suggesting that does not merely stagnate challenges posed by limited observability, the learning still proceeds slower than for the agent with full observations, which achieves the score 4.106 5M steps.
trivial policy. However, due the
Generalization of SUGARL We apply SUGARL to Soft Actor-Critic (SAC) 36 and evaluate its performance environments without peripheral observation. As before, compare it with Random View, Raster Scanning, and SUGARL without intrinsic reward. According to Table 2c we find that SUGARL-SAC outperforms the naive version without the guidance of sugart further emphasizing the significance of our method. Moreover, SUGARL-SAC also surpasses the random view and raster scanning policies. However, when employing max-entropy- -based methods like SAC, it is necessary to reduce the weight of the entropy term to ensure consistency in the sensory policy. We will further discuss in Section 5.3 based the analysis of the learned sensory policy behavior.
DMC Tasks We also evaluate SUGARL on DMC [97] tasks based DrQ (108), improved version of DDPG [100]. We use six environments in DMC [97]: ball in cup-catch, cartpole-swingup. cheetah-run. dog- feich. feich.fish-swim, fish-swim, and walker-walk. Three foveal observa- tion resolutions (20x20, 30x30, and 50x50) without pe- ripheral observation are explored. We use relative control for sensory action and decide the discrete sensory options by thresholding the continuous output. We compare our approach with baselines with joint modeling of sensory and motor action spaces (Single Policy), Raster Scanning, and Random View. Results shown in Table 3 SUGARL outperforms all baselines in three foveal settings in DMC, showing it is also applicable continuous control tasks.
Table 3: SUGARL on DMC
Model
20
SUGARL-DAA
0.686 0.717 1.052
DrQO-Single Policy
0.540
0.570 0.776
DrO-Raster Scanning
0.913
DrQ-Random View
0.569 0.591 0.768
BUGARL--rO. w/o PVM
0.672 0.620 0.930
SUGARL w/o Joint Learning 0.377 0.446 0.355

---

5.3 Sensory Policy Analysis
Sensory Policy Patterns In Figure 7 we present examples of learned sensory policies from SUGARL-DON in four Atari games, in settings w/o peripheral observations. These policies are visualized as heat maps based on the frequency of observed pixels. We discover that the sensory policies learn both fixation and movement (similar to tracking) behaviours [12 11Q depending on the specific task requirements. In the first two examples, the sensory policy tends to concentrate on fixed regions. In battle Lone,, the policy learns to focus on the regions where enemies appear and the fixed front sight needed for accurate firing. By contrast, in highly dynamic environments like boxing and freeway, the sensory policies tend to observe broader areas in order to get timely observations. Though not being a perfect tracker, the sensory policy learns to track the agent or the object of interest in these two environments, demonstrating the learned capability akin to humans' Smooth Pursuit Movements. Recorded videos for entire episodes are available our project page
Sensory Policy Distribution We quantitatively assess the distributions of learned sensory policies. There are 16 sen- sory actions, i.e. the observable area options in our setup (Figure 5). We compare the distributions against uniform distribution using KL-divergence, across 26 games x 10 eval runs 5 seeds. The resulting histogram are shown in Figure We observe that the learned policies consistently deviate from the uniform distribution, suggesting that sen- sory policies prefer specific regions in general. The high peak at the high KL end supports the "fixation" behavior identified in the previous analysis. As the observation size increases, the divergence distribution shifts towards smaller KL end, while remaining 0.5 for all policies. This trend indicates that with larger observable sizes, the policy does not need to adjust its attention frequently, corroborating the benefit of using relative control shown in Table 2a
Figure 8: KI. divergence distributions of learned sensory policies.
Pitfalls on Max-Entropt Based Methods In the previous analysis, we both qualitatively and quantitatively demonstrate that sensory policies are not uniformly dispersed across the entire sensory action space. This observation implies that sensory policy should exercise caution when adopting the max-entropy-De -based methods. We conduct an experiment on varying the usage of a, the entropy coefficient SAC in all three foveal observation size settings. Results in T show that simply disabling autotune and setting a small value to the sensory policy's 's a improves. This finding tells that max-entropy may not be suitable assumption in modeling sensory policies.
Table 4: Varing
of
SUGARL-SAC.
Model
20
30
50
autotune a
0.271 0.358 0.444
fixed-a 0.2 0.424 0.730 0.785
5.4 Ablation Studies
We conduct ablation studies in the setting without peripheral observation and with 50x50 observation size. Five crucial design components investigated to validate their effectiveness by incrementally adding or removing them from the full model. The results are presented in Table 5 From the results, we demonstrate the importance of penalizing the agent for inaccurate self-understanding predictions rather than rewarding accurate predictions (y-suarrl (positive) - مuugadt (negative)). By imposing penalties, the maximum return is bounded by the original maximum possible return per episode, allowing motor and sensory policies to better coordinate each other and achieve optimal task performance. Reward balance significantly improves the policy, indicating its effectiveness in coordinating two policies as well. PVM also considerably enhances the algorithm by increasing the effective observable area as expected.
Table 5: Ablation study results based on DQN 50x50 foveal res w/o peripheral observation. The blank fields means there are no such modifications for that model.
Model
Learning
PVM
Reward Balance (3)
IQM
Random View
0.367
Base RL algorithm
SUGARL
Naive positive intrinsic reward
positive
0.281
Joint learning
positive nogative negative
0.322
Positive negative pagat
0.360
-PPM
0.423
Reward Balance
negative
0.805
SUGARL wño paaai
0.421
SUGARL w/o aar and w/o PVM
0,231

---

6 Related Work
Active Learning the concept that an agent decides which data are taken into its learning and may ask for external information in comparison to fitting a fixed data distribution [ 6 10
21
22
31 46, 49
55
157 60
64
66
73
81
88 90 101, 105 113]. Active Vision focuses on continuously acquiring
new visual observations that is helpful for the vision task like object classification, recognition and detection 4 7 18 28
63
98 T06 T09], segmentation [T3
58 70,, and action
recognition [45]. The active vision is usually investigated under robot vision scenario that a robot
moves around in a scene. However, the policy is usually not required to accomplish a task with physical interactions such as manipulating objects compared to reinforcement learning.
Active Reinforcement Learning (Active-RL), RL), at a high level, is that the agent is allowed actively gather new perceptual information of interest simultaneously through an RL task, which can also be called active perception [[02] The extra perceptual information could be reward signal 23 50
87 other input modalities [6]. and language instructions [19 69 94] Though these work may not explicitly use the term Active-RI, we find that they can be uniformly organized in the general Active-RI. formulation and we coin the term here. In our work, we study the ActiveVision Vision-RL task in a limited visual observability environment, where at each step the agent is only able to partially observe the environment. The agent should actively seek the optimal observation at each step. Therefore, our setting is more close to [32 33 and Active Vision problems, unlike research incorporating attention-like inductive bias given full observation [34 42
566 62], visual observations from new viewpoints 33
54
67
93 104]. The Vision-RL agent must learn an observation selection policy, called sensory policy, to effectively choose the optimal partial observation for executing the task-specific policy (motor policy). The unique challenge for Active Vision-R -LL is the coordination between sensory and motor policies given there mutual influence. In recent works, the sensory policy can be either trained in the task-agnostic way (3 with enormous exploration data, or trained jointly with the task [32) with naive environmental reward only. In this work we investigate the joint learning case because of the high cost and availability concern of pre-training tasks [33].
43
79
91
Robot Learning with View Changes Viewpoint changes and gaps in visual observations are the 53 77 84 [22) especially for the embodied agents To address those challenges, previous works proposed to map visual observation from different viewpoints to a common representation space by contrastive encoding (26
common challenges for robot learning 40
that uses its first-person view [30
35 80
or build implicit neural representations [53 [07]. In many first-person view tasks, the viewpoint control is usually modeled together with the motor action like manipulation and movement 30 80
82
3
In contrast, in our Active Vision-RL setting, explore the case where the agent can choose where to observe independently to the motor action inspired by the humans' ability.
7 Limitations
In this work, we assume that completely independent sensory and motor actions are present in an embodied agent. But in a real-world case, the movement of the sensors may depend on the motor actions. For example, fixed camera attached to the end-effector of robot manipulator, or to mobility robot. To address the potential dependence and conflicts between two policies in this case, extensions like voting or weighing across two actions to decide the final action may be required. The proposed algorithm also assumes a chance to adjust viewpoints at every step. This could be challenging for applications where the operational or latency costs for adjusting the sensors are high like remote control. To resolve this, additional penalties on sensory action and larger memorization capability are potentially needed. Last, the intrinsic reward currently only considers the accuracy of agent-centric prediction. Other incentives like gathering novel information or prediction accuracy over other objects in the environment can be further explored.
8 Conclusion
We present SUGARL, a framework based on existed RL algorithms to jointly learn sensory and motor policies through the Active RL task. In SUGARL, an intrinsic reward determined by sensorimotor understanding effectively guides the learning of two policies. Our framework is validated in both 3D and 2D benchmarks with different visual observability settings. Through the analysis on the learned sensory policy, it shows impressive active vision skills by analogy with human's fixation and tracking that benefit the overall policy learning. Our work paves the initial way towards reinforcement learning using active agents for open-world tasks.
10

---

Acknowledgments and Disclosure of Funding
We appreciate the fruitful discussions on the methodology with Xiang Li and Wensheng Cheng, on experimental designs with Kumara Kahatapitiya and Ryan Burgert. We also appreciate the inspiring feedbacks from Kanchana Ranasinghe and Varun Belagali.
References
[1] S. Agarwal, H. Arora, S. Anand, and C. Arora. Contextual diversity for active learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 137-153. Springer, 2020.
[2] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's 's cube with robot hand. arXiv preprint arXiv:1910.07133. 2019.
[3] Akrour, M. Schoenauer, and M. Sebag. April: Active preference learning-basse baaed reinforce- ment learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK. September 24-28, 2012. Proceedings, Part II 23, pages 1 116-131. Springer, 2012.
[4] A. Andreopoulos and J. K. Tsotsos. A theory of active object localization. In 2009 IEEE 12th International Conference on Computer Vision, pages 903-910. IEEE, 2009.
[5] A. Andreopoulos and J. K. Tsotsos. A computational learning theory of active object recogni- tion under uncertainty. International journal of computer vision, 101:95-144 2013.
[6] J. Ash, Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020.
[7] N. Atanasov, B. Sankaran, J. Le Ny, G. Pappas, and K. Daniilidis. Nonmyopic view planning for active object classification and pose estimation. IEEE Transactions on Robotics, 30(5); 1078-1090, 2014.
[8] R. Bajcsy, Y. Aloimonos, and J. K. Tsotsos. Revisiting active perception. Autonomous Robots, 42:177-196, 2018.
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 153-279,110 jun 2013.
[10] W. H. Beluch, T. Genewein, Nürbberger, and J. M. Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9368 9377, 2018
[1]] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning, arXiv preprini arXiv:1912.06680, 2019.
[12] A. Borji and L. Itti. State-of tht in visual attention modeling. IEEE transactions on pattern analysis and machine intelligence, 35(1):185-25) 207, 2012.
[13] A. Casanova, P. O. Pinheiro, N. Rostamzadeh, and C. J. Pal. Reinforced active learning for image segmentation. arXiv preprint arXiv:2002.06583 2020.
[14] E. Cetin, P. J. Ball, S. Roberts, and O. Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pages 2784-2810. PMLR, 2022.
[15] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), Dec. 2021.

---

[16] S. Chen, H. Cao, Q. Ouyang, X. Wu, and Q. Qian. Alds: An active learning method for multi-souree materials data screening and materials design. Materials & Design, 223:111092. 2022. ISSN 0264-1275.
[17] R. Cheng, A. Agarwal, and K. Fragkiadaki. Reinforcement learning of active vision for manipulating objects under occlusions. In Conference on Robot Learning, pages 422-431 4311. PMLR, 2018.
[18] R. Cheng, Z. Wang, and K. Fragkiadaki. Geometry-aware recurrent neural networks for active visual recognition. Advances Neural Information Processing Systems, 31, 2018.
[19] T.-C. Chi, M. Shen, M. Eric, S. Kim, and D. Hakkani-tur. Just ask: An interactive learning framework for vision and language navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2459-2466, 2020.
[20] J. Choi and S.-e. Yoon. Intrinsic motivation driven intuitive physics learning using deep reinforcement learning with intrinsic reward normalization. arXiv preprint arXiv: UrXiv:1907.000 19070090711111 16, 2019.
[21] J. Choi, K. M. Yi, J. Kim, J. Choo, B. Kim, J. Chang, Y. Gwon, and H. J. Chang- Vab-al: Incorporating class imbalance and difficulty with variational bayes for active learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6749-6758, 2021.
[22] D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. Journal of artificial intelligence research, 1:129-145, 1996.
[23] C. Daniel, M. Viering. J. Metz, O. Kroemer, and J. Peters. Active reward learning. In Robotics: Science and systems, volume 98, 2014.
[24] R. Dodge. Five types eye movement in the horizontal meridian plane of the field of regard. American journal of physiology-legucy content, 8(4):307-32,, 1903.
[25] P. W. Duncan, M. Propst, and S. G. Nelson. Reliability of the fugl-meyer assessment of sensorimotor recovery following cerebrovascular accident. Physical therapy, 63(10):6606 1610, 1983.
[26] D. Dwibedi. J. Tompson, C. Lynch, and P. Sermanet. Learning actionable representations from visual observations. JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1577-1584. IEEE, 2018.
[27] A. Epshteyn, A. Vogel, and G. DeJong, Active reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 296-303, 2008
[28] L. Fan and Y. Wu. Avoiding lingering in learning active recognition by adversarial disturbance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4612-4621, January 2023.
[29] L. Fan, P. Xiong, W. Wei, and Y. Wu. Flar: A unified prototype framework for few-sample lifelong active recognition. In Proceedings of the IEÉE/CVF International Conference on Computer Vision (ICCV), pages 15394-15403, October 2021.
[30] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang. H. Zhu, A. Tang. D.-A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems, volume 35, pages 18343- 18362. Curran Associates, Inc., 2022.
[31] Y. Gal, R. Islam, and Z. Ghahramani. Deep bayesian active learning with image data. In International conference machine learning, pages 1183-1192. PMLR, 2017.
[32] R. Göransson. Deep reinforcement learning with active vision atari environments. In Master's 's Thesis. Lund University, 2022.
[33] M. K. Grimes, J. V. Modayil, W. Mirowski, D. Rao, and R. Hadsell. Learning to look by self-prediction. Transactions on Machine Learning Research, 2023, ISSN 2835-8856.
12

---

[34] S. S. Guo, R. Zhang. B. Liu, Y. Zhu, D. Ballard, M. Hayhoe, and P. Stone. Machine versus human attention in deep reinforcement learning tasks. Advances in Neural Information Processing Systems, 34:25370-225 25385, 2021.
[35] W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. Twent-Eiighh International Joinl Conference on Artificial Intelligence, 2019. URL http:://miierr..io
[36] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), 2018.
[37] M. J. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2015.
[38] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforce- ment learning that matters. In Proceedings of the AAAI conference on artificial intelligence. volume 32, 2018.
[39] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
[40] K. Hsu, M. J. Kim, R. Rafailov, J. Wu, and C. Finn. Vision-based manipulators need to also see from their hands. arXiv preprint arXiv:2203. 12677, 2022.
[41] W. Huang. I. Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pages 4455 4464. PMLR, 2020.
[42] S. James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1612-1619, 2022.
[43] Jame,, K. Wada, T. Laidlow, and J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[44] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances Neural Information Processing Systems (NeurIPS), Dec. 2021.
[45] Jayaraman and K. Grauman. Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion. In Computer Vision-ECC ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 489-505. Springer, 2016.
[46] A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Multi-class active learning for image classification. In 2009 ieee conference on computer vision and pattern recognition, pages 2372-2379. IEEE, 2009.
[47] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Camphell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model based reinforcement learning for atari. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[48] S. Kasaci, J. Sock, L. S. Lopes, A. M. Tomé, and T.-K. Kim. Perceiving, learning, and recognizing 3d objects: An approach to cognitive service robots. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[49] T. Kim, I. Hwang, H. Lee, H. Kim, W.S. Choi, J. J. Lim, and B.-T. Zhang. Message passing adaptive resonance theory for online active semi- saiisuueeeisee learning. In International Conference on Machine Learning, pages 5519- 5529. PMLR, 2021.
[50] D. Krueger, J. Leike, O. Evans, and J. Salvatier. Active reinforcement learning: Observing rewards at cost. arXiv preprint 2:Xiv:2011.06709, 2020.
13

---

[51] S. LLevine P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421-436 2018.
[52] X. Ii, Shang. S. Das, and M. Ryoo. Does self-supervised learning really improve reinforce- ment learning from pixels? In Advances in Neural Information Processing Systems, volume 35, pages 30865-30881, 2022.
[53] Y. Li, Li, V. Sitzmann, Agrawal, and Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 112-123. PMLR, 2022.
[54] L. Liu, S. Frye, L. Wu, T. L. Vu, G. Paul, and T. Vidal-Calleja. Active and interactive mapping with dynamic gaussian process implicit surfaces for mobile manipulators. IEEE Robotics and Automation Letters, 6(2):3679-3666, 2021.
[55] Z. Liu, H. Ding, H. Zhong, W. Li, Dai, and C. He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9274-9283, 2021.
[56] M. Lopes, F. Melo, and L. Montesano. Active learning for reward estimation in inverse rein- forcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2009, Bled, Slovenia, September 7-11, 2009, Proceedings, Part II 20, pages 31-46. Springer Berlin Heidelberg, 2009.
[57] W. Luo, A. Schwing, and R. Urtasun. Latent structured active learning. Advances Neural Information Processing Systems, 26, 2013.
[58] D. Mahapatra, B. Bozorgtabar, J.-P. Thiran, and M. Reyes. Efficient active learning for image classification and segmentation using sample selection and conditional generative adversarial network. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 580-588. Springer, 2018
[59] J. S. Matthis, J. L. Yates, and M. M. Hayhoe. Gaze and the control of foot placement when walking in natural terrain. Current Biology, 28(8): 122--123, 2018.
[60] C. Mayer and R. Timofte. Adversarial sampling for active learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3071-3079, 2020.
[61] M. K. McBeath, M. Shaffer, and M. K. Kaiser. How baseball outfielders determine where to run to catch fly balls. Science, 268(5210):569
1995.
[62] P. Ménard, O. D. Domingues, A. Jonsson, F. Kaufmano, E. Leurent, and M. Valko. Fast active learning for pure exploration in reinforcement learning. In International Conference on Machine Learning, pages 7599-7608. PMLR, 2021.
[63] M. B. Mirza, R. Adams, C. D. Mathys, and K. J. Friston Scene construction, visual foraging, and active inference. Frontiers in computational neuroscience, 10:56, 2016.
[64] S. Mittal, M. Tatarchenko, Ö. Çięek, and T. Brox. Parting with illusions about deep active learning. arXiv preprint arXiv:1912.05366) 2019.
[65] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning, 2013. arXiv: 132.552..
[66] P. Munjal, N. Hayat, M. Hayat, Sourati, and Khan. Towards robust and reproducible active learning using neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 223-232, 2022.
[67] A. Narr, R. Triebel, and D. Cremers. Stream active learning for efficient and adap- tive classification of 3d objects. In 2016 IEFE International Conference on Robotics and Automation (ICRA), pages 227-233 IEEE, 2016.
14

---

[68] K. Nguyen and H. Daumé III. Help, anna! visual navigation with natural multimodal assistance retrospective the 2019 Conference Empirical Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 614-695,2 2019.
imitation learning. In Proceedings of
[69] K. Nguyen, D. Dey, C. Brockett, and B. Dolan. Vision-based navigation with language-basee assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12527-12537, 2019.
[70] D. Nilsson, A. Pirinen, E. Gärtner, and C. Sminchisescu. Embodied visual active learning for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2373-2383, 2021.
[71] J. K. O'regan and A. Noë. A sensorimotor account of vision and visual consciousness. Rehavioral and brain sciences, 24(5):939-973, 2001.
[72] J. Piaget. Piaget's theory, 1976.
[73] R. Pinsler, J. Gordon, E. Nalisnick, and J. M. Hernández-Lobato. Bayesian batch active learning as sparse subset approximation. Advances neural information processing systems, 32, 2019.
[74] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[75] A. Rangel and A. Clithero. Value normalization decision making: theory and evidence. Current opinion in neurobiology, 22(6):970-99) 981000999111111 2012.
[76] E. Rumelhart, G. E. Hinton, and Williams. Learning internal representations by error propagation. In J. W. Shavlik and T. G. Dietterich, editors, Readings in Machine Learning, pages 115-137, San Mateo, CA, 1990. Kaufmann.
[77] M. S. Ryoo, T. J. Fuchs, L. Xia, J. K. Aggarwal, and L. Matthies. Robot-centric activity prediction from first-person videos: What will they do to me? In Proceedings of the tenth annual ACM/IEEE international conference on human-robot interaction, pages 295-302, 2015.
[78] M. S. Ryoo, K. Gopalakrishnan, K. Kahatapitiya, T. Xiao, K. Rao, A. Stone, Y. Lu, J. Ibarz, and A. Arab.. Token turing machines. arXiv preprint arXiv:2211.09111 2022.
[79] S. Salter, Rao, M. Wulfmeier, Hadsell, and I. Posner. Attention-privilett reinforcement learning. In Conference on Robot Learning, pages 394 408. PMLR, 2021.
[80] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, F. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: platform for embodied ai research. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Oct 2019.
[81] O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 17x0v:7008.00489, 20048922011 2017.
[82] P. Sermanet, C. Lynch. Y. Chebotar, Hsu, E. Jang, S. Schaal, Levine, and G. Brain. Time- contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation (ICRA), pages 1134-1141. IEEE, 2018
[83] J. Shang and M. S. Ryoo. Self-supervised disentangled representation learning for third-person imitation learning. In 202/ IEFF/RSJ International Conference Intelligent Robots and Systems (IROS), pages 214-221, 2021.
[84] J. Shang, S. Das, and M. Ryoo. Learning viewpoint-agnostic visual representations by recovering tokens in 3d space. Advances in Neural Information Processing Systems, 35: 31031-31044, 2022.
15

---

[85] J. Shang, K. Kahatapitiya, X. Li, and M. S. Ryoo. Starformer: Transformer with state-action- reward representations for visual reinforcement learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[86] J. Shang, X. Li, K. Kahatapitiya, Y.-C. Lee, and M. S. Ryoo. Starformer: Transformer with state- reward representations for robot learning. IEEE transactions on pattern analysis and machine intelligence, 2022.
[87] J. Shields, O. Pizarro, and S. Williams. Towards adaptive benthic habitat mapping. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 9263-9270, IEEE, 2020.
[88] Shui, F. Zhou, C. Gagné, and B. Wang. Deep active learning: Unified and principled method for query and training. In International Conference on Artificial Intelligence and Statistics, PMLR, 2020.
pages
[89] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. arXiv preprint arXiv:
2017.
[90] S. Sinha, S. Ebrahimi, and T. Darrell. Variational adversarial active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
[91] I. Sorokin, A. Seleznev, M. Pavlov, A. Fedorov, and A. Ignateva. Deep attention recurrent q-network, 2015.
[92] B. Stadie, P. Abbeel, and I. Sutskever. Third arXiv:1703.01700 2017.
Third-person person imitation learning. arXiv preprint
[93] Y. Tang, D. Nguyen, and D. Ha. Neuroevolution of self-interpretable agents. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pages 414-24, 2020.
[94] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394-406. PMLR, 2020.
[95] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:/805.0/9: 009554 2018.
[96] A. Tucker, A. Gleave, and S. Russell. Inverse reinforcement learning for video games. arXiv preprint arXiv:1810.1052: 2018.
[97] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess. and Y. Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638.
[98] T. Van de Maele, T. Verbelen, O. Çatal, and B. Dhoedr. Embodied object representation learning and recognition. Frontiers in Neurorobotics, 16:840658, 2022.
[99] H. P. van Hasselt, A. Guez, M. Hessel, V. Mnih, and D. Silver. Learning values across many orders of magnitude. Advances in neural information processing systems, 29, 2016.
[100] M. Vecerik, T. Hester, Scholz, Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 1eXiv:1707.0880 08817, 2017.
[101] F. Wan, T. Yuan, M. Fu, X. Ji, Q. Huang, and Q. Ye. Nearest neighbor classifier embedded network for active learning. In Proceedings of the AAAI Conference Artificial Intelligence. volume 35, pages 10041-10048, 2021.
[102] S. D. Whitehead and D. H. Ballard. Active perception and reinforcement learning. In Machine Learning Proceedings 1990, pages 179-188. Elsevier, 1990.
[103] M. Wolpert, J. Diedrichsen, and R. Flanagan. Principles of sensorimotor learning. Nature reviews neuroscience, 12(12):739-751, 2011.
16

---

[104] H. Wu, K. Khetarpal, and D. Precup. Self-supervised attention aware reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10311
10319, 2021.
[105] Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining pretrinning-imettninng paradigm. arXiv preprint WxXiv:2303.1438. 2023.
[106] J. Yang. Z. Ren, M. Xu, X. Chen, D. J. Crandall, D. Parikh, and D. Batra. Embodied amodal recognition: Learning to move to perceive objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2040-2050, 2019.
[107] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Conference on Computer Vision and Pattern Recognition, 2023.
[108] D. Yarats, R. Fergus, A. Lazaric, and L.. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint (xXi:2200700645, 2021.
[109] T. Yuan, F. Wan, M. Fu, J. Liu, S. Xu, X. Ji, and Q. Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1330-5339, 2021.
[110] G. Zelinsky, W. Zhang, B. Yu, X. Chen, and D. Samaras. The role of top-down and bottom-up processes in guiding eye movements during visual search. Advances in neural information processing systems, 18, 2005.
[111] T. Zhang, Li, C. Wang, G. Xie, and Z. Lu. Fop: Factorizing optimal joint policy of maximum- entropy multi-agent reinforcement learning. In International Conference on Machine Learning. pages 1 12491-1 15000. PMLR, 2021.
[112] A. Zhou, M. J. Kim, L. Wang, P. Florence, and C. Finn. Nerf in the palm of your hand: Cor- rective augmentation for robotics via novel-view synthesis. arXiv preprint arX15:2301.08556, 2023.
[113] J.JJ Zhu and J. Bento. Generative adversarial active learning. arXiv preprint ar Xiv: arXiv: 1702.07556. 2017.
[114] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martin, A. Joshi, S. Nasiriany, and Y. Zhu. robo- suite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.1222 2020.
17

---

A Appendix
A.1 Learning Curves
Figure 9: Learning curves of 26 Atari games, under the setting of 50x50 foveal observation size and 20x20 peripheral observation.
Figure 10: Learning curves of 26 Atari games, under the setting of 30x30 foveal observation size and 20x20 peripheral observation.
A.2 Hyper-parameter Settings
18

---

SUGARL
Figure 11: Learning curves of 26 Atari games, under the setting of 20x20 foveal observation size and 20x20 peripheral observation.
DON'U
SUGAILOOH
CON--aatte
DON-Bundem
DON-Crrter
DONA"SX
2007
Figure 12: Learning curves of 26 Atari games, under the setting of 50x50 foveal observation size and w/o peripheral observation.
19

---

furden
DONATEKATO
60
Figure 13: Learning curves of 26 Atari games, under the setting of 30x30 foveal observation size and w/o peripheral observation.
DON'UE
SUGAILOOH
SUGNI,
COl--latte
DON-fundem
DON-Crrter
DONA"SX
2027
Figure 14: Learning curves of 26 Atari games, under the setting of 20x20 foveal observation size and w/o peripheral observation.
20

---

DIO
DYGAL
flaiz Sca Scanting
DrD Randen Viow
DYO DYOA"SXAT
malavve
299
405
383
200
150
210
150
200
Figure 15: Learning curves of 6 DMC environments, under the setting 50x50 foveal observation size and w/o peripheral observation.
Fal
SUKAAR DHQ
DAO RNOW SONTRES
DO Aantem View
DRO ATE
200
300
400
150
1.0
20W0 5000
1500 20000
1500
7530
2500 7380 200C
75ICE 10000
2500 75000 1900
2500
750M
Figure 16: Learning curves of 6 DMC environments, under the setting of 30x30 foveal observation size and w/o peripheral observation.
200
180
1.5
150
200
33303 30000
15000 200000
35009 J8303 15000 S00ES
1500 5000 2303 325
200 5000
PUICE 100000
250C JUDK 7500 330000
21000 ScERc TIRE
Figure 17: Learning curves of 6 DMC environments, under the setting of 20x20 foveal observation size and w/o peripheral observation.
Table 6: Hyper-parametee for DQN SUGARL-DQN (on Atari)
Total steps
1,000,000 or 5,000,000
Replay buffer size
100,000
start
1.0
end
0.01
min e step
100,000
?
0.99
Learning start 80,000
Q network train frequency
Target network update frequency
1,000
Learning rate
10 10-4
Batch size
32
Self-understanding module train frequency
Self Seff-understanding module learning rate
10-4
21

---

Table 7: Hyper- pareeeeeraeeerrss for SAC (on Atari)
Total steps
1,000,000
Replay buffer size
100,000
Y
0.99
Learning start
80,000
Actor train frequency
4
Critic train frequency
4
Target network update frequency
8,000
Actor Learning rate
3
-4
Critie Learning rate
3 x 10
Batch size
64
Self-understanding module train frequency
4
soff-understanding module learning rate
3 x 10
Visual policy alpha
0.2
Physical policy alpha
autotune
Physical policy target entropy scale
0.2
Table 8: Hyper- for DrQv2 (on DMC)
Total steps
100,000
Replay buffer size
100,000
? 0.99
Standard deviation start
1.0
Standard deviation end
0.1
Standard deviation end step
50,000
Standard deviation clip
0.3
Learning start 2,000
Actor train frequency
Critic train frequency
Target network update frequency
Target network exponential moving average weight
0.01
Actor Learning rate
10-4
Critic Learning rate 10-4
Batch size 256
Self-understanding module train frequency
Self-understandie module learning rate
10
Multiple-step reward
22

---

Table 9: Hyper-paan ppper-parameters for DrQv2 (on Robosuite)
Total steps
100,000
Replay buffer size
100,000
?
0.99
Standard deviation start
1.0
Standard deviation end
0.1
Standard deviation end step
50,000
Standard deviation clip
0.3
Learning start
8,000
Actor train frequency
Critic train frequency
Target network update frequency
Target network exponential moving average weight
0.01
Actor Learning rate
10-4 44
Critic Learning rate
10
Batch size
256
Seff-understanding module train frequency
Self-understanding module learning rate
10-4
Multiple-step reward
Table 10: Environment Settings
Atari
Gray-scale
True
Full observation size
84x84
Frame stacking
4
Action repeat (frame skipping)
Observable area initial location
(0,0)
Sensory action options
4 x grid
Sensory action space size
16 (abs) or (rel)
PVM number of steps 3
DMC
Gray-scale
True
Full observation size
84x84
Frame stacking
3
Action repeat (frame skipping)
2
Observable area initial location
(0, 0)
Sensory action options
4 x grid
Sensory action space size
5 (rel)
PVM number of steps 3
Robosuite
Gray-scale
False
Full observation size
84x84
Frame stacking
3
Action repeat (frame skipping)
2
Observable area initial location
Side View
Sensory action space PVM number of steps
continuous relative 5-DoF control
3
23


Reviews:
[{"review_id": "QreCOQPOcR", "reviewer": "Reviewer_LTdh", "summary": "This work focuses on reinforcement learning with a controllable perceptive field where the agent not only learns a motor policy but learns a sensor policy at the same time. The sensor policy controls the information to be obtained. This work proposes to coordinate two policies by introducing an intrinsic reward encouraging the sensor policy to provide observation correlates to the decision making. The proposed method is evaluated on modified DMC and atari benchmarks to show the effectiveness of the proposed method.", "questions": "1. How does DrQ or DQN with separate heads for sensor policy and motor policy work in different benchmarks? \n2. How does other forms of memory work in this case, like LSTM which is much more generalizable to other applications compared with the proposed PVM module.", "limitations": "Potential negative societal impact is not applicable to this work. For POMDP, providing some kind of memory mechanism for agent to learn a better policy is quite standard practice in the community, some baseline with LSTM or other memory mechanism should be compared.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is intuitive and easy to implement, enough technical details are included for future reproducement. \n2. This work performed extensive ablation study over different components of the proposed method and showed effectiveness of different components.\n3. This work provides clear visualization of learned sensor policy for better understanding of the work, and discusses the behavior of the sensor policy under different observation resolutions. \n4. The overall writing of the work is good, the paper is easy to follow and easy to understand.", "weaknesses": "1. The problem itself is not novel, previous work[1] has studied similar settings with more challenging tasks, separately modeling the sensor policy and the motor policy should not be considered a contribution as well. With only proposed new intrinsic reward functions, the novelty of the work is relatively limited.\n2. The proposed Persistence-of-Vision Memory module is specifically designed for the given task, which can hardly apply to more realistic tasks like moving cameras with different poses. \n3. According to the detailed experiment results in supplementary materials, the experiment results are super noisy and the performance varies a lot according to the environment. \n4. Some baselines like DrQ / DQN with separate heads for motor policy and sensory policy should be compared to validate the method. These baselines should not be considered as part of the ablation study.\n\n[1] Cheng, et al; Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments and questions. Please see below for our response that addresses the concerns.\n\n**1\\. Novelty of this research**\n\nWe would like to highlight our contributions as follows:\n\n- The decomposition of policies governing sensory and motor actions. It’s a different approach compared to conventional single-policy approaches including [Cheng et al., 2018]. We experimentally show that ours works much better (Table 1 in the rebuttal PDF) on difficult Robosuite manipulator tasks. \n- The new approach to enable learning of such decomposed policies, based on joint learning with intrinsic reward. And we empirically show the joint learning approach is easy to implement by simple extensions from many existing RL algorithms, and it works well.\n- We bring up the concept of PVM and show different instantiations of it. We confirm their effectiveness through experiments in 2D and 3D environments.\n- We provide comprehensive analysis on sensory policy behavior that helps understanding and is beneficial for further sensory policy / joint learning algorithm design.\n\nAnd another key difference is that [Cheng et al., 2018] was originally designed for goal-conditioned RL, which has access to the privileged information.\n\n**2\\. More instantiations of Persistence-of-Vision Memory (PVM)**\n\nThe Persistence-of-Vision Memory is proposed to be a general framework that combines multiple recent partial observations into one using a function $f()$. What we present in the paper is one implementation of it (i.e., 2D-image stitching), highlighting its necessity. Following the suggestion, we further provide more instantiations of $f()$. We additionally implemented:\n\n- **LSTM-based PVM**: Each image is first encoded by CNN and then fed into LSTM\n- **3D Transformation + Stacking**: We use camera parameters to align pixels from different images to the current camera frame. Then the transformed images are simply stacked on the channel axis.\n- **3D Transformation + LSTM**: Similar 3D transformation as above, but using an LSTM to encode the image sequence.\n\nThe results in Table 1 in rebuttal PDF show that they can handle 3D environments. In particular, the approach of using both temporal LSTM and 3D alignment for PVM is most effective. It can effectively handle the camera motion and improve policy learning.\n\nThe motivation behind the PVM design is to combine visual observations from multiple **spatio-temporal** locations. We study its influence on the challenging active vision + RL problems. While recurrent models like LSTMs could provide a memorization capability to a certain degree, explicit spatial-temporal encoding (e.g., 2D stitching or 3D Transformation) further improves the memory.\n\n\n**3\\. Performance and learning curves**\n\nThe performance varies across environments due to different environment properties, which is very common in RL algorithms. For example, in Appendix E of DreamerV2 [Hafner et al., 2021], Figure 3 of SimPle [Kaiser et al., 2020], the performance also varies a lot. \n\nFor the curves, we show the raw episodic return during training without smoothing. They are noisy due to random explorations in the learning process. We will improve the visualization to make it clear.\n\n**4\\. More comparison using separate heads**\n\nFollowing this suggestion, we provide more results of DrQ with separate heads (i.e. **SUGARL-DrQ w/o Joint Learning**) in Robosuite (Table 1 in the rebuttal PDF) and in DMC (Table 2 in the rebuttal PDF). Both experiments show separated heads are not producing better results than full SUGARL.\n\nThank you for reading our responses.\n\n**References**\n\n- Cheng et al., Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions, CoRL 2018\n- Hafner et al., Mastering Atari with Discrete World Models, ICLR 2021\n- Kaiser et al., Model Based Reinforcement Learning for Atari, ICLR 2020\n\n"}, {"review_id": "DpovZFar0B", "reviewer": "Reviewer_SANf", "summary": "This paper presents a method for what the authors call \"active RL\", a flavor of RL where the agent not only needs to pick a (motor) action, but also needs to select the (sensory) action where to look. The authors do so by training a DQN or SAC agent with separate heads for the motor and sensory policies respectively. The sensory actions are learned using a separate reward function, which is based on the prediction error when predicting the motor action from two consecutive observations. The authors evaluate on Atari and DMC equipped with limited observability.", "questions": "- In the case no PVM is used, to you still provide the complete frame size, but with the unseen pixels black-ed out. Would it also work when just providing the foveal observation?\n\n- The sugarl reward is focused on predicting the agent's motor action. However, in many games you should rather pay attention to the surroundings, since you \"know\" your action (i.e. especially when standing still for instance), but you'd want to know if there are new obstacles popping up at the edges. Have you also tried other incentives, for instance expected information gain, which supposedly also underpins human saccading, i.e.\nhttps://www.frontiersin.org/articles/10.3389/fncom.2016.00056/full\nhttps://www.frontiersin.org/articles/10.3389/fnbot.2022.840658/full\n\n- It's interesting that SUGARRL outperforms vanilla RL with full observations on some environments. Do you have any insights on why this might be the case? Is there anything common on the particular environments where the gap is largest?\n\n- Would it help to provide the policy with the current sensory action, i.e. so it knows where it is currently looking at. Although it probably won't make a difference in case the PVM / full observation with black pixels is used, as it can derive it from that. Still, it feels like a waste of compute to process all these black pixels.\n\n- Could the sensory policy be generalized across environments?", "limitations": "The main limitation mentioned is the slower training. I'd say the limited set of fixed fixation points is also a limitation of the current implementation. It would be nice to scale this to an agent that e.g. has a continuous pan/tilt control for its camera.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Active vision is an important skill that we humans use all the time, but is often overlooked in the RL literature. It's nice to see an \"active RL\" approach addressing this issue.\n\n- The paper is well written, and presents nice quantitative and qualitative results on both DMC and Atari.", "weaknesses": "- I found the very coarse discretization of sensory actions limiting.\n\n- The method seems very dependent on the reward balance hyperparameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our technical contribution and our analysis. Please find our response below.\n\n**1\\. Sensory action space**\n\nWe follow the suggestion to test SUGARL on continuous camera control on Robosuite, a simulated robotics environment. The agent uses a 5-DoF relative (x, y, z, yaw, pitch) to control the camera. Our results are available in general response and Table 1 in rebuttal PDF. We find that SUGARL is also effective in continuous sensory action setting for challenging robotics tasks.\n\n**2\\. Reward balance**\n\nSimilar to other research [Hasselt et al., Henderson et al., Choi et al.], the reward balance hyperparameter is necessary to make rewards at different scales to work. We follow those previous research to select this balance. The hyperparameter is computed as the max/average environmental return normalized by the length of the trajectory. In our experiments on Atari and DMC, we use only one hyperparameter per game/task for all visual settings. SUGARL performs well consistently. And this hyperparameter strategy also works for our new Robosuite experiments. These verify the robustness of balance selection.\n\n**3\\. Pixel utilization / foveal observation without black padding**\n\nWe find out that providing the foveal observation without padding (IQM=0.400) is not better than the approach with PVM (IQM=0.805). Providing those blacked pixels is helping the performance.\n\n\n**4\\. Sensory policy incentives and behaviors**\n\nThanks for these inspirational references. We will look into the connections between ours and the references, and will investigate new forms of incentives in the future.\n\nThe sensory policy is learned and will focus on the necessary part. It is not forced to pay attention to specific parts like surroundings. The presence of environmental reward encourages both policies to be task-oriented. In this case, the sensory action won't overly satisfy the intrinsic reward. Furthermore, due to the existence of PVM, the agent is able to access observation several steps before. This allows the agent to use information gathered from different locations, and thus supports the potential \"saccading\" skills.\n\n**5\\. Insights on outperforming some full observation baselines**\n\nOur insight is that the partially observable area excludes noisy information that may not be helpful for decision making.\nFor example, the “scores” either on the top or the bottom of the game screen can be noisy. They are changes in pixels, but they do not mean the change of actual “state” like object locations.\n\nBattle_zone is actually one of them with the largest performance improvements than the full observation model (>3x). As shown in Figure 6, the “fixation” behavior makes visual observation fixed to the center of the screen, which is the only direction the agent can fire a bullet. This \"simplified\" observation allows easier policy learning. More visualization is available in the caption of Figure 6.\n\n**6\\. Providing sensory action location as input**\n\nThanks for the thoughtful question! PVM indeed implies the sensory action location. And it is able to precisely combine observations spatially. We tried to provide such information to the policy but it does not further improve.\n\n\n**7\\. Sensory policy transfer**\n\nThis work is under the single-task setting, so it’s hard to measure the transferability of the learned agent. In visual RL, the transfer of the visual encoder itself could be challenging before studying the transfer of the policy [Hansen et al.], as it has more parameters. To make it work, multi-task (pre-)training might be needed and it is a completely different setting from this paper. The transfer of learned policy is a valuable direction and we will leave it for future work. Thanks for the valuable question!\n\n**8\\. Continuous camera control**\n\nWe follow the suggestion and conduct new experiments on Robosuite, with continuous control of the camera. According to the results in Table 1 in the general rebuttal PDF, SUGARL works the best. Please also refer to our general response for more details.\n\nThank you for reading our responses.\n\n**References**:\n\n- Henderson et al., Deep reinforcement learning that matters, AAAI 2018\n- Hasselt et al., Learning values across many orders of magnitude, NeurIPS 2016\n- Choi et al., Intrinsic motivation driven intuitive physics learning using deep reinforcement learning with intrinsic reward normalization, 2019\n- Hansen et al., Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation, NeurIPS 2021\n"}, {"review_id": "EihegwoXo6", "reviewer": "Reviewer_Wwp7", "summary": "The authors develop an RL framework for problems where the agent must control its perception in order to solve the task. Their framework, SUGARL, decouples the task into a sensory and motory policy, and propose a heuristic reward for training the sensory policy stably. They evaluate in Active Vision versions of Atari and DMC.\n\n---\n(8/14/23) The rebuttal has addressed my major concern, which is showing SUGARL results in more realistic robotics tasks in 3d scenes. I also appreciate the additional baselines the authors added in. \n\nUpdating my score from 3 to a 6. \n\nI would recommend the authors to focus their time now on making the paper easier to read, as well as performing experiments on two more realistic robotic settings - a setting where moving the camera has a cost, and a setting where there are physical occluders. \n\n", "questions": "## Questions from the limitations:\nCan the authors provide results on a more realistic benchmark where active perception is required, like manipulation or benchmark?\n\nCan the authors provide a more \"fair\" comparison with baselines by either removing or adding PVM? \n\nDoes the SUGARL reward work in environments where action prediction is not helpful for solving the task? \n\n\n## Clarification questions:\n\nIt says that both policies are trained with a shared reward function. Does this mean that the motor policy can also optimize the sugarl reward? If so, couldn't this lead to degenerate case where motor policy only moves in very predictable ways?", "limitations": "The authors say one sentence for limitations - that Active-RL learns more slowly than full observations. This is fairly obvious, and I would like to see more discussion on other limitations. I can suggest some.\n\nThis framework makes many assumptions about the active perception itself. First, it assumes a clean separation between motor and sensory actions - they do not affect each other. However, in many tasks, *interactive perception* is required - a robot may choose to move its arm away from the camera or open a box to remove occlusion. These actions can count as both motor and sensory actions. Another assumption is that the active perception actions happen instantly alongside motor actions, and have no cost. This is not true in the real world. Can SUGARL handles these assumptions, or what extensions would be necessary to address these assumptions?\n \nNext, what are the compute and resource constraints of training an Active-RL policy versus a normal policy? Can the authors provide more details about GPU usage, runtime of experiments, etc.?", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- An RL framework that sensibly handles active perception by decoupling the overall task into sensory and motor tasks. \n\t- The authors propose a heuristic dense reward for the sensory policy that seems to work well in their tasks, although I have some questions about its generality. Nevertheless, I appreciate its simplicity and effectiveness.\n- Interesting and diverse experiments, although experiments are in rather artificial active vision settings (see concerns).\n\t- Sometimes, Active-RL can even outperform Full Observation baseline with joint training of the active vision policy.  This is impressive, since training both active vision and motor policies from scratch can easily destabilize the RL agent.\n\t- I like the qualitative analysis of the sensory policy behavior.", "weaknesses": "## Major\n**Experimental section does not evaluate active perception in natural settings.**\n - There is an obvious omission of tasks in natural domains where active perception is necessary. All tasks are currently done in a modified version of Atari or DMC where the agent can instantly change the view per timestep. In more realistic settings, such as robotics, changing the sensors have costs - energy cost, delays, etc. These constraints are not mentioned or studied at all. \n - Neurips is a more theoretical conference, but this paper focuses on solving Active RL problems, which is mainly a robotics problem. So while the Atari/DMC benchmark adds some scientific value, the experimental section needs a more realistic Active RL benchmark to show practical value. The authors should show results on a robotics task, like manipulation or navigation as done in related work [1,2]. I am also open to discussing this point, but the authors would need to convince me that the Atari/DMC tasks are somehow reflective of actual Active-RL applications.\n\nThis is the most pressing issue for me, and I believe if addressed, will make this paper very compelling. If SUGARL can show good trends (outperform or competitive with full view baseline) in realistic applications, it will be a good method since joint training the sensory and motor policies together is nontrivial. \n\n## Moderate\n**Experiment section is messy.** It has the feeling of being cobbled together rather than presenting a unified analysis. There are many small experiments and their figures are all over the place. This makes it hard for the reader to understand key takeaways. For example, section 5.1 seems to have 5 \"mini\" experiments, each asking a different question. Some experiments seem more tangential (e.g. training for more steps) and can be moved to appendix.  While it's hard for me to tell exactly what needs to be fixed here, overall this section needs to be rewritten so the reader can easily get key takeaways. Can the authors attempt to make this section easier to read?\n\n\n**PVM is an implementation trick for 2D environments rather than a contribution.** Related to my major concern on using only artificial 2D environments - the authors propose Persistence of Vision memory, which stitches together multiple recent images into one. This is more of an implementation trick for 2D environments. Can we give baselines access to PVM, or remove PVM from SUGARL for comparison? I see there is already an ablation for PVM, but I would like to see this across all tasks. Alternatively, the authors can evaluate in a 3D environment (like robotics, navigation, etc.) and show a 3D version of PVM. Either way would make it more general.\n\n**SUGARL reward on non agent-centric tasks.**  The SUGARL reward rewards the sensory policy for choosing views with low action prediction error. However, will this still work in tasks where relevant parts of the image are unrelated to the action? For example, Pong and Pinball have moving small balls that are important to look at, but the balls give little information about the action.\n\n**Reproducibility.** No codebase or promise of code release of the Active-Gym benchmark or SUGARL algorithm.\n\n1. Cheng, Ricson, Arpit Agarwal, and Katerina Fragkiadaki. \"Reinforcement learning of active vision for manipulating objects under occlusions.\" _Conference on Robot Learning_. PMLR, 2018.\n2. Wang, Hanqing, et al. \"Active visual information gathering for vision-language navigation.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16. Springer International Publishing, 2020. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the effectiveness of our approach and providing thoughtful comments. Following the suggestions, we conducted addtional experiments on more realistic robot environments. \n\n**1\\. Active perception settings**\n\nFollowing the suggestion, we tested our algorithm on a more realistic robot control (simulated) environment, Robosuite [Zhu et al., 2020]. In this environment, the camera motion is controlled by 5-DoF control: relative (x, y, z, yaw, pitch), and is constrained by maximum linear and angular velocities. Under this constrained active perception setting, we confirm that SUGARL works the best. \n\nIn the Atari results we provided, we also test SUGARL with relative control (SUGARL(rel), Table 1(a) in the paper). This control only allows the view to move to nearby locations. From the experimental result we observe that it sometimes can outperform absolute control. \n\n**2\\. Robotics task**\n\nWe followed the suggestion and conducted new experiments on Robosuite. Results are available in the general response and Table 1 in the rebuttal PDF. In short, SUGARL achieves best performance on 3 of 5 challenging manipulation tasks (are all harder tasks), and outperforms hand-coded views most of the time on all tasks. \n\n**3\\. Organization of experiment section**\n\nThanks for the comment, we will improve this section accordingly. For example, having a list of questions we try to answer at the beginning of this section.\n\n**4\\. Contribution of PVM**\n\nPlease see our general response (2. Designs of PVMs). We conducted extra experiments on Robosuite where we make several new instantiations of PVM for this 3D case. The results are in Table 1 of the rebuttal PDF. We find the version for the 3D environment is effective in combining partial observations spatially and temporally. PVM is about finding an appropriate function to combine partial observations spatially and temporally, and we provide several versions of in these new experiments.\n\nFollowing the suggestion, we tested SUGARL w/o PVM on Robosuite and DMC (Tables 1, 2 in the rebuttal PDF). We confirm that PVM is an important component.\n\n**5\\. Non-agent centric tasks/sensory policy behavior**\n\nThe sensory policy is learned and will focus on the necessary part. It is not forced to pay attention to specific parts like surroundings. In the meantime, the presence of environmental reward encourages both policies to be task-oriented. So the sensory action won't overly satisfy the intrinsic reward.\n\nOne example of this is the visualization of Assault (the middle of Row 2) in the supplementary website available in the caption of Figure 6. The sensory policy focuses on the enemies (top) rather than the agent (bottom). In the visualization of Pong (Row 7), the sensory policy observes the ball many times. \n\n**6\\. Effects of shared reward function**\n\nThe final reward we use is the combination of environmental reward $r^{env}$ and our intrinsic reward $r^{sugarl}$. With the presence of environmental reward, the motor policy won’t overly satisfy intrinsic reward. The reward balance is the key and can be determined in a principled way from previous research [Hasselt et al., 2016, Henderson et al., 2018, Choi et al., 2019]. We verify its effectiveness and robustness across different visual settings in Atari and DMC. We didn’t observe a highly predictable motor policy from our current study.\n\n**7\\. Discussion on limitations**\n\nFollowing the suggestion, we provide a limitation section which we will incorporate to the revised paper.\n\nIn this work, we assume that completely independent sensory and motor actions are present in an embodied agent. But in a real-world case, the movement of the sensors may depend on the motor actions. For example, a fixed camera attached to the end-effector of a robot manipulator, or to a mobility robot. To address the potential dependence and conflicts between two policies in this case, extensions like voting or weighing across two actions to decide the final action may be required. The proposed algorithm also assumes a chance to adjust viewpoints at every step. This could be challenging for applications where the operational or latency costs for adjusting the sensors are high like remote control. To resolve this, additional penalties on sensory action and larger memorization capability are potentially needed. Last, the intrinsic reward currently only considers the accuracy of agent-centric prediction. Other incentives like gathering novel information or prediction accuracy over other objects in the environment can be further explored.\n\n**8\\. Reproducibility and resources**\n\nWe will open-source our approach and the benchmark, including the robotics environment.\nThe resources required are listed below, measured using a NVIDIA A5000 GPU. The speed is mostly bottlenecked by the underlying simulator (if only one thread is used), not our algorithm.\n| Environment | Algorithm    | Steps | RAM | GPU VRAM | Wall-clock Time/Task |\n|-------------|--------------|-------|-----|----------|----------------------|\n| Atari       | DQN          | 1M    | 18G | 1.3G     | 1.2 hrs              |\n| Atari       | SUGARL-DQN   | 1M    | 18G | 1.7G     | 1.6 hrs              |\n| DMC         | DrQv2        | 100K  | 18G | 2.5G     | 1.4 hrs              |\n| DMC         | SUGARL-DrQv2 | 100K  | 18G | 2.8G     | 1.7 hrs              |\n| Robosuite   | DrQv2        | 100K  | 54G | 3.9G     | 2.5 hrs              |\n| Robosuite   | SUGARL-DrQv2 | 100K  | 54G | 4.2G     | 3 hrs                |\n\nThank you for reading our responses.\n\n**References**\n- Henderson et al., Deep reinforcement learning that matters, AAAI 2018\n- Hasselt et al., Learning values across many orders of magnitude, NeurIPS 2016\n- Choi et al., Intrinsic motivation driven intuitive physics learning using deep reinforcement learning with intrinsic reward normalization, 2019\n- Zhu et al., robosuite: A modular simulation framework and benchmark for robot learning, 2020\n"}, {"review_id": "sDtI5JeCXc", "reviewer": "Reviewer_3Naa", "summary": "The paper introduces an RL setting where an agent take actions to simultaneously perform a task and control its vision. The authors design an algorithm for learning two policies for task performing and vision controlling. They test the algorithm on two domains: Atari and DMC. Results show that the proposed algorithm works better than baselines that do not intentionally control the vision. The authors conduct a comprehensive study to analyze the behavior of the algorithm.  ", "questions": "I do not have any questions. ", "limitations": "There is no limitations section. I suggest discussing the limitations of the simulated environments in modeling real-world scenarios, the drawbacks of RL in the face of rich information and action spaces. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The proposed setting is interesting. While it is not clear in the paper, I can envision potential significant impact.\n* The paper is very well-written. The methods are clearly presented and the results are thoroughly analyzed. \n* The experiment analyses are thoughtful and comprehensive. The behavior of the algorithm is dissected extensively, bringing valuable insights to the readers.   ", "weaknesses": "My concerns are about the high-level framing of the paper and the lack of comparison with highly relevant baselines:\n* The motivation of the setting is still unconvincing in terms of practicability. Scientifically, it is interesting to replicate human abilities to intentionally adapt their field of vision, but it is unclear why acquiring the same abilities would be pragmatically beneficial for an AI agent, compared with (i) observing all information or (ii) end-to-end-learned attentions. \n   * The paper seems to be missing a notion of \"cost\" (e.g., time budget, cognitive effort, amount of information), which makes it easier to motivate and compare different methods. For example, in figure 5(b), the method outperforms observing the full state, but it is unclear whether the gain is due to observing more pixels in general or having the ability to control the vision. A more fair comparison would fix a budget of pixels (or some other type of cost) for all methods. \n   *  The authors should include baselines with end-to-end-learned attention. Without any cost of gathering new information, one can design a baseline that scans the full state, and applies attention on top of that. With this baseline, I am currently not convinced why learning attention through RL would be more effective than through end-to-end optimization. \n* I suggest changing the framework to \"Active*Vision*-RL\", which fits better to the scope of the paper.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. We will address the concerns below.\n\n**1\\. The necessity of active perception**\n\nWe thank the reviewer for the constructive comments. In order to highlight the necessity of active perception and effectiveness of our approach. \n  - We ran new experiments on Robosuite [Zhu et al., 2020] where an active perception is naturally beneficial for manipulation. Our results show that active agents are outperforming hand-coded views most of the time. Please refer to our general response for the details of the experiments. \n  - We also compared our proposed approach with baselines with (i) observing all information (using an object detector) or (ii) end-to-end-learned attentions. SUGARL outperforms them. Please also refer to Tables 1 and 3 in our rebuttal PDF for detailed results. For the attention-based baseline, we notice there are several possible approaches for RL like [James et al., 2022, Tang et al., 2020, Wu et al., 2021]. We choose [Tang et al., 2020] because it is differentiable and with only a small amount of computation overhead for fair comparison.\n\n\n\n**2\\. Pixel budget**\n\nPlease see the table below for comparison on the number of observed pixels versus performance under Atari. In Atari and DMC, the number of actual pixels observed by the full observation model is 84x84=7056. For the foveal only settings, SUGARL uses 3x20x20=1200 (17.0%), 3x30x30=2700 (38.3%), 3x50x50=7500 (106%) pixels **at most**, where 3 is the introduced PVM length, and 20, 30, 50 are the foveal resolutions. We note that there are overlaps (because of stitching) between old and new observations, so the actual numbers will be smaller than those, especially for the 50x50 case. SUGARL models have lower pixel budgets than the full observation model in general. In Robosuite, all methods have the same pixel budget. We will add these comparisons to the paper.\n\n| Visual Settings                 | # Observed Pixels   | Percentage of a Full Obs. | Performance on Atari |\n|---------------------------------|---------------------|:--------------------------:|:----------------------:|\n| 20x20 foveal only               | <20x20x3=1200       | <17.0%                   | 0.475                |\n| 30x30 foveal only               | <30x30x3=2700       | <38.3%                   | 0.810                |\n| 50x50 foveal only               | <50x50x3=7500       | <106%                    | 0.805                |\n| 20x20 foveal + 20x20 peripheral | <20x20x3+20x20=1600 | <22.7%                   | 1.010                |\n| 30x30 foveal + 20x20 peripheral | <30x30x3+20x20=3100 | <43.9%                   | 1.116                |\n| 50x50 foveal + 20x20 peripheral | <50x50x3+20x20=7900 | <112%                    | 1.289                |\n| Full                            | 84x84=7056          | 100%                     | 1.000                |\n\n**3\\. Term ActiveVision-RL**\n\nThanks for the suggestion, we will follow it.\n\n**References**\n\n- James et al., Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation, RA-L 2022\n- Tang et al., Neuroevolution of Self-Interpretable Agents, GECCO 2020\n- Wu et al., Self-Supervised Attention-Aware Reinforcement Learning, AAAI 2021\n- Zhu et al., robosuite: A modular simulation framework and benchmark for robot learning, 2020\n\n"}, {"review_id": "1rp9M3m2B9", "reviewer": "Reviewer_Rg6w", "summary": "This paper studies an interesting setting where an RL agent needs to simultaneously decide how to act in the motor space and how to act to change the 2D observation space. Under this setting, the authors propose a new framework named SUGARL, mainly consisting of three technical contributions: \n- a two-branch joint learning framework;\n- a definition of the sensorimotor reward using inverse dynamics model\n- a technique to accumulate history observations named Persistence-of-Vision Memory\n\nThe authors conduct experiments on Atari with DQN and on DMControl with DrQv2, to show the effectiveness of their framework compared to Random View and Raster Scanning baselines.", "questions": "Please see `weaknesses` for questions.", "limitations": "The limitations of the environments are not fully mentioned in the conclusion section, which however are my major concern in this work. Other parts are good.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. An interetsting problem named Active RL is reformulated and well-studied. This problem has its real-world meaning and could lead to possibly large impact for the real-world applications of Visual RL.\n2. To drive for the better learning of the sensorimotor policy, a new intrinsic reward is defined based on IDM, which is well motivated and also effective in experiments.\n3. The evaluation looks extensive, conducted on two kinds of environments with both continous and discrete action space, and every component is well ablated.\n4. The learned sensorimotor policy gives relatively interesting visualizations, where the task-oriented objects are more focused.", "weaknesses": "1. **Environments are not matching the problem setting.** Although the proposed setting and the motivation example (i.e., robotic manipulation) are interesting to me, my major concern is the mismatch between the proposed setting and the experiment environments, where authors only focus on the 2D RL environments (DMC and Atari). Recent years have seen great advances in robotic simulation environments (e.g, MetaWorld [1], RoboSuite [2], ManiSkill [3], RLBench [4], ...), and it would be good to see the active RL setting is studied on these robotic environments, instead of the toy 2D environments such as video game.\n2. **Baselines are weak.** The presented experiment results show that SUGARL outperforms Random View and Raster Scanning, which are two random weak baselines that could not show the advantage of the proposed framework. It would be good if authors could devise some stronger baseline (such as using a pre-trained detection network [5] to get the target, or using some hand-crafted features like color and shape to get the target) and compare.\n\n[1] Yu, Tianhe, et al. \"Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.\" *Conference on robot learning*. PMLR, 2020.\n\n[2] Zhu, Yuke, et al. \"robosuite: A modular simulation framework and benchmark for robot learning.\" *arXiv preprint arXiv:2009.12293* (2020).\n\n[3] Mu, Tongzhou, et al. \"Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations.\" *arXiv preprint arXiv:2107.14483* (2021).\n\n[4] James, Stephen, et al. \"Rlbench: The robot learning benchmark & learning environment.\" *IEEE Robotics and Automation Letters* 5.2 (2020): 3019-3026.\n\n[5] https://github.com/facebookresearch/detectron2", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. Please find our response below.\n\n**1\\. Experiments on robotics tasks**\n\nFollowing the suggestion, we conducted new experiments on Robosuite [Zhu et al., 2020] and found that SUGARL performs best, especially on harder manipulator tasks. Please see our general response and Table 1 in our rebuttal PDF.\n\n**2\\. Baselines**\n\nWe follow the suggestion and introduce more SOTA baselines. Please also refer to our general response and Table 1 in our rebuttal PDF for details. We implement SOTA baselines including (a) DrQ with object detection using pre-trained DETR [Carion et al., 2020], and (b) DrQ with learned attention [Tang et al., 2020]. The former is our replication of a relevant approach from [Cheng et al., 2018], with a stronger object detection module and a stronger RL algorithm. We find that SUGARL outperforms both baselines by a large margin. \n\n**3\\. Discussion on limitations**\n\nFollowing the suggestion, we provide a limitation section which we will incorporate to the revised paper.\n\nIn this work, we assume that completely independent sensory and motor actions are present in an embodied agent. But in a real-world case, the movement of the sensors may depend on the motor actions. For example, a fixed camera attached to the end-effector of a robot manipulator, or to a mobility robot. To address the potential dependence and conflicts between two policies in this case, extensions like voting or weighing across two actions to decide the final action may be required. The proposed algorithm also assumes a chance to adjust viewpoints at every step. This could be challenging for applications where the operational or latency costs for adjusting the sensors are high like remote control. To resolve this, additional penalties on sensory action and larger memorization capability are potentially needed. Last, the intrinsic reward currently only considers the accuracy of agent-centric prediction. Other incentives like gathering novel information or prediction accuracy over other objects in the environment can be further explored.\n\nWe appreciate your consideration of our clarifications.\n\n**References**\n\n- Carion et al., End-to-End Object Detection with Transformers, ECCV 2020\n- Tang et al., Neuroevolution of Self-Interpretable Agents\n- Cheng et al., Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions, CoRL 2018\n- Zhu et al., robosuite: A modular simulation framework and benchmark for robot learning, 2020\n"}]

Author rebuttal:
**General response:**

We thank all reviewers for inspiring comments and questions. In this general response, we address the common concerns of more experiments on (1) robot manipulator tasks, (2) more baselines, and (3) more designs of proposed Persistence-of-Vision Memory (PVM). 

1. **New experiments on robot manipulator tasks**

    In addition to Atari and DMC, following the suggestions from the reviewers, we test SUGARL on Robosuite [Zhu et al., 2020], a simulated robotics environment.

    **Tasks**: We selected five of available tasks, namely block lifting (Lift), block stacking (Stack), nut assembling (NutAssembleSquare), door opening (Door), wiping the table (Wipe). The first two are easier compared to the later three. Example observations are available in Figure 1 in rebuttal PDF.

    **Camera / sensory action setting**: 
    Active RL agent is allowed to control the camera. We use a 5-DoF **continuous** control: relative (x, y, z, yaw, pitch). The maximum linear and angular velocities are constrained to 0.05/step and 5 degrees/step, respectively. For reference, the dimension of the table is 0.8x0.8. 

    **Results**:
    We compare baselines including RL with object detection (a replication of [Cheng et al., 2018]), learned attention [Tang et al., 2020], and standard RL with hand-coded views. Results are in Table 1 in the new rebuttal PDF. We confirm that our SUGARL works outperforms SOTA baselines all the time, and also outperforms the hand-coded views most of the time. Specifically, for the harder tasks including Wipe, Door, and NutAssemblySquare, SUGARL gets the best scores. 

2. **More SOTA baselines**

    Following the reviewer suggestion, we implement two SOTA baselines.
    - **DrQ w/ Object Detection**: Since [Cheng et al., 2018] requires goal-conditioning unlike our setting, we extend [Cheng et al., 2018] with a stronger object detector and RL algorithm. We try our best to replicate [Cheng et al., 2020] using a pre-trained object detector DETR [Carion et al., 2020] that provides object bounding boxes in addition to visual observation. Compared to the original version of [Cheng et al., 2018], the reimplemented method has (a) a stronger detector DETR and (b) DrQv2 which is an improved version of DDPG. We remove the goal conditioning in [Cheng et al., 2018].
    - **DrQ w/ End-to-End Attention**: We use [Tang et al., 2020] to perform an learnable attention. This approach divides the input image to patches and selects K patches with the highest attention scores for the policy input. We use it along with DrQv2 in order to perform a fair comparison.

    From the experimental results in Table 1 in the rebuttal PDF, we find that SUGARL outperforms them by a large margin. They perform similarly or slightly better than the Single Policy baseline. In Table 3 in the rebuttal PDF, we also find that SUGARL outperforms attention approach in Atari.

3. **Designs of PVMs**

    The Persistence-of-Vision Memory (PVM) is proposed to be a general framework, and what we present in the paper is one implementation of it (i.e., 2D-image stitching). Following the suggestion of the reviewers, we further provide more instantiations of it. We additionally implemented it using
    - **LSTM**: Each image is first encoded by CNN and fed into LSTM.
    - **3D Transformation + Stacking**: We use camera parameters to align pixels from different images to the current camera frame. Then the transformed images are simply stacked on the channel axis. CNN encodes the stacked images.
    - **3D Transformation + LSTM**: Similar 3D transformation as above, but using an LSTM to encode the images after going through CNN.

    We compare these new instantiations to a simple stacking PVM in Robosuite, and results are available in the first 4 rows of Table 1 in rebuttal PDF. We find that 3D Transformation + LSTM works the best, because it tackles spatial aligning and temporal merging together. LSTM also works well in general.

    We also test LSTM PVM on Atari to compare it with our originally proposed Stitching PVM. The results are available in Table 3 of the rebuttal PDF. We find that LSTM PVM does not outperform Stitching PVM because Stitching can combine observations exactly according to their spatial locations.


**Details of robosuite experiments**

  - **Hand-coded view baselines:** We select three hand-coded views that come with Robosuite -- **Front View**, **Agent View**, and **Eye-in-hand View**. These are used for the standard RL. Front View has the camera facing towards the robot. Agent View has the camera closer to the robot than Front View and more concentrated to the working area (e.g. table top or the door). The Eye-in-hand View is from the camera attached to the end effector of the robot. Front View and Agent View are static views while the Eye-in-hand View camera moves together with the end effector. Examples are available in Figure 1 in rebuttal PDF.

  - **Training:** Each task is trained for 100k steps. The size of the replay buffer is 100k. Each task is trained for 3 seeds, with 10 evaluation runs at the end of the training.

  - **Metrics:** We report the absolute IQM value from a total of 30 evaluations.

Thank you for reading our responses.

**References**
- Carion et al., End-to-End Object Detection with Transformers, ECCV 2020
- Cheng et al., Reinforcement Learning of Active Vision for Manipulating Objects under Occlusions, CoRL 2018
- Tang et al., Neuroevolution of Self-Interpretable Agents, GECCO 2020
- Zhu et al., robosuite: A modular simulation framework and benchmark for robot learning, 2020


Meta-review: This paper proposes a method for active reinforcement learning where two control policies are learned, one for motor actions and one for sensor actions. The motor and sensor policies are learned jointly by optimizing a combination of environmental and intrinsic rewards. In initial reviews, the primary concerned shared by the reviewers was  the evaluation tasks used. The authors addressed this concern in their rebuttal by performing additional experiments on Robotsuite. I encourage the authors to include these experiments in the final version as their inclusion increases the impact and value of the paper.
Decision: Accept (poster)