SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6: Results with different observation size and peripheral observation settings. The green bars are results from SUGARL. The red dashed lines stand for the DQN baseline trained on full observations using the same amount of data. We compare SUGARL against two dynamic views: Random View and Raster Scanning, and three static view baselines. In (b) with peripheral observation, we compare a baseline using peripheral observation only indicated by the cyan line.
(b) With peripheral observation
(c) Static policies comparison
the hand-coded views most of the time. Specifically, for the harder tasks including Wipe, Door, and NutAssemblySquare, SUGARL gets the best scores.
Designs of PVM We compare different instantiations of proposed PVMs including: Stacking: naively stacking multiple frames; LSTM: Each image is first encoded by CNN and fed into LSTM; 3D Transformation + LSTM: we use camera parameters to align pixels from different images to the current camera frame. Then an LSTM encodes the images after going through CNN. We find that 3D Transformation + LSTM works the best, because it tackles spatial aligning and temporal merging together. LSTM also works well in general.
5.2 2D Benchmark Results
We evaluate the policy learning on Atari under two primary visual settings: with and without peripheral observation. In each visual setting we explore three sizes of observable area (set equivalent to foveal resolution): 20x20, 30x30, and 50x50. In with peripheral observation setting, the peripheral resolution is set to 20x20 for all tasks. We use DQN [65]-based SUGARL (SUGARL-DQN) and compare it against two variants by replacing the learnable sensory policy in SUGARL with: Random View and Raster Scanning. Random View always uniformly samples from all possible crops. Raster Scanning uses a pre-designed sensory policy that chooses observable areas from left to right and top to down sequentially. Raster Scanning yields relatively stable observation patterns and provides maximum information under PVM. We also provide a DQN baseline trained on full observations (84x84) as a soft oracle. In with peripheral observation settings, another baseline trained on peripheral observation only (20x20) is compared as a soft lower bound.
In Figure 6a and 6b, we find that SUGARL performs the best in all settings, showing that SUGARL learns effective sensory and motor policies jointly. More importantly, SUGARL with peripheral observation achieves higher overall scores  $(+0.01\sim 0.2)$  than the full observation baselines. In details, SUGARL gains higher scores than the full observation baseline in 13 out of 26 games with  $50\times 50$  foveal resolution (details are available in the Appendix). This finding suggests the untapped potential of ActiveVision-RL agents which leverage partial observations better than full observations. By actively selecting views, the agent can filter out extraneous information and concentrate on task-centric information.
Compare against Static Sensory Policies We also examine baselines with static sensory policies, which consistently select one region to observe throughout all steps. The advantage of this type baseline lies in the stability of observation. We select 3 regions: Center, Upper Left, and Bottom Right, and compare them against SUGARL in environment w/o peripheral observation. As shown in Figure 6c, we observe that SUGARL still surpasses all three static policies. The performance gaps between SUGARL and Center and Bottom Right are relatively small when the observation size is larger (50x50), as the most valuable information is typically found at these locations in Atari
Table 2: Evaluation results on different conditions and algorithm backbones
(a) Action modeling
Model	20	30	50
SUGARL (abs)	0.475	0.810	0.805
SUGARL (rel)	0.367	0.745	0.945
Single Policy	0.132	0.222	0.171
(b) Train more steps
Steps	Model	20	30	50
1M	SUGARL	0.475	0.810	0.805
Single Policy	0.132	0.222	0.171
5M	SUGARL	1.170	1.121	1.553
Single Policy	0.332	0.640	1.145
(c) SUGARL with SAC
Model	20	30	50
SUGARL	0.424	0.730	0.785
SUGARL w/o r-sugar1	0.300	0.307	0.504
SAC-raster scanning	0.117	0.195	0.136
SAC-random view	0.155	0.104	0.134
(d) Different PVMs
Model	20	30	50
Stitching PVM	0.475	0.815	0.810
LSTM PVM	0.397	0.448	0.470
Figure 7: Examples of learned sensory policies from SUGARL-DQN. Top: frequency heat maps of pixels being observed. Bottom: sequences of observations showing fixation-like and tracking-like behaviors. More videos are available at this link
environments. Static observation is therefore quite beneficial for decision-making. However, as the observation size decreases, an active policy becomes essential for identifying optimal observations, leading SUGARL to outperform others by a significant margin.
Sensory-Motor Action Spaces Modeling SUGARL models the sensory and motor action spaces separately inspired by the human abilities. An alternative approach is jointly modeling two action spaces  $\mathcal{A} = \mathcal{A}^s \times \mathcal{A}^o$  and learning one single action policy, with environmental reward only (referred as Single Policy). We compare Single Policy and display the results in Table 2a, which reveal that modeling one large action space fails to learn the policy properly due to the expanded action space. Additionally, we examine two types of sensory action spaces: absolute (abs) and relative (rel). Absolute modeling allows the view to jump across the entire space while relative modeling restricts the view to moving only to the neighboring options at each step. Results in Table 2a indicate that the absolute modeling performs better in smaller observable regions, as it can select the desired view more quickly than relative modeling. In contrast, relative modeling demonstrates better performance in larger observable region setting as it produces more stable observations across steps.
Training More Steps We subsequently explore SUGARL performance when trained with more environmental steps, comparing outcomes between 1M and 5M steps. Results in Table 2b confirm that SUGARL continuous to improve with more steps and consistently outperforms the single policy baseline, suggesting that it does not merely stagnate at a trivial policy. However, due to the challenges posed by limited observability, the learning still proceeds slower than for the agent with full observations, which achieves the score 4.106 at 5M steps.
Generalization of SUGARL We apply SUGARL to Soft Actor-Critic (SAC) [36] and evaluate its performance on environments without peripheral observation. As before, we compare it with Random View, Raster Scanning, and SUGARL without intrinsic reward. According to Table 2c, we find that SUGARL-SAC outperforms the naive version without the guidance of  $r^{\text{sugar}}$ , further emphasizing the significance of our method. Moreover, SUGARL-SAC also surpasses the random view and raster scanning policies. However, when employing max-entropy-based methods like SAC, it is necessary to reduce the weight of the entropy term to ensure consistency in the sensory policy. We will further discuss it in Section 5.3 based on the analysis of the learned sensory policy behavior.
DMC Tasks We also evaluate SUGARL on DMC [97] tasks based on DrQ [108], an improved version of DDPG [100]. We use six environments in DMC [97]: ball_in_cup-catch, cartpole-swingup, cheetah-run, dogfetch, fish-swim, and walker-walk. Three foveal observation resolutions (20x20, 30x30, and 50x50) without peripheral observation are explored. We use relative control for sensory action and decide the discrete sensory options
Table 3: SUGARL on DMC
Model	20	30	50
SUGARL-DrQ	0.686	0.717	1.052
DrQ-Single Policy	0.540	0.570	0.776
DrQ-Raster Scanning	0.609	0.566	0.913
DrQ-Random View	0.569	0.591	0.768
SUGARL-DrQ w/o PVM	0.672	0.620	0.930
SUGARL w/o Joint Learning	0.377	0.446	0.355
by thresholding the continuous output. We compare our approach with baselines with joint modeling of sensory and motor action spaces (Single Policy), Raster Scanning, and Random View. Results are shown in Table 3. SUGARL outperforms all baselines in three foveal settings in DMC, showing it is also applicable to continuous control tasks.
5.3 Sensory Policy Analysis
Sensory Policy Patterns In Figure 7, we present examples of learned sensory policies from SUGARL-DQN in four Atari games, in settings w/o peripheral observations. These policies are visualized as heat maps based on the frequency of observed pixels. We discover that the sensory policies learn both fixation and movement (similar to tracking) behaviours [12, 110] depending on the specific task requirements. In the first two examples, the sensory policy tends to concentrate on fixed regions. In *battle-zone*, the policy learns to focus on the regions where enemies appear and the fixed front sight needed for accurate firing. By contrast, in highly dynamic environments like boxing and freeway, the sensory policies tend to observe broader areas in order to get timely observations. Though not being a perfect tracker, the sensory policy learns to track the agent or the object of interest in these two environments, demonstrating the learned capability akin to humans' Smooth Pursuit Movements. Recorded videos for entire episodes are available at our project page.
Sensory Policy Distribution We quantitatively assess the distributions of learned sensory policies. There are 16 sensory actions, i.e. the observable area options in our setup (Figure 5). We compare the distributions against uniform distribution using KL-divergence, across 26 games x 10 eval runs x 5 seeds. The resulting histogram are shown in Figure 8. We observe that the learned policies consistently deviate from the uniform distribution, suggesting that sensory policies prefer specific regions in general. The high
peak at the high KL end supports the "fixation" behavior identified in the previous analysis. As the observation size increases, the divergence distribution shifts towards smaller KL end, while remaining  $>0.5$  for all policies. This trend indicates that with larger observable sizes, the policy does not need to adjust its attention frequently, corroborating the benefit of using relative control shown in Table 2a.
Figure 8: KL divergence distributions of learned sensory policies.
Pitfalls on Max-Entropy Based Methods In the previous analysis, we both qualitatively and quantitatively demonstrate that sensory policies are not uniformly dispersed across the entire sensory action space. This observation implies that sensory policy should exercise caution when adopting the max-entropy-based methods. We conduct an experiment on varying the usage of  $\alpha$ , the entropy coefficient in SAC in all three foveal observation size settings. Results in Table 4
show that simply disabling autotune and setting a small value to the sensory policy's  $\alpha$  improves. This finding tells that max-entropy may not be a suitable assumption in modeling sensory policies.
Table 4: Varing  $\alpha$  of SUGARL-SAC.
Model	20	30	50
autotune α	0.271	0.358	0.444
fixed-α = 0.2	0.424	0.730	0.785
We conduct ablation studies in the setting without peripheral observation and with  $50 \times 50$  observation size. Five crucial design components are investigated to validate their effectiveness by incrementally adding or removing them from the full model. The results are presented in Table 5. From the results, we demonstrate the importance of penalizing the agent for inaccurate self-understanding predictions rather than rewarding accurate predictions ( $r^{\text{sugar}}(\text{positive}) \to r^{\text{sugar}}(\text{negative})$ ). By imposing penalties, the maximum return is bounded by the original maximum possible return per episode, allowing motor and sensory policies to better coordinate each other and achieve optimal task performance. Reward balance significantly improves the policy, indicating its effectiveness in coordinating two policies as well. PVM also considerably enhances the algorithm by increasing the effective observable area as expected.
Table 5: Ablation study results based on DQN 50x50 foveal res w/o peripheral observation. The blank fields means there are no such modifications for that model.
Model	r\( ^{\text{sugar1}} \)	Joint Learning	PVM	Reward Balance (β)	IQM
Random View	✓	0.367
SUGARL =	Base RL algorithm	-
+Naive positive intrinsic reward	positive	0.281
+Joint learning	positive	✓	0.322
Positive → negative r\( ^{\text{sugar1}} \)	negative	✓	0.360
+PVM	negative	✓	✓	0.423
+Reward Balance	negative	✓	✓	0.805
SUGARL w/o r\( ^{\text{sugar1}} \)	✓	✓	0.421
SUGARL w/o r\( ^{\text{sugar1}} \) and w/o PVM	✓	0.231
Active Learning is the concept that an agent decides which data are taken into its learning and may ask for external information in comparison to fitting a fixed data distribution [1, 6, 10, 21, 22, 31, 46, 49, 55, 57, 60, 64, 66, 73, 81, 88, 90, 101, 105, 113]. Active Vision focuses on continuously acquiring new visual observations that is helpful for the vision task like object classification, recognition and detection [4, 5, 7, 8, 18, 28, 29, 48, 63, 98, 106, 109], segmentation [13, 58, 70], and action recognition [45]. The active vision is usually investigated under a robot vision scenario that a robot moves around in a scene. However, the policy is usually not required to accomplish a task with physical interactions such as manipulating objects compared to reinforcement learning.
Active Reinforcement Learning (Active-RL), at a high level, is that the agent is allowed to actively gather new perceptual information of interest simultaneously through an RL task, which can also be called active perception [102]. The extra perceptual information could be reward signal [3, 23, 27, 50, 56, 62], visual observations from new viewpoints [33, 54, 67, 87], other input modalities [16], and language instructions [19, 68, 69, 94]. Though these work may not explicitly use the term Active-RL, we find that they can be uniformly organized in the general Active-RL formulation and we coin the term here. In our work, we study the ActiveVision-RL task in a limited visual observability environment, where at each step the agent is only able to partially observe the environment. The agent should actively seek the optimal observation at each step. Therefore, our setting is more close to [32, 33] and Active Vision problems, unlike research incorporating attention-like inductive bias given a full observation [34, 42, 43, 79, 91, 93, 104]. The ActiveVision-RL agent must learn an observation selection policy, called sensory policy, to effectively choose the optimal partial observation for executing the task-specific policy (motor policy). The unique challenge for ActiveVision-RL is the coordination between sensory and motor policies given there mutual influence. In recent works, the sensory policy can be either trained in the task-agnostic way [33] with enormous exploration data, or trained jointly with the task [32] with naive environmental reward only. In this work we investigate the joint learning case because of the high cost and availability concern of pre-training tasks [33].
Robot Learning with View Changes Viewpoint changes and gaps in visual observations are the common challenges for robot learning [40, 53, 77, 84, 92, 112], especially for the embodied agents that uses its first-person view [30, 35, 80]. To address those challenges, previous works proposed to map visual observation from different viewpoints to a common representation space by contrastive encoding [26, 82, 83] or build implicit neural representations [53, 107]. In many first-person view tasks, the viewpoint control is usually modeled together with the motor action like manipulation and movement [30, 80]. In contrast, in our ActiveVision-RL setting, we explore the case where the agent can choose where to observe independently to the motor action inspired by the humans' ability.
7 Limitations
In this work, we assume that completely independent sensory and motor actions are present in an embodied agent. But in a real-world case, the movement of the sensors may depend on the motor actions. For example, a fixed camera attached to the end-effector of a robot manipulator, or to a mobility robot. To address the potential dependence and conflicts between two policies in this case, extensions like voting or weighing across two actions to decide the final action may be required. The proposed algorithm also assumes a chance to adjust viewpoints at every step. This could be challenging for applications where the operational or latency costs for adjusting the sensors are high like remote control. To resolve this, additional penalties on sensory action and larger memorization capability are potentially needed. Last, the intrinsic reward currently only considers the accuracy of agent-centric prediction. Other incentives like gathering novel information or prediction accuracy over other objects in the environment can be further explored.
We present SUGARL, a framework based on existed RL algorithms to jointly learn sensory and motor policies through the ActiveVision-RL task. In SUGARL, an intrinsic reward determined by sensorimotor understanding effectively guides the learning of two policies. Our framework is validated in both 3D and 2D benchmarks with different visual observability settings. Through the analysis on the learned sensory policy, it shows impressive active vision skills by analogy with human's fixation and tracking that benefit the overall policy learning. Our work paves the initial way towards reinforcement learning using active agents for open-world tasks.
We appreciate the fruitful discussions on the methodology with Xiang Li and Wensheng Cheng, on experimental designs with Kumara Kahatapitiya and Ryan Burgert. We also appreciate the inspiring feedbacks from Kanchana Ranasinghe and Varun Belagali.
[1] S. Agarwal, H. Arora, S. Anand, and C. Arora. Contextual diversity for active learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 137-153. Springer, 2020.
[2] I. Akkaya, M. Andrychowicz, M. Chogiej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
[3] R. Akrour, M. Schoenauer, and M. Sebag. April: Active preference learning-based reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pages 116-131. Springer, 2012.
[4] A. Andreopoulos and J. K. Tsotsos. A theory of active object localization. In 2009 IEEE 12th International Conference on Computer Vision, pages 903-910. IEEE, 2009.
[5] A. Andreopoulos and J. K. Tsotsos. A computational learning theory of active object recognition under uncertainty. International journal of computer vision, 101:95-142, 2013.
[6] J. T. Ash, C. Zhang, A. Krishnamurthy, J. Langford, and A. Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations, 2020.
[7] N. Atanasov, B. Sankaran, J. Le Ny, G. J. Pappas, and K. Daniilidis. Nonmyopic view planning for active object classification and pose estimation. IEEE Transactions on Robotics, 30(5): 1078-1090, 2014.
[8] R. Bajcsy, Y. Aloimonos, and J. K. Tsotsos. Revisiting active perception. Autonomous Robots, 42:177-196, 2018.
[9] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, jun 2013.
[10] W. H. Beluch, T. Genewein, A. Nurnberger, and J. M. Kohler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9368-9377, 2018.
[11] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
[12] A. Borji and L. Itti. State-of-the-art in visual attention modeling. IEEE transactions on pattern analysis and machine intelligence, 35(1):185-207, 2012.
[13] A. Casanova, P. O. Pinheiro, N. Rostamzadeh, and C. J. Pal. Reinforced active learning for image segmentation. arXiv preprint arXiv:2002.06583, 2020.
[14] E. Cetin, P. J. Ball, S. Roberts, and O. Celiktutan. Stabilizing off-policy deep reinforcement learning from pixels. In International Conference on Machine Learning, pages 2784–2810. PMLR, 2022.
[15] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems (NeurIPS), Dec. 2021.
[104] H. Wu, K. Khetarpal, and D. Precup. Self-supervised attention-aware reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10311-10319, 2021.
[105] Y. Xie, H. Lu, J. Yan, X. Yang, M. Tomizuka, and W. Zhan. Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm. arXiv preprint arXiv:2303.14382, 2023.
[106] J. Yang, Z. Ren, M. Xu, X. Chen, D. J. Crandall, D. Parikh, and D. Batra. Embodied amodal recognition: Learning to move to perceive objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2040–2050, 2019.
[107] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control. In Conference on Computer Vision and Pattern Recognition, 2023.
[108] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645, 2021.
[109] T. Yuan, F. Wan, M. Fu, J. Liu, S. Xu, X. Ji, and Q. Ye. Multiple instance active learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5330-5339, 2021.
[110] G. Zelinsky, W. Zhang, B. Yu, X. Chen, and D. Samaras. The role of top-down and bottom-up processes in guiding eye movements during visual search. Advances in neural information processing systems, 18, 2005.
[111] T. Zhang, Y. Li, C. Wang, G. Xie, and Z. Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference on Machine Learning, pages 12491–12500. PMLR, 2021.
[112] A. Zhou, M. J. Kim, L. Wang, P. Florence, and C. Finn. Nerf in the palm of your hand: Corrective augmentation for robotics via novel-view synthesis. arXiv preprint arXiv:2301.08556, 2023.
[113] J.-J. Zhu and J. Bento. Generative adversarial active learning. arXiv preprint arXiv:1702.07956, 2017.
[114] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martin, A. Joshi, S. Nasiriany, and Y. Zhu. robotsuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293, 2020.
A.1 Learning Curves
Figure 9: Learning curves of 26 Atari games, under the setting of  $50 \times 50$  foveal observation size and  $20 \times 20$  peripheral observation.
Figure 10: Learning curves of 26 Atari games, under the setting of  $30 \times 30$  foveal observation size and  $20 \times 20$  peripheral observation.
A.2 Hyper-parameter Settings
Figure 11: Learning curves of 26 Atari games, under the setting of  $20 \times 20$  foveal observation size and  $20 \times 20$  peripheral observation.
Figure 12: Learning curves of 26 Atari games, under the setting of  $50 \times 50$  foveal observation size and w/o peripheral observation.
Figure 13: Learning curves of 26 Atari games, under the setting of  $30 \times 30$  foveal observation size and w/o peripheral observation.
Figure 14: Learning curves of 26 Atari games, under the setting of  $20 \times 20$  foveal observation size and w/o peripheral observation.
Figure 15: Learning curves of 6 DMC environments, under the setting of  $50 \times 50$  foveal observation size and w/o peripheral observation.
Figure 16: Learning curves of 6 DMC environments, under the setting of  $30 \times 30$  foveal observation size and w/o peripheral observation.
Figure 17: Learning curves of 6 DMC environments, under the setting of  $20\mathrm{x}20$  foveal observation size and w/o peripheral observation.