SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinghuan Shang Michael S. Ryoo  
Department of Computer Science, Stony Brook University  
{jishang, mryoo}@cs.stonybrook.edu
In this work, we investigate Active Vision Reinforcement Learning (ActiveVision-RL), where an embodied agent simultaneously learns action policy for the task while also controlling its visual observations in partially observable environments. We denote the former as motor policy and the latter as sensory policy. For example, humans solve real world tasks by hand manipulation (motor policy) together with eye movements (sensory policy). ActiveVision-RL poses challenges on coordinating two policies given their mutual influence. We propose SUGARL, Sensorimotor Understanding Guided Active Reinforcement Learning, a framework that models motor and sensory policies separately, but jointly learns them using with an intrinsic sensorimotor reward. This learnable reward is assigned by sensorimotor reward module, incentivizes the sensory policy to select observations that are optimal to infer its own motor action, inspired by the sensorimotor stage of humans. Through a series of experiments, we show the effectiveness of our method across a range of observability conditions and its adaptability to existed RL algorithms. The sensory policies learned through our method are observed to exhibit effective active vision strategies.
Although Reinforcement Learning (RL) has demonstrated success across challenging tasks and games in both simulated and real environments [2, 9, 11, 89, 97], the observation spaces for visual RL tasks are typically predefined to offer the most advantageous views based on prior knowledge and can not be actively adjusted by the agent itself. For instance, table-top robot manipulators often utilize a fixed overhead camera view [51]. While such fixed viewpoints can potentially stabilize the training of an image feature encoder [14], this form of perception is different from humans who actively adjust their perception system to finish the task, e.g. eye movements [24]. The absence of active visual perception poses challenges on learning in highly dynamic environments [59, 61], open-world tasks [30] and partially observable environments with occlusions, limited field-of-views, or multiple view angles [40, 83].
We study Active Reinforcement Learning (Active-RL) [102], the RL process that allows the embodied agent to actively acquire new perceptual information in contrast to the standard RL, where the new information could be reward signals [3, 23, 27, 50, 56, 62], visual observations [32, 33], and other forms. Specifically, we are interested in visual Active-RL tasks, i.e. ActiveVision-RL, that an agent controls its own views of visual observation, in an environment with limited visual observability [33]. Therefore, the goal of ActiveVision-RL is to learn two policies that still maximize the task return: the motor policy to finish the task and the sensory policy to control the observation.
Our project page, code, and library are available at this link
Figure 1: ActiveVision-RL with limited visual observability in comparison with standard RL, with exemplary process in Atari game *Boxing*. Red arrows stand for additional relationships considered in ActiveVision-RL. In the example on the right, the highlighted regions are the actual observations visible to the agent at each step. The rest of the pixels are not visible to the agent.
ActiveVision-RL tasks present a considerable challenge due to the coordination between motor and sensory policies, given their mutual influence [102]. The motor policy requires clear visual observation for decision-making, while the sensory policy should adapt accordingly to the motor action. Depending on the sensory policy, transitioning to a new view could either aid or hinder the motor policy learning [14, 40, 102]. One notable impediment is the perceptual aliasing mentioned by Whitehead and Ballard [102]. An optimal strategy for sensory policy should be incorporating crucial visual information while eliminating any distractions. In the real world, humans disentangle their sensory actions, such as eye movements, from their motor actions, such as manipulation, and subsequently learn to coordinate them [59, 61]. Despite being modeled separately, these two action policies and the coordination can be learned jointly through the interaction during sensorimotor stage [25, 71, 72, 103].
Taking inspiration from human capabilities, we propose SUGARL: Sensorimotor Understanding Guided Active Reinforcement Learning, an Active-RL framework designed to jointly learn sensory and motor policies by maximizing extra intrinsic sensorimotor reward together with environmental reward. We model the ActiveVision-RL agent with separate sensory and motor policies by extending existing RL algorithms with two policy/value branches. Inspired by sensorimotor stage [71, 72, 103], we use the intrinsic sensorimotor reward to guide the joint learning of two policies, imposing penalties on the agent for selecting sub-optimal observations. We use a learned sensorimotor reward module to assign the intrinsic reward. The module is trained using inverse dynamics prediction task [52, 95], with the same experiences as policy learning without additional data or pre-training.
In our experiments, we use modified Atari [9] and DeepMind Control suite (DMC) [97] with limited observability to comprehensively evaluate our proposed method. We also test on Robosuite tasks to demonstrate the effectiveness of active agent in 3D manipulation. Through the challenging benchmarks, we experimentally show that SUGARL is an effective and generic approach for ActiveRL with minimum modification on top of existed RL algorithms. The learned sensory policy also exhibit active vision skills by analogy with humans' fixation and tracking.
2 Active Vision Reinforcement Learning Settings
Consider a vanilla RL setting based on a Markov Decision Process (MDP) described by  $(S, \mathcal{A}, r, P, \gamma)$ , where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $r$  is the reward function,  $P$  describes state transition which is unknown and  $\gamma$  is the discount factor. In this work, we study ActiveVision-RL under limited visual observability, described by  $(S, \mathcal{O}, \mathcal{A}^s, \mathcal{A}^o, r, P, \gamma)$ , as shown in Figure 1.  $\mathcal{O}$  is the actual partial observation space the agent perceives. In particular, we are interested in visual tasks, so each observation  $\mathbf{o}$  is an image contains partial information of an environmental state  $\mathbf{s}$ , like an image crop in 2D space or a photo from a viewpoint in 3D space. To emulate the human ability, there are two action spaces for the agent in Active-RL formulation.  $\mathcal{A}^s$  is the motor action space that causes state change  $p(\mathbf{s}'|\mathbf{s}, \mathbf{a}^s)$ .  $\mathcal{A}^o$  is the sensory action space that only changes the observation of an environmental state  $p(\mathbf{o}|\mathbf{s}, \mathbf{a}^o)$ . In this setting, the agent needs to take  $(\mathbf{a}^s, \mathbf{a}^o)$  for each step, based on observation(s) only. An example is shown in Figure 1.
Our goal is to learn the motor and sensory action policies  $(\pi^s,\pi^o)$  that still maximize the return  $\sum r_t$ . Note that the agent is never exposed to the full environmental state s. Both policies are completely based on the partial observations:  $\mathbf{a}^s = \pi^s (\cdot |\mathbf{o})$ ,  $\mathbf{a}^o = \pi^o (\cdot |\mathbf{o})$ . Therefore the overall policy learning is challenging due to the limited information per step and the non-stationary observations.
Original Algorithm
SUGARL
Figure 2: Overview of SUGARL and the comparison with original RL algorithm formulation. SUGARL introduces an extra sensory policy head, and jointly learns two policies together with the extra sensorimotor reward. We use the formulation of SAC [36] as an example. We introduce sensorimotor reward module to assign the reward. The reward indicates the quality of the sensory policy through the prediction task. The sensorimotor reward module is trained independently to the policies by action prediction error.
3 SUGARL: Sensorimotor Understanding Guided Active-RL
3.1 Active-RL Algorithm with Sensorimotor Understanding
We implement Active-RL algorithms based on the normal vision-based RL algorithms with simple modifications regarding separated motor and sensory policies  $(\pi^s,\pi^o)$ , and the sensorimotor reward  $r^{\mathrm{suar}}$ . We use DQN [65] and SAC [36] as examples to show the modifications are generally applicable. The example diagram of SAC is shown in Figure 2. We first introduce the policy then describe the sensorimotor reward in Section 3.2
Network Architecture The architectural modification is branching an additional head for sensory policy, and both policies share a visual encoder stem. For DQN [65], two heads output  $Q^{s}$ ,  $Q^{o}$  for each policy respectively. This allows the algorithm to select  $\mathbf{a}^s = \arg \max_{\mathbf{a}^s} Q^s$  and  $\mathbf{a}^o = \arg \max_{\mathbf{a}^o} Q^o$  for each step. Similarly for SAC [36], the value and actor networks also have two separate heads for motor and sensory policies. The example in the form of SAC is in Figure 2.
Joint Learning of Motor and Sensory Policies Though two types of actions are individually selected or sampled from network outputs, we find that the joint learning of two policies benefits the whole learning [41, 111]. The joint learning here means both policies are trained using a shared reward function. Otherwise, the sensory policy usually fails to learn with intrinsic reward signal only. Below we give the formal losses for DQN and SAC.
For DQN, we take the sum of Q-values from two policies  $Q = Q^{s} + Q^{o}$  and train both heads jointly. The loss is the following where we indicate our modifications by blue:
$$
\begin{array}{l} \mathcal {L} _ {i} ^ {Q} \left(\theta_ {i}\right) = \mathbb {E} _ {\left(\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {s}, \mathbf {a} _ {t} ^ {o}\right) \sim \mathcal {D}} \left[ \left(y _ {i} - \left(Q _ {\theta_ {i}} ^ {s} \left(\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {s}\right) + Q _ {\theta_ {i}} ^ {o} \left(\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {o}\right)\right)\right) ^ {2} \right] \\ y _ {i} = \mathbb {E} _ {\mathbf {o} _ {t + 1}} \left[ r _ {t} ^ {\mathrm {e n v}} + \beta r _ {t} ^ {\mathrm {s u g a r l}} + \gamma \left(\max  _ {\mathbf {a} _ {t + 1} ^ {s}} Q _ {\theta_ {i - 1}} ^ {s} (\mathbf {o} _ {t + 1}, \mathbf {a} _ {t + 1} ^ {s}) + \max  _ {\mathbf {a} _ {t + 1} ^ {o}} Q _ {\theta_ {i - 1}} ^ {o} (\mathbf {o} _ {t + 1}, \mathbf {a} _ {t + 1} ^ {o})\right) \right], \\ \end{array}
$$
where  $L^Q$  is the loss for Q-networks,  $\theta$  stands for the parameters of both heads and the encoder stem,  $i$  is the iteration, and  $\mathcal{D}$  is the replay buffer.  $\beta r_t^{\mathrm{suar}}$  is the extra sensorimotor reward with balancing scale which will be described in Section 3.2.
For SAC, we do the joint learning similarly. The soft-Q loss is in the similar form of above DQN which is omitted for simplicity. The soft-value loss  $L^V$  and is
$$
\begin{array}{l} \mathcal {L} ^ {V} (\psi) = \mathbb {E} _ {\mathbf {o} _ {t} \sim \mathcal {D}} \left[ \frac {1}{2} \left(V _ {\psi} ^ {s} (o _ {t}) + V _ {\psi} ^ {o} (o _ {t}) - \right. \right. \\ \left. \mathbb {E} _ {\mathbf {a} _ {t} ^ {s} \sim \pi_ {\phi} ^ {s}, \mathbf {a} _ {t} ^ {o} \sim \pi_ {\phi} ^ {o}} \left[ Q _ {\psi} ^ {s} (\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {s}) + Q _ {\psi} ^ {o} (\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {o}) - \log \pi_ {\phi} ^ {s} (\mathbf {a} _ {t} ^ {s} | o _ {t}) - \log \pi_ {\phi} ^ {o} (\mathbf {a} _ {t} ^ {o} | o _ {t}) \right]\right) ^ {2} \left. \right], \\ \end{array}
$$
and the actor loss  $\mathcal{L}^{\pi}$  is
$$
\mathcal {L} ^ {\pi} (\phi) = \mathbb {E} _ {\mathbf {o} _ {t} \sim \mathcal {D}} \left[ \log \pi_ {\phi} ^ {s} (\mathbf {a} _ {t} ^ {s} | \mathbf {o} _ {t}) + \log \pi_ {\phi} ^ {o} (\mathbf {a} _ {t} ^ {o} | \mathbf {o} _ {t}) - Q _ {\psi} ^ {s} (\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {s}) - Q _ {\psi} ^ {o} (\mathbf {o} _ {t}, \mathbf {a} _ {t} ^ {o}) \right],
$$
where  $\psi, \phi$  are parameters for the critic and the actor respectively, and reparameterization is omitted for clearness.
3.2 Sensorimotor Reward
The motor and sensory policies are jointly trained using a shared reward function, which is the combination of environmental reward and our sensorimotor reward. We first introduce the assignment of sensorimotor reward and then describe the combination of two rewards.
The sensorimotor reward is assigned by the sensorimotor reward module  $u_{\xi}(\cdot)$ . The module is trained to have the sensorimotor understanding, and is used to indicate the goodness of the sensory policy, tackling inspiration of human sensorimotor learning [72]. The way we obtain such reward module is similar to learning an inverse dynamics model [95]. Given a transition  $(\mathbf{o}_t, \mathbf{a}_t^s, \mathbf{o}_{t+1})$ , the module predicts the motor action  $\mathbf{a}^s$  only, based on an observation transition tuple  $(\mathbf{o}_t, \mathbf{o}_{t+1})$ . When the module is (nearly) fully trained, the higher prediction error indicates the worse quality of visual observations. For example, if the agent is absent from observations, it is hard to infer the motor action. Such sub-optimal observations also confuse agent's motor policy. Since the sensory policy selects those visual observations, the quality of visual observations is tied to the sensory policy. As a result, we can employ the negative error of action prediction as the sensorimotor reward:
$$
r _ {t} ^ {\text {s u g a r l}} = - \left(1 - p \left(\mathbf {a} _ {t} ^ {s} \mid \mathbf {o} _ {t}, \mathbf {o} _ {t + 1}; u _ {\xi}\right)\right). \tag {1}
$$
This non-positive intrinsic reward penalizes the sensory policy for selecting sub-optimal views that do not contribute to the accuracy of action prediction and confuse motor policy learning. In Section 5.4 we show that naive positive rewarding does not guide the policy well. Note that the reward is less noisy when the module is fully trained. However, it is not harmful for being noisy at the early stage of training, as the noisy signal may encourage the exploration of policies. We use the sensorimotor reward though the whole learning.
The sensorimotor reward module is implemented by an independent neural network. The loss is a simple prediction error:
$$
\mathcal {L} ^ {u} (\xi) = \mathbb {E} _ {\mathbf {o} _ {t}, \mathbf {o} _ {t + 1}, \mathbf {a} _ {t} ^ {s} \sim \mathcal {D}} \left[ \operatorname {E r r o r} \left(\mathbf {a} _ {t} ^ {s} - u _ {\xi} \left(\mathbf {o} _ {t}, \mathbf {o} _ {t + 1}\right)\right) \right], \tag {2}
$$
where the  $\mathrm{Error}(\cdot)$  can be a cross-entropy loss for discrete action space or L2 loss for continuous action space. Though being implemented and optimized separately, the sensorimotor reward module uses the same experience data as policy learning, with no extra data or prior knowledge introduced.
Combining Sensorimotor Reward and Balancing The sensorimotor reward  $r^{\mathrm{suagrl}}$  is added densely on a per-step basis, on top of the environmental reward  $r^{\mathrm{env}}$  in a balanced form:
$$
r _ {t} = r _ {t} ^ {\text {e n v}} + \beta r _ {t} ^ {\text {s u g a r l}}, \tag {3}
$$
where  $\beta$  is the balance parameter varies across environments. The reward balance is very important to make both motor and sensory policies work as expected, without heavy bias towards one side [28], which will be discussed in our ablation study in Section 5.4. Following the studies in learning with intrinsic rewards and rewards of many magnitudes [20, 38, 75, 96, 99], we empirically set  $\beta = \mathbb{E}_{\tau}[\sum_{t=1}^{T} r_t^{\mathrm{env}} / T]$ , which is the average environmental return normalized by the length of the trajectory. We get these referenced return data from the baseline agents trained on normal, fully observable environments, or from the maximum possible environmental return of one episode.
3.3 Persistence-of-Vision Memory
To address ActiveVision-RL more effectively, we introduce a Persistence-of-Vision Memory (PVM) to spatio-temporally combine multiple recent partial observations into one, mimicking the nature
Figure 3: Examples for different instantiations of PVM with  $B = 3$ .
Figure 4: Five selected Robosuite tasks with examples on four hand-coded views.
of human eyes and the memory. PVM aims to expand the effective observable area, even though some visual observations may become outdated or be superseded by more recent observations. PVM stores the observations from  $B$  past steps in a buffer and combines them into a single PVM observation according to there spatial positions. This PVM observation subsequently replaces the original observation at each step:
$$
\operatorname {P V M} \left(\mathbf {o} _ {t}\right) = f \left(\mathbf {o} _ {t - B + 1}, \dots , \mathbf {o} _ {t}\right),
$$
where  $f(\cdot)$  is a combination operation. In the context of ActiveVision-RL, it is reasonable to assume that the agent possesses knowledge of its focus point, as it maintains the control over the view. Therefore, the viewpoint or position information can be used. In our implementations of  $f(\cdot)$  shown in Figure 3, we show a 2D case of PVM using stitching, and a PVM using LSTM which can be used in both 2D and 3D environments. In stitching PVM, the partial observations are combined like a Jiasaw puzzle. It is worth noting that combination function  $f$  can be implemented by other pooling operations, sequence representations [15, 44, 85, 86], neural memories [37, 39, 76, 78], or neural 3D representations [74, 107], depending on the input modalities, tasks, and approaches.
4 Environments and Settings
4.1 Active-Gym: An Environment for ActiveVision-RL
We present Active-Gym, an open-sourced customized environment wrapper designed to transform RL environments into ActiveVision-RL constructs. Our library currently supports active vision agent on Robosuite [114], a robot manipulation environment in 3D, as well as Atari games [9] and the DeepMind Control Suite (DMC) [97] offering 2D active vision cases. These 2D environments were chosen due to the availability of full observations for establishing baselines and upper bounds, and the availability to manipulate observability for systematic study.
4.2 Robosuite Settings
Observation Space In the Robosuite environment, the robot controls a movable camera. The image captured by that camera is a partial observation of the 3D space.
Action Spaces Each step in Active-Gym requires motor and sensory actions  $(\mathbf{a}^s, \mathbf{a}^o)$ . The motor action space  $\mathcal{A}^s$  is the same as the base environment. The sensory action space  $\mathcal{A}^o$  is a 5-DoF control: relative (x, y, z, yaw, pitch). The maximum linear and angular velocities are constrained to 0.01/step and 5 degrees/step, respectively.
4.3 2D Benchmark Settings
Observation Space In the 2D cases of Active-Gym, given a full observation  $\mathbf{O}$  with dimensions  $(H,W)$ , only a crop of it is given to the agent's input. Examples are highlighted in red boxes in Figure 5. The sensory action decides an observable area by a location  $(x,y)$ , corresponding to the top-left corner of the bounding box, and the size of the bounding box  $(h,w)$ . The pixels within the observable area becomes the foveal observation, defined as  $\mathbf{o}^{\mathrm{f}} = \mathbf{o}^{\mathrm{c}} = \mathbf{O}[x:x + h,y:y + w]$ . Optionally, the foveal observation can be interpolated to other resolutions  $\mathbf{o}^{\mathrm{f}} = \operatorname{Interp}(\mathbf{o}^{\mathrm{c}};(h,w)\rightarrow (r_h^f,r_w^f))$ , where  $(r_h^f,r_w^f)$  is the foveal resolution. This design allows for flexibility in altering the observable area size while keeping the effective foveal resolution constant. Typically we set  $(r_h^f,r_w^f) = (h,w)$  and fixed them during a task. The peripheral observation can be optionally provided as well, obtained by interpolating the non-foveal part  $\mathbf{o}^{\mathrm{p}} = \operatorname{Interp}(\mathbf{O}\setminus \mathbf{o}^{\mathrm{c}};(H,W)\rightarrow (r_h^p,r_w^p))$ , where  $(r_h^p,r_w^p)$  is the peripheral resolution. The examples are at the even columns of Figure 5. If the peripheral observation is not provided,  $\mathbf{o}^{\mathrm{p}} = 0$ .
Figure 5: Left: observations from Active-Gym with different foveal resolutions (F) and peripheral settings (P) in Atari and DMC. The red bounding boxes show the observable area (foveal) for clarity. Right:  $4 \times 4$  discrete sensory action options in our experiments.
Table 1: Results on Robosuite. We report the IQM of raw rewards from 30 evaluations. Highlighted task names are harder tasks. Bold numbers are the best scores of each task and underscored numbers are the second best.
Approach	Wipe	Door	NutAssemblySquare	Lift	Stack
SUGARL-DrQ (Stacking PVM)	56.0	274.8	78.0	79.2	12.7
SUGARL-DrQ (LSTM PVM)	58.5	266.9	108.6	88.8	31.5
SUGARL-DrQ (3D Transformation+LSTM PVM)	74.1	291.0	65.2	87.5	32.4
SUGARL-DrQ w/o Joint Learning	43.6	175.4	58.0	107.2	12.0
SUGARL-DrQ w/o PVM	52.8	243.3	37.9	55.6	7.7
Single Policy	12.4	22.8	8.42	10.7	0.53
DrQ w/ Object Detection (DETR)	15.2	43.1	54.8	15.4	7.5
DrQ w/ End-to-End Attention	14.2	141.4	28.5	33.0	13.6
Eye-in-hand View (hand-coded, moving camera)	16.1	114.6	102.9	233.9	73.0
Front View (hand-coded, fixed camera)	49.4	240.6	39.6	69.0	13.8
Agent View (hand-coded, fixed camera)	12.7	190.3	49.9	122.6	14.7
Side View (hand-coded, fixed camera)	25.9	136.2	34.5	56.6	12.8
Action Spaces The sensory action space  $\mathcal{A}^o$  includes all the possible (pixel) locations on the full observation, but can be further formulated to either continuous or discrete spaces according to specific task designs. In our experiments, we simplify the space by a 4x4 discrete grid-like anchors for  $(x,y)$  (Figure 5 right). Each anchor corresponds to the top-left corner of the observable area. The sensory policy chooses to place the observable area among one of 16 anchors (absolute control), or moves it from one to the four neighbor locations (relative control).
4.4 Learning Settings
In our study, we primarily use DQN [65] and SAC [36] as the backbone algorithms of SUGARL to address tasks with discrete action spaces (Atari), and use DrQv2 [108] for continuous action spaces (DMC and Robosuite). All the visual encoders are standardized as the convolutional networks utilized in DQN [65]. To keep the network same, we resize all inputs to  $84 \times 84$ . For the sensorimotor understanding model, we employ the similar visual encoder architecture with a linear head to predict  $\mathbf{a}_t^s$ . Each agent is trained with one million transitions for each of the 26 Atari games, or trained with 0.1 million transitions for each of the 6 DMC tasks and 5 Robosuite tasks. The 26 games are selected following Atari-100k benchmark [47]. We report the results using Interquartile Mean (IQM), with the scores normalized by the IQM of the base DQN agent under full observation (except Robosuite), averaged across five seeds (three for Robosuite) and all games/tasks per benchmark. Details on architectures and hyperparameters can be found in the Appendix.
5.1 Robosuite Results
We selected five of available tasks in Robosuite [114], namely block lifting (Lift), block stacking (Stack), nut assembling (NutAssembleSquare), door opening (Door), wiping the table (Wipe). The first two are easier compared to the later three. Example observations are available in Figure 4. We compare against a straightforward baseline that a single policy is learned to govern both motor and sensory actions. We also compare to baselines including RL with object detection (a replication of Cheng et al. [17]), learned attention [93], and standard RL with hand-coded views. Results are in 1. We confirm that our SUGARL works outperforms baselines all the time, and also outperforms
(a) Without peripheral observation