Learning to Look by Self-Prediction
Authors: Matthew Koichi Grimes
Joseph Varughese Modayil
Piotr W Mirowski
Dushyant Rao
Raia Hadsell
Venue: Accepted by TMLR ()

Abstract: We present a method for learning active vision skills, to move the camera to observe a robot's sensors from informative points of view, without external rewards or labels. We do this by jointly training a visual predictor network, which predicts future returns of the sensors using pixels, and a camera control agent, which we reward using the negative error of the predictor. The agent thus moves the camera to points of view that are most predictive for a chosen sensor, which we select using a conditioning input to the agent. We observe that despite this noisy learned reward function, the learned policies a exhibit competence by reliably framing the sensor in a specific location in the view, an emergent location which we call a behavioral fovea. We find that replacing the conventional camera with a foveal camera further increases the policies' precision.

Published in Transactions on Machine Learning Research (02/2023)
Learning to Look by Self-Prediction
Matthew Koichi Grimes, Joseph Modayil, Piotr Mirowski, Dushyant Rao, Raia Hadsell DeepMind London, UK
{mkg, modayil, piotrmirowski, dushyantr, raiaH@deepminndcom
Reviewed on OpenReview: httpes//ppaarrrieeeeeee//////usd
Abstract
We present method for lcarning active vision skills, move the camera to observe a robot's sensors from informative points of view, without external rewards or labels. We do this by jointly training visual predictor network, which predicts future returns of the sensors using pixels, and a camera control agent, which reward using the negative error of the predictor. The agent thus moves the camera to points of view that are most predictive for a chosen sensor, which we select using a conditioning input to the agent. We observe that despite this noisy learned reward function, the learned policies avoid occlusions, and reliably frame the sensor in a specific location in the view, an emergent location which we call a behavioral fovea. We find that replacing the conventional camera with a foveal camera further increases the policies' precision.
1 Introduction
Computer vision, as commonly employed in embodied RL and robotics, does not closely resemble human vision. Human vision has moving eyes, fovea, movements such as saccades to frame targets in view, and smooth pursuit to track them (Dodge 1903). In the animal kingdom, active vision in the form of background tracking is widely demonstrated across disparate branches of the tree of life, for example by goldfish, rock crabs, cut clefishh, and blowflies. This background tracking reduces molion blur, skill without which a high- resolution retina goes to waste. It also unlocks another common ability, to detect the relative motion of small moving objects against the tracked background (Land 1999). By contrast, in robotics and embodicd RL, the camera is often rigidly fixed to the environment, limiting the agent to operate within the camera's fixed field of view (Levine et al., 2)16). With the camera fixed to the environment or robot, moving objects cause large amounts of input variance as they traverse pixels. This can be a source of training instability (Cetin al., 2 often leading researchers to avoid learning vision altogether and use object features off- tte-hh--heff vision modules instead (OpenAl et al. 2019).
An agent that has learned to visually frame objects in consistent image location could simplify the ac- quisition of visually-guided manipulation policies, as they can then focus on the manipulation aspect of the policy. This intuition has recently seen evidence robotic manipulation research on hand-mounted cameras, where the object's apparent position roughly stays the same as the hand is about to grasp (Hsu et al. 2022 Cheng al.] 2018 Gualtieri & Platt, 2018 Szot al., 2021 Jangir et al. 2022). This "hand-chosen" camera mount consistently frames objects about to be grasped, but not necessarily other elements of the environment, such as footholds to step obstacles to duck, or other agents to collaborate or compete with.
By contrast, humans benefit. from decoupling the kinematic chain of the eye from those of the limbs, allowing them to flexibly choose their visual focus in highly dynamic tasks ranging from fielding baseballs (MeBeath et al. 1995) to traversing challenging terrain (Matthis al. 2018). These visual policies all share common building blocks in the form of fixation and tracking (saccades and smooth pursuit). One inspiration our paper the question: if fixation is general visual skill that is key to acquiring more task-specific

---

Published in Transactions on Machine Learning Research (02/2023)
View from the foveal camera.
loints pos WRJ2
(b) The environment with hand, block prop and camera (high- (c) Stacked visualization scaled faveal images.
lighted by the yellow circle).
Figure 1: The environment, with hand, box prop, and foveal camera. Fig. la shows the foveal image, stack of 5 RGB images, with shape (5, height, width, 3). The images have the same dimension, but cover differing fields of view: 90, 45, 22, 11, and degrees. Fig. shows another foveal view.
visual policies, how can agent learn it independently of specific tasks? This paper demonstrates how an embodied agent can acquire visual skills in the absence of external rewards.
We propose that learning to observe onc's interactions with the world can start with learning to look at the interface those interactions, namely the parts of one's own body. We bootstrap this by training a predictor network predict the body's sensors from vision alone. At the same time, use this predictor's errors negative rewards for an RL agent that moves the camera. In other words, reward the agent for moving the camera to viewpoints that yield better predictions for chosen sensor. We communicate this choice of "target sensor" to the agent using simple one-hot conditioning input. In this manner, we show that single agent can learn distinct look-at policies, one per target sensor, without hand-designed external rewards. The predictor network, and the agent's policy and critic networks, can be trained simultaneously without manual scheduling. The learned camera. policies are competent. relative to baselines, navigating around occluding objects and preciscly framing the targeted sensor consistent location within the view. We call this emergent. area of the retina "behavioral fovea."
Motivated by this emergent behavioral fovea, implement a foveated camera (Cheung et al. 2017 Harris ct al. 2019 Deza & Konkle, 2020a by with exponentially higher resolution near the center of the field of view. We show that this results in more precise framing behavior, as the agent learns to position the subject near the center, where can observe it at the highest resolution.
2 Related Work
Unsupervised Learning of Tracking. Among the wider literature on tracking objects on video, some recent work focused unsupervised learning: conditional object-centric tracking given initial bounding box cue (Kipf et al. 2021 segmentation using Object Discovery and Representation Networks (Hénaff et al. 2022 and segmentation using motion-based (optical flow) and appearance-based information (Choudhury et al. 2022 Our unsupervised approach learns to execute large viewpoint changes in 3D environments, rather than tracking or annotating within the fixed view given by a video.
Computational Models of Foveation. Cheung et al. (2017) introduce neural attention model with a learnable retinal sampling lattice, with multiple extensions (Harris al.] 2019: Deza. & Konkle 2020al b).
2

---

Published in Transactions on Machine Learning Research (02/2023)
Other work uses spatial attention (Kosiorek et al. 2017] virtual fovea (Burt et al. 2021 to imitate the equivalent of what (bottom-up saliency) and where (top-down attention) pathways in animal vision. Rivkind (et al. (2022) introduce a low-resolution dynamical sensor that moves with drift-like tiny steps mimicking the microsaccades of a human eye. By contrast, we implement visual attention as a motor policy capable of driving the camera. to informative viewpoints 3D, rather than controlling which pixels to attend to within given 2D image. Instead of cropping or masking out the non-foveal part of the image, we retain the low-resolution wide-angle periphery to aid the camera agent in navigating through a 3D environment.
Visual Attention in Reinforcement Learning. Attention mechanisms for agents playing RL games were introduced by (Sookiin et al. (2015). They observed that top-down attention mechanisms forced agents to focus on task-relevant information by sequentially querying the environment (Mott et al. 2019) and helped generate virtual goals to replay (Liu et al., 2020). Guo et al. (222I) studied analogies between the visual attention of human experts and saliency maps in RL agents. Recently, self-supervised attention in RI. agents has been used to select regions of interest without explicit annotations (Wu et al.) 2021) and has provided robustness as well as increased learning efficiency and interpretability for visual tasks (Salter et al.) 2020:
James & Davison 2022 James et al. 2022 Tang et al.) 2020). Our active vision approach uses RL to control the camera, instcad of controlling top- attention over fixed views.
Active Vision in Reinforcement Learning. Embodied perception in a navigating agent enables it to move around an object to perceive better (Yang et al.) 2019) or to solve semantic segmentation tasks (Chaplot et al. 2020: Nilsson et al.) 2021). In a panoptic camera rig, RL can be used to select the best viewpoint for human pose estimation (Gärtner a 2020), 3D reconstruction (Pirinen et al., 2019), fore- casting the effects of motion (Jayaraman & Grauman, 2016), and more generally, learning to look around to efficiently gather information about the agent's 's surroundings (Ramakrishnan et al. 2019 Jayaraman & Grauman 2018). Our work focuses on learning active vision skills without external task rewards.
3 Methods
Our method teaches a camera-controlling agent to visually frame parts of its own body. It does this by simultancously training three networks: the agent's policy and critic networks, and a separate predictor network that predicts the body's sensor values from the camera's pixels. We use the predictor's 's errors as reinforcement learning penaltios for the camera agent, incentivizing it to move the camera to more informative views. Below, we describe each of these components.
3.1 Environment
Our enviromment uses the MuJoCo (Todorov et al. 2012) physics simulator, in which we place a camera at the end of an invisible armature (the camera bot), controlled by the camera agent. This camera bot shares the scene with a manipulator, and block prop randomly positioned within reach of the fingers, which gives the touch sensors at the fingertips something to touch (fig. Ibp. Both the camera bot and manipulator are driven by velocity control, i.e. target velocity for each actuator.
(PID) control in which the actions specify a
We treat the camera bot and manipulator as belonging to one robot that is conceptually split into two separate entities. The manipulator runs a fixed random behavioral policy throughout the experiment, but sensors on the manipulator are made available to define the losses of the predictor network (section 3.3) multi-headed generalized value function (GVF) whose losses define the endogenous penalty function for the camera agent (section 3.4p. The camera agent has to learn move to look at selected parts of the manipulator to better predict the future values of the targeted sensor in the manipulator. The training process only modifies the behavior of the camera bot, not the manipulator.
At the start of each episode, we randomize the joint angles of the manipulator and camera bot, and the position and dimensions of the box prop. We also randomly choose a sensor on the manipulator for the camera agent to target. We use an episode length sufficient to allow the camera enough time to reach an arbitrary final pose from its initial randomized pose.
3

---

Published in Transactions on Machine Learning Research (02/2023)
Figure 2: Predictor network imput and out- puts. The input is a stack of the previ- ous two camera observations (to enable pre- dicting velocities). The outputs {oz fur)er the distributions of the estimated return val- ues of N proprioceptive sensors, such joint angles or touch sensor readings. For clarity, we have lightened the background this high- resolution image rendered in MuJoCo.
v
T
servior predictor
e
Distributional
TD
pred
Vjrrnd
Bum
0:-1
qi-1
Distributional TD
LN
Tt-1
Distributiona
Leritic
TO
selact
largel
at
1
Ypread
policy
critic
Tt
L
q
(a) Predictor loss L.red and agent reward T2-
(b) Agent critic loss Leritse-
Figure 3: a) Distributional TD losses L; are computed for each sensor reading si, in a loop over N sensors (the figure only shows the N'th sensor's reading s(²) ). The predictor loss L,red is the sum of the N distributional TD losses. A different target sensor is chosen at the start of each episode, whose prediction error Laaret is used as negative agent reward rt. b) D4PG (Barth-Marou al., 2018)) critic Leritie depends on the learned agent reward rt. The target sensor (see Section 3.4) is specilied as one-hot vector, fed to the policy as conditioning imput.
3.2 Cameras
The camera bot can be equipped with conventional camera foveal one. Like Muih et al. (2014) we implement the foveal camera as N conventional cameras, all with the same position, orientation, and pixel dimensions, but differing in their field of view. Fig. shosowws an example of the resulting multi-scale images as scen by the agent, and fig. Ic demonstrates the scales' relative fields of view by stacking them on top of each other.
3.3 Predictor network
The predictor network shares no weights with the policy critic networks in the agent. It is convolutional residual network (ResNet) followed by a multi-layer perceptron (MLP). Its architecture and layer sizes are taken from IMPALA's (Espeholt et al.) 2018) convolutional residual network, replacing the LSTM at the end with an MRP. This predictor network takes two consecutive images as input, and outputs predictions for all potential target sensors. Like the critic network, its predictions take the form of discrete distributions v, of estimated future-discounted returns y/ of sensor signals rather than of rewards.
1/2 = Σ 1a 844+11 1=
(1)
v. (v) P(ue = y)
(2)
This was inspired by multi-timescale nexting MModayil et al. 2014). and the predictor network can be thought of as a multi-headed GVF (Sutonn et al.] 2011) with distributional outputs. Predicting a decaying sun of sensor readings, rather than a future sensor reading at a particular point in time, makes our experiments less dependent on the exact choice of frame rate or prediction timescale. Furthermore, using distributional TD losses instead of L2 losses serves as a principled means of normalizing the prediction losses across multiple sensor modalities with very different numerical ranges and distributions.
4

---

Published in Transactions on Machine Learning Research (02/2023)
Driving behavior to maximize prediction errors has been suggested form of curiosity (Whtte et al. 2014 Our agent does the opposite of this, as it is rewarded for minimizing error. Instead of maximizing short-term surprise, it explores the state space through the diversity of its prediction target sensors. This is partially a reflection of our different sctting. When an agent physically interacts with the world under a fixed camera, it may make sense lo explore by maximizing prediction error. When the agent moves the camera itself, it can trivially increase prediction error by staring in uninformative directions, without meaningful exploration.
3.3.1 Agent networks
The agent consists of a policy network and a critic network. Like the predictor network, each of these an MLP stacked on an IMPALA-style convolutional ResNet. The critic and policy networks share a ResNet, but have different MLP heads. Both networks take a mix of images and other inputs. The images are fed through the ResNet, and the other inputs are concatenated with the ResNet's output and fed to the MLP. The policy network takes imput the observation (two consecutive image frames), and one-hot vector specifying the target. It outputs bounded continuous target velocities for the camera bot's four joints (section 3.1p. The state-action critic additionally takes the action, and outputs a discrete distribution over the estimated return. The agent's ' training objective is to minimize the predictor network's error for the target sensor, by modifying the policy network to move the camera to look at the associated body part.
3.4 Training
For each batch of transitions sampled from the replay buller, we compute losses for the predictor, agent critic, and agent policy networks, then perform gradient updates on all three.
To train the predictor, we compute the prediction error of sensor i using the distributional TD loss BBlllmaaee et al.] 2017) from D4PG (Barth-Maron et al. 2018). This is analogous to the standard TD error Tt yos =)-1(x)) (x)), except that values are represented not by scalars, but by discrete distributions over the range of possible return values (eq. 2
1 DistributionalT D(s Ypred, ui, 1
(3)
The predictor discount Ypred is separate from the discount used for training the critic. Setting "prred 0 amounts to performing next-frame prediction, while setting it to larger values predicts its future sum over a decaying time window with half-life h = At in(0.5) [n("prez] It is possible to predict in Horde (Sutton et al. 2011,, which may be useful in environments with predictable dynamics over multiple frames. For our environment, where the camera and manipulator have little momentum, we use decay Ppred chosen to have a short half-life of 0.1s. The predictor network's loss is then the sum of prediction losses across all target sensors, L,rrd Σ, Li.
multiple time windows
The camera agent networks share no parameters with the predictor. Each episode randomly chooses a proprioceptive sensor on the manipulator, to serve as the camera agent's taryet for that episode. The camera agent's 's task is to position the camera in a manner that reduces the predictor's error for that target sensor. We therefore define the reward to be r; = -T. -ffarget where I. I.target the prediction error for that episode's target sensor, on timestep t. This it is learned. The agent critie loss analogous to eq. 3 substituting ra for sensor reading of y for "pred: and critic outputs 4 and 4-11 for predictor outputs Vi and u--1 We train the policy network using the deterministic policy gradient loss Siivvr et al. 2014).
dense reward, as is available on every timestep, but a noisy one,
4 Experimental Results
We describe our experimental setup, then present the claims supported by our results. All results are from the trained camera agent, evaluated without the exploration noise used during e-greedy training.
5

---

Published in Transactions on Machine Learning Research (02/2023)
Little finger touch
Forefinger root joint
Thumb root joint
Wrist flexion
Wrist deviation
(a) Target sensor prediction error improvement over an episode (higher is better, indicates camera agent competence)
Little finger tauch
Forefinger root joint
Thumb root joint
Wrist flexion
Wrist deviation
(b) Target sensor prediction error average over an episode (lower is better, indicates predictor competence)
Figure 4: Agent and predictor training. X axis is wall time in seconds, spanning 13 hours of training. In fig. 4a
the Y axis the improvement of the largel sensor's prediction error (L target eq. 3 from the beginning to the end of the episode, as the agent moves the camera to better point of view. In fig. 4b the Y axis is the average value of L larget over the course of an episode. Unlike in fig. da the Y axes have different ranges, to better show the progression of predictor training. T.lurget is a KT. divergence between distributions over possible return values so unitless, and its range is independent of the sensor value range. All curves are taken from single agent conditioned to target one of the above five target sensors, chosen randomly on each episode, Bold curve: mean of experiments with different RNG seeds. Shaded area: +1.96r.
4.1 Experimental setup
Environment
The manipulator is a model of the 20 degrees of freedom (DOF) tendon-driven hand by
Shadow Robotics (Shi et al. 2011 Plappert et al. 2018). We drive the manipulator using Perlin noise (Perlin 1985), which provides temporally smooth control by interpolating random keypoint velocities with splines. The keypoint velocities are spaced second apart and are uniformly sampled from the range of joint velocitios.
Noise drives all the tendons, causing the hand to writhe about. The camera bot has four DOF with different
rangos: elevation [0, æ]. azimuth [-n, distance [.2m, 2m], and yaw [- x). The first three DOFs move the camera in spherical coordinates centered around the manipulator. The yaw DOF rotates the camera around its local vertical axis, allowing it to look away from the hand. The values for these DOFs are randomized at the beginning of each episode. Randomizing the yaw DOF causes the manipulator to be entirely outside the field of view half the time. The episodes are 12 seconds long. The camera agent selects actions every 200 ms, yielding episodes of 60 limesteps.
Cameras In our experiments, the foveal camera has N cameras with 21 21 pixels each. Camera has FOV of 90 degrees in the vertical and horizontal directions, camera 2 has FOV of 43 degrees, and on, down to camera 5 with FOV of 5.26 degrees (90 x 2-1). The conventional camera is 21 x 21 pixels, with FOV of 90 x 90 degrees.
Training We train the agent and predictor networks simultaneonsly. We employ 512 processes that each run an independent copy of the environment, pushing experience transitions (labeled with their episode's sensor targets) into the replay buffer. The buffer runs in own process, and has maximum capacity of 1M transitions (168 GB of RAM when using for 5 x 21 x 21 foveal images). A single learner process samples mini-batches of 256 transitions from the replay buffer. The policy network trains with learning rate of 10-5, while the critic and predictor networks use learning rate of 10-4. We use Adam (Kingma & Ba 2015) to minimize the losses. Each environment process runs on machine with CPU and 1.1 GB of RAM. The learner runs on a machine with 2 CPUs, 4.9 GB of RAM, and a TPU. Training takes roughly 72 hours to converge for both foveal and conventional agents.
Networks The agent's policy and critic networks share a convolutional ResNet, which acts as a visual feature extractor. Its architecture and layer sizes are taken from IMPALA's (Espeholt et al. 2018). The policy and

---

Published in Transactions on Machine Learning Research (02/2023)
critic have separate MLP heads, with hidden layer sizes (256, 256, 256) and (512, 512, 256), respectively. The predictor uses the same layer sizes as the critic, but has its own convolutional network weights.
Inputs and outputs The predictor takes pair of successive frames from the camera, and outputs N discrete distributions for possible target sensors, over the range of possible return values. In our experiments, N = 5. These five sensors were selected to represent both touch and joint angle sensors. We selected the predicted joints to cover a range of sizes and directions of movement when the joints bend. The critic takes an N-dimensional one-hot vector in addition to the pair of pixel frames. This vector indicates which sensor to target. The critic outputs distribution over the possible return values of the learned reward. The predictor and critic's s output distributions are 51-atom discrete distributions evenly spanning the range of possible returns. The policy network takes the larget one-hot and a pair of pixel frames, and outputs continuous target velocities for the camera bot's four DOFs: azimuth, elevation, distance, and yaw.
4.2 The trained camera policy improves target prediction accuracy
Figure 4a shows the improvement of the target sensor's prediction error from the start to the end of an episode, plotted throughout training. We plot the error improvement rather than the final error (seen in figure 4b and table D, because the latter would jointly evaluate the predictor and the camera agent. By contrast, the error improvement within an episode evaluates the camera policy, independent of the predictor. Regardless of predictor quality, a policy selecting velocities from uniform distribution centered at zero does not, average, improve prediction accuracy over the course of episode. A camera policy trained to improve this prediction error will do so, as shown. The wrist joint angles (two rightmost plots) have the most visual impact, as they move the whole hand. The agent learns to improve those errors first, while policies targeting the fincr sensors improve later.
Little finger touch
Porefinger root joint angle 4.12 1 0.065
Thumb root roint angle
Wrist flexion angle
Wrist deviation angle
Blind (c)
0,680 1 0.020
4.11 1 0.073
3.84 1 0.059
4.08 1 0.051
Random (c)
0.667 0.067
4.16 + 0.034
4.16
0.056
3.51 0.069
3.61 0.050
Random (f)
0.648 + 0.048
4.07 : 0.033
4.09 + 0.068
3.41 0.054
3.58 + 0.077
Geometric (c)
0.620 1. 0.001
4.12 1: 0.033
4.10 1: 0.026
3.57 1: 0.038
3.79 1: 0.027
Coometric (f)
0.566 0.046
3.75 1: 0.037
3.61 +: 0.044
2.55 0.043
2.84 0.057
Ours (c)
0.582 + 0.062
3.24 + 0.070
2.98 + 0.061
1.73 - 0.054
2.17 # 0.048
Ours (f)
0.606 + 0.046
3.11 + 0.050
2.75 + 0.071
1.65 -: 0.047
1.93 + 0.051
Oracle (c)
0.480 0.020
2.80 1 0.096
2.53 4 0.025
1.45 - 0.017
1.62 0.026
Oracle (f)
0.587 +: 0.022
2.80 + 0.030
2.53 + 0.032
1.45 0.009
1.66 0.010
Table Target sensor's prediction error at episode end (lower better). The "(c)" and "(f)" indicate conventional or loveal camera. Blind and Oracle give upper and lower bounds to the error, Random shows the prediction error of a randomly posed camera. Geometric shows the prediction error of a camera with random position, but pointed at the hand, whose position is inferred by known body geometry. See section 4.2.1 for more detail. Error is measured as the TD error for predictions given as distributions over the range of possible return values. Confidence bounds indicate 11,66a, calculated from 10 runs with different RNG seeds.
4.2.1 Comparison with baselines
Table shows the target's prediction error at the end of the episode (lower is better). Unlike the relative measure of the agent plotted in fig. 4al this is an absolute measure of the joint quality of the predictor and agent at the end of training. It compares trained agents against the following baselines:
Blind: The Blind baseline is a predictor trained on camera pointed away from the hand. It can do no better than learn to output each sensor's prior distribution, and serves an upper bound to the expected agent prediction error.
Oracle: For a lower bound on the prediction error, we run a sweep over a series of hand-chosen fixed camera poses surrounding and looking at the hand, training a separate predictor for each pose. The Oracle entries show the minimum prediction error over all viewpoints for that target sensor. The Oracle benefits from not only specializing a single point of view, but also from not moving, which significantly reduces data variance and improves prediction error even in the moving-camera agent. In practice, moving agents cannot
7

---

Published in Transactions on Machine Learning Research (02/2023)
fonefinger root
wrist flexion
61
a =
0.05
= 0.1
thumb root jo nt
wrist deviation
0.2
0.4
Figure 5: Agent performance under sensor noise. Vertical axis: prediction improvement of the target sensor over the course of an episode, due to the camera agent repositioning the camera to better view. Horizontal axis: Number of actor episodes. Colors represent different amounts of noise added to the sensor readings. The noise is parameterized by a, the fraction of the sensor range covered by + standard deviation of the noise distribution. See section 1.3 for details.
always maintain static view, nor an agent with multiple static cameras usually know a priori which camera will yicld the most accurate predictions.
Random: The Random agent is our agent with its policy and critic learning rates set to zero. The camera spawns randomly, as usual, but hardly moves thereafter (brownian actions do little to move the camera bot, which has high inertia). The predictor must learn to predict from the resulting random camera views. These views are mostly static, advantageously reducing input variance in a similar manner to the still images of the Oracle. Our agent outperforms the Blind and Random baselines by statistically significant margin. Geometric: The Geometric agent is identical to Random, except instead of randomizing the camera's position and orientation, Geometric only randomizes the position. The orientation is pointed at the hand, using the known relative geometry between it and the camera. The fact that the Geometric baseline policy performs worse than the learning policy shows that using externally specified geometrie knowledge of the sensor location suboptimal in this setting. This serves as an upper bound on what a classical control policy of orientation could achieve. The foveal version gets lower errors than its conventional counterpart, as its multi-scale camera allows it to see detail even when randomly positioned a distance.
4.3
The method is robust to epistemic and aleatoric uncertainty
As our method uses prediction accuracy as a reward, here ask how it fares in the face of uncertain predictions. Broadly, predictions may be uncertain due to external, aleatoric sources of uncertainty, such as sensor noise, internal, epistemic uncertainty, such as sub-optimal prodictor.
Aleatoric uncertainty Simulating sensor noise accurately is an active field of work, much of it idiosyncratic to each sensor type or even instance. However, we may still use simple noise models to make general probes into our mothod's response to sensor noise. In fig. 5 we plot the target sensor's prediction error improvement over the course of an episode vs actor steps. section 4.2, we use this improvement as a measure of the camera agent's competence. We run a sweep of five levels of noise, with three random seeds each, for a total of 15 experiments. Here used conventional camera. with 47 47 resolution. We added noise to all non-visual sensors, sampled from zero-centered normal distributions with variances parameterized by hyper-parameter a as follows. The standard deviation of each sensor's noise is set to a = a(smar - -mmin/ where Smaz and Smin are the sensor bounds. For example, if a = 1, then the noise's +1 standard deviations would span the whole sensor range. After adding the noise, we clamped the sensor value to its range. Fig. 5 shows the performance curves under a = 0, 0.05, 0.1, 0.2, and 0.4. All plots show that while increasing noise diminishes performance as expocted, the camera still Iearns competent policies without being derailed, even under sensor noise variances that would be considered severe in a robotic context.
Epistemic uncertainty The predictor, which learns the reward function, and the camera policy, which learns from this reward, are trained from scratch in parallel with no effort made to schodule the reward training to precede the camera agent training. The agent neleth learns a compelent policy, indicating robustness to the high epistemic uncertainty exhibited near the beginning of training by the untrained prodictor. Fig. La| shows the camera agent's competence, as mcasured by the improvement of the target sensor prediction over an episode due to camera motion. Below it, fig. 4b shows the predictor competence, measured by the average prediction error of the target sensor over the episode. In all instances, the agent
8

---

Published in Transactions on Machine Learning Research (02/2023)
forefinge root joint
thumb root joint
wrist Aexton (curl)
Wrist deviation
Littl fnger touch
Farelinger nood jaii
Thumb nool jont
What Texion (curl)
Whit deriation
lorefinger joint
thumb reat
wrist flexion (curli
wrist devistion
(b) Histograrms of targets on conventional careera
Litle Enger loucl
Forwligge
Thumbo
Whatt Desionn cleviation
(a) First (top row) and last (bottom row) frames
(c) Histograms of largets on foveal camreaa
Figure Fig. a: the first and last frames (lop and bottom row) from episodes where the sensor the caption is the visual target. The figure highlights the target sensor with yellow circles (the agent does not see this). Note that the agent chooses to observe the wrist flexion angle (3rd column) from the side, from where while it observes the wrist's side-to-side deviation angle (4th column) from above. Fig. b and show histograms of target location at the end the episode, in image space, for conventional and foveal agents. The red box covers the 95% confidence interval (2.5% 97.5% of the cumulative probability distribution) along each axis. The histograms accumulate the final target position over episodes collected after convergence, or the last 8 days out of an 11 day training run. This amounts to total of 25000 episodes, roughly 5000 episodes per target.
most visible,
performance starts to plateau well before the predictor performance has converged. For example, when the target is the wrist deviation, the camera agent curve enters its plateau stage at 10k seconds of training, at which point the corresponding predictor is only 51% of its way from its peak value (4.0) to its value convergence (..7).
4.4 The trained camera policy frames the subject
A consistent outcome in our experiments that camera agents trained on a fixed choice of target tend to place the target in a particular position on the screen. This position varies from one training run to another, though it is usually near the center (fig. (a)) This is not an instance of the camera agent having memorized particular set of preferred values for its own joint angles. The camera agent is unaware of the camera bot's joint angles, as the only inputs are pixels and the one-hot vector specifying the target sensor. Furthermore, it controls the camera bot joints by velocity control, unlike a position-controlled camera, which may learn to output a constant target camera. pose regardless of the input. Figure 0b shows the emergence of this behavioral fovea from framing.
4.5 Training the camera policy with a foveal camera yields more precise framing
As shown in fig. 6b agents learn to frame the target in a specific region of the image, even when equipped with a non-foveal camera. We call these regions behavioral fovea. Equipping agents with foveal camera induces more focused and centered framing behavior (fig. 6c). as the agent can see a target at full resolution only when framing at the center. A stronger tendency to center the subject further lessens the need to spend network capacity on position-equivarincce Tf the target moves around the workspace, tracking behavior may emerge as a sidc-effoct of this tendency to keep centered in the image. We discuss this future work in section 5
The non-foveal camera. places the behavioral fovea near the center despite having no physical fovea there, Therefore, one might ask whether the foveal camera has tightly centered behavioral fovea primarily because that is the location of its physical fovea, or whether this is mostly an enhancement of a preexisting rentral bias in the learning dynamics. Figure 7 addresses this question. Each row shows the behavioral fovea for a single experiment, with the physical fovca horizontally shifted to the left, center, and right. In each case, the agent learns to position the target sensor on the physical fovea, showing it be the determining factor in where the agent chooses to frame the target.
9

---

Published in Transactions on Machine Learning Research (02/2023)
Littl finger touch
Forefinger root joint
Thuma root joint
Wrist flexion (curl)
Wrist deviation
(a) Fovea offset = -0.75
Littl finger touch
Foretinger root joint
Thuma root Joint
Wrist flexion (curl)
Wrist deviation
(b) Fovea offset
Little fingar touch
Foretinger root joint
Thuma root Joint
Wrist flexion (curl)
Wrist deviation
(c) Fovea offset 0.75
Figure Histograms of the larget an off-center fovea. Each row is labeled by the horizontal offset of the fovea, in normalized image coordinates (-1 and correspond to the left and right edges of the image). Each column corresponds different target sensor. The red box covers the 95% confidence interval along each axis.
location the end of the episode, in image space, for foveal with
Figure G also helps answer a fundamental question: why use RL to learn to look at sensors, as opposed to supervised policy training? After all, the positions of sensors are calculable from the robot's 3D geometry, which could provide image-space targets to direct the camera towards during training. Our RL-based method has the following advantages over such a supervised approach: (1) it needs no such prior information on the robot's geometry, (2) the agent learns not just what 3D point to look al, but from which direction, and (3) the most informative point of focus is not always the sensor itself. Fig. 6a illustrates point 2. The agent observes wrist flexion and deviation from orthogonal directions (i.c., along their orthogonal bending axes), despite looking at the same wrist. The "geometric" baseline in section 4.2.1 perfectly orients the camera at the hand, and thus represents an upper bound on the performance of an agent supervised on orientation alone. As seen in table [ underperforms our agent, which learns a suitable orientation and position for the camera. The rightmost image of fig. be illustrates point The histogram shows two poaks; the wrist is framed off-center to the left right, and the camera centers the hand instead. This is because the wrist primarily causes motion to the hand, making the latter more informative to look at. (Some sensors have a larger decoupling between the sensor position and the most informative view, such IMU-based orientation sensor attached to some arbitrary location in large rigid body.)
4.6 The camera policy adopts distinct camera positions for different sensor targets
Sections 4.4 and 4.5 showed that the policy learns to orient the camera to frame the target sensor at a specific image location. While it is obviously important look in the right direction, looking from the right position also matters. Fig. 8 shows the distribution of the camera position the end of the episode for single trained agent, showing separate plot for each sensor target. Figure Sa shows that the camera. has learned to observe the wrist flexion in profile, i.c., along the axis of rotation, from which the visual flow of flexing the wrist is most apparent. It observes the wrist from one side or the other, hence the bi-modal distribution seen in the red projection. By contrast, observing the same wrist joint, but predicting its side-to-side deviation angle, behooves the agent adopt a top-down view, as shown in fig. 8b The differences distribution for the camera positions when observing the finger root joints (fig. 8c fig. 8d] are more subtle, observable in the blue ionjection along the floor. Fig. Se shows the agent to be position-agnostic when targeting the
10

---

Published in Transactions on Machine Learning Research (02/2023)
(a) Wrist flexion (curl)
(b) Wrist deviation
(c) Forefinger root joint (d) Thumb root joint (e) Little finger touch
Figure 8: Positions of the camera the end of the episode, for different larget sensors. Positions are in purple, with their projections to axis-aligned planes shown in red, green, and blue.
little finger's touch sensor. As shown in fig. 1a the little finger's touch sensor the most difficult to predict, due to the severe label imbalance presented to the predictor (the touch sensor touches the block infrequently under the hand's random policy). This leads to noisy learned reward, which is good enough to teach the agent to center the touch sensor in the view (fig. Gc but is insufficient to narrow down the camera position. In section D we propose a method to improve this.
4.7 The camera policy learns to circumnavigate occlusions
We introduced a randomized occluder to our training environment to lest the generality of our self-supervised training. The occluder is flat rectangle with random color, position, and dimensions. We uniformly sampled its height and width from the range [0.5, 1.0], and sampled its position in spherical coordinates from azimuth ~ -, x]. elevation ~ [0, x]. distance ~ [.4, .6.. These spherical coordinates are roughly centered around the hand. For reference, the size of the environment's floor as an occluder, we orient the board to face the hand. Fig. shows two trajectories of the camera agent, starting from the same initial conditions. The resulting camera policy is able to sidestep the occluder when it blocks the camera's 's view, thereafter exhibiting similar viewpoint preferences as in the unoccluded case.
2. To maximize its effectiveness
5 Discussion
Many animals do not have fovea. Some, like rats, have roughly even distribution of optical receptors spread over a nearly spherical field of view. When objects in all directions might ask if rats need to move their eyes to look al subjects at all. Yet they do, exhibiting similar visual skills to humans, such as centering and fixating the subject in a specific area of the retina (Holmgren et al.) 2021). As with the rat, we find that our non-foveal agents exhibit framing behavior, positioning the subject in a specific area of the image, giving rise to a behaviorul fovea despite there being no intrinsic acuity advantage in one area of the image versus another.
with equal acuity, one
This suggests that the dynamics of the training encourage positive feedback loop between the predictor and agent: the predictor improves its expertise in specific region of the field of vision, and the agent learns to move the relevant subject into this visual region to receive the better prediction reward. This in turn provides to the predictor even more training data with the target in that image position, further improving its predictions there. This behavior is in contrast to the usual emphasis placed on learning position-invann or position-equivariant representations in computer vision research that uses static datasets of images rather than an active camera (LeCun al., 1998). That said, even static facial recognition has been shown to benefit from normalizing the facial feature locations (Taigman et al., 2014); domain--pecifie form of framing.
We simulated the Shadow hand in the MuJoCo environment and thus have yet to investigate real-world complexities such as sensor timing, synchronization, noise, resolution, and discretization. Predict- ing distributions of returns instead of sensor readings isolates our framework from some of these real-world complexities but not all (e.g. motion blur). We use a random policy for the hand, under which the finger- tips rarely touch the block, resulting in highly skewed distributions for their touch sensors. This results in relatively poor touch prediction compared to joint angle prediction. In future work, we plan to reward the hand for maximizing target sensor entropy, yielding more pedagogical hand behavior with flatter sensor
Limitations
11

---

Published in Transactions on Machine Learning Research (02/2023)
(a) Wrist flexion viewed in profile.
(b) Wrist deviation is viewed from above.
Figure 9: Occlusion-avoidanl trajectories, by the camera agent trained with random occlusions.
distribution. We randomize the lighting and expect that other standard data-rante ranaamizaoiiiiooon techniques could enable more robust sim2real transfers, as demonstrated in appendix E Lee et al. (2022). At higher level, this work does not address the open research question of how an agent can form an expanding conceptual space of internal GVF questions beyond predicting its own sensors.
Future work Giving our agent foveal camera incentivizes to position the subject more prociscly at the center of the image. One direction for future work is to investigate whether the agent can mainlain this centering as the target body part makes large movements, resulting in tracking behavior from the camera as side-effect of framing. We demonstrate that a single agent can be trained to fixate on one of several targets, specified with conditioning input. In this work, we limit the sensors to a single complex hand, where inferring joint angles of individual fingers can be challenging amidst the occlusions and visual distractions offered by the other fingers. We are therefore optimistic that our method would work on simpler morphologies such as parallel grippers, which we would like to explore in future work. A realistic body provides rich variety of visual scales and distances, from shoulders to toe tips. We posit that these present a means learn rich repertoire of visual fixation and tracking skills without the need to design their reward functions. basis for exploration while learning higher-level skills, demonstrated with
These skills could serve SAC-X [Riedmiller et al. 2018), which used manually designed reward functions.
For example, some tasks are alrcady known to benefit from fixating the eye on body parts as they touch the environment. Manipulation lasks such as pick-and-place benefit from a camera mounted on the wrist and fixated on the fingers (Hsu al. 2022). Such tasks, however, are already well served by passive cameras. For future work, we envision active vision primarily be of use in dynamic tasks with mobile robots, where the subject of visual interest change rapidly.
Our learned pelf-prediction reward, while noisy, is shaped and temporally dense, which are attractive qualities for an auxiliary reward. Such rewards see extensive use where sparse task reward alone is insufficient to learn dynamic motor policies, such as playing football/se /socccr (Vezzani et al. 2022 Lin et al. 2022). We emphasize that the sensor prediction reward not means to observe one's own body parts per se, but rather a means dircct the gaze to whatever conducive predicting sensory inputs, such as the aural and tactile experience of making contact with a football. A reward for predicting contact with the ball incentivizes robot to keep auxiliary reward. Such downstream tasks exhibit different
the ball in view, and is thus worth exploring sensor distributions than the random policy used in this paper, and open interesting questions to whether to fine-tune the predictor on the new distribution, or freeze it after pretraining on an entropy-maximizing policy.
Parisi et al. (2022) show that visual RI. agents benefit from pretraining on static image datasets, even when the dataset imagery looks nothing like the tasks. Self-supervised RL presents a means of feature learning that is more natural, in lifelong learning sense, than using human-annotated image datasets. Our method
12

---

Published in Transactions on Machine Learning Research (02/2023)
learns features that are sufficient for the task of precise visual fixation. Whether they improve performance general control tasks remains to be seen.
6 Conclusion
Learning motor skills without externally defined rewards or tasks is one path towards lifelong learning, Externally specified rewards require privileged information that is unavailable to real embodied agent. This is especially true of skills such as active vision, employed across wide variely of embodied visual tasks.
In this paper, we demonstrate means of learning visual fixation from self-prediction alone. We train a single embodied agent to visually fixate on different parts of its own body, as chosen by a conditioning input from set of proprioceptive sensors. We show that this encourages the emergence of a behavioral fovea, where the fixated body part typically appears in a specializod region of the image, even when using conventional cameras. We show that when provided with an actual foveated camera, the samn agent more strongly constrains the target to the center of the fovea.
We show that the agent learns to adopt distinct points of view for observing different sensor targets, fixating on them precisely, while circunnavigating occlusions. Taken logether, these results present a means to learn a variety visual skills, up to one per proprioceptive sensor, without the need for hand-designed rewards.
References
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. 6th. International Conference on Learning Representations, ICLR, 2018. URL http://arxiv/ httpaaaxivv..og/aaaa abs/
1804. 086677
Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449-458. PMLR, 2017.
Ryan Burt, Nina N Thigpen, Andreas Keil, and Jose C Principe. Unsupervised foveal vision neural archi- tecture with top-down allention. Neural Networks, 141:145 151:145-159, 2021. URL htttps://arxa org/ aba/
2910.09108
Edoardo Cetin, Philip J Ball, Stephen Roberts, and Oya Celiktutan. Stabilizing off-po deep reinforcement learning from pixcls. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabalo (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2784 2810. PMLR, 17-23 Jul 2022. URL https: //proceedings. mlr, prress/v122/cetin222a html
Devendra Singh Chaplot, Helen Jiang, Saurabh Gupta, and Abhinav Gupta. Semantic curiosity for active visual learning. In European Conference on Computer Vision, pp. 309- 099-326. Springer, 2020.
Ricson Cheng, Arpil Agarwal, and Katerina Fragkiadaki. Reinforcement learning of active vision for ma- nipulating objects under occlusions. In Conference on Robot Learning, pp. 422-431. PMI.R, 2018. URL mlr http:pee/ree///cccnnaaaa////eeee http:://ppeeiiiiiaaappprrsrrsrrr
pdf
Brian Cheung, Eric Weiss, and Bruno Olshausen. Emergence of foveal image sampling from learning attend in visual scenes. 5th International Conference on Learning Representations, ICLR, 2017. URL
https://arxivv org/ abs/
Subhabrata Choudhury, Laurynas Karazija, Tro Laina, Andrea Vedaldi, and Christian Rupprecht. Guess what moves: Unsupervised video and image segmentation by anticipating motion. arXiv preprint oxXiv:2205.078, 2022. URL
Arturo Deza and Talia Konkle. arXiv:200.007991, 2020a. URL http:/wwwww...g/ab/0/.000000
Emergent properties of foveated perceptual systems. arxiv preprint
13

---

Published in Transactions on Machine Learning Research (02/2023)
Arturo Deza and Talia Konkle. Foveation induces robussnesss to scene occlusion in deep neural networks. Jour- nal of Vision, 20(11):442 4422, 2020b. URL /ttp:tts////oo.aroooooooo wtt.:aall.....eaa...e.eee.e.e.e. http/////oo/////////o////a/aa/ar attp:/icccccc.pppppacicccceeeit 2771566
Raymond Dodge. Five lypes of eye movement in the horizontal meridian plane of the field of regard. American journal of physiology-legaey content, 8(4):307-322 1903.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Tain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407 1416. PMLR, 2018.
Erik Gärtner, Alcksis Pirinen, and Cristian Sminchiscscu. Deep reinforcement learning for active human pose estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10835 10844, 2020. URL hwtpt:ttt////ttt aaai btg/index..o php/
Marcus Gualtieri and Robert Platt. Learning 6-dof grasping and pick-place using attention focus. In Conference on Robot Learning, pp. 477-486, PMLR, 2018. URL http://ppoocediigs.. mlr leiiggs..mr....e.r.ee. press/v87/ pdf
Suna Sihang Guo, Ruohan Zhang, Bo Liu, Yifeng Zhu, Dana Ballard, Mary Hayhoe, and Peter Stone. Machine versus human attention in deep reinforcement learning tasks. Advances in Newral Information
Processing Systems, 14:25370-25385 2021. URL lattss///axxiv org/ abs//aa0.. 15942.
Ethan William Albert Harris, Mahesan Niranjan, and Jonathon Hare, Foveated convolutions: improving spatial transformer networks by modelling the retina. In Shared Visual Representations in Human g Machine Intelligence Workshop at NeurIPS 2019, 2019. URL https: ptttps://eprints.... sotes.ssoooooooo ac. wwt.oo.......//1//.0.0..
Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. arXiv preprint org/ htrggabb//2200 08777
arXiv:2203.0877 2022. URL https://arxiy: http://rwww.oorab/////aaa
Carl D Holmgren, Paul Stahr, Damian Wallace, Kay-Michael Voit, Emily Matheson, Juergen Sawinski, Giacomo Bassetto, and Jason ND Kerr. Visual pursuit behavior in mice maintains the pursued prey on the retinal region with least optic flow. Elife, 10:c70838, 2021. URL htttp:://elifescienees wtggartireess////o/oooooooooosst 778838
Kyle Hsu, Moo Jin Kim, Rafacl Rafailov, Jiajun Wu, and Chelsca Finn. Vision-based manipulators need to also see from their hands. 10th International Conference Learning Represendations, ICLR, 2022. URL https://sites.
wwt./ooooo//aceeeeeeeeeeeeeeeeee cttp//iee///eeeeeeeeeoceooeeeeet seeing
Slephen James and Andrew J. Davison. Q-allention: Enabling efficient learning for vision-based robotic manipulation. IEFE Robotics and Automation Letters, 1(2):1612-1619, 2022. doi: 10.1109/T.RA.200 3140817.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew J. Davison. Coarse-to-five q-attention: Efficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13739-13748 June 2022.
Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, and Xiaolong Wang. Look closer: Bridg- ing egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 2022. URL httpp:://axxi.. http://xxiv..rr/abb//.0.........
Dinesh Jayaraman and Kristen Grauman. Look-ahead before you leap: End-to-end active recognition by forecasting the effect of motion. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 489-505, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46454
1.
Dinesh Jayaraman and Kristen Grauman. Learning to look around: Intelligently exploring unseen environ- ments for unknown tasks. In Conference on Computer Vision and Pattern Recognition, pp. 1238-12281 1238-1247,4 06 2018. doi: 10.109/CVVP:2218.00013
14

---

Published in Transactions on Machine Learning Research (02/2023)
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 3rd International Con- ference on Learning Represenlations, TCLR, 2015. URL
6980
Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional objoct-centric learning from video. arXiv preprint Arxiv:2111. 12594, 2021. URL ptttp::/aaxiv.orgge
12594
Adam Kosiorck, Alex Bewley, and Ingmar Posner. Hicrarchical attentive recurrent tracking. Advances in neural information processing systems, 30, 2017. URL http:://arriv..og/aa/////t 09262
Michael F Land. Motion and vision: why animals move their eyes. Journal of Compurative Physiology A, 185(4):341 352, 1999. URL https::/ppubmed.nottt ncbi .nlm .ih gov/1055526///
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Alex X. Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost Tobias Springenberg. Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, Claudio Fantacci, Jose Enrique Chen, Akhil Raju, Rae Jeong, Michael Neunert, Antoine Laurens, Stefano Saliceti, Federico Casarini, Martin Riedmiller, raia hadsell, and Francesco Nori. Beyond pick-and-place: Tackling robotic stacking of diverse shapes. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1131 PMLR, 08-11 Nov 2022. URL https://proceeddiggg mlr. sreie/rpeees///111/ee22bbb html
Sergey Levine, Peter Pastor, Alex Krizhevsky, and Deirdre Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection, 2016. URL https://arxiv.orrg org/ /bsg/
(603.02199)
Peng Liu, Chenjia Bai, Yingnan Zhao, Chenyao Bai, Wei Zhao, and Xianglong Tang. Generating attentive goals for prioritized hindsight reinforcement learning. Knowledge- Based Systems, 203:106140, 2020. URL
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, S. M. Ali Eslami, Daniel Hennes, Wojciech M. Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, Noah Y. Siegel, Leonard Hasenclever, Luke Marris, Saran Tunyasuvunakool, H. Francis Song, Markus Wulfmeier, Paul Muller, Tuomas Haarnoja, Brendan Traccy, Karl Tuyls, Thore Graepel, and Nicolas Heess. From motor control to team play in simulated hunanoid football. Science Robolics, 7(69) 2022. doi: 10.1126/scirbbotissaaooooo https://wwis
URL
https://www.science. science. www.ssiiiiieeneooo....o.
Jonathan Samir Matthis, Jacob I, Yates, and Mary M Hayhoe. Gaze and the control of foot placement when walking in natural terrain. Current Biology, 28(8))1224 1233, 2018. URL https:/www.iiennncdrrettt wcw.ecediiecc...................
Michael K McBeath, Dennis M Shaffer, and Mary K Kaiser. How bascball outfielders determine where run to catch fly balls. Science, 268(5210):569 573, 1995. URL httts::/www.biiteeet wuweaachh..b..... ca/
Volodymyr Mnih, Nicolas Heess, Alex Graves, and koray kavukcuoglu. Recurrent models of visual attention. Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (ods.), Advances in Newral Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL htttpp:://proccedingg.
.dd
Joseph Modayil, Adam White, and Richard S Sutton. Multi-timescale nexting in reinforcement learning robot. Adaptive Behavior, 22(2):146 160, 2014. URL httpp::/aaraiv.org/dd////// 1tt.//111.111111..111.1 pdf
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32, 2019. URL https::/://// httts:///arrt:: http:///www..rr//bb////......
15

---

Published in Transactions on Machine Learning Research (02/2023)
David Nilsson, Alcksis Pirinen, Erik Gärtner, and Cristian Sminchisescu. Embodied visual active learning for semantic segmentation. Proceedings of the AAAT Conference on Artificial Intelligence, 2021. URL
OpenAI, 1ige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019. URL httpps///axxii httppp////aaiiooo/a/b/////////// hatp://91100000 07113
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The (un)surprising effec- tiveness of pre- pre-trained vision models for control. In Workshop on Learning from Diverse, Offline Data, 2022. URL http://pparreeeeeeeeeeerrrrrraaa
Ken Perlin. An image synthesizer. SIGGRAPH Compul. Graph., 19(3):287 296, jul 1985. ISSN 0097-8930. doi:
URL bsttp:://do..rrg/iie wttp///.....//....... 1151/325113556... 35521/3222565535.577777
Alcksis Pirinen, Erik Gärtner, and Cristian Sminchisescu. Domes to drones: Self-supervised active triangu- lation for 3d human pose reconstruction. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Chal- lenging robotics environments and roquest for research. arXiv preprint arXiv:1802.094p0 2018. URL
httts::/arzize /ttp:////ara....r///////2222
Santhosh K. Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman. Emergence of exploratory look- around behaviors through active observation completion. Science Robolics, 4(30))aa 2019. doi: 10. 1126 wpcerroboiis..aww6::3.
1111////ciooooto URL https://www.sciett http://wwwwiiieeeerrrr////////// wttpoo//gg////////otoooo/o
aaw6326
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost. Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volune 80 of Procesdings of Machine Learning Research, pp. 4344 4353. PMLR, 10-15 Jul 2018. URL https::/
mlr Sttps///80//redmmieee
html
Alexander Rivkind, Or Ram, Eldad Assa, Michael Kreiserman, and Ehud Ahissar. Visual hyperacuity with moving sensor and recurrent neural computations. In International Conference on Leurning Represenla- tions, 2022. URL httpp:// www.:eeenttfiiieee http:/pparrrrreeeeeeeforrurrrdde Piddepon
Sasha Salter, Dushyant Rao, Markus Wulfmeier, Raia Hadsell, and Ingmar Posner. Attentioon-piivilgged reinforcement learning. Conference on Robot Learning (CoRT), 2020. URI. https::/arxxv.org/ 1911.08363
J. Shi, J. Z. Woodruff, and P. B. Umb. The shadow dextrous hand. 2011. [Online; accessed 24-January
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Determinis- tic policy gradient algorithms. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 387 395, Bejing, China, 22-24 Jun 2014. PMLR. URL https: httpps:///poceeeegss.oo mlr. httpeeeiiiinnm...s.ssseeeee.ee
Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, and Anastasiia Ignateva. Deep attention recurrent nentwwrr. eep Reinforcement Learning Workshop, NIPS, 2015. URL https:://arxivv org/ abs/
1512.01693
Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensori- motor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume 2. pp. 761-768, 2011. URL huttps://aaat csc. 1iv. ac. wwt.://wwww....oooooo//////a//aa A6 pdf
16

---

Published in Transactions on Machine Learning Research (02/2023)
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustala Mukadam, Devendra Singh Chaplot, Oleksandr Maksymeis, et al. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34, 2021. URL
attps://arxiae wttp:////awww..o/aaaaa////aaa/// 2106. 14405
Yaniv Thigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In 2014 IEEE Conference Computer Vision and Pollern Recogrition, pp. 1701-1708, 2014. doi: 10.1109/ W0.109/CVPPR.. 2VP..201..220.
Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of self-interpretable agents. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, GECCO '20, pp. 414-424, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371285. doi: 10.1455/3379930.38888.. URL https://doi.orge
1145/3877930 3389847)
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp.
IEEE, 2012. URL
nttps://iesexplore. ieee. org/ /tcp///mtttttme110
Giulia Vezzani, Dhruva. Tirumala, Markus Wulfmeier, Dushyant Rao, Abbas Abdolmaleki, Ben Moran, Tuomas Haarnoja, Jan Humplik, Roland Hafner, Michael Neunert, Claudio Fantacci, Tim Hertweck, Thomas Lampe, Fereshtch Sadeghi, Nicolas Heess, and Martin Riedmiller. Skills: Adaptive skill sequencing for efficient lentorally-estennded exploration, 2022. URL https://arxiv. org/ www./rg//////eeeeeeeeeeeeeeeee/e aws//b///11 /t/p//2211..1//// 13743
Adam White, Joseph Modayil, and Richard S Sutton. Surprise and curiosity for big data robotics. Tn Workshops at the Soventy-igghun AAAI Conference on Artificial Intelligence, 2014.
Haiping Wu, Khimya Khetarpal, and Doina Precup. Self-supervised attention-aware reinforcement learn- ing. Proceedings of the AAAI Conference on Artificial Intelligence, 2021. URL https://www aaai.org/ org/ AAAI21Papers/ AAAIAAAAII-111 WuH.
Jianwei Yang, Zhile Ren, Mingze Xu, Xinlci Chen, David J Crandall, Devi Parikh, and Dhruv Batra. Embodied amodal recognition: Learning to move to perceive objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2050, 2019. URL org/dbcument/900889
Leee.
17


Reviews:
[{"review_id": "bJMzwZoI9x", "reviewer": "Reviewer_WqZV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ The paper is well written, motivated and presented. Experiments are complete.\n+ Insightful qualitative analysis on the emergent behavior.\n\n### Weaknesses\n- The intrinsic reward formulation with the prediction errors might be susceptible to sensor reading noise, which is common in the real world. Would the policies learn to simply posit cameras to regions that are less susceptible to sensor noise as opposed to learning a useful behavior? Concretely, I would like to see more discussions on how this formulation can handle the aleatoric and epistemic uncertainty of the underlying system.\n- I would like to see how the method works in a more complex setting, for example, pick and place. When the camera is free to look at regions so as to better perform the manipulation task, what behavior would be the policy learned under that setting? Overall, I would like to see more discussions on how such a framework could be used for downstream robotics tasks.\n- Comparison with a classical policy that uses the robot’s geometry, as mentioned in 4.4.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a framework to learn visual servoing policies without specifying an explicit reward function. It trains a GVF that predicts discounted, distributional cumulative sensor readings. Policies are trained with D4PG by maximizing the negative GVF prediction error with the selected joint. Experiments show that 1. The policy learns emergent fovea behavior and 2. The policy learns emergent occlusion circumventing behavior.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Also as mentioned above\n1. More discussions on how the presented framework handle the aleatoric and epistemic uncertainty of the underlying system.\n2. More discussions on how the presented framework work in a more complicated manipulation setting.\n3. Comparison with a classical policy that uses the robot’s geometry, as mentioned in 4.4.\n", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9E4lUvPiMU", "reviewer": "Reviewer_mHZx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\nThis is an interesting application of using a learning-based reward function (i.e., predicting the sensing information) to train an RL agent to actively perceive the environment.\n\nThe connection to biology, especially computational models of foveation, is also interesting and could be of relevance to a broader audience beyond the machine learning community.\n\nThe study of a foveal camera further verifies the effectiveness of the proposed algorithm, where the agent frames the sensor on the camera regions with the highest resolution.\n\n[Weaknesses]\n\nThis paper only contains simulation results. It would greatly increase the paper’s significance if the authors conducted real-world experiments using real robots to control the cameras.\n\nRelated to my previous point, it is unclear how large the sim-to-real gap is using the current simulation setup. As a result, I’m not entirely convinced if the observations presented in this paper still hold true when extending to the real world.\n\nThis paper only considers touch as the sensing modality to make the prediction. It would greatly strengthen the paper if the authors could consider other sensing modalities, like sound (which might require the agent to focus more on the motion of the hand) and vision (which might require the camera agent to cover the field of view of the predicted vision sensors).\n\nThis paper currently also only considers a dexterous hand. Will the conclusion still hold true for other manipulators like parallel grippers or dexterous hands with other morphologies?\n\nAs has been mentioned in the limitation section, the hand is moving in a random fashion, which is very different from how humans would use their hands. Therefore, the sensing information distribution would be very different from a policy that can do meaningful tasks. Training the models in scenarios where the hand is doing practical tasks can improve the impact of this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a method for active perception that encourages an agent to move the camera to observe a robot’s sensors without explicit labels on the sensors’ locations. Specifically, the authors focus on a dexterous hand and aim to train camera agents to look at the tactile sensors on the fingertips.\n\nThe authors propose to do this task by training a camera control agent that obtains visual observations, based on which a visual predictor network predicts the future returns of a specific sensor. Intuitively, the camera control agent wants to focus on regions that are helpful for the predictor network to predict sensing information.\n\nExperimental results suggest that the agent moves the camera to points of view that are most predictive for a chosen sensor, even with learning-based noisy reward functions. The learned policies reliably place the sensor in a specific location in the view. By replacing the conventional camera with a foveal camera, the policies’ precision can be further increased.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides the points in the weakness section, I hope to see the following two additional changes.\n\nWhat will happen if the highest resolution region is not at the center of the foveal camera? Will the agent still place the sensor at the location with the highest resolution?\n\nI’m expecting more discussions and connections to the biology findings, which I believe can further extend the paper’s impact to a broader community.", "broader_impact_concerns": "I do not see any concerns regarding ethical aspects of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tm4lIJukoV", "reviewer": "Reviewer_mMVG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The overall “behavioral fovea” approach to active perception for robotics is quite novel and an interesting direction for the community. Quite often, methods use explicit 3D representations (like pointclouds, or voxels) or ignore 3D information entirely with RGB-only encoders. Multi-modal alignment is a general prior for robotics, and it’s great to see this paper tackle this area without making additional assumptions.\n+ The evaluations cover a good range of baselines and ablations. This discussions accompanying the results do a good job of motivating the design choices and clearly state the benefits of the proposed approach over alternatives.\n+ In general, the paper is well written and easy to understand. The video and figures are informative. Limitations are acknowledged.\n+ If the authors decide to open-source the framework, or even just model, the robotics community could potentially extend it to other embodiments like legged robots and parallel-jaw manipulators. \n\nWeaknesses\n- While the paper focuses on predicting proprioception, it’s unclear how useful the framework would be for downstream manipulation tasks. Can the trained RL agent be directly used to find better viewpoints for BC or RL control policies, and if so, is it better than 1-3 fixed cameras (which is the common setup in robotics)? Or can the vision encoders be used as a pre-trained network for fine-tuning manipulation policies? In the natural world, prediction errors are closely tied with an agent’s need to achieve certain behaviors or goals other than just viewpoint correction. It would be great to explicitly investigate this.\n- It’s also unclear how accurate proprioceptive predictions need to be in order to act capably. While the proposed approach outperforms all baselines in Table 1 in terms of joint angle errors, what is the best error rate that’s desirable? Clearly lower is better, but it’s unlikely that humans/animals make fine-grained predictions at the metric level. Perhaps a single, fixed, high-resolution camera is sufficient for “good-enough” prediction errors, but without knowing the underlying task this is hard to investigate. \n- As acknowledged in the limitations, sim2real and/or real-world reproducibility is a concern. Domain randomization might be feasible in no-object scenes, but it could get complicated with one or multi-object scenes where interactions with the environment and objects adds another degree of complexity.  Further, the action-space of a free-floating camera is easy to simulate but hard to replicate in the real-world where collision avoidance and unreachable viewpoints have to be accounted for. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an active vision framework for predicting proprioceptive senses. An RL agent learns to navigate to better viewpoints for predicting proprioceptive states of a target sensor (eg. the thumb joint). Unlike supervised methods, this agent is trained purely with TD losses from multi-modal alignment and doesn’t rely on any prior 3D information or demonstrations. The framework is conceptually simple, and also motivated by ideas from human/animal perception (like “behavioral fovea”). Experiments include evaluations in Mujoco with a simulated multi-fingered hand. These evaluations measure proprioceptive errors in joint angles or binary variables. Baselines include blind agents (that guess the most likely proprioceptive states), oracle agents (that used fixed cameras), and also an ablation investigating the usefulness of foveal over standard cameras. The proposed framework makes substantially lower prediction errors than all baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper as is already relevant and interesting to the robotics community. The following changes could potentially improve incentives for broader adoption:\n+ Show results on usefulness for downstream manipulation tasks.\n+ Study the importance of low proprioceptive predictive error on success rates of downstream tasks.\n+ Reproduce the results with real-robots.\n", "broader_impact_concerns": "As such, the reviewer does not see any major concerns. Training such an agent in the real-world with RL is a safety concern, but the authors seem keen on taking the sim2real route, which should be safer. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers posed a number of questions about uncertainty, robustness, and how these techniques may transfer to real robotic applications or be extended beyond the sensors used. Authors addressed some of these, but the others remain as interesting follow-up work.
Decision: Accept (poster)