Real-World Reinforcement Learning of Active Perception Behaviors
Authors: Edward S. Hu
Jie Wang
Xingfang Yuan
Fiona Luo
Muyao Li
Gaspard Lambrechts
Oleh Rybkin
Dinesh Jayaraman
Venue: NeurIPS 2025 poster ()
Keywords: robotics
reinforcement learning
imitation learning
active perception
privileged information
Primary area: reinforcement_learning
TL;DR: We propose a simple robot learning recipe leveraging privileged information to train active perception policies on real robots.

Abstract: A robot's instantaneous sensory observations do not always reveal task-relevant state information. Under such partial observability, optimal behavior typically involves explicitly acting to gain the missing information.
Today's standard robot learning techniques struggle to produce such active perception behaviors.
We propose a simple real-world robot learning recipe to efficiently train active perception policies. Our approach, asymmetric advantage weighted regression (AAWR), exploits access to "privileged" extra sensors at training time. The privileged sensors enable training high-quality privileged value functions that aid in estimating the advantage of the target policy. Bootstrapping from a small number of potentially suboptimal demonstrations and an easy-to-obtain coarse policy initialization, AAWR quickly acquires active perception behaviors and boosts task performance. In evaluations on 8 manipulation tasks on 3 robots spanning varying degrees of partial observability, AAWR synthesizes reliable active perception behaviors that outperform all prior approaches. When initialized with a "generalist" robot policy that struggles with active perception tasks, AAWR efficiently generates information-gathering behaviors that allow it to operate under severe partial observability for manipulation tasks. Website: 
https://penn-pal-lab.github.io/aawr/

Real- Reinforcement Learning of Active Perception Behaviors
Edward S. Hu Jie Wang"./
Xingfang Yuan".l
Fiona Luo'
Muyao Li'
Gaspard Lambrechts
Oleh Rybkin
Dinesh Jayaraman'
Equal contribution University of Pennsylvania
University of Liège
SUC Berkeley
Abstract
A robot's instantancous sensory observations do not always reveal task-relevant state information. Under such partial observability, optimal behavior typically involves explicitly acting to gain the missing information. Today's standard robot learning techniques struggle to produce such active perception behaviors. We propose simple real-world robot learning recipe to efficiently train active percep- tion policies. Our approach, asymmetric advantage weighted regression (AAWR), exploits access "privileged" extra sensors at training time. The privileged sen- sors enable training high-quality privileged value functions that aid in estimating the advantage of the target policy. Bootstrapping from a small number of poten- tially suboptimal demonstrations and an casy-to-obtain coarse policy initialization, AAWR quickly acquires active perception behaviors and boosts task performance. evaluations on manipulation tasks on robots spanning varying degrees partial observability. AAWR synthesizes reliable active perception behaviors that outperform all prior approaches. When initialized with a "generalist" robot policy that struggles with active perception tasks, AAWR efficiently generates information- gathering behaviors that allow it to operate under severe partial observability for wtt.://nnn.allllaaaai.u..b...b..
manipulation tasks. Website:
Introduction
Any organism needs to extract information from the world via its sensory apparatus to make decisions, solve tasks, and survive. One strategy have high bandwidth and sophisticated sensors like the human eye, to sense as much information as possible and embrace the "blooming, buzzing confusion of the senses" [ this entails, subject to natural limits, such as a local field view. Another strategy is use the ability to move around in the world to gather new information and overcome our sensory limitations we scan our eyes across crowded party to find a friend, and polish our glasses to get clearer view. Such information gathering behaviors perception based on whether they only move sensor around the world or they also alter the world. In the following, except when the distinction is particularly pertinent. In this work, we are interested in learning active perception behaviors to compensate for the limitations of various sensory setups in robots, ranging from entirely blind robots operating purely from proprioception, to robots operating with sophisticated multi-camera setups.
called active [2] interactive [
will use "active perception" as shorthand to refer all such behaviors,
has been hard learn useful active perception behaviors robotics, and not for lack of trying 12 Of the techniques commonly in vogue for robotics, imitation learning is ill-suited because acquiring optimal active perception demonstrations can be cumbersome and unnatural (c.g. forcing a teleoperator to look through wrist cameras). In theory, RL should be able to learn active perception behaviors from interaction, but in practice it too sample-inefficien even in fully observed settings, leave alone the partially observed settings where active perception is relevant. Moreover, sim-to-real
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Object?
Found
Figure I: Left: A robot with single wrist image struggles find objects in heavily occluded scene. Right: An active perception policy searches through potential hiding spots to find the target object.
transfer hard for such tasks because it tied closely sensory capabilities, and many sensory observations of interest, including depth, LIDAR, RGB cameras, audio, touch, contact, force sensors, etc., are all hard simulate well enough reliably learn transferable behaviors. This difficulty is also reflected in the inability of today's 's state-of-the art generalist policies trained on massive amounts of robot tele-operation data perform even simple search tasks, as will show in experiments.
We demonstrate that bootstrapping from suboptimal demonstrations and incorporating rewards in an cifline-to-online RL algorithm is viable approach for synthesizing active perception behaviors. Crit- ically, the RL algorithm must be designed appropriately, and we theoretically derive that asymmetric access extra sensors for critics and value functions is important correctly estimate supervision signals for the policy in partially observed settings. Specifically, we take a weighted behavior cloning approach, by extending advantage weighted regression (AWR) (33166 incorporate privileged, training-time time only observations. We call this approach Asymmetric AWR (AAWR). As an example, in one experiment, we give critics privileged access to object detectors to train open-loop policies that only receive proprioception and initial object positions. In another experiment, we give critics privileged access to privileged segmentation masks train search policies visually cluttered scenes.
Our key contributions are as follows: (1) We efficiently train real-world active perception policies by introducing AAWR, which uses privileged value functions to better supervise the policy. (2) We provide theoretical justification for using privileged advantage estimates for AWR in POMDPs, by showing that maximizing the expected policy improvement in POMDP results in the AAWR objective. (3) We demonstrate that AAWR effectively learns a variety of active and interactive perception behaviors in 8 different settings types of simulated and real robots, and varied tasks.
varying types of partial observability, multiple
Related Work
Active perception policies are frequently trained to optimize information--heorriiic objectives such as uncertainty reduction and next best viewpoint selection [77 21) Such approaches used for task- agnostic applications like object tracking [8] 222,, scene reconstruction [1] 23 24 pose estimation
[12] or free-space navigation 20 However, such approaches are not applicable to our setting in variety of ways. First, many assume the ability to freely query views of a scene, without regard to task constraints [9]. In our manipulation settings with clutter, there are many informative viewpoints that are difficult to reach due to physical constraints. Next, such information- locate a toy, human may naturally look in drawers, shelves, cabinets, or other metrics do not incorporate such task information, and
metrics are
task-relevant
storage areas. But these information-theone may find the unseen back of shelf just as interesting. In short, for training active perception policies for robots, we desire a more task-centric active perception approach that considers the constraints of the task and does perception-improving behavior to maximize the task success rate, instead of task-agnostic metric.

---

Imitation and reinforcement learning are natural ways to synthesize task-relevant active perception policies, by using demonstrations and reward functions. Some prior works 25] use imitation learning to train active perception policies on real robots, but performance is bounded by the demonstrator. However, acquiring optimal active perception demonstrations can be quite cumbersome (e.g. forcing human tele-operators to look through wrist cameras). On the other hand, RI. approaches do not impose this burden, but real-world RL approaches for active perception [26] 27 require heavy instrumentation (e.g. for constructing task-specific volumetric maps), limiting generality. Without such assumptions, RL methods are often limited to simulation due to sample inefficiencies [8 [2]]. Sim2real transfer, however, is not easily applicable to active perception tasks. This is because active perception tasks are closely related to sensory capabilities, and accurately simulating sensors (e.g. RGB, depth, touch, etc.) is difficult. Relative to prior work, we propose an RL method that efficiently learns active perception behavior on real robots while requiring minimal instrumentation (e.g. uncalibrated RGB cameras) at inference time.
To do this, we operate in an asymmetric training setting, exploiting privileged information during training time to improve policy training [31 32. Privileged information approaches have been widely successful solving partially observed tasks [33 34 and have been deployed on real world robots with sim2real transfer K5 35 As mentioned above however, it is difficult to perform sim-to-real transfer for active perception problems. Further, asymmetric RL approaches for sim2real 37) are designed to exploit billions of privileged simulator state transitions [39], infeasible for privileged training in the real world, where we only have small amounts of potentially noisy privileged observations. We develop new "asymmetric advantage weighted regression" RL algorithm that is more capable of learning efficiently in the real world, exploiting privileged additional sensors.
B0
3 Asymmetric Reinforcement Learning in Active Perception POMDPs
Consider a robot tasked with finding a toy in a cluttered room using just its wrist camera, as seen in [Figure T, The toy's 's location is hidden to the robot, and it must scan the scene with its wrist camera in an efficient search path to find the toy quickly. These types of tasks where the robot has limited sensing but the reward and dynamics is dependent some hidden environment state (e.g. toy location), are naturally modelled by partially observed Markov decision processes (POMDPs) [40]
A POMDP is represented by the tuple action space, and O the observation space. The dynamics are described by the transition density T(s-1 $t, az), the reward density R(r, st. at)).). the observation density E(o; | st), and the initial state density P(sp). For the search task of Figure the state would include the robot position and toy location, while the observation would be the wrist camera view. The goal of policy synthesis in POMDP to find an optimal policy π that maximizes the expected return J(TT) = E Ei=p ?'re). where the discount factor weights the importance of future rewards. In POMDP, such a n² generally requires access to the complete history h = (oo, 4g..... 0t) E H of past observations and actions. This contrasts with an MDP, a special case of POMDP with Or, in which the optimal policy depends only on the current state.
where S is the state space. A the
Back to POMDPs, it is usually impractical to learn policy conditioned on the full history, since its input space would grow exponentially with time. Instead, it is common to consider an "agent state" f: H + Z that is recurrent in the sense that 2, f (h+.) sliding window. Then, the policy π E Z + A A(A) must map from the agent state. Interestingly, when using an agent state and such policies, the POMDP can be transformed into an equivalent MDP whose state (st, 2x) includes both the environment state and the agent state, with policies E II conditioned on the latter state only [41444]
u(f(h,,1). 0,-1, such as a
3.1 Background: Advantage Weighted Regression (AWR) for Markov Decision Processes
policy iteration algorithm for fully observed MDPs whose policy update objective is written as a behavior cloning loss, weighted by the estimated advantage of the transition. AWR is presented as a versatile algorithm that is able to leverage offline / off-policy data as well as on-policy data. More formally, at each iteration, AWR seeks to find a policy m: S + ((A) that maximizes the expected surrogate improvement, 7(T) = Es-dy(s) Ea-s(a)s) (s, a) x ((x) (p) with respect a behavior policy μa, under KL constraint Eewd,(() [L[(T() s) p( s)] < E. The behavior policy μ typically corresponds to the mixture
Advantage weighted regression (AWR) 13
3

---

of all past policy iterates that generated the dataset of online interactions Don When relaxing the KL constraint with multiplier / O, optimizing this soft-constrained objective is equivalent to maximizing the final AWR objective:
LAwR(T)
E (e) ampp(a)s)
E
exp (A" x(a 8s
The original AWR algorithm 14] used either a return-based estimate or TD(A) estimate of the advantage, by learning a value function with Monte Carlo estimation. Followup works [13 T6] used a critic-based estimate of the advantage by learning a Q-function with TD learning, which improves sample efficiency by better leveraging off-policy samples. We build these latter works. For more detailed background on AWR and related approaches, see Appendixx B
3.2 The Need for Asymmetric Training in POMDPs
We now derive the AWR objective for POMDPs, showing why requires asymmetry during training. We also show that the unprivileged value functions associated with naive application of symmetric AWR cannot be learned by TD learning.
Asymmetric and Symmetric AWR for POMDPs. We aim to train policy Z - A(A), conditioned on the agent state & (equivalent to history h = (o)...o:)) to maximize the return in POMDPs with AWR-like objective. We consider the asymmetric learning paradigm in which the environment state available during training (offline or online) but not during policy deployment. We introduce the asymmetric AWR (AAWR) objective:
E CAAWR(T) (s.z) (5,2) a>a(al=)
exp (A"(s, 2, a) log (a 2)
(2)
where A",s, 2, = (", a) (M 2) is the privileged advantage function, with Q' (s, 2, a)
and ("(s, 2) the privileged critic and value functions, formally defined in Appendix See Figure
for visual overview of the loss.
Privileged Supervision V
Policy
Ot
π
po 01)
Privileged Critics
Advantage
LAAWR As X Polor or)
ot
at
V
A
Weighted Regression
Figure 2: Top row: The policy receives the partial observation. Bottom row: Privileged observations or state, available only during training, are given the critic networks to estimate the advantage. The advantage estimates are used as weights in the loss, providing privileged supervision to the policy.
the environment state s is unavailable during training, one natural strategy is to solely use agent state to estimate the advantage. We call this unprivileged variant the symmetric AWR (SAWR) objective, which is justuuuuuaate Equation (2) with environment state s removed from all terms.
Why is LAAWR the right objective to implement AWR for POMDPs? To show this, we start by observing that the original AWR objective was derived as constrained policy improvement in an MDP setting. To apply AWR to the POMDP setting with policy x E II Z - A(A), we can consider the equivalent MDP with state (s, z), as discussed in [Section We then closely follow the original derivation in this MDP by additionally constraining the policy be in II Z A(A).

---

Theorem 1 (Asymmetric Advantage Weighted Regression). For any POMDP and agent state f: H Z, the Lagrangian relaxation with Lagrangian multiplier strained optimization problem,
> of the following con-
max EEl (s.))dd((ss)))
R
E (a)()
[A* z, n,)
S.L. (3,4))
E
[L(T( |2) ual: 2)] s
(4)
equivalent to the following optimization problem: maX LAAWR(T)..
The proof is given in Appendix C Cand concludes the validity of the AAWR objective. In addition, we also show in (npppeddix tht optimizing the SAWR objective does not recover the correct solution, because an advantage estimator depending on agent state & only is insufficient for estimating the advantage of the equivalent MDP, whose state is (8, z). In the example in Figure it is clear that privileged advantage estimator with access to toy locations will better estimate success.
Implementation Details. To instantiate asymmetric advantage weighted regression, train Vi and critic Q" networks to compute the advantage, mirroring extensions [65] of AWR that train critics to better leverage off-policy data instead relying on MC returns. To train the networks, we choose IQL [5]. a well known Q-learning algorithm known for its effectiveness in offline RL, offline-to-online RL finetuning [46) and real robot RL (47] tasks. The networks are trained using IQL's expectile regression objective, see[Appendis A for details.
In our POMDPs, in the symmetric setting, the unprivileged advantage estimator would be Aōv (2t, az) (" ae) Vz)(.. In the asymmetric setting, the privileged advantage esti- mator would instead be W0r(ss: 2t,0a) ae) (5,,,, 2t, a.) V/"(s,,, 21). In Appendix E, we show that the privileged value functions
the fixed point of the Bellman equations described by IQL's objective. In contrast, we show that the unprivileged value functions are not the fixed point of their corresponding Bellman equations, which further motivates the use of AAWR instead of SAWR.
We consider an asymmetric learning setting which the state 8, or privileged observations of from additional sensors are available during offlin / online training time, but not during policy deployment. The privileged critics take in either observation and state (0,, 8t), or the augmented observation (o (o,, of)) while the policy only receives Ot.
Al Algorithm 1 AAWR Offline-to-Online Training
Algorithm 2 Deployment
Require: privileged ot, partial o, (- (). Q, V
Require: partial
1: for i= = 1 to Not do
I: for = to T do
2: Update Q, V using Don and IQI. loss
2: f env.step(x(( o))
3: Update æ using Don and Eq. 2
4: for i = 1 to Non do
5:
Collect {(o, of ae Tz, O1+1 D with π
6: Dan + Don U ((o,, of at, Tt, 0t+1, o 1
7: Update Q, V using Dan: Dar and IQL loss
8:
Update π using Don, Don and Eq. 2
Figure 3: Left: The policy is trained using privileged sensors on offline / online data. Right: After training, privileged sensors are longer available and only the policy is deployed.
Privileged Training We follow the offline-o-oonline RL paradigm [15 45 47 50] where the policy and value functions are first pre-trained offline data using offline RL, and then are further fine-tuned with online interaction in the environment. Following lines 1-3 of Alggorihmm
Given the offline data D.n. we update Q, V using the IQL objective and with the Equation (2) for Nor gradient steps. Next, in lines 4-8 of Algorithm I We execute the policy in the environment and store online transitions into buffer Don- We sample equal number of transitions from both buffers to form batch, following best practice from prior work [47 5II. Using the batch, we update Q, V using the IQL objective and स with the [Equation (2)
Unprivileged Deployment: During deployment, only partial observations remain available, which the policy uses to output actions, seen in Algorithm 2
5

---

Experiments
Camouflage Pick
Fully Ob. Pick
Active Perception Koch
Blind Pick
10
Bookshelf-P
Bookshelf-D
Shelf-Cabinet
Complex
Figure 4: We setup 8 different environments with diverse sensor setups and tasks to evaluate active perception behavior. Bottom row, we label the hiding spots for target objects.
In our experiments, we evaluate AAWR's ability learn active and interactive perception behaviors in variety of tasks. We aim to answer the following questions. Does AAWR learn active perception behaviors more efficiently than other approaches? Does AAWR work in both offline-to-online and purely offline training settings? Does active perception improve the ability of modern generalist policies solve partially observed tasks?
4.1 Task setups
We evaluate AAWR in 8 different tasks, spanning both simulated and real world setups, see Figure for images of all tasks. The tasks are grouped into simulated active perception tasks: Camouflage Pick, Fully Obs. Pick, Active Pereeption Koch, and real active/interactive perception tasks: Blind Pick, Bookshelf-P, Bookshelf-D, Shelf-Cabinet, Complex. In Table I sensor setups, rewards, demonstrations. training budget, etc.
we detail task properties like
In the handoff tasks (Blind Pick, Bookshelf-P, Bookshelf-D, Shelf-Cabinet, Complex), we use three metrics to evaluate the search behavior. First, Search a score the policy behavior with the following rubric: 1) the target appears in wrist camera frame, 2) the policy approaches the target, and 3) the policy remains fixated on the target after 5 timesteps. Next, Completion is the rate which To successfully grabs the object after switching from the active perception policy. Finally, Steps is amount of steps the policy takes to complete the task. See Appendices H for comprehensive descriptions of the tasks.
4.2 Baselines
We compare against symmetric advantage weighted regression (AWR) without privileged information. Its implementation is identical that of AAWR, except for inputs of the critic and value networks. Next, we compare against standard behavior cloning (BC), which performs imitation learning on the successful trajectories in the dataset.
4.3 Results
Simulated Active Perception tasks. Fig. 5 shows these results, and videos are Supp and website. First, compare against AWR and BC two simulated active perception tasks with varied degrees

---

Table 1: The tasks vary in embodiment, nature of partial observability, privileged sensors, reward, demo quantity and quality. and training budget.
Task
Target Obs.
Reward Demos
Offline Steps
Description
Platform
Privileged Obs.
Online Steps
Camouflage Pick
Side Cam True Obj. Pos
Sparse
20K
Pick up barely visible object
Sim. Koch
100 suboptimal
80K
Fully Obs. Pick
Side Cam
Sparse
20K
Pick up fully visible object
Sim. xArm
True Obj. Pos
100 suboptimal
20K
AP Koch
Wrist Cam
Sparse
100K
Locate then pick up object
Sim. Koch
True Obj. Pos
100 suboptimal
900K
Blind Pick
Joints, Init Obj. Pos Obj. Pos Estimate
Dense
20K
Pick object from proprioception
Real Koch
100 suboptimal
1.2K
Bookshelf-P
Wrist Cam, Joints Bbox, Mask
Dense
100K
Look for object & switch to xo
Real Franka
150 suboptimal
0
Bookshelf-D
Wrist Cam, Joints Bbox, Mask
Dense
100K
Look for object switch to To
Real Franka
~ 100 suboptimal
0
Shelf-Cabinet
Wrist Cam, Joints Bbox, Mask
Dense
100K
Look for object & switch to To
Real Franka
~3 suboptimal
0
Complex
Wrist Cam, Joints Bbox, Mask
Dense 50 expert
100K
Look for object switch to To
Real Franka
0
Camouflaged Pick
Fully Obs. Pick
AP Koch
1.0
1.0
0.3
AAWR
AANR AWR
AWR
0.8
Success Rate
Success Rate
BC
BC
0.2
0.6
0.6
Success
0.4
0.1
AAWR
0.2
0.2
Distilatoo
VIB
0.0
0.0
10K
60K
-10K
200K
400K
Stepps
Steps
Steps
Figure 5: Evaluation curves for the simulated experiments, over 10 seeds per method. The shaded
regions indicate the offline pretraining phase. AAWR outperforms baselines in all simulated tasks.
of observability, Camouflage Pick and Fully Obs. Pick. In both tasks, AAWR outperforms its non- privileged counterpart AWR, and BC in Camouflage Pick and Fully Obs. Pick, by approximately 2x and 3x respectively. While the gains from using privileged observations are in line with expectations for Camouflage Pick, where inferring the tiny marble from RGB is difficult, it is interesting to gains even in a fully observable task, where the object position is always clearly inferable from vision. We hypothesize this is because the non- privileged critic needs to learn to extract the object position from pixels, whereas the privileged critic does not.
One natural question is: how does AAWR compare against other popular approaches that leverage privileged information? On the simulated Active Perception Koch task, we compare AAWR against Distillation [39]. which first trains a privileged expert policy and then distills it into partially observed policy. We also compare against a variational information bottleneck approach (VIB) [52). which gives the policy regularized access to privileged information.
As seen in Figure 5] only AAWR achieves 100% on the task by learning to scan the workspace, while other privileged baselines stagnate. Distillation gets high initial success but plateaus at 80%, learning a local maxima strategy of approaching the center of the workspace. This fails when the object is in the workspace corners, out of camera view. This behavior arises due to the privileged expert, unaware of the camera's limited field of view, that goes straight for the object. AAWR is a policy iteration algorithm that works with both offline and online data, which allows it to discover the
7

---

active perception behavior through online exploration, despite starting with suboptimal demos. VIB collapses during evaluation because privileged information is not available. even though was trained to minimally use privileged information. See Appendix H for details and the wehsite for videos.
Real Interactive Perception task. Here, con- tinue comparing to AWR and BC, this time including purely offline variants of AAWR and AWR, on the
Table 2: Koch Interactive Perception.
Method
Grasp %
Pick %
Blind Pick task in the real world. As seen in Table 2 both offline and online variants of AAWR outperform its unprivileged counterparts, and BC. Among the of- fline methods, BC performs worst, exhibiting jerky and inaccurate movements. Both offline AWR and offline AAWR demonstrate better approaching and picking behavior, but offline AWR missed grasps and released the block frequently.
BC
47
41
Off. AWR
65
62
On. AWR
71
55
Off. AAWR (ours)
88
71
On. AAWR (ours)
89
Offline AAWR demonstrates more suboptimal behavior, such as releasing the candy after grasping. We observed that after online finetuning, the suboptimal behavior of offline AAWR is reduced, and online AAWR demonstrates the most consistent and robust open-loop picking behavior. Online AAWR reliably places its gripper over the object for grasping. In cases when the object slips from its grasp, the policy attempts regrasp at the original location. See videos on the website.
Table 3: In the active perception handoff tasks, AAWR consistently outperforms baselines in terms of search behavior, completion, and speed. Bold = best, underline second best.
Method
Bookshelf-P
Bookshelf-D
Shelf-Cabinet
Camplex
Search
Completion
Steps
Search
Completion
Steps
Search
Completion
Steps
Search
Completion 1
Steps
1
1
AAWR
92.4
44.4
36.6
81.3
44.4
26.9
78.2
40.0
46.3
73.2
50.0
43.0
AWR
79.6
0.0
34,0
62.6
16.7
30.2
52.3
10.0
38.0
33.2
40.0
67.0
29.9
20.0
84.0
47.7
16.7
22.5
28.1
15.0
125.0
15.0
77.0
To
11.0
16.7
263.3
66.7
33.3
229.7
10.0
10.0
280.0
29.6
20.0
252.5
Exhaustive VLM+.
64.2
44.0
105.4
96.0
22.2
106.7
$2.8
45.0
183.0
78.2
30.0
297.0
31.4
27.8
322.3
33.2
16.7
281.8
28.2
15.0
382.0
14,8
10.0
374.7
Handholding Foundation VLA Policies for Real Active Perception tasks. We find that To. generalist foundation policy for manipulation tasks, is not good at searching tasks such as finding target objects in cluttered scene from just a wrist camera (see in /Figure 4). Tu is fundamentally limited by not having memory, which hampers its ability to handle POMDPs. We now evaluate our approach's 's ability to generate helper policies that condition on history (see AseenAppedixx F) handhold To policies up to a configu configuration from which they could reasonably succeed. To achieve this, we propose switching framework for the policy where an active perception policy is first run, and an object detector periodically checks the wrist image for the target object. Once the object is detected across two consecutive intervals, the robot switches to To to grasp the object.
We set up four realistic active perception tasks where the robot must search through a cluttered scene to find target object and grasp it up (see Figure 4). In the Bookshelf-P and Bookshelf-D tasks, place a target object (either a toy pineapple or duck) on one of the three shelves, requiring the robot to the shelves both vertically and horizontally and stop at good viewpoint. In the Shelf-Cabinet task, we add an additional cabinet with drawers and hiding spots near its top and inside cabinet door, increasing the complexity of the search. Finally, in Complex task, we add additional shelf on the floor, whose shelves are completely occluded from all side camera viewpoints.
We compare against several basclines. Exhaustive is an engineered controller that methodically goes through all hiding spots in the scene, but is slow due to its heuristic pathfinding. Next, we again compare unprivileged AWR and BC as helper policies. We compare against To itself, as well as VLM+ variant that queries the Gemini-2.5 VLM (53) to generate language commands that are executed by Ty. similar to the Hierarchical VLM-VLA baseline proposed in [54]. As mentioned earlier in [Section 4.1, we use the Search metric to judge the searching behavior, Completion metric to measure how useful is the handoff for no. and Steps for speed. See Ace|Appedixx Foro more details.
For the Bookshelf-P, Bookshelf-D tasks, we collected 250 demonstrations split among four demon- strators with varying qualities, resulting in suboptimal dataset of about 50% To success rate. For
8

---

AAWR
AWR
Figure 6: Example rollouts in the cabinet shelf task. AAWR explores the top left of the cabinet and locates the target object (red arrow) while AWR only finds small glimpse before drifting away.
Shelf-Cabinet, Complex, we collected smaller but higher quality datasets of 35 trajectories with 74% success rate and 50 trajectories with 94% success rate respectively. We perform offline training of AAWR, AWR, and BC on the same demonstration dataset, and for the same amount of gradient steps, before evaluating in the real world.
AAWR consistently outperforms baselines all metrics, learning sensible active perception behavior to aid generalist policy. AAWR always outperforms non privileged AWR and BC, thus validating the usefulness of privileged information and the use of offline RI. over supervised learning. We find that To and VLM+#o both search poorly they tend to take inefficient movements and fail to track the object. Exhaustive has decent search and completion rates, but takes much longer than AAWR. In fact, when normalized for time taken, AAWR scores 2-8 times higher in Search and Completion metrics than Exhaustive. See Table 5 for the time- normalized relative metrics.
As seen Table 4.
In the Bookshelf tasks, AAWR first learns to zoom out of the scene to see multiple shelves, then seans from bottom to up, and then approaches the target object once located. The AWR and BC baselines follow relatively fixed search path that approaches the shelf, but the policies failed to efficiently scan the shelves. Even if the target object appears the frame, the policies do not fixate on the object, reducing their search score and To success rate. In the Shelf-Cabinet task, AAWR searches through the right bookshelf, before moving to the left cabinet. Both AWR and BC do not thoroughly search the scene, preventing them from finding objects placed in the left cabinet's drawer. In Complex, AAWR searches the bottom shelf, the right shelf, and then the left cabinet (see Figure I). See Figure 6 and the website for examples.
5 Conclusion and Limitations
We aim to train active perception policies in the real world to allow robots to overcome their sensory limitations, useful problem for which current approaches have shown limited success. We propose asymmetric advantage weighted regression (AAWR), simple weighted behavior cloning technique that leverages privileged observations during training to efficiently train active perception policies. We provide theoretical justification for AAWR, by deriving the validity of the AAWR objective for POMDP as opposed to its symmetric counterpart. Then, we show that AAWR successfully Icarns active and interactive perception behaviors in 8 different simulated and real world tasks.
Despite our promising results, there is much room for improvement. Instead of switching between policies to execute active perception behaviors, AAWR could be used to directly fine-tune generalist foundation policies. Other forms of privileged information could be considered, such as foundation model outputs. Instead of using prespecified privileged information, useful features from the addi- tional information could also be explicitly selected through representation learning. Because this work overcomes limitations of existing methods for active perceptions tasks, it would be worth exploring the scalability of AAWR to tasks with longer horizons where information -athiriig challenges compound. Finally, AAWR could be applied to many other partially observed robotic (or other) tasks.

---

Acknowledgments
This work was supported by the DARPA TIAMAT HRO01 1249042, NSF CAREER 2239301, ONR 100014-2-11-2677, and NSF SLES 2331783 grants. It used computing resources from the National Artificial Intelligence Research Resource Pilot (NAIRR 240077). Gaspard Lambrechts is postdoc- toral researcher of the Fund for Scientific Research (FNRS) from the Wallonia--russee Federation in Belgium. The authors would like to thank the GRASP lab, anonymous NeurIPS reviewers, Antonio Loquercio, Kostas Daniilidis, Lei Zhang. Tianyou Wang, Siyu Li and Jiahui Lei for constructive feedback and support. The authors thank James Springer, Will Liang, Johnny Wang. Sam Wang and Tau Robotics for robot support.
References
[1] William James. The principles of psychology. Henry Holt, 1890.
[2] Ruzena Bajesy. Active perception. Proceedings of the IEEE, 76(8):966-100) 1988.
[3] Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. Interactive perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6):1233 1291, 2017.
[4] Ricson Cheng, Arpit Agarwal, and Katerina Fragkiadaki. Reinforcement learning of active vision for manipulating objects under occlusions. In Aude Billard, Anea Dragan, Jan Peters, and Jun Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedinngs of Machine Learning Research, pages 422-431, PMLR, 29-31 Oct 2018. URL httpp://pproceedinge. mlr ppress/v7/chos chengg8he html
[5] Wenyan Yang, Alexandre Angleraud, Roel S Pieters, Joni Pajarinen, and Joni-Kristian Kämärää- nen. Seq2seq imitation learning for tactile feedback-baad based manipulation. In 2023 IEEE International Conference on Roboties and Automation (ICRA), pages 5829-5836. IEEE, 2023.
[6] Ian Chuang, Andrew Lee, Dechen Gao, and Iman Soltani. Active vision might be all you need: Exploring active vision in bimanual robotic manipulation, 2024.
arxiv org/
abs/2409. 17435
[7] Michael Krainin, Brian Curless, and Dieter Fox. Autonomous generation of complete 3d object models using next best view manipulation planning. In 2011 IEEE international conference on rubotics and automation, pages 5031-5037. IEEE, 2011.
[8] Ehud Rivlin and Héctor Rotstein. Control of a camera for active vision: Foveal vision, smooth tracking and saccade. International Journal of Computer Vision, 39:81-96, 2000.
[9] Yang Liu, Pansiyu Sun, and Akio Namiki. Target tracking of moving and rotating object by high-speed monocular active vision. IEEE Sensors Journal, 20(12))6777-6744, 2020.
[10] Akshay K Burusa, Eldert J van Henten, and Gert Kootstra, Attention-driven next bext-best-view planning for efficient reconstruction of plants and targeted plant parts. Biosystems Engineering, 246:248-262, 2024.
[11] Andrew J Davison, Walterio W Mayol, and David William Murray. Real-time localization and mapping with wearable active vision. In The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings., pages 18-27. 27. IEEE, 2003.
[12] Kanzhi Wu, Ravindra Ranasinghe, and Gamini Dissanayake. Active recognition and pose estimation of household objects in clutter. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2015.
[13] Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. Advances in neural information processing systems, 21, 2008.
[14] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
10

---

[15] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.099353 2020.
[16] Ziyu Wang, Alexander Novikov, Konrad Zoina, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[17] Dinesh Jayaraman and Kristen Grauman. Learning to look around. CoRR, abs/1709.00507,
2017. URL http://arxiv, org/ hotgabbs/7709 00507
[18] Matthew Koichi Grimes, Joseph Varughese Modayil, Piotr W Mirowski, Dushyant Rao, and Raia Hadsell. Learning to look by peee-prri Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL http/tttt//ppeeeeiiiiee http:/ppnnreei....oooooroooooaaa
net www.neeeeottiiiie 9aaKdd-KXXKKKKU
[19] Wen Jiang, Boshu Lei, and Kostas Daniilidis. Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information. arXiv preprint arXiv:2311.17874 2023.
[20] Siming He, Christopher D Hsu, Dexter Ong. Yifei Simon Shao, and Pratik Chaudhari. Active perception using neural radiance fields. arXiv preprint arXiv:2310.09892, 2023.
[21] Tao Yi, Dongbo Zhang, Lufeng Luo, and Jiangtao Luo. View planning for grape harvesting based on active vision strategy under occlusion. IEEE Robotics and Automation Letters, 9()): 2535-2542, 2024.
doi: 10.1009/RA42222443377777
[22] Björ Browatzki, Vadim Tikhanoff, Giorgio Metta, Heinrich H Bülthoff, and Christian Wall- raven. Active object recognition humanoid robot. In 2012 IEEE international conference on robotics and automation, pages 2021-2023 IEEE, 2012.
[23] Andrew J Davison and David W. Murray, Simultaneous localization and map-building using active vision. IEEE transactions on pattern analysis and machine intelligence, 24(7):865-88) 880, 2002.
[24] Siyan Dong, Kai Xu, Qiang Zhou, Andrea Tagliasacchi, Shiqing Xin, Matthias Nießner, and Baoquan Chen. Multi-robot collaborative dense scene reconstruction. ACM Transactions on Graphics (TOG), 38(4):1-16, 2019.
[25] Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jeannette Bohg, and Shuran Song. Vision in action: Learning active perception from human demonstrations. arXiv preprint arXiv:2506.15660 15666, 2025.
[26] Tonci Novkovic, Remi Pautrat, Fadri Furrer, Michel Breyer. Roland Siegwart, and Juan Nieto. Object finding in cluttered scenes using interactive perception. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1338-8344- 2020. doi: 10.1109/ 0TKX440945.2020.99971000
[27] Thomas Pitcher, Julian Förster, and Jen Jen Chung. Reinforcement learning for active search and grasp in clutter. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7301-7306, 2024. doi: 10. 10.1100//00558592.2222222 01000008855522222108888833330 10801366.
[28] Youssef Zaky, Gaurav Paruthi, Bryan P. Tripp, and James Bergstra. Active perception and representation for robotic manipulation. CoRR, abs/2003.06734, 2020. URL https://arxiiv org/abs/2003.0677
[29] Jinghuan Shang and Michael S Ryoo. Active vision reinforcement learning under limited visual observability. In Thirty-seventth Conference on Newral Information Processing Systems, 2023. URL
[30] Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural networks, 22(5-6):544-555 2009.
[31] Gaspard Lambrechts, Adrien Bolland, and Damien Ernst. Informed POMDP: Leveraging Additional Information in Model-Based RL. arXiv preprint arXiv:2306.11488 2023.
11

---

[32] Edward S. Hu, James Springer, Oleh Rybkin, and Dinesh Jayaraman. Privileged sensing scaffolds reinforcement learning. In The Twelfth International Conference on Learning Repre- sentations, 2024. W www..neeoooounndddBeeeeee
[33] Junjie Li, Sotetsu Koyamada, Qiwei Ye, Guoqing Liu, Chao Wang, Ruihan Yang. Li Zhao, Tao Qin, Tie- Yan Liu, and Hsiao-Wuen Hon. Suphx: Mastering mahjong with deep reinforcement learning. arXiv preprint arXiv:2003. /3590, 2020.
[34] Miguel Vasco, Takuma Seno, Kenta Kawamoto, Kaushik Subramanian, Peter R Wurman, and Peter Stone. A super-human vision-based reinforcement learning agent for autonomous racing in gran turismo. arXiv preprint arXiv:2406.1256. 2024.
[35] Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asymmetric actor critic for image-based robot learning. In Robotics: Science and Systems (RSS), 2018.
[36] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning Quadrupedal Locomotion over Challenging Terrain. Science Robotics, 2020, doi: 10.1126/ scirobotics.abc5986..
[37] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: Rapid Motor Adaptation for Legged Robots. In Roboties: Science and Systems (RSS), 2021. doi: 10. 10.15607/RSS.2021
XVII.01I.
[38] Sasha Salter, Dushyant Rao, Markus Wulfmeier, Raia Hadsell, and Ingmar Posner. Attention- privileged reinforcement learning. In Conference on Robot Learning, pages 394-408 PMLR, 2021.
[39] Yuanpei Chen, Chen Wang, Li Fei-Fei, and C Karen Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. arXiv preprint wrxiv:2309.00983 2023.
[40] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2)999-134, 1998.
[41] Nan Jiang. On value functions and the agent-enviromment boundary. 19Xiv:1905.13322 2019.
arXiv preprint
[42] Shi Dong, Benjamin Van Roy, and Zhengyuan Zhou Simple agent, complex environment: Efficient reinforcement learning with agent states. Journal of Machine Learning Research, 23 1255);1-54, 2022.
[43] Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen, et al. Reinforcement learning, bit by bit. Foundations and Trends@ in Machine Learning. 16(6):733 865, 2023.
[44] Amit Sinha and Aditya Mahajan. Agent-State Based Policies in POMDPs: Beyond Belief-State MDPs. IEEE 63rd Conference on Decision and Control (CDC), pages 6722-6735, 2024.
[45] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q g-learning. In International Conference on Learning Representations, 2022. URL httpss
//openeviee iopemreeies..eet/foutttt-------e
[46] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprini arXiv:2410.20099, 2024.
[47] Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. In Proceedings of the 7th Conference on Robot Learning (CoRI.), 2023.
[48] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble, In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
12

---

[49] Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36x62244-62269, 2023.
[50] Zishun Yu and Xinhua Zhang. Actor-critic alignment for offlnneto-online reinforcement learning. In International Conference on Machine Learning, pages
PMLR, 2023.
[51] Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
[52] Kyle Hsu, Moo Jin Kim, Rafael Rafailov, Jiajun Wu, and Chelsea Finn. Vision-based manipula- tors need to also see from their hands. In Intermational Conference on Learning Representations, 2022. URL httppp///ppppereeeeeeeeee/rrtrrp
[53] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- SanntBattstt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
[54] Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang. Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, and Chelsea Finn. Hi robot: Open-ended instruction following with hierarchical vision-language-ccioo models, 2025. URL 19417
[55] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, pages 267-274, 2002.
[56] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Young- woon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Matco Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao. David Antonio Herrera, Minho Hco, Kyle Hsu, Jiaheng Hu, Donovon Jackson, Charlotte Le, Yunshuang Li. Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang. Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martin, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, and Chelsea Finn. Droid: A large-scale in-the-wild robot manipulation dataset. In Proceedings of Robotics: Science and Systems, 2024.
[57] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. pi0.5: a vision- language-action model with open-world generalization. arXiv preprint 2504:255041600 2025.
[58] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/99. /93, 2023.
[59] Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng. Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song, Yuhong Zhang, Hongjie Huang, Han Gao, Shilong Liu, Hao Zhang, Feng Li, Kent Yu, and Lei Zhang. Dino- x: unified vision model for open-world object detection and understanding. 2024. URL. https: // arriiv org/ abs/ 14347
13

---

[60] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
[61] Jess Moss. koch-vl-1,
wttpj//sssooss/ooc--------- 2025.
[62] Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszez, and Max Jaderberg. Distilling policy distillation. In The 22nd conferenc on artificial intelligence and statistics, pages 1331-1340. PMLR, 2019.
14

---

A AAWR Implementation Details
Instead of the standard policy evaluation of the AWR algorithm [6], we use implicit Q-learning (IQL) [45] and its optimistic policy evaluation, for its effectiveness offline, and offlne-to-online 46 and real robots 47 tasks. The IQL algorithm learns both a value function Vi and a critic
2 Extending IQL to our POMDPs, in the symmetric setting, the unprivileged advantage estimator
would be (z, (e) (= at) (=x). In the asymmetric setting, the privileged advantage estimator would instead be AQv Ze; ur) = Q7 Ee, az) Vo (8t,
The privileged Q-function is trained using the 1-step TD error.
LØ(Ø)=
E
(Te + 2V (5411,2411 (1,
(5)
(1
The privileged value function is trained conservatively approximate the maximization max, (g 2, a) using an asymmetric L2 loss (expectile regression):
(v(0)=
( |T 1 ( (Q) 2: (st, =))² (BL)
E
(6)
where 7 (0, 1) the expectile. As r + O, the loss increasingly penalizes overestimates of The unprivileged value functions are trained analogously.
Note that when to the standard 1 1-step TD update. In An pendix H show that the privileged value functions are the fixed point of the Bellman equations described by Eq. 5
0.5, it corresponds
Algorithm 3 AAWR Oflline-to-nllin
Require: policy n, critics Q, V, buffers Dotr Don 1: for = to Nan do
2: Update Q, V using Dorr and Eq. 5 and Eq. 6
3:
Update π using Dotf and Eq. 2
and Eq. 6 In contrast, we show that the the fixed point of their corresponding Bell- man equations, which further motivates the use of AAWR instead of SAWR.
unprivileged value functions
4:
for i= to Non do
5:
Collect {(ot, of ag Tr, oc+1: 2++1 ++)) with m
6:
Dan +- D U {(or, of ar,TT, 0++1, 1, o 1
7:
Update Q, V using Dan, Dotr and Eq. Eq.
8: Update using Don Donr and Eq.
We train the models in offline-to-
online manner. Following lines 1-3 of Algorrhhh 3] during the offline stage, Q, V, π are updated Dar for Not gradient steps the offline dataset. After offline training, on-policy trajectories
collected by executing the target policy in the environment. These trajectories are added to the online buffer Don- Following [47), we use symmetric sampling where 50% of samples are from Doff and the other 50% are from D on-
B Advantage Weighted Regression Estimators
The original AWR algorithm 14] used a return-based estimate of the advantage At (st, a:) Ex- 7 Tz4* (*(),) where (*()s, is an approximation of the value function, learned by mini- mization of,
20
(vc(V)=
E
7 Tet* -
(oi)
k=0
Future works have instead used aritic-based estimate of the advantage (5,,,, (s,, 0,)
(7 a.)
Ea-x(als.) (se, a) [5] 16 where Q" az) is an approximation of the critic function, learned by minimization of,
(T)
E
E
(re 7 (81+1, a441) (sr, a.)))
(8)
(+4) a (arle))
The typical learning procedure of AWR is summarized in Algorithm 4
15

---

Algorithm 4 Advantage Weighted Regression
Require: policy critic V, buffer Don for == to oe do
2: Collect {(st, de, P2. P2, St+2))11 with
3: Don t- Do {(8,, de, rs, $141)]1=1
4: Update V using Don and minimizing Eq.
5: Update a using Don and maximizing Eq.
C Asymmetric Advantage Weighted Regression Derivation
In this section, we derive the AWR objective for POMDPs, which results the AAWR objective. Since consider POMDP and agent state f: H + Z, consider the equivalent cnvironment-agent state MDP 44), whose state is (s, z), and restrain the class of fully observable policies 11 S x Z (A) for this MDP to the agent-state policies,
3m 11:: x1::=((a (a 8,2) = x(a V2) E E II (at 1I
(9)
Since 11 1I we have x- E 11 and we can derive the AWR objective using this restricted set
of policies following similar steps as Peng et al. [14]. In the following, we use π E II to denote the partially observable policy x(a z) corresponding to æ II with (a s, s. <a 2).
Before deriving the AWR objective for the equivalent environment-agent state MDP, let us define the (normalized) discounted visitation measure of policy TI in this MDP as:
(s, (1
p/p(st 3, 2).
(10)
t=0
In the following, we denote the current policy मे with μ and its corresponding partially observable policy with μ Since work in the environment-agen state MDP, consider the usual value functions definitions, where the state (s,
Q" (8,2,0) =E" E²
rr = 8, 2, 2, =
(11)
Va (8, = BB
rr
=
(12)
Now, let us derive an objective to improve the policy. We want to maximize the policy improvement,
(( -(T J(x
(13)
16

---

The policy improvement is related to the advantage function of
55
20
7(TT)) =
E
r'r
1
(14)
EE
r
(50, 20)
(15)
8
E
7/1 Vi (s0: 20)
(16)
E
7 (re + YVH (5141, - V (s), + Va (s0, 20) Va
(s0: 20)
(17)
E
(r, + yV (8441,2411) V (+))
(18)
Ex
A4 (st, Zz, ae)
(19)
A" (st, (st, 21) an)
(20)
t=0
(1 ?)
E
E
A" 2s)
(21)
(5.z) (. (8,2) (a/2)
where A (s, = Q* - (s, 2). The objective 77xxxx may be inefficient to optimize, due to the dependence of the expectation on through d,. Instead, we thus choose to optimize an off- off-policy surrogate objective where the samples are generated from policy μl,
(1 - 7)
E (s.s) a>)((a=))
E
A" (s, (,)22,a)
(22)
In practice, we thus seek to approximately solve the following constrained optimization problem at each iteration,
Tk+1
E arg max
E
E
A" (s, z, a))
(23)
x ElT
(8.2) (8.2) (x,2) am)((a)s)
s.t. KL( (-1 || p (- s,)), Se
(24)
Now that we have identified the desired constrained optimization problem, let prove Theorem which states that its relaxation corresponds to the AAWR objective.
Theorem (Asymmetric Advantage Weighted Regression). For any POMDP and agent state f: H. + Z, the Lagrangian relaxation with Lagrangian multiplier 3 > 0 of the following con- strained optimization problem,
max NENT
E (d.))dd (s.2) (a]=(
E
[A" 2.)
(3)
(L(T( =) || ml: [)] Se
(s..))
equivalent the following optimization problem: max LaAWR(T).
Proof. Starting from Eq. 23 we that additional constraints are hidden in .. II The first
one is SacA TTaa =) I, Vs S, Vz E Z. The second one is π E I C II+,, or equivalently. πt - 21,2) x < 52. 2), Vsl, 82 E S, Vz E Z, Va A.
17

---

Following similar steps as Peng et al. [1]. by relaxing the KL constraint in Lagrangian multiplier with multiplier S, (a 8, x i (a 8 exp
(3) (0,2;a)
(25)
x p(a 2) exp (3 A" (5, {AA
(26)
We now substitute back the two additional constraints as additional constraints, so that we project the solution on the manifold of policies II_ By minimizing the KL divergence
(ee,)))d,, (I (- 8,2) ll x(- =)] to that target, we obtain
loga(a e)e (7AM (s, (812210)
E Tk+1 arg (s.)) emp((a=s)
(27)
This concludes the proof, for μ TT.-
With the additional constraint that we use parametrized policy тe E lll we obtain
arg max E deo
log To(a 2) cxp
8 A" (s. (8,2,a))
(28)
(s.))~d (~)(a)))
Eq. 27 corresponds to the AAWR objective, and Eq. 28 is the AAWR objective in the context of parametrized function approximators.
D Problem with Symmetric Advantage Weighted Regression
While the asymmetric value functions followed the classical definitions in the environment-ageen state MDP, the symmetric value functions are not standard because is not a Markovian state. We select the following definition: A" a) Q" a) V²(2) with.
Q" (z,a)=
E
80:
8,20
2,
(29)
snnd (+/z)
VP (z)
E
80
8,
20
(30)
spid (*|=)
By definition, this choice provides unbiased symmetric value functions under the distribution dy- (8,2) induced by the current policy μ7
Q" (z,
[Q" (s,
(31)
andd (0/2)
V" (2) andd (0|2)
E
VV (s,
(32)
Let us now prove Theorem 2, which proves that the symmetric AWR (SAWR) objective different from the asymmetric AWR (AAWR) objective.
Theorem (Symmetric Advantage Weighted Regression). In general, for a POMDP and an agent state f: H Z, we have arg max LSaWr arg max LAAWR-
Proof. Combining Eq 29 and Eq. 30 we have,
A (z,0)=
E
A"
(s,
(33)
and
(a12)
Now, it is straightforward to see that the SAWR objective.
CSAWR(T) =
E
exp (A" (2.3)/3) log (ogr(alz))
(34)
(s.)) amp((l))
E
exp
A" (s, 18 log 2)
(35)
(1.3) a>م((/>))
svdd n (812)
18

---

does not correspond to the AAWR objective,
(A = E
E (s.z)=dp (5.2) amp
exp (A" (s,2,a)) log
(a lo 2)
(36)
Indeed, we apply Jensen's strict inequality over the the strictly convex exponential function, under the assumption that the distribution dz- (s 2) is nondegenerate.
E
Problem with Symmetric Temporal Difference Learning
Let us consider the asymmetric Bellman equations for the environment-agent state MDP,
Q" (s, (8,2,00 a)
B"
[r- + 7Q" (81+1, Fe4 ac-1) 84 2t 2, 0z 0
(37)
Since the underlying Bellman operator is 7-contractive, these equations have unique solution Q" Because the environment and agent states form a Markovian variable (s, z), by definition of Q we have Q" = Q" As a consequence, we also have 7 = V* where V is the unique fixed point of its analogous Bellman operator.
Let us now consider the symmetric Bellman equations for the environment-agent state MDP,
Qu (z,a)
E yodd (0'12)
RJ
(r. + ~ (e++1,0+11) = a
(38)
It is interesting note that, by bootstrapping with Q" this Q-function considers the distribution of the state (8++1, Zt+1 8t, at) from the second timestep to be p( 8t: 2t, az)dy (8++1 12++1) instead of the true distribution p(8++1, Ze+1 8t, Zz, az). As result, by telescoping, obtain,
Q" (z, a)
E
Σ B"
r(1 2z, az) so s, 20 al (s((=))
2, 00
(39)
and (1/2)
where r(s, 2) =Eir 1t S, 2
2, ae = a]. contrasts with the true Q-function, which writes,
Q" (s, a) 1E"
7'r
80 8,20 2,00
(40)
y BB" Σ
[r(st, 24, 20, 20 2, a]
(41)
and (a|=)
It can be concluded that the unprivileged fixed point Q" and the unprivileged Q-function Q" can be different, as soon as the distribution p(s,, 2ı, az 80, 20, un) is different from p(a,, at so, 20, 4o) dd (st | 2x) at any timestep t.
19

---

Active Perception Experimental Details
F1 Task Definition
In these tasks, the robot must find objects placed out of view in cluttered environments. Similar to how human may try find ingredients in messy kitchen, the robot must move its camera around occluders, zoom into hard to see spots (i.e. behind drawers), and zoom out to increase its overall view of the scene.
Data Collection
Privileged Information
AAWR
O2
W
Mask
Spacemmuse
A. Training Phase
DINOv2
B. Inference Phase
Occupncy
Search Rubric:
33% object appear in scene
66% object folls torget region
New Environment
Perception & Robot State
100% object footes region
Search...
Remember... ....
Find...
Focus...!
Handaver...
Grasp
active perception policy
pio policy
Figure 7: Franka Robot Search Diagram: A visual overview of the Franka Active Perception experiments. Top half: Offline trajectories are collected through teleoperation collecting privileged masks and unprivileged wrist image and history features. The policy is trained using AAWR. Bottom half: During evaluation, only the unprivileged search policy is run to perform active perception to find the object, before switching the generalist policy. The policy's search behavior is graded with the rubric.
In this experiment, we train "helper" active perception policy that searches the scene for the target object, and once located, hands off control generalist VLA policy to pick up the object. This addresses weakness of such generalist policies- that they are typically trained in fully observed situations, and do not generalize to partial observations.
We set up four tasks, ordered in increasing complexity, where the robot must either find toy pineapple or duck. Objects and visual distractors are randomly placed at the beginning of each episode, the bookshelf and cabinet are placed in same position.
Bookshelf-P and Bookshelf-D: The robot must find either the pineapple or duck in three-tier vertical bookshelf with multiple visual distractors (Fig. Figure 9).
Shelf-Cabinet: We make the scene more difficult by adding a cabinet on the left side. The cabinet creates additional hiding spots its top drawer and shelf over the drawer.
Complex: In addition to the bookshelf and cabinet, we add horizontal bookshelf, There are several completely occluded spots, such the bottom cabinet drawer and objects placed in the horizontal bookshelf.
20

---

(a) detection: score
[b) first detection: score 33
(c) move into region: score 66
(d) stay & center: score 100
(a) starting position
(b) zoom out to find pineapple
(c) once find, approach closer
(d) hand-off to pio pick up!
Figure 8: The Search % metric gives points for spotting, approaching, and fixating the object.
the UTrrpt Object and
Figure 10: The Complex task has heavy occlu- sion in the left cabinet, and bookshelf on the floor.
Figure 9: Robot hardware configuration: Franka Panda with wrist & side view camera.
F.2 Metrics
Search %: 3-point rubric for grading search behavior, Figure 8 for example.
1. Policy spots the target object anywhere in the image [33%]]
2. Policy moves until target object falls into the target region of viewpoint [66%
3. Policy fixates on the target object inside the target region [100%].
the grasping success rate of тo (100 timestep limit), after switching from the active perception policy.
Completion:
Steps: mean number of steps for the policy to complete the first two stages of the rubric (finding and approaching). Episodes that fail reach the first two stages count timeout (Tmaz 300 steps).
we also compute normalized metrics to take into account time efficiency, by dividing Search and Completion by the Steps. We then compare all methods relative to the Exhaustive baseline, by dividing the time-normalized Search and Completion metrics by the Exhaustive baselines" time- normalized Search and Completion.
Table 5
F.3 Hardware and Scene Setup
We used the DROID robot setup which consists of 7 DoF Franka Emika Panda Robot Arm, Robotiq 2F-85 parallel-jaw gripper, wrist-mounted ZED Mini RGB-D camera and two side-mounted ZED stereo cameras. The DROID set-up enables the usage of the generalist VL.A policy xo [57] specifically the FAST-DROID checkpoint.
21

---

F4 Policy Design
1. Observation Space:
Partial observation: wrist RGB 84x 84 image, end effector position of last 6 timesteps,
occupancy grid feature.
Privileged observation: Segmentation mask and bounding box of the target object.
2. Action Space: Cartesian and angular velocities of end effector frame,
dt = [Var Ugs May Wholl, Wpitch; Wyaw]. We use action chunks of length
The search policy and privileged critic networks constructed using encoder head architecture. We first detail the input processing steps below, as they are shared for AAWR, AWR and BC network:
The wrist image is first fed into frozen DINO-V2 58 encoder (ViT-S14). and the resulting DINO- V2 features are reduced into a 256 x 16 dimensional latent using PCA. Next, the occupancy grid feature is constructed by projecting the historical camera rays (inferred through the gripper position) into the XZ. dimension of the robot frame, resulting in 2D occupancy grid where elements are the camera ray has passed through it in the episode.
The search policy takes in the wrist image features, a history of the last 6 gripper positions, and the occupancy grid feature. The occupancy grid is first processed through a convolutional encoder, and then is concatenated alongside the gripper position history and wrist image features. This latent then fed into small MLP to generate the action prediction.
The privileged critic networks also use the same inputs as the search policy, and take in an additional privileged target object segmentation mask. The mask is processed using small convolutional encoder, and concatenated with the other inputs.
To handoff from the search policy to To, we implement the following logic. Every 5 timesteps, query object detector to see if target object is detected. If the target object detected in the previous query from 5 timesteps ago and the current query, then we handoff to m-- This consecutive mechanism implemented to prevent premature switching to no, since the object detector is perfect and sometimes gives false positives. We find that the consecutive criteria rules out false positives and switches correctly when the target object within view.
E,5 Reward Design
The reward function incentivizes the robot to locate, approach, and fixate on the target object. Viewpoints that score highly under this reward function feature the object prominently in the top- center region of the wrist image. We chose to incentivize putting the object in this target region because the grippers occupy the bottom half of the wrist image, and find that this particular viewpoint optimizes for grasping success of тo-
To define the reward, we first need privileged information about the object location and size in the wrist view image. To obtain object detection and segmentation of the target object, we used the DINO-X (59] API and the GroundedSAM [60] Model for Open-World Object Detection and segmentation. We use a color segmentation mask to break the tie when the object detector detects multiple potential target objects.
During the training phase, we infer the wrist camera images with DINO-X, obtain the bounding box and mask of the target object, and label the reward for offline RL training. See Figure II for examples of the object detection and reward pipeline.
The reward function consists of three terms:
Distance reward
T Jiss = - tanh( 10
D(c,e")
D(c, c²) E [o, 1000] px,
1000
where c, c* are the centroids of the bounding box, and target region, and D is the LI distance. This term smoothly saturates to 1 as the bounding box center approaches the image center.
(ii) Mask-area reward
clip (masmmasaaaaaa area, 1000, 50000)
F Tarza
50000
22

---

(a) DINO-X / GSAM Detection & Segmentation
(b) Color Segmentation & Reward
(d) Reward Feature Raw Value
(c) Fused Segmentation Bbox
Rossed
(e) Weighted Reward Curve
An example reward trajectory in Complex scene. Notice the trajectory is given high peak reward when the detected pineapple overlap on top of the square area.
Figure 11:
which proportional to the mask area subject to lower and upper limits, encouraging the robot to find viewpoints where the object is prominently in view.
(iii) Mask-overlap reward
Toverlap m_ask_ovvrlaap 0.10].
gives a sparse binary bonus once the object mask intersects with the target region, defined as loU > 10% with 128 128 region centered at c².
The per-step reward is then composed as
R₁ 0.5 Fais + raraa 10 Toverlap
Demonstrations We collect demonstrations using the 5- 5-DoF 3Dconnexion SpaceMouse. During teleoperation, we label trajectories as successful mo successfully grabs the target object after switching from the teleoperator. The demonstrations collected with four different teleoperators, with success rates for the demonstrations roughly between 50-70%. We initially collect up to 250 demonstrations per task, but then curate the dataset, dropping out trajectories with mislabeled object detections, noisy/faulty sensor readings, etc. After filtering, we end up with 152 demonstrations for Bookshelf-P, 109 for Bookshelf-D, 35 for Shelf-Cabinet, and 195 for Complex.
F.6 Baselines
Please watch the videos on our website (WW-RI. Project Page) to better compare the differences among baselines.
23

---

nove the grean black to the side right 1. AWR: Advantage- -eeihhte Regression, access to privileged observations.
2. BC: Filtered Behaviour Cloning, trained successful trajectories only.
3. Exhaustive: Hard-coded baseline that goes over every possible hiding location fixed order. This "brute-force" method gets high Search% score but takes much longer to search search.
4. VLM+TO: This baseline decomposes the task using a VLM for high level task planning and the тт VLA for low level movement as proposed HiRobot (54]. This approach commonly used in works that solve long-horizon tasks with only foundation models. In practice, we query the Gemini-2.5-Fiie of searching-relatee instructions that the low-level VLA can follow. Then, we ask Gemini to choose among them. The prompt template attached below.
[53) model with task prompt template, which includes series
expert operator using pio policy, , general purpose foundation model. It receives natural language command and executes the Franka Panda arn.
Your job is to provide naturel language instructions to help single single-arued robot with parallel- -jaw gripper complate tabletap manipulation
overall find SHARGET_OBJECT_KAN
in the scene. The robot has mounted canera and sequences actions, such as opening dravers, acanning
can perforn compartnente, and noving its camera viespoint.
Given the following constraints:
The <[>TARGET_OBJER MAME >AME< night be +spartislly or fully occluded*.. located siinsde dravers*e, sebehind objecte*e, or
Your job is to break this task doun into amaller instructions that rabot can complete
Every Iew seconds will a8k to provide natural language instruction for the robot
Ne will provide
(1) external view of the robat and (2) camera nounted on the robot 's vrist.
fron
Your instruction should refer to relevant objacts that see in the images, and should help the robot nake progress towards completing the everall task (S)>OVERALL, TASK <|>).
To help you, se've prepared list of instructions for you to choose from:
look around for the <|>TARGET_ORG BBRTTTDBBBR DBCTT___BBEEEEEEEEEEEEEE <
open top
open the botton draver
look inside the top draver
look inside the botton draver
look bebind the tays bahind the blue block
look bebind the green block an the top shelf look the botton shelf nove the red block to the side
the side the blue block
griper
nove griper to the left nove griper the center griper to the front nove griper to the back griper the top nove griper to the botton <|>TARGET_ pick up the </>>TAGET__ DBW._CBBETCCCEEE NAME
pick up
Bere is the externsl viev: <I>CURRENT_IMAGEE IIAGE<I>>> Hare is the vrist viev: <|>CURRENT_ BIIIEETT_KRITT_IAGGE provide instruction case with no punctuation. Just provide the instruction; do not provide additional explanation.
follow. instruction
Listing Prompt Template for VLM+#g baseline
24

---

Bookshelf-P
Bookshelf-D
(celfff-Cabinet
Complex
ooject found but not approached Object approached but not fixated completed
Figure 12:
Failure analysis AAWR, AWR, and BC policies in all 4 tasks. For each policy, we show the number of times each policy completes the first, second and third stage of the Search % rubric. AAWR completes all three stages the most, while AWR and BC fail to consistently approach and fixate on the target object.
Table 4: Metrics for active perception handoff tasks, AAWR consistently outperforms baselines. Bold best, underline second best.
Method
BookshelEP
BonishelFDD
Shelf Callect
Cumples
Scarch R
Steps 1
Search/
Scarch St
Stops
Soorch S
58
Steps
AAWR AWR
92.4.so
44.4mm
36.6=2
44.4:12
26.9 9:2b
78.2
40.0=114
46 1:44
Sd.8ssss
20.0=xx
121.0am1
29.6
0.0ana
34,0.2>
62.6us5
16.7asa
50.2
52.316.1
10.0s67
38.0
13.2a50
10.0
217.0s213
BC
29.9a12
20.0s12
84,0=92
47.7
16.7asa
22.5-11
28.1as5
15.0sam
125.0a215
46.4555
10.0ax7
138 00000
263.3am
229.7
20.0s1t
253.5 253.5am1
410m
105.4:9a
96.0.31
22.2.44
52.8156
45.0g11
183.0::53
78.2224
300
VLM
31.4
27.31t
522.3
33.2
16.7a167
281.8t18.1
28.2.75
15.0usa
382
14.8
10.0.ss
394.7
F.7 Results
As seen in Table 4 AAWR consistently outperforms baselines in all metrics, learning sensible active perception behavior aid generalist policy. We report the mean and standard error over trials for all metrics. AAWR always outperforms non privileged AWR and BC, thus validating the usefulness of privileged information and the use of offline often has high search progress and mo % success rate, but is much slower than AAWR, showing that AAWR Iearns to search efficiently. We find that mo and VLM+#p both search poorly they tend to take inefficient movements and fail to track the object. In Figure 12] we break down the failures of the active perception policies in each task by recording the number of stages completed the 3-point rubric. Across all tasks, AAWR completes the task (all three stages) the most. AWR and BC often spot the object, but do not consistently approach and fixate on the object.
over supervised learning. The Exhaustive baseline
In the Bookshelf tasks, AAWR first learns to zoom out of the scene to see multiple shelves, then scans from bottom to up, and then approaches the target object once located. The AWR and BC baselines follow relatively fixed search path that approaches the shelf, but the policies failed to efficiently scan the shelves. Even they luckily glimpse the target object, they do not fixate on the object, reducing their search score and success rate. See Figure 6 for visualization.
25

---

Table 5: Active perception handoff tasks: Metrics normalized by time, and then compared to the exhaustive search. AAWR consistently is ~2-8x better than exhaustive search in such metrics, while other baselines have mixed results.
Bookshelf-P
Bookshelf-D DD
Shelf-Cabing Cabinet
Complex
Method
Search
Completion
Search
Completion
Search
Completion
Search
Completion
AAWR
4.14
2.91
3.36
7.93
5,86
3,51
1.72
1.64
AWR
3.84
0.00
2.30
2.65
4.77
1.07
0.23
0.46
BC
0.58
0.57
2.36
3.56
0.78
0.49
1.28
0.72
To
0.07
0.15
0.32
0.70
0.12
0.15
0.45
0.78
VLM+*D
0.16
0.21
0.13
0.28
0.26
0.16
0.15
0.26
Exhaustive
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
In the Shelf-Cabine task, AAWR searches through the right bookshelf, before moving to the left cabinet. Both AWR and BC do not thoroughly search the scene, preventing them from finding objects placed in the left cabinet's drawer.
In Complex, AAWR searches the bottom shelf, the right shelf, and then the left cabinet (see Figure 1j.
See the website for comprehensive success and failure recordings of the policies over all tasks.
F88 Dataset ablation on the Complex task.
We ablate the demonstration quality by collecting two demonstration datasets. First, suboptimal dataset 195 demonstrations is collected by four different teleoperators. The dataset is quite diverse, suboptimal (success rate of 60%), ), and potentially hard to learn behaviors from. We found that all policies trained on the initial dataset struggled to reproduce certain behaviors, particular moving to the left side of the scene, despite the existence of such trajectories in the dataset.
Next, we collected small 50 trajectory dataset from only one expert teleoperator, with a high success rate of 94%. As seen in Table 6) all approaches improve using the smaller, more optimal optimal dataset. AAWR still outperforms baselines, as it consistently approaches and fixates on the objects, maximizing the success rate of after handoff. In contrast, AWR and BC do not approach and fixate target object as well as AAWR, often switching To when the target object is barely in view or in an odd location with respect to the gripper.
Table 6: We ablate the demonstration dataset of the Complex task, and find that all approaches benefit from smaller but more optimal demonstration source. AAWR still outperforms all baselines.
Complex (Suboptimal Demos)
Complex (Expert Demos)
Method
Search % (?)
To % (f)
Steps (1)
Search %
To % (f)
Steps (1)
(t)
AAWR
54.8
20.0
121.0
73.2
50.0
43
AWR
13.2
10.0
217.0
33.2
40.0
67
BC
46.4
10.0
138.0
31.5
15.0
77
T
29.6
20.0
252.5
29.6
20.0
252.5
Exhaustive VLM+To
78.2
30.0
297.0
78.2
30.0
297.0
14.8
10.0
374.7
14.8
10.0
374.7
F9 Reward Quality Analysis
We further analyze how reward signal quality and annotation noise affect search behaviors. In experiments, all episode begins without the target object in view, and the robot must actively search until it detects the object with visible and confident masks. Ideally, the object is in a clear view see Figure IT where we give high terminal reward. Consequently, early reward spikes typically indicate mislabeled frames.
26

---

For each dataset, we label the mislabeled episode using heuristic. An episode is flagged as mislabeled if its reward exceeds a fixed threshold of 5.0 within the first 30% timesteps. We counted the mislabeling rate of success trials, failure trials and total trials of each search datasets below:
Table 7: Per-task policy performance and reward noise.
Search (1)
Completion
Task
Mislabel (Suce/Faal/Tonal)
Observation
(1)
Bookshelf-P
92.4 5.0
44.4 + 16.6
0.2.0/2.7
Clean reward
Bookshclf-D
81.3 = 6.2
44.4 = 11.7
75 56.88 68.8
Noisy reward
Shelf-Cabine
78.2 7.0
40.0 1 11.0
17.1 30.8 23.8
Harder scene with longer horizon
Complex: Suboptimal
54.8 8.5
8.9
13.4 11.8 12.8
Diverse dirty data. Expert clean
Complex: Expert
73.2
12.0
50.0 + 15.8
6
As the table shows, higher quality data (e.g. clean, sparse reward with less detection error) correspond to better search and completion rates. Despite noisy rewards Bookshelf-D, AAWR is still able to learn good search behavior.
G Blind Pick Details
Task Definition In this real world experiment, the Koch robot must pick up a small rectangular candy (1 cm x cm x cm). It operates blindly since is given only the initial object position and its own joint positions during the episode. Interactive perception required solve the task, since the robot must sense when the object is gripped using its proprioception and then proceed to lift it up.
Observation: Initial object position, and current robot joint positions.
Partial observation: joint positions and initial Cartesian position of the target.
Privileged observation: real time Cartesian position of the target at each timestep.
Action Space: Cartesian position commands relative to robot base and gripper joint control.
Figure 13: Hardware configuration: Koch robot with RGB-D camera in the back.
Figure 14: Koch robot picking up the target ob- ject.
Hardware and Scene Setup We utilized Kochvl.1 robot[6]], open- low-cost, 5 DoF robotic arm. The robot arm is operated via a Cartesian position controller respect to the robot base frame. The forward and inverse kinematic computations is computed in a MuJoCo simulation model synchronized with real robot's joint positions real-time.
To get the privileged cartesian position of the object, we set up a RealSense D455i RGBD camera pointed towards the robot workspace. We calibrate the D455i using an ArUco marker, and then use color segmentation to filter the point cloud to estimate the 3d position of the object on the table. The target object is randomly placed within 10cm square region front of the robot the start of each trial.
27

---

Data for the Koch robot experiment was collected by executing approximately 100 demonstration episodes, total containing around 3000 transitions. The demonstrations were gathered from a noisy hand-coded script, resulting in a success rate of approximately 20%.
Data Collection
Reward Design
1. Distance penalty The distance penalty is the reward term where we introduce privileged information: the real-time position of the target object. Th reward term is computed by:
Ft ||
where 2y is the current Euclidean position of the target object computed via color seg- mentation and the depth camera, x is the current Euclidean position of the robot's end effector.
2. Grasp reward Using robot proprioception, grasp on an object. More specifically, the gripper receives a closing command, but the actuator cannot rotate the gripper to the commanded position, firm grasp detected by proprioception. Therefore, we have:
determine if the gripper has a firm
'grasp = kgrasp 1 (grasped, True)
3. Success reward A larger reward when the robot fully accomplishes the task: picking up the target object and lifts it em above the robot's s base. In particular, the reward is given by:
T success
kigrasp [Eee [See-Phaee g0.0
where zee the z-axis position of the end effector, zbase is the z-axis position of the robot base
Baselines
1. BC: Behavior Cloning using offline successful demonstrations only.
2. AWR: Advantage Weighted Regression trained both offline and online.
3. AAWR: Asymmetric Advantage Weighted Regression leveraging privileged information during offline and online training phases.
Metries
Performance is evaluated across 40 trials per method with the following metrics:
Grasp %: Percentage of trials which the robot successfully grasped the object.
2. Pick %: Percentage of trials where the robot successfully grasped and lifted the object.
Online Training and Evaluation All methods underwent an initial 20.000-step offline pretrain- ing phase followed by online fine-tuning using 1,200 transitions ( (40-50 episodes), each lasting approximately 20 minutes.
During evaluation, for each trial, policy has 30 timesteps accomplish the task. Evaluation results confirmed AAWR's superior performance in grasping and picking success rates and demonstrated notably effective retrying behaviors following failed initial attempts.
H Simulated Experimental Details
H.1 Camouflage Pick
The Camouflage Pick experiment requires simulated Koch robot to pick up a tiny, hard-to-see marble initialized in 10x 10 cm region (see (see Figure 15).
Observation Space:
Partial observation: 3rd person 84 x 84 image of the robot and marble.
Privileged observation: robot and marble positions using simulator state
28

---

Action Space: Cartesian velocity of end effector frame and gripper position
Reward Function: Sparse reward that gives +1 if the marble is in gripper and altitude is over 7cm.
Demonstrations: We collect 100 demos using hand-coded script. The script is not perfect and gets around 30% success rate.
Offlin Online budget: 20K offline, 80K online
(a) AAWR
(b) AWR
(c) BC
Figure 15: Camouflage Pick (third-person camera only). (a) AAWR: succeeds with search then grasp. (b) AWR: fails due to mis-grasps. (c) BC: fails due to overfit on grasping without search
We compare against symmetric AWR, the non-privileged version of AAWR, and BC. We train BC for 20,000 offline steps, periodically checkpointing and evaluating it, and report the highest performing checkpoint. We pretrain AWR and AAWR for 20,000 offline steps. Then, we do online finetuning for 80.000 environment steps. While training, we periodically evaluate the policies by recording their average success over 100 trials. The success metric is the sparse reward function.
We train all models with a batch size of 256, learning rate of 0.0001, and the Adam optimizer. For online finetuning following [47], we use an update-to-date ratio of performing gradient updates after every episode. For AWR and AAWR, we use an advantage temperature of 10.
We instantiate separate networks for the for the policy and value/critic networks. We use the same encoder head recipe for all models, following [47]. We use a CNN to process the RGB image into a 50-dimensional latent, and MLP to process the privileged information into a 50-dimensional latent. The latents are then fed into a MLP to get the output.
AAWR outperforms baselines. AWR and BC frequently completely miss the marble, while AAWR displays more accurate picking behavior. Even after picking, the small marble frequently slips out of the grasp, making the success rate for all of the policies rather low. See the website for videos.
As seen in Figure 5
H.2 Fully Observed Pick
The Fully Observed Pick experiment requires simulated xArm6 robot to pick up block. To make the scene as fully observable as possible, we make the xArmó robot invisible except for its grippers, making occlusion of the object by the robot impossible (see Figure 16). The object is randomly initialized a 25 x 25cm region in front the arm.
Observation Space:
Partial observation: 3rd person 84 84 image of the robot and block.
Privileged observation: robot and object positions using simulator state
Action Space: Cartesian velocity of end effector frame and gripper position
Reward Function: Sparse reward that gives +1 if the block is in gripper and altitude is over 10cm.
Demonstrations: 100 demos using a hand-coded script. The script is not perfect and gets around
30% success rate.
Offline Online budget: 20K offline, 20K online

---

We use the same baselines, hyperparameters, network, and evaluation configuration as the Camouflage Pick experiment. The only change the offline / online budget 20K steps offline, and 20K steps online.
(a)
(b)
(c)
(d)
Figure 16: Visually ambiguous pick (fully observed). (a-b) AAWR: grasp centers then lifts (success). (c-d) AWR: hovers off-center, closes mid-air (fail).
Results are in Figure AAWR effectively solves the task with near perfect success rate, learning to accurately localize and grasp the object. AWR shows two failure modes. First, it struggles with positioning the gripper over the block. It often places the gripper in front of the block, which looks like reasonable grasp from the camera angle, but is in reality quite off from the block Figure 16).
Next when is able to grasp the block, it does not lift up the block. BC displays similar failure modes. See the website for videos of the policies.
H.3 Active Perception Koch
In this task, we equip the simulated Koch robot with a wrist camera with small field of view, and task it to pick up cube randomly initialized in 10x20 em region in front of it. (see FFigure 17). As the wrist camera has narrow field of view and becomes self-occluded during grasp closure, the robot must first scan to rediscover the cube before executing the pick.
Observation Space:
Partial observation: Frame-stacked (past 3) grayscale wrist camera images of size 84 84.
Privileged observation: object positions using simulator state
Action Space: Cartesian velocity of end effector frame and gripper position
Reward Function: Sparse reward that gives +1 the object in gripper and altitude is over 7cm.
Demonstrations: We collect 100 demos using a hand-coded script. The scripted behavior uses state information to command the robot to go over the block and pick it up. The script is not perfect and gets around 30% success rate.
Offline Online budget: 100K offline, 900K online
Wrist camera with limited field of view. When the gripper approaches and closes on the cube, the wrist camera becomes self-occluded, The side view is shown for risualizaaiinnevaaluatinn only and is not provided to the policy.
Figure 17:
We compare against other approaches that use privileged information. The first Distillation which features two-stage training process. In the first stage, teacher acquisition, privileged teacher
30

---

policy is trained on the collected successful demonstrations. In the second phase, distillation, the teacher is distilled into student policy. Following [62]. the distillation phase is performed over online rollouts from the student policy. In our setup, after the first stage, the teacher policy is able to get near perfect success rate on the task, although note that using privileged information to do so.
The second baseline is variational information bottleneck approach VIB [52] that trades off the RL return with KL penalty for accessing privileged information. Concretely, this penalty is implemented by defining a privileged latent that comes from the posterior 2 p(- ot)) and constraining the posterior to the unprivileged gaussian prior N (0,, I) via KL divergence. During training, the policy (a o, 2) uses the latent from the privileged posterior, and during evaluation uses a latent sampled from the unprivileged prior. The RL agent should learn minimally use privileged information, since usage will negatively impact its overall return. We conduct sweeps over different weights of the KI. term 3 0.01, 0.1, 0.5, 1, 10, report the performance of the best performing weight (3 0.5).
All baselines are implemented in the same codebase, using the same encoder head architecture configuration AAWR. We conduct sanity checks make sure the baselines work, such making sure the privileged teacher and VIB policy with the privileged latent get high success rates.
(a) AAWR
(b) Distillation
(c) VIB
Resulting behaviors on the Koch task. (a) AAWR actively scans the workspace, recenters the cube in view, grasps, and lifts with near 100% success at evaluation. (b) Distillation learns suboptimal ggo-to-cented strategy and often closes off target due to the absence of scanning in the teacher. (c) VIB degrades evaluation without privileged information; using only prior latent leads to drift and low success.
Figure 18:
As in Figure 18
only AAWR learns to do active perception by scamning the workspace, getting near 100% success rate during evaluation time. The distilled student learns a suboptimal behavior of just approaching the center of the workspace, because its privileged teacher never displays the scanning behavior. VIB does poorly during evaluation with no access to privileged information, even though it trained minimally use privileged information.
Additional video examples are available on our project page at battps://penn. pal- lab. github co/aawr/
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide experimental evidence and theoretical analysis for our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
31

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We outline limitations in the conclusions section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide proof' and assumptions for all theoretical claims the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
32

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide experimental details in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release code for the algorithm and environments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
hublic/guidess http://uuiddse/eeeubmmiseee Coode@Subisssiooooll1..
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips http:h/pww////eiieeee//eeetaaaaa
for more details.
33

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See appendix for all details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
See appendix for all details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2iggma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
34

---

Justification:
We provide details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We have reviewed it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss such things in a section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
35

---

Answer: [NA]
Justification:
We don't foresee high risk for misuse..
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly credit the assets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release the trained weights of our model.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
36

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No human experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No humans experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs as core part the research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy /bttps://neurba ce / for what should or should be described.
37


Reviews:
[{"review_id": "vRZ7WFiwbg", "reviewer": "Reviewer_CA6i", "summary": "The paper proposes Asymmetric Advantage Weighted Regression (AAWR), a reinforcement learning method for training robots to perform active perception behaviors in real-world environments with partial observability. AAWR leverages \"privileged\" extra sensor information available only during training to train value functions that provide better supervision signals for the policy. The approach combines a small number of suboptimal demonstrations with a coarse policy initialization and uses offline-to-online RL to efficiently acquire information-gathering behaviors. The method is evaluated on eight manipulation tasks involving different robots under varying sensory setups. The results show that AAWR learns active perception policies and outperforms baselines.", "questions": "Practically, if or why AAWR would be better than some search-based heuristics in active perception in your tasks performed? Or are there other tasks that cannot be simply done by heuristics search but well solved by AAWR?", "limitations": "Partially, but limitations are not sufficiently addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ The paper introduces privileged information at training time to enable robots to efficiently learn active perception behaviors. This is a meaningful advancement, as it addresses the sample inefficiency and partial observability issues.\n+ The authors provide a solid theoretical justification for their approach, deriving the AAWR objective for POMDPs and clearly explaining why asymmetric advantage estimation is crucial in these settings.\n+ Empirical results are strong: Across a variety of tasks and robot platforms, AAWR demonstrates good performance compared to both imitation learning and standard RL baselines.\n\nWeaknesses:\n\n- The validity and generalizability of the approach are somewhat unclear, as the experimental tasks are relatively simple or \"toy\" problems. In many of these settings, a hard-coded exploratory policy or a straightforward exhaustive search could plausibly solve the task. While such a policy might not be efficient at first sight, it might also be justified considering the cost to train and collect data for AAWR.\n- The real-world benefit of sophisticated active perception may be less apparent in scenarios where domain knowledge or engineered exploration strategies could also succeed. The comparison to such baselines (e.g., exhaustive search or heuristic exploration) is mentioned, but the gap between learned and non-learned approaches may not be as meaningful in these simplified settings.", "was_revised": true, "final_justification": "I've read the author response. While I still do not agree with some of the comments, I would like to sync with fellow reviewers and remain on the accept side.", "rebuttal": "We thank the reviewer for their helpful feedback, and discuss their concerns below. The reviewer asked “why AAWR would be better than **search-based heuristics** in …your tasks?”. They raise the point that non-learning-based approaches like a “hard-coded exploratory policy or a straightforward exhaustive search”, could plausibly do well, without requiring data collection and training costs. \n\nIndeed, active and interactive perception tasks are search problems, and all search problems do have brute force “solutions” that will eventually succeed. The measure of good active and interactive perception strategies is how *efficiently* they can search. This is reflected in our performance metrics, where we measure not just if the policy succeeds at finding the object, but also the number of steps it takes. \n\nAAWR is a learning based approach for acquiring active perception policies, and we seek to learn policies rather than engineer them, for the following compelling reasons.\n\n1. Unlike a fixed, engineered policy, AAWR, like any other RL algorithm, can improve over time from its own experience towards an optimal policy. In contrast, a heuristic strategy may solve the task, but is likely inefficient. For example, we find that our engineered exhaustive search baseline, and a VLM-powered heuristic search baseline, both take over 3 times longer than AAWR to find the objects. \n\n2. Engineered solutions often require more expansive sensor setups than what we afford to AAWR in our experimental setups. For example, [1] uses depth sensors and [2] uses calibrated cameras. In contrast, AAWR only uses a single uncalibrated camera in the $\\pi_0$ active perception task, and only proprioception encoders in the Blind Pick task. \n\n3. Engineered heuristic solutions tend to be very specific for each task and sensor setup, and may need to be completely reimplemented if anything changes. AAWR policies are acquired using the same recipe across different robots, sensors, and tasks, through collecting data and optimizing the AAWR objective. \n\nNext, the reviewer is concerned about the scope of our experiments, in that “the experimental tasks are relatively simple or **toy problems**”. Our tasks in sim and in real are relatively hard compared to recent prior work on learning active perception policies [3,4,5], and are thus meaningfully advancing the state of the art. However, as mentioned in the paper, the field as a whole has struggled to produce useful active perception behaviors for robotic manipulation, and this is reflected in the complexity of tasks that are commonly investigated. \n\nAAWR offers an improved technique and meaningfully advances the complexity of active perception behaviors that can be learned for real robotic setups, even showing the capacity to improve the behavior of a state-of-the-art generalist robot policy. Even so, we agree that there is much more room for growth, and hope that our paper will be an important stepping stone in this direction! \n\nWe will aim to add this context into the paper, thank you again for the thoughtful queries.\n\n[1] Krainin, Michael, Brian Curless, and Dieter Fox. \"Autonomous generation of complete 3D object models using next best view manipulation planning.\" 2011 IEEE international conference on robotics and automation. IEEE, 2011.\n\n[2] Liu, Yang, Pansiyu Sun, and Akio Namiki. \"Target tracking of moving and rotating object by high-speed monocular active vision.\" IEEE Sensors Journal 20.12 (2020): 6727-6744.\n\n[3] Grimes, Matthew Koichi, et al. \"Learning to look by self-prediction.\" Transactions on Machine Learning Research (2023).\n\n[4] Cheng, Ricson, Arpit Agarwal, and Katerina Fragkiadaki. \"Reinforcement learning of active vision for manipulating objects under occlusions.\" Conference on robot learning. PMLR, 2018.\n\n[5] Dass, Shivin, et al. \"Learning to Look: Seeking Information for Decision Making via Policy Factorization.\"Conference on robot learning. PMLR, 2024."}, {"review_id": "zJk5IuqbbF", "reviewer": "Reviewer_u7RD", "summary": "This paper presents a method called Asymetric Advantage Weighted Regression that allows for training of active perception policies under partial observability. It uses information provided by sensors during training to efficiently train value functions that are more accurate for providing advantage estimates. The method extends Advantage Weighted Regression to this type of setting, in which the deployed policy has asymmetric perception capabilities. The paper tests this method on a variety of simulated and real world tasks and find significant improvements over baseline techniques.", "questions": "1. Do the authors have a sense of how \"clean\" the privileged information has to be for the method to be useful?\n2. How much does the type of privileged information impact the learned policies (for a given task)?\n3. Do the failure modes of the method likely stem more from the demonstration noise or privileged information noise?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is very well-written and clear, and presents a clear motivation for their method in the context of integrating active perception with techniques using foundation VLA models. \n2. The real-world experiments are very helpful in illustrating the advantage of their method.\n3. Although I am not very well-versed in this area (and therefore this comparison may not be accurate), the method itself seems efficient in terms of clock time and number of demonstrations needed for training. \n\nWeaknesses:\n1. I'll add the caveat that this particular area of active perception is not my strongest expertise, it seems that the novelty of the work is rather incremental, as the bedrock of the method is AWR, an existing method. \n2. There is a lack of comparison with other approaches that use privileged information as mentioned in related work.", "was_revised": true, "final_justification": "I feel that this paper should be accepted, as it has a high standard of quality of work and my concerns have been addressed by the rebuttals.", "rebuttal": "Thank you for the feedback!\n> novelty may be incremental relative to AWR?\n\nAs we argued around line 160 in the paper, Advantage Weighted Regression (AWR) was originally formulated for fully observed settings. One contribution of ours is showing that its naive application to POMDPs would result in suboptimal learning, see lines starting from 160 for an intuitive explanation and full theorem in Appendix D. We then show both theoretically and empirically why AAWR is the right instantiation of AWR for partially observed tasks\n\nAAWR is a mechanistically simple modification of AWR, but it is precisely the simplicity of implementation that makes the AAWR recipe easy to evaluate on many diverse tasks, including with real-world robots, and alongside state-of-the-art generalist robot policies.\n\n> There is a lack of comparison with other approaches that use privileged information as mentioned in related work\n\nOur task setup involves learning directly on a real world robot platform, with no access to a simulator. Due to the slow data collection time of real robots, the sample budget is very small (e.g. two hundred offline trials, and 50 online trials). Furthermore, the privileged observations are estimated using noisy, real world sensors.\n\nTrue, we did indeed mention other privileged RL approaches in related work, but as we have argued there (Lines 82-95), these approaches are designed for sim-to-real RL, and (1) sim-to-real RL is not easily applicable to active perception due to the difficulty of effectively simulating most sensors, and (2) the privileged RL approaches designed for sim-to-real do not easily transfer to our settings, because they assume access to millions to billions of episodes of training data, and exploit low-dimensional ground truth simulator state as privileged information. As such, these methods are not empirically comparable to ours. \n\n> Do the authors have a sense of how \"clean\" the privileged information has to be? \n\nWe have some empirical sense of the robustness of our method to noise in the privileged information. In our real world experiments, we show that AAWR is able to learn from privileged observations, which have some irreducible noise due to imperfect sensing and environmental constraints. \n\nFor example, in the active vision pi0 experiments, we used off-the-shelf object detectors [1,2] to get privileged segmentation masks of the target objects (see figure 10 in appendix). These detectors would sometimes misclassify objects (e.g. classifying distractor objects with similar appearances as the target class), with a non-trivial misclassification rate of ~22%. We will include more discussion and details of the privileged sensor noise in the experimental detail sections (appendix F and G).\n\nNext, in the Blind Pick experiment, we used a depth sensor (Realsense D435i) to get 3D position estimates of the object. On average, the estimates were off by 0.5-1cm, which is  non-trivial considering the small size of the object (around 1.5cm) and the workspace (around 10cm). There are also cases when the 3D position estimates are invalid, such as when the robot gripper occludes the object.\n\nIt may also be interesting to try to theoretically characterize the sensitivity / robustness of AAWR to noise in the privileged observations. We will look into this in future work.\n\n> …does the type of privileged information impact policies? \n\nDue to our limited resources on running real world RL experiments, we did not test different types of privileged observations for a given task. But we can give a few guidelines for choosing privileged information.\n\nFirst, for the privileged critic, the privileged information should aid in predicting returns. This implication is drawn from appendix C and D, where we show the advantage is determined by both the agent state and environment state. In our active perception tasks, the location of the hidden object is an important privileged feature since the return is dependent on the object location.\n\nNext, one should consider how easy it is to learn from the privileged sensor. For example, high dimensional sensors like cameras and depth sensors would require higher capacity encoders, which in turn demand more data samples to learn good representations. \n\n> Do the failure modes … stem more from the demonstration noise or privileged information noise?\n\nWe observe that in the offline case, the policy is more dependent on the particularities of the demonstrated behaviors, since it cannot improve from on-policy trajectories. \n\nIn the online RL phase, demonstration noise is no longer a problem since the agent is learning from its own on-policy trajectories. Here, we believe privileged information noise may impact the performance more, since noisy privileged information would result in less accurate advantage estimates, which would affect the policy gradient."}, {"review_id": "b4pQISVeIp", "reviewer": "Reviewer_3ct1", "summary": "This paper proposes an extended version of AWR, termed Asymmetric AWR (AAWR), for policy improvement in POMDP settings with access to privileged information during training. The goal is to leverage this privileged information to promote active perception behaviors. The main novelty on the algorithmic side lies in the asymmetric structure: the critic has access to privileged information (e.g., additional observations such as position, masks, or bounding boxes), while the policy is trained using only standard observations. The method is evaluated on both simulated robotics tasks and real-world robot learning tasks, with comparisons to vanilla BC, the original AWR, active perception baselines, and VLA-based models. \n\nOverall, I find the proposed method to be simple and practical, and the experiments are well-executed with a set of realistic, real-world benchmarks.", "questions": "I have listed my questions (Q1–Q5) in the weakness section above, as they are closely related to the identified weaknesses and grouping them together improves coherence. In the rebuttal, it would be helpful to see clear answers to Q1, empirical validation for Q3, a discussion addressing Q2, and concrete plans for how the authors intend to revise the paper to address Q4.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n(+) [About novelty and significance] The problem setting itself is important. While there has been prior work on learning active perception, this paper applies AAWR to address it, and I feel that the algorithmic novelty is reasonable within this context.\n\n(+) [Evaluation] I think the paper provides a solid set of experiments, covering both simulated and real-world scenarios, and overall, the evaluation is convincing. That said, the inclusion of additional relevant baselines could further strengthen the empirical results.\n\n(+) [Clarity] In general, the writing is clear and easy to follow, and most of the necessary details are provided. I also checked the appendix, which includes the full derivations of the objective functions as well as comprehensive experimental details.\n\n**Weaknesses and questions**\n\n**W1** [Model] In general, I don't find specific technical problems in the proposed AAWR, however, I do have questions on the setting:\n\n*Q1*: What is the fundamental difference between the proposed method and existing approaches that learn active perception by introducing an additional motor policy? In those cases, privileged information is also often used to train the critic. One example is Shang et al., 2023.\n\n*Q2*: Is it possible to allow the agent to actively select the privileged information to use also during training time, rather than relying on a pre-specified set during policy improvement?\n\n**W2** [Evaluation]\n\n*Q3*: Similarly to Q1,I think it would be sensible to also compare with Shang et al., 2023 to empirically demonstrate the differences. If this is not feasible within the current timeframe or it is not doable, providing a justification in the rebuttal would be sufficient.\n\n**W3** [Presentation] I think the presentation and clarity could be further improved:\n\n*Q4*: I feel it would be sensible to present the AAWR component with more details. At a minimum, the beginning of Section 3 should explicitly define the problem setting, clearly distinguishing what constitutes privileged information, what is considered sensory input, what the final objective is, and conceptually, how the side information is used during training. \n\n*Q5*: Minor: Line 147, V(s,a) -> V(s,z); Line 188: missing citation.\n\n\nShang, Jinghuan, and Michael S. Ryoo. \"Active vision reinforcement learning under limited visual observability.\" Advances in Neural Information Processing Systems 36 (2023): 10316-10338.", "was_revised": true, "final_justification": "All my concerns have been addressed in the rebuttal, and I also checked the rebuttal to other reviewers - all look sensible to me.", "rebuttal": "Thank you for your comments!\n\n> Q1: What is the fundamental difference between the proposed method and existing approaches that learn active perception by introducing an additional motor policy? \n> Q3: …it would be sensible to also compare with Shang et al., 2023 to empirically demonstrate the differences\n\n**Shang et al. requires a factorized action space, while AAWR does not.** The fundamental difference between AAWR and such prior work like Shang et al. [1] is that Shang et al. only works on POMDPs with a “factorized” action space (explained below), whereas AAWR is derived with general POMDPs in mind, including those without factorized action spaces.\n\nSpecifically, Shang et al. requires that the action space can be factorized into “sensing” actions that move the sensor without affecting the scene, and “motor” actions that move the robot without affecting the sensor’s viewpoint. In their simulated robotic tasks, they assume access to a 3rd-person camera that can freely move around the scene to observe the robot. This assumption is mentioned in their limitation section, where they mention future work is needed to address tasks where there is a dependence between sensing and motor actions. \n\n**AAWR tasks cannot be factorized.** The tasks evaluated in AAWR do not meet Shang et al.’s requirements. In our active perception tasks, the only moving camera is a wrist camera that is attached to the same robot arm that must manipulate the scene. This is very common in modern manipulation setups including diffusion policies, VLA models such as pi-zero, and more. In these settings, any action will simultaneously change the viewpoint and affect the scene. To take another example from our paper, in our Blind Pick task, the robot needs to perform *interactive* perception by grasping a block to detect it before picking it up. There is no clear distinction between sensor and motor actions, since the robot needs to physically affect the scene by grasping the block to detect it. Since AAWR does not assume a factorized action space, it can solve these tasks, while Shang et al. cannot.\n\n\n**Shang et al. is not ready for real world RL.** Aside from assumptions in task formulation, extensive algorithmic modifications would be needed to make Shang et al. work in the real world. Shang et al. was trained only in simulation with online RL. In contrast, AAWR learns strictly in the real world from offline and online data, with a sample budget several orders of magnitude below the sample budgets of Shang et al.\n\nIn summary, we do not think it is feasible to apply Shang et al. to our tasks since our tasks’ action spaces cannot be factorized, which is a key assumption of Shang et al. Additional and substantial algorithmic modifications would be needed to make it more sample efficient for real world learning. Extending Shang et al. to real world tasks, potentially using ideas introduced in AAWR to make it efficient, would be an interesting future direction, but is not feasible within our rebuttal timeline. \n\n> In those cases, privileged information is also often used to train the critic. One example is Shang et al., 2023.\n\nAfter reading Shang et al. 2023, we could not find any equation or text mentioning a privileged critic. It is possible we missed something, could the reviewer point to where it mentions the usage of a privileged critic?  \n\n> Q2: Is it possible to allow the agent to actively select the privileged information to use also during training time, rather than relying on a pre-specified set during policy improvement?\n\nWe agree that this would be a cool direction to pursue! In fact, this is close to our own motivations for pursuing this work, and so we’ve already  mentioned it as a potential future direction in the conclusion in line 349. One major challenge we anticipate is learning efficiency. Learning to choose sensors through RL might be  too sample-inefficient for real-world settings. We believe alternative strategies from the active feature acquisition [2] and co-design [3]  literature could be promising ways to perform sensor selection efficiently. \n\n> Q4: I feel it would be sensible to present the AAWR component with more details. \n\nThank you for the feedback on the presentation, and here’s our plan for improving clarity. \n\n> …the beginning of Section 3 should explicitly define the problem setting, clearly distinguishing what constitutes privileged information, what is considered sensory input,\n\n- Line 104: The paragraph before motivated modelling active perception tasks as POMDPs. At the start of this paragraph, we will explicitly define the problem setting as “finding a policy that maximizes return in such POMDPs”. \n- Next, we will clearly define what is the sensory input and privileged information. The sensory input is the observation space of the POMDP. The privileged information is either the hidden state of the POMDP or additional observations that are informative of the state.\n\n> what the final objective is, and conceptually, how the side information is used during training.\n\n- The final objective is equation 3, and we will explicitly write that it is the final objective of our method. To make it clear where the privileged information is used, we will highlight the privileged input $s$ in blue in equation 3,  and in lines 147-148. \n- In the next version of the manuscript, we will add a figure visualizing the computation graph of equation 3. It will show two components, a policy and a privileged critic. The policy will just take in the observation, while the privileged critic will take in the observation and privileged information.\n\n> Q5: Minor: Line 147, V(s,a) -> V(s,z); Line 188: missing citation.\n\nThank you, we will update it in the next version of the manuscript.\n\n[1] Shang, Jinghuan, and Michael S. Ryoo. \"Active vision reinforcement learning under limited visual observability.\" Advances in Neural Information Processing Systems 36 (2023): 10316-10338.\n\n[2] Rahbar, Arman, Linus Aronsson, and Morteza Haghir Chehreghani. \"A Survey on Active Feature Acquisition Strategies.\" arXiv preprint arXiv:2502.11067 (2025).\n\n[3] Antonova, Rika, and Jeannette Bohg. \"Learning to be multimodal: co-evolving sensory modalities and sensor properties.\" Conference on Robot Learning. PMLR, 2022."}]

Author rebuttal:

Meta-review: This paper presents an approach for RL in partially-observed environments in cases where "privileged" full observations are available during the first phase, and not available afterwards. This problem setting is a natural fit for sim2real settings in which the simulator provides privileged information and the real environment does not. The proposed approach is based on advantage-weighted regression, and adapts it by privileging the advantage function, but not the policy.

Reviewers found that the approach was a meaningful advancement with clear and grounded motivation. Some concerns were raised about the simplicity of the settings not supporting the generalizability of the approach. Concerns were also raised about real-world benefits relative to heuristic methods for adapting to partial observability.

I myself have lingering concerns about the definition of the approach and an important missing baseline. Regarding approach definition: I encourage the authors to clarify in the paper why the privileged information ($o_t^p$) in $o_t^+$ is available in the online phase, as it results in what seems to be a different setting from the motivated setting of the policy not having access to privileged information during online finetuning. Because this information is available throughout both the offline and online phases, it raises the possibility of an approach that simply uses both $(z, o_t^p)$ as input to the policy. This was not tested, instead, the closest method of comparison is the  ablation of $o_t^+$ as input to the critics (denoted "AWR" in S4.2). I therefore think this is an important missing baseline (a policy that also conditioned on $o_t^p$). Without this method of comparison, the performance difference between AAWR and AWR shown in Fig. 3 (and elsewhere) could simply be due to including the extra information $o_t^p$ at all, rather than ensuring that it's only added to the critics. It is possible that adding that information also to the policy results in comparable or superior performance to AAWR, thereby undermining the original formulation of the approach as needing to treat "privileged" information as separate, since the privileged information is in fact available from the environment at all phases of the method.

Relatedly, the theoretical connection between $s$ and $o_t^+$ appears to be missing, and I'll note that $z$ appears nowhere in Alg. 1, yet, it appears as a term in Eq. 3 and the IQL loss for the critics, both of which are part of Alg. 1. I did not see $z$ defined in the implementation details (in either S3.2 or Appendix A).

Given the otherwise positivity of the reviews, my understanding of the paper's contributions, and that the authors can no longer provide feedback to the concerns raised above, I hesitate to recommend rejection on this basis.
Decision: Accept (poster)