Title: Real-World Reinforcement Learning of Active Perception Behaviors
Venue: NeurIPS 2025
Accepted: yes
Keywords: robotics

reinforcement learning

imitation learning

active perception

privileged information
Abstract: A robot's instantaneous sensory observations do not always reveal task-relevant state information. Under such partial observability, optimal behavior typically involves explicitly acting to gain the missing information.
Today's standard robot learning techniques struggle to produce such active perception behaviors.
We propose a simple real-world robot learning recipe to efficiently train active perception policies. Our approach, asymmetric advantage weighted regression (AAWR), exploits access to "privileged" extra sensors at training time. The privileged sensors enable training high-quality privileged value functions that aid in estimating the advantage of the target policy. Bootstrapping from a small number of potentially suboptimal demonstrations and an easy-to-obtain coarse policy initialization, AAWR quickly acquires active perception behaviors and boosts task performance. In evaluations on 8 manipulation tasks on 3 robots spanning varying degrees of partial observability, AAWR synthesizes reliable active perception behaviors that outperform all prior approaches. When initialized with a "generalist" robot policy that struggles with active perception tasks, AAWR efficiently generates information-gathering behaviors that allow it to operate under severe partial observability for manipulation tasks. Website: 
https://penn-pal-lab.github.io/aawr/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an extended version of AWR, termed Asymmetric AWR (AAWR), for policy improvement in POMDP settings with access to privileged information during training. The goal is to leverage this privileged information to promote active perception behaviors. The main novelty on the algorithmic side lies in the asymmetric structure: the critic has access to privileged information (e.g., additional observations such as position, masks, or bounding boxes), while the policy is trained using only standard observations. The method is evaluated on both simulated robotics tasks and real-world robot learning tasks, with comparisons to vanilla BC, the original AWR, active perception baselines, and VLA-based models. \n\nOverall, I find the proposed method to be simple and practical, and the experiments are well-executed with a set of realistic, real-world benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n(+) [About novelty and significance] The problem setting itself is important. While there has been prior work on learning active perception, this paper applies AAWR to address it, and I feel that the algorithmic novelty is reasonable within this context.\n\n(+) [Evaluation] I think the paper provides a solid set of experiments, covering both simulated and real-world scenarios, and overall, the evaluation is convincing. That said, the inclusion of additional relevant baselines could further strengthen the empirical results.\n\n(+) [Clarity] In general, the writing is clear and easy to follow, and most of the necessary details are provided. I also checked the appendix, which includes the full derivations of the objective functions as well as comprehensive experimental details.\n\n**Weaknesses and questions**\n\n**W1** [Model] In general, I don't find specific technical problems in the proposed AAWR, however, I do have questions on the setting:\n\n*Q1*: What is the fundamental difference between the proposed method and existing approaches that learn active perception by introducing an additional motor policy? In those cases, privileged information is also often used to train the critic. One example is Shang et al., 2023.\n\n*Q2*: Is it possible to allow the agent to actively select the privileged information to use also during training time, rather than relying on a pre-specified set during policy improvement?\n\n**W2** [Evaluation]\n\n*Q3*: Similarly to Q1,I think it would be sensible to also compare with Shang et al., 2023 to empirically demonstrate the differences. If this is not feasible within the current timeframe or it is not doable, providing a justification in the rebuttal would be sufficient.\n\n**W3** [Presentation] I think the presentation and clarity could be further improved:\n\n*Q4*: I feel it would be sensible to present the AAWR component with more details. At a minimum, the beginning of Section 3 should explicitly define the problem setting, clearly distinguishing what constitutes privileged information, what is considered sensory input, what the final objective is, and conceptually, how the side information is used during training. \n\n*Q5*: Minor: Line 147, V(s,a) -> V(s,z); Line 188: missing citation.\n\n\nShang, Jinghuan, and Michael S. Ryoo. \"Active vision reinforcement learning under limited visual observability.\" Advances in Neural Information Processing Systems 36 (2023): 10316-10338.\n\nQUESTIONS:\nI have listed my questions (Q1–Q5) in the weakness section above, as they are closely related to the identified weaknesses and grouping them together improves coherence. In the rebuttal, it would be helpful to see clear answers to Q1, empirical validation for Q3, a discussion addressing Q2, and concrete plans for how the authors intend to revise the paper to address Q4.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method called Asymetric Advantage Weighted Regression that allows for training of active perception policies under partial observability. It uses information provided by sensors during training to efficiently train value functions that are more accurate for providing advantage estimates. The method extends Advantage Weighted Regression to this type of setting, in which the deployed policy has asymmetric perception capabilities. The paper tests this method on a variety of simulated and real world tasks and find significant improvements over baseline techniques.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is very well-written and clear, and presents a clear motivation for their method in the context of integrating active perception with techniques using foundation VLA models. \n2. The real-world experiments are very helpful in illustrating the advantage of their method.\n3. Although I am not very well-versed in this area (and therefore this comparison may not be accurate), the method itself seems efficient in terms of clock time and number of demonstrations needed for training. \n\nWeaknesses:\n1. I'll add the caveat that this particular area of active perception is not my strongest expertise, it seems that the novelty of the work is rather incremental, as the bedrock of the method is AWR, an existing method. \n2. There is a lack of comparison with other approaches that use privileged information as mentioned in related work.\n\nQUESTIONS:\n1. Do the authors have a sense of how \"clean\" the privileged information has to be for the method to be useful?\n2. How much does the type of privileged information impact the learned policies (for a given task)?\n3. Do the failure modes of the method likely stem more from the demonstration noise or privileged information noise?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Asymmetric Advantage Weighted Regression (AAWR), a reinforcement learning method for training robots to perform active perception behaviors in real-world environments with partial observability. AAWR leverages \"privileged\" extra sensor information available only during training to train value functions that provide better supervision signals for the policy. The approach combines a small number of suboptimal demonstrations with a coarse policy initialization and uses offline-to-online RL to efficiently acquire information-gathering behaviors. The method is evaluated on eight manipulation tasks involving different robots under varying sensory setups. The results show that AAWR learns active perception policies and outperforms baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ The paper introduces privileged information at training time to enable robots to efficiently learn active perception behaviors. This is a meaningful advancement, as it addresses the sample inefficiency and partial observability issues.\n+ The authors provide a solid theoretical justification for their approach, deriving the AAWR objective for POMDPs and clearly explaining why asymmetric advantage estimation is crucial in these settings.\n+ Empirical results are strong: Across a variety of tasks and robot platforms, AAWR demonstrates good performance compared to both imitation learning and standard RL baselines.\n\nWeaknesses:\n\n- The validity and generalizability of the approach are somewhat unclear, as the experimental tasks are relatively simple or \"toy\" problems. In many of these settings, a hard-coded exploratory policy or a straightforward exhaustive search could plausibly solve the task. While such a policy might not be efficient at first sight, it might also be justified considering the cost to train and collect data for AAWR.\n- The real-world benefit of sophisticated active perception may be less apparent in scenarios where domain knowledge or engineered exploration strategies could also succeed. The comparison to such baselines (e.g., exhaustive search or heuristic exploration) is mentioned, but the gap between learned and non-learned approaches may not be as meaningful in these simplified settings.\n\nQUESTIONS:\nPractically, if or why AAWR would be better than some search-based heuristics in active perception in your tasks performed? Or are there other tasks that cannot be simply done by heuristics search but well solved by AAWR?\n\nLIMITATIONS:\nPartially, but limitations are not sufficiently addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}