Title: Focus-Then-Reuse: Fast Adaptation in Visual Perturbation Environments
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning; visual domain adaptation
Abstract: Visual reinforcement learning has shown promise in various real-world applications. However, deploying policies in complex real-world environments with visual perturbations remains a significant challenge. We notice that humans tend to filter information at the object level prior to decision-making, facilitating efficient skill transfer across different contexts. Inspired by this, we introduce Focus-Then-Reuse (FTR), a method utilizing a novel object selection mechanism to focus on task-relevant objects, and directly reuse the simulation-trained policy on them. The training of the object selection mechanism integrates prior knowledge from a vision-language model and feedback from the environment. Experimental results on challenging tasks based on DeepMind Control Suite and Franka Emika Robotics demonstrate that FTR enables rapid adaptation in visual perturbation environments and achieves state-of-the-art performance. The source code is available at https://github.com/LAMDA-RL/FTR.


Peer reviews:
comments: SUMMARY:
The paper presents an approach for fast policy adaptation in environments with noisy/varying backgrounds. The approach uses a combination of image segmentation and tracking to extract only the information relevant for the task, while ignoring the uncontrollable components in the background. The representation extraction, in particular the image segmentation selection, is trained using a combination of supervised and RL training. The supervised signal comes from a VLM model that is used to get a high level scene understanding.

STRENGTHS AND WEAKNESSES:
Strengths
- Results. Policy adaptation happens very fast

Weaknesses
- Complex approach, a lot of hyper-parameters to deal with
- Requires a specific policy pretraining (black background)

QUESTIONS:
While I think the idea of the paper is interesting, I have concern around the complexity of the approach and scalability.

- Why is it necessary to have both a tracking and segmentation model given that they are used in an exclusive way? Is it related to cost trade-off for the usage of the VLM?
- The fact that you need to train the frozen policy using no background seems quite a big limitation to me. Would it be possible to remove such constraint?
- How would you scale your approach to more complicated setting, eg navigation in Habit, robocasa, ManiSkill-HAB, Carla. In this case it seems quite complicated to have a background free training.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors propose Focus-Then-Reuse (FTR) a framework for visual reinforcement learning under changes of background information. In particular, FTR is composed of two phases: (i) the focus stage, responsible for segmenting and tracking relevant information for the RL task (e.g., the agent, robot and objects on interest in the workspace) through pre-trained segmentation models and a trainable high-level segment selection policy; (ii) the reuse stage, where a standard RL pipeline is employed. The authors evaluate FTR in Mujoco environments and in three robotic tasks in simulation. The results show that their approach outperforms the baselines in robustness to visual disturbances in the background.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- Paper is well-written and clear;\n- Experimental results show a clear improvement over the selected baselines.\n\n**Weaknesses**:\n- Limited novelty (See \"Comments\" below);\n- Some claims require support (See \"Comments\" below).\n\n**Comments**:\n- One major concern with the paper is the limited novelty of the work: using segmentation models to remove background distractors have been extensively explored in literature (e.g., [1, 2] for object-centric learning). From the discussion presented in the related work, and it is not clear where the novelty in the work lies. Moreover, the paper also introduces no novelty in terms of policy training as the authors employ standard PPO and DrQ-v2 algorithms for the focus and reuse stage policies.\n- While not a weakness per se, I do have concerns regarding the significance of the work: the authors propose a method for removing visual background distractors by segmenting objects relevant to the task. This induces the policy to overfit to these elements and their visual appearance, unable to deal with potential color changes, different lighting conditions, different positions of the camera, amongst other visual perturbations that are common in real-world scenarios. It is also unclear how the method would perform in face of distractor objects in the scene (such as other cubes and/or robot manipulators, for example). While this is not the scope of the paper, the concern remains.\n- Some claims in the paper are not supported by references in the current version of the paper: (i) in Lines 34-39, the authors claim that humans perform object-level filtering in decision making tasks, where object selection is initially driven by prior knowledge and subsequently fine-tuned by environmental feedback. It would improve the strength of the argument if the authors could support it with references. In line 225-226 the authors also claim that \"the higher the accuracy of the segment selector, the greater the cumulative reward obtained by the low-level policy, and vice-versa\". However, in the current version of the paper, the authors do not support this claim with any experimental validation.\n\n**References**:\n\n[1] - Shi, Junyao, et al. \"Composing Pre-Trained Object-Centric Representations for Robotics From\" What\" and\" Where\" Foundation Models.\" 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.\n\n[2] - Wang, Ziyu, et al. \"Generalizable visual reinforcement learning with segment anything model.\" arXiv preprint arXiv:2312.17116 (2023).\n\nQUESTIONS:\nI am willing to revise my score if the authors can address my concerns (see \"Strengths and Weaknesses/Comments\") and the following questions:\n\n1) What is the performance of your method when dealing with standard visual perturbances (e.g., color changes, different lighting conditions and camera changes) and distractor objects?\n\n2) Why did the authors not show the statistical results of the DrQ-v2 baseline? Also, in Table 1 which results are significantly better than the baselines?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework called Focus-Then-Reuse (FTR) to address the challenge of deploying reinforcement learning policies trained in simulation to real-world environments with visual perturbations. FTR has two stages: (1) focus stage: dynamically selects task-relevant objects using a trainable segment selector (trained via RL + VLM supervision), (2) reuse stage: executes actions with a fixed, pre-trained policy on the filtered observations. FTR outperforms baselines like SimGRL (generalisation) and PAD (adaptation), especially in complex visual perturbations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Integration of foundation models into RL:\n    - The paper innovatively leverages large-scale pretrained models (Qwen-VL and SAM 2) to address visual RL challenges, bridging the gap between high-level perception and low-level control. This is a promising direction, as foundation models offer rich prior knowledge but remain underutilised in RL for dynamic decision-making.\n    - The implementation (released in the supp.) is practically valuable, providing a valuable pipeline for combining VLMs with RL. This could inspire follow-up work on foundation-model-augmented RL.\n\n2. The fusion of VLM supervision and RL fine-tuning for the segment selector is a novel approach. The VLM provides an initial object-selection prior, while RL adapts it to task-specific rewards. The dynamic weighting mechanism (transitioning from SL to RL) is well-justified and could generalise to other hierarchical RL settings.\n\n**Weakness:**\n\n1. Sim-to-real gaps in the high-level policy:\n    - While the focus stage mitigates visual noise, the high-level policy (segment selector) is still trained purely in simulation. Real-world perceptual challenges (e.g., lighting changes, motion blur) may degrade its performance, necessitating additional adaptation.\n    - The paper does not test FTR on real-world deployment or quantify how the segment selector’s sim-trained priors (from Qwen-VL/SAM) transfer to real images. This limits claims about real-world readiness.\n\n2. Dependence on a pretrained low-level policy:\n    - FTR assumes access to a low-level policy trained in a clean simulation, which may not exist for many real-world systems. Even with a \"perfect\" simulation twin, the filtered observations may still differ from the simulation’s training distribution.\n    - The low-level policy is frozen during adaptation; FTR cannot correct for dynamics mismatches (e.g., sim-to-real mechanical discrepancies).\n\nQUESTIONS:\n1. How does FTR handle cases where SAM2 or the tracker fails to segment/task-relevant objects? Since the low-level policy depends on the selected segments, what is the impact on performance in real-world settings where such failures are common?\n\n2. Are the initial segmentation masks (or tracking updates) generated by SAM2 without prompts? Could the target object (e.g., a robot arm or manipuland) be oversegmented or contaminated with background pixels? If so, how does this affect policy performance?\n\n3. Instead of feeding raw pixels to the low-level policy, have you explored using SAM2’s object features (e.g., mask embeddings) as inputs? Would this introduce bottlenecks (e.g., latency, feature misalignment) compared to the current pipeline?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a VLM/segmentation-model assisted background segmentation scheme to help combat visual distractions in reinforcement learning. First, an optimal control policy is trained in the clean simulation environment. Then, an adaptation phase is conducted in the target environment. SAM-2 is used to propose candidate image segments to be provided to the frozen simulation control policy. To choose the image segments, a selection policy is trained in the target environments via downstream target environment RL rewards (collected by the control policy). To jumpstart training, the selection policy is additionally trained with supervised learning to match VLM suggestions in early stages of training. Segment selection is only performed occasionally, and currently selected segments are tracked between frames using the pretrained segmentation model. This method outperforms a variety of adaptation, generalization, and target-environment training approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well-written, proposing a straightforward and effective idea that other practitioners should be able to readily reproduce. \nMy only notable complaint is that the novelty is somewhat incremental due to VLMs having been used for background segmentation before, for example in [1], however the introduction of the segment-selection policy trained in the target-domain is new and significant, so I'm leaning towards acceptance.\n\nExperiments and ablations seem sufficient, except for one concern listed in *Questions*.\n\nThis is a missing reference that also uses VLMs to suggest segmentation masks while training with domain-randomized video backgrounds:\n\n[1] Kim et al. Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions. 2024.\n\nQUESTIONS:\nIt would be helpful to add a sanity check for all baselines (not just DrQ-v2) on the clean environment to show that they all can correctly learn an optimal policy and that any failures are due to the distractions. Could this be added?\n\nLIMITATIONS:\nThe authors state that other methods, when facing unseen disturbances, may drop performance significantly.\nThey should add more clear acknowledgements that their method still relies on the simulation policy generalizing to the target environment appearance for the segmented foreground objects. Likewise, this method can also drop in performance if the in-focus objects have a different appearance (lighting, texture, occlusion, etc.) compared to the simulation.\n\nAnother downside is that they require deployment-time segmentation/tracking, which can limit control frequency. Other baselines as well as [1] do not require this, and this should be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}