Focus-Then-Reuse: Fast Adaptation in Visual Perturbation Environments
Authors: Jiahui Wang
Chao Chen
Jiacheng Xu
Zongzhang Zhang
Yang Yu
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning; visual domain adaptation
Primary area: reinforcement_learning
TL;DR: We introduce a novel object selection mechanism to allow sim-trained policies to rapidly adapt to real-world visual perturbations.

Abstract: Visual reinforcement learning has shown promise in various real-world applications. However, deploying policies in complex real-world environments with visual perturbations remains a significant challenge. We notice that humans tend to filter information at the object level prior to decision-making, facilitating efficient skill transfer across different contexts. Inspired by this, we introduce Focus-Then-Reuse (FTR), a method utilizing a novel object selection mechanism to focus on task-relevant objects, and directly reuse the simulation-trained policy on them. The training of the object selection mechanism integrates prior knowledge from a vision-language model and feedback from the environment. Experimental results on challenging tasks based on DeepMind Control Suite and Franka Emika Robotics demonstrate that FTR enables rapid adaptation in visual perturbation environments and achieves state-of-the-art performance. The source code is available at https://github.com/LAMDA-RL/FTR.

Focus-Then-Reuse: Fast Adaptation in Visual Perturbation Environments
Jiahui :; Chao Chen 12; Jiacheng Xu Zongzhang Zhang Yang Yu!.²
National Key Laboratory for Novel Software Technology. Nanjing University, Nanjing, China "cchool of Artificial Intelligence, Nanjing University, Nanjing, China *Nayyang Technological University, Singapore {wangjh, h, chench 01 amda nju. edu. cn, jiacheng005@e.c int..ntu.eeei edu.s. {zzzhang, yuy)@nju. edu.uuuu.uuuu.....e.... edu. on
Abstract
Visual reinforcement learning has shown promise in various real-world applica- tions. However, deploying policies in complex real-world environments with vi- sual perturbations remains significant challenge. We notice that humans tend to filter information at the object level prior decision-making, facilitating efficient skill transfer across different contexts. Inspired by this, we introduce Focus- Then- Reuse (FTR), method utilizing novel object selection mechanism focus task-relevant objects, and directly reuse the simulation-trained policy on them. The training the object selection mechanism integrates prior knowledge from a vision-language model and feedback from the environment. Experimental re- sults on challenging tasks based on DeepMind Control Suite and Franka Emika Robotics demonstrate that FTR enables rapid adaptation in visual perturbation environments and achieves state-of- the- performance. The source code is avail- able at https: // http://iittub...m.omMaaMaaaa www. COm/LAMMA-RL/FTE
Introduction
Visual Reinforcement Learning (RL) has achieved breakthroughs in a wide range of real-world ap- plications, including robotic manipulation [[-B]] and autonomous navigation [[], DJ. However, bridg- ing the gap between simulation and real-world environments remains pivotal challenge. In this work, we focus on distracting background, which is typical form of visual disturbance. Taking robotic grasping as an example, although existing methods can achieve good performance in simu- lation [6., [7), directly deploying the learned policies in the real world may suffer from performance degradation when there exists complex backgrounds [8] g].
Recent studies have explored ways to address the challenge of deploying policies in complex en- vironments with visual perturbations. Existing approaches can categorized into three groups: (1) training policies directly in noisy real-world environments [100-22; (2) learning generalizable policies that are robust environment variations [[3:22]; and (3) adapting pre-trained policies to target environments with visual perturbations [23 25J. Each of these approaches presents potential limitations. First, policies trained directly in visually disturbed environments often incur high costs and risks, and the policy may generalize poorly to unseen settings. Second, generalization-bal baased methods rely on techniques such as data augmentation to simulate real-world variations during train- ing [26]] However, if the training process fails to capture the diversity of the deployment environ- ment, severe performance degradation may occur. Lastly, existing adaptation methods struggle to
These authors contributed equally. Zongzhang Zhang is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fully preserve policy performance achieved in simulation. We concentrate on an adaptation approach that aims to preserve the source policy performance with minimal degradation.
Inspired by the stark contrast with the limitations of existing adaptation methods, we investigate hu- man skill transfer. Notably, humans possess natural ability to transfer skills across diverse contexts. We believe this ability can be attributed to two key aspects. First, during perception, humans perform object-level filtering distinguish between task-relevant and irrelevant objects, effectively reducing the complexity of information processing [27]]. Second, in decision-making, humans can leverage both prior knowledge and environmental feedback to guide their actions. Initially, object selection is driven by prior knowledge; if the outcome is unsatisfactory, humans will iteratively adjust their selection policy based on environmental feedback until the task is completed [28]]
Humans acquire prior knowledge by summariz- ing large number of past experiences. Similarly, foundation models gain strong prior knowledge through pre-training on large-scale and diverse datasets, achieving remarkable success in natu- ral language processing [29, [S0) and computer vi- sion [31+35]] Recently, many studies have also explored using foundation models as a source of prior knowledge for downstream tasks [64] 35
Focus
Reuse
Inspired by humans' ability transfer skills and the success of foundation models, we introduce paradigm for visual domain adaptation RL. We propose Focus-Then-Reuss. (FTR), method that directly applies a simulation-trained policy to observations focused by a novel object selec- tion mechanism (Fig. M.). The object selection mechanism consists of trainable segment selec- tor, fixed segmentation model, and fixed track- ing model. The training of the segment selec- tor synthesizes prior knowledge from a Vision- Language Model (VLM) and feedback from the environment. Experiments on DeepMind Control Suite and Franka Emika Robotics show that FTR facilitates rapid policy adaptation from clean training environments to visually perturbed target en- vironments and achieves state-of-the-art (SOTA) performance.
Figure 1: Directly deploying the learned pol- icy in real-world environments may degrade per- formance. We propose Focus-Then-Reuse. method that utilizes novel object selection mechanism focus on task-relevant objects, and directly reuses the learned policy on them.
We summarize the contribution of this paper as:
We introduce a novel Mrcus-Then- Reuse framework for policy adaptation, with the focus stage filtering task-relevant objects and the reuse stage employing a fixed policy for faster and more stable adaptation.
We propose an adaptation RL method that integrates supervised learning and reinforcement learning synthesize VLM's prior knowledge and environmental feedback.
Experiment on challenging tasks demonstrates that our method facilitates quick and effec- tive policy adaptation to visual perturbation environments, achieving SOTA performance.
2 Related works
The pursuit robust policy deployment across visually perturbed domains has catalyzed research interest in recent years. Some works try to train policies in visually distracted environments directly. DBC [66] first explores this pathway by learning a representation robust to distractions. Building on DBC, -earning [[ decouples policy learning from behavioral metric learning for stable training. Some model- based methods [[22 37-69]] explicitly recognize task-relevant parts. However, policies trained directly in visually perturbed environments often suffer from high training costs and risks, as well as limited generalization to other scenarios.
Generalization methods explore ways enhance policy performance, which can be classified into three types: data augmentation, inductive bias, and learning invariances [26]. Data augmentation methods [|-- apply techniques like cropping and color jittering to training images to reduce

---

the distribution gap between training and testing. Inductive bias methods [19 22, 26 incorporate assumptions about task-relevance (e.g., foreground is of higher importance), among which Sim- GRI. [T3] achieves the SOTA performance. Invariance-based methods |90 focus on extracting information consistent across diverse training environments. We point out that generalization meth- ods face two issues. First, added perturbations or inductive biases may hurt training performance. Second, when facing unseen disturbances, the performance may drop significantly.
Domain adaptation RL focuses on transferring policies trained in the source domain to the target do- main [9G 50.. Of particular relevance to this paper is visual domain adaptation RL, which remains relatively under-expe under-explored. PAD [[X]] is the work most closely related to our setting. It introduces a self-supervised objective during training and performs online adaptation at deployment through this objective, enabling policy udaptation to target domains with simple backgrounds. Other studies have explored policy adaptation to target domains with varying camera viewpoints [244], changes in color and object scale [25]] or new visual dynamics [5]]]. However, we suggest that current adapta- tion methods fail fully recover the policies' performance, with severe degradation in challenging environments like video background [[_]]
Foundation models can leverage their prior knowledge to support perception and decision-making in downstream tasks [52, D.3]. For perception, some approaches [>4] T employ foundation mod- els as pre-trained feature extractors, while others |I|I 56 59 incorporate promptable segmenta- tion models to enhance scene understanding and representation learning for visual RL agents. For decision- making, common approach is to employ foundation models as reward generators to pro- vide learning signals for policy optimization [60), 61]]. Our method leverages foundation models to assist RL. from both aspects. On one hand, we use pre-trained segmentation and tracking models to process complex inputs in zero-shot manner. On the other hand, we employ Vision-Language Model (VLM) as a supervision signal in policy training.
3 Preliminaries
Reinforcement learning Traditional RL considers the task in the form of Markov Decision Process (MDP) M = (S, =(S,,44 A. RR.. S state space, A is action space. P:S x A S + [0, 1] is the transition function that defines the conditional probability distribution P(s |s, az) over next states given state s E S and action a E A at time t, R S x A + R is a reward function. RL aims to learn policy T[ass) that maximizes the expected discounted cumulative reward E,, (E.77))) where 7 E [o, 1] is the discount factor. In visual RL tasks, the agent can only have access to the observation on rendered from 8,, and thus the policy becomes (alo). In this paper, we use Proximal Policy Optimization (PPO) [62]]. PPO is an effective algorithm for solving general RI. problems. Given policy me (a)ss) and value function (s), PPO maximizes the following objectives:
E [())- (s,0,1,")) J-F
[min ( ae (s, a), clip(r(0), e, A (s, a))],
[(w)= (s.arr,))) J-TTR
E
[Va(s) - (V (s) + M (s, (s)a))]
salala) (s]) is the probability ratio measuring how much the current policy deviates from the old policy used for data sampling, while the clip(-) operation prevents overly large policy updates. The hyperparameter defines the clipping boundary. The value function (s)(s) is a param- eterized estimate of the expected discounted cumulative reward starting from state s and following policy ne thereafter. The advantage AMM (s, a) is usually estimated from (s) via Generalized Advantage Estimation (GAE) [[3]]. 1b- denotes the target network parameter, which is periodically synchronized with & for stable training.
Here r(0)
Visual foundation models Visual foundation models are commonly pre-trained on massive datasets and can adapt to various downstream tasks in zero-shot way. In this work, we fo- cus specifically on foundation models designed for image understanding [B3), image segmenta- tion [64]] and object tracking [b5]]. For image understanding task, we use Qwen-VL-Ne
[33], a multimodal version of the Qwen large model series that outperforms current SOTA general- ist models on multiple vision-language tasks. It takes combination of image and text as in- put and generate structured textual outputs following a specified format (e.g.. ISON). For im- age segmentation and object tracking, we use SAM 2 [2), which achieves SOTA performance
3

---

Focus Stage
Reuse Stage
Observation:
Previsus
Original policy: ae
segntent
ervation: of
Reinforcement Learning
Tracking Model
Update
trudking?
Train
Yes Segmentation Model
Seguent selectors Chigh policy)
Lew-levd polleys af
Segments o
Selected segments: 02
Action
Reward
Environment
Supervised
Learning
Reinforcement
iB
Vislon-Language
Reuse zp1r1
Model
Label:
Figure 2: Architecture overview. FTR comprises a focus stage and a reuse stage, depicted in the or- ange and purple boxes, respectively. The focus stage utilizes a novel object selection mechanism filter task-relevant segments, and the reuse stage applies a fixed, simulator-trained policy to generate actions based on the selected objects. The object selection mechanism consists of trainable seg- ment selector, a fixed segmentation model, and fixed tracking model. The training of the segment selector synthesizes prior knowledge from VLM and feedback from the environment.
in both tasks. In image segmentation task, the model takes an image o E RC (C. W. H. are respectively channels, width, and height) as input and outputs set of binary masks m² i (1.....ke .k), m² 13 "x] Each mask m' corresponds to single object instance, with indicating a pixel of the object. Segmented images are then obtained by element-wise multiplication (O), Ooe E (1.... ,k,,, o m' RCCW*H] In object tracking task, the model takes a video and masks as input and propagates the masks across the video. Given video with frames for E (1. .T....Thee
,T, or e gCxW'H) and object masks the first frame
mi i E (I, k}. ml E 10 Wx#] j. the goal of tracking is to predict the object masks in subsequent frames, {m; E f [
4
Method
We present Tocus-Then-Rewie (FTR), hierarchical framework designed to quickly deploy policies target domains with visual perturbations. FTR facilitates efficient policy adaptation by maintain- ing the core functionality of the original policy while dynamically compensating for visual pertur- bations through learned "focus" module. In this section, we first provide a brief overview of the FTR framework. Following this, introduce the forward process of our method, which is divided into two stages: the focus stage and the reuse stage. For the focus stage, we combine two filtering techniques identify the task-relevant objects. For the reuse stage, explain the training and reusing of the original policy. Finally, we present the training approach for the segment selector, which incorporates both supervised learning objective and reinforcement learning objective.
4.1 Architecture overview
An overview of FTR is provided in Fig. DL FTR comprises two distinct stages: a high-level focus stage and a low-level reuse stage. They are in the orange and purple boxes, respectively. the focus stage, image observation with visual disturbance is processed. Based whether the tracked objects require updating, task-relevant segments are obtained either through a segment selector (high- level policy *²) or a tracking model. In the reuse stage, the selected objects og are fed into the

---

original policy, which is acquired via pre-training in clean environment, to obtain the action. The training objective of n" combines RL with VLM supervision to enable efficient adaptation.
4.2 Focus stage: segmentation, selection and tracking
In this section, we introduce the focus stage in detail, as shown in the orange box in Fig. 2. At time step 4, the image observation of the visually disturbed environment denoted as or E RR RCx The focus stage takes as input and outputs the filtered task-relevant objects, denoted as of xC Depending on whether the tracked objects need update (blue diamond box in Fig. D), there are two approaches: The first approach uses segmentation model and segment selector (the selection pathway). The second approach employs a tracking model and historical selected images (the tracking pathway). We introduce these two pathways and explain their necessities.
4.2.1 Selection pathway
The selection pathway corresponds to the downward arrow labeled "Yes" Fig. D. This pathway filters objects through image segmentation model and segment selector xh. First, image seg- mentation model is employed to segment the disturbed observation os, yielding a set of k: segments: ot {or seg, | E {1, ,r seg. RCXW*H) Subsequently, x" takes the l segments and the original observation o, as input, and output the mean values 14, M2. ......)))) E (0, 1)*.
By sampling from a diagonal Gaussian distribution W(, o with pt as mean, hyperparam- eter d variance (Øt 0.1 by default), and I the identity matrix, we get action at (at Lael ..... aj aell)) ). Using 0.5 as the threshold, the i-th segment is selected its value atdl: 0.5; otherwise, it is discarded. The selected segments are then integrated to form of E RCxW* rep- resenting the union of focused objects:
gel
U o0 te IEI.
where Ij = =iilie((1,.. kk,, at > 0.5)
By obtaining of through the above sampling method, we can effectively explore different segment selection patterns and prevent premature convergence local optima.
4.2.2 Tracking pathway
While the selection pathway alone is sufficient to derive ofel this pathway suffers from a major issue: the inconsistency in segment selection. A segment selector that has not been sufficiently trained to develop a stable selection pattern is likely to produce results lacking short-term consistency, resulting in inconsistent actions and low-quality reward signals. Therefore, we propose another approach to derive ordl, the tracking pathway. Whenever we obtain the selected objects in the selection pathway, we simultaneously record them the tracking model. The tracking model then recognizes the selected objects of
del using previous selection results.
Till now, have fully introduced the pathways for selecting images. We propose a simple yet effective mechanism to combine these two pathways. We introduce a selection interval Ted (Tue 20 by default). If the current timestep is divisible by Tes the selection pathway is called; otherwise, the tracking pathway is used. In a comerr case where no segment is selected by xh, tracking must also be refreshed in the subsequent timestep, regardless of Tee.
4.3 Reuse stage
Before adaptation, the original policy " is trained in a clean environment without visual perturba- tion using the DrQ-v2 algorithm [[6]]. In the reuse stage, х is copied and frozen as the lower-level policy -'. As a common practice in visual RL, the last 3 frames of focused objects otd of of are stacked and fed into n¹, yielding action a, for environment interaction to obtain reward rt. Notably, the focus stage and the reuse stage are highly decoupled, meaning that our method is compatible with range of RI. algorithms. Additionally, our method requires no modifications to r!
5

---

4.4 Segment selector trained with supervised learning and RL
We introduce the training process of FTR in this section. During training, only the parameters the segment selector T" are updated, while those of the segmentation model, tracking model, and low- level policy n' remain frozen. The training objective of the segment selector consists of two parts: the first is a supervised learning objective based on VLM (blue dashed line in Fig. 2); the second an RI. objective based on environmental rewards (green dashed line in Fig. E). Recalling the way humans handle complex environments, as mentioned in Section II, supervised learning can be seen as leveraging prior knowledge, while RL resembles adjusting based on environmental feedback.
4.4.1 Supervised learning objective
We use the discrepancy between the VLM's selection and the output of π" as the supervised loss. Given segments of an example image from the source domain, and output format, the VLM returns Yt = (yt, vi: ,... : .tt), where y E {0, 1). indicating whether each segment should be focused on. (Or, seg .Y) are added to Dst. the supervision dataset, for training. We provide more details about VLM Appendix A.5
With the selection result of VLM, an intuitive way of training is to use the Binary Cross Entropy (BCE) loss. However, we point out that the rigidity binarize of μe will degrade the subsequent RL optimization. Instead, we use a "softer" margin- regularized loss function:
(8) = E (0.0" y)-Da y max(0, 0.5 + d - a a²d)) + + (1-y) max(0, a - (0.5-6)) 6))] (3) (10,,0
where ò a predefined margin hyperparameter (8 0.1 by default). For positive samples (y² = 1). the loss penalizes predictions afl that fall below 0.5 + 5 with linearly increasing loss. For negative samples (y² = 0), penalizes predictions exceeding 0.5 5. 5.
4.4.2 RL objective
Although the VLM exhibits impressive reasoning capabilities, its predictions can be unreliable in certain scenarios, and its latency several seconds is unacceptable for real-time control tasks. These limitations motivate the integration of RL to refine the policy. Specifically, n² is optimized using the PPO algorithm within a hierarchical framework. The state, action, and reward of π^ are:
State includes observation og and segments of when h called on the selection pathway.
Action is the selection vector at sampled from N( 1)
Reward is the cumulative reward EttTa-1 r, between two consecutive invocations of hh
Based on the MDP formulation, we train the high-level policy n with the PPO algorithm, and the loss function fRI is defined in Eq. II The RL objective relies on simple yet valid assumption: the higher the accuracy of the segment selector, the greater the cumulative reward obtained by the low-level policy, and vice versa.
4.4.3 Combination of the objectives
Both objectives involve trade-offs: supervised learning enables rapid convergence but is limited by the VLM's accuracy and latency, whereas RL can achieve better performance but may result in slower training. The total loss function is defined as follows:
L TTLL + CRL,
where the weights of CSL and CRL are Tss and MRL respectively. To complement the two objectives, we propose a dynamic adjustment mechanism of 1/s and MRL- This strategy allows the training to benefit from supervised learning for fast convergence in the early phase, while gradually shifting towards reinforcement learning for fine-tuning in the later phase. Specifically, when < T1, 7sL
1, /RI. = 0. During Ti < < T2, 1JsI. decays linearly from to 0 while TR.. rises from 0 to
1, mitigating gradient misalignment between objectives. Note that the VLM is called only when
1 Ti with its outputs added to DsI. When L. > T2, 1/SL. = 0, f/RL.
By default, we set T = 5000 and Tž 10000, thereby restricting VLM calls to TTa 250 times.

---

Experiments
In this section, we present the experimental results of FTR on 11 tasks, including tasks of Deep- Mind Control Suite (DMC) [67] and tasks of Franka Emika Robotics [6X. 69]. DMC is widely used benchmark of visual RL, while Franka Emika Robotics can better reflect real-world training scenarios. Finally, conduct ablation studies validate each module's effectiveness.
5.1 Experimental setup
We 8 tasks from DMC: pendulum-swingup (ps), cartpole-swingup (cs), finger-spin (fs), hopper- stand (hs), hopper-hop (hh), cheetah-run (cr), walker-walk (ww), walker-run (wr), and tasks for robotic manipulation: franka-reach (fr), branka-push (fp). and franka-door (fd). These tasks are di- verse and comprehensive, ranging from single objects to multiple objects and from easy locomotion tasks to dexterous manipulation tasks.
Our research explores fast adaptation from source to target domains. The source domain corresponds environments with no background. Policies are trained using the DrQ-v2 algorithm. We perform three independent runs and choose the policy with the best performance as the original policy -. To simulate real-world visual disturbances, we use five diverse videos from the AMC--eneralization Benchmark [[Z]] shown in Fig. B These videos cover range of indoor and outdoor scenes and serve the backgrounds of five target domains. We perform domain adaptation independently on each target domain and report the average performance.
Figure Source domain (leftmost) and five target domains task franka franka-push.
We compare FTR with the following baseline methods:
DrQ-v2 (clean) [66]. DrQ-v2 is a classic visual RL algorithm. is also used as the default method by FTR for training the original policy in the source domain. DrQ-v2 (clean) can be regarded as the potential upper bound for FTR's 's domain adaptation performance.
SimGRL [3] SimGRL achieves SOTA performance in visual generalization RL, demon-
strating effectiveness in test environments with video backgrounds.
PAD [23]. PAD is classical visual domain adaptation method in reinforcement learning, and has shown effectiveness in adapting target domains with simple backgrounds.
- -learning [[ Q -e-leannng is representation learning-bast baased method for performing RL directly on complex visual inputs.
We also compare against FTR w/o SL and FTR w/o RL. FTR w/o SL refers to the variant of FTR trained without supervised learning, relying solely on the RI. objective. Conversely, FTR w/o RI. denotes the variant trained without RI.
FTR and the baseline methods can be categorized into four groups: classical visual RL method DrQ- v2, generalization method that requires no interaction with target domains, adaptation methods that involve limited interaction, and robust training method that is trained directly on the target domains. Although we present the results of all methods in the same table, we clarify their differences:
Classical visual RL method (DrQ-v2 (clean)): We perform three independent runs in the source domain and report the best performance.
Generalization method (SimGRL): Policies are trained in the source domain for 500k steps across three runs. The trained policies are then evaluated in the five target domains.

---

Adaptation methods (FTR, PAD): We first conduct three training runs in the souree do- main and select the best-performing policy as the initial policy. This policy is then adapted to each of the five target domains using three different seeds for 200k steps.
Robust training method (Q'-learnigg) Policies are trained directly in each target domain for 500k steps.
We report the mean and standard deviation of performance across the five target domains for all methods except DrQ-v2 (clean).
5.2 Experiment results
Table 0I and Fig. show the performance for FTR and baseline methods. Table records the final performance. The solid lines and shaded area in Fig. ! correspond to the mean and variance. FTR achieves the best performance on 10 out of 11 tasks.
Compared to baselines, FTR demonstrates clear advantages in both sample efficiency and perfor- mance. In terms of sample efficiency, FTR leverages prior knowledge from the VLM achieve strong initial performance and improves rapidly using environmental feedback, achieving conver- gence within 50k steps in most cases. Regarding final performance, FTR outperforms baselines on all tasks except cartpole-swinge tial upper bound represented by DrQ-v2 (clean). On the most complex robotic manipulation tasks, FTR shows considerable advantages over other methods. SimGRL, the SOTA visual generalization RL method, performs decently on most tasks, but compared FTR, it only holds an advantage on the cartpole-swingup task, task with sparse rewards and challenging object segmentation. This suggests that even with strong augmentation during training, encountering unseen disturbances at deployment can still lead to substantial performance degradation. Similarly, Qe-learning obtain a certain level of performance on most tasks. However, its performance remains inferior to FTR except cartpole-swingup. This indicates that traditional representation learning methods still struggle to handle complex visual perturbations. Despite being classic visual domain adaptation RL method, PAD performs poorly all 11 tasks. This indicates that existing adaptation methods struggle with complex, near-real-word target domains with video background.
On average across all tasks, FTR retains over 85% of the poten-
As ablation study to evaluate the respective contributions of supervised learning and reinforce- ment learning in FTR, we compare FTR with its two variants: FTR w/o SL and FTR w/o RL. Comparing FTR and FTR w/o SL, we observe similar final performance; however, FTR converges significantly faster. This suggests that the prior knowledge from VLM offers a well-informed ini- tialization. On the cartpole-swingup task, the performance of FTR w/o SL exhibits notable per- formance drop. We speculate that this due to the task's 's reward being sensitive to state variations, which causes excessive noise in the RL objective. Comparing FTR and FTR w/o RL, can be ob- served that incorporating RL consistently improves performance. This indicates that the supervisory signal alone is insufficient, as errors from the VLMM lead to sub-optimal policies. Therefore, RL. essential for achieving optimal domain-adaptive performance.
Table 1: Performance comparison of FTR and baselines (mean + std). Note that train and evaluate DrQ-v2 in the clean source domain, and test all other methods in visually perturbed target environments. For DrQ-v2 (clean), we report the best performance over three runs as the potential upper bound for other methods.
Task
DrQ-v2 (clean)
PAD
Q" d'-leaming
SimGRL
FTR w/o RL
FTR w/o SL
FTR (ours)
829.0
0.5 1.
0.4
436.6 331.2
46.1
49.8
708.1 1 1. 145.5
770.9 1
85.4
786.7
82.3
5.9
807.3
578.6
541.2
187.2
373.7
646.0
168.5
Is
973.3
0.8
0.7
605.8
121.8
351.6
273.8
718.0 1 183.8
909.1
62.3
903.9
71.2
2.2
2.2
264.2 125.8
746.9
139.3
563.0 + 209.3
68.0
bh
336.4
0.5
0.7
91.7= 18.9
121.2
20.3
231.1 77.2
300.9
17.1
307.3
20.8
6.8
6.8
221.6= 49.8
221.4
110.6
344.9
124.7
427.3
70.5
446.1
957.1
26.3
10.5
325.3= 62.2
73100
41.6
558.6
212.2
891.5
41.3
899,7
45,4
25.8
141.5= 17.9
312.1
27.8
244.3 + 94.3
359.2
36.7
366.5
657.7 = 220.0
10.3
161.9
52.0
121.5
10.4 11.6
19.0 12.9
4.2 91.5
6.5.L.
64.2 47.4
107.4
15.1
96.8 1: 33.8
0.3
2.7
28.5
8

---

Steal
pendulum-swee swinguu
cartpole-swingup
(c) finger-spin
(d) hopper-stand
2.4
200
Steas
Stere
Stens
(e) hopper-hop
(l) cheetah-run
walker-walk
(h) walker-run
DiQu22 (clean)
FTR
FTR w/o St.
FTR w/o RL
SimGRL
Q*-lerring
RAD
(i) franka-reach
( franka-push
(k) franka-door
(I) Legends
Figure 4: Training curves DeepMind Control Suite (a-h) and Franka Emika Robotics (i-k). Note that for the adaptation methods (FTR, PAD), we set the adaptation duration to 200k steps, and for other methods, train for 500k steps. FTR has converged by 100k steps across all tasks.
5.3 Further ablation studies
We conduct ablation studies on different selection intervals Twa SL-to-RL transition timesteps Ti, and supervised learning objectives LSL on the finger-spin task.
The results under different values of Taa are in Fig. bla). When Tea (using only the selection pathway while disabling the tracking pathway). the RL process fails to converge, leading to poor final performance. This highlights the critical role the tracking pathway in FTR. As mentioned in Section 9.2.2, the tracking pathway helps maintain consistency in segment selection, thereby enhancing the stability in RI. training. For Tee E (10, 20, 40), performance exhibits minor variation.
In Fig. bbb), the performance shows negligible differences between Ti 1000 and Ti 5000, demonstrating the effectiveness of our method even with limited supervision from the VLM. When Ti = 10000, slight performance drop is observed, which may be attributed to overfitting to DsL.
As shown Fig. Blc),, using BCE loss as the supervised learning objective leads to a significant degradation in performance, especially during the transition from supervised learning to reinforce- ment learning. This underscores the importance of the proposed margin-regularizer loss in Eq. B for ensuring a stable transition from supervised learning to reinforcement learning.
1200
100
100
200
600
100
400
200
200
Liss 11
10k
DCE lms
0.0
0.5
1.0
20
5.0
0.5
1.0
1.5
20
0.0
0.5
1.0
1.5
20
(a) Ablation Ted
(b) Ablation on Ti
(c) Ablation on Lst.
Figure 5: Ablation studies on different selection intervals Tea SL-to-RL transition timesteps Ti, and
supervised learning objectives LsL the finger-spin task.

---

Conclusion
In this work, we propose a novel Focus-Then-Reute (FTR) framework achieve rapid policy de- ployment in real-world environments with background disturbances. The core of FTR lies in train- ing segment selector using both environmental rewards and VLM's supervision identify task- relevant objects, while directly applying source domain policy on the filtered visual inputs. The focus stage and the reuse stage are highly decoupled, meaning that FTR compatible with range of generalization RL algorithms and has the potential to handle complex distribution shift. Fur- thermore, we propose novel object selection mechanism that combines segmentation model and tracking model to improve object selection consistency and enhance the stability in RL training. Experimental results on the DeepMind Control Suite and Franka Emika Robotics indicate that our method effectively synthesizes prior knowledge from the VLM and environmental feedback, demon- strating ad advantages over baseline methods in performance, efficiency, and interpretability.
For future work, we suggest three aspects worth improving and exploring. First, while FTR demon- strates effective deployment target domains with visual perturbation, it struggles in additional disturbances such as camera pose variations. A potential solution involves incorporating visual generalization RL approaches enhance the robustness of the source domain policy. Second, the fixed selection interval Teel relies manual configuration. Developing an adaptive scheduler for Tel could potentially enhance performance. Finally, the performance of FTR depends on the seg- mentation and tracking model, suggesting the need for foundation models specifically tailored for downstream RL tasks.
Acknowledgments
We thank the reviewers for their insightful and valuable comments. This work is supported by the National Science Foundation of China (62276126, 62250069).
References
[1] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
[2] Rongrong Liu, Florent Nageotte, Philippe Zanne, Michel de Mathelin, and Birgitta Dresp- Langley. Deep reinforcement learning for the control of robotic manipulation: A focussed mini-review. Robotics, 10(1):22, 2021.
[3] OpenAl: Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learn- ing dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1): 3-20, 2020.
[4] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. In International Conference Learning Representations, 2017.
[5] Yuke 7hu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In IEEE International Conference on Robotics and Automation, pages 3357-3364, 2017.
[6] Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4737-4746. 2023.
[7] Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-speccalis learning. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 3891-3902, 2023.
10

---

[8] Naomi Chukwurah, Abiodun Sunday Adebayo, and Olanrewaju Oluwaseun Ajayi. Sim-to- real transfer in roboties: Addressing the gap between simulation and real-world performance. International Journal of Robotics and Simulation, 6(1):89-102, 2024.
[9] Longchao Da, Justin Tumnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods RL: Progress, prospects and challenges with foundation models. arXiv preprini arXiv:2502. 13187, 2025.
[10] Chao Chen, Jiacheng Xu, Weijian Liao, Hao Ding, Zongzhang Zhang, Yang Yu, and Rui Zhao. Focus-then-decide: Segmentation-assisedd reinforcement learning. AAAI Conference on Artificial Intelligence, pages 1240-11248, 2024.
[11] Weijian Liao, Zongzhang Zhang, and Yang Yu. Policy-independent behavioral metric-based representation for deep reinforcement learning. In AAAI Conference on Artificial Intelligence, pages 8746-8754, 2023.
[12] Tongzhou Wang, Simon S. Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. Denoised MDPs: Learning world models better than the world itself. In International Conference Machine Learning, pages 22591-22612, 2022.
[13] Wonil Song, Hyesong Choi, Kwanghoon Sohn, and Dongbo Min. A simple framework for gen- eralization in visual RL under dynamic scene perturbations. Advances in Neural Information Processing Systems, pages 21199--12824 2024.
[14] Roberta Raileanu, Maxwell Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Auto- matic data augmentation for generalization in reinforcement learning. In Advances in Neural Information Processing Systems, pages 5402-5415, 2021.
[15] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforce- ment learning with mixture regularization. In Advances in Neural Information Processing Systems, pages
2020.
[16] Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. In Advances in Neural Information Processing Systems, pages 3680-3693, 2021.
[17] Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data augmentation. In International Conference on Robotics and Automation, pages 1361 1-13617,
2021.
[18] Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. Secant: Self-expert cloning for zero-shot generalization of visual policies. In International Conference on Machine Learning. pages 3088-3099, 2021.
[19] Xudong Wang, Long Lian, and Stella X Yu. Unsupervised visual attention and invariance for reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Cision and Pattern Recognition, pages 6677-6687, 2021.
[20] David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. Look where you look! Saliency-guides q-networks for generalization in visual reinforcement learning. In Ad- vances in Neural Information Processing Systems, pages 50693-30706 2022.
[21] Bram Grooten, Tristan Tomilin, Gautham Vasan, Matthew E Taylor, A Rupam Mahmood. Meng Fang, Mykola Pechenizkiy, and Decebal Constantin Mocanu. MaDi: Learning to mask distractions for generalization in visual deep reinforcement learning. In International Confer- ence on Autonomous Agents and Multiagent Systems, pages 733-742, 2024.
[22] Di Zhang, Bowen Lv, Hai Zhang, Feifan Yang. Junqiao Zhao, Hang Yu, Chang Huang, Hongtu Zhou, Chen Ye, et Focus on what matters: Separated models for visual-based RL general- ization. In Advances in Neural Information Processing Systems, pages 1 169660-116986 2024.
[23] Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. In International Conference on Learning Representations, 1.2..

---

[24] Sizhe Yang, Yanjie Ze, and Huazhe Xu. MoVie: Visual model-based policy adaptation for view generalization. Advances Neural Information Processing Systems, pages 21507-21523, 2023.
[25] Biwei Huang, Fan Feng, Chaochao Lu, Sara Magliacane, and Kun Zhang. AdaRL: What, where, and how adapt in transfer reinforcement learning. In International Conference Learning Representations, 2022.
[26] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning, Journal of Artificial Intelligence Research, 76: 201-264, 2023.
[27] Courtney Stevens and Daphne Bavelier. The role of selective attention on academic founda- tions: cognitive neuroscience perspective. Developmental cognitive neuroscience, pages 2012.
[28] Jeremy Roschelle. Learning in interactive environments: Prior knowledge and new experience. Museums Institutions for Personal Learning, 1999.
[29] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[30] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, et al. DeepSeek-Ri: incentivizes reasoning in Ilms through reinforcement learning. Nature, 645:633-638, 2025.
[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEESCVF International Conference on Computer Vision, pages 4015- 4026, 2023.
[32] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma. Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. SAM Segment anything in images and videos. In International Conference Learning Representations, 2025.
[33] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL. VL: versatile rision-lanuue model for understanding. localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
[34] Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, King-Hsuan Yang, and Fahad Shahbaz Khan. Foundation mod- els defining a new era vision: a survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2245-2264, 2025.
[35] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pages 1-65, 2024.
[36] Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. Inter- national Conference on Learning Representations, 2021.
[37] Xiang Fu, Ge Yang, Pulkit Agrawal, and Tommi Jaakkola. Learning task informed abstractions.
In International Conference on Machine Learning, pages 3480-3491, 2021.
[38] Minting Pan, Xiangming Zhu, Yitao Zheng, Yunbo Wang, and Xiaokang Yang. Model-based reinforcement learning with isolated imaginations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):2788-2003 2023.
12

---

[39] Miles Richard Hutson, Isaac Kauvar, and Nick Haber. distractions in model-based reinforcement learning. In Advances in Neural Information Pro- cessing Systems, pages 13124-13148, 2024.
prediction: Avoiding
[40] Rishabh Agarwal, Marlos C Machado, Pablo Samuel Castro, and Mare G Bellemare. Con- trastive behavioral similarity embeddings for generalization in reinforcement learning. In In- ternational Conference on Learning Representations, 2021.
[41] Donghu Kim, Hojoon Lee, Kyungmin Lee, Dongyoon Hwang, and Jaegul Choo. Investigating pre-training objectives for generalization in vision-based reinforcement learning. In Interna- rional Conference on Machine Learning, pages 24294 -43226, 2024.
[42] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI Con- ference on Artificial Intelligence, pages 10674-10681, 2021.
[43] Jinwei Xing, Takashi Nagata, Kexin Chen, Xinyun Zou, Emre Neftci, and Jeffrey L Krichmar. Domain adaptation in reinforcement learning via latent unified state representation. In AAAI Conference on Artificial Intelligence, pages 10452-10459, 2021.
[44] Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan Adams, and Sergey Levine. Why generalization in RL is difficult: Epistemic pomdps and implicit partial observability. In Advances in Neural Information Processing Systems, pages
2021.
[45] Bonnie Li, Vincent François-Lavet, Thang Doan, and Joelle Pineau. Domain adversarial rein- forcement learning. arXiv preprint arXiv:2102.07097 2021.
[46] Lanqing Li, Rui Yang, and Dijun Luo. FOCAL: Efficient fully-offline meta-reinnorreemnnt learning via distance metric learning and behavior regularization. In International Conference on Learning Representations, 2021.
[47] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcement learning via probabilistic context variables. In International Con- ference on Machine Learning, pages 5331-5340, 2019.
[48] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. In Advances in Neural Information Processing Systems, pages 29374-29385, 2022.
[49] Fan-Ming Luo, Shengyi Jiang, Yang Yu, Zongzhang Zhang, and Yi-Feng Zhang. Adapt to environment sudden changes by learning a context sensitive policy. In AAAI Conference on Artificial Intelligence, pages 7637-7646, 2022.
[50] Xinyu Zhang, Wenjie Qiu, Yi-Chen Li, Lei Yuan, Chengxing Jia, Zongzhang Zhang, and Yang
Yu. Debiased offline representation learning for fast online adaptation non-stationary dy- namies. In International Conference Machine Learning, pages 59741-59755 2024.
[51] Lin Yen-Chen, Maria Bauza, and Phillip Isola. Experience-embeeded visual foresight. In Conference on Robot Learning, pages 1015-1024, 2019.
[52] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255 2021.
[53] Sherry Yang. Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.04129, 2023.
[54] Che Wang, Xufang Luo, Keith Ross, and Dongsheng Li. Vr13: A data-driven framework for visual deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 32974-32982 2022.
[55] Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. Pre-trained image encoder for generalizable visual reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pages 13022-13037, 2022.
13

---

[56] Junjie Zhang, Chenjia Bai, Haoran He, Zhigang Wang, Bin Zhao, Xiu Li, and Xuelong Li. SAM-E: Leveraging visual foundation model with sequence imitation for embodied manipula- tion. In International Conference on Machine Learning, pages 58579-58599 2024.
[57] Ziyu Wang, Yanjie Ze. Yifei Sun, Zhecheng Yuan, and Huazhe Xu. Generalizable visual reinforcement learning with segment anything model. arXiv preprint arXi::2112.17116, 2023.
[58] Kyungmin Kim, JB Lanier, and Roy Fox. Make the pertinent salient: Task-relevant reconstruc- tion for visual control with distractions. In Reinforcement Learning Conference, 2025.
[59] Junyao Shi, Jianing Qian, Yecheng Jason Ma, and Dinesh Jayaraman. Composing pre- trained object-centric representations for robotics from" what" and" where" foundation models. In IEEE International Conference on Robotics and Automation, pages 15424-15432, 2024.
[60] Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Er- iekson. RL-VLM-FF: Reinforcement learning from vision language foundation model feedback. In International Conference on Machine Learning, pages 51484-51501, 2024.
[61] Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, and Sikai Chen. VLM-RL: A unified vision language models and reinforcement learning framework for safe autonomous driving. arXiv preprint 4rXiv:2412.15544, 2024.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (iti::1707.064477 2017.
[63] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Con- ference on Learning Representations, 2016.
[64] Abdul Mueed Hafiz and Ghulam Mohiuddin Bhat. A survey on instance segmentation: State of the art. International Journal of Multimedia Information Retrieval, 933):111-19, 2020.
[65] Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: A survey, ACM Computing Surveys, 33, 2006.
[66] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representationss 2022.
[67] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801.00690, 2018.
[68] Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. RL-ViGen: A reinforcement learning benchmark for visual generalization. In Advances in Neural Informa- tion Processing Systems, pages 6720-6747, 2023.
[69] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martin Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. Robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
14

---

Appendix
A Implementation Details
In this section, we describe the implementation details of our work.
A.1 FTR implementation
The source code is available https://githu: COm/LAMDA-RL/ED
The code is mod- ified from DMC-Gennralizztion Benchmark [077] and FTD [T]] The PPO algorithm used track github........ 10/2022/03/25. The DrQ-v2 algorithm is implemented based on https://
in FTR is implemented based https://iclr-bs
ppo- wmw.mmmmmmnttattondtttalllll github. cca//ffacbookrreeaccccccec arqq2 For details hyperparameters and network architec- ture, please refer to the "Hyperparametens" section and the "Network Architecture" section. Algo- rithm for FTR is shown in Algorithm III Most experiments are conducted
server outfitted with
2 AMD EPYC 7542 32-Core Processor CPUs, 504GB of RAM, and GPUs, each with a perfor- mance of over 35 TFLOPS, running Ubuntu 22.04. Training the source domain using the DrQ-v2 algorithm for 500k steps takes about day. Adapting in the target domain for 200k steps takes about hours.
Algorithm Focus- Then-Reut
1: Initialize: segmentation model, tracking model, VLM, segment selector n², original policy trained in clean environment -" replay buffer DRL, VLM supervision dataset DsL
2: for t = to N do
3: if 1.%% Tel == 0 then
4: use the segmentation model to generate o seg from
5: generate high-level action a: = (or, seg
6: select of according at
7:
initialize the tracking model with of and
8: if < Ti then
9:
gain VLM's supervision information y: add (or, of y.) to Dst.
11:
end if
add (for-Ta: Tr. }. for, oz 808
TTaa: E(= rt) to DRI.
Taa
13: else
14:
use the tracking model to recognize of from Or
15:
end if
input (o oi-1; o(d) into प obtain d;
17:
take action a, in the environment to receive the next observation 01411 and reward sample a batch BsL from Ds and calculate Ls according to Eq. B
18:
19:
sample batch BEL from DRL and calculate LRL according to Eq. MI
20:
calculate according to Eq. @ and update n
21:
update Y/st. and TRR
22:
end for
Segment Anything Model (SAM 2) serves as the default segmentation model and tracking model in FTR. The official implementation of SAM only supports offline video object tracking, not live streaming video. Therefore, we utilize the implementation from armment-ayythnngg 2-ral-----r which makes SAM 2 possible for real-time video applica- tions. To improve training speed and memory efficiency, we align with FTD's preprocessing [I[C]] to heuristically filter the segmentation model's outputs into k instances.
http:///iitauuu..o/////ae/// com/Gy920/
The default VLM used in FTR is Qwen-VI--Max VL-Max [33]. For more details, please refer to the "VLM Details" section.
15

---

A.2 Baseline implementation
SimGRL and PAD are implemented following the apen-source code, Table Ø includes the links of them. leemmnni is implemented according to the original paper, and is included in our code.
Table 2: Links the open-source code of baseline methods.
Method
Open-Source URL
SimGRL
PAD
A.3
Hyperparameters
Table E shows the hyperparameters for reproducing the experiments.
Table 3: Hyperparameters.
Hyperparameters of environments
frame size
168 x 168 (franka-pusi) franka-door), 84 x 84 (otherwise)
frame stack
3
episode length
200 (franka-push, franka-door), 1000 (otherwise)
action repeat
2 (finger-spin, pendulum-swingup). (otherwise)
Hyperparameters of DrQ-v2
train steps replay buffer size
5 x 10a
1 x 105
exploration steps
1 x 104
n-step returns
3
batch size
256
optimizer
Adam
actor & critic learning rate
1 x 10
discount factor
0.99
critic Q-function soft-update rate
0.01
exploration stddev. clip
0.3
exploration stddev. schedule
(incar[1.0001,,10000
Hyperparameters of focus stage
SAM 2 checkpoint
sam2_hiera_ hiera_tiny
adapt steps
2 x 105
number of segments k
9
selection interval Tee SL-to-RL transition timestep Ti
20
5000
transition end timestep T2
10000
policy stddev. da optimizer batch size learning rate clip ratio of PPO discount factor
0.1
Adam
128
3 x 10
0.2
0.5
GAE lambda
0.95
EsL objective margin 5
0.1
A.4 Network architecture
Below are the network architectures of the main components of FTR. Here, MLP(n) denotes fully-connected layer with output size of n; LayerNorm() () denotes applying layer normalization; Conv2D(c, k, s, p) denotes and padding p; Maxpoo12D (k, s) denotes 2D max-pooling layer of kernel size k and stride 8; Flatten() denotes flatten layer; ReLU() denotes rectified linear unit; Tanh () denotes hyperbolic tangent function; (...} x l denotes repeating the layers within brace for & times.
convolution layer of output channel c, kernel size k stride
16

---

A.4.1 Network architecture of the original policy
The original policy л is trained in the clean environment without visual perturbation using the DrQ-v2 algorithm. The actor and critic share the same image encoder.
Encoder:
Conv2D (32, 3, 2, 0) => R)LU() => {Conv2D(32, 3, 1, 0) => ReLU())x3 => Flatten
Actor:
MLP (50) => LayerNorm() => Tanh () => (MLP ReLU())x2 =>
MLP (action_dim)
Critic:
MLP (50)
LayerNorm () => Tanh () (MLP(1024) ReLU()}22 => MLP(1)
A.4.2 Network architecture of the segment selector
The segment selector (high-level policy T²) is trained with the PPO algorithm. The actor x² and value function V share the same embedding module Ø5.
Embedding module Φ:
Conv2D 3,2,,,2,1 => {ReLU() => Conv2D (32,3,1,, => (axpoo12D(2,2)3x< => Flatten => MLP (128)
Actor ==
Th adopts an attention-like mechanism to capture the relationship between o we and oz. First, the inputs are transformed into latent representation Φ(or) E k and @(0, E R through an embedding module o, where D the dimension of the latent space. Next, Φ(o,) undergoes linear projection generate query vector qu RIXD, while ((o, are mapped to key vectors k, E Rx Scaled dot-produce scores are computed between q, and kj
q.k. DD
score;
ERIXK
(5)
Linear projection Wht and W 1, 4): MLP (128)
The can be interpreted the relevance of segments the task, where a higher value indicates stronger task relevance of the corresponding segments. Then, a sigmoid function is used generate the probability sampling each segment:
μr = (4t, 22 ... px) = Sigmoid(score:) E (0, 1)
(6)
Value V/:
V adopts an attention mechanism. Given latent representation Φ(o:) and Φ(o, through the embed- ding module é, Φ(o:) undergoes linear projection to generate query vector, while o(o² are mapped key vectors and value vectors.
seg)
Linear projection W and WT, (i = 1,2,3, 3,44) 4), W": MLP (128)
The outputs of the attention mechanism are fed into value head to get the value of current state.
Value head: MLP(1)
A.5
VLM details
We use Qwen- VL-Max [6>3]] by default. Given segments of and prompt, the VLM returns y: (yt ................................ ot yf E (0, indicating whether each segment should be focused on. The prompt seg template is shown in Fig. R including an example image from source domain, segments o and output format. The example images for each task used in the prompt are shown in Fig. Z. The total cost of API calls is below $100 for our experiment. The average response time for API calls is
17

---

approximately seconds. The number of VLM calls for a single run in our experiment is Ti/Tet 5000/20 250.
Task: Determine the object each of the candidate images 1-9 part the "erticulated" object(s) the image. Note that the articulated object may *different poses* or joint configurations across images. Output the results JSDN format.
Please return the results in ISON format as of objects. "Thhe order objects the arrey must order candidate Ifrom to 9), Each object should contain the following fields:
correspond
provided
"Imago_ id": Candidate image number (1-9)
Image: [Image Placeholder]
"r__sama_objcct": false if different)
"Is_same_ object": Boolean value
object,
Candidate Image 1: [Image Placeholder]
Candidate Image [Image Placeholder)
Candidate Image 3: [Image Placeholder]
ISON Output Template: "[son[
Candidate image [Image
Candidate image 5: [Image Placeholder]
{ ('Image_ide [i "1", "ls_smee_eooeeee "i__same_objct".. boolean
Candidate Image 6: [Image Placeholder]
("Image_iee '2", "Is sasmemmmjcct"" boolean
Candidate image 7: [Image Placeholder]
{ ('Image_id id": "3",
("Imago_id" "is_same__oa oaje__tjjctt" boolean
Candidate Image [Image Placeholder)
"k_same__de object": boolean )
Candidate image 9: [Image Placeholder]
{ ('image_dd": "5", "k_same_object" boolean
{ 'mmage "immag_id":: "6", "B_same, object". boolean
''imago _dd "7", "is st__sam__oo "o__same_ojjet".. boolean
{ ''mage id": "8", "Bs_sam,_ object": boolean
{ ''mage id": "9",
object": boolean lI"
Figure VLM prompt template.
T
(a) pendulum-sswngup
(b) cartpole-swingup
(c) finger-spin
(d) hopper-stand
(c) hopper-hop
(f) cheetah-run
(g) walker-walk
(h) walker-run
(i) franka-reach
(j) franka-push
(k) franka-door
Figure 7: Source domain of the DeepMind Control Suite (a-h) and Franka Emika Robotics (i-k) lasks.
18

---

B Additional Experimental Results
B.1 Further ablation studies
conduct ablation studies on different selection intervals Tel SL-to-RL transition timesteps Ti. and supervised learning objectives Lst. on the finger-spin task. Results are shown in Table EI, Table S Table 6, and Fig. E.
Table Ablation on Tat on the finger-spin task.
Tee
Performance (mean = std)
249.6 244.0
10
915.7 + 55.9
20
903.9 + 71.2
40
928.9 47.8
Table 5: Ablation on Ti on the finger-spin task.
Ti
Performance (mean + std)
1000
922.5 + 44.4
5000
903.9 = 71.2
10000
802.8 + 229.5
Table 6: Ablation on Lst. on the finger-spin task.
Lst.
Performance (mean + std)
The proposed loss in Eq. Binary Cross Entropy (BCE) loss
903.9 + 71.2 510.4 + 337.8
B.2 Performance of the baselines in the clean environments
Table Z shows the results of the baselines in the clean environment. The experiments conducted with three random sceds. PAD and SimGRL perform well on most tasks, while Q2-Icarning performs decently except in franka-push and franka-door.
Table 7: Performance of the baselines in the clean environments (mean + std).
Task
PAD
Q²-learning
SimGRL
pendulum--swiggup
665.0 i 437.7
912.0 1 32.0
910.0 1 6.0
cartpole-swingup
859.6 + 14.5
874.2 +
0.7
862.8 + 7.2
finger-spin
908.5 + 18.5
653.2
262.9
979.5 + 6.4
hopper-stand
814.7 + 25.6
845.0
18.3
866.2 5.8
hopper-hop
114.6 + 5.3
170.7
14.0
150.9 5,2
cheetah-run
343.2 + 36.8
385.5
13.3
318.8 4.6
walker-walk
915.4
20.4
616.9
43.9
879.1
walker-run
318.3
7.3
251.2
37.3
41.3
franka-reach
942.7
3.3
884.2
44.6
032.7
franka-push
102.8
2.1
54.8
5.6
91.2
franka-door
152.4
4.3
20.4
0.3
163.3 + 0.8
B.3 VLM accuracy
We calculate the accuracy of Owen-VL--Me across different tasks in Table N. The VLM's judg- ment is considered accurate when selects task-related segments and refrains from selecting task- irrelevant ones.
19

---

Table 8: Accuracy of Qwen- VL-Max.
Task
Accuracy (%)
pendulum-swingup
97.84
cartpole-swingup
88.67
finger-spin
86.40
hopper-stand
94.23
hopper-hop
94.33
cheetah-run
96.32
walker-walk
89.77
walker-run
89.11
franka-reach
89.04
franka-push
85.52
franka-door 91.41
B.4 Relationship between segment selection accuracy and reward
We conduct supplementary experiments the franka-reach task validate the relationship between segment selection accuracy and cumulative reward in an episode over five random seeds in Table B. The results in the table demonstrate clear correlation: When the segment selector accurately fo- cuses on task-relevant objects, achieves the highest rewards. Conversely, incorrect selection leads substantially lower rewards.
Table 9: Relationship between segment selection accuracy and performance (mean + std).
segment selection
Performance (mean + std)
Only task-relevant objects
892.4 22.9
Task-relevant objects irrelevant object
570.9 + 223.1
Task-relevant objects 2 irrelevant objects
495.3 + 259.3
All objects
234.4 + 282.9
All task-irrelevant objects
9.6 + 17.2
B.5 Experiments on more distractions
B.5.1 Task-similar objects
To verify FTR's ability to handle task-similar objects in the target domain that could easily be misidentified, we introduce two scenarios: another cube in the background and another uncontrolled robotic arm in the background. We conduct experiments on the franka-reach task over three ran- dom seeds. The results, shown Table II indicate that even with such objects in the background, FTR can still filter out task-relevant objects using environmental reward, thereby maintaining high performance.
Table 10: Performance of FTR on the franka-reach task in face of distracting objects (mean + std). Another cube Another uncontrolled robotic arm
913.0 + 18.5
871.8 + 38.6
B.5.2 Illumination variation
We evaluate FTR's 's illumination generalization the franka-reach task over three random seeds. To enhance the adapted policy's performance, as discussed the conclusion of our paper, we introduce illumination perturbation during low-level policy x training enhance robustness. The results in Table I
Row shows performance without illumination- roobust training, while Row 2 demonstrates FTR's s adaptation under both illumination and background perturbations, maintaining the performance of m despite illumination sensitivity. We introduce varying directions of illumination perturbations
20

---

during the training of тi in the source domain to enhance its robustness. As presented in Row 3, it's performance improved significantly. Row 4 shows FTR's adaptation performance under both illumination and background perturbations, demonstrates that FTR consistently maintained the performance Ti. Furthermore, benefiting from the enhanced robustness of my, the performance showed notable improvement compared to Row 2.
Table 11: Performance of FTR in face of illumination variation (mean + std).
With background perturbation? illumination variation?
Ty robust to
With FTR adaptation?
Performance (mean + std)
no no yes yes
no
659.3 = 52.5
yes
yes
599.1 = 62.1
no
952.3 4.1
yes
yes
929.5 = 11.6
B.6 Fine-tuning after adaptation
When my lacks robustness or is deployed in an unforeseen perturbed environment, we show fine- tuning कy after adaptation (while fixing the selector) can further improve performance.
To demonstrate this, we conduct experiments on the franka-reach task over three random seeds and introduce two types of target domain perturbations: a 15 rotation of the camera around the z-axis and 15e horizontal inclination of the robot arm's base. After training the selector for 200k steps, we fix its parameters and unfreeze 's' parameters to fine-tune for 50k steps using environmental reward. The results are in Table MM The adapted FTR performance initially degrades. Nevertheless, after the 50k-step fine-tuning, the performance recovers to a commendable level.
Table 12:
Performance after adaptation and fine-tuning и (mean + std). After FTR adaptation
After fine-tuing m
Camera rotation
733.3 + 15.7 716.2 = 51.0
904.6 8.7
Base inclination
906.0 + 13.7
We posit that the reason for the effectiveness of the "adapt fine-tune" process and the performance maintenance with non-robust mi is the same: The selector's RL training does not impose stringent requirements on the optimality of my The selector can be guided towards correct learning as long as Ji satisfies a key condition: the reward for correctly chosen action by the selector is greater than that for an incorrectly chosen one. As shown in the experiment of Appendix B4,, this condition can be easily satisfied. Once the selector learns the correct selection patterns, fine-tuning क. based on the filtered images becomes highly efficient.
C Visualization
and the selection result of seg across all tasks (Figs. N to We visualize o,, o. o seg
We show images from t = 0 to f = 36, sampled every 4 time steps. Recall that we set selection interval T = 20. When = 0 and 20, the selection pathway called: otherwise, the tracking pathway is used. In the selection pathway, we displays the segments o, E {0, 20). and their selection probabilities. Here, a segment is selected if its probability
seg
0.5. The selected segments are then integrated to form ozel / {0, 20), representing the union of focused objects. Whenever obtain the selected objects of / E (0, 20) in the selection pathway, we simultaneously record them in the tracking model. In the tracking pathway, the tracking model recognizes the selected objects of using observation oz.
21

---

4
8
12
16
20
28
32
36
0.
0:07
0.24
0.28
0.21
0.91
Se I ect on Pathway
Se lectii on Pathway
0.27
0.32
011 7
seg
0.7
0.35
0.16
sek
00
20
0.92
0.07
0.31
0.4.
0.28
0.37
T
T
T
%
mll
Tracking ing Pathway
Track ing Pathway
Figure 8: Visualization pendulum-swingup.
8
12
16
20
24
28
32
36
O
0.33
0.08
0.0
0.28
0.24
0.31
Se l ect ion Pathway
Select in Pathway
0.9/
0.33
seg %
0.22
0.07
0.22
seg 0.0
0.23
0.31
0.0
100
0.2c
0.0
1
Rel
Track ing Pathway
Track ing Pathway
Figure 9; Visualization of cartpole-swingu
4
12
16
20
24
28
32
36
00.7
0.96
0.99
0.95
0.05
0.27
Se ect ion Pathway
Se lectii i on Pathway
9.06
0.27
0.13
seg
0.05
0.22
0.31
seg O2n
%
0.2-
0.26
0
0.27
0.98
0.12
sel
8
Track ing Pathway
Track ing Pathway
Figure 10: Visualization of finger-spin.
22

---

4
8
12
16
20
28
32
36
0.19
0.13
0.88
0.17
0.29
0.22
Se ect on Pathway
Se lectiiin on Pathway
0.18
0.0
0.24
00
seg
0.24
0.86
0:31
20
sek
0.1
0.27
0.18
0.0
00.31
0.29
%
mll
Tracking ing Pathway
Track ing Pathway
Figure 11: Visualization hopper-stand.
8
12
16
20
24
28
32
36
O
0.02
0.17
0.16
0.02
0.23
0.22
Se colleciion ion Pathway
Se lectiiinn i on Pathway
0.0
0,19
%
seg
0.$1
0.09
seg 20
100
L..0
0.45
0.69
0.12
0.1
0.14
6
Track ing Pathway
Track ing Pathway
Figure 12: Visualization of hopper-hop.
4
8
12
16
20
24
28
32
36
O
0.2
0.29
0.3
0.18
b.27
0.34
Se ect ion Pathway
Se lecti i on Pathway
0.38
0.24
0.33
86g
0.38
0:78
0.34
seg O2
%
0.78
0.31
0.0
0.33
0.34
0.31
sel
K
I
~
T
Tracking Pathway
Track ing Pathway
Figure 13: Visualization of cheetah-run.
23

---

4
8
12
16
20
28
32
36
0:08
0.14
0.17
8,04
0.17
0.24
Se ect on Pathway
Se lelticinn on Pathway
0.17 0.07
0.0
00
SEg
0.07
0.13
0.72
20
sek
0.06
0.95
0.0
0.14
0,93
0.54
/
ト
Tracking ing Pathway
Track ing Pathway
Figure 14: Visualization of walker-walk.
4
8
12
16
20
24
28
32
36
Ot
0.08
0.09
.93
0.08
0.11
0.13
Se pelleciion ion Pathway
Selection inntiion Pathway
0.03
0.08
%
seg
.02
0.16
0.19
seg 2.0
.002
0.04
0.13
001
0.79
0.0
Track ing Pathway
Track ing Pathway
Figure 15: Visualization of walker-run.
4
8
12
16
20
24
28
32
36
Q,01
1.0
0.27
0.01
1.0
0.14
Se ect ion Pathway
Se Selection i on Pathway
0.02
0.89
0.25
seg
0.0
1.0
0.23
seg
%
O
0.29
0.17
0.23
0.41
0.19
0.0
Track ing Pathway
Track ing Pathway
Figure 16: Visualization of franka-reach.

---

4
8
12
16
20
21
28
32
36
Ot
0.0
0.0
0.52
0.0
0.03
0.01
Se l ect in Pathway
Se l ect i on Pathway
1.0
0.08
%
f
1.0
R0.0
0.20
seg
0.0
1,0
0.06
0.0
00.0
0106
Tracking ing Pathway
Track ing Pathway
Figure 17: Visualization of franka-push.
1
8
12
16
20
24
28
32
36
Ot
0.88
0.19
0.03
0.
0.12
0.08
Se lot in Pa Pathway
Se I ect i on Pathway
0.
0.16
seg %
1
0.04
0.0
seg 220
0.2
0.01
0.05
0.07
0.52
0.01
Gel
Track ing Pathway
Track ing Pathway
Figure 18: Visualization of franka-door.
25

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction include the claims made in the paper, which accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the possible limitations in Section F
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We present the implementation details in Section M and Appendix Al.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
The source code available at https://github. com/ htw../oo.//.AAAA. LAMDA RL/FIR.
LAMDA-RL/FIRR
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guides//Cotee www.ooddewumiisisoooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /mips.ccc/pblli/uuiddde Seweiideeee/oSSSmmmiiiie http://uddeeeeoeeeeeeaaaaaaaaaaa
www.iiooiiiiyyyyoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specifyy all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present the experimental setting in Section B and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The results and accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
28

---

is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes] We provide sufficient information the computer resources in Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics litpss:
Answer:
[Yes]
Justification: I have read and followed the ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Creators or original owners of assets used in the paper, are properly credited in Appendix AT
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, Saperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New assets
30

---

Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
31

---

Answer: [Yes]
Justification: The paper use the Vision-Language Model for the component of the method. Details are provided in Section ! and Appendix A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to our LLM policy for what should or should not be described.
32


Reviews:
[{"review_id": "gv428GDnQ5", "reviewer": "Reviewer_HyXv", "summary": "The authors propose a VLM/segmentation-model assisted background segmentation scheme to help combat visual distractions in reinforcement learning. First, an optimal control policy is trained in the clean simulation environment. Then, an adaptation phase is conducted in the target environment. SAM-2 is used to propose candidate image segments to be provided to the frozen simulation control policy. To choose the image segments, a selection policy is trained in the target environments via downstream target environment RL rewards (collected by the control policy). To jumpstart training, the selection policy is additionally trained with supervised learning to match VLM suggestions in early stages of training. Segment selection is only performed occasionally, and currently selected segments are tracked between frames using the pretrained segmentation model. This method outperforms a variety of adaptation, generalization, and target-environment training approaches.", "questions": "It would be helpful to add a sanity check for all baselines (not just DrQ-v2) on the clean environment to show that they all can correctly learn an optimal policy and that any failures are due to the distractions. Could this be added?", "limitations": "The authors state that other methods, when facing unseen disturbances, may drop performance significantly.\nThey should add more clear acknowledgements that their method still relies on the simulation policy generalizing to the target environment appearance for the segmented foreground objects. Likewise, this method can also drop in performance if the in-focus objects have a different appearance (lighting, texture, occlusion, etc.) compared to the simulation.\n\nAnother downside is that they require deployment-time segmentation/tracking, which can limit control frequency. Other baselines as well as [1] do not require this, and this should be mentioned.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is well-written, proposing a straightforward and effective idea that other practitioners should be able to readily reproduce. \nMy only notable complaint is that the novelty is somewhat incremental due to VLMs having been used for background segmentation before, for example in [1], however the introduction of the segment-selection policy trained in the target-domain is new and significant, so I'm leaning towards acceptance.\n\nExperiments and ablations seem sufficient, except for one concern listed in *Questions*.\n\nThis is a missing reference that also uses VLMs to suggest segmentation masks while training with domain-randomized video backgrounds:\n\n[1] Kim et al. Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions. 2024.", "was_revised": true, "final_justification": "All of my concerns originally mentioned in this review have been adequately addressed. I recommend this paper for acceptance.", "rebuttal": "We sincerely appreciate your insightful feedback, which has greatly improved our work. We hope the following clarifications can further address your concerns.\n\n## Q1: Sanity check for all baselines\n\nThe table below shows the results of the baselines on the clean environment. Due to time constraints, the experiments were performed with a single random seed. PAD and SimGRL perform well on almost all the tasks, while Q$^2$-learning performs decently except fp and fd. We will include the sanity check in the appendix.\n| Task | PAD | Q$^2$-learning | SimGRL |\n| --- | --- | --- | --- |\n| ps | 964 | 944.0 | 904 |\n| cs | 845 | 874.9 | 873 |\n| fs | 927 | 390.3 | 986 |\n| hs | 789.1 | 863.2 | 872.0 |\n| hh | 116.2 | 184.7 | 145.7 |\n| cr | 380 | 372.2 | 322 |\n| ww | 895 | 660.7 | 923 |\n| wr | 311.0 | 288.5 | 338.1 |\n| fr | 939.4 | 928.8 | 955.1 |\n| fp | 100.7 | 60.4 | 84.1 |\n| fd | 148.0 | 20.1 | 162.4 |\n\n## Q2: The novelty is incremental for background segmentation[1]\n\nWe agree that segmentation model have indeed been explored for background segmentation in prior works, including [1]. The method in [1] primarily focuses on using segmentation masks in the auxiliary task to improve representation learning in distracting environments. Compared with the method proposed in [1], our **fusion of VLM supervision and RL finetuning in the target domain for the segment selector is a novel approach**.\n\nWe would like to summarize the contributions of our paper:\n- We propose a novel policy adaptation framework employing an object selection mechanism to focus on task-relevant objects and then directly reusing a simulation-trained policy on them.\n- The focus stage and the reuse stage are highly decoupled, meaning that FTR is compatible with a range of generalization RL algorithms and able to handle complex distribution shift.\n- Our method integrates supervised learning and RL to synthesize VLM’s prior knowledge and environmental feedback, enabling efficient and effective adaptation.\n\nWe are grateful to you for your careful reading of our paper and for pointing out the missing reference on VLM and segmentation masks. We will include a dedicated discussion of this aspect in the Related Work section of our future version of the paper.\n\n## Q3: Needs clearer acknowledgment that the method relies on simulation policy generalizing to target environment foreground appearance. Performance may drop if in-focus objects' appearance (lighting, texture, occlusion, etc.) differs from simulation.\n\nWe appreciate the reviewer for pointing out this limitation. We acknowledge that when the appearance of in-focus objects in the target domain differs from that in the source domain, the adaptation performance of FTR relies on the generalization capability of the low-level policy $\\pi_l$. To demonstrate the feasibility of our approach under such conditions, we conducted experiments involving illumination perturbations. Due to time constraints, the experiments were performed with a single random seed.\n\nThe table below presents our experimental results. We choose franka-reach as the environment. The performance of DrQ-v2 in a clean environment reported in our paper is 948.3, which represents the potential upper bound for FTR's performance.\n\nAs shown in **row 1**, we initially evaluated the performance when $\\pi_l$ was trained without illumination perturbations but deployed with their presence. By comparing this with **row 2**, which presents the adaptation performance of FTR in the target domain with both illumination and background perturbations, it is evident that FTR can effectively maintain the performance of $\\pi_l$ under background disturbances, even when $\\pi_l$ itself is not robust to illumination changes.\n\nSubsequently, we introduced varying directions of illumination perturbations during the training of $\\pi_l$ in the source domain to enhance its robustness. As presented in **row 3**, $\\pi_l$'s performance in environments with illumination perturbations significantly improved. **Row 4**, detailing FTR's adaptation performance in the target domain under both illumination and background perturbations, demonstrates that **FTR consistently maintained the performance of the low-level policy $\\pi_l$**. Furthermore, benefiting from the enhanced robustness of $\\pi_l$, the performance showed a notable improvement compared to the second row, closely approaching DrQ-v2's performance of 948.3 in a clean environment.\n\n| With background perturbation? | Low-level policy robust to illumination variation? | With FTR adaptation? | Performance |\n| --- | --- | --- | --- |\n| no | no | no | 611.5 |\n| yes | no | yes | 604.8 |\n| no | yes | no | 956.0 |\n| yes | yes | yes | 925.1 |\n\n## Q4: Require deployment-time segmentation/tracking\n\nWe acknowledge the reviewer's concern regarding the deployment-time overhead introduced by segmentation/tracking. While this indeed incurs an additional ~50% computational cost during evaluation as we measured, we argue that the trade-off is justified due to significant improvements. Specifically, our approach demonstrates superior performance compared to the baseline, as shown in Section 5.2. If long-term policy deployment in noisy environments is required, one way to reduce computational overhead is to collect interaction data and distill a policy that relies solely on noisy inputs. We acknowledge this could have been more clearly stated as part of our system's assumptions in the Limitations part.\n\n## References\n\n[1] Kim et al. Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractions. arXiv. 2024."}, {"review_id": "zs1nS6uZOM", "reviewer": "Reviewer_8K8e", "summary": "The paper introduces a framework called Focus-Then-Reuse (FTR) to address the challenge of deploying reinforcement learning policies trained in simulation to real-world environments with visual perturbations. FTR has two stages: (1) focus stage: dynamically selects task-relevant objects using a trainable segment selector (trained via RL + VLM supervision), (2) reuse stage: executes actions with a fixed, pre-trained policy on the filtered observations. FTR outperforms baselines like SimGRL (generalisation) and PAD (adaptation), especially in complex visual perturbations.", "questions": "1. How does FTR handle cases where SAM2 or the tracker fails to segment/task-relevant objects? Since the low-level policy depends on the selected segments, what is the impact on performance in real-world settings where such failures are common?\n\n2. Are the initial segmentation masks (or tracking updates) generated by SAM2 without prompts? Could the target object (e.g., a robot arm or manipuland) be oversegmented or contaminated with background pixels? If so, how does this affect policy performance?\n\n3. Instead of feeding raw pixels to the low-level policy, have you explored using SAM2’s object features (e.g., mask embeddings) as inputs? Would this introduce bottlenecks (e.g., latency, feature misalignment) compared to the current pipeline?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. Integration of foundation models into RL:\n    - The paper innovatively leverages large-scale pretrained models (Qwen-VL and SAM 2) to address visual RL challenges, bridging the gap between high-level perception and low-level control. This is a promising direction, as foundation models offer rich prior knowledge but remain underutilised in RL for dynamic decision-making.\n    - The implementation (released in the supp.) is practically valuable, providing a valuable pipeline for combining VLMs with RL. This could inspire follow-up work on foundation-model-augmented RL.\n\n2. The fusion of VLM supervision and RL fine-tuning for the segment selector is a novel approach. The VLM provides an initial object-selection prior, while RL adapts it to task-specific rewards. The dynamic weighting mechanism (transitioning from SL to RL) is well-justified and could generalise to other hierarchical RL settings.\n\n**Weakness:**\n\n1. Sim-to-real gaps in the high-level policy:\n    - While the focus stage mitigates visual noise, the high-level policy (segment selector) is still trained purely in simulation. Real-world perceptual challenges (e.g., lighting changes, motion blur) may degrade its performance, necessitating additional adaptation.\n    - The paper does not test FTR on real-world deployment or quantify how the segment selector’s sim-trained priors (from Qwen-VL/SAM) transfer to real images. This limits claims about real-world readiness.\n\n2. Dependence on a pretrained low-level policy:\n    - FTR assumes access to a low-level policy trained in a clean simulation, which may not exist for many real-world systems. Even with a \"perfect\" simulation twin, the filtered observations may still differ from the simulation’s training distribution.\n    - The low-level policy is frozen during adaptation; FTR cannot correct for dynamics mismatches (e.g., sim-to-real mechanical discrepancies).", "was_revised": true, "final_justification": "See the rebuttal comment.", "rebuttal": "Thank you for your constructive comments and suggestions. Below are further clarifications for your concerns.\n## Q1: Sim-to-real gaps in the high-level policy\n- **Q1.1: Simulation-trained high-level policy may degrade in real-world perception.**\n\nWe would like to clarify that our experiment follows the study protocol of DMControl Generalization Benchmark [1], PAD [2], and SimGRL [3]. To address your concern, we conduct further experiments in a target domain with more intricate, realistic conditions. We choose franka-reach as the environment. Due to time constraints, these experiments were performed with one random seed.\n\nFirst, to verify FTR's ability to handle task-similar objects in the target domain that could easily be misidentified, we introduced cubes or uncontrolled robotic arms into the background and use FTR to adapt in this environment. The results, shown in the table below, indicate that even with such objects in the background, FTR can still filter out task-relevant objects using environmental reward, thereby maintaining high performance.\n|DrQ-v2 (clean background)|FTR (background perturbation)|FTR (background + another box)|FTR (background + another robot arm)|\n|-|-|-|-|\n|948.3|860.5|892.5|883.1|\n\nGoing a step further, we evaluated FTR's illumination generalization. To enhance the adapted policy's performance, as discussed in the conclusion of our paper, we introduce illumination perturbation during low-level policy $\\pi_l$ training to enhance robustness.\n\n**Row 1** shows performance without illumination-robust training, while **row 2** demonstrates FTR's adaptation under both illumination and background perturbations, maintaining the performance of $\\pi_l$ despite illumination sensitivity.\n\nSubsequently, we introduced varying directions of illumination perturbations during the training of $\\pi_l$ in the source domain to enhance its robustness. As presented in **row 3**, $\\pi_l$'s performance improved significantly. **Row 4** shows FTR's adaptation performance under both illumination and background perturbations, demonstrates that **FTR consistently maintained the performance of the low-level policy $\\pi_l$**. Furthermore, benefiting from the enhanced robustness of $\\pi_l$, the performance showed a notable improvement compared to row 2, closely approaching DrQ-v2's performance of 948.3 (as a potential performance upper bound for FTR performance) in the source domain.\n|With background perturbation?|$\\pi_l$ robust to illumination variation?|With FTR adaptation?|Performance|\n|-|-|-|-|\n|no|no|no|611.5|\n|yes|no|yes|604.8|\n|no|yes|no|956.0|\n|yes|yes|yes|925.1|\n\n- **Q1.2: The paper does not test FTR on real-world deployment or quantify how the segment selector’s priors from VLM/SAM transfer to real images**\n\nDue to experimental constraints, testing FTR in real-world scenarios is beyond our current scope. As an alternative validation, we perform the following evaluations on six real robotic arm videos from Open-X-Embodiment [4] to assess:\n- **SAM2's segmentation robustness:** SAM2 successfully segments the robotic arm in almost all cases, demonstrating strong real-world applicability. Minor limitations are observed in gripper detection with a failure rate of 4.3%, which might sightly affect the performance of FTR in real-world.\n- **VLM’s recognition reliability with segments:** The VLM demonstrates reliable identification of robotic arms in real-world scenarios, achieving an accuracy of 90.9%. This performance aligns with our experimental result in appendix B.2, confirming the VLM's capability in real-world scenarios.\n\nThese demonstrate promising real-world transfer potential, though real-world deployment testing would further strengthen our claims.\n## Q2: Dependence on a pretrained low-level policy $\\pi_l$\n- **Q2.1: FTR assumes $\\pi_l$ trained in clean simulation, which may not exist for real-world systems.**\n\nHere are our justifications for this approach:\n- Many practical applications (e.g., robotics [5]) involve policies trained in structured/clean simulation settings that must later operate in noisy, real-world environments. Our setup mirrors this paradigm.\n- Many complex tasks, like robotic arm manipulation, are impractical or impossible to train directly in complex real-world environments as they require extensive interaction. While our method involves a two-stage training process, it offers much high sample efficiency. A $\\pi_l$ trained in a clean background can rapidly adapt to diverse real-world target environments. For instance, in the franka-reach task, $\\pi_l$ requires 200k interactions with the simulation environment. Building on this, the high-level selector then only needs 50k real-world interactions to complete its training.\n\nWhile our current experiments focus on adaptation from clean to perturbed domains, as stated in the conclusion of the paper, the framework could be extended to other scenarios (e.g., training with synthetic perturbations). To demonstrate the feasibility of our approach under such conditions, we conducted experiments involving illumination perturbations, as shown in Q1.\n\n- **Q2.2: Frozen $\\pi_l$ during adaptation limits dynamics mismatch correction**\n\nWe acknowledge that while Q1's source-domain perturbation enhance $\\pi_l$'s robustness, performance may degrade if perturbations don't fully cover target domain variations. However, we show **finetuning $\\pi_l$ after adapting** (while fixing the selector) can further improve performance.\n\nTo demonstrate this, we selected two types of target domain perturbations:\n1. **Camera rotation**: 15° around the z-axis.\n2. **Base inclination angle**: The robot arm's base is tilted 15° horizontally.\n\n$\\pi_l$ was trained in an unperturbed source domain, rendering it non-robust to these specific target domain perturbations.\n\nAfter **training the selector for 200k steps**, we **fix its parameters and unfreeze $\\pi_l$'s parameters** to **finetune $\\pi_l$ for 50k steps** using environmental reward. Due to time constraints, these experiments were performed with one random seed. As the table below shows, the adapted FTR performance initially exhibits a gap compared to the potential upper bound of 948.3. Nevertheless, after the 50k-step finetuning, the performance recovers to a commendable level.\n\nWe posit that this **\"adapt + finetune\" process is effective because the selector's RL training does not impose stringent requirements on the optimality of $\\pi_l$**. The selector can be guided towards correct learning as long as $\\pi_l$ satisfies a key condition: the reward for a correctly chosen action by the selector is greater than that for an incorrectly chosen one. This condition is typically met even in the presence of a domain gap. Once the selector learns the correct selection patterns, finetuning $\\pi_l$ based on the filtered images becomes highly efficient.\n||After FTR adaptation|After finetuing $\\pi_l$|\n|-|-|-|\n|Camera angle|755.1|903.3|\n|Base inclination angle|776.0|896.5|\n\n## Q3: How does FTR handle SAM2's failures? What's the performance impact in real-world?\nWe appreciate this important question regarding failure cases in our pipeline. In our experiments, SAM2 demonstrated near-perfect segmentation accuracy on the benchmark, which is why we didn't emphasize this in the paper. Also, the evaluations on real-world videos from the Open-X-Embodiment dataset stated in Q1 shows that SAM2 remains robust in real-world scenarios.\n\nWe acknowledge that the performance of FTR depends on SAM2, as the pipeline depends on proper object segmentation in the focus stage. Should SAM2 fail to produce valid segmentation masks (a rare occurrence in our experiments but theoretically possible), FTR would consequently be unable to function as intended. We acknowledge this could have been more clearly stated as part of our system's assumptions in the Limitations part.\n\n## Q4: Prompt requirements in SAM2.\nIn our paper, SAM2 generates segmentation masks and updates tracking without prompt.\n\n## Q5: Could the target object be oversegmented or contaminated with background pixels? If so, how does this affect policy performance?\nOur method filters the masks using the method in [6]: discard smaller segments if they overlap more than 95% with a larger segment, and select up to nine segments in descending order of size. In this case, the target objects are barely oversegmented. As shown in the visualization of franka-reach in supplementary, the target object do sometimes be contaminated with background pixels. The experimental results on franka-reach in Section 5.2 indicate that this hardly affects performance.\n\n## Q6: What about using SAM2’s object features as inputs? Would this introduce bottlenecks (e.g., latency, feature misalignment) compared to the current pipeline?\nFor $\\pi_l$, the task of decision making based on filtered images isn't particularly difficult. Given that DrQ-v2 already achieves good performance, incorporating SAM2's object features as input isn't necessary. \n\nFurthermore, because SAM2's training objective is object segmentation and tracking, it doesn't focus on other information like color or relative position. Therefore, using SAM2's object features as input might cause $\\pi_l$ to lose crucial information, leading to a performance drop. \n## References\n[1] Hansen et al. Generalization in Reinforcement Learning by Soft Data Augmentation. ICRA. 2021.\n\n[2] Hansen et al. Self-Supervised Policy Adaptation during Deployment. ICLR. 2021.\n\n[3] Song et al. A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations. NeurIPS. 2024.\n\n[4] O’Neill et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models: Open X-Embodiment Collaboration. ICRA. 2024.\n\n[5] Choi et al. On the Use of Simulation in Robotics: Opportunities, Challenges, and Suggestions for Moving Forward. Proceedings of the National Academy of Sciences. 2021.\n\n[6] Chen et al. Focus-Then-Decide: Segmentation-Assisted Reinforcement Learning. AAAI. 2024."}, {"review_id": "iqa0U02ZmD", "reviewer": "Reviewer_WqUh", "summary": "In this work the authors propose Focus-Then-Reuse (FTR) a framework for visual reinforcement learning under changes of background information. In particular, FTR is composed of two phases: (i) the focus stage, responsible for segmenting and tracking relevant information for the RL task (e.g., the agent, robot and objects on interest in the workspace) through pre-trained segmentation models and a trainable high-level segment selection policy; (ii) the reuse stage, where a standard RL pipeline is employed. The authors evaluate FTR in Mujoco environments and in three robotic tasks in simulation. The results show that their approach outperforms the baselines in robustness to visual disturbances in the background.", "questions": "I am willing to revise my score if the authors can address my concerns (see \"Strengths and Weaknesses/Comments\") and the following questions:\n\n1) What is the performance of your method when dealing with standard visual perturbances (e.g., color changes, different lighting conditions and camera changes) and distractor objects?\n\n2) Why did the authors not show the statistical results of the DrQ-v2 baseline? Also, in Table 1 which results are significantly better than the baselines?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- Paper is well-written and clear;\n- Experimental results show a clear improvement over the selected baselines.\n\n**Weaknesses**:\n- Limited novelty (See \"Comments\" below);\n- Some claims require support (See \"Comments\" below).\n\n**Comments**:\n- One major concern with the paper is the limited novelty of the work: using segmentation models to remove background distractors have been extensively explored in literature (e.g., [1, 2] for object-centric learning). From the discussion presented in the related work, and it is not clear where the novelty in the work lies. Moreover, the paper also introduces no novelty in terms of policy training as the authors employ standard PPO and DrQ-v2 algorithms for the focus and reuse stage policies.\n- While not a weakness per se, I do have concerns regarding the significance of the work: the authors propose a method for removing visual background distractors by segmenting objects relevant to the task. This induces the policy to overfit to these elements and their visual appearance, unable to deal with potential color changes, different lighting conditions, different positions of the camera, amongst other visual perturbations that are common in real-world scenarios. It is also unclear how the method would perform in face of distractor objects in the scene (such as other cubes and/or robot manipulators, for example). While this is not the scope of the paper, the concern remains.\n- Some claims in the paper are not supported by references in the current version of the paper: (i) in Lines 34-39, the authors claim that humans perform object-level filtering in decision making tasks, where object selection is initially driven by prior knowledge and subsequently fine-tuned by environmental feedback. It would improve the strength of the argument if the authors could support it with references. In line 225-226 the authors also claim that \"the higher the accuracy of the segment selector, the greater the cumulative reward obtained by the low-level policy, and vice-versa\". However, in the current version of the paper, the authors do not support this claim with any experimental validation.\n\n**References**:\n\n[1] - Shi, Junyao, et al. \"Composing Pre-Trained Object-Centric Representations for Robotics From\" What\" and\" Where\" Foundation Models.\" 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.\n\n[2] - Wang, Ziyu, et al. \"Generalizable visual reinforcement learning with segment anything model.\" arXiv preprint arXiv:2312.17116 (2023).", "was_revised": true, "final_justification": "The authors introduce a novel method to allow RL agents to execute policies under visual distractors, outperforming the previous baselines. My main concerns with the paper (limited novelty, limited applicability, lacking references to claims) were successfully addressed by the authors. As such, I have increased my score, and I believe the paper to be relevant to the RL community.", "rebuttal": "We would like to thank you for your constructive feedback. We hope the following clarifications can further address your concerns and enhance your evaluation of our paper.\n## Q1: Limited novelty\n- **Q1.1: Segmentation models have been explored in [1,2]**\n\nThank you for pointing out these relevant works; we'll add citations to them in future version. We would like to highlight a key distinction between our approach and the method in [1]. It employs a rule-based strategy to select task-relevant segments. It lacks adaptability, since the rules cannot be dynamically adjusted to new environments. Compared with the method proposed in [1], our fusion of VLM supervision and RL finetuning for the segment selector is a novel approach, addressing the limitation of rule-based methods.\n\nThe method in [2] requires post-training of SAM, which introduces additional overhead. Also, it relies on human annotation of task-relevant objects for SAM to generate segmentation masks. Our approach operates with SAM without finetuning and leverages SAM in a zero-shot manner.\n\n- **Q1.2: No novelty in policy training (PPO, DrQ-v2)**\n\nThis paper aims to address the rapid adaptation of policies in environments with complex backgrounds. Instead of directly improving classic algorithms like PPO and DrQ-v2, we adopt a plug-in approach. We innovatively propose rapidly training a high-level selector during adaptation. Our method enables fast and efficient domain adaptation without needing to retrain the low-level policy $\\pi_l$. Furthermore, it's compatible with existing visual RL algorithms, meaning any policy can serve as FTR's $\\pi_l$ and be quickly deployed to the target domain.\n\nWe would like to summarize the contributions of our paper:\n- We propose a novel policy adaptation framework employing an object selection mechanism to focus on task-relevant objects and then directly reusing a simulation-trained policy on them.\n- The focus stage and the reuse stage are highly decoupled, meaning that FTR is compatible with a range of generalization RL algorithms and able to handle complex distribution shift.\n- Our method integrates supervised learning and RL to synthesize VLM’s prior knowledge and environmental feedback, enabling efficient and effective adaptation.\n\n## Q2: Policy may overfit to focused objects and backgrounds, unable to handle real-world perturbations like color changes, lighting, or camera position changes. What's the performance in such perturbations?\nTo address this concern, we conduct additional experiments in a target domain featuring complex, real-world conditions. We choose franka-reach as the environment. Due to time constraints, the experiments were performed with one random seed.\n\nFirst, to verify FTR's ability to handle task-similar objects in the target domain that could easily be misidentified, we introduced cubes or uncontrolled robotic arms into the background and use FTR to adapt in this environment. The results, shown in the table below, indicate that even with such objects in the background, FTR can still filter out task-relevant objects using environmental reward, thereby maintaining high performance.\n|DrQ-v2 (clean background)|FTR (background perturbation)|FTR (background + another box)|FTR (background + another robot arm)|\n|-|-|-|-|\n|948.3|860.5|892.5|883.1|\n\nWhen deploying policies in other distractions like illumination, camera position, we propose two solutions: enhancing the robustness of $\\pi_l$, and finetuning $\\pi_l$ after adaptation.\n\n### Enhancing the robustness of $\\pi_l$\n\nAs discussed in the conclusion section of our paper, improving the robustness of $\\pi_l$ is a promising way to handle many other real-world disturbances. We introduce illumination perturbation during low-level policy $\\pi_l$ training to enhance robustness.\n\n**Row 1** shows performance without illumination-robust training, while **row 2** demonstrates FTR's adaptation under both illumination and background perturbations, maintaining the performance of $\\pi_l$ despite illumination sensitivity.\n\nSubsequently, we introduce varying directions of illumination perturbations during the training of $\\pi_l$ in the source domain to enhance its robustness. As presented in **row 3**, $\\pi_l$'s performance improved significantly. **Row 4** shows FTR's adaptation performance under both illumination and background perturbations, demonstrates that **FTR consistently maintained the performance of $\\pi_l$**. Furthermore, benefiting from the enhanced robustness of $\\pi_l$, the performance showed a notable improvement compared to row 2, closely approaching DrQ-v2's performance of 948.3 (as a potential performance upper bound for FTR performance) in the source domain.\n|With background perturbation?|$\\pi_l$ robust to illumination variation?|With FTR adaptation?|Performance|\n|-|-|-|-|\n|no|no|no|611.5|\n|yes|no|yes|604.8|\n|no|yes|no|956.0|\n|yes|yes|yes|925.1|\n    \n### Finetuning $\\pi_l$ after adapting\n\nWhen $\\pi_l$ lacks robustness or is deployed in an unforeseen perturbed environment, we show **finetuning $\\pi_l$ after adapting** (while fixing the selector) can further improve performance.\n    \nTo demonstrate this, we selected two types of target domain perturbations:\n1. **Camera rotation**: 15° around the z-axis.\n2. **Base inclination angle**: The robot arm's base is tilted 15° horizontally.\n\n$\\pi_l$ was trained in an unperturbed source domain, rendering it non-robust to these specific target domain perturbations.\n\nAfter **training the selector for 200k steps**, we **fix its parameters and unfreeze $\\pi_l$'s parameters** to **finetune $\\pi_l$ for 50k steps** using environmental reward. As the table below shows, the adapted FTR performance initially falls short of the potential upper bound of 948.3. Nevertheless, after the 50k-step finetuning, the performance recovers to a commendable level.\n||After FTR adaptation|After finetuing $\\pi_l$|\n|-|-|-|\n|Camera angle|755.1|903.3|\n|Base inclination angle|776.0|896.5|\n    \nWe posit that the reason for the effectiveness of the \"adapt + finetune\" process and the performance maintenance with non-robust $\\pi_l$ is same: **The selector's RL training does not impose stringent requirements on the optimality of $\\pi_l$**. The selector can be guided towards correct learning as long as $\\pi_l$ satisfies a key condition: the reward for a correctly chosen action by the selector is greater than that for an incorrectly chosen one. As shown in the experiment of Q3, this condition can be easily satisfied. Once the selector learns the correct selection patterns, finetuning $\\pi_l$ based on the filtered images becomes highly efficient.\n\n## Q3: Claims not supported by references\n\n- **Lines 34-39: \"humans perform object-level filtering in decision making tasks, where object selection is initially driven by prior knowledge and subsequently finetuned by environmental feedback\"**\n\n    We sincerely appreciate your constructive comments.We would like to cite the following literature that supports our claims above:\n    1. Reference [3] draws on research on the neurobiology of attention and suggests that human cognition involves selective attention mechanisms, where only the most important signals are processed for action.\n    2. Reference [4] offers theoretical support through its summarization on research on the roles of prior knowledge in learning, noting that \"Learning proceeds primarily from prior knowledge\" and \"Prior knowledge is the bane of learning. Mere absorption cannot account for the revolutionary changes\".\n\n    We will add these citations in the future version of the paper.\n- **Lines 225-226: \"the higher the accuracy of the segment selector, the greater the cumulative reward obtained by the low-level policy, and vice-versa\"**\n\n    We conduct supplementary experiments in franka-reach environment to validate the relationship between segment selection accuracy and cumulative reward in an episode over five random seeds. The results in the table below demonstrate a clear correlation: When the segment selector accurately focuses on task-relevant objects, $\\pi_l$ achieves the highest rewards. Conversely, incorrect selection leads to substantially lower rewards. \n    We will discuss these experimental results and their implications in the appendix.\n    |Only task-relevant objects|Task-relevant objects + 1 irrelevant object|Task-relevant objects + 2 irrelevant objects|All objects|All task-irrelevant objects|\n    |-|-|-|-|-|\n    |**892.4±22.9**|570.9±223.1|495.3±259.3|234.4±282.9|9.6±17.2|\n\n## Q4: Where are the statistical results of DrQ-v2? & Which results are significantly better than the baselines in Table 1?\nWe appreciate the reviewer's attention to methodological details. To clarify our experimental design for DrQ-v2, as detailed in Section 5.1, we conduct three independent runs in the clean environment using DrQ-v2, selecting the best-perfroming policy as $\\pi_l$ in FTR. This represents the potential upper bound for domain adaptation performance. \n\nThe performance gap between DrQ-v2 and other baselines in Table 1 stems from their different evaluation conditions: while DrQ-v2 was evaluated in the clean source domain, all other methods were tested in visually perturbed target environments. We recognize that this methodological distinction could have been more prominently highlighted, and will add an explanatory footnote to Table 1 in the revised version.\n\n## References\n[1] Shi et al. Composing Pre-Trained Object-Centric Representations for Robotics From What and Where Foundation Models. ICRA. 2024.\n\n[2] Wang et al. Generalizable Visual Reinforcement Learning with Segment Anything Model. arXiv. 2023.\n\n[3] Stevens et al. The Role of Selective Attention on Academic Foundations: A Cognitive Neuroscience Perspective. Developmental Cognitive Neuroscience. 2012.\n\n[4] Roschelle J. Learning in Interactive Environments: Prior Knowledge and New Experience. Exploratorium Institute for Inquiry. 1997."}, {"review_id": "EpYs5C4nEM", "reviewer": "Reviewer_R2gz", "summary": "The paper presents an approach for fast policy adaptation in environments with noisy/varying backgrounds. The approach uses a combination of image segmentation and tracking to extract only the information relevant for the task, while ignoring the uncontrollable components in the background. The representation extraction, in particular the image segmentation selection, is trained using a combination of supervised and RL training. The supervised signal comes from a VLM model that is used to get a high level scene understanding.", "questions": "While I think the idea of the paper is interesting, I have concern around the complexity of the approach and scalability.\n\n- Why is it necessary to have both a tracking and segmentation model given that they are used in an exclusive way? Is it related to cost trade-off for the usage of the VLM?\n- The fact that you need to train the frozen policy using no background seems quite a big limitation to me. Would it be possible to remove such constraint?\n- How would you scale your approach to more complicated setting, eg navigation in Habit, robocasa, ManiSkill-HAB, Carla. In this case it seems quite complicated to have a background free training.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Results. Policy adaptation happens very fast\n\nWeaknesses\n- Complex approach, a lot of hyper-parameters to deal with\n- Requires a specific policy pretraining (black background)", "was_revised": true, "final_justification": "After checking the other reviews and the authors' rebuttal, I decided to increase my score. The authors addressed all my concerns and I think the paper is novel and interesting for the community.", "rebuttal": "Thank you for providing valuable feedback. Below, we provide further clarification and results to address your concern and we hope these materials can enhance your evaluation of our paper.\n\n## Q1: Too many hyper-parameters\nTo address this concern, we highlight two key points from our experiments:\n- Stability: Our ablation studies (Section 5.3) demonstrate that FTR’s performance remains robust across hyper-parameters, suggesting low sensitivity to their exact values.\n- Consistency: The majority of experiments were conducted using the same fixed set of hyper-parameters (detailed in Table 3, Line 571). This also reduces the practical tuning burden.\n\nWe acknowledge that the number of hyper-parameters might appear large at first glance, but their empirical stability and shared use across experiments mitigate the risk of overfitting or extensive tuning.\n\n## Q2: Why is it necessary to have both a tracking and segmentation model given that they are used in an exclusive way? Is it related to cost trade-off for the usage of the VLM?\nWe sincerely appreciate the reviewer's insightful question regarding our design. While the segmentation model alone could theoretically provide complete functionality, our empirical analysis in Section 4.2.2 (Line 176) and the ablation study on $T_{\\text{sel}} = 1$ in Section 5.3 (Line 318) demonstrate that introducing the tracking model significantly improves segment selection consistency. This enhancement leads to more stable RL training.\n\nWhile the tracking mechanism does incidentally reduce VLM computation overhead (as noted by the reviewer), this gain is not our primary design motivation. Our key objective is to improve training stability through more consistent object selection across frames, as evidenced by our experimental results.\n\n## Q3: Requires training the frozen policy using clean background. Would it be possible to remove such constraint?\nWe appreciate the reviewer’s thoughtful question regarding the constraint of training the frozen policy in a clean-background setting. Indeed, this design choice is intentional and stems from our study’s core focus on domain adaptation under visual perturbations—specifically, investigating how a policy trained in a clean environment (source domain) can effectively adapt to environments with visual perturbation (target domain). Here are our justifications for this approach:\n- Many practical applications (e.g., robotics [1]) involve policies trained in structured/clean simulation settings that must later operate in noisy, real-world environments. Our setup mirrors this paradigm.\n- Many complex tasks, like robotic arm manipulation, are impractical or impossible to train directly in complex real-world environments as they require extensive interaction. While our method involves a two-stage training process, it offers much high sample efficiency. A low-level policy trained in a clean background can rapidly adapt to diverse real-world target environments. For instance, in the franka-reach task, the low-level policy requires 200k interactions with the simulation environment. Building on this, the high-level selector then only needs 50k real-world interactions to complete its training.\n\nWhile our current experiments focus on adaptation from clean to perturbed domains, as stated in the conclusion of the paper, the framework could be extended to other scenarios (e.g., training with synthetic perturbations). To demonstrate the feasibility of our approach under such conditions, we conducted experiments involving illumination perturbations. Due to time constraints, the experiments were performed with a single random seed.\n\nThe table below presents our experimental results. We choose franka-reach as the environment. The performance of DrQ-v2 in a clean environment reported in our paper is 948.3, which represents the potential upper bound for FTR's performance.\n\nAs shown in **row 1**, we initially evaluated the performance when the low-level policy was trained without illumination perturbations but deployed with their presence. By comparing this with **row 2**, which presents the adaptation performance of FTR in the target domain with both illumination and background perturbations, it is evident that FTR can effectively maintain the performance of the low-level policy under background disturbances, even when the low-level policy itself is not robust to illumination changes.\n\nSubsequently, we introduced varying directions of illumination perturbations during the training of the low-level policy in the source domain to enhance its robustness. As presented in **row 3**, the low-level policy's performance in environments with illumination perturbations significantly improved. **Row 4**, detailing FTR's adaptation performance in the target domain under both illumination and background perturbations, demonstrates that **FTR consistently maintained the performance of the low-level policy**. Furthermore, benefiting from the enhanced robustness of the low-level policy, the performance showed a notable improvement compared to the second row, closely approaching DrQ-v2's performance of 948.3 in a clean environment.\n\n| With background perturbation? | Low-level policy robust to illumination variation? | With FTR adaptation? | Performance |\n| --- | --- | --- | --- |\n| no | no | no | 611.5 |\n| yes | no | yes | 604.8 |\n| no | yes | no | 956.0 |\n| yes | yes | yes | 925.1 |\n\nWe will emphasize these points in the paper’s limitations part to provide broader context. Thank you for raising this important discussion.\n\n## Q4: How would you scale your approach to more complicated setting, e.g. navigation in Habit, robocasa, ManiSkill-HAB, Carla. In this case it seems quite complicated to have a background free training.\n\nFirst, we acknowledge that our experimental evaluation follows the same protocol established by prior works including the DMControl Generalization Benchmark [2], PAD [3], and SimGRL [4]. We sincerely appreciate the reviewer's insightful comment regarding the current limitations of the benchmark environments used in this line of research.\n\nTo address the scalability challenge in more complex environments, taking navigation in Habit for example, we propose a multi-stage adaptation framework that maintains effectiveness while handling richer visual inputs:\n1. Train a navigation policy in simulation using ground truth environmental information (e.g., full terrain maps)\n2. Distill the terrain-aware policy into a vision-based policy using only egocentric RGB/RGB-D inputs\n3. Apply our FTR approach to adapt to real-world environments, by automatically identifying and focusing on task-relevant visual inputs.\n\n## References\n\n[1] Choi et al. On the Use of Simulation in Robotics: Opportunities, Challenges, and Suggestions for Moving Forward. Proceedings of the National Academy of Sciences. 2021.\n\n[2] Hansen et al. Generalization in Reinforcement Learning by Soft Data Augmentation. ICRA. 2021.\n\n[3] Hansen et al. Self-Supervised Policy Adaptation during Deployment. ICLR. 2021.\n\n[4] Song et al. A Simple Framework for Generalization in Visual RL under Dynamic Scene Perturbations. NeurIPS. 2024."}]

Author rebuttal:

Meta-review: This paper proposes a two-stage framework to help reinforcement learning agents adapt from a clean simulation to environments with distractions, using a trainable object selector to focus on task-relevant objects, followed by pre-trained policy on filtered input.
The paper's main strength is its practical integration of large pre-trained models (a VLM and a segmentation model) into a reinforcement learning adapation pipeline. A primary weakness is the reliance on a low-level policy trained in a clean environment without distractions. While the authors show this can be made more robust with domain randomization or a final finetuning step, it remains a core assumption of the approach.

The discussion period was productive, reviewers raised important questions about the method's novelty (HyXv, WqUh) and ability to handle perturbations beyond background noise (8K8e, WqUh). The authors provide a comprehensive rebuttal including new results demonstrating robustness to distractor objects and lighting variations as well as short finetuning stage to address dynamics mismatches. The response addressed the reviewers' main concerns leading to positive consensus.
Decision: Accept (poster)