When would Vision-Proprioception Policies Fail in Robotic Manipulation?
Authors: Jingxian Lu
Wenke Xia
Yuxuan Wu
Zhiwu Lu
Di Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Robotic Manipulation
Vision-Proprioception Policy
Primary area: applications to robotics, autonomy, planning

Abstract: Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision-proprioception policies. 
In this work, we investigate this by conducting temporally controlled experiments. We found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy plays a limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. 
To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to robust and generalizable vision-proprioception policies. 
The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies in robotic manipulation.
Videos and code are available at https://gewu-lab.github.io/GAP/.

Published as a conference paper at ICLR 2026
WHEN WOULD VISION-PROPRIOCEPTION POLICIES FAIL IN ROBOTIC MANIPULATION?
Jingxian Lu Wenke Xia! Yuxuan Wi', Zhiwu L Di Hul
Gaolligg School of Artificial Intelligence, Renmin University of China
BBijjig Key Laboratory of Research on Large Models and Intelligent Governance School of Artificial Intelligence, Beihang University
ABSTRACT
Proprioceptive information is critical for precise servo control by providing real- time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision- proprioception policies. In this work, we investigate this by conducting tempo- rally controlled experiments. We found that during task sub-phases that robot's s motion transitions, which require target localization, the vision modality of the policy plays limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and sup- pressing the learning of the visual modality during motion-transition phases. To alleviate this, propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision- propooopppioontiionn policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of propriaception's gradient based on estimated probabilities, leading to robust and generalizable vision- proprioception policies. The comprehensive experiments demonstrate GAP is applicable both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision- Language-Actin Action models. We believe this work can offer valuable insights into the development of iision-proprioccption policies in robotic manipulation. Videos and code are available at https: http://gwwwweeee lab. git.gittii qithub. ioo./Ao/aa
INTRODUCTION
Proprioceptive information has long been recognized cornerstone of low-level robotic control, enabling smooth motor behavior through immediate access to the robot's internal state. This ca- pability is especially critical in tasks requiring high accuracy and fast correction, such posture
control (Allum et al. 1998 Henze et al. 2014) and locomotion (Bjelonic et al. 2016 [ee et al. 2020
Yang et al.) (003). In recent years, there has been growing interest in introducing proprioception to learning-base based manipulation (Levine et al. 2016 Cong et al. 2022 Jiang et al., 2025a). Despite the expectations that its inclusion will empower manipulation policies maintain precision and robust- ness across various scenarios, existing works have reported inconsistent observations: HPT Wang al.] (2024) demonstrated clear improvements under the joint utilization of vision and proprioception, while Octo Model Team et al. (2024) observed policies trained with additional propioception seemed generally worse than vision-only policies. This discrepancy exposes a critical obstacle to understanding: when vision-proprioception policies would fail in robotic manipulation?
Extensive prior studies have revealed that the importance of visual and proprioceptive information could change over time within manipulation (Sarlegna & Sainburg 2009 Feng et al. 2024 He et al.)
2025), which referred to as Modality Temporality. For example, during caation-consistem phases
Equal contribution, Corresponding Author (dihu ruc.ccu.cc))

---

Published as a conference paper at ICLR 2026
Performance Real OreuArm
execured acDon
Real 000
Real Curl-kem
46 7.6
Vaion-coly policy
failure
Vision Progrio policy
switch
Stutesa Rate Clange
+20%
grzsp
besemmiee them
80%
part
forward
69%
680mmm
2%
0%
-14%
$6,35
Sin OOO
85.2%
MenlWorkd
73.35%
-20%%
KoboSulte
0-10
10-20
20-30
30-40
40-50
50-00
60-70
70-80
50-90
90-100
Inervention (oaly
timesteps)
Propeio policy
Vision only policy
GAP
Vision only policy without intervention
Switch to Vision Proprio policy
Figure I: The generalization of rision-proprio policies. (left) Assion-Proprioceptine policies perform 15.8% worse than Vision-only policies. (right) We explore this through intervening the task execution of vision-only policy during different periods, by switching intervention has minimal impact during motion-consistent phases. However, during motion-transition phases, switching leads noticeable degradation, indicating the vision modality fails take effect.
iisionpropprioeen
policy. Such
where the robot performs ongoing movements, the policy can benefit more from proprioceptive signals. In contrast, during the transition intervals where the robot's motion shifts, it is required to rely more on visual cues for accurate target localization. To verify whether the vision- prooo-prorriccptioon policy exhibits such collaboration, we conduct an intervention experiment in the controlled simulation enviromment. Concretely, execute the "assembly" task using the vision-only policy, but for specific 10-timestep period, we replace executed actions with those predicted by the rision-proprioception policy under the same observations. As shown in Figure I (right), the intervention brings minimal impact during motion-consister phases like "move forward". during motion--ransiiioe phases like "locate base" and "assemble them". the switching leads to noticeable degradation. suggests that the vision modality of the vision-propriinini proprioception policy fails to take effect during motion-tranee transition phases.
We further investigate the underlying cause from an optimization perspective. During motion- transition phases, visual cues tend to be subtle and may only differ at the pixel level (Tsagkas et al. 2025). As a result, the iision-proproccpptin policy naturally gravitates toward the more concise proprioceptive signals to minimize the training loss, thereby dominating the optimization (Huang) ct al. 2022; Fan et al. 2023). This dominance suppresses the learning of the vision modality and ultimately leads to under-utilized visual information during motion-transition phases.
To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration between vision and proprioception. Specifically, we first define the motion of the robot using concise proprio- ception signals and segment the trajectory into motion-consistent phases. Robot's motion transits within the intervals between these phases, we thus employ a temporal network like LSTMM to model transition processes. It helps eliminate potential errors introduced by the segmentation and estimates the probability that each timestep belongs to traiii-trnnnttinn phases. During policy learning, we guide the prsiin-pporriocettion policy to focus essential visual cues of motion-transition phases, by applying fine-graines gradient adjustment that reduces the magnitude of proprioception' 's gradient.
Our GAP algorithm facilitates the lision-proprioceptionn policy to effectively utilize proprioception without suppressing the lcarning of visual modality. GAP is compatible with both conventional and Vision-Lanne Lation-Languag-Actttin
models, and its versatility and effectiveness have been validated by extensive experiments in both simulated and real-world environments. Our evaluations span multiple policy architectures, including MLP-based, diffusion-based and transformer- based policies. Further, they cover articulated object manipulation, contact-rich interactions, rotation-sensitiv tasks, and soft object manipulation, across both one-arm and dual-arm robotic setups. Across all experimental settings and configurations, vision-propriost proprioception policies equipped with proposed GAP con- sistently lead to superior performances. We believe this work can offer valuable insights into the development of proriopropiio policies in robotic manipulation.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Paiion-roprriceppion Policy in Manipulation. Vision has been the most commonly used modality in robotic manipulation policies ZZikooiich et al. 2023] Kim et al. 2024; Zeng et al. 2024). While it provides sufficient information to complete many manipulation tasks, visual data often includes a large amount of noise, such as irrelevant background distractions (Tsagkas et al. (025). Therefore, more concise proprioceptive information has been introduced by many works to assist robotic manipulation policy, with the expectation that it can provide complementary and physically grounded information for precise and robust task execution (Cong et al. 2022 Mandlekar et al.) [022] Chi et al. 2023] Fu et al. 2024 Wang et al., 2024| Liu et al. 2024). However, existing studies have reported confused observations: some works demonstrate clear improvements when integrating proprioceptive
information with vision (Cong et al. 2022 Wang et al. 2024), others observe limited gains or even
detrimental effects (Mandlekar et al. 2022 Octo Model Team et al. (002). Fu et al. (2024) attributes this to overfitting while Octo Model Team et al. (2024) suggests it arises from causal confusion between the proprioceptive information and the target actions. In this study, we further explore the utilizations of each modality and introduce a tomporryyt perspective to offer valuable insights into the development of vision- vision-propriocee proprioception policies for robotic manipulation.
Modality Temporality. In manipulation tasks, each modality's 's contribution to decision decision-making can vary significantly over time. For example, in "pick-place" task. policy must first rely on vision to locate the target object. When moving toward the object, proprioception becomes more critical for executing consistent and precise actions. It is proven by strong correlations between variations in modality data and task stages (Lee et al. 2019 He et al. 2025 Jiang et al., 2025b). Feng et al. (2024) summarizes such property of manipulation tasks as modality temporality. Given this nature of robotic manipulation tasks, recent works have proposed approaches based on dynamic fusion LL al. 2023 Feng al., 2024 He et al. 2025) and modality selection Uiang al.) (025b) to improve the performance of multimodal manipulation policies. In this study, we introduce the modality- temporality perspective to understand the roles of vision and propriocetion and propose the gradient adjustment algorithm to enhance dynamic collaboration within the vision- proprioception policy.
3 OPTIMIZATION ANALYSIS OF DISION-PROPRIOCEPTTOR POLICIES
In this section, we first formalize the problem and further analyze the generalization of vision- proprioception policies from an optimization perspective. The sssionproorioccption policy is learned under the Behavior Cloning (BC) paradigm, which can be formulated as the Markov Decision Process (MDP) framework (Torabi et al. 2018). Formally, the policy л takes the environment observation or E O as input at each timestep l. In this work, o, includes RGB-sensor readings Ut, and for policy uu+s. it includes robot proprioceptive information 8, additionally. This proprioceptive information consists of the 6D pose of robot's 's gripper (pf ,i. př. O7 0r, 02) E IR6 in Cartesian space and orientation, and a continuous value 9a E [O, 1 representing the degree of gripper opening, with ge = 1 denoting fully open and 9t 0 denoting fully closed.
The policy π maps the observation history to sequence of actions: âe+L= = (+:) where L and H indicate the length of predicted action sequence and observation history respectively. For simplicity, we set omit them in the following discussion. The training objective can be formulated as:
π = argmin, (oo..aa)rn.LCeeee
at),
where Tę expert demonstration dataset and a action labels. In vanilla BC, L typically represents the Mean Squared Error (MSE) loss for continuous action spaces, or Cross-Entropy (CE) loss for discrete action spaces. We focus solely on the vanilla MSE loss here.
In this work, we adopt standard joint-learning architecture to design the vision- proprricooiiooiioon policy, which extracts features from both vision and proprioception modalities using two separate chunks p., &.. These features from two modalities are then concatenated and fed into the policy head th. Although some recent works have tried exploring alternative modality fusion approaches (Wang et al.
2024; Feng et al.) 2024), concatenation remains the most widely used approach (Levine et al. 2016
Cong et al. [022] Mandlekar et al. 2022). To support our analysis under this fusion approach, we
split the first layer of MI.P-based policy head tb into vh., the and rewrite the action prediction as:
3

---

Published as a conference paper at ICLR 2026
ã= (f.. vo(l))) Wshore + b,
(2)
where So is the feature extracted by (o), (o) respectively. Under Gradient Descent (GD)- based policy learning, the optimization of the vision chunk's parameters wy is influenced by:
BC Ow.
a a - all ((v.(fa) + vo(fo)) Wehare + b) Of. Owi
Ofe
Within the execution trajectory of the task, changes in visual cues are usually subtle compared to proprioceptive signals. For example, when the gripper is closing, visual cues differ only at pixel-level (Tsagaas et al., 2025), while concise and low-dimension proprioceptive signals directly represent this process via changes in opening degree g. As result, the vision-proprioceptioo policy naturally gravitates toward proprioceptive signals to minimize the training loss. leads to optimization dominated by proprioception and suppresses the learning of wy due vision modality's low contribution to action prediction (Huang et al. 2022 Fan et al. 2023).
As shown in Figure (right), such overreliance to proprioception brings negligible impact during motion-consistent phases, since the execution of ongoing movements benefits significantly from proprioceptive signals. However, the initial positions of the target objects vary during testing and the proprioceptive signal does not contain object-related information. During motion-transition phases, the policy is required accurately locate the target objects. The suppressed learning of vision modality thus regretfully impairs generalization of the ssioo-proprioception policy]]
METHOD
To alleviate the suppression of the learning of vision modality during propose the Gradient Adjustment with Phase-gun guidance (GAP) algorithm. As shown in Figure we initially define the representation robot's motion and identify motion-consistent phases. Motion- transition phase indicators are then predicted to estimate the probability that each timestep belongs to motion-transition phases. Based on these indicators, we apply fine-grained gradient adjustment during policy learning, facilitating dynamic collaboration within the assionprropiocepttinn policy.
phases. we
4.1 MOTION REPRESENTATION OF ROBOT
Proprioceptive signals of the trajectory [81, .2,...... sx] directly provide the state of the gripper's position p. orientation 0, and opening degree g. The variations in them effectively capture the motion of the robot arm over time. We first define the representation of motion for further motion- transition phase estimation. Specifically, the motion between timestep and timestep j is defined as: Maj [Pr.j. Oz.j. Srj). where Paj = Pj denotes the change in the gripper's 3D position, Biy 8 0g denotes the change in orientation, and gi. = gj 9: denotes the change in gripper opening. Together, these three dimensions provide a complete representation of the robot's motion.
4.2 MOTION-TRANSITION PIIASE ESTIMATION
The represented motion captures the movement of robot arm, allowing expert demonstrations be segmented into sequences of continuous states that correspond to semantically similar motions. To leverage this property for identifying motion-consistent phases, we employ the simple yet effective Change Point Detection (CPD) algorithm (Liu et al., 2013 Aminikhanghahi & Cook (017). The overall motion of a trajectory phase Tt,t,, can be characterized by m Based on whether the directions of these changes consistent, we define the following distance between phase motion m4 and adjacent motion m
d(merit Mant))))) -cos( 1 accos(Or.ityy O.i4 i) (sgn(g)aat)) = sgn (g(a
'For experimental results of UNet-based diffusion policies, please refer to Appendix C.3
4

---

Published as a conference paper at ICLR 2026
Modion Reprosntation
Motina transiinnii
Phase Fatimaatio
motixa Syw
Eiqr, d'
trjjetoory
miniazee
phyaee
migles
concepanding prophociptin saquanee Ing moyl
informing
plase untion
statc
segmen
stalc
motins trassts
tmmestep
trasstion
timesicp
Bision
motion reprosmtation
Imput Ma
proppto Inpur
IPay Bry)
policy heed
Magnitude (I - Ar
tratoiiion cxlinatioon
Pej pretiimn change
6iy orientation change
H opening degree change
Rackward
14 a.ll
Gradlent Adjustment
himostopy
Figure 2: The pipeline our Gradient Adjustment with Phase- Phase-guidand -asieenuudncce (GAP) algorithm. We define the motion representation and identify the cost between phase motion and each adjacent motion. Motion-transition phase indicators are then estimated reduce the magnitude of proprioception's 's backward gradient. GAP facilitates vision- proprioception policies to effectively utilize proprioception without suppressing vision modality.
phases by minimizing the total
where sgn( denotes the sign function, and a, are weighting factors for the orientation and opening degree component respectively. The statistic that measure the motion inconsistency of phase Ttys defined Ct E d(me, m.++1). The Change Point Detection algorithm leverages dynamic programming to identify set of indices that minimize the total cost Σ, Crz segmenting the trajectory into motion-consister phases.
Motion of the robot transits within the intervals between these phases, requiring the policy locate target object. Vision is therefore expected to play more significant role. However, transition is continuous process and the discrete labels from CPD can't capture this continuous nature effectively, which is demonstrated by our ablation experiments in Section B.2 of the Appendix. Therefore, we further utilize the temporal differences of proprioceptive information As; = 8;+1 s; and leverage their sequential context with an temporal network such LSTM. It predicts motion-transition phase indicators Pj to estimate the probability that timestep belongs to motion- nottiontraanstionn phases. The predicted indicators p is under the supervision of indices set J. Additionally, for timesteps within a range near the transition, we reduce the penalty applied to them in order to better capture the inherently continuous and gradual transition process.
4.3 GRADIENT ADJUSTMENT FOR MODALITY COLLABORATION
The vision-proprioeee proprioception policy extracts features from both vision and proprioception modalities using two separate chunks Φ.., &., which consist an encoder and temporal transformer, these features are then fused and fed into policy head to predict the action. However, since visual cues during motion-transition phases may be subtle, the policy tends to rely heavily on features of proprioception. As result, the gradient optimization for corresponding samples becomes dominated by proprioceptive inputs, which in turn constrains the learning of the vision modality chunk Φ
To mitigate this, employ gradient adjustment to control the optimization of proprioceptive chunk 6e during motion- motion-transition transition phases, thereby guiding the iision-proprioception policy to focus more on visual cues and preventing the degradation of its generalization. Concretely, in the j-th epoch of Gradient Descent (GD)-based optimization, the parameters of the proprioceptive feature chunk w updated according to the following formula:
wبth wi X (1 p) (Ww
(5)
5

---

Published as a conference paper at ICLR 2026
where 7) is the learning rate, À is hyper-parameter that controls the degree of adjustment. For each timestep, we modulate the magnitude the proprioception backward gradient based on its indicator of belonging notion-transition phases. The higher value of p leads to greater degree of modulation.
By applying gradient adjustment with phase-guidance as illustrated Algorithm T the vision- proprioception policy is enabled to effectively leverage proprioceptive information without compro- mising its generalization ability.
Algorithm Vision-Proos Prorroopropiooopttionn Policy Learning with Gradient Adjustment
Notations: Expert demonstrations de. proprioceptive signals Se. epoch number T,, vision- proprioception policy Tets. proprioception chunk parameters wa, vision chunk parameters we-
Motion- Transitionnnn Phase Estimation
Identify motion-consistent phases by Change Point Detection CPD(s.):
Predict motion-transition phase indicators 4- LSTM(A8s))
Gradient Adjustment during Policy Learning
for j = 0, 1.... T
do
Sample fresh mini-bated B, from expert demonstrations 0z;
ferward the batched data B, to To4si
Calculate average indicator Pj of By;
Update proprioception chunk wi+1 using Equation
Update vision chunk w
end for
5 EXPERIMENTS
In this section, we validate the versatility and effectiveness of our proposed Gradient Adjustment with Phase-guie crase-guidance (GAP) algorithm through series of question-driven experiments. The evaluations comprehensively cover a wide range of manipulation tasks, including articulated object manipulation, rotation-sensitivee tasks, well as long-horizon and contact-rich tasks. Ablations on each modules and hyperparameters are presented in Section B of the Appendix due to space limitations.
5.1 EXPERIMENTAL SETUP
We select two simulated environments our benchmarks: Meta-World Yu et al. (2020) and Robo- Suite Zhu et al. (202)). Tasks Meta-World are relatively simple, featuring a 4-dimensional action space that includes the gripper's position and its opening degree, while tasks in RoboSuite involve complex scenarios, longer task sequence horizons and richer physical interactions, with the action space further including the orientation of the gripper. For real-world experiments shown in Figure 3 we use 6-DoF xArm 6 robotic arm equipped with a Robotiq gripper. Moreover, we utilize the open-source Cobot Magic platform support tasks that require dual-arm collaboration. In all tasks, the initial position of target object varies randomly in each validation, while the initial position of gripper remains fixed. Tasks in simulation and real-world are evaluated with 100 and 20 rollouts, respectively. The detailed task descriptions provided in Section of the Appendix.
5.2 CAN GAP LEAD TO MORE ROBUST VISION- POLICIES?
Vision-Proprioceppe policies perform generally worse than vision-only policies. Can GAP lead to more robust vision- propriiccociiinnnnn policies? To answer this, we conducted comparative analyses between our algorithm and the following baselines:
MS-Bot Feng et al. (2024); this method uses state tokens with stage information to guide the dynamic collaboration of modalities within multi-modality policy.
Auxiliary Loss (Aux): following HumanPlus Fu et al. (2024), we use visual feature to
predict the next frames auxiliary loss, which tries to enhance the vision modality.
Mask: to prevents the overfitting specific modality, RDT-1B Liu et al. (2024) randomly and independently masks each uni-modal input with a certain probability during encoding. We adapt the algorithm by masking only proprioception modality instead.

---

Published as a conference paper at ICLR 2026
Oce Art Tocks
Dual Tasks
pros
hundowee
pur
tug sweep
pour
Figure 3: Visualization of real-world tasks. Our experiments cover wide range of manipulation tasks, including both One-Arm and Dual-Arm Setups.
Results in Table demonstrate that vision-press prapoooppppiionttioon policies with our GAP applied outperform vision-only policies and other methods. Although MS-Bot achieves overall improvements over the vision-only policy by incorporating stage information, focuses on the semantic stage instead of motion-transition phases. As result, its benefits are marginal in tasks like "push-wall" and "lift lid and pour", where motion frequently transits. This highlights the necessity of fnn-ggrined gradient adjustment during motion-transiteen phases. Auxiliary loss forces the vision-pon proprioception policy to concentrate on visual input during the whole task, which falls short in tasks requiring proprioception to enhance the precision and robustness of manipulation, such "threading". Meanwhile, masking the proprioceptive input with a fixed probability overlooks the modality temporality of manipulation tasks, resulting in minimal improvement. By adaptively applying fine-grained gradient adjustment phases, GAP enables the vision sssion-roprioccption policy to effectively leverage these two modalities and outperform both the vision-only policy and other methods.
during
Table Comparisons with other methods both simulated and real-world environments. Average success rate and standard deviation of simulation results are calculated over 5 seeds. The vision- proprioception policies after our gradient adjustment significantly outperform other methods.
RoboSuite
Task
pick-place
assembly
disassemble
pash-wall
bin-picking
hammer
threading
Visioe-only Concatenation
91 01.81.5
82.6133
84,4=222
63.2+119
63:2215
83.8
65.8+224
43.6::..7
7..41:15
74.612.1
79.6.1..1
54.4229
48.812.2
78.6.1
54.6:113
33.211.9
MS- et
91.+116
91.0+2211
66.211
694419
87.6+1.11
69.4++1.9
Aux Fu ot a. 2021
88.21:2.6
9044218
77.82:.9
51.01:.2
$4.4115
71.21:9
$4.0:22.2
45.41:2.1
Mask
4024
8..41.88
8..21:16
83.8.1.77
79.21.37
60411.1
78.6.118
61.8119
47.2116
(Qurs)
73.6+1.3
77.6+4099
Sctup
Real One-Arm
Real Dual-Ar
Task
Method
press button
cube
rag sweep table
handover
put thermos into bug
lift lid and pour
Vision-only
18/20
14/20
9/20
15/20
11/20
9/20
Concatenation
12/20
11/20
5/20
12/20
7/20
5/20
MS-Bet Feng et 2024
20/20
16/20
11/20
16/20
13/20
10/20
16/20
11/20
15/20
8/20
Mask Lii et 2024
18/20
14/20
7/20
15/20
9/20
7/20
GAP (Ours)
20/20
17/20
13/20
18/20
16/20
15/20

---

Published as a conference paper at ICLR 2026
localo
Brggni
Sucoes Rate Changs
0%%
-2065
0.10
10-20
20-30
30-40
40:30
30-60
60-70
70-80
80-90
90-100
Interecntion (only lists limeslepe)
Vision only policy without intervention
Swiich to Vision Proprio policy
Switch to Regulaned Visien Prappio policyy
Figure 4: The intervention experiment of the GAP-equipped vision-proprioceptionn policy. The sight changes in success rate indicate that GAP does enhance the utilization of vision modality.
5.3 DOES GAP ENHANCE THE UTILIZATION OF VISION MODALITY?
Although vision- propriooooiiiiiooonn policies outperform vision-only policies after apply GAP, it re- mains unclear whether GAP truly enhances the utilization of the vision modality within vision- proprioception policies. To answer this, first conducted intervention experiment under the same settings as described Section [ As shown in Figure the degrees of suppression of vision transition phases GAP does enhance the utilization of vision modality. We further evaluated the generalization of the vision-proys propppprpriioettionn policies in out-of-distribution (OOD) scenarios. In each scenario, the initial distribution of object positions differs from that the training dataset expert demonstrations. The vision-only policies are less affected by such changes due well-utilized vision modality as demonstrated in T2 crsion-proprocception policies exhibit poor generalization with suppressed vision. Meanwhile, Our algorithm alleviates this by regulating the optimization of the proprioceptive, preventing the suppression. The maintained superior performance over vision-only policy also indi- cates the effectiveness of introducing proprioception modality for precise and robust manipulation. The linear-probing experiments that directly evaluate the vision modality are provided in Section C.1
modality during
significantly reduced after applying GAP, indicating
Table 2: Experiments under out-if-distribution settings. For each task, our proposed GAP algorithm
enhances the generalization of the vision-proprioccetioon policies.
Setup
Meta-World
RoboSuite
Real One-Arm
Real Dual-Arm
Task
assembly
bin-picking
stack
threading
cube
handover
Method
Vision-only
78%
59%
63%
32%
12/20
12/20
Concatenation
62%
32%
49%
28%
7/20
9/20
GAP (Ours)
88%
67%
72%
49%
15/20
15/20
5.4
GAP COMPATIBLE WITH VISION- ACTION MODELS?
Above experiments have demonstrated that our algorithm facilitates dynamic collaboration within conventional ission-propriocenne models. We further investigate is GAP compatible with Vision- Language-Action (VLA) models. Specifically, we compare fine-tuned Octo model Octo Model Team et al. (2024) using only visual information (Octo-V) versus using both vision and proprioception (Octo- VP), and tries to apply our gradient adjustment algorithm during fine-tuning. As reported in the original paper, policies trained with additional propioception seemed generally worse than vision-only policies. We observe the same trend across various tasks Table 3 However, after applying our gradient adjustment algorithm, Octo-VPt achieves average improvement of 17%
8

---

Published as a conference paper at ICLR 2026
and exhibits stronger generalization ability than Octo- V. These results suggest that algorithm effectively enhances dynamic collaboration between vision and proprioception within VLA models.
Table 3: Performances of fine-tuned Octo. i indicates GAP is applied.
Suite
Meta- World
RoboSuite
Task
disussemble
push-wall
put hammer into drawer
threading
Model
Octo-V
95%
77%
92%
Octo-VP
82%
65%
88%
57%
Octo-VPi
100%
85%
97%
78%
5.5 CAN GAP BE APPLIED TO VARIOUS MODALITY FUSION APPROACHES?
The preliminary results in Section 3 reveal that the vision- pitionnpppriocpttion policy using straightforward concatenation tends perform worse than the vision-only policy. We further explore broader set of fusion approaches and validate the versatility of our algorithm. Specifically, we apply GAP to three typical and widely used fusion approaches: Concatenation, Summation and FiLM Perez et al. (018))
As reported in Table 4 vision-only policies outperforms all three fusion approaches in tasks such "pick-place" and "put hammer into drawer", indicating that vision modality suffice for certain tasks. However, they fail drastically in task "threading" due to demands for precise manipulation and exhibits suboptimal performance in task "push-wall", which involves visual occlusions the target location, highlighting the necessity of the inclusion of proprioceptive information for precise and robust manipulation.
Table Performance of typical fusion approaches combined with GAP. indicates GAP is applied.
Suite
Meta: Workd
RoheSuite
Task
Method
pick-place
nssembly
disnssemble
push-wall
bin bin-picking
put hammer into drawer
stack
threading
Vision-only
Concatenation
79%
76%
80%
56%
49%
79%
56%
34%
Summation
78%
95%
80%
54%
61%
75%
49%.
30%
FiLM
75%
91%
47%
67%
59%
76%
53%
41%
Concatenation/
94%
96%
91%
73%
70%
91%
77%:
52%
Summation]
92%
97%
93%
66%
70%
88%
82%
48%
FILM;
90%
94%
85%
74%
68%
95%
72%
46%
Concatenation preserves raw features from both modalities, but the high-dimensional redundancy hinders the policy to dynamically utilize each modality. As a result, underperforms tasks like "push-wall', where effective coordination is required. Simple summation may obscure critical details, whose limitation is evident in precise manipulation tasks such as "threading" and "push-wall". Meanwhile, FiLM applies affine transformations conditionally adjust features, making it more suitable for tasks requiring modality collaboration. For instance, it achieves notably higher score in "push-wall" task. However, performance tends degrade simpler tasks where such complex conditioning may be unnecessary. Conversely, GAP successfully unlocked the full potential of the vision- proproppopiooctttion policy, outperforming vision-only policies all three fusion approaches.
VISUALIZATION
6.1 VISUALIZATION OF MOTION-TRANSEE PHASE ESTIMATION
To better understand the Motion- Transition Phase Estimation, we present visualizations that display the estimated transition probabilities alongside corresponding RGB images of "threading" task in this section. We also show the relationship between and visual uncertainty. Specifically, we leverage R3M (Nair et al. 2023) extract universal visual features of this task, and calculate the

---

Published as a conference paper at ICLR 2026
local entropy of features based on sliding window covariance, denoted The visual uncertainty u is then normalized to the range [0, 1] using (tanh(u) + 1)/2. As shown in F the value of increases sharply at key moments, such as when the robotic arm grasps object or transitions from a vertical to a rotational movement. This demonstrates that our method accurately captures motion transitions during the robot's operation. Moreover, accompanied by decreases in scaled visual uncertainty, indicating clear inverse relationship between them. implies that changes in the visual input tend to be relatively subtle during motion transitions, highlighting the importance of enhancing the learning of the visual modality.
observe that increases in p are consistently
A
1.0
0.8
0.6
0.4
0.2
0.0
100
175
simesepp
phase
uncerainty
Figure 5: Visualization of Motion- Phase Estimation.
6.2 VISUALIZATION OF Loss CURVES
Furthermore, we demonstrate the effect of GAP on the loss curves of lisoonproorioccetionn policy in Figure 6 Due the adjustment of GAP the gradient updates for proprioceptive parameters, the loss decreases more slowly during the early and middle stages of training. However, the policy ultimately converges to lower loss, indicating that GAP effectively enhances performance of vision-proprioccption policies.
Task "assembly"
Task "threading"
original
Figure Visualization of loss curves.
CONCLUSION AND LIMITATION
In this work, we illustrate that the vision modality of the iision-proprioceptin policy plays a limited role during motion-trane transition phases due to suppression. To alleviate this, propose the Gradient Adjustment with gaase-guidance (GAP) algorithm, enabling dynamic collaboration between vision and proprioception within vision- praionnproorioctttion policy. We believe this work can offer valuable insights into the development vision- protoonproriicettionn policies for robotic manipulation.
Limitations. All vision-prevs proprioccooiiiiionn policies are trained single embodiment in this work. As existing large-scale datasets often contain diverse embodiments, exploring the role proprioception in cross-embodiment datasets would be promising for future research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Beijing Natural Science Foundation (4262050), in part by National Natural Science Foundation of China (62376274) and the fund for building world-class universities (disciplines) of Renmin University of China.
REPRODUCIBILITY STATEMENT
Extensive efforts have been made ensure the reproducibility of work. Descriptions of datasets and experiment settings are provided in S. and Appendix[A.]] Hyperparameters are presented in Appendix A.2 Furthermore, the algorithmic procedure clearly described in Algorithm
REFERENCES
JHJ Allum, BR Bloem, MG Carpenter, M Hulliger, and Hadders-Algra. Proprioceptive control of posture: a review of new concepts. Gait & posture, 8(3):214-242, 1998.
Samaneh Aminikhanghahi and Diane J. Cook. A survey of methods for time series change point detection. Knowledge and Information Systems, 51 (2):3339-367, 2017.
Marko Bjelonic, Navinda Kottege, and Philipp Beckerle. Proprioceptive control of an over-actuated hexapod robot in unstructured terrain. In Proceedings of International Conference Intelligent Robots and Systems (IROS), pp. 2042-2049, 2016.
Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Lin Cong, Hongzhuo Liang, Philipp Ruppel, Yunlei Shi, Michael Görner, Norman Hendrich, and Jianwei Zhang. Reinforcement learning with ssiio-proprioception model for robot planar pushing. Frontiers in Neurorobotics, 16:829437, 2022.
Yunfeng Fan, Wenchao Xu, Haozhao Wang. Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20029-20038, 2023.
Ruoxuan Feng, Di Hu, Wenke Ma, and Xuelong Li. Play to the score: Stage-guided dynamic multi-sensory fusion for robotic manipulation. In Proceedings of Conference on Robot Learning (CoRL),
Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. In Proceedings of Conference on Robot Learning (CoRL), 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778, 2016.
Zihao He, Hongjie Fang, Jingjing Chen, Hao-Shu Fang, and Cewu Lu. Foar: Force-aware reactive policy for contact-rich robotic manipulation. IEEE Robotics and Automation Letters, 2025.
Bernd Henze, Christian Ott, and Maximo A. Roa. Posture and balance control for humanoid robots in multi-contact scenarios based on model predictive control. In Proceedings of International Conference on Intelligent Robots and Systems (IROS), pp. 3253-3258, 2014.
Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and I Longbo Huang. Modality competition: What makes joint training of molti-modal network fail in deep "ourning((prvvallii In Proceedings of International Conference Machine Learning (ICML), pp. 9226-9259, PMLR, 2022.
Zhiwei Jia, Vineet Thumuluri, Fangchen Liu, Linghao Chen, Zhiao Huang, and Hao Su. Chain- of of- thought predictive control. In International Conference on Machine Learning (ICML), pp- 21 PMLR, 2024.
11

---

Published as a conference paper at ICLR 2026
Guangqi Jiang. Yifei Sun, Tao Huang, Huanyu Li, Yongyuan Liang, and Huazhe Xu. Robots pre- mreitrain robots: Manipulation-cemne centric robotic representation from large-scale robot datasets. In Proceedings of International Conference on Learning Representations (ICLR), 2025a.
Jiawei Jiang, Keita Ota, Devesh K. Jha, and Asako Kanezaki. Modality selection and skill segmenta- tion via mrosatmy attention. https://arrihorrr//aa2554444444 2025b.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source Vision-language-actionn model. In Proceedings of Conference Robot Learning (CoRL), 2024.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47))eabc5986, 2020.
Michelle A. Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In Proceedings of International Conference on Robotics and Automation (ICRA), pp. 8943-8950, 2019.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-40 2016.
Hao Li, Yizhi Zhang, Junzhe Zhu, Shaoxiong Wang, Michelle A Lee, Huazhe Xu, Edward Adelson, Li Fei-Fei, Ruohan Gao. and Jiajun Wu. See. hear, and feel: Smart sensory fusion for robotic manipulation. In Proceedings of Conference on Robot Learning (CoRL), pp. 1368-1378, 2023.
Song Liu, Makoto Yamada, Nigel Collier, and Masashi Sugiyama. Change-point detection in time- series data by relative density-ratio estimation. Neural Networks, 43:72-83, 2013. ISSN 0893-6080. doi: http:///cioooe/eerree//////////0 com/
URL hettps:/www..ccennee.
Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. Rdt-lb: diffusion foundation model for bimanual manipulation. arXiv preprint 2rXiv:2410.07864 2024.
Jingxian Lu, Wenke Xia, Dong Wang, Zhigang Wang. Bin Zhao, Di Hu, and Xuelong Li. Koi: Accelerating online imitation learning via hybrid key-state guidance. In Proceedings of Conference on Robot Learning (CoRL), pp. 3847-3865. PMLR, 2025.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang. Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Proceedings of Conference on Robot Learning (CoRL., pp. 1678-1690, 2022.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning (CoRL.), pp. 1820-1864. PMLR, 2023.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In Proceedings of Conference on Robot Learning (CoRL), pp. 892-909. PMLR, 2023.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems (RSS), Delft, Netherlands, 2024.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
12

---

Published as a conference paper at ICLR 2026
Fabrice R Sarlegna and Robert L Sainburg. The roles of vision and proprioception in the planning of reaching movements. Progress in Motor Control: A Multidisciplinary Perspective, pp. 317-335, 2009.
Qi Sun, Pengfei Hong. Tej Deep Pala, Vernon Toh, U Tan, Deepanway Ghosal, Soujanya Poria, et al. Emma-x: An embodied multimodal action model with grounded chain of thought and look-ahead spatial reasoning. arXiv preprint arXiv:2412.11974, 2024.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. Proceedings of International Joint Conference on Arlificial Intelligence (IJCAI), pp. 4950-4957, 2018.
Nikolaos Tsagkas, Andreas Sochopoulos, Duolikun Danier, Sethu Vijayakumar, Chris Xiaoxuan Lu, and Oisin Mac Aodha. When pre-trained visual representations fall short: Limitations in Visuo-motor robot learning. arXiv preprint arXiv:2502.03277, 2025.
Lirui Wang. Xinlei Chen, Jialiang Zhao, and Kaiming He. Scaling proprioceptive-visaal learning with heterogeneous pre-trained transformers. Advances in Neural Information Processing Systems (NeurIPS), 1774:22420-122 124450, 2024.
Yanhao Yang. Joseph Norby, Justin Yim, and Aaron Johnson. Proprioception and tail control enable extreme terrain traversal by quadruped robots. In Proceedings of International Conference on Intelligent Robots and Systems (IROS), pp. 735-742, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Proceedings of Conference on Robot Learning (CoRL), pp. 1094-1100 100, 2020.
Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, al. Learning manipulation by predicting interaction. In Proceedings of Robotics: Science and Systems (RSS), 2024.
Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine fine-grained bimanual manipulation with low-cost hardware. In Proceedings of Robotics: Science and Systems (RSS), 2023.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, Yifeng Zhu, and Kevin Lin. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009.12253 122933, 2020.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of Conference on Robot Learning (CoRL), pp. 2183. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A IMPLEMENTATION DETAILS
In this section, we provide implementation details, including experimental settings and algorithmic specifics, to facilitate the reproduction of our work.
A.1 EXPERIMENT SETTINGS
Simulation experiments are conducted on two representative environments: Meta-World Yu et al. (2020) and RoboSuite Zhu et al.] (2020). Tasks in Meta-World World are relatively simple, featuring 4-dimensional action space that includes the gripper's position and its opening degree, while RoboSuite tasks involve complex scenarios, longer task sequence horizons and richer physical interactions, with the action space further including the orientation of the gripper. During policy learning, task in Meta- World uses 100 hard-coded expert demonstrations from its official open-source implementation, while task in RoboSuite leverages 500 synthesized trajectories that generated from human demonstrations (Mandlekar et al. 2023).
We use an observation history window H of 5 with the learning rate 3 to train the vision- proprioception policy network. The parameters of the policy are optimized by Adam. The vision- proprioception policy consists of two feature extraction chunks and a policy head. For the visual input, spatial features are first extracted using a ResNet- 18 He et al. (2016) backbone, then projected into a 512-dimensional hidden representation. 4-layer temporal transformer with hidden dimension of 256 is applied to capture temporal features. For proprioceptive input, spatial features are extracted using a 3-layer MLP. These features are fused via concatenation and then passed through a 3-layer MLP to output a sequence of actions of length L = 9. We train with batch size 128 on a single NVIDIA RTX 3090 GPU for 100 epochs (around 2 hours for 100 Werld demonstrations and hours for 500 RoboSuite demonstrations).
For real-world experiments, we use a 6-DoF xArm robotic arm equipped with a Robotiq gripper for all one-arm tasks. The visual information is provided by RGB images captured by an Intel RealSense D435i camera mounted on the wrist. Moreover, we utilize the open-source Cobot Magic platform to support tasks requiring dual-arm collaboration. It features four robot arms and three Intel RealSense D435 RGB-D cameras. Due to extended reach of the dual-arm setup, the two wrist-mounted cameras can't 't fully capture the task scene. Therefore, we also incorporate the front camera on the platform to obtain a more comprehensive RGB observation of the task. Policies of real-world experiments
trained with 50 demonstrations collected by teleoperation. For one-arm tasks, the vision- proprioception policy is the same as in simulation. For dual-arm tasks, we modified the ACT Zhao al. (2023) architecture, we train the left-arm policy using observations from the left-wrist camera, the front camera, and the left-arm proprioceptive information, and similar settings are applied to the right-arm policy.
In all tasks, the initial position of target object varies randomly in each validation, while the initial position of gripper remains fixed. In out-of-distribute (OOD) scenarios, the initial distribution of object positions differs from that in the training dataset of expert demonstrations.
A.2 DETAILS OF GAP
In this study, we illustrate that vision proprioception policy would fail during motion- trrioonttttnniiionn phases due to its suppressed vision modality. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm, enabling dynamic collaboration between vision and proprioception within iisoonprrorriocettion policy.
The complete list of hyperparameters used in Equations 4 and 5 is provided in Table Since the motion inconsistency of the gripper's opening state is measured using binary (0/I) values, we set 3 to lower value to balance the numerical scale, while keeping gripper motion. To enable fine- reee-graindd gradient adjustment, we set the parameter À controlling the degree of adjustment to moderate level, avoiding excessive modulation leading unstable gradient updates and potential policy collapse. Additionally, we apply gradient adjustment only during the early stage of policy learning, such as the first 50 epochs. Ablation studies these hyperparameters
to account for the overall
14

---

Published as a conference paper at ICLR 2026
For bimanual tasks, the proprioceptive information from each arm differs in physical meaning. The trajectories of the two arms thus have distinct motion transition phases. Therefore, we train policies for each arm independently. In transformer-based models, modalities become highly integrated within the transformer, we thus only applied gradient adjustment to the proprioception feature extractor (i.e., the parameters before the transformer) for Octo-VP.
provided in Section B.3
Table 5: List of hyperparameters uesd in GAP.
Hyperparameters
Value
1
8
2e
0.3
B ABLATION EXPERIMENTS
B.I ABLATION ON MOTION
MOTION-TRANSITUN TRANSITION PHASE ESTIMATION
In our proposed GAP framework, the Aatiio-Transitio Phase Estimation module essential for enhancing vision- prooppprorriocettion policies. To thoroughly analyze its effectiveness and robustness, conducted comprehensive ablation studies.
First, kept other components unchanged and tested the impact of different trajectory decomposition methods on policy learning. The detailed descriptions of the comparison methods are as follows:
Human: Since obtaining ground truth phase transition labels is infeasible due the com- plexity and diversity of manipulation tasks, we instead employ human experts to manually label phase transitions.
HDBSCAN: Following Emma-X X [un et al. (0244) utilize the HDBSCAN algorithm to cluster our distance metric defined in Equation 4, decomposing trajectories into phases.
CoTPC et al. (2024): To decompose task trajectories into temporally close and function- ally similar subskills, CoTPC applies the Change Point Detection (CPD) algorithm on the sequence of robot actions using Cosine distance.
As shown in Table 6. using human-labeled phase transitions for gradient adjustment does provide a slight improvement in performance. However, since human annotators tend to decompose tra- jectories from semantic perspective rather than based actual motion changes, this approach may miss certain transition points, resulting in only limited gains. The cluster-based HDBSCAN method disrupts the temporal structure inherent manipulation tasks, leading even worse policy performance compared to human labeling. Notably, both CoTPC and our GAP framework employ the CPD algorithm for phase estimation and generally outperform human annotation across most tasks, suggesting that automated phase estimation can capture motion transitions more effectively than manual labeling. However, CoTPC relies on simple cosine distance, which lacks the ability to represent the nuanced characteristics of robot motion, thus yielding less significant performance improvements compared GAP. our approach, the motion-consisse distance defined in Equation 4 captures spatial position changes, rotational changes, and gripper opening state changes, enabling more accurate and efficient prediction of motion transitions.
To further understand the performance gap between GAP and human labeling, we calculated the precision and recall between GAP outputs and human labels. Our results show that GAP achieves a high recall (97% average) but relatively lower precision (73% on average), suggesting that the CPD algorithm not only accurately detects semantic changes during manipulation but also identifies motion transitions that may be overlooked by humans. This experiment highlights that method offers an effective and efficient approach, given that obtaining ground truth phase transition labels is infeasible due the complexity and diversity of manipulation tasks.
15

---

Published as a conference paper at ICLR 2026
Table 6: Performance Comparison between GAP and other decomposition methods.
Suite
Mera- World
RoboSuite
Task
Method
pick-place
assembly
disassemble
push-wall
bin-picking
put hammer into drawer
stack
threading
Vision-only
92%
85%
64%
6756
Human
9556
835%
8856
68%
65%
88%
70%
47%
HDBSCAN
CoTPC erar. 4024
91%
85%
90%
66%
62%
85%
69%
504
GAP (Owrs)
94%%
96%
91%
73%
70%
91%
77%
52%
B.2 ABLATION ON TEMPORAL NETWORK
We further validate the effectiveness the LSTM component by conducting comprehensive ablation studies. Specifically, we compared the LSTM-based phase transition modeling with the following alternatives:
Fixed p: For the phase transition indices output by CPD, we applied a fixed gradient adjustment magnitude p to the corresponding samples during policy learning.
Smooth: Following KOI Lu al. 2025). modeled the transition process by treating the CPD output index as the mean of a Gaussian distribution.
The results in Table show that while alternative methods such as smoothing may serve as substitutes for modeling transitions, they regretfully fail to provide significant performance improvements.
Table 7: Ablation Studies on the LSTM component.
Task
put hammer into drawer
threading
cube
Method
Vision-only
86%
44%
70%
Fixed p=0.3
77%
44%
65%
Fixed p=0.5
89%
48%
75%
Fixed p=0.7
82%
38%
65 %
Smooth
88%
46%
80%
GAP (Ours)
91%
52%
85%
Additionally, demonstrate the LSTM's effectiveness in eliminating potential estimation errors, we conducted noise injection experiments by randomly selecting direction and shifting each CPD output index with a probability of 0.5, repeating this shift process recursively. As shown in Table 8 even when considerable noise is injected into the CPD outputs, the policy performance remains robust. We attribute this robustness the LSTM's capability to model continuous transitions, which effectively alleviates the impact of errors introduced by the CPD algorithm.
Table 8: Noise Injection Experiments.
Task
put hammer into drawer
threading
cube
Method
Vision-only
86%
44%
70%
Concatenation
79%
34%
55%
Noise-Injected
88%
49%
85%
GAP (Ours)
91 %
52%
85%
16

---

Published as a conference paper at ICLR 2026
Table 9: Ablation Studies on a and B.
Task
assembly
threading
put the banana in the box
Method
Vision-only Concatenation
83%
44%
55%
75%
33%
50%
a = 0.5, 2
93%
49%
65%
a 2, 2e 23-
91 %
57%
60%
1,3 10c
88%
51%
60%
GAP (a 1, = 10-3)
94%
53%
65%
B.3 ABLATIONS ON HYPERPARAMETERS
To validate the effectiveness and robustmess of our proposed GAP framework, we conduct compre- hensive ablation studies the key hyperparameters of the gradient adjustment.
First, we examine the selection of hyperparameters a and f3, which serve to balance the relative contribution of each term in Equation 4, thus providing a more accurate measurement of whether the directions of these changes are consistent. As described Section A.T] we set a so that displacement consistency and rotation consistency consistency is measured via sign function (which outputs or 1), we set 3 = 2e -3 to ensure this term would not overshadow the first two terms. The results in Table show that when is small, displacement consistency contributes more to overall motion consistency, which leads to higher success of tasks where displacement dominates, such "assembly" and "put the banana in the box". In contrast, when a 2, the final policy performs better on the "threading" task, where rotation is more important. Increasing 3 10 10 to emphasize the importance gripper opening does not yield substantial improvement. Notably, all three sets of hyperparameter choices result in prsiion-roprioccttionn policies that outperform policies without GAP, and are also superior to vision-only policies, indicating that our proposed GAP method is robust to hyperparameter selection.
equally weighted. Since gripper opening
We also perform ablation studies on the À in Equation In our GAP framework, we set À 0.3 by default to control the degree of gradient adjustment. As shown in Table [O] the policy performance not highly sensitive to changes insufficient attenuation of the proprioception gradients, causing the policy performance to become similar to or even worse than the simple concatenation bascline. Conversely, if is set too large, leads to instability and even collapse during policy training.
between 0.2 and 0.4. However, setting À too small results in
Table 10: Ablation Studies X.
Task
assembly
threading
put the banana in the box
Method
Vision-only Concatenation
83%
44%
55% 50%
75%
33%
= 0.1
77% %
41 %
50%
= 0.2
91 %
57%
60%
= 0.4
96%
49%
65%
= 0.8
70%
37%
50%
GAP (A- 0.3)
94%
53%
65%%
The impact of stages of applying GAP has been evaluated as well. In all of our experiments, policies trained for 100 epochs. We apply GAP during the first 50 epochs avoid excessive modulation, which may cause unstable gradient updates and potentially lead to policy collapse. Table [I presents the impact of the number of training stages æ during which GAP is applied the final policy
17

---

Published as a conference paper at ICLR 2026
performance. Applying GAP in the early stages of training improves the performance of vision- proprioception policies. However, the gradient updates for proprioception are suppressed for too many epochs, the performance of policies deteriorates.
Table 11: Ablation Studies on the number of stages of applying GAP æ.
Task
assembly
threading
put the banana in the box
Method
Vision-only Concatenation
83%
44%
55% 50%
75%
33%
2' = 10
77%
38%
50%
2 = 30
88%
47%
60%
2' = 70
83%
45%
55%
2' = 90
72%
35%
50%
GAP (x 50)
94%
53%
65%
C ADDITIONAL EXPERIMENTS
C.I LINEAR-PROBING EXPERIMENTS ON THE VISUAL FEATURES
To directly evaluate the visual features and verify whether our proposed GAP framework enhances the learning of visual representations, conduct finear-prbbig experiments the vision features extracted before and after applying GAP. Specifically, we extract and freeze the visual branch of the ssiionprrorioceptioon policy and train separate policy head to predict actions using only these frozen visual features. This linear- probaingooinggggo setup ensures that any performance differences directly reflect the quality of the learned visual representations.
These policies are evaluated in both in-distribution and -uttof-distribution settings, where object positions test time differ from those in the training data. As shown in Table T2 the visual branch trained with GAP significantly outperforms that without GAP in both in-distribution and out-of-distribution scenarios across different tasks.
Table 12: Linear- Probarr Experiments on the Visual Features.
Task
assembly
assembly (OOD)
threading
threading (OOD)
Method
Visual branch without GAP
61%
45%
26% 41% %
14%
Visual branch with GAP
74%
69%
27%
C.2 MORE OUT-OF-DISTRIBUTTON (OOD) EVALUATIONS
To further assess the robustness of GAP under diverse visual distribution shifts, we conducted additional out-of-distribution experiments beyond the object placement variations reported in Table Specifically, we evaluated three additional types of visual perturbations:
Table Color: The color of the table was modified during testing, altering the overall scene appearance.
Object Color: Objects with different colors from those used during training were employed, introducing color-based perturbations.
Lighting: Colored spot lighting was introduced to create additional visual disturbances during testing.
18

---

Published as a conference paper at ICLR 2026
The results in Table 13 and Table 14 show that GAP consistently outperforms baseline methods under all tested OOD conditions. Among different perturbations, placement changes exert the most significant impact performance, underscoring the importance of accurate target object localization for successful manipulation. While concatenation benefits from proprioceptive information that remains unaffected by visual disturbances, the lack of precise position perception results in overall suboptimal performance.
In contrast, GAP achieves superior performance compared vision-only approaches, highlighting that leveraging proprioceptive information in collaboration with vision enhances policy robustness under challenging OOD scenarios.
Table 13: OOD evaluation of the task "assembly" in Meta- Meta-World.
Task
Placement
Table Color
Object Color
Lighting
Method
Vision-only
78%
80%
75%
73%
Concatenation
62%
72%
76%
74%
GAP (Ours)
88%
85%
83%
90%
Table 14: OOD evaluation of the task "cube" in the Real- World.
Task
Placement
Table Color
Object Color
Lighting
Method
Vision-only
60%
65%
60%
55%
Concatenation
35%
50%
55%
50%
GAP (Ours)
75%
80%
85%
80%
C.3 EXPERIMENTS WITH VARIOUS POLICY HEADS
The previous experiments have demonstrated the effectiveness of our GAP on policies with MLP- based policy heads. Additionally, we conducted experiments using the popular UNet-based diffusiio policy head Chi et al. (2023) and reported the results in Table [5] The results show that policies using a ceffuusion-bassee policy head are generally better those with MLP-based heads, while applying the GAP method still outperforms vision-only policies and other methods.
Table 15: Comparisons with other methods in simulated environments. Vision-Proorioccptioon policies are equipped with UNet-based diffusion policy head.
Suite
Meta- World
RoboSuite
Task
disassemble
push-wall
bin-picking
stack
threading
Method
Vision-only Concatenation
92%
70%
66%
73%
51%
82%
62%
51%
62%
40%
MS-Bot Feng al. (2024)
93%
75%
72%
77%
58%
Aux Fu et al. (2024
93%
71%
75%
72%
55%
Mask Liu al. 2004
87%
83%
67%
66%
53%
GAP (Ours)
97%
88%
81%
87%
63%
C.4 EXPERIMENTS OF EXTENDING GAP TO ON-POLICY UPDATES
To verify whether GAP can be effectively applied on-policy updates, conduct experiments using online learning settings in this section. In these experiments, we use vision- ciroon-prpriicpttion policies initialized with behavior cloning and train the LSTM to predict motion transition probabilities in
19

---

Published as a conference paper at ICLR 2026
real-time during exploration, thereby adjusting proprioception gradient updates dynamically in online learning. This setup allows us to assess whether GAP maintains its effectiveness when the policy is updated based on newly collected experience, rather than fixed offline dataset.
Table 16: Experiments with On-Policy Updates.
Task
assembly
bin-picking
threading
Method
Vision-only
88%
71%
50%
Concatenation
78%
50%
37%
GAP
94%
82%
61% %
The results in Table 16 demonstrate that policies trained with online learning are more robust, and GAP can effectively applied to on-policy updates across different tasks and environments. Additionally, the online learning experiments demonstrate that GAP does not over-dampen necessary proprioception reliance, as the policies maintain strong performance while still benefiting from the enhanced visual representations.
D TASK DESCRIPTIONS
In this subsection, we provide detailed descriptions of tasks we evaluated and visualize additionally simulation tasks in Figure 7 The evaluations comprehensively cover a wide range of manipulation tasks, including simple pick-and-place tasks, contact-rich tasks and includes one-arm and dual-arm robot setups.
tasks, as well as long-horizon and
pick-place: The robot arm first needs locate the position of the cylinder, then move toward it. After grasping the object, it locates the target circle and moves the object to the target position.
The robot arm first locates the position of the part, then moves toward it. After grasping the part, it locates the position of the base, moves the part above it, and finally assembles them.
assembly:
disassemble:
The robot arm first locates the position the part, then moves toward After grasping the part, it disassembles the part from the base.
push-wall: The robot
first locates the position of the cylinder, then moves toward it. After grasping it, the arm pushes it to target point behind the wall. The target point is not visible through visual observation, but its location remains fixed across all evaluations.
bin-picking:
The robot arm first locates the position of the green cube, then it. After grasping it, the arm removes from the red container and finally places it into the blue container.
toward
put hammer into drawer: The robot arm first locates the position of the drawer and then moves toward it. After opening the drawer, it locates the position of the hammer, grasps it, places it inside the drawer, and finally closes the drawer.
The robot arm first locates the position of the red cube and then moves toward After grasping it, it locates the position of the green cube and finally stacks the red cube on top of the green cube.
stack:
The robot arm first locates the position of the pin and then moves toward After grasping it, it locates the position of the pinhole, then moves and rotates to align the pin with the hole, and finally inserts the pin into the hole.
threading:
press button: The robot arm first locates the position of the button, then moves toward it, closes the gripper, and presses the button.
20

---

Published as a conference paper at ICLR 2026
cube:
The robot arm first locates the position of the cube, then moves toward it. After grasping the cube, it locates the drawer and finally places the cube in the drawer.
use rag to sweep table: The robot arm first locates the position the rag, then moves toward the rag. It rotates the arm to grasp the rag, takes it down from the shelf, and finally rotates the rag to sweep the table.
handover: The robot first locates the position of the rectangular block, then its right arm moves toward it. After grasping the block, the right arm moves it to the center, where the left arm takes over the block handover. Finally, the left. arm places it on the left side of the table.
put thermos into bag: The robot first locates the position of the bag, then its right arm moves toward lifts it, and rotates the wrist open the bag. Next, robot locates the thermos, uses the left arm to grasp it, and puts it into the bag.
lift lid and pour: The robot first locates the position of the cup, then its left arm moves toward grasp and lift the lid. Next, robot locates the graduated cylinder, uses the right arm to grasp it, moves it above the cup, and finally rotates the cylinder to pour.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
To improve the clarity and quality, this work uses LLMs for refining grammar.
21

---

Published as a conference paper at ICLR 2026
pick-place
assembly
disassemble
push-wall
bin- bin-picking
hammer
stack
threading
Figure Visualization of tasks we evaluated in simulation environments.
22


Reviews:
[{"review_id": "HjBuiGgY9S", "reviewer": "Reviewer_mBpw", "summary": "This paper explores an efficient strategy for combining vision observations with proprioception observations in robotic manipulation. The authors first empirically show that naïve vision-proprioception policies tend to underutilize vision during motion-transition phases, which degrades performance compared to pure vision policies. To address this, they propose a Gradient Adjustment with Phase-guidance (GAP) method that dynamically scales the proprioceptive gradient updates during training, particularly for parameters related to motion-transition estimation. Experimental results demonstrate that GAP effectively improves both in-distribution and out-of-distribution (OOD) performance.", "questions": "1) What is the reasoning behind the choice of GAP hyperparameters ($\\alpha $, $\\beta$, and stage schedule)? How sensitive is the method to these values?\n2) What is the relationship between GAP and GradNorm-based gradient balancing? Would a GradNorm-style adjustment (e.g., scaling by grad/∥grad∥) yield similar improvements?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper tackles an important and practical challenge in robot learning—how to effectively combine vision and proprioception for efficient and accurate manipulation.\n2. The empirical findings are well-motivated, and the proposed gradient adjustment method, GAP, is simple, interpretable, and effective.\n3.  Both simulation and real-world experiments demonstrate the effectiveness of GAP. The approach shows consistent gains over both vision-only and vision-proprioception baselines, and the improved OOD performance is particularly promising.", "weaknesses": "Overall, this is a good paper for me, but there are a few minor concerns:\n1. The paper lacks a deeper investigation into the form of the gradient adjustment function $(1-p)$ and the choice of key hyperparameters. Since the paper mentions that gradient adjustment is applied only during the early stage of training, it would be beneficial to discuss the rationale for choosing specific hyperparameters ($\\alpha $, $\\beta$, and the number of stages for applying GAP) and to include an ablation study on parameter sensitivity.\n2. GAP adjusts gradients for certain parameters of the network. This may have connections to GradNorm-based balancing methods [1]. For instance, if early in training the gradient magnitude of the proprioceptive branch is larger than that of the vision branch, then GAP might behave similarly to gradient normalization. It would be helpful to clarify whether GAP implicitly aligns with GradNorm principles. Including statistics of gradient norms for both branches and showing how the adjustment parameter $p$ evolves during training would strengthen the analysis.\n\n[1] Chen, Zhao, et al. “GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks.” ICML 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pLWcNEapUM", "reviewer": "Reviewer_eFq6", "summary": "The paper argues that in vision+proprioception policies trained by BC, proprio signals dominate optimization (especially at motion-transition phases where vision should matter for target localization) so the visual pathway under-trains and generalization suffers. They propose GAP, which (i) detects motion-consistent phases via CPD on proprio traces, refines transition probabilities with an LSTM, and (ii) down-scales the proprio branch’s gradients during transitions to re-balance learning. GAP improves success rates across Meta-World, RoboSuite, and several real tasks; it also helps a VLA (Octo) variant.", "questions": "### Q1: Does GAP play well with on-policy updates (SAC/BCQ) where exploration changes state visitation?\n\n**Action:** Maybe add a small on-policy appendix experiment, and a disturbance test (pushes/perturbations) to ensure GAP doesn’t over-dampen necessary proprio reliance.\n\n### Q2: Is Octo-VP better because the visual branch actually learned more useful features, or simply because the policy became less overfit to proprioception?\n\n**Action:** Linear-probe vision features pre/post GAP and report OOD splits where only visual scene factors change.\n\n### Q3: Does GAP transfer across arms/cameras?\n\n**Action:** Train on robot A, test on robot B (and different wrist cameras) with no re-tuning. This is hard so not required.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The key idea is that during optimization, proprio-encoder parameters are updated as $\\omega_s^{j+1} \\leftarrow \\omega_s^j - \\lambda (1 - \\rho)\\eta \\nabla_{\\omega_s^j} \\mathcal{L}_{\\text{BC}}$, where $\\lambda$ controls scaling, $\\eta$ is the learning rate, and $\\rho \\in [0,1]$ measures transition likelihood. A higher $\\rho$ (likely transition) down-scales proprio gradients, forcing the visual encoder to learn those steps more effectively. GAP improves success rates across simulated and real tasks (Meta-World, RoboSuite) and enhances vision–proprio fusion even in pretrained VLAs such as Octo. The approach is simple yet effective, but relies on proprio-derived phase signals that might misidentify transitions and suppress useful proprio cues.\n- The phase-guided reweighting is simple yet principled. It directly addresses the modality competition problem well known in multimodal learning. The LSTM refinement of $\\rho_t$ mitigates the discretization artifacts introduced by CPD.  Together, these are reasonable, low-friction additions to standard behavioral cloning pipelines.\n- Key findings of this work include (i) vision-proprio policies can underperform vision-only due to optimization bias toward concise proprio signals; GAP reverses this by reweighting gradients at the right time; (ii) transition phases are the crux: improvements concentrate where target localization is required. Intervention plots show reduced degradation after GAP.", "weaknesses": "- Hyperparameters ($\\alpha$, $\\beta$, $\\lambda$) and LSTM size require task-specific tuning, which is a limitation of this work.\n- From my understanding, CPD/LSTM finds proprio change points, not visual evidence of new targets. Showing that $\\rho_t$ tracks visual uncertainty (e.g., entropy over detectors) would strengthen the claim.\n- One minor concern is that the current phase-detection method and the gradient-adjustment strategy both rely on proprioceptive signals (joint positions, velocities, gripper states). Because the change-point detector (CPD) and the transition probability $\\rho_t$​ are derived entirely from proprio data, the system may inadvertently bake in proprioceptive priors (already dominate the optimization process). This creates a potential feedback loop: proprio cues define where \"transitions\" occur, and then proprio gradients are scaled according to those same cues, reinforcing proprio's influence instead of balancing it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ki07SdLQ3Q", "reviewer": "Reviewer_X7TA", "summary": "This paper proposes GAP, a gradient adjustment algorithm that uses phase-guidance to dynamically modulate proprioception's gradient magnitude during training. This paper identifies a critical issue in vision-proprioception policies: the suppression of visual modality learning during motion-transition phases due to the dominance of more concise proprioceptive signals in optimization. Comprehensive experiments in simulation and real-world settings, across single-arm, dual-arm, and VLA models, demonstrate that GAP significantly improves policy robustness and generalization.", "questions": "1. Are the analysis and methods presented in this paper still effective when applied to more advanced policy architectures, such as generalist VLA models built upon imitation learning frameworks with multimodal large language models as their backbone?\n2. The paper also includes experiments with a transformer-based VLA (Octo). How did the authors determine parameter attribution and differentiate between parameters belonging to the vision chunk versus the proprioception chunk? Given that Octo's architecture is highly integrated, requiring deep fusion of both observation modalities within the transformer, this distinction seems non-trivial.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A detailed analysis was conducted to investigate the reasons behind the suppressed learning of the visual modality;\n2. Phase-guided gradient adjustment offers a principled approach to dynamic modality balancing;\n3. Comprehensive evaluation across different environments, platforms, and models.", "weaknesses": "The assessment lacks a direct evaluation of whether the visual modality is better utilized. While performance metrics provide some evidence, incorporating qualitative visualizations or explainable analysis of the GAP-trained model would more effectively demonstrate the method's effectiveness to readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q16mKVZHQM", "reviewer": "Reviewer_rSWj", "summary": "The paper claims that in vision-proprioception policies, the proprioceptive modality, which is relatively easier to learn, suppresses the visual modality during the motion-transition phases of a task. The authors propose a gradient adjustment algorithm that effectively addresses this issue, supported by well-designed experiments in both real and simulated environments. In particular, the proposed method first performs an initial segmentation of the task's motion-transition phases using a Change Point Detection (CPD) algorithm on proprioceptive data. It then uses a temporal model, such as an LSTM, to refine this and calculate the probability of each time step belonging to a motion-transition phase. Based on this probability, it then applies dynamic gradient adjustment to regulate learning from proprioception, thereby preventing the suppression of the visual modality. The authors validate the effectiveness of this method across various settings, including simulation, the real world, and with different architectures like conventional policies and Vision-Language-Action models.", "questions": "1. Please include comparison plots (in the supplement) of the loss curves with and without your method for a given task to clearly demonstrate the effectiveness of the proposed approach.\n2. Please add some plots to show how the phase changes  (the estimated transition probabilities) along with scene RGB images for validation. \n3. In Figure 4, the regulated policy still shows slightly lower performance than the vision-only policy during some motion transition stages. Could you analyze the reason for this?\n4. In Table 4, for the “disassemble” task, the success rate of the FiLM fusion method is significantly lower than other methods. What might be the reason for this specific underperformance?\n5. In the real-world bimanual task experiment, why were the policies for the left and right arms trained separately instead of as a single, unified policy?\n6. The paper states that for training stability, gradient adjustment is only applied during the early stages of policy learning. Could you provide a more precise analysis of this reasoning, ideally supported by evidence such as training-related plots?\n7. To further enhance the paper’s replicability and transparency, the reviewer recommends submitting the source code as a part of the supplementary material.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper provides a crucial diagnosis for the inconsistent performance of vision-proprioception policies, a known but previously under-explained problem.\n2. The paper's argument is well-structured and easy to follow, building a logical methodology directly from its clear problem diagnosis.\n3. The paper validates the effectiveness of the proposed methodology by demonstrating high task success rates throughout its extensive experiments and ablation studies.", "weaknesses": "1. Reliance on Proprioceptive Cues: The reliance on proprioceptive cues for phase detection could be a failure point in tasks where transitions are defined by external, non-physical events (e.g., waiting for a visual cue), as the algorithm may not detect these critical moments.\n2. Lack of Direct Evidence for Phase Estimation: The paper's central mechanism, i.e., the phase estimation, is not empirically validated with visualizations. The absence of plots showing the phase changes along with RGB images and/or the estimated transition probabilities makes it difficult to verify that this core component is functioning as intended, independently of the final task outcomes.  The final success rates are not enough to validate the rationale of this proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)