Title: When Would Vision-Proprioception Policy Fail in Robotic Manipulation?
Venue: NeurIPS 2025
Accepted: no
Keywords: Robotic Manipulation

Vision-Proprioception Policy
Abstract: Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported confused observations that vision-proprioception policies frequently suffer from poor generalization. In this work, we attempt to answer the question: When would vision-proprioception policy fail? To this end, we conducted temporally controlled experiments and found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy fails to take effect. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to improved generalization of vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies for robotic manipulation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the failure of imitation learning policies that leverage both vision and proprioception information as inputs. Empirical results demonstrate that policies conditioned on both modalities tend to fail in motion-transition phases where the policy has to pay attention to detailed visual cues. Motivated by this result, the authors propose the Gradient Adjustment with Phase-guidance (GAP) framework. GAP first identifies motion-transition phases in demonstration trajectories using Change Point Detection (CPD) and an LSTM. The learned LSTM outputs a continuous scalar score that predicts how likely the robot is in a motion-transition phase. Based on this prediction, the magnitude of the gradient of the imitation learning loss with respect to the proprioceptive inputs is modulated. This modulation encourages the policy to pay more close attention to visual cues in motion-transition phases. Experiments in both simulation and the real-world demonstrate the effectiveness of GAP.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* The paper first conducts an investigative study on when vision-proprio policies tend to be more vulnerable than vision-only policies to form a scientific hypothesis. This hypothesis leads to the design of GAP, which is shown to be effective through a series of experiments. Such a hypothesis-driven approach to robot learning research is highly appreciative.\n\n* The proposed GAP framework seems versatile and applicable to various imitation learning architectures (e.g., MLP-based, diffusion, flow-matching, etc.).\n\n* The paper conducts thorough experiments, including ablation of multiple policy architectures and fusion techniques across multiple tasks in both simulation and the real-world. \n\n\n**Weaknesses:**\n\n* Lack of key information in the experiments: I was unable to identify the number of rollouts per policy per task in the Meta-World and RoboSuite simulation benchmarks. Lack of sample size makes it impossible to judge the significance of empirical results. (For instance, 90% vs. 96% success rates may be insignificant under 50 rollouts but are likely significant with 500 rollouts, due to smaller statistical noise.)\n\n* The OOD experiment in Section 5.3 is a bit contrived, since the authors only test GAP under different object placements. More visual distribution shifts such as changes in background color, object color, or object shape might lead to poorer performance of GAP, since GAP-trained policies rely less on proprioceptive information that is unaffected by those visual distribution shifts.\n\nQUESTIONS:\n* It remains a bit unclear how CPD and LSTM are leveraged together to perform motion-transition phase estimation. Specifically:\n\n  * What is the target output for the LSTM training? Is it a binary 0-1 label indicating the motion-consistent and motion-transition phases as computed by the CPD algorithm?\n  * Could we possibly eliminate the LSTM training phase, for instance by simply smoothing the motion phase outputs of CPD? The versatility of GAP would improve if it did not require the additional LSTM training phase.\n\n* I wonder whether the ignorance of vision inputs is still an issue for models like Open-VLA, where strong general-purpose vision/language features such as DinoV2 and SigLIP are employed. Some comments on this point would be appreciated.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the impact of proprioceptive measurements in visual-proprioceptive policy learning. The paper hypothesizes that the gradients from the proprioceptive signals dominate the optimization process during motion transition phases, where the visual cues are less prominent, leading to under-utilized visual information. This hypothesis is supported by several experiments in both real-world and simulation settings. The paper further proposes Gradient Adjustment with Phase-guidance (GAP) to alleviate such an issue. The experimental results show that GAP greatly improves the success rate of visual-proprioceptive policies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper shows that visual-proprioceptive policy can suffer from performance degradation when visual cues are subtle compared to proprioceptive signals. It further demonstrates that such an issue is caused by imbalanced gradient flows. In my opinion, this finding provides great insights into training visual-proprioceptive policies.\n\n2. The proposed GAP method is simple, yet it is shown to effectively improve the success rate of visual-proprioceptive policies. \n\n3. The paper presents 4 different experiments to help understand the effect of GAP. The experiments show that:\na) GAP can improve the performance of visual-proprioceptive policies. \nb) GAP enhances the focus on visual inputs, allowing better generalization to different object locations. \nc) GAP is compatible with vision-language-model Octo-VP.\nd) GAP can be applied to different fusion approaches, and in general, improve the performance.\n\nWeaknesses\n1. The description of the motion transition phase in the introduction is intuitive. However, the definition is unclear. An explanation of why the motion transition phase can be captured by the Change Point Detection with heuristic rules is important for readers to understand the paper.\n2. In addition to the above point, evaluations on how well the CPD algorithm can capture the motion transition phase can greatly improve the paper.\n3. The proposed GAP method depends on a trained LSTM to predict the motion transition phase. However, there is no evidence on how well the network performs, as well as it’s effect on the overall performance.\n\nQUESTIONS:\n1. Motion transition phases can be one of the causes for gradient imbalance. Do you think it is possible to extend the proposed method to a more general framework, where the system automatically figures out when to put attention on proprioceptive signals or visual cues?\n\nLIMITATIONS:\n1. The paper primarily focuses on methods with two detached heads for visual and proprioceptive inputs. It is unclear if the same results can be obtained from systems with a shared encoder. \n2. As mentioned in the paper, all the visual-proprioceptive policies are trained on single embodiment. There is no guarantee that the results can be extrapolated to methods trained on cross-embodiment datasets.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper claims that adding proprioceptive information to VLA policies tends to hurt performance. The authors hypothesize this is because proprioceptive information is more useful when continuing the same skill, but visual information is more relevant when transitioning between skills, and the latter is down-weighted by learning algorithms in these transition phase due to the relatively subtler changes in feature values. They attempt to mitigate this by increasing the relative weighting of the gradients from visual features during motion-transition phases, which they define using a dynamic programming based change point detection algorithm on the end-effector state. \nThe approach is compared with previous ones on a series of manipulation benchmark tasks and show improved task success rate. Additional experiments also demonstrate an enhanced usage of the vision modality, and a compatibility with other VLA models and modality fusion approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This paper presents a clear narrative, with a well-defined problem, a well reasoned mechanism causing this problem based on previous work, and a clear and simple solution to address this. They present a very comprehensive set of experiments to test the efficacy and applicability of their method with experiments in real and in sim. The questions answered in the experiments section are clear and well-motivated.\nWeaknesses: The most significant weakness of this paper is the importance of the motivating problem and the marginality of the benefit it produces. In the authors’ own related work, they state that previous literature has mixed opinions of whether this is a real problem. The results in Table 1 show only minor increases in success for most tasks, possibly not being statistically significant enough to justify the importance of the underlying problem. In addition to this the paper has several grammatical errors, including the title, which could be changed to “when would visual-proprioceptive policies fail for robotic manipulation”. Other mistakes are “To support our analyze” on line 120 and the title for section 5.2.\n\nQUESTIONS:\n1. How do you reconcile the fact that previous literature is mixed on whether proprioception hurts performance and your statement that it is a real problem? I am not familiar with this as a problem in the field so definitive evidence of this could change my evaluation.\n\n2. Why do you train a network to predict transitions indicators if you are already labeling your training set with them.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, we attempt to answer the question: When would vision-proprioception policy fail? The authors conduct temporally controlled experiments and found that during task sub-phases that robot’s motion transitions, which require target localization, the vision modality of the vision-proprioception policy fails to take effect and concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. The paper proposes the Gradient Adjustment with Phase-guidance (GAP) algorithm.  The authors conduct  comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The question of when to rely on vision versus proprioception signals during policy training is a critical and timely challenge in the robot learning community. The visualizations in Figures 1 and 4 effectively highlight this problem and provide compelling motivation.\n2. To address this, the authors introduce a novel algorithm Gradient Adjustment with Phase-guidance (GAP), which adaptively modulates proprioceptive optimization. This approach enables dynamic and context-aware collaboration between visual and proprioceptive inputs within a unified policy.\n3. Notably, GAP is compatible with both conventional policy architectures and emerging Vision-Language-Action models. Its versatility and effectiveness are well-supported by extensive experiments across both simulated and real-world environments.\n\n**Weaknesses**\n\nOverall, this work presents a promising direction for advancing multimodal policy learning in robotic manipulation. However, several critical points require clarification and refinement before the contributions can be fully appreciated and appropriately scoped.  \n\n1. Overgeneralization of the Core Research Question: The central question posed—“When do vision-proprioception policies fail in robotic manipulation?”—is framed too broadly given the scope of the study. The vision-proprioception policies evaluated in this work represent only a subset of the broader policy design space, excluding other relevant architectures such as diffusion-based policies. Furthermore, the experimental tasks are limited in complexity and diversity; contact-rich interactions and articulated object manipulation, both of which pose unique challenges to vision and proprioception integration, are not addressed. As such, the authors are strongly encouraged to revise the framing of their research question and claims to better reflect the actual coverage and limitations of the current study.\n2. Lack of Analysis on Motion-Transition Phase Estimation Robustness: The Motion-Transition Phase Estimation module appears to be a critical component of the proposed GAP framework, as it governs when and how proprioception is emphasized during training. However, the paper lacks an in-depth analysis of its accuracy and impact on overall performance. Specifically:\n    - What is the standalone performance of this estimation module?\n    - How sensitive is the system to estimation errors?\n    - If ground truth phase labels were provided, how would the policy performance differ?\n\nIncluding such an ablation or comparison would greatly enhance the reader’s understanding of the module’s influence on the final results and provide insights into the robustness and generalizability of the proposed method.\n\nQUESTIONS:\n1. What is the relationship between the proposed method with guided diffusion policy [1]? While they are not the same, it seems to me there are some similarity between the two ideas. It would be great to have a connect them. Can guided diffusion be another implementation to realize the same concept as in this work?\n[1] Xu et al., Dynamics-Guided Diffusion Model for Robot Manipulator Design, CoRL 2024\n\n2. What is the standalone performance of this estimation module?\n\n3. How sensitive is the system to estimation errors?\n\n4. If ground truth phase labels were provided, how would the policy performance differ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}