Title: Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention
Venue: NeurIPS 2025
Accepted: yes
Keywords: video object segmentation

egocentric

vision and language

multi-modal

causal learning
Abstract: Egocentric Referring Video Object Segmentation (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos. This task is critical for understanding egocentric human behavior. However, achieving such segmentation robustly is challenging due to ambiguities inherent in egocentric videos and biases present in training data. Consequently, existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets and fundamental visual confounding factors of the egocentric perspective, such as rapid motion and frequent occlusions. To address these limitations, we introduce Causal Ego-REferring Segmentation (CERES), a plug-in causal framework that adapts strong, pre-trained RVOS backbones to the egocentric domain. CERES implements dual-modal causal intervention: applying backdoor adjustment principles to counteract language representation biases learned from dataset statistics, and leveraging front-door adjustment concepts to address visual confounding by intelligently integrating semantic visual features with geometric depth information guided by causal principles, creating representations more robust to egocentric distortions. Extensive experiments demonstrate that CERES achieves state-of-the-art performance on Ego-RVOS benchmarks, highlighting the potential of applying causal reasoning to build more reliable models for broader egocentric video understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses Egocentric Referring Video Object Segmentation (Ego-RVOS), a task focused on segmenting objects involved in human actions in egocentric vision, guided by language queries. The task is challenging due to language biases and limitations of the egocentric view. In this work, the authors introduce Causal Ego-Referring Segmentation (CERES), a novel framework leveraging causal inference. CERES applies principles based on back door adjustment and front door adjustment to counteract the confounding issues. The proposed model was evaluated on three public egocentric vision datasets and achieved state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The clarity of this paper is very good. Overall, this paper is well-written, with clear format, tables and math.\n2. The proposed method is evaluated across multiple datasets, achieving state-of-the-art performance.\n3. The ablation study results reported in this work support the framework design choices and helps understanding.\n\nMajor weakness:\n\nN/A\n\nMinor weakness:\n\nLine 31: \"because because\".\n\nSome tables (1,2,4) may be inaccurate and requires clarification (see questions).\n\nOver all, this paper is well-written. It introduces an interesting idea and the experimental results show potential . Due to limited understanding behind the math in causal reasoning, my current rating for this paper is borderline accept, and will be adjusted after rebuttal.\n\nQUESTIONS:\n1. In table 1, why is \"Ours\" in bold but have higher mIoU⊖ and cIoU⊖?\n2. In table 2, ReferFormer+ the second best mIoU⊖, why is \"ours\" underlined?\n3. In table 4, adding LBD shows clear advantage when not using DAttn and MAttn. However, the difference in numbers between the last two rows (w/ and w/o LBD) is small. Are there more analyses that justify the effectiveness of LBD in the final design?\n4. This work identified two confounding sources: observable language bias and unobservable visual confounding. Is there existing work or analysis supporting these as the primary confounding factors?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework, CERES, for Egocentric Referring Video Object Segmentation (Ego-RVOS). The core idea is to improve model robustness by addressing two sources of spurious correlations using principles from causal inference. \n\nFirst, to counteract language biases in datasets (e.g., \"knife\" always co-occurring with \"cut\"), the framework uses back-door adjustment on the text features, which fix the bias using dataset statistics. \nSecond, to handle visual confounding factors common in egocentric video (e.g., motion blur, occlusions), it applies front-door adjustment. A key component of this is a novel visual mediator that uses geometric depth features, which are assumed to be more robust to confounders, to guide the aggregation of semantic RGB features via an attention mechanism. \n\nThe authors demonstrate that CERES achieves state-of-the-art performance on several Ego-RVOS benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The performance of this paper is good, showing significant improvement over the baseline ActionVOS.\n- The method used in this paper is effective, as it takes into account the bias in text-based datasets and integrates depth information visually.\n\n\nWeakness：\n- This paper lacks a well-defined motivation for incorporating depth information, which provides the largest improvement, and makes an unfair comparison with the baselines. In Table 4, it can be observed that the introduction of depth information through DAttn yields the most significant improvement, with the mIoU increasing from 59.9 in the first row to 63.3 in the third row. Moreover, it is widely acknowledged that depth information provides a strong prior for segmentation, as evidenced both intuitively and through experiments such as those in DepthAnything. Therefore, I think the way this paper addressing motion blur is somewhat trivial, and I remain skeptical of the theoretical framework presented in this paper.\n- Using a pre-trained model instead of training from scratch reduces the originality of this paper. Line 269 mentions the use of a pre-trained model, and the number in the first row of Table 4 is identical to that of the ActionVOS method with a Res101 backbone in Table 1. This indicates that the pre-trained model from ActionVOS was used. In other words, the method's performance improvement is built upon the combination of a pre-trained model and depth information, which, to some extent, affects the validity of this paper.\n- This illustrations, especially the egocentric images, are very difficult to see clearly, including Figure 1, Figure 3, and Figure 4.\n\nQUESTIONS:\nMy major concern is that this paper should have been built upon a comprehensive set of suitable baselines due to its heavy reliance on the pretrained models, followed by incremental experiments incorporating +CERES to demonstrate its generalizability, rather than being presented as a standalone SOTA method for comparison. \nFor example, this paper demonstrates the feasibility on ActionVOS, but does not provide evidence of its applicability to other pre-trained models, such as RefFormer or HOS.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper proposes a new RVOS method, named CERES. CERES addresses two challenges: confounders in visual images and language representation biases, respectively. Specifically, the visual confounders are addressed by intergrating depth data, while the language presentation biases is addressed by intergrating data statistics.

STRENGTHS AND WEAKNESSES:
Strengths:
1. CERES achieves state-of-the-art results and significantly outperforms ActionVOS, demonstrating its strong effectiveness.
2. The consideration of biases in both visual and textual information is novel and interesting.
3. CERES performs well across different network backbones, highlighting its practical applicability and flexibility.

Weaknesses:
1. The integration of data statistics and depth information seems somewhat detached from the earlier causal bias analysis. The method mainly improves performance by adding depth data and data statistics to the RVOS model through mechanisms like attention or concatenation, but the connection to the prior bias framework is not clearly established.
2. While ActionVOS includes analysis of state-changing objects, it remains unclear whether CERES can be effectively applied to such scenarios.
3. It would be valuable to understand how CERES performs in open-vocabulary settings, particularly on unseen objects or actions.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a causal inference-based framework, CERES, to address robustness issues in Egocentric Referring Video Object Segmentation (Ego-RVOS). Ego-RVOS aims to segment target objects involved in human actions based on language queries (e.g., \"knife used to cut carrot\") in first-person videos. CERES introduces dual-modal causal intervention, including a Linguistic Back-door Deconfounder (LBD) for language debiasing and a Visual Front-door Deconfounder (VFD) for visual confounding mitigation. Experiments on VISOR, VOST, and VSCOS datasets validate CERES's state-of-the-art performance. Ablation studies confirm the efficacy of each module, with theoretical proofs and implementation details provided in the appendix.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Standardized Presentation. Well-structured writing, clear figures, and intuitive framework diagrams.\n2. Rigorous Technical Foundation. Complete theoretical derivations (e.g., MMSE optimality proof in the appendix) and transparent implementation details.\n3. Practical Research Value. The focus on robust Ego-RVOS addresses a meaningful challenge in egocentric vision with real-world applicability.\n\nWeaknesses:\n1. Ambiguous Causal Rationale for Language Bias. Is the \"Language Bias\" solely defined as \"certain object categories frequently co-occur with specific actions\"? Is all knowledge of such co-occurrences harmful? Some correlations represent valid prior knowledge (e.g., \"open\" strongly associates with objects like \"fridge\" or \"cabinet\"), which can be beneficial for prediction. The paper lacks a clear distinction between harmful spurious correlations and potentially beneficial prior knowledge learned from data statistics. What specific types of \"spurious correlations\" does CERES target?\n2. Insufficient Novelty in Core Methodology. The key components of the VFD (Depth encoder, Memory-bank) and LBD modules rely heavily on established techniques. The overall RVOS segmentation backbone shows minimal innovation. Moreover, the Attention-Linear Family (ALF) assumption for mediator fusion may constrain representation power. Exploring nonlinear fusion mechanisms (e.g., gating) is recommended, or at minimum, ablation studies comparing ALF to nonlinear alternatives should be included.\n3. Incomplete Computational Analysis. Integrating the Depth encoder and Memory-bank (MAttn) likely increases computational cost. The paper lacks metrics quantifying this impact (e.g., inference speed/FPS, FLOPs, parameter count comparisons vs. baselines).\n4. Inadequate Experimental Explanation. For Fig.4: It is unclear how depth information specifically resolves occlusion. Visualizations (e.g., attention maps or depth overlays) illustrating the role of depth in handling occlusion are needed. And depth maps from Depth Anything V2 can be error-prone. The paper does not analyze CERES's sensitivity to depth estimation errors. Experiments testing robustness under noisy/depth maps or with weaker depth estimators are crucial.\n5. Inflexible & Dataset-Dependent Temporal Context. The fixed window size (W=5) for MAttn relies on a short-term stationarity assumption. Its performance (Fig. 5) appears highly dataset-dependent. This fixed window is likely suboptimal for sequences with long-term dependencies (e.g., objects leaving and re-entering the view). An adaptive window mechanism is strongly recommended instead of a fixed value.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}