Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention
Authors: Haijing Liu
Zhiyuan Song
Hefeng Wu
Tao Pu
Keze Wang
Liang Lin
Venue: NeurIPS 2025 poster ()
Keywords: video object segmentation
egocentric
vision and language
multi-modal
causal learning
Primary area: applications
TL;DR: This paper introduces CERES that uses dual-modal causal intervention (backdoor adjustment for language bias and front-door adjustment with vision-depth fusion for visual bias) to achieve robust, SOTA egocentric referring video object segmentation.

Abstract: Egocentric Referring Video Object Segmentation (Ego-RVOS) aims to segment the specific object actively involved in a human action, as described by a language query, within first-person videos. This task is critical for understanding egocentric human behavior. However, achieving such segmentation robustly is challenging due to ambiguities inherent in egocentric videos and biases present in training data. Consequently, existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets and fundamental visual confounding factors of the egocentric perspective, such as rapid motion and frequent occlusions. To address these limitations, we introduce Causal Ego-REferring Segmentation (CERES), a plug-in causal framework that adapts strong, pre-trained RVOS backbones to the egocentric domain. CERES implements dual-modal causal intervention: applying backdoor adjustment principles to counteract language representation biases learned from dataset statistics, and leveraging front-door adjustment concepts to address visual confounding by intelligently integrating semantic visual features with geometric depth information guided by causal principles, creating representations more robust to egocentric distortions. Extensive experiments demonstrate that CERES achieves state-of-the-art performance on Ego-RVOS benchmarks, highlighting the potential of applying causal reasoning to build more reliable models for broader egocentric video understanding.

Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention
Haijing Liu,
Zhiyuan Song, Hefeng Wu, Tao Pu,
Keze Wang,
Liang Lin"
Sun Yat-sen University, Guangzhou 510006, China
Liuhj660mai12..m www.. edu.cn, cn, songzhy290mai12.cm sysu. edu. wuhefengdgmail. putao30mai12 aysu. edu. en, kezewangdgmail.com linliang@iese org
Abstract
Egocentric Referring Video Object Segmentation (Ego-RVOS) aims to segment the specific object actively involved human action, as described by lan- guage query, within first-person videos. This task is critical for understanding egocentric human behavior. However, achieving such segmentation robustly challenging due to ambiguities inherent in egocentric videos and biases present in training data. Consequently, existing methods often struggle, learning spurious correlations from skewed object-action pairings in datasets and fundamental visual confounding factors of the egocentric perspective, such as rapid motion and fre- quent occlusions. To address these limitations, we introduce Causal Ego REferring Segmentation (CERES), a plug-in causal framework that adapts strong, pre-trained RVOS backbones to the egocentric domain. CERES implements dual-modal causal intervention: applying backdoor adjustment principles to counteract language representation biases learned from dataset statistics, and leveraging front-door adjustment concepts to address visual confounding by intelligently integrating semantic visual features with geometric depth information guided by causal prin- ciples, creating representations more robust to egocentric distortions. Extensive experiments demonstrate that CERES achieves state-of-the-art performance on Ego-RVOS benchmarks, highlighting the potential of applying causal reasoning to build more reliable models for broader egocentric video understanding.
Introduction
Egocentric vision, capturing the world from first-person perspective, offers invaluable data for understanding human interaction and behavior. Within this domain, Egocentric Referring Video Object Segmentation (Ego-RVOS) [36] presents key task: segmenting the specific object actively involved in human action, as identified by a natural language query combining object and action descriptions (Figure (())). Successfully addressing Ego-RVOS paves the way for machines develop a deeper comprehension of dynamic scenes, integrating visual perception, language understanding, and temporal reasoning. Prior work, such as Action VOS [36], established settings for this task, notably utilizing action descriptions alongside object names and sometimes adapting pre-trained models with specialized loss functions to focus on active objects.
However, developing robust Ego-RVOS models faces significant hurdles. Current approaches often struggle because they learn spurious correlations rather than genuine cause-and-effect relation- ships [18, 60, 50, 6, 27]. These spurious correlations stem from two main sources, as illustrated by typical failure cases in Figure 1(b). First, dataset biases often exist where certain object categories frequently co-occur with specific actions, Icading models to rely on these statistical shortcuts instead
"Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Close cereal bot
Visual occlusion
11
do
72
Earret loue 3.88)
Video Frames
C.00)
twrn
Unobservable Confounder
Front-door Adjustment
Optimization
Masks
Model
Cut corret
Rare object
Observable Confounder
Back-door Adjustment
Object Referring
Tano
knll
do
(a) Example of Ego-RVOS (b) Failures due bias
(c) Overview of the CERES causal framework
Figure I: Motivation and overview the CERES addressing biases. (a) Ego-RVOS needs segment the objects related to action (positive objects, colored) instead of ohjects unrelated to action (negative ohjects, gray). (b) Example failure cases of baseline [36] because of visual occlusion and rare objects outside the training set. (c) Our CERES from text and visual modal performs causal intervention to achieve robust Ego-RVOS.
of truly grounding the language query [5, 40, 55]. Second, the inherent nature of egocentric video introduces fundamental visual confounding factors: rapid camera movements, frequent hand-object occlusions, and perspective distortions create complex visual patterns that can mislead models, partic- ularly given the domain shift from typical third-person pre-training data [46, 19, 44, 12, 7, 56]. This reliance on superficial cues renders models brittle and unreliable.
To overcome these challenges and foster robust segmentation, we propose Causal Ego- Beo-RReerring Segmentation (CERES), plug-in causal framework that adapts strong, pre-trained RVOS backbones to the egocentric domain by employing dual-modal causal intervention.
Instead of merely learning correlations, CERES aims to identify and model the underlying causal pathways from the dual-modal imputs (vision X, text ') to the segmentation output ()), intervening to remove confounding influences. As illustrated in Figure (c)), we conceptualize the Ego-RVOS process using a causal graph and outline CERES framework. We identify two primary confounding issues: (1) For the observable language bias, stemming from dataset statistics (confounder 2), CERES applies principles inspired by backdoor adjustment [38, 39J. This aims to block the spurious path T T Z + y and estimate the direct causal effect of the text query T on the output y, P()) do(T)). (2) For the unobservable visual confounding, originating from inherent egocentric factors (L)), CERES utilizes principles based front-door adjustment [39]. This requires identifying mediator variable M that captures the causal effect flowing from vision X' to the output ) The goal is to estimate the causal effect of the visual input X' on y, P() do(2))) by adjusting for the effect mediated through M while blocking the confounding path X' < +
Implementing the front-door adjustment necessitates carefully chosen mediator M. Egocentric vi- sual features are susceptible to confounding (Lf) through factors like rapid motion and occlusion. purely visual mediator risks inheriting this confounding, potentially violating front-door requirements. We hypothesize that incorporating geometric structure can yield a more robust mediator, less sensitive to U. Therefore, propose mediator M integrating semantic visual knowledge (Ma)) with geometric depth information (Ma), both derived from X. Leveraging depth cues provides robustness against visual distortions inherent in U4 aiming to better isolate the back-door path X' M -> y CERES employs attention [60] to realize this vision-depth fusion and implement the necessary causal adjustments within an end-to-end framework. The main contributions of this work are:
We propose CERES, novel framework applying causal inference principles to tackle key robustness challenges in Ego-RVOS.
We employ backdoor adjustment concepts mitigate language biases arising from spurious correlations object-action dataset statistics.
We utilize front-door adjustment concepts, implemented via a novel vision-depth mediator, to address fundamental visual confounding inherent in the egocentric perspective.
Extensive experiments across diverse backbones demonstrate that CERES achieves state-of-the-art performance on VISOR, VOST and VSCOS datasets, significantly improving robustness against both linguistic and visual biases.

---

2 Related Work
Referring Video Object Segmentation (RVOS) Referring Video Object Segmentation (RVOS) aims to segment the object referred to by a natural language expression throughout video. Existing RVOS tasks [17, 22, 34, 43, 58J are typically constructed by annotating referring expressions on existing video segmentation benchmarks. These expressions often describe static attributes of single target object. The recent McViS dataset [13] introduces motion- based language descriptions for video object segmentation. Various methods have been proposed for RVOS [3, 14, 8, 54, 52, 24]. For instance, SLVP [32] extends RVOS to the VISOR dataset [12], while Action [36] incorporates action narrations to segment active objects in egocentric videos. Despite these advances, most approaches overlook critical challenges such as occlusion and label bias in referring expressions. In this work, we propose a causal inference- based framework to address these issues, enabling more robust and generalizable RVOS models.
Causal Inference Causal inference has become an increasingly popular tool for uncovering task causality [39]. and has been widely integrated into deep learning systems, especially in vision- language tasks such as image recognition [51, 53, 66, 65, 33], image captioning [61, 25], and visual question answering [23, 60]. A common approach is to apply adjustment techniques to mitigate the influence of confounding variables, with some studies exploring counterfactual reasoning [37, 1, 35]. In this work, we focus intervention- based methods due their practicality. However, most existing causal learning frameworks are limited to relatively simple tasks and rarely consider complex embodied settings like RVOS. Moreover, current approaches typically apply either back-door [25, 50, 64, 68] or front-door [28, 60, 61] adjustments independently across modalities, failing to account for both observable and unobservable confounders in a unified manner. Unlike prior works such as GOAT [49], which tackles confounders in vision, language, and action history, we propose the first causal framework tailored for RVOS. Specifically, we introduce a novel front-door adjustment that integrates depth information and adjacent frames features to refine segmentation decisions under occlusion. Mcanwhile, we design a back-door blocking strategy statistically correct biases in referring expressions and action labels. Our approach effectively addresses both visual and linguistic confounding effects, leading to more robust and generalizable Ego-RVOS models.
3 Preliminary
Task Formulation. The Egocentric Referring Video Object Segmentation (Ego-RVOS) task [36] requires segmenting a specific object instance involved in an action within first-person video. The input consists of an egocentric video clip, represented as a sequence of frames X {=, where x2 E TR. H x3 is the RGB frame realization at time t. Alongside the video, a language query T, realized as tra- provides both the object category name and a description of the action (e.g., "knife used to cut carrot").
u
Z
M
X
M
y
T
Ma
Vision
y
Prediction
Text
Unobservable Confounder
M
Mediator
z
Observable Confounder
Figure 2: The proposed SCM for Ego-RVOS. (Dashed lines indicate confounding paths; solid lines indicate causal paths.)
The objective is to predict a sequence of binary seg- mentation masks Y = (yr)- where each realiza- tion E (0, 1) H XW precisely delineates the pixels belonging the object instance specified by the query ( and actively participating in the described action within frame Tt.- This task demands robust integration of visual perception (AY), language understanding (T), and reasoning about object-action relationships over time.
Structural Causal Model of RVO-RVOS. To systematically address the biases inherent in Ego- RVOS, we formulate the task using Structural Causal Model (SCM) [38], as depicted in Figure 2. This model posits that the visual input X' and text query T are the direct causes the segmentation output D. However, this ideal relationship is often confounded in practice.
We identify two primary confounders. First, an unobserved confounder iI encapsulates intrinsic egocentric visual characteristics (e.g., rapid motion, occlusions) [44]. 1 affects both the visual input X and the output V/ creating a spurious backdoor path X' < 24 + y. Second, an observable confounder Z represents dataset-level statistical biases, such as skewed object-action co-occurrences [40]. Z
3

---

influences both the text queries T and the labels V, forming another backdoor path T + 2 - y. These backdoor paths lead models learn superficial correlations rather than true causal relationships.
To mitigate the visual confounding from 4, we employ the front-door criterion. This involves an intermediate mediator M that captures the causal effect from X' to y. Recognizing that purely visual knowledge might still be tainted by U, we propose a more robust mediation strategy. As shown in Figure 2, we conceptualize the visual information X' as giving rise to distinct semantic knowledge M, (what objects are present) and geometric depth knowledge Ma (their spatial layout and structure). We hypothesize that M. forms the primary pathway to an intermediate representation M (X' MM + M + y), while Ma offers a complementary, potentially more robust, pathway influencing D (X' + Ma + y). This decomposition aims to leverage the stability of geometric cues (Ma) to buttress the semantic interpretation (M,),, leading to a mediator M (or a combined influence on }) that is less susceptible to the distortions introduced by U. The specific mechanisms for realizing these causal adjustments will be detailed in our method section.
4 Methodology
The CERES (Causal Egocentric Referring- -Segmnnttttmmtnii framework implements distinct causal adjustment strategies to address the language and visual biases inherent in Ego-RVOS, as outlined in our Structural Causal Model (SCM, Figure 2). Specifically, to counteract:
Language Bias: Stemming from the observable confounder Z in the T + + y pathway, CERES applies back-door adjustment.
Visual Bias: Originating from the unobserved confounder UI affecting the X + U y pathway, CERES employs front-door adjustment. This is operationalized using mediator MM which is carefully constructed from semantic visual features (M,) and geometric depth features (Ma)) derived from the visual input X.
The subsequent sections detail the specific formulations for these back-door and front-door adjust- ments and describe their neural network implementations within CERES.
4.1 Language De-biasing via Back-Door Adjustment
Dataset statistics often correlate a textual query T with its target mask y through visible confounder 2 (e.g., frequent "knife-cut" pairs). Following Pearl's back-door criterion [39], the interventional distribution is
P(Y | do(T = ()) = Σ - = B2(P [4:)].
(1)
approximation. A modern segmenter first maps inputs to pre- activvivation scores (logits) 2t, and then applies the Softmax function to obtain probabilities. For many practical score distributions, the expectation of Softmax outputs can be closely approximated by applying the Softmax function to the expected scores. This is because for any function of the form f(z) exp(g(=)), the weighted geometric mean II, f(2) is equal to exp EE' [=))). Given the exponential nature of the Softmax function, this leads to the following approximation (often referred to as the Normalized Weighted Geometric Mean or NWGM approximation [2, 25]):
Ez] (o
~ Softmax( [sy 2)).
Additive score assumption. We further assume the pre-activation scores decompose sy(t, z) ~ (I) + (2)), a standard design when text and bias features are fused by summation before the final classifier. Substituting into Equation (2) gives the de de-confounded score:
s5y(1) ST + Bz(sz(=)).
Practical estimator. We instantiate (t) w (y with text encoder fy. A dictionary {f) K of confounder embeddings is built once from the training set (each 2; is a unique object-action pair); the empirical frequency of each 2; serves as P(z:). The expectation becomes the fixed vector fz = > P(z,) fz(z)). Finally, the diabed text representation
(t) (t) + fz,
and s'y(t) w (t).
(4)
4

---

This implements Equation (3), providing an approximation of P()) do(T)) that is provably back- door adjusted under the stated assumptions.
4.2
Visual De-biasing via Front-Door Adjustment
The visual pathway is confounded by an unobserved confounder U (c.g., rapid camera motion, occlusions), rendering back-door adjustment strategy impossible. Instead, we exploit the two-step mediator process X' + (M,,,a) 7 M + ) to apply front-door identification, where M, represents semantic visual features and Ma represents geometric features.
Front-door estimand. For the causal chain X + (Me.Ma) + M + y. Pearl's front-door theorem yields:
P( | =o(x= = x)) = EEPON [ M
m, x | === P(X ='). (5)
122
To implement this, two key expectations need to approximated using their feature embeddings (denoted by bold capitals): (i) the effect of general visual context P(z'), approximated via X RJ
Ex [X], and (ii) the effect of the current visual input x on the mediator P(m a)), approximated via M R: Ex (M].
Mediator design. My encodes high-level semantics from RGB features but can be sensitive to the confounder U. .Ma encodes geometric information (e.g., from pretrained monocular estimation model) and empirically more robust to U. These are extracted by modality encoders, producing sets of token vectors: M,)x) = {ma )رر1 from the current visual input æ, and similarly Ma(x) = {ma,))=
Self- Deff-normalizing token aggregation for mediator components. To approximate the conditional expectations E[M, X and || æ] (which contribute forming M), we employ weighted aggregation tokens. The weights are derived from a normalized exponential of dot-product similarities between query projections (derived from r)) and key projections of the tokens:
M. (x)((g=(2). kny (m.,)))
Ld
exp( K.. (.....))))
Mejs M.
Md.
p=1 exp( (4o (x), kp.p (m)r)))
g= exp(pp (1)(()) Ka, (ma.a))
(6)
where q-(-) and k., represent learned query and key projection functions.
latent mediator M a convex combination of the visual tokens M, = (m..1..... .mmL.. } whose coefficients depend only on the geometric cue Ma. Restricting the predictor this attention-linen -ineer family (ALF) yields the set:
Cross-modal aggregation for the final mediator M. We assume
FALF (M,)) = Σ m.me.
Softmax(s(Ma,, M.))
With squared error the loss, the minimum-meant sqque enror (MMSE) estimator inside FALF is obtained by choosing the scaled dot cot-ppt score s(M,, m..)) = (WoMa, Wgm..)/Val
Appendix proves this result via Lagrange- hgrange-multiplier solution of the following the arguments of Perez et al. [41].
quadratic program [4].
Hence the conditional expectation takes the familiar cross-attention form [48]:
Le expl (W))
E|M M. M.())
-h. p1 exp( (WQM.(z) Wxm..)) m..j.
Because [L(x)==]]M)) XX = x] is purposely designed to be insensitive the egocentric bias U. obtain bias-robust estimate of the overall mediator by plugging Ma(z) into Equation (8): M(x) Attn(Q M., K = V M.). Equation (8) concretely instantiates our claim that "Ma guides the aggregation of M.". ", while ensuring that the learned weights are less exposed to the confounder U. A complete derivation and additional empirical justification provided in Appendix.
Memory-bank estimator for X. Most front-door implementations [60, 49] pre-compute global dictionary of training frames and apply coss-attention against that dictionary approximate the expectation 1 [x]]
5

---

Act-City
Datase
Depth
Visual Front-Door
CLS
class prediction
POS
positive prediction
Tvl
())
(a) Linguistic Back-door Deconfounder
VED
RVOS
Tempori Memory Attention (MAttn)
Mask Dec.
Image Facoder
Masks
Video Frames
X
Iing
M
(x)
Cereal box
Close
Text
Enc
LBD
Self Atta
SA
Prosen
Trainable
Depth-guided Attention (DAttn)
Revedinn
Object Referving (with action)
Linguistic Back-Door
Cross-AAlt
CA
Train val
Train only
(b) Visual Front-door Deconfounder
(c) Overall Architecture of CERES framework
Figure 3: Overview of the CERES framework. The Linguistic Back-door Deconfounder (LBD) de-biases the input text query into features (t). Concurrently, the Visual Front-door Deconfounder (VFD) processes the video frame Xx; ; it forms vision-depth mediator M(x;) using Depth-guided Attention (DAttm) and estimates temporal visual context X via Memory Attention (MAttn). yielding de-biased visual features Ex (zt). These de-biased multimodal features are then used by the RVOS model to predict the segmentation mask Y..
Such static lookup is impractical for long. dynamic egocentric streams. We instead assume (AI) short-range stationarity: within window of W frames, the marginal distribution of the raw frames is approximately unchanged. Under (AI), the last W frames form i.i.d. Monte-Carlo sample of the stationary distribution, so their empirical average is an unbiased estimate of Ex (X'). Concretely, we keep a sliding memory bank B; ( and define
exp exp(sim(x,, Xt-r)) W
o= exp(sim(xx, x1-+)) Xx-7i
(9)
where x, (and x(-r) is the frame-level emhedding of zt and sim(;,) is learnable dot-product score standard attention [48]. Equation (9) is nothing hut a soft weighting Monte Carlo samples, similar to temperature-scaled importance sampling [63].
Theoretical guarantee. By the law of large numbers, the empirical mean over Br converges to Ex [X'] as W + cc. Because the softmax weights in Equation (9) satisfy Σ, w, and are bounded, the same convergence holds for X,. Combined with the Attention- Linear- Family (ALF) model used for M, the overall network consistent estimator of the front-door integral.
4.3 Overall Architecture
The CERES framework applics our proposed causal adjustment modules, the Linguistic Back-door Deconfounder (LBD) and the Visual Front-door Deconfounder (VFD), to pre-trained Referring Vidco Object Segmentation (RVOS) model, typically trained third-person datasets. An overview of the architecture is depicted in Figure 3.
Linguistic Back-door Deconfounder (LBD). To mitigate language bias, the LBD module (Section 4.1) first constructs confounder dictionary. This dictionary comprises embeddings of unique object-action pairs (2;) their empirical frequencies ((z)) derived from the training dataset statistics. The text encoder from pre-trained RVOS model is used to obtain the initial text query embedding fr(() and the confounder embeddings (z,).. During both training and inference, the de- de-biased text representation (t) is computed using Eq. (4), etfectively adjusting for spurious correlations learned from dataset statistics.
Visual Front-door Deconfounder (VFD). The VFD module (Section 4.2) addresses visual confounding, For each vidco frame 2t, visual fcatures X are extracted using the image encoder of the pre-trained RVOS model. Concurrently, geometric depth features x- ddlpthh are obtained from a pre-trained monocular depth estimation model's encoder.
To construct the mediator component M(xx). features from the last n layers of both the RGB encoder (M,, (ax) for layer !) and depth encoder (M,((r)) for layer I) are utilized. For each of these layers, Depth-guided mediator Attention (DAttn) combines these modalitics:
mi(x:) DAttu(Q Mar(zs), K= V= =Mo,(2:)),
(10)

---

Table 1: Comparison (%) with state-of-the-art methods VISOR. Best results highlighted in bold, second- best in underline within same backhone, 1 indicates higher better, & indicates lower better.
Method
Backbone m cloU@1
mloU
cloU
gloUt
Acct
F11
ReferFormer
R101
59,9
66.4
30.5
52.1
55.3
58.6
64.2
ReferFormer+
R101
58.2
64.8
14.3
18.9
63.1
67.6
68.9
Action VOS
R101
59.9
67.2
16.3
28.5
69.9
73.4
73.7
Ours
R101
64.0
72.8
15.3
25.6
72.4
76.3
77.1
ReferFormer+
VSwinB
61.1
68.9
19.2
36.8
68.4
73.2
74.0
Action VOS
VSwinB
62.9
70.9
20.0
38.8
69.5
70.7
74.3
Ours
VSwinB
65.4
72.5
19.1
35.1
72.1
74.7
75.9
HOS
Swinl.
55.1
59.2
13.5
17.3
66.5
70.3
69.7
VotionVOS
SwinL 66.3
71.9
22.8
42.5
68.7
73.4
75.5
Ours
SwinL
67.0
73.6
16.9
28.6
71.8
75.2
76.2
Table 2: Comparison (%) with state-of-the offte-of-theaatt methods on the novel subset of VISOR. Best results are in bold. second-best in underline. f indicates higher is better, J indicates lower is better.
Method
m
cloU@
m
cToU
gloUt
Acef
F11
ReferFormer+
47.2
54.1
13.5
21.8
50.1
51.9
57.7
HOS
45.8
49.7
7.8
11.1
61.9
64.5
66.0
ActionVOS
55.3
62.8
14.5
65.8
69.4
71.9
Ours
60.0
69.9
14.4
20.4
67.9
72.2
75.8
where Ma,1 (xx) is the aggregated depth feature representation for layer (analogous to Eq. (6) applied layer- wise). This yields layer-specific mediator representations (i,{(,)){ For the final mask prediction, we typically use the mediator from the last processed layer, M(x;)) = m,,))
To estimate the general visual context X., memory bank stores recent frame features (from Xub), augmented with temporal positional encodings. Temporal Memory Attention (MAttn) then computes X, as per Eq. (9). The de-biased visual feature (x;) is then formed by integrating the mediator and context information.
Specifically, the final mediator M(x)) and the context X, are concatenated and processed through an MLP followed by gated residual connection:
(2,) OMX MLP((M(z.). X,)) + (1-OM.x) X,-
(11)
This foe (3rx) represents the visual input adjusted for egocentric confounders, O'.x the gate residual path for easier to train [45, 20J. For auxiliary losses during training, similar de-biased features (x,( are computed using the intermediate layer-specific mediators (r,).
Output Generation and Training. The de-biased text features (t) and visual features (x (or f'x (a.) for intermediate layers) are then utilized by the subsequent components of the RVOS model. These typically include classification head to predict the object category. "positive" head identify if the object is actively involved the action, using standard segmentation loss. Following prior work, auxiliary segmentation losses are applied to the outputs derived from the de-biased visual features of the n intermediate layers during training. During inference, only the de-biased visual feature from the final considered layer is used for prediction.
mask decoder to generate final segmentation mask Pt. The model trained
5 Experiment
5.1 Experiment Settings
Datasets. Following previous work [36], we evaluate our method on three public egocentric video datasets:
VISOR [12], VOST [46], and VSCOS [62]. VISOR, derived from EPIC-KITCHENS [10, 11], provides utilize its training and validation splits. After pre- processing, this yields 13,205 videos (76.873 objects) training 467 videos (1,841 objects) for validation. where validation objects are manually annotated as positive or negative. VOST and VSCOS are used for validation only. VOST assesses performance on objects undergoing transformations. VSCOS focuses on state- mate-chaangin objects; its validation data is filtered prevent overlap with the VISOR training set.
annotations for hands and active object interactions;

---

Metriiss We follow established evaluation protocols [26, 36]. Key metries include mean Intersection over Union (mloU) and cumulative ToU (cloU), reported separately for positive (mloU cloU and negative (mIoU9, mloUS) objects to assess segmentation of interacted and non-interacted instances, respectively. We also report generalized IoU (gloU) [26] for a combined assessment of segmentation and target classification. Additionally, Precision (P), Recall (R), and Accuracy (Acc) are used for the binary classification of object activity. For these classification--elee related evaluations (gloU, P. R, Acc), a prediction is considered a True Positive (TP) if its IoU with the ground truth exceeds 0.5. More metries details are in the Appendix D.
Implementation Details. CERES builds upon the ReferFormer [57] architecture, initializing with its pre-trained weights. We evaluate ResNet101 [20], Swin-Traasformer-LL. [30], and Vidco Wwin-Transformer- [31] image encoder backbones, paired with a RoBERTa [29] text encoder. Input frames are resized to 448 x 448. For the Depth-guided Attention (DAttn) of Visual Front-door Deconfounder (VFD), depth features are extracted using the encoder of frozen pre-trained Depth Anything V2 model [59]. The Linguistic Back-door Deconfounder (LBD) defines confounders 2, based on unique "verb-noun" " pairs identified in the training set queries. The VFD's Temporal Memory Attention (MAtn) employs a window W 5 recent frames. The model is trained with a batch size of 4, Following the previous implement, auxiliary losses during training utilize visual features from the last three layers of the image encoder. During inference, predictions are made online, without access to future frames. Further optimization and hyperparameter details are provided in the Appendix.
5.2 Comparison with State-of-the-arr Methods
We compare our CERES against several leading methods. ReferFormer [57] serves as a foundational baseline, rep- resenting a model pre-trained on third-person data and subsequently fine-tuned for Ego-RVOS, ReferFormer+ ex- tends this by incorporating an auxiliary prediction head for "positiveness" (identifying the object actively involved in the action) and by including action descriptions in the referring query. EgoHOS [67] is hand-object segmen- tation model; for our comparison, we train it on VISOR, treating hand-associated objects as positive targets. Ac- tion VOS [36] a strong recent baseline that also builds on pre-trained models and employs specialized losses for active objects.
Table 3: Comparison (%) of positive objects IoU on VSCOS and VOST.
VSCOS
VOST
Method
mIOUt
clOUt mIOUt cIOUf
ReferFormer+
$3.0
54.2
30.6
16.1
HOS ActionVOS
42.1
31.2
21.9
15.2
52.5
57.7
30.2
17.6
Ours
55.3
62.5
32.0
21.7
On the VISOR benchmark (Table 1), CERES consistently
outperforms prior methods across backhones. With ResNer101, CERES achieves 64.0% mloU (+4.1 over ActionVOS) and 72.4% gloU (+2.5%), alongside improved accuracy and F1 scores for positive object classification. CERES also generally shows lower mloU and C indicating better discrimination against non-target objects. While HOS has low e due to its focus on hand aandiproximate objects, its @ is consequently limited.
CERES's advantages particularly evident on VISOR subset with novel objects actions seen during training (Table 2). This subset serves as an open-vocabulan evaluation within VISOR, and CERES significantly surpasses previous state-of-the-an results, demonstrating strong generalization capabilities likely due to its mitigation of ladaset language biases.
Evaluations on VSCOS and VOST (Table 3), which feature significant object transformations, further underscore CERES's robustness. We treat these zero-shot open-vocabulary generalization: models are trained only on VISOR and evaluated on VSCOS/VOST without any fine-tuning. For instance, on VSCOS. CERES achieves 55.3% m and 62.5% cloU@ exceeding Action VOS (52.5% and 57.7%). This superior performance in challenging scenarios highlights the effectiveness of CERES's causal intervention strategies in addressing both linguistic and visual confounding factors egocentric videos.
5.3 Ablation Study
We conduct ablation studies on the VISOR dataset using the ResNet101 backbone to analyze the contribution of each key component in CERES.
Ablation of Proposed Modules. In Table 4, the bascline model (first row) achieves 59.9% m @ and 69.9% gloU. Introducing only the LBD (second row) improves mloU@ to 61.2% (+1.3%) and gloU to 71.4% (+1 (+1.5%), demonstrating its effectiveness in mitigating language bias. A nonlincar MLP-based depth fusion (third row) for the mediator offers a 62. 1% mloU@ but increases @ indicating limited discriminative benefit. In contrast, our DAttn depth integration (fourth row) significantly boosts @ to 63.3% (+3.4% over baseline) and reduces e to 15.8%, yielding substantial gains in gloU (71.8% and Acc (75.3%). This confirms
8

---

Table 4: Ablation study (%) of proposed modules on VISOR (ResNer101). (O indicates M-.P-based depth fusion)
Table 5: Performance comparison "hard" subset VISOR. (RF means ReferFormer [57],
ActV means Action VOS [36])
DAtin MAttn LBD ? mloU
gloUt
Acct
Method m mloU
gloUt
Acc
59,9
16.3
69.9
73.4
61.2
16.0
71.4
RF
53.4
14.9
56.3
58.5
62.1
17.5
70.5
73.6
RF+
54.2
14.7
56.7
58.7
63.3
15.8
71.8
75.3
ActV
58.4
15.1
69.9
73.1
63.1
14.9
72.1
76.1
Ours
62.3
14.3
72.2
75.6
64.0
15.3
72.4
76.3
Put box
Deel carrol
Open lortilla pack
Action VOS
DO0 DICORUE
Tortis pacs
mlou (%)
Accuracy (%)
63.5
63.0
BOK BIBRE
p-mlol
Ours
unie Pouh 0
62.5
Accuracy
Window Size (W)
Figure Qualitative analysis Action VOS our CERES.
Figure 5: Effecc of MAtmn.
the superiority of our causally-inspired depth mediator design within the VFD. Adding MAttn (fifth row) to complete the VFE (DAutn + MAttn) further refines performance. This configuration achieves the lowest @ (14.9%) and strong 72. % gloU, highlighting TCA's role in improving discrimination by modeling broader visual context. The full CERES model (last row), integrating both the complete VFE and LBD, attains the best overall performance, with 64.0% mI 72.4% gloU, and 76.3% Acc. While its mloU (15.3%) is slightly higher than VFE-only (14.9%). the LBD's inclusion enhances the recall of positive instances. This trade-off results in superior overall identification and segmentation of target objects. These ablations validate the individual and synergistic contributions of our causal adjustment modules.
Temporal Context Window Size MAttn. We analyze impact of temporal window size MAttn module Figure 5. Setting W = 0 (i.e., no MAttn module) results in lower performance compared to using MAtt. As W increases, mlnU@ and Acc generally improve. We found W 5 provides robust balance and consistently strong results.
Performance on Rare Concepts. To assess robustness against data scarcity biases, we evaluate models on a "hard" subset of VISOR, comprising 159 clips with actions or objects appearing <50 times in training. Table shows that, compared to Action CERES improves @ 2 by +3.9% (62.3% vs 58.4%) gloU by +2.3% (72.2% vs 69.9%). These gains are mainly attributed to LBD blocking the spurious path (back-door), while VFD mitigates egocentric visual confounders l/ via the vision-depth mediator (front-door). This highlights the effectiveness of CERES, particularly the IBBD component, in generalizing to less frequent. concepts by mitigating reliance spurious statistical correlations.
Qualitative Analysis. Figure 4 qualitatively compares CERES with Action VOS [36]. showcasing CERES's superior robustness. Our method yields more accurate segmentation challenging egocentric scenarios. including those with visual distortions like motion blur (e.g.. "box", column 1) and occlusions (e.g.. "knife", column 2). This highlights the VFD modulc's effectiveness in mitigating visual biases. Furthermore, CERES demonstrates improved handling of textual queries involving uncommon objects actions (c.g., "tortilla pack", column 3), where ActionVo VOS may falter due dataset biases. This underscores LBD modulc's contribution better language grounding. Overall, these visual examples corroborate our quantitative results, illustrating how CERES's ' causal interventions lead to more robust Ego-RVOS.
6 Conclusion
This paper introduced Causal Ego- Reoo-REfennng Segmentation (CERES), novel framework that applies causal inference principles to address critical robustness challenges in Egocentric Referring Video Object Segmentation. We identified two primary sources of error: language biases stemming from dataset statistics and visual confounding inherent in the egocentric perspective. CERES tackles these by employing backdoor adjustment

---

mitigate spurious correlations between textual queries and segmentation outputs, and by utilizing front-door adjustment with novel vision-depth mediator counteract the effects of unobserved visual confounders. This dual-pronged causal intervention allows CERES learn more robust representations, less susceptible to dataset- specific biases and egocentric visual distortions. Extensive experiments on standard Ego-RVOx benchmarks demonstrate that CERES achieves state-of-the-as art performance, significantly improving segmentation accuracy and reliability, particularly in challenging scenarios with novel concepts significant visual ambiguity. Our work underscores the potential of causal reasoning build more generalizable and trustworthy models for complex egocentric video understanding tasks.
Acknowledgments
This work was supported by National Key R&D Program of China under Grant No. 202 10001111600 1601. National Natural Science Foundation China (NSFC) under Grant No. 62272494, Guangdong Basic and Applicd Basic Research Foundation under Grant No. 2023A1515012845 and 2023A151501133 and Guangdong Province Key Laboratory of Information Security Technology.
References
[1] Ehsan Abbasnejad, Damien Teney, Amin Parvaneh, Javen Qinfeng Shi, and Anton van den Hengel. Counterfactual vision and language learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10041-10051, 2020. doi: 10.1 11111///VVY44260000000000 20..0000000000000000.0
[2] Pierre Baldi and Peter Sadowski. The dropout learning algorithm. Artificial Intelligence, 210:78-122, 2014. ISSN 0004-3702. doi: 10. (0.1016jj.rtint.2011 201.0l..0.0..00004..
[3] Adam Botach, Evgenii Zheltomozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4975-4985, 2022. doi: 10. (D9/CCVPR5288.2022..000
[4] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Dibyadip Chatterjee, Fadime Sener, Shugao Ma, and Angela Yao. Opening the vocabulary of egocentric actions. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems (NeurIPS), 2023.
[6] Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin. Learning semantic-specific graph representation multi-label image recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). pages 522-531, 2019.
[7] Tianshui Chen, Tao Pu, Hefeng Wu, Yuan Xie, Lingho Liu, and Liang Lin. Cross-domain facial expression recognition: unified evaluation benchmark and adversarial graph learning, IEEE Transactions on Puttern 2022.
Analysis and Machine Intelligence (TPAMI). 44(12):9887-99)
[8] Ho Kei Cheng, Seoung Wug Oh, Brian L. Price, Alexander G. Schwing. and Loon-Yourg Lee. Tracking anything with decoupled video segmentation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1316-1326, 2023. doi: 10. /BD//CCCVS1007..222.001111
[9] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. volume 26, 2013.
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Scaling egocentric vision: The epic-kitchens datasct. Proceedings of the European Conference on Computer Vision (ECCV), pages 0011:/978---0--1---
720-736, 2018. doi:
[11] Dima Damen, Hazel Doughty. Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma. Davide Moltisanti, Jonathan Munro, Toby Perreut Will Price, et al. Rescaling egocentric vision: Collection. pipeline and challenges for epic-kitchens- 100. International Journal of Computer Vision (IJV), pages 1-23, 2022. doi: 10. 100 10.1007/4112233-00220 12200001111222-0015555333
[12] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitcher visor benchmark: Video segmentations and object relations. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 13745-13758, 2022. doi: 00.555/3300270.366013
[13] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang. and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 2023. doi: 10.11009/CCCV110002223....
10

---

[14] Henghui Ding. Chang Liu, Suchen Wang, and Xudong Jiang. VLT: vision-languan transformer and query generation for referring segmentation. IEEE Trunsactions Pattern Analysis and Machine Intelligence (TPAMI), 45(6))7900-7116, 2023, doi: 2L0.9//TA...222...222.....
[15] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[16] Rick Durrett. Probability: Theory and Examples. Cambridge Series Mathematics, Cambridge University Press, 5 edition, 2019.
in Statistical and Probabilistic
[17] Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees G. M. Snoek. Actor and action video segmentation from sentence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2018. doi: 10.1 109/CVPR 20E.E..R.201.1... 0.0244.
[18] Robert Geirhos, Jörn-Herrik Jacobsen, Claudio Michaelis, Richard S. Zemel. Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelli- gence, ((11)6665-65 2020. doi: 10. 1038//422266-020 020-00057000222222 URL https: //tp:////dooooo wtg//.ooo.///oooo 10. 1038/ s42256 0200000000000 00257-2.
[19] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Procedingss of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18995-19012, 2022.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1 20.1..
[21] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns:: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 5156-5165, 2020.
[22] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Asian Conference on Computer Vision (ACCV), pages 123-141, 2019. doi: 10. 1007/ 978-3-030-20877 00000000020077---0 8.
[23] Jiangtong Li, Li Niu, and Liqing Zhang. From representation to reasoning: Towards both evidence and commonsense reasoning for video question-answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21241-21250, 2022. doi: 10.1 1999CCS522.22........ 2022.02059.
[24] Xiang Li. Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu. Robust referring video object segmentation with cyelic structural consensus. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 22179-22188. IEEE, 2023. doi: 10. 1009/CCCV11000.222...000
[25] Bing Liu, Dong Wang, Xu Yang, Yong Zhou, Rui Yao, Zhiwen Shao, and Jiaqi Zhao. Show, deconfound and tell: Image captioning with causal inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 18020-18029, IEEE, 2022. doi: 10.1109/CVPR522888. 2022.01751.
[26] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 23592 23601, 2023. doi: 10. N0.1109/CVPR2272..00 2222000
[27] Haijing Liu, Tao Pu, Hefeng Wu, Keze Wang, and Liang Lin. DART: dual adaptive refinement transfer for apen-vocabulary multi- recognition. Proceedings of the ACM International Conference on Multimedia (MM), 2025.
[28] Yang Liu, Guanbin Li, and Liang Lin. Cross-modal causal relational reasoning for event-level visual question answering. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(10); 11624-116A4 11641, 2023. doi: 10. 109/TPAMM..023.377 233.328038...
[29] Yinhan Liu, Myle Ott, Naman Goyal. Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: //txxvaaivvvv///////// org/ wwtpaa/////// 1907 11692.

---

[30] Ze Liu, Yutong Lin. Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9992-10002, 2021. doi: 10.1 N0.1109/1CC48892. 4021.00986
[31] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 3192-3201, 2022. doi: 10.1 2DD0/CCVP52688.2222.00020
[32] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L.. Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11-20, 2016, doi: 10.1 10.PPR.. 2016.9.
[33] Lei Meng. Xiangxian Li, Xiaoshuo Yan, Haokai Ma, Zhuang Qi. Wei Wu, and Xiangxu Meng. Causal inference over disual-semante semantic-aligned graph for image classification. In Toby Walsh, Julie Shah, and Zico Kolter, editors. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 19449-19455 2025. doi: 10. V..91AAA..33111....1111
[34] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE/CVT Conference on Computer Vision and Pattern Recognition (CVPR), pages 891-898, 2014. doi: 10.1 10.1109/VVPR.2244.119
[35] Yulei Niu, Kaihua Tang, Hanwang Zhang. Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: cause- efffcc look at language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12695 12705, 2021. doi: 10. 10.1109/CVPR444437.22222 2.02001
[36] Liangyang Ouyang, Ruicong Liu, Yifci Huang, Ryosuke Furuta, and Yaichi Sato. Actionvos: Actions as prompts for video object segmentation, In Proceedings of the European Conference Computer Vision (ECCV), volume 15068, pages 216-235, 2024. doi: 10. 1000/007/78--11 0-017------011-77422-2-
[37] Amin Parvaneh, Ehsan Abbasnejad, Damien Teney, Javen Qinfeng Shi, and Anton van den Hengel. Counterfactual vision-and-languuaee navigation: Unravelling the unseen. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 5296-5307. Curran Associates, Inc., 2020.
[38] Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009.
[39] Judea Pearl and Dana Mackenzie. The Book of Why: The New Science Inc., USA, 1st edition, 2018. ISBN 046509760X.
of Cause and Effect. Basic Books,
[40] Simone Alberto Peirone, Gabriele Goletto, Mirco Planamente, Andrea Bottino, Barbara Caputo, and Giuseppe Averta. Egocentric zone-aware action recognition across environments. Pattern Recognition Letters, 188:140- 147, 2025. ISSN 0167-8655. doi: http:://doi.or//1000 p/to//.attree.2222. 12.008.
[41] Guillaume Perez, Sebastian Ament, Carla Gomes, and Michel Barlaud, Efficient projection algorithms onto the weighted I1 hall. Artificial Intelligence, 306: 1036833,,, 2022. ISSN 0004-3702. doi: 10, 10.1016/j.artint.
2022. 1 103683.
[42] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44(3))1623-113333 2022. doi: 200. 3019967.
[43] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection union: A metric and loss for bounding box regression. In Proceedings of the JEFF//CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 658-666, 2019. doi: 10.1 WttCCVP..009........ 2019.00075.
[44] Yuhan Shen, Huiyu Wang, Xitong Yang, Matt Feiszli, Ehsan Elhamifar, Lorenzo Torresani, and Effrosyni Mavroudi. Learning segment referred objects from narrated egocentric videos. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14510-14520, 2024. doi: 10.1 A1110//CVPS2233.22222222 20.70.0.7.......
[45] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks, 2015. URL https: //arxi/ arrg/abs/15t 00387.
12

---

[46] Pavel Tokmakov, Jie Li. and Adrien Gaidon. Breaking the "object" in video object segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22836-22845, 2023. doi: D0.1099CCVRRS27292222203. 2023.02187.
[47] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang. J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Anna Korhonen, David R. Traum, and Lluís Märquez, editors, Proceedings of the Annual Meeting of the Association for Computational Linguisticy (ACL), pages 6558-6569. Association for Computational Linguistics, 2019. doi: 10.18653/V1/1111 1/P19-1656.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg. S. Bengio. H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnet, editors, Advances in Neural Information Processing Systems (NeurlPS), volume 30. Curran Associates, Inc., 2017.
[49] Liuyi Wang, Zongtao He, Ronghao Dang. Mengjiao Shen, Chengju Liu, and Qijun Chen. Vision-and- language navigation via causal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages
2024. doi: 10. N0.110//VVPS22733.2200 202.3333.222.122444
[50] Tan Wang. Jianqiang Huang, Hanwang Zhang, and Qianru Sun, Visual commonsense r-cnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10757- 107677 2020. doi: 10.1 L009/CCV422000022222 2020.01077
[51] Tan Wang, Chang Zhou, Qianru Sun, and Hanwang Zhang. Causal attention for unbiased visual recognition. In 2021 IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 3071-3080, 2021. doi: 10. 1091111 200000000.
[52] Wenxuan Wang, Yisi Zhang, Xingjian He, Yichen Yan, Zijia Zhao, Xinlong Wang, and Jing Liu. Beyond literal descriptions: Understanding and locating open-world objects aligned with human intentions. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics (ACL Finding), pages 762-776, 2024. doi: 10.18653/V 1/2024. 2.0224.FINNNNGS SECDINGS-ACLL.3.
[53] Yuqing Wang. Xiangxian Li, Zhuang Qi, Jingyu Li, Xuelong Li, Xiangxu Meng, and Lei Meng- Mcta- causal feature learning for out-of-distribution generalization. In Leonid Karlinsky. Tomer Michaeli, and Ko Nishino, editors, Proceedings of the European Conference on Computer Vision Workshops (ECCV Workshops), volume 13806 of Lecture Notes in Computer Science, pages 530-545, 2022. doi: 10.1007/ 978-3-031 100-3-031-25075-00 25075-01-36.
[54] Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language- image segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10213-10222, 2020. doi: 10.1 00.100//CCP44266022200000
[55] Hefeng Wu, Weifeng Chen, Zhibin Liu, Tianshui Chen, Zhiguang Chen, and Liang Lin. Contrastive transformer learning with proximity data generation for text-based person search. JEEE Transactions on Circuits and Systems for Video Technology, 3408)-7005-7016, 2024.
[56] Hefeng Wu, Yandong Chen, Lingho Liu, Tianshui Chen, Keze Wang, and Liang Lin. SQLNet: Scale- modulated query and localization network for few-shot class-agnostic counting. IEEE Transactions on Image Processing (TIP), 34:4631-4645 2025.
[57] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring vidco object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4964-4974, 2022. doi: 10.1 (00////VPR2288.22222.000
[58] Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Segment every reference object in spatial and temporal spaces. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages
2023. doi: 10. www.///CC1000...2.......
[59] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pages 21875-2191 06
2024. doi: 10.48550/arxiv.2444 wtw..220060944..
[60] Xu Yang, Hanwang Zhang, Guojun Qi. and Jianfei Cai. Causal attention for vision-language tasks. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9842-9852, 2021. doi: 10.1 109/CVPR46437.2021 100972..
[61] Xu Yang. Hanwang Zhang, and Jianfei Cai. Deconfounded image captioning: A causal retrospect. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 45(1 12:1999 13010, 2023. doi: 10.1109/TPAMI.00 2021..2..2211111111 310
13

---

[62] Jiangwei Yu, Xiang Li. Xinran Zhao, Hongming Zhang. and Yu-Xiong Wang. Video state-changing object segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20439-20448, 2023. doi: 10.1 10.10000CCVV510000200..0111
[63] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 19922-15932, 2023. doi: 200
[64] Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Kian-Sheng Hua. Interventional few shot learning. In Advances in Neural Information Processing Systems (NeurlPS), 2020. ISBN 9781713829546.
[65] Zhongqi Yue, Tan Wang. Qianru Sun, Xian- Shen-Sheng Hua, and Hanwang Zhang. Counterfactual zero-shot and open-set visual recognition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15404-1544 15414, 2021. doi: 10. 109/CCVP444437.222...000 115115.
[66] Hua Zhang. Xiaohong Li. Xiaochun Cao, and Hassan Foroosh. Multiple adverse weather conditions adaptation for object detection via causal intervention. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 46(3):1742-111 2024. doi: 10.1 (000TPAAM..2222.31666677
[67] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In Proceedings of the European Conference Computer Vision (ECCV), pages 127 145, 2022. doi: 10.1007/978- 3-031- 19811-99988--111 2_8..
[68] Shengyu Zhang, Tan Jiang, Tan Wang. Kun Kuang, Zhou Zhao, Jianke Zhu, Jin Yu, Hongxia Yang, and Fei Wu. Devlbert: Learning deconfounded visio-linguistic representations. In Chang Wen Chen, Rita Cucchiara, Kian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, Proceedings of the ACM International Conference on Multimedia (MM), pages 4373-4382, 2020. doi: 10. 11145/3394171.413355
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and socictal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] mcans either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While
"[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars not reported because it would be too computationally expensive" or used"). In general, answering "[No] "o "[NA] " is
"we were unable to find the license for the dataset grounds for rejection. While the questions are phrased in binary way. we acknowledge that the true answer often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction claim CERES introduces a novel causal inference frame- work for Ego-RVOS, using backdoor adjustment for language biases and front-door adjustment with a novel vision-depth mediator for visual confounding. The paper's 's methodology (Section 4) details these causal mechanisms. Experimental results (Section 5), including SOTA performance standard benchmarks (VISOR, VOST, VSCOS) and ablation studies, directly support these contributions and demonstrate the effectiveness the proposed interventions within the defined scope of Ego-RVOS. The claims regarding improved robustness against these specific biases are also substantiated by comparative and ablation experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We dedicated paragraph this (Limitations") part of the Appendix.
15

---

Guidelines:
The answer NA means that paper has no limitation while the answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Key assumptions stated in the main text, and complete proofs detailed derivations for our theoretical claims, such as the mediator construction and overall estimator consistency, are provided in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should he clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper details the datasets, evaluation metrics, base model, specific architectural choices for CERES components (LBD, DAttn, MAtt),, key hyperparameters like batch size and window size, and explicitly states that further implementation details are available in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
16

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same datasct, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproducc that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of glosed-soure models, it may be that access to the model is limited in some way (e.g.- registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will release our code and detailed instructions for data preparation and reproducing the main experimental results after the review period.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp) http:/iiss.ccppplppeeppp cc/pubiic/ guides/CodeSubmn Coo///oeeeSeiieic asionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips.i...///////// cc/ cc/publcc/ guides/ quides/CodeSubent Cet///oooSuiiiiie ssiooPolcyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper outlines key experimental settings including datasets, metries, backbone architectures, and critical module parameters in the main text, and explicitly states that further optimization and hyperparameter details are provided in the Appendix, ensuring reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Experiments were too costly to be run multiple times, however, lot of ablations available and the model is tested on multiple datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the bars should be explained (closed form formula, call library function, bootstrap, etc.)
assumptions made should given (e.g.. Normally distributed errors).
should be clear whether error the standard deviation or standard error of mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures rates).
symmetric error bars that would yield results that are of range (c.g. negative
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resourees
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The machine which the experiments, well the computing time needed is mentioned in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpss////euii...e
Answer: [Yes]
Justification: Our work complies with NeurIPS ethics guidelines. No human subjects or sensitive data were involved.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure preserve anonymity (e.g., if there is special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
The paper explicitly discusses potential positive impacts such as advancements in assistive technologies, human-robot interaction, AR, and embodied AI, alongside negative socictal impacts including privacy concerns from surveillance, potential for bias propagation, and risks of misuse, while also suggesting the need for ethical guidelines and mitigation strategies
Justification:
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.- disinformation, generating fake profiles, surveillance), fairness considerations (e-g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out that a generic algorithm optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is heing used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: No model in this paper is with high risk for misuse. The collected datasets all open-source and accessible
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code. data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All datasets and code libraries are properly cited with licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for cach asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode- com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datascts that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code for the CERES framework and the corresponding trained models assets. Documentation provided with the code release, detailing implementation, training procedures, and dependencies ensure reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their sub- missions via structured templates. This includes details about training. license, limitations. etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, rememher to anonymize your assets (if applicahle). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We did not use crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our research did not involve experiments on humans subjects.
Guidelines:
answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research conducted. IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
20

---

For initial submissions, do not include information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it important, original, non-standard component
core methods this research? Note that if the LLM used only for writing. editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [No]
Justification: LLMs were only used for proofreading: they did not contribute to methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: //neur///sesseeee cc//oonferences/2225///////////e for what should should not be described.
21

---

Appendix
A
Limitations
While CERES demonstrates notable advancements in robust Ego-RVOS, we acknowledge certain aspects for future considcration. The LBD module's current definition of confounders (2) object-action pairs, while effective for the targeted dataset biases, represents one specific strategy; future work could explore more nuanced or automatically discovered confounder definitions. The additive score assumption used the back-door adjustment, though common, is an approximation of the true deconfounded score.
For the VFD, its performance is influenced by the capabilities of the underlying pre-trained depth estimator. While we empirically show that depth aids in mitigating visual confounding, formally verifying full front-door conditions for the chosen mediator structure is nontrivial. The Attention- Linear Family (ALF) assumption explicitly trades expressivity for identifiability; restricting fusion weights to be functions of Ma helps prevent leakage of the unobserved confounder U and yields a minimal, identifiable realization of the mediator, but narrows the function class. Empirically, our Al.e- DAttn outperforms a nonlinear MI.P fusion (Table 4), indicating superior robustness under egocentric confounding. Exploring richer mediator parameterizations that preserve front-door validity is left for future work.
The MAttn module approximates temporal context using sliding window, technique effective for many dynamic egocentric scenes, though its generalization to scenarios with extremely long-range temporal depen- dencies could be further investigated. The integration of depth features and additional attention mechanisms introduces computational costs relative to simpler baselines, common trade-off for enhanced robustness, and further optimization could be explored.
Finally, the current evaluation of CERES is focused on the Ego-RVOS task. Extending and rigorously evaluating its applicability and potential adaptations for broader spectrum of egocentric video understanding challenges, such as egocentric action recognition or long-term activity understanding, presents a valuable avenue for future research.
B Broader Impact
The advancements in robust Ego-RVOS achieved by CERES hold potential for significant positive impacts. More reliable egocentric video understanding can directly benefit assistive technologies, enhancing contextual awareness for individuals with visual impairments, and can enable more intuitive human-robot interaction by allowing machines to better grasp human-object interactions from first-person view. In fields like augmented reality, precise segmentation of actively manipulated objects can lead to more seamless and responsive user experiences. Crucially, the causal principles and architectural components developed in CERES, particularly the strategies for mitigating dataset and visual biases, may offer a foundational approach for enhancing robustness and generalizability in other egocentric video analysis tasks. This is particularly relevant for embodied Al, where a nuanced understanding of human actions and object interactions from an egocentric perspective is critical for agents to learn from human demonstrations, predict intentions, and operate safely and intelligently within complex human environments.
However, as with any technology capable of detailed scene analysis, responsible development paramount. The potential for misuse in surveillance or intrusive monitoring, especially with personal egocentric data, necessitates strong ethical guidelines and underlying pre- mre-trained models might still harbor unaddressed biases, emphasizing the need for continuous auditing and fairness considerations Al system development. The deployment of such technologies should thus proceed with a commitment to ethical practices and ongoing research into comprchensive bias mitigation.
measures. While CERES aims to reduce specific biases, the
C More Implementation Details
All experiments were conducted using PyTorch 2.2 and CUDA 11.8 system with four NVIDIA V100 GPUs. Models were trained for epochs with total batch size of 4, where cach batch item was single vidco clip. We initialized the lcarning rate to 1 x 10 for our CERES modules and 1 x 10 for pre-trained components, decaying it by 0.1 at epochs and 5, using the AdamW optimizer. The primary segmentation loss combined bounding box, Dice and Focal losses. Input frames were resized to 148 x 448 for both training and inference.
We use ReferFormer pretrained weight Youtube- VOS dataset. For the VFD, depth features were extracted using a frozen pre-trained Depth Anything V2 encoder. The LBD module defined confounders from unique "verb-noun" pairs in training queries. The VFD's DAttn utilized features from the last three layers of the image and depth encoders, while MAttn employed temporal window of W = 5 recent frames. During inference, an online strategy adopted. processing frames sequentially without access future information.
22

---

D
Metric Details
This section provides further details the evaluation metrics used in this work, complementing the descriptions Section 5.1 of the main paper.
Accuracy and Fl-score. These metrics evaluate the binary classification of whether an object is actively involved the queried action. prediction is considered a True Positive (TP) if its Intersection over Union (IoU) with the ground-truth active object mask exceeds 0.5. This stricter threshold, compared to some prior works like VOS [36] which might use IoU 0, ensures more accurate reflection of identification and localization. True Negatives (TN) correctly identified non-active objects correct "no target" predictions. False Positives (FP) are non-active objects misclassified as active, or incorrect "target present" predictions, False Negatives (FN) are active objects missed or misclassified as non- non-active.
Based on these, Accuracy (Acc) and Fl-score are calculated as:
Accuracy (Acc):
TP TN TP + TN FP
Acc
Fl-score:
2 TP
F1
2 TP FP+ FN
The F1-score is the harmonic mean of Precision and Recall, providing balanced measure for active object classification.
Union (mIoU) and Cumulative IoU (cIoU). These are standard segmentation quality metrics, mIoU calculates the average IoU across all individual object instances the dataset. For each instance, loU is the ratio of the area overlap between the predicted mask and the ground-truu mask to the area of their union. cloU computes single IoU value over the entire dataset (or subset) by summing all intersection areas and dividing by the sum of all union areas. As stated in the main paper. these metrics reported separately for positive objects (target objects actively involved in the action: mloU@ cloU@) and negative objects (other objects present the scene not involved in the queried action: m cloU Lower scores for negative objects indicate better discrimination against non-targ non-targets.
Mean Intersection
Generalized IoU (gloU), Proposed by Liu et al. [26], gloU offers a combined assessment of both segmentation quality and the model's 's ability to correctly classify target presence, It is calculated as the mean of per-sample scores, making it less sensitive to object size variations compared to cloU. The per-sample score for gloU is determined follows:
If ground-truth target object exists for the query in a given sample: The prediction for this object is first evaluated against the TP criterion (loU > 0.5 with the ground truth). If it qualifies as TP, its actual IoU value contributes to the gloU average for that sample. If the prediction's IoU is S 0.5, or if no object predicted by the model, the contribution for that sample O.
If the ground truth indicates no target object for the query in given sample (a "no-target" sample): If the model correctly predicts that no target object is present, the sample's 's contribution to gloU the model incorrectly predicts an object, the contribution is O.
This definition ensures that gloU comprchensively evaluates the model's performance in segmenting correctly identified targets as well its ability to correctly handle scenarios where the queried object absent.
E Theoretical Details
E.1
Proof of the Cross-Modal MMSE Estimator
This appendix provides the technical details that underpin Eq. (8) in the main text, showing that the scaled dot- not-ppr cross-attention
Le expl (WoM WK Wk.m..)/VV m.j.
Atta(Q = Ma. K - - - M.) = Σ
(E.I)
1 E, exp( (WaMa, Wxm.p)/VVl
the minimum mean- square-ere error (MMSE) estimator of the latent mediator M within attention linear family FALF defined in Sec. 4.2. The derivation follows the general recipe of simplex-constrainee constrained quadratic optimization [4] and re- uses the causal decomposition arguments of Perez et al. [41]. Throughout the appendix all expectations are conditional the observed pair (Ma, M.) unless stated otherwise.
23

---

E.1.1 Problem Set-Up
Let E R" be the latent mask mediator that triggers the downstream decoder. For fixed input we observe
Ma E IR"" M. {m..1, m..L.) } C R..
Inside the family FALF every candidate estimator is combination fa (M.) E, ayme, with non- negative coefficients a A where A = {a, 2 E, 1} the probability simplex. The restricted MMSE problem is therefore
argmin asa ((a), [(a) =B M Σ, a m..
(F.2)
E.1.2 Quadratic Form of the Risk
Let μ = E[ be the (unknown) conditional mean of the latent mask. Expanding the square in Eq. (E.2) gives
L(a)
| 226 a a Ga,
(E.3)
const
where b, (u, m..)) and G., (m.... mos). Because the first term does not depend on a, Eq. (E.2) reduces the simplex--onstrained quadratic program
(E.4)
E.1.3 Lagrange-Multiplier Solution
We solve Eq. (E.2) by introducing Lagrangian
J(ax A, n) = -2 o--a
T Ca
X aj -0- "
(E.5)
where À R and 7 L are dual variables that enforce the simplex constraints. Differentiating Eq. (F.2) and using the KKT conditions [4, Ch. 5] yields
2 2(Ga), - 2b,
- 173 =0
Vj.
(F.6a)
' = 0,
O3 2 0, n3 2 0,
(E.6b)
Ea -a==1. 1.
(F.6c)
When (true after layer normalization [48]), solution lies in the open simplex-namely a; > implying = 0. Subtracting the i-th and j-th rows of Eq. (E.2) cancels À and gives b, bb = (Ga); - (Ga). Re-ordering yields
= a: exT b -G- (Ga),
(F.7)
where we have inserted an infinitesimal temperature > 0 for differentiability (the same trick entropic regularization [9]). Imposing E, a, converts Eq. (E.2) into the softmax-style fixed-point
exp( 26, - -(Ga),
0g Ep exp 2hp-2(Go),
(E.8)
E.1.4 SuorropicTToken Approximation
After layer normalization, nigh-dimensional visual tokens (m.,) are often nearly orthogonal [15, 21J, implying Gy (m.,4 m..3) > x 7,04y, where 7/ = mm.. 11.. Thus. (Ga), Rs 75ay. The exponent Eq. (E.8)
hecomes (2h, 2//0))/T.
To achieve the common softmax form based on linear scores, we approximate by assuming the cross-intcraction terms = (t,, m..)) dominate the self-interaction terms 2yja;/7 in determining the relative attention weights. This simplification, common in deriving attention mechanisms [41], effectively neglects the quadratic self- influence terms or treats their impact as uniform, yielding:
exp(2h,//1 Σ, exp((22p/7)
aj
Softmax (n, m/))
(E.9)
This results in scores linear in μ and maj within the softmax, aligning with standard attention designs.

---

E.1.5 Substituting a Geometric Surrogate for
The inner products in Eq. (E.9) still involve the unknown mean μ.. We therefore introduce linear surrogate driven by the geometric cue Mg:
1 WaMa:
maj W..
(E.10)
a standard choice multimodal transformers [47]. Choosing T = vd converts Eq. (F.9) into the scaled dot-product of Vaswani et al. [48], and re-inserting the value tokens my. finally yields Eq. (B.1).
E.1.6 Consistency as +
Let a be the solution of the regularized optimization with temperature т. By the Ieconvergencc of entropic regularization 19, Thm. 1J. a² - oó 7 0, where aj is the exact Buclidean projection solution of Eq. (E.2). Because Wo. WK trainable and M. is fed through temperature-scaling LayerNorm block, the network approximate arbitrarily small practice, the learned weights converge the optimal MMSE estimator.
E.1.7 Connecting Back Causal Front-Door
Finally, note our estimator (E.1) uses only the bias-free query Ma(x) Rs E. | x]] and the raw visual tokens M, Because Ma U/ by design, the attention weights (xx) are conditionally independent of the confounder, guarantecing that M(x) = Attn(Ma, M,) is a pront-door-adjussee proxy for the latent mediator,
exactly as required by Perez et al. [41].
E.2 Derivation of the NWGM Approximation
This section provides a derivation for Normalized Weighted Geometric Mean (NWGM) approximation used main text Section 4.1 to implement the back- adjustment for language de-biasing. The goal is to estimate the causal effect P() do t))-
The back-door adjustment formula (Eq. 1) is:
P(Y do(T ()) = Σ -
B2( 2,
(E.11)
typical neural network classifier or segmenter, the conditional prohability P(Y f, obtained by applying a Softmax function to pre-activation scores (logits), denoted as (f, Thus, we are interested in computing:
P(Y do(T t)) =B<(5oltmax(s>((33)
(E.12)
The NWGM approximation (Eq. 2) states:
Ez Softmax(ey(tt =))] R (oztmax(BBz(oy(t,,,
(E.13)
To justify this approximation, we first consider the relationship between the arithmetic mean and the weighted geometrie mean (WGM). Let f(2) be function of random variable Z which takes values a with probabilities P(=). The arithmetic mean
Ez [=]s(=)] Σ (z))
(E.14)
The weighted geometric mean is:
WGMz[f(#)) P()
(E.15)
For many distributions, especially when the number of samples for Z is large or f(=) does not vary excessively. the arithmetic mean can be approximated the WGM:
Ez[s(=) ss [GM)[
(E.16)
Now, f(z) exp(g(=)) for some function (2). Substituting into WGM definition (Eq. (E.15)):
WGMzlexp(o(2)) Il (explo(=)))
II exp(o(=)
[ Eg(a)((a)
=ex
exp(B2lo(2))).
(E.17)
25

---

Combining Eq. (6) and Eq. (E.17):
Ez [2(ex((((2)) x exp(E2(b(=))).
(E.18)
The Softmax function for particular class output, given s, proportional exp(sx) (where ss is the exp(ss)
score for class k). Specifically, Softmax:(s) I, exp(s;)" The NWGM approximation essentially applies the relationship in Eq. (E.18) the logits *before* the normalization step inherent in Softmax, effectively moving the expectation inside exponential terms that dominate Softmax behavior. This leads to approximation in Eg. (3.13):
Es (sftmaxxxsy( =)) & (22
(E.19)
This approximation is commonly used and has been discussed in works like Baldi and Sadowski [2] and Liu [25].
With the additive score assumption (Section 4.1), (t, (1) (2)2), the expected score becomes:
Ez(sy((,,) Ez [s( s2))
= st) B2
(E.20)
Substituting this into Eq. (E.19) gives:
P(Y do(T = ()) zs Softmaxx(57(0 Ez(sz(=)])
(E.21)
The de-confounded score used for prediction is therefore (ty(t) = (t (2]s2(2)], presented in main text
Eq. 3. The term Ez [==(=)] is practically estimated by averaging the confounder embeddings fz(2) weighted by their empirical probabilities P(z;) from the training set, leading the de-biased text representation ff (t) in main text Eq. 4.
E.3 Proof of Consistency of the Memory- -Bnn Estimator
In this appendix provide rigorous justification of the Memory bank estimator defined in Eq. 9 of the main text. Recall that, for a fixed time index N, the estimator is
W
exp(sim(xx,
Wt,7 Xt-To
We,
(E.22)
E exp(sim(xx, Xe-o))
Here X2 E R is the fraee-level embedding for æt, W is the memory horizon, and sim(..) bounded dot-produce similarity.
Our goal show X, is consistent estimator of the front door expectation Ex (X'). The proof proceeds two steps:
1. Show the unweighted empirical mean
X,
X
W
is unbiased and converges Ex X') under short-range stationarity.
2. Bound the bias introduced by the importance weights {wr,r } and prove vanishes as + oc.
E.3.1 Preliminaries and Assumptions
(A1) Short-range stationarity. Over the window size W, sequence (X:-x W. is wide-sense stationary, ie. for any lag f, the joint law of (Xi.v, Xx-r-r) does not depend on т.
(A2) Finite second moment.
E Xa
[3]
80.
(A3) Bounded similarity. There is x > 0 such that (sim(x, S almost surely.
E.3.2 Unweighted Empirical Mean
Under (A1), the past embeddings X, X.- w form an i..d. sample from the marginal Px. Hence
B Ex [X'),
and by Kolmogorov's 's strong law of large numbers [16]
Ex |
(E.23)
26

---

E.3.3 Effect of Importance Weights
Write the weighted estimator as
X, X.
We.r W Xc-r-
1.)
Then
X X,Il2 I max X 112 l| w 10-1112
(B.24)
where W. (w.,,...,, W.,w). From (A3) the softmax weights satisty
e
I Wasr I We-n! VT,
We
|w: w 1|1, I 2(c²* 1) W Combining Eq. (E.24) with Eq. (E.25) taking expectation gives
(E.25)
B|J
X+Ila]
2(c 1) w E max X |la]
(B.26)
where Doob's maximal inequality and (A2) ensure the last expectation finite. Thus the weight- induced bias decays at rate OWW-)).
E.3.4 Consistency
By the triangle inequality.
|| - Ex [X']]
S
|X
The first term converges to O in 7." by Eq. (F.26), the second converges a.s. by Eq. (F.23). Hence
X.
prob.
Ex X'),
W-000
establishing consistency.
Remark. In practice W is fixed (e.g. W 8). Eq. (E.26) shows the residual bias is proportional (e 1)/W, and the model can trade off sharpness vs. bias the learned scale K.
E.3.5
Necessity of Softmax Weights
In Eq. main paper the memory-bank feature is a convex combination X, = Ete wy X w E
A W
Why choose the weights with Softmax?
Entropy-regullrizzed MMSE derivation. Let z' = E|X' xx] be the ideal feature we would predict if the full distribution were known. Inside the attention-linear family minimize the mean-square error while discouraging single frame from monopolizing the weight:
Σ. W, W WEAR
z- - (ww),
11
H(w) -, wy log Wry
(F.27)
where H(w) is the Shannon entropy and À > 0 is temperature balancing accuracy and diversity, Introducing a Lagrange multiplier for Σ, w, and taking derivatives gives
expl X1-))/-))/AA
W+ En exp( (xx, xA-a)/A)
which exactly the Softmax weight scale vd/R used multi-head attention. Thus Softmax is unique optimum of the entropy-regee rrguuaazee MMSE problem: it gives higher importance to frames more similar to the current one, yct avoids collapsing onto single frame.
More Qualitative Analysis
This section provides further qualitative evidence to support the claims made in the main paper. We first delive into an analysis of the inherent biases present in the training dataset and then showcase additional comparative segmentation results that highlight the superior performance of our CERES framework.
27

---

Top 30 Action-Object Co-occurrence Heatmap
bkg
chopsing beadd
gd
ORLICE
PX
12021
14
cles once fooge
up pude
diaer avti
put dlwer sharl
gaonge
thef
las tork
uo
the
pur
pite
Figure F.1: Co-occurrence frequency of object categoriis and actions in the VISOR training set. Darker cells indicate more frequent pairings. This visualization clearly shows strong correlations between certain objects and actions, demonstrating the statistical bias LBD module aims mitigate.
F1 Analysis of Dataset Bias
As discussed in the main paper (Section 1), one of the primary challenges in Ego-RVOS the presence of dataset biases, where certain object categories frequently co-occur with specific actions. This can lead models to learn spurious correlations rather than truly grounding the language query in the visual scene. To empirically demonstrate this, conducted a statistical analysis of the action-object pairings within the VISOR training set.
Figure F.2 illustrates the distribution of the most frequent actions (verbs) in the training queries. It is evident that actions such "cut". "take". and "put" are predominant.
Furthermore, Figure F.1 presents heatmap (or co-occurrence matrix analysis) showing the frequency of specific object categories (nouns) appearing with particular actions. This statistical imbalance naturally biases models trained on such data to favor common pairings, potentially failing on queries involving rarer but equally valid combinations.
These distributional statistics underscore the necessity for de- dic-biasing mechanisms like the Linguistic Back-door Deconfounder (LBD) in CERES. By explicitly modeling and adjusting for these confounders (as defined by object-action pairs and their frequencies, Section 4.1), CERES can achieve more robust language grounding and generalize better less frequent or novel combinations.
F.2 More Comparison of Segmentation Results
To further illustrate the robustness and accuracy of CERES, we present additional qualitative comparisons against the strong baseline, Action VOS [36], on challenging sequences from the VISOR validation set. These examples complement those shown in Figure in the main paper.
28

---

2500
800
2000
Frequency
600
Frequency
1500
400
1000
200
500
Median:
0
1000 2000
3000 4000 5000
200
400
800
1000
Index (sorted by frequency)
Index (sorted by frequency)
(a) Action Distribution
(b) Object Distribution
Figure F.2: Distribution of frequent actions and objects in the VISOR training set queries. This highlights skewed distribution, where few actions (objects) are significantly more common.
Table G.I: Performance (%) of CERES (ResNet101 backbone) on VISOR with different depth encoders for DAttn. Results are for the full CERES model. DAv2 refers to Depth Anything V2. The chosen configuration for main experiments (DAv2 ViT-B) is highlighted. (Bold indicates the best performance; underlined indicates the second-best.
Depth Encoder
mloU@1
mloU
gloUt
Acct
MiDaS (BEIT-L)
60.8
15.0
70.6
73.7
DAv2 (ViT-S)
61.1
14.9
71.1
74.5
DAv2 (VIT-B)
64.0
15.3
72.4
DAv2 (ViT-L)
64.7
16.2
72.5
76.2
Figure F.3 showeases scenarios involving (1) significant hand-object occlusion, (2) rapid camera motion leading motion blur, and (3) objects with subtle state changes.
These additional qualitative results, in conjunction reinforee the conclusion that CERES's ' dual-modal causal intervention strategy effectively addresses key biases and confounding factors, lcading to a more robust and reliable Ego-RVOS model. The VFD module enhances resilience to visual distortions common in egocentric video, while the LBD module improves generalization by mitigating reliance spurious statistical correlations learned from biased training data.
quantitative improvements reported the main paper.
G More Quantitative Analysis
This section provides additional quantitative analyses to further investigate the components and robustness of our CERES framework. All experiments are conducted on the VISOR dataset using ResNet101 backbone, unless otherwise specified.
G.1 Impact of Depth Encoder Choice in DAttn
Visual Front-door Deconfounder (VFD) utilizes depth features to guide aggregation visual semantic features via septh-guided Attention (DAttn). The quality and nature these depth features can influence the effectiveness of the mediator construction. To assess this, we evaluated CERES with different pre-trained monocular depth estimation models as the source of depth features. We compared MiDaS 1v3.1, BETT-L backhone) [42] with various sizes of Depth Anything V2 (DAv2) [59] models, specifically those based on ViT-Small (vit-s), ViT-Base (vit-b), and ViT-Large (vit-l). The results are presented Table G.
As shown Table G. the choice depth encoder impacts performance. Models from the Depth Anything family generally outperform MiDaS in this application. Within the DAv2 family, there a trend of improved performance with larger model sizes (ViT-S ViT-B VIT-L), with DAv2 (VTT-L. achieving the highest m (64.7%) and gloU (72.5%). However, DAv2 (ViT-B) provides a strong balance between performance (64.0% m 72.4% gloU) and computational cost/model size. Given this trade-off, selected DAv2 (VIT-B) the default depth encoder for CERES in our main experiments, its results are highly competitive while being essuuree-ffficien than the ViT-L variant. The slightly higher m for larger models might indicate
29

---

Original frame
Ground truth
Baseline
Ours
Clean
Opent microware
Mix ingreelien
Pick up Lray
Place chortoo
bag
Acress
pol
Figure F.:: Additional qualitative comparisons between ActionVOS (Baseline) and CERES on challenging Ego-RVOS scenarios. CERES consistently demonstrates robust and accurate segmentation in the presence occlusions, motion blur, and subtle interactions.
more complex feature space that could require further fine-tuning or regularization negative object suppression a primary concern. Overall, these results confirm that higher- quality depth information, as provided by more recent and powerful depth estimation models, contributes positively to the VFD module's ability to bia visual features.
G.2 Robustness to Depth Feature Degradation
To further the robustness of DAttn mechanism imperfections in depth information, simulated scenarios where depth map quality might compromised (e.g., due to challenging scenes, sensor noise, or limitations of the depth estimator). We conducted an experiment by adding varying levels of Gaussian noise to the normalized depth features extracted by the DAv2 (VIT-B) B) encoder before they are fed into the DAttn module. Specifically, for each depth feature vector fe. added noise A/(0, o I). where is the standard
30

---

80
Metric Value (%)
70
miou@D
55
GloU
Accuracy
50
0.1
0.2
0.8
1.0
Noise Level
Figure G.1: Performance of CERES (ResNet101, DAv2 ViT-B depth) on VISOR under varying levels of Gaussian noise added to depth features. m (blue), gloU (orange), and Accuracy (green) plotted against the noise standard deviation (o,). (a,, = means no noise.)
Table G.2: Computational overhead
on RTX 3090. (Bold indicates best; underlined indicates the second-best. Params include the frozen depth encoder (DAv2-B); FPS uses a memory window W =5)
Method
Backbone
Params (M)
FPS1
mIoU@
Action VOS
ResNet101
195
23.8
58.4
Action VOS
VSwin-B
237
15.4
62.9
CERES (Ours)
ResNet101 + DAv2-B
306
18.2
64.0
deviation of the noise. The performance of CERES on VISOR evaluated across range of noise levels on [0.0, 0.1, 0.2, 0.5, 0.8, 1.0, 2.0, 5.0).
The results, depicted in Figure G. show clear trend. For low moderate noise levels, CERES exhibits notable resilience. Specifically, when the noise standard deviation d. less than (i.c., for a, S 0.8 in tested discrete levels), the impact on performance is minimal. At 63.5% (a relative decrease of 0.78%), gloU drops from 72.4% 72.3% (a relative decrease of 0.14%), and Accuracy drops from 76.3% to 76.1% (a relative decrease of 0.26%) In all these cases, the relative performance degradation is less than 1% compared to the no-noise baseline. Even at aa = 1.0, gloU remains remarkably stable (72.4%) and Accuracy only slightly decreases 76.1 %, while mloU sees a modest drop to 63.2% (a 1.25% relative decrease).
0.8, @ drops from 64.0% to
As the noise intensity increases further (e.g.. d. = 2.0 and o., 5.0), the performance degradation becomes pronounced, particularly for m which drops 61.8% and 51.2% respectively. This indicates that while DAttn can effectively handle minor inaccuracies in depth features, highly comupted geometric information will naturally lead to a more significant decline in segmentation quality.
This graceful degradation under low to moderate noise levels (with performance loss under 1% for a. < 1.0) demonstrates the robustness of our causally-inn innspre vision-depth fusion approach. The DAttn mechanism appears capable of leveraging the general structure provided by depth accurate, while still underscoring the overall benefit of reasonably high-quality depth information for optimal performance.
when they perfectly
G.3 Overhead Comparison
quantify computational overhead introduced by our causal framework relative established baselines, we report end end-to-end throughput and parameter counts at 448 x 448 input resolution measured on an NVIDIA RTX 3090. The table below reproduces the setup described in the rebuttal and serves as the basis for overhead comparison.
31

---

According to Table G.2, relative to the ResNet101 Action baseline, our method increases the parameter count by approximately +56.9% (from 195M to 306M) and reduces throughput by 23.5% (from 23.8 18.2 FPS), reflecting the added depth pathway. In contrast, when compared to stronger VSwin-B Action baseline, our method carries a smaller parameter overhead +29. 29.1% (from 2 to 306M) yet delivers higher throughput by +18.2% (from 15.4 to 18.2 FPS), indicating that the observed efficiency gains stem from the causal design rather than scaling the RGB backbonc alonc.
Overall, these measurements isolate the overhead attributable to the depth encoder and the lightweight attention modules. While the separate depth pathway increases parameters and reduces FPS versus ResNet101-only baseline, the framework remains more efficient than upgrading RGB backhone Video Swin-B, achieving higher throughput at comparable or better accuracy.
32


Reviews:
[{"review_id": "WmTaHbeF1h", "reviewer": "Reviewer_NduK", "summary": "The paper proposes a causal inference-based framework, CERES, to address robustness issues in Egocentric Referring Video Object Segmentation (Ego-RVOS). Ego-RVOS aims to segment target objects involved in human actions based on language queries (e.g., \"knife used to cut carrot\") in first-person videos. CERES introduces dual-modal causal intervention, including a Linguistic Back-door Deconfounder (LBD) for language debiasing and a Visual Front-door Deconfounder (VFD) for visual confounding mitigation. Experiments on VISOR, VOST, and VSCOS datasets validate CERES's state-of-the-art performance. Ablation studies confirm the efficacy of each module, with theoretical proofs and implementation details provided in the appendix.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Standardized Presentation. Well-structured writing, clear figures, and intuitive framework diagrams.\n2. Rigorous Technical Foundation. Complete theoretical derivations (e.g., MMSE optimality proof in the appendix) and transparent implementation details.\n3. Practical Research Value. The focus on robust Ego-RVOS addresses a meaningful challenge in egocentric vision with real-world applicability.\n\nWeaknesses:\n1. Ambiguous Causal Rationale for Language Bias. Is the \"Language Bias\" solely defined as \"certain object categories frequently co-occur with specific actions\"? Is all knowledge of such co-occurrences harmful? Some correlations represent valid prior knowledge (e.g., \"open\" strongly associates with objects like \"fridge\" or \"cabinet\"), which can be beneficial for prediction. The paper lacks a clear distinction between harmful spurious correlations and potentially beneficial prior knowledge learned from data statistics. What specific types of \"spurious correlations\" does CERES target?\n2. Insufficient Novelty in Core Methodology. The key components of the VFD (Depth encoder, Memory-bank) and LBD modules rely heavily on established techniques. The overall RVOS segmentation backbone shows minimal innovation. Moreover, the Attention-Linear Family (ALF) assumption for mediator fusion may constrain representation power. Exploring nonlinear fusion mechanisms (e.g., gating) is recommended, or at minimum, ablation studies comparing ALF to nonlinear alternatives should be included.\n3. Incomplete Computational Analysis. Integrating the Depth encoder and Memory-bank (MAttn) likely increases computational cost. The paper lacks metrics quantifying this impact (e.g., inference speed/FPS, FLOPs, parameter count comparisons vs. baselines).\n4. Inadequate Experimental Explanation. For Fig.4: It is unclear how depth information specifically resolves occlusion. Visualizations (e.g., attention maps or depth overlays) illustrating the role of depth in handling occlusion are needed. And depth maps from Depth Anything V2 can be error-prone. The paper does not analyze CERES's sensitivity to depth estimation errors. Experiments testing robustness under noisy/depth maps or with weaker depth estimators are crucial.\n5. Inflexible & Dataset-Dependent Temporal Context. The fixed window size (W=5) for MAttn relies on a short-term stationarity assumption. Its performance (Fig. 5) appears highly dataset-dependent. This fixed window is likely suboptimal for sequences with long-term dependencies (e.g., objects leaving and re-entering the view). An adaptive window mechanism is strongly recommended instead of a fixed value.", "was_revised": true, "final_justification": "Overall, after weighing the authors’ rebuttal and the material they have promised to add in the revision, I find the paper well written, methodologically innovative, and experimentally sound. I therefore lean toward a borderline-accept recommendation.", "rebuttal": "We sincerely thank you for your insightful feedback and constructive comments. We are especially encouraged that you acknowledged our work for its **rigorous technical foundation**, with **complete theoretical derivations** and **transparent implementation**; its **practical research value** in addressing a meaningful challenge in egocentric vision; and its well-structured and clear presentation.\n\n---\n\n> `Weakness 1 / W1` Ambiguous Causal Rationale for Language Bias. Is the \"Language Bias\" solely defined as \"certain object categories frequently co-occur with specific actions\"? Is all knowledge of such co-occurrences harmful? Some correlations represent valid prior knowledge (e.g., \"open\" strongly associates with objects like \"fridge\" or \"cabinet\"), which can be beneficial for prediction. The paper lacks a clear distinction between harmful spurious correlations and potentially beneficial prior knowledge learned from data statistics. What specific types of \"spurious correlations\" does CERES target?\n\nThank you for this excellent point.\n\nWe agree that some statistical priors (e.g., ‘knives cut’) are beneficial. Our work targets the model’s **over-reliance** on these priors, a shortcut learning behavior that is especially harmful in Ego-RVOS. The core task is to determine if an object is actively involved in an action (aka. positive), and over-reliance on language priors causes failures on less common pairs (e.g., segmenting a 'knife' even if it's just lying on the counter, not being used for 'cutting'). \n\nBy de-biasing the text features, our LBD module dismantles these language-based shortcuts. **This prevents the model from defaulting to common pairings and instead necessitates a greater reliance on visual evidence.**\n\nTo validate that LBD specifically addresses this issue, we performed a new ablation on the VISOR \"hard\" subset (rare concepts where shortcuts fail):\n\nTable: Ablation on the VISOR \"hard\" subset.\n|Method|IoU⊕|gIoU|\n|-|-|-|\n|ActionVOS|58.40|69.90|\n|+ LBD only|61.4 (+3.0)|71.8 (+1.9)|\n|+ VFD only|60.5 (+2.1)|71.3 (+1.4)|\n|CERES|62.3 (+3.9)|72.2 (+2.3)|\n\nThis result demonstrates that LBD effectively addresses the targeted bias, which is also grounded in the statistical analysis of the dataset's co-occurrence bias detailed in **More Appendix D.1 of Supplementary Material zip**. We will revise the main text (Sec. 1 & 4.1) to state this critical distinction more explicitly.\n\n---\n> `W2` Insufficient Novelty in Core Methodology. The key components of the VFD (Depth encoder, Memory-bank) and LBD modules rely heavily on established techniques. The overall RVOS segmentation backbone shows minimal innovation. Moreover, the Attention-Linear Family (ALF) assumption for mediator fusion may constrain representation power. Exploring nonlinear fusion mechanisms (e.g., gating) is recommended, or at minimum, ablation studies comparing ALF to nonlinear alternatives should be included.\n\nThank you for this insightful comment regarding methodological novelty. We appreciate the opportunity to clarify our contribution and design choices.\n\n1.  **Novelty is the Causal Framework, Not the Base Architecture:** We respectfully clarify that our core contribution is the novel **causal framework (CERES)** itself, designed to adapt powerful, pre-trained models to the unique biases of the egocentric domain. The use of established components is intentional, as our goal is to demonstrate how our causal interventions can robustly enhance existing SOTA models for this challenging data shift. Our consistent gains across multiple backbones (Table 1) support this claim.\n\n2.  **Justification for Depth Fusion (DAttn):** We agree that the choice of fusion mechanism is critical. We opted for our Depth-guided Attention (DAttn) for two principled reasons:\n - **Causal Grounding:** The Attention-Linear Family (ALF) assumption allows our DAttn to be a direct implementation of Pearl’s front-door adjustment (see Appendix A), making it a theoretically grounded choice, not an ad-hoc one.\n - **Empirical Superiority:** In **Table 4 (row 3 vs. 4)**, we compare our DAttn against **a standard nonlinear MLP fusion (RGB+Depth)**. Our causally-guided DAttn (63.3% mIoU⊕) not only outperforms the MLP fusion (62.1% mIoU⊕) but also better discriminates against distractor objects (**15.8 vs. 17.5 mIoU⊖**).\n\nThis result reveals a key architectural insight. A standard MLP performs \"blind\" fusion, lacking the structure to leverage depth. In contrast, our DAttn explicitly uses depth as a geometric query to attend to relevant visual features while actively suppressing distractors. This structured reasoning is something a simple MLP cannot easily replicate, explaining the superior robustness of our causally-motivated design.\n\n---\n> `W3` Incomplete Computational Analysis. Integrating the Depth encoder and Memory-bank (MAttn) likely increases computational cost. The paper lacks metrics quantifying this impact.\n\nThank you for this excellent point. We have added the requested computational analysis, which shows that our causal framework is a more efficient path to state-of-the-art performance than simply scaling the backbone.\n\nTable R1: Computational Cost vs. Performance on the VISOR dataset.\n(FPS measured on an NVIDIA 3090 GPU at 448x448.)\n|Method|Backbone|Param. (M)|FPS|mIoU⊕ (%)|\n|-|-|-|-|-|\n|ActionVOS|ResNet101|195|23.8|58.4|\n|ActionVOS|VSwin-B|237|15.4|62.9|\n|Ours|ResNet101 + DAv2-B|306|18.2|64.0|\n\nCrucially, our method outperforms the stronger ActionVOS (VSwin-B) baseline in both accuracy (**+1.1% mIoU⊕**) and speed (**18% faster**). This demonstrates that our method’s efficiency is rooted in its design, not just model size.\n\nWhile the separate depth encoder adds overhead, these results validate our approach and strongly motivate future work on integrating these causal principles into a single or more efficient encoder, potentially through knowledge distillation or structured pruning.\n\n\n---\n> `W4` Inadequate Experimental Explanation. For Fig.4: It is unclear how depth information specifically resolves occlusion. Visualizations (e.g., attention maps or depth overlays) illustrating the role of depth in handling occlusion are needed. And depth maps from Depth Anything V2 can be error-prone. The paper does not analyze CERES's sensitivity to depth estimation errors. Experiments testing robustness under noisy/depth maps or with weaker depth estimators are crucial.\n\nThank you for these insightful questions, which allow us to clarify the core mechanism and robustness of our method.\n\n1. **On the Role of Depth in Handling Occlusion:**\n\nOur approach is founded on a key principle: occlusion severely degrades the semantic understanding derived from RGB features, but the object's spatial geometry remains far more stable when captured by a depth map.\n\nFor example, when an object like a \"knife\" is occluded by a \"hand,\" its visual semantics are confounded by the hand's appearance, leading to feature bias and segmentation errors. However, the depth map preserves the distinct spatial planes, clearly delineating the foreground occluder from the background object. Our Depth-guided Attention (DAttn) directly leverages this by using the stable depth features (Md) to guide attention over the less reliable RGB features (Mv). This implements a front-door adjustment, forcing the model to ground its segmentation in the object's consistent spatial location.\n\nTo provide direct visual evidence, we will add the attention map visualizations you suggested to the appendix.\n\n2. **On Robustness to Depth Estimation Errors:**\n\nThis is an excellent point, and we agree it is crucial to validate. We have performed this exact analysis in **Supplementary Material zip file of More Appendix E.2 (Figure E.1)**.\n\nOur experiments show that CERES is highly robust to imperfections in the depth map. We tested this by corrupting the depth features with progressively stronger Gaussian noise. The results show that even with significant noise (std. dev. σn = 0.8), the performance degradation is minimal: **mIoU drops by less than 1% relative to the baseline (64.0% → 63.5%)**. This graceful degradation confirms that our DAttn mechanism effectively uses the general geometric structure and does not depend on pixel-perfect depth. We will add a sentence to the main paper to more clearly reference this appendix analysis.\n\n---\n> `W5` Inflexible & Dataset-Dependent Temporal Context. The fixed window size (W=5) for MAttn relies on a short-term stationarity assumption. Its performance (Fig. 5) appears highly dataset-dependent. This fixed window is likely suboptimal for sequences with long-term dependencies (e.g., objects leaving and re-entering the view). An adaptive window mechanism is strongly recommended instead of a fixed value.\n\nThank you for this insightful comment regarding our temporal modeling.\n\nOur use of a fixed window (W=5) was a deliberate design choice, representing a balance between model simplicity and strong performance on established Ego-RVOS benchmarks.\n\nWe agree that an adaptive mechanism is more powerful for general, long-form videos. However, current standard datasets are not structured to support this; for instance, clips in the VISOR benchmark are typically sampled to an average of **10 frames with a 0.8-second interval**. This short-clip structure makes it very difficult to meaningfully train or evaluate a more complex long-term memory module. Our simpler fixed-window approach is thus well-suited and demonstrably effective for this data regime (as shown in Figure 5), avoiding unnecessary complexity.\n\nWe concur that as long-form egocentric video datasets become available, developing an adaptive temporal model will be a crucial next step. Architectures like X-Mem [1] would be a promising direction for such future work, and we will clarify this motivation in our final manuscript.\n\n[1] Ho Kei Cheng, et al. X-Mem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model. In ECCV 2022."}, {"review_id": "r5Jxf4WbpQ", "reviewer": "Reviewer_myER", "summary": "This paper proposes a new RVOS method, named CERES. CERES addresses two challenges: confounders in visual images and language representation biases, respectively. Specifically, the visual confounders are addressed by intergrating depth data, while the language presentation biases is addressed by intergrating data statistics.", "questions": "Please refer to weaknesses.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. CERES achieves state-of-the-art results and significantly outperforms ActionVOS, demonstrating its strong effectiveness.\n2. The consideration of biases in both visual and textual information is novel and interesting.\n3. CERES performs well across different network backbones, highlighting its practical applicability and flexibility.\n\nWeaknesses:\n1. The integration of data statistics and depth information seems somewhat detached from the earlier causal bias analysis. The method mainly improves performance by adding depth data and data statistics to the RVOS model through mechanisms like attention or concatenation, but the connection to the prior bias framework is not clearly established.\n2. While ActionVOS includes analysis of state-changing objects, it remains unclear whether CERES can be effectively applied to such scenarios.\n3. It would be valuable to understand how CERES performs in open-vocabulary settings, particularly on unseen objects or actions.", "was_revised": true, "final_justification": "Thanks for the authors' reply and extra analysis of my questions. Now I have a clearer understanding of CERES's ability in state-change scenarios and open-vocabulary settings. I have decided to raise my rating.", "rebuttal": "We sincerely thank you for your thorough evaluation and constructive feedback. We are encouraged that you found our work to be effective, novel, and practical, highlighting that CERES achieves **state-of-the-art results**, that our **dual-modal causal approach is novel and interesting**, and that its **strong performance across different backbones shows its practical applicability**.\n\n\nYour main suggestions provide an excellent opportunity for us to clarify key aspects of our method. Below, we address each point in detail, using existing experimental evidence to: (1) further clarify the principled connection between our causal theory and its implementation `Weakness 1 / W1`; (2) demonstrate CERES's effectiveness on state-changing objects `W2`; and (3) highlight its strong open-vocabulary performance on unseen concepts `W3`.\n\n---\n\n> `W1` The integration of data statistics and depth information seems somewhat detached from the earlier causal bias analysis. The method mainly improves performance by adding depth data and data statistics to the RVOS model through mechanisms like attention or concatenation, but the connection to the prior bias framework is not clearly established.\n\nWe sincerely thank you for this insightful comment. It highlights a crucial point: the connection between our causal framework and its practical implementation must be explicit and well-justified. We agree that this link is the most important part of our paper, and we appreciate the opportunity to clarify it.\n\nOur architectural choices are not ad-hoc additions but are, in fact, principled and direct implementations of the causal adjustment formulas derived from our analysis. The key contribution is ***how*** **we handle linguistic bias and visual bias, which is dictated entirely by our causal graph (Fig. 1c)** to solve two distinct problems.\n\n1. **Linguistic Bias → Back-door Adjustment (Using Data Statistics):**\n   - **The Problem:** Models learn dataset shortcuts, like assuming the query \"cut carrot\" means *any* knife in the scene should be segmented, even one lying inactive on the counter. This is an observable confounding bias `Z` linking text `T` to output `Y`.\n   - **The Causal Solution:** To break this shortcut, we apply the back-door adjustment formula, $\\mathbb{E}_z[P(Y|T,z)]$.\n   - **Our Principled Implementation (LBD):** Our LBD module (Eq. 4) implements this by calculating an \"average confounding effect\" from the dataset statistics and using it to de-bias the text features. This is not arbitrary concatenation; it is a specific operation designed to remove the statistical shortcut. This forces the model to ground the language query in *visual evidence of the action*, rather than just co-occurrence statistics.\n2.  **Visual Bias → Front-door Adjustment (Using Depth-guided Attention):**\n    - **The Problem:** Egocentric video is plagued by occlusions and motion blur. This is an **unobserved** confounder `U` that corrupts the visual input `X`. We cannot adjust for `U` directly.\n    - **The Causal Solution:** We apply the front-door adjustment, which requires a mediator `M` that is robust to the confounder `U`.\n    - **Our Principled Implementation (DAttn):** A standard visual feature `M_v` (from RGB) is confounded by occlusion. However, an object's geometric structure `M_d` (from depth) remains far more stable. Our Depth-guided Attention (DAttn) is the direct embodiment of this principle: we use the **robust depth features (`M_d`) as the Query to guide attention over the confounded RGB features (`M_v` as Key/Value)**. Intuitively, the model is forced to ask using stable geometry: *\"Where in this occluded, blurry visual scene is the object that has this specific shape and spatial location?\"*\n\nTo provide empirical evidence that this principled design is superior to simple feature fusion, our ablation in **Table 4** shows that our causally-guided DAttn (**63.3% mIoU⊕**) substantially outperforms a naive MLP-based depth fusion (**62.1% mIoU⊕**). This result confirms that *how* depth is integrated, guided by the front-door principle, is more critical than its mere presence.\n\nWe sincerely thank you again for this valuable feedback. In our final version, we will revise the methodology section to more explicitly and forcefully articulate these connections, ensuring the link between our causal theory and each architectural choice is unmistakable.\n\n---\n> `W2` While ActionVOS includes analysis of state-changing objects, it remains unclear whether CERES can be effectively applied to such scenarios.\n\nThank you for this insightful question regarding the model's performance on state-changing objects. We agree that this is a critical aspect of robust video object segmentation.\n\nTo address this, we have evaluated our method on the VSCOS dataset, which is specifically **designed to assess performance on objects undergoing significant transformations**. Our results, presented in Table 3, show that CERES achieves a higher **mIoU of 55.3% compared to 52.5%** for ActionVOS on this dataset.\n\nTo further highlight this capability, we will add qualitative visualizations from the VSCOS dataset to the appendix, showcasing successful segmentation of objects during state changes.\n\n---\n> `W3` It would be valuable to understand how CERES performs in open-vocabulary settings, particularly on unseen objects or actions.\n\nThank you for this insightful question regarding the model's open-vocabulary performance. We agree that evaluating on unseen concepts is crucial for demonstrating robustness.\n\nWe addressed this in two ways in our experiments, which we will clarify further in the final version:\n1. **Unseen Concepts within VISOR (Table 2):** Thank you for highlighting the need to better explain this. Table 2 evaluates performance on a challenging subset of the VISOR validation data that we created, containing only objects or actions not seen during training. This directly tests the model's ability to generalize beyond learned statistical correlations. On this \"novel\" subset, CERES achieves a **60.0% mIoU⊕**, a significant improvement of **+4.7%** over the strong ActionVOS baseline, highlighting its robustness.\n2. **Generalization to New Datasets (Table 3):** The evaluations on VSCOS and VOST are zero-shot generalization experiments. Our model is trained only on VISOR data and then evaluated directly on the VSCOS and VOST validation sets without any fine-tuning. These datasets feature different objects, actions, and visual distributions. Our consistent state-of-the-art performance here demonstrates a strong capability to generalize open-vocabulary scenarios.\n\nWe appreciate the feedback and will revise Section 5 to more explicitly frame these results in the context of open-vocabulary and zero-shot generalization for the camera-ready version."}, {"review_id": "ZScvclspeH", "reviewer": "Reviewer_gqxt", "summary": "This paper proposes a new framework, CERES, for Egocentric Referring Video Object Segmentation (Ego-RVOS). The core idea is to improve model robustness by addressing two sources of spurious correlations using principles from causal inference. \n\nFirst, to counteract language biases in datasets (e.g., \"knife\" always co-occurring with \"cut\"), the framework uses back-door adjustment on the text features, which fix the bias using dataset statistics. \nSecond, to handle visual confounding factors common in egocentric video (e.g., motion blur, occlusions), it applies front-door adjustment. A key component of this is a novel visual mediator that uses geometric depth features, which are assumed to be more robust to confounders, to guide the aggregation of semantic RGB features via an attention mechanism. \n\nThe authors demonstrate that CERES achieves state-of-the-art performance on several Ego-RVOS benchmarks.", "questions": "My major concern is that this paper should have been built upon a comprehensive set of suitable baselines due to its heavy reliance on the pretrained models, followed by incremental experiments incorporating +CERES to demonstrate its generalizability, rather than being presented as a standalone SOTA method for comparison. \nFor example, this paper demonstrates the feasibility on ActionVOS, but does not provide evidence of its applicability to other pre-trained models, such as RefFormer or HOS.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The performance of this paper is good, showing significant improvement over the baseline ActionVOS.\n- The method used in this paper is effective, as it takes into account the bias in text-based datasets and integrates depth information visually.\n\n\nWeakness：\n- This paper lacks a well-defined motivation for incorporating depth information, which provides the largest improvement, and makes an unfair comparison with the baselines. In Table 4, it can be observed that the introduction of depth information through DAttn yields the most significant improvement, with the mIoU increasing from 59.9 in the first row to 63.3 in the third row. Moreover, it is widely acknowledged that depth information provides a strong prior for segmentation, as evidenced both intuitively and through experiments such as those in DepthAnything. Therefore, I think the way this paper addressing motion blur is somewhat trivial, and I remain skeptical of the theoretical framework presented in this paper.\n- Using a pre-trained model instead of training from scratch reduces the originality of this paper. Line 269 mentions the use of a pre-trained model, and the number in the first row of Table 4 is identical to that of the ActionVOS method with a Res101 backbone in Table 1. This indicates that the pre-trained model from ActionVOS was used. In other words, the method's performance improvement is built upon the combination of a pre-trained model and depth information, which, to some extent, affects the validity of this paper.\n- This illustrations, especially the egocentric images, are very difficult to see clearly, including Figure 1, Figure 3, and Figure 4.", "was_revised": true, "final_justification": "Thanks the authors for their detailed responses, which partly resolves my major concern stated in “Questions”. I decide to raise my score to “borderline accept”.", "rebuttal": "We sincerely thank you for your detailed feedback and for recognizing the key strengths of our work. We are pleased that you found our causally-motivated approach **effective** and our results **good**, noting the **significant improvement** over the baseline. We are particularly encouraged that you acknowledged the core of our contribution: a principled method that corrects for text-based dataset bias and visually integrates depth information via a **novel visual mediator**. \n\nIn our response, we will address your valid concerns regarding the motivation for depth `Weakness 1 / W1`, the fairness of our comparisons `W1`, and the generalizability of our framework `W2, Question 1 / Q1`.\n\n---\n\n> `W1` This paper lacks a well-defined motivation for incorporating depth information, which provides the largest improvement, and makes an unfair comparison with the baselines. In Table 4, it can be observed that the introduction of depth information through DAttn yields the most significant improvement, with the mIoU increasing from 59.9 in the first row to 63.3 in the third row. Moreover, it is widely acknowledged that depth information provides a strong prior for segmentation, as evidenced both intuitively and through experiments such as those in DepthAnything. Therefore, I think the way this paper addressing motion blur is somewhat trivial, and I remain skeptical of the theoretical framework presented in this paper.\n\nWe sincerely thank you for your thoughtful and challenging questions. These points are crucial, and we appreciate the opportunity to clarify the motivation for our work, the fairness of our evaluation, and the novelty of our theoretical framework.\n\n1. **Causal Motivation for Integrating Depth**.\n\nWe agree that depth is a strong prior for segmentation. However, our motivation is not simply to leverage this prior for a performance boost. Instead, our core contribution is a **causal framework** designed to mitigate specific confounding effects in egocentric video, and depth information is the chosen instrument to realize our proposed causal intervention.\n\nOur central hypothesis is this: unobserved confounders ($U$) like **occlusion and motion blur** primarily corrupt the semantic information available in RGB frames ($X$), leading to biased features. However, the scene's underlying geometric structure, captured by a depth map, is significantly more robust to these specific visual distortions. For example, when a hand occludes a knife, the knife's semantic RGB features become entangled with the hand's, but the depth map preserves their distinct spatial planes.\n\nTheoretically, we frame this using **front-door adjustment**. Our Depth-guided Attention (DAttn) module is the explicit mechanism for this intervention. It uses the robust geometric mediator ($M_d$ from depth) to guide and disambiguate the less reliable semantic mediator ($M_v$ from RGB). This principled design directly blocks the confounding path from the visual input to the output, making it far more than a trivial application of a known prior.\n\n2. **On the Fairness of Comparison.**\n\nTo demonstrate that our contribution lies in how depth is integrated, not merely its presence, we respectfully point to the ablation study already in our paper.\n\n- Evidence from Table 4: We have explicitly compared our DAttn against **a naive MLP-based RGB+depth fusion (Row 3 vs. Row 4 in Table 4)**. The results are telling:\n  - The simple MLP fusion improves target segmentation (mIoU⊕ from 59.9% to 62.1%) but **worsens discrimination**, increasing errors on non-target objects (mIoU⊖ from 16.3 to 17.5).\n  - In contrast, our causally-motivated DAttn not only achieves superior target segmentation (mIoU⊕ 63.3%) but also improves discrimination, reducing non-target errors (mIoU⊖ to 15.8).\n\nThis directly validates that the structure of our intervention is **critical and non-trivial**. The comparison with baselines is therefore a fair evaluation of our causal framework against non-causal alternatives. The performance gain is evidence that our method for addressing confounding is effective.\n\n3. **New Supporting Evidence for our Theoretical Framework.**\n\nTo provide more direct, intuitive evidence for our claims, we have produced new attention visualizations for challenging occlusion scenarios. We observe that:\n- In a representative case, the baseline model's attention incorrectly shifts to the occluding \"hand\".\n- In contrast, our model, guided by DAttn, maintains a tight and accurate focus on the occluded \"knife\".\n\nWe will add these compelling visualizations to the appendix. \n\n---\n\n> `W2` Using a pre-trained model instead of training from scratch reduces the originality of this paper. Line 269 mentions the use of a pre-trained model, and the number in the first row of Table 4 is identical to that of the ActionVOS method with a Res101 backbone in Table 1. This indicates that the pre-trained model from ActionVOS was used. In other words, the method's performance improvement is built upon the combination of a pre-trained model and depth information, which, to some extent, affects the validity of this paper.\n\n> `Q1` My major concern is that this paper should have been built upon a comprehensive set of suitable baselines due to its heavy reliance on the pretrained models, followed by incremental experiments incorporating +CERES to demonstrate its generalizability, rather than being presented as a standalone SOTA method for comparison. For example, this paper demonstrates the feasibility on ActionVOS, but does not provide evidence of its applicability to other pre-trained models, such as RefFormer or HOS.\n\n\nWe appreciate your insightful feedback and appreciate the opportunity to clarify our contribution.\n\nFirst, we would like to precisely clarify our model's heritage, as our method builds upon the **pre-trained weights of ReferFormer, not those of ActionVOS**. To be more specific, we inherited the successful **task-specific framework** proposed by ActionVOS for Ego-RVOS, which includes their designs for using an \"action as prompt\", a \"positive prediction head\", and an \"Action-guided Focal Loss\".\n\nThe identical performance between our baseline (Table 4, row 1) and the ActionVOS results (Table 1) is intentional and serves to demonstrate that we have faithfully reimplemented their framework as a strong starting point.\n\nHowever, our key observation, which motivates this work, is that these foundational models are often **pre-trained on large-scale third-person video datasets** and ***do not*** **explicitly account for the unique biases inherent in the egocentric perspective**. Our core innovation then builds upon this baseline by introducing a principled causal framework (CERES) specifically designed to address the unique visual and linguistic biases in egocentric video that are overlooked by standard models.\n\nTo directly address your valid concern about generalizability, we have conducted new experiments showing our framework's effectiveness on the original ReferFormer model, further demonstrating its modularity. We agree completely with your excellent suggestion that framing our work as a generalizable framework is a more powerful way to convey its value. The results below confirm that our modules provide a notable performance lift on a different architecture, proving they are ***not*** **overfitted to the ActionVOS design**:\n\nTable: Applying CERES to ReferFormer (ResNet101) on VISOR\n|Method|mIoU⊕ (%) ↑|gIoU (%) ↑|\n|-|-|-|\n|ReferFormer|59.9|63.1|\n|+CERES (Ours)|62.5 (+2.6)|66.8 (+3.7)|\n\nThese positive results encourage us to view CERES as a **foundational approach for improving robustness** across various egocentric vision tasks. We believe the same causal principles could be highly effective for other challenges like egocentric referring expression comprehension or video question answering.\n\nIn summary, we thank you for your valuable guidance. We will revise the manuscript to better articulate these details and frame our contributions as a generalizable causal framework, and we look forward to exploring these broader applications in future work.\n\n---\n\n> `W3` These illustrations, especially the egocentric images, are very difficult to see clearly, including Figure 1, Figure 3, and Figure 4.\n\nThank you for your valuable feedback regarding the clarity of our figures. These images indeed need to be enlarged to be clear.\n\nTo rectify this, we will add high-resolution, enlarged versions of these figures to the appendix in our revised manuscript. We will also improve their size and layout within the main text for the camera-ready version as space allows."}, {"review_id": "DpSCMhk9M7", "reviewer": "Reviewer_VF42", "summary": "This paper discusses Egocentric Referring Video Object Segmentation (Ego-RVOS), a task focused on segmenting objects involved in human actions in egocentric vision, guided by language queries. The task is challenging due to language biases and limitations of the egocentric view. In this work, the authors introduce Causal Ego-Referring Segmentation (CERES), a novel framework leveraging causal inference. CERES applies principles based on back door adjustment and front door adjustment to counteract the confounding issues. The proposed model was evaluated on three public egocentric vision datasets and achieved state-of-the-art performance.", "questions": "1. In table 1, why is \"Ours\" in bold but have higher mIoU⊖ and cIoU⊖?\n2. In table 2, ReferFormer+ the second best mIoU⊖, why is \"ours\" underlined?\n3. In table 4, adding LBD shows clear advantage when not using DAttn and MAttn. However, the difference in numbers between the last two rows (w/ and w/o LBD) is small. Are there more analyses that justify the effectiveness of LBD in the final design?\n4. This work identified two confounding sources: observable language bias and unobservable visual confounding. Is there existing work or analysis supporting these as the primary confounding factors?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The clarity of this paper is very good. Overall, this paper is well-written, with clear format, tables and math.\n2. The proposed method is evaluated across multiple datasets, achieving state-of-the-art performance.\n3. The ablation study results reported in this work support the framework design choices and helps understanding.\n\nMajor weakness:\n\nN/A\n\nMinor weakness:\n\nLine 31: \"because because\".\n\nSome tables (1,2,4) may be inaccurate and requires clarification (see questions).\n\nOver all, this paper is well-written. It introduces an interesting idea and the experimental results show potential . Due to limited understanding behind the math in causal reasoning, my current rating for this paper is borderline accept, and will be adjusted after rebuttal.", "was_revised": true, "final_justification": "The rebuttal answered all my questions and I will maintain my original rating.", "rebuttal": "We sincerely thank you for your valuable time and the positive and constructive assessment. We are delighted that you found our paper to be **well-written and clear**, our results to achieve **state-of-the-art performance** across multiple datasets, and our ablation studies to be **supportive and insightful**.\n\nCrucially, we appreciate that you recognized the core contribution of our work as a **\"novel framework leveraging causal inference\"** to tackle the complex challenges of Egocentric Referring Video Object Segmentation. Your thoughtful questions have allowed us to further clarify the foundations and impact of our method, and we have addressed each point in detail below.\n\n---\n\n> `Minor weakness`\n\nWe are sincerely grateful for your meticulous review and conscientious attention to detail. Your careful and responsible reading has helped us identify and correct several minor yet important errors, significantly improving the quality and precision of our paper.\n\n> `Weakness 1 / W1` Line 31: \"because because\".\n>\n> `Question 1 / Q1` In table 1, why is \"Ours\" in bold but have higher mIoU⊖ and cIoU⊖?\n>\n> `Q2` In table 2, ReferFormer+ the second best mIoU⊖, why is \"ours\" underlined?\n\nYou are absolutely correct on all three points you raised:\n- The typo on Line 31 (\"because because\") was an oversight.\n- The bolding in **Table 1** for `mIoU⊖` and `cIoU⊖` was incorrect. As lower is better, ReferFormer+ indeed achieves the best scores for these metrics with the R101 backbone.\n- Similarly, the underlining in **Table 2** for `mIoU⊖` was a mistake. HOS and ReferFormer+ have the first and second-best scores, respectively.\n\nOur main performance claims are based on the primary metrics (mIoU↑, gIoU↑, etc.), where our method does achieve state-of-the-art results. We have corrected the bolding and underlining in the final version to accurately reflect the top performers for every metric. We are very grateful for this careful correction.\n\n\n---\n\n> `Q3` In table 4, adding LBD shows clear advantage when not using DAttn and MAttn. However, the difference in numbers between the last two rows (w/ and w/o LBD) is small. Are there more analyses that justify the effectiveness of LBD in the final design?\n\nThank you for this insightful question.\n\nThe primary role of Linguistic Back-door Deconfounder (LBD) is to mitigate the model's over-reliance on spurious language-level correlations, a bias that is most detrimental when encountering rare object-action pairs. To isolate and highlight this specific contribution, we performed a new ablation study on the VISOR \"hard\" subset, which is composed of such rare concepts. The results below demonstrate that LBD is, in fact, the most impactful component for improving generalization in these challenging cases.\n\nTable: Ablation on the VISOR \"hard\" subset.\n|Method|IoU⊕|gIoU|\n|-|-|-|\n|ActionVOS|58.40|69.90|\n|+ LBD only|61.4 (+3.0)|71.8 (+1.9)|\n|+ VFD only|60.5 (+2.1)|71.3 (+1.4)|\n|CERES|62.3 (+3.9)|72.2 (+2.3)|\n\nAs shown, adding LBD alone provides a larger gain in mIoU⊕ (+3.0%) than adding the VFD alone (+2.1%) on this difficult subset. This directly confirms that LBD is highly effective at addressing its targeted bias, forcing the model to ground its decisions in visual evidence rather than statistical shortcuts. While the Visual Front-door Deconfounder (VFD) provides a significant general performance boost, the LBD is a crucial component for robust generalization.\n\nWe will add this new table and analysis to the paper to better clarify the distinct and critical role of LBD.\n\n---\n> `Q4` This work identified two confounding sources: observable language bias and unobservable visual confounding. Is there existing work or analysis supporting these as the primary confounding factors?\n\nThank you for this important question, which allows us to clarify the established foundation of our work. \n\nOur selection of confounders is a direct response to two distinct challenges well-documented in vision-language research. Specifically, we address (1) **observable language biases arising from dataset statistics**, a known problem tackled with backdoor adjustment in prior work [1, 2], and (2) **unobservable visual confounding from inherent egocentric distortions** like occlusion and motion blur, which have been identified as key challenges in egocentric video analysis [3]. Our unified framework, which applies the appropriate causal tool to each problem, draws inspiration from comprehensive causal approaches [4]. Applying a formal causal model to diagnose and address these specific challenges in the context of egocentric video is a key innovation of our work.\n\nBelow, we elaborate on these two pillars of our methodology:\n\n1. **Observable Language Bias (Z)**:\n\nThe problem of models exploiting statistical shortcuts instead of learning genuine relationships is a known source of failure. In the image captioning domain, Liu et al. (2022) [1] explicitly identify and address a \"linguistic confounder\" using backdoor adjustment to prevent the model from relying on spurious object-attribute correlations. We argue that **Ego-RVOS faces an analogous challenge**, where frequent verb-noun pairings in the training data (e.g., \"cut\" consistently appearing with \"knife\") create a powerful but brittle shortcut. We model this as an observable confounder Z, as its statistics are directly computable from the dataset. Applying backdoor adjustment is, therefore, a principled and established strategy to block this spurious causal path.\n\n2. **Unobservable Visual Confounding (U):**\n\nThe inherent nature of egocentric video introduces a separate class of confounders. As Shen et al. [3] and others have noted, factors like severe **\"occlusion,\" \"blur,\" and \"frequently changing object appearance\"** are defining characteristics of egocentric data. These visual artifacts are not explicitly labeled, yet they directly impact both the visual input X and the final segmentation Y. This makes them a classic unobserved confounder U, for which backdoor adjustment is impossible. This motivates our use of the front-door criterion, and our primary innovation here is the design of a novel vision-depth mediator specifically engineered to be robust to these distortions, thereby isolating the true causal effect of the visual input.\n\nIn the final version of our paper, we will revise the related work section to incorporate this discussion and these citations more explicitly, making it clear how our work builds upon these foundational insights to address the specific, dual challenges of robust Ego-RVOS.\n\n\n[1] Bing Liu, et al. Show, deconfound and tell: Image captioning with causal inference. In CVPR, 2022.\n\n[2] Xu Yang, et al. Causal Attention for Vision-Language Tasks. In CVPR, 2021.\n\n[3] Yuhan Shen, et al. Learning to segment referred objects from narrated egocentric videos. In CVPR, 2024.\n\n[4] Liuyu Wang, et al. Vision-and-language navigation via causal learning. In CVPR, 2024."}]

Author rebuttal:

Meta-review: This paper received final ratings of 4 borderline accepts. Initially, the reviewers praised the paper for its practical nature and value; state of the art results across many backbones; and thorough evaluations. Regardless, the reviewers raised some issues in their reviews, namely: the definition of bias and how this was added into the causal framework and the motivation for including depth and whether this makes the results fairly comparable with previous methods. During the rebuttal stage, a lot of the time was spent discussing these two aspects, particularly on the need for depth and how biases were integrated. As a result of this, all reviewers ended with a positive recommendation towards the paper. The AC agrees with the reviewers' comments and final scores and sees no reason to overturn the majority decision for this paper and recommends acceptance for this paper.

The AC reminds the authors to add in the additional discussions from the rebuttal and update the camera ready with the promises that they made during the rebuttal.
Decision: Accept (poster)