Interaction-aware Representation Modeling With Co-Occurrence Consistency for Egocentric Hand-Object Parsing
Authors: YUEJIAO SU
Yi Wang
Lei Yao
Yawen Cui
Lap-Pui Chau
Venue: ICLR 2026 Poster (2026)
Keywords: Egocentric vision
human-environment interaction
hand-object parsing
consistency
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: A fine-grained understanding of egocentric human-environment interactions is crucial for developing next-generation embodied agents.
One fundamental challenge in this area involves accurately parsing hands and active objects.
While transformer-based architectures have demonstrated considerable potential for such tasks, several key limitations remain unaddressed:
1) existing query initialization mechanisms rely primarily on semantic cues or learnable parameters, demonstrating limited adaptability to changing active objects across varying input scenes;
2) previous transformer-based methods utilize pixel-level semantic features to iteratively refine queries during mask generation, which may introduce interaction-irrelevant content into the final embeddings;
and 3) prevailing models are susceptible to ``interaction illusion'', producing physically inconsistent predictions.
To address these issues, we propose an end-to-end Interaction-aware Transformer (InterFormer), which integrates three key components, i.e., a Dynamic Query Generator (DQG), a Dual-context Feature Selector (DFS), and the Conditional Co-occurrence (CoCo) loss. 
The DQG explicitly grounds query initialization in the spatial dynamics of hand-object contact, enabling targeted generation of interaction-aware queries for hands and various active objects.
The DFS fuses coarse interactive cues with semantic features, thereby suppressing interaction-irrelevant noise and emphasizing the learning of interactive relationships.
The CoCo loss incorporates hand-object relationship constraints to enhance physical consistency in prediction. Our model achieves state-of-the-art performance on both the EgoHOS and the challenging out-of-distribution mini-HOI4D datasets, demonstrating its effectiveness and strong generalization ability.
Code and models are publicly available at https://github.com/yuggiehk/InterFormer.

Published as a conference paper at ICLR 2026
LNTERACTION-AWARY REPRESENTATION MODELING WITH CO-OCCURRENCE CONSISTENCY FOR EGOCEN- TRIC HAND-OBJECT PARSING
Yuejiao Su, Wang, Lei Yao, Yawen Cui, Lap-Pui Chau The Hong Kong Polytechnic University rayyoh. su, fyi-eis wang, yawen. cui, lap-pui. chau chauu@polyu.cco edu,
wwapeeecttolly..... polyu.
ABSTRACT
A fine-grained understanding of egocentric Human-environneen interactions is crucial for developing next-generation embodied agents. One fundamental chal- lenge this area involves accurately parsing hands and active objects. While transformer-based architectures have demonstrated considerable potential for such tasks, several key limitations remain unaddressed: I) existing query initialization mechanisms rely primarily on semantic cues or learnable parameters, demonstrat- ing limited adaptability to changing active objects across varying input scenes: previous transformer-based methods utilize pixel-level semantic features to iter- atively refine queries during mask generation, which may introduce interaction- irrleevatt content into the final embeddings; and 3) prevailing models ceptible to "interaction illusion", producing physically inconsistent predictions. To address these issues, we propose an end-to-end Interaction-aware Transformer (InterFormerr, which integrates three key components, i.e. a Dynamic Query Gen- erator (DQG), Dual-context Feature Selector (DFS), and the Conditional Co- occurrence (CoCo) loss. The DQG explicitly grounds query initialization in the spatial dynamics of hand-object contact, enabling targeted generation of interac- tion-aware queries for hands and various active objects. The DFS fuses coarse interactive cues with semantic features, thereby suppressing noise and emphasizing the learning of interactive relationships. The CoCo loss incorporates hand-object relationship constraints to enhance physical consistency in prediction. Our model achieves state-of-the-an performance both the Ego- HOS and the challenging cidt---distribution mini-HO14D datasets, demonstrating its effectiveness and strong generalization ability. Code and models are publicly available at https:::///. /ttttp:://gttub..... wwt. www.//uggieh///nnnnoooooo wwweeehwwwinteeoeeeee
INTRODUCTION
Recent advances in personal terminal devices such as GoPros and head-mounted devices (HMDs)
have driven significant increase in sharing
view (FPV, or egocentric) images and videos
on various social media platforms (Xu et al.) 2024a] Fan et al.
2025 Chen et al. 2025 Cartillier
al. 2021). response, the research community has released large-scale FPV datasets including
Ego4D
Grauman et al.] 2022). SPIC-KITCHENS (Damen et al. 2018). and HOI4D (Liu et al.
2022b). Unlike third-person view (TPV) or exocentric perspective data (Kim et al. 2025 Lei al,
2024 egocentric content directly captures the immersive visual information experienced by the
camera wearer, providing details about their interactions with the surrounding environment (uang et al. 2024 Shi et al. 2024). To better understand human behavior from the egocentric perspective, a fundamental challenge lies accurately parsing the hands and objects involved in interaction, which is the main objective of the Egocentric Hand-Object Segmentation (EgoHOS) task (Zhang)
et al. 2022 Su et al. 2025a). By focusing on pixel-level segmentation of hands and interacting objects, this fine-graing greigga analysis is able to interpret the complex dynamics of human-environment engagement, forming a foundational capability for next- cext-eeneration technologies such as assistive
Corrrsponndng author.

---

Published as a conference paper at ICLR 2026
agents Yang et al. Zhao et al. 2024).
2025
Zhou et al. 2025 embodied Al DDang et al. 2025). and AR/VR systems
75
or Fland Miture Rind Mlothlued Dijpct Right Oiticc Twodlad Diject
752
Para
MaskaFormer
Maskformer
65
SCTNat
Segrenter
mioti
Upertiat
05
Care-Epr
Seq
InterFormer (Ours)
45
Indomain Test
hopvvrr
60
120
180
240
OGD
Input
Interctioon
With CoCo Loss
Ground Truth
Model Siize (M)
Image
Illusion
Figure 1: Model size vs. mloU for InterFormer com- Figure 2: Illustration of interaction il- pared to other methods. Evaluations use EgoHOS in- lusion, in which segmentation results vi- domain (In (In-domain), EgoHOS out-of-domain (OOD Test olate real-world causal dependencies be- 1), and mini-HO14D (OOD Test 2) datasets.
tween hands and objects.
Existing approaches for parsing hands and interacting objects can be broadly categorized into convolution-based transformer-based. and multi-modal large language model (MLLM)-based meth- ods. Convolution- models (Zhao et al., 2025 Xu et al., 2024b) are proposed earlier stages, exhibiting limited performance handling the complexities of the egocentric vision due to their inherent constraints in capturing long-range dependencies. With the recent advancement of Multi- modal Large Language Models (MLLMss, ML.L.-based methods (Su et al. 2025b) have gained at- tention for their powerful representational capacity. However, MLLM- models typically incur substantial computational overhead and parameter costs. In contrast, transformertbased approaches
Zhang al. 2022 Su et al. 22255a Leonardi al. 2024 Cheng et al. 2022a Xie et al. 2021 offer more favorable trade-off between accuracy and model complexity, effectively capturing raage interactions while maintaining manageable parameter efficiency.
Despite the promising progress of transformer- bassedd approaches, several key limitations remain un- solved. First, primary issue lies in query initialization. which plays an essential role in transformer- based frameworks such as DETR (Carion et al.) 2020 Zhu et al.] (221)). Current methods typically initialize queries using either sampled image features (Zhou et al. 2022) or learnable parameters (Cheng et al. 2022a Shah et al. 2024). The former often introduces irrelevant background informa- tion, while the latter provides set of stable but static queries after sufficient training. Consequently, both approaches exhibit limited adaptability to dynamically changing active objects across diverse input scenes. Second, current methods predominantly rely on dense pixel-level semantic features from the input image, implicitly extracting target information through attention operations in the transformer decoder for mask prediction (Cheng et al., 2022a Su et al., 2025a). However, such generic semantic features fundamentally limited answering "whar it" rather than whether it is in interaction. This semantic bias inevitably introduces substantial interaction -ireeeevant noise, ultimately degrading segmentation accuracy. Third, existing methods (Zhang et al. 2022 Cheng cet al. 2022a Su et al. 2025a) often exhibit a logical defect termed as the interaction illusion. As illustrated in Figure 2] object is predicted as manipulated by both hands even when the right hand is not detected. Such outcomes are inconsistent with real-world physical plausibility, leading to a significant drop in performance.
To address the aforementioned limitations, we propose the Interaction-aware TransFormer (In- ferFormer), a novel end-to-end framework for parsing hands and interacting objects from an ego- centric perspective. In contrast to prior methods that rely primarily on semantic features (Su et al.) 2025a Cheng et al. 2022a Zhang et al. 2022). we first introduce the Interaction Prior Predic- tor (IPP), an auxiliary branch trained to estimate interaction boundaries. This branch extracts pre- liminary houndary-guidke features that coarsely localize hand-object contact regions and capture initial interaction characteristics. However, these rough boundary-guld guided features alone are insuffi- cient for accurately distinguishing hands and their interacting objects. Therefore, we further propose Dynamic Query Generator (DQG) and Dual-context Feature Synthesizer (DFS) to shift the
2

---

Published as a conference paper at ICLR 2026
model's focus to distinguishing interaction representation learning. Specifically, the DQG grounds query initialization within the dynamic spatial cues of ongoing interactions. This module selects semantic embeddings that demonstrate strong similarity with coundary-guided features and inte- grates them with learnable parameters, producing intrinsically interaction mieraction-aware aware queries that enable flexible adaptation to diverse hands and interactive objects across varying scenes. To address the noise caused by relying solely on pixel-level semantic features for query refinement, we propose the DFS. This module synthesizes coarse interaction boundary cues with semantic features, effec- tively suppressing interaction-irreevve information and refocusing the model on essential contact relationships. Furthermore, to mitigate the interaction illusion problem, we design a Conditional Co- occurrence (CoCo) loss that incorporates hand-object contact constraints to ensure physically plau- sible and accurate segmentation. We conduct extensive experiments to evaluate the effectiveness of method. The results show that our InterFormer consistently surpasses all competing approaches across all evaluation metrics, achieving relative improvements of 2.42%, 5.09%, and 11.4% on the EgoHOS in-domain test set (Zhang al. 2022). the EgoHOS out-of out-of-domain test set (Zhang al., 2022]. and the -fitt--diitiiuution (OOD) Mimi-HO14D dataset (Su al. (025a)). respectively. These findings demonstrate the superior performance and robust generalization capability of our approach, as illustrated in Fig. The main contributions of our InterFormer can be concluded as:
We establish a novel query initialization paradigm, DQG, which generates intrinsically interac- tion-aware queries by fusing coarse interaction-alignedd semantic embeddings with learnable pa- rameters, enabling dynamic adaptation to hands and diverse active objects across varying scenes.
The proposed DFS introduces an interaction-centric refinement mechanism that purifies semantic embeddings through guuidd feature fusion, effectively suppressing interaction-ine irrele- vant noise and refocusing the model on contact relationships.
We introduce novel CoCo loss that encodes intuitive hand-object contact constraints into the learning process. By penalizing physically implausible co-occurrences, the CoCo loss significantly mitigates the "interaction illusion" problem and improves segmentation consistency.
Extensive evaluations on EgoHOS and mini-HO14D benchmarks confirm that InterFormer achieves state-of oh-the-art (SOTA) performance and exhibits strong generalization across in-domain and -id-tri-iitiittiinn settings.
2 RELATED WORK
Egocentric images and videos Narasimhaswamy et al. (2024) provide a unique perspective on human-environman interactions, capturing how individuals manipulate objects with their hands in natural, unstructured, real-world settings (Plizzari et al., 2025 Dang et al., 2025). The study of egocentric vision is essential for developing advanced intelligent embodied agents and has attracted growing interest from academic and industrial research communities. In response, several large- scale egocentric datasets have been introduced support data-driven modeling of human behavior, such as Ego4D (Grauman et al. 2022), SPIC-KITCHENS (Damen et al.) 2018), and HOI4D (Liu) let al. 2022b). These datasets provide foundational resources for a wide range of tasks, including action recognition (Peirone et al., 2025), video captioning (Ohkawa et al. 2022). action anticipation (Lai et al. (0224b). affordance learning (Luo al. 2024).. and hand-object interaction interpreta- tion (Leonardi et al. (2024). Despite the availability of these new datasets, the volume of FPV data remains considerably smaller than that of TPV datasets, limiting the training of deep models. To mitigate this data scarcity, several studies have explored cross-view representation learning. aim- ing to transfer view-in invviiitt features from the TPV domain to the FPV domain (Jia et al. [004] Liu et al. 2024 Li et al., 2024). However, such approaches typically rely on precisely aligned multi-view recordings, which are challenging to collect at scale due to hardware and synchroniza- tion constraints. Concurrently, some methods have integrated complementary multi-modal signals to enrich egocentric representation (Wang et al. 2025 Ramazanova et al.] (025), c.g., gaze (Lai et al., 2024a), audio (Jia et al. 2024). and textual descriptions (Hong et [025 Wang al. 2025). These modalities provide auxiliary contextual cues that can compensate for visual ambiguities in FPV data.
Recent advances in transformer architectures have significantly promoted the research in egocentric hand-object interaction (EgoHOO) by enabling more effective modeling of long-range dependencies
3

---

Published as a conference paper at ICLR 2026
Backbone
pox
Interaction Prior Predictor (IPP)
+
M
2
pix
Pint
pet
Pinc
Input Image
Interaction GT So
Fint
InterFormer Decoder
DQG
int
o²
Pix
Fint
lomaabl Querz
C
Class Head
Mask Head
Dual-context Feature Selector (DFS)
Mr
Fint
pix
M
gf-1
Mask Prod.
inf
Helle Lage
Pcco
Tramsformer
Decoder Block
Inturaction Query 2
o
InterFormer Decoder Block
XX
Simularity Map S
Multiplication
Sine Positional Encoding
Addition
-+ Supervision
Data Flow
Figure 3: Architecture of our end-to-end InterFormer. bone network first extracts global and multi-scale pixel-level features. We add an additional IPP branch extract coarse boundary-guided representations that characterize the interaction. Subse- quently, the DQG produces robust and dynamic queries by integrating interaction-reles tual information with learnable parameters. Finally, these queries and extracted features fed into the InterFormer decoder, which employs the DFS to refine interaction-awart representations and generate the final segmentation masks. The overall end-to-end architecture is supervised by the classification loss Cdss, dice loss Ldics cross entropy loss Lees IPP loss Ls, and CoCo loss Lco-
Given an input egocentric image, a back-
contex-
and complex visual relationships (Lai al. 2024a Roy al.) 2024 Cao et al. 2024). Despite
these improvements, fundamental limitation remain: most existing methods lack explicit and structured mechanisms for capturing the interactive relationships between hands and objects Cheng al. (2022a); (Su et al. (2025a); [Zhang al. (2022). As result, they often produce inaccurate physically implausible predictions.
3 METHODOL.OGY
The EgoHOS task aims parse the left hand Mu. right hand My,, and objects in contact within an egocentric image I E Hxx x3, The categories of interacting objects include left-hand objects Mio (objects that interact only with the left hand), right-hand objects Mro (objects that interact only with the right hand), and two-hand objects Mro (objects that are contacted by both hands).
3.1 OVERVIEW
This paper presents InterFormer, novel approach for precisely parsing hands and interacting objects egocentric images. The overall architecture shown Figure Given an egocentric image, the backbone first extracts global and multi-scale pixel-level features. Unlike existing approaches that rely solely on semantic features, we introduce an additional interaction prior predictor, which is ex- plicitly supervised by the interaction boundary ground truth G, to guide the network to concentrate on hand-object contact regions and model sounday--guided cues. However, these rough bound- ary-guided features alone are insufficient for accurately distinguishing hands and their interacting objects. Therefore, we further design dynamic query generator and a dual-context feature selector to explicitly refine and enrich the interaction representation. Specifically, the DQG grounds query initialization within the dynamic spatial cues ongoing interactions, generating robust and dynamic queries for hands and manipulated objects. The generated queries along with the extracted features transmitted into the InterFormer decoder, which integrates the DFS to refine interaction-awer representations and produce the final segmentation masks. Details of each component are provided in the following subsections.

---

Published as a conference paper at ICLR 2026
Backbone. The backbone of InterFormer extracts global and pixel-level features from an input egocentric image I. Specifically, we use a Swin (Liu et al., 2021) Transformer encoder { 0c) to obtain global features F, xB,,, where be the parameter. Next, deformable DETR transformer (Zhuu et al. 2021 serves as the pixel decoder DD pias generating multi-scale pixel-level F pia E I H xw I {1,2, ... ..).. [1,2,...,f]] where H W and C₁., denote the height, width, and channel dimensions at each scale.
features ppiz
Interaction Prior Predictor. After extracting pixel-level features, most existing methods send them directly into the transformer decoder for prediction Cheng al. (2022a); [Su et al. (2025a); Zhang itt al. (022). However, since different actions involve different interacting objects, identifying active objects cannot rely solely on semantic information, but must depend on their relationship with the hand. Therefore, we introduce an interaction prior predictor branch to roughly localize the hand-ob- ject interaction. This branch takes the global feature Fg as input and predicts the interaction boundary (Zhang et al. 2022), i.e., the overlapping region between hands and interacting objects, using cascaded U-Net-style decoder Oine followed by stacked convolutional layers @head. The output is an interaction boundary map MA supervised using binary cross-entropy loss: La Choe Cane(Me, (Ma, Go), where Gú denotes the ground truth of the interaction boundary generated by the intersection of dilated hand and object masks. By predicting interaction boundaries, the model achieves coarse spatial lo- calization of hand-object interaction regions, thereby generating preliminary boundary-guided fea- tures Firt = F ine E # I (1,2....,f) ()) which provide essential spatial constraints for subsequent interaction modeling. However, these features are insufficient for accurately distin- guishing hands and their interacting objects. Therefore, further design a dynamic query generator and dual-context feature selector to explicitly refine and enrich the interaction representation.
3.2 DYNAMIC QUERY GENERATOR
Query initialization is crucial in transformer-based methods, as it dominates the model's attention to the most relevant information and significantly influences the learning procedure. Some approaches
CCeng et al.) 2022a; Shah et al.) 2024 Li et al. 2023 Jain et al.) 2023 Zhang et al. 2023 2021)
employ learnable parameters queries, offering robustness and stability but often leading to slower convergence due to delayed feature alignment. In contrast, others use sampled features (Zhou et al.)
2022 Cheng et al., 2022b| c Fu al. 2024) to initialize queries, which offers adaptability input content but potentially introduces noise from irrelevant or ambiguous regions. More importantly, neither method explicitly encodes hand-object interactive relationships in queries, resulting in a lack of adaptation to hands and diverse interacting objects in varying input images.
To address these limitations, we propose the Dynamic Query Generator (DQG). The key innova- tion of this module lies grounding query initialization in the dynamic spatial cues of interactions through a two-stage process. First, extracts interaction-relevant content by selecting semantic em- beddings that demonstrate strong correspondence with boundary-guided features, ensuring the se- lection captures genuine contact relationships rather than relying solely on semantic information as in traditional samplig methods. Second, it synthesizes these selected features with learn- able parameters generate the final interaction- queries. Specifically, the last-layer level-level feature pia
xc5 and boundary-guided feature F 12 HE Il HS xw.
xwf xc are first aligned
in the channel dimension via a multi-layer perceptron (MLP), resulting in F Fins E " we xC5 Consequently, uniformly partition FL. into and width dimensions and compute the cosine similarity between each sub-region and Fint. This procedure produces dense similarity map S E TR
non-overlap sub-regions along the height
HH xw defined as follows:
(Fint; FL pise (i,
S = Fin
1,J E (1,2.....
(1)
[FS (1,
where F (i. denotes the feature vector at the (i.j) - th sub-region. Next, we select the N highest similarity values in and extract the corresponding feature vectors from F to form query vector 2, E Nxce, which encodes salient interactive regions. This intermediate query is
5

---

Published as a conference paper at ICLR 2026
then combined with learnable parameter vector through element- addition to produce the final interaction query E R-xcc, which serves as the initial input to the transformer decoder.
The proposed DQG extracts interaction-relevant content by measuring the similarity between image features and boundary guuded representations. The extracted features, which correspond to hand- object interaction regions, are combined with learnable parameters to construct robust and adaptive queries. This approach enables the model to explicitly generate queries based on dynamic interac- tion contexts rather than static object categories. Consequently, for input images with varying active objects, queries can constructed according to the hand-object interaction regions. The inclusion of learnable parameters further enhances the flexibility of the query formulation process.
3.3 DUAL-CONTEXT FEATURE SELECTOR
After the query initialization process, most current methods rely on dense pixel-level semantic fea- tures to implicitly learn target information through attention operations in the transformer decoder for mask prediction. However, such generic semantic features are fundamentally limited to answer- ing "what is if" rather than whether it is in interaction. This semantic bias inevitably introduces substantial interaction irrelevant interactio-irrelevant noise, ultimately degrading segmentation accuracy.
To address this limitation, context feature selector within each InterFormer de- coder layer to explicitly enhance interaction un- derstanding. As depicted in Fig. 4 for the 1- th layer, the DFS inputs the pixel-level feature pir E #, and the corresponding prelim- inary boundary-guisa guiddd feature F'.r E H Both features are first projected to the same dimen- sion and reshaped to size h x w x dim. Next, they flattened along the spatial dimensions h and w,
introduce dual-
Flnt
pltz
ppro)
Dropout
t
yat
Interaction: Enhanced
drarm
T
Self Attention
o0o 000 1
000
yielding F'... and F' piiz- A learnable positional pa- rameter T E (g(hw) is then added F Fuiz to improve robustness. Subsequently, an interaction- guided cross-attention mechanism is deployed to fuse semantic and interactive information, where the query Q is derived from the boundary- -ggided inter- action feature, while the key and value 10 are com- puted from the pixel-level feature. Specifically:
Interaction Crirrction-Gidee Cross Alienttion
Oharre
Figure 4: Detailed architecture of Dual-
context Feature Selector (DFS).
piz (ttaal0mra (F (), dcoo (T F ))
drav @onor (efiae@pprra (ime))))
(3)
where fltats Sproj' Poos, and @norm denote the flatten, linear projection, convolution, and normal- ization operations, respectively. Then, the interaction-uuide cross-attention operation is performed using Q, IC and Do:
QKT V
F soflmar
Vdim
Subsequently, the fused feature Fl cos is passed through a dropout layer Ddrop and a normalization layer, followed by an interaction-enhaneed self-attention module (.). This operation is identical to standard self-attention operation, which refines the feature representation by modeling long-range dependencies within the interaction-ant aeeaee context, yielding a more discriminative output F F. The final integrated feature F! F', is then computed through residual connection and normalization:
F! (onomm (oorrmm@drrg (Ddrop (F' (Froa)))
(5)
pår Onorm (F + @norm (ddrop (F' )))
(6)
By leveraging both interaction-guidee cross-attention and interaction-enhanced self-attention, the DFS effectively fuses semantic content with structural interaction cues, thereby suppressing inter- action-irrelevant information and learning more representative and interaction-aware features. In

---

Published as a conference paper at ICLR 2026
each InterFormer decoder layer, the DFS feature F'n/ f is used as the key and value within the transformer decoder block, while the query Q'-1 from the previous layer serves as the input query. This hierarchical attention mechanism enables progressive refinement of target localization through iterative feature alignment and interaction modeling. After decoder layers, the final set of queries Q encodes rich target-specific representations, which are independently decoded into class predictions and mask reconstructions. Specifically, the predicted category distribution C and the class-agnoste mask MM are generated from Q The final segmentation output M is obtained by multiplication: M = C @ Mc, where @ denotes channel-wis wise multiplication between the class logits and the corresponding masks.
3.4 CONDITIONAL CO-OCCURRENCE LOSS
Interaction Illusion. In real-world scenarios, the presence of a hand is fundamental prerequisite for any hand object interaction. For example, an object manipulated by the left hand can only be involved in interaction if the left hand itself is present and detected. However, existing methods (Zhang et al., 2022 Cheng et al. 2022a [Su et al. 2025a) often suffer from phenomenon termed as interaction illusion, which predicted interactions violate causal dependencies between hands and objects. As shown in the first row of Fig. 2 when the right hand is missing in the prediction, current models may incorrectly classify the interacting object as being operated by both hands, despite the absence of one hand. Such errors contradict basic physical constraints and undermine the reliability of segmentation systems in embodied Al applications.
To address this issue, we propose the Conditional Co-occurrence (CoCo) Loss, novel supervision mechanism that explicitly enforces physically plausible hand-object segmentation by conditioning object predictions the presence of the corresponding hand. Unlike probability-basee penalties, CoCo loss operates directly the spatial extent of the predictions, i.e., the number of pixels in the predicted hand and object masks. We opt for this design based on the observation that the "interaction illusion" is fundamentally macro-level logical error, which is more directly and et- fectively measured by the physical presence or absence (i.e., the pixel count) of the mask, rather than the average classification confidence across pixels. Guided by this principle, our CoCo loss is as follows: if the predicted mask for given hand contains fewer pixels than a predefined threshold T (indicating the absence of that hand), the loss penalizes any prediction of objects associated with that hand. This discourages implausible co- co-occurrence patterns, such as recognizing an object as the lefi-hand object when the left hand is not detected, Conversely, when the hand is confidently present (i.e., pixel count exceeds r), the penalty is deactivated, allowing legitimate interactions to be learned without interference. This dynamic constraint guides the model toward causally consistent hand-object associations. The CoCo loss for the left and right hands is defined as:
Clest (1 [(N(x)r) (Nio I(NN -No) = (V-I(N))))))) Noo. Bright = -I(()>>)))) (Nro - TWNSs .Nra) = -I(Ns>>)) -No, Lo
(8)
where Ni. Ni.. Nys, and Wr. denote the number of predicted pixels for the left hand, left-hand interacting object, right hand, and right-hand interacting object, respectively. I[m) is the indicator function, which returns if the condition is satisfied and otherwise. Equations ( illustrate the core mechanism of the CoCo loss, i.e., the hand-first principle. To further extend this regulation to objects manipulated by both hands, we define the CoCo loss for two-hand interactions:
Ctwo = =(1 (1- II--LNN..TANN,>20
(Noo - II.. 2TNN...+A....277 Noo) = (1 (1-14p (AA....A....)))))) -ioo (9)
where No. denotes the number of predicted pixels for the object involved in two-hands manipula- tion, and the indicator function I[ [[N.. >TANN equals to only when both hands are present. Thus, Ciro penalizes predictions of two-hand objects unless both hands are simultancously detected, en- forcing a physically grounded co-activation prior. The proposed CoCo loss incorporates real-world physical constraints into the learning process, guiding the model toward logically consistent and physically plausible hand-object relationships.
Overall Training, The InterFormer framework is trained in fully end-to-end manner, which is jointly supervised by the interaction boundary loss L, and the proposed CoCo loss L,p. Following established ransformer-based segmentation approaches (Cheng al. 2022a Zhang et al. 2022
7

---

Published as a conference paper at ICLR 2026
Table I: Comparison with SOTA methods on the EgoHOS in-domain test set measured by IoU
Left Hand
Right Hand
Left-hand Object
Right-hand Object
Two-hand Object
Method
Type
Overall
Segformer (Xic al.
2021
62.49
64.77
4.03
3.01
5.13
27.89, +4.389)
SCTNet (Ku 2024b
81.94
82.12
17.77
16.60
21.74
44.03, 29.19)
Para (Zhang el al 2022
69.08
73.50
48.67
36.21
37.46
52.98
Segmenter (Strudel ml
2021
T
82.20
83,28
46.22
34.79
51.10
59. 59.52;
UperNet Xtao el al. 2018
89.88
91.39
36.22
40.55
45.54
60.71,+12.51)
Multi-UNet (aaa et al.
2025
86.35
87.64
44.80
45.29
46.72
+2.16
MaskFormer (Cheng ct al.
2022a
90.45
91.95
43.51
41.04
54.65
(4.32
OneForrer al. 2022
90.38
91.95
43.88
44.37
52.64
(4.64(--8.6)
Mask2Former (Cheng al.
2022a
90.74
92.25
44.22
46.05
64.88 64.88(-8.50)
Sey (Zhang el al. 2022
87.70
88.79
62.20
44.40
52.77
67. 67.17(+6.05)
ANNEXE (Su al.
2025b
91.50
92.73
58.94
57.32
56.41
71.38
Care-Ego (Su 2025a
92.34
93.64
60.07
56.69
54.73
71.49 (1.73)
InterFormer (Ours)
92.51
93.50
60.86
55.04
64.17
73.22
Table 2: Comparison results on the EgoHOS out-of-domain test set measured by IoU f.
Right Hand
Left-hand Object
Right-hand Object
Two-hand Object
Overall
Method
Type
Hand
Segformer(XL Xie et al
2021
T
71.97
71.44
7.60
5.00
4.91
32.18 42.18(+40.60))
SCTNet Xu al.
2024b
T
87.12
86.29
31.18
19.70
13.32
47.52
UperNet Xiao al.
93.17
93.96
42.53
28.88
24.35
56.58
Multi-UNet (Zhao ct al.
2025
C
92.76
83.08
44.31
39.07
37.15
(9.27(+13.68))
Maskformer Caaammeerhhheee Cheng ct
2022a
T
92.69
94.02
51.81
39.84
39.43
(3.56
Musk2former al.
2022a
91.46
93.04
53.41
44.90
35.61
63.68(+9.14)
Segmenter Strudel al
2021
T
89.40
90.58
52.73
43.88
42.33
63.78(+9.03)
Zegghang al. 2022
T
81.77
78.82
46.93
26.40
42.38
(5.26
ANNEXE al.
2025b
92.45
93.18
54.39
46.60
40.71
65.36
CaRe-Ego (Su al.
2025a
T
94.47
94.41
51.56
36.80
41.84
63.82
InerFormer (Ours)
94.38
94.87
66.79
55.79
52.25
72.82
incorporate standard task-specific losses; the classification loss Lets, the dice loss Caics and the mask cross-entropy loss Lee, to optimize class prediction and mask quality. The overall training objective is formulated as weighted combination of these components:
Lco Cleft + Cright + +tww
As Ls Noo Lco + \ote Cets Adic Ldie Ace
Lees (10)
where the Abs Acor Nctos Adio; and Aoe are non-negative hyperparameters that balance the contribu- tions of each loss term. These weights are kept fixed throughout training in our experiments.
4 EXPERIMENTS
4.1 DATASETS AND METRICS
To evaluate the effectiveness and generalization capability of our InterFormer, we conduct compre- hensive comparisons with various state-of-the-art (SOTA) methods on the EgoHOS (Zhang et al.) 2022) and mini-HO14D (Su et al. 2025a) datasets.
EgoHOS. This dataset is an egocentric dataset with pixel-Ievel annotations for hands and interacting objects, containing 8,993 training, 1,124 validation, 1,126 in-domain test, and 500 out-of-domain test image-mast pairs.
mini-HO14D. This dataset is derived from the HOI4D dataset (Liu et al. 2022a), which consists of 1,095 egocentric images with corresponding mask annotations for hands and active objects. We use this dataset to evaluate the generalization ability of the proposed method under OOD conditions.
Evaluation Metrics and Implementation Details. We assess segmentation performance using standard metrics: (mean) citersection-over-iieee (IoU) and pixel accuracy (Acc). Due to space limitations, we present the loU and mloU results in this paper. The Ace results are provided in Appendix A. The implementation details described in Sec.
8

---

Published as a conference paper at ICLR 2026
Table Comparison with SOTA methods on the mini-HOI4D dataset measured by IoU 1.
Left Hand
Right Hand
Right-hand Object
Two-hand Object
Overall
Method
Type
Object
Segformer Xie et al
2021
T
30.16
56.44
5.17
12.02
25.95
SCTNett Xu et al. 2024b]
T
35.83
66.29
17.72
20.98
( (-30.86)
Multi- UNet (Zhao et
2025
52.15
83.64
25.70
41.60
42.36; 23.71)
UperNeet Xiao al. 2018
54.82
84.43
20.34
29.34
47.23, 18.84)
MaskFormer(Chenie et al
2022a
T
58,50
83.66
35.28
56.91
58.59 (+7.48)
Segmenter Strudel et al
2021
74.70
85,58
22.38
58.67
60.33
Zeang 2022
8.74
34.60
23.88
53.96
30.30,-35. 771
Mask2Former et al.
2022a
T
70.13
88.57
32.37
55.72
61. 61.70
ANNEXE (Su et al.
202.5
L
68.06
85.13
40.93
57.36
62.87
CaRe- Lggee--gooee (Su et al.
2025a
70.39
89.76
27.56
60.08
61.95(+4.11)
interrormer (Ours)
66.44
87.07
46.30
64.48
66.07
Baseline
Ours (4)
GT
Baseline
Ours
GT
Raseline
Ours (c)
GT
(b)
Figure 5: Visualization results on (a) EgoHOS in-domain test set, (b) EgoHOS out-of-domain test
set, and (c) dustrf-dist mini-HOl4D dataset.
4.2 COMPARISONS WITH STATE-OF-THE:
METHODS
We conduct comprehensive in-domain and OOD comparison of the InerFormer with SOTA Ego- HOS models in Tables 33 including convolution-basee (C), transformer- based (T), and large lan- guage model-based (L). The best results are in bold, and the second best is underlined.
4.2.1 IN-DOMAIN COMPARISON RESULTS
D presents a comparative analysis of InterFormer against methods on the EgoHOS in-domain benchmark. InterFormer achieves superior performance across all categories, attaining an impres- sive mloU 73.22%. The most pronounced advantage observed object segmentation, par- ticularly for two-handed objects, where it achieves outstanding IoU of 64.17%, representing substantial improvement of 7.76% loU over the second-best method. This significant progress can be attributed to interaction-cenrrc design, which leverages DQG to adapt queries to diverse interacting objects, employs DFS to enhance the feature representation of interactions, and utilizes the CoCo loss enforce robust hand-object correlations.
Table
4.2.2 OUT-OF-DIITRRIUTION COMPARISON RESULTS.
Evaluation the EgoHOS out-of-domain test set. To evaluate the generalization ability of our model, we assessed InerFormer on the out-of-domain EgoHOS test set using the best saved checkpoint, as shown Table Iner- Former achieved the highest overall mloU of 72.82%, outperforming the second-be best method by 7.46%. Notably, our approach also attained the highest IoU scores for all three categories of interacting objects.
Table 4: Ablation study results on the EgoHOS in-domain test set.
Performance (%
IPP
DQG
DFS
CoCo
mloU 1 mAce
70.72
77.48
70.95
79.02
3
71.23
79.97
4
1
71.50
79.68
5
71.26
79.11
72.35
80.
Evaluation on the mini-HO14D dataset. We also conducted OOD testing on the challenging
Ours
73.22
80.68

---

Published as a conference paper at ICLR 2026
mini-HO14D dataset in Table Our proposed InerFormer method achieved the highest mloU of 66.07%, surpassing the second-bes best method by 3.20%. In conclusion, these results underscore the generalization ability of InerFormer in challenging out-of- settings, confirming that the pro- posed module can dynamically understand the interactive relationships hetween hands and objects.
4.3 ABLATION STUDY
Efficacy of InerFormer. We present an ablation study to assess the contributions of InerFormer and its core components. For fair comparison, all experiments used identical model configurations. Since DQG and DFS are built upon the IPP Branch, we also ablate this branch. Table summarizes the results, leading to three key observations: 1) The IPP branch significantly improves performance, as it can localize hand-object interaction regions. 2) Each additional component brings further incre- mental improvements. 3) Integrating all components, the complete InerFormer framework achieves the best performance, proving the effectiveness of the framework and its individual components.
Hyperparameter study. In CoCo loss, the thresh- old determines that class is considered present only if the predicted pixel number exceeds т. as- sess its impact, we varied from 50 to 300 in steps of 50 in Table 5 Performance peaks at т = 100, consistent with theoretical expectations: when is too small, the model becomes overly sensitive, yield-
Table 5: Hyperparameter experiments of on the EgoHOS in in-domain test set.
Hyper Parameter
Performance mAce
#
mloU
50
71.62
80.62
2
100
73.22
80.68
ing spurious hand detections (false positives). Con-
3
150
71.97
80.51
versely, when т is too large, the model may miss partially visible hands (false negatives). Thus, this
4
200
71.62
80.94
trade-off explains the observed optimum a 100.
5
250
71.10
78.78
Due page limits, analysis of λo Aco, Adtes Adice. and Ace is presented in Appendix A.2
6
300
72.38
80.12
4.4 VISUALIZATION RESULTS.
For visual comparison, we show results against the CaRe-Ego baseline (Su et al. 2025a). Fig. 5)- (c) demonstrates that our method achieves superior segmentation of interacting objects by explicitly modeling interaction-aware representations. More visualization results can be found in Appendix.
CONCLUSION
We propose a novel InerFormer approach for the EgoHOS task. Specifically, we introduce the DQG module to create robust queries that can adapt various interacting objects in different images. We further design the DFS module to encourage the network explicitly perceive interaction-awan features. Additionally, our CoCo loss guides the network to learn interaction relationships that are consistent with real-world logic. Experimental results on three public test sets demonstrate the remarkable effectiveness and generalization of InerFormer.
6 ACKNOWLEDGEMENT
The research work described in this paper was conducted in the JC STEM Lab of Machine Learning and Computer Vision funded by The Hong Kong Jockey Club Charities Trust. This research re- ceived partially support from the Global STEM Professorship Scheme from the Hong Kong Special Administrative Region.
ETHICS STATEMENT
Our research work fully adheres to the ICLR Code of Ethics and embodies its core principles through the following commitments: 1) We maintain the highest standards of scientific rigor by providing comprehensive experimental results, detailed methodology descriptions, and open-source code im- plementation to ensure full reproducibility and verification of our findings. 2) As fundamental algo- rithmic research, our work contributes to the advancement of egocentric vision without targeting any specific application domain that might raise ethical concerns. The proposed techniques designed to generally applicable across diverse contexts. 3) Our research utilizes only publicly available benchmark datasets with proper licensing, ensuring no privacy violations or discriminatory impacts.
10

---

Published as a conference paper at ICLR 2026
The algorithmic improvements focus technical efficiency without embedding biases against any demographic groups. 4) We faithfully acknowledge all referenced works and properly cite prior re- search contributions. Our comparisons with existing methods are conducted fairly under standard- ized evaluation protocols. 5) By open-soureing our code and models, we promote equitable access our research outcomes and encourage broader community participation in further development.
8 REPRODUCIBILITY STATEMENT
All experiments were carried out on four NVIDIA RTX 4090 GPUs, using a total batch size of 8. During data preparation, images were cropped to 448x448 pixels and normalized using a mean of [106.011. 95,400, 87.429] and standard deviation of [64.357, 60.889, 61.419]. The maximum number of training iterations was set to 180k. Following previous methods (Cheng al.) 2022a), The number of queries is set to the number of target categories, i.e., 5. The values of Ao. Aco. and Adt were set to 1, while Ace and Adic were set to The hyperparameter 100. The model was trained end-to-end using the AdamW optimizer with an initial weight decay of 0.01. More implementation details can found the Appendix. B. The learning rate followed a two-phase schedule: it was linearly warmed up from 0 to le-4 over the first 10,000 iterations, and then decayed polynomially to zero by the end of training.
CoCo loss is set to
To ensure the reproducibility of our results and promote further research in the community, will release all code, models, and implementation details upon paper acceptance. The repository will include comprehensive documentation, training scripts, and inference instructions facilitate easy adoption and validation of our method.
REFERENCES
Congqi Cao, Ze Sun, Qinyi Lv. Lingtong Min, and Yanning Zhang. Vs-transgru: novel framework enhanced by visual-semantic fusion for egocentric action an- ticipation. IEEE Transactions on Circuits and Systems for Video Technology. 2024.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European conference computer vision, volume 12346, pp. 213-229, 2020.
Vincent Cartillier, Zhile Ren, Neha Jain, Stefan Lee, Irfan Essa, and Dhruv Batra. Semantic mapnet: Building allocentric semantic maps and representations from egocentric views. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 964-972, 2021.
Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 2159- 2127,
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 1290-1299, 2022a.
Bowen Cheng, Omkar Parkhi, and Alexander Kirillov. Pointly-supervised instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
2617-2626, 2022b.
Tianheng Cheng, Xinggang Wang, Shaoyu Chen, Wenqiang Zhang, Qian Zhang, Chang Huang, Zhaoxiang Zhang, and Wenyu Liu. Sparse instance activation for real-time instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
4433 4442, 2022c.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The spic-kitchens dataset. In Proceedings of the European conference on computer vision, pp. 720-736, 2018.
11

---

Published as a conference paper at ICLR 2026
Ronghao Dang, Yuqian Yuan, Wenqi Zhang. Yifei Xin, Boqiang Zhang, Long Li, Liuyi Wang, Qinyang Zeng. Xin Li, and Lidong Bing. Ecbench: Can multi-modal foundation models under- stand the egocentric world? a holistic embodied cognition benchmark. In Proceedings of the IEFE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 24593- 24602, 2025.
Zhen Fan, Peng Dai, Zhuo Su, Xu Gao, Zheng Lv, Jiarui Zhang, Tianyuan Du, Guidong Wang, and Yang Zhang. Emhi: A multimodal egocentric human motion dataset with hmd and body-worn imus. In AAAL, volume 39, pp. 2879-2887, 2025.
Stephanie Fu, Mark Hamilton, Laura Brandt, Axel Feldman, Zhoutong Zhang, and William T Freeman. Featup: A model-agnostic framework for features at any resolution. arXiv preprint MrXiv:2403.1051 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 18995- 19012, 2022.
Fangzhou Hong. Vladimir Guzov, Hyo Jin Kim, Yuting Ye, Richard Newcombe, Ziwei Liu, and Lingni Ma. Egolm: Multi-modal language model of egocentric motions. In Proceedings of the IEEE/CVF International Conference Computer Vision and Pattern Recognition, pp. 5344- 5354, 2025.
Yifei Huang, Guo Chen, Jilan Xu, Mingfang Zhang, Lijin Yang, Baoqi Pei, Hongjie Zhang, Lu Dong, Yali Wang. Limin Wang, et al. Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 22072-22086, 2024.
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. One- former: One transformer rule universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2989-2998, 2023.
Wenqi Jia, Miao Liu, Hao Jiang, Ishwarya Ananthabhotla, James M Rehg, Vamsi Krishna Ithapu, and Ruohan Gao. The audio-visual conversational graph: From an egocentric-exoceetrii perspec- tive. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 26396 26405, 2024.
Chanyoung Kim, Dayun Ju, Woojung Han, Ming-Hsuan Yang, and Seong Jae Hwang. Distilling spectral graph for object-context aware veccbvccayl semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp- 15033-15042, 2025.
Bolin Lai, Miao Liu, Fiona Ryan, and James M Rehg. In the eye of transformer: Global-local correlation for egocentric gaze estimation and beyond. International Journal Computer Vision, 132(3):854-877) 2024a.
Bolin Lai, Fiona Ryan, Wenqi Jia, Miao Liu, and James M Rehg. Listen look into the fu- ture: Audio-visual egocentric gaze anticipation. In Proceedings the European conference on computer vision, pp. 192-210, 2024b.
Ting Lei, Shaofeng Yin, and Yang Liu. Exploring the potential of large foundation models for open- vocabulary hoi detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 16657-16667, 2024.
Rosario Leonardi, Antonino Furnari, Francesco Ragusa, and Giovanni Maria Farinella. Are syn- thetic data useful for egocentric hand-object interaction detection? In Proceedings of the European conference on computer vision, pp. 36-54,11 2024.
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards unified transformer- based framework for object detection and segmenta- tion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3041-3050 2023.
12

---

Published as a conference paper at ICLR 2026
Yuan-Ming Li, Wei-Jin Huang. An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, and Wei-Shi Zheng. Egoexo-fitme fitness: Towards egocentric and exocentric full-body action understanding. In Proceedings of the European conference on computer vision, pp. 363-382, 2024.
Jei Liu, Weijia Mao, Zhongcong Xu, Jussi Keppo, and Mike Zheng Shou. Exocentric-to- egocentric video generation. Advances in Neural Information Processing Systems, 37:136149- 136172, 2024.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interac- tion. In Proceedings of the IEEE/CVF International Conference Computer Vision and Pattern Recognition, pp. 21013 21012-21022, 2022a.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang I. Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interac- tion. In Proceedings of the IEEE/CVF International Conference Computer Vision and Pattern Recognition, pp. 21013-21022 2022b.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9992-10002, 2021.
Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Grounded affordance from exocentric view. International Journal of Computer Vision, 13266):1955-199, 2024.
Supreeth Narasimhaswamy, Huy Anh Nguyen, Lihan Huang, and Minh Hoai. Hoist-former: Hand- held objects identification segmentation and tracking in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2351 2351-2361, 2024.
Takehiko Ohkawa, Takuma Yagi, Taichi Nishimura, Ryosuke Furuta, Atsushi Hashimoto, Yoshitaka Ushiku, and Yoichi Sato. Exo2egodve: Dense video captioning of egocentrie procedural activities using web instructional videos. In IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 8324 8335, 2025.
Simone Alberto Peirone, Gabriele Goletto, Mirco Planamente, Andrea Bottino, Barbara Caputo, and Giuseppe Averta. Egocentric zone-aware action recognition across environments. Pattern Recognition Letters, 188:140-147) 2025.
Chiara Plizzari, Alessio Tonioni, Yonggin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal Ilms in egocentric videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 24129-24138 2025.
Merey Ramazanova, Alejandro Pardo, Humam Alwassel, and Bernard Ghanem. Exploring miss- ing modality in multimodal egocentric datasets. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 75 75, 2025.
Debaditya Roy, Ramanathan Rajendiran, and Basura Fernando. Interaction region visual trans- former for egocentric action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 6740-6750, 2024.
Nisarg A. Shah, Vibashan VS, and Vishal M. Patel. Lqmformer: Language-aware query mask transformer for referring image segmentation. In Proceedings of the IEEE/CVF International Conference Computer Vision and Pattern Recognition, pp. 12903-12913, 2024.
Zhaofeng Shi, Heqian Qiu, Lanxiao Wang. Fanman Meng. Qingbo Wu, and Hongliang Li. Cog- nition transferring and decoupling for text-supervised egocentric semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 7262-7272, 2021.
13

---

Published as a conference paper at ICLR 2026
Yuejiao Su, Yi Wang, and Lap-Pui Chau. Care-ego: Contact-aware relationship modeling for ego- centric interactive hand-object segmentation. Expert Systems with Applications. pp. 129148, 2025a.
Yuejiao Su, Yi Wang, Qiongyang Hu, Chuang Yang, and Lap-Pui Chau, Annexe: Unified analyzing. answering, and pixel grounding for egocentric interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 9027-9038, 2025b.
Jian Wang. Rishabh Dabral, Diogo Luvizon, Zhe Cao, Lingjie Liu, Thabo Beeler, and Christian Theobalt. Ego4o: Egocentric human motion capture and understanding from multi-modal in- put. In Proceedings the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 22668 22678,, 2025.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision, pp. 418 434, 2018.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 34:12077-12090, 2021.
Lingjing Xu, Yang Gao, Wenfeng Song, and Aimin Hao. Weakly supervised multimodal affor- dance grounding for egocentric images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6324-6332, 2024a.
Zhengze Xu, Dongyue Wu, Changqian Yu, Xiangxiang Chu, Nong Sang, and Changxin Gao. Sct- net: Single-branch enn with transformer semantic information for real-time segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2024b.
Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong. Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 28885-28900, 2025.
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A sim- ple framework for open- vocabulayy segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1020-1031 2023.
Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. In Proceedings of the European conference computer vision, pp. 127-145, 2022.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34: 14:00326-10338, 2021.
Amy Zhao, Chengeheng Tang. Lezi Wang. Yijing Li, Mihika Dave, Lingling Tao, Christopher D Twigg, and Robert Y Wang. Egobody3m: Egocentric body tracking on a headset using diverse dataset. In Proceedings of the European conference on computer vision, pp. 375-392, 2024.
Qiangwei Zhao. Jingjing Cao, Junjie Ge, Qi Zhu, Xiaoming Chen, and Wenxi Liu. Multi-unet: An effective multi-u convolutional networks for semantic segmentation. Knowledge- Systems, 309: 1 12854, 2025.
Sheng Zhou, Junbin Xiao, Qingyun Li, Yicong Li, Xun Yang, Dan Guo, Meng Wang, Tat-Seng Chua, and Angela Yao. Egotextvqa: Towards egocentric scene-text aware video question answer- ing. Proceedings of the IEFE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 3363-3373, 2025.
Tianfei Zhou, Wenguan Wang, Ender Konukoglu, and Lue Van Gool. Rethinking semantic segmen- tation: A prototype view. In Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition, pp. 2572-2555 2583, 2022.
14

---

Published as a conference paper at ICLR 2026
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In Proceedings of the International Conference on Learning Representations, 2021.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
In this supplementary material, we first introduce the comparison experimental results of our Inter- Former against other SOTA methods measured by Acc in Sec. A.I The implementation details introduced in Sec. Finally, the hyperparameter experiments of loss weights are described in Sec.
A.2
A.1
COMPARISONS WITH STATE-OF- THE METHODS
We conduct a comprehensive comparison of the proposed InterFormer with SOTA egocentric hand- object segmentation models, including convolution-basee (C), waasformer-basee (T), and large lan- guage model-based (L) methods. This evaluation performed on the EgoHOS in-domain test set. Furthermore, to assess the generalization capability of our approach, we evaluate all methods using their best saved training checkpoints on the EgoHOS out-of-domain test set and the mini-HO14D dataset. We exhibit the comparison results three test sets using IoU in the main paper, we add the comparison results on three test sets using Acc this supplementary material.
A.1.1 IN-DOMAIN COMPARISON RESULTS
Table 6: Comparison with SOTA methods on the EgoHOS in-domain test set measured by Ace f (%) mAce t (%). The best results are in bold and the second best is underlined. T: transformer-basee
methods, C: convolution-ban methods, L: MLLM-based methods.
Left
Right Hand
Left-hand Objects
Right-hand Objects
Two-hand Objects
Overall
Method
Type
Hand
mAce f
Segformer Xie al.
2021
T
75.47
78.13
4.57
3.17
5.57
33.38
SCTNet (Xu et al. 20240
T
90.25
89.92
24.49
20.79
29.08
50.91
Para iZhang al. 2022
T
75.57
75.93
39.50
39.33
42.58
54.58
Segmenter (Strudel el al. 2021
89.87
91.92
62.69
45.59
62.78
70.57
(xperNet(Xino et al_ 2018
89.86
91.32
37.24
42.26
49.27
61.99
Mulii-UNet (Zhao el al., 2022
89.01
91.37
60.71
43.76
48.35
66.64
MaskFormer (Cheng al. 20224
93.90
96.41
67.08
$2.91
64.86
75.43
OncFormen Zhang ct al. 2022
96.21
96.33
61.19
53.06
63.75
74.71
Mask2Former (Cheng ot al.
2022a
T
96.01
96.20
53.97
58.10
60.48
72.95
ANNEXE (Su al. 2025b
L
95.87
94.81
73.28
66.54
68.50
79.80
Sog [Zhang et al. 2022
95.77
91.29
66.67
59.85
62.21
75.16
Care-Ego (Su el al., 2025a
96.64
96.81
71.79
68.71
63.85
79,96
InterFormer (Ours)
96.58
96.55
74.06
65.93
70.26
80.68
Table 7: Comparison with SOTA methods on the EgoHOS out-of-domain test set using Acc 1 (%)
mAce t (%). The best results are in bold, and the second best underlined. T: transformer-baser
methods, C: convolution-base. methods, L: MLLM-based methods.
Left
Right Hand
Left-hand Object
Right-hand Object
Two-hand Object
Overall
Method
Type
Hand
Segformer[Xie et
2021
T
86.80
79.44
21.77
9.56
11.67
41.85
SCTNet
2024b
94.62
90.92
49.14
17.12
UperNet al.
2018
C
96.89
96.00
64.83
54.59
27.83
68.03
Multi-UNet (Zhao 2025
94.75
95.80
45.16
47.89
16.75
60.07
Maskformer Cheng et al.
2022a
T
95.58
96.10
70.53
60.49
46.52
73.84
Musk2former Cheng
2022a
T
97.05
96.38
64.39
64.18
39.78
72.36
Zhang et al.
2022
87.83
85.98
57.17
43.85
54.76
65.92
ANNEXR Su et al.
2025h
I.
97.03
96.80
76.57
60.35
52.35
76.62
CaRe-Ego (Su et al.
2025a
97.09
96.69
72.30
46.28
InterFormer (Ours)
T
97.21
97.10
79.22
68.19
58.74
80.09
Table 6 presents comparative analysis of InterFormer against leading convolution-based
transformer- based, and large language model-based methods on the EgoHOS in-domain benchmark. As shown, InterFormer achieves superior performance across all categories, attaining impressive
16

---

Published as a conference paper at ICLR 2026
mAce of 80.68%. While InterFormer excels in hand segmentation, its most pronounced advantage observed in object segmentation, particularly for left-hand objects and two-handed objects, where it achieves outstanding Acc of 74.06% and 70.26%, respectively. This significant progress be attributed to our interaction-centre design, which leverages DQG to adapt queries to diverse in- teracting objects, employs DFS enhance the feature representation of interactions, and utilizes the CoCo loss to enforce robust hand-object correlations.
A.1.2 OUT-OFF-DISTRIRUTION COMPARISON RESULTS
Comparison on EgoHOS out-of-domain test set measured by Acc. To thoroughly evaluate the generalization capability of our model, we conducted assessment of InterFormer on the out-of- domain EgoHOS test set, utilizing the best saved checkpoint for this purpose. The results this evaluation are presented in Table 7 Notably, InterFormer achieved an impressive overall accuracy score of 80.09%, which enables to outperform the second- best method by a margin of 3.47%. This achievement is particularly significant, the highest Acc scores across all categories hands and interacting objects. These findings strongly indicate the robust generalization ability of our InterFormer method across diverse scenarios.
approach not only excelled overall but also secured
Table 8: Comparison with SOTA methods the mini-HO14D dataset measured by Acc (%) mAcc t (%). The best results are in bold, and the second best is underlined. transformer based methods, C: convolution-basee methods, L: MILM MLsMe methods.
Left Hand
Right Hand
Right-hand Object
Two-hand Object
Overall Object
Method
Type
Segformer Xic et al.
2021
92.13
72.42
5.52
13.41
45.87
SCTNet Xu al. 2024b
95.25
71.27
22.68
29.98
54.90
Multi-UNa (Zhao et al.
2025
C
94.37
70.25
30.41
47.65
60.67
UperNet(Xiao et al. 2018
C
97.71
86.04
25.77
36.58
61.53
MaskFormer Cheng et al.
2022a]
96.63
87.83
44.81
73.47
75.69
Zeqng et al. 2022
40.90
38.05
28.99
61.67
42.40
Mask2Former Cheng
2022a
97.48
89.38
45.22
74.17
76.56
ANNEXE Su et al
2025b
I.
96.54
93.18
48.77
75.19
78.42
CaRe-Ego (Su ct al.
2025a
T
97.79
93.09
49.35
68.01
77.06
InterFormer (Ours)
96.55
91.71
59.75
74.70
80.68
Comparison mini-HO14D dataset measured by Acc. In addition the EgoHOS test set, also performed out-of-distribution testing on the challenging mini-HOI4D dataset, the results of which are summarized in Table 8 Our proposed InterFormer method demonstrated its efficacy by attaining the highest mean Accuracy (mAcc) of 80.68%, thereby surpassing the second-best method by 2.26%. These results further prove the generalization ability of InterFormer demanding out- of-domain settings. They confirm that the proposed module is adept at dynamically understanding and interpreting interactive relationships between hands and objects, showcasing potential for real-world applications in egocentric scenarios.
Table 9: Comparison of computational complexity and performance on EgoHOS test set
Method
Type
FLOPss
mloU (%)
SegFormer
Transformer-baae based
71.961G
27.89
Segmenter
Transformer-hase
70.074G
59.52
Mask2Former
Transformer-ber based
96.093G
64.88
Seq
Transformer-basee
392.483G
67.17
ANNEXE
MLLM-based
610.500G
71.38
Interormer (Ours)
Transformer based
122.996G
73.22
17

---

Published as a conference paper at ICLR 2026
Input Egg-Immage
ANNEXE
GT
Figure 6: The MLLM-basee methods (ANNEXE) show limited spatial precision in mask generation for parsing hands and interacting objects.
A.1.3 COMPLEXITY OF INTERFORMER
Figure compares the model sizes different methods. It shows that our InterFormer achieves sate-offtheeart performance with manageable increase in parameters, effectively trading a com- pact model structure for superior accuracy.
To evaluate computational complexity, present comparative analysis of our InterFormer against other methods in terms of FLOPs. The results (Tab. 99) are based testing conducted the Ego- HOS in-domain test set. These results demonstrate that method achieves favorable balance between FLOPs and mean Intersection over Union (mloU).
1.1.4
ERROR ANALYSIS: INTERFORMER VS. MLLMs.
To better understand the comparative strengths and limitations of our InterFormer versus Multi- modal Large Language Models (MLLMs), such as ANNEXE, we conducted a detailed failure mode analysis.
Spatial Precision in Mask Generation. Our analysis confirms that MLLMs consistently produce masks with coarse and inaccurate boundaries. shown Figure 6 MLLM-generated masks (AN- NEXE row) can classify the categories each predicted entity. However, the predicted masks for hands and objects often exhibit poor alignment with true edges, which is caused by the lack of inter- action-centric representation learning. In contrast, our proposed InterFormer is specifically designed for egocentric hands and active object parsing, enabling the generation of more precise masks with clear boundaries.
Prompt Sensitivity. All evaluated MLLMs require text prompts and show high sensitivity to prompt design. In our experiments, used detailed prompt specifying five distinct mask types (left/right hands, corresponding interacting objects, and two-hand objects). However, ANNEXE struggled to understand and generate the required masks under this instruction, revealing limitations in following detailed segmentation tasks.
A.2 HYPERPARAMETER STUDY
The InterFormer framework is supervised by the interaction boundary loss Ls, CoCo loss Cco. the classification loss Lats, dice loss Ldies and mask loss Lee for model training. The overall loss function is defined as:
Lco clest + Cright + Liwo
(11)
L >b Lb + Neo Lcot Adss Lats Adic Late + Dee Lees
(12)
18

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameter study of different loss weights on the EgoHOS in-domain set. The best results are shown in bold.
Performance (%)
A
Ava
Acis
Adic
Aoe
mIoU
mAce
1
1
71.74
78.40
71.79
78.21
3
1
5
1
71.88
78.59
4
5
5
1
71.08
79.71
1
71.30
78.79
5
5
5
72.08
79.71
Ours
5
73.22
80.68
where the λ6 Acc. Acts. Adie; and Ace are hyperparameters used to balance the contributions of each loss item. In this section, we conducted hyperparameter experiments to verify the impact of dif- ferent loss weights on the EgoHOS in-domain test set. To ensure experimental fairness, all other parameters except the loss weight were the same as those described in Section X The experimental results are shown in Table TO Our model achieved the highest mean intersection over union (mloU) of 73.22% and mean average accuracy (mAcc) of 80.68% with the hyperparameter configurations of λ 1, Amo 1, Ar' 1, Adic 5, and Aoe 5. This result demonstrates that appropri- ate loss weights can significantly improve model performance. Experimental results highlight the importance of balancing the weights of different loss functions in model.
A.3 EFFECTIVENESS OF CoCo Loss
Qualitative results. To specifically validate the con- tribution of the CoCo loss, Figure 7 demonstrates the qualitative improvements achieved by incorpo- rating this component. The visualization contrasts model predictions without ("w/o coco") and with ('w/ coco") the CoCo loss across two datasets. With- out the CoCo loss, the model erroneously predicts objects as interacting with the left hand even when no left hand present. In contrast, incorporating the CoCo loss substantially enhances the physical con- sistency of hand-object interactions in the predictions.
Table 11: Quantitive results of CoCo loss.
#
Method
Rate J
Seq Zhang et al. 2022)
9.80%
Care-Ego Su al. (2025a)
5.45%
3
Ours w/o CoCo
2.19%
4
Ours w/ CoCo
1.55%
Quantitative results. We conducted systematic evaluation quantify the effectiveness of the CoCo loss in mitigating the interaction illusion problem. As summarized in Table II measured the frequency of this phenomenon by calculating the percentage of predictions containing interaction illusions across all outputs generated by our model. The results demonstrate that model trained without the CoCo loss produces such artifacts in 2.19% of its predictions. whereas incorporating the CoCo loss ("w/ CoCo") reduces this rate to 1.55%, which represents significant reduction of 0.64%. This quantitative evidence strongly confirms the critical role of the CoCo loss in suppressing spurious non-interactive object segments.
A.4 VISUAALIZATION RESULTS
This section presents comparative visualizations between our method and baseline approaches in Figure 8 Additionally, Figure showcases representative failure cases that reveal the current lim- itations of our InterFormer framework. Based on the presented failure cases, we observe that our method tends to miss small objects overlook parts of objects with significant appearance varia- tions during parsing.
19

---

Published as a conference paper at ICLR 2026
Left Hland
Right Hand
Lefrband Object
Right-lon Object
Two-hand Object
(a) Wio CoCo
(b) GT
(c) W/ CoCo
(u) W/o CoCo
(b) GT
(e) W/ CoCo
Figure 7: Qualitative comparison with/without employing CoCo Loss on EgoHOS (left) and mini- HOI4D (right) datasets.
A.5 LIMITATIONS AND FUTURE WORK
A.5.1 LIMITATIONS.
Limitations of hand-object interaction understanding in egocentric images. One significant lim- itation is that in more realistic and complex scenarios, hand may not be visible in a specific frame, yet it can still actively engage in interactions between objects. This situation underscores the chal- lenge of relying solely on static images for understanding dynamic interactions.
Limitations about pixel count CoCo loss. The proposed CoCo loss relies on an absolute pixel threshold r to determine the presence of hands. Although simple and efficient, it introduces sensi- tivity to imaging conditions such as hand-camera distance and image resolution.
Occulion. While the proposed method demonstrates strong performance, it shares a common limi- tation with existing approaches in handling hands and objects that are heavily occluded. Our Inter- Former does not incorporate explicit mechanisms for occlusion reasoning or recovery. In scenarios where hand-object interactions are significantly obscured (e.g., by other objects or self-occlusion), the model may struggle accurately segment boundaries infer interaction contexts.
A.5.2 FUTURE WORK.
Hand-object interaction understanding multi-view and/or video. Our future work will focus on extending egocentric hand-object segmentation to encompass both video and multi-view scenar- ios. By integrating more information, we aim develop a more robust framework that can accurately capture interactions, even when the hands are occluded or not visible in specific frames. This ad- vancement will not only enhance the reliability of hand-object interaction detection but also expand the applicability of our research fields such as roboties, augmented reality, and human-computer interaction.
20

---

Published as a conference paper at ICLR 2026
Left Hond
Right Hand
Lefl-land Object
Right-band Object
Two-land Objoct
(n) Baseline
(b) GT
(c) Ours
(a) Baseline
(b) GT
(c) Ours
Figure 8: Qualitative comparison of our InterFormer against baseline on 0OD mini-HOl4D (left)
and EgoHOS out-of-domain (right) test sets.
Ours
GT
Ours
GT
Figure 9: Failure cases of our method.
Replace the pixel count in CoCo loss. In future work, we plan to explore adaptive thresholding mechanisms to replace the pixel count in CoCo loss to improve generalization. A direction is to nor- malize the pixel count of hands, making the threshold condition relative to the visual scene. Addi- tionally, we will investigate learning-baid based approaches where the presence threshold is dynamically determined by the network rather than pre-defined.
Deploying and evaluating on AR/MR devices. Evaluating the model's performance in dynamic real-world settings will provide valuable insights into its robustness and adaptability, further in- forming improvements. We anticipate that this research could signilicantly advance the state of the art in interactive applications, paving the way for more intuitive and engaging user experiences in AR and MR technologies.
21

---

Published as a conference paper at ICLR 2026
Addressing the occlusion problem. Future work will explore dedicated strategies to address this challenge, such as introducing occlusion-aware attention mechanisms, leveraging temporal consis- tency in video sequences, integrating generative models to reconstruct plausible structures in oc- cluded regions. We believe these directions will further enhance the robustness of egocentric hand- object interaction analysis in real-world settings.
A.6 THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large language models (LLMs) were used exclusively for language refinement and proofreading during the preparation of this manuscript. The model assisted in improving grammar, clarity, and overall readability of the text without altering the original technical content, data, or scientific con- clusions. All research ideas, analysis, and writing conducted by the authors; the model was not involved any aspect of content generation or decision-makinn
22


Reviews:
[{"review_id": "OCKyny1RVK", "reviewer": "Reviewer_96hC", "summary": "To solve inflexible query initialization, pixel-level feature noise, and \"interaction hallucinations\" of Transformer-based EgoHOS methods, this paper proposes InterFormer. It integrates three key components: Prototype Query Generator (PQG) for adaptive queries, Dual Context Feature Selector (DFS) for noise reduction, and Conditional Co-occurrence (CoCo) Loss for physical consistency. The model achieves SOTA on EgoHOS in-domain, out-of-domain, and mini-HOI4D, with core contributions in explicit interaction-aware modeling and improved accuracy/generalization.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Targeted Problem-Solving: It accurately identifies three critical EgoHOS pain points (e.g., interaction hallucinations harming agent safety) and designs solutions accordingly, ensuring tight problem-solution alignment.\n- Logical Component Design: PQG, DFS, and CoCo Loss form a cohesive pipeline (query generation -> feature processing -> result optimization), with progressive and rigorous logic.\n- Comprehensive Experiments: It outperforms baselines on out-of-domain/OOD datasets (+5.09%/+11.4%) and uses ablation experiments to verify each component’s effectiveness, ensuring credible conclusions.\n- Reproducibility & Openness: The paper promises reproducibility and open-source availability, which are crucial for facilitating further research.", "weaknesses": "Section 3.4 mentions that the presence of a hand is a fundamental prerequisite for any hand-object interaction; when the right hand is not detected in the prediction results, current models may incorrectly classify the interacting object as being 'operated by both hands' despite the absence of one hand (the right hand). In terms of results, the CoCo Loss designed based on this premise has indeed achieved good performance in tests. However, this assumption has a critical flaw: the fact that a hand is not detected in the frame does not mean the hand is absent from the current interaction. If hastily determine whether a hand is involved in the interaction based solely on whether it is detected in the frame, the inferences drawn will violate physical common sense in more realistic and complex scenarios (e.g., large changes of viewpoint , or when the perspective can only observe one of the hands).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Vs5arEDp9", "reviewer": "Reviewer_s6Yi", "summary": "Authors propose a new architecture which explicitly models interaction by using explicit hand-object boundaries, and the results features are used along with pixel level features to extract hand-object interactions. The proposed approach also claims to address the problem of interaction illusion where the model incorrectly predicts objects being interacted with under implausible scenarios (hand not visible generally). Experiments demostrate improved performance over state of the art approaches on in-domain and out of domain datasets.", "questions": "1. Please clearly focus on 2-3 contributions - multiple small contributions are harder to validate. For example why is the number of pixels being minimized instead of the cross entropy loss directly by penalizing the prediction probability corresponding to object undergoing interaction.\n2. Why is $\\tau$ in pixels instead of normalized coverage in the image? Moreover hands can be much closer to the camera where pixel based coverage might not be enough and could be normalized based on the hand size (GT hand mask can provide that).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea to explicitly model interaction for extract hand-object segmentation masks under interaction is highly intuitive.\n2. The experimental results support the claim of improving performance over SOTA on various datasets; qualitative evidence is also shown to strengthen that claim.\n3. Paper tries to address the problem of interaction illusion which forces the model to learn causality over correlation (weakly speaking).", "weaknesses": "1. The paper uses a lot of ambiguous and vague terms - \"enrichment\", \"structural priors\", \"task-relevant\", \"preliminary coarse interactive representations\". I urge the authors to not use them since it causes confusion and distracts from understanding the actual core contributions. I would recommend a proper rewriting of the paper.\n2. The idea to use interaction cues (from Zhang et al., 2022, EGOHOS) is useful, but it is important to highlight if other methods lack this supervision. If they do, then the comparison is not fair - it is still useful to compare but important to disambiguate the role of architectural changes vs supervision. Some ablations do cover the impact of just introducing interaction loss (IPP) - focusing on mAcc in Table 4, most of the improvement is actually from IPP. Can authors retrain any baseline by adding another head that predicts hand-object boundaries, then measure how much is the improvement.\n3. Can authors show qualitative comparison on the models training with and without CoCo loss, how much does that help with interaction illusion. If authors do claim improvement, it would be useful to do a small quantitive study comparing the prominence of this issue in other methods vs the proposed one.\n4. Equations 7, 8, 9 are unnecessarily complex, please just write $I_{N_{lh}<\\tau} \\cdot N_{lo}$, this is simpler to read which means penalize interaction when the hand is not visible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cko7tL4HL2", "reviewer": "Reviewer_tKL9", "summary": "This paper presents InterFormer, a claimed novel transformer-based framework for Egocentric Hand-Object Segmentation (EgoHOS) task which is now a popular egocentric tasks under CV area. The authors address key limitations in SOTA: inflexible query initialization, reliance on noisy pixel-level features, and physically inconsistent predictions; by incorporating 3 major components: (i) a Prototypical Query Generator (PQG) that creates adaptive queries by fusing learnable parameters with interaction-relevant context ; (ii) a Dual-context Feature Selector (DFS) that refines feature representations by combining semantic and interactive cues ; (iii) a Conditional Co-occurrence (CoCo) loss that enforces physical plausibility by penalizing object predictions when corr. hand is not detected. The proposed model is evaluated on the EgoHOS and mini-HOI4D datasets, where it achieves good performance. This is an overall good paper, only lacking in theoretical analysis and qualitative illustrations.", "questions": "1. The figures could be explained with a simple end to end example instead of just architecture.\n2. Equation on Coco loss is straight-forward. any weights on the combined loss?\n3. How actually is interaction-relevant context extracted and how to map perception to this, in order to represent context?\n4. Any specific reason for DFS, there can be multiple parallel feature selector methods.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper points to a key area, i.e. handling contact regions with a prior predictor for interactions.\n2. Instead of a one size fits all - the approach of handling specific SOTA technique drawback by the most optimal way is a fresh look -- PQG, DFS, and CoCo loss.\n3. CoCo loss is a simple solution to handle the interaction illusion problem.\n4. Codebase will be released if accepted and appendix section helps some clarifications.\n5. Comparison with extensive SOTA establishes good empirical results.", "weaknesses": "1. Instead of an end-end pipeline, the proposed modular approach may lead to complexity in time, space and a cut in error backprop.\n2. The dependency of the later modules on IPP, makes the contextual training necessary for IPP for generalization.\n3. Anonymous codebase was expected for validity along with more supplementray to show qualitative results and failure cases.\n4. Certain variables like CocoLoss params were found by ablation on specific datasets, thereby reducing generalization in lack of non-empirical justification. Hard thresholds can cause problems in other datasets.\n5. Real life deployment on a actual glass should have proved robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YvPA7KocJs", "reviewer": "Reviewer_yyjQ", "summary": "This paper presents InterFormer, a transformer-based framework for egocentric hand–object parsing that introduces interaction-aware representation learning. It integrates four key modules: the Interaction Prior Predictor (IPP) to capture contact boundaries, the Prototypical Query Generator (PQG) to create dynamic interaction-grounded queries, the Dual-context Feature Selector (DFS) for fusing local and global context, and the Conditional Co-occurrence (CoCo) loss to enforce physical consistency. The method achieves superior performance on EgoHOS and mini-HOI4D benchmarks", "questions": "Refer weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed IPP–PQG–DFS–CoCo framework is non-trivial and well-structured, explicitly modeling physical contact and interaction awareness within a transformer-based segmentation pipeline.\n\nBy introducing boundary-guided priors (IPP) and the Conditional Co-occurrence loss, the method embeds physical and causal constraints that significantly reduce implausible  errors.\nThe paper is well written and easy to follow.", "weaknesses": "Incomplete SOTA coverage. The authors do not discuss or compare to HOIST-Former (CVPR 2024), a recent and stronger transformer-based approach that explicitly models spatio-temporal hand–object relations. This omission weakens the manuscript’s literature coverage and contextualization.\n\nInsufficient treatment of occlusion. High occlusion is a core challenge in egocentric settings, yet the paper does not analyze how the proposed method handles heavily occluded hands or objects. In particular, there is no discussion or visualization showing how the transformer attention mechanism behaves on occluded regions or how IPP/PQG mitigates occlusion-induced errors.\n\nLimited out-of-distribution evaluation. The model is trained in a fully supervised manner, which can limit generalizability. This concern is reinforced by the absence of evaluations on large, diverse egocentric benchmarks such as EPIC-KITCHENS and Ego4D. The authors should either justify this omission or provide cross-dataset results.\n\nShallow comparison to large multimodal / foundation models. Although the paper includes MLLM baselines (e.g., ANNEXE, Care-Ego), the comparison and accompanying analysis are brief. The manuscript lacks an exploration of why InterFormer outperforms these models or where MLLMs might still be advantageous (for example, open-vocabulary categories or rare classes). A detailed error analysis contrasting InterFormer and MLLM failures would strengthen the evaluation.\n\nWeak theoretical motivation for design choices. Some architectural decisions are insufficiently justified. For example, the PQG selects the top-N similarity subregions using a fixed partition—why this fixed pooling strategy rather than a learned selection mechanism? Provide either theoretical intuition or ablation results showing sensitivity to NNN and partition size", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHeQjczGid", "reviewer": "Reviewer_NB12", "summary": "This paper presents InterFormer, a transformer-based model for egocentric hand–object segmentation. It introduces three key components, Prototypical Query Generator (PQG), Dual-context Feature Selector (DFS), and Conditional Co-occurrence (CoCo) loss to improve interaction-aware representation and reduce physically inconsistent predictions. The model achieves state-of-the-art results on EgoHOS and mini-HOI4D datasets.", "questions": "See weakness.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper provides clear figures to clarify the motivation and methodology. \n2. The method achieves competitive performance on the reported benchmarks.", "weaknesses": "1. Limited novelty and over-engineered design. The paper mainly achieves performance gains through stacking multiple existing components, relying on a cascaded U-Net-style decoder to generate priors and a sequence of post-processing steps. The overall method appears as an incremental engineering integration rather than a conceptually novel contribution.\n2. Inconsistent motivation and contribution. The authors claim that existing query initialization methods “lack adaptability to diverse categories of contacting objects,” and thus propose the Prototypical Query Generator (PQG). However, PQG only guides the network toward hand–object contact regions via similarity-based subregion selection and has little to do with modeling category diversity.\n3. Misleading use of the term “Prototypical”. The proposed Prototypical Query Generator (PQG) constructs queries by combining interaction priors, subregion Top-N selection, and a learnable bias，yet the connection to 'prototypical' concepts appears tenuous. \n4. Lack of model efficiency analysis. The method adds multiple stages and operations (e.g., Top-K feature selection), yet provides no analysis or comparison of model efficiency in terms of computational cost, or inference speed. \n5. Insufficient diversity of datasets. The experiments are mainly conducted on mini-HOI4D and EgoHOS, which are highly similar in structure and data distribution. Incorporating additional datasets such as Ego-IRGBench[1] and VISOR-NVOS[2] is recommended. \n[1] ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction. CVPR, 2025.\n[2] Learning to segment referred objects from narrated egocentric videos. CVPR, 2024.\n6. Poor writing quality. For instance, the notation system is inconsistent (L362), the references to Leonardi et al. (2024) and EgoHOS (L44) are unrelated, the citation for mini-HOI4D (L383) appears incorrect, and the claim \"most existing methods\" (L206) lacks supporting citations. \n7. Limited practical applicability and broader impact. While the study is technically well-executed, its potential influence on future research directions appears modest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)