EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
Authors: Yuhang Yang
Wei Zhai
Chengfeng Wang
Chengjun Yu
Yang Cao
Zheng-Jun Zha
Venue: NeurIPS 2024 poster (2024)
Keywords: egocentric human-object interaction
3D human contact
3D object affordance
embodied AI
Primary area: machine_vision
TL;DR: We propose estimating 3D human-object interaction regions from egocentric videos, including human contact and object affordance.

Abstract: Understanding egocentric human-object interaction (HOI) is a fundamental aspect of human-centric perception, facilitating applications like AR/VR and embodied AI. For the egocentric HOI, in addition to perceiving semantics e.g., ''what'' interaction is occurring, capturing ''where'' the interaction specifically manifests in 3D space is also crucial, which links the perception and operation. Existing methods primarily leverage observations of HOI to capture interaction regions from an exocentric view. However, incomplete observations of interacting parties in the egocentric view introduce ambiguity between visual observations and interaction contents, impairing their efficacy. From the egocentric view, humans integrate the visual cortex, cerebellum, and brain to internalize their intentions and interaction concepts of objects, allowing for the pre-formulation of interactions and making behaviors even when interaction regions are out of sight. In light of this, we propose harmonizing the visual appearance, head motion, and 3D object to excavate the object interaction concept and subject intention, jointly inferring 3D human contact and object affordance from egocentric videos. To achieve this, we present EgoChoir, which links object structures with interaction contexts inherent in appearance and head motion to reveal object affordance, further utilizing it to model human contact. Additionally, a gradient modulation is employed to adopt appropriate clues for capturing interaction regions across various egocentric scenarios. Moreover, 3D contact and affordance are annotated for egocentric videos collected from Ego-Exo4D and GIMO to support the task. Extensive experiments on them demonstrate the effectiveness and superiority of EgoChoir.

EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
Yuhang Yang', Wei Zhailt, Chengfeng Wang', Chengjun Yu', Yang Caol.2, Zheng-Jun Zha
University of Science and Technology of China Institute of Artificial Intelligence, Hefei Comprehensive National Science Center htps://yyvhang
htess/yyyhaagggiihhub.. github www.uuuboo//oooooooo Eooggohoiir
Visullte
Human Contact
Egocentric Video
Capturing 3D Interaction Regions
Motion
Object
Object Affordance
EgoChoir
Figure I: EgoChoir takes egocentric frames and head motion from sou-teddteed devices, along with the 3D object, to capture 3D interaction regions, including human contact and object affordance. The human motion just visualized for intuitive observation of contact, yet is not utilized by EgoChoir.
Abstract
Understanding egocentric human-object interaction (HOI) is fundamental aspect of human-centric perception, facilitating applications like AR/VR and embodied AI. For the egocentric HOI, in addition to perceiving semantics e.g., "what" interaction is occurring, capturing "where" the interaction specifically manifests in 3D space is also crucial, which links the perception and operation. Existing methods primarily leverage observations of HOI capture interaction regions from an exocentric view. However, incomplete observations of interacting parties in the egocentric view introduce ambiguity between visual observations and interaction contents, impairing their efficacy. From the egocentric view, humans integrate the visual cortex, cerebellum, and brain internalize their intentions and interaction concepts of objects, allowing for the premformulation of interactions and making behaviors even when interaction regions are out sight. In light of this, we propose harmo- nizing the visual appearance, head motion, and 3D object to excavate the object interaction concept and subject intention, jointly inferring 3D human contact and object affordance from egocentric videos. To achieve this, we present EgoChoir, which links object structures with interaction contexts inherent in appearance and head motion reveal object affordance, further utilizing it model human contact. Additionally, a gradient modulation is employed to adopt appropriate clues for capturing interaction regions across various egocentric scenarios. Moreover, 3D contact and affordance are annotated for egocentric videos collected from Ego- Exo4D and GIMO support the task. Extensive experiments them demonstrate the effectiveness and superiority of EgoChoir.
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Envision
Sit
Subject Intention
Interaction Regions
Object Concept
The subject intention, conveyed through synergistic visual appearances and head movements, along with the object interaction concept revealed by its structure and functionality, pre-formulate an interaction body image, which enables interaction regions to be envisioned.
Figure 2:
Introduction
Human-object interaction (HOI) understanding aims to excavate co-occurrence relations and inter-
action attributes between humans and objects [91, 103]. For egocentric interactions, in addition to capturing interaction semantics like what the subject is doing or what the interacting object is [5, 25], knowing where the interaction specifically manifests in space egg., human contact [9, 80] and object affordance [16, 24] is also crucial. The precise delineation of spatial regions constitutes a pivotal component in numerous applications, like interacting with the scene in embodied AI [17, 71],
interaction modeling in graphics [28, 95], robotics manipulation [52, 115], and AR/VR [11].
Most existing methods isolate the human and object to estimate contact or affordance regions [9, 29, 63, 80, 96, 101], capturing one aspect of interaction regions but neglecting the synergistic nature of interaction regions between interacting parties [59]. They delineate the region where objects should be operated without specifying the region of subjects intended for executing such operations, and vice versa. This oversight limits their efficacy in shaping final interactions. Some studies explore correlations between interacting parties to jointly estimate interaction regions for both the subject and object [34, 100, 102], in which observations of the interacting parties are quite crucial, whether appearances within exocentric visuals or compatible structures formed by geometries of the subject and object. However, the egocentric view possesses incomplete observations interacting parties, for instance, when sitting on a chair or interacting with hands accompanied by head rotation, the interacting parties are only partially visible or even completely invisible. This leads to ambiguity between visual observations and interaction contents, which undermines the effectiveness of these methods, resulting in gaps when directly applied to egocentric scenarios.
Studies in cognitive science illustrate that humans make egocentric behaviors through coordination of the visual cortex, cerebellum, and brain to correlate visual observations, self-movement, and concep- tual understanding. thereby revealing complementary interaction clues that link their embodiment and surroundings [1, 23, 66]. This motivates us to ponder: what clues could drive machines capture effective interaction contexts and infer interaction regions from the egocentric view? Analogous to humans, in this paper, we propose harmonizing the visual appearance, head motion, and 3D object to infer 3D human contact and object affordance from egocentric videos (Fig. 1). Normally, objects are designed to fulfill certain human needs, the linkage between their functionalities and structures reveals their interaction concepts, implying the intention and interaction regions. When engaging in interactions with specific objects, visual observation synergistically changes with head movement, conveying the interaction intention [51]. The subject intention and object concept formulate an interaction "body image" [72, 75], with it, the region humans intend to contact, and the region that objects afford for such interactions could be pre-envisioned during forming the interaction (Fig. 2). This guides the estimation interaction regions even when interacting parties move out of sight, eliminating the ambiguity between egocentric visual observations and interaction contents.
To consolidate the above insight, we present EgoChoir, a novel framework that integrates the visual appearance, head motion, and 3D object to excavate the object interaction concept and subject intention, collaboratively capturing 3D interaction regions. EgoChoir first links the semantic functionality and structures of the object by correlating interaction contexts within the appearance and motion with object geometry, thus mining the object interaction concept. Specifically, the appearance and motion features are mapped into interaction clues, and the object geometric feature, along with semantic token that represents the functionality, queries these clues calculate the affordance through parallel cross-attention. With affordance, the appearance feature is taken to

---

query complementary interaction clues from head motion and 3D affordance in parallel, excavating the subject intention and modeling the contact representation. Despite the framework being heuristic, egocentric interaction scenarios are quite distinct e.g., with hand or body, which leads to varying effects of multiple interaction clues on modeling interaction regions in different scenarios. To adapt to this variability, EgoChoir employs modulation tokens to adjust gradients of specific layers that map interaction clues the parallel cross-attention, endowing the model to adopt appropriate interaction clues for robustly estimating interaction regions across various egocentric scenarios.
Furthermore, we collect egocentric videos including 12 types of interactions with 18 different objects, and over 20K corresponding 3D object instances. 3D human contact and object affordance are also annotated for the collected data, which could serve as the first test bed for estimating 3D human-object interaction regions from egocentric videos. The key contributions are summarized as follows:
We propose harmonizing the visual appearance, head motion, and 3D object to infer human contact and object affordance regions in 3D space from egocentric videos. It furnishes essential spatial representations for egocentric human-object interactions.
We present EgoChoir, a framework that correlates complementary interaction clues to mine the object interaction concept and subject intention, thereby modeling the object affordance and human contact through parallel cross-attention with gradient modulation.
We construct the dataset that contains paired egocentric interaction videos and 3D objects, as well as annotations of 3D human contact and object affordance. It serves as the first test bed for the task, extensive experiments on demonstrate the effectiveness and superiority of EgoChoir.
2 Related Work
Embodied Perception. Embodied perception emphasizes actively understanding the surroundings
and facilitates intelligent agents in learning and improving human-like skills through interactions
[71]. This involves perceiving various attributes of the scene, egg., object functionality [18, 22, 58, 89, 98, 101, 109], scene semantics geometry [15, 44, 36, 60, 61, 86], and sound [6, 7, 8, 21, 76). Meanwhile, perceiving the embodied subject is also crucial, which involves anticipating the intention of the interacting subject [30, 83, 99] and the way interact with objects [38, 82, 97, 115] or scene [28, 39, 50, 110]. These methods achieve significant progress in perceiving a certain side of the embodiment or surroundings. However, when embodied agents interact with their surroundings, the interaction manifests in both the interacting subject and the facing object. Capturing synergistic interaction between the interacting parties is crucial. EgoChoir aims to explore the synergy perception of interaction regions from egocentric videos, including human contact and object affordance.
Egocentric Interaction Understanding, So far, methods have made significant progress in several proxy tasks for understanding egocentric interactions, such as action recognition [33, 65, 77, 88), anticipation [53, 70, 93], moment query [41, 73], semantic affordance detection [47, 104, 106], and temporal localization [105, 107]. They endow machines to understand the semantic ("what"") and temporal ("when"") aspects of the interaction. Despite their importance in egocentric interaction understanding, the lack of spatial perception ("where") makes it challenging to form interactions in the physical world. Some methods explore grounding spatial interaction regions at the instance-level [2, 40, [14] or part-level [48, 49, 57], but only in 2D space, resulting in gaps when extrapolating to the real 3D environment. contrast, EgoChoir captures the spatial aspect of egocentric interactions, and jointly estimates object affordance and human contact in 3D space.
Perceiving Interaction Regions in 3D Space. For 3D interaction regions, dense human contact [80] and 3D object affordance [16, 24] recently get much attention in the field. Methods estimate them typically follow two paradigms, one of which is to directly establish mapping between geometries and semantics [28, 55, 56, 89, 96, 1 111], e.g., "sit" links the seat of chairs, as well as the buttocks and thighs of humans. This paradigm establishes category-level connections between semantics and geometric regions, but it possesses limited generalization to unseen categories. Another paradigm explores correlations between geometries and interaction contents in 2D visuals e.g., exocentric images [29, 62, 74, 80, 101, 102], taking correlations to guide the estimation. This endows the model to actively anticipate based on interaction contents, which generalizes better unseen cases. However, incomplete observations in the egocentric view lead to ambiguous visual appearances for modeling the correlation, which affects their effectiveness. EgoChoir mitigates this influence by harmonizing multiple interaction clues that could provide effective interaction contexts.
3

---

EgoChoir
Object Interaction Concept e..
modulation tokens
aaf
Pipeline
hadlamard product
Modalitywiis wise
to
Fusion
Encoders
Decader
fs
M
Attention
The Mae---r a- dn
Cross Attention
Cross Attention
Fl F
F
Fd
Reu
Subject Interaction Intention θ
O+O
Motion Encoder
Pre-train
C Concat
Spin
Semantic Taken
FM
1/
F FO/v Fv/o
Figure 3: Method. EgoChoir first employs modality-wiss encoders to extract features, in which the motion encoder pre-trained by minimizing the distance between visual disparity and motion disparity. Then, it takes them to excavate the object interaction concept and subject intention, modeling the affordance and contact through parallel cross-attention with gradient modulation.
3 Method
The pipeline of EgoChoir is shown in Fig. 3, including extracting modality-wise features (Sec. 3.2), modeling the object affordance and human contact (Sec. 3.3), and the gradient modulation that enables to adopt appropriate clues estimate interaction regions across various scenarios (Sec. 3.4).
3.1 Preliminaries
Given the inputs M, O), where E RTXXHxWx indicates a video clip with T frames of size H x W, .M E RT'x22 denotes the translation vectors and rotation matrixes of head poses. O E R an object point cloud with N points. The goal is to learn a model f that outputs temporal dense human contact de E 7xxx8000x 3D object affordance on E N along with interaction category O., expressed as: dc. Oa. o. = f(V,M, O). 6890 the number of SMPL [46] vertices.
3.2 Modality-wise feature extraction
Employing video backbones that are pre-trained by specific tasks like action recognition or contrastive learning [41] on egocentric datasets [13, 26] is candidate approach to encode V. However, we find that they tend to homogenize features across sequence in our task (Sec. 4.3), this is detrimental to estimating temporally dynamic interaction regions. Thus, referring HPS from videos [32, 35], EgoChoir adopts the paradigm that correlates per-frame features. Specifically, per- per-frame features are extracted through pre-trained HRNet (f+) [85], then, the joint space-time attention (fat) is applied to establish temporal and spatial correlations among features, expressed as: Fy ((())) R H, Wi xe, where Hi, W, are height and width, C the feature dimension.
The relative change in head poses is crucial clue for providing interaction contexts [37]. Thus, the relative head pose difference between each frame and the first frame calculated, including translation difference f. and rotation difference R. It could be formulated as: T, t; to, R. Rg R.
where to. 4 E 1xx and Ro, R4 E 3333 indicate the head translations and rotations at the first frame and i-th frame, i E [1, ('. The calculated and I are concatenated into the relative head motion M. Despite calculating relative changes head pose, motion encoder capable of encoding the variation is still needed. EgoChoir achieves this by associating the feature discrepancy between encoded motion features with the discrepancy in visual appearance features [79]. In detail, appearance features FL, F are extracted from two random frames V by the frozen fi, where j, k means j-th, k-th frame and j < Then, the corresponding j-th, k-th head poses are selected from M and encoded by fM that is composed of MLP layers, obtaining motion features F Ek The fM trained by minimizing KL divergences calculated by F Fby and Fy, FY, the loss can be formulated as:
F F'M)) Σ FFfloge Fyyooge
Fú F2 112
Lm = Σ F Fyylogee
(1)
c+ FM
C+F,
H,W,
where is regularization constant. By constraining the distance hetween the visual discrepancies and motion discrepancies in feature space, the variation in appearance features moderately transmitter to

---

motion features, allowing fM to extract motion features F FM E RTxC with variations and associate with appearances. The object geometric feature Fo N is extracted through the DGCNN [90]. Each encoder is further fine-tuned during the optimization of affordance and contact estimation.
3.3 Modeling object affordance and human contact
Object interaction concept. With the modality-wiss features, EgoChoir correlates Fv, F M with Fo to link the object functionality and structure, revealing the object interaction concept and calculating the affordance feature through parallel cross-attention. In specifie, semantic token Ty R qxxx representing the functionality is concatenated with Fo as the query, while Fy, F. are used as two parallel key-value pairs. In the parallel cross- attennt,,n,,on, Fy, F, M scaled by learnable modulation tokens Te, Tm E R This modulates gradients of mapping layers and enables the model to extract effective interaction contexts from appropriate interaction clues across various scenarios, which is clarified in Sec. 3.4. The cross-attention is employed to model correlations among the query and key- value pairs parallelly, expressed as: F. = O₂ (T Fol: To Fv. Ti F M) E [[N+I)xC where Oe denotes the transformer with parallel cross-attentinn shown in Fig. 3, the fusion is composed of concatenation and MLP layers, I' indicates the concatenation, "." is the hadamard product. F, is split into the affordance feature F. E N and semantic feature of the functionality F.f BIXC
Subject interaction intention. As a manifestation of the object interaction concept, affordance implies the subject intention and assists in modeling intention semantics and human contact. With it, the Fy queries complementary interaction clues from the motion feature M and affordance feature F. to derive the subject intention, and calculate the human contact and intention semantic features. Analogous to the affordance extraction, can be expressed as: F. = (T,,,,, Fy + pez], To Fa. Tm (F M + pez)) E (T +IxCC where +₂ is similar with O2, T, E RIxC is a token that represents the intention semantics, pe, E RT'xC' is a temporal position encoding which introduces temporal dynamics into human contact and is expanded to H, W; WXC, F. is split into semantic feature of the intention F,; TR 1xC and contact feature F. TTH, Wi xC Furthermore, to maintain the synergy between contact and affordance, Fe is then mapped to key-value features, and Fa queries the synergistic interaction regions from them through
-otssnattnnionn fea.
Decoder. The semantics of functionality and intention are correlated, thus, the Fss, Fsi are concate- nated to the semantic feature F., then F. is decoded into the categorical logits d. E RR" through MLP layers, n is the number of interaction category. The affordance feature F, is decoded in the feature dimension and projected to object affordance da E Xx For the Fe,, in addition to decoding the feature dimension, the spatial dimension is mapped to the sequence of SMPL vertices. The human contact de RT x6800x is output through two shallow MLP layers that decode the feature and spatial dimension. The overall loss is formulated as: L = La L. + L,, where C. is a cross-entropy loss, it constrains synergistic interaction semantics of the human and object. La and Le optimize the affordance and contact respectively, both are a focal loss [42] plus a dice loss [54].
3.4 Gradient modulation
Egocentric interaction scenarios exhibit differences, e.g., with hands or body, which affect the effectiveness of distinct interaction clue features for extracting interaction contexts in the parallel cross-attention. Assuming sitting down or operating with hands in the egocentric view, the former hardly observes interaction regions, in which the variation of head motion is a more effective clue for extracting interaction contexts. In contrast, the latter has less head movement but can derive rich contexts from the object interaction concept and visual appearances. Vanilla cross-attention presents limitations for adapting to diverse egocentric interactions.
Our goal is enable the model adopt appropriate interaction clues for modeling interaction regions across various egocentric scenarios. Some methods [20, 67, 87] balance information from distinct modalities by calculating the discrepancy of consistent output (e.g., category logits), they compute a scaling factor x from logits output by different modalities and take the k to modulate gradients in each modal branch, thereby adjusting the weight to balance each modality, it can be simplified as:
OL
f1(1))
O.t1 0, 77
00,"
f2
where 0 is the optimize parameter, y is the learning rate, L is the loss function. 31; 22 represent inputs of distinct modalities, fi, fa indicates layers and calculations to get the logits, and denotes
5

---

Quantitative Results. Metrics of baselines and ours on human contact and object affordance. The best results are covered with the mask, indicates the relative improvement to the first row.
Table I:
Human Contact
Object Affordance
Method
Precision
Recall t
F1 f
geo. (em)
Method
AUC f
alOu t
SIM t
BSTRO [29]
0.42
0.45
0.42
43.21
O20 [56]
71.52
9.65
0.387
DECO [80]
0.54 628%
0.57 620%
0.53 0263
29.57
631%
IAG [101]
74.30 63%
11.21
616%
0.402 64%
LEMON [102]
0.65 654%
0.70
655%
0.67
009%
21.43
650%
75.97 06%
12.31
627%
0.410 08%
Ours
0.78 085%
0.79
675%
0.76 681%
12.62
618%
78.02 69%
14.94
063%
0.436
01398
an activation function. This manner seeks to equally weigh multi-modal inputs and avoid being dominated by a certain modality. While it differs from our expectations for the model, EgoChoir is expected adopt appropriate clue features in different egocentric interaction scenarios by modulating gradients of specific layers. Actually, in addition to adding a scaling factor the gradient can also be modulated by manipulating DC
in Eq. 2, which could be expanded into:
00
Oc OC don Ozn. Oc do, = Sn 20mm do,, Oz..
Ozn
80 Son, Ozn 30mn"
Om
Bn = Omn Om + b,
(3)
where Om, Zn are input and output connected by layer weight Omn b is the bias, On is the value of on through an activation function. To clarify the formula, the partial derivative of L on certain z is defined as ō, taking the 6n as an example, it represents the partial derivative of L with respect to 2n in the subsequent layer that contains certain nodes. With the 6,, the first part of Eq. can be rewritten 88mm
BC 6n Own seen from this formulation, scaling the feature on also modulates
the gradient, and this manner can primarily adjust the gradients of specific layers by selecting features to scale. Eventually, the parameters are updated as: 8+t1 < 8e 8To, where 7 represents the learnalle tokens to scale features in Sec. 3.3.
4 Experiment
4.1 Experimental setup
Dataset. 1) Source data: we collect video clips with egocentric interactions from Ego-Exo4D) [27] and GIMO [1 13], encompassing over 300K frames across 12 interactions with 18 object categories. The paired ego-exo videos in Ego-Exo4D enable the annotation of human contact from exocentric views. GIMO includes aligned 3D human bodies and scene, the human contact can be calculated based on distance [29]. Besides, over 20K 3D object instances spanning 18 categories appearing in egocentric videos, are collected from multiple 3D datasets [14, 43, 81, 92, 94]. 2) Annotation: we adopt remi-automated approach to annotate human contact, involving multiple rounds of manual annotation and fine-tuning off the-hhh model for inference. The calculated contacts in GIMO are also manually refined. Furthermore, we refer to the annotation pipeline [16, 102] to annotate the 3D object affordance. The statistical information about the dataset, including interaction categories distribution of video clips, the distribution of object affordance annotations, and the distribution of contact on different human body parts, are shown in Fig. 4.
WDed MiAnn
RChi
250
Wfort
Castat Uasan Part
SPAN
(h)
(ch
Figure 4: Dataset Distribution. (a) The distribution of different interaction categories and objects in video clips. (b) Category distribution of 3D object affordance annotation. (c) Distribution of contact annotations on human body parts.
Annotation. We annotate both 3D human contact and object affordance for the collected egocentric videos. Referring to DECO [80] and LEMON [102], the contact vertices are drawn on SMPL

---

A Affondance Anoiaaion
Chair Sit
Computad A Amootation
Aunotation
Kaife Cut
Spoon W Wroggrapp
Ego IxoAD
GIMO
Figure 5: Annotation of 3D human contact and object affordance. (a) Annotate contact for data in Ego-Exo4D. (b) Contact annotation for GIMO dataset, including calculations and manual refinement. (c) 3D object affordance annotation, with the red region denoting that with higher interaction probability, while the blue region indicates the adjacent propagable region.
[46] through Meshl.ab [64]. corresponding to the human contact in exocentric frames. For data in Ego-Exo4D, we select the best exocentric perspective and initially annotate the contact for 150 video clips, the process is shown in Fig. 5 (a). In detail, we select frames with stride of 16 to manually annotate the contact and the remaining frames are consistent with the adjacent annotated frames. Next, annotators check per-frame annotations and refine those with slight changes. Then, We fine-tune LEMON [102] through the annotated contact, the human body needed by LEMON is obtained by SMPLer-X X [4]. The remaining data is divided into groups for every 200 clips, and the fine-tuned model is used to predict human contact along with the manual refinement. Multiple rounds conducted to obtain the final annotations. Please note that the annotations for each round accumulated, and LEMON is fine-tuned each round, takes exocentric frames as the input.
For data in GIMO, first set distance threshold [29, 31] to calculate the contact between the human body and the 3D scene, the threshold is set to 2em. However, we find that there a deviation in the accuracy of human-scene alignment, which makes it hard to calculate all contacts using unified threshold, shown in Fig. 5 (b). Besides, the scanned geometry cannot reflect deformation, which also affects the contact annotation. Therefore, we locate key frames of the interaction and visualize human bodies in the 3D scene for these frames, then manually refine the calculated contacts.
For 3D object affordance, shown in Fig. 5 (c), we annotate high probability interaction region (red) and an adjacent propagable region (blue) on a object, and calculate the 3D affordance annotation S through a symmetric normalized laplacian matrix [16], formulated as:
S (I-aa(D a-D-=((DDDD WD 0.55-111 W 0 Aug Ao=
{ ||v. v,l2, NN (v.) O, otherwise
(4)
where Y E {0. is the one-hot label vector and indicates positive label, a is tlyper-paramete controlling decreasing speed, set 0.995. A represents the adjacency matrix sampled points in KNN graph, W is the symmetric matrix and D is the degree matrix. v is the zyz spatial coordinate of the point in the red region and N denotes the set of nearest neighbors in the blue region.
Metrics and baselines. Referring to advanced work estimating interaction regions [16, 29, 80, 101], the object affordance is evaluated through AUC, alOU, and SIM. Precision. Recall, F1, and geodesic errors (geo.) are used to evaluate human contact estimation. Since there is no existing method to estimate 3D human contact and object affordance from the egocentric view, the constructed dataset is utilized to retrain methods that estimate interaction regions based on observations for comparison, including DECO [80], LEMON [102], etc. Note: some methods require certain modifications their raw frameworks, details of metrics and each comparison method are provided in the appendix.
4.2 Experimental results
Quantitative results. Tab. shows that our method outperforms baselines across all metrics in both human contact and object affordance estimation from egocentric videos. The gap between visual
7

---

LEMON
Ours
GT
Kettle Grasp
Piano play
LEMON
Ours
GT
Afford. Figure Qualitative Results. Contact vertices are colored yellow, and 3D object affordance are colored red, with the depth of red representing the affordance probability. Note: for intuitive visualization, the contact GT of body interactions are visualized on posed humans (last row) from GIMO [113]. Please zoom in for better visualization and refer to the Sup. Mat. for video results.
Contact
Chair Sit
Afford.
Contact
Bed Lay
appearance and interaction content, caused by incomplete observations of interacting parties, hinders the performance of methods that rely on visual cues, e.g., BSTRO, DECO, O20, and IAG, leading to suboptimal results for both contact and affordance. LEMON gets moderate results owing modeling human-object geometric correlations. However, due to the parallel architecture visual appearances and geometries in its framework, the incomplete appearance in the egocentric view diminishes its performance. In contrast, EgoChoir correlates interaction regions by linking the object interaction concept and subject intention, thereby bridging the gap and achieving better results. Semantics are primarily used to constrain the synergy of interaction regions and are not included in the main evaluation. The comparison of category prediction accuracy reported in the appendix.
Qualitative results. Fig. 6 presents a qualitative comparison of contact and affordance estimated by method and LEMON. As can be seen, method yields more precise results and captures the temporal variation of contact, e.g., playing the piano with two hands or one. Besides, in cases where the interaction regions invisible (the below row), LEMON gets poor results due to the ambiguous guidance provided by visual observations. Our method adopts appropriate interaction clues to extract interaction contexts under different interaction scenarios and still infers plausible results.
4.3
Ablation study
We conduct a thorough ablation study to validate the effectiveness of the framework design and some implementation mechanisms, both quantitative and qualitative results provided.
Framework design. The metrics when detaching certain framework designs are recorded in Tab. The head motion M and affordance F. crucial interaction clues to excavate the subject intention

---

Table 2: Quantitative Ablations. Metrics when detaching the head motion M, affordance F. in O. gradient modulation т, the fca for semantics and region synergy, and pe,. As well as ablations of several implementations, including randomly initialize (ri.) fM without pre-train, video extractors e.g. SlowFast (S.F.) and Lavila (La.), divided space- timee-timeae attention (d. fat), X means without.
Metries
Ours
X M
X Fa
X T
X F
X fco
X pe
ri. fM
S.F.
La.
d. fst
Prec.
0.78
0.68
0.71
0.72
0.74
0.72
0.75
0.73
0.67
0.70
0.76
Recall
0.79
0.73
0.64
0.71
0.71
0.75
0.78
0.69
0.77
0.79
0.75
F1
0.76
0.69
0.66
0.71
0.73
0.72
0.74
0.70
0.72
0.74
0.75
geo.
12.62
19.86
19.13
17.68
15.53
15.73
13.43
14.57
21.37
19.22
13.04
AUC
78.02
74.36
75.21
75.34
76.12
76.61
77.75
76.05
76.35
76.62
77.88
aTOU
14.94
11.75
12.05
12.36
13.04
13.63
12.86
12.92
12.52
13.10
14.62
SIM
0.436
0.403
0.410
0.113
0.422
0.425
0.429
0.423
0.422
0.127
0.431
Mug
high
w/o M
Art
W
Viualive Head Molion
Afford.
low
Epoch
(a)
(c)
Figure 7: Qualitative Ablations. (a) Results of the human contact and object affordance ur w//o and head motion, along with the visualized head motion. (b) The lack of 3D affordance leads to over-prediction and temporal inconsistency of human contact. (c) Gradients of layers mapping different interaction clues under distinct input interactions during sampled 30 training epochs.
and object interaction concept, the performance significantly declines without them. The gradient modulation т enables the model to robustly adapt to various interaction scenarios, the absence of this mechanism also impacts performance. The semantic feature F, and the fee establish semantic and regional synergy between interacting parties, the metrics drop when detaching any of them. The temporal position encoding pe, introduces disparity in temporal dimension and eliminates some false positives, removing it decreases the precision and alOU.
Additionally, qualitative results provided for further analysis. Fig. (a) demonstrates the results w and w w/o head motion, as observed, the model can hardly anticipate interaction regions without the head motion, particularly for body interactions. The ablation of F, in modeling human contact is shown in Fig. 7 (b), which shows that the 3D affordance constrains the contact scope and maintains the temporal coherence of contact. Even if the object disappears in some frames, the model still plausibly infers based on the interaction concept provided by 3D affordance. The effectiveness of gradient modulation illustrated in Fig. (c). As can be seen, the gradients of layers mapping different interaction clues in the parallel cross-attention exhibit significant differences across inputs with distinct interactions, indirectly reflecting that the modulation endows the model to adopt appropriate clues for interaction context modeling and generalize to various interaction scenarios.
Implementation mechanisms. The metrics of some implementation mechanisms are also shown in Tab. 2. Randomly initializing the motion encoder makes it difficult to capture variations in motion features, adversely affecting the extraction of interaction contexts and resulting in performance decline. For the extraction of video features Fv, we also test video backbones such as SlowFas egocentric datasets [13, 26]. The precision and recall of contact estimation reveal that they tend to predict consistent results across all frames (see qualitative results in appendix), leading lower precision. Divided space-time attention [3] is also implemented to replace the joint one, while the joint space-time attention demonstrates superior performance.
[19], Lavila [112] pre-trained
4.4 Performance analysis
Here, we outline several heuristic attributes of the model that can robustly reason interaction regions from egocentric videos, and provide insights for further improving the model performance.

---

Insfances
Graep
Wrap.
Play
rolate
rotale
Cut
Visualized Head Motion
Chair
(b)
Figure 8: Analysis. (a) The changing interaction contents correspond dynamic 3D object af- fordances e.g., from grasp to cut. (b) Dynamic contact, e.g., left-right change can be implied by the head rotation. (c) Results of distinct object categories and different instances in interaction scenario. Note that slight differences exist in contact results with different object instances, but overall consistency, of the inferred contacts is visualized. Wrap. means wrapgrasp.
Dynamic region. Human contact and object affordance regions vary as the interaction evolves. We conduct experiment to validate whether the model captures this attribute. Fig. 8 (a) shows the results of changing object affordances and Fig. 8 (b) demonstrates the estimated changing human contact, unlike methods that distinguish left-right hands by masks or boxes, EgoChoir leverages the head motion e.g., rotations, for inference. This provides a way to get rid of intermediary models.
Multiplicity, The multiplicity is another crucial attribute of the interaction, encompassing interacting with multiple objects and instances. This requires the model to differentiate interactions with distinct objects and generalize across various instances. As shown in Fig. 8 (c), our method infers credible interaction regions with different objects and instances, which indicates that our model effectively captures interaction contexts with specific objects. It completes the estimation through the interaction contexts rather than merely mapping to specific categories or instances.
Whole-body motion. Recently, significant progress has been made in estimating human pose from the egocentric view [12, 37, 84], fa- cilitating the capture of egocentric whole-body motion. We test replacing motion features in the existing framework with global geometric features derived from sequence of human bodies (SMPI. vertices), and find that the performance continues to improve, shown in Tab. 3. This validates the effectiveness of harmonizing multiple clues for estimating interaction regions and suggests the potential for boosting performance in the future.
Table 3: Metrics when using whole-body motion.
Precision
gen.
atOu:
0.80
0.82
0.79
11.24
78.54
15.46
0.448
Discussion and conclusion
We propose harmonizing the visual appearance, head motion, and 3D object to infer 3D human contact and object affordance regions from egocentric videos. furnishes spatial representation of the interaction facilitate applications like embodied AI and interaction modeling. Through the constructed data and anmotations, we train EgoChoir, a novel framework that mines the object interaction concept and subject intention, to estimate object affordance and human contact by correlating multiple interaction clues. With the gradient modulation in parallel cross-attention, adopts appropriate clues extract interaction contexts and achieves robust estimation of interaction regions across diverse egocentric scenarios. Extensive experiments show that EgoChoir could infer dynamic and multiple egocentric interactions, as well its superiority over existing methods. EgoChoir offers fresh insights into the field and facilitates egocentric 3D HOI understanding.
Limitations and future work. Currently, EgoChoir may estimate the interaction region slightly before or after the exact contact frame, possibly due to a lack of spatial relation perception between interacting parties. Future work could consider incorporating 3D scene conditions and estimated whole-body motion [37, 108] to better constrain the spatial relation, achieving more tine-grained estimation of interaction regions and promoting egocentric human-scene interaction modeling.
Acknowledgments. This work is supported by the National Natural Science Foundation of China (NSFC) under Grants 62306295 and 62225207.
10

---

References
[1] Ralph Adolphs. Cognitive neuroscience of human social behaviour. Nature reviews neuro- science, 165-1165-, 2003.
[2] Peri Akiva, Jing Huang, Kevin Liang, Rama Kovvuri, Xingyu Chen, Matt Feiszli, Kristin Dana, and Tal Hassner. Self-supervised object detection from egocentric videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5225-5237, 2023. 3
[3] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[4] Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, et al. Smpler-x: Scaling up expressive human pose and shape estimation. Advances in Neural Information Processing Systems, 36, 2024. 7, 20
[5] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. 2018 ieee winter conference on applications of computer vision (wacr), pages 381 381-389. IEEF, 2018. 2
[6] Changan Chen, Kumar Ashutosh, Rohit Girdhar, David Harwath, and Kristen Grauman. Soundingactions: Learning how actions sound from narrated egocentric videos. arXiv preprint arXiv:2404. 05206, 2024.
[7] Changan Chen, Ruohan Gao, Paul Calamia, and Kristen Grauman. Visual acoustic matching. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18858-18868, 2022. 3
[8] Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for iisual-acoustic learning. Advances in Neural Information Processing Systems, 35:8896- 8911. 2022. 3
[9] Yixin Chen, Sai Kumar Dwivedi, Michael Black, and Dimitrios Tzionas. Detecting human- object contact in images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17100-17110, 2023. 2
[10] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022. 20
[11] Kun-Hung Cheng and Chin-Chung Tsai. Affordances of augmented reality in science learning: Suggestions for future research. Journal of science education and technology, 22:449 462, 2013. 2
[12] Hanz Cuevas-Velaspee Velasquez, Charlie Hewitt, Sadegh Aliakbarian, and Tadas Baltrušaitis. Sim- pleego: Predicting probabilistic body pose from egocentric cameras. arXiv preprint vrXiv:2401.14786 2024. 10
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(11): 4141, 2021. 4,
[14] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142-13153, 2023. 6, 19
[15] Alexandros Delitzas, Ayca Takmaz, Federico Tombari, Robert Sumner, Mare Pollefeys. and Francis Engelmann. Scenefun3d: Fine- fine-grained functionality and affordance understanding in 3d scenes. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2024.
[16] Shengheng Deng, Xun Xu, Chaozheng Wu, Ke Chen, and Kui Jia. 3d affordancenet: A benchmark for visual object affordance understanding. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1778-1787 2021. 2,3, 6,
[17] Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics Computatoonal Intelligence, 6(2):230-244 2022. 2
11

---

[18] Kuan Fang. Te-Lin Wu, Daniel Yang, Silvio Savarese, and Joseph J Lim. Demo2vec: Rea- soning object affordances from online videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2139-2147, 2018.
[19] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211 6211, 2019. 9,
[20] Jie Fu, Junyu Gao, Bing- Bao, and Changsheng Xu. Multimodal imbalance-aware gradient modulation for weakly-supervised audio-visual video parsing. IEEE Transactions on Circuits and Systems for Video Technology. 2023.
[21] Rishabh Garg, Ruohan Gao, and Kristen Grauman. Visually-guided audio spatialization in video with geometry- aware multi-task learning. International Journal of Computer Vision, 131(10):2723-22)) 2023. 3
[22] Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. Gapartnet: Cross-category domain uamain-generalizabll object perception and manipulation via generalizable and actionable parts, arXiv preprint arXiv:2211.0527,, 2022. 3
[23] Raymond W Gibbs Jr. Embodiment and cognitive science. Cambridge University Press, 2005.
2
[24] James Gibson. The ecological approach to visual perception: classic edition. Psychology
press, 2014. 2,3
[25] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-object interactions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8359-8367, 2018. 2
[26] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995 19012, 2022. 4,9
[27] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri- antafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311 182599, 2023. 6, 19
[28] Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, and Michael J Black. Populating 3d scenes by learning human-scent interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021. 2,3
[29] Chun-Hao P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13274-13288 2022. 2, 3, 6, 7, 20
[30] Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie, and Ser-Nam Lim. Intentonomy; a dataset and study towards human intent understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12986- 2021.
[31] Nan Jiang, Zhiyuan Zhang, Hongjic Li, Xiaoxuan Ma. Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scem interaction modeling. arXiv preprint arXiv:2403 arXiv:2403.08629, 2024.7 7
[32] Angjoo Kanazawa, Jason Y Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019. 4
[33] Evangelos Kazakos, Arsha Nagrani. Andrew Zisserman, and Dima Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the
IEEE/CVF international conference on computer vision, pages 5492-5501, 2019. 3
[34] Hyeonwoo Kim, Sookwan Han, Patrick Kwon, and Hanbyul Joo. Zero-shot learning for the primitives of 3d affordance in general objects. arXiv preprint 1711:2240..12978, 2024. 2
12

---

[35] Muhammed Kocabas, Nikos Athanasiou, and Michael Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognifion, pages
2020. 4
[36] Gen Li, Kaifeng Zhao, Siwei Zhang, Xiaozhong Lyu, Mihai Dusmanu, Yan Zhang, Marc Pollefeys, and Siyu Tang. EgoGie: An Egocentric Synthetic Data Generator. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3
[37] Jiaman Li, Karen Liu, and Jiajun Wu. Ego-body pose estimation via ego-head pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17142-17151, 2023. 4, 10
[38] Jiaman Li, Jiajun Wu, and C Karen Liu, Object motion guided human motion synthesis. ACM Trans. Graph., 42(6), 2023,
[39] Xueting Li, Sifei Liu, Kihwan Kim, Xiaolong Wang, Putting humans in scene: Learning affordance in 3d indoor environments. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12368-12376, 2019. 3
Yang, and Jan Kautz.
[40] Fanqing Lin, Brian Price, and Tony Martinez. Ego2hands: A dataset for egocentric two-hand segmentation and detection. arXiv preprint arXiv:201 1.00222,, 2020.
[41] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difci Gao, Coon-Chhng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems, 35:7575-7586, 2022. 3, 4
[42] Tsung-Yr Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[43] Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Qiaojun Yu, Yang Han, and Cewu Lu. Akb-48: A real-world articulated object knowledge base. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14809~14816 18, 2022. 6, 19
[44] Miao Liu, Lingni Ma, Kiran Somasundaram, Yin Li, Kristen Grauman, James M Rehg, and 3d map. In European Conference
Chao Li. Egocentric activity recognition and localization on Computer Vision, pages 621-638. Springer, 2022. 3
[45] Jorge M Lobo, Alberto Jiménez-Valverde, and Raimundo Real. Auc: a misleading measure of the performance of predictive distribution models. Globul ecology and Biogeography,
17(2):145-11, 2008. 20
[46] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons- Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), ((66):24::1224: October 2015. 4,
[47] Liangsheng Lu, Wei Zhai, Hongchen Luo. Yu Kang, and Yang Cao. Phrase-based affordance detection via cyclic bilateral interaction. IEEE Transactions on Artificial Intelligence, 2022.
[48] Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Learning affordance grounding from exocentric images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2261, 2022. 3
[49] Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao. Grounded affordance from exocentric view. International Journal of Computer Vision, pages 1-25, 2023. 3
[50] Zhengyi Luo, Shun Iwase, Ye Yuan, and Kris Kitani. Embodied scene-aware human pose estimation. Advances in Neural Information Processing Systems, 35:6815 586811-6828, 2022. 3
[51] Bertram F Malle and Joshua Knobe. The folk concept of intentionality, Journal of experimental
social psychology, 33(2):001-121 1997. 2
[52] Priyanka Mandikal and Kristen Grauman. Learning dexterous grasping with object-centric visual affordances. In 2021 IEEE international conference on robotics and cutomation (ICRA), pages 6169-6176. 76. IEEE, 2021. 2
[53] Esteve Valls Mascaró, Hyemin Ahn, and Dongheui Lee. Infention-conditioned long-term human egocentric action anticipation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6048-6057, 2023. 3
13

---

[54] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. 2016 fourth international conference 3D vision (3DV), pages 565-571, IEEE, 2016.
[55] Kaichun Mo, Leonidas J Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. WhereZact: From pixels to actions for articulated 3d objects. In Proceedings the International Conference on Computer Vision, pages 6813-6823, 2021.
the IEEF/CVF
[56] Kaichun Mo, Yuzhe Qin, Fanbo Xiang, Hao Su, and Leonidas Guibas. O2o-afford: Annotation- free large-scale object-object affordance learning. In Conference on robot learning, pages 1666-1677. PMLR, 2022. 3,
[57] Lorenzo
Jose J Guerrero, and Ruben Martinez-Cantnn. Multi-label affordance mapping from egocentric vision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5238-5249, 2023. 3
[58] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688 (888-8697,44 2019.
[59] Tushar Nagarajan and Kristen Grauman. Shaping embodied agent behavior with activity- context priors from egocentric video. Advances in Neural Information Processing Systems, 34:29794-29805, 2021.
[60] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 163-172, 2020. 3
[61] Tushar Nagarajan, Santhosh Kumar Ramakrishnan, Ruta Desai, James Hillis, and Kristen Grau- man. Egoenv: Human-centen environment representations from egocentric video. Advances in Neural Information Processing Systems, 36, 2024. 3
[62] Hyeongjin Nam, Daniel Sungho Jung, Gyeongsik Moon, and Kyoung Mu Lee. Joint recon- struction of 3d human and object via contact-based refinement transformer. arXiv preprint urXiv:2404.048) 2024.
[63] Supreeth Narasimhaswamy Trung Nguyen, and Minh Hoai Nguyen. Detecting hands and recognizing physical contact in the wild. Advances in neural information processing systems, 33:7841-7851, 2020.
[64] Cignoni Paolo, Muntoni Alessandro, Ranzuglia Guido, and Callieri Marco. Meshlab.
[65] Simone Alberto Peirone, Francesca Pistilli, Antonio Alliegro, and Giuseppe Averta. A backpack full of skills: Egocentric video understanding with diverse task perspectives, 2024.
[66] Jeff Pelz, Mary Hayhoe, and Russ Loeber. The coordination of eye, head, and hand movements in a natural task. Experimental brain research, 139:266-277, 2001.
[67] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 8238-8241 2022. 5
[68] Md Atiqur Rahman and Yang Wang. Optimizing iieesection-over-uniog in deep neural networks for image segmentation. In International symposium on visual computing, pages
234 244. Springer, 2016. 20
[69] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-wer open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 14159, 2024. 20
[70] Debaditya Roy, Ramanathan Rajendiran, and Basura Fernando. Interaction region visual transformer for egocentric action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6740-6750, 2024.
[71] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied research. In Proceedings of the IFEF/CVF international conference on computer vision, pages 9339-9347 9347, 2019. 2,3
[72] Paul Schilder. The image and appearance of the human body. Routledge, 2013. 2
14

---

[73] Jiayi Shao, Xiaohan Wang, Ruijie Quan, and Yi Yang. Action sensitivity learning for the ego4d episodic memory challenge 2023. arXiv preprint arXiv:2306.09117 2023. 3
[74] Soshi Shimada, Vladislav Golyanik, Zhi Li, Patrick Pérez, Weipeng Xu, and Christian Theobalt. Hulc: 3d human motion capture with pose manifold sampling and dense contact guidance. In European Conference on Computer Vision. pages 516-533. Springer, 2022. 3
[75] Peter David Slade. What is body image? Behaviour research and therapy, 1994, 2
[76] Arjun Somayazulu, Changan Chen, and Kristen Grauman. Self-supervised visual acoustic
matching. Advances in Neural Information Processing Systems, 36, 2024. 3
[77] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Lsta: Long short-term attention for egocentric action recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9954-9963, 2019. 3
[78] Michael Swain and Dana H Ballard. Color indexing. International journal of computer
vision, 7(1):1 1-32, 1991. 20
[79] Shuhan Tan, Tushar Nagarajan, and Kristen Grauman. Egodistill: Egocentric head motion distillation for efficient video understanding. Advances in Neural Information Processing Systems, 36:33485-33493 2023. 4
[80] Shashank Tripathi, Agniv Chatterjee, Jean-Claude Passy, Hongwei Yi, Dimitrios Tzionas, and Michael J Black. Deco: Dense estimation of 3d human-scene contact in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8001-8013, 2023. 2, 3, 6, 7, 18, 20
[81] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai- Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In Proceedings of the IEEE/CVF international conference computer vision, pages I 1588-1597, 2019. 6, 19
[82] Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-awant aware curriculum and iterative generalist-specialle learning. arXiv preprint arXiv:2304.0044 2023. 3
[83] Hongeheng Wang, Andy Guan Hong Chen, Xiaoqi Li, Mingdong Wu, and Hao Dong. Find what you want: Learning demand-conditionee object attribute space for demand-driven navi- gation. Advances in Neural Information P Processing Systems, 2023. 3
[84] Jian Wang, Zhe Cao, Diogo Luvizon. Lingjie Liu, Kripasindhu Sarkar, Danhang Tang, Thabo Beeler, and Christian Theobalt. Egocentric whole-body motion capture with fisheyevit and diffusion-basee motion refinement. arXiv preprint arXiv:2311. 16495,, 2023. 10
[85] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43(10))3349-3333 -364,, 2020. 4
[86] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. arXiv preprint arXiv:2312.16170 2023. 3
[87] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal elassification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695-12705, 2020. 5
[88] Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang. Interactive prototype learning for egocentric action recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3168-8177, 2021.
[89] Yian Wang, Ruihai Wu, Kaichun Mo, Jiaqi Ke, Qingnan Fan, Leonidas Guibas, and Hao Dong. Adaafford: Learning to adapt manipulation affordance for 3d articulated objects via few-shot interactions. In European conference on computer vision, pages 90-107. Springer, 2022. 3
[90] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38 2019.
15

---

[91] Ping Wei, Yibiao Zhao, Nanning Zheng, and Song-Chur Zhu. Modeling 4d humen-object interactions for event and object recognition. In Proceedings of the IEEE international conference computer vision, pages 3272-3279, 2013. 2
[92] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang. Jiaqi Wang, Chen Qian, al. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 803-814, 2023. 6, 19
[93] Yu Wu, Linchao Zhu, Xiaohan Wang. Yi Yang, and Fei Wu. Learning to anticipate egocentric actions by imagination. IEEE Transactions on Image Processing. 30: 142-114- 2020.
[94] Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015. 6,
19
[95] Xianghui Xie, Bharat Lal Bhatnagar, and Gerard Pons-Moll. Chore: object reconstruction from a single rgb image. In European Conference on Computer Vision, pages 125-145, Springer, 2022. 2
Contact, human and
[96] Chao Xu, Yixin Chen, He Wang, Song-Chun Zhu, Yixin Zhu, and Siyuan Huang. Partafford: Part-level affordance discovery from 3d objects. arXiv preprint arXiv:2202.13519 2022. 2,
[97] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang- Yan Gui. Interdiff: Generating 3d human-object interactions with physics-informed diffusion. In ICCV, 2023. 3
[98] Zihui Xue, Kumar Ashutosh, and Kristen Grauman. Learning object state changes in videos:
An open-world perspective. arXiv preprint arXiv:2312. 2023. 3
[99] Zihui Xue, Yale Song, Kristen Grauman, and Lorenzo Torresani. Egocentric video task translation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 131022220,22 2023.
[100] Lixin Yang. Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, and Cewu Lu. Cpft Learning a contact potential field to model the hand-object interaction. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 1199--1 106, 2021. 2
[101] Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, Jiebo Luo, and Zheng-Jun Zha Grounding
3d object affordance from 2d interactions in images. In Proceedings of the IEEE/CVF
International Conference on Computer Vision, pages 10905-10915, 2023. 2, 3, 6, 7, 7,20
[102] Yuhang Yang. Wei Zhai, Hongchen Luo, Yang Cao, and Zheng-Jun Zha. Lemon: Learning 3d human-object interaction relation from 2d images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16284-16294 2024. 2,3, 6,
7, 2,
[103] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of object and human pose in human- object interaction activities. In 2010 IEEE Computer Suciety Conference Computer Vision and Puttern Recognition, pages 17-24. IEEE, 2010.
[104] Zecheng Yu, Yifei Huang, Ryosuke Furuta, Takuma Yagi, Yusuke Goutsu, and Yoichi Sato. Fine- rene-grained affordance annotation for egocentric hand-object interaction videos. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2155-2161 2023. 3
[105] Yingsen Zeng, Yujie Zhong, Chengjian Feng, and Lin Ma. Unimd: Towards unifying moment retrieval and temporal action detection. arXiv preprint arXiv:2404.0493. 2024. 3
[106] Wei Zhai, Hongchen Luo, Jing Zhang, Yang Cao, and Dacheng Tao. One-shot object affordance detection in the wild. International Journal of Computer Vision, 130(10))2472-2500 2022.
[107] Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In European Conference on Computer Vision, pages 192-510. Springer, 2022. 3
[108] Siwei Zhang, Qianli Ma, Yan Zhang, Sadegh Aliakbarian, Darren Cosker, and Siyu Tang. Prohabilistic human mesh recovery in 3d scenes from egocentric views. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7989-8000, 2023. 10
[109] Zichen Zhang, Hongchen Luo, Wei Zhai, Yang Cao, and Yu Kang. Bidirectional progressive transformer for interaction intention anticipation. arXiv preprint arXiv:2405.0555 2024. 3
16

---

[110] Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14738-14749, 2023. 3
[111] Yan Zhao, Ruihai Wu, Zhehuan Chen, Yourong Zhang, Qingnan Fan, Kaichun Mo, and Hao Dong. Dualafford: Learning collaborative visual affordance for dual-gripper manipulation. arXiv preprint arXiv:2207. 2tt.::2220....911,, 2022.
[112] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6586-6597, 2023. 9,22,,2 23
[113] Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yehin Liu, C Karen Liu, and Leonidas J Guibas. Gimo: Gaze-informed human motion prediction in context. In European Conference on Computer Vision, pages 676-694. Springer, 2022. 6, 8, 19
[114] Chenchen Zhu, Fanyi Xiao, Andrés Alvarado, Yasmine Babaei, Jiabo Hu, Hichem El-Mohri. Sean Culatana, Roshan Sumbaly, and Zhicheng Yan. Egoobjects: A large-scale egocentric dataset for fine- mne-grained object understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 201 10-20120, 20, 2023. 3
[115] Xinghao Zhu, JingHan Ke, Zhixuan Xu, Zhixin Sun, Bizhe Bai, Jun Lv, Qingtao Liu, Yuwei Zeng, Qi Ye, Cewu Lu, et al. Diff-Ifd: Contact-aware model-based learning from visual demon- stration for robotic manipulation via differentiable physics-based simulation and rendering. In Conference on Robot Learning, pages 499-512. PMLR, 2023. 2, 3
17

---

Appendix
A Implementation Details
A.1 Method details
Visual encoder. The backbone to extract visual features is HRNet-W440 pre-trained on the ImageNet, which takes V as the input and outputs the feature with the shape of R2Tx208xTXT, then, 1 x convolution kernel is employed to obtain Fy, E x6777777777 In our implementation, T is set to 32, C' in the main paper is 768, and H1, W1 are both 7. Fyy is flattened to RT--197688 to enter a transformer with joint space-time attention fat. The depth of fet is set it contains 12 heads with each head dimension 64, the mapping dimension in the FeedForward is 768. Through the fat. we obtain the feature Fy E IR -497769
Motion encoder. The motion encoder fM is composed of MLP layers, projecting a sequence of head poses to the feature space, and obtaining the motion feature FM E R T768, For the training of the motion encoder fM. there is also an online training mechanism, which directly adds Lm to the whole loss function L for end-to-end training. However, this mechanism carries the risk of model collapse, when the visual backbone is being fine-tuned, the KI. divergence among visual features varies, this causes situation where f... optimizes the KI. divergence between visual features to then, the KL divergence between motion features also tends to O, affecting the entire model. We also conduct an experiment to validate this point. Please refer to Sec. C.3.
Point encoder. The DGCNN is employed to extract the geometric features of 3D objects. The number of nearest neighbor points in KNN is 20. We fuse local graph features with the global feature to obtain the geometric feature Fo E N x 768, in our implementation, N is set to 2000.
Transformer with parallel cross-attention. We take θ, as an example to introduce the implementa- tion details of the transformer with parallel cross-attention, and θ, similar to O Te: Tm, To E 767 are learnable tokens to scale the appearance, motion, and geometry features respectively. In Oa. the Fy, M multiply with Tu, Tm before mapping to kv vectors. Fo mapped to the query and Fv, M after scaling are mapped to two key-value pairs, the query vector performs cross-attention with these two key-value pairs separately. The cross-attention also contains 12 heads, each head with a dimension of 64, and the dropout ratio is 0.1. The fusion layer is composed of MLP layers to fuse query features, obtaining the affordance feature Fa E x The way to calculate F. E RT 768 is similar, but there are some details that need to be clarified. In Oc, the Fo. FM are multiplied with To, Tm and mapped to the key-value pairs, F y is mapped to the query. Besides, Fy and F are added with a temporal position encoding pe, R T766 pe, only applies to the temporal dimension, and the spatial dimension at each time step of Fy adds the same value.
The affordance feature F, and semantic feature F, concatenated by Fsf, F. are decoded directly through MLP layers, obtaining the object affordance da E R2000 I and semantic interaction category d, 12 For the contact feature Fe, the decoding of spatial and feature dimensions is decoupled. Specifically, the feature dimension is first mapped from 768 to 1, representing the probability of contact, and then the spatial dimension mapped to the vertex sequence of SMPL, ultimately obtaining the human contact &e E R. 6890x1 DECO [80] decodes the contact by directly mapping the feature dimension to 6890. We also conduct the ablation experiment and find that the decoupled decoding works better, please refer to Sec. C.3.
Decoder.
Loss. Here, we further clarify the loss function. For the output Φ.. Ocs Ps. each one has the corresponding ground truth Φe. de, d.. The Lg is a cross-entropy calculated through de and &., the C,, L, possess the same formulation, expressed as:
Caje =1 Σj yr +
ΣS (1- -x)) +
Σn ++r+c Σ 2- + N Σ1- (1 (1- (1-- x) log (1 - x) any (1 x)" log (x)),
(5)
where N indicates the number of points or vertices in d. or Φe. x is the prediction (Ø., dc), y is the ground truth (d., S.), is set to 1e-10, a, 7 are set to 0.25 and 2 respectively.
18

---

Table 4: The collected 12 different interactions with 18 different objects. Obj. indicates objects, Int. denotes interactions, wrap. is wrapgrasp and Refrige. is Refrigerator.
Obj.
Bottle wrap.
Kettle grasp
Bowl
Bed sit
Fork
Faucet open
Guitar
Chair sit
Dishwasher
wrap.
wrap. stab
play
open
Int.
open open
contain
lay
contain
contain
pour
pour
Obj.
Mug Knife grasp
Spoon wrap. mix contain
Spatula wrap. mix
Piano play
Violin play
Vase wrap.
Suitcase pull
Refrige.
wrap.
open
Int.
grasp
cut
pour
stab
contain
A.2 Training and inference
EgoChoir implemented by Py PyTorch and trained with the Adam optimizer. The training epoch is set to 100, the training batch size is set 8, and the initial learning rate is 1c-4 with cosine annealing. All training processes are on NVIDIA A40 GPUs (20 GPU hours). The HRNet backbone is initialized with the weights pre-trained ImageNet.
For each video, 32 frames are sampled for training. maximize data usage and maintain degree of randomness, the start frame is randomly selected from the first n-32 frames, where n is the total frame number the video, and the last frame is the n-th frame the video, 32 frames are uniformly sampled between the start and end frame. The corresponding head poses are also indexed for training. This ensures that the sampled frames basically cover the whole process of an interaction. Additionally, for each video sample, 3D instance corresponding the category of interacting object in the video randomly selected for training. This strategy helps the model generalize across instances while allowing for many-to ta-yyny combinations between videos and 3D objects, enhancing the diversity of training samples. Note that for videos involving multiple interacting objects, the ground truth of contact selected based on the input 3D object category, which enables the model estimate object-specific interaction regions. During the inference, the entire video is segmented into clips, each containing frames. The last clip will be padded with the final frame if has fewer than 32 frames. Each clip is paired with the same 3D object, allowing the inference of interaction regions for the whole video, while preserving the dynamic nature of both contact and affordance.
B Dataset Construction
B.1 Collection
We collect videos that contain egocentric hand and body interactions from Ego-Exo4D [27] and GIMO [113], encompassing 12 different interactions with 18 different objects, shown Tab. 4. The original video has long duration, which either contains content without interaction context for a long time. Therefore, we segment the collected videos to ensure that each clip has clear interaction contents, with a duration of 5-15 seconds. Both Ego-Exo4D and GIMO provide head trajectories. Trajectories from GIMO are aligned with the video frames, while trajectories Ego-Exo4D are sampled 1/ HZ. Therefore, we select the middle one every 33 head poses as the head pose which aligns with the video frames. Besides, we collect over 20k: 3D object instances from multiple 3D object datasets [14, 43, 81, 92, 94], corresponding to categories of interacting objects in collected egocentric videos. Ultimately, we construct dataset containing 1570 egocentric video clips, exceeding 300K frames, which can trained a to-nnyy combination manner with over 20K 3D instances. Among them, 1216 video clips are used for training, and 354 are used for testing. Note: to validate the model's generalization ability to unseen scenes, the egocentric scene in the training set and test set are almost non-overlapping.
much redundant interaction content
19

---

C
Experiments
Here, we provide further detailed explanation of the experimental setup, including the baselines and evaluation metrics. Besides, additional experimental results are also provided.
C.1 Baselines
BSTRO [29]: BSTRO concatenates downsampled vertices of the SMPL template onto the image features extracted by HRNet, then it employs a multi-layer transformer to estimate the contact vertex. We take egocentric frames to train BSTRO with its mask mechanism, the vertex of SMPI. is downsampled to 431.
DECO [80]: DECO utilizes two branches with cross-attention to parse the human part and scene semantics in images, thus facilitating the estimation of human contact. Following its instructions, we take the Fask2Former [10] to create scene segmentation maps for egocentric frames and use the scene and contact branches to train the DECO, the HRNet used as the image backbone, align with the encoder used in the EgoChoir.
O2O-Afford [80] O2O-Afford aims to ground the 3D affordance through the object-object interaction relation. It calculates the correlation between different 3D object features accompanied by spatial distance constraint. Although the input format differs from our setting, the insight of this method can still be used to form a comparison. The key modulation is we take pixel features of egocentric images as the kernel to slide over sampled seed point features of the object, obtaining content-aware seed point features. The pixel features, content-aware seed point features and the object global feature are aggregated and upsampled as the final representation of object affordance.
IAG-Net [101]: IAG-Net anticipates object affordance by establishing the correlations between interaction contents in the image and the geometric features of object point cloud. We use Grounded- SAM [69] to get the bounding boxes of interacting subject and object, and train IAG-Net by inputting egocentric frames.
LEMON [102]: LEMON correlates human and object geometries with images to jointly estimate 3D human contact, object affordance and their relative spatial relation, it needs posed human bodies as the input. We directly use the provided humans for cases in the GIMO dataset and take SMPLer-X [4] to estimate posed humans from exocentric frames in Ego-Exoo Exo4D. The posed human body, egocentric images, and 3D objects are used as inputs to train LEMON. The curvature of humans and objects needed by LEMON is calculated by Trimesh and the software Cloudcompare. Note that the layers used to predict relative human-object spatial relation in the original LEMON structure are removed.
C.2 Evaluation metrics
The metrics for evaluating human contact prediction include Precision, Recall, F1, and geodesic distance. Precision is the ratio of correctly predicted positive observations to the total predicted positives and measures the accuracy of the positive predictions made by model. Recall is the ratio of correctly predicted positive observations to all observations in the actual class and measures the ability of a model to capture all the positive instances. F1-score is the harmonic mean of Precision and Recall. provides a balance between Precision and Recall, making it suitable metric when there is an imbalance between classes. They could be formulated
TP Precision TP +++F FP* Recall
TP
2 Precision Recall Precision + Recall
TP - FN F1
(6)
where FP, and FN denote the true positive, false positive, and false negative counts, respec- tively. The geodesic distance is utilized to translate the count-based scores to errors in metric space. For each vertex predicted in contact, its shortest geodesic distance to a ground-truth vertex in contact is calculated. If it is true positive, this distance is zero. If not, this distance indicates the amount of prediction error along the body.
Object affordance are evaluated by AUC [45], alOU [68] and SIM [78]. The Area under the ROC curve, referred to as AUC, is the most widely used metric for evaluating saliency maps. The saliency map is treated as binary classifier of fixations at various threshold values (level sets), and an ROC curve swept out by measuring the true and false positive rates under each binary classifier, loU is the most commonly used metric for comparing the similarity between two arbitrary shapes. The IoU
20

---

Table 5: Metries of LEMON and our method for each interaction category. Prec. indicates Precision, wrap. wrapgrasp.
Metries
lay
sit
pour
pull
play
stab
contain
cut
mix
Pres.
0.68
0.80
0.34
0.45
0.72
0.79
0,80
0.64
0.73
0.72
0.77
0.78
Recall
0.72
0.77
0.42
0.53
0.69
0.72
0.28
0.58
0.64
0.75
0.80
0.68
LEMON
FI
0.78
0.70
0.74
0,41
0.61
0.66
0.72
0.71
geo.
29.52
18.15
41.62
37.31
25.73
13.47
17.23
14.95
19.82
23.75
11.90
21.39
AUC
58.01
72.12
55.16
58.92
61.79
39.77
$4.05
66.34
47.37
49.84
74.38
61.57
aIOU
2,30
6.15
2.23
3.19
5.91
3:25
1.22
6.36
1.93
3.17
4.04
6.40
SIM
0.17
0.18
0.09
0.15
0.42
0.15
0.04
0.27
0.20
0.25
0.31
0.29
Recall
0.80
0.83
0.74
0.80
0.74
0.73
0.33
0.71
0.88
0.75
0.89
0.73
Ours
F1
0.75
0.82
0.62
0.79
0.74
0.77
0.48
0.70
0.83
0.76
0.87
0.77
goo.
25.28
12.74
26.60
7.52
21.4
6.7
6.94
4.22
13.68
17.80
3.49
14.14
AUC
atOU
5.33
8.83
19.48
36.89
8.91
6.75
7.32
9.56
3.92
6.02
8.02
8.34
SIM
0.24
0.23
0.56
0.56
0.43
0.21
0.30
0.26
0.38
0.50
measure gives the similarity between the predicted region and the ground-truth region, and is defined as the size of the intersection divided by the union of the two regions. It can be formulated as:
TP
IOU + +
The similarity metric (SIM) measures the similarity between the prediction map and the ground truth map. Given a prediction map P and a continuous ground truth map Q (-)) is computed as the sum of the minimum values at each element, after normalizing the input maps:
(P. (P.QP)
Σ min(P.,QP) where PP==
(8)
C.3 Additional results
Here, provide more quantitative and qualitative experimental results to further demonstrate the effectiveness and superiority of the method.
Metrics for each category. The overall metrics are provided in the main paper, metrics for each category of LEMON and our method are reported in Tab. 5. can be seen that our method outperforms the comparison baseline across almost all categories. For body interactions like "sit" and "lay", the results are much better than the baseline. This is because these interaction scenarios have significant ambiguity between visual observation and the interaction content. Observation- based methods struggle to predict plausible results, while our method overcomes this ambiguity by extracting effective interaction context from appropriate interaction clues, leading better results.
Detach the foot contact. For human contact, in most scenarios, the feet are in contact with the ground, while certain hand contact regions are relatively small. This if it only
Table 6: Contact metrics when detaching the foot contact.
Precision
Recall
F1
geo.
results in situation where the model, predicts foot contact, could achieve favorable evaluation metrics. To further validate the model's performance of contact estimation, we retrain the model and comparison baselines without considering foot contact, reported in Tab 6. As can be seen, our method still exhibits the best performance.
BSTRO DECO
0.29
54.24
0.48
0.45
0.44
37.62
LEMON
0.50
0.50
31.27
Ours
0.67
0.63
0.65
22.67
Additional quantitative results. As illustrated in Sec. A.1, conduct experiment validate the performance when training the motion encoder fM online, directly adding C. to the entire loss C. the metries are reported in Tab.
For the head motion, we also attempt to calculate the relative head pose between adjacent two frames as M', in this manner, the frame sampling strategy during training requires slight modification. Because only by sampling consecutive frames does the head pose make sense. Specifically, the
21

---

Table Metrics when training fM online (adding Lm to L), using adjacent relative head pose M', and directly decode (D>) F. feature dimension. As well the error bar (e.b.) of EgoChoir.
Precision
Recall
F1
geo. (cm)
AUC
aIOU
SIM
+Lm M
0.73
0.75
0.72
16.76
76.32
12.05
0.423
0.74
0.71
0.71
15.59
76.23
12.72
0.425
> F.
0.75
0.76
0.74
15.10
e.b.
0.78 1 0.02
0.79 + 0.01
0.76 + 0.01
12.62 1 1.5
78.02 = 0.5
14.94 1 0.3
0.436 1 0.02
Hour
Vidk
15e:
Stouford
Visalavd Masin
Mision
Cluir
Bed
Lay
More results inferred by our method. For egocentric body interactions, the whole-body motion is visualized for intuitive observation.
Figure 9:
strategy for obtaining the start frame remains unchanged, and the subsequent T-1 frames are directly selected for training after obtaining the start frame. The results are reported in Tab. 7.
The result of directly decoding the feature dimension F, to 6890 also shown the table, the performance is inferior to decoupling spatial and feature dimensions.
The metries in the main paper and appendix report the best results, also provide the error bar of EgoChoir as a reference, shown in Tab. 7. For the prediction of interaction categories, we compare the top-1 accuracy with video backbones like SlowFast [19] and Lavila [112]. Acc_ for SlowFast 0.79, Lavila is 0.84, and ours is 0.80, our method still performs comparably well.
Additional qualitative results. More qualitative results are visualized in Fig. 9, involving various hand and body interactions with distinct objects from the egocentric videos. As mentioned in the main paper, using pre-trained video backbones tends homogenize the features across sequence, resulting in almost the same contact estimation for the whole sequence. We visualize the human contact when taking Lavila [112] as the backbone to extract Fy, as shown in Fig. 10, this case, contact is predicted even before it actually occurs.
22

---

Figure 10: The estimated human contact when taking Lavila [112] as the backbone to extract video features. In this case, the contact is predicted even before it actually occurs.
Affestanc
Atfordooce
Conten
Book Worggrop
Mal Cang.
Katla 19ar
15ay
Weopghop
Figure 11: More results that demonstrate interactions with multiple object categories, including the human contact and object affordance.
Furthermore, present more results showcasing interactions with multiple object categories, as depicted Fig. 11. The ability to infer the affordance and contact associated with specific objects when interacting with multiple objects is crucial, which facilitates interaction modeling with various objects, and assists embodied agents in operating specific objects.
D Society Impact
The method unleashes the ability to estimate 3D interaction regions from the egocentric view, benefiting downstream applications such as embodied Al and interaction modeling. However, currently cannot cover all interaction types, which may lead confusing predictions in certain interactions and introduce risks to the entire system.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Introduction (Sec. 1)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussion and Conclusion (Sec. 5)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The main results of this paper are experimental results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Method (Sec. 3) and Appendix (Sec. A.1)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Supplemental material
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https: httppps://niip../e cc/
publ / guides
des CodeSuumm i ssi assioPoll liyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. cc/ cttppcccppccccccccc pubppcpliooooo hwtp///giesseeeeeeee Code Submi asmiissinn ssissiionnn ceiionnoliinn ooelllyyyyy...... for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental setup (Sec. 4.1) and Appendix (Sec. A.2)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appendix (Sec. C.3)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix (Sec. A.2)
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /neurips..... cc/ppubllcc/tthhccccllelee.ee
Answer: [Yes]
Justification: conform
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix (Sec. D)
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Reference
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper swi hhoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
28

---

Answer: [Yes]
Justification:
Code and demo are in supplementary materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work is entirely completed by the authors and does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "jEiZRAAS6B", "reviewer": "Reviewer_PyYF", "summary": "This egocentric paper, aims to capture 3D interactions such as human contact and objects affordance, to achieve this it uses head motion, 3d object and the visual appearance", "questions": "The work provides good performance and is clearly written, what is the key contribution to the model fusion network?", "limitations": "nothing more than whats written above", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The add to Ego-Exo-4D the interaction data through a semi-automated process\nThe approach appears to beat other works on this newly labelled data.\nThere is extensive analysis of the results around the backbones", "weaknesses": "How accurate is the semi-automated annotation process\nIt's not clear where the right-hand side of Figure 3 is in the central system diagram\nThe fusion of the modality-wise encoders is basically a few cross-attention layers\nIt would be interesting to quantify the importance of each modality to the process", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1. How accurate is the semi-automated annotation process?**\n\nAs described in Appendix B.2, during the semi-automated annotation process, we conduct a manual check and refinement after each round of model prediction to ensure the accuracy of contact annotations. This is because the model estimates approximate regions in the initial round but performs limitedly in fine-grained regions. To further validate the accuracy of contact annotations (semi-automated part: Ego-Exo4D), we conduct a cross-dataset validation. Specifically, we train LEMON to estimate contact in two settings using __exocentric__ images:  \n1) train on 3DIR dataset [1] and test on annotated Ego-Exo4D part\n2) train on Ego-Exo4D part and test on 3DIR   \n\nWe use overlap object categories in the two datasets for testing, and the evaluation metrics of the two groups are shown in the table:\n|      | Precision | Recall |  F1  | geo. (cm) |\n|:----:|:---------:|:------:|:----:|:---------:|\n| 1    | 0.72      | 0.73   | 0.73 |   12.32   |\n| 2    | 0.70      | 0.74   | 0.71 |   13.14   |\n\nNote: the contact annotations in 3DIR are done entirely manually and are relatively more accurate. Meanwhile, the two groups are close in terms of metrics, confirming the accuracy of semi-automated annotations. This also suggests a potential way to scale up the dataset.  \n[1] LEMON: Learning 3D Human-Object Interaction Relation from 2D images\n\n---\n\n> **W2. It's not clear where the right-hand side of Figure 3 is in the central system diagram.**  \n\nSorry for the confusion. Actually, the right of Fig. 3 represents two modules that have the same model structure but different parameters and inputs. We denote them as $\\Theta_{a}$ and $\\Theta_{c}$. $\\Theta_{a}$ is in the \"Object Interaction Concept,\" while $\\Theta_{c}$ is in the \"Subject Interaction Intention\", modeling affordance and contact, respectively. We will add notations of these modules to the central part of Fig. 3.\n\n---\n\n> **W3/Q1. The key contribution to the model fusion network. Quantify the importance of each modality to the process.**\n\nThe core to fuse modality-wise features lies in what information flow is used to combine them and adapt appropriate clue features (e.g. appearance, head motion, or object concept), to model spatial regions for distinct egocentric interaction scenarios (e.g. body or hand). In our fusion model, the object feature $F_\\mathcal{O}$ is taken to capture interaction context from the appearance $F_\\mathcal{V}$ and head motion $F_\\mathcal{M}$, modeling the affordance region. The appearance feature $F_\\mathcal{V}$ then queries complementary interaction contexts from the mined object concept (affordance $F_{a}$) and $F_\\mathcal{M}$ to model contact region. Parallel cross-attention is a technical mechanism that completes the above process. In addition, unlike employing vanilla cross-attention directly, we introduce gradient modulation in the parallel cross-attention, enabling the model to adapt to various egocentric interaction scenarios.\n\nRegarding the importance of each modality, Tab. 2 in the main paper shows the quantitative impact of different modalities, we copy the modality-related results to the table below. It includes conditions without head motion ($\\bar{\\mathcal{M}}$) and affordance $F_{a}$ (source from 3D objects). The right part of Tab. 2 shows different encoding methods for the head motion, e.g. randomly initialize motion encoder (ri. $f_\\mathcal{M}$), and use divided space-time attention (d. $f_{st}$) for extracting video features, also reflecting the impact of these modalities.\n| Metrics   | all modality | w/o $\\bar{\\mathcal{M}}$ | w/o $F_{a}$ | ri. $f_\\mathcal{M}$ | d. $f_{st}$ |\n|:-----------:|:-----------:|:--------:|:-------:|:----------:|:-------:|\n| Precision   | 0.78        |   0.68   |   0.71  |   0.73     |   0.76  |\n| Recall      | 0.79        |   0.73   |   0.64  |   0.69     |   0.75  |\n| F1          | 0.76        |   0.69   |   0.66  |   0.70     |   0.75  |\n| geo. (cm)   | 12.62       |   19.86  |   19.13 |   14.57    |   13.04 |\n| AUC         | 78.02       |   74.36  |   75.21 |   76.05    |   77.88 |\n| aIOU        | 14.94       |   11.75  |   12.05 |   12.92    |   14.62 |\n| SIM         | 0.436       |   0.403  |   0.410 |   0.423    |   0.431 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "poOoHDJ7Qd", "reviewer": "Reviewer_paDU", "summary": "This paper investigates inferring 3D human contact and object affordance from a combination of egocentric video, human head motion, and 3D object point cloud. Inspired by real human behavior, which is based on visual observations, self-movement, and conceptual understanding, the authors propose a framework called EgoChoir. This framework utilizes modality-wise encoders to extract features from different input modalities and then infers object interaction concepts and subject intentions. The authors also construct a dataset comprising paired egocentric interaction videos (from EgoExo4D and GIMO) and 3D objects as the first test bed, demonstrating that EgoChoir outperforms existing baselines in inferring 3D Human-Object Interaction Regions.", "questions": "Please refer to the weaknesses part.", "limitations": "The authors discuss the limitations in Section 5.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, with a clear and concise background and motivation.\n- The key idea is ingenious and intuitive, drawing inspiration from real human behaviors and making it easy to understand.\n- The empirical evaluation is comprehensive, covering a range of existing baselines and two recent datasets EgoExo4D and GIMO, demonstrating the effectiveness of the proposed approach.", "weaknesses": "- The authors could leverage the existing annotations in EgoExo4D, such as 3D hand pose and scene annotation, to improve the quality of synchronization between the annotated affordance and video. Since most human-object interactions in EgoExo4D involve only two hands, utilizing these annotations could enhance the accuracy of human contact and object affordance annotation.\n- The authors mention that the egocentric scenes in the training and test sets are \"almost non-overlapping,\" but the meaning of this term is unclear. Could the authors clarify whether \"non-overlapping\" refers to: no exact same videos, no same scenes or no same activities (e.g., training on cooking videos and evaluating on music videos). And how did they determine this train-test split.\n- It would be beneficial to provide more elaboration on the application examples of the proposed method, e.g. illustrating its potential use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1. Leverage the existing annotations in EgoExo4D, such as 3D hand pose and scene annotation.**\n\nThanks for the advice. Combining 3D hand and body poses with the scene could indeed improve the annotation accuracy, which enables the contact to be calculated through the spatial distance at first, with only manual refinement needed. However, while building the dataset, we found that not all data in EgoExo-4D have these pose annotations, which hinders us from making a unified annotation workflow. As a result, we ultimately adopt a semi-automated annotation approach. In the future, as the dataset gradually updates the annotations, we will consider scaling up the dataset in this way.\n\nPlus, integrating 3D hand or body poses with the scene can potentially improve the prediction accuracy of the region in temporal. As we mentioned in the limitations (Sec. 5), due to the lack of spatial relation between the subject and object, the current model may estimate the interaction region slightly before or after the exact frame, while 3D poses and scenes can provide this spatial relation, offering a potential solution.\n\n---\n\n> **W2. Explanation of the scenes \"non-overlapping.\"**\n\nHere, the \"scenes\" refer to the background. Both the EgoExo-4D and GIMO datasets include the same interaction in different backgrounds (e.g. cut food in different kitchens). We realize that randomly splitting the training and test sets causes interaction clips with the same background to appear in both sets. In this case, the evaluation metrics cannot reflect whether the model is reasoning based on background information or interaction contents. Therefore, we manually split the training and test sets to ensure that clips with the same interaction have different backgrounds in two sets. This partition tests whether the model is inferring based on interactions and its generalization across different backgrounds.\n\n---\n\n> **W3. Elaboration on the application examples.**\n\nThanks for the constructive suggestion. We list some potential application scenarios:\n\n1) Interaction Modeling: some studies [1,2] provide datasets to facilitate interaction reconstruction and generation from egocentric videos. These tasks usually encounter problems such as misalignment, floating, and penetration of the interacting subject and object in spatial. Our method provides a spatial surface constraint, such as the subject contact and object affordance. These representations can be extracted from our method as a condition for those modeling methods to improve the authenticity and spatial rationality of the interaction.\n2) Embodied AI: learning skills from human demonstrations is an important part of embodied intelligence [3]. A feasible solution to achieve this is to first parse elements related to interactions from the embodied perspective (egocentric) demonstration, including the contact body part and object functional region. Then, retargeting the perceived elements to the agent's own configuration (e.g. dexterous hand) and facing objects. Taking the middle-level spatial perception to drive low-level control signals to complete the interaction.\n\nWe will add a section about applications in the future version.\n\n[1] OAKINK2 : A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion  \n[2] HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction  \n[3] DexMV: Imitation Learning for Dexterous Manipulation from Human Videos", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rGmfpu2BdQ", "reviewer": "Reviewer_fLMx", "summary": "This paper deals with the problem of inferring human-object interaction regions from egocentric views. It tackles the challenge of incomplete observations of interacting parties in the egocentric view by integrating the information from the visual appearance, head motion, and 3D object. It jointly infers 3D human contact and object affordance by exploring parallel cross-attention between the two parts. Moreover, 3D\ncontact and affordance are annotated for egocentric videos collected from existing datasets to evaluate the task.", "questions": "1) The latter part of equation (4) seems unnecessary and even confusion. Actually, it is easy to understand that scaling the features would affect the gradient on the model parameters.  \n2) The right part of Figure 3 is difficult to understand. More details are needed to explain the notations and procedure.  \n3) Writing needs further refinement. For example,  \n    a) line-158, Obejct -> Object  \n    b) line-186, ...that decoupled \"decode\" the feature...  \n    3) line-233, punctuations error", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper first tackles the task of estimating 3D contact and affordance from egocentric view by harmonizing multiple interaction clues including visual appearance, head motion, and 3D object.  \n2) A new framework is proposed to jointly infer human contact and object affordance regions in 3D space from egocentric videos. In the framework, object affordance and human contact are modeled through parallel cross-attention with gradient modulation to deal with the challenge of incomplete visual appearance in egocentric view.  \n3)  To evaluate the new task, a dataset is constructed that contains paired egocentric interaction videos and 3D objects, as well as annotations of 3D human contact and object affordance.  Extensive experiments are conducted on the constructed dataset to demonstrate the effectiveness of the proposed framework.", "weaknesses": "Several important information regarding the technical details and the dataset are missing.  For example,  \n1) Since a new dataset is constructed to evaluate the new task, the statistical information about the dataset (e.g., the proportations of interaction categories, ratio of contact and affordance annotations for different body parts and object categories) should be presented.  \n2) In Section 3.4, gradient modulation is achieved with learnable tokens to scale features to adapt to different scenarios. However, it is not clear whether these learnable tokens depend on input features. If yes, how? If not, then the tokens are fixed once the training is over and it is not possible to adapt to new scenarios during testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1. The statistical information about the dataset.**\n\nWe follow the advice and provide statistical information about the dataset **in the newly added PDF**, including interaction categories distribution of video clips, the distribution of object affordance annotations, and the distribution of contact on different human body parts. We will add this information in the future version.\n\n---\n\n> **W2. Modulation tokens that scale features to adapt to different scenarios.**\n\nThe model is expected to adaptively capture effective interaction contexts from different interaction clues when inputting distinct scenarios (body or hand interaction). Different input scenarios result in distinct distributions of clue features, such as the appearance $F_\\mathcal{V}$ and head motion $F_\\mathcal{M}$. To achieve adaptation, the model should differentiate which clue feature is effective for the input scenario based on their feature distributions. However, such specific feature distribution is hard to capture. Therefore, we employ learnable tokens to enhance or weaken certain feature dimensions (e.g. for hand interaction, some dimensions in $F_\\mathcal{V}$ represent hand features, while for body interaction, certain dimensions in $F_\\mathcal{M}$ express significant head translation and rotation), we provide a schematic figure about this **in the newly added PDF**. By doing so, clues exhibit significant differences in feature distribution across distinct interaction scenarios. It simultaneously adjusts gradients of mapping layers in the parallel cross-attention, making the layers handle the clue feature discrepancies of different input scenarios. During testing, the key to adapting to different scenarios is the learned mapping layers, the key-value input of the parallel cross-attention depends on the combination of input clue features and modulation tokens, these tokens densely scale clue features (at feature dimension) that are extracted from different scenarios, aligning the feature distribution that the mapping layers handled. Thus, these layers differentiate the utilization of multiple clue features (e.g. whether querying from appearance, object concept, or head motion) under distinct scenarios, achieving the adaption.\n\n---\n\n> **Q1-3. The latter part of Eq. 4 and the right part of Figure 3.**\n\n* The latter part of Eq.4 may indeed cause confusion, we will remove it and integrate the former part of Eq. 4 into Eq. 3 to make it more concise and clearer. Additionally, we will correct the typo you mentioned, thanks again for pointing it out.\n\n* The right of Fig. 3 actually represents two modules $\\Theta_{a}$ and $\\Theta_{c}$, which model affordance and contact respectively. They have the same model structure but different parameters and inputs. $\\Theta_{a}$ corresponds to object interaction concept and $\\Theta_{c}$ corresponds to subject interaction intention (middle of Fig. 3). As described in Sec. 3.3, $\\Theta_{a}$ takes 3D object feature $F_\\mathcal{O}$ as the query, head motion $F_\\mathcal{M}$ and appearance $F_\\mathcal{V}$ as two parallel key-value pairs to calculate the affordance feature. For $\\Theta_{c}$, the $F_\\mathcal{V}$ is the query, while affordance feature $F_{a}$ and $F_\\mathcal{M}$ are parallel key-value pairs, it calculates the contact feature. Both $\\Theta_{a}$ and $\\Theta_{c}$ perform the gradient modulation. We will add notations in Fig.3 and more details in Sec. 3.3 to clarify the procedure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kg9KmWuOrZ", "reviewer": "Reviewer_L6qS", "summary": "This paper describes an approach to estimate interaction between humans and objects from egocentric videos. To this end, head movement, 3D object meshes and 2D egocentric video are used as input and processed individually before being combined to predict contact regions in both object and subject. Experiments are carried out on Ego-Exo4D and GINO datasets, with additional annotations provided for both.", "questions": "1. What is the (technical) novelty of the current approach over related works (in particular LEMON)?\n2. What is the influence of a learned prior on the contact regions of both object and subject? Could the authors show the variation in estimations across videos of the same interaction?", "limitations": "Partly. While some limitations have been addressed, it's clear that the system is rather specific in the way it's been evaluated.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ novel model in which the flow of information makes sense\n+ quantitative and qualitative results give a significant amount of insight into the method's performance and inner workings. It's clear that all modules contribute to the increased performance over baselines.", "weaknesses": "- contributions of the paper a bit unclear, in terms of novelty over related work\n- critical discussion missing\n- paper hard to read\n\nContribution: The approach seems to be heavily influenced by LEMON, and the comparisons in the experiment section are also predominantly with LEMON. Still, differences between the two models are not outlined. It is only when really reading through the text and the appendix that differences in architecture and (training) details because more clear. But it's left to the reader. Linking Figures 1-3 more tightly with the text can help to better understand the particular novelties of the method. Now, these figures are relatively disjoint from the story.\n\nCritical discussion missing: I would have appreciated a more critical discussion. While statements on Lavila's anticipation bias are reported, I don't see critical reflections on the performance reported for the main model. Still, given that human-object contact is modeled, the results depicted in Fig. 8 (for example) raise a number of questions, such as why the blade of the knife is identified (rather than the handle), or why the 3D model of the bottle is off, and why one of the two hands is not considered in the interaction with the bottle). Also, from the ablation study, the influence of various key components seems marginal. Leaving out many seemingly important terms still leaves a performance well higher than LEMON. Is there a very strong prior encoded? The experimental section gives me the feeling that the authors have been more critical to other works than to their own. This raises questions about the merits.\n\nPaper hard to read: I found the paper particularly verbose, and it seems the authors have prioritized colorful language over concise, precise descriptions. The result of that choice is that many parts, including abstract and introduction, are particularly hard to read. Given the complexity of the work and the sheer number of concepts and symbols, it's easy to get confused when structure and language are not to-the-point. As a result, it's difficult to understand the true contributions of the work. Evidently, there is a host of related work on capturing estimating contact in HOI, also from the first person perspective. What seems to be novel is the more connected processing of the inputs, including head motion. But I found little motivation into the specific design choices to do so. For one, it's not motivated why a geometry-semantics approach is not preferred. While it's clear that this only works for objects that have been considered during training, it seems the used data features objects that are rather standard, with affordances that can be well predicted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1. Contributions, in terms of novelty over related work.**\n\nWe summarize the types of existing methods that estimate interaction regions. A distinct type involves estimating 2D regions [1]. For methods in 3D, the types include: 1) part-level contact [2]; 2) estimate human or object in isolation [3]; 3) estimate regions for a single exocentric image [4]; 4) estimate from the egocentric view, but only focus on hands [5].\n\nDifferent from them, we propose estimating **3D** interaction regions at dense **vertex-level** for **both** humans and objects from **egocentric videos**. Additionally, we extend egocentric estimation to the **body**. In this case, interacting subjects and objects may disappear from view, the model should address ambiguity between visual observations and interaction contents, which is more challenging.\n\n[1] Detecting human-object contact in images  \n[2] CHORE: Contact, Human and Object REconstruction from a single RGB image  \n[3] DECO: Dense estimation of 3D human-scene contact in the wild    \n[4] LEMON: Learning 3D Human-Object Interaction Relation from 2D images  \n[5] ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation\n\n--- \n> **W2. Differences with LEMON.**\n\n* **Task and interaction clues**: LEMON estimates interaction regions for an exocentric image, where humans are observed relatively intact, and human mesh recovery (HMR) methods can be employed to obtain human geometries. It leverages clues such as the object and posed human geometries, as well as the interaction content (semantics) in images. EgoChoir estimates the regions for egocentric videos (temporally dense human contact). From this view, appearances alone may not provide effective interaction contexts, and human geometry is also hard to obtain. Thus, EgoChoir combines head movement, visual appearance, and 3D object as clues to model interaction regions. Note: a significant performance gap remains between HMR methods on egocentric and exocentric views, addressing the concern of \"why a geometry-semantics approach is not preferred.\"\n* **Technical differences**: LEMON regards clues (semantics and geometries) as equally important for modeling. Its core technique involves building semantic correlations between geometries using image content as a shared key-value and constructing geometric correlations through a handcraft feature: curvature. In contrast, Egochoir seeks to differentiate the effectiveness of multiple clues (e.g. appearance, motion) based on interactions (hand or body). It achieves this by introducing a gradient modulation mechanism in the parallel cross-attention that correlates these clue features.\n\n---\nBased on responses to **W1/W2**, we briefly reiterate contributions:\n\n* We first propose estimating interaction regions of the whole human body and objects from egocentric videos.\n* Introducing a heuristic framework EgoChoir. Analogous to human behavior, it harmonizes the visual appearance, head motion, and 3D object to model interaction regions and adapt to distinct egocentric interaction scenarios through a gradient modulation mechanism.\n* Annotating contact for collected video clips and affordance for 3D objects, which supports the training and evaluation of the task. \n\n---\n> **W3. The experiments mentioned and critical discussion.**\n\n1) **Comparisons are predominantly with LEMON**: We report quantitative comparison metrics with five baselines in Tab. 1. LEMON performs better than other baselines, so we just chose it for qualitative comparison.\n2) **Critical discussion missing**: In lines 317-321, we critically clarify that the current model may estimate the region slightly before or after the exact frame due to the lack of spatial relation perception. This problem is currently difficult to solve because recovering human meshes from an egocentric view is challenging. So that the spatial relation is hard to capture.\n3) **Marginal influence of components, leaving out still better than LEMON**: In Tab.2, some metrics, e.g. precision and F1, are shown in percentages, the improvement is actually significant. For instance, head motion $\\bar{\\mathcal{M}}$ increases the precision from 0.68 to 0.78, a 10-percentage point. Plus, without certain key designs, e.g. object concept $F_{a}$, some metrics (F1:0.66, AUC:75.21) are not higher than LEMON (F1:0.67, AUC:75.97). LEMON relies on image content as guidance, while in egocentric videos, the subject and object disappear in many frames. Its framework is hard to address in this case and leads to lower performance. That's why EgoChoir leaves out some designs still better than it.\n4) **Questions about Fig. 8**: The definition of object affordance is key to understanding the case in Fig. 8. Affordance is first defined as \"opportunities of interaction\" [1]. In the computer vision field, object affordance refers to the functional regions that support certain interactions [2], it contains implicit functional semantics and is not completely equivalent to the contact region. In Fig. 8, videos display cutting with a knife and pouring from a bottle, so affordance regions are the blade and top of the bottle.\n5) **Only works for standard objects**: The 3D objects we collect also include those from scanning and multi-view reconstruction. We present inference results on these noisy 3D objects **in the added PDF**.\n6) **Learned prior? Same interaction across videos**: EgoChoir actually learns to reasonably adopt appropriate clue features based on distinct egocentric interactions, to capture effective interaction contexts. These clues are complementary, enabling it to handle situations such as appearance variations and missing subjects or objects. We provide qualitative results about the same interaction across videos **in the added PDF**, and quantitative results are reported in Appendix Tab. 5.\n\n[1] The ecological approach to visual perception  \n[2] 3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to all the reviewers for their effort and constructive feedback. We are encouraged that the reviewers appreciate our work, including:

* The key idea of the method is ingenious for the proposed task [Reviewer paDU, fLMx]
* The superiority of performance over baselines [Reviewer PyYF, paDU, fLMx, L6qS]
* Extensive and comprehensive analysis of the proposed method [Reviewer PyYF, paDU, fLMx]
* Clear and well written [Reviewer PyYF, paDU]

There are questions prompting additional investigations. We provide here a pdf containing the results of those investigations. Its contents are:

* Figure 1: Statistical information of the dataset and schematic figure of modulation for reviewer fLMx  
* Figure 2: Inference results of noisy 3D objects for reviewer L6qS  
* Figure 3: The same interaction across videos for reviewer L6qS  

Specific context and discussions are presented in the corresponding review rebuttals. We are open to addressing any issues from reviewers during the discussion stage. We will adapt the paper based on your insightful comments, feedback, and questions.

Many thanks,

The authors

Meta-review: The paper presents a novel approach to estimating 3D human-object interaction regions from egocentric videos, which is a crucial yet under-explored area. The reviewers find the paper's presentation clear. The method is novel and experimental validations are comprehensive.  
While some reviewers initially had concerns about the technical novelty and clarity of the contributions, these were adequately addressed during the rebuttal phase, leading to an increase in their ratings.  

The proposed method's ability to integrate visual appearance, head motion, and 3D object information to infer interaction regions is seen as a significant contribution to the field.  Additionally, the creation of a new dataset with 3D contact and affordance annotations further strengthens the paper's impact.  Overall, the positive reviews, the authors' responsiveness to feedback, and the potential impact of the work justify accepting this paper.
Decision: Accept (poster)