Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment
Authors: Zihui Xue
Kristen Grauman
Venue: NeurIPS 2023 poster (2023)
Keywords: fine-grained video understanding
egocentric video
self-supervised learning
temporal alignment
TL;DR: a self-supervised embedding approach to learn fine-grained action features invariant to the egocentric and exocentric viewpoints

Abstract: The egocentric and exocentric viewpoints of a human activity look dramatically different, yet invariant representations to link them are essential for many potential applications in robotics and augmented reality.  Prior work is limited to learning view-invariant features from paired synchronized viewpoints.  We relax that strong data assumption and propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time, even when not captured simultaneously or in the same environment. To this end, we propose AE2, a self-supervised embedding approach with two key designs: (1) an object-centric encoder that explicitly focuses on regions corresponding to hands and active objects; (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context, comprising four datasets---including an ego tennis forehand dataset we collected, along with dense per-frame labels we annotated for each dataset. On the four datasets, our AE2 method strongly outperforms prior work in a variety of fine-grained downstream tasks, both in regular and cross-view settings.

Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment
Zihui Xue Kristen Grauman
'he University of Texas at Austin
FFAIR, Meta
Abstract
The egocentric and exocentric viewpoints of a human activity look dramatically different, yet invariant representations to link them are essential for many potential applications in robotics and augmented reality. Prior work limited to learning view-invariant features from paired synchronized viewpoints. We relax that strong data assumption and propose to learn fine-grained action features that are invariant the viewpoints by aligning egocentric and exocentric videos in time, even when not captured simultaneously or in the same environment. To this end, we propose AE2, self-supervised embedding approach with two key designs: (1) object-centric encoder that explicitly focuses on regions corresponding to hands and active objects; and (2) contrastive-based alignment objective that leverages temporally reversed frames as negative samples. For evaluation, we establish a benchmark for fine-gree fine-grained video understanding the ego-exo context, comprising four cntasets-includig an ego tennis forehand dataset we collected, along with dense per-frame labels we annotated for each dataset. On the four datasets, our AE2 method strongly outperforms prior work in variety of fine-grained downstream tasks, both in regular and cross-view settings.
1 Introduction
Fine-grained video understanding aims to extract the different stages an activity and reason about their temporal evolution. For example, whereas coarse-grained action recognition task [31, 16, 68, 20, 7] might ask, "is this sewing or snowboarding or...?", a fine-gratd fine-grained action recognition task requires detecting each of the component steps, e.g., "the person threads the needle here, they poke the needle through the fabric here..." Such fine-grained understanding of video is important in numerous applications that require step-by-step understanding, ranging from robot imitation learning [42, 63] to skill learning from instructional "how-to" videos [86, 73, 47, 46).
The problem is challenging on two fronts. First, the degree of detail and precision eludes existing learned video representations, which are typically trained to produce global, clip-level descriptors of the activity [69, 74, 18, 5]. Second, in many scenarios of interest, the camera viewpoint and background will vary substantially, making the same activity look dramatically different. For example, imagine user wearing AR/VR glasses who wants to compare the egocentric recording of themselves doing a tennis forehand with a third-person (exocentric) expert demonstration video YouTube, or robot tasked with pouring drinks from its egocentric perspective but provided with multiple third-person human pouring videos for learning. Existing view-invariant feature learning [76, 27, 78, 45, 57, -nclunnuu methods targeting ego-exo views [63, 660 aasume that training data contains synchronized video pairs capturing the same action simultaneously in the same environment. This is strong assumption. It implies significant costs and physical constraints that can render these methods inapplicable in complex real-world scenarios.
edu/ http://rjecctseeee https/AAgggggggoggee 37th Conference on Neural Information Processing Systems (NcurIPS 2023).
Project webpage: https://vissonncs utexas edu/

---

Fine-grained Action Understanding To recognize fine-grained actions, supervised learning ap- proaches [82, 30, 59, 65] employ fine-grained action datasets annotated with sub-action bound- aries. In contrast, our self-supervised approach requires much lighter annotations to pretrain for ego video
video
Temporal Alignment
embedding space
Figure I: Left: We propose AE2, self-supervised approach for learning frame-wise action features that are invariant the egocentric and exocentric viewpoints. The key is find correspondences across the viewpoints via temporal alignment. Right: Before training, the frame embeddings are clustered by viewpoint (pink and green); after training, our learned AE2 embeddings effectively capture the progress of an action and exhibit viewpoint invariance. Shading indicates time.
Our goal is to learn fine-grained frame-wise video features that are invariant to both the ego and exo views,2 2 from unpaired data. the unpaired setting, we know which human activity occurs in any given training sequence (e.g-. pouring, breaking eggs), but they need not be collected simultaneously or in the same environment. The main idea our self-supervised approach is to infer the temporal alignment between ego and exo training video sequences, then use that alignment to learn a view- invariant frame-wise embedding. Leveraging unpaired data means our method is more flexible-both in terms of curating training data, and in terms of using available training data more thoroughly, since with our approach any ego-exo pairing for an action becomes a training pair. Unlike existing temporal alignment for third-person videos [15, 54, 22, 23, 41], where canonical views inherently similar, ego-exo views are so distinct that finding temporal visual correspondences non-trivial (see Fig. 1).
To address this challenge, we propose two main ideas for aligning ego and exo videos in time. First, motivated by the fact that human activity in the first-person view usually revolves around hand-object interactions, we design object-centric encoder to bridge the gap between ego-exo viewpoints. The encoder integrates regional features corresponding to hands and active objects along with global image features, resulting in more object-centric representations. Second, we propose a contrastive- based alignment objective, where we temporally reverse frames to construct negative samples. The rationale is that aligning ego-exo video pair should be easier (i.e., incur lower aligmment cost) compared to aligning the same video pair when one is played in reverse. We bring both ideas together in single embedding learning approach call AE2, for "align ego-exo".
To evaluate the learned representations, we establish an ego-exo benchmark for fine-grained video understanding. Specifically, we assemble four datasets of atomic human actions, comprising ego and exo videos drawn from five public datasets and an ego tennis forehand dataset that we collected. In addition, annotate these datasets with dense per-frame labels (for evaluation only). We hope that the enriched data and labels that we release publicly can support progress in fine-grate nne-grained temporal understanding from the ego-exo perspective. This is valuable dataset contribution alongside technical approach contribution. Furthermore, we present a range of practical downstream tasks that demand frame-wist wise ego-exo aew-invariant features: action phase classification and retrieval across views, progress tracking, and synchronous playback of ego and exo videos. Experimental results demonstrate considerable performance gains of AF2 across all tasks and datasets.
2 Related Work
We use "ego" and "exo" shorthand for egocentric (first-person) and exocentric (third-person).

---

action phase classification, namely, video-level action labels. Keystep recognition in instructional videos [86, 73, 47, 46, 2] is a related form of fine- aen-grained action understanding, where the goal is to name the individual steps of some longer activity (e-g., making brownies requires breaking eggs, adding flour, etc.). In that domain, multi-modal representation learning a powerful tool that benefits from the spoken narrations that accompany how-to videos [47, 72, 38, 84]. In any of the above, handling extreme viewpoint variation is not an explicit target.
Self-supervised Video Representation Learning Various objectives for self- video features have been explored [61]. This includes using accompanying modalities like audio [51, 29, 49] and text or speech [47, 72, 38, 37, 84] as training signals, as well as temporal coherence priors [28, 26, 63, 44, 10] or self-supervised tasks that artificially alter the temporal order of videos [48, 19, 35]. Alignment-
objectives have also been explored [15, 54, 22, 23, 41, 33], with ideas based on cycle consistency [15, 54], soft dynamic time warping (DTW) [22, 231, and optimal transport (OT) [41]. Our model also has alignment stage; however, unlike our work, all these prior methods are explored for third-person videos only (where views are generally more similar across videos than ego-exo) without attending to the viewpoint discrepancy problem. Frame-wise representations invariant to both ego and exo viewpoints remain unresolved.
View-invariant Action Representation Learning Many prior works have explored view-invariance in action recognition [76, 9, 53J. One popular approach is to employ multi-view videos during training to learn view-invariant features [76, 77, 27, 78, 45, 71J. Several works [17, 39, 83, 55] discover latent feature space independent of viewpoint, while others use skeletons [81], 2D/3D poses [56. 71] and latent 3D representations [53] extracted from video to account for view differences.
As advances in AR/VR and robotics unfold, ego-exo viewpoint invariance takes on special interest. Several models relate -aarse-grained action information across the two drastically different do- mains [70, 1, 80, 36), for action recognition using both ego and exo cameras [70] or cross-viewpon video recognition [80, 36]. Closer to our work are Time Contrastive Networks (TCN) [63] and CharadesEgo [66, 67], both of which use embeddings aimed at pushing (ir)relevant ego-exo views closer (farther). However, unlike our approach, both employ videos that concurrently record the same human actions from ego and exo viewpoints. In fact, except for [36], all the above approaches rely on simultancously recorded, paired multi-view videos. However, even in the case of [36], the goal is a coarse, clip-wise representation. In contrast, our method leverages temporal alignment as the pretext task, enabling fine-grained representation learning from unpaired ego and exo videos.
Object-centric Representations Several works advocate object-centric representations to improve visual understanding [40, 85, 50, 36], such as an object-centric transformer for joint hand motion and interaction hotspot prediction [40], or using a pretrained detection model for robot manipulation tasks [85, 50]. Hand-object interactions are known to dominate ego-video for human-won worn cam- eras [11, 211, and recent work explores new ways to extract hands and active objects [64, 43, 12]. Our proposed object-centric transformer shares the rationale focusing on active objects, though in our case as a key to link the most "matchable" things in the exo view and bridge the ego-exo gap.
3 Approach
We first formulate the problem (Sec. 3.1) and introduce the overall self-supervised learning objective (Sec. 3.2). Next we present our ideas for object-centric transformer (Sec. 3.3) and an alignment-basee contrastive regularization (Sec. 3.4), followed by a review of our training procedure (Sec. 3.5).
3.1 Learning Frame-wise Ego-Exo View-invariant Features
Our goal is to learn an embedding space shared by both ego and exo vidcos that characterizes the progress of an action, depicted in Fig. (right). Specifically, we aim to train view-invariant encoder network capable of extracting fine-grained frame- wise features from a given (ego or exo) video. Training is conducted in a self-supervised manner.
Formally, let o(-;8) represent a neural network encoder parameterized by 9. Given an ego video sequence S with M frames, denoted by S 5 /S,....,,MM. where denotes the i-th input frame, we apply the encoder to extract frame-wise features from S. i.e. x; = Φ(s;; 0), Vi E (l...., M). The resulting ego embedding is denoted by X = [x|,.... xas]. Similarly, given an exo video sequence V .V,...., VN) with N frames, we obtain the frame-wist wise exo embedding Y = (y1,..., N,), where y o(v.;0) Vi e [I, using the same encoder Φ.
3

---

The self-supervised learning objective is to map S and V to shared feature space where frames representing similar action stages are grouped together, regardless of the viewpoint, while frames depicting different action stages (e.g., the initial tilting of the container versus the subsequent decrease in the liquid stream as the pouring action nears completion) should be separated. We present series of downstream tasks to evaluate the quality of learned feature embeddings X and Y (Sec. 4).
An intuitive approach to utilize paired ego-exo data collected simultaneously, adopting loss function that encourages similarities between frame representations at the same timestamp while maximizing the distance between frame representations that are temporally distant [63, 66]. However, this approach relies on
views, which is often challenging to realize in practice.
Instead, we propose to achieve view-invariance from umpaired ego-exo data; the key is to adopt temporal alignment as the pretext task. The encoder & learns to identify visual correspondences across views by temporally aligning ego and videos that depict the same action. Throughout the learning process, no action phase labels or time synchronization is required. To enforce temporal alignment, we assume that multiple videos portraying the same action (e.g., pouring, tennis forehand), captured from different ego and exo viewpoints (and potentially entirely different environments), are available for training.
3.2 Ego-Exo Temporal Alignment Objective
To begin, we pose our learning objective as a classical dynamic time warping (DTW) problem [4]. DTW has been widely adopted to compute the optimal alignment between two temporal sequences [8, 6, 23,,,22 22, 79] according to a predefined cost function. In our problem, we define the cost of aligning frame in ego sequence S and frame j in exo sequence V the distance between features extracted by our encoder é, x; and y,, so that aligning more similar features results a lower cost.
Formally, given the two sequences of embeddings X of length M and Y of length N, we calculate cost matrix E IR M XxX,, with each element G.j computed by a distance function D. We adopt the distance function from [22], and define Cij to be the negative log probability of matching x, to selected y, in sequence Y:
C, = D(x,, y,) = log expl XT //13
X; = =xx//xxl ÿ. = y./llylll
(I)
E cxp(x/vn/B))
where 3 is hyper-paramette controlling the softmax temperature, which we fix at 0.1.
DTW calculates the alignment cost between X and Y by finding the minimum cost path in C. This optimal path can be determined by evaluating the following recurrence [60]]
R(i.j) = G.j min[R( 1,j 1), R(i R(ijj
where R(i,)) denotes the cumulative distance function, and R,O, 0) = O. Given that the min operator non-differentiable, we approximate it with smooth differentiable operator, defined in [34, 22]. This yields a differentiable loss function that can be directly optimized with pack-propagation:
Calign XX Y) N).
While DTW has been applied frame-wise action feature learning [23, 22] with some degree view invariance across exo views, it is inadequate to address the ego-exo domain gap. As we will show in results, directly adopting the DTW loss for learning ego-exo view-invariant features is sub-optimal since the ego and exo views, even when describing the same action, exhibit significant differences (compared to differences within ego views or within exo views). In Sec. 3.3-3.4, we propose two designs of AE2 that specifically account for the dramatic ego-exo view differences.
3.3 Object-centric Encoder Network
The visual appearance in ego and exo views may look dramatically different due to different back- grounds, fields of view, and (of course) widely disparate viewing angles. Yet a common element shared between these two perspectives is the human action itself. In this context, hand-object interac- tion regions are particularly informative about the progress of an action. Hand-object interactions are prominent in the first-person viewpoint; the wearable camera gives a close-up view of the person's 's near-field field manipulations. In exo vidcos, however, these active regions may constitute only small

---

band
ROI
Align
V (Vi."
object
Prnuafenar
Callgn (X, Y)
Pas
Input Frame 84
Xi
global
Object-centric Encoder
feliin (X, Y)
Figure 2: Overview of AE2. We extract frame-wis wise representations from the two video sequences (i.e., S and V) using the encoder & and temporally align their embeddings (i.e., X and Y) using DTW. Left: Our proposed encoder network design emphasizes attention to hand and active object regions, leading more object-centric representations, Right: As form of regularization, we reverse the video sequence S and enforce that the cost of aligning (S, V) is less than that of aligning (S, V).
portion of the frame and are not necessarily centered, depending on the third-person camera position and angle. Inspired by these observations, we propose an object-centric encoder design that explicitly focuses on regions corresponding hands and active objects to better bridge the ego-exo gap.
Our proposed AE2 encoder network & consists of two main components: (I) ResNet-50 encoder [25], denoted by @base. extract features from the input video frame; (2) a transformer encoder [75], denoted by Ptramsoommer to fuse global features along with regional features through self-attention.
As preprocessing step. employ off shellf detector to localize hand and active object regions in ego and exo frames [64]. For frame i, we select the top two hand and object proposals with the highest confidence scores, resulting in a total of four bounding boxes (BB.,)==
First, given one input frame 8; at timestamp we pass to chase to obtain global features: g: = (b,)) Next, we apply ROI Align [24] to intermediate feature maps produced by Øbasc for the input frame St. This yields regional features ri for each bounding box BB,,k: Fi ROI Align( (oose (s:), BB,,)),, Vk.
Both the global and regional features are flattened as vectors and passed through projection layer unify their dimensions. We denote the projection layer to map global and local tokens ppooeess, and Oprojeet, respectively. The input tokens are then enriched with two types of embeddings: (1) learnable spatial embedding, denoted by espatial. which encodes the bounding box coordinates and confidence scores obtained from the hand-object detector and (2) a learnable identity embedding, denoted by Cidentiyy which represents the category of each feature token, corresponding to the global image, left hand, right hand, and object. This yields the tokens: Zi.globul project, (g:) Cadentity and
Zi,ooalk = preojett, (I,,)) + espatial + eieentity Vk.
The transformer encoder processes one global token along with four regional tokens as input, leveraging self-attention to conduct hand-object interaction reasoning. This step enables the model to effectively capture action-related information within the scene, yielding the transformer outputs: f.A @mmnsformme (zi.glohal [Zi,localk ha). Finally, we average the transformer output tokens and adopt one embedding layer (embeddiing to project the features to our desired dimension, yielding the final output embedding for the input frame s;: @embedding (3 Σ3 ft,:).
Fig. 2 (left) showcases our proposed object-centric encoder. This design inherently bridges the ego- exo gap by concentrating on the most informative regions, regardless of the visual differences between the two perspectives. As result, the object-centric encoder can better align features extracted from both ego and exo videos, enhancing the learned features in downstream tasks.
3.4 Contrastive Regularization With Reversed Videos
As shown previous studies [22, 23), directly optimizing the DTW loss can result in the collapse of embeddings and the model settling into trivial solutions. Morcover, aligning ego and exo videos
5

---

presents new challenges due to their great differences in visual appearance. Thus, careful design of regularization, supplementing the alignment objective in Eqn. (3), is imperative.
We resort to discriminative modeling as the regularization strategy. The crux lies in how to con- struct positive and negative samples. A common methodology in previous fine-grained action approaches [63, 66, 23, 10], is to define negative samples on a frame-by- close frames are considered as positive pairs (expected to be nearby in the embedding space), while temporally far away frames are negative pairs (expected to be distant). This, however, implicitly assumes clean data and strict monotonicity in the action sequence sequence: if video sequence involves certain degree of periodicity, with many visually similar frames spread out over time, these approaches could incorrectly categorize them as negative samples.
basis: temporally
To address the presence of noisy frames in real-world videos, we introduce a global perspective that creates negatives at the video level. From given positive pair of sequences (S, V), we derive negative pair (S, V), where $ [SM, .,, si] denotes the reversed video sequence S. The underlying intuition is straightforward: aligning an ego-exo video pair in their natural temporal order should yield lower cost than aligning the same pair when one video played in reverse. Formally, we employ a hinge loss to impose this regularization objective:
Leg [X Y) max(Calign (X, Y) Calign X Y)
where X denotes the embeddings of S, which is essentially X in reverse. Fig. 2 (right) illustrates the alignment cost matrices of (X, Y) and (X, Y). A lighter color indicates a smaller cost. Intuitively, Calign corresponds to the minimal cumulative cost traversing from the top-left to the bottom-right of the matrix. As seen in the figure, the alignment cost increases when X is reversed, i.e., Latign (X, Y) should be larger than Lalign X, Y).
Compared with frame-level negatives, our formulation is inherently more robust to repetitive and background frames, allowing for degree of temporal variation within video. This principle guides two key designs: (1) In creating negative samples, we opt for reversing the frames in S rather than randomly shuffling them. The latter approach may inadvertently generate plausible (hence, not negative) temporal sequence, particularly when dealing with short videos abundant in background frames. (2) To obtain informative frames the positive sample, we leverage the hand-object detection results from Sec. 3.3 to perform weighted sampling of the video sequences S and V. Specifically, we sample frames proportionally their average confidence scores of hand and object detections. The subsampled S and V thus compose positive pair. This way, frames featuring clear hand and object instances are more likely be included.
3.5
Training and Implementation Details
Our final loss is combination of the DTW alignment loss (Eqn. (3)) and the contrastive regularization loss (Eqn. (4)):
C(X, Y) Calign (X Y) + ACrog (X, Y),
(5)
where À is syper-paramette controlling the ratio of these two loss terms.
During training. we randomly extract 32 frames from each video to construct a video sequence. The object-centric encoder network o. presented in See. 3.3, is optimized to minimize the loss above for all pairs of video sequences in the training set, including ego-ego, exo-exo, and ego-exo pairs -all of which are known contain the same coarse action. During evaluation, we freeze the encoder and use it to extract 128-dimensional embeddings for each frame. These representations are then assessed across variety of downstream tasks (Sec. 4). See Supp. for full implementation details.
Datasets and Evaluation
Datasets Existing datasets for fnneggained video understanding (e.g. [82, 65, 3)) are solely com-
posed of third-person vidcos. Hence we propose a new ego-exo benchmark by assembling ego and exo videos from five public datasets CMU-MMAC [13], H2O [32], EPICCKitchens [11], HMDB51 [31]
'I Sec. 3.1. we initially denote X and Y as ego and exo embeddings. However, in this context, we extend the meaning of X and Y to represent any pair of embeddings from the training set.

---

Stet
Phanc
Ligud exit cortaterr
Plaae
Complete
Pouting
2
Sten
Phose
Loadaoi ball
Racket
Figure 3: Example labels for Pour Liquid (left) and Tennis Forehand (right). Key events displayed in boxes below sequences, with the phase label assigned to each frame between two key events.
and Penn Action [82]-plux a newly collected ego tennis forehand dataset. Our benchmark consists of four action-specific ego-exo datasets: (A) Break Eggs: ego-exo videos of 44 subjects breaking
eggs, with 1 11//30 train/val+test samples from CMU-MMAC [13]; (B) Pour Milk [32]: ego-exo videos of 10 subjects pouring milk, with 77/35 train/val+tes from H2O [32]; (C) Pour Liquid: we extract ego "pour water" clips from EPIC-Kitchens [11] and exo "pour" clips from HMDB51 [31],
yielding 137/56 train/val+test. (D) Tennis Forehand: we extract exo tennis forehand clips from Penn Action [82] and collect ego tennis forehands by 12 subjects, totaling 173/149 train/val+test clips. We obtain ground truth key event and action phase labels for all of them (used only in downstream evaluation), following the labeling methodology established in [15]. Fig. illustrates examples of these labels. Altogether, these four datasets provide unpaired ego-exo videos covering both tabletop (i.e., breaking eggs, pouring) and physical actions (i.e., tennis forehand). While (A) and (B) from more controlled environments, where exo viewpoints are fixed and both views are collected in the same environment, (C) and (D) represent in-the-wild scenarios, where ego and exo are neither synchronized nor collected in the same environment. Our model receives them all as unpaired.
Evaluation We aim to demonstrate the practical utility of the learned ego-exo view-invariaat. features by addressing two questions: (1) how well do the features capture the progress of the action? (2) how view-invariant are the learned representations? To this end, test four downstream tasks: (I) Action Phase Classification: train an SVM classifier on top of the embeddings to predict the action phase labels for each frame and report Fl score. We also explore few-shot and cross-view zero-shot settings (i.e., "exo2ego" when training the SVM classifier on exo only and testing on ego, and vice versa). (2) Frame Retrieval: parallels the classification task, but done via retrieval (no classifier), and evaluated with mean average precision (mAP)@ K, for 10,510.15. We further extend this task to the cross-view setting to evaluate view-invariance. (3) Phase Progression: train linear regressor the frozen embeddings to predict the phase progression values, defined as the difference in time-stamps between any given frame and each key event, normalized by the number of frames in that video [15], evaluated by average R R-square. (4) Kendall's Tau: to measure how well-aligned two sequences are in time, following [15, 22, 23, 10, 41].
Note that: (1) Departing from the common practice of splitting videos into train and validation
only [15], our benchmark includes dedicated test set to reduce the risk of model overfitting. (2)
We opt for the Fl score instead of accuracy for action phase classification to better account for the label imbalance in these datasets and provide more meaningful performance evaluation. (3)
Phase progression assumes a high level of consistency in actions, with noisy frames diminishing the performance greatly. Due to the challenging nature of Pour Liquid data, we observe a negative progression value for all approaches. Thus, we augment the resulting embeddings with temporal dimension, as 0.001 times the time segment as the input so that the regression model can distinguish repetitive (or very similar) frames that differ in time. We report modified progression value for all approaches on this dataset. (4) Kendall's Tau assumes that there are no repetitive frames a video. Since we adopt in-the-wild videos where strict monoticity is not guaranteed, this metric may not faithfully reflect the quality of representations. Nonetheless, we report them for completeness and for consistency with prior work. (5) With focus being frame-wist representation learning, the total number of training samples equals the number of frames rather than the number videos, reaching a scale of few thousands (8k-30k per action class). See Supp. for more dataset and evaluation details.
Experiments
Baselines We compare AE2 with total baselines of three types: (1) Naive baselines based on random features or ImageNet features, produced by a ResNet-50 model randomly initialized
Although Break Eggs (and a portion of Pour Milk) offers synchronized ego-exo pairs, AE2 ignores the pairing and is trained in sulf-super manner. AE2 never uses this synchronization supervision signal.
7

---

Table 1: Benchmark evaluation on four datasets: (A) Break Eggs; (B) Pour Milk: (C) Pour Liquid; (D) Tennis Forehand. Top results are highlighted in bold, while second- results are underlined. AE2 performs best across all tasks and datasets, in both regular and cross-view scenarios.
Classification (Fl score)
Frame Retrieval (mAP@ 10)
Phase
Kendall's
Dataset
Method
regular
ego2exo
exolego
regular
ego2exo
exo2ego
Progression
Tau
Random Features
19.18
18.93
19.45
47.13
41.74
38.19
-0.0572
0.0018
ImageNet Features
50.24
21.48
32.25
50.49
33.09
37.80
-0.1446
0.0188
ActerObserverNet [66]
36.14
36.40
31.00
50.47
42.70
41.29
-0.0517
0.0024
single-view TCN [63]
56.90
18.60
35.61
53.42
32.63
34.91
0.0051
0.1206
(A)
multi-view TCN [63]
59,91
48.65
56.91
58.83
47.04
52.68
0.2669
0.2886
CARL [10]
43.43
28.35
29.22
46.04
37.38
39,94
-0.0837
-0.0091
TCC [15]
59.84
54.17
52.28
58.75
61.11
62.03
0,2880
0.5191
GTA [22]
56.86
52.33
58.35
61.55
56.25
53.93
0,3462
0.4626
AE2 (ours)
66.23
57.41
71.72
65.85
64,59
62.15
0,5109
0,6316
Random Features
36.84
33.96
41.97
52.48
50.56
51.98
-0.0477
0.0050
ImageNet Features
41.59
39.93
45.52
54.09
27.31
43.21
-2.6681
0.0115
single-view TCN [63]
47.39
43.44
42.28
57.00
46.48
47.20
-0,3238
-0.0197
(B)
CARL [10]
48.79
52.41
43.01
55.01
52.99
51.51
-0.1639
0.0443
TCC [15]
77.91
72.29
81.07
80.97
75.30
80.27
0.6665
0.7614
GTA [22]
81.11
74.94
81.51
80.12
72.78
75,40
0.7086
0.8022
AE2 (ours)
85.17
84.73
82.77
84,90
78.48
83.41
0,7634
0,9062
Random Features
45.26
47.45
44.33
49.83
55.44
55.75
-0.1303
-0.0072
ImageNet Features
53.13
22.44
44.61
$1,49
52.17
30.44
-1.6329
-0.0053
single-view TCN [63]
54.02
32.77
51.24
48.83
55,28
31.15
-0.5283
0.0103
(C)
CARL [10]
56.98
47.46
52.68
55.29
59.37
36.80
-0.1176
0.0085
TCC [15]
52.53
43.85
42.86
62.33
56.08
57.89
0.1163
0.1103
GTA [22]
56.92
42.97
59.96
62.79
58.52
53.32
-0.2370
0.1005
AE2 (ours)
66.56
57.15
65.60
65.54
65.79
57.35
0.1380
0.0934
Random Features
30.31
33.42
28. 10
66.47
58.98
59.87
-0.0425
0.0177
ImageNet Features
69.15
42.03
58.61
76.96
66.90
60.31
-0.4143
0.0734
single-view TCN [63]
68.87
48.86
36.48
73.76
55.08
56.65
-0.0602
0.0737
(D)
CARL [10]
59.69
35.19
47.83
69.43
54.83
63.19
-0.1310
0.0542
TCC [15]
78.41
53.29
32.87
80.24
55.84
47.27
0,2155
0,1040
GTA [22]
83.63
82,91
81.80
85,20
78.00
79.14
0.4691
0.4901
AE2 (ours)
85.87
84.71
85.68
86.83
81.46
82.07
0,5060
0.6171
or pretrained on ImageNet [14]; (2) Self-supervised learning approaches specificallyy designed for learning ego-exo view-invariant features, time-contrastive networks (TCN) [63] and ActorOb- serverNet [66]. Both require synchronized ego-exo videos for training, and are thus only applicable to Break Eggs data. Additionally, TCN offers a single-view variant that eliminates the need for synchronized data, by taking positive frames within a small temporal window surrounding the anchor, and negative pairs from distant timesteps. We implement single-viey view TCN for all datasets. (3) Fine- grained action representation learning approaches, CARL [10] which utilizes sequence contrastive loss, and TCC [15] and GTA [22], which also utilize alignment as the pretext task but do not account for ego-exo viewpoint differences.
Main Results In Table 1, we benchmark all baseline approaches and AE2 on four ego-exo datasets. See Supp. for full results, including the Fl score for few-shot classification, and mAP@S, 15 for frame retrieval. From the results, it is clear that AE2 greatly outperforms other approaches across all downstream tasks. For instance, on Break Eggs, AF2 surpasses the multi-view TCN [63], which utilizes perfect ego-exo synchronization as a supervision signal, by +6.32% in Fl score. Even when we adapt the TCN to treat all possible ego-exo pairs as perfectly synchronized, AE2 still excels, showing a considerable margin of improvement (see Supp. for discussion). Pour Liquid poses the most substantial challenge due to its un-the-wild nature, as reflected by the low phase progression and Kendall's s Tau values. Yet, AE2 notably improves on previous works, achieving, for instance, +9.64% Fl score. Regarding Tennis Forehand, another in-the-wild dataset, we note that some methods like TCC perform well identifying correspondences within either the ego or exo domain, but struggle to connect the two. Consequently, while it achieves an 80.24% mAP@ 10 for standard frame retrieval, falls below 60% for cross-view frame retrieval. In contrast, AF2 effectively bridges these two significantly different domains, resulting in high performance of over 81% mAP@ 10
8

---

original
Hame
Fame
Haro
home
Hrame
Prane
shallked regellow
reggell
Fame
Trme 2
Prame
Proree
Prame
Frame
Fraind
Frame-
Pramno
France
sequence
Pame TY/R2 5
Pamez
Porroc
Pome
Hrancc
PRAFIO
Pran
Figure 4: Visualization of (a) the original video sequence; (b) randomly shuffled sequence; and (c) a temporally reversed sequence on Break Eggs (left) and Pour Liquid (right). The randomly shuffled sequence may still preserve the action progression order due many similar frames in the video
sequence, failing to be truly "negative". In contrast, temporally reversing frames results in a more distinct and suitable negative example.
both standard and cross-view settings. These results demonstrate the efficacy AE2 learning fine-grained ego-exo view-invariant features.
Ablation Table 2 provides an ablation of the two de- sign choices of AE2. Starting with baseline approach (denoted as "Base DTW" in the table), we optimize ResNet-50 encoder with respect to DTW alignment ob- jective as defined in Eqn. (3). Next, we replace the feature encoder with our proposed object-centric encoder design and report its performance. Finally, we incorporate the contrastive regularization from Equ. (4) into the objective. is evident that object-centric representations are instru- mental in bridging the ego-exo gap, leading substantial performance improvements. Furthermore, incorporating contrastive regularization provides additional gains.
Table 2: Ablation study of AE2 on four datasets (FI score). See Supp. for other metrics.
Dataset (B)
Base DTW object
58.53 82.91 59.66
79.56
62.86 84.04 63.28
84.14
object + contrast 66.23 85.17 66.56 85.87
Negative Sampling We conduct experiments to validate the efficacy of creating negative samples by temporally reversing frames, as opposed to random shuffling and report results in Table In general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. The inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to a positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur real videos), temporally reversing frames is apt to create negative sample that deviates from the correct action progression order. To further illustrate this point, we visualize two video sequences in Fig. 5. is evident that the randomly shuffled sequence seems to preserve the sequence of actions like breaking eggs or pouring liquid, thereby resembling a "positive" example. Consequently, incorporating such negative samples into training may confuse the model and lead to diminished performance.
Table 3: Comparison of AE2 employing randomly shuffled frames versus tempo- rally reversed frames as the negative sam- ples (Frame Retrieval mAP@ 10). See Supp. for other metrics.
Dataset
(A)
(D)
Random Shuffle 61.66 81.92 63.48 86.73
Temporal Reverse 65.85 84.90 65.54 86.83
Qualitative Results Fig. 5 (left) shows examples of cross-view frame retrieval on Pour Liquid and Tennis Forehand datasets. Given a query image from one viewpoint in the test set, we retrieve its nearest neighbor among all test frames in another viewpoint. We observe that the retrieved frames closely mirror the specific action state of the query frame. For example, when the query is frame

---

Query (cxo)
Retrieved Nearest Neighbors (ezo)
Video (ogo) Vidao
Video (eno) Video 4 (cso)
Prosparr lifling the
Pro-erack: eggshell and ready broken
Active Pour: liquid exitin the conteiner
Frame embeldings betire training se
Query (ogo)
Retrieved Nearest Neighbors (exo)
racket poised
Post-crak:: eggsshell crckkdl contents relessed
following thoough
Figure 5: Qualitative results of AE2. Left: Cross-view frame retrieval results Pour Liquid and Tennis Forehand. Our view-invariant embeddings can be effectively used for fine-grained frame retrieval; Right: tSNE trajectories of 4 test videos (2 ego and exo) in the embedding space Break Eggs. Shading indicates time. The learned embeddings successfully capture the progress of an action and maintain invariance across the ego and exo viewpoints. See Supp. for more examples.
capturing the moment before the pouring action, where the liquid is still within the container (row 1), or when the query shows pouring actively happening (row 2), the retrieved exo frames capture that exact step as well. Fig. (right) shows tSNE visualizations of embeddings for 4 test videos on Break Eggs. Our learned embeddings effectively capture the progress of action, while remaining view-invariant. These learned embeddings enable to transfer the pace of one video to others depicting the same action. We demonstrate this capability with examples of both ego and exo videos, collected in distinct environments, playing synchronously in the Supp. video.
Limitations While AF2 has demonstrated ability to handle in-the-wild data, there remains much room for improvement, particularly on challenging in-the-wild datasets such as Pour Liquid. Here, we observe instances of failure in cross-view frame retrieval and synchronizing the pace of two videos. Although we highlight successful examples in discussion, it is important to note that some attempts have been unsuccessful, largely due to the inherent noise in real-world videos. This represents key area for future refinement and development of AE2.
6 Conclusion
We tackle fine-grained temporal understanding by jointly learning from umpaired egocentric and exocentric videos. Our core idea is leverage temporal alignment as the self-supervised learning objective for invariant features, together with a novel object-centric encoder and contrastive regularizer that reverses frames. On four datasets we achieve superior performance compared state-of-the-an ego-cxo and alignment approaches, This fundamental technology has great promise for both robot learning and human skill learning in AR. where an ego actor needs understand (or even match) the actions of a demonstrator that they observe from the exo point of view. Future work includes generalizing AE2 train from pooled data of multiple actions, extending its applicability more complex, longer action sequences and exploring its impact for robot imitation learning.
10

---

Acknowledgements
UT Austin is supported in part by the IFML NSF AT Institute. KG is paid as a research scientist at Meta. The authors extend heartfelt thanks to the following individuals from UT Austin for their vital role collecting the ego-tennis forehand dataset: Bowen Chen, Zhenyu Jiang, Hanwen Jiang, Xixi Hu, Ashutosh Kumar, Huihan Liu, Bo Liu, Mi Luo, Sagnik Majumder, Shuhan Tan, and Yifeng Zhu. We are also grateful to Zhengqi Gao (MIT), Xixi Hu (UT Austin) and Yue Zhao (UT Austin), for their constructive discussions and feedback.
References
[1] Shervin Ardeshir and Ali Borji. An exocentric look at egocentric actions and vice versa. Computer Vision and Image Understanding, 171:61-68, 2018.
[2] Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric vidcos. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 27, 2022, Proceedings, Part XIII, pages 657 657-675 675. Springer, 2022.
[3] Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Camphell, Cristian Rodrigucz-Opazo, Hongdong Li, and Stephen Gould. The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pages 847-859, 2021.
[4] Donald Berdtt and James Clifford. Using dynamic time warping find patterns in time series. KDD workshop, volume 10, pages 359-370, Seattle, WA, USA:, 1994.
[5] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding" In ICML, volume 2. page 4, 2021.
[6] Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Yi Chang, and Juan Carlos Niebles. Few-shot video classifica- tion via temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10618-10621 2020.
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics datasct. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[8] Chien- Yi Chang. An Huang. Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3546-3555,
2019.
[9] Chao-Yeh Chen and Kristen Grauman, Inferring unseen views of people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2003-2010, 2014.
[10] Minghao Chen, Fangyun Wei, Chong Li, and Deng Cai. Frame-wise action representations for long videos via sequence contrastive learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13801-13810 10, 2022.
[11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018
[12] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Vidco segmentations and object relations. Advances in Neural Information Processing Systems, 35:13745-1377 2022.
[13] Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. Guide to the carnegie mellon university multimodal activity (cmu- cnu-mmac) database.
[14] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[15] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson. Pierre Sermanct, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1801-1810, 2019.
[16] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
11

---

[17] Ali Farhadi and Mostafa Kamali Tabrizi. Learning to recognize activities from the wrong view point. In Computer Vision-ECCV 2008: 10th Europeun Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10. pages 154-166. Springer, 2008
[18] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202- 6211, 2019.
[19] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. 2017.
[20] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller- et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pages 5842-5850. 2017.
[21] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19012, 2022.
[22] Isma Hadji, Konstantinos G Derpanis, and Allan D Jepson. Representation learning via global temporal aligmment and cycle-consistency. In Proceedings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pages 1068-1 1077, 2021.
[23] Sanjay Haresh, Satcesh Kumar, Huseyin Coskun, Shahram N Syed, Andrey Konin, Zeeshan Zia, and Quoc-Huy Tran. Learning by aligning videos time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5548-5558, 2021.
[24] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770 778,, 2016.
[26] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In NewrIPS, 2016.
[27] Alexandros Iosifidis, Anastasios Tefas, and loannis Pitas. View-invariant action recognition based on artificial neural networks. IEEE transactions on neural networks and learning systems, 23(3):412 12-424, 2012.
[28] D. Jayaraman and K. Grauman. Slow and steady feature analysis: Higher order temporal coherence in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[29] Bruno Korbar, Du Tran, and Lorenzo Torresani. Co-training of audio and video representations from self-supervised temporal synchronization. In NeurlPS, 2018.
[30] Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activitics. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 780-787, 2014.
[31] Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision, pages 2556-2563. IEEE, 2011.
[32] Tacin Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10138 10148, 2021.
[33] Taein Kwon, Bugra Tekin, Siyu Tang. and Marc Pollefeys. Context-aware sequence alignment using 4d skeletal augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8172-8182, 2022.
[34] Mandy Lange, Dietlind Zühlke, Olaf Holz, Thomas Villmann, and Saxonia-Germann Mittweida. Appli- cations of Ip-norms and their smooth approximations for gradient based learning vector quantization. In ESANN, pages 271-276. Citeseer, 2014.
[35] Jun Li and Sinisa Todorovic, Action shuffle alternating learning for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12628- 12636, 2021.
[36] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representa- tions from third perrdn to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943-6953 2021.
12

---

[37]
Kevin Qinghong Lin. Jinpeng Wang. Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao. Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems, 35:7575-7586, 2022.
[38] Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13853-13883 2022.
[39]
Jingen Liu, Mubarak Shah, Benjamin Kuipers, and Silvio Savarese. Cross-view action recognition via view knowledge transfer. In CVPR 20 pages 3209-3216. IERE, 2011.
[40] Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 3282-322 3292, 2022.
[41] Weizhe Liu, Bugra Tekin, Huseyin Coskun, Vibhav Vineet, Pascal Fua, and Mare Pollefeys. Learning to align sequential actions in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2181-2191, 2022.
[42] YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1118-1125. IEEE, 2018.
[43] Jian Ma and Dima Damen. Hand-object interaction reasoning. In 2022 18th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1-8. IEEE, 2022.
[44] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Towards universal visual reward and representation via value-implicir pre-training. In ICLR, 2023.
[45] Behrooz Mahasseni and Sinisa Todorovic. Latent multitask learning for view-invarian action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 3128-3135, 2013.
[46] Antoine Miech, Jean-Baptiste Alayrae, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[47] Antoine Miech, Dimitri Zhukov, Jean- Baan-Bapistte Alayrac, Makarand Tapaswi, Ivan I Laptev, and Josef Sivic. Howto100m I Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
[48] Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffic and learn: unsupervised learning using temporal order verification. In ECCV, 2016.
[49] Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In NeurlPS, 2020.
[50] Tushar Nagarajan and Kristen Grauman. Shaping embodied agent behavior with activity priors from egocentric video. Advances in Neural Information Processing Systems, 34:29794-29988 2021.
[51] Andrew Owens and Alexei A Efros, Audio-visual scene analysis with self-supervised multisensory fcatures. In ECCV, 2018
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[53] AJ Piergiovanni and Michael S Ryoo. Recognizing actions in videos from unseen viewpoints. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 4124-4132, 2021.
[54] Senthil Purushwalkam, Tian Ye. Saurabh Gupta, and Abhinav Gupta. Aligning videos in space and time. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXVI 16, pages 262-278 278.. Springer, 2020.
[55] Hossein Rahmani and Ajmal Mian. Learning a non-linear knowledge transfer model for cross-view action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2458 2466, 2015.
[56] Hossein Rahmani and Ajmal Mian. 3d action recognition from novel viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Patiern Recognition, pages 1506-1515, 2016.
[57] Cen Rao and Mubarak Shah. View-invariance in action recognition. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 2, pages II-II. IEEE, 2001.
[58] Cen Rao, Alper Yilmaz, and Mubarak Shah. View-invariant representation and recognition of actions. International Journal of Computer Vision, 50(2):203, 2002.
13

---

[59] Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Recognizing fine-grained and composite activities using hand-centric features and script data. International Journal of Computer Vision, 119:346-373, 2016.
[60]
Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing, 26(1):43-49, 1978.
[61] Madeline C. Schiappa, Yogesh S. Rawat, and Muharak Shah. Self-supervised learning for videos: A survey. arXiv:2207.004113 2022.
[62] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101 101: A large-scale multi-view vidco dataset for understanding procedural activitics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21096- 21106, 2022.
[63] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Fric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference rabotics and automation (ICRA), pages 1 113-11141. IEFE, 2018.
[64] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9869-9878, 2020.
[65] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hicrarchical video dataset for fine Ene-grained action understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2616-2625, 2020.
[66] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. In proceedings of the IEEE conference on computer vision and pattern recognition, pages 7396-7404, 2018.
[67]
Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteck Alahari, Charades-ego: A large-scale datasct of paired third and first person vidcos. arXiv preprint 184402180..0966,, 2018.
[68] Gunnar A Sigurdsson, Gii Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsoureing data collection for activity understanding. In Computer 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 510-526. Springer, 2016.
2016:
[69] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[70] Bilge Soran, Ali Farhadi, and Linda Shapiro. Action recognition in the presence of one egocentric and multiple static cameras. In Computer Vision-ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part V 12, pages 178 193. Springer, 2015.
[71] Jennifer J Sun, Jiaping Zhao, Chinh Chen, Florian Schroff, Hartwig Adam, and Ting Liu. View- invariant probabilistic embedding for human pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part 16, pages 53-70. Springer. 2020.
[72] Reuben Tan, Bryan Plummer, Kate Saenko, Hailin Jin, and Bryan Russell. Look at what i'm doing: Self-supervised spatial grounding of narrations in instructional videos. Advances in Neural Information Processing Systems, 34:14476- 14487, 2021.
[73] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili 7hao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1207-1216, 2019.
[74] Du Tran, Lubomir Bourdey, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedingy of the IEEE international conference computer vision, pages 4489-4497 2015.
[75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[76] Daniel Weinland, Mustafa Özuysal, and Pascal Fua. Making action recognition robust to occlusions and viewpoint changes. In European Conference on Computer Vision, 2010.
[77] Daniel Weinland, Remi Ronfard, and Edmond Boyer. Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding (CVIU), 2006.
[78] Xinxiao Wu, Han Wang. Cuiwei Liu, and Yunde Jia. Cross-vien action recognition over heterogeneous feature spaces. In Proceedings of the IEEE International Conference on Computer Vision, pages 609 6166, 2013.
14

---

[79] Yuncong Yang. Jiawei Ma, Shiyuan Huang, Long Chen, Xudong Lin, Guangxing Han, and Shih-Fu Chang. Tempelr: Temporal alignment representation with contrastive learning. arXiv preprint arXiv:2212.13778. 2022.
[80] Huangyue Yu, Minjie Cai. Yunfei Liu, and Feng Lu. What see is what you see: Joint attention learning for first and third person video co-analysis. ACM MM, 2019.
[81] Pengfei Zhang, Cuiling Lan, Junliang Xing. Wenjun Zeng. Jianru Xue, and Nanning Zheng. View adaptive recurrent neural networks for high performance human action recognition from skeleton data. In Proceedings of the IEEE international conference computer vision, pages 21 2117-2126, 2017.
[82] Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly- supervised representation for detailed action understanding. In Proceedings of the IEEE international conference compuder vision, pages 2248-2255, 2013.
[83] Zhong Zhang, Chunheng Wang. Baihua Xiao, Wen Zhou, Shuang Liu, Cunzhao Shi. Cross-view action recognition via a continuous virtual path. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2690-2697, 2013.
[84] Honglu Zhou, Roberto Martin-Martin, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. arXiv preprint arXiv:23031182300
2023.
[85] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for bsson-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11333 2022.
[86] Dimitri Zhukov, can-Baptistt Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545, 2019.
15

---

A Video Containing Qualitative Results
We invite the reader to view the video available https:/ http::///viioont cs utexas edu/pro ctt/projects// AlignEgoExo/, where we show qualitative examples of one practical application enabled by our learned ego-exo view-invariant representations Synchronns playback of egocentric and exocentric videos. We randomly select ego-exo video pairs from the test set, and use the frozen encoder ø extract frame-wise embeddings. We then match each frame of one video (the reference) to its closest counterpart in the other video using nearest neighbor. As demonstrated across all datasets with several examples, AE2 effectively aligns two videos depicting the same action, overcoming substantial viewpoint and background differences. The video also includes examples of synchronizing two egocentric two exocentric videos.
B Experimental Setup
B.1
Datasets
Since existing video datasets for fine-grained video understanding (e.g., PennAction [82], Fine- Gym [65], IKEA ASM [3]) are solely composed of third-person videos, we curate video clips from five public datasets: Break Eggs [13], H2O [32], EPIC-Kitchens [11], HMDB51 [31] and Penn Action [82] and collect egocentric Tennis Forehand dataset. Our data selection criteria is to find videos that depict the same action from distinct egocentric and exocentric viewpoints. Consequently, TCN pouring [63] is excluded due to its small scale and significant similarity between the egocentric and exocentric views; Assembly 101
101 [62] is not selected since there there are large variations within a
single atomic action and the egocentric video is monochromatic.
In all, our dataset compilation from five public data sources, along with our collected egocentric tennis videos, results in four distinct ego-exo datasets, each describing specific atomic actions:
(A) Break Eggs. The dataset contains 44 subjects cooking five different recipes (brownies, pizza, sandwich, salad, and scrambled eggs), captured from one egocentric and four exocentric views simultaneously in CMU-MMAC [13]. We use the temporal keystep boundaries provided in [2] to extract clips corresponding to the action of breaking eggs from all videos. Among the four exocentric viewpoints, we adopt videos from the right-handed view, as this viewpoint captures the action being performed most clearly. We randomly split the data into training and validation sets across subjects, with 35 subjects (118 videos) for training and subjects (30 videos) for validation and test. There strict synchronization between egocentric and exocentric video pairs in this dataset.
(B) Pour Milk. The dataset features 10 subjects interacting with a milk carton using both hands, captured by one egocentric camera and four static exocentric cameras in H2O [32]. We utilize the keystep annotations provided in [33] to extract clips corresponding to the pouring milk action. All four exocentric views included, as they clearly capture the action. We follow the data split in [33], with 7 subjects (77 videos) in the training set and 3 subjects (35 videos) the validation and test set. A portion of this dataset (the first 3 subjects) contains video pairs, while the remaining part does not.
synchronized egocentric-exocemee
(C) Pour Liquid. To evaluate our methods on in-the-wild data, we assemble a Pour Liquid dataset by extracting clips from one egocentric dataset, EPIC- Kitchens [11] and one exocen- tric dataset, HMDB51 [3 [31]. We utilize clips from the "pour water" class in EPIC-Kitchens and "pour" category in HMDB51. Following the data split the original datasets, we obtain 137 videos for training and 56 videos for validation and test. is important to note that
Table 4: Dataset summary.
Train
Val
Test exo
Fixed exo view?
Ego-exo time-syne?
Dataset
ego
ego
ego
(A) Break Figss
61
57
5
5
10
10
(B) Pour Milk
29
4
16
/
(C) Pour Liquid
70
67
10
9
19
18
X
X
(D) Tennis Forehand
94
79
25
24
50
50
X
16

---

the egocentric and exocentric videos are neither synchronized nor collected in the same enviromment, providing a challenging testbed.
(D) Tennis Forehand. To include physical activities in our study, we leverage exocentric video sequences of the tennis forehand action from Penn Action [82] and collect an egocentric dataset featuring the same action performed by 12 subjects using Go Pro HERO8. We adopt the data split from [15] for Penn Action, and divide egocentric tennis forehand dataset by subject: for training and for validation and testing. This results in a total of 173 clips for training, and 149 clips for validation and testing. The egocentric and exocentric videos, gathered from a range of real-world scenarios, are naturally unpaired.
Table provides a summary of these four datasets. In addition, we recognize that the original datasets lack frame-wise labels and provide dense frame-level annotations to enable a comprehensive evaluation of the learned representations. See Table for complete list of all the key events we annotate and Fig. 6 for illustrative examples. We will release our collected data and labels for academic usage."
Table 5: Number of actions phases and list of key events for each dataset.
Dataset
# phases
List of key events
(A) Break Eggs
4
hit egg, visible crack the eggshell; egg contents released into bowl liquid starts exiting, pouring complete liquid starts exiting, pouring complete
(B) Pour Milk
3
(C) Pour Liquid
3
(D) Tennis Forehand
racket touches ball
B.2 Evaluation
We provide detailed description of the four downstream tasks below and their corresponding evaluation metries:
Action Phase Classification. We train an SVM classifier on top of the embeddings to predict the action phase labels for each frame and report Fl score on test data. Besides the regular setting, we investigate (1) few-shot; and (2) cross-view zero-shot settings.
(1) Few-shot. We assume that only a limited number of training vidcos have annotations and can be used for training the SVM classifier.
(2) Cross-view zero-shot. We assume that per-frame labels of training data are only available one view for training the SVM classifier, and report the test performance on the other view. We use the terms "exo2ego" to describe the case where we use exocentric data for training the SVM classifier and test its performance on egocentric data, while "ego2exo" represents the reverse case.
2. Phase Progression. We train a linear regressor on the frozen embeddings to predict the phase progression values, defined the difference in time-stamps between any given frame and each key event, normalized by the number of frames in that video [15]. Average R- square measure test data is reported. This metric evaluates how well the progress of an action is captured by the embeddings, with the maximum value being
3. Frame Retrieval. We report the mean average precision (mAP)@K K (K=5,10 10, ,15). For each query. average precision computed by determining how many frames among the retrieved K frames have the same action phase labels as the query frame, divided by K. Furthermore, evaluate ieeainvariaance, we propose the cross-view frame retrieval task (i.e., ego2exo and exo2ego frame retrieval). For each query in one view, the goal is to retrieve K frames from another view. No additional training required for this task.
4. Kendall's Tau. This metric calculated for every pair of test videos by sampling two frames in the first video and retrieving the corresponding nearest frames in the second video, then checking whether their orders are shuffled. measures how well-aligned two sequences are in time. No additional training or frame-wise labels are necessary for this evaluation.
Thiis dataset is owned by UT Austin and involves participation from Meta.
17

---

(A) Break Eggs
Sturt
Phase
Hit
Phose
Visible eneck the
Lag contents relessed
Phose
Find
(B)
Plaae
Liquid exis costaiser
Phase
Pouring Complese
Start
(C) Liquid
Liquid caiss
Plase
Pouring Compleee
Phane 2
End
Pleoe 0
Racket
Phuse
Start
touches ball
Figure 6: Example labels for all datasets. Key events are displayed in boxes below sequences, with the phase label assigned each frame between two key events.
Table 6: Hyperparameters summary. 'Lr' stands for learning rate, and 'Wd' denotes weight decay.
Optimizer
Transformer Encoder
Regularization
Datasets
Lr
Wd
Hidden Dim.
Layers
# Frames
# Pos. Frames
Ratio X
(A) Break Eggs
5e-5
le-5
256
1
32
32
1
(B) Pour Milk
lc-4
le-5
256
2
(C) Pour Liquid
5e-5
le-5
128
3
32
16
2
Tennis Forehand
5e-5
le-5
128
20
4
B.3
Implementation
For all video sequences, frames resized 224 224. During training, randomly extract 32 frames from each video to construct a video sequence. We train the models for a total number of 300 epochs with batch size of 4, using the Adam optimizer. The base encoder dbase is initialized with ResNet-50 pretrained on ImageNet, and jointly optimized with the transformer encoder Orransformr throughout the training process. The model checkpoint demonstrating the best performance on validation data is selected, and its performance on test data is reported. In terms of the encoder network, global features stack the features of any given frame and its context frames along the dimension of time, followed by convolutions for aggregating temporal information and max pooling. In all experiments, the number and stride of context frames are set as and 15, respectively. For local features, we take output of the Conel layer Ôhose and apply max pooling aggregate temporal information from the given frame and its context frame. The features are then fed input of ROI Align.
taken from the output of Cone4c layer in druse. Following [15], we
18

---

Table 7: Results of shot action phase classification (FI score) and frame retrieval (mAP@5.10,15)
Fow-shot Cls.
Frame Retrieval
Dataset Method
10%
50%
100%
mAP@S
mAP@ 10
mAP@ 15
Random Features
19.18
19.18
19.18
48,26
47.13
45.75
ImageNet Features
46.15
48.80
50.24
49,98
50.49
50.08
ActorObserverNet [66]
31.40
35.63
36.14
50.92
50.47
49.72
single-view TCN [63]
52.30
54.90
56.90
52.82
53.42
53.60
multi-view TCN [63]
56.88
59.25
59.91
59.11
58.83
58.44
(A)
multi-view TCN (unpaired) [63]
56.13
56.65
56.79
58,18
57.78
57.21
CARL[[1)
39.18
41.92
43.43
47.14
46.04
44.99
TCC [15]
57.54
59.18
59.84
59.33
58.75
57.99
GTA [22]
56.89
56.77
56.86
62.79
61.55
60.38
AE2 (ours)
63.95
64.86
66.23
66,86
65.85
64.73
Random Features
36.84
36.84
36.84
52.94
52.48
51.59
ImageNet Features
39.29
40.83
41.59
53.32
54.09
54.06
single-view TCN [63]
43.60
46.83
47.39
56.98
57.00
56.46
(B)
CARL [10]
48.73
48.78
48.79
55.59
55.01
54.23
TCC [15]
78.69
77.97
77.91
81,22
80.97
80.46
GTA [22]
79.82
80.96
81.11
80.65
80.12
79.68
AE2 (ours)
85.17
85.12
85.17
85.25
84.90
84.55
Random Features
45.26
45.26
45.26
49.69
49.83
49.18
ImageNet Features
55.53
54.43
53.13
50.52
51.49
51.89
single-view TCN [63]
54.62
55.08
54.02
48.50
48.83
49.03
(C)
CARL [10]
51.68
55.67
56.98
55.03
55.29
54.93
TCC [15]
52.37
$1.70
52.53
62.93
62.33
61.44
GTA [22]
55.91
56.87
56.92
62.83
62.79
62.12
AE2 (ours)
65.88
66.53
66.56
66.55
65.54
64.66
Random Features
31.54
30.31
30.31
69.57
66.47
64.34
ImageNet Features
65.48
68.03
69.15
78.11
76.96
75.84
single-view TCN [63]
65.78
69.19
68.87
74.05
73.76
73.10
(D) CARL [10]
58.89
59.38
59.69
72.94
69.43
67.14
TCC [15]
67.71
77.07
78.41
82.78
80.24
78.59
GTA [22]
80.31
83.04
83.63
86.59
85.20
84.33
AE2 (ours)
85.24
85.72
85.87
87.94
86.83
86.05
During evaluation, we freeze the encoder and use to extract 128-dimensional embeddings for each frame. These representations are then assessed across variety of downstream tasks (Sec. 4). Detailed hyperparameters specific each dataset are provided in Table Noteworthy adjustments include: (1) In the case of Tennis Forehand, we utilize single object proposal, as the active object is only the tennis racket (the tennis ball is too small to be detected reliably). Furthermore, given the shorter video lengths, we sample 20 frames from each video as opposed to the usual 32. (2) For datasets featuring non-monotonic actions (i.e., Pour Milk and Pour Liquid), we construct the negative sequence randomly reversing either the first the last half of the sequence, rather than the whole sequence. This is due to the cyclic nature of the pouring action present in some videos within these datasets. All experiments conducted using Py PyTcchh [52] on 2 Nvidia V100 GPUs.
C Further Results and Visualizations
Results Supplementing Table in the main paper, Tables and present comprehensive results of AE2 and baseline models on few-shot action phase classification and frame retrieval tasks. For few-shot classification, we train the SVM classifier with 10% (or 50%) of the training data, averaging results over 10 runs. AE2 demonstrates superior performance in learning fine -raiinnde,, view- new-innariiant ego-exo features when compared with ego-exo [66, 63], frame- wise contrastive learning [10], and alignment-based [15, 22] approaches.
Break Eggs, AF2 greatly outperforms the multi-view TCN [63], which utilizes perfect ego-exo synchronization supervision signal. We hypothesize that the strict supervision requirement of
19

---

Table 8: Results of cross-view frame retrieval (mAP@5,10,15),
Ego2exo Frame Retrieval
Exo2ego Frame Retrieval
Dataset
Method
mAP@5 mAP@ 10
mAP@ 15
mAP@5 mAP@ 10
mAP@@ 15
Random Features
42.51
41.74
40.51
38,08
38.19
37.10
ImageNet Features
33.32
33.09
32.78
38.99
37.80
36.71
ActorObserverNet [66]
43.57
42.70
41.56
42.00
41.29
40.48
single-view TCN [63]
31.12
32.63
33.73
34.67
34.91
35.31
multi-view TCN [63]
46.38
47.04
46.96
52.50
52.68
52.43
multi-view TCN (unpaired) [63]
55.34
$4.64
53.75
58.79
57.87
57.07
CARL [10]
37.89
37.38
36.57
40.37
39,94
39.38
TCC [15]
62.11
61.11
60.33
62.39
62.03
61.25
GTA [22]
57.11
56.25
55.10
54.47
53,93
53.22
AE2 (ours)
65.70
64.59
63.76
62.48
62.15
61.80
Random Features
51.46
50.56
48.93
52.78
51.98
50.82
ImageNet Features
25.72
27.31
28.57
41.50
43.21
43.06
single-view TCN [63]
47.00
46.48
45.42
47.94
47.20
46.59
(B)
CARL [10]
54.35
52.99
51.99
51.14
51.51
51.00
TCC [15]
75.54
75.30
75.02
80.44
80,27
80.18
GTA [22]
72.55
72.78
72.96
75.16
75.40
75.48
AE2 (ours)
78.21
78.48
78.78
83.88
83.41
83.05
Random Features
55.78
55,44
54.77
56.31
55.75
54.56
ImageNct Features
51.44
52.17
52.38
30.18
30.44
30.40
single-view TCN [63]
$3.60
55.28
55.46
29.16
31.15
31.95
(C)
CARL [10]
59.59
59.37
59.19
34.73
36.80
38.10
TCC [15]
$5.98
56.08
56.13
58.11
57.89
$7.15
GTA [22]
57.03
58.52
59,00
51.71
53.32
53.54
AE2 (ours)
66.23
65.79
65.00
57.42
57.35
57.03
Random Features
61.24
58.98
56.94
63.42
59.87
57.57
ImageNet Features
69.34
66.90
64.95
61.61
60,31
58.55
single-view TCN [63]
$4.12
55.08
55.05
56.70
56.65
55.84
CARL [10]
$2.18
54.83
55.39
65.94
63.19
60.83
TCC [15]
57.87
55.84
53.81
48.62
47.27
46.11
GTA [22]
78.93
78.00
77.01
79.95
79.14
78.52
AF2 (ours)
82.58
81.46
80.75
82.82
82.07
81.69
(A) CMUMMACC
820
(C) Pou Liqid
(D) Teris Forzlaad
500r0
labeled
Number
labeled
Number
Figure 7: Few-shot action phase classification results. AE2 achieves superior performance across wide range of labeled training videos, particularly under the most challenging conditions where less than 10 videos are labeled. Results are averaged over 50 runs, and confidence bars represent one standard deviation.
TCN might be limiting, as it can not utilize many ego-exo pairs as AE2 due to its reliance on ego-exo synchronization.
contrast, AE2 capitalizes on broader set unpaired ego-exo data,
Even when we modify multi-view TCN to consider all potential ego-exo pairs as synchronously perfect (termed as multi-view TCN (unpaired) the tables), does not outperform its regular version, indicating a lack of robustness towards non-synchronized ego-exo pairs. Consequently, it appears that multi-view TCN ill-equipped learn desired view-invariant representations from unpaired, real-world ego-exo videos.
20

---

Table 9: Ablation Study of AE2.
Classification (F1 score)
Frame Retrieval (mAP@ 10)
Phase
Kendall's
Dataset Method
regular ego2exo exo2ego
regular ego2exo
exolego
Progression
Tau
Base DTW
58.53
57.78
54.23
58,36
55,36
58,95
0.1920
0.5641
(A)
object
62.86
60.88
58.52
62.66
61.26
60.69
0.4235
0.5484
+ object contrast
66.23
57.41
71.72
65.85
64.59
62.15
0,5109
0.6316
Base DTW
82,91
81.82
81.83
81.49
74.63
80.21
0.7525
0.8199
(B)
object
84.04
84.20
84.23
83,03
81.42
81.57
0.7646
0,8886
+ object contrast
85.17
84.73
82.77
84.90
78.48
83.41
0.7634
0,9062
Base DTW
59.66
55.48
59.49
52.57
54.12
52.23
0.0553
0.0609
(C) + object
63,28
57.60
62.42
63.40
67.15
63.05
0.2231
0.1339
+ object contrast
66.56
57.15
65.60
65.54
65.79
57.35
0.1380
0.0934
Base DTW
79.56
81.38
72.54
82.65
75.36
76.74
0.4022
0.4312
(D) + object
84.14
85,36
83.32
88.22
79.07
82.61
0.5431
0.6477
object contrast
85.87
84.71
85.68
86.83
81.46
82.07
0,5060
0.6171
Few-shot Learning Besides the few-shot results in Table 7, we vary the number of labeled training videos, ranging from extremely sparse (a single labeled video) to the case where all training videos are labeled. Note that each labeled video equates to multiple labeled frames. AE2 is compared with top-performing baselines, TCC [15] and GTA [22], across all four datasets in Fig. 7. The results are averaged over 50 runs and include a +- one standard deviation error bar. As shown, AE2 excels in low-label scenarios. For instance, on Pour Milk, a single labeled video yields an action phase classification Fl score over 80%. This suggests that AE2 effectively aligns representations across all training videos, enabling a robust SVM classifier for the downstream task, even with minimal labeling.
Ablation Table 9 presents an ablation of AE2 on all four datasets, which a comprehensive version of Table 2 in the main paper. From the results, we can see that object-centric representations are instrumental in bridging the ego-exo gap, leading to substantial performance improvements. For instance, frame retrieval mAP@ 10 improves by +10.83% on Pour Liquid and +5.57% on Tennis Forehand. Furthermore, incorporating contrastive regularization provides additional performance boosts for several downstream tasks such as regular action phase classification. These results demonstrate the integral contributions of both components of AE2 to achieve optimal performance.
Negative Sampling (Temporal Reverse versus Random Shuffle) Table 10 presents compar- ison of two negative sampling strategies (random shuffling versus temporal reversing), which is a comprehensive version of Table 3 in the main paper. The results reveal that, in general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. For example, on Break Eggs data, random shuffling results in a decrease of -5.62% in F1 score and -4. 4.19% in mAP@ 10 compared with regular AE2. The inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur in real videos), temporally reversing frames is apt to create a negative sample that deviates from the correct action progression order.
To dissect the performance gains further, the large improvement observed on dataset (A) when using temporally reversed frames over randomly shuffled frames stems from its inherent ability to be more robust to the repetitive nature the breaking egg action (e.g., the camera wearer may hit the egg twice before breaking it, resulting in many similar frames in the video sequence). On the other hand, the subtler improvement on dataset (D), dealing with the tennis forehand action, is likely due to the strictly monotonic nature of this action, leaving less room for our method to outperform. However, it essential to emphasize that this does not diminish the value of our approach, our negative sampling strategy still proves preferable consistently across all four datasets.
Negative Sampling (Sequence-level versus Frame-level) We emphasize the unique advantages of proposed sequence-level negatives over the frame-level negatives used in prior works [63, 66, 10].
21

---

Temporally away but visually similar
frame-level negatives
sequencs-level negatives
Figure 8: Visualization video sequence (from CMU-MMAC) exhibiting a certain amount of periodicity, where the camera wearer hits the egg to the bowl's edge twice. In the upper figure, frame-level negatives are defined based on temporal distance, leading to the possibility of two visually similar frames (marked by red border) being misclassified as a "negative" pair. In contrast, the lower figure illustrates how sequence-level negatives provide a more robust solution to accomodate the temporal variations within video.
Table 10: Comparison of AE2 employing randomly shuffled frames versus temporally reversed frames the negative samples.
Classification (FI score)
Frame Retrieval (mAP@ 10)
Phase
Dataset
Method
regular
ego2exo
exo2ego
regular
ego2exo
exo2ego
Progression
Random Shuffle
60.61
58.63
54.46
61.66
57.57
59.61
0.3385
(A)
Temporal Reverse
66.23
57.41
71.72
65.85
64.59
62.15
0.5109
Random Shuffle
82.05
82,74
79.48
81,92
80.60
79.71
0.6659
(B)
Temporal Reverse
85.17
84.73
82.77
84.90
78.48
83,41
0.7634
(C) Random Shuffle
68.13
64.96
58.44
63.48
62.73
60.27
-0.0121
Temporal Reverse
66.56
57.15
65.60
65.54
65.79
57.35
0.1380
(D) Random Shuffle
84.12
84.83
82.59
86.73
80.49
83.75
0.5304
Temporal Reverse
85.87
84.71
85.68
86.83
81.46
82.07
0.5060
While periodic actions present great challenges for all negative sampling techniques, sequence- level negatives offers more robust solution to tackle the intricacies of repetitiveness within an action. As motivating example, Fig. 8 depicts video sequence of breaking eggs with certain amount of periodicity (where the camera wearer hits the egg to the bowl's 's edge twice). Frame-level sampling approaches treat temporally close frames as positive pairs and temporally distant ones as negative pairs. As consequence, the two visually similar frames (marked by red border) would be incorrectly identified negative pair by those existing methods. Contrarily, our method of constructing negative samples adopts global perspective, addressing temporal fluctuations within video more effectively. By reversing the entire sequence, we create a challenge aligning the reversed ego view video with the original exo view video, thus forming valid negative sample.
This methodology is not only conceptually appealing but also empirically validated. Table in the main paper illustrates the effectiveness of our proposed AE2, demonstrating that it outperforms prior techniques utilizing frameewiss negatives [63, 66, 10]. These results offer compelling empirical evidence that our sequence-level negative sampling strategy successfully mitigates the issues related periodicity and improves performance, affirming its superiority over conventional frame-by-fin sampling methods.
Visualizations addition to the cress-view frame retrieval results for Pour Liquid and Tennis Forehand presented in the main paper (Fig. 5), we showcase results for the other two datasets (i.e., Break Eggs and Pour Milk) in Fig. 9. For any given query frame from one view, the retrieved nearest neighbors closely match the action stage of the query, regardless of substantial differences in viewpoints. These results underline AE2's efficacy learning fine- eneingrieed action representations that transcend ego-exo viewpoint differences.
22

---

Query (ego)
Retrieved Nearest Neighbors (exo)
Pre-pour: milk carton distance from the cup. upright
Imminent pour: milk carton closer the cup, tilted
Active pour: milk carton touching the cup, nearly vertical
Query (exo)
Retrieved Nearest Neighbors (ego)
Pre-orack: egg tapped on bowl's edge
Mid-crack; preparing open, egg remuins whole
Post-cruck: eggsshell cracked, contents released
Figure 9: Cross-view frame retrieval results Pour Milk (rows 1-3) and Break Eggs (rows 4-6). AE2 leads to representations that encapsulate the fine-grained state of an action and are invariant to the ego-exo viewpoints.
23


Reviews:
[{"review_id": "6AfjqUukYm", "reviewer": "Reviewer_YpGy", "summary": "The manuscript proposes a self-supervised learning approach for view-invariant representation learning from egocentric and exocentric videos. Unlike prior approaches, the proposed method is applicable to videos that are not aligned and captured in different environments. The key idea is to learn the alignment between the frames from two views using contrastive learning by using the temporally reversed version of the video as the negative sample. The manuscript also develops an object-centric encoder that focuses on the hands and active object regions in the frames. This allows the model to ignore the less informative regions such as background. The manuscript also develops a benchmark for evaluating the effectiveness of view-invariant representation approaches. Comparison of the proposed approach on the new benchmark with prior approaches clearly reveal its effectiveness.", "questions": "Please see the weaknesses section for the main queries.\n\nThe learnable spatial embedding $\\textbf{e}_{spatial}$ is mentioned as the embedding to encode the bounding box coordinates and the confidence score obtained from the hand-object detector. It is not clear how this is obtained for the global image feature $\\phi _{project_g}(g_i)$.\n\nSome implementation details such as how the base encoder is initialized and whether it is frozen/finetuned during training are missing.\n", "limitations": "The manuscript has adequately addressed the limitations.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript addresses a problem that is not widely studied. Unlike prior view-invariant learning approaches that sets hard constraints such as availability of time-synchronized videos or videos captured in the same environment, the proposed approach can be successfully applied to unpaired videos captured at different environments. \n\nThe benchmark developed could also be useful to the research community.\n\nThe manuscript is written in a clear way explaining the various motivations behind the key ideas and the methodology used. \n\nThe manuscript also presents an extensive set of experiments to validate its effectiveness by comparing against existing approaches. Moreover, most of the design choices adopted in the proposed approach are also validated with empirical analysis.\n", "weaknesses": "One of the contributions of the manuscript is the use of time reversed frames as the negative sample for contrastive learning instead of random shuffling. However, there is no experimental validation of the proposed negative sampling strategy against random shuffling.\n\nContinuing from the above, the proposed negative sampling strategy could affect actions that are constituted of repetitive subactions such as cutting something, playing drums, etc. where the time reversed frames could look more or less similar to its original version. The authors mention in the supplementary that for videos containing non monotonic actions (pouring water) only the first/last half of the frames are reversed. However, it is not clear even such an approach would work on repetitive actions. It seems the proposed negative strategy will work only on a selected set of actions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer YpGy for the helpful comments and for providing thoughtful feedback on our work.  \n\n***\n\n**1. Comparing temporally reversing frames with randomly shuffling frames as the negative sample**\n\n> no experimental validation of the proposed negative sampling strategy against random shuffling\n\nPlease refer to our general response.\n\n***\n\n**2. Negative sampling strategy**\n\n> the proposed negative sampling strategy could affect actions that are constituted of repetitive subactions such as cutting something, playing drums, etc. where the time reversed frames could look more or less similar to its original version\n\nWe agree that periodic actions present specific challenges, as we discussed in Ln 208-211.  However, we would like to emphasize that our negative sampling strategy offers a more robust solution to tackle the intricacies of repetitiveness within an action, when compared with frame-by-frame sampling approaches adopted in prior works [10, 61, 64].  As noted in Ln 223-228, instead of creating negatives at the frame level, we opt for a more holistic, global perspective to construct negative samples, yielding more robust negatives and stronger representations. \n\n**Qualitative Insights.** Figure R2 (available in the rebuttal PDF) depicts a video sequence of breaking eggs with a certain amount of periodicity (where the camera wearer hits the egg to the bowl’s edge twice). Frame-by-frame sampling approaches [10, 61, 64] treat temporally close frames as positive pairs and temporally distant ones as negative pairs. As a consequence, the two visually similar frames (marked by a red border) would be incorrectly identified as a negative pair by those existing methods. Contrarily, our method of constructing negative samples adopts a global perspective, addressing temporal fluctuations within a video more effectively. By reversing the entire sequence, we create a challenge in aligning the reversed ego view video with the original exo view video, thus forming a valid negative sample.\n\n**Quantitative Evidence.** Beyond this illustrative example, Table 1 in the main paper demonstrates the efficacy of our proposed AE2. It showcases the leading performance of AE2 compared to prior techniques that utilize frame-wise negatives [10, 61, 64]. These results provide strong empirical evidence for the superiority of our proposed negative sampling strategy.\n\n**Clarification on Training Setting.** It’s pivotal to understand the context in which AE2 operates. Consistent with current fine-grained action feature learning approaches [15, 22, 23, 41, 53],  AE2 focuses on videos portraying a single action instance (e.g. one instance of breaking an egg) rather than videos involving multiple takes of the same action (e.g., breaking multiple eggs), as highlighted in Ln 57 and Ln 136-138. Within this context, our negative sampling technique excels in generating valid negative samples and proves its mettle against the challenge of repetitive and background frames. Finally, it remains an open question in the community to generalize existing techniques to longer videos featuring multiple occurrences of the same action. Addressing this challenge remains on our horizon for future work, as indicated in Ln 336-338.\n\n***\n\n**3. Implementation details: spatial embedding and base encoder initialization**\n\nWe appreciate your attention to the details. Concerning the learnable spatial embedding $e_{spatial}$, it is designed for local image features and not incorporated in global feature computation. We will rectify the equation in Ln 187 of our manuscript to exclude $e_{spatial}$. \n\nRegarding the base encoder, it is initialized with a ResNet-50 pretrained on ImageNet, and is jointly optimized along with the transformer encoder throughout the training process. In addition, the architecture of our proposed encoder is detailed in Ln 240-328 within models/embedder.py in the submitted Appendix zip. \n\nWe will include these details in our updated manuscript, and pledge to open-source our code for complete reproducibility. "}, {"review_id": "GdqXaD6ju9", "reviewer": "Reviewer_CYjc", "summary": "The authors proposed a novel self-supervised embedding approach AE2 to learn fine-grained view-invariant frame-wise video features from unpaired egocentric and exocentric videos. AE2 consists of two core designs: (1) an object-centric encoder that explicitly focuses on \nobjects; (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. Four datasets exhibit improvement in experiments, demonstrating a feasible framework for learning fine-grained view-invariant features.", "questions": "See Weaknesses", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a well-motivation idea and sound arguments in general. It makes sense to learn view-invariant video features form unpaired data.\n2. Ablation studies are conducted to evaluate the effectiveness of each design.\n3. The visualization of learned view-invariant video features for test videos is very impressive, which demonstrates effectively capture the progress of an action while remaining view-invariant.", "weaknesses": "1. In Lines 225-226 “ In creating negative samples, we opt for reversing the frames in S rather than randomly shuffling them.” Although the authors explain the superiority of the reversing frames, authors need to provide further experiments comparing reversing frames with randomly shuffling them.\n2. Baseline's experimental results are so good that the baseline achieves SOTA results on most of the datasets. why is the baseline's experimental performance so good?\n3. Lack of analysis of experimental results in the ablation experiment section.\n4. In Training and Implementation Details, “the object-centre encoder network is optimized to minimize the loss above for 240 all pairs of video sequences in the training set, including ego-ego, exo-exo, and ego-exo pairs.”  The training process introduces data from the same viewpoint and paired data, but the network structure is designed specifically for unpaired data from different viewpoints. Can separate experiments be performed only on unpaired data from different viewpoints to demonstrate the validity of the proposed method?\n5. The language should be further improved by correcting some language errors, such as “ See Supp. for full implementation deatils ...” (Lines 232 in the paper).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer CYjc for the helpful comments and for providing thoughtful feedback on our work. \n\n***\n\n**1. Comparing temporally reversing frames with randomly shuffling frames**\n\nPlease refer to our general response.\n\n***\n\n**2. Baseline’s experimental results**\n\n> Baseline's experimental results are so good that the baseline achieves SOTA results on most of the datasets. \n\nWe appreciate your observation regarding the Base DTW row in Table 2. Base DTW is indeed a strong baseline, as it adopts dynamic time warping (DTW) loss [4] to align video pairs. Specifically, we implemented a smooth differentiable version of the DTW loss, positioning Base DTW as a variant of the SOTA baseline GTA [22]. This strong design foundation and the well-established efficacy of DTW in video self-supervised learning [6, 8, 22, 23, 77] contribute to Base DTW's robust performance.\n\nFurthermore, when considering the performance of Base DTW, it is useful to examine both Table 2 in the main paper and Table 8 in the Appendix — the latter being a complete version of Table 2. While Base DTW ranks strongly on the action phase classification task in Table 2 (i.e., 5th on dataset (A), 2nd on datasets (B) and (C), and 3rd on dataset (D)), its performance is more moderate when observed across a broader range of evaluations in Table 8. For instance, it ranks 4th on (A), 3rd on (B), 7th on (C), and 3rd on (D) in ego2exo frame retrieval. \n\nWhile acknowledging the strength of Base DTW, it's essential to highlight that our proposed AE2 consistently outperforms it. In both Table 2 & 8, AE2 secures the top position across various evaluations. \n\nWe hope the discussion above clarifies why Base DTW exhibits good performance in specific instances and provides a balanced view of its overall capabilities.\n\n***\n\n**3. Lack of analysis of experimental results in ablation**\n\nDue to space constraints, we offer a brief discussion of our ablation study in the main paper (Ln 301-311) and a more comprehensive analysis in the Appendix (Ln 727-733). Our findings highlight that the object-centric encoder greatly boosts downstream task performance. This boost can be attributed to the encoder's emphasis on shared information between ego and exo views (i.e., hand-object interaction regions), which effectively bridges the ego-exo domain gap. Furthermore, contrastive regularization amplifies the quality of the learned representations by creating valid negative sample pairs, leading to additional improvements.\n\nIf there are particular aspects or questions that Reviewer CYjc would like us to elaborate on, we would be more than happy to address them.\n\n***\n\n**4. Experiments on unpaired data from different viewpoints**\n\n> The training process introduces data from the same viewpoint and paired data, but the network structure is designed specifically for unpaired data from different viewpoints. \n\nIn general, unpaired and different-viewpoint video pairs pose more complex constraints than their counterparts (i.e., paired, same-view) data. With this understanding, we designed AE2 to be versatile enough to accommodate all scenarios (i.e., paired same-view, unpaired same-view, paired different-view, and unpaired different-view). As we detail below, AE2 demonstrates consistently superior performance across these cases, outperforming baseline approaches. \n\n**Paired & Unpaired Data.** AE2 is designed to encompass a diverse set of training scenarios. While the first two datasets (CMU-MMAC and H2O) include a proportion of paired (ego, exo) videos, the latter two (Pour Liquid and Tennis Forehand) contain naturally unpaired videos. Note that even for the first two datasets, we never utilize the ego-exo synchronization as an explicit supervision signal in the training phase (unlike baselines [61, 64], which *do* directly exploit the pairing information). Rather, we treat all pairs as if they are unpaired (Ln 256). As shown in Table 1 of our manuscript, we observe superior performance of AE2 across all four datasets, indicating its generalizability on both scenarios. \n\n**Same-Viewpoint & Different-Viewpoint.** Including both same-viewpoint and different-viewpoint video pairs in our training data is a deliberate choice to foster generalizability. Considering $n_1$ ego videos and $n_2$ exo videos, different-viewpoint pairs give $n_1 \\times n_2$ data, while incorporating all pairs grows this to $(n_1 + n_2) \\times (n_1 + n_2)$ training data. This expansion ensures the model learns meaningful representations that can temporally align both types of videos, enhancing its flexibility. As shown in our supplementary video (Ln 320-322), AE2 succeeds in aligning not only ego-exo (different-viewpoint) videos but also unpaired ego or exo (same-viewpoint) videos; this lends support for the effectiveness of including same-viewpoints training pairs in AE2.  \n\n**Separate Experiment on Unpaired Data from Different Viewpoints.** In line with Reviewer CYjc’s suggestion, we conducted focused experiments on the (unpaired, different-viewpoint) setting — using only different-viewpoint data pairs for training on the two naturally unpaired datasets, Pour Liquid and Tennis Forehand. The results reveal that when applied to cross-viewpoint pairs, AE2 consistently outpaces the strongest baseline GTA [22]. Specifically, for Pour Liquid, the comparison between GTA (cross-viewpoint pairs) and AE2 (cross-viewpoint pairs) yielded F1 scores of 63.24 vs 63.80 and mAP@10 scores of 59.17 vs 62.73. Similarly, for Tennis Forehand, the F1 scores were 84.23 vs 84.48 and mAP@10 scores were 86.38 vs 87.17. These results further validate AE2’s ability to adeptly handle the challenges posed by unpaired data from different viewpoints. \n\nWe thank Reviewer CYjc for this insightful question, and will include the discussion and additional experiments in the manuscript. \n\n***\n\n**5. Language error “deatils’’**\n\nWe will thoroughly review and correct this typo and any other language errors in our manuscript. \n\n"}, {"review_id": "opifCbsk1y", "reviewer": "Reviewer_ghSs", "summary": "The paper introduces a novel approach for learning view-invariant features from exo and ego views using temporal alignment. The key idea is to utilize unpaired ego-exo data for the pretext task of temporal alignment in a self-supervised setting. Furthermore, an object-centric encoder was used to focus on hand and object regions. The authors also introduce a regularization method to maximize the margin between positive pair and their corresponding negative pairs. The negative pair is constructed by replacing the one video of the pair with its reversed.\n", "questions": "See weakness.", "limitations": "Yes, the limitation is addressed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow\n- The paper introduces a novel framework for view-invariant representation learning\n- The proposed approach is capable of utilizing the unpaired data where the videos of a pair do not have to be synchronized or be in the same environment \n- The method is compared with a reasonable set of baseline methods\n \n", "weaknesses": "  - The idea of sequence alignment as self-supervised video representation learning has been studied by e.g. [22, 23]. It is not clear why the methods of [22, 23] cannot be applied to ego-exo data.\n- In Line205-211 it is argued that frame-by-frame sampling in contrastive loss can lead to incorrect negative samples due to the periodicity of some videos, however, it is not clear how the proposed approach of using reversed videos for the negative sample does not suffer from periodicity.     \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ghSs for the helpful comments and for providing thoughtful feedback on our work. \n\n***\n\n**1. Applicability of existing sequence alignment approaches**\n\n>  The idea of sequence alignment as self-supervised video representation learning has been studied by e.g. [22, 23]. It is not clear why the methods of [22, 23] cannot be applied to ego-exo data.\n\nWe would like to clarify that these prior approaches *can* be directly applied to ego-exo data. Specifically, we have provided these comparisons in Table 1 of our manuscript, where we compared a total of 8 baselines (including two sequence alignment approaches: TCC [15] and GTA [22]) with AE2. Note that LAV [23] shares similarities with GTA in terms of the dynamic time warping (DTW) alignment loss, albeit with a different regularization term. Since LAV [23] does not have publicly released code, we implemented GTA [22] using their officially released code in our experiments. \n\nFrom Table 1, we observe the inferior performance of these baselines compared with AE2, since these sequence alignment approaches are developed for third-person videos only and do not account for the ego-exo viewpoint discrepancy. Contrarily, our proposed AE2 effectively addresses the unique challenges in the ego-exo domain, outperforming all baselines across a wide range of downstream tasks. These results substantiate our claim that AE2 is superior in learning fine-grained ego-exo view-invariant features. \n\n***\n\n**2. Sequence-level negatives vs. frame-level negatives**\n\n> it is not clear how the proposed approach of using reversed videos for the negative sample does not suffer from periodicity.\n\n\nWe thank Reviewer ghSs for this insightful question, and clarify below how our proposed sequence-level negatives (using reversed videos) can better address the periodicity within a video sequence. \n\nAs a motivating example, Figure R2 (available in the rebuttal PDF) depicts a video sequence of breaking eggs with a certain amount of periodicity (where the camera wearer hits the egg to the bowl’s edge twice). Conventional frame-by-frame sampling approaches [10, 61, 64] treat temporally close frames as positive pairs and temporally distant ones as negative pairs. As a consequence, the two visually similar frames (marked by a red border) would be incorrectly identified as a negative pair by those existing methods. Contrarily, our method of constructing negative samples adopts a global perspective, addressing temporal fluctuations within a video more effectively. By reversing the entire sequence, we create a challenge in aligning the reversed ego view video with the original exo view video, thus forming a valid negative sample.\n\nThis methodology is not only conceptually appealing but also empirically validated. Table 1 in the main paper illustrates the effectiveness of our proposed AE2, demonstrating that it outperforms prior techniques utilizing frame-wise negatives [10, 61, 64]. These results offer compelling empirical evidence that our sequence-level negative sampling strategy successfully mitigates the issues related to periodicity and improves performance, affirming its superiority over conventional frame-by-frame sampling methods.\n"}, {"review_id": "58ShrDafcv", "reviewer": "Reviewer_G2AN", "summary": "This paper proposes a self-supervised approach for learning view-invariant action features by aligning the egocentric and exocentric videos in time. The proposed method has two main components: 1)  an object-centric encoder to learn discriminative features around the hands and active objects; 2) a contrastive learning paradigm to align videos from different viewpoints. \n\nApart from the proposed method, this paper also establishes a new benchmark to evaluate the proposed method, and extensive experiments have been conducted to validate the effectiveness of the proposed method.", "questions": "As the authors generate the benchmark datasets on their own, please elaborate more on the details, such as the number of action classes in each dataset.", "limitations": "Please see the weakness.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated and well-written. Learning view-invariant features for fine-grained action recognition is important in video understanding. \n\n2. Experiments are conducted on four different downstream tasks to show the effectiveness of the proposed method.", "weaknesses": "1. The proposed method has limited novelty, and it is a combination of different existing modules. For example, emphasizing the hand-region features and contrastive learning with inverse temporal orders are widely used in many existing works.\n\n2. The proposed benchmark contains four datasets, but each dataset has a very limited number of samples, conversing a few action classes.\n(a) Limited samples: e.g., the CMU-MMAC dataset contains 118 videos for training and 30 videos for validation and testing. This makes the model training easy to overfit. \n(b) Limited diversity: e.g., the CMU-MMAC dataset contains only a few actions related to 'breaking eggs', which casts doubt on the generalizability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer G2AN for the helpful comments and for providing thoughtful feedback on our work. \n\n***\n\n**1. Limited novelty**\n\n> The proposed method has limited novelty, and it is a combination of different existing modules. For example, emphasizing the hand-region features and contrastive learning with inverse temporal orders are widely used in many existing works.\n\nUnfortunately the reviewer does not indicate which specific existing work he/she is concerned about.  We would be glad to elaborate with those reference(s) in mind if the reviewer can provide references.\n\nIn our related works section, we thoroughly discuss existing techniques and indicate their differences with AE2 (video contrastive learning approaches with inverse temporal orders in Ln 76-85 and works that emphasize the hand-region features in Ln 101-108).\n\nWe respectfully disagree with the comment of limited novelty and provide clarifications below. As recognized by other reviewers, in this work, we address a less studied research problem (Reviewer YpGy), and propose a novel self-supervised embedding approach to learn fine-grained view-invariant features (Reviewer CYjc and ghSs).\n\nOur innovation arises from two main aspects:\n\n**Problem Space.** We focus on learning ego-exo view-invariant features from in-the-wild videos that are not captured simultaneously, or in the same environment — a problem that has been unexplored thus far.\n\n**Technical Contributions.** In response to the unique challenges presented by the ego-exo domain, we have developed two techniques: (1) Recognizing the substantial differences between ego and exo views, we devise an object-centric encoder design that emphasizes hand-object interaction regions, effectively bridging the ego-exo domain gap. (2) To address the complexities in aligning different-viewpoint video pairs and the potential temporal variations within a video, we introduce a global perspective for creating negative samples. This contrasts with the frame-by-frame sampling methods used in previous approaches. Both techniques are specifically tailored to our new problem space and have not been previously discussed in the context of fine-grained action feature learning [15, 22, 23, 41, 53, 61, 64].\n\nIf Reviewer G2AN knows of specific references that we might have overlooked in our related work, we would gladly examine them and discuss how they differentiate from our research.\n\n***\n\n**2. Issues of the proposed benchmark**\n\n> (a) Limited samples…CMU-MMAC…This makes the model training easy to overfit. (b) Limited diversity... which casts doubt on the generalizability of the proposed method.\n\n> please elaborate more on the details, such as the number of action classes in each dataset.\n\nWe recognize the concerns regarding the dataset scale and diversity in our proposed benchmark. Since the field of ego-exo fine-grained feature learning is new, the development of our benchmark was both an essential and challenging step. Here's how we've approached these issues:\n\n**Comparative Scale and Avoiding Overfitting.** We carefully designed our benchmark to align with the scale of datasets widely adopted by the fine-grained action feature learning community [15, 22, 23, 33, 41, 53], such as:\n+ Pouring [63] (70 for training, 14 for validation), 1 action class \n+ Penn Action [82] (40-134 for training, 42-116 for validation), 13 action classes\n+ IKEA ASM [3] (61 for training, 29 for validation), 1 action class\n+ H2O ego-only set [32] (27 for training, 11 for validation), 1 action class\n\nMoreover, unlike these examples that split data only into training and validation, we included a separate test set (Ln 684-685 in the Appendix) to provide a more robust evaluation and mitigate the risk of overfitting. We believe this approach contributes to a more fair and comprehensive assessment within the field.\n\n**Dataset Clarifications and Method Generalizability.** As described in Ln 243-259 in the main paper and Section A.2.1 in Appendix, our benchmark includes four action-specific datasets (not only CMU-MMAC), each linked to one action category (Ln 247 and Ln 608-637), encompassing videos from both controlled and in-the-wild environments. Importantly, the action-specific focus of our benchmark aligns with existing fine-grained action feature learning literature [15, 22, 23, 33, 41, 53]. The emphasis here is on understanding and reasoning about the temporal evolution of one specific action rather than classifying the action category (Ln 18-24). In addition, we would like to emphasize the diversity of these four datasets in our benchmark, and AE2’s superior performance across them. As shown in Table 1, AE2 consistently outperforms all baseline approaches on all datasets; these results offer strong support for AE2’s generalizability. \n\n**Cost constraints.** While our approach is entirely self-supervised and does not require frame-wise labels during training, *evaluating* the learned representations on downstream tasks necessitates per-frame labels for every video in the dataset. We incurred significant expense to create the new frame-labeled ground truth sets for evaluation by us and others down the road. \n \nIn conclusion, we believe our benchmark provides a valuable starting point for the emerging field of ego-exo fine-grained feature learning. We appreciate Reviewer G2AN’s suggestions, and will strive to enhance its scale and diversity in future iterations. \n"}]

Author rebuttal:
We thank all reviewers for their thoughtful and constructive review of our manuscript. 

Three of the four reviewers recommend accepting.  We were encouraged to hear that the reviewers found our work novel (CYjc, ghSs), sound (Cyjc), well-motivated (CYjc, G2AN), backed up with extensive experiments (YpGy, G2AN), detailed ablation studies (YpGy, CYjc), and compelling visualizations (CYjc); and that they view our proposed ego-exo benchmark to be new (G2AN) and useful to the community (YpGy). 

In response to the feedback received, we provide a general response here to address a query shared by two reviewers, and individual responses below to specific points from each reviewer.  Please refer to the attached rebuttal PDF, where supplementary figures and tables have been provided to substantiate our responses.
***
**1. Comparing temporally reversing frames with randomly shuffling frames as the negative sample**

We thank Reviewer YpGy and CYjc for raising this point. In compliance with their suggestions, we conducted additional experiments to validate the efficacy of creating negative samples by temporally reversing frames, as opposed to random shuffling. Table R1 (available in the rebuttal PDF file) compares these two frame sampling approaches across all four datasets. 

The results in Table R1 reveal that, in general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. For example, on (A) CMU-MMAC data, random shuffling results in a decrease of -5.62% in F1 score and -4.19% in mAP@10 compared with regular AE2. 


As explicated in Ln 223-228 of our manuscript, the inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to a positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur in real videos), temporally reversing frames is apt to create a negative sample that deviates from the correct action progression order.
 
To further illustrate this point, we visualize two video sequences in Figure R1 (available in the rebuttal PDF file). From this figure, it is evident that the randomly shuffled sequence seems to preserve the sequence of actions like breaking eggs or pouring liquid, thereby resembling a “positive” example. Consequently, incorporating such negative samples into training may confuse the model and lead to the observed diminished performance.

Going back to our results in Table R1, we can dissect the performance gains further. The large improvement observed on dataset (A) when using temporally reversed frames over randomly shuffled frames stems from its inherent ability to be more robust to the repetitive nature of the breaking egg action (e.g., the camera wearer may hit the egg twice before breaking it, resulting in many similar frames in the video sequence). On the other hand, the subtler improvement on dataset (D), dealing with the tennis forehand action, is likely due to the strictly monotonic nature of this action, leaving less room for our method to outperform. However, it is essential to emphasize that this does not diminish the value of our approach, our negative sampling strategy still proves preferable consistently across all four datasets. 

We will include the new baseline, the corresponding figure and a detailed discussion regarding these two frame sampling strategies in the manuscript. 
***
We would again like to thank all reviewers for their time and feedback, and we hope that our responses adequately address all concerns. Any further questions are highly welcomed. 


Meta-review: All reviewers were in agreement, recommending acceptance. The AC reviewed the paper, reviews, and rebuttals, and agrees with the consensus. The paper presents a novel approach to view-invariant representation learning, addressing a unique problem not widely studied before - unlike previous methods that rely on stringent data constraints, this approach is adaptable to unpaired videos from diverse environments. The experimental analysis provides robust validation of the method's efficacy. The introduced benchmark can also be a significant contribution to the community. Important concerns from reviewers were resolved during the rebuttal period. Please, ensure that the new figures, baselines, and other clarifications provided in the rebuttal are added to the main manuscript.
Decision: Accept (poster)