Title: Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment
Venue: NeurIPS 2023
Accepted: yes
Keywords: fine-grained video understanding

egocentric video

self-supervised learning

temporal alignment
Abstract: The egocentric and exocentric viewpoints of a human activity look dramatically different, yet invariant representations to link them are essential for many potential applications in robotics and augmented reality.  Prior work is limited to learning view-invariant features from paired synchronized viewpoints.  We relax that strong data assumption and propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time, even when not captured simultaneously or in the same environment. To this end, we propose AE2, a self-supervised embedding approach with two key designs: (1) an object-centric encoder that explicitly focuses on regions corresponding to hands and active objects; (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context, comprising four datasets---including an ego tennis forehand dataset we collected, along with dense per-frame labels we annotated for each dataset. On the four datasets, our AE2 method strongly outperforms prior work in a variety of fine-grained downstream tasks, both in regular and cross-view settings.

Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment

Abstract

The egocentric and exocentric viewpoints of a human activity look dramatically different, yet invariant representations to link them are essential for many potential applications in robotics and augmented reality. Prior work is limited to learning view-invariant features from paired synchronized viewpoints. We relax that strong data assumption and propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time, even when not captured simultaneously or in the same environment. To this end, we propose AE2, a self-supervised embedding approach with two key designs: (1) an object-centric encoder that explicitly focuses on regions corresponding to hands and active objects; and (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context, comprising four datasets-including an ego tennis forehand dataset we collected, along with dense per-frame labels we annotated for each dataset. On the four datasets, our AE2 method strongly outperforms prior work in a variety of fine-grained downstream tasks, both in regular and cross-view settings. 1

1 Introduction

Fine-grained video understanding aims to extract the different stages of an activity and reason about their temporal evolution. For example, whereas a coarse-grained action recognition task [31, 16, 68, 20, 7] might ask, 'is this sewing or snowboarding or...?', a fine-grained action recognition task requires detecting each of the component steps, e.g. , 'the person threads the needle here, they poke the needle through the fabric here...' Such fine-grained understanding of video is important in numerous applications that require step-by-step understanding, ranging from robot imitation learning [42, 63] to skill learning from instructional 'how-to' videos [86, 73, 47, 46].

The problem is challenging on two fronts. First, the degree of detail and precision eludes existing learned video representations, which are typically trained to produce global, clip-level descriptors of the activity [69, 74, 18, 5]. Second, in many scenarios of interest, the camera viewpoint and background will vary substantially, making the same activity look dramatically different. For example, imagine a user wearing AR/VR glasses who wants to compare the egocentric recording of themselves doing a tennis forehand with a third-person (exocentric) expert demonstration video on YouTube, or a robot tasked with pouring drinks from its egocentric perspective but provided with multiple third-person human pouring videos for learning. Existing view-invariant feature learning [76, 27, 78, 45, 57, 58]-including methods targeting ego-exo views [63, 66]-assume that training data contains synchronized video pairs capturing the same action simultaneously in the same environment . This is a strong assumption. It implies significant costs and physical constraints that can render these methods inapplicable in complex real-world scenarios.

ego video

embedding space

exo video

Our goal is to learn fine-grained frame-wise video features that are invariant to both the ego and exo views, 2 from unpaired data. In the unpaired setting, we know which human activity occurs in any given training sequence ( e.g. , pouring, breaking eggs), but they need not be collected simultaneously or in the same environment. The main idea of our self-supervised approach is to infer the temporal alignment between ego and exo training video sequences, then use that alignment to learn a viewinvariant frame-wise embedding. Leveraging unpaired data means our method is more flexible-both in terms of curating training data, and in terms of using available training data more thoroughly, since with our approach any ego-exo pairing for an action becomes a training pair. Unlike existing temporal alignment for third-person videos [15, 54, 22, 23, 41], where canonical views are inherently similar, ego-exo views are so distinct that finding temporal visual correspondences is non-trivial (see Fig. 1).

To address this challenge, we propose two main ideas for aligning ego and exo videos in time. First, motivated by the fact that human activity in the first-person view usually revolves around hand-object interactions, we design an object-centric encoder to bridge the gap between ego-exo viewpoints. The encoder integrates regional features corresponding to hands and active objects along with global image features, resulting in more object-centric representations. Second, we propose a contrastive-based alignment objective, where we temporally reverse frames to construct negative samples. The rationale is that aligning an ego-exo video pair should be easier ( i.e. , incur a lower alignment cost) compared to aligning the same video pair when one is played in reverse. We bring both ideas together in a single embedding learning approach we call AE2, for 'align ego-exo'.

To evaluate the learned representations, we establish an ego-exo benchmark for fine-grained video understanding. Specifically, we assemble four datasets of atomic human actions, comprising ego and exo videos drawn from five public datasets and an ego tennis forehand dataset that we collected. In addition, we annotate these datasets with dense per-frame labels (for evaluation only). We hope that the enriched data and labels that we release publicly can support progress in fine-grained temporal understanding from the ego-exo perspective. This is a valuable dataset contribution alongside our technical approach contribution. Furthermore, we present a range of practical downstream tasks that demand frame-wise ego-exo view-invariant features: action phase classification and retrieval across views, progress tracking, and synchronous playback of ego and exo videos. Experimental results demonstrate considerable performance gains of AE2 across all tasks and datasets.

2 Related Work

Fine-grained Action Understanding To recognize fine-grained actions, supervised learning approaches [82, 30, 59, 65] employ fine-grained action datasets annotated with sub-action boundaries. In contrast, our self-supervised approach requires much lighter annotations to pretrain for

action phase classification, namely, video-level action labels. Keystep recognition in instructional videos [86, 73, 47, 46, 2] is a related form of fine-grained action understanding, where the goal is to name the individual steps of some longer activity ( e.g. , making brownies requires breaking eggs, adding flour, etc.). In that domain, multi-modal representation learning is a powerful tool that benefits from the spoken narrations that accompany how-to videos [47, 72, 38, 84]. In any of the above, handling extreme viewpoint variation is not an explicit target.

Self-supervised Video Representation Learning Various objectives for self-supervised video features have been explored [61]. This includes using accompanying modalities like audio [51, 29, 49] and text or speech [47, 72, 38, 37, 84] as training signals, as well as temporal coherence priors [28, 26, 63, 44, 10] or self-supervised tasks that artificially alter the temporal order of videos [48, 19, 35]. Alignment-based objectives have also been explored [15, 54, 22, 23, 41, 33], with ideas based on cycle consistency [15, 54], soft dynamic time warping (DTW) [22, 23], and optimal transport (OT) [41]. Our model also has an alignment stage; however, unlike our work, all these prior methods are explored for third-person videos only (where views are generally more similar across videos than ego-exo) without attending to the viewpoint discrepancy problem. Frame-wise representations invariant to both ego and exo viewpoints remain unresolved.

View-invariant Action Representation Learning Many prior works have explored view-invariance in action recognition [76, 9, 53]. One popular approach is to employ multi-view videos during training to learn view-invariant features [76, 77, 27, 78, 45, 71]. Several works [17, 39, 83, 55] discover a latent feature space independent of viewpoint, while others use skeletons [81], 2D/3D poses [56, 71] and latent 3D representations [53] extracted from video to account for view differences.

As advances in AR/VR and robotics unfold, ego-exo viewpoint invariance takes on special interest. Several models relate coarse-grained action information across the two drastically different domains [70, 1, 80, 36], for action recognition using both ego and exo cameras [70] or cross-viewpoint video recognition [80, 1, 36]. Closer to our work are Time Contrastive Networks (TCN) [63] and CharadesEgo [66, 67], both of which use embeddings aimed at pushing (ir)relevant ego-exo views closer (farther). However, unlike our approach, both employ videos that concurrently record the same human actions from ego and exo viewpoints. In fact, except for [36], all the above approaches rely on simultaneously recorded, paired multi-view videos. However, even in the case of [36], the goal is a coarse, clip-wise representation. In contrast, our method leverages temporal alignment as the pretext task, enabling fine-grained representation learning from unpaired ego and exo videos.

Object-centric Representations Several works advocate object-centric representations to improve visual understanding [40, 85, 50, 36], such as an object-centric transformer for joint hand motion and interaction hotspot prediction [40], or using a pretrained detection model for robot manipulation tasks [85, 50]. Hand-object interactions are known to dominate ego-video for human-worn cameras [11, 21], and recent work explores new ways to extract hands and active objects [64, 43, 12]. Our proposed object-centric transformer shares the rationale of focusing on active objects, though in our case as a key to link the most 'matchable' things in the exo view and bridge the ego-exo gap.

3 Approach

We first formulate the problem (Sec. 3.1) and introduce the overall self-supervised learning objective (Sec. 3.2). Next we present our ideas for object-centric transformer (Sec. 3.3) and an alignment-based contrastive regularization (Sec. 3.4), followed by a review of our training procedure (Sec. 3.5).

3.1 Learning Frame-wise Ego-Exo View-invariant Features

Our goal is to learn an embedding space shared by both ego and exo videos that characterizes the progress of an action, as depicted in Fig. 1 (right). Specifically, we aim to train a view-invariant encoder network capable of extracting fine-grained frame-wise features from a given (ego or exo) video. Training is conducted in a self-supervised manner.

Formally, let ϕ ( · ; θ ) represent a neural network encoder parameterized by θ . Given an ego video sequence S with M frames, denoted by S = [ s 1 , . . . , s M ] , where s i denotes the i -th input frame, we apply the encoder to extract frame-wise features from S , i.e. , x i = ϕ ( s i ; θ ) , ∀ i ∈ { 1 , . . . , M } . The resulting ego embedding is denoted by X = [ x 1 , . . . , x M ] . Similarly, given an exo video sequence V = [ v 1 , . . . , v N ] with N frames, we obtain the frame-wise exo embedding Y = [ y 1 , . . . , y N ] , where y i = ϕ ( v i ; θ ) , ∀ i ∈ { 1 , . . . , N } using the same encoder ϕ .

The self-supervised learning objective is to map S and V to a shared feature space where frames representing similar action stages are grouped together, regardless of the viewpoint, while frames depicting different action stages ( e.g. , the initial tilting of the container versus the subsequent decrease in the liquid stream as the pouring action nears completion) should be separated. We present a series of downstream tasks to evaluate the quality of learned feature embeddings X and Y (Sec. 4).

An intuitive approach is to utilize paired ego-exo data collected simultaneously, adopting a loss function that encourages similarities between frame representations at the same timestamp while maximizing the distance between frame representations that are temporally distant [63, 66]. However, this approach relies on time-synchronized views, which is often challenging to realize in practice.

Instead, we propose to achieve view-invariance from unpaired ego-exo data; the key is to adopt temporal alignment as the pretext task. The encoder ϕ learns to identify visual correspondences across views by temporally aligning ego and exo videos that depict the same action. Throughout the learning process, no action phase labels or time synchronization is required. To enforce temporal alignment, we assume that multiple videos portraying the same action ( e.g. , pouring, tennis forehand), captured from different ego and exo viewpoints (and potentially entirely different environments), are available for training.

3.2 Ego-Exo Temporal Alignment Objective

To begin, we pose our learning objective as a classical dynamic time warping (DTW) problem [4]. DTWhas been widely adopted to compute the optimal alignment between two temporal sequences [8, 6, 23, 22, 79] according to a predefined cost function. In our problem, we define the cost of aligning frame i in ego sequence S and frame j in exo sequence V as the distance between features extracted by our encoder ϕ , x i and y j , so that aligning more similar features results in a lower cost.

Formally, given the two sequences of embeddings X of length M and Y of length N , we calculate a cost matrix C ∈ R M × N , with each element c i,j computed by a distance function D . We adopt the distance function from [22], and define c i,j to be the negative log probability of matching x i to a selected y j in sequence Y :

where β is a hyper-parameter controlling the softmax temperature, which we fix at 0.1.

DTWcalculates the alignment cost between X and Y by finding the minimum cost path in C . This optimal path can be determined by evaluating the following recurrence [60]:

where R ( i, j ) denotes the cumulative distance function, and R (0 , 0) = 0 . Given that the min operator is non-differentiable, we approximate it with a smooth differentiable operator, as defined in [34, 22]. This yields a differentiable loss function that can be directly optimized with back-propagation:

While DTW has been applied in frame-wise action feature learning [23, 22] with some degree of view invariance across exo views, it is inadequate to address the ego-exo domain gap. As we will show in results, directly adopting the DTW loss for learning ego-exo view-invariant features is sub-optimal since the ego and exo views, even when describing the same action, exhibit significant differences (compared to differences within ego views or within exo views). In Sec. 3.3-3.4, we propose two designs of AE2 that specifically account for the dramatic ego-exo view differences.

3.3 Object-centric Encoder Network

The visual appearance in ego and exo views may look dramatically different due to different backgrounds, fields of view, and (of course) widely disparate viewing angles. Yet a common element shared between these two perspectives is the human action itself. In this context, hand-object interaction regions are particularly informative about the progress of an action. Hand-object interactions are prominent in the first-person viewpoint; the wearable camera gives a close-up view of the person's near-field manipulations. In exo videos, however, these active regions may constitute only a small

hand

object

global

ROI

Align

Reverse

Object-centric  Encoder

S={S1,···,SM}

V={V1,···,VN}

Lalign (X, )

x←

S={sM,.·.,S1}

Lalign(X, )

Si

portion of the frame and are not necessarily centered, depending on the third-person camera position and angle. Inspired by these observations, we propose an object-centric encoder design that explicitly focuses on regions corresponding to hands and active objects to better bridge the ego-exo gap.

Our proposed AE2 encoder network ϕ consists of two main components: (1) a ResNet-50 encoder [25], denoted by ϕ base, to extract features from the input video frame; (2) a transformer encoder [75], denoted by ϕ transformer, to fuse global features along with regional features through self-attention.

As a preprocessing step, we employ an off-the-shelf detector to localize hand and active object regions in ego and exo frames [64]. For frame i , we select the top two hand and object proposals with the highest confidence scores, resulting in a total of four bounding boxes { BB i,k } 4 k =1 .

First, given one input frame s i at timestamp i , we pass it to ϕ base to obtain global features: g i = ϕ base ( s i ) . Next, we apply ROI Align [24] to intermediate feature maps produced by ϕ base for the input frame s i . This yields regional features r i,k for each bounding box BB i,k : r i,k = ROI Align ( ϕ interm. base ( s i ) , BB i,k ) , ∀ k .

Both the global and regional features are flattened as vectors and passed through a projection layer to unify their dimensions. We denote the projection layer to map global and local tokens as ϕ project g and ϕ project l , respectively. The input tokens are then enriched with two types of embeddings: (1) a learnable spatial embedding, denoted by e spatial, which encodes the bounding box coordinates and confidence scores obtained from the hand-object detector and (2) a learnable identity embedding, denoted by e identity, which represents the category of each feature token, corresponding to the global image, left hand, right hand, and object. This yields the tokens: z i, global = ϕ project g ( g i ) + e identity and

The transformer encoder processes one global token along with four regional tokens as input, leveraging self-attention to conduct hand-object interaction reasoning. This step enables the model to effectively capture action-related information within the scene, yielding the transformer outputs: f i,k = ϕ transformer ( z i, global , { z i, local k } k ) . Finally, we average the transformer output tokens and adopt one embedding layer ϕ embedding to project the features to our desired dimension, yielding the final output embedding for the input frame s i : x i = ϕ embedding ( 1 5 ∑ k f i,k ) .

Fig. 2 (left) showcases our proposed object-centric encoder. This design inherently bridges the egoexo gap by concentrating on the most informative regions, regardless of the visual differences between the two perspectives. As a result, the object-centric encoder can better align features extracted from both ego and exo videos, enhancing the learned features in downstream tasks.

3.4 Contrastive Regularization With Reversed Videos

As shown in previous studies [22, 23], directly optimizing the DTW loss can result in the collapse of embeddings and the model settling into trivial solutions. Moreover, aligning ego and exo videos

presents new challenges due to their great differences in visual appearance. Thus, a careful design of regularization, supplementing the alignment objective in Eqn. (3), is imperative.

We resort to discriminative modeling as the regularization strategy. The crux lies in how to construct positive and negative samples. A common methodology in previous fine-grained action approaches [63, 66, 23, 10], is to define negative samples on a frame-by-frame basis: temporally close frames are considered as positive pairs (expected to be nearby in the embedding space), while temporally far away frames are negative pairs (expected to be distant). This, however, implicitly assumes clean data and strict monotonicity in the action sequence: if a video sequence involves a certain degree of periodicity, with many visually similar frames spread out over time, these approaches could incorrectly categorize them as negative samples.

To address the presence of noisy frames in real-world videos, we introduce a global perspective that creates negatives at the video level. From a given positive pair of sequences ( S , V ) , we derive a negative pair ( ˜ S , V ) , where ˜ S = [ s M , . . . , s 1 ] denotes the reversed video sequence S . The underlying intuition is straightforward: aligning an ego-exo video pair in their natural temporal order should yield a lower cost than aligning the same pair when one video is played in reverse. Formally, we employ a hinge loss to impose this regularization objective:

where ˜ X denotes the embeddings of ˜ S , which is essentially X in reverse. Fig. 2 (right) illustrates the alignment cost matrices of ( X , Y ) and ( ˜ X , Y ) . A lighter color indicates a smaller cost. Intuitively, L align corresponds to the minimal cumulative cost traversing from the top-left to the bottom-right of the matrix. As seen in the figure, the alignment cost increases when X is reversed, i.e. , L align ( ˜ X , Y ) should be larger than L align ( X , Y ) .

Compared with frame-level negatives, our formulation is inherently more robust to repetitive and background frames, allowing for a degree of temporal variation within a video. This principle guides two key designs: (1) In creating negative samples, we opt for reversing the frames in S rather than randomly shuffling them. The latter approach may inadvertently generate a plausible (hence, not negative) temporal sequence, particularly when dealing with short videos abundant in background frames. (2) To obtain informative frames as the positive sample, we leverage the hand-object detection results from Sec. 3.3 to perform weighted sampling of the video sequences S and V . Specifically, we sample frames proportionally to their average confidence scores of hand and object detections. The subsampled S and V thus compose a positive pair. This way, frames featuring clear hand and object instances are more likely to be included.

3.5 Training and Implementation Details

Our final loss is a combination of the DTW alignment loss (Eqn. (3)) and the contrastive regularization loss (Eqn. (4)):

where λ is a hyper-parameter controlling the ratio of these two loss terms.

During training, we randomly extract 32 frames from each video to construct a video sequence. The object-centric encoder network ϕ , presented in Sec. 3.3, is optimized to minimize the loss above for all pairs of video sequences in the training set, including ego-ego, exo-exo, and ego-exo pairs 3 -all of which are known to contain the same coarse action. During evaluation, we freeze the encoder ϕ and use it to extract 128-dimensional embeddings for each frame. These representations are then assessed across a variety of downstream tasks (Sec. 4). See Supp. for full implementation details.

4 Datasets and Evaluation

Datasets Existing datasets for fine-grained video understanding ( e.g. [82, 65, 3]) are solely composed of third-person videos. Hence we propose a new ego-exo benchmark by assembling ego and exo videos from five public datasets-CMU-MMAC [13], H2O [32], EPIC-Kitchens [11], HMDB51 [31]

Liquid exits

container

Phase 0

Start

Pouring

Complete

End

...

...

...

Phase 1

Phase 2

Racket

touches ball

Phase 0

Start

End

Phase 1

...

...

and Penn Action [82]-plus a newly collected ego tennis forehand dataset. Our benchmark consists of four action-specific ego-exo datasets: (A) Break Eggs : ego-exo videos of 44 subjects breaking eggs, with 118/30 train/val+test samples from CMU-MMAC [13]; (B) Pour Milk [32]: ego-exo videos of 10 subjects pouring milk, with 77/35 train/val+test from H2O [32]; (C) Pour Liquid : we extract ego 'pour water' clips from EPIC-Kitchens [11] and exo 'pour' clips from HMDB51 [31], yielding 137/56 train/val+test. (D) Tennis Forehand : we extract exo tennis forehand clips from Penn Action [82] and collect ego tennis forehands by 12 subjects, totaling 173/149 train/val+test clips. We obtain ground truth key event and action phase labels for all of them (used only in downstream evaluation), following the labeling methodology established in [15]. Fig. 3 illustrates examples of these labels. Altogether, these four datasets provide unpaired ego-exo videos covering both tabletop ( i.e. , breaking eggs, pouring) and physical actions ( i.e. , tennis forehand). 4 While (A) and (B) are from more controlled environments, where exo viewpoints are fixed and both views are collected in the same environment, (C) and (D) represent in-the-wild scenarios, where ego and exo are neither synchronized nor collected in the same environment. Our model receives them all as unpaired.

Evaluation We aim to demonstrate the practical utility of the learned ego-exo view-invariant features by addressing two questions: (1) how well do the features capture the progress of the action? (2) how view-invariant are the learned representations? To this end, we test four downstream tasks: (1) Action Phase Classification : train an SVM classifier on top of the embeddings to predict the action phase labels for each frame and report F1 score. We also explore few-shot and cross-view zero-shot settings ( i.e. , 'exo2ego' when training the SVM classifier on exo only and testing on ego, and vice versa). (2) Frame Retrieval : parallels the classification task, but done via retrieval (no classifier), and evaluated with mean average precision (mAP)@ K , for K =5,10,15. We further extend this task to the cross-view setting to evaluate view-invariance. (3) Phase Progression : train a linear regressor on the frozen embeddings to predict the phase progression values, defined as the difference in time-stamps between any given frame and each key event, normalized by the number of frames in that video [15], evaluated by average R-square. (4) Kendall's Tau : to measure how well-aligned two sequences are in time, following [15, 22, 23, 10, 41].

Note that: (1) Departing from the common practice of splitting videos into train and validation only [15], our benchmark includes a dedicated test set to reduce the risk of model overfitting. (2) We opt for the F1 score instead of accuracy for action phase classification to better account for the label imbalance in these datasets and provide a more meaningful performance evaluation. (3) Phase progression assumes a high level of consistency in actions, with noisy frames diminishing the performance greatly. Due to the challenging nature of Pour Liquid data, we observe a negative progression value for all approaches. Thus, we augment the resulting embeddings with a temporal dimension, as 0.001 times the time segment as the input so that the regression model can distinguish repetitive (or very similar) frames that differ in time. We report modified progression value for all approaches on this dataset. (4) Kendall's Tau assumes that there are no repetitive frames in a video. Since we adopt in-the-wild videos where strict monoticity is not guaranteed, this metric may not faithfully reflect the quality of representations. Nonetheless, we report them for completeness and for consistency with prior work. (5) With our focus being frame-wise representation learning, the total number of training samples equals the number of frames rather than the number of videos, reaching a scale of a few thousands (8k-30k per action class). See Supp. for more dataset and evaluation details.

5 Experiments

Baselines We compare AE2 with 8 total baselines of three types: (1) Naive baselines based on random features or ImageNet features , produced by a ResNet-50 model randomly initialized

or pretrained on ImageNet [14]; (2) Self-supervised learning approaches specifically designed for learning ego-exo view-invariant features, time-contrastive networks (TCN) [63] and ActorObserverNet [66]. Both require synchronized ego-exo videos for training, and are thus only applicable to Break Eggs data. Additionally, TCN offers a single-view variant that eliminates the need for synchronized data, by taking positive frames within a small temporal window surrounding the anchor, and negative pairs from distant timesteps. We implement single-view TCN for all datasets. (3) Finegrained action representation learning approaches, CARL [10] which utilizes a sequence contrastive loss, and TCC [15] and GTA [22], which also utilize alignment as the pretext task but do not account for ego-exo viewpoint differences.

Main Results In Table 1, we benchmark all baseline approaches and AE2 on four ego-exo datasets. See Supp. for full results, including the F1 score for few-shot classification, and mAP@5,15 for frame retrieval. From the results, it is clear that AE2 greatly outperforms other approaches across all downstream tasks. For instance, on Break Eggs, AE2 surpasses the multi-view TCN [63], which utilizes perfect ego-exo synchronization as a supervision signal, by +6.32% in F1 score. Even when we adapt the TCN to treat all possible ego-exo pairs as perfectly synchronized, AE2 still excels, showing a considerable margin of improvement (see Supp. for a discussion). Pour Liquid poses the most substantial challenge due to its in-the-wild nature, as reflected by the low phase progression and Kendall's Tau values. Yet, AE2 notably improves on previous works, achieving, for instance, a +9.64% F1 score. Regarding Tennis Forehand, another in-the-wild dataset, we note that some methods like TCC perform well in identifying correspondences within either the ego or exo domain, but struggle to connect the two. Consequently, while it achieves an 80.24% mAP@10 for standard frame retrieval, it falls below 60% for cross-view frame retrieval. In contrast, AE2 effectively bridges these two significantly different domains, resulting in a high performance of over 81% mAP@10

(a)originalsequence

(a)originalsequence

Frame 0

Frame 1

Frame 2

Frame 3

Frame 4

Frame 0

Frame 1

Frame 2

Frame 3

Frame 4

(b)randomly shuffled sequence as negative

(b)randomly shuffled sequenceasnegative

Frame 0

Frame 2

Frame 1

Frame 4

Frame 3

Frame 1

Frame 0

Frame 3

Frame 2

Frame 4

(c) temporallyreversed sequence as negative

(c) temporally reversed sequence as negative

Frame 4

Frame 3

Frame 2

Frame 1

Frame 0

Frame 4

Frame 3

Frame 2

Frame 1

Frame 0

in both standard and cross-view settings. These results demonstrate the efficacy of AE2 in learning fine-grained ego-exo view-invariant features.

Ablation Table 2 provides an ablation of the two design choices of AE2. Starting with a baseline approach (denoted as 'Base DTW' in the table), we optimize a ResNet-50 encoder with respect to a DTW alignment objective as defined in Eqn. (3). Next, we replace the feature encoder with our proposed object-centric encoder design and report its performance. Finally, we incorporate the contrastive regularization from Eqn. (4) into the objective. It is evident that object-centric representations are instrumental in bridging the ego-exo gap, leading to substantial performance improvements. Furthermore, incorporating contrastive regularization provides additional gains.

Negative Sampling We conduct experiments to validate the efficacy of creating negative samples by temporally reversing frames, as opposed to random shuffling and report results in Table 3. In general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. The inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to a positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur in real

Table 2: Ablation study of AE2 on four datasets (F1 score). See Supp. for other metrics.

Dataset

(A)

(B)

(C)

(D)

Random Shuffle

61.66 81.92 63.48 86.73

Temporal Reverse

65.85 84.90 65.54 86.83

videos), temporally reversing frames is apt to create a negative sample that deviates from the correct action progression order. To further illustrate this point, we visualize two video sequences in Fig. 5. It is evident that the randomly shuffled sequence seems to preserve the sequence of actions like breaking eggs or pouring liquid, thereby resembling a 'positive' example. Consequently, incorporating such negative samples into training may confuse the model and lead to diminished performance.

Qualitative Results Fig. 5 (left) shows examples of cross-view frame retrieval on Pour Liquid and Tennis Forehand datasets. Given a query image from one viewpoint in the test set, we retrieve its nearest neighbor among all test frames in another viewpoint. We observe that the retrieved frames closely mirror the specific action state of the query frame. For example, when the query is a frame

Pre-pour: lifting the container

Active Pour: liquid exiting the container

Query (exo)

Retrieved Nearest Neighbors (ego)

Query (ego)

Retrieved Nearest Neighbors (exo)

Pre-stroke: racket poised to strike ball

Pre-crack: eggshell intact

and ready to be broken

Post-crack: eggshell

cracked, contents released

Video 1 (ego)

Video 2 (ego)

Video 3 (exo)

Video 4 (exo)

Frame embeddings

before training

Post-stroke: following through after the ball is hit

capturing the moment before the pouring action, where the liquid is still within the container (row 1), or when the query shows pouring actively happening (row 2), the retrieved exo frames capture that exact step as well. Fig. 5 (right) shows tSNE visualizations of embeddings for 4 test videos on Break Eggs. Our learned embeddings effectively capture the progress of an action, while remaining view-invariant. These learned embeddings enable us to transfer the pace of one video to others depicting the same action. We demonstrate this capability with examples of both ego and exo videos, collected in distinct environments, playing synchronously in the Supp. video.

Limitations While AE2 has demonstrated its ability to handle in-the-wild data, there remains much room for improvement, particularly on challenging in-the-wild datasets such as Pour Liquid. Here, we observe instances of failure in cross-view frame retrieval and synchronizing the pace of two videos. Although we highlight successful examples in our discussion, it is important to note that some attempts have been unsuccessful, largely due to the inherent noise in real-world videos. This represents a key area for future refinement and development of AE2.

6 Conclusion

We tackle fine-grained temporal understanding by jointly learning from unpaired egocentric and exocentric videos. Our core idea is to leverage temporal alignment as the self-supervised learning objective for invariant features, together with a novel object-centric encoder and contrastive regularizer that reverses frames. On four datasets we achieve superior performance compared to state-of-the-art ego-exo and alignment approaches. This fundamental technology has great promise for both robot learning and human skill learning in AR, where an ego actor needs to understand (or even match) the actions of a demonstrator that they observe from the exo point of view. Future work includes generalizing AE2 to train from pooled data of multiple actions, extending its applicability to more complex, longer action sequences and exploring its impact for robot imitation learning.

References

Shervin Ardeshir and Ali Borji. An exocentric look at egocentric actions and vice versa. Computer Vision and Image Understanding , 171:61-68, 2018.

Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII , pages 657-675. Springer, 2022.

Yizhak Ben-Shabat, Xin Yu, Fatemeh Saleh, Dylan Campbell, Cristian Rodriguez-Opazo, Hongdong Li, and Stephen Gould. The ikea asm dataset: Understanding people assembling furniture through actions, objects and pose. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 847-859, 2021.

Donald J Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In KDD workshop , volume 10, pages 359-370. Seattle, WA, USA:, 1994.

Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML , volume 2, page 4, 2021.

Kaidi Cao, Jingwei Ji, Zhangjie Cao, Chien-Yi Chang, and Juan Carlos Niebles. Few-shot video classification via temporal alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10618-10627, 2020.

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6299-6308, 2017.

Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3tw: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3546-3555, 2019.

Chao-Yeh Chen and Kristen Grauman. Inferring unseen views of people. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2003-2010, 2014.

Minghao Chen, Fangyun Wei, Chong Li, and Deng Cai. Frame-wise action representations for long videos via sequence contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13801-13810, 2022.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 720-736, 2018.

Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. Advances in Neural Information Processing Systems , 35:13745-13758, 2022.

Fernando De la Torre, Jessica Hodgins, Adam Bargteil, Xavier Martin, Justin Macey, Alex Collado, and Pep Beltran. Guide to the carnegie mellon university multimodal activity (cmu-mmac) database.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1801-1810, 2019.

Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A largescale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 961-970, 2015.

Ali Farhadi and Mostafa Kamali Tabrizi. Learning to recognize activities from the wrong view point. In Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part I 10 , pages 154-166. Springer, 2008.

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 62026211, 2019.

Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. 2017.

Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision , pages 5842-5850, 2017.

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18995-19012, 2022.

Isma Hadji, Konstantinos G Derpanis, and Allan D Jepson. Representation learning via global temporal alignment and cycle-consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11068-11077, 2021.

Sanjay Haresh, Sateesh Kumar, Huseyin Coskun, Shahram N Syed, Andrey Konin, Zeeshan Zia, and Quoc-Huy Tran. Learning by aligning videos in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5548-5558, 2021.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In NeurIPS , 2016.

Alexandros Iosifidis, Anastasios Tefas, and Ioannis Pitas. View-invariant action recognition based on artificial neural networks. IEEE transactions on neural networks and learning systems , 23(3):412-424, 2012.

D. Jayaraman and K. Grauman. Slow and steady feature analysis: Higher order temporal coherence in video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Bruno Korbar, Du Tran, and Lorenzo Torresani. Co-training of audio and video representations from self-supervised temporal synchronization. In NeurIPS , 2018.

Hilde Kuehne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 780-787, 2014.

Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 International conference on computer vision , pages 2556-2563. IEEE, 2011.

Taein Kwon, Bugra Tekin, Jan Stühmer, Federica Bogo, and Marc Pollefeys. H2o: Two hands manipulating objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10138-10148, 2021.

Taein Kwon, Bugra Tekin, Siyu Tang, and Marc Pollefeys. Context-aware sequence alignment using 4d skeletal augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8172-8182, 2022.

Mandy Lange, Dietlind Zühlke, Olaf Holz, Thomas Villmann, and Saxonia-Germany Mittweida. Applications of lp-norms and their smooth approximations for gradient based learning vector quantization. In ESANN , pages 271-276. Citeseer, 2014.

Jun Li and Sinisa Todorovic. Action shuffle alternating learning for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1262812636, 2021.

Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6943-6953, 2021.

Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems , 35:7575-7586, 2022.

Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13853-13863, 2022.

Jingen Liu, Mubarak Shah, Benjamin Kuipers, and Silvio Savarese. Cross-view action recognition via view knowledge transfer. In CVPR 2011 , pages 3209-3216. IEEE, 2011.

Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, and Xiaolong Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3282-3292, 2022.

Weizhe Liu, Bugra Tekin, Huseyin Coskun, Vibhav Vineet, Pascal Fua, and Marc Pollefeys. Learning to align sequential actions in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2181-2191, 2022.

YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 1118-1125. IEEE, 2018.

Jian Ma and Dima Damen. Hand-object interaction reasoning. In 2022 18th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) , pages 1-8. IEEE, 2022.

Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Towards universal visual reward and representation via value-implicit pre-training. In ICLR , 2023.

Behrooz Mahasseni and Sinisa Todorovic. Latent multitask learning for view-invariant action recognition. In Proceedings of the IEEE International Conference on Computer Vision , pages 3128-3135, 2013.

Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9879-9889, 2020.

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2630-2640, 2019.

Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV , 2016.

Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In NeurIPS , 2020.

Tushar Nagarajan and Kristen Grauman. Shaping embodied agent behavior with activity-context priors from egocentric video. Advances in Neural Information Processing Systems , 34:29794-29805, 2021.

Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In ECCV , 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

AJ Piergiovanni and Michael S Ryoo. Recognizing actions in videos from unseen viewpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4124-4132, 2021.

Senthil Purushwalkam, Tian Ye, Saurabh Gupta, and Abhinav Gupta. Aligning videos in space and time. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16 , pages 262-278. Springer, 2020.

Hossein Rahmani and Ajmal Mian. Learning a non-linear knowledge transfer model for cross-view action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2458-2466, 2015.

Hossein Rahmani and Ajmal Mian. 3d action recognition from novel viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1506-1515, 2016.

Cen Rao and Mubarak Shah. View-invariance in action recognition. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 , volume 2, pages II-II. IEEE, 2001.

Cen Rao, Alper Yilmaz, and Mubarak Shah. View-invariant representation and recognition of actions. International Journal of Computer Vision , 50(2):203, 2002.

Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, and Bernt Schiele. Recognizing fine-grained and composite activities using hand-centric features and script data. International Journal of Computer Vision , 119:346-373, 2016.

Hiroaki Sakoe and Seibi Chiba. Dynamic programming algorithm optimization for spoken word recognition. IEEE transactions on acoustics, speech, and signal processing , 26(1):43-49, 1978.

Madeline C. Schiappa, Yogesh S. Rawat, and Mubarak Shah. Self-supervised learning for videos: A survey. arXiv:2207.00419 , 2022.

Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly101: A large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2109621106, 2022.

Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 1134-1141. IEEE, 2018.

Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9869-9878, 2020.

Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2616-2625, 2020.

Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. In proceedings of the IEEE conference on computer vision and pattern recognition , pages 7396-7404, 2018.

Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626 , 2018.

Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 510-526. Springer, 2016.

Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems , 27, 2014.

Bilge Soran, Ali Farhadi, and Linda Shapiro. Action recognition in the presence of one egocentric and multiple static cameras. In Computer Vision-ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part V 12 , pages 178-193. Springer, 2015.

Jennifer J Sun, Jiaping Zhao, Liang-Chieh Chen, Florian Schroff, Hartwig Adam, and Ting Liu. Viewinvariant probabilistic embedding for human pose. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16 , pages 53-70. Springer, 2020.

Reuben Tan, Bryan Plummer, Kate Saenko, Hailin Jin, and Bryan Russell. Look at what i'm doing: Self-supervised spatial grounding of narrations in instructional videos. Advances in Neural Information Processing Systems , 34:14476-14487, 2021.

Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1207-1216, 2019.

Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 4489-4497, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Daniel Weinland, Mustafa Özuysal, and Pascal Fua. Making action recognition robust to occlusions and viewpoint changes. In European Conference on Computer Vision , 2010.

Daniel Weinland, Remi Ronfard, and Edmond Boyer. Free viewpoint action recognition using motion history volumes. Computer Vision and Image Understanding (CVIU) , 2006.

Xinxiao Wu, Han Wang, Cuiwei Liu, and Yunde Jia. Cross-view action recognition over heterogeneous feature spaces. In Proceedings of the IEEE International Conference on Computer Vision , pages 609-616, 2013.

Yuncong Yang, Jiawei Ma, Shiyuan Huang, Long Chen, Xudong Lin, Guangxing Han, and Shih-Fu Chang. Tempclr: Temporal alignment representation with contrastive learning. arXiv preprint arXiv:2212.13738 , 2022.

Huangyue Yu, Minjie Cai, Yunfei Liu, and Feng Lu. What i see is what you see: Joint attention learning for first and third person video co-analysis. ACM MM , 2019.

Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng. View adaptive recurrent neural networks for high performance human action recognition from skeleton data. In Proceedings of the IEEE international conference on computer vision , pages 2117-2126, 2017.

Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A stronglysupervised representation for detailed action understanding. In Proceedings of the IEEE international conference on computer vision , pages 2248-2255, 2013.

Zhong Zhang, Chunheng Wang, Baihua Xiao, Wen Zhou, Shuang Liu, and Cunzhao Shi. Cross-view action recognition via a continuous virtual path. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2690-2697, 2013.

Honglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. arXiv preprint arXiv:2303.18230 , 2023.

Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11339 , 2022.

Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3537-3545, 2019.

A Video Containing Qualitative Results

We invite the reader to view the video available at https://vision.cs.utexas.edu/projects/ AlignEgoExo/ , where we show qualitative examples of one practical application enabled by our learned ego-exo view-invariant representations-synchronous playback of egocentric and exocentric videos. We randomly select ego-exo video pairs from the test set, and use the frozen encoder ϕ to extract frame-wise embeddings. We then match each frame of one video (the reference) to its closest counterpart in the other video using nearest neighbor. As demonstrated across all datasets with several examples, AE2 effectively aligns two videos depicting the same action, overcoming substantial viewpoint and background differences. The video also includes examples of synchronizing two egocentric or two exocentric videos.

B Experimental Setup

B.1 Datasets

Since existing video datasets for fine-grained video understanding ( e.g. , PennAction [82], FineGym [65], IKEA ASM [3]) are solely composed of third-person videos, we curate video clips from five public datasets: Break Eggs [13], H2O [32], EPIC-Kitchens [11], HMDB51 [31] and Penn Action [82] and collect a egocentric Tennis Forehand dataset. Our data selection criteria is to find videos that depict the same action from distinct egocentric and exocentric viewpoints. Consequently, TCN pouring [63] is excluded due to its small scale and significant similarity between the egocentric and exocentric views; Assembly101 [62] is not selected since there there are large variations within a single atomic action and the egocentric video is monochromatic.

In all, our dataset compilation from five public data sources, along with our collected egocentric tennis videos, results in four distinct ego-exo datasets, each describing specific atomic actions:

(A) Break Eggs . The dataset contains 44 subjects cooking five different recipes (brownies, pizza, sandwich, salad, and scrambled eggs), captured from one egocentric and four exocentric views simultaneously in CMU-MMAC [13]. We use the temporal keystep boundaries provided in [2] to extract clips corresponding to the action of breaking eggs from all videos. Among the four exocentric viewpoints, we adopt videos from the right-handed view, as this viewpoint captures the action being performed most clearly. We randomly split the data into training and validation sets across subjects, with 35 subjects (118 videos) for training and 9 subjects (30 videos) for validation and test. There is strict synchronization between egocentric and exocentric video pairs in this dataset.

(B) Pour Milk . The dataset features 10 subjects interacting with a milk carton using both hands, captured by one egocentric camera and four static exocentric cameras in H2O [32]. We utilize the keystep annotations provided in [33] to extract clips corresponding to the pouring milk action. All four exocentric views are included, as they clearly capture the action. We follow the data split in [33], with 7 subjects (77 videos) in the training set and 3 subjects (35 videos) in the validation and test set. A portion of this dataset (the first 3 subjects) contains synchronized egocentric-exocentric video pairs, while the remaining part does not.

(C) Pour Liquid . To evaluate our methods on in-the-wild data, we assemble a Pour Liquid dataset by extracting clips from one egocentric dataset, EPIC-Kitchens [11] and one exocentric dataset, HMDB51 [31]. We utilize clips from the 'pour water' class in EPIC-Kitchens and 'pour' category in HMDB51. Following the data split in the original datasets, we obtain 137 videos for training and 56 videos for validation and test. It is important to note that

the egocentric and exocentric videos are neither synchronized nor collected in the same environment, providing a challenging testbed.

(D) Tennis Forehand . To include physical activities in our study, we leverage exocentric video sequences of the tennis forehand action from Penn Action [82] and collect an egocentric dataset featuring the same action performed by 12 subjects using Go Pro HERO8. We adopt the data split from [15] for Penn Action, and divide our egocentric tennis forehand dataset by subject: 8 for training and 4 for validation and testing. This results in a total of 173 clips for training, and 149 clips for validation and testing. The egocentric and exocentric videos, gathered from a range of real-world scenarios, are naturally unpaired.

Table 4 provides a summary of these four datasets. In addition, we recognize that the original datasets lack frame-wise labels and provide dense frame-level annotations to enable a comprehensive evaluation of the learned representations. See Table 5 for a complete list of all the key events we annotate and Fig. 6 for illustrative examples. We will release our collected data and labels for academic usage. 5

B.2 Evaluation

We provide a detailed description of the four downstream tasks below and their corresponding evaluation metrics:

Action Phase Classification . We train an SVM classifier on top of the embeddings to predict the action phase labels for each frame and report F1 score on test data. Besides the regular setting, we investigate (1) few-shot; and (2) cross-view zero-shot settings.

Few-shot. We assume that only a limited number of training videos have annotations and can be used for training the SVM classifier.

Cross-view zero-shot. We assume that per-frame labels of training data are only available on one view for training the SVM classifier, and report the test performance on the other view. We use the terms 'exo2ego' to describe the case where we use exocentric data for training the SVM classifier and test its performance on egocentric data, while 'ego2exo' represents the reverse case.

Phase Progression . We train a linear regressor on the frozen embeddings to predict the phase progression values, defined as the difference in time-stamps between any given frame and each key event, normalized by the number of frames in that video [15]. Average Rsquare measure on test data is reported. This metric evaluates how well the progress of an action is captured by the embeddings, with the maximum value being 1.

Frame Retrieval . We report the mean average precision (mAP)@K (K=5,10 ,15). For each query, average precision is computed by determining how many frames among the retrieved K frames have the same action phase labels as the query frame, divided by K. Furthermore, to evaluate view-invariance, we propose the cross-view frame retrieval task ( i.e. , ego2exo and exo2ego frame retrieval). For each query in one view, the goal is to retrieve K frames from another view. No additional training is required for this task.

Kendall's Tau . This metric is calculated for every pair of test videos by sampling two frames in the first video and retrieving the corresponding nearest frames in the second video, then checking whether their orders are shuffled. It measures how well-aligned two sequences are in time. No additional training or frame-wise labels are necessary for this evaluation.

(A) Break Eggs

...

...

...

...

Visible crack on

the eggshell

Egg contents

released

Hit egg

Start

End

Phase 0

Phase 1

Phase 2

Phase 3

(B) Pour Milk

Liquid exits

container

Phase 0

Start

Pouring

Complete

End

Phase 1

Phase 2

...

...

...

Liquid exits

container

Phase 0

Start

Pouring

Complete

End

...

...

...

Phase 1

Phase 2

(C) Pour Liquid

Racket

touches ball

Phase 0

Start

End

Phase 1

...

...

(D) Tennis Forehand

B.3 Implementation

For all video sequences, frames are resized to 224 × 224. During training, we randomly extract 32 frames from each video to construct a video sequence. We train the models for a total number of 300 epochs with a batch size of 4, using the Adam optimizer. The base encoder ϕ base is initialized with a ResNet-50 pretrained on ImageNet, and jointly optimized with the transformer encoder ϕ transformer throughout the training process. The model checkpoint demonstrating the best performance on validation data is selected, and its performance on test data is reported. In terms of the encoder network, global features are taken from the output of Conv 4 c layer in ϕ base . Following [15], we stack the features of any given frame and its context frames along the dimension of time, followed by 3D convolutions for aggregating temporal information and 3D max pooling. In all experiments, the number and stride of context frames are set as 1 and 15 , respectively. For local features, we take output of the Conv 1 layer in ϕ base and apply 3D max pooling to aggregate temporal information from the given frame and its context frame. The features are then fed as input of ROI Align.

During evaluation, we freeze the encoder ϕ and use it to extract 128-dimensional embeddings for each frame. These representations are then assessed across a variety of downstream tasks (Sec. 4). Detailed hyperparameters specific to each dataset are provided in Table 6. Noteworthy adjustments include: (1) In the case of Tennis Forehand, we utilize a single object proposal, as the active object is only the tennis racket (the tennis ball is too small to be detected reliably). Furthermore, given the shorter video lengths, we sample 20 frames from each video as opposed to the usual 32. (2) For datasets featuring non-monotonic actions ( i.e. , Pour Milk and Pour Liquid), we construct the negative sequence by randomly reversing either the first or the last half of the sequence, rather than the whole sequence. This is due to the cyclic nature of the pouring action present in some videos within these datasets. All experiments are conducted using PyTorch [52] on 2 Nvidia V100 GPUs.

C Further Results and Visualizations

Results Supplementing Table 1 in the main paper, Tables 7 and 8 present comprehensive results of AE2 and baseline models on few-shot action phase classification and frame retrieval tasks. For few-shot classification, we train the SVM classifier with 10% (or 50%) of the training data, averaging results over 10 runs. AE2 demonstrates superior performance in learning fine-grained, view-invariant ego-exo features when compared with ego-exo [66, 63], frame-wise contrastive learning [10], and alignment-based [15, 22] approaches.

On Break Eggs, AE2 greatly outperforms the multi-view TCN [63], which utilizes perfect ego-exo synchronization as a supervision signal. We hypothesize that the strict supervision requirement of

F1 score

Number of labeled videos

Number of labeled videos

Number of labeled videos

Number of labeled videos

TCC

GTA

AE2

TCC

GTA

AE2

TCC

GTA

AE2

TCC

GTA

AE2

TCC

GTA

AE2

(A)CMU-MMAC

(B)H2O

(C) Pour Liquid

(D) Tennis Forehand

85

60

80

60

80

50

70

50

75

40

60

40

70

30

50

0

25

50

75

100

0

20

40

60

80

0

25

50

75

100

125

0

50

100

150

TCN might be limiting, as it can not utilize as many ego-exo pairs as AE2 due to its reliance on ego-exo synchronization. In contrast, AE2 capitalizes on a broader set of unpaired ego-exo data. Even when we modify multi-view TCN to consider all potential ego-exo pairs as synchronously perfect (termed as multi-view TCN (unpaired) in the tables), it does not outperform its regular version, indicating a lack of robustness towards non-synchronized ego-exo pairs. Consequently, it appears that multi-view TCN is ill-equipped to learn desired view-invariant representations from unpaired, real-world ego-exo videos.

Few-shot Learning Besides the few-shot results in Table 7, we vary the number of labeled training videos, ranging from extremely sparse (a single labeled video) to the case where all training videos are labeled. Note that each labeled video equates to multiple labeled frames. AE2 is compared with top-performing baselines, TCC [15] and GTA [22], across all four datasets in Fig. 7. The results are averaged over 50 runs and include a +- one standard deviation error bar. As shown, AE2 excels in low-label scenarios. For instance, on Pour Milk, a single labeled video yields an action phase classification F1 score over 80%. This suggests that AE2 effectively aligns representations across all training videos, enabling a robust SVM classifier for the downstream task, even with minimal labeling.

Ablation Table 9 presents an ablation of AE2 on all four datasets, which is a comprehensive version of Table 2 in the main paper. From the results, we can see that object-centric representations are instrumental in bridging the ego-exo gap, leading to substantial performance improvements. For instance, frame retrieval mAP@10 improves by +10.83% on Pour Liquid and +5.57% on Tennis Forehand. Furthermore, incorporating contrastive regularization provides additional performance boosts for several downstream tasks such as regular action phase classification. These results demonstrate the integral contributions of both components of AE2 to achieve optimal performance.

Negative Sampling (Temporal Reverse versus Random Shuffle) Table 10 presents a comparison of two negative sampling strategies (random shuffling versus temporal reversing), which is a comprehensive version of Table 3 in the main paper. The results reveal that, in general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. For example, on Break Eggs data, random shuffling results in a decrease of -5.62% in F1 score and -4.19% in mAP@10 compared with regular AE2. The inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to a positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur in real videos), temporally reversing frames is apt to create a negative sample that deviates from the correct action progression order.

To dissect the performance gains further, the large improvement observed on dataset (A) when using temporally reversed frames over randomly shuffled frames stems from its inherent ability to be more robust to the repetitive nature of the breaking egg action (e.g., the camera wearer may hit the egg twice before breaking it, resulting in many similar frames in the video sequence). On the other hand, the subtler improvement on dataset (D), dealing with the tennis forehand action, is likely due to the strictly monotonic nature of this action, leaving less room for our method to outperform. However, it is essential to emphasize that this does not diminish the value of our approach, our negative sampling strategy still proves preferable consistently across all four datasets.

Negative Sampling (Sequence-level versus Frame-level) We emphasize the unique advantages of our proposed sequence-level negatives over the frame-level negatives used in prior works [63, 66, 10].

frame-levelnegatives

sequence-levelnegatives

While periodic actions present great challenges for all negative sampling techniques, sequencelevel negatives offers a more robust solution to tackle the intricacies of repetitiveness within an action. As a motivating example, Fig. 8 depicts a video sequence of breaking eggs with a certain amount of periodicity (where the camera wearer hits the egg to the bowl's edge twice). Frame-level sampling approaches treat temporally close frames as positive pairs and temporally distant ones as negative pairs. As a consequence, the two visually similar frames (marked by a red border) would be incorrectly identified as a negative pair by those existing methods. Contrarily, our method of constructing negative samples adopts a global perspective, addressing temporal fluctuations within a video more effectively. By reversing the entire sequence, we create a challenge in aligning the reversed ego view video with the original exo view video, thus forming a valid negative sample.

This methodology is not only conceptually appealing but also empirically validated. Table 1 in the main paper illustrates the effectiveness of our proposed AE2, demonstrating that it outperforms prior techniques utilizing frame-wise negatives [63, 66, 10]. These results offer compelling empirical evidence that our sequence-level negative sampling strategy successfully mitigates the issues related to periodicity and improves performance, affirming its superiority over conventional frame-by-frame sampling methods.

Visualizations In addition to the cross-view frame retrieval results for Pour Liquid and Tennis Forehand presented in the main paper (Fig. 5), we showcase results for the other two datasets ( i.e. , Break Eggs and Pour Milk) in Fig. 9. For any given query frame from one view, the retrieved nearest neighbors closely match the action stage of the query, regardless of substantial differences in viewpoints. These results underline AE2's efficacy in learning fine-grained action representations that transcend ego-exo viewpoint differences.

Retrieved Nearest Neighbors (exo)

Active pour: milk carton touching the cup, nearly vertical

Post-crack: eggshell cracked, contents released

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a self-supervised approach for learning view-invariant action features by aligning the egocentric and exocentric videos in time. The proposed method has two main components: 1)  an object-centric encoder to learn discriminative features around the hands and active objects; 2) a contrastive learning paradigm to align videos from different viewpoints. \n\nApart from the proposed method, this paper also establishes a new benchmark to evaluate the proposed method, and extensive experiments have been conducted to validate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The paper is well-motivated and well-written. Learning view-invariant features for fine-grained action recognition is important in video understanding. \n\n2. Experiments are conducted on four different downstream tasks to show the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The proposed method has limited novelty, and it is a combination of different existing modules. For example, emphasizing the hand-region features and contrastive learning with inverse temporal orders are widely used in many existing works.\n\n2. The proposed benchmark contains four datasets, but each dataset has a very limited number of samples, conversing a few action classes.\n(a) Limited samples: e.g., the CMU-MMAC dataset contains 118 videos for training and 30 videos for validation and testing. This makes the model training easy to overfit. \n(b) Limited diversity: e.g., the CMU-MMAC dataset contains only a few actions related to 'breaking eggs', which casts doubt on the generalizability of the proposed method.\n\nQUESTIONS:\nAs the authors generate the benchmark datasets on their own, please elaborate more on the details, such as the number of action classes in each dataset.\n\nLIMITATIONS:\nPlease see the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper introduces a novel approach for learning view-invariant features from exo and ego views using temporal alignment. The key idea is to utilize unpaired ego-exo data for the pretext task of temporal alignment in a self-supervised setting. Furthermore, an object-centric encoder was used to focus on hand and object regions. The authors also introduce a regularization method to maximize the margin between positive pair and their corresponding negative pairs. The negative pair is constructed by replacing the one video of the pair with its reversed.

STRENGTHS:
- The paper is well-written and easy to follow
- The paper introduces a novel framework for view-invariant representation learning
- The proposed approach is capable of utilizing the unpaired data where the videos of a pair do not have to be synchronized or be in the same environment 
- The method is compared with a reasonable set of baseline methods

WEAKNESSES:
- The idea of sequence alignment as self-supervised video representation learning has been studied by e.g. [22, 23]. It is not clear why the methods of [22, 23] cannot be applied to ego-exo data.
- In Line205-211 it is argued that frame-by-frame sampling in contrastive loss can lead to incorrect negative samples due to the periodicity of some videos, however, it is not clear how the proposed approach of using reversed videos for the negative sample does not suffer from periodicity.

QUESTIONS:
See weakness.

LIMITATIONS:
Yes, the limitation is addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a novel self-supervised embedding approach AE2 to learn fine-grained view-invariant frame-wise video features from unpaired egocentric and exocentric videos. AE2 consists of two core designs: (1) an object-centric encoder that explicitly focuses on \nobjects; (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. Four datasets exhibit improvement in experiments, demonstrating a feasible framework for learning fine-grained view-invariant features.\n\nSTRENGTHS:\n1. This paper presents a well-motivation idea and sound arguments in general. It makes sense to learn view-invariant video features form unpaired data.\n2. Ablation studies are conducted to evaluate the effectiveness of each design.\n3. The visualization of learned view-invariant video features for test videos is very impressive, which demonstrates effectively capture the progress of an action while remaining view-invariant.\n\nWEAKNESSES:\n1. In Lines 225-226 “ In creating negative samples, we opt for reversing the frames in S rather than randomly shuffling them.” Although the authors explain the superiority of the reversing frames, authors need to provide further experiments comparing reversing frames with randomly shuffling them.\n2. Baseline's experimental results are so good that the baseline achieves SOTA results on most of the datasets. why is the baseline's experimental performance so good?\n3. Lack of analysis of experimental results in the ablation experiment section.\n4. In Training and Implementation Details, “the object-centre encoder network is optimized to minimize the loss above for 240 all pairs of video sequences in the training set, including ego-ego, exo-exo, and ego-exo pairs.”  The training process introduces data from the same viewpoint and paired data, but the network structure is designed specifically for unpaired data from different viewpoints. Can separate experiments be performed only on unpaired data from different viewpoints to demonstrate the validity of the proposed method?\n5. The language should be further improved by correcting some language errors, such as “ See Supp. for full implementation deatils ...” (Lines 232 in the paper).\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a self-supervised learning approach for view-invariant representation learning from egocentric and exocentric videos. Unlike prior approaches, the proposed method is applicable to videos that are not aligned and captured in different environments. The key idea is to learn the alignment between the frames from two views using contrastive learning by using the temporally reversed version of the video as the negative sample. The manuscript also develops an object-centric encoder that focuses on the hands and active object regions in the frames. This allows the model to ignore the less informative regions such as background. The manuscript also develops a benchmark for evaluating the effectiveness of view-invariant representation approaches. Comparison of the proposed approach on the new benchmark with prior approaches clearly reveal its effectiveness.\n\nSTRENGTHS:\nThe manuscript addresses a problem that is not widely studied. Unlike prior view-invariant learning approaches that sets hard constraints such as availability of time-synchronized videos or videos captured in the same environment, the proposed approach can be successfully applied to unpaired videos captured at different environments. \n\nThe benchmark developed could also be useful to the research community.\n\nThe manuscript is written in a clear way explaining the various motivations behind the key ideas and the methodology used. \n\nThe manuscript also presents an extensive set of experiments to validate its effectiveness by comparing against existing approaches. Moreover, most of the design choices adopted in the proposed approach are also validated with empirical analysis.\n\nWEAKNESSES:\nOne of the contributions of the manuscript is the use of time reversed frames as the negative sample for contrastive learning instead of random shuffling. However, there is no experimental validation of the proposed negative sampling strategy against random shuffling.\n\nContinuing from the above, the proposed negative sampling strategy could affect actions that are constituted of repetitive subactions such as cutting something, playing drums, etc. where the time reversed frames could look more or less similar to its original version. The authors mention in the supplementary that for videos containing non monotonic actions (pouring water) only the first/last half of the frames are reversed. However, it is not clear even such an approach would work on repetitive actions. It seems the proposed negative strategy will work only on a selected set of actions.\n\nQUESTIONS:\nPlease see the weaknesses section for the main queries.\n\nThe learnable spatial embedding $\\textbf{e}_{spatial}$ is mentioned as the embedding to encode the bounding box coordinates and the confidence score obtained from the hand-object detector. It is not clear how this is obtained for the global image feature $\\phi _{project_g}(g_i)$.\n\nSome implementation details such as how the base encoder is initialized and whether it is frozen/finetuned during training are missing.\n\nLIMITATIONS:\nThe manuscript has adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}