SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Compared with frame-level negatives, our formulation is inherently more robust to repetitive and background frames, allowing for a degree of temporal variation within a video. This principle guides two key designs: (1) In creating negative samples, we opt for reversing the frames in S rather than randomly shuffling them. The latter approach may inadvertently generate a plausible (hence, not negative) temporal sequence, particularly when dealing with short videos abundant in background frames. (2) To obtain informative frames as the positive sample, we leverage the hand-object detection results from Sec. 3.3 to perform weighted sampling of the video sequences S and V. Specifically, we sample frames proportionally to their average confidence scores of hand and object detections. The subsampled S and V thus compose a positive pair. This way, frames featuring clear hand and object instances are more likely to be included.
3.5 Training and Implementation Details
Our final loss is a combination of the DTW alignment loss (Eqn. (3)) and the contrastive regularization loss (Eqn. (4)):
$$
\mathcal {L} (\mathbf {X}, \mathbf {Y}) = \mathcal {L} _ {\text {a l i g n}} (\mathbf {X}, \mathbf {Y}) + \lambda \mathcal {L} _ {\text {r e g}} (\mathbf {X}, \mathbf {Y}), \tag {5}
$$
where  $\lambda$  is a hyper-parameter controlling the ratio of these two loss terms.
During training, we randomly extract 32 frames from each video to construct a video sequence. The object-centric encoder network  $\phi$ , presented in Sec. 3.3, is optimized to minimize the loss above for all pairs of video sequences in the training set, including ego-ego, exo-exo, and ego-exo pairs<sup>3</sup>—all of which are known to contain the same coarse action. During evaluation, we freeze the encoder  $\phi$  and use it to extract 128-dimensional embeddings for each frame. These representations are then assessed across a variety of downstream tasks (Sec. 4). See Supp. for full implementation details.
4 Datasets and Evaluation
Datasets Existing datasets for fine-grained video understanding (e.g. [82, 65, 3]) are solely composed of third-person videos. Hence we propose a new ego-exo benchmark by assembling ego and exo videos from five public datasets—CMU-MMAC [13], H2O [32], EPIC-Kitchens [11], HMDB51 [31]
<sup>3</sup>In Sec. 3.1, we initially denote  $\mathbf{X}$  and  $\mathbf{Y}$  as ego and exo embeddings. However, in this context, we extend the meaning of  $\mathbf{X}$  and  $\mathbf{Y}$  to represent any pair of embeddings from the training set.
Figure 3: Example labels for Pour Liquid (left) and Tennis Forehand (right). Key events are displayed in boxes below sequences, with the phase label assigned to each frame between two key events.
and Penn Action [82]—plus a newly collected ego tennis forehand dataset. Our benchmark consists of four action-specific ego-exo datasets: (A) Break Eggs: ego-exo videos of 44 subjects breaking eggs, with 118/30 train/val+test samples from CMU-MMAC [13]; (B) Pour Milk [32]: ego-exo videos of 10 subjects pouring milk, with 77/35 train/val+test from H2O [32]; (C) Pour Liquid: we extract ego "pour water" clips from EPIC-Kitchens [11] and exo "pour" clips from HMDB51 [31], yielding 137/56 train/val+test. (D) Tennis Forehand: we extract exo tennis forehand clips from Penn Action [82] and collect ego tennis forehands by 12 subjects, totaling 173/149 train/val+test clips. We obtain ground truth key event and action phase labels for all of them (used only in downstream evaluation), following the labeling methodology established in [15]. Fig. 3 illustrates examples of these labels. Altogether, these four datasets provide unpaired ego-exo videos covering both tabletop (i.e., breaking eggs, pouring) and physical actions (i.e., tennis forehand). While (A) and (B) are from more controlled environments, where exo viewpoints are fixed and both views are collected in the same environment, (C) and (D) represent in-the-wild scenarios, where ego and exo are neither synchronized nor collected in the same environment. Our model receives them all as unpaired.
Evaluation We aim to demonstrate the practical utility of the learned ego-exo view-invariant features by addressing two questions: (1) how well do the features capture the progress of the action? (2) how view-invariant are the learned representations? To this end, we test four downstream tasks: (1) Action Phase Classification: train an SVM classifier on top of the embeddings to predict the action phase labels for each frame and report F1 score. We also explore few-shot and cross-view zero-shot settings (i.e., "exo2ego" when training the SVM classifier on exo only and testing on ego, and vice versa). (2) Frame Retrieval: parallels the classification task, but done via retrieval (no classifier), and evaluated with mean average precision  $(\mathrm{mAP})@K$ , for  $K = 5,10,15$ . We further extend this task to the cross-view setting to evaluate view-invariance. (3) Phase Progression: train a linear regressor on the frozen embeddings to predict the phase progression values, defined as the difference in time-stamps between any given frame and each key event, normalized by the number of frames in that video [15], evaluated by average R-square. (4) Kendall's Tau: to measure how well-aligned two sequences are in time, following [15, 22, 23, 10, 41].
Note that: (1) Departing from the common practice of splitting videos into train and validation only [15], our benchmark includes a dedicated test set to reduce the risk of model overfitting. (2) We opt for the F1 score instead of accuracy for action phase classification to better account for the label imbalance in these datasets and provide a more meaningful performance evaluation. (3) Phase progression assumes a high level of consistency in actions, with noisy frames diminishing the performance greatly. Due to the challenging nature of Pour Liquid data, we observe a negative progression value for all approaches. Thus, we augment the resulting embeddings with a temporal dimension, as 0.001 times the time segment as the input so that the regression model can distinguish repetitive (or very similar) frames that differ in time. We report modified progression value for all approaches on this dataset. (4) Kendall's Tau assumes that there are no repetitive frames in a video. Since we adopt in-the-wild videos where strict monoticty is not guaranteed, this metric may not faithfully reflect the quality of representations. Nonetheless, we report them for completeness and for consistency with prior work. (5) With our focus being frame-wise representation learning, the total number of training samples equals the number of frames rather than the number of videos, reaching a scale of a few thousands (8k-30k per action class). See Supp. for more dataset and evaluation details.
Baselines We compare AE2 with 8 total baselines of three types: (1) Naive baselines based on random features or ImageNet features, produced by a ResNet-50 model randomly initialized
<sup>4</sup>Although Break Eggs (and a portion of Pour Milk) offers synchronized ego-exo pairs, AE2 ignores the pairing and is trained in a self-supervised manner. AE2 never uses this synchronization as a supervision signal.
Table 1: Benchmark evaluation on four datasets: (A) Break Eggs; (B) Pour Milk; (C) Pour Liquid; (D) Tennis Forehand. Top results are highlighted in bold, while second-best results are underlined. AE2 performs best across all tasks and datasets, in both regular and cross-view scenarios.
Dataset	Method	Classification (F1 score)	Frame Retrieval (mAP@10)	Phase Progression	Kendall's Tau
regular	ego2exo	exo2ego	regular	ego2exo	exo2ego
(A)	Random Features	19.18	18.93	19.45	47.13	41.74	38.19	-0.0572	0.0018
ImageNet Features	50.24	21.48	32.25	50.49	33.09	37.80	-0.1446	0.0188
ActorObserverNet [66]	36.14	36.40	31.00	50.47	42.70	41.29	-0.0517	0.0024
single-view TCN [63]	56.90	18.60	35.61	53.42	32.63	34.91	0.0051	0.1206
multi-view TCN [63]	59.91	48.65	56.91	58.83	47.04	52.68	0.2669	0.2886
CARL [10]	43.43	28.35	29.22	46.04	37.38	39.94	-0.0837	-0.0091
TCC [15]	59.84	54.17	52.28	58.75	61.11	62.03	0.2880	0.5191
GTA [22]	56.86	52.33	58.35	61.55	56.25	53.93	0.3462	0.4626
AE2 (ours)	66.23	57.41	71.72	65.85	64.59	62.15	0.5109	0.6316
(B)	Random Features	36.84	33.96	41.97	52.48	50.56	51.98	-0.0477	0.0050
ImageNet Features	41.59	39.93	45.52	54.09	27.31	43.21	-2.6681	0.0115
single-view TCN [63]	47.39	43.44	42.28	57.00	46.48	47.20	-0.3238	-0.0197
CARL [10]	48.79	52.41	43.01	55.01	52.99	51.51	-0.1639	0.0443
TCC [15]	77.91	72.29	81.07	80.97	75.30	80.27	0.6665	0.7614
GTA [22]	81.11	74.94	81.51	80.12	72.78	75.40	0.7086	0.8022
AE2 (ours)	85.17	84.73	82.77	84.90	78.48	83.41	0.7634	0.9062
(C)	Random Features	45.26	47.45	44.33	49.83	55.44	55.75	-0.1303	-0.0072
ImageNet Features	53.13	22.44	44.61	51.49	52.17	30.44	-1.6329	-0.0053
single-view TCN [63]	54.02	32.77	51.24	48.83	55.28	31.15	-0.5283	0.0103
CARL [10]	56.98	47.46	52.68	55.29	59.37	36.80	-0.1176	0.0085
TCC [15]	52.53	43.85	42.86	62.33	56.08	57.89	0.1163	0.1103
GTA [22]	56.92	42.97	59.96	62.79	58.52	53.32	-0.2370	0.1005
AE2 (ours)	66.56	57.15	65.60	65.54	65.79	57.35	0.1380	0.0934
(D)	Random Features	30.31	33.42	28.10	66.47	58.98	59.87	-0.0425	0.0177
ImageNet Features	69.15	42.03	58.61	76.96	66.90	60.31	-0.4143	0.0734
single-view TCN [63]	68.87	48.86	36.48	73.76	55.08	56.65	-0.0602	0.0737
CARL [10]	59.69	35.19	47.83	69.43	54.83	63.19	-0.1310	0.0542
TCC [15]	78.41	53.29	32.87	80.24	55.84	47.27	0.2155	0.1040
GTA [22]	83.63	82.91	81.80	85.20	78.00	79.14	0.4691	0.4901
AE2 (ours)	85.87	84.71	85.68	86.83	81.46	82.07	0.5060	0.6171
or pretrained on ImageNet [14]; (2) Self-supervised learning approaches specifically designed for learning ego-exo view-invariant features, time-contrastive networks (TCN) [63] and ActorObserverNet [66]. Both require synchronized ego-exo videos for training, and are thus only applicable to Break Eggs data. Additionally, TCN offers a single-view variant that eliminates the need for synchronized data, by taking positive frames within a small temporal window surrounding the anchor, and negative pairs from distant timesteps. We implement single-view TCN for all datasets. (3) Fine-grained action representation learning approaches, CARL [10] which utilizes a sequence contrastive loss, and TCC [15] and GTA [22], which also utilize alignment as the pretext task but do not account for ego-exo viewpoint differences.
Main Results In Table 1, we benchmark all baseline approaches and AE2 on four ego-exo datasets. See Supp. for full results, including the F1 score for few-shot classification, and mAP@5,15 for frame retrieval. From the results, it is clear that AE2 greatly outperforms other approaches across all downstream tasks. For instance, on Break Eggs, AE2 surpasses the multi-view TCN [63], which utilizes perfect ego-exo synchronization as a supervision signal, by  $+6.32\%$  in F1 score. Even when we adapt the TCN to treat all possible ego-exo pairs as perfectly synchronized, AE2 still excels, showing a considerable margin of improvement (see Supp. for a discussion). Pour Liquid poses the most substantial challenge due to its in-the-wild nature, as reflected by the low phase progression and Kendall's Tau values. Yet, AE2 notably improves on previous works, achieving, for instance, a  $+9.64\%$  F1 score. Regarding Tennis Forehand, another in-the-wild dataset, we note that some methods like TCC perform well in identifying correspondences within either the ego or exo domain, but struggle to connect the two. Consequently, while it achieves an  $80.24\%$  mAP@10 for standard frame retrieval, it falls below  $60\%$  for cross-view frame retrieval. In contrast, AE2 effectively bridges these two significantly different domains, resulting in a high performance of over  $81\%$  mAP@10
Figure 4: Visualization of (a) the original video sequence; (b) a randomly shuffled sequence; and (c) a temporally reversed sequence on Break Eggs (left) and Pour Liquid (right). The randomly shuffled sequence may still preserve the action progression order due to many similar frames in the video sequence, failing to be truly "negative". In contrast, temporally reversing frames results in a more distinct and suitable negative example.
in both standard and cross-view settings. These results demonstrate the efficacy of AE2 in learning fine-grained ego-exo view-invariant features.
Ablation Table 2 provides an ablation of the two design choices of AE2. Starting with a baseline approach (denoted as "Base DTW" in the table), we optimize a ResNet-50 encoder with respect to a DTW alignment objective as defined in Eqn. (3). Next, we replace the feature encoder with our proposed object-centric encoder design and report its performance. Finally, we incorporate the contrastive regularization from Eqn. (4) into the objective. It is evident that object-centric representations are instrumental in bridging the ego-exo gap, leading to substantial performance improvements. Furthermore, incorporating contrastive regularization provides additional gains.
Negative Sampling We conduct experiments to validate the efficacy of creating negative samples by temporally reversing frames, as opposed to random shuffling and report results in Table 3. In general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. The inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to a positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur in real
videos), temporally reversing frames is apt to create a negative sample that deviates from the correct action progression order. To further illustrate this point, we visualize two video sequences in Fig. 5. It is evident that the randomly shuffled sequence seems to preserve the sequence of actions like breaking eggs or pouring liquid, thereby resembling a "positive" example. Consequently, incorporating such negative samples into training may confuse the model and lead to diminished performance.
Table 2: Ablation study of AE2 on four datasets (F1 score). See Supp. for other metrics.
Dataset
(A)	(B)	(C)	(D)
Base DTW	58.53	82.91	59.66	79.56
+ object	62.86	84.04	63.28	84.14
+ object + contrast	66.23	85.17	66.56	85.87
Table 3: Comparison of AE2 employing randomly shuffled frames versus temporally reversed frames as the negative samples (Frame Retrieval mAP@10). See Supp. for other metrics.
Dataset
(A)	(B)	(C)	(D)
Random Shuffle	61.66	81.92	63.48	86.73
Temporal Reverse	65.85	84.90	65.54	86.83
Qualitative Results Fig. 5 (left) shows examples of cross-view frame retrieval on Pour Liquid and Tennis Forehand datasets. Given a query image from one viewpoint in the test set, we retrieve its nearest neighbor among all test frames in another viewpoint. We observe that the retrieved frames closely mirror the specific action state of the query frame. For example, when the query is a frame
Pre-pour: lifting the container
Retrieved Nearest Neighbors (ego)
Active Pour: liquid exiting the container
Query (ego)
Retrieved Nearest Neighbors (exo)
Pre-stroke: racket poised to strike ball
Post-stroke: following through after the ball is hit
Figure 5: Qualitative results of AE2. Left: Cross-view frame retrieval results on Pour Liquid and Tennis Forehand. Our view-invariant embeddings can be effectively used for fine-grained frame retrieval; Right: tSNE trajectories of 4 test videos (2 ego and 2 exo) in the embedding space on Break Eggs. Shading indicates time. The learned embeddings successfully capture the progress of an action and maintain invariance across the ego and exo viewpoints. See Supp. for more examples.
capturing the moment before the pouring action, where the liquid is still within the container (row 1), or when the query shows pouring actively happening (row 2), the retrieved exo frames capture that exact step as well. Fig. 5 (right) shows tSNE visualizations of embeddings for 4 test videos on Break Eggs. Our learned embeddings effectively capture the progress of an action, while remaining view-invariant. These learned embeddings enable us to transfer the pace of one video to others depicting the same action. We demonstrate this capability with examples of both ego and exo videos, collected in distinct environments, playing synchronously in the Supp. video.
Limitations While AE2 has demonstrated its ability to handle in-the-wild data, there remains much room for improvement, particularly on challenging in-the-wild datasets such as Pour Liquid. Here, we observe instances of failure in cross-view frame retrieval and synchronizing the pace of two videos. Although we highlight successful examples in our discussion, it is important to note that some attempts have been unsuccessful, largely due to the inherent noise in real-world videos. This represents a key area for future refinement and development of AE2.
We tackle fine-grained temporal understanding by jointly learning from unpaired egocentric and exocentric videos. Our core idea is to leverage temporal alignment as the self-supervised learning objective for invariant features, together with a novel object-centric encoder and contrastive regularizer that reverses frames. On four datasets we achieve superior performance compared to state-of-the-art ego-exo and alignment approaches. This fundamental technology has great promise for both robot learning and human skill learning in AR, where an ego actor needs to understand (or even match) the actions of a demonstrator that they observe from the exo point of view. Future work includes generalizing AE2 to train from pooled data of multiple actions, extending its applicability to more complex, longer action sequences and exploring its impact for robot imitation learning.
UT Austin is supported in part by the IFML NSF AI Institute. KG is paid as a research scientist at Meta. The authors extend heartfelt thanks to the following individuals from UT Austin for their vital role in collecting the ego-tennis forehand dataset: Bowen Chen, Zhenyu Jiang, Hanwen Jiang, Xixi Hu, Ashutosh Kumar, Huihan Liu, Bo Liu, Mi Luo, Sagnik Majumder, Shuhan Tan, and Yifeng Zhu. We are also grateful to Zhengqi Gao (MIT), Xixi Hu (UT Austin) and Yue Zhao (UT Austin), for their constructive discussions and feedback.
[79] Yuncong Yang, Jiawei Ma, Shiyuan Huang, Long Chen, Xudong Lin, Guangxing Han, and Shih-Fu Chang. TempCLR: Temporal alignment representation with contrastive learning. arXiv preprint arXiv:2212.13738, 2022.
[80] Huangyue Yu, Minjie Cai, Yunfei Liu, and Feng Lu. What i see is what you see: Joint attention learning for first and third person video co-analysis. ACM MM, 2019.
[81] Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, and Nanning Zheng. View adaptive recurrent neural networks for high performance human action recognition from skeleton data. In Proceedings of the IEEE international conference on computer vision, pages 2117-2126, 2017.
[82] Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly-supervised representation for detailed action understanding. In Proceedings of the IEEE international conference on computer vision, pages 2248-2255, 2013.
[83] Zhong Zhang, Chunheng Wang, Baihua Xiao, Wen Zhou, Shuang Liu, and Cunzhao Shi. Cross-view action recognition via a continuous virtual path. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2690-2697, 2013.
[84] Honglu Zhou, Roberto Martín-Martín, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. arXiv preprint arXiv:2303.18230, 2023.
[85] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision-based manipulation with object proposal priors. arXiv preprint arXiv:2210.11339, 2022.
[86] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramadan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545, 2019.
A Video Containing Qualitative Results
We invite the reader to view the video available at https://vision.cs.utexas.edu/projects/ AlignEgoExo/, where we show qualitative examples of one practical application enabled by our learned ego-exo view-invariant representations—synchronous playback of egocentric and exocentric videos. We randomly select ego-exo video pairs from the test set, and use the frozen encoder  $\phi$  to extract frame-wise embeddings. We then match each frame of one video (the reference) to its closest counterpart in the other video using nearest neighbor. As demonstrated across all datasets with several examples, AE2 effectively aligns two videos depicting the same action, overcoming substantial viewpoint and background differences. The video also includes examples of synchronizing two egocentric or two exocentric videos.
Since existing video datasets for fine-grained video understanding (e.g., PennAction [82], FineGym [65], IKEA ASM [3]) are solely composed of third-person videos, we curate video clips from five public datasets: Break Eggs [13], H2O [32], EPIC-Kitchens [11], HMDB51 [31] and Penn Action [82] and collect a egocentric Tennis Forehand dataset. Our data selection criteria is to find videos that depict the same action from distinct egocentric and exocentric viewpoints. Consequently, TCN pouring [63] is excluded due to its small scale and significant similarity between the egocentric and exocentric views; Assembly101 [62] is not selected since there there are large variations within a single atomic action and the egocentric video is monochromatic.
In all, our dataset compilation from five public data sources, along with our collected egocentric tennis videos, results in four distinct ego-exo datasets, each describing specific atomic actions:
(A) Break Eggs. The dataset contains 44 subjects cooking five different recipes (brownies, pizza, sandwich, salad, and scrambled eggs), captured from one egocentric and four exocentric views simultaneously in CMU-MMAC [13]. We use the temporal keystep boundaries provided in [2] to extract clips corresponding to the action of breaking eggs from all videos. Among the four exocentric viewpoints, we adopt videos from the right-handed view, as this viewpoint captures the action being performed most clearly. We randomly split the data into training and validation sets across subjects, with 35 subjects (118 videos) for training and 9 subjects (30 videos) for validation and test. There is strict synchronization between egocentric and exocentric video pairs in this dataset.
(B) Pour Milk. The dataset features 10 subjects interacting with a milk carton using both hands, captured by one egocentric camera and four static exocentric cameras in H2O [32]. We utilize the keystep annotations provided in [33] to extract clips corresponding to the pouring milk action. All four exocentric views are included, as they clearly capture the action. We follow the data split in [33], with 7 subjects (77 videos) in the training set and 3 subjects (35 videos) in the validation and test set. A portion of this dataset (the first 3 subjects) contains synchronized egocentric-exocentric video pairs, while the remaining part does not.
(C) Pour Liquid. To evaluate our methods on in-the-wild data, we assemble a Pour Liquid dataset by extracting clips from one egocentric dataset, EPIC-Kitchens [11] and one exocentric dataset, HMDB51 [31]. We utilize clips from the "pour water" class in EPIC-Kitchens and "pour" category in HMDB51. Following the data split in the original datasets, we obtain 137 videos for training and 56 videos for validation and test. It is important to note that
Table 4: Dataset summary.
Dataset	# Train	# Val	# Test	Fixed exo view?	Ego-exo time-sync?
ego	exo	ego	exo	ego	exo
(A) Break Eggs	61	57	5	5	10	10	✓	✓
(B) Pour Milk	29	48	4	8	7	16	✓	✗
(C) Pour Liquid	70	67	10	9	19	18	✗	✗
(D) Tennis Forehand	94	79	25	24	50	50	✗	✗
the egocentric and exocentric videos are neither synchronized nor collected in the same environment, providing a challenging testbed.