SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(D) Tennis Forehand. To include physical activities in our study, we leverage exocentric video sequences of the tennis forehand action from Penn Action [82] and collect an egocentric dataset featuring the same action performed by 12 subjects using Go Pro HERO8. We adopt the data split from [15] for Penn Action, and divide our egocentric tennis forehand dataset by subject: 8 for training and 4 for validation and testing. This results in a total of 173 clips for training, and 149 clips for validation and testing. The egocentric and exocentric videos, gathered from a range of real-world scenarios, are naturally unpaired.
Table 4 provides a summary of these four datasets. In addition, we recognize that the original datasets lack frame-wise labels and provide dense frame-level annotations to enable a comprehensive evaluation of the learned representations. See Table 5 for a complete list of all the key events we annotate and Fig. 6 for illustrative examples. We will release our collected data and labels for academic usage.
Table 5: Number of actions phases and list of key events for each dataset.
Dataset	# phases	List of key events
(A) Break Eggs	4	hit egg, visible crack on the eggshell; egg contents released into bowl
(B) Pour Milk	3	liquid starts exiting, pouring complete
(C) Pour Liquid	3	liquid starts exiting, pouring complete
(D) Tennis Forehand	2	racket touches ball
B.2 Evaluation
We provide a detailed description of the four downstream tasks below and their corresponding evaluation metrics:
1. Action Phase Classification. We train an SVM classifier on top of the embeddings to predict the action phase labels for each frame and report F1 score on test data. Besides the regular setting, we investigate (1) few-shot; and (2) cross-view zero-shot settings.
(1) Few-shot. We assume that only a limited number of training videos have annotations and can be used for training the SVM classifier.
(2) Cross-view zero-shot. We assume that per-frame labels of training data are only available on one view for training the SVM classifier, and report the test performance on the other view. We use the terms "exo2ego" to describe the case where we use exocentric data for training the SVM classifier and test its performance on egocentric data, while "ego2exo" represents the reverse case.
2. Phase Progression. We train a linear regressor on the frozen embeddings to predict the phase progression values, defined as the difference in time-stamps between any given frame and each key event, normalized by the number of frames in that video [15]. Average R-square measure on test data is reported. This metric evaluates how well the progress of an action is captured by the embeddings, with the maximum value being 1.
3. Frame Retrieval. We report the mean average precision (mAP)@K (K=5,10,15). For each query, average precision is computed by determining how many frames among the retrieved K frames have the same action phase labels as the query frame, divided by K. Furthermore, to evaluate view-invariance, we propose the cross-view frame retrieval task (i.e., ego2exo and exo2ego frame retrieval). For each query in one view, the goal is to retrieve K frames from another view. No additional training is required for this task.
4. Kendall's Tau. This metric is calculated for every pair of test videos by sampling two frames in the first video and retrieving the corresponding nearest frames in the second video, then checking whether their orders are shuffled. It measures how well-aligned two sequences are in time. No additional training or frame-wise labels are necessary for this evaluation.
This dataset is owned by UT Austin and involves no participation from Meta.
Figure 6: Example labels for all datasets. Key events are displayed in boxes below sequences, with the phase label assigned to each frame between two key events.
Table 6: Hyperparameters summary. 'Lr' stands for learning rate, and 'Wd' denotes weight decay.
Datasets	Optimizer	Transformer Encoder	Regularization
Lr	Wd	Hidden Dim.	# Layers	# Frames	# Pos. Frames	Ratio λ
(A) Break Eggs	5e-5	1e-5	256	1	32	32	1
(B) Pour Milk	1e-4	1e-5	256	1	32	8	2
(C) Pour Liquid	5e-5	1e-5	128	3	32	16	2
(D) Tennis Forehand	5e-5	1e-5	128	1	20	10	4
For all video sequences, frames are resized to  $224 \times 224$ . During training, we randomly extract 32 frames from each video to construct a video sequence. We train the models for a total number of 300 epochs with a batch size of 4, using the Adam optimizer. The base encoder  $\phi_{\mathrm{base}}$  is initialized with a ResNet-50 pretrained on ImageNet, and jointly optimized with the transformer encoder  $\phi_{\mathrm{transformer}}$  throughout the training process. The model checkpoint demonstrating the best performance on validation data is selected, and its performance on test data is reported. In terms of the encoder network, global features are taken from the output of Conv4c layer in  $\phi_{\mathrm{base}}$ . Following [15], we stack the features of any given frame and its context frames along the dimension of time, followed by 3D convolutions for aggregating temporal information and 3D max pooling. In all experiments, the number and stride of context frames are set as 1 and 15, respectively. For local features, we take output of the Conv1 layer in  $\phi_{\mathrm{base}}$  and apply 3D max pooling to aggregate temporal information from the given frame and its context frame. The features are then fed as input of ROI Align.
Table 7: Results of few-shot action phase classification (F1 score) and frame retrieval (mAP@5,10,15).
Dataset	Method	Few-shot Cls.	Frame Retrieval
10%	50%	100%	mAP@5	mAP@10	mAP@15
(A)	Random Features	19.18	19.18	19.18	48.26	47.13	45.75
ImageNet Features	46.15	48.80	50.24	49.98	50.49	50.08
ActorObserverNet [66]	31.40	35.63	36.14	50.92	50.47	49.72
single-view TCN [63]	52.30	54.90	56.90	52.82	53.42	53.60
multi-view TCN [63]	56.88	59.25	59.91	59.11	58.83	58.44
multi-view TCN (unpaired) [63]	56.13	56.65	56.79	58.18	57.78	57.21
CARL [10]	39.18	41.92	43.43	47.14	46.04	44.99
TCC [15]	57.54	59.18	59.84	59.33	58.75	57.99
GTA [22]	56.89	56.77	56.86	62.79	61.55	60.38
AE2 (ours)	63.95	64.86	66.23	66.86	65.85	64.73
(B)	Random Features	36.84	36.84	36.84	52.94	52.48	51.59
ImageNet Features	39.29	40.83	41.59	53.32	54.09	54.06
single-view TCN [63]	43.60	46.83	47.39	56.98	57.00	56.46
CARL [10]	48.73	48.78	48.79	55.59	55.01	54.23
TCC [15]	78.69	77.97	77.91	81.22	80.97	80.46
GTA [22]	79.82	80.96	81.11	80.65	80.12	79.68
AE2 (ours)	85.17	85.12	85.17	85.25	84.90	84.55
(C)	Random Features	45.26	45.26	45.26	49.69	49.83	49.18
ImageNet Features	55.53	54.43	53.13	50.52	51.49	51.89
single-view TCN [63]	54.62	55.08	54.02	48.50	48.83	49.03
CARL [10]	51.68	55.67	56.98	55.03	55.29	54.93
TCC [15]	52.37	51.70	52.53	62.93	62.33	61.44
GTA [22]	55.91	56.87	56.92	62.83	62.79	62.12
AE2 (ours)	65.88	66.53	66.56	66.55	65.54	64.66
(D)	Random Features	31.54	30.31	30.31	69.57	66.47	64.34
ImageNet Features	65.48	68.03	69.15	78.11	76.96	75.84
single-view TCN [63]	65.78	69.19	68.87	74.05	73.76	73.10
CARL [10]	58.89	59.38	59.69	72.94	69.43	67.14
TCC [15]	67.71	77.07	78.41	82.78	80.24	78.59
GTA [22]	80.31	83.04	83.63	86.59	85.20	84.33
AE2 (ours)	85.24	85.72	85.87	87.94	86.83	86.05
During evaluation, we freeze the encoder  $\phi$  and use it to extract 128-dimensional embeddings for each frame. These representations are then assessed across a variety of downstream tasks (Sec. 4). Detailed hyperparameters specific to each dataset are provided in Table 6. Noteworthy adjustments include: (1) In the case of Tennis Forehand, we utilize a single object proposal, as the active object is only the tennis racket (the tennis ball is too small to be detected reliably). Furthermore, given the shorter video lengths, we sample 20 frames from each video as opposed to the usual 32. (2) For datasets featuring non-monotonic actions (i.e., Pour Milk and Pour Liquid), we construct the negative sequence by randomly reversing either the first or the last half of the sequence, rather than the whole sequence. This is due to the cyclic nature of the pouring action present in some videos within these datasets. All experiments are conducted using PyTorch [52] on 2 Nvidia V100 GPUs.
C Further Results and Visualizations
Results Supplementing Table 1 in the main paper, Tables 7 and 8 present comprehensive results of AE2 and baseline models on few-shot action phase classification and frame retrieval tasks. For few-shot classification, we train the SVM classifier with  $10\%$  (or  $50\%$ ) of the training data, averaging results over 10 runs. AE2 demonstrates superior performance in learning fine-grained, view-invariant ego-exo features when compared with ego-exo [66, 63], frame-wise contrastive learning [10], and alignment-based [15, 22] approaches.
On Break Eggs, AE2 greatly outperforms the multi-view TCN [63], which utilizes perfect ego-exo synchronization as a supervision signal. We hypothesize that the strict supervision requirement of
Table 8: Results of cross-view frame retrieval (mAP@5,10,15).
Dataset	Method	Ego2exo Frame Retrieval	Exo2ego Frame Retrieval
mAP@5	mAP@10	mAP@15	mAP@5	mAP@10	mAP@15
(A)	Random Features	42.51	41.74	40.51	38.08	38.19	37.10
ImageNet Features	33.32	33.09	32.78	38.99	37.80	36.71
ActorObserverNet [66]	43.57	42.70	41.56	42.00	41.29	40.48
single-view TCN [63]	31.12	32.63	33.73	34.67	34.91	35.31
multi-view TCN [63]	46.38	47.04	46.96	52.50	52.68	52.43
multi-view TCN (unpaired) [63]	55.34	54.64	53.75	58.79	57.87	57.07
CARL [10]	37.89	37.38	36.57	40.37	39.94	39.38
TCC [15]	62.11	61.11	60.33	62.39	62.03	61.25
GTA [22]	57.11	56.25	55.10	54.47	53.93	53.22
AE2 (ours)	65.70	64.59	63.76	62.48	62.15	61.80
(B)	Random Features	51.46	50.56	48.93	52.78	51.98	50.82
ImageNet Features	25.72	27.31	28.57	41.50	43.21	43.06
single-view TCN [63]	47.00	46.48	45.42	47.94	47.20	46.59
CARL [10]	54.35	52.99	51.99	51.14	51.51	51.00
TCC [15]	75.54	75.30	75.02	80.44	80.27	80.18
GTA [22]	72.55	72.78	72.96	75.16	75.40	75.48
AE2 (ours)	78.21	78.48	78.78	83.88	83.41	83.05
(C)	Random Features	55.78	55.44	54.77	56.31	55.75	54.56
ImageNet Features	51.44	52.17	52.38	30.18	30.44	30.40
single-view TCN [63]	53.60	55.28	55.46	29.16	31.15	31.95
CARL [10]	59.59	59.37	59.19	34.73	36.80	38.10
TCC [15]	55.98	56.08	56.13	58.11	57.89	57.15
GTA [22]	57.03	58.52	59.00	51.71	53.32	53.54
AE2 (ours)	66.23	65.79	65.00	57.42	57.35	57.03
(D)	Random Features	61.24	58.98	56.94	63.42	59.87	57.57
ImageNet Features	69.34	66.90	64.95	61.61	60.31	58.55
single-view TCN [63]	54.12	55.08	55.05	56.70	56.65	55.84
CARL [10]	52.18	54.83	55.39	65.94	63.19	60.83
TCC [15]	57.87	55.84	53.81	48.62	47.27	46.11
GTA [22]	78.93	78.00	77.01	79.95	79.14	78.52
AE2 (ours)	82.58	81.46	80.75	82.82	82.07	81.69
Figure 7: Few-shot action phase classification results. AE2 achieves superior performance across a wide range of labeled training videos, particularly under the most challenging conditions where less than 10 videos are labeled. Results are averaged over 50 runs, and confidence bars represent one standard deviation.
TCN might be limiting, as it can not utilize as many ego-exo pairs as AE2 due to its reliance on ego-exo synchronization. In contrast, AE2 capitalizes on a broader set of unpaired ego-exo data. Even when we modify multi-view TCN to consider all potential ego-exo pairs as synchronously perfect (termed as multi-view TCN (unpaired) in the tables), it does not outperform its regular version, indicating a lack of robustness towards non-synchronized ego-exo pairs. Consequently, it appears that multi-view TCN is ill-equipped to learn desired view-invariant representations from unpaired, real-world ego-exo videos.
Table 9: Ablation Study of AE2.
Dataset	Method	Classification (F1 score)	Frame Retrieval (mAP@10)	Phase Progression	Kendall's Tau
regular	ego2exo	exo2ego	regular	ego2exo	exo2ego
(A)	Base DTW	58.53	57.78	54.23	58.36	55.36	58.95	0.1920	0.5641
+ object	62.86	60.88	58.52	62.66	61.26	60.69	0.4235	0.5484
+ object + contrast	66.23	57.41	71.72	65.85	64.59	62.15	0.5109	0.6316
(B)	Base DTW	82.91	81.82	81.83	81.49	74.63	80.21	0.7525	0.8199
+ object	84.04	84.20	84.23	83.03	81.42	81.57	0.7646	0.8886
+ object + contrast	85.17	84.73	82.77	84.90	78.48	83.41	0.7634	0.9062
(C)	Base DTW	59.66	55.48	59.49	52.57	54.12	52.23	0.0553	0.0609
+ object	63.28	57.60	62.42	63.40	67.15	63.05	0.2231	0.1339
+ object + contrast	66.56	57.15	65.60	65.54	65.79	57.35	0.1380	0.0934
(D)	Base DTW	79.56	81.38	72.54	82.65	75.36	76.74	0.4022	0.4312
+ object	84.14	85.36	83.32	88.22	79.07	82.61	0.5431	0.6477
+ object + contrast	85.87	84.71	85.68	86.83	81.46	82.07	0.5060	0.6171
Few-shot Learning Besides the few-shot results in Table 7, we vary the number of labeled training videos, ranging from extremely sparse (a single labeled video) to the case where all training videos are labeled. Note that each labeled video equates to multiple labeled frames. AE2 is compared with top-performing baselines, TCC [15] and GTA [22], across all four datasets in Fig. 7. The results are averaged over 50 runs and include a  $+ -$  one standard deviation error bar. As shown, AE2 excels in low-label scenarios. For instance, on Pour Milk, a single labeled video yields an action phase classification F1 score over  $80\%$ . This suggests that AE2 effectively aligns representations across all training videos, enabling a robust SVM classifier for the downstream task, even with minimal labeling.
Ablation Table 9 presents an ablation of AE2 on all four datasets, which is a comprehensive version of Table 2 in the main paper. From the results, we can see that object-centric representations are instrumental in bridging the ego-exo gap, leading to substantial performance improvements. For instance, frame retrieval mAP@10 improves by  $+10.83\%$  on Pour Liquid and  $+5.57\%$  on Tennis Forehand. Furthermore, incorporating contrastive regularization provides additional performance boosts for several downstream tasks such as regular action phase classification. These results demonstrate the integral contributions of both components of AE2 to achieve optimal performance.
Negative Sampling (Temporal Reverse versus Random Shuffle) Table 10 presents a comparison of two negative sampling strategies (random shuffling versus temporal reversing), which is a comprehensive version of Table 3 in the main paper. The results reveal that, in general, temporally reversing frames yields superior and more consistent performance than randomly shuffling. For example, on Break Eggs data, random shuffling results in a decrease of  $-5.62\%$  in F1 score and  $-4.19\%$  in mAP@10 compared with regular AE2. The inferior performance of random shuffling can be related to the abundance of similar frames within the video sequence. Even after shuffling, the frame sequence may still emulate the natural progression of an action, thereby appearing more akin to a positive sample. Conversely, unless the frame sequence is strictly symmetric (a scenario that is unlikely to occur in real videos), temporally reversing frames is apt to create a negative sample that deviates from the correct action progression order.
To dissect the performance gains further, the large improvement observed on dataset (A) when using temporally reversed frames over randomly shuffled frames stems from its inherent ability to be more robust to the repetitive nature of the breaking egg action (e.g., the camera wearer may hit the egg twice before breaking it, resulting in many similar frames in the video sequence). On the other hand, the subtler improvement on dataset (D), dealing with the tennis forehand action, is likely due to the strictly monotonic nature of this action, leaving less room for our method to outperform. However, it is essential to emphasize that this does not diminish the value of our approach, our negative sampling strategy still proves preferable consistently across all four datasets.
Negative Sampling (Sequence-level versus Frame-level) We emphasize the unique advantages of our proposed sequence-level negatives over the frame-level negatives used in prior works [63, 66, 10].
Figure 8: Visualization of a video sequence (from CMU-MMAC) exhibiting a certain amount of periodicity, where the camera wearer hits the egg to the bowl's edge twice. In the upper figure, frame-level negatives are defined based on temporal distance, leading to the possibility of two visually similar frames (marked by a red border) being misclassified as a "negative" pair. In contrast, the lower figure illustrates how sequence-level negatives provide a more robust solution to accommodate the temporal variations within a video.
Table 10: Comparison of AE2 employing randomly shuffled frames versus temporally reversed frames as the negative samples.
Dataset	Method	Classification (F1 score)	Frame Retrieval (mAP@10)	Phase Progression
regular	ego2exo	exo2ego	regular	ego2exo	exo2ego
(A)	Random Shuffle	60.61	58.63	54.46	61.66	57.57	59.61	0.3385
Temporal Reverse	66.23	57.41	71.72	65.85	64.59	62.15	0.5109
(B)	Random Shuffle	82.05	82.74	79.48	81.92	80.60	79.71	0.6659
Temporal Reverse	85.17	84.73	82.77	84.90	78.48	83.41	0.7634
(C)	Random Shuffle	68.13	64.96	58.44	63.48	62.73	60.27	-0.0121
Temporal Reverse	66.56	57.15	65.60	65.54	65.79	57.35	0.1380
(D)	Random Shuffle	84.12	84.83	82.59	86.73	80.49	83.75	0.5304
Temporal Reverse	85.87	84.71	85.68	86.83	81.46	82.07	0.5060
While periodic actions present great challenges for all negative sampling techniques, sequence-level negatives offers a more robust solution to tackle the intricacies of repetitiveness within an action. As a motivating example, Fig. 8 depicts a video sequence of breaking eggs with a certain amount of periodicity (where the camera wearer hits the egg to the bowl's edge twice). Frame-level sampling approaches treat temporally close frames as positive pairs and temporally distant ones as negative pairs. As a consequence, the two visually similar frames (marked by a red border) would be incorrectly identified as a negative pair by those existing methods. Contrarily, our method of constructing negative samples adopts a global perspective, addressing temporal fluctuations within a video more effectively. By reversing the entire sequence, we create a challenge in aligning the reversed ego view video with the original exo view video, thus forming a valid negative sample.
This methodology is not only conceptually appealing but also empirically validated. Table 1 in the main paper illustrates the effectiveness of our proposed AE2, demonstrating that it outperforms prior techniques utilizing frame-wise negatives [63, 66, 10]. These results offer compelling empirical evidence that our sequence-level negative sampling strategy successfully mitigates the issues related to periodicity and improves performance, affirming its superiority over conventional frame-by-frame sampling methods.
Visualizations In addition to the cross-view frame retrieval results for Pour Liquid and Tennis Forehand presented in the main paper (Fig. 5), we showcase results for the other two datasets (i.e., Break Eggs and Pour Milk) in Fig. 9. For any given query frame from one view, the retrieved nearest neighbors closely match the action stage of the query, regardless of substantial differences in viewpoints. These results underline AE2's efficacy in learning fine-grained action representations that transcend ego-exo viewpoint differences.
Pre-pour: milk carton at a distance from the cup, upright
Imminent pour: milk carton closer to the cup, tilted
Active pour: milk carton touching the cup, nearly vertical
Query (exo)
Pre-crack: egg tapped on bowl's edge
Mid-crack: preparing to open, egg remains whole
Post-crack: eggshell cracked, contents released
Figure 9: Cross-view frame retrieval results on Pour Milk (rows 1-3) and Break Eggs (rows 4-6). AE2 leads to representations that encapsulate the fine-grained state of an action and are invariant to the ego-exo viewpoints.