SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihui Xue<sup>1,2</sup> Kristen Grauman<sup>1,2</sup>
1The University of Texas at Austin 2FAIR,Meta
The egocentric and exocentric viewpoints of a human activity look dramatically different, yet invariant representations to link them are essential for many potential applications in robotics and augmented reality. Prior work is limited to learning view-invariant features from paired synchronized viewpoints. We relax that strong data assumption and propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time, even when not captured simultaneously or in the same environment. To this end, we propose AE2, a self-supervised embedding approach with two key designs: (1) an object-centric encoder that explicitly focuses on regions corresponding to hands and active objects; and (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context, comprising four datasets—including an ego tennis forehand dataset we collected, along with dense per-frame labels we annotated for each dataset. On the four datasets, our AE2 method strongly outperforms prior work in a variety of fine-grained downstream tasks, both in regular and cross-view settings.<sup>1</sup>
Fine-grained video understanding aims to extract the different stages of an activity and reason about their temporal evolution. For example, whereas a coarse-grained action recognition task [31, 16, 68, 20, 7] might ask, "is this sewing or snowboarding or...?", a fine-grained action recognition task requires detecting each of the component steps, e.g., "the person threads the needle here, they poke the needle through the fabric here..." Such fine-grained understanding of video is important in numerous applications that require step-by-step understanding, ranging from robot imitation learning [42, 63] to skill learning from instructional "how-to" videos [86, 73, 47, 46].
The problem is challenging on two fronts. First, the degree of detail and precision eludes existing learned video representations, which are typically trained to produce global, clip-level descriptors of the activity [69, 74, 18, 5]. Second, in many scenarios of interest, the camera viewpoint and background will vary substantially, making the same activity look dramatically different. For example, imagine a user wearing AR/VR glasses who wants to compare the egocentric recording of themselves doing a tennis forehand with a third-person (exocentric) expert demonstration video on YouTube, or a robot tasked with pouring drinks from its egocentric perspective but provided with multiple third-person human pouring videos for learning. Existing view-invariant feature learning [76, 27, 78, 45, 57, 58]—including methods targeting ego-exo views [63, 66]—assume that training data contains synchronized video pairs capturing the same action simultaneously in the same environment. This is a strong assumption. It implies significant costs and physical constraints that can render these methods inapplicable in complex real-world scenarios.
<sup>1</sup>Project webpage: https://vision.cs.utexas.edu/projects/AlignEgoExo/.
Figure 1: Left: We propose AE2, a self-supervised approach for learning frame-wise action features that are invariant to the egocentric and exocentric viewpoints. The key is to find correspondences across the viewpoints via temporal alignment. Right: Before training, the frame embeddings are clustered by viewpoint (pink and green); after training, our learned AE2 embeddings effectively capture the progress of an action and exhibit viewpoint invariance. Shading indicates time.
Our goal is to learn fine-grained frame-wise video features that are invariant to both the ego and exo views, from unpaired data. In the unpaired setting, we know which human activity occurs in any given training sequence (e.g., pouring, breaking eggs), but they need not be collected simultaneously or in the same environment. The main idea of our self-supervised approach is to infer the temporal alignment between ego and exo training video sequences, then use that alignment to learn a view-invariant frame-wise embedding. Leveraging unpaired data means our method is more flexible—both in terms of curating training data, and in terms of using available training data more thoroughly, since with our approach any ego-exo pairing for an action becomes a training pair. Unlike existing temporal alignment for third-person videos [15, 54, 22, 23, 41], where canonical views are inherently similar, ego-exo views are so distinct that finding temporal visual correspondences is non-trivial (see Fig. 1).
To address this challenge, we propose two main ideas for aligning ego and exo videos in time. First, motivated by the fact that human activity in the first-person view usually revolves around hand-object interactions, we design an object-centric encoder to bridge the gap between ego-exo viewpoints. The encoder integrates regional features corresponding to hands and active objects along with global image features, resulting in more object-centric representations. Second, we propose a contrastive-based alignment objective, where we temporally reverse frames to construct negative samples. The rationale is that aligning an ego-exo video pair should be easier (i.e., incur a lower alignment cost) compared to aligning the same video pair when one is played in reverse. We bring both ideas together in a single embedding learning approach we call AE2, for "align ego-exo".
To evaluate the learned representations, we establish an ego-exo benchmark for fine-grained video understanding. Specifically, we assemble four datasets of atomic human actions, comprising ego and exo videos drawn from five public datasets and an ego tennis forehand dataset that we collected. In addition, we annotate these datasets with dense per-frame labels (for evaluation only). We hope that the enriched data and labels that we release publicly can support progress in fine-grained temporal understanding from the ego-exo perspective. This is a valuable dataset contribution alongside our technical approach contribution. Furthermore, we present a range of practical downstream tasks that demand frame-wise ego-exo view-invariant features: action phase classification and retrieval across views, progress tracking, and synchronous playback of ego and exo videos. Experimental results demonstrate considerable performance gains of AE2 across all tasks and datasets.
Fine-grained Action Understanding To recognize fine-grained actions, supervised learning approaches [82, 30, 59, 65] employ fine-grained action datasets annotated with sub-action boundaries. In contrast, our self-supervised approach requires much lighter annotations to pretrain for
2We use "ego" and "exo" as shorthand for egocentric (first-person) and exocentric (third-person).
action phase classification, namely, video-level action labels. Keystep recognition in instructional videos [86, 73, 47, 46, 2] is a related form of fine-grained action understanding, where the goal is to name the individual steps of some longer activity (e.g., making brownies requires breaking eggs, adding flour, etc.). In that domain, multi-modal representation learning is a powerful tool that benefits from the spoken narrations that accompany how-to videos [47, 72, 38, 84]. In any of the above, handling extreme viewpoint variation is not an explicit target.
Self-supervised Video Representation Learning Various objectives for self-supervised video features have been explored [61]. This includes using accompanying modalities like audio [51, 29, 49] and text or speech [47, 72, 38, 37, 84] as training signals, as well as temporal coherence priors [28, 26, 63, 44, 10] or self-supervised tasks that artificially alter the temporal order of videos [48, 19, 35]. Alignment-based objectives have also been explored [15, 54, 22, 23, 41, 33], with ideas based on cycle consistency [15, 54], soft dynamic time warping (DTW) [22, 23], and optimal transport (OT) [41]. Our model also has an alignment stage; however, unlike our work, all these prior methods are explored for third-person videos only (where views are generally more similar across videos than ego-exo) without attending to the viewpoint discrepancy problem. Frame-wise representations invariant to both ego and exo viewpoints remain unresolved.
View-invariant Action Representation Learning Many prior works have explored view-invariance in action recognition [76, 9, 53]. One popular approach is to employ multi-view videos during training to learn view-invariant features [76, 77, 27, 78, 45, 71]. Several works [17, 39, 83, 55] discover a latent feature space independent of viewpoint, while others use skeletons [81], 2D/3D poses [56, 71] and latent 3D representations [53] extracted from video to account for view differences.
As advances in AR/VR and robotics unfold, ego-exo viewpoint invariance takes on special interest. Several models relate coarse-grained action information across the two drastically different domains [70, 1, 80, 36], for action recognition using both ego and exo cameras [70] or cross-viewpoint video recognition [80, 1, 36]. Closer to our work are Time Contrastive Networks (TCN) [63] and CharadesEgo [66, 67], both of which use embeddings aimed at pushing (ir)relevant ego-exo views closer (farther). However, unlike our approach, both employ videos that concurrently record the same human actions from ego and exo viewpoints. In fact, except for [36], all the above approaches rely on simultaneously recorded, paired multi-view videos. However, even in the case of [36], the goal is a coarse, clip-wise representation. In contrast, our method leverages temporal alignment as the pretext task, enabling fine-grained representation learning from unpaired ego and exo videos.
Object-centric Representations Several works advocate object-centric representations to improve visual understanding [40, 85, 50, 36], such as an object-centric transformer for joint hand motion and interaction hotspot prediction [40], or using a pretrained detection model for robot manipulation tasks [85, 50]. Hand-object interactions are known to dominate ego-video for human-worn cameras [11, 21], and recent work explores new ways to extract hands and active objects [64, 43, 12]. Our proposed object-centric transformer shares the rationale of focusing on active objects, though in our case as a key to link the most "matchable" things in the exo view and bridge the ego-exo gap.
We first formulate the problem (Sec. 3.1) and introduce the overall self-supervised learning objective (Sec. 3.2). Next we present our ideas for object-centric transformer (Sec. 3.3) and an alignment-based contrastive regularization (Sec. 3.4), followed by a review of our training procedure (Sec. 3.5).
3.1 Learning Frame-wise Ego-Exo View-invariant Features
Our goal is to learn an embedding space shared by both ego and exo videos that characterizes the progress of an action, as depicted in Fig. 1 (right). Specifically, we aim to train a view-invariant encoder network capable of extracting fine-grained frame-wise features from a given (ego or exo) video. Training is conducted in a self-supervised manner.
Formally, let  $\phi (\cdot ;\theta)$  represent a neural network encoder parameterized by  $\theta$ . Given an ego video sequence  $\mathbf{S}$  with  $M$  frames, denoted by  $\mathbf{S} = [\mathbf{s}_1,\dots ,\mathbf{s}_M]$ , where  $\mathbf{s}_i$  denotes the  $i$ -th input frame, we apply the encoder to extract frame-wise features from  $\mathbf{S}$ , i.e.,  $\mathbf{x}_i = \phi (\mathbf{s}_i;\theta),\forall i\in \{1,\ldots ,M\}$ . The resulting ego embedding is denoted by  $\mathbf{X} = [\mathbf{x}_1,\dots ,\mathbf{x}_M]$ . Similarly, given an exo video sequence  $\mathbf{V} = [\mathbf{v}_1,\dots ,\mathbf{v}_N]$  with  $N$  frames, we obtain the frame-wise exo embedding  $\mathbf{Y} = [\mathbf{y}_1,\dots ,\mathbf{y}_N]$ , where  $\mathbf{y}_i = \phi (\mathbf{v}_i;\theta),\forall i\in \{1,\dots ,N\}$  using the same encoder  $\phi$ .
The self-supervised learning objective is to map  $\mathbf{S}$  and  $\mathbf{V}$  to a shared feature space where frames representing similar action stages are grouped together, regardless of the viewpoint, while frames depicting different action stages (e.g., the initial tilting of the container versus the subsequent decrease in the liquid stream as the pouring action nears completion) should be separated. We present a series of downstream tasks to evaluate the quality of learned feature embeddings  $\mathbf{X}$  and  $\mathbf{Y}$  (Sec. 4).
An intuitive approach is to utilize paired ego-exo data collected simultaneously, adopting a loss function that encourages similarities between frame representations at the same timestamp while maximizing the distance between frame representations that are temporally distant [63, 66]. However, this approach relies on time-synchronized views, which is often challenging to realize in practice.
Instead, we propose to achieve view-invariance from unpaired ego-exo data; the key is to adopt temporal alignment as the pretext task. The encoder  $\phi$  learns to identify visual correspondences across views by temporally aligning ego and exo videos that depict the same action. Throughout the learning process, no action phase labels or time synchronization is required. To enforce temporal alignment, we assume that multiple videos portraying the same action (e.g., pouring, tennis forehand), captured from different ego and exo viewpoints (and potentially entirely different environments), are available for training.
3.2 Ego-Exo Temporal Alignment Objective
To begin, we pose our learning objective as a classical dynamic time warping (DTW) problem [4]. DTW has been widely adopted to compute the optimal alignment between two temporal sequences [8, 6, 23, 22, 79] according to a predefined cost function. In our problem, we define the cost of aligning frame  $i$  in ego sequence  $\mathbf{S}$  and frame  $j$  in exo sequence  $\mathbf{V}$  as the distance between features extracted by our encoder  $\phi$ ,  $\mathbf{x}_i$  and  $\mathbf{y}_j$ , so that aligning more similar features results in a lower cost.
Formally, given the two sequences of embeddings  $\mathbf{X}$  of length  $M$  and  $\mathbf{Y}$  of length  $N$ , we calculate a cost matrix  $\mathbf{C} \in \mathbb{R}^{M \times N}$ , with each element  $c_{i,j}$  computed by a distance function  $\mathcal{D}$ . We adopt the distance function from [22], and define  $c_{i,j}$  to be the negative log probability of matching  $\mathbf{x}_i$  to a selected  $\mathbf{y}_j$  in sequence  $\mathbf{Y}$ :
$$
c _ {i, j} = \mathcal {D} (\mathbf {x} _ {i}, \mathbf {y} _ {j}) = - \log \frac {\exp \left(\tilde {\mathbf {x}} _ {i} ^ {T} \tilde {\mathbf {y}} _ {j} / \beta\right)}{\sum_ {k = 1} ^ {N} \exp \left(\tilde {\mathbf {x}} _ {i} ^ {T} \tilde {\mathbf {y}} _ {k} / \beta\right)}, \quad \tilde {\mathbf {x}} _ {i} = \mathbf {x} _ {i} / \| \mathbf {x} _ {i} \| _ {2} \quad \tilde {\mathbf {y}} _ {i} = \mathbf {y} _ {i} / \| \mathbf {y} _ {i} \| _ {2} \tag {1}
$$
where  $\beta$  is a hyper-parameter controlling the softmax temperature, which we fix at 0.1.
DTW calculates the alignment cost between  $\mathbf{X}$  and  $\mathbf{Y}$  by finding the minimum cost path in  $\mathbf{C}$ . This optimal path can be determined by evaluating the following recurrence [60]:
$$
\mathbf {R} (i, j) = c _ {i, j} + \min  [ \mathbf {R} (i - 1, j - 1), \mathbf {R} (i - 1, j), \mathbf {R} (i, j - 1) ] \tag {2}
$$
where  $\mathbf{R}(i,j)$  denotes the cumulative distance function, and  $\mathbf{R}(0,0) = 0$ . Given that the min operator is non-differentiable, we approximate it with a smooth differentiable operator, as defined in [34, 22]. This yields a differentiable loss function that can be directly optimized with back-propagation:
$$
\mathcal {L} _ {\text {a l i g n}} (\mathbf {X}, \mathbf {Y}) = \mathbf {R} (M, N). \tag {3}
$$
While DTW has been applied in frame-wise action feature learning [23, 22] with some degree of view invariance across exo views, it is inadequate to address the ego-exo domain gap. As we will show in results, directly adopting the DTW loss for learning ego-exo view-invariant features is sub-optimal since the ego and exo views, even when describing the same action, exhibit significant differences (compared to differences within ego views or within exo views). In Sec. 3.3-3.4, we propose two designs of AE2 that specifically account for the dramatic ego-exo view differences.
3.3 Object-centric Encoder Network
The visual appearance in ego and exo views may look dramatically different due to different backgrounds, fields of view, and (of course) widely disparate viewing angles. Yet a common element shared between these two perspectives is the human action itself. In this context, hand-object interaction regions are particularly informative about the progress of an action. Hand-object interactions are prominent in the first-person viewpoint; the wearable camera gives a close-up view of the person's near-field manipulations. In exo videos, however, these active regions may constitute only a small
Figure 2: Overview of AE2. We extract frame-wise representations from the two video sequences (i.e.,  $\mathbf{S}$  and  $\mathbf{V}$ ) using the encoder  $\phi$  and temporally align their embeddings (i.e.,  $\mathbf{X}$  and  $\mathbf{Y}$ ) using DTW. Left: Our proposed encoder network design emphasizes attention to hand and active object regions, leading to more object-centric representations. Right: As a form of regularization, we reverse the video sequence  $\mathbf{S}$  and enforce that the cost of aligning  $(\mathbf{S},\mathbf{V})$  is less than that of aligning  $(\tilde{\mathbf{S}},\mathbf{V})$ .
portion of the frame and are not necessarily centered, depending on the third-person camera position and angle. Inspired by these observations, we propose an object-centric encoder design that explicitly focuses on regions corresponding to hands and active objects to better bridge the ego-exo gap.
Our proposed AE2 encoder network  $\phi$  consists of two main components: (1) a ResNet-50 encoder [25], denoted by  $\phi_{\mathrm{base}}$ , to extract features from the input video frame; (2) a transformer encoder [75], denoted by  $\phi_{\mathrm{transformer}}$ , to fuse global features along with regional features through self-attention.
As a preprocessing step, we employ an off-the-shelf detector to localize hand and active object regions in ego and exo frames [64]. For frame  $i$ , we select the top two hand and object proposals with the highest confidence scores, resulting in a total of four bounding boxes  $\{\mathbf{BB}_{i,k}\}_{k=1}^4$ .
First, given one input frame  $\mathbf{s}_i$  at timestamp  $i$ , we pass it to  $\phi_{\mathrm{base}}$  to obtain global features:  $\mathbf{g}_i = \phi_{\mathrm{base}}(\mathbf{s}_i)$ . Next, we apply ROI Align [24] to intermediate feature maps produced by  $\phi_{\mathrm{base}}$  for the input frame  $\mathbf{s}_i$ . This yields regional features  $\mathbf{r}_{i,k}$  for each bounding box  $\mathrm{BB}_{i,k}$ :  $\mathbf{r}_{i,k} = \mathrm{ROI~Align}(\phi_{\mathrm{base}}^{\mathrm{interm.}}(\mathbf{s}_i),\mathrm{BB}_{i,k}),\forall k$ .
Both the global and regional features are flattened as vectors and passed through a projection layer to unify their dimensions. We denote the projection layer to map global and local tokens as  $\phi_{\mathrm{project}_g}$  and  $\phi_{\mathrm{project}_l}$ , respectively. The input tokens are then enriched with two types of embeddings: (1) a learnable spatial embedding, denoted by  $\mathbf{e}_{\mathrm{spatial}}$ , which encodes the bounding box coordinates and confidence scores obtained from the hand-object detector and (2) a learnable identity embedding, denoted by  $\mathbf{e}_{\mathrm{identity}}$ , which represents the category of each feature token, corresponding to the global image, left hand, right hand, and object. This yields the tokens:  $\mathbf{z}_{i,\mathrm{global}} = \phi_{\mathrm{project}_g}(\mathbf{g}_i) + \mathbf{e}_{\mathrm{identity}}$  and  $\mathbf{z}_{i,\mathrm{local}k} = \phi_{\mathrm{project}_l}(\mathbf{r}_{i,k}) + \mathbf{e}_{\mathrm{spatial}} + \mathbf{e}_{\mathrm{identity}}, \forall k$ .
The transformer encoder processes one global token along with four regional tokens as input, leveraging self-attention to conduct hand-object interaction reasoning. This step enables the model to effectively capture action-related information within the scene, yielding the transformer outputs:  $\mathbf{f}_{i,k} = \phi_{\mathrm{transformer}}(\mathbf{z}_{i,\mathrm{global}},\{\mathbf{z}_{i,\mathrm{local}k}\}_{k})$ . Finally, we average the transformer output tokens and adopt one embedding layer  $\phi_{\mathrm{embedding}}$  to project the features to our desired dimension, yielding the final output embedding for the input frame  $\mathbf{s}_i$ :  $\mathbf{x}_i = \phi_{\mathrm{embedding}}\left(\frac{1}{5}\sum_k\mathbf{f}_{i,k}\right)$ .
Fig. 2 (left) showcases our proposed object-centric encoder. This design inherently bridges the ego-exo gap by concentrating on the most informative regions, regardless of the visual differences between the two perspectives. As a result, the object-centric encoder can better align features extracted from both ego and exo videos, enhancing the learned features in downstream tasks.
3.4 Contrastive Regularization With Reversed Videos
As shown in previous studies [22, 23], directly optimizing the DTW loss can result in the collapse of embeddings and the model settling into trivial solutions. Moreover, aligning ego and exo videos
presents new challenges due to their great differences in visual appearance. Thus, a careful design of regularization, supplementing the alignment objective in Eqn. (3), is imperative.
We resort to discriminative modeling as the regularization strategy. The crux lies in how to construct positive and negative samples. A common methodology in previous fine-grained action approaches [63, 66, 23, 10], is to define negative samples on a frame-by-frame basis: temporally close frames are considered as positive pairs (expected to be nearby in the embedding space), while temporally far away frames are negative pairs (expected to be distant). This, however, implicitly assumes clean data and strict monotonicity in the action sequence: if a video sequence involves a certain degree of periodicity, with many visually similar frames spread out over time, these approaches could incorrectly categorize them as negative samples.
To address the presence of noisy frames in real-world videos, we introduce a global perspective that creates negatives at the video level. From a given positive pair of sequences  $(\mathbf{S},\mathbf{V})$ , we derive a negative pair  $(\tilde{\mathbf{S}},\mathbf{V})$ , where  $\tilde{\mathbf{S}} = [\mathbf{s}_M,\dots,\mathbf{s}_1]$  denotes the reversed video sequence  $\mathbf{S}$ . The underlying intuition is straightforward: aligning an ego-exo video pair in their natural temporal order should yield a lower cost than aligning the same pair when one video is played in reverse. Formally, we employ a hinge loss to impose this regularization objective:
$$
\mathcal {L} _ {\text {r e g}} (\mathbf {X}, \mathbf {Y}) = \max  \left(\mathcal {L} _ {\text {a l i g n}} (\mathbf {X}, \mathbf {Y}) - \mathcal {L} _ {\text {a l i g n}} (\bar {\mathbf {X}}, \mathbf {Y}), 0\right) \tag {4}
$$
where  $\tilde{\mathbf{X}}$  denotes the embeddings of  $\tilde{\mathbf{S}}$ , which is essentially  $\mathbf{X}$  in reverse. Fig. 2 (right) illustrates the alignment cost matrices of  $(\mathbf{X},\mathbf{Y})$  and  $(\tilde{\mathbf{X}},\mathbf{Y})$ . A lighter color indicates a smaller cost. Intuitively,  $\mathcal{L}_{\mathrm{align}}$  corresponds to the minimal cumulative cost traversing from the top-left to the bottom-right of the matrix. As seen in the figure, the alignment cost increases when  $\mathbf{X}$  is reversed, i.e.,  $\mathcal{L}_{\mathrm{align}}(\tilde{\mathbf{X}},\mathbf{Y})$  should be larger than  $\mathcal{L}_{\mathrm{align}}(\mathbf{X},\mathbf{Y})$ .