SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dibyadip Chatterjee 1 Fadime Sener 2 Shugao Ma 2 Angela Yao 1
<sup>1</sup>National University of Singapore <sup>2</sup>Meta Reality Labs Research  
{dibyadip, ayao}@comp.nus.edu.sg {famesener, shugao}@meta.com  
https://dibschat.github.io/openvocab-egoAR/
Human actions in egocentric videos often feature hand-object interactions composed of a verb (performed by the hand) applied to an object. Despite their extensive scaling up, egocentric datasets still face two limitations — sparsity of action compositions and a closed set of interacting objects. This paper proposes a novel open vocabulary action recognition task. Given a set of verbs and objects observed during training, the goal is to generalize the verbs to an open vocabulary of actions with seen and novel objects. To this end, we decouple the verb and object predictions via an object-agnostic verb encoder and a prompt-based object encoder. The prompting leverages CLIP representations to predict an open vocabulary of interacting objects. We create open vocabulary benchmarks on the EPIC-KITCHENS-100 and Assembly101 datasets; whereas closed-action methods fail to generalize, our proposed method is effective. In addition, our object encoder significantly outperforms existing open-vocabulary visual recognition methods in recognizing novel interacting objects.
Egocentric or first-person videos captured from body-mounted cameras often feature the person manipulating objects with one or both hands. An action is thus conveniently defined as the composition of a verb performed by the hand with an interacting object, e.g., the action "slicing apple" is defined as "slice" and "apple". Egocentric datasets [17, 8, 51] have scaled up by increasing the variety of verbs and objects. Theoretically, the increase in feasible actions should be quadratic, yet actually observed compositions are sparse, e.g., EPIC-KITCHENS-100 [8] has 97 verbs and 300 objects, but only  $14\%$  of possible verb-object compositions are observed as actions. Although not all compositions are feasible, e.g., "eat dishwasher", many feasible actions are not observed, e.g., "serve chicken", "crush ice".
We observe that the verbs in egocentric datasets tend to be domain-agnostic. For example, even though EPIC100 is set in the kitchen, its verbs like pull, remove, and shake are also applicable outside of the kitchen. Ideally, we want to learn the verb concepts and generalize the verb to any object. Yet the dataset bounds the type and number of objects and the capture environment, e.g., EPIC100 has cooking utensils and food, whereas Assembly101 [51] has toy vehicle parts. Simply put, the objects restrict the subsequent action space of the dataset.
Existing action recognition models [61, 11, 43, 48, 22] are designed for a closed set of actions. A few works have studied open-set action recognition [4, 65], but they simply reject actions not seen during training. Inspired by the recent success of open vocabulary object detection [63, 18, 68], a more practical objective would be to provide novel actions as text inputs during inference. To that end, we address this problem of open vocabulary action recognition where a video model trained on a base vocabulary of actions needs to recognize novel actions outside of that vocabulary.
We tackle open vocabulary action recognition as two problems – (1) generalizing verbs to novel objects and (2) recognizing novel objects from egocentric videos. Note that this differs from a zero-shot setup, where some actions are not observed during training. These unobserved actions are not novel because the vocabulary is known and fixed during training, hence closed. More recently, [25, 57]
Figure 1: Open vocabulary action recognition for verb-object composed actions. During training, the model observes a predefined vocabulary of verbs and objects. During testing, the model is presented with seen verbs with any object (described by a text prompt). Objects in green are base classes  $(\mathcal{O}_{base})$  seen during training; those in blue are novel classes  $(\mathcal{O}_{novel})$  and unknown during training.
proposed recognizing an open vocabulary of verbs by leveraging large-scale vision-language models (VLMs) [46]. Contrary to their task, we are interested in the scenario of generalizing verbs to an open vocabulary of objects.
Generalizing verbs to novel object categories is challenging when the classes are naturally long-tailed [31, 44]. Models trained on such imbalanced datasets are biased towards head verb-object composed actions (when a verb is frequently seen with a particular object). Furthermore, these models suffer from static bias [7, 28] and overfit to the appearance of fixed frames without learning the transformation of objects over time [35]. To alleviate these biases, we decouple the verb and object predictions via two separately trained encoders. We propose an Object Agnostic Pretraining (OAP) for the verb encoder. OAP is based on a supervised contrastive loss [26], to which we add a set of guiding augmentations. These augmentations are specially designed for egocentric videos and allow the verb encoder to learn object-agnostic representations to improve transfer to novel objects. To our knowledge, this work is the first to explore contrastive learning of video backbones end-to-end on egocentric videos. In contrast, the augmentations in existing video contrastive learning [12, 45] are designed for Kinetics [6] styled videos where actions can be predicted from a single frame or a short clip sampled anywhere from the video. Such augmentations are ineffective for egocentric footage (Sec. C of Supplementary).
To recognize novel objects, we design an open-vocabulary object encoder that leverages a pretrained CLIP [46] model. We are specifically interested in recognizing active objects – those that undergo a state change from the actor's interactions. Active objects are not necessarily handheld [64, 9] e.g., while “beating egg”, the actor holds a whisk and a bowl in his hand, but the egg is the active object undergoing a state change. Recognizing active objects in egocentric videos is challenging even under closed settings due to the many distractor objects in natural environments. To this end, we propose Active Object Prompting (AOP) that guides the CLIP model to understand which object is active. Given a frame containing a whisk, bowl, egg batter, hands, and other distractors, AOP aims at learning the context i.e. “beating egg” by generating a set of learnable verb-conditioned prompts. Conditioning the prompts on the verb features generated by OAP helps the CLIP recognize the active object.
Egocentric datasets [8, 51] focus on a closed set evaluation of actions where a handful of objects are unseen. However, for a realistic open vocabulary evaluation, a variety of novel objects is expected to be encountered for the first time during inference. To solve this, we create open vocabulary benchmarks on EPIC100 and Assembly101 by repurposing their original split. Our contributions can thus be summarized as: (i) an object agnostic pretraining of video backbones to improve its verb generalization to novel objects, (ii) an active object prompting of CLIP to recognize novel active objects and (iii) open vocabulary action recognition benchmarks on EPIC100 and Assembly101.
Egocentric Action Recognition. Egocentric computer vision has garnered huge interest in recent years, and massive efforts at scaling up have resulted in the development of datasets like Ego4D [17],
EPIC-KITCHENS-100 [8] and Assembly101 [51]. It has also aided the development of sophisticated video recognition architectures [11, 3, 43, 58]. Recent works [48, 22, 33] focus on learning more object-centric representations by explicitly modeling object transformations over time with off-the-shelf detectors [50] and trackers [5]. However, these models are restricted to a closed set of predefined actions and do not scale to the diversity of objects in the real world.
Contrastive Learning in Videos. Videos provide unique opportunities for contrastive learning [19, 41] due to its multimodal and highly redundant nature. Most popular methods follow the self-supervised instance discrimination task [59] where a single positive of an anchor video is contrasted against a set of negatives. This positive can be an augmented view of the RGB stream [12, 45, 49] or sampled from a complementary modality like optical flow [40, 60], text [14, 36, 62, 21, 30], audio [2, 27, 34, 15] or even all three at once [1, 53]. Contrary to instance discrimination, recent works [26, 20] propose supervised contrastive learning. They show that sampling multiple positives from the same class as the anchor outperforms instance discrimination. Another line of research uses multiple positives to combat noisy anchor-positive alignment [37]. In this paper, we propose a set of lightweight augmentations to train video backbones for verb prediction. These augmentations result in a noisy alignment due to the camera motion inherent in egocentric videos. Thus, we leverage multiple positives by modifying [37] to combat it.
Learning for an Open vocabulary. The remarkable zero-shot capabilities of large language models (LLMs) [10, 47] and vision language models (VLMs) [46, 23] have widened the possibility of recognition and detection tasks in the visual domain. Of particular interest is the open vocabulary setting, which can recognize novel classes unknown during training. By presenting the novel classes as a textual phrase, it becomes possible to probe the huge corpora of knowledge in LLMs and VLMs. It differs from the traditional zero-shot setting where the novel classes are assumed to be known during training [63] and hence is restricted to the closed vocabulary of predefined classes. Most works for open vocabulary recognition are limited to the image domain [63, 67, 66, 18, 68, 39]. Recently, some works [56, 32, 25, 57] have adapted VLMs for video tasks but primarily focus on zero-shot verb recognition. In contrast, we are interested in a practical hand-object interaction scenario of generalizing verbs performed by hands to any object.
Prompting VLMs. Large Scale Vision Language Models (VLMs) like CLIP [46] and ALIGNN [23] are trained on an enormous corpus of web data and contain semantically grounded knowledge of images. Prompting [24] refers to designing textual instructions to steer the VLM towards some desired output. Manually engineering discrete text prompts is laborious and requires domain expertise. One alternative is fine-tuning the VLM encoder for the task at hand; this approach incurs significant computation and is also at risk of catastrophic forgetting [29]. The more widely adopted approach is simply learning the prompts by backpropagating the task-specific loss while freezing the VLM encoders [67, 66, 13]. Such an approach is parameter-efficient since only a few prompt embeddings are learned while keeping the encoders frozen. In this paper, we are interested in such parameter-efficient prompt learning to steer the VLM toward active object recognition.
Consider a video dataset consisting of verbs  $\mathcal{V}$  applied to a base set of objects  $\mathcal{O}_{base}$ . The possible set of base actions is  $\mathcal{A}_{base} = \mathcal{V} \times \mathcal{O}_{base} = \{(v, o) \mid v \in \mathcal{V}, o \in \mathcal{O}_{base}\}$ . During training, only a sparse subset of actions are observed i.e.  $\mathcal{A}_{train} \subset \mathcal{A}_{base}$ . The goal is to learn a model  $f: \mathbb{X}_{train} \to \mathcal{A}_{base}$  where  $\mathbb{X}_{train}$  is the set of videos seen during training. During inference, along with base objects, we also encounter novel objects  $\mathcal{O}_{novel}$  that form novel actions  $\mathcal{A}_{novel} = \{(v, o) \mid v \in \mathcal{V}, o \in \mathcal{O}_{novel}\}$ . We want our model  $f$  to predict both base and novel actions i.e.  $\mathcal{A}_{test} = \mathcal{A}_{base} \cup \mathcal{A}_{novel}$ . Conventional action recognition follows a closed vocabulary evaluation where  $\mathcal{A}_{test} = \mathcal{A}_{base}$ .
We train two encoders - one for predicting verbs and another for predicting an open vocabulary of objects as depicted by the pink and yellow blocks in Fig. 2a. During inference, predictions from the verb and object encoders are composed to generate the action prediction.
not all unobserved actions are feasible
Figure 2: (a) Framework. We train separate encoders for predicting verb  $\hat{v}$  and object  $\hat{o}$ . The verb encoder operates on a closed set of verbs, whereas the object encoder, trained on  $\mathcal{O}_{\text{base}}$ , can predict any object described by a text prompt. (b) Object Agnostic Pretraining of the Verb Encoder. We design an object mixing augmentation to facilitate learning object-agnostic verb representations. Multiple positives are sampled for each anchor video, including our proposed object mixing (mixed with "cut orange"), temporal gradients of the anchor video, and in-class positives i.e. videos featuring the same verb ("cut cheese"). The two augmentations corresponding to the same point in the embedding space illustrate that the anchor only needs to minimize its distance from a softmax weighted average of the two (softmax over the similarity of the anchor with the two).
Verb Encoder. Encoding verbs requires spatiotemporal modeling, so we use a video backbone as the verb encoder. We want to learn object-agnostic verb representations to encourage generalization to novel objects. We propose Object Agnostic Pretraining (OAP) (described in Sec. 4.1), which is a general pretraining strategy that can be applied to any video backbone and does not add parameters or increase the model size.
Object Encoder. We aim to recognize an open vocabulary of active objects. Even with a closed vocabulary, finding the active object amongst the many other objects in a cluttered scene is difficult. As such, state-of-the-art open vocabulary object detectors [18, 68] perform poorly. One option for support is to spatially locate a region of interaction in the frame; we use a pretrained hand-object interaction (HOI) detector, 100DOH [52] as the base of our object encoder. It provides bounding boxes of the left and right hands and the corresponding object(s)-in-contact in a class-agnostic fashion. Note that the object bounding boxes may contain more than one object (Fig. 3). Thus, we propose an Active Object Prompting (AOP) strategy (described in Sec. 4.2) by leveraging CLIP [46] to generate active object labels from the interaction region for both base and novel classes.
4.1 Object Agnostic Pretraining (OAP)
(d)
Figure 3: Example frame of "cut onion" from EPIC. (a) DETIC [68] output with EPIC noun vocabulary. (b) HOI Detector [52] output. (c) & (d) are crops of (a) & (b) respectively for better visualization. The object-in-contact crop O alongside onion also captures knife.
OAP aims to pull instances with the same verb
as the anchor close in the embedding space (irrespective of its object class) and contrast them against instances with different verbs. We randomly sample a batch  $B = \{(x_{1},y_{1}),\dots,(x_{i},y_{i}),\dots,(x_{N},y_{N})\}$  where  $y_{i}$  is the verb class for the raw video clip  $x_{i}\in \mathbb{X}_{train}$  and  $N$  is the batch size. We first use
random photometric augmentations like Gaussian blur and color jittering to create two views of each anchor, thus doubling the batch size to  $2N$ . Each view is then fed to the verb encoder  $f_{\theta}$  followed by a projection head  $g_{\theta}$  which generates features for the whole batch  $\{z_1,z_2,\dots,z_{2N}\}$ . These features are then normalized to the unit hypersphere. The verb encoder is trained with the standard supervised contrastive loss [26]. Videos sampled from  $B$  with the same verb as the anchor are considered positives. For each anchor  $z_{i}$  we have the bag of positives  $\mathcal{P}_i = \{z_j|y_j = y_i,\forall j\in [1,2N]\}$ . We use the loss  $\mathcal{L}_{out}$  with temperature parameter  $\tau_{out}$ :
$$
\mathcal {L} _ {\text {o u t}} ^ {(i)} = - \frac {1}{| \mathcal {P} _ {i} |} \sum_ {j \in \mathcal {P} _ {i}} \log \frac {\exp \left(z _ {i} \cdot z _ {j} / \tau_ {\text {o u t}}\right)}{\sum_ {k \in B \backslash i} \exp \left(z _ {i} \cdot z _ {k} / \tau_ {\text {o u t}}\right)}. \tag {1}
$$
Guiding Augmentations. Sampling in-class positives is still bounded by the verb-object compositions in the training set. Our intuition is that as long as a verb is being performed in a video, exchanging objects and the presence of other distractor objects in the background should not affect the outcome of the verb encoder. To reduce the reliance on such objects for verb prediction, we propose a set of guiding augmentations to diversify the set of positives. The augmentations are lightweight and allow new positives to be generated online during training.
To achieve this, we first mask static information (objects in the background) from a frame of the video clip. The most obvious choice is to use temporal gradient that eliminates any static information and focuses primarily on the region with non-zero motion vectors, which we refer to as the active region  ${}^{2}$  . In hand-object interactions, this active region is expected to contain the active object. Furthermore, we exploit the compositional nature of actions and propose a new object mixing augmentation. The idea is simple - we combine two clips with the same verb but different objects to generate new object mixed clips (Fig. 2b). Formally, given two clips from the batch,  ${x}_{i}$  and  ${x}_{j}$  ,having the same verb,we get the object mixed clips  ${x}_{i}^{\prime }$  and  ${x}_{j}^{\prime }$  .
$$
x _ {i} ^ {\prime} = \alpha (\nabla x _ {i} \odot x _ {i}) + (1 - \alpha) [ (\nabla x _ {j}) ^ {- 1} \odot x _ {j} ]
$$
$$
x _ {j} ^ {\prime} = \alpha (\nabla x _ {j} \odot x _ {j}) + (1 - \alpha) [ (\nabla x _ {i}) ^ {- 1} \odot x _ {i} ] \tag {2}
$$
where  $\nabla x_{i}$  is the first-order temporal gradient of  $x_{i}$  (which masks static regions) and  $\odot$  is the element-wise matrix multiplication;  $(\nabla x_{j})^{-1}$  refers to the inverted gradient image of  $x_{j}$  and  $\alpha$  is the mixing hyperparameter. To summarize, masking aims at minimizing the impact of distractor objects outside of the active region, and mixing prevents the model from being biased towards low-level object information, e.g., color, shape, and hence to the observed verb-object compositions.
Learning from Noisy Augmentations. The basis of our guiding augmentations is the temporal gradient. In egocentric videos, the camera (ego) motion will result in non-zero gradients on static parts of the scene in addition to the true active region. While it is feasible to specially design expensive augmentations that account for camera motion [55, 54]; instead, we propose to adjust the contrastive loss so that it can learn from these noisy but lightweight augmentations. In  $\mathcal{L}_{out}$ , all the positives for an anchor contribute equally to the loss; this is desirable for in-class positives where the condition for being a positive is well-defined and fixed, but undesirable for the noisy guiding augmentations. Now, if we move the summation inside the log, the logit for each anchor becomes the average cosine similarity of all the guiding augmentations. The contribution of each augmentation is thus dependent on its similarity with the anchor, where noisy augmentations having low cosine similarity will have less contribution to the gradient. With a sufficiently large bag of augmentations, this is expected to reduce the noise implicitly. Such a loss is formulated as  $\mathcal{L}_{in}$  in [26]. If  $\mathcal{G}_i$  is our bag of guiding augmentations, our  $\mathcal{L}_{in}$  loss is as follows:
$$
\mathcal {L} _ {i n} ^ {(i)} = - \log \frac {\frac {1}{| \mathcal {G} _ {i} |} \sum_ {j \in \mathcal {G} _ {i}} \exp \left(z _ {i} \cdot z _ {j} / \tau_ {i n}\right)}{\sum_ {k \in B \backslash i} \exp \left(z _ {i} \cdot z _ {k} / \tau_ {i n}\right)} \tag {3}
$$
where  $\tau_{in}$  is the temperature parameter. This loss is also similar to MIL-NCE [37] for  $\tau_{in} = 1$ , which is used for learning from noisy video-text pairs. In our case, a lower softmax temperature ( $< 1$ )
2active regions may still contain static objects, e.g., for the action of "cutting onions", the cutting board which is present in the active region is static. This object is not a distractor and is relevant for action prediction.
Figure 4: (Top) Parameter-efficient training for Active Object Prompting. Keeping the CLIP encoders frozen, the base vocabulary initialized with the CLIP-pretrained word embeddings is fine-tuned on the training set. A set of prefix and postfix prompt vectors conditioned on the verb representation are also learned. (Bottom) Testing with Active Object Prompting. The fine-tuned embeddings are used for the base objects, whereas the CLIP-pretrained embeddings are used for novel ones. The learned prefix and postfix prompts are shared among all object classes.
results in better performance. Please refer to the Supplementary for theoretical insights into the loss formulation (Sec. D) and illustrations of guiding augmentations (Sec. F). Thus, the overall loss function is  $\mathcal{L} = \mathcal{L}_{out} + \lambda \mathcal{L}_{in}$ , where  $\lambda$  is a weighting hyperparameter.
4.2 Active Object Prompting (AOP)
This section discusses leveraging CLIP to predict the active object from a crop with multiple objects. We assume no prior knowledge of the set of objects observed during inference i.e. any object is expected to be predicted by our model as long as it is described by a textual phrase (open vocabulary).
The overall framework of AOP is shown in Fig. 4. First, the given set of object classes  $\mathcal{O}$  are converted into text prompts by prepending and appending each class with a set of prefix and suffix tokens. For object  $o_i$  with word embedding  $W_{o_i}$ , a prompt text  $t_i$  can be defined as  $t_i = \{\mathrm{pre}_1,\dots,\mathrm{pre}_{k_1},W_{o_i},\mathrm{post}_1,\dots,\mathrm{post}_{k_2}\}$ , where  $\mathrm{pre}_j$  are  $k_{1}$  prefix tokens,  $\mathrm{post}_j$  are  $k_{2}$  postfix tokens. The prefix and postfix prompts, together known as context prompts [67], are essential in guiding a pretrained CLIP to solve a desired task. The context prompt tokens are shared across all classes and can be fixed or learned.
Fixed prompt tokens must be manually engineered by humans, which is laborious and requires domain expertise. Also, prompting CLIP, which operates in a continuous space with discrete human-readable text prompts, limits its usage. Hence, we propose to learn our prompts via two prompting strategies – (i) verb-conditioned context prompt generation and (ii) fine-tuning base object vocabulary.
Verb-Conditioned Context Prompt Generation. As shown in Fig. 3, given a crop containing onion, knife cutting board and hands, we want the CLIP to predict onion as the active object by understanding the context i.e. the action of cut onion. To this end, we propose verb-conditioned context prompts. Given the video  $x$ , we add the verb representation of  $x$  learned using OAP to the context prompts as shown in Fig. 4 i.e.  $\mathrm{pre}_j = \mathrm{pre}_j + \Phi_L(f_\theta(x))$  and  $\mathrm{post}_j = \mathrm{post}_j + \Phi_L(f_\theta(x))$ , where  $f_\theta$  is the pretrained verb encoder and  $\Phi_L$  is a projection network used to project the visual feature to the language domain.
Fine-tuning the Base Object Vocabulary. Egocentric videos often capture many different objects in the same environment. Semantic variability of the objects can be very fine-grained e.g. "boom" vs. "arm" in Assembly101 [51]. Such fine granularity is not fully captured by CLIP pretrained on data crawled from the web. We treat this as a vocabulary learning problem with the goal of making fine-grained object categories discriminative in the continuous CLIP feature space. To this end, we fine-tune the pretrained object embeddings  $W_{o_i}$  for all base objects seen during training i.e.  $\forall o_i \in \mathcal{O}_{base}$ . This simple fine-tuning allows CLIP to adjust the object embeddings in its continuous space to make them discriminative for optimal performance.