Opening the Vocabulary of Egocentric Actions
Authors: Dibyadip Chatterjee
Fadime Sener
Shugao Ma
Angela Yao
Venue: NeurIPS 2023 poster (2023)
Keywords: video understanding
egocentric videos
open vocabulary
TL;DR: We propose the novel task of open vocabulary action recognition for verb-object composed actions from egocentric videos and a novel approach to address it.

Abstract: Human actions in egocentric videos often feature hand-object interactions composed of a verb (performed by the hand) applied to an object. Despite their extensive scaling up, egocentric datasets still face two limitations — sparsity of action compositions and a closed set of interacting objects. This paper proposes a novel open vocabulary action recognition task. Given a set of verbs and objects observed during training, the goal is to generalize the verbs to an open vocabulary of actions with seen and novel objects. To this end, we decouple the verb and object predictions via an object-agnostic _verb encoder_ and a prompt-based _object encoder_. The prompting leverages CLIP representations to predict an open vocabulary of interacting objects. We create open vocabulary benchmarks on the EPIC-KITCHENS-100 and Assembly101 datasets; whereas closed-action methods fail to generalize, our proposed method is effective. In addition, our object encoder significantly outperforms existing open-vocabulary visual recognition methods in recognizing novel interacting objects.

Opening the Vocabulary of Egocentric Actions
Dibyadip Chatterjee
Fadime Sener" 2
Shugao Ma* 2
Angela Yao
Nattional University of Singapore
Meta Reality Labs Research
(dibyadip, ayao @ @comp. edu.sg sg
famesener, shugao @meta. https: / bachat. gi github. / io/ppenvccab-egokkk
Abstract
Human actions in egocentric videos often feature hand-object interactions com- posed of verb (performed by the hand) applied an object. Despite their extensive scaling up, egocentric datasets still face two limitations sparsity of action compositions and closed set of interacting objects. This paper proposes a novel open vocabulary action recognition task. Given set of verbs and objects observed during training, the goal is to generalize the verbs to an open vocabu- lary of actions with seen and novel objects. To this end, we decouple the verb and object predictions via an object-agnostic verb encoder and a prompt-based object encoder. The prompting leverages CLIP representations to predict open vocabulary interacting objects. We create open vocabulary benchmarks on the EPIC-KITCHENS 100 and Assembly 101 datasets; whereas closed-action methods fail to generalize, our proposed method effective. In addition, our object encoder significantly outperforms existing open- ppen-vocabllary visual recognition methods in recognizing novel interacting objects.
Introduction
Egocentric first-person videos captured from body-mounted cameras often feature the person manipulating objects with one or both hands. An action thus conveniently defined as the composi- tion of verb performed by the hand with an interacting object, e.g., the action "slicing apple" is defined as "slice" " and "apple". Egocentric datasets [17, 8, 51] have scaled up by increasing the variety of verbs and objects. Theoretically, the increase in feasible actions should be quadratic, yet actually observed compositions are sparse, e.g., SEPC-KITCHENS-III 100 [8] has 97 verbs and 300 objects, but only 14% of possible verb-object compositions observed actions. Although not all compositions are feasible, e.g., "eat dishwasher" many feasible actions are not observed, e.g..
"serve chicken", "crush ice".
We observe that the verbs in egocentric datasets tend to be domain-agnostic For example, even though EPIC100 is set in the kitchen, its verbs like pull, remove, and shake are also applicable outside of the kitchen. Ideally, we want to learn the verb concepts and generalize the verb to any object. Yet the dataset bounds the type and number of objects and the capture environment, e.g., EPIC100 has cooking utensils and food, whereas Assembly 101 [51] has toy vehicle parts. Simply put, the objects restrict the subsequent action space of the dataset.
Existing action recognition models [61, 11, 43, 48, 22] are designed a closed set of actions. few works have studied open-set action recognition [4, 65], but they simply reject actions not seen during training. Inspired by the recent success of open vocabulary object detection [63, 18, 68), more practical objective would be to provide novel actions text inputs during inference. To that end, we address this problem of open vocabulary action recognition where a video model trained on base vocabulary of actions needs to recognize novel actions outside of that vocabulary.
We tackle open vocabulary action recognition two problems - (1) generalizing verbs to novel objects and (2) recognizing novel objects from egocentric videos. Note that this differs from a zero- shot setup, where some actions are observed during training. These unobserved actions are not novel because the vocabulary is known and fixed during training, hence closed. More recently, (25,57)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Text prompt
haze nove!
Open Vocabulary
model predictions
Video Model
wash plate
remove lid
throw paper
throw plate
remove cap
wash bag
Training Vocabulary: Vx Obess
Testing Vocabulary: Vx (Orare (Onoe)))
Figure/ 1: Open vocabulary action recognition for verb-object composed actions. During training, the model observes predefined vocabulary of verbs and objects. During testing, the model is presented with seen verbs with any object (described by a text prompt). Objects in green are base classes (Ohase) seen during training; those in blue are novel classes (O and unknown during training.
proposed recognizing an open vocabulary of verbs by leveraging large-scale vision-language models (VLMs) [46]. Contrary to their task, we are interested in the scenario of generalizing verbs to an open vocabulary of objects.
Generalizing verbs to novel object categories is challenging when the classes are naturally long- tailed [31, 44]. Models trained on such imbalanced datasets biased towards head verb-object composed actions (when a verb is frequently seen with a particular object). Furthermore, these models suffer from static bias [7, 28] and overfit to the appearance of fixed frames without learning the transformation of objects over time [35]. To alleviate these biases, we decouple the verb and object predictions via separately trained encoders. We propose Object Agnostic Pretraining (OAP) for the verb encoder. OAP is based on a supervised contrastive loss [26]. to which we add a of guiding augmentations. These augmentations are specially designed for egocentric videos and allow the verb encoder to learn object-agnostic representations to improve transfer to novel objects. To knowledge, this work is the first explore contrastive learning of video backbones end-to-end on egocentric videos. In contrast, the augmentations in existing video contrastive learning [12, 45] are designed for Kinetics [6] styled videos where actions can be predicted from a single frame or short clip sampled anywhere from the video. Such augmentations are ineffective for egocentric footage (Sec. of Supplementary).
To recognize novel objects, we design an open-vocabulary object encoder that leverages a pretrained CLIP [46] model. We specifically interested in recognizing active objects those that undergo a state change from the actor's interactions. Active objects are not necessarily handheld [64, 9] e.g., while "beating egg", the actor holds whisk and bowl in his hand, but the egg is the active object undergoing a state change. Recognizing active objects in egocentric videos is challenging even under closed settings due to the many distractor objects in natural environments. To this end, we propose Active Object Prompting (AOP) that guides the CLIP model to understand which object is active. Given a frame containing whisk, bowl, egg batter, hands, and other distractors, AOP aims at learning the context i.e. "beating egg" by generating set of learnable verb-conditions prompts. Conditioning the prompts on the verb features generated by OAP helps the CLIPP recognize the active object.
Egocentric datasets [8, 51] focus on closed set evaluation of actions where a handful of objects unseen. However, for a realistic open vocabulary evaluation, a variety of novel objects is expected be encountered for the first time during inference. To solve this, we create open vocabulary benchmarks on EPIC100 and Assembly 101 by repurposing their original split. Our contributions can thus be summarized as: (i) an object agnostic pretraining of video backbones improve its verb generalization to novel objects, (ii) an active object prompting of CLIP to recognize novel active objects and (ii) open vocabulary action recognition benchmarks on EPIC100 and Assembly 101.
2 Related Works
Egocentric Action Recognition. Egocentric computer vision has garnered huge interest recent years, and massive efforts at scaling up have resulted in the development of datasets like Ego4D [17],

---

SIIC-KITCHENS-AO 100 [8] and Assembly 101 [51]. It has also aided the development of sophisticated video recognition architectures [11, 3, 43, 58]. Recent works [48, 22, 33] focus on learning more object-centric representations by explicitly modeling object transformations over time with off-the- shelf detectors [50] and trackers [5]. However, these models are restricted to closed set of predefined actions and do not scale to the diversity of objects in the real world.
Contrastive Learning in Videos. Videos provide unique opportunities for contrastive learning [19, 41] due to its multimodal and highly redundant nature. Most popular methods follow the self- supervised instance discrimination task [59] where a single positive of an anchor video is contrasted
against a set of negatives. This positive can be an augmented view of the RGB stream [12, 45, 49] or sampled from complementary modality like optical flow [40, 60], text [14, 36, 62, 21, 30), audio [2, 27, 34, 15] or even all three at once [1, 533. Contrary to instance discrimination, recent works [26, 20] propose supervised contrastive learning. They show that sampling multiple positives from the same class as the anchor outperforms instance discrimination. Another line of research uses multiple positives to combat noisy anchor-positive alignment [37]. In this paper, we propose a set of lightweight augmentations to train video backbones for verb prediction. These augmentations result in a noisy alignment due to the camera motion inherent in egocentric videos. Thus, we leverage multiple positives by modifying [37] to combat it.
Learning for an Open vocabulary. The remarkable zero-shot capabilities of large language models (LLMs) [10, 47] and vision language models (VLMs) [46, 23] have widened the possibility of recognition and detection tasks in the visual domain. Of particular interest is the open vocabulary setting, which can recognize novel classes unknown during training. By presenting the novel classes as a textual phrase, it becomes possible to probe the huge corpora of knowledge in LLMs and VLMs. It differs from the traditional zero-shot setting where the novel classes are assumed to be known during training [63] and hence is restricted to the closed vocabulary of predefined classes. Most works for open vocabulary recognition are limited to the image domain [63, 67, 66, 18, 68, 39j. Recently, some works [56, 32, 25, 57] have adapted VLMs for video tasks but primarily focus on zero-shot verb recognition. In contrast,
are interested in a practical hand-object interaction scenario of generalizing verbs performed by hands to any object.
Prompting VLMs. Large Scale Vision Language Models (VLMs) like CLIP [46] and ALIGN [23] trained on an enormous corpus of web data and contain semantically grounded knowledge of images. Prompting [24] refers to designing textual instructions to steer the VLM towards some desired output. Manually engineering discrete text prompts is laborious and requires domain expertise. One alternative is fine-tuning -eeinunigg the VLM encoder for the task at hand; this approach incurs significant computation and is also at risk of catastrophic forgetting [29].The more widely adopted approach is simply learning the prompts by backpropagating the task-specific loss while freezing the VLM encoders [67, 66, 13.. Such an approach is parameter-efficien since only few prompt embeddings are learned while keeping the encoders frozen. In this paper, we are interested in such parameter- efficient prompt learning steer the VL. toward active object recognition.
3 Preliminaries
Consider video dataset consisting of verbs V applied to base set of objects Obuse. The possible set of base actions is Abase = V x Obave = {(v, o) V u V.O Obase) }- During training, only sparse subset of actions are observed
i.e. Atrain C Abase. The goal is to learn model f Xtrain - Abese where Xsrain is the set of videos seen during training. During inference, along with base objects, we also encounter novel objects Onowel that form novel actions Anovel ((v,0) IV E V,O E )novet). We want our model ʃ to predict both base and novel actions Lee.Aest Abase U Anonet. Conventional action recognition follows closed vocabulary evaluation where Atest- = Abase-
4 Method
We train two encoders- - one for predicting verbs and another for predicting an open vocabulary of objects as depicted by the pink and yellow blocks in Fig. 2a. During inference, predictions from the verb and object encoders are composed to generate the action prediction.
not all unobserved actions feasible
3

---

anchor
negatives
verbs (X)
video of
in-cless positive
Verb Encoder
Object Encoder
objectt mixling
objects
Text prompt +
(a)
(b)
gradient
Figure 2: (a) Framework. We train separate encoders for predicting verb i and object ô. The verb encoder operates on a closed set of verbs, whereas the object encoder, trained on Obases can predict any object described by text prompt. (6) Object Agnostic Pretraining of the Verb Encoder. We design an object mixing augmentation to facilitate learning object-agnostic verb representations. Multiple positives sampled for each anchor video, including proposed object mixing (mixed with "cut orange"), temporal gradients of the anchor video, and in-class positives iee videos featuring the same verb ("cut cheese"), The two augmentations corresponding to the same point in the embedding space illustrate that the anchor only needs to minimize its distance from a softmax weighted average of the two (softmax the similarity of the anchor with the two).
Verb Encoder. Encoding verbs requires spatiotemporal modeling, the verb encoder. We want to learn object-agnostic verb representations to encourage generalization to novel objects. We propose Object Agnostic Pretraining (OAP) (described in Sec. 4. which is general pretraining strategy that can be applied to any video backbone and does not add parameters or increase the model size.
use a video backbone as
Object Encoder. We aim to recognize an open vocabulary of active objects. Even with closed vocabulary, finding the active object amongst the many other objects in cluttered scene is difficult. As such, state-of-the
open vocab-
ulary object detectors [18, 68] perform poorly, One option for support is to spatially locate region of interaction in the frame; we pretrained hand-object interaction (HOI) detec- tor, 100DOH [52] as the base of our object en- coder. It provides bounding boxes of the left and right hands and the corresponding object(s)-in -nn contact in a class-agnostic fashion. Note that the object bounding boxes may contain more than one object (Fig. 3). Thus, we propose an Active Object Prompting (AOP) strategy (described in Sec. 4.2) by leveraging CLIP [46] to generate active object labels from the interaction region for both base and novel classes.
Figure 3: Example frame of "cul onion" from
EPIC. (a) DETIC [68] output with EPIC noun vo- cabulary. (b) HOI Detector [52] output. (c) & (d) are crops of (a) & (b) respectively for better visu- alization. The object-in-contact crop O alongside onion also captures knife.
4.1 Object Agnostic Pretraining (OAP)
OAP aims to pull instances with the same verb as the anchor close in the embedding space (irrespective of its object class) and contrast them against instances with different verbs. We randomly sample batch = ((x)) y1 Ui).... (Ix; yi),.., (NN, N, yN JN)) where is the verb class for the video clip 2; E Xtrain and N is the batch size. We first use

---

random photometric augmentations like Gaussian blur and color jittering to create two views of each anchor, thus doubling the batch size to 2.V. Each view then fed the verb encoder fo followed by a projection head go which generates features for the whole batch {21, =2, ..... 22N ). These features are then normalized to the unit hypersphere. The verb encoder is trained with the standard supervised
contrastive loss [26]. Videos sampled from B with the same verb as the anchor are considered
positives. For each anchor 2, we have the bag of positives P. = {2ا1,ト = the loss Lout with temperature parameter Tout:
yc, Vj E [1, 2N]]. We use
exp (2: 2z/ 25/Toue) Tout)
Σ JEP:
log
PPj
exp (2i 2/ (ouss)
REBR
Guiding Augmentations. Sampling in-class positives is still bounded by the verb-object com- positions in the training set. Our intuition is that as long as a verb is being performed in a video, exchanging objects and the presence of other distractor objects in the background should not affect the outcome of the verb encoder. To reduce the reliance on such objects for verb prediction, we propose a set of guiding augmentations diversifyy the set of positives. The augmentations are lightweight and allow new positives to be generated online during training.
To achieve this, first mask static information (objects in the background) from frame of the video clip. The most obvious choice is to use temporal gradient that eliminates any static information and focuses primarily the region with non-zero motion vectors, which refer to the active region" 2. In hand-object interactions, this active region is expected to contain the active object. Furthermore, we exploit the compositional nature of actions and propose a new object mixing augmentation. The idea is simple- we combine two clips with the same verb but different objects to generate new object mixed clips (Fig. 2b). Formally, given two clips from the batch, xy and zy,, having the same verb, we get the object mixed clips F; and T..
x. a V2, - (1-)) (Va)) 1
(2)
2, a (Zz
2 x2) + (1- (V
where Va, is the first-order temporal gradient of r; (which masks static regions) and is the element- wise matrix multiplication; (Vz,)- refers to the inverted gradient image of zj and a is the mixing hyperparameter. To summarize, masking aims at minimizing the impact of distractor objects outside of the active region, and mixing prevents the model from being biased towards lowelevel object information, e.g., color, shape, and hence to the observed verb-object compositions.
Learning from Noisy Augmentations. The basis of our guiding augmentations is the temporal gradient. In egocentric videos, the camera (ego) motion will result in non-zero gradients on static parts of the scene in addition to the true active region. While it is feasible to specially design expensive augmentations that account for camera motion [55, 54]; instead, we propose to adjust the contrastive loss so that it can learn from these noisy but lightweight augmentations. In Lout, all the positives for an anchor contribute equally to the loss; this is desirable for in-class positives where the condition for being a positive is well-defined and fixed, but undesirable for the noisy guiding augmentations. Now, if we move the summation inside the log, the logit for each anchor becomes the average cosine similarity of all the guiding augmentations. The contribution of each augmentation is thus dependent on its similarity with the anchor, where noisy augmentations having low cosine similarity will have less contribution the gradient. With a sufficiently large bag of augmentations, this is expected to reduce the noise implicitly. Such a loss is formulated as Lin in [26]. If G, is our bag of guiding augmentations, our Lin loss is as follows:
exp (z) Zj 2T/Tm)) jEC:
Σ
C.
c log
Σ exp (24 2x T(n)
KEBY
where Tan is the temperature parameter. This loss is also similar MIL-NCE [37] for T. 1, which is used for learning from noisy video-text pairs. In our case, a lower softmax temperature (< 1)
Pattive regions may still contain static objects, e.g., for the action of "cutting onions", the cutting board which is present in the active region is static. This object is not a distractor and is relevant for action prediction.
5

---

Training
Learnable
Image Encoder
Frozen
Encoder
Detector
Osans
object
almond
apple
posti perfic
CLIP Text
B: B2
fa
Br f. B3 fo
yoghurt
Prompt Tokens
cross-entropy loss
alarm
Me
pott un pont
CLIP Text Encoder
A1 N2
No.
B3 B1
B3
turkey
orange
nond
f. fo No
N. Br Bg fo
Testing
Image Encoder
Encoder
Detector
object crop
Figure 4: (Top) Parameter-efficient training for Active Object Prompting, Keeping the CLIP encoders frozen, the base vocabulary initialized with the CLIP-pretrained word embeddings is fine-tuned on the training set. A set of prefix and postfix prompt vectors conditioned on the verb representation are also learned. (Bottom) Testing with Active Object Prompting. The fine-tuned embeddings used for the base objects, whereas the CLIP- LLIP-pretrained embeddings used for novel ones. The Iearned prefix and postfix prompts are shared among all object classes.
results in better performance. Please refer to the Supplementary for theoretical insights into the loss formulation (Sec. D) and illustrations of guiding augmentations (Sec. F). Thus, the overall loss function is L = Lout + ALin, where is a weighting hyperparameter.
4.2
Active Object Prompting (AOP)
This section discusses leveraging CLIP predict the active object from crop with multiple objects. We assume no prior knowledge of the set of objects observed during inference i.e. any object is expected be predicted by our model long is described by textual phrase (open vocabulary).
The overall framework of AOP shown in Fig. First, the given set of object classes O are converted into text prompts by prepending and appending each class with set of prefix and suffix tokens. For object o, with word embedding Wo,, a prompt text t, can defined as {pre,- ..... prey, Wo, post, .... post,, ). where pre, are k prefix tokens, post, are k2 post- fix tokens. The prefix and postfix prompts, together known as context prompts [67], are essential in guiding a pretrained CLIP to solve desired task. The context prompt tokens shared across all classes and can fixed or learned.
Fixed prompt tokens must be manually engineered by humans, which is laborious and requires domain expertise. Also, prompting CLIP, which operates in a continuous space with discrete human-readable text prompts. limits its usage. Hence, we propose to learn our prompts via two prompting strategies (i) verb-conditioned context prompt generation and (ii) fine-tuning base object vocabulary.
Verb-Conditioned Context Prompt Generation. As shown in Fig. 3, given a crop containing onion, knife cutting board and hands, want the CLIP to predict onion as the active object by understanding the context i.e. the action of cut onion. To this end, we propose verb-conditioned context prompts. Given the video æ, add the verb representation of learned using OAP the context prompts as shown in Fig. 4 f.e. pre, = pre, + (f(()()) and post, = post, + $ (fo(())) where So is the pretrained verb encoder and + r. is a projection network used project the visual feature to the language domain.

---

Fine-tuning the Base Object Vocabulary. Egocentric videos often capture many different objects in the same environment. Semantic variability of the objects can be very fine-graiin grained e.g. "boom" " vs. "arm" in Assembly 101 [51]. Such fine granularity is not fully captured by CLIP pretrained on data crawled from the web. We treat this as a vocabulary learning problem with the goal of making fine- grained object categories discriminative in the continuous CLIP feature space. To this end, we fine- tune the pretrained object embeddings Wo, for all base objects seen during training i.e. Vo, Obase- This simple fine-tuning allows CLIP to adjust the object embeddings in its continuous space make them discriminative for optimal performance.
The CLIP-text encoder is kept frozen for both the prompting strategies, making them parameter- efficient. During testing (Fig. 4 bottom), we use the fine-tuned embeddings for the base objects and the original pretrained embeddings for the novel objects. The verb cordiiondt context prompts are unique for every instance.
Experiments
Datasets. We conduct experiments on three datasets: EPIC100 [8] features 100 hours of ego- centric footage of daily kitchen activities annotated with 97 verbs and 300 interacting objects. Assembly 101 [51] is multi-view procedural activity dataset featuring 4321 videos of people as- sembling and disassembling 101 take-apart toy vehicles and is annotated with 24 verbs and 90 objects. We use the egocentric view e3 (see [5I]), which best captures the hands and interacting objects. Something-Else [35], is a auman-object interaction subset of SSv2 [16] with 174 verb phrases e.g. "picking something up". Something Else is not strictly egocentric but shares interesting properties such as diverse hand-object interactions and active objects undergoing state change. We use the compositional split, where all verb phrases are seen during training but applied to novel objects during testing. Interacting objects per video are not annotated; hence, it is unsuitable for our open vocabulary action recognition task but is an ideal testhed for evaluating our verb encoder and OAP with other baselines.
Implementation Details. We use S3D [61] as the verb encoder for the majority of our experiments and ablations as it is popular architecture for video contrastive learning [20, For all datasets, the verb encoder is pretrained with the OAP contrastive scheme for 300 epochs and then fine-tuned for verb classification for another 100 epochs. To see how OAP scales with larger backbones, we For the hand-object interaction support, we use the pretrained 100DOH [52] detector without tine-tuning on the target dataset. For the CLIP Image Encoder, we experiment with ViT-B/16 and the larger ViT-L/14 backbones. Keeping both the CLIP image and text encoders frozen, prompts are learned using AOP for 20 epochs. Unless otherwise stated, we choose h and kg, the number of learnable prefix and postfix prompts, to be 16 each. For additional implementation details, please refer to Sec. B of the Supplementary.
also evaluate Mformer-B [43] on Something-Elke
Table 1: Proposed open vocabulary benchmark splits.
Dataset
V
Obase
Onove!
#train seg
Meett seg.
EPIC100-OV
97
190
110
63k
13.8k
Assembly10--Ov
24
60
30
56.2k
15k
5.1 Open Vocabulary Benchmark
Since the original action recognition split of EPICI00 and Assembly101 are unsuitable for open vocabulary (OV) evaluation, we create OV benchmarks EPICI00-OV and Assembly101-OV, re- spectively, by dividing the complete set of objects into base and novel categories. The base objects are seen during training, whereas the novel objects are kept only for testing. We also incorporate base actions in our test set to evaluate the closed- worldd performance of the models. The benchmark statistics are provided in Tab. 1. We provide more detailed explanation of the OV splits Sec. A of the Supplementary. We use Top-1 Accuracy [8, 51] as our evaluation metric and report results for both closed and novel classes. Additionally, we report the harmonic mean (HM) [38] of the closed and novel accuracies consolidated metric for comparing different methods.
7

---

Table Comparison of methods for open- vocab object recognition on EPIC100-OV. Zero-shot evaluation.
Table 2: Compositional Action Recognition on Something-EBe Elss [35]. Extra Module refers to if any additional parameterized module has been used on top of the video backbone.
Object Top- (%)
Model
Base
Novel
HM
Extm Module
Top-1 (5R
Top-5 (S)
B3D [6]
X
46.8
72.2
S3DI
52.2
1.6
3.1
Slowfast [1]]
52.2
80.3
CLIP (VIT-B/16))
20.2
0.6
1.2
STRG,STIN [35]
Slowtsst-RN [33]
56.2
52.9
81.3
80.8
CLIP (VIT-B/16)
7,9
16.3
10.6
DETIC [68]
12.0
2.3
3,9
[61] OAP (83D)
49.3
77.3
C2 (VTT-B/16) [25]
19.1
9.9
10.3
X
53.7
80.5
C4 (VIT-B/16) [25]
39.9
7.3
12.3
Mformer- B [43]
60.2
85.8
OAP (Morrmrr-B)
61.8
86.2
AOP (CLIP VIT-B/16)
41.9
19.2
26.3
AOP (CLIP VIT-L/14)
47.8
22.6
30.7
5.2
Object-Agnostic Pretraining (OAP) vs State-of-the-Art
How effective is OAP in generalizing verbs to novel objects? Tab. 2 shows OAP applied to video backbones S3D [61] and Mformer-B [43] the Something-Elss compositional split. S3D with OAP outperforms Slowfast [11] by 1.5% in Top-1 acc. even though Slowfast is a stronger video backbone. It also outperforms. by 0.8%, the sophisticated Slowfast+IRN [33], which adds an extra transformer to SlowFast to explicitly model the hand-object interactions. Finally, S3D with OAP still competitive with respect STRG.STIN+OIE-NL [35], which is an ensemble of various video models, despite being a single backbone without any extra parameters or compute. Mformer-B state-of-the -h-ttheart video architecture comfortably outperforming all other models; adding OAP improves its Top-1 performance by 1.6%, demonstrating that our pretraining is robust to the scale of the backbone.
5.3 Active Object Prompting (AOP) vs. State-of-the-Art
Tab. compares AOP with similar open vocabulary baselines. The first two rows with i refer to zero-shot baselines of S3D and CLIP, where training is done using cross-entropy over all classes (base+novel), CLIP refers to the linear probing of CLIP's frozen visual encoder. Both models overfit to the base classes and fail to generalize to novel ones. DETIC [68], a of-te-of-theart open vocabulary object detector, improves novel object performance over S3D marginally by 1.3%. It is non-trivial for DETIC to recognize which object is active from a set of per-frame object detections e.g. for the action "beating egg" ", a whole egg is not present in any frame. C4 [25] adapts CLIP for video tasks by adding transformer layer top of per-frame CLIP features; again, it overfits to the base classes. Finally, our AOP using both CLIP backbones significantly outperforms all other baselines AOP improves over the closest competitor (C4) having the same backbone in ViT-B/16 by 14.0% in HM.
5.4 Open Vocabulary Action Recognition
Tab. 4 presents action recognition on open vocabulary splits of EPIC and Assembly101. S3D predicts verbs and objects using two heads but share the same backbone. For a fair comparison with our decoupled verb and object encoders, we also decouple the S3D training, one for predicting verbs and another for objects (reported as 2x S33).. As mentioned in Sec. 5.3, the S3D baseline predicts novel objects in zero-shot fashion where the novel categories are already assumed to be known during training. Both S3D baselines fail to predict novel objects on both benchmarks. OAP improves verb generalization to novel objects by 1.3% and 4.8% over S3D on EPIC100-OV and Assembly101-00 respectively. Additionally, the dlosed-world verb prediction is also enhanced by 1.6% and 7.0% over S3D on EPIC100-OV and Assembly 101-OV, object prediction performance in both benchmarks. The novel object prediction performance on Assembly 101 could be further enhanced by adopting an HOI detector that is robust to domain shifts in the target datasct since Assembly 101 egocentric videos monochrome.
Assembly101-OV, respectively. AOP significantly improves novel
8

---

Table 4: Open Vocabulary Action Recognition Results.
Closed Top-1 1op-1(0 (96)
Novel Top-1 (%)
HM Top-1 1pp1(c (%)
Dataset
Model
Verb
Object
Action
Verb
Object
Action
Verb
Object
Action
S3D
62.5
50.8
37.6
40.1
48.8
3.3
0
EPIC100-OV
2x
61.5
52.2
37.9
0.1
46.9
0.2
OAP AOP
64.1
47.8
35.9
41.4
22.6
11.2
50.3
30.7
17.0
S3D
50.6
34.6
19.9
40.3
0
44.9
0
0
Assembly10--01
2x
50.7
37.9
20.2
41.8
0
45.8
0
OAP + AOP
57.6
30.1
21.6
45.1
6.7
3.7
50.5
11.0
6.6
Table 5: Effect of guiding augmentations for OAP on Something Elase.
Loss
Temporal Gradients
Object Mixing
Top-I (%)
Top-5 (%)
Supervised
X
49.3
77.3
Lout
X
50.1
78.1
Cout
/
52.1
79.3
Lout + ALin Cout Xiin
X
50.3
78.4
/
53.7
80.5
5.5 Ablations
Guiding Augmentations. We study the effect of using guiding augmentations on Something-Be in Table 5. Using Lout already improves performance over supervised training, consistent with the findings in supervised contrastive learning [26, 20]. Introducing verb-specific guidance via temporal gradients and especially proposed object mixing augmentations boosts the performance further by 3.6%. This demonstrates that OAP with guiding augmentations enhances the generalization of a backbone like S3D without requiring additional parameters.
Active Object Prompting. We thoroughly ablate various learnable prompting strategies in Tab, on EPIC100-OV. VIT-B/16 has been used the image encoder for all experiments here. For the rows where the context prompts are not learnable, we use the fixed prompt "A picture of <object>." Introducing the HOI detector generate active object crops improves HM by 3.1% full images. Our AOP mainly consists of two proposals fine-tuning base object vocabulary and conditioning context prompts verb representations from the video. Most of the gain in Tab. 6 comes from these two proposals. Fine-tuning Obase is expected to improve closed set performance, but we also find 2% improvement over novel categories. This alludes to the fact that learning object names decreases false positives when tasked to predict a novel category. Verb-conditioned prompting brings further improvement of 3.9% in HM. In particular, boosts the performance over base categories, which is consistent with the findings of conditional prompting learning [66, 67].
using
Next, analyze using temporal modeling top of the CLIP features. Similar C4 [25], add single-layer transformer with eight heads over the per-frame CL.IP features. As expected, the model overfits the base categories seen during training, only achieving 4.8% over novel categories.
Table 6: AOP ablations on EPIC100-OV. ViT-B/16 is used as the image encoder. AOP-Base and
AOP-Novel refer to the best performef models on base and novel object categories.
HOI
Learnable
Verb Conditioned
Finetune
Temporal modelling
Base
Novel
HM
Detector
Context Prompts
Obase
x
X
X
5.4
12.1
7.5
x
x
x
7.9
16.3
10.6
x
/
x
20.2
18.3
19.2
/
X
21.6
16.6
Nov-Novel
/
X
29.6
18.4
22.7
/
x
46.9
4.8
AOP-Base
/
/
51.0
11.9
AOP-Base + AOP-Novel
41.9
19.2
26.3

---

Finally, we ensemble the predictions of the best-performing model on base (AOP-Bass (oBeesse) and novel (AOP-Novel) categories to obtain the best HM.
5.6 Limitations and Outlook
In this paper, assume that every verb-object composition is feasible. The compositions could be pruned to feasible subset to reduce the action search space. Obtaining feasibilities, and distinguishing them from common-sense settings, however, is non-trivial, e.g., "cut thermometer", albeit feasible, is unlikely to be observed. Our verb-conditioned prompting of CLIP provides a weak feasibility check. However, CLIP is well known to have a strong object bias [42] over verbs, and may still yield infeasible active object predictions. One direction to improve the feasibility of actions is to learn object affordances; leave this for future work.
Conclusion
This paper proposed a new task of open vocabulary action recognition for verb-object composed actions and a novel approach to address it. We decouple the verb and object predictions. The verb encoder trained with an object-agnostic pretraining strategy to improve its generalization to novel objects. Our object encoder leverages pretrained CLIP model using active object prompting. This guides the CLIP to predict an open vocabulary of interacting objects. Our experiments demonstrate that the proposed approach can recognize hand-object interactions even when the specific actions objects are not in the training data.
Acknovledgements: This research/project is supported by the National Research Foundation, Singapore and DSO National Laboratories under its AI Singapore Programme (AISG Award No: LISG2-RP-2020-016 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
References
[1] lean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapu- ram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervieed multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[2] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435-451, 2018.
[3] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun. Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836-6846, 2021.
[4] Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13349 13358, 2021.
[5] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In 2016 IEEE international conference image processing (ICIP), pages 3464-3466 IEFE, 2016.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308. 6308, 2017.
[7] Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. Advances Neural Information Processing Systems, 32, 2019.
10

---

[8] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision: collection, pipeline and challenges for epic-kitchens-100. International
Journal of Computer Vision, 130(1))33-55, 2022.
[9] Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Amlan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. Epic-kitchens visor benchmark: Video segmentations and object relations. Advances Neural Information Processing Systems, 35: 15:15545-,3758, 2022.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019.
[12] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3299-3309, 2021.
[13] Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video relation detection. arXiv preprint urXiv:2302.00268, 2023.
[14] Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox Coot: Co- operative hierarchical transformer for video-text representation learning. Advances in neural information processing systems, 33:22605-22618, 2020.
[15] Yuan Gong, Andrew Rouditchenko, Alexander H Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, and James R Glass. Contrastive audio-visual masked autoencoder. In The Eleventh International Conference Learning Representations, 2022.
[16] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitte al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842- 5850, 2017.
[17] Kristen Grauman, Andrew Westbury, Eugene Bymee Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995- 19012, 2022.
[18] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference Learning Representations, 2022.
[19] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) volume 2, pages 1735-1742. IEEE, 2006.
[20] Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video rep- resentation learning. Advances in Neural Information Processing Systems, 33:5679-5690 2020.
[21] Tengda Han, Weidi Xie, and Andrew Zisserman. Temporal alignment networks for long- term video. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 2906-2916, 2022.
[22] Roei Herzig, Elad Avr-har,,am, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In Pro- ceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pages 3158-3159, 2022.
11

---

[23] Chao Jia, Yinfei Yang, Ye Xia. Yi-Ting Chen. Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and isson-languaan representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMIR, 2021.
[24] Zhengbao Jiang, Frank F Xu, Jum Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423 4388, 2020.
[25] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 105-124, Springer, 2022.
[26] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samn,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.
[27] Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018.
[28] Matthew Kowal, Mennatullah Siam, Md Amirul Islam, Neil DB Bruce, Richard P Wildes, and Konstantinos G Derpanis, deeper dive into what deep spatiotemporal networks encode: Quantifying static vs. dynamic information. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13999-14009, 2022.
[29] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang, Fine- tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022.
[30] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems, 35:7575-7586, 2022.
[31] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailes recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2537-2546, 2019.
[32] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508:293 304, 2022.
[33] Jian Ma and Dima Damen. Hand-object interaction reasoning. In 2022 18th IEEE International Conference Advanced Video and Signal Based Surveillance (AVSS), pages 1-8. IEEE, 2022.
[34] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song. Active contrastive learning of audio-visual video representations. In International Conference on Learning Representations, 2021.
[35] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else, Compositional action recognition with spatial-temporaa interaction networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1049-1059, 2020.
[36] Antoine Miech, Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivie, and Andrew Zisserman. Think- ing fast and slow: Efficient text-to-visual retrieval with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826-9836, 2021.
[37] Antoine Miech, Jean-Bap Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEFE//VV Conference on Computer Vision and Pattern Recognition, pages 9879-9889, 2020.
12

---

[38] Tushar Nagarajan and Kristen Grauman. Attributes as operators: factorizing unseen attribute- object compositions. In Proceedings of the European Conference on Computer Vision (ECCV). pages 169-185, 2018.
[39] Nihal V. Nayak, Peilin Yu, and Stephen Bach. Learning to compose soft prompts for composi- tional zero-shot learning. In The Eleventh International Conference on Learning Representa- rions, 2023.
[40] Jingcheng Ni, Nan Zhou, Jie Qin. Qian Wu, Junqi Liu, Boxun Li, and Di Huang. Motion sensitive contrastive learning for self-supervised video representation. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 457 457-474. Springer, 2022.
[41] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1880000778, 2018
[42] Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach, Exposing the limits of video-text models through contrast sets. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3574-3586, 2022.
[43] Mandela Patrick, Dylan Campbell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichten- hofer, Andrea Vedaldi, and João Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. Advances in neural information processing systems, 34:12493-12506, 2021.
[44] Toby Perrett, Saptarshi Sinha, Tilo Burghardt, Majid Mirmehdi, and Dima Damen. Use your head: Improving long-tail video recognition. arXiv preprint arXiv:2304.01143 2023.
[45] Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964-6974, 2021.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, (8):9, 2019.
[48] Abhinav Rai, Fadime Sener, and Angela Yao. Transformed rois for capturing visual transforma- tions in videos. Computer Vision and Image Understanding, 224:10355 2022.
[49] Adria Recasens, Pauline Luc, Dean-Baptiste Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pārräucean, Florent Altché, Michal Valko, al. Broaden your views for self-supervised video learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1255-1265, 2021.
[50] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[51] Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang. and Angela Yao. Assembly101: large-scale multi-view video dataset for understanding procedural activities. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11096-221066 2022.
[52] Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9869-9878, 2020.
13

---

[53] Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Roge- rio S Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at once-multi-moda fusion transformer for video retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 20020-20029, 2022.
[54] Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. In 2021 International Conference on 3D Vision (3DV), pages 110-919. IEEE, 2021.
[55] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision, pages 3551-3558, 2013.
[56] Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action 2021.
recognition. arXiv preprint arXiv:2109.0880
[57] Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. arXiv preprint arXiv:2304.03307, 2023.
[58] Yuan Wu, Yanghao Li. Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory- augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13587-13597, 2022.
[59] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non parammereitic instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
[60] Fanyi Xiao, Joseph Tighe, and Davide Modolo. Maclr: Motion-aware contrastive learning of representations for videos. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pages 353-370. Springer, 2022.
[61] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spa- tiotemporal feature learning: Speed accuracy trade-offs in video classification. In Proceedings of the European conference computer vision (ECCV), pages 305-321, 2018.
[62] Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:2109, 14084, 2021.
[63] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open- vpcablocabylary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14444 2021.
[64] Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand- object segmentation: Dataset, model, and applications. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 127-145. Springer, 2022.
[65] Chen Zhao, Dawei Du, Anthony Hoogs, and Christopher Funk. Open set action recognition via labei evidential learning. arXiv preprint arXiv:2303. /2698, 2023.
[66] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[67] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. International Journal of Computer Vision, 13009):3337-244,, 2022.
[68] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting owenty-thousane classes using image- level supervision. In Computer Esion-ECCYV 2022: 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 350-368. Springer, 2022.
14


Reviews:
[{"review_id": "bERbvZ3gtU", "reviewer": "Reviewer_jmBw", "summary": "Learning representations that generalise across novel objects and actions is an important topic of research that is of great practical utility. This is especially true due to most datasets containing only a few combinations of object-action pairs or triplets, which often leads to model overfitting. A key question is, how to overcome this dataset limitation. The approach tackles this by decoupling the object representation and verb representation by using object word embeddings and verb-agnostic visual representations. \n\n- Task: Given objects and actions seen during training, generalize to new actions that are performed with either seen or novel objects.\n- Approach: Decompose predictions of objects and verbs. This overcomes the verb-object overfitting that might cause lack of generalization to novel verb-object combinations.\n    - Object-agnostic pretraining of a verb encoder\n    - Prompt-based object encoder (CLIP features for a given prompt). This enables an open vocabulary (of objects that CLIP is trained with).\n- Contributions:\n    - Open vocab. benchmarks on EPIC-KITCHENS-100, Assembly101\n    - Approach that decomposes object and verb encodings is better than prior approaches at recognizing novel verb-object combinations.", "questions": "**Clarity / Details**\n    1. Fig. 4 and text don’t go well together, or I’m confused. f_\\theta = f_v and g_theta (f_theta(.)) = f_v^L ? \n    2. In Fig. 4, the verb encoder is shown to be frozen. This confuses me. My understanding was that the training procedure in Sec. 4.1 indeed updates the parameters of the verb encoder. \n    3. The input to the verb encoder is the whole video. Is it the same for the HOI detector? Then how are image crops sampled from this video (to pass through CLIP Image Encoder)?", "limitations": "Yes, they discuss limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Decoupling the object representation into a function of (a) object word embedding, and (b) object-agnostic verb embedding is a promising formulation. While the specific details of the verb-agnostic encoder training algorithm and prompting of the text encoder might change and improve, the basic formulation that decouples the two different semantic concepts, makes sense. This  leverages CLIP’s pretrained object-text representation that is trained over large datasets containing many variations of objects.  \n2. CLIP text encoder’s input “context prompt” into a function of (noun CLIP word embedding, object-agnostic verb encoding ) is interesting. The paper mentions that the “pre” and “post” are necessary for the encoding to work. Is this a novel observation or is there prior literature on the topic? \n3. In addition to the verb encoder, the paper aims to more accurately represent the differences between fine-grained objects via CLIP’s text encoder. To achieve this, the approach also involves fine-tuning the object embeddings (on seen object classes). The paper demonstrates that this achieves a significant improvement in performance.", "weaknesses": "1. There are many small decision choices involved in the successful implementation of the approach. There are different types of augmentation and contrastive sampling. Further, there are different representations for nouns, verbs. The noun object representation is also fine-tuned. It makes the overall approach appear to be the sum of a series of smaller tricks. While it makes the paper a bit harder to read and understand, from a practitioner’s standpoint, this might indeed contribute to new knowledge / understanding of what’s possible. Further, it’s nice that Table 5 evaluates the effects of these different decision choices. \n2. The visual feature extraction heavily relies on a HOI detector to obtain the “active object”. This only works on hand-based interactions. This is a fairly narrow set of actions compared to overall possible object-verb interactions. However, one can imagine that this can also be extended to object detectors and relationship-detectors in non-ego-centric RGB videos. Yet again, the approach would have to rely on the high performance of these detectors. A concrete question along these lines is: how good is the HOI detector? What’s the UB performance of the proposed approach assuming perfect HOI detection? \n3. There’s an assumption that non-static ⇒ active objects ⇒ objects involved in action. And vice-versa, i.e., static objects are *not* of interest. The masking (or not) of objects is based on this assumption? Is this a valid assumption? I raise this point because the validity of this assumption requires some discussion, and the implementation of this assumption (masking), is also understandably imperfect, and results in noisy augmentations. \n4. To produce a training set containing diverse objects for the same verb, the approach does data augmentation by masking out certain objects, and alpha-blending other objects in. This results in unrealistic, noisy images. The MIL-NCE-like loss appears to counter this effectively. However, is it possible to perform better augmentation like [54, 55]? How much will performance improve with better augmentation?\n5. Limitations: The approach is only as “open” as the vocabulary of objects that CLIP’s feature space encodes. It is also heavily dependent on the performance of the HOI detector.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and acknowledging our open vocabulary benchmark, our method’s potential and our ablations in Table 5. \n\n**S1**\n\n *“This paper mentions that the “pre” and “post” are….”* - Using learnable prefix and postfix context prompts [13, 25, 66] are used as a (parameter-efficient) alternative to manually engineering prompts [24, 46] which is laborious and requires domain expertise. Some papers like ViLD [18] use an ensemble of 63+ prompt templates to report their final result.\n\n**W1**\n\nIn our approach, the main contributions are:\n- **object mixing augmentations** to generate object-agnostic verb representations\n- **use of L_in** to mitigate the noise in object mixing augmentations introduced as a result of the persistent camera motion in egocentric videos. In\n- **verb-conditioned** prompt generation to provide additional context to CLIP when the active object is ambiguous e.g. for the action of “beating egg”, the objects being interacted with are whisk and bowl, but the active object is egg.\n- **fine-tuning** base object vocabulary which allows CLIP to adjust the object class names in a continuous space that helps differentiate discrete fine-grained concepts like ‘boom’ vs ‘arm’.\n\n**W2**\n- *\"The visual feature extraction heavily relies...\"* - In presently available large-scale egocentric datasets like Ego4D, EPIC-KITCHENS and Assembly101, all annotated actions are hand-object interactions where the motion of the hand is described by a verb and the object by a noun. One advantage of our pipeline is that it is sufficiently general and individual components can be replaced with a better one in the future. In this essence, our HOI detector can be replaced with object/relationship -detectors.\n- *\"Yet again, the approach would have to rely on...\"* - We acknowledge that using an HOI detector contributes to our success as can be observed from Tab. 5 (+3.1%). However, its baseline performance is still poor and further ablations demonstrate that design strategies such as fine-tuning base object vocabulary (+8.6%) and verb-conditioned prompting (+3.9%) are extremely beneficial for AOP.\n- *\"A concrete question along these lines is...\"* - Thank you for this great suggestion. However, the HOI detector [52] cannot be evaluated directly on EPIC100 nor Assembly101 because these datasets do not have ground truth bounding box ground annotations. The reason for choosing [52] as our HOI detector is because of its strong cross-dataset generalization (at worst, it obtains 92.9% of the mAP of training and testing on the same dataset). Also, [52] is trained on a subset of EPIC and hence will not confound the AOP ablation study on EPIC100.\n\n**W3**\n\nWe don’t assume that static objects are not of interest. Rather, we assume that actions are related to non-static regions that we refer to as active regions in the paper `(L167-L168)`. Active regions may still contain static objects, e.g. for the action of “cutting onions”, the cutting board itself is static but relevant for the action. Our use of masking aims to minimize the impact of distractor objects outside of the active region.\n\n**W4**\n\nWe thank the reviewer for this suggestion. We expect expensive handcrafted augmentations, especially NeuralDiff [54] to improve performance as it can mitigate the impact of camera motion. However they cannot be generated online during mini-batch training and need to be pre-computed offline. Generating static views from egocentric videos is a research topic in itself [54] and is beyond the scope of our work.\n\n**W5**\n\n- CLIP’s feature space encodes a variety of “concepts” which are not necessarily objects but certainly have a bias toward them i.e. low-level properties like texture, shape, color, etc. to high-level properties like scene and context. Exploring the limitations of CLIP [a, 42] and addressing them to perform better compositional reasoning [b], verb reasoning [25, 57], etc. is a broad research topic. Our main aim is to recognize an open vocabulary of active objects using a VLM like CLIP and we are the first to do so. However, our entire pipeline and training strategies are general where individual components like CLIP can be replaced with a better VLM if it is proposed in the future.\n- Moreover, targeting open vocabulary using CLIP is already established in object detection [18, 68] and our proposed AOP outperforms all these baselines as shown in Tab. 3.\n\n**Questions:**\n\nFig.4 shows training/testing protocol for AOP. The verb encoder, already trained using OAP `(Sec. 4.1)` is kept frozen during AOP `(Sec. 4.2)` to prevent the object-agnostic verb features from getting corrupted. HOI detector takes each frame of the video as input. Each frame is then cropped on the basis of the detected hand-object interaction and then passed to the CLIP image encoder. Further details for the HOI detector are provided in `Sec. B` and `Sec. E.2` of the supplementary.\n\n[a] M. Lewis et al., “Does CLIP Bind Concepts? Probing Compositionality in Large Image Models”, arXiv 2022\n\n[b] N. Nayak et al., “Learning to Compose Soft Prompts for Compositional Zero-shot Learning”, ICLR 2023"}, {"review_id": "84yT2Z1bfO", "reviewer": "Reviewer_KvE6", "summary": "Existing approaches for egocentric action recognition considers the problem as recognizing actions from a closed set. The manuscript instead proposes to consider this as an open set recognition problem where some objects are unseen during training. Towards, a novel benchmark for evaluating open set egocentric action recognition is proposed using two egocentric action recognition benchmarks. The manuscript also proposes an approach for open set egocentric action recognition. The key idea behind the approach is to decouple the verb representation from interacting objects followed by using a prompt based object encoder. The proposed approach shows strong improvements in the proposed benchmark when compared against existing closed set action recognition baselines.\n", "questions": "Please see the weaknesses section for the main questions regarding the manuscript.\n\nThe main problem that the proposed approach is trying to address is related to the fact that existing approaches consider that there is only a fixed set of actions (verb-noun pairs that are observed in the training set). Instead the proposed approach is trying to develop an approach for open set recognition (some object classes are not observed during training). However, the AOP based object encoder still functions in a way that the set of possible objects are already known since the object names are required for the fixed prompt. This contradicts the main objective of the paper to develop an approach for open vocabulary egocentric action recognition. A true open set recognition approach requires a generative model instead of the discriminative one used in the proposed approach.\n", "limitations": "One of the limitations of the proposed approach is the fact all verb-noun pairs cannot constitute a plausible action. This is discussed in the limitations section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The open set egocentric recognition problem addressed in the manuscript is novel, non trivial and highly relevant to the research community. The development of the two benchmark datasets will be highly beneficial for future research.\n\nThe paper is written in a clear and concise manner making it easy to understand the motivations and the proposed approach. An extensive set of ablations are also presented in the manuscript making it clear the impact of the various contributions. Evaluation of the proposed approach on the benchmark and comparison against the existing close set recognition approaches shows its effectiveness in addressing open set egocentric action recognition.\n", "weaknesses": "The main weakness of the manuscript is that the proposed approach is compared only against simple baselines based on standard action recognition approaches. The proposed approach makes use of the CLIP model for object recognition. CLIP is widely known for its performance on zero shot and few shot object recognition. One important baseline would be to use the S3D model for verb recognition (without the proposed OAP training) and the CLIP model for object recognition. The CLIP model can also be used for both verb and object recognition. Without this, it is not clear how effective the proposed approach is for open set egocentric action recognition. Similarly since the novel objects are composed of the tail classes from the respective datasets, using CLIP model would be highly effective since it is good at recognizing few shot and zero shot classes. So the comparisons reported in the paper do not reveal the true impact of the proposed approach.\n\nSome of the design choices behind the proposed approach are not motivated well or empirically proven. For example, why use pre and post context prompts? Why not add them as prefix or suffix to the fixed prompt? For AOP, the verb features are added to the context tokens. Why is this preferable over concatenating the verb features to the input embeddings of the text encoder?\n\nAfter the OAP pretraining, the verb encoder is further finetuned for supervised verb recognition. Wouldn’t this result in the model losing its object agnostic verb prediction capabilities?\n\nEven though the manuscript presents various ablation studies to validate the effectiveness of various contributions, the experiments are conducted on different datasets. This makes it difficult to analyze the impact of the different contributions of the proposed approach. The reviewer recommends the authors to select a single dataset to present in the ablation analysis. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and acknowledging our novel task, benchmarks and the set of ablations.\n\n**W1**\n\n- In the main paper, we have provided the results for *“One important baseline would be...”* The 1st row of `Tab. 4` provides S3D (without OAP) results on EPIC100-OV. The 2nd and 3rd row of `Tab. 3` provide the CLIP-only baseline for active object recognition. Please refer to `Sec. 5.3` for a detailed description of the CLIP-only baseline.\n- *“The CLIP model can also be used...”* - We ran preliminary experiments with CLIP for verb recognition. However due to CLIP’s inherent object bias [42] and lack of temporal knowledge, we omitted these results from our paper as it would not be a fair comparison. Additionally, our verb set is neither open vocabulary nor zero-shot, therefore existing baselines already outperform CLIP. We are happy to add this result in our supplementary.\n- *“Similarly since the novel objects are ….”* - We thank the reviewer for acknowledging our motivation for using CLIP. However, as observed in `Tab. 5`, naively using CLIP doesn't result in good performance, hence we propose AOP to improve CLIP’s performance on both base and novel object classes. Furthermore, we also provide comparisons with DETIC [68] (a SoTA open vocabulary object detector) and C4 [25] (a SoTA zero-shot verb recognition model), both of which are based on CLIP. We outperform DETIC by *26.8%* and C4 by *18.4%*.\n\n**W2**\n\n- *\"For example, why use pre and post context prompts?\"* - We use pre and post context prompts because it can be learned directly via backpropagation rather than manually engineering different versions of fixed prompts [24, 46] which is laborious and requires domain expertise. \n- *\"Why not add them as prefix or suffix to the fixed prompt?\"* - The pre and post context prompts are added as prefix and suffix respectively to the fixed prompt where the content of the fixed prompt might change. Since our purpose is to learn the prefix and postfix context prompts, keeping the fixed prompt simple results in better performance. In the following table, we provide the comparison between the two prompt versions which empirically supports our claim. Other AOP components are restricted.\n\n| **Prompt**     | **Base** | **Novel** |  **HM**  |\n|----------------|:--------:|:---------:|:--------:|\n| \"$W_{pre}, W_{a\\ photo \\ of \\ a \\ <object>.} , W_{post}$\"     |   21.4   |    7.2    |   10.8   |\n|  \"$W_{pre}, W_{<object>} , W_{post}$\" | **22.0** |  **9.6**  | **13.4** |\n\n- *\"For AOP, the verb features are added to the context tokens.\"* - If we understood correctly, the reviewer asks if we add the verb features to the context tokens. The verb features are indeed added to the embeddings of the context tokens. We empirically found that addition is better than concatenation although the difference in performance is negligible. Also it is a standard practice in conditional prompting [66, 67]. In the following table, we provide concatenation vs addition results, where for concatenation, the verb feature from S3D (2048-D) is first concatenated to the context prompt embedding (512-D for CLIP-ViT-B/16) and then the projection network $\\Phi_L$ is applied to project it back to 512-D. Other AOP components are restricted.\n\n| **Method**    | **Base** | **Novel** |  **HM**  |\n|---------------|:--------:|:---------:|:--------:|\n| concatenation |   24.9   |    13.1   |   17.2   |\n| addition      | **25.4** |  **13.2** | **17.4** |\n\n**W3**\n\nWe thank the reviewer for raising this important question. The model does not lose its object agnostic capabilities when fine-tuned for supervised verb recognition as is evident from our experiments and ablations. To ensure this, we adopt the following fine-tuning strategy. After OAP pretraining, a linear classification layer is added to the encoder whose weights are initially randomly sampled from a normal distribution with 0 mean and 0.01 standard deviation. Now during fine-tuning, if the classification layer is trained with learning rate L, then the base network (which has OAP pretrained weights) is fine-tuned with a learning rate of L/10. This ensures that object-agnostic verb representations are not forgotten. The learning rate L used for fine-tuning itself is very low (1e-3). We find that fine-tuning using the above strategy outperforms linear probing the object-agnostic verb features. This is also a standard practice in UberNCE [20].\n\n**W4**\n\nWe thank the reviewer for this suggestion. In an ideal setting, we would like to be able to use the same dataset for all ablation studies. However, Something-Else does not have object classes annotated per video as stated in `L228-L229`. We perform all OAP ablations on Something-Else and all AOP ablations on EPIC-OV split. We purposely choose Something-Else for OAP ablations since it already has existing compositional baselines which we can compare to. For AOP, we choose EPIC-KITCHENS since the HOI detector [52] is already trained on a subset of it and is less likely to confound the ablation results.\n\n**Questions:**\n\nWe think there might be a confusion here between open-set recognition versus open-vocabulary recognition. Indeed, the former would require a generative model instead of a discriminative one. However, in an open vocabulary setting, it is assumed that during inference, the class names are provided and there is no prior assumption as such on the vocabulary of the names provided during inference, hence *open vocabulary*. This is a standard practice followed by other open-vocabulary works [18, 68]. We make this distinction in `L30-31`, but will be sure to further highlight the differences in our revision."}, {"review_id": "5YcJrgKYpD", "reviewer": "Reviewer_DGnw", "summary": "The paper proposes a novel open vocabulary action recognition task, and this task aims to generalize the ability of the model to an open vocabulary of actions with seen and novel objects. To do this, the authors propose a new method, which mainly contains two components: Object Agnostic Pretraining (OAP) and Active Object Prompting (AOP). The authors also create open vocabulary benchmarks on the EPIC-KITCHENS-100 and Assembly101 datasets, and the results on these benchmarks demonstrate the effectiveness of this method.", "questions": "It seems that only the objects in this task can be open-vocabulary. Can the verbs in the task be also open-vocabulary?\n", "limitations": "The authors have discussed limitations in chapter 5.6, such as, bias in CLIP.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The paper proposes a novel open vocabulary action recognition task, which is important in practice.\n2) The method presented in the paper is simple and straightforward. \n3) The paper presents sufficient experiments. The results of comprehensive experiments enhance the validity and reliability of the proposed approach.", "weaknesses": "1) Motivation is unclear, and many modules are designed without proper motivation, e.g., why conduct object agnostic pretraining, why interested in recognizing active objects.\n2) The authors claim that we are the first to explore end-to-end contrastive learning on the video backbone of egocentric videos, but contrastive learning is already widely employed in the fields of video and image. \n3) Since this paper focuses on open vocabulary action recognition, why can't the verb category be open vocabulary as well?\n4) The proposed methods lack novelty. Firstly, mixup operations are widely used in image augmentation [1]. Secondly, the idea of Parameter-efficient training for Active Object Prompting appears to resemble a modified version of COCOOP [2]. Thirdly, Active Object Prompting (AOP) is not very novel, just a simple combination of the HOI model and CLIP. CLIP has been widely used in open-vocabulary tasks.\n5) The description of the paper's method is insufficiently detailed, making it difficult to reproduce. For example, in Line 205, the author mentions ΦL as a projection network without further interpretation or elaboration.\n\n[1] Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation, CVPR2022\n[2] Conditional Prompt Learning for Vision-Language Models, CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments and acknowledging the importance of our proposed open vocabulary action recognition task, the simplicity of the proposed approach and sufficient experimentation.\n\n**W1**\n\n- *\"why interested in recognizing active objects\"* - Our task is action recognition and active objects correspond to the noun in the action labels of our datasets, therefore we want to recognize these objects (please refer to `L54-L63` and `Sec. 4 - Object Encoder`).\n- *\"why conduct object agnostic pretraining\"* - Our focus is open vocabulary action recognition where actions are composed of a seen verb with a seen or novel object. Video models trained with a set of seen verb-object compositions get biased towards the seen compositions which hampers the generalization ability of the model (please refer to `L42-L53` and `Sec. 4 - Verb Encoder`) i.e. generalizing verbs to novel objects (as can be seen from our experiments in `Tab. 4`). Hence, we conduct object agnostic pretraining (OAP) to remove the dependency of objects on predicting a verb category thus improving its generalization to novel object categories not seen during training.\n\n**W2**\n\nThanks for noticing! We claim on `L51-L52` that *“To our knowledge, we are the first to explore contrastive learning of video backbones end-to-end on egocentric videos.”* End-to-end (backbone not frozen) contrastive pre-training has been widely explored for Kinetics-style videos [6, 19]. Very recently, it has also been adopted in egocentric videos [30] but for video-text representation learning. On the other hand, some methods only learn a parameterized module over a frozen backbone using contrastive learning [a]. Since egocentric videos in EPIC, Assembly101 and non-ego videos of Something-Else have properties different from Kinetics-style videos (`L224-L226`), the spatio-temporal augmentations needed to train a video backbone end-to-end need to be adapted. We study the nature of such augmentations in `Sec. C` of the supplementary and in this regard, to our knowledge, we are the first to do so.\n\n**W3 \\& Question**\n\n*\"...why can't the verb category be open vocabulary as well?\"* - We address the concern regarding why an open vocabulary of verbs was not considered in the global author rebuttal.\n\n**W4**\n\n- *\"Firstly, mixup operations are widely used...\"* - [1] uses mixup in the embedding space as opposed to our object mixing which is applied in the pixel space. Kindly note that this is the first time lightweight mixing augmentations of this form are being used for egocentric videos (featuring fine-grained hand-object interactions in the real world).\n- *\"Thirdly, Active Object Prompting (AOP) is not very novel...\"* - Our goal is to show what design choices are important for prompting a CLIP model to recognize active objects. In this regard, we have found that using an HOI detector and using the union of hand and object crop improves the performance over a base CLIP operating on full image by 3.1 %. Conditional Prompting of a CoCoOp [2] style is another facet of active object prompting but unlike CoCoOp that projects CLIP-Image encoder feature to the text embedding space, we project the verb features learnt via OAP to the text embedding space. This requires no extra computation since the verb encoder is already being used for verb prediction. The most important aspect of AOP which brings about the biggest improvement is *fine-tuning the base object vocabulary*. This is extremely crucial for fine-grained action datasets like EPIC, Assembly101 (`L207-L216`).\n- Additionally,  we are the first to propose open vocabulary action recognition which has many potential applications and might encourage the action recognition community to explore beyond predefined action classes used during training.\n\n**W5**\n\nThanks for pointing this out! We follow the established CoCoOP [66] meta-network design for $\\Phi_L$ i.e. a 2-layer bottleneck MLP (Linear-ReLU-Linear) where the hidden dimension is (input dimension)/16. We will add this information in our supplementary and also release our source code.\n\n[1] A. Goel et al., \"Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation\", CVPR 2022\n\n[a] D. Singhania et al., “Iterative Contrast-Classify For Semi-supervised Temporal Action Segmentation”, AAAI 2022"}, {"review_id": "1EW7qyWiAH", "reviewer": "Reviewer_FCUw", "summary": "The paper tackles open vocabularly action recognition where novel objects may be seen at test time and will be composed with a known verb. Note that the paper does not investigate the open vocabularly of verbs. To address this problem the paper proposes 'object agnostic pretraining' to become less reliant on the objects seen in training as well as 'active object prompting' a method to prompt CLIP for the specific problem of recognizing unseen objects in combination with seen verbs. This approach is tested on compositional action recognition with SomethingElse and open vocabularly action recognition on EPIC-Kitchens-100 and Assembly101.", "questions": "Why weren't prior works [A] and [B] on compositional action recognition compared to in Table 1?\n\nWhy aren't additional open vocabularly object recognition methods compared to in Table 2? E.g. [63] and [18]\n\nDoes L_in have any impact to the final results?\n\nIs object mixing better than class-balancing the object in the (verb) in-class positives?\n\nAfter reading the rebuttal some of my concerns have been alleviated, namely the comparison with compositional action recognition, the additional ablation, the promise of an open-vocabulary verb baseline in the final version and the clarification on the open vocabulary object detector comparisons. As reading the other reviews has not raised any additional concerns I have upgraded my rating to borderline accept. I have not raised it higher as I am not convinced by the arguments for not considering an open vocabulary of verbs. I do think it is an interesting problem for the paper to focus on an open vocabulary of objects in actions however, the title, abstract and introduction should be updated to clearly indicate this is the focus of the paper if it is accepted. I was also not convinced by the rebuttal response on the proposed object mixing vs. class balancing. An experiment would demonstrate the point more clearly.", "limitations": "Limitations are addressed well in a separate section of the main paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The set up of open vocabularly benchmarks is thoughtful. For instance, keeping objects from imagenet and kinetics in the base split to avoid leak from pretraining to open vocabularly in test and using tail clases as the open vocabularly ones to make the setting more realistic ensures it will be a useful baseline for future work.\n\n***\n\n- Reporting results on SomethingElse allows comparison to prior work on the related task of compositional action recognition. This is useful as there is no prior work on open vocabularly action recognition to compare to.\n\n***\n\n- The ablations demonstrate the effectiveness of the majority of proposed components", "weaknesses": "- Only focusing on the open vocabulary of objects, not verbs, limits the usefulness of the baseline to future works. I appreciate the paper is focused on unseen objects however setting up the baseline in a way that allowed future works to study unseen verbs would have been valuable\n\n***\n\n- The approach is tailored to the specific scenario of egocentric vision and recognizing unseen objects in combination with seen verbs. Testing on SomethingElse, which is not egocentric, did partially alleviate this concern however, the method is still quite specific to the target task.\n\n***\n\n- The baselines in Table 2 are mostly standard action recognition backbones rather than works for compositional action recognition.\n- [A] and [B] provide results on SomethingElse and are more recent than [35] so it is unclear why these results weren't compared to.\n\n***\n\n- Limited open vocabulary object recognition baselines in Table 3\n- It is unclear why [63] and [18] aren't compared to as the introduction sets these up as some of the most relevant works.\n- CLIP performs quite well on novel objects already so makes sense that an existing open vocabulary object detection method built on CLIP would perform even better.\n\n***\n\n- Table 6 ablation, some components ineffective\n- From Table 6 the benefit of temporal gradients and L_in seem negligible: +0.2% top-1 accuracy\n- Furthermore, it would be useful to assess the impact of these two components separately. For instance, can the object mixing be added without L_in to assess whether L_in is effective or whether the object mixing is the only effective component in this section\n- I'm also curious as to whether the success of the object mixing is mainly due to the imbalance of objects in training. How would class-balancing the negative object selected compare to the object mixing.\n\n***\n\n[A] Is an Object-Centric Video Representation Beneficial for Transfer? ACCV 2022.\n[B] Revisiting spatio-temporal layouts for compositional action recognition. BMVC 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review, great suggestions and acknowledging the additional experiment on Something-Else. We also thank the author for appreciating the motivation behind our benchmark design. Due to space constraints, kindly find the required results (tables) in the attached rebuttal pdf.\n\n**W1**\n\nWe address the concern regarding why an open vocabulary of verbs was not considered in the global author rebuttal. A baseline with unseen verbs is a very nice recommendation. To facilitate future research on both novel verbs and objects, we can release such a benchmark and prepare a zero-shot (verb) + open vocabulary object baseline in the supplementary.\n\n**W2**\n\n- Indeed, our approach is limited to recognizing actions which are defined as a combination of verbs and nouns. However, this type of definition, while prominent in egocentric datasets, is not limited to it and is plausible for any hand-object interaction datasets e.g. Something-Something [16], Charades [a], Breakfast [b] etc.\n- Additionally, contrastive spatiotemporal augmentations for egocentric videos are presently lacking in the community and working with them provides an opportunity for such exploration. Please refer to Sec. C of the supplementary for a detailed explanation.\n\n**W3 \\& Q1**\n\n- *\"Why weren't prior works [A] and [B]...\"* - Thanks for bringing this to our notice! Our OAP is a general pretraining strategy for any video backbone without using any extra parameterized module. Our goal in `Tab. 2` was to show how OAP scales with larger backbones (S3D → Motionformer).\n- [A] uses Motionformer as backbone and [B] uses R3D as backbone on top of which they add a parameterized module specialized for compositional recognition (like STIN [35]). The design of such specialized modules is orthogonal to our approach. In theory, specialized modules on top of our OAP pretrained backbone, should improve the overall performance. Unfortunately [A] do not release their code but in `Tab. 1 of rebuttal pdf` we show this by using CACNF [B] on top of supervised S3D features and then on top of OAP pretrained features. We observe that OAP improves performance since it provides better video features to CACNF. However, the margin of improvement decreases due to CACNF’s inherent capabilities of better compositional recognition. Not that we use S3D with 16 frame input instead of the heavier R3D with 32 frame input as reported in [B].\n\n**W4 and Q2**\n\nAs mentioned and visualized `L135` onwards, these SOTA open vocabulary object detectors fail to detect active objects which correspond to the nouns of the action labels. We provide comparison with DETIC which is a better baseline than [63, 18] in open vocabulary object detection. Our method outperforms DETIC by *18.8%*. We could also prepare baselines for [63] and [18] and include it in our revision. Kindly note that naively using CLIP results in poor performance and our proposed AOP-Novel (w/o temporal modeling) finds a good balance between improving both base and novel class performances. Please refer to `Tab. 2 of the rebuttal pdf` for the corresponding comparison.\n\n**W5 and Q3**\n\n- *\"...the benefit of temporal gradients...\"* - With this ablation we show that how different augmentation affects the Top-1 performance and indeed using temporal gradients does not result in much improvement in accuracy. However, our proposed object mixing strategy brings an improvement of *3.4%* Top-1 Acc.\n\n- *\"Does L_in have any impact to the final results?\"* - Thanks for pointing this out and this is indeed an important experiment to assess the performance of L_in. In `Tab. 3 of rebuttal pdf`, it can be observed that although object mixing augmentations are extremely effective, using L_in provides a performance boost of *1.6%* Top-1 Accuracy compared to when all the augmentations are used along only with L_out. Here $|G_i| = 5$ and $\\tau_{in} = 0.1$.\n\n**W5 and Q4**\n\n- *\"Is object mixing better than class-balancing...\"* - This is indeed an interesting observation. However, class-balancing the object classes for verb (in-class) positives will require object labels per video which is not provided for Something-Else.\nSecondly, if we assume the object labels are available e.g. EPIC100-OV, Assembly101-OV, our goal of introducing object mixing augmentations and broadly OAP is to remove the dependency of the model on the object for verb recognition. Dependency of object for verb recognition can be introduced in a video model broadly by two ways:\n    - **class imbalance of the dataset:** Some verb-object compositions are seen more frequently than others during training.. Hence, a supervised training with no class-frequency based weighting/sampling will bias the model towards the head compositions. Here, class-balancing might improve performance on the few-shot classes.\n    - **static bias of video models:** A class-balanced model will still be biased towards certain object appearances for verb recognition e.g. assuming *“slicing apple”* is the only action with apple seen during training, during inference if the action *“washing apple”* is shown to the model, the model has a high chance of incorrectly predicting the verb as  *“slicing”* by overfitting to the red and round appearance of the apple. This is widely explored in literature as static bias of video models [7, 28]. Since our motivation is not improving few-shot performance but providing a reasonable solution to the open vocabulary setting, we require the verb model to predict the verb in a novel action composition. Here OAP helps by reducing the dependency of not only the appearance of the object being manipulated but also any distractor objects present in the background (scene).\n\n\n[a] G. Sigurdsson et al., “Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding”, ECCV 2016\n\n[b] H. Kuehne et al., “The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities”, CVPR 2014"}, {"review_id": "9hch2uxL4t", "reviewer": "Reviewer_TsuW", "summary": "In this paper, a new method is proposed for open vocabulary object understanding within egocentric videos for the task of action recognition. The method uses CLIP features to generalise to new objects which haven't been seen during training by focusing on objects using a HoI model, combining nouns via object mixing and creating learned prompts. The method is tested on three datasets, EPIC-Kitchens-100, Assembly 101, and Something-Else. The proposed method outperforms other methods across all datasets on both seen and unseen nouns.", "questions": "Here are the list of questions that I would like to see answered, for more context of each question please see the weaknesses section.\n\n1. More information regarding k1 and k2, what are the values and how were they chosen?\n2. What do the learnt prompts look like? Do they have a relation to any words that are within the context of the datasets? How much do the learned prompts change per dataset?\n3. Could line 210 be clarified as to how the object embeddings are tuned?\n4. How were the values of alpha and lambda chosen and how sensitive is the model to these?", "limitations": "Yes, the authors have addressed some limitations in the method such as the assumption that every verb/object pair makes a viable action and that CLIP has a strong object bias.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Expanding action recognition beyond a closed vocabulary of objects is good to see and the proposed method makes sense to use the CLIP aligned image text space. The results perform well on all three datasets and most aspects of the method are ablated.", "weaknesses": "Figure 2 could be improved such that it shows the examples of the anchor, positive, and mixed objects (as well as the negatives). Even with labels, currently, the frames in the figure are quite small and it's difficult to understand what is going on.\n\nSection 4.2: What are the values of k1 and k2 and how were these chosen? How sensitive is the model to these values? In a similar vein, has an investigation between the learnt prompts and what these represent from the learnt embedding space been found?\n\nLine 210: It isn't clear to me what it means by fine-tuning the object embeddings but the CLIP text encoder is kept frozen? \n\nSection 5 the value of alpha and lambda haven't been explored within the paper, I assume these values set to 0.5 have been empirically chosen? If so, how sensitive is the method to these different hyperparameters?\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and acknowledging our task of expanding egocentric action recognition beyond a closed vocabulary of objects.\n\n**W1**\n\n*\"Figure 2 could be improved...\"* - Thank you for the feedback, we will improve this figure in our revision and add a more detailed one in the supplementary.\n\n**W2 \\& Q1**\n\n- *\"What are the values of k1 and k2 and how were these chosen?\"* - k1 and k2 refers to the number of learnable prefix and postfix context prompts. We ran experiments with a wide range of k1 and k2. The results provided in all tables of the main paper are for k1=k2=16.\n- k1=k2=0 refers to CLIP+HOI baseline. Initially, increasing k1,k2 (4→16) improves both base class and overall (HM) performance, until it is too large (32), at which point the model overfits to the base set and cannot generalize well to novel classes. A good balance is achieved (highest HM) for k1=k2=16. See results in the following table.\n\n| **k1,k2** | **Base** | **Novel** | **HM** |\n|:------:|:--------:|:---------:|:------:|\n|  **0** |    7.9   |    **16.3**   |  10.6  |\n|  **4** |  13.2   |    10.5   |  11.7  |\n|  **8** |   21.4   |    7.6    |  11.2  |\n| **16** |  22.0   |    9.6    |  **13.4**  |\n| **32** | **24.2**   |    4.6    |   7.7  |\n\nFor the sake of simplicity we assume k1=k2. To measure model sensitivity to the values of only k1 & k2, we refrain from using other AOP strategies like fine-tuning base vocabulary, verb-conditioned prompting and temporal modeling. ViT-B/16 is used as the CLIP-Image encoder backbone. We stop at k1=k2=32 since the maximum context length of CLIP is 77.\n\n**W2 \\& Q2**\n\n*\"In a similar vein, has an investigation between the learnt prompts\"* - While retrieving the learnt prompts, we observed words like *video* and *table*. However, since these prompts are learnt from scratch, most of them are not human-readable when retrieved as discrete tokens. As a result, it is difficult to verify how they behave with the changing of datasets. This is consistent with findings in previous literature [25, a].  Alternatively, we did analyze the interpretability of the fine-tuned base object embedding; this is given in Supplementary `Sec. E.4` which shows base object adjustments like synonym and plurality corrections.\n\n**W3 \\& Q3**\n\n*\"Could line 210 be clarified as to how the object embeddings are tuned?\"* - Similar to learning context prompts by backpropagating the task-specific loss while keeping the CLIP-encoder frozen, we can also do the same for the base object embeddings. The main difference is that unlike the context prompts which are randomly initialized from a normal distribution with 0 mean 0.01 and standard deviation (i.e. learnt from scratch), the object embeddings are initialized as pretrained CLIP text embeddings. Thus backpropagating the loss finetunes the object embeddings and allows the CLIP to adjust the object class names in the continuous space that helps differentiate discrete fine-grained concepts like *‘boom*’ vs *‘arm’*. Also refer to `Sec. E.4` of the supplementary for interpretability analysis of the fine-tuned base object vocabulary.\n\n**W4 \\& Q4**\n\n*\"How were the values of alpha and lambda chosen and how sensitive is the model to these?\"* - $\\alpha$ is chosen to be 0.5 and $\\lambda$ is chosen to be 1 for all the experiments across all the datasets. We did a hyperparameter sweep for $\\lambda = \\{0.1, 1, 10\\}$ with best performance for $\\lambda=1$ where the range of $L_{out}$ and $L_{in}$ were equally scaled. We ran some preliminary experiments with $\\alpha=0.2$, however the results were poor hence we aborted the run. Due to compute constraints for end-to-end pretraining of video backbones on large datasets like Something-Else, EPIC and Assembly, we did not perform a full grid-search on $\\alpha$. We found $\\alpha=0.5$ to be a good estimate since it mixes both the videos with the same weight. We have focused our compute on ablating the main methodological contributions (Sec. E of Supplementary) so that our results are interpretable and reproducible. Please refer to `Sec. B` of the supplementary for additional implementation details.\n\n[a] B. Lester et al., “The Power of Scale for Parameter-Efficient Prompt Tuning”, EMNLP 2021"}]

Author rebuttal:
We thank all the reviewers for their valuable time and constructive feedback. We also thank them for appreciating our novel problem formulation of open vocabulary action recognition. 

We appreciate the concern from R3 (DGnw) and R4 (FCUw) for not considering an open vocabulary of verbs; here we explain our rationale for keeping a closed verb set:

- **A closed set is logical, semantically:** Verbs are characterized by the hand movements when manipulating an object. Yet fine-grained hand motions are inherently bounded to a limited set of categories. This is confirmed by haptics research, which define only 17 [a] to 34 [b] different hand grasps.  Furthermore, only 5-10 of these grasps are in use on a regular basis [b].

- **Limitations of Datasets:** Existing datasets do not lend themselves well to investigate open verb vocabularies.  Verbs in datasets of interest (EPIC-KITCHENS-100 [8], Assembly101 [51], Ego4D [17]), are limited in number (24 in Assembly, ~100 for EPIC100 and Ego4D).  They are also inherently redundant in the underlying movements, e.g. consider the same hand motion from *screwing a wheel*, *opening a bottle cap*, and *juicing a lemon*.  As such, any partitioning is still likely to have information leakage from the closed to the open set.

In conclusion, we appreciate the idea of open vocabulary verbs.  We leave it for future work, where the topic can be addressed with due consideration, in a more meaningful setting and with better data.  Furthermore, we wish to emphasize that using a closed verb set does not diminish our current contribution - we are the first to open the vocabulary of objects for hand-object interaction, and see our paper as a starting point for investigating open vocabulary action recognition.

[a] T. Feix et al., “The GRASP Taxonomy of Human Grasp Types”, IEEE Transactions on Human-machine Systems 2014

[b] IM Bullock et al., "Grasp Frequency and Usage in Daily Household and Machine Shop Tasks", IEEE Transactions on Haptics 2013

Meta-review: The paper studied open-vocabulary recognition of human-object interactions in egocentric videos. The initial review ratings were mixed. The reviewers raised concerns about the problem formulation (e.g., open vocabulary recognition for nouns but not for verbs) and the empirical evaluation (e.g., lack of comparison to certain baselines). They also questioned several technical details. The authors provided a detailed response. All reviewers were satisfied with the rebuttal, and leaned toward the acceptance of the paper. Reviewer TsuW expressed the support in the discussion, yet did not update the rating. 

After going through the paper, the reviewers, and the responses, the AC considers that the paper addressed an interesting problem and presented a reasonable solution, and thus recommend its acceptance, despite an overall borderline rating.
Decision: Accept (poster)