SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The CLIP-text encoder is kept frozen for both the prompting strategies, making them parameter-efficient. During testing (Fig. 4 bottom), we use the fine-tuned embeddings for the base objects and the original pretrained embeddings for the novel objects. The verb-conditioned context prompts are unique for every instance.
Datasets. We conduct experiments on three datasets: EPIC100 [8] features 100 hours of egocentric footage of daily kitchen activities annotated with 97 verbs and 300 interacting objects. Assembly101 [51] is a multi-view procedural activity dataset featuring 4321 videos of people assembling and disassembling 101 take-apart toy vehicles and is annotated with 24 verbs and 90 objects. We use the egocentric view  $e3$  (see [51]), which best captures the hands and interacting objects. Something-Else [35], is a human-object interaction subset of SSv2 [16] with 174 verb phrases e.g. "picking something up". Something-Else is not strictly egocentric but shares interesting properties such as diverse hand-object interactions and active objects undergoing state change. We use the compositional split, where all verb phrases are seen during training but applied to novel objects during testing. Interacting objects per video are not annotated; hence, it is unsuitable for our open vocabulary action recognition task but is an ideal testbed for evaluating our verb encoder and OAP with other baselines.
Implementation Details. We use S3D [61] as the verb encoder for the majority of our experiments and ablations as it is a popular architecture for video contrastive learning [20, 37]. For all datasets, the verb encoder is pretrained with the OAP contrastive scheme for 300 epochs and then fine-tuned for verb classification for another 100 epochs. To see how OAP scales with larger backbones, we also evaluate Mformer-B [43] on Something-Else. For the hand-object interaction support, we use the pretrained 100DOH [52] detector without fine-tuning on the target dataset. For the CLIP Image Encoder, we experiment with ViT-B/16 and the larger ViT-L/14 backbones. Keeping both the CLIP image and text encoders frozen, prompts are learned using AOP for 20 epochs. Unless otherwise stated, we choose  $k_{1}$  and  $k_{2}$ , the number of learnable prefix and postfix prompts, to be 16 each. For additional implementation details, please refer to Sec. B of the Supplementary.
Table 1: Proposed open vocabulary benchmark splits.
Dataset	V	Obase	Onovel	#train seg	#test seg.
EPIC100-OV	97	190	110	63k	13.8k
Assembly101-OV	24	60	30	56.2k	15k
5.1 Open Vocabulary Benchmark
Since the original action recognition split of EPIC100 and Assembly101 are unsuitable for open vocabulary (OV) evaluation, we create OV benchmarks EPIC100-OV and Assembly101-OV, respectively, by dividing the complete set of objects into base and novel categories. The base objects are seen during training, whereas the novel objects are kept only for testing. We also incorporate base actions in our test set to evaluate the closed-world performance of the models. The benchmark statistics are provided in Tab. 1. We provide a more detailed explanation of the OV splits in Sec. A of the Supplementary. We use Top-1 Accuracy [8, 51] as our evaluation metric and report results for both closed and novel classes. Additionally, we report the harmonic mean (HM) [38] of the closed and novel accuracies as a consolidated metric for comparing different methods.
Table 2: Compositional Action Recognition on Something-Else [35]. Extra Module refers to if any additional parameterized module has been used on top of the video backbone.
Model	Extra Module	Top-1 (%)	Top-5 (%)
I3D [6]	X	46.8	72.2
Slowfast [11]	X	52.2	80.3
Slowfast+IRN [33]	✓	52.9	80.8
STRG,STIN+OIE+NL [35]	✓	56.2	81.3
S3D [61]	X	49.3	77.3
OAP (S3D)	X	53.7	80.5
Mformer-B [43]	X	60.2	85.8
OAP (Mformer-B)	X	61.8	86.2
Table 3: Comparison of methods for open-vocab object recognition on EPIC100-OV. † Zero-shot evaluation.
Model	Object Top-1 (%)
Base	Novel	HM
S3D†	52.2	1.6	3.1
CLIP (ViT-B/16)†	20.2	0.6	1.2
CLIP (ViT-B/16)	7.9	16.3	10.6
DETIC [68]	12.0	2.3	3.9
C2 (ViT-B/16) [25]	19.1	9.9	10.3
C4 (ViT-B/16) [25]	39.9	7.3	12.3
AOP (CLIP ViT-B/16)	41.9	19.2	26.3
AOP (CLIP ViT-L/14)	47.8	22.6	30.7
5.2 Object-Agnostic Pretraining (OAP) vs State-of-the-Art
How effective is OAP in generalizing verbs to novel objects? Tab. 2 shows OAP applied to video backbones S3D [61] and Mformer-B [43] on the Something-Else compositional split. S3D with OAP outperforms Slowfast [11] by  $1.5\%$  in Top-1 acc. even though Slowfast is a stronger video backbone. It also outperforms, by  $0.8\%$ , the sophisticated Slowfast+IRN [33], which adds an extra transformer to SlowFast to explicitly model the hand-object interactions. Finally, S3D with OAP is still competitive with respect to STRG,STIN+OIE+NL [35], which is an ensemble of various video models, despite being a single backbone without any extra parameters or compute. Mformer-B is a state-of-the-art video architecture comfortably outperforming all other models; adding OAP improves its Top-1 performance by  $1.6\%$ , demonstrating that our pretraining is robust to the scale of the backbone.
5.3 Active Object Prompting (AOP) vs. State-of-the-Art
Tab. 3 compares AOP with similar open vocabulary baselines. The first two rows with  $\dagger$  refer to zero-shot baselines of S3D and CLIP, where training is done using cross-entropy over all classes (base+novel).  $\mathrm{CLIP}^{\dagger}$  refers to the linear probing of CLIP's frozen visual encoder. Both models overfit to the base classes and fail to generalize to novel ones. DETIC [68], a state-of-the-art open vocabulary object detector, improves novel object performance over S3D marginally by  $1.3\%$ . It is non-trivial for DETIC to recognize which object is active from a set of per-frame object detections e.g. for the action "beating egg", a whole egg is not present in any frame. C4 [25] adapts CLIP for video tasks by adding a transformer layer on top of per-frame CLIP features; again, it overfits to the base classes. Finally, our AOP using both CLIP backbones significantly outperforms all other baselines - AOP improves over the closest competitor (C4) having the same backbone in ViT-B/16 by  $14.0\%$  in HM.
5.4 Open Vocabulary Action Recognition
Tab. 4 presents action recognition on open vocabulary splits of EPIC and Assembly101. S3D predicts verbs and objects using two heads but share the same backbone. For a fair comparison with our decoupled verb and object encoders, we also decouple the S3D training, one for predicting verbs and another for objects (reported as  $2 \times \mathrm{S}3\mathrm{D}$ ). As mentioned in Sec. 5.3, the S3D baseline predicts novel objects in a zero-shot fashion where the novel categories are already assumed to be known during training. Both S3D baselines fail to predict novel objects on both benchmarks. OAP improves verb generalization to novel objects by  $1.3\%$  and  $4.8\%$  over S3D on EPIC100-OV and Assembly101-OV, respectively. Additionally, the closed-world verb prediction is also enhanced by  $1.6\%$  and  $7.0\%$  over S3D on EPIC100-OV and Assembly101-OV, respectively. AOP significantly improves novel object prediction performance in both benchmarks. The novel object prediction performance on Assembly101 could be further enhanced by adopting an HOI detector that is robust to domain shifts in the target dataset since Assembly101 egocentric videos are monochrome.
Table 4: Open Vocabulary Action Recognition Results.
Dataset	Model	Closed Top-1 (%)	Novel Top-1 (%)	HM Top-1 (%)
Verb	Object	Action	Verb	Object	Action	Verb	Object	Action
EPIC100-OV	S3D	62.5	50.8	37.6	40.1	1.7	0	48.8	3.3	0
2×S3D	61.5	52.2	36.7	37.9	1.6	0.1	46.9	3.1	0.2
OAP + AOP	64.1	47.8	35.9	41.4	22.6	11.2	50.3	30.7	17.0
Assembly101-OV	S3D	50.6	34.6	19.9	40.3	0	0	44.9	0	0
2×S3D	50.7	37.9	20.2	41.8	0	0	45.8	0	0
OAP + AOP	57.6	30.1	21.6	45.1	6.7	3.7	50.5	11.0	6.6
Table 5: Effect of guiding augmentations for OAP on Something-Else.
Loss	Temporal Gradients	Object Mixing	Top-1 (%)	Top-5 (%)
Supervised	X	X	49.3	77.3
Lout	X	X	50.1	78.1
Lout	✓	✓	52.1	79.3
Lout + λLin	✓	X	50.3	78.4
Lout + λLin	✓	✓	53.7	80.5
5.5 Ablations
Guiding Augmentations. We study the effect of using guiding augmentations on Something-Else in Table 5. Using  $L_{out}$  already improves performance over supervised training, consistent with the findings in supervised contrastive learning [26, 20]. Introducing verb-specific guidance via temporal gradients and especially our proposed object mixing augmentations boosts the performance further by  $3.6\%$ . This demonstrates that OAP with guiding augmentations enhances the generalization of a backbone like S3D without requiring additional parameters.
Active Object Prompting. We thoroughly ablate various learnable prompting strategies in Tab. 6 on EPIC100-OV. ViT-B/16 has been used as the image encoder for all experiments here. For the rows where the context prompts are not learnable, we use the fixed prompt "A picture of a <object>". Introducing the HOI detector to generate active object crops improves HM by  $3.1\%$  over using full images. Our AOP mainly consists of two proposals - fine-tuning base object vocabulary and conditioning context prompts on verb representations from the video. Most of the gain in Tab. 6 comes from these two proposals. Fine-tuning  $\mathcal{O}_{base}$  is expected to improve closed set performance, but we also find  $2\%$  improvement over novel categories. This alludes to the fact that learning object names decreases false positives when tasked to predict a novel category. Verb-conditioned prompting brings further improvement of  $3.9\%$  in HM. In particular, it boosts the performance over base categories, which is consistent with the findings of conditional prompting learning [66, 67].
Next, we analyze using temporal modeling on top of the CLIP features. Similar to C4 [25], we add a single-layer transformer with eight heads over the per-frame CLIP features. As expected, the model overfits to the base categories seen during training, only achieving  $4.8\%$  over novel categories.
Table 6: AOP ablations on EPIC100-OV. ViT-B/16 is used as the image encoder. AOP-Base and AOP-Novel refer to the best-performing models on base and novel object categories.
HOI Detector	Learnable Context Prompts	Verb Conditioned	Finetune Obase	Temporal modelling	Base	Novel	HM
X	X	X	X	X	5.4	12.1	7.5
✓	X	X	X	X	7.9	16.3	10.6
✓	X	X	✓	X	20.2	18.3	19.2
✓	✓	X	✓	X	21.6	16.6	18.8
AOP-Novel	✓	✓	✓	✓	X	29.6	18.4	22.7
✓	✓	X	X	✓	46.9	4.8	8.7
AOP-Base	✓	✓	✓	✓	✓	51.0	11.9	19.3
AOP-Base + AOP-Novel	✓	✓	✓	✓	✓	41.9	19.2	26.3
Finally, we ensemble the predictions of the best-performing model on base (AOP-Base) and novel (AOP-Novel) categories to obtain the best HM.
5.6 Limitations and Outlook
In this paper, we assume that every verb-object composition is feasible. The compositions could be pruned to a feasible subset to reduce the action search space. Obtaining feasibilities, and distinguishing them from common-sense settings, however, is non-trivial, e.g., "cut thermometer", albeit feasible, is unlikely to be observed. Our verb-conditioned prompting of CLIP provides a weak feasibility check. However, CLIP is well known to have a strong object bias [42] over verbs, and it may still yield infeasible active object predictions. One direction to improve the feasibility of actions is to learn object affordances; we leave this for future work.
This paper proposed a new task of open vocabulary action recognition for verb-object composed actions and a novel approach to address it. We decouple the verb and object predictions. The verb encoder is trained with an object-agnostic pretraining strategy to improve its generalization to novel objects. Our object encoder leverages a pretrained CLIP model using active object prompting. This guides the CLIP to predict an open vocabulary of interacting objects. Our experiments demonstrate that the proposed approach can recognize hand-object interactions even when the specific actions or objects are not in the training data.
Acknowledgements: This research/project is supported by the National Research Foundation, Singapore and DSO National Laboratories under its AI Singapore Programme (AISG Award No: AISG2-RP-2020-016). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
[1] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelovic, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. Advances in Neural Information Processing Systems, 33:25-37, 2020.
[2] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435-451, 2018.
[3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836-6846, 2021.
[4] Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13349-13358, 2021.
[5] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and real-time tracking. In 2016 IEEE international conference on image processing (ICIP), pages 3464-3468. IEEE, 2016.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[7] Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. Advances in Neural Information Processing Systems, 32, 2019.