SlowGuess/NAACL_2022_OCR huggingface dataset:
Dataset and split: We conduct experiments on two datasets, Epic-kitchen v-2 (Damen et al., 2020, 2018) and Charades (Sigurdsson et al., 2016). On Epic-kitchen benchmark, we create the compositional split for compositional action recognition. To avoid inductive bias brought by pretrained backbones (e.g. Faster R-CNN (Girshick, 2015) pretrained on ImageNet (Deng et al., 2009), or Inception3D (I3D) (Carreira and Zisserman, 2017) pretrained on Kinetics (Kay et al., 2017)) as discussed in (Wang et al., 2020), we ensure all nouns/verbs seen during pre-training stay in  $V_{s}N_{s}$  when creating compositional split on Epic-kitchen benchmark. For Charades, we follow the same splits in (Kato et al., 2018) for fair comparison.
Charades dataset (Sigurdsson et al., 2016) contains 9848 videos, and many involve compositional human-object interaction. We use the compositional benchmark proposed by (Kato et al., 2018): they remove "no interaction" action categories, leaving 9625 videos with 34 verbs and 37 nouns. Those verbs and nouns are further partitioned into two verb splits  $V_{s}, V_{u}$  (number of classes being
Table 1: DARK's results on Epic-Kitchen dataset, compared with baselines and GCNCL (Kato et al., 2018)
-	AUC Macro Open	AUC Open	mAP Open
Top1	Top2	Top3	Top1	Top2	Top3	All	Zero-shot class
Chance	6.4× 10-7	2.9× 10-6	7.8× 10-6	1.9× 10-5	6.7× 10-5	1.2× 10-4	0.00065	0.00067
Triplet	0.021	0.060	0.095	0.021	0.060	0.094	0.051	0.053
SES	0.091	0.25	0.42	0.16	0.45	0.73	1.83	0.99
DEM	0.0068	0.019	0.040	0.022	0.074	0.14	0.52	0.30
GCNCL+GT	0.044	0.11	0.19	0.044	0.11	0.19	0.46	0.31
GCNCL+Affd	0.064	0.16	0.27	0.082	0.22	0.37	0.48	0.15
GCNCL+Both	0.061	0.16	0.26	0.07	0.17	0.27	0.47	0.27
DARK (ours)	1.69	3.64	5.45	2.04	4.67	7.05	2.39	1.22
Table 2: Dataset statistics of proposed benchmark based on Epic-kitchen and previous benchmark on Charades.
VsNs	VsNu	VUns	VUns	samples
Epic-	840	896	1073	820	76605
Charades	49	47	22	31	9625
20/14),and two noun splits  $N_{s},N_{u}$  (18/19).The total number of compositional actions is 149.
Epic-kitchen version 2 dataset (Damen et al., 2020, 2018) contains videos recorded in kitchens, where people demonstrate their interaction with objects like pan, etc. The diversity of actions in this dataset makes it especially challenging. We follow the steps in similar previous works (Rahman et al., 2018; Wang et al., 2020) to create our compositional split. We first make sure that classes seen in pre-training stay in the seen split. Then for the remaining classes we sort them based on the number of instances in descending order, and pick the last  $20\%$  to be unseen classes, because (Rahman et al., 2018) pointed out that zero-shot learning targets the classes not easy to collect (especially those in the tail part of the long tail class distribution). We show dataset statistics in Table 2. We get a total number of 76605 videos, including 90 verbs, 249 nouns, and 3629 compositional actions. Compared to Charades, our proposed benchmark is at a larger scale in terms of classes involved and sample size.
Baselines: We establish our baselines following previous work (Kato et al., 2018). Here we briefly summarize their architectures, and readers can refer to (Kato et al., 2018) or original papers for details. These baselines are based on Inception3D features.
Triplet Siamese Network (Triplet) by (Kato et al., 2018): verb/noun embeddings are concatenated, and transformed by fully connected (FC) layers. The output is concatenated with visual features to predict scores through one FC layer with the training of BCE loss.
Semantic Embedding Space (SES) (Xu et al., 2015): The model projects visual features into embedding space through FC layers and then matches output with corresponding action embeddings (average of verb/noun embeddings) using L2 loss.
Deep Embedding Model (DEM) (Zhang et al., 2017): Verb/noun embeddings are transformed separately via FC layers and summed together. Then output is matched with visual features via L2 loss.
4.3 Results on Epic-kitchen dataset
The results of the proposed DARK model, as well as the aforementioned baselines (Triplet, SES and DEM) and previous model GCNCL (Kato et al., 2018) on the Epic-kitchen dataset are listed in Table 1. We report the results in the proposed AUC metric (Chao et al., 2016) with precision calculated at top  $1/2/3$  prediction for both open world and macro open world settings, which evaluates the overall trade-off between seen/unseen class. We also report the mean average precision (mAP) used in (Kato et al., 2018) on all and zero-shot compositional action classes for reference.
Our best performing DARK model outperforms all baselines and GCNCL by a large margin under all metrics, illustrating the benefit of disentangled action representation for compositional action recognition. DARK is also more scalable, and reduces the number of graph nodes from 22749 (GCNCL with no external knowledge) to 339 (ours).
DARK considers the type constraint of verbs and nouns when composing verb and noun into compositional action label by training an affordance scoring module, while GCNCL considers the constraint when building compositional action nodes by collecting the existing verb-noun pairs from NEIL (Chen et al., 2013). For fair comparison, we re-implement three versions of KG in GCNCL model. In "GT", we use the ground-truth verb-noun relationships that are presented in the dataset
Table 3: Combination for zero-shot learning in verb and noun classifiers. The top1 "open world" AUC is reported on the Epic-kitchen dataset.
Verb-KG	Verb-SES	Verb-ConSE
Noun-KG	1.81	1.24	0.49
Noun-SES	0.40	0.44	0.32
Noun-ConSE	0.25	0.18	0.13
(open world setting). In "Affd", we only consider relationships in the same corpus with DARK. We use the relationships as a hard look-up since GC-NCL only contains unweighted "hard" edges in its knowledge graph. In "Both", we use the union of the constraints in "GT" and "Affd". Under all the three circumstances, our DARK model outperforms other models by a large margin. For all experiments, we report the best results.
4.4 Ablations on different components
Zero-shot learning in verb/noun classifier: In DARK model, we do separate verb and noun classification in two branches. We investigate different implementations of zero-shot learning in verb/noun classifier. Specifically, we consider three options for both verb and noun, namely "KG", "SES" and "ConSE". "KG" stands for zero-shot learning by using knowledge graph to predict classification weights for unseen component with GloVe embedding (Pennington et al., 2014) as in (Kato et al., 2018). "SES" (Xu et al., 2015) is the best common embedding baseline in Table 1 using better BERT word embedding (Devlin et al., 2018) (based on observation in Table 5, BERT tends to have better performance). "ConSE" (Norouzi et al., 2013) learns a semantic structure aware embedding space compared to original word embeddings, which is modeled with graph. "ConSE" (Norouzi et al., 2013) is used as the zero-shot learning component in previous image-based action recognition task (Xu et al., 2019). It learns a semantic structure aware embedding space and we also use GloVe embedding. For better comparison of zero-shot learning component, we report the "open world" AUC on the Epic-kitchen dataset without using affordance (same as "ground-truth" affordance in macro open setting), thus excluding the influence of affordance prior. Different zero-shot learning combinations for verbs and nouns are reported in Table 3. Using "KG" for both verb / noun outperforms others by a large margin, and we take this approach in the rest experiments.
Table 4: Different verb knowledge graphs. We report the top1 AUC for verbs under GZL setting.
WN dis	VN group	VN tree
one-way	1.86	0.83	1.93
two-way	×	×	1.79
Construction of verb knowledge graph: Compared to nouns, the concept of verb is relatively abstract and the relationship between verbs is hard to capture. We explore different ways of constructing the verb KG, namely, "WN dis", "VN group" and "VN tree". (The details of noun KG are discussed in the appendix.) In "WN dis", we use WordNet (Miller, 1995) structure and add edges between nodes if their LCH (Leacock and Chodorow, 1998) distance is bigger than a threshold. We also explore VerbNet (Kipper et al., 2008) which is designed to capture the semantic similarity of verbs. VerbNet categorizes verbs into different classes, and each class contains multiple verb members. To resolve the duplication in each class, we add edges between verbs in the same class, and denote this as "VN group". We also try adding a meta node for each class and connecting all its members to the meta node, denoted as "VN tree". Graphs of "WN dis" and "VN group" are naturally undirected. For "VN tree", we consider an additional "two-way" setting as in (Kampffmeyer et al., 2019), where a GCN model separates the parent-to-children and children-to-parent knowledge propagation into two stages to better model hierarchical graph structure. However, we do not observe performance improvement in this setting. In Table 4, we report top1 AUC for verbs using different KGs under the GZL(Chao et al., 2016) setting. "VN tree" in "one-way" gives the best prediction for verb, and we keep this configuration in rest experiments.
Affordance learning: We consider the compositional constraints between verbs and nouns (affordance) when composing the compositional action. We explore various ways of learning affordance in Table 5. In "Word-only" we train a word-embedding only model. And "Visual" represents the approach in method section where an additional projection module maps visual features to embedding space. For each, we explore two word embeddings, GLoVe (Pennington et al., 2014) and BERT (Devlin et al., 2018). In terms of score calculation, we try three different methods each. In "Concat-Scoring", we concatenate verb/noun features, and
Table 5: Top1 "marco open world" AUC on Epic-kitchen based on different affordance learning methods.
Word-only	Visual
GLoVe	BERT	GLoVe	BERT
Concat-Scoring	1.49	1.59	1.48	1.60
Context-Scoring	1.49	1.59	1.49	×
Proj-Cosine	1.42	1.59	1.42	1.69
Lookup Table	1.35
Uniform	1.46
Ground Truth	1.81
Table 6: Results (mAP) on Charades under GZL setting. Baselines and GCNCL results from (Kato et al., 2018).
Model	All	Zero-shot
Baseline	Chance	1.37	1.00
Triplet	10.41	7.82
SES	10.14	7.81
DEM	9.57	7.74
GCNCL	GCNCL-I+A	10.48	7.95
GCNCL+A	10.53	8.09
Ours	DARK	11.21	8.38
train a scoring model. In "Context-Scoring", instead of concatenating, for BERT the scoring model embeds verb-noun phrase together and averages their embeddings, and for GloVe we simply average their embeddings. In "Proj-Cosine", we project verb embedding to noun embedding space and calculate the cosine distance. We also try a lookup table, where affordance is one if the compositional pair exists in train set or knowledge bases, and zero otherwise. "Uniform" sets all affordance to be one, which means no weighting is applied. "Ground Truth" sets one for pairs existing in the dataset (train/test), equivalent to "open world". In all experiments, we use best configuration from Table 3, and label compositional pairs seen in training to one. BERT constantly improve affordance relationships in different methods. Lookup table performs worse than "Uniform" (no affordance) since some valid pairings are missing in knowledge bases.
4.5 Results on Charades dataset
We report results on Charades in Table 6. Following (Kato et al., 2018), we report mean average precision (mAP) and compare our model to theirs and baselines. We also report zero-shot classes  $(V_{s}N_{u} + V_{u}N_{s} + V_{u}N_{u})$  separately but all predictions are made under GZL setting. Unlike Epic
Figure 5: Qualitative Analysis. Underlines are unseen concepts, green for right predictions and red for wrong.
Kitchen which contains ego-centric actions, Charades contains third-person view videos and cannot detect the mask of person's hand. Thus we directly learn the verb and noun feature disentanglement leveraging a discriminator and a disentanglement loss. Following (Peng et al., 2018), discriminator tries to adversarially classify noun label  $y_{n}$  from its verb feature, and feature extractor  $\mathcal{F}$  goes against it. To better capture the multi-label property in Charades, we use an un-factorized classification model for actions in  $V_{s}N_{s}$  so they can be treated separately. Since we report the mAP results for fair comparison with GCNCL, we do not use affordance in our model. As indicated in (Kato et al., 2018), we also notice that the amount of improvement over baselines is not large, possibly because Charades is relatively small and easy to overfit. And this motivated us to propose a large-scale zero-shot compositional action recognition benchmark.
4.6 Qualitative error analysis
We also visualize some examples in Figure 5. The model misclassifies coriander as leaf, and foil as plastic wrap due to visual similarity.
In this paper, we propose DARK, a novel compositional action recognition model that reduces complexity from quadratic to linear, making the training more scalable. DARK generalizes to unseen verb-noun pairs, and can be combined with knowledge bases to produce state-of-the-art compositional action recognition results.
Prof. Darrell's group was supported in part by DoD, BAIR and BDD.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. 2015. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):1425-1438.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261.
Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. 2016. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In European conference on computer vision, pages 52-68. Springer.
Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740-750.
Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. 2013. Neil: Extracting visual knowledge from web data. In Proceedings of the IEEE international conference on computer vision, pages 1409-1416.
Jinwoo Choi, Chen Gao, Joseph CE Messou, and Jia-Bin Huang. 2019. Why can't i dance in the mall? learning to mitigate scene bias in action recognition. Advances in Neural Information Processing Systems, 32.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2018. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. 2020. Rescaling egocentric vision. arXiv preprint arXiv:2006.13256.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Terrance DeVries and Graham W Taylor. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.
Junyu Gao, Tianzhu Zhang, and Changsheng Xu. 2019. I know the relationships: Zero-shot action recognition via two-stream graph convolutional networks and knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8303-8311.
Yuxia Geng, Jiaoyan Chen, Zhuo Chen, Zhiquan Ye, Zonggang Yuan, Yantao Jia, and Huajun Chen. 2020a. Generative adversarial zero-shot learning via knowledge graphs. arXiv preprint arXiv:2004.03109.
Yuxia Geng, Jiaoyan Chen, Zhiquan Ye, Zonggang Yuan, Wei Zhang, and Huajun Chen. 2020b. Explainable zero-shot learning via attentive graph convolutional network and knowledge graphs. Semantic Web, (Preprint):1-28.
James J Gibson. 1978. The ecological approach to the visual perception of pictures. *Leonardo*, 11(3):227-235.
Ross Girshick. 2015. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. 2019. Rethinking knowledge graph propagation for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11487-11496.
Keizo Kato, Yin Li, and Abhinav Gupta. 2018. Compositional learning for human object interaction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 234-251.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of english verbs. Language Resources and Evaluation, 42(1):21-40.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2013. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intelligence, 36(3):453-465.
Claudia Leacock and Martin Chodorow. 1998. Combining local context and wordnet similarity for word sense identification. WordNet: An electronic lexical database, 49(2):265-283.
Chung-Wei Lee, Wei Fang, Chih-Kuan Yeh, and Yu-Chiang Frank Wang. 2018. Multi-label zero-shot learning with structured knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1576-1585.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In European conference on computer vision, pages 852-869. Springer.
Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. 2020. Something-else: Compositional action recognition with spatial-temporal interaction networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1049-1059.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Ishan Misra, Abhinav Gupta, and Martial Hebert. 2017. From red wine to red tomato: Composition with context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1792-1801.
Tushar Nagarajan and Kristen Grauman. 2018. Attributes as operators: factorizing unseen attribute-object compositions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 169-185.
Nihal V Nayak and Stephen H Bach. 2020. Zero-shot learning with common sense knowledge graphs. arXiv preprint arXiv:2006.10713.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. 2013. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. 2018. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv:1810.00821.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc'Aurelio Ranzato. 2019. Taskdriven modular networks for zero-shot compositional learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3593-3602.
Shafin Rahman, Salman Khan, and Nick Barnes. 2018. Polarity loss for zero-shot object detection. arXiv preprint arXiv:1811.08982.
Saurabh Sahu, Palash Goyal, Shalini Ghosh, and Chul Lee. 2020. Cross-modal non-linear guided attention and temporal coherence in multi-modal deep video models. In Proceedings of the 28th ACM International Conference on Multimedia, pages 313-321.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. 2020. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869-9878.
Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer.
Krishna Kumar Singh and Yong Jae Lee. 2017. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In 2017 IEEE international conference on computer vision (ICCV), pages 3544-3553. IEEE.
Suchen Wang, Kim-Hui Yap, Junsong Yuan, and Yap-Peng Tan. 2020. Discovering human interactions with novel objects via zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11652-11661.
Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-37.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. 2018. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6857-6866.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. 2019. Detector2. https://github.com/facebookresearch/detectron2.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. 2018. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-2265.