SlowGuess/NAACL_2022_OCR huggingface dataset:
Yongqin Xian, Bernt Schiele, and Zeynep Akata. 2017. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4582-4591.
Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S Kankanhalli. 2019. Learning to detect human-object interactions with knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Xun Xu, Timothy Hesperides, and Shaogang Gong. 2015. Semantic embedding space for zero-shot action recognition. In 2015 IEEE International Conference on Image Processing (ICIP), pages 63-67. IEEE.
Muli Yang, Cheng Deng, Junchi Yan, Xianglong Liu, and Dacheng Tao. 2020. Learning unseen concepts via hierarchical decomposition and composition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10248-10256.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032.
Li Zhang, Tao Xiang, and Shaogang Gong. 2017. Learning a deep embedding model for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2021-2030.
Bohan Zhuang, Lingqiao Liu, Chunhua Shen, and Ian Reid. 2017. Towards context-aware interaction recognition for visual relationship detection. In Proceedings of the IEEE international conference on computer vision, pages 589-598.
Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545.
A Implementation Detail
To build the object feature extractor  $\mathcal{F}_n$ , we first use an off-the-shelf class-agnostic object detector to detect the bounding box of interacting objects. For the Charades (Sigurdsson et al., 2016) dataset, we use the detected object boxes generated by HOID (Wang et al., 2020). HOID model first detects the box of human, then based on the human's bounding box it detects interacting objects. We use the code publicly released by (Wang et al., 2020), with the default parameters provided by the author. We use the weights of the class-agnostic object detector provided on the project page. For the Epic-kitchen dataset (Damen et al., 2020, 2018), we use the pre-computed class-agnostic boxes provided by the authors. They use the model from (Shan et al., 2020), which detects human hands and locates the interacting objects. We use different models for object detection because Epic-kitchen (Damen et al., 2018) contains mostly ego-centric action videos while Charades (Sigurdsson et al., 2016) contains third-person view videos. For the Epic-kitchen dataset, we additionally use its precomputed detected masks of human hands.
After the object boxes are detected, we run the Faster-RCNN (Girshick, 2015) model with the ResNet-101 backbone to extract the object features. We use the implementation provided by the Detector2 (Wu et al., 2019) library, with the weights pre-trained on ImageNet (Deng et al., 2009). We obtain the features before the classifier layer in the Faster-RCNN model, which results in 2048 dimension object features. If we detect multiple boxes in one frame, we conduct max-pooling over their extracted features to obtain one feature representation for each frame. If no box is detected for a particular frame, we then extract the feature of the whole image as its object feature.
We sample several frames along the temporal axis of the video to conduct object detection. Since videos in the Charades (Sigurdsson et al., 2016) dataset may contain more than one action, we treat each frame in Charades dataset as one sample for training. For the Epic-kitchen (Damen et al., 2020, 2018) dataset whose videos contain only one action, we instead simply apply mean pooling over object features of sampled frames to obtain one feature representation for the whole video. We add fully-connected (FC) layers upon the fixed Faster-RCNN backbone to conduct feature extraction.
For verb features, we build our feature extrac
tor  $\mathcal{F}_v$  based on a standard two-stream Inception3D (Carreira and Zisserman, 2017) backbone pre-trained on the Kinetics dataset (Kay et al., 2017). We use both the RGB branch and the opticalflow branch, each producing a 1024 dimension feature in the layer Mixed_5c. We then concatenate them, resulting in a feature representation of 2048 dimension. For the Epic-kitchen dataset, we generate features first using the video input with the object cropped out. Then we do the same using the video input with everything cropped except for the detected hands in order to obtain hand gesture movement information. We further concatenate these two features to get a 4096 dimension feature. Similar to the object feature extractor, we add FC layers to features generated by the fixed Inception3D backbone. For the Charades dataset, since another disentanglement approach is used, we simply use the 2048 dimension feature.
Following common practice, we split the whole video into video clips with a small duration, and generate features for each clip during training and inference. For the Charades dataset, we sample 10 clips per video to conduct training and we treat each clip as a sample. Whereas for the Epic-kitchen dataset, we apply max pooling to the features of all the clips generated from one video to obtain one feature representation for the whole video.
Our model is implemented in PyTorch with Adam optimizer. We used in total around 20 GPUs through out the experiments. But a single run only needs 5 GPUs. (we launch parallel experiments)
B The Proposed Epic-kitchen Benchmark
We build our compositional action recognition benchmark based on the Epic-kitchen(Damen et al., 2020, 2018) dataset version two. We take the class that overlaps with pre-trained backbones into consideration when creating seen/unseen class splits. We find that there are 95 noun classes overlapping with ImageNet classes, and 23 verb classes overlapping with Kinetics classes, where the backbones that we use have been pre-trained. We make sure these overlapping classes stay in the seen split.
We then remove the tail verb and noun classes with less than 10 instances. The remaining dataset contains a total number of 76605 videos, including 90 verbs, 249 nouns, and 3629 compositional actions. We have 29 verbs in the seen category, and 61 verbs in the unseen category. On the other hand, 102 nouns are seen and 147 nouns are unseen.
The  $V_{s}N_{s}$  split contains 840 compositional actions, and 51228 samples. The  $V_{u}N_{s}$  split has 1073 compositional actions, and 10105 samples. For the  $V_{s}N_{u}$  split, there are 896 compositional actions and 11073 samples. And for the  $V_{u}N_{u}$  split, there are 820 compositional actions and 4199 samples.
Epic-Kitchen dataset is realized under the Creative Commons Attribution-NonCommercial 4.0 International License. The licence for nonCommercial use of Charades dataset can be found at http://vuchallenge.org/charades.html. We follow the intended usage of these two dataset.
C Noun Knowledge Graph Construction
We discuss the construction of the verb knowledge graph in the paper, due to the space limit, we present the details of the noun knowledge graph in this section. We construct the noun knowledge graph following (Wang et al., 2018)'s approach. We begin from the nouns presented in the dataset, and recursively search their hyper-norms using WordNet (Miller, 1995)'s lexical relationship to add to the graph. In addition, we augment the knowledge graph by adding nouns from ImageNet's ((Deng et al., 2009)) class labels.
When building noun knowledge graphs, we add an edge if two entities are direct synonyms or hypernorms. Our model is built upon the graph convolution model implemented by (?) . We use its plain GCN version without attention. And we use the "two-way" approach, which separates parent-to-children and children-to-parent knowledge propagation into two stages to better model the hierarchical graph structure. For noun knowledge graph learning, we use 300d GloVe (Pennington et al., 2014) embeddings as input.
D Disentanglement in Charades Dataset
Let  $X_{v}$  denote the input to the verb feature extractor  $\mathcal{F}_v$ , and  $X_{v}^{\prime}$  denote the extracted verb features. Similarly,  $X_{n}$  is the input to  $\mathcal{F}_n$  and  $X_{n}^{\prime}$  is the extracted noun features.
$$
X _ {v} ^ {\prime} = \mathcal {F} _ {v} \left(X _ {v}\right) \tag {8}
$$
$$
X _ {n} ^ {\prime} = \mathcal {F} _ {n} (X _ {n}) \tag {9}
$$
To obtain disentangled verb / noun features, we take the idea from the previous paper (Peng et al., 2018). We use a discriminator to limit the information verb and noun features contain. The discriminator  $\mathcal{D}_v$  tries to adversarially classify noun
label  $y_{n}$  from its verb feature  $X_{v}^{\prime}$ , and the feature extractor  $\mathcal{F}_v$  goes against it via a minimax process. The discriminator helps to limit the information which verb feature  $X_{v}^{\prime}$  contains about the nouns in the video. The same procedure happens for  $D_{n}$ . We use one layer linear classifier for discriminator  $D_{v}$  and  $D_{n}$  and they output class predictions for the opposite branch. This leads to the disentanglement loss:
$$
\mathcal {L} _ {d i s} ^ {v} = - \mathbf {C E} \left(\mathcal {D} _ {v} \left(X _ {v} ^ {\prime}\right); y _ {n}\right) \tag {10}
$$
$$
\mathcal {L} _ {d i s} ^ {n} = - \mathbf {C E} \left(\mathcal {D} _ {n} \left(X _ {n} ^ {\prime}\right); y _ {v}\right) \tag {11}
$$
The CE refers to the cross-entropy loss. The overall loss for training the feature extractor  $\mathcal{F}_v, \mathcal{F}_n$  and the classifier for seen classes  $\mathcal{W}_v^{seen}, \mathcal{W}_n^{seen}$  is:
$$
\mathcal {L} _ {v} = \mathcal {L} _ {c l s} ^ {v} + \mathcal {L} _ {d i s} ^ {v} \tag {12}
$$
$$
\mathcal {L} _ {n} = \mathcal {L} _ {c l s} ^ {n} + \mathcal {L} _ {d i s} ^ {n} \tag {13}
$$
The definitions of  $\mathcal{L}_{cls}^v$ ,  $\mathcal{L}_{cls}^n$  are the same as discussed in the main paper.