SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhekun Luo
zhekun_luo@berkeley.edu
Shalini Ghosh*
ghoshsha@amazon.com
Devin Guillory
Dguillory@berkeley.edu
Keizo Kato†
Fujitsu Laboratories Ltd.
kato.keizo@jp.fujitsu.com
Trevor Darrell
trevor@eecs.berkeley.edu
Huijuan Xu
hkx5063@psu.edu
Action in video usually involves the interaction of human with objects. Action labels are typically composed of various combinations of verbs and nouns, but we may not have training data for all possible combinations. In this paper, we aim to improve the generalization ability of the compositional action recognition model to novel verbs or novel nouns that are unseen during training time, by leveraging the power of knowledge graphs. Previous work utilizes verb-noun compositional action nodes in the knowledge graph, making it inefficient to scale since the number of compositional action nodes grows quadratically with respect to the number of verbs and nouns. To address this issue, we propose our approach: Disentangled Action Recognition with Knowledge-bases (DARK), which leverages the inherent compositionality of actions. DARK trains a factorized model by first extracting disentangled feature representations for verbs and nouns, and then predicting classification weights using relations in external knowledge graphs. The type constraint between verb and noun is extracted from external knowledge bases and finally applied when composing actions. DARK has better scalability in the number of objects and verbs, and achieves state-of-the-art performance on the Charades dataset. We further propose a new benchmark split based on the Epic-kitchen dataset which is an order of magnitude bigger in the numbers of classes and samples, and benchmark various models on this benchmark.
Understanding human-object interaction is crucial for modeling human behavior, and plays a key role in developing robotic agents that interact with humans. In videos, many of these interactions can be described using the combination of verbs and
Figure 1: DARK extracts disentangled features of verbs/nouns and leverages knowledge graphs (KG) to generate classifiers for unseen verbs and nouns. Predictions are then composed under constraints of object affordance priors (Affd) from knowledge bases (e.g. banana can be cut).
nouns, e.g., move chair, peel apple. Recently researchers have been focusing on the task of compositional action recognition with the goal of recognizing actions represented by such verb-noun pairs. The key challenge comes from the extremely large label space of various combinations. The number of possible verb-noun pairs grows quadratically with respect to the number of verbs and nouns. It is infeasible to collect training data for all possible actions. This motivates us to study the problem of zero-shot compositional action recognition, which aims to predict action with components beyond the vocabularies in train data.
To conduct zero-shot learning, we propose our Disentangled Action Recognition with Knowledgebases (DARK), which leverages knowledge graphs. A knowledge graph (KG) encodes semantic relationships between verb or noun nodes. We apply graph convolutional network (GCN) on KG to predict classifier weights for unseen nodes in graphs.
*Work done prior to Amazon
$\dagger$  Work was done when Keizo Kato was at CMU
Human Language Technologies, pages 573 - 586
Previous work (Kato et al., 2018) has explored using knowledge graphs for zero-shot compositional action learning. However, their model builds a graph containing verb, noun and compositional action nodes (which link verbs and nouns). It learns features of novel action nodes by propagating information from connected verb/noun nodes. The number of compositional action nodes during training is in the order of  $\mathcal{O}(n^2)$ , and the memory consumption may become prohibitively expensive as we scale this approach to large vocabularies. To overcome this issue, we propose to learn separate classifiers for verbs and nouns, which scales linearly with respect to the vocabulary size.
Specifically, DARK extracts verb and noun features separately, and relies on separate verb and noun knowledge graphs to predict unseen concepts before composing the action label. Activity recognition is particularly well-suited for such a factorized approach, because nouns may be better captured using object detection-based approaches and verbs may be represented by motion. Compared to prior work (Kato et al., 2018; Zhukov et al., 2019) that use the same feature representation for both verb and noun, our separate features model noun and verb more precisely. In addition, we adopt disentanglement between the learned verb and noun features, so they compose more readily and improve generalization on unseen actions.
Though scalability is achieved using our factorized approach, verbs and nouns are actually not fully independent. For instance, the process of sanding an object and scrubbing an object are visually similar, however you are more likely to be scrubbing a car than sanding it. Prior work (Kato et al., 2018) models the verb-noun relationships by constructing quadratic compositional action nodes. In our model, when composing action labels, we take object affordances (Gibson, 1978), namely the commonsense relationship of verbs and nouns, into consideration. We extract affordance knowledge from a caption corpus and build a scoring component to consider the relationship between verbs and nouns, to further improve the generalization ability. The basic idea of our proposed model is illustrated in Figure 1.
Furthermore, we investigate the evaluation of zero-shot compositional action recognition task and identify the drawback of existing metrics. With  $N_{v}$  verbs,  $N_{n}$  nouns, it constructs a  $N_{v} \times N_{n}$  label space for possible actions. Among these actions,
some are invalid (e.g. peel a car) and some are valid but not presented in the dataset. In real-world applications, the model would need to make predictions in the whole  $N_v \times N_n$  label space. But current evaluation protocols, implicitly or explicitly, only evaluate on compositional classes that are valid and presented in the dataset, which does not reflect the real difficulty of this task. We propose a new setting, where predictions are made and evaluated in the full  $N_v \times N_n$  label space.
The Charades (Sigurdsson et al., 2016) dataset is relatively small scale for testing zero-shot compositional action recognition (Kato et al., 2018). To promote further research, we propose a new benchmark based on the Epic-kitchen (Damen et al., 2018, 2020) dataset, which is an order of magnitude bigger both in number of classes and sample size. The key contributions of our paper are:
1. We propose a novel factorized model that learns disentangled representation separately for verbs and nouns, facilitating scalability.
2. We further improve the model's generalization performance by learning the interaction constraints between verbs and nouns (affordance priors) from an external corpus.
3. We propose a new evaluation protocol for zero-shot compositional learnings, which better reflects the real-world application setting.
4. We propose a new large-scale benchmark based on the Epic-Kitchen dataset and achieve state-of-the-art results.
Zero-shot learning with knowledge graphs: Zero-shot learning has been widely studied in computer vision (Akata et al., 2015; Lampert et al., 2013; Lee et al., 2018; Sahu et al., 2020; Wang et al., 2019; Xian et al., 2018). We will focus on related work relevant to our approach. (Wang et al., 2018) proposes to distill both the implicit knowledge representations (e.g., word embedding) and explicit relationships (e.g., knowledge graph) to learn a visual classifier for new classes through GCN (Kipf and Welling, 2016). (Kampffmeyer et al., 2019) later proposes to augment the knowledge graph (KG) with dense connections which directly connects multi-hop relationship and distinguishes between parent and children nodes. The graph learning of our model mostly follows their
work. Recently there have been improvements on GCN models. (Nayak and Bach, 2020) designs a novel transformer GCN to learn representations based on common-sense KGs. (Geng et al., 2020b) uses an attentive GCN together with an explanation generator to conduct explainable zero-shot learning. Instead of generating classifier for unseen classes directly, (Geng et al., 2020a) uses a generative adversarial network to synthesize features for unseen classes to conduct classification. These directions could be potentially explored in our problem setting to further improve performance. (Gao et al., 2019) conducts zero-shot action recognition based on KGs, but unlike our problem setting, their verb-noun relationship is not compositional and objects are used as attributes to infer action.
Compositional action recognition: Many prior works aim to understand actions through interaction with objects. (Wang et al., 2020; Xu et al., 2019) tackle zero-shot human-object interaction in images. (Zhukov et al., 2019) conduct weakly-supervised action recognition, leveraging compositionality of verb-noun pairs to decompose tasks into a set of verb/noun classifiers. This shares certain similarities with our factorized model, but it is not a zero-shot setting, nor does it enforce feature disentanglement. (Materzynska et al., 2020) conduct zero-shot compositional action recognition, where individual verb/noun concepts have been seen during training but not in the same interaction with each other. Although it cannot deal with unseen verbs or nouns, using object detector to explicitly model object features inspires our approach. One of the closest works to our proposed approach is (Kato et al., 2018). It constructs a KG that contains verb nodes, noun nodes and compositional action nodes, and learns the feature representation for each action node to match visual features. Novel actions' features are inferred jointly during training through GCN. The number of action nodes grows quadratically with respect to the number of nouns or verbs, which makes this approach difficult to scale, especially considering that GCN's forward pass needs to learn all features simultaneously.
We propose DARK - Disentangled Action Recognition with Knowledge-bases (Figure 2). It extracts disentangled feature representations for verbs and nouns, then predicts classifier weights for unseen components using knowledge graphs, and com
poses them under object affordance priors.
3.1 Factorized verb-noun classifier
Given a video  $X$ , we first use a verb feature extractor  $\mathcal{F}_v$  to extract verb feature, and a noun feature extractor  $\mathcal{F}_n$  for noun feature. Subsequently, we learn one-layer predictors  $\mathcal{W}_v^{seen}$  and  $\mathcal{W}_n^{seen}$  for predicting the final verb/noun class.  $\mathcal{F}_v$ ,  $\mathcal{F}_n$  and  $\mathcal{W}_v^{seen}$ ,  $\mathcal{W}_n^{seen}$  are trained via cross entropy (CE) losses  $\mathcal{L}_{cls}^v$  and  $\mathcal{L}_{cls}^n$  with verb / noun labels  $y_v$ ,  $y_n$ .
$$
\mathcal {L} _ {c l s} ^ {v} = \mathbf {C E} \left(\mathcal {F} _ {v}; \mathcal {W} _ {v} ^ {\text {s e e n}}; y _ {v}\right) \tag {1}
$$
$$
\mathcal {L} _ {c l s} ^ {n} = \mathbf {C E} \left(\mathcal {F} _ {n}; \mathcal {W} _ {n} ^ {\text {s e e n}}; y _ {n}\right) \tag {2}
$$
We extract disentangled features for verbs and nouns, so that verbs and nouns can be treated as separate entities. If verb features contain much information about nouns, it would overfit to seen actions and would not generalize to unseen compositions. Standard networks like Inception3D (I3D) (Carreira and Zisserman, 2017) can rely on scene or object information to predict verbs (Battaglia et al., 2018; Materzynska et al., 2020). To decouple verb's representation from noun's, we add explicit regularization to the model input. We first used an off-the-shelf class-agnostic object detector to detect the bounding box of interacting objects. Then, we crop the object from videos and use I3D backbone to extract verb features from the cropped videos. (Yun et al., 2019; DeVries and Taylor, 2017; Choi et al., 2019; Singh and Lee, 2017) use similar cropping technique to remove bias in other tasks. We also detect hand masks and add hand regions separately to the verb input because the class-agnostic detector tends to crop out the hands as well. Adding hand gesture information back provides hints for verbs. Disentanglement method on Charades dataset (Sigurdsson et al., 2016) is different as it contains third-person view videos, and relevant details are discussed in Section 4.5.
3.2 GCN for learning novel concept
After training on seen concepts, we can infer the classifier for unseen ones. In this subsection, we drop the subscript  $v / n$  as the same process applies for both verb and noun. After learning feature extractor  $\mathcal{F}$  and classifier of seen concept  $\mathcal{W}^{seen}$ , learning classifier for unseen concepts is equivalent to learning the weight  $\mathcal{W}^{unseen}$ . This step leverages graph convolution network (GCN) following previous work (Kampffmeyer et al., 2019; Kato
Figure 2: Overall training process. We first jointly train factorized disentangled feature extractor  $\mathcal{F}_v$ ,  $\mathcal{F}_n$ , and classifier weights for seen class  $\mathcal{W}_n^{seen}$ ,  $\mathcal{W}_v^{seen}$ . We then take word embedding  $s_{seen}$ ,  $s_{unseen}$  as input, and use GCN to predict classifier weights of unseen class  $\mathcal{W}_n^{unseen}$ ,  $\mathcal{W}_v^{unseen}$  based on knowledge graphs (KGs). Note that the same GCN learning applies to both verbs / nouns, and in the figure we only show the noun's part for brevity.
et al., 2018; Xian et al., 2017). It takes the word embedding of unseen/seen concepts  $s_{seen}, s_{unseen}$  and conducts graph convolution on the KG, with previously learned classifier weight  $\mathcal{W}^{seen}$  as supervision. In each layer, it calculates:
$$
Z _ {i + 1} = \hat {A} Z _ {i} W _ {i} \tag {3}
$$
$Z_{i}$  and  $Z_{i + 1}$  are input and output of the layer  $i$ ,  $\hat{A}$  is the adjacency matrix of the graph. Following (Kampffmeyer et al., 2019; Kato et al., 2018; Wang et al., 2018), we normalize the adjacency matrix.  $W_{i}$  is a learnable parameter. GCN first transforms features linearly, then aggregates information between nodes via graph edges. The  $0^{th}$  layer's input  $Z_{0}$  is the word embedding  $[s_{seen}, s_{unseen}]$ . The last layer's output  $Z_{n}$  is the classifier weight  $[\mathcal{W}^{\text{seen}}, \mathcal{W}^{\text{unseen}}]$ . We use the  $\mathcal{W}^{\text{seen}}$  learned previously as supervision, and calculate the mean square error loss between  $\mathcal{W}^{\text{seen}}$  and  $\mathcal{W}^{\text{seen}}$ . The training process is illustrated in Figure 2. Only the GCN learning of nouns is shown for brevity.
$$
\mathcal {L} _ {G C N} = \mathcal {L} _ {m s e} (\hat {\mathcal {W} ^ {s e e n}}, \mathcal {W} ^ {s e e n}) \tag {4}
$$
3.3 Incorporating affordance prior
Not all verb-noun pairs are equally important — some objects can only admit certain actions but not others. (Gibson, 1978) proposed the notion of “affordance” — the shape of an object may provide hints on how humans should use it, which
induces the set of suitable actions. Affordance can be extracted from the language source, e.g. we will often say peel the apple but rarely peel the chair. Prior works (Zhuang et al., 2017; Lu et al., 2016) used language information as prior to improve their performance. In this paper, we use captions of HowTO100M dataset (Miech et al., 2019) which records human-object interaction. We run the Standford NLP parser (Chen and Manning, 2014) to extract nouns/verbs from captions automatically.
After extracting verb-noun pairs, we train a scoring function  $\mathcal{A}$  to calculate the verb-noun affordance matching score. We project verb embedding  $s_v$  to the noun embedding space and calculate cosine distance with  $s_n$ , followed by sigmoid to output a scalar value indicating whether this verb-noun compositional action is plausible. For training, we generate positive/negative pairs and use binary cross-entropy loss  $\mathcal{L}_{affd}$ . Note that there underlies an open-world assumption (Nickel et al., 2015): the verb-noun pairs missing are not entirely infeasible, but could be unobserved. Further research can be explored to develop a more precise way of modeling the affordance constraint.
A scoring function based on only word-embedding is similar to a static look-up table for verb-noun pairs, and may fail to encode diverse action visual features. Thus we train a mapping function  $\mathcal{M}$  to transform verb's visual input to its word embedding  $s_v$  using mean square error loss.
Figure 3: We train a scoring function  $\mathcal{A}$  to calculate the affordance matching between verb and noun. In test time, the matching score is computed between mapped verb visual feature  $\mathcal{M}(X_v)$  and  $s_n$ .
Algorithm 1 Training process of our DARK model
1. Train the feature extractor and corresponding classification weight  $(\mathcal{F}_v,\mathcal{W}_v)$ $(\mathcal{F}_n,\mathcal{W}_n)$  via classification loss  $\mathcal{L}_v,\mathcal{L}_n$  separately.
2. Use the word embeddings  $s_v, s_n$  as input and the learned classification weight  $\mathcal{W}_v, \mathcal{W}_n$  as supervision, to train the GCN model  $(\mathcal{G}_v, \mathcal{G}_n)$  with mean square error (MSE) loss (Equation 4).
3. Use extracted affordance pairs to train scoring function  $\mathcal{A}(s_v,s_n)$  (Figure 3), via binary cross entropy (BCE) loss  $\mathcal{L}_{affd}$ .
4. Train mapping function  $\mathcal{M}$  to map visual verb inputs to semantic embedding space (Equation 5).
$$
\mathcal {L} _ {m s e} \left(\mathcal {M} \left(X _ {v}\right), s _ {v}\right) \tag {5}
$$
The separation of  $\mathcal{A}$ ,  $\mathcal{M}$  also adds interpretability and allows learning from different data.  $\mathcal{A}$  can be trained on a language corpus without video data. Also,  $\mathcal{A}$  deals with textual affordance relationship directly and adds interpretability. In test time we map verb's visual input to verb embedding space and calculate affordance score with target noun's embedding  $s_n$  (Figure 3). The model is asymmetric, since we use object proposals with false detection and verb visual input is more reliable.
3.4 Overall algorithm and inference
The training of our DARK model is shown in Algorithm 1. During inference, we calculate the probability of a video containing the compositional ac
Figure 4: The seen compositional actions correspond to  $V_{s}N_{s}$  (the upper left part), and unseen actions include  $V_{s}N_{u}, V_{u}N_{s}$  and  $V_{u}N_{u}$  (the rest). We visualize the scope of close / open / macro-open world settings.
tion  $(v,n)$  using following equations:
$$
\mathcal {P} (v, n) = \mathcal {P} (v) * \mathcal {P} (n) * \mathcal {A} (\mathcal {M} \left(X _ {v}\right), s _ {n}) \tag {6}
$$
$$
\mathcal {P} (v) = \sigma \left(\mathcal {W} _ {v} * \mathcal {F} _ {v} (X)\right) \tag {7}
$$
$\sigma$  is sigmoid function. For the classification weights  $\mathcal{W}_v$  used in verb prediction  $\mathcal{P}(v)$ , we use the learned classification weight  $\mathcal{W}_v^{\text{seen}}$  for seen classes, and  $\mathcal{W}_v^{\text{unseen}}$  predicted by GCN for unseen classes. Similar equation applies for noun prediction  $\mathcal{P}(n)$ , which is omitted.
In this section, we discuss experiment evaluation, setup and results. Some implementation details are in appendix.
4.1 Evaluation of zero-shot compositional task
Following previous work (Kato et al., 2018), we partition verbs into two disjoint sets for seen/unseen classes,  $V_{s} / V_{u}$ , and same for nouns,  $N_{s} / N_{u}$ . Thus, "seen compositional actions" correspond to  $V_{s}N_{s}$ , while "unseen (zero-shot) actions" correspond to  $V_{s}N_{u}, V_{u}N_{s}$  and  $V_{u}N_{u}$ .
Prior to (Chao et al., 2016), most works (e.g. (Norouzi et al., 2013)) on zero-shot learning adopt an evaluation protocol where predictions are made and evaluated only among unseen classes.
$^{1}$ Code and proposed benchmark are in https://github.com/airmachine/DARK.
This is later denoted as close world setting. (Chao et al., 2016) points out that it does not reflect the difficulty of recognition in practice, and there exists a trade-off between classification accuracies of seen / unseen classes. They propose generalized zero-shot learning (GZL) setting — test set contains samples of both seen / unseen classes. Predictions are made and evaluated on both categories. By adding different biases to unseen classes' prediction, one can draw a curve depicting the trade-off between accuracies of seen/unseen samples. They use area under the curve (AUC) to better reflect model's overall performance on both seen and unseen classes. Our evaluation follows this setup.
Currently, relatively few prior works tackle zero-shot compositional action recognition (Kato et al., 2018; Materzynska et al., 2020). Taking other zero-shot compositional learning tasks such as zero-shot attribute-object classification (Misra et al., 2017; Nagarajan and Grauman, 2018; Purushwalkam et al., 2019; Yang et al., 2020) and image-based zero-shot human-object interaction (Wang et al., 2020; Xu et al., 2019) into consideration, we find that zero-shot compositional learning task poses extra challenges due to its combinatorial label space: not all compositional labels are valid, and for the valid ones, there may be no samples in the dataset. For brevity, we continue to use the term "verb" and "noun", but the following discussion could be also applied to other zero-shot compositional learning tasks (e.g., attribute-object). As in Figure 4, with  $N_v$  verbs,  $N_n$  nouns, we can construct a  $N_v \times N_n$  action label space. Among these actions, some are invalid, because the verb-noun pair contradicts our common sense (e.g. peel a car). And some are valid but not presented in the dataset. Most previous works only consider labels that contain samples in the dataset, namely compositional classes that are valid and presented in the dataset.
Prior works (Materzynska et al., 2020; Misra et al., 2017; Nagarajan and Grauman, 2018; Yang et al., 2020) use disjoint label spaces in training and test sets, which corresponds to the close world setting. (Purushwalkam et al., 2019)'s test set contains both seen and unseen classes (GZL setting) and uses the AUC metric like ours, but their prediction is made and evaluated only among compositional classes with samples in dataset. (Kato et al., 2018; Wang et al., 2020) also follow GZL setting, but they use mean average precision (mAP) over compositional classes presented in dataset, which
implicitly only considers valid and presented classes. We denote this as open world setting: test set contains samples from seen/unseen classes, but prediction is made and/or evaluated among valid and presented classes.
Neither close world setting nor open word setting reflects the difficulty of zero-shot compositional action recognition task. When deploying recognition models in the real world, it would need to make predictions in the whole  $N_v \times N_n$  label space. Thus, compositional constraints between verbs and nouns (affordance) should be properly modeled to exclude invalid classes. In addition, the evaluation protocol should not distinguish between classes that are valid but not presented and valid and presented in the dataset, because models would not have access to that information beforehand. We propose the marco open world setting. In test time, sample can be from all seen/unseen classes, including  $V_sN_s$ ,  $V_sN_u$ ,  $V_uN_s$  and  $V_uN_u$ , and model receives no information about where the sample comes from. Predictions are made and evaluated in the whole  $N_v \times N_n$  label space, and the AUC metric (Chao et al., 2016) considering the trade-off between seen/unseen classes is reported. Figure 4 compares these three settings.