SlowGuess/NAACL_2021_OCR huggingface dataset:
Jihyung Kil
kil.5@osu.edu
Wei-Lun Chao
chao.209@osu.edu
Zero-shot learning aims to recognize unseen objects using their semantic representations. Most existing works use visual attributes labeled by humans, not suitable for large-scale applications. In this paper, we revisit the use of documents as semantic representations. We argue that documents like Wikipedia pages contain rich visual information, which however can easily be buried by the vast amount of non-visual sentences. To address this issue, we propose a semi-automatic mechanism for visual sentence extraction that leverages the document section headers and the clustering structure of visual sentences. The extracted visual sentences, after a novel weighting scheme to distinguish similar classes, essentially form semantic representations like visual attributes but need much less human effort. On the ImageNet dataset with over 10,000 unseen classes, our representations lead to a  $64\%$  relative improvement against the commonly used ones.
Algorithms for visual recognition usually require hundreds of labeled images to learn how to classify an object (He et al., 2016). In reality, however, the frequency of observing an object follows a long-tailed distribution (Zhu et al., 2014): many objects do not appear frequently enough for us to collect sufficient images. Zero-shot learning (ZSL) (Lampert et al., 2009), which aims to build classifiers for unseen object classes using their semantic representations, has thus emerged as a promising paradigm for recognizing a large number of classes.
Being the only information of unseen objects, how well the semantic representations describe the visual appearances plays a crucial role in ZSL. One popular choice is visual attributes (Lampert et al., 2009; Patterson and Hays, 2012; Wah et al., 2011) carefully annotated by humans. For example, the bird "Red bellied Woodpecker" has the "capped head pattern" and "pointed wing shape". While
Tiger Wikipage
...its dark vertical stripes on orange-brown fur .... has a muscular body with powerful forelimbs, a large head and a tail. ...listed as endangered on the IUCN Red List. ... The tiger is the national animal of India ...
Lion Wikipage
... a muscular, deep-chested cat with a rounded head, a reduced neck and round ears. ... male lions have a prominent mane, ... Lion hunting has occurred since ancient times... In Africa, the lion has been a common character in stories...
Figure 1: An illustration of our ZSL approach, which recognizes the input image by comparing it to the visual sentences of documents. Here we show two documents, one for "Tiger" and one for "Lion". The gray area highlights the extracted visual sentences (red: by section headers; blue: by clustering).
strictly tied to visual appearances, visual attributes are laborious to collect, limiting their applicability to small-scale problems with hundreds of classes.
For large-scale problems like ImageNet (Deng et al., 2009) that has more than 20,000 classes, existing ZSL algorithms (Frome et al., 2013; Norouzi et al., 2013) mostly resort to word vectors of classes names (Mikolov et al., 2013; Pennington et al., 2014) that are automatically extracted from large corpora like Common Crawl. While almost labor free, word vectors are purely text-driven and barely aligned with visual information. As a result, the state-of-the-art ZSL accuracy on ImageNet falls far behind being practical (Changpinyo et al., 2020).
Is it possible to develop semantic representations that are as powerful as visual attributes without significant human effort? A feasibility study by representing a class with its Wikipedia page shows some positive signs — Wikipedia pages do capture rich attribute information. For example, the page "Red-bellied Woodpecker" contains phrases "red cap going from the bill to the nape" and "black and white barred patterns on their back, wings and tail" that exactly match the visual attributes mentioned above. In other words, if we can identify visual sentences from a document to represent a class, we are likely to attain much higher ZSL accuracy<sup>1</sup>.
$^{1}$ Representing a class by a document has been studied in (Zhu et al., 2018; Elhoseiny et al., 2013; Qiao et al., 2016), but they use all sentences instead of extracting the visual ones.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3117-3128 June 6-11, 2021. ©2021 Association for Computational Linguistics
To this end, we present a simple yet effective semi-automatic approach for visual sentence extraction, which leverages two informative semantic cues. First, we leverage the section structures of Wikipedia pages: the section header indicates what kind of sentences (visual or not) appear in the section. Concretely, we search Wikipedia pages of common objects following the sysnets in ImageNet (e.g., fish, room), and manually identify sections that contain visual information (e.g., characteristics, appearance). We then apply these visual headers to the Wikipedia pages of the remaining ImageNet classes. Second, we observe that visual sentences share some common contextual patterns: for example, they contain commonly used words or phrases of visual attributes (e.g., red color, furry surface). To leverage these patterns, we perform K-means sentence clustering using the BERT features (Devlin et al., 2018) and manually select clusters that contain visual information. We keep sentences in these clusters and combine them with those selected by section headers to represent a document. See Figure 1 for an illustration.
To further increase the discriminative ability of the visual sentences between similar object classes (e.g., breeds of dogs), we introduce a novel scheme to assign weights to sentences, emphasizing those that are more representative for each class.
We validate our approach on three datasets: ImageNet Fall 2011 dataset (Deng et al., 2009), which contains 14,840 unseen classes with Wikipedia pages; Animals with Attributes 2 (AwA2) (Xian et al., 2018a), which has 50 animal classes; Attribute Pascal and Yahoo (aPY) (Farhadi et al., 2009), which has 32 classes. Our results are promising: compared to word vectors on ImageNet, we improve by  $64\%$  using visual sentences. On AwA2 and aPY, compared to visual attributes annotated by humans, we improve by  $8\%$  and  $5\%$ , respectively. Moreover, our new semantic representations can be easily incorporated into any ZSL algorithms. Our code and data will be available at https://github.com/heendung/vs-zsl.
Semantic representations. Visual attributes are the most popular semantic representations (Lampert et al., 2009; Patterson and Hays, 2012; Wah et al., 2011; Zhao et al., 2019). However, due to the need of human annotation, the largest dataset has only 717 classes. Reed et al. (2016b,a) collect vi
sual sentences for each image, which is not scalable. For large-scale recognition, word vectors (Mikolov et al., 2013) have been widely used. Lu (2015); Kampffmeyer et al. (2019); Wang et al. (2018) explore the use of WordNet hierarchy (Miller, 1995), which may not be available in other applications.
Similar to ours, Akata et al. (2015b); Elhoseiny et al. (2013); Qiao et al. (2016); Zhu et al. (2018) represent classes by documents, by counting word frequencies but not extracting visual sentences. Al-Halah and Stiefelhagen (2017) extract single word attributes, which are not discriminative enough (e.g., "red cap" becomes "red", "cap"). None of them works on ZSL with over 1,000 classes.
Hessel et al. (2018); Le Cacheux et al. (2020) collect images and tags of a class and derives its semantic representation from tags, which is not feasible for unseen classes on ZSL.
Zero-shot learning algorithms. The most popular way is to learn an embedding space in which visual features and semantic representations are aligned and nearest neighbor classifiers can be applied (Changpinyo et al., 2017; Romera-Paredes and Torr, 2015; Akata et al., 2015a; Kodirov et al., 2017; Schonfeld et al., 2019; Zhu et al., 2019; Xie et al., 2019; Socher et al., 2013). These algorithms consistently improve accuracy on datasets with attributes. Their accuracy on ImageNet, however, is saturated, mainly due to the poor quality of semantic representations (Changpinyo et al., 2020).
3 Visual Sentence Extraction
3.1 Background and notation
ZSL algorithms learn to align visual features and semantic representations using a set of seen classes  $S$ . The alignment is then applied to the test images of unseen classes  $\mathcal{U}$ . We denote by  $\mathcal{D} = \{(x_{n},y_{n}\in S)\}_{n = 1}^{N}$  the training data (i.e., image feature and label pairs) with the labels coming from  $S$ .
Suppose that we have access to a semantic representation  $\pmb{a}_c$  (e.g., word vectors) for each class  $c \in S \cup \mathcal{U}$ , one popular algorithm DeViSE (Frome et al., 2013) proposes the learning objective
$$
\begin{array}{l} \sum_ {n} \sum_ {c \neq y _ {n}} \max \{0, \Delta - f _ {\pmb {\theta}} ^ {\top} (\pmb {x} _ {n}) M g _ {\phi} (\pmb {a} _ {y _ {n}}) \\ + f _ {\boldsymbol {\theta}} ^ {\top} \left(\boldsymbol {x} _ {n}\right) M g _ {\phi} \left(\boldsymbol {a} _ {c}\right) \}, \tag {1} \\ \end{array}
$$
where  $\Delta \geq 0$  is a margin. That is, DeViSE tries to learn transformations  $f_{\theta}$  and  $g_{\phi}$  and a matrix  $M$  to maximize the visual and semantic alignment of
Characteristics, Description, Appearance, Habitat, Diet,
Construction and Mechanics, Materials for utensil,
Design for appliance, Furnishings for room, Fabrication,
Feature for geological formation, Design, Equipment for sport
History, Health, Terminology, Mythology, Conservation, Culture,
References, External links, Further reading
Table 1: Visual (top) & Non-Visual (bottom) sections.
the same classes while minimizing that between classes. We can then classify a test image  $\pmb{x}$  by
$$
\arg \max  _ {c \in \mathcal {U}} f _ {\boldsymbol {\theta}} ^ {\top} (\boldsymbol {x}) M g _ {\phi} (\boldsymbol {a} _ {c}). \tag {2}
$$
Here, we consider that every class  $c \in \mathcal{S} \cup \mathcal{U}$  is provided with a document  $H_{c} = \{\pmb{h}_{1}^{(c)}, \dots, \pmb{h}_{|H_{c}|}^{(c)}\}$  rather than  $\pmb{a}_{c}$ , where  $|H_{c}|$  is the amount of sentences in document  $H_{c}$  and  $\pmb{h}_{j}^{(c)}$  is the  $j$ th sentence, encoded by BERT (Devlin et al., 2018). We mainly study DeViSE, but our approach can easily be applied to other ZSL algorithms.
3.2 Visual section selection
We aim to filter out sentences in  $H_{c}$  that are not describing visual information. We first leverage the section headers in Wikipedia pages, which indicate what types of sentences (visual or not) are in the sections. For example, the page "Lion" has sections "Description" and "Colour variation" that are likely for visual information, and "Health" and "Cultural significance" that are for non-visual information.
To efficiently identify these section headers, we use ImageNet synsets (Deng et al., 2009), which group objects into 16 broad categories. We randomly sample  $30 \sim 35$  classes per group, resulting in a set of 500 classes. We then retrieve the corresponding Wikipedia pages by their names and manually identify section headers related to visual sentences. By sub-sampling classes in this way, we can quickly find section headers that are applicable to other classes within the same groups. Table 1 shows some visual/non-visual sections gathered from the 500 classes. For example, "Characteristics" frequently appears in pages of animals to describe their appearances. In contrast, sections like "History" or "Mythology" do not contain visual information. Investigating all the 500 Wikipedia pages carefully, we find 40 distinct visual sections. We also include the first paragraph of a Wikipedia page, which often contains visual information.
3.3 Visual cluster selection
Our second approach uses K-means for sentence clustering: visual sentences often share common
It has large ears that help the fox lower its body temperature.
It usually has a gray coat, with rusty tones, and a black tip to its tail.
It has distinct dark patches around the nose.
It is most recognisable for its dark vertical stripes on orangish-brown fur.
muscular body with powerful forelimbs, a large head and a tail.
They have a mane-like heavy growth of fur around the neck and jaws . . .
The kit fox is a socially monogamous species.
Male and female kit foxes usually establish monogamous mating
The average lifespan of a wild kit fox is 5.5 years.
Tiger mates all year round, but most cubs are born between March  $\cdot \cdot \cdot$
The father generally takes no part in rearing.
The mortality rate of tiger cubs is about  $50\%$  in the first two years.
Table 2: Sentence clusters. The top cluster is visual and the bottom one is non-visual. The sentences from a class kit-fox are in red and those from a class tiger are in blue.
words and phrases of visual attributes, naturally forming clusters. We represent each sentence using the BERT features (Devlin et al., 2018), and perform K-means (with  $K = 100$ ) over all the sentences from Wikipedia pages of ImageNet classes. We then manually check the 100 clusters and identify 40 visual clusters. Table 2 shows a visual (top) and a non-visual (bottom) cluster. We highlight sentences related to two classes: "kit-fox" (red) and "tiger" (blue). The visual cluster describes the animals' general appearances, especially about visual attributes "dark", "black", "tail", "large", etc. In contrast, the non-visual cluster describes mating and lifespan that are not related to visual aspects.
3.4 Semantic representations of documents
After we obtain a filtered document  $\hat{H}_c$ , which contains sentences of the visual sections and clusters, the next step is to represent  $\hat{H}_c$  by a vector  $\mathbf{a}_c$  so that nearly all the ZSL algorithms can leverage it.
A simple way is average,  $\bar{a}_c = \frac{1}{|\hat{H}_c|}\sum_{h\in \hat{H}_c}h$  where  $h$  is the BERT feature. This, however, may not be discriminative enough to differentiate similar classes that share many common descriptions (e.g., dog classes share common phrase like "a breed of dogs" and "having a coat or a tail").
We therefore propose to identify informative sentences that can enlarge the difference of  $\pmb{a}_c$  between classes. Concretely, we learn to assign each sentence a weight  $\lambda$ , such that the resulting weighted average  $\pmb{a}_c = \frac{1}{|\hat{H}_c|} \sum_{\pmb{h} \in \hat{H}_c} \lambda(\pmb{h}) \times \pmb{h}$  can be more distinctive. We model  $\lambda(\cdot) \in \mathbb{R}$  by a multi-layer perceptron (MLP)  $b_\psi$
$$
\lambda (\boldsymbol {h}) = \frac {\exp \left(b _ {\boldsymbol {\psi}} (\boldsymbol {h})\right)}{\sum_ {\boldsymbol {h} ^ {\prime} \in \hat {H} _ {c}} \exp \left(b _ {\boldsymbol {\psi}} \left(\boldsymbol {h} ^ {\prime}\right)\right)}. \tag {3}
$$
We learn  $b_{\psi}$  to meet two criteria. On the one hand, for very similar classes  $c$  and  $c'$  whose similarity  $\cos(\pmb{a}_c, \pmb{a}_{c'})$  is larger than a threshold  $\tau$ , we want
Sentence clusters
$\cos (\pmb{a}_c, \pmb{a}_{c'})$  to be smaller than  $\tau$  so they can be discriminable. On the other hand, for other pairs of less similar classes, we want their similarity to follow the average semantic representation  $\bar{a}_c^2$ .
To this end, we initialize  $b_{\psi}$  such that the initial  $a_{c}$  is close to  $\bar{a}_{c}$ . We do so by first learning  $b_{\psi}$  to minimize the following objective
$$
\sum_ {c \in \mathcal {S} \cup \mathcal {U}} \max  \{0, \epsilon - \cos \left(\boldsymbol {a} _ {c}, \bar {\boldsymbol {a}} _ {c}\right) \}. \tag {4}
$$
We set  $\epsilon = 0.9$ , forcing  $a_{c}$  and  $\bar{a}_{c}$  of the same class to have  $\cos(a_{c}, \bar{a}_{c}) > 0.9$ . We then fine-tune  $b_{\psi}$  by minimizing the following objective
$$
\sum_ {c} ^ {\mathcal {S} \cup \mathcal {U}} \sum_ {c \neq c ^ {\prime}} ^ {\mathcal {S} \cup \mathcal {U}} \max  \left\{0, \cos \left(\boldsymbol {a} _ {c}, \boldsymbol {a} _ {c ^ {\prime}}\right) - \tau \right\}. \tag {5}
$$
We assign  $\tau$  a high value (e.g., 0.95) to only penalize overly similar semantic representations. Please see the appendix for details.
Comparison. Our approach is different from DAN (Iyyer et al., 2015). First, we learn an MLP to assign weights to sentences so that their embeddings can be combined appropriately to differentiate classes. In contrast, DAN computes the averaged embedding and learns an MLP to map it to another (more discriminative) embedding space. Second, DAN leans the MLP with a classification loss. In contrast, we learn the MLP to reduce the embedding similarity between similar classes while maintaining the similarity for other pairs of classes.
4.1 Dataset and splits: ImageNet
We use the ImageNet Fall 2011 dataset (Deng et al., 2009) with 21,842 classes. We use the 1K classes in ILSVRC 2012 (Russakovsky et al., 2015) for DeViSE training and validation (cf. Equation 1), leaving the remaining 20,842 classes as unseen classes for testing. We follow (Changpinyo et al., 2016) to consider three tasks, 2-Hop, 3-Hop, and ALL, corresponding to 1,290, 5,984, and 14,840 unseen classes that have Wikipedia pages and word vectors and are within two, three, and arbitrary tree hop distances (w.r.t. the ImageNet hierarchy) to the 1K classes. On average, each page contains 80 sentences. For images, we use the 2,048-dimensional ResNet visual features (He et al., 2016) provided
Model	Type	Filter	2-Hop	3-Hop	ALL
Random	-	-	0.078	0.017	0.007
DeViSE	w2v-v2	-	6.45	1.99	0.78
BERTp	No	6.73	2.23	0.83
DeViSE*	w2v-v2	-	11.55	3.07	1.48
BERTp	No	13.84	4.05	1.75
Vissec	15.56	4.41	1.82
Visclu	15.72	4.49	2.01
Vissec-clu	15.86	4.65	2.05
BERTp-w	Vissec-clu	16.32	4.73	2.10
BERTf	No	17.70	5.17	2.29
Vissec	19.52	5.20	2.32
Visclu	19.74	5.37	2.36
Vissec-clu	19.82	5.39	2.39
BERTf-w	Vissec-clu	20.47	5.53	2.42
EXEM	w2v-v2	-	16.04	4.54	1.99
BERTf	Vissec-clu	21.22	5.42	2.37
HVE	w2v-v2	-	8.63	2.38	1.09
BERTf-w	Vissec-clu	18.42	5.12	2.07
Table 3: Comparison of different semantic representations on ImageNet. We use per-class Top-1 accuracy(%) The best is in red and the second best in blue.
by Xian et al. (2018a). For sentences, we use a 12-layer pre-trained BERT model (Devlin et al., 2018). We denote by  $\mathrm{BERT_p}$  the pre-trained BERT and  $\mathrm{BERT_f}$  the one fine-tuned with DeViSE. Please see the appendix for details.
4.2 Baselines, variants, and metrics
Word vectors of class names are the standard semantic representations for ImageNet. Here we compare to the state-of-the-art w2v-v2 provided by Changpinyo et al. (2020), corresponding to a skipgram model (Mikolov et al., 2013) trained with ten passes of the Wikipedia dump corpus. For ours, we compare using all sentences (NO), visual sections (Vissec) or visual clusters (Visclu), and both (Vissec-clu). On average, Vissec-clu filters out  $57\%$  of the sentences per class. We denote weighted average (Section 3.4) by  $\mathrm{BERT}_{\mathfrak{p - w}}$  and  $\mathrm{BERT}_{\mathrm{f - w}}$ .
The original DeViSE (Frome et al., 2013) has  $f_{\theta}$  and  $g_{\phi}$  as identity functions. Here, we consider a stronger version, DeViSE*, in which we model  $f_{\theta}$  and  $g_{\phi}$  each by a two-hidden layers multi-layer perceptron (MLP). We also experiment with two state-of-the-art ZSL algorithms, EXEM (Chang-pinyo et al., 2020) and HVE (Liu et al., 2020).
We use the average per-class Top-1 classification accuracy as the metric (Xian et al., 2018a).
Table 3 summarizes the results on ImageNet. In combining with each ZSL algorithm, our semantic representations  $\mathbf{Vis}_{\mathrm{sec - clu}}$  that uses visual sections
2The purpose of introducing  $\lambda (\cdot)$  is to improve  $a_{c}$  from the average representation  $\bar{a}_c$  to differentiate similar classes.
Model	Type	AwA2	aPY
ZSL	GZSL	ZSL	GZSL
U	S	H	U	S	H
DeViSE	Visual attributes	59.70	17.10	74.70	27.80	37.02	3.54	78.41	6.73
w2v-v2	39.56	2.18	69.29	4.22	27.67	1.68	85.53	3.22
BERTp + Vissec-clu	64.32	19.79	72.46	31.09	38.79	3.94	71.60	7.51
Table 4: Results on AwA2 and aPY. We compare different semantic representations. Visual attributes are annotated by humans. GZSL is the generalized ZSL setting (Xian et al., 2018a). In GZSL, U, S, H denote unseen class accuracy, seen class accuracy, and their harmonic mean, respectively. We use per-class Top-1 accuracy (\%).
and visual clusters for sentence extraction outperforms w2v-v2. More discussions are as follows.
BERT vs. w2v-v2. For both DeViSE\* and DeViSE,  $\mathrm{BERT_p}$  by averaging all the sentences in a Wikipedia page outperforms w2v-v2, suggesting that representing a class by its document is more powerful than its word vector.
DeViSE* vs. DeViSE. Adding MLPs to DeViSE largely improves its accuracy: from  $0.78\%$  (DeViSE + w2v-v2) to  $1.48\%$  (DeViSE* + w2v-v2) at ALL. In the following, we then focus on DeViSE*. Visual sentence extraction. Comparing different strategies for  $\mathrm{BERT}_{\mathrm{p}}$ , we see both VisCLU and Vissec largely improves NO, demonstrating the effectiveness of sentence selection. Combining the two sets of sentences (Vissec-clu) leads to a further boost.
Fine-tuning BERT. BERT can be fine-tuned together with DeViSE\*. The resulting  $\mathrm{BERT_f}$  has a notable gain over  $\mathrm{BERT_p}$  (e.g.,  $2.39\%$  vs.  $2.05\%$ ). Weighted average. With the weighted average  $(\mathrm{BERT}_{\mathrm{p - w}},\mathrm{BERT}_{\mathrm{f - w}})$ , we obtain the best accuracy. ZSL algorithms. EXEM + w2v-v2 outperforms  $\mathrm{DeViSE}^{\star} + \mathrm{w2v - v2}$ , but falls behind  $\mathrm{DeViSE}^{\star} + \mathrm{BERT}_{\mathrm{p - w}}$  (or  $\mathrm{BERT_f}$ ,  $\mathrm{BERT_{f - w}}$ ). This suggests that algorithm design and semantic representations are both crucial. Importantly, EXEM and HVE can be improved using our proposed semantic representations, demonstrating the applicability and generalizability of our approach.
4.4 Results on other datasets
Table 4 summarizes the results on AwA2 (Xian et al., 2018a) and aPY (Farhadi et al., 2009). The former has 40 seen and 10 unseen classes; the latter has 20 seen and 12 unseen classes. We apply DeViSE together with the 2,048-dimensional ResNet features (He et al., 2016) provided by Xian et al. (2018a). Our proposed semantic representations (i.e.,  $\mathbf{BERT}_{\mathbf{p}} + \mathbf{Vis}_{\mathbf{sec - clu}}$ ) outperform w2-v2 and the manually annotated visual attributes on both the ZSL and generalized ZSL (GZSL) settings. Please see the appendix for the detailed experimental setup. These improved results on Ima
Model	Type	Filter	2-Hop	3-Hop	ALL
DeViSE*	BERTp	No	13.84	4.05	1.75
BERTp-w-direct	No	14.85	4.25	1.79
BERTp	Par1st	13.48	4.10	1.78
Clsname	14.82	3.31	1.40
Vissec	15.56	4.41	1.82
Visclu	15.72	4.49	2.01
Vissec-clu	15.86	4.65	2.05
BERTp-w	Vissec-clu	16.32	4.73	2.10
Table 5: The effectiveness of our visual sentence extraction.  $\mathbf{BERT}_{\mathbf{p - w - direct}}$  directly learns visual sentences without our sentence selection.  $\mathbf{Par}_{1st}$  and  $\mathbf{Cls}_{\mathbf{name}}$  use the first paragraph and sentences containing the class name, respectively.
geNet, AwA2, and aPY demonstrate our proposed method's applicability to multiple datasets.
4.5 Analysis on ImageNet
To further justify the effectiveness of our approach, we compare to additional baselines in Table 5.
- BERTp-w-direct: it directly learns  $b_{\psi}$  (Equation 3) as part of the DeVise objective. Namely, we directly learn  $b_{\psi}$  to identify visual sentences, without our proposed selection mechanisms, such that the resulting  $a_{c}$  optimizes Equation 1.
-  $\mathbf{Par}_{\mathbf{lst}}$ : it uses the first paragraph of a document.
- CIsname: it uses the sentences of a Wikipedia page that contain the class name.
As shown in Table 5, our proposed sentence selection mechanisms (i.e., Vissec, Visclu, and Vissec-clu) outperform all the three baselines.