SlowGuess/NAACL_2021_OCR huggingface dataset:
ZSL relies heavily on the quality of semantic representations. Most recent work, however, focuses solely on algorithm design, trying to squeeze out the last bit of information from the pre-define, likely poor semantic representations. Changpinyo et al. (2020) has shown that existing algorithms are trapped in the plateau of inferior semantic representations. Improving the representations is thus more crucial for ZSL. We investigate this direction and show promising results by extracting distinctive visual sentences from documents for representations, which can be easily used by any ZSL algorithms.
This research is supported by the OSU GI Development funds. We are thankful for the support of computational resources by Ohio Supercomputer Center and AWS Cloud Credits for Research.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. 2015a. Label-embedding for image classification. TPAMI, 38(7):1425-1438.
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. 2015b. Evaluation of output embeddings for fine-grained image classification. In CVPR.
Ziad Al-Halah and Rainer Stiefelhagen. 2017. Automatic discovery, association estimation and learning of semantic attributes for a thousand categories. In CVPR.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. 2016. Synthesized classifiers for zero-shot learning. In CVPR.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. 2020. Classifier and exemplar synthesis for zero-shot learning. IJCV, 128(1):166-201.
Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. 2017. Predicting visual exemplars of unseen classes for zero-shot learning. In ICCV.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In CVPR. IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. 2013. Write a classifier: Zero-shot learning using purely textual descriptions. In ICCV.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. 2009. Describing objects by their attributes. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1778-1785. IEEE.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. In NIPS.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Jack Hessel, David Mimno, and Lillian Lee. 2018. Quantifying the visual concreteness of words and topics in multimodal datasets. arXiv preprint arXiv:1804.06786.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. arXiv preprint arXiv:1902.00751.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In ACL.
Michael Kampffmeyer, Yinbo Chen, Xiaodan Liang, Hao Wang, Yujia Zhang, and Eric P Xing. 2019. Rethinking knowledge graph propagation for zero-shot learning. In CVPR.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. 2017. Semantic autoencoder for zero-shot learning. In CVPR.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.08593.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In CVPR.
Yannick Le Cacheux, Adrian Popescu, and Herve Le Borgne. 2020. Webly supervised semantic embeddings for large scale zero-shot learning. In Proceedings of the Asian Conference on Computer Vision.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. 2020. Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9273-9281.
Yao Lu. 2015. Unsupervised learning on neural network outputs: with application in zero-shot learning. arXiv preprint arXiv:1506.00990.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. 2013. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650.
Genevieve Patterson and James Hays. 2012. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton Van Den Hengel. 2016. Less is more: zero-shot learning from online textual documents with noise suppression. In CVPR.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016a. Learning deep representations of fine-grained visual descriptions. In CVPR.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016b. Generative adversarial text to image synthesis. In ICML.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
Bernardino Romera-Paredes and Philip Torr. 2015. An embarrassingly simple approach to zero-shot learning. In ICML.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252.
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. 2019. Generalized zero-and few-shot learning via aligned variational autoencoders. In CVPR.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pages 935-943.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2008. Yago: A large ontology from wikipedia and wordnet. Journal of Web Semantics, 6(3):203-217.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The caltech-ucsd birds-200-2011 dataset.
Xiaolong Wang, Yufei Ye, and Abhinav Gupta. 2018. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. 2018a. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. TPAMI, 41(9):2251-2265.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. 2018b. Feature generating networks for zero-shot learning. In CVPR.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. 2019. f-vaegan-d2: A feature generating framework for any-shot learning. In CVPR.
Guo-Sen Xie, Li Liu, Xiaobo Jin, Fan Zhu, Zheng Zhang, Jie Qin, Yazhou Yao, and Ling Shao. 2019. Attentive region embedding network for zero-shot learning. In CVPR.
Amrapali Zaveri, Dimitris Kontokostas, Mohamed A Sherif, Lorenz Buhmann, Mohamed Morsey, Soren Auer, and Jens Lehmann. 2013. User-driven quality evaluation of dbpedia. In Proceedings of the 9th International Conference on Semantic Systems.
Bo Zhao, Yanwei Fu, Rui Liang, Jiahong Wu, Yonggang Wang, and Yizhou Wang. 2019. A large-scale attribute dataset for zero-shot learning. In CVPRW.
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama 2019. Generalized zero-shot recognition based on visually semantic embedding. In CVPR.
Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. 2014. Capturing long-tail distributions of object subcategories. In CVPR.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. 2018. A generative adversarial approach for zero-shot learning from noisy texts. In CVPR.
In this appendix, we provide details omitted in the main text.
- Appendix A: contribution
- Appendix B: more related work (cf. Section 2 in the main text)
- Appendix C: detailed statistics of Wikipedia pages (cf. Section 4.1 in the main text)
- Appendix D: weighted average representations (cf. Section 3.4 in the main text)
- Appendix E: dataset, metrics, and ZSL algorithms (cf. Section 4.2 in the main text)
- Appendix F: implementation details (cf. Section 4.3 in the main text)
- Appendix G: ablation study (cf. Section 4.3 in the main text)
- Appendix H qualitative results (cf. Section 3 in the main text)
A Contribution
Our contribution is not merely in the method we developed, but also in the direction we explored. As discussed in Section 5 of the main paper, most of the efforts in ZSL have focused on algorithm design to associate visual features and pre-defined semantic representations. Yet, it is also important to improve semantic representations. Indeed, one reason that ZSL performs poorly on large-scale datasets is the poor semantic representations (Changpinyo et al., 2020). We therefore chose to investigate this direction by revisiting document representations, with the goal to make our contributions widely applicable. To this end, we deliberately kept our method simple and intuitive, but also provided insights for future work to build upon. Our manual inspection identified important properties of visual sentences like the clustering structure, enabling us to efficiently extract them. We chose to not design new ZSL algorithms but make our semantic representations compatible with existing ones to clearly demonstrate the effectiveness of improving semantic representations.
Zero-shot learning (ZSL) algorithms construct visual classifiers based on semantic representations. Some recent work applies generative models to generate images or visual features of unseen classes (Xian et al., 2019, 2018b; Zhu et al., 2018), so that conventional supervised learning algorithms can be applied.
Knowledge bases usually contain triplets of entities and relationships. The entities are usually objects, locations, etc. For ZSL, we need entities to be fine-grained (e.g., "beaks") and capture more visual appearances. YAGO (Suchanek et al., 2008) and DBpedia (Zaveri et al., 2013) leverage Wikipedia infoboxes to construct triplets, which is elegant but not suitable for ZSL since Wikipedia infoboxes contain insufficient visual information. Thus, these datasets and construction methods may not be directly applicable to ZSL. Nevertheless, the underlying methodologies are inspiring and could serve as the basis for future work. The datasets also offer inter-class relationships that are complementary to visual descriptions, and may be useful to establish class relationships in ZSL algorithms like SynC (Changpinyo et al., 2016).
C Statistics of Wikipedia Pages
We use a Wikipedia API to extract pages from Wikipedia for ImageNet 21,842 classes. Among 21,842 classes, we find that some classes have multiple Wikipedia pages because of their ambiguous class names. For example, a class "black widow" in ImageNet refers to a spider with dark brown or a shiny black in colour, but it also refers to the name of a "Marvel Comics" character in Wikipedia. We therefore exclude such classes and also classes that do not have word vectors, resulting in 15,833 classes. The Wikipedia pages of the 15K classes contain 1,260,889 sentences where each class has 80 sentences on average. We also investigate the number of sentences by our filters (i.e. Vissec, Viscls, Vissec-clu). As a result, we correspondingly find 213,585, 534,852, 542,645 sentences, which are  $16\%$ ,  $42\%$ ,  $43\%$  of all sentences in 15K classes, respectively (See Figure 2).
D Weighted Average Representations
D.1 Observation
Two similar classes may have similar averaged visual sentence embeddings since they share many
Figure 2: Statistics of Wikipedia pages.
common descriptions. For example, Figure 3 shows that the averaged embedding (i.e.,  $\mathrm{BERT_p}$  and  $\mathrm{BERT_f}$ ) between "Kerry Blue Terrier" and "Soft-coated Terrier" are overly similar since they share a number of sentences containing the common dog features such as "a breed of dog" or "having a coat or a tail". Thus, if we represent their semantic representations  $a_{c}$  as the averaged embeddings, ZSL models may not differentiate them.
D.2 Algorithm
In Section 3.4 of the main text, we introduce  $\lambda(\cdot)$  to give each sentence  $h$  of a document a weight. We note that, while learning  $\lambda(\cdot)$  can enlarge the distance of  $a_{c}$  between similar classes, we should not overly maximize the distance to prevent semantically similar classes (e.g., different breed of dogs) end up being less similar than dissimilar classes (e.g., dogs and cats). To this end, we introduce a margin loss with  $\tau$  in Equation 5, which only penalize overly similar semantic representations.
We also note that, the purpose of  $\lambda(\cdot)$  is to improve  $a_{c}$  from the simple average embedding  $\bar{a}_{c}$ . We therefore initialize  $\lambda(\cdot)$  such that the initial  $a_{c}$  is similar to  $\bar{a}_{c}$ . We do so by first learning  $b_{\psi}$  with the following objective:
$$
\sum_ {c \in \mathcal {S} \cup \mathcal {U}} \max  \{0, \epsilon - \cos \left(\boldsymbol {a} _ {c}, \bar {\boldsymbol {a}} _ {c}\right) \}. \tag {6}
$$
We set  $\epsilon = 0.9$ , forcing  $a_{c}$  and  $\bar{a}_{c}$  to have a similarity larger than 0.9.
Figure 3 demonstrates the effectiveness of the weighted average embedding  $\mathrm{BERT}_{\mathrm{f - w}}$ . While other semantic representations predict "Kerry Blue
Terrier" as other similar dog, "soft-coated Terrier",  $\mathrm{BERT}_{\mathrm{f - w}}$  is able to classify the image correctly. In addition, based on the attention weights, we report the Top 3 sentences and the Bottom 3 sentences. The Top 1st sentence contains the inherent features for "Kerry Blue Terrier" such as long head or soft-to-curly coat while the Top 2nd and 3rd sentences describe general features of dogs. On the other hand, the Bottom 3 sentences do not have visual appearance of the object. This suggests that our weighted representation  $\mathrm{BERT}_{\mathrm{f - w}}$  is more representative to "Kerry Blue Terrier" than other semantic representations.
E Dataset, Features, Metrics, and ZSL Algorithm
For visual features, we use the 2,048-dimensional ResNet visual features (He et al., 2016) provided by Xian et al. (2018a). Word vectors can be found in (Changpinyo et al., 2020). Followed by (Xian et al., 2018a), we use the average per-class Top-1 accuracy as our metric. Instead of simply averaging over all test images (i.e. the average per-sample Top-1 accuracy), this accuracy is obtained by first taking average over all images in each test class independently and then taking average over all test classes. Compared to the average per-sample accuracy, the per-class accuracy is a more suitable for ImageNet since the dataset is highly imbalanced (Changpinyo et al., 2020). The state-of-the-art algorithms in ZSL are EXEM and HVE proposed by (Changpinyo et al., 2020) and (Liu et al., 2020), respectively. To make fair comparison with our models, we evaluate their algorithms on the same number of our test classes using their official codes.
We follow (Xian et al., 2018a; Changpinyo et al., 2016) to consider three tasks, 2-Hop, 3-Hop, and ALL, corresponding to 1, 509, 7, 678 and 20, 345 unseen classes that have word vectors and are within two, three, and arbitrary tree hop distances to the 1, 000 seen classes.
We search Wikipedia and successfully retrieve pages for 15,833 classes, of which 1,290, 5,984, and 14,840 are for 2-Hop, 3-Hop, and ALL.
E.2 AwA2
Animals with Attributes2 (AwA2) provides 37,322 images of 50 animal classes. On average, each class
Kerry Blue Terrier
Figure 3: Qualitative analysis of a class Kerry Blue Terrier. w2v-v2, BERTp, and BERTf can not distinguish between Kerry Blue Terrier and Soft-coated Terrier since two classes share the common features of dogs such as "a breed of dog" or "having a coat or a tail". On the other hand, our weighted average BERTf-w is able to differentiate them by weighting on the sentences. We report the Top 3 sentences and the Bottom 3 sentences based on the attention weights.
Model	Type	Prediction
DeViSE	w2v-v2	soft-coated Terrier
BERTp	soft-coated Terrier
BERTf	soft-coated Terrier
BERTf-w	Kerry Blue Terrier
Top 3 sentences
Kerry Blue Terrier include a long head … a wavy-to-curly coat … shades of blue …
The Kerry Blue Terrier is a breed of dog.
The coat is the key feature of the Kerry.
Bottom 3 sentences
Kerries require daily exercise.
Kerry Blue requires very regular grooming.
Many live to 13 or 14 years.
includes 746 images. It also provides 85 visual attributes that are manually annotated by humans. In AwA2, classes are split into 40 seen classes and 10 unseen classes. For GZSL, a total of 50 classes is used for testing.
E.3 aPY
Attribute Pascal and Yahoo (aPY) contains 15,339 images of 32 classes with 64 attributes. The classes are split into 20 seen classes and 12 unseen classes. A total of 32 classes is used for testing on GZSL.
E.4 DeViSE (Frome et al., 2013) vs.  
EXEM (Changpinyo et al., 2020) vs.  
HVE (Liu et al., 2020)
All algorithms learn feature transformations to associate visual features  $\pmb{x}$  and semantic representations  $\pmb{a}_{c}$ . The key differences are what and how to learn. DeViSE* learns two MLPs  $f_{\theta}$  and  $g_{\phi}$  to embed  $\pmb{x}$  and  $\pmb{a}_{c}$  into a common space, while HVE embeds them into a hyperbolic space. EXEM learns kernel regressors to embed  $\pmb{a}_{c}$  into the visual space. On how to learn, DeViSE* and HVE force each image  $\pmb{x}$  to be similar to the true class  $\pmb{a}_{c}$  by a margin loss and a ranking loss respectively, while EXEM learns to regress the averaged visual features of a class from  $\pmb{a}_{c}$ .
F.1 Sentence representations from BERT
Sentence representations can be defined in multiple ways such as a [CLS] token embedding or an average word embedding from different layers in BERT (Reimers and Gurevych, 2019). In our experiments, the average word embedding from the second last layer of BERT achieve the best results in all cases.
Model	Type	Filter	Threshold τ	2-Hop
DeViSE*	BERTp-w	Vissec-clu	0.98	15.97
0.97	16.09
0.96	16.32
0.95	16.13
BERTf-w	Vissec-clu	0.88	20.34
0.86	20.44
0.82	20.33
0.80	20.47
Table 6: Results of per-class Top-1 accuracy(%) on 2-Hop with different thresholds  $\tau$  and semantic representation types. The best is in red and the second best in blue.
F.2 Hyperparameters
DeViSE (Frome et al., 2013) has a tunable margin  $\Delta \geq 0$  (cf. Section 3.1 in the main text) which its default value is 0.1. We try multiple values 0.1, 0.2, 0.5, and 0.7 to find the best setting. DeViSE uses Adam optimizer which its learning rate is  $1e^{-3}$  by default. We try different possible values,  $1e^{-3}, 5e^{-4}, 2e^{-4}$ , and  $1e^{-4}$ . Among all 16 possible combinations of the margin and learning rate, we find that margin of 0.2 and learning rate of  $2e^{-4}$  achieve the best results on all our cases.
F.3 Fine-tuned models
For fine-tuning, DeViSE\* is first attached to a BERT model. Then, we train the model with jointly fine-tuning BERT parameters based on the DeViSE\* objective. Regards to BERT training, Houlsby et al. (2019) demonstrates that fine-tuning only last few  $n$  layers (e.g. 2 or 4) can outperform fine-tuning all layers in some NLP tasks. Kovaleva et al. (2019) also shows that the fine-tuning procedure is more effective to the last few layers than earlier layers. Considering the computational resources and time, we therefore set  $n$  equal to 2. After fine-tuning, we freeze BERT parameters and further train DeViSE\*.
Class	Top3 Similar Classes	Similarity
BERTp	BERTp-w
Sea boat	Scow	0.94	0.91
Row boat	0.93	0.91
Canoe	0.93	0.91
Table 7: Similarity of Top 3 similar classes with Sea boat drops after applying the weighting approach.
Table 6 shows the results on 2-Hop with different thresholds  $\tau$  introduced in Equation 5. We obtain the weighted average  $\mathrm{BERT}_{\mathrm{p - w}}$  by taking an input  $h$  from  $\mathrm{BERT}_{\mathrm{p}}$  and learning MLP  $b_{\psi}$  with different  $\tau$  (similar for  $\mathrm{BERT}_{\mathrm{f - w}}$ ). Then, we measure 2-Hop accuracy based on  $\mathrm{BERT}_{\mathrm{p - w}}$  (or  $\mathrm{BERT}_{\mathrm{f - w}}$ ). Note that  $\mathrm{BERT}_{\mathrm{p}}$  and  $\mathrm{BERT}_{\mathrm{f}}$  have different ranges of  $\tau$ , since  $\mathrm{BERT}_{\mathrm{f}}$  already has lower similarity between classes. This is because  $\mathrm{BERT}_{\mathrm{f}}$  is trained with images (from seen classes) during fine-tuning, which makes  $\mathrm{BERT}_{\mathrm{f}}$  more aligned with visual features and thus is more representative. We choose  $\tau$  based on the ImageNet validation set of the seen classes.
Table 7 shows that the weighted average embedding  $\mathrm{BERT}_{\mathbf{p - w}}$  makes similar classes less similar. Originally, a class "Sea boat" has overly similar semantic representations with other type of boats (i.e.  $\mathrm{BERT}_{\mathbf{p}}$ ). After applying our weighting approach, the classes become less similar (e.g. 0.94 to 0.91 between "Sea boat" and "Scow").
H Qualitative Results
H.1 Visual sections and clusters
We provide additional illustrations of visual sections and clusters of Section 3 in the main text.
Figure 4 shows visual and non-visual sections in a Wikipedia page Siberian Husky. We note that the summary paragraph and sections such as Description contain visual sentences while sections such as Health or History do not. Similarly, Table 8 shows two clusters: the top cluster is visual, consisting of information about hunting and preys of animals while the bottom cluster includes mythology sentences not visually related.
H.2 On ImageNet
Figure 5 shows the qualitative results of our  $\mathrm{BERT}_{\mathrm{f - w}}$  and w2v-v2 on ImageNet. For each image, we provide its label and the Top 5 prediction by  $\mathrm{BERT}_{\mathrm{f - w}}$  and w2v-v2. While w2v-v2 is not able
... hunt shortly after sunset, eating small animals ...
if food is scarce, it has been known to eat tomatoes  $\cdot \cdot \cdot$
Tigers are capable of taking down larger prey like adult gaur  $\cdot \cdot \cdot$
Tigers will also prey on such domestic livestock as cattle, horses,
Panda is a Roman goddess of peace and travellers
The Ibex is also a national emblem of the great ancient Axum empire.
In Aztec mythology, the jaguar was considered to be the totem animal of  $\cdot \cdot \cdot$
It is the national animal of Guyana, and is featured in its coat of arms . . .
Table 8: K-means sentence clusters. The top cluster has visual information about hunting and preys while the bottom one contains non-visual description such as mythology.
to differentiate the similar classes (e.g. Predicting "Scooter" as "Tandem bicycle"), our  $\mathrm{BERT}_{\mathrm{f - w}}$  can distinguish them. We also note that the Top 5 classes predicted by  $\mathrm{BERT}_{\mathrm{f - w}}$  are similar (e.g. "Grey whale" and "Killer whale"). This suggests that our approach maintains the order of similarity among classes but make their semantic representations more distinctive.
Figure 4: Visual sections on Siberian Husky.
Figure 5: Qualitative results between  $\mathbf{BERT}_{\mathbf{f - w}}$  and  $\mathbf{w2v - v2}$  on ImageNet. For each image, we report Top 5 prediction. While  $\mathbf{w2v - v2}$  is not able to distinguish similar classes (e.g. Predicting "Scooter" as "Tandem bicycle"), our  $\mathbf{BERT}_{\mathbf{f - w}}$  differentiates them.