SlowGuess/NAACL_2022_OCR huggingface dataset:
We believe that this is because the captions describe the similarities and differences between the images more obviously than the images themselves do. For example, to determine that two birds belong to different species, one piece of evidence is the belly color. The captions can explain this information clearly, e.g., "yellow belly" and "white belly". From the image, however, the extraction of this information requires multiple steps such as locating the belly and specifying its color.
Inner-Class Dist.	Inter-Class Dist.	LID
No Text (Image)	0.504	0.592	17.8
No Text Enc. (Image)	0.526	0.609	19.0
LIDE (Fusion)	0.459	0.709	6.73
What are characteristics of latent feature spaces? Second, we examined the dimensions of the feature spaces. Lee and Chung (2021) observed that the features embedded in a manifold with a smaller latent dimension are more generalized. They evaluated the latent dimension by using the average of the local intrinsic dimensions (LID) of the features. The LID measures the number of dimensions of a feature manifold in the neighbor of  $x$ , and it can be estimated as
$$
\mathrm {L I D} (x) = - \left\{\frac {1}{n _ {n n}} \sum_ {i} ^ {n _ {n n}} \log \left(\frac {r _ {i} (x)}{r _ {n _ {n n}} (x)}\right) \right\} ^ {- 1}
$$
by maximum likelihood estimation, where  $n_{nn}$  is the number of the nearest neighbors and  $r_i$  is the Euclidean distance from  $x$  to the  $i$ -th nearest neighbor (Levina and Bickel, 2005; Amsaleg et al., 2015). We set  $n_{nn} = 20$  in accordance with the previous studies.
Table 6 lists the estimated LIDs of the features of each model. In addition, we applied principal component analysis (PCA) to the features, and Figure 4 shows the cumulative contribution rates. All features were embedded in  $\mathbb{R}^{1600}$ .
The multi-modal features existed in a manifold with a smaller latent dimension than those of the image-only representations. Therefore, the text representation contributed to shrinking the representation manifold to a smaller dimension. We assume that the text controlled the main focus among the many objects in an image. For example, the captions in the CUB dataset describe the characteristics of birds. As a result, the model can extract the important information from captions for the downstream image classification task. In contrast, an image has much information, such as the background, and image features thus require a larger-dimension manifold.
Are multi-modal representations robust for noisy images? Next, we hypothesize that language descriptions can help to classify noisy or obscure images. To verify our hypothesis, we performed experiments in two heuristic noisy settings: One consisted of grayscale images and the
Figure 4: Cumulative contribution rates for PCA of each representations.
Table 6: Distribution of feature representations.
Original	Grayscale	Adversarial
No Text	62.22 ±0.92	38.04 ±0.78	33.09 ±0.69
No Text Enc.	63.10 ±0.90	38.55 ±0.80	31.30 ±0.65
LIDE	78.09 ±0.79	59.81 ±0.97	56.31 ±1.01
Table 7: Performance in noisy settings.
other consisted of images that were adversarially attacked via the fast gradient sign method (Goodfellow et al., 2015). We compared LIDE with gold captions to the ablated models using the image representations and evaluated their performance with the noisy images in the test phase.
Table 7 lists the result. When a caption was not provided, the classification accuracy dropped greatly in both settings. However, LIDE reduced the decline by virtue of the textual information. These results indicate that the text representation may be useful for classifying certain ill-conditioned images, such as an image in which the bird is extremely small.
What information do text representations have? Finally, we performed a probing test for each modality of representation, as in previous natural language processing (NLP) studies that discovered linguistic properties (Conneau et al., 2018; Hupkes and Zuidema, 2018) in the text representations. The CUB dataset has annotations of the birds' attributions, and we recover the attribution labels from  $h_{img}$  and  $h_{text}$ . Each image has  $\{0,1\}$  labels for 312 attributions.
We used our trained model to obtain  $h_{img}$  and  $h_{text}$  for all images and captions. Then, we used linear classifiers  $W_{img,attr} \in \mathbb{R}^{1600 \times 312}$  and  $W_{lang,attr} \in \mathbb{R}^{768 \times 312}$ , which were trained the linear classifiers with binary-cross-entropy loss in the training split. Next, we determined the thresholds in the development split and obtained prediction results in the test split. Finally, we performed a Wilcoxon signed-rank test between the image re
Image	Text	No Significance
43	17	8
Table 8: Numbers of significant attributions.
Table 9: Attributions that were significantly recovered from text representations.
sults and the text results.
Table 8 lists the numbers of attributions having significance at a  $p$ -value of 0.05. Among the 68 attributions, 60 were recovered more easily from one modality than from the other modality. In other words, the image and text representations complemented each other. Most of the attributes favored the image representation, but 17 of them favored the text representation. Table 9 lists those 17 attributes, and we can observe two main characteristics among them. First, the colors black and white were recovered from the text representation. These attributes may be difficult to recover from an image because of light and shadow. The second characteristic was a spotted pattern, which is obscure in an image with  $84 \times 84$  pixels.
Image classification with language Several studies have provides gold language information in the test phase. For zero-shot learning or few-shot learning, class-label words are used as additional information (Frome et al., 2013; Socher et al., 2013; Xing et al., 2019). Moreover, He and Peng (2017); Liang et al. (2020) used language descriptions for the standard image classification problem, in which the classes in the training and test phases are the same. The few-shot image classification is a more challenging problem that requires the capability
of learning novel concepts from language descriptions.
Textual explanation of image representation Explainability of artificial intelligence (XAI) has attracted much attention (Bastings et al., 2021). Papers have proposed methods to generate an image description for XAI in the image classification task and visual question answering (VQA) task (Hendricks et al., 2016, 2018; Li et al., 2018). In contrast to those studies, the motivation of this paper is to decode and encode such descriptions to improve the few-shot image classification performance. We could integrate the findings of XAI studies into LIDE.
Analysis of image and text representations Previous papers have investigated why language information is useful in vision and language tasks. Collell Talleda and Moens (2016) also found that the image and text representations complemented each other; for example, taxonomic attributes are captured well in the language. Li et al. (2020) observed that the attention heads of the multi-modal pre-trained models ground elements of language to image regions.
We tackled the few-shot image classification task through learning of novel concepts from language descriptions of images. We observed that machine-and user-generated descriptions improved the few-shot image classification performance. We also found that the generated captions explained the input image and were consistent with the prediction performance.
Our experiments also revealed four reasons why the text representation improved the performance: the inner-class distances of the multi-modal representations are smaller and the inter-class distances are larger than those of image representations; multi-modal representations are embedded in a space with a smaller latent dimension; multimodal representations are robust for noisy images; and certain types of knowledge are easily recovered from text representations.
Humans can learn concepts from language and explain them with language, but this is still difficult for machine learning models. This study sheds light on the importance of interactivity in explaining with language in machine learning.
Mohamed Afham, Salman Khan, Muhammad Haris Khan, Muzammal Naseer, and Fahad Shahbaz Khan. 2021. Rich semantics improve few-shot learning. arXiv preprint arXiv:2104.12709.
Laurent Amsaleg, Oussama Chelly, Teddy Furon, Stephane Girard, Michael E. Houle, Ken-ichi Kawarabayashi, and Michael Nett. 2015. Estimating local intrinsic dimensionality. In KDD, page 29-38.
Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language. In *NAACL-HLT*, pages 2166-2179.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In *Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization* @ACL, pages 65-72.
Jasmijn Bastings, Yonatan Belinkov, Emmanuel Dupoux, Mario Giulianielli, Dieuwke Hupkes, Yuval Pinter, and Hassan Sajjad, editors. 2021. Proceedings of the Fourth BlackboxNLP@EMNLP.
Steven Bird, Ewan Klein, and Edward Loper. 2009. *Natural Language Processing with Python*. O'Reilly Media, Inc.
Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, and Min Sun. 2017. Show, adapt and tell: Adversarial training of cross-domain image captioner. In ICCV, pages 521-530.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734.
Sahil Chopra, Michael Henry Tessler, and Noah D Goodman. 2019. The first crank of the cultural ratchet: Learning and transmitting concepts through language. In CogSci, pages 226-232.
Guillem Collel Talleda and Marie-Francine Moens. 2016. Is an image worth more than a thousand words? on the fine-grain semantic differences between visual and linguistic representations. In COLING, pages 2807-2817.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In ACL, pages 2126-2136.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. In NIPS, volume 26.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In *ICLR*.
Xiangteng He and Yuxin Peng. 2017. Fine-grained image classification via combining vision and language. In CVPR, pages 5994-6002.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In ECCV, pages 3-19.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. 2018. Grounding visual explanations. In ECCV, pages 264-279.
Dieuwke Hupkes and Willem Zuidema. 2018. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure (extended abstract). In *IJCAI*, pages 5617-5621.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Dong Hoon Lee and Sae-Young Chung. 2021. Unsupervised embedding adaptation via early-stage feature reconstruction for few-shot classification. In ICML, volume 139, pages 6098-6108.
Elizaveta Levina and Peter Bickel. 2005. Maximum likelihood estimation of intrinsic dimension. In NIPS, volume 17.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What does BERT with vision look at? In ACL, pages 5265-5275.
Qing Li, Jianlong Fu, Dongfei Yu, Tao Mei, and Jiebo Luo. 2018. Tell-and-answer: Towards explainable visual question answering using attributes and captions. In EMNLP, pages 1338-1346.
Weixin Liang, James Zou, and Zhou Yu. 2020. Alice: Active learning with contrastive natural language explanations. In EMNLP, pages 4380-4391.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out@ACL, pages 74-81.
TY Lin, M Maire, S Belongie, J Hays, P Perona, D Ramanan, P Dollar, and CL Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740-755.
Jesse Mu, Percy Liang, and Noah Goodman. 2020. Shaping visual representations with language for few-shot classification. In ACL, pages 4823-4830.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311-318.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In *Autodiff@NIPS*.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21:1-67.
Sachin Ravi and Hugo Larochelle. 2017. Optimization as a model for few-shot learning. In ICLR.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016. Learning deep representations of fine-grained visual descriptions. In CVPR, pages 49-58.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In NIPS, volume 30, pages 4077-4087.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In NIPS, pages 935-943.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, volume 29.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The caltech-ucsd birds-200-2011 dataset.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. 2019. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04623.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In ACL: System Demonstrations, pages 38-45.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O Pinheiro. 2019. Adaptive cross-modal few-shot learning. In NeurIPS, volume 32, pages 4847-4857.
We trained all the models on an NVIDIA Quadro RTX 8000 (48GB), and each experiment took almost one day. The hyperparameter settings are listed in Table 10. We used the Adam optimizer (Kingma and Ba, 2014), PyTorch (Paszke et al., 2017), and transformers (Wolf et al., 2020). Stop words were implemented with NLTK (Bird et al., 2009), and "bird" was added to the stop words. The training of the "No Image" ablated model was unstable due to the transformer architecture, so we used greedy decoding in the test phase to reduce the discrepancy between the train and test phases.
Pre-Training	Fine-Tuning
Batch size	128	100
# Epochs	100	1500
Learning rate for main model	1e-3	1e-3
Learning rate for text encoder	1e-3	1e-4
Learning rate for text decoder	1e-5	1e-5
λtext	—	10
λcntr	—	0.1
τ	—	0.05
Table 10: Hyperparameters.
For the experiments on recovering attributions (Section 4.8), we used sigmoid activation and determined the thresholds for each attribution. We sampled one caption for each image in the development and test splits. When an attribution was not described in a caption even though the label was 1, the example was ignored. We also removed attributions when the number of correct predictions was less than 20 or the number of positive examples was less than 20. Here, a positive example means one for which the label is 1 and the attribution is described in the caption. As a result, 68 attributions remained.