SlowGuess/NAACL_2022_OCR huggingface dataset:
Kosuke Nishida, Kyosuke Nishida, Shuichi Nishioka
NTT Human Informatics Laboratories, NTT Corporation
kosuke.nishida.ap@hco.ntt.co.jp
Humans can obtain the knowledge of novel visual concepts from language descriptions, and we thus use the few-shot image classification task to investigate whether a machine learning model can have this capability. Our proposed model, LIDE (Learning from Image and Description), has a text decoder to generate the descriptions and a text encoder to obtain the text representations of machine- or user-generated descriptions. We confirmed that LIDE with machine-generated descriptions outperformed baseline models. Moreover, the performance was improved further with high-quality user-generated descriptions. The generated descriptions can be viewed as the explanations of the model's predictions, and we observed that such explanations were consistent with prediction results. We also investigated why the language description improved the few-shot image classification performance by comparing the image representations and the text representations in the feature spaces.
Humans can efficiently learn about new concepts from language (Chopra et al., 2019). Hence, in this paper, we focus on the few-shot image classification problem to verify machine learning models' capability to understand new concepts from language. This problem is a kind of meta-learning problem in which a model first learns from the concepts of classes by training on a few instances and then learns unseen classes in the same way.
In our problem setting, the model can use language descriptions of images as additional information. This setting is similar to teaching a new concept to others by explaining it deductively from small amounts of data, unlike most machine learning models that learn inductively from large amounts of data.
For this setting, we propose a new model, called LIDE (Learning from Image and Description). As
Figure 1: Concept of the LIDE model.
shown in Figure 1, LIDE consists of an image encoder, an image classifier, a text decoder, and a text encoder.
LIDE has the advantage of providing explainability. It passes an image representation encoded by the image encoder to the text decoder, which then generates a language description of the image as an explanation of the model's prediction. The image classifier then classifies the image in accordance with the text representation, which is encoded by the text encoder, in addition to the image representation.
LIDE also provides high accuracy due to its explainability. It has been difficult to use machine-generated descriptions to improve image classification performance because of their low quality (Mu et al., 2020). Therefore, we design a training algorithm and a text encoding method to obtain robust text representations. In addition, LIDE includes a feature fusion module added to the image classifier to combine the information from both the image representation and the text representation.
Moreover, LIDE can take user-generated descriptions as input instead of machine-generated descriptions. We can use a description that contains textual features captured from an input image by human perception or a post-edited text in the text decoder output. The resulting high-quality descriptions provided by users can improve the image classification accuracy.
- We confirmed that LIDE with machine-generated descriptions outperformed previous models, and thus the explanations of the
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1421 - 1430
model's predictions were helpful to improve the classification accuracy.
- We observed that the performance improved further when gold captions were fed to LIDE as users' high-quality descriptions.
- We investigated whether the generated explanations were consistent with the image classification predictions, and we found a positive correlation between the quality of the generated captions and the classification accuracy.
- We thoroughly investigated why the text representations explaining the input image contributed to the image classification task, specifically in terms of the distributions of the representations in the feature space, the robustness of the representations for noisy images, and the knowledge of concepts that can be extracted from the representations.
2.1 Few-Shot Image Classification
$N$ -way  $K$ -shot classification involves three data splits,  $\mathcal{T}_{train}$ ,  $\mathcal{T}_{dev}$ , and  $\mathcal{T}_{test}$ , and each split consists of many classes and instances. The classes in the splits are disjoint. Each  $N$ -way  $K$ -shot classification task is a classification problem with  $N$  classes. Each task provides  $K$  training instances for each class, called support instances. A task entails the evaluation of  $M$  instances for each class, called query instances. The tasks, which consist of the classes and instances, are randomly sampled from the data splits.
This problem is a meta-learning problem. In the training phase, we use the episodic training (Ravi and Larochelle, 2017), where many mini-batches of size  $B$  consisting of  $B$  tasks, each of which consists of  $N$  classes and  $N(K + M)$  instances, are independently sampled from  $\mathcal{T}_{train}$ . In the test phase, the model learns new  $N$ -way  $K$ -shot classification tasks with unseen classes and instances sampled from  $\mathcal{T}_{dev}$  or  $\mathcal{T}_{test}$ . For each sampled task, a model learns from the supports, and we evaluate the classification performance of the queries.
A major approach for  $N$ -way  $K$ -shot classification is the prototypical network (ProtoNet) (Snell et al., 2017). In both the training and test phase, instead of updating the model parameters for each sampled task with few support instances, the prototypical network computes the class prototypes. Let
$h_k^c$  be the  $k$ -th support feature of class  $c$ . Here, the class prototype  $z^c$  is
$$
z ^ {c} = \frac {1}{K} \sum_ {k} W _ {p r o t o} h _ {k} ^ {c}.
$$
In the training phase, the model is trained with the cross-entropy loss of the queries from  $\mathcal{T}_{train}$ :
$$
L _ {c l a s s} = - \frac {1}{M} \sum_ {i} \sum_ {c} y _ {i} ^ {c} \log \frac {\exp [ s (z ^ {c} , h _ {i}) ]}{\sum_ {c ^ {\prime}} \exp [ s (z ^ {c ^ {\prime}} , h _ {i}) ]},
$$
where  $h_i$  is the  $i$ -th query feature,  $s(z_c, h_{mm,i}) = z_c^\top W_{proto} h_{mm,i}$  is a score function containing  $W_{proto}$  as a trainable parameter, and  $y_i^c \in \{0,1\}$  is the ground-truth label of the  $i$ -th query. In the test phase, the class prototypes are obtained from the supports, and the average score over the sampled tasks is reported.
2.2 Few-Shot Image Classification with Language Description
We focus on the few-shot image classification to verify the machine learning models' capability to learn new concepts from language descriptions in addition to the images. Previous studies have used the language description to improve the few-shot image classification. The classifiers in the studies are based on ProtoNet, which corresponds to the model with the image encoder and prediction module in Figure 2. Mu et al. (2020) proposed LSL by introducing a text decoder to ProtoNet to avoid overfitting by training the image encoder with a language generation loss. They observed that a text representation from a noisy machine-generated description was harmful for image classification. Accordingly, they viewed the text decoder as a regularizer and did not use a text encoder. RS-FSL (Afham et al., 2021) replaced the GRU (Cho et al., 2014) text decoder of LSL with a bi-directional transformer (Vaswani et al., 2017).
Andreas et al. (2018) were interested in describing the hidden states with natural language, but not in improving the image classification performance. They proposed L3 by adding both a text decoder and encoder to ProtoNet. They encoded an image into an image representation and decoded it into an explanation. The input of their image classifier was only the text representation encoded by the text encoder. L3 provided the explainability, but their model performance decreased.
All of the aforementioned papers assumed that the image encoder was not pre-trained in a super-
Figure 2: Overall structure of LIDE. The text decoder generates a caption on the basis of the image representation encoded by the image encoder. The text encoder obtains the text representation from the generated caption or a gold caption. The feature fusion module combines the image and text representation to generate the multi-modal representations. The prediction module outputs the classification probability.
vised or self-supervised fashion with external images. Our motivation is to clarify the benefit of the language description to learn novel visual concepts, and so we follow their setting.
We show the overall model structure of LIDE in Figure 2. The model components are explained in the following.
Image Encoder We can use any network as the image encoder, and we used a 4-layer CNN as in the previous studies. The output is the image feature  $h_{img}$ .
Text Decoder First, we map the image feature to the text feature space as follows:
$$
f _ {I 2 T} (h _ {i m g}) = \operatorname {L i n e a r} (\operatorname {L a y e r N o r m} (h _ {i m g})),
$$
where LayerNorm is layer normalization (Ba et al., 2016). Then, we pass the text feature vector to the text decoder as an encoder hidden state sequence of length 1. The text decoder autoregressively generates the  $j$ -th token  $t_j$ . The  $j$ -th token generation probability  $p_j$  is written as
$$
p _ {j} = \Pr (t _ {j}; f _ {I 2 T} (h _ {i m g}), t _ {0: j - 1}).
$$
We used a uni-directional three-layer transformer.
Text Encoder We use BERT (Devlin et al., 2019) for the text encoder, which outputs the last hidden states  $H_{BERT}$ . The text feature  $h_{text}$  is a weighted average pooling of  $H_{BERT}$ :
$$
h _ {t e x t} = \frac {1}{\sum p _ {j} w _ {j}} \sum p _ {j} w _ {j} h _ {B E R T, j},
$$
where the weight  $p_j$  is the token generation probability in the text decoder. If a caption is user-generated, then  $p_j$  is 1 for all tokens. The weight
$w_{j}$  is 1 if the  $j$ -th token is not a stop word; otherwise,  $w_{j} = 0$ .
The use of weighted average pooling with the text generation probability has two advantages. First, the text encoder can ignore the low-confidence tokens. Second, the image classification loss back-propagates to the text decoder through the weight  $p_j$ . Because the discrete operation of text generation breaks the computation graph, we cannot back-propagate the gradient of  $H_{BERT}$  to the decoder without the method.
Feature Fusion Module The feature fusion module combines the single-modal features  $h_{img}$  and  $h_{text}$  into a multi-modal feature  $h_{mm}$ . Let  $[;]$  be the vector concatenations,  $f_{T2I}$  be a mapping function from the text feature space to the image feature space, and  $g$  be a linear function to  $\mathbb{R}^2$ .  $f_{T2I}$  is a three-layer FFNN with ReLU activation. The feature fusion operation is the weighted sum of the two features:
$$
[ w _ {i m g}; w _ {l a n g} ] = \operatorname {s o f t m a x} (g ([ h _ {i m g}; h _ {t e x t} ])),
$$
$$
h _ {m m} = w _ {i m g} h _ {i m g} + w _ {t e x t} f _ {T 2 I} \left(h _ {t e x t}\right).
$$
Prediction Module We use ProtoNet for the prediction module and replace  $h$  with  $h_{mm}$ .
Loss Function For image classification, we compute two image classification losses:  $L_{\text{class,gold}}$  is computed from the image feature and the gold caption, while  $L_{\text{class,gen}}$  is computed from the image feature and the generated caption. For text generation, we compute  $L_{\text{text}}$  with teacher-forcing and cross-entropy loss.
To enrich the mapped text feature  $f_{T2I}(h_{text})$ , we use the contrastive loss (Sohn, 2016; Oord et al., 2018) between the gold and generated captions.
Let  $v_{gold}^{c}$  and  $v_{gen}^{c}$  be the averages of the mapped support text features in class  $c$  from the gold and generated captions, respectively. The similarity is  $\cos(v_{gold}^{c} \top v_{gen}^{c'})$ . Then, the contrastive loss  $L_{cntr}$  is
$$
\begin{array}{l} L _ {c n t r} = - \frac {1}{2 N} \sum_ {c} \log \frac {\exp [ \cos (v _ {g o l d} ^ {c} \top v _ {g e n} ^ {c}) / \tau ]}{\sum_ {c ^ {\prime}} \exp [ \cos (v _ {g o l d} ^ {c} \top v _ {g e n} ^ {c ^ {\prime}}) / \tau ]} \\ - \frac {1}{2 N} \sum_ {c ^ {\prime}} \log \frac {\exp [ \cos (v _ {g o l d} ^ {c ^ {\prime}} \top v _ {g e n} ^ {c ^ {\prime}}) / \tau ]}{\sum_ {c} \exp [ \cos (v _ {g o l d} ^ {c} \top v _ {g e n} ^ {c ^ {\prime}}) / \tau ]}, \\ \end{array}
$$
where  $\tau$  is a temperature parameter.
The total loss that we minimize in the training phase to optimize the whole model parameters in an end-to-end manner is
$$
L = L _ {\text {c l a s s , g o l d}} + L _ {\text {c l a s s , g e n}} + \lambda_ {\text {t e x t}} L _ {\text {t e x t}} + \lambda_ {\text {c n t r}} L _ {\text {c n t r}},
$$
where  $\lambda_{text}$  and  $\lambda_{cntr}$  are hyperparameters.
Pre-Training Following Afham et al. (2021); Wang et al. (2019), we pre-train the model with the training data for the downstream task. The pre-training consists of the standard image classification task, and we replace the prediction module with a linear classifier for all training classes. The loss function is  $L = L_{\text{class,gen}} + \lambda_{\text{text}} L_{\text{text}}$ .
Caption Generation In the training phase, we use a greedy algorithm and random sampling for computational reasons. In each step, we uniformly and randomly choose between the two algorithms. In random sampling, we restrict the candidate tokens to the top 20 tokens at each position.
In the test phase, we input the generated captions to the text encoder in the setting where the user-generated description is not available. The generation algorithm is beam search with a beam width of five and a length penalty of 0.5. Thus, the token sequence  $t_{1:l}$  is generated as
$$
\operatorname {a r g m a x} _ {l, t _ {1: l}} \frac {1}{l ^ {0 . 5}} \sum_ {j = 1} ^ {l} \log \operatorname * {P r} (t _ {j}; f _ {I 2 T} (h _ {i m g}), t _ {1: j - 1}),
$$
where  $l$  is the text length. The length penalty reduces the preference for words consisting of multiple subwords, such as 'point_y' ('pointy').
We used the Caltech-UCSD Birds (CUB) dataset (Wah et al., 2011) for evaluation. It contains 200
bird species (classes) and 40-60 images for each class. The classes are split into 100 training classes, 50 development classes, and 50 test classes. We used this dataset for the  $N = 5$ -way  $K = 1$ -shot classification problem. The number of query instances  $M$  per class was 15.
The CUB dataset has 10 captions for each image (Reed et al., 2016). For each step, we randomly sampled one caption from the 10 gold captions.
For the image classification, we report the average accuracy over 600 tasks, following the previous studies. To evaluate the generated caption quality, we used major metrics for image captioning,  $\mathrm{BLEU}_4$  (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE $_{\mathrm{L}}$  (Lin, 2004).
We pre-processed the images in the same way as Mu et al. (2020). The dimension of the image feature  $h_{img}$  was 1600. The text encoder and tokenizer were the pre-trained BERT-base-uncased model. The text encoder output dimension was 768. The configuration of the transformer layers in the text decoder was the same as that of the T5-base decoder model (Raffel et al., 2020), but we did not use the pre-trained parameters for the text decoder because they did not contribute to the performance. The parameter size of  $W_{proto}$  was  $1600 \times 1600$ . The other hyperparameters and optimization details are given in Appendix A.
4.4 Compared Models
ProroNet (Snell et al., 2017) was the baseline, with only the image encoder. As for the other compared models, L3 (Andreas et al., 2018) used a 200-dimensional GRU text encoder and decoder but did not use image representation for classification. LSL (Mu et al., 2020) used a 200-dimensional GRU text decoder for regularization. RS-FSL (Afham et al., 2021) used a 2-layer, 768-dimensional, bidirectional transformer as the text decoder. All models used a 4-layer CNN as the image encoder, along with the ProtoNet-based prediction module.
4.5 Ablated Models
To evaluate the models that use a single-modal representation for image classification, we introduced the models "No Text," "No Image," and "No Text Encoder" by removing the feature fusion module
Model	Text Dec.	Text Enc.	Modal	Baseline
No Text	Image	ProtoNet
No Image	✓	✓	Text	L3
No Text Enc.	✓	Image	LSL, RS-FSL
Table 1: Setting of ablated single-modal models.
Model	Accuracy	Img. Enc.	Text Enc.	Fusion	Text Dec.
ProtoNet	57.97 ±0.96	✓
L3	53.96 ±1.06	✓	✓
LSL	61.24 ±0.96	✓	✓
RS-FSL	65.66 ±0.90	✓	✓
LIDE	67.53 ±0.91	✓	✓	✓	✓
from LIDE. Each model corresponds to our implementation of the baseline models, as shown in Table 1.
4.6 Evaluation Results for LIDE as Few-Shot Image Classification Model
4.6.1 Main Results
Performance with Machine-generated Description Table 2 summarizes the results. LIDE outperformed ProtoNet, LSL, and RS-FSL, which use an image classifier using image representations only. LIDE improved the image classification performance with the mechanism of encoding text and combining multi-modal representations, while L3, which uses an image classifier with text representations only, harmed their performance.
Performance with User's Description The advantage of the text encoder is that it enables textual input by users. Specifically, a user can use language as an explanation from humans to machines by feeding the textual features captured from the input image to the model. In addition, if a user objects to the model's explanations, the user can edit them to correct the model's misunderstanding.
To evaluate LIDE in this setting, we viewed the gold captions as user descriptions. As the CUB dataset has 10 gold captions per image, we selected one gold caption and fed it to the model to maximize the similarity to a generated caption and thus simulate a user editing the generated caption. The similarity was defined as the bi-gram precision of the gold caption with respect to the generated caption.
Table 3 shows that the performance was improved significantly when a high-quality gold caption was given. However, the performance declined
Table 2: Performance of the compared models.
Random Description	58.89 ±0.93
Generated Description	67.53 ±0.91
Gold Description	73.08 ±0.88
Table 3: Performance with each kind of description.
when a wrong caption was given, which was a randomly sampled caption from all captions in  $\mathcal{T}_{test}$ . We conclude that the model's output depends on the quality of the description.
4.6.2 Ablation Study
Evaluation on modalities The first set of rows in Table 4 lists ablation study results for the different modalities that LIDE uses. We confirmed that LIDE using all the modalities outperformed the compared models using part of the modalities. The image and text representations complemented each other, as will be discussed later.
Evaluation on introduced techniques The second set of rows in Table 4 lists the techniques that were introduced in LIDE: image classification loss with generated captions, contrastive learning, weighted average pooling, and random sampling during caption generation. We confirmed that all of these techniques contributed to the performance of LIDE.
We assume that the loss with the generated captions decreases the discrepancy between the train and test phases. The contrastive loss enriches the text representation and makes the representations of the generated text and the gold caption close. The weighted average pooling reduces the effect of noisy generated text. The random sampling in the training phase contributes to the generation of diverse captions.
Evaluation on pre-training of text encoder We also found that the multi-modal feature was useful even when the text encoder was trained from scratch. The text representation could assist in the image representation without BERT pre-training, because the captions in the CUB dataset are restricted to the descriptions of birds, and the training data thus covers the space of the captions well.
4.7 Evaluation Results for LIDE as Interpretable Machine Learning Model
Evaluation on quality of generated captions First, generated captions are insufficient as explanations if they are not accurate. The upper bound was the CNN-LSTM model pre-trained with the
LIDE	67.53 ±0.91
No Text (ProtoNet)	62.22 ±0.92
No Image (L3)	49.60 ±1.03
No Text Encoder (LSL, RS-FSL)	63.10 ±0.90
No Lclass,gen	61.96 ±0.90
No Contrastive Loss	64.60 ±0.88
No Weighted Average Pooling	66.16 ±0.93
No Random Sampling	66.40 ±0.93
None of the Above	59.68 ±0.98
No BERT Pre-Training	66.42 ±0.94
Table 4: Ablation study results.
BLEU4	METEOR	ROUGE L
UB: Caption	59.0	36.1	69.7
No Text Enc.: Caption	50.0	34.6	67.2
Correlation	0.114	0.201	0.217
LIDE: Caption	48.1	34.1	66.4
Correlation	0.309†	0.468*	0.436**
MSCOCO (Lin et al., 2014) dataset from Chen et al. (2017). Although we had no training data without the CUB dataset, Table 5 shows that the differences between LIDE and the upper bound were only 2.0 points for METEOR and 3.3 points for ROUGE.
Evaluation on consistency between generated captions and image classification When the generated captions are correct (respectively, incorrect), the classification results should also be correct (incorrect) in terms of the consistency between explanations and classification results.
Accordingly, we calculated the Spearman rank-order correlation coefficient between the captioning scores and the classification scores. We divided 2,953 test-split images into 30 bins in accordance with the ascending order of each captioning score, and we computed the average image classification accuracy in each bin.
Table 5 lists the results. We confirmed a positive correlation between the quality of the generated captions and the prediction accuracy of LIDE. However, LIDE without the text encoder, which corresponds to LSL (Mu et al., 2020) and RS-FSL (Afham et al., 2021), showed a low correlation. This result demonstrated the importance of the text encoder and the feature fusion modules introduced in LIDE.
Qualitative analysis on generated captions Figure 3 shows examples of the generated captions. We found that the captions captured the birds' char
Table 5: Captioning scores and correlations to the prediction scores.†:  $p < {0.1}, *  : p < {0.05}, *  *  : p < {0.01}$
Ours	this bird is yellow with black and has a very short beak.
Gold	the bird has a small black bill and a yellow crown
Ours	this bird has wings that are brown and has a white belly.
Gold	a small bird with a red face black crown white cheeks brown back and black and yellow wings
Figure 3: Examples of generated captions and gold captions.
acteristics. However, the structures of the captions were uniform, and they could not describe a birds' most distinctive element, such as the red face in the second example. We believe that overfitting to the 5-class classification problems with the small dataset caused this problem.
We restricted the image encoder to a 4-layer CNN for fair comparison to the existing models, and we did not use external training data to validate the ability to learn novel classes from language descriptions. Removal of the limitations would improve the performance of LIDE as an explainable machine learning model.
4.8 Discussion
In this section, we clarify four reasons why text representations are useful in the few-shot image classification task. Specifically, we compared the multi-modal feature space of LIDE with the gold captions to the image feature space of the "No Text" and "No Text Encoder" models.
How are classes distributed in each modal feature space? First, we calculated the inner- and inter-class distances in each feature space. Table 6 lists the results. The inner-class distances in the multi-modal feature space of LIDE were smaller than those in the image feature space; the inter-class distances of LIDE were larger. As a result, the clusters were distributed well in the multi-modal feature space.