SlowGuess/ACL_2020_OCR huggingface dataset:
PDCNN PDCNN (Lei Ba et al., 2015) is the most relevant prior method as it use the natural language descriptions of class labels to generate classification layers capable of distinguishing between objects described. Note that PDCNN is distinct from our work in that it dynamically generates fully connected layers and/or additional convolutional layers to be appended to a pre-trained deep neural network (VGG-19) whilst ours generates parameter adaptations to be combined directly with all existing layers within deep neural network models, effectively "fine-tuning" the pre-trained model. We compare with two variants of PDCNN, with  $\mathrm{PDCNN_{FC}}$  generating a fully connected layer only for classification and  $\mathrm{PDCNN_{FC + Conv}}$  producing an
additional convolutional layer to help with classification. To make our works comparable, we replaced the TF-IDF feature extractor in PDCNN with a BERT-based document embedding (specifically, a BERT token embedding followed by max-pooling) and changed the pre-trained model from VGG-19 to ResNet-18.
MEGAZSL Due to the scarcity of prior work leveraging natural language descriptions for zero-shot classifications in a metadata-efficient way, we also adapted less metadata-efficient methods to function with stricter metadata-efficiency requirements. Specifically, ZSLPP (Elhoseiny et al., 2017), GAZSL (Zhu et al., 2017) and CorrectionNetwork (Hu et al., 2019) all utilize natural language object class descriptions to produce classifiers capable of distinguishing between images belonging to unseen categories during training. However, all of them require significantly more metadata: specifically, parts annotations of each sample image are used to provide extra supervision of the training procedure. Among these methods, GAZSL (Zhu et al., 2017) stands out as the most cited work; therefore we adapted the code released by its authors to learn from only natural language metadata, without using parts annotations; to distinguish our modified version from the original, we will refer to our modified version as MEGAZSL (Metadata-Efficient GAZSL). To make our works comparable, we also updated its language representation from TF-IDF to BERT-based ones and used ResNet-18 as the image feature embedding module. It is worth noting the CorrectionNet(Hu et al., 2019) is orthogonal to our work as it is designed to improve any existing zero-shot classification task modules, and in its original setup, GAZSL (Zhu et al., 2017) was used as the main task module, which we do include in our experimental comparison.
For all experiments, hyper-parameters are tuned
6999
Method	CUB-50	AWA2-10	NAB-40	FS-10
SS	PS	SS	PS	SS	PS	SS	PS
PDCNNFC(Lei Ba et al., 2015)	6.1	6.1	12.8	18.4	6.7	7.4	13.1	11.9
PDCNNFC+CONV(Lei Ba et al., 2015)	7.5	6.5	22.6	17.2	8.9	5.6	7.7	13.6
MEGAZSL(Zhu et al., 2017)	2.9	1.8	14.0	10.4	2.8	3.7	10.3	12.3
N3 (Proposed)	17.6	9.5	34.0	37.5	14.1	20.7	16.4	17.6
Table 2: Zero-Shot Classification Accuracy (Defined by Eq. 7) On Various Datasets/Meta-Splits Combinations; number of classification labels used in the test set are recorded next to the name of each dataset.
with the same exact algorithms (random search) and for the same number of runs (10).
In this section, we compare  $\mathbf{N}^3$  method with prior work on 4 zero-shot image classification benchmark datasets. Furthermore, we provide a simple approach to help interpret what part of the language input is being taken as evidence by the synthesized visual classifier to make predictions. We then show through ablation studies the importance of adaptation of all parameters in the pre-trained network, the necessity of pre-trained networks, and the effect of language representation choices on the success of  $\mathbf{N}^3$ .
5.1 Benchmark Evaluations
We report performance in per-class averaged accuracy, as shown in Table 2. In these experiments, we standardized the language representations, dataset-splits, and other factors orthogonal to our model design to ensure fairness of comparison. We also include experimental results comparing  $\mathbf{N}^3$  to models in their respective original settings in the Appendix. From Table 2, we can clearly observe that  $\mathbf{N}^3$  outperforms all competing methods by a significant margin on all 8 dataset/meta-split combinations. Noticing the large performance gap between MEGAZSL and the original GAZSL, we performed additional investigations to pinpoint the cause: we reproduced one of their experiments (on CUB dataset with SCE meta-split), and replaced their TF-IDF module with BERT document embedding module. The modified module performed noticeably better (from  $10.3\%$  to  $11.3\%$ ); however, when we replaced its image feature embedding module, which is trained with additional parts annotations of bird images, the performance dropped significantly (from  $11.3\%$  to  $3.3\%$ ), confirming our conjecture that such methods cannot be easily
adapted to work without extra supervision in the form of additional data annotation.
5.2 Interpreting Model Predictions
In this section, we explore how  $\mathbf{N}^3$  model architecture can help interpret the predictions made by the adapted model. Specifically, the design of  $\mathbf{N}^3$  is unique in that it examines all object class descriptions in order to adapt neural network parameters. This means that  $\mathbf{N}^3$  can adapt neural networks to seek both positive and negative evidence for an image to be classified. Naturally, we want to understand how the model is using these object class descriptions. In Figure. 3, we present our findings by visualizing the magnitude of  $\frac{\partial E}{\partial D_{ij}}$  where  $E$  is the loss value computed on a test example (in our experiment, this example is correctly predicted to be an Acadian Flycatcher) and  $D_{ij}$  is the BERT representation of the  $j$ -th word in the  $i$ -th object class description. We present two patterns in the data that are indicative of the model behavior:
Top Positive Evidence: top positive evidence is identified as tokens in the description of the ground truth label with large gradients. Intuitively, these tokens are top supporting evidence that encourages the prediction of the correct label. We locate them by ranking all tokens in the description of the ground truth label by their magnitude of gradients and take the top few. Top Negative Evidence: top negative evidence is identified as tokens with large gradients descriptions of the negative labels that also has a low predicted probability. Intuitively, these tokens are the keywords deemed as important evidence when rejecting to predict a label. We locate such evidence by first ranking the descriptions by the ratio between their largest token gradients and the softmax probability of the corresponding label. Then, within the set of class descriptions with the largest aforementioned ratios, we locate the tokens with the largest gradients.
Top Positive Evidence
(The image is a Acadian Flycatcher because ...)
the breast is washed with olive
adults have olive upperparts , darker on the wingsand tail
they also have a call similar to that of the northern flicker
Top Negative Evidence
(The image is not a [tern, oriole, warbler] because ...)
it is a small term , 22 - 24 cm (8.7-9.4 in) long
(Least Tern)
immature males are yellow - orange on the breast
(Baltimore Oriole)
females feature a similar coloration pattern , but the
black is replaced with light grey
(Golden Winged Warbler)
Acadian Flycatcher (Ground Truth)
Northern Flicker (Related Label)
Least Tern (Negative Label)
Baltimore Oriole (Negative Label)
Golden Winged Warbler (Negative Label)
Examples for both types of keyword evidence are presented in 3 with their context. Several observations can be drawn about how  $\mathbf{N}^3$  uses textual descriptions to make classification decisions. Firstly, we can observe that distinguishing features described with keywords such as "olive", "darker", "yellow" and "grey" are used to support both positive and negative identifications, which support our conjecture that  $\mathbf{N}^3$  examines descriptions from all object class descriptions to make classification decisions, sometimes employing the process of elimination. Secondly, we can observe that some label (or label word piece) like "flicker" and "tern" is used as evidence to support or refute a classification decision, which seems to suggest that some task-specific knowledge is learned and employed in the language representations.
5.3 Ablation Study: Task-specificness of Synthesized Models
To account for the improved accuracy of models synthesized with whole-model parameter adaptations, we performed additional analysis to understand the features models are utilizing when making predictions. Two variants of  $\mathbf{N}^3$  metamodels sharing the same set of hyper-parameters are trained on CUB-50 with the standard split. One is allowed to adapt parameters of every layer whilst the other is ablated only to generate the classification layer. Saliency map (as described in (Si
Figure 3: When classifying an image of Acadian Flycatcher, we visualized the magnitude of gradients of description tokens w.r.t the loss. Higher magnitude is colored red and lower magnitude blue. Intuitively, words with higher magnitude of gradient represent textual features that are more important for the classification decision.
Figure 4: Saliency map showing magnitude of gradients of loss w.r.t. every pixel in the input image.
Left: original image. Middle: saliency map from a model with every layer adapted by  $\mathbf{N}^3$ . Right: saliency map from a model where  $\mathbf{N}^3$  is restricted to act on only its fc layer. Purple indicates small values while blue/green indicates large values.
monyan et al., 2014)) visualizing the importance of each pixel w.r.t. predictions is plotted as a heatmap in Figure. 4. Clearly, fully adapted models show a greater concentration on task-specific regions.
5.4 Ablation Study: Importance of pre-trained Model
To adapt generic pre-trained model parameters to task-specific ones,  $\mathbf{N}^3$  mixes the pre-trained model parameters with a set of generated model parameter adaptations, using a trainable mixing ratio as illustrated in Equation 6. It is natural to question whether the pre-trained parameters are necessary. In other words, can  $\mathbf{N}^3$  generate parameters for a task-specific model from scratch and achieve reasonable classification accuracy? To study the importance of pre-trained model parameters, we ex
7001
γ	initial μ	Acc@1
0.999	0.001	16.2 %
0.99	0.01	18.5 %
0.9	0.1	16.2 %
0.5	0.5	6.2 %
0.1	0.9	2.1 %
0.01	0.99	2.3 %
Experiment with a modified version of Equation 6:
$$
\Theta^ {\prime} = \gamma \cdot \Theta + \mu \cdot \Delta \Theta
$$
Where  $\gamma$  is a fixed scalar weight given to the pre-trained model,  $\mu$  is still the trainable scaling factor. The choice of  $\gamma$  and the initial value of  $\mu$  affects the extent to which the pre-trained model contribute to the parameter-adapted one.
In our main experiments, we have fixed  $\gamma$  to be 1 and initialized  $\mu$  to be  $1^{-3}$ , such choice of value initialization proves to work well, yet it remains unclear to what extent the superior performance of  $\mathbf{N}^3$  depends on these two hyperparameters. To answer this question, we decide to vary  $\gamma$  and  $\mu$ ; In order to keep the magnitude of parameters relatively stable, we always set the initial value of  $\mu$  to be  $1 - \gamma$  across different settings here. We trained  $\mathbf{N}^3$  to produce a 50-way classification model with varying  $\gamma$ , and the resultant zero-shot classification performance are shown in Table 3. Such an ablation experiment demonstrates that setting a small initial  $\mu$  is crucial for performance, yet it is not the smaller the better, re-affirming the crucial role of parameter adaptation in achieving good performance.
5.5 Ablation Study: Impact of Different Word Embeddings
To understand the importance of pre-trained BERT module, and whether  $\mathbf{N}^3$  can be generalized to be used with pre-trained word embedding modules other than BERT, we compared the classification accuracy of models adapted by variants of  $\mathbf{N}^3$  using different word embeddings. Concretely, we experimented with BERT (Devlin et al., 2018), ELMo (Peters et al., 2018) and GloVe (Pennington et al., 2014) embeddings. Results are shown in Table 4. While BERT prevails as expected, ELMo seems to under-perform GloVe. We hypothesize
Table 3: Comparison of different  $\gamma$  and initial  $\mu$  values. The pre-trained model is ignored when  $\gamma = 0$ .
Embedding Method	Acc @ 1
ELMo (paragraph)	4.1 %
ELMo (sentence)	5.0 %
GloVe	8.9 %
BERT	17.6 %
Table 4: Comparison of different embedding methods for  ${\mathrm{N}}^{3}$  in zero-shot classification task.
that ELMo might have been impacted by unexpectedly long sequence lengths (here each description is a paragraph up to 512 tokens), since LSTM-based models are known to be worse at capturing long-range dependencies. To further investigate, we trained a variant of  $\mathbf{N}^3$ , where we limit the context of ELMo to each sentence instead of the entire paragraph. As expected, the performance increases noticeably, but the resulting performance is still not on par with other methods. In contrast, GloVe embeddings worked better since such static embeddings are not affected by the long context windows.
In this paper, we have demonstrated that small amount of unstructured natural language descriptions for object classes can provide enough information to fine-tune an entire pretrained neural network to perform classification task on class labels unseen during training. We have achieved state-of-the-art performance for natural language guided zero-shot image classification tasks on 4 public datasets with practical metadata requirements. In addition, we presented in-depth analysis and extensive ablation studies on various aspects of the model functioning mechanism and architecture design, showing the necessity of our design contribution in achieving good results.
This material is based upon work partially supported by the National Science Foundation (Awards #1750439 #1722822) and National Institutes of Health. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation or National Institutes of Health, and no official endorsement should be inferred.
7002
2019. albanie/convnet-burden :memory consumption and flop count estimates for convnets. https://github.com/albanie/convnet-burden.
2019. torchvision.models. https://torch.org/docs/stable/torchvision/models.html.
Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. 2016. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):1425-1438.
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. 2015. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2927-2936.
Maruan Al-Shedivat, Avinava Dubey, and Eric P. Xing. 2017. Contextual explanation networks. CoRR, abs/1705.10301.
Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. 2016. Synthesized classifiers for zero-shot learning. CoRR, abs/1603.00550.
Sasank Chilamkurthy. Transfer learning for computer vision tutorial.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Kun Duan, Devi Parikh, David Crandall, and Kristen Grauman. 2012. Discovering localized attributes for fine-grained recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3474-3481. IEEE.
Mohamed Elhoseiny, Ahmed M. Elgammal, and Babak Saleh. 2016. Write a classifier: Predicting visual classifiers from unstructured text descriptions. CoRR, abs/1601.00025.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. 2013. Write a classifier: Zero-shot learning using purely textual descriptions. In Proceedings of the IEEE International Conference on Computer Vision, pages 2584-2591.
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed M. Elgammal. 2017. Link the head to the "beak": Zero shot learning from noisy text description at part precision. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6288-6297.
Jonathan Goldsmith. Wikipedia
David Ha, Andrew M. Dai, and Quoc V. Le. 2016. Hypernetworks. CoRR, abs/1609.09106.
R. Lily Hu, Caiming Xiong, and Richard Socher. 2019. Correction networks: Meta-learning for zero-shot learning.
P. Kankuekul, A. Kawewong, S. Tangruamsub, and O. Hasegawa. 2012. Online incremental attribute-based zero-shot learning. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3657-3664.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. 2011. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. 2017. Semantic autoencoder for zero-shot learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
B. Kulis, K. Saenko, and T. Darrell. 2011. What you saw is not what you get: Domain adaptation using asymmetric kernel transforms. In CVPR 2011, pages 1785-1792.
C. H. Lampert, H. Nickisch, and S. Harmeling. 2014. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453-465.
Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. 2015. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the IEEE International Conference on Computer Vision, pages 4247-4255.
M-E. Nilsback and A. Zisserman. 2006. A visual vocabulary for flower classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 1447-1454.
Devi Parikh and Kristen Grauman. 2011. Relative attributes. In Proceedings of the 2011 International Conference on Computer Vision, ICCV '11, pages 503-510, Washington, DC, USA. IEEE Computer Society.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. CoRR, abs/1802.05365.
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. 2018. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 425-435. Association for Computational Linguistics.
7003
Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. 2016. Less is more: zero-shot learning from online textual documents with noise suppression. CoRR, abs/1604.01146.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016. Learning deep representations of fine-grained visual descriptions. In *The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.
Bernardino Romera-Paredes and Philip H. S. Torr. 2015. An embarrassingly simple approach to zero-shot learning. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pages 2152-2161. JMLR.org.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 6000-6010.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. 2011. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. 2010. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. 2017. Zero-shot learning - the good, the bad and the ugly. CoRR, abs/1703.04394.
Z. Zhang and V. Saligrama. 2016. Zero-shot learning via joint latent similarity embedding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6034-6042.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, and Ahmed M. Elgammal. 2017. Imagine it for me: Generative adversarial approach for zero-shot learning from noisy texts. CoRR, abs/1712.01381.
A Impact of Different Base Model Architectures
In this section, we provide an additional ablation experiment, we will be using the Standard Split on CUB dataset. With the superior results of  $\mathbf{N}^3$  obtained on the above datasets using ResNet-18 as the base architecture, it is reasonable to wonder whether the power of  $\mathbf{N}^3$  can extend to other base architectures and whether  $\mathbf{N}^3$  can work effectively on other base model architectures. To answer this question, we trained  $\mathbf{N}^3$  with 2 additional base model architecture: GoogleNet and SqueezeNet. We compared them in terms of zero-shot classification accuracy. Results are tabulated in Table 5.
Based on the results shown in Table. 5, we were surprised to find out that SqueezeNet performs as well as ResNet as  $\mathrm{N}^3$  's base model and GoogleNet can even out-perform ResNet-18 as a better base model. One explanation for the superior performance of GoogleNet is that parameters of batch normalizations are not known to correlate with task semantics and therefore can hardly be fine-tuned. GoogleNet, on the other hand, does not make use of batch normalization, thus avoiding the potential performance degradations caused by lack of fine-tuning on Batch Normalization layers.
B Additional Comparison with Prior Work
In our main experimental evaluation section, we performed extensive experiments comparing our methods with prior arts using newly established rigorous zero-shot learning evaluation guidelines (Xian et al., 2017). To adhere to the stricter guidelines, and to compare fairly with prior arts, we have to reproduce the results of prior methods ourselves; one may wonder how do we compared with each of these prior methods in their own experimental setup, and more importantly, how do we compare with their originally published performance numbers? In this section we seek to assure readers that our method continues to perform noticeably better than prior methods in their original experimental setup. To begin with, we compare the zero-shot classification performance of  $\mathbf{N}^3$  with prior work which also generates neural network parameters directly (Lei Ba et al., 2015).
To make our results comparable, we adopted the meta-training/testing splits from (Lei Ba et al., 2015), where in CUB-2010/CUB-2011, the 200
7004
Model Arch	Acc @ 1	ImageNet Test Acc. (tor, 2019)	Param Size (MB) (con, 2019)
GoogleNet	21.3 %	69.78 %	51
ResNet18	17.6 %	69.76 %	45
SqueezeNetV1.1	17.7 %	58.19 %	5
Table 5:  ${\mathrm{N}}^{3}$  Zero-Shot Classification Accuracy on CUB2011 with Different Base Model Architectures