SlowGuess/ACL_2020_OCR huggingface dataset:
Albert Gordo and Diane Larlus. 2017. Beyond instance-level image retrieval: Leveraging captions to learn a global visual representation for semantic retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6589-6598.
Prasoon Goyal, Scott Niekum, and Raymond J. Mooney. 2019. Using natural language for reward shaping in reinforcement learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 2385-2391.
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1884-1895.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778.
Xiangteng He and Yuxin Peng. 2017. Fine-grained image classification via combining vision and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5994-6002.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. 2018. Grounding visual explanations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 264-279.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Alexander Kuhnle and Ann Copestake. 2017. Shapeworld-a new test methodology for multimodal language understanding. arXiv preprint arXiv:1704.04517.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! Leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4932-4942, Florence, Italy.
4828
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 49-58.
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR).
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 4077-4087.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems (NeurIPS), pages 935-943.
Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2017. Joint concept learning and semantic parsing from natural language explanations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1527-1536.
Pavel Tokmakov, Yu-Xiong Wang, and Martial Hebert. 2019. Learning compositional representations for few-shot recognition. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 6372-6381.
Michael Tomasello. 1999. The Cultural Origins of Human Cognition. Harvard University Press, Cambridge, MA.
Vladimir Vapnik and Akshay Vashist. 2009. A new learning paradigm: Learning using privileged information. Neural Networks, 22(5-6):544-557.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The Caltech-UCSD Birds-200-2011 dataset.
Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O Pinheiro. 2019. Adaptive cross-modal few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 4848-4858.
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using annotator rationales to improve machine learning for text categorization. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference (NAACL-HLT), pages 260-267.
A Model and training details
A.1 ShapeWorld
$f_{\theta}$ . Like Andreas et al. (2018),  $f_{\theta}$  starts with features extracted from the last convolutional layer of a fixed ImageNet-pretrained VGG-19 network (Simonyan and Zisserman, 2015). These 4608-d embeddings are then fed into two fully connected layers  $\in \mathbb{R}^{4608\times 512}$ ,  $\mathbb{R}^{512\times 512}$  with one ReLU nonlinearity in between.
LSL. For LSL, the 512-d embedding from  $f_{\theta}$  directly initializes the 512-d hidden state of the GRU  $g_{\phi}$ . We use 300-d word embeddings initialized randomly. Initializing with GloVe (Pennington et al., 2014) made no significant difference.
L3.  $f_{\theta}$  and  $g_{\phi}$  are the same as in LSL and Meta.  $h_{\eta}$  is a unidirectional 1-layer GRU with hidden size 512 sharing the same word embeddings as  $g_{\phi}$ . The output of the last hidden state is taken as the embedding of the description  $\mathbf{w}^{(t)}$ . Like Andreas et al. (2018), a total of 10 descriptions per task are sampled at test time.
Training. We train for 50 epochs, each epoch consisting of 100 batches with 100 tasks in each batch, with the Adam optimizer (Kingma and Ba, 2015) and a learning rate of 0.001. We select the model with highest epoch validation accuracy during training. This differs slightly from Andreas et al. (2018), who use different numbers of epochs per model and did not specify how they were chosen; otherwise, the training and evaluation process is the same.
Data. We recreated the ShapeWorld dataset using the same code as Andreas et al. (2018), except generating 4x as many test tasks (4000 vs 1000) for more stable confidence intervals.
Note that results for both L3 and the baseline model (Meta) are 3-4 points lower than the scores reported in Andreas et al. (2018) (because performance is lower for all models, we are not being unfair to L3). This is likely due to differences in model initialization due to our PyTorch implementation and/or recreation of the dataset with more test tasks.
A.2 Birds
$f_{\theta}$ . The 4-layer convolutional backbone  $f_{\theta}$  is the same as the one used in much of the few-shot literature (Chen et al., 2019; Snell et al., 2017). The
model has 4 convolutional blocks, each consisting of a 64-filter  $3 \times 3$  convolution, batch normalization, ReLU nonlinearity, and  $2 \times 2$  max-pooling layer. With an input image size of  $84 \times 84$  this results in 1600-d image embeddings. Finally, the bilinear matrix  $\mathbf{W}$  used in the similarity function has dimension  $1600 \times 1600$ .
LSL. The resulting 1600-d image embeddings are fed into a single linear layer  $\in \mathbb{R}^{1600\times 200}$  which initializes the 200-d hidden state of the GRU. We initialize embeddings with GloVe. We did not observe significant gains from increasing the size of the decoder  $g_{\phi}$ .
L3.  $f_{\theta}$  and  $g_{\phi}$  are the same.  $h_\eta$  is a unidirectional GRU with hidden size 200 sharing the same embeddings as  $g_{\phi}$ . The last hidden state is taken as the concept  $\mathbf{c}_n$ . 10 descriptions per class are sampled at test time. We did not observe significant gains from increasing the size of the decoder  $g_{\phi}$  or encoder  $h_\eta$ , nor increasing the number of descriptions sampled per class at test.
Training. For ease of comparison to the few-shot literature we use the same training and evaluation process as Chen et al. (2019). Models are trained for 60000 episodes, each episode consisting of one randomly sampled task with 16 query images per class. Like Chen et al. (2019), they are evaluated on 600 episodes. We use Adam with a learning rate of 0.001 and select the model with the highest validation accuracy after training.
Data. Like Chen et al. (2019), we use standard data preprocessing and training augmentation: ImageNet mean pixel normalization, random cropping, horizontal flipping, and color jittering.