SlowGuess/NAACL_2022_OCR huggingface dataset:
Comparison with other models. First, we compare against supervised SVM regression models, which have shown strong performance on the Brysbaert data in prior work (Charbonnier and Wartena, 2019). Following their work, we use two feature configurations: (1) POS tags + suffixes, (2) POS tags + suffixes + pre-trained FastText embeddings (Joulin et al., 2017). We train the SVMs on the full Brysbaert data.
Second, we compare with a minimally supervised text-only model. As in Sec 5.1, we create word vector representations from co-occurrence counts. Next, following prior work (Turney et al., 2011), we select concrete (abstract) representative words by taking the 20 words with the highest (lowest) concreteness value in the Brysbaert data that occur more than 10 times in the MSCOCO training set. We predict a word's concreteness by computing its average cosine similarity to the concrete representative words minus the average of its cosine similarity to the abstract representative words.
Results. Figure 2 presents the results in terms of Pearson correlation when evaluated on words of varying minimum frequency in MSCOCO. When considering frequent tokens only, our model predicts word concreteness with an accuracy higher than the SVM with POS and suffix features, although additional embedding features improve SVM performance further. Note that the supervised baseline was trained on the full data set, and hence evaluated on a subset of its training set. Our multimodal model performs better than its text-only variant for tokens that occur at least 100 times, even though the text-only model has received some supervision (by selecting the representative words).
See appendix for the full list of clusters of all clustering algorithms.
For example, in the concreteness dataset built by Bryssbaert et al. (2013), in which human annotators rated the concreteness of words on a scale of 1 to 5,  $85.6\%$  of the words with an average concreteness rating above 4 are nouns.
Figure 2: Pearson correlation between predicted word concreteness and gold standard human ratings, evaluated over test sets with increasing minimum frequency in the MSCOCO data. Results are averaged across 5 random initializations (std was consistently  $< 0.03$ ).
5.3 Visual Multi-Label Classification
In addition to linguistic knowledge, infants acquire visual semantic knowledge with little explicit supervision, i.e., they learn to segment and classify objects. To test whether our model also acquires such knowledge we evaluated it on the multi-label classification task: For each image in the MSCOCO test set, predict the classes of objects in the image.
In a zero-shot setting, we mapped the induced clusters to predicted lists of MSCOCO classes as follows. We first provided the name of each class to our model as text input and retrieved the assigned cluster, thus obtaining a (one-to-many) cluster-to-classes mapping. Now, for each test image, we used the visual encoder to predict the assigned cluster(s). The predicted set of MSCOCO classes is the union of the lists of classes to which the predicted clusters are mapped.
Comparison with CLIP. We compare our results against CLIP. To ensure comparability with our model we use CLIP with ResNet50. We use CLIP as a point of comparison to provide perspective on the capabilities of our model despite differences in modeling and assumptions. However, we note two caveats regarding this comparison. First, CLIP was trained on a much larger training set and has more parameters than our model (see Table 1). Second, CLIP has only been used for single- (not multi-) label classification, by inferring encodings of both input images and prompts representing the ground-truth classes (e.g., "a photo of a bus" for the ground truth class bus) and assigning the image to the class
Model	Precision	Recall	F-Score
Ours	0.43 ± 0.04	0.21 ± 0.02	0.28 ± 0.01
CLIP	0.52	0.39	0.45
Table 5: Visual multi-label classification results on the MSCOCO data for our model and CLIP in terms of precision, recall and F1 score. We report mean (std) over 5 random initializations for our model. CLIP experiments are deterministic (we use the pretrained model directly, unlike Sec 5.1 where we used KMeans on top of CLIP).
with highest cosine similarity to its encoding. We adapt CLIP to a multi-label setting as follows: Instead of assigning the image to the class with the highest cosine similarity, we take into account the cosine similarity with all classes for each image. We consider a class as predicted if its cosine similarity exceeds a threshold, tuned on the MSCOCO training split.
Results. Table 5 presents the results. As expected, CLIP outperforms our model. However, our model achieves impressive results considering its simplicity, its size, and that CLIP is the current state-of-the-art in self-supervised vision and language learning. Training a CLIP model of comparable size and exposed to similar training data as our model is beyond the scope of this paper, but an interesting direction for future work.
5.4 Object Localization
Another important task performed by infants is visual object localization. To test our model's ability to reliably localize objects in images we use Class Activation Maps (CAM) described by Zhou et al. (2016). Each CAM indicates how important each pixel was during classification for a specific cluster.
Quantitative analysis. Most previous studies of zero-shot segmentation (Bucher et al., 2019) trained on a subset of "seen" classes, and evaluated on both seen and unseen classes. We use a more challenging setup previously referred to as annotation-free segmentation (Zhou et al., 2021), where we evaluate our model without any training for the segmentation task. We use MSCOCO's ground-truth bounding boxes, which are human annotated and mark objects in the image, for evaluation. Following the original CAM paper, we use a heuristic method to predict bounding boxes: Given a CAM, we segment the pixels of which the value is
Model	Precision	Recall	F-Score
Ours	0.178 ± 0.01	0.025 ± 0.004	0.044 ± 0.006
Rand	0.027 ± 0.001	0.027 ± 0.001	0.027 ± 0.001
Table 6: Mean F-score and standard deviation across 5 random initializations on bounding box prediction. Our model (Ours) improves precision (bold) over random (Rand) significantly while achieving similar recall (might improve by tuning the visual threshold).
above  $50\%$  of the max value of the CAM and take the bounding box that covers the largest connected component in the segmentation map.
We use precision and recall for evaluation. A pair of bounding boxes is considered a match if the intersection over union (IoU) of the pair exceeds 0.5. Given lists of predicted and ground-truth bounding boxes, we consider each matched pair as a true positive and a prediction (ground-truth) for which no matching ground-truth (prediction) was found as a false positive (negative). We compare our model to a random baseline: Sample  $k$  random bounding boxes (where  $k$  is the number of ground-truth bounding boxes in the current image). This baseline uses the number of ground-truth bounding boxes in each image (our model is not exposed to this information).
The results are presented in Table 6. Our model is significantly more precise than the random baseline, but achieves similar recall: the entire MSCOCO test split contains a total of 164,750 bounding boxes, while our model predicted 38,237 bounding boxes. This problem could be addressed by lowering the visual threshold. We leave this direction for future research.
Qualitative analysis. Fig. 3 shows a selection of CAMs, plotted as heatmaps and associated with class predictions (see Sec. 5.3). The heatmaps extracted by the model were better when the model predicted a correct class in the visual classification task (top six images and bottom left image in Fig 3). In the bottom two images two clusters were predicted for the same original image, one correct and one incorrect (with an, unsurprisingly, meaningless heatmap).
We proposed a model for unsupervised multimodal language acquisition, trained to jointly cluster text and images. Many of our design choices were guided by findings from cognitive studies of infant language acquisition: The joint learning of
Figure 3: Examples of heatmaps of CAM values. Above each image we print the class predicted by the model.
multiple modalities; learning word-level semantics (e.g., Fisher et al., 1994, suggest that children first learn to identify nouns and use this information to learn sentence-level semantics); and cross-situational learning (counting how many times each word co-occurred with each cluster, see Gleitman, 1990). After training, our model demonstrates capabilities typical of infant language acquisition: Word concreteness prediction and identification and segmentation of objects in a visual scene.
However, we do not stipulate that infants begin their acquisition of language by clustering words. It would be interesting to design experiments to test this hypothesis, e.g., by connecting our work with laboratory work on joint word and category learning (Borovsky and Elman, 2006), or work on the emergence of syntagmatic vs. taxonomic categories in young children (Sloutsky et al., 2017).
While our model is cognitively more plausible compared to previous studies, the gap from a realistic setting of language acquisition is still large: (1) we assume the language input is segmented into words; (2) the input data, while naturalistic, is not typical of infants at the stage of language acquisition.
tion; (3) the input only includes the visual and textual modality, but not, e.g., pragmatic cues like gesture; and (4) the model learns in a non-interactive setting, whereas physical and social interactions are considered crucial for language learning, and learns (and is evaluated) in a batch fashion while human learning is typically incremental (Frermann and Lapata, 2016).
In the semantic word categorization and concreteness prediction experiments, we compared our multimodal model to unimodal text-only baselines, which we chose to be as similar as possible to our model. The results suggest that multimodality improves performance on both text tasks. However, it is unclear which specific information is encoded in the visual modality that benefits these text tasks. We leave this question for future research.
Syntagmatic categories, although highly intuitive in the context of human memory, were not the subject of many previous computational studies. We propose to further investigate this type of categories and its use. One interesting direction is to combine syntagmatic categories with interactivity: Given a relevant signal from the environment the model can cycle through concepts in the syntagmatic category triggered by the signal, speeding up the extraction of relevant concepts in real time. One possible application of this direction is modelling the construction of ad-hoc categories, described by Barsalou (1983).
While all experiments are conducted in English, our setting supports future work on other languages. The small training set and the nature of the data (image-sentence pairs that might be, to some extent, collected from the Internet) allow our model to be extended to low-resource languages, while the minimal structural assumptions of the text encoder may imply some degree of language-agnosticity.
In future work, we plan to improve the cognitive plausibility of our model by (1) incorporating typical input observed by children (by using videos taken in real scenes of child language acquisition, see Sullivan et al., 2021); and (2) changing the setting to an interactive one, where the model is transformed into a goal-driven agent that uses interactivity to learn and produce language.
We would like to thank the anonymous reviewers for their helpful comments and feedback. This work was supported in part by the Israel Science
Foundation (grant no. 2424/21), by a research gift from the Allen Institute for AI and by the HUJI-UoM joint PhD program.
Ethical Considerations
We used publicly available resources to train our model (Lin et al., 2014). As with other statistical methods for word representations, our approach may capture social biases which manifest in its training data (e.g., MSCOCO was shown to be biased with respect to gender, Zhao et al., 2017). Our code includes a model card (Mitchell et al., 2019) which reports standard information regarding the training used to produce our models and its word clusters.
Lawrence W. Barsalou. 1983. Ad hoc categories. Memory & Cognition, 11(3):211-227.
Arielle Borovsky and Jeff Elman. 2006. Language input and semantic categories: A relation between cognition and early word learning. Journal of child language, 33(4):759-790.
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2013. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods, 46(3):904-911.
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez. 2019. Zero-shot semantic segmentation. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 466-477.
Tyler A Chang and Benjamin K Bergen. 2021. Word acquisition in neural language models. ArXiv preprint, abs/2110.02406.
Jean Charbonnier and Christian Wartena. 2019. Predicting word concreteness and imagery. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 176-187, Gothenburg, Sweden. Association for Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020a. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
Grzegorz Chrupa, Lieke Gelderloos, and Afra Alishahi. 2017. Representations of language in a model of visually grounded speech signal. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 613-622, Vancouver, Canada. Association for Computational Linguistics.
Simon De Deyne, Danielle J Navarro, Amy Perfors, Marc Brysbaert, and Gert Storms. 2019. The "small world of words" english word association norms for over 12,000 cue words. Behavior research methods, 51(3):987-1006.
Ferdinand De Saussure. 1916. Cours de linguistique générale. Payot, Paris.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Society.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Loekie Elbers and Anita van Loon-Vervoorn. 1999. Lexical relationships in children who are blind. Journal of Visual Impairment & Blindness, 93(7):419-421.
Susan M Ervin. 1961. Changes with age in the verbal determinants of word-association. The American journal of psychology, 74(3):361-372.
Cynthia Fisher, D.geoffrey Hall, Susan Rakowitz, and Lila Gleitman. 1994. When it is better to receive than to give: Syntactic and conceptual constraints on vocabulary growth. *Lingua*, 92:333-375.
Trevor Fountain and Mirella Lapata. 2010. Meaning representation in natural language categorization. In Proceedings of the 32nd annual conference of the Cognitive Science Society, pages 1916-1921.
Michael C. Frank, Noah D. Goodman, and Joshua B. Tenenbaum. 2007. A bayesian framework for cross-situational word-learning. In Advances in Neural Information Processing Systems 20, Proceedings of the Twenty-First Annual Conference on Neural Information Processing Systems, Vancouver, British
Columbia, Canada, December 3-6, 2007, pages 457-464. Curran Associates, Inc.
Lea Frermann and Mirella Lapata. 2016. Incremental bayesian category learning from natural language. Cognitive Science, 40(6):1333-1381.
Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kimbrough Oller, John HL Hansen, and Terrance D Paul. 2017. Mapping the early language environment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):248-265.
Lila Gleitman. 1990. The structural sources of verb meanings. Language Acquisition, 1(1):3-55.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear, 7(1):411-420.
Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624-646, Online. Association for Computational Linguistics.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, Valencia, Spain. Association for Computational Linguistics.
Ákos Kádár, Afra Alishahi, and Grzegorz Chrupał. 2015. Learning word meanings from images of natural scenes. *Traitément Automatique des Langues*.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. Computer Vision - ECCV 2014 Lecture Notes in Computer Science, page 740-755.
Alexander H. Liu, SouYoung Jin, Cheng-I Jeff Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. 2021. Cross-modal discrete representation learning.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13-23.
Nawoja Mikolajczak-Matyja. 2015. The associative structure of the mental lexicon: hierarchical semantic relations in the minds of blind and sighted language users. Psychology of Language and Communication, 19(1):1.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220-229.
Mitja Nikolaus and Abdellah Fourtassi. 2021. Evaluating the acquisition of semantic knowledge from cross-situational learning in artificial neural networks. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 200-210, Online. Association for Computational Linguistics.
Alexander Ororbia, Ankur Mali, Matthew Kelly, and David Reitter. 2019. Like a baby: Visually situated neural language acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5127-5136, Florence, Italy. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR.
Deb K Roy and Alex P Pentland. 2002. Learning words from sights and sounds: A computational model. Cognitive science, 26(1):113-146.
Ava Santos, Sergio E. Chaigneau, W. Kyle Simmons, and Lawrence W. Barsalou. 2011. Property generation reflects word association and situated simulation. Language and Cognition, 3(1):83-119.
Carina Silberer and Mirella Lapata. 2014. Learning grounded meaning representations with autoencoders. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 721-732, Baltimore, Maryland. Association for Computational Linguistics.
Vladimir M Sloutsky, Hyungwook Yim, Xin Yao, and Simon Dennis. 2017. An associative account of the development of word learning. Cognitive Psychology, 97:1-30.
Suzy J. Styles and Kim Plunkett. 2009. How do infants build a semantic system? Language and Cognition, 1(1):1-24.
Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C. Frank. 2021. Saycam: A large, longitudinal audiovisual dataset recorded from the infant's perspective. Open Mind : Discoveries in Cognitive Science, 5:20 - 29.
Michael Tomasello, Randi Strosberg, and Nameera Akhtar. 1996. Eighteen-month-old children learn words in non-ostensive contexts. Journal of child language, 23(1):157-176.
Peter Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011. Literal and metaphorical sense identification through concrete and abstract context. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 680-690, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2979-2989, Copenhagen, Denmark. Association for Computational Linguistics.
Bolei Zhou, Aditya Khosla, Ågata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2921-2929. IEEE Computer Society.
Chong Zhou, Chen Change Loy, and Bo Dai. 2021. Denseclip: Extract free dense labels from clip. ArXiv preprint, abs/2112.01071.
A.1.1 Training
For the visual encoder, we used the ResNet50 implementation from the torchvision package with ADAM optimizer and a learning rate of  $10^{-4}$ .
A.1.2 Semantic Word Categorization
For clustering of word embeddings, we use the KMeans implementation in scikit-learn. We use the word2vec google-news-300 model from thegensim package, the BERT<sub>BASE</sub> model from the transformers library and CLIP's official implementation.
A.1.3 Concreteness Estimation