SlowGuess/NAACL_2022_OCR huggingface dataset:
Uri Berger $^{1,2}$ , Gabriel Stanovsky $^{1}$ , Omri Abend $^{1}$ , and Lea Frermann $^{2}$
$^{1}$ School of Computer Science and Engineering, The Hebrew University of Jerusalem  
 $^{2}$ School of Computing and Information Systems, University of Melbourne  
{uri.berger2, gabriel.stanovsky, omri.abend}@mail.huji.ac.il lea.frermann@unimelb.edu.au
Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition, which is believed to rely heavily on cross-modal signals. However, prior studies have been limited by their reliance on vision models trained on large image datasets annotated with a pre-defined set of depicted object categories. This is (a) not faithful to the information children receive and (b) prohibits the evaluation of such models with respect to category learning tasks, due to the pre-imposed category structure. We address this gap, and present a cognitively-inspired, multimodal acquisition model, trained from image-caption pairs on naturalistic data using cross-modal self-supervision. We show that the model learns word categories and object recognition abilities, and presents trends reminiscent of those reported in the developmental literature. We make our code and trained models public for future reference and use<sup>1</sup>.
To date, the mechanisms underlying the efficiency with which infants learn to speak and understand natural language remain an open research question. Research suggests that children leverage contextual, inter-personal and non-linguistic information. Visual input is a case in point: when spoken to, infants visually perceive their environment, and paired with the input speech, the visual environment could help bootstrap linguistic knowledge (Tomasello et al., 1996). Unlike social cues, visual input has a natural physical representation, in the form of pixel maps or videos.
Previous multimodal language acquisition studies either considered toy scenarios with small vocabularies (Roy and Pentland, 2002; Frank et al., 2007), or used visual encoders that were pretrained
Figure 1: Model overview. Given an image-caption pair (left), both the visual (top) and textual encoder (bottom) generate a binary vector indicative of the clusters associated with the current input. The text/visual model predicts a probability vector over clusters per word/image. Vectors are mapped to a binary space using thresholding. The modality-specific vectors provide mutual supervision during training (right).
on large labeled data bases such as ImageNet (Deng et al., 2009) or Visual Genome (Krishna et al., 2017). This has two drawbacks: first, systematic access to labeled data is a cognitively implausible assumption in a language acquisition setting; second, imposing a pre-defined categorization system precludes studying categories that emerge when learning from unlabeled multimodal data. This type of setting more closely resembles the data underlying early language learning at a time when the child has only acquired little conceptual information. Although the subject of much psycholinguistic work, the computational study of multimodal word categories, formed without recourse to manual supervision has been scarcely addressed in previous work.
We present a model that learns categories as clans
$^{1}$ github.com/SLAB-NLP/multimodal_clustering
Human Language Technologies, pages 3819 - 3835
Model	parameters	token	(img, capt)	train time
BERT	110M	3300M	-	64 TPU days
word2vec	1B	100B	-	60 CPU days
CLIP	63M	-	400M	10K GPU days
Ours	3M	3M	280K	4 GPU days
Children	-	13K/day	-	-
Table 1: Comparing the size, training data and training time of popular self-supervised models (BERT<sub>base</sub>, word2vec (300d, google-news) and CLIP with ResNet50X64) against our model (Ours, bold) and typical child input (Children; Gilkerson et al., 2017). For the multimodal models (CLIP and Ours) we only mention the size of the text encoder.
ters of words from large-scale, naturalistic multimodal data without any pre-training. Given (image, caption)-pairs as input, the model is trained to cluster words and images in a shared, latent space, where each cluster pertains to a semantic category. In a self-supervised setup, a neural image classifier and a co-occurrence based word clustering module provide mutual supervision through joint training with an expectation-maximization (EM) style algorithm. Figure 1 illustrates our model.
Our input representation is cognitively plausible in that we use raw image data (pixels), and train our model on a comparatively small data set (Table 1). However, we follow previous work (Kádár et al., 2015; Nikolaus and Fourtassi, 2021) in using written and word-segmented text input. Young infants do not have access to such structure, and this assumption therefore deviates from cognitive plausibility (but opens avenues for future work).
We show that semantic and visual knowledge emerges when training on the self-supervised categorization task. In a zero-shot setup, we evaluate our model on (1) word concreteness prediction as a proxy for noun identification, one of the first cues for syntax in infant language acquisition (Fisher et al., 1994); (2) visual classification and object segmentation. We also study the emerging latent word clusters and show that words are clustered syntagmatically (De Saussure, 1916): words representing entities that are likely to occur together are more likely clustered together (e.g., dog-bark), than words that share taxonomic categories (e.g., dog-cat). This concurs with findings that young children acquire syntagmatic categories more readily than taxonomic categories (Sloutsky et al., 2017).
We briefly review previous studies of unimodal learning without pretraining (for both text and vision) and multimodal learning (studied mainly in acquisition implausible settings) to highlight the gap that this work addresses.
Unimodal Learning. Self-supervised learning without pre-training has been extensively studied, but predominantly in a unimodal scenario or under cognitively implausible assumptions. For the text modality, large language models have been developed in recent years (e.g., BERT; Devlin et al., 2019), trained on large unlabeled text corpora (Table 1). A more cognitively motivated model is BabyBERTa (Huebner et al., 2021), a smaller version of RoBERTa (Liu et al., 2019) (also) trained on transcribed child directed speech. In the visual domain, self-supervision is typically implemented as contrastive learning, training the model to align corrupted images with their original counterparts (Chen et al., 2020a), with subsequent fine-tuning.
Multimodal Language Learning. Early language acquisition studies (Roy and Pentland, 2002) considered toy scenarios with small vocabularies and used heuristics for image processing. Silberer and Lapata (2014) model multi-modal human categorization using human-annotated feature vectors as input for a multimodal self-supervised autoencoder, while we learn the features from raw images.
Unlike our work, recent work on cross-modal language learning (Kádár et al., 2015; Chrupaţa et al., 2017; Ororbia et al., 2019; Nikolaus and Fourtassi, 2021) typically use Convolutional Neural Networks, pre-trained on large labeled data bases like ImageNet (Deng et al., 2009), or alternatively (e.g., Lu et al., 2019; Chen et al., 2020b) use object detectors pre-trained on Visual Genome (Krishna et al., 2017) as the visual model.
Few studies assume no prior knowledge of the grounded modality. Most related to our study is CLIP (Radford et al., 2021), a pre-trained off-the-shelf model trained to project matching images and captions to similar vectors. CLIP assumes a multimodal joint space which is continuous, unlike our binary space. Liu et al. (2021) use CLIP pretrained encoders to learn cross-modal representations with a similar training objective as ours. They discretize the output of the encoders by mapping it to the closest vector from a finite set of learned vectors, which can be viewed as a form of categorization. CLIP-
based works are trained to match entire sentences to images and have no explicit representation of words and phrases. We therefore view it as a cognitively less plausible setting than is presented in the current study. Nevertheless, we include CLIP as a point of comparison when applicable.
Our goal is to learn the meaning of words and raw images through mutual supervision by mapping both modalities to a joint representation. Intuitively, given a visual input paired with relevant text (approximating a typical learning scenario), the output for each of the modalities is a binary vector in  $\{0,1\}^N$ , with non-zero dimensions indicating the clusters<sup>2</sup> to which the input is assigned and  $N$  is the total number of clusters (a predefined hyper-parameter). The clusters are unknown a priori and are formed during training. The goal is to assign matching text and image to the same clusters. In order to minimize assumptions on innate knowledge or pre-imposed categories available to the language learner, and enable the study of emerging categories from large-scale multi-modal input data, we deliberately avoid any pre-training of our models.
3.1 Visual Encoder
The visual encoder (Figure 1, top) is a randomly initialized ResNet50 (He et al., 2016), without pretraining. We set the output size of the network to  $N$  (the number of clusters) and add an element-wise sigmoid layer. To produce the binary output and predict the clusters given an input image, we apply a hyper-parameter threshold  $\theta_v$  to the output of the sigmoid layer.
3.2 Text Encoder
The text encoder (Figure 1, bottom) is a simple probabilistic model based on word-cluster cooccurrence, which is intuitively interpretable and makes minimal structural assumptions. Given a sentence, the model assigns each word to at most one cluster. The sentence is assigned to the union of the clusters to which the words in it are assigned.
Formally, given a sentence  $s = (w_{1}, w_{2}, \ldots, w_{n})$  of words  $w_{i}$ , and an assignment of the words to clusters  $f: \{w_{1}, \ldots, w_{n}\} \rightarrow \{1, \ldots, N\} \cup \{\emptyset\}$ , the clusters to which the sentence is assigned are:  $\{c | \text{if } \exists w_{i} \text{ s.t. } f(w_{i}) = c\}_{c=1}^{N}$ . When assigning words to clusters, we make two simplifying assumptions: (1) the probability that a word is assigned to a specific cluster is independent of the linguistic context, meaning that we assign to clusters on the type- rather than the token level (a reasonable assumption given that children learn single words first); (2) a single word cannot be assigned to more than one cluster, but it might be assigned to no cluster at all if it does not have a visual correspondent in the image (e.g., function words). Under these assumptions, the encoder estimates  $P(c|w)$  for each  $c \in \{1, \ldots, N\}$  and for each  $w \in V$ , where  $V$  is the vocabulary. If the probability of assigning a given word in a sentence to any of the clusters exceeds a hyper-parameter threshold  $\theta_{t}$ , it is assigned to the cluster with the highest probability, otherwise it is not assigned to any cluster. Formally:
$$
f (w) = \left\{ \begin{array}{l l} \underset {c \in [ N ]} {\operatorname {a r g m a x}} P (c | w) & \text {i f} \underset {c \in [ N ]} {\operatorname {m a x}} P (c | w) \geq \theta_ {t} \\ \emptyset & \text {e l s e} \end{array} \right.
$$
In the next step, we define the word-cluster associations  $P(c|w)$ . We estimate these using Bayes Rule,
$$
P (c | w) = \frac {P (w | c) P (c)}{P (w)} \tag {1}
$$
$P(w|c)$  is defined as the fraction of all predictions of cluster  $c$  from the visual encoder, in which  $w$  occurred in the corresponding caption. We instantiate the prior cluster probability  $P(c)$  as uniform over all clusters.
Finally, for a given word  $w$ , we estimate  $P(w) = \sum_{i=1}^{N} P(c_i)P(w|c_i)$ . Intuitively, we expect that a concrete word would repeatedly occur with similar visual features (of the object described by that word), therefore repeatedly co-occurring with the same cluster and receiving a high assignment probability with that cluster, whereas abstract words would co-occur with multiple clusters, therefore not being assigned to any cluster.
4We also tried instantiating  $P(c)$  as the empirical cluster distribution as predicted by the visual encoder. However, the noisy initial predictions lead to a positive feedback loop leading to most clusters being unused.
2We use the term clusters for the output of models and the term categories for ground truth classification of elements (e.g., human defined categories).
3This is a multi-label categorization task (i.e., an image can be assigned to multiple clusters): Cluster assignments do not compete with one another, which is why we chose sigmoid over softmax.
4 Training
At each training step, the model observes a batch of (image, caption)-pairs. We first perform inference with both encoders, and then use the results of each encoder's inference to supervise the other encoder.
Text Encoder. Given the list of clusters predicted by the visual encoder and a tokenized caption  $s = \{w_1, \dots, w_n\}$ , for each  $w_i \in s$  and for each cluster  $c_j$  predicted by the visual encoder, we increment  $count(c_j)$  and  $count(w_i, c_j)$ . These are needed to compute the probabilities in equation (1).
Visual Encoder. For each input image and corresponding cluster vector predicted by the text encoder, we use binary cross entropy loss comparing the output of the sigmoid layer with the predicted cluster vector and use Backpropagation to update the parameters of the ResNet model.
We trained our model on the 2014 split of MSCOCO (Lin et al., 2014), a dataset of naturalistic images with one or more corresponding captions, where each image is labeled with a list of object classes it depicts. MSCOCO has 80 object classes, 123K images and 616K captions (split into  $67\%$  train,  $33\%$  test). We filtered out images that did not contain any labeled objects, and images that contained objects with a multi-token label (e.g., "fire hydrant"). After filtering, we are left with 65 ground-truth classes. The filtered training (test) set contains 56K (27K) images and 279K (137K) captions. We set apart  $20\%$  of the training set for hyper-parameter tuning.
We trained our model with a batch size of 50 until we observed no improvement in the F-score measure from Section 5.1 (40 epochs). Training took 4 days on a single GM204GL GPU. We used  $N = 150$  clusters,  $\theta_t = 0.08$ , and  $\theta_v = 0.5$ . The visual threshold  $\theta_v$  was first set heuristically to 0.5 to avoid degenerate solutions (images being assigned to all or no clusters initially). Then,  $N$  and  $\theta_t$  were determined in a grid search, optimizing the F-score measure from Section 5.1. We used spaCy (Honnibal and Montani, 2017) for tokenization.
5.1 Semantic Word Categorization
Background. Semantic word categorization is the clustering of words based on their semantic features. Psycholinguistic studies have shown that children use semantic word categories to solve linguistic tasks by the end of the second year of life (e.g., Styles and Plunkett, 2009). There is a long established fundamental distinction between syntagmatic (words that are likely to co-occur in the same context) and paradigmatic relations (words that can substitute one another in a context without affecting its grammaticality or acceptability) (De Saussure, 1916). Each relation type invokes a different type of word categories (syntagmatic relations invoke syntagmatic categories, or associative categories; paradigmatic relation invoke taxonomic categories). Despite an acknowledgement that infants, unlike adults, categorize based on syntagmatic criteria more readily than on paradigmatic criteria ("The Syntagmatic-Paradigmatic shift"; Ervin, 1961), and empirical evidence that syntagmatic categories might be more important for word learning than taxonomic categories (Sloutsky et al., 2017), computational categorization studies and datasets predominantly focused only on taxonomic hierarchies (Silberer and Lapata, 2014; Frermann and Lapata, 2016).
Setting. Our model's induced clusters are created by using the text encoder to predict, for each word, the most likely cluster.
We evaluated induced clusters against a taxonomic and a syntagmatic reference data set. First, we followed Silberer and Lapata (2014), used the categorization dataset from Fountain and Lapata (2010), and transformed the dataset into hard categories by assigning each noun to its most typical category as extrapolated from human typicality ratings. The resulting dataset contains 516 words grouped into 41 taxonomic categories. We filtered the dataset to contain only words that occur in the MSCOCO training set and in the word2vec (Mikolov et al., 2013) dictionary, obtaining the final dataset with 444 words grouped into 41 categories.
In order to quantify the syntagmatic nature of the induced clusters, we used a large dataset of human word associations, the "Small World of Words" (SWOW, De Deyne et al., 2019). SWOW was compiled by presenting a cue word to human participants and requesting them to respond with the first three words that came to mind. The association
5This was enforced because multi-token labels do not necessarily map to a single cluster, rendering the evaluation more difficult. We plan to address this gap in future work.
strength of a pair of words  $(w_{1}, w_{2})$  is determined by the number of participants who responded with  $w_{2}$  to cue word  $w_{1}$ . Prior work has shown that word associations are to a large extent driven by syntagmatic relations (Santos et al., 2011).
Comparison with other models. We compare against several word embedding models, where for each model we first induce embeddings, which we then cluster into  $\mathrm{K} = 41$  clusters (the number of taxonomic gold classes) using K-Means. We compare against a text-only variant of our model by creating a co-occurrence matrix  $C$  where  $C_{i,j}$  is the number of captions in which tokens  $i, j$  in the vocabulary co-occur. The normalized rows of  $C$  are the vector embeddings of words in the vocabulary. We compare against off-the-shelf word2vec and BERTBASE embeddings. For BERT, given a word  $w$ , we feed an artificial context ("this is a  $w$ ) and take the embedding of the first subword of  $w$ . We also include the multi-modal CLIP, using prompts as suggested in the original paper ("a photo of a  $w$ "). Finally, we include a randomized baseline, which assigns each word at random to one of 41 clusters. Implementation details can be found in Appendix A.1.
Taxonomic categorization. We use the F-score metric following Silberer and Lapata (2014). The F-value of a (gold class, cluster)-pair is the harmonic mean of precision and recall defined as the size of intersection divided by the number of items in the cluster and the number of items in the class, respectively. The F-score of a class is the maximum F-value attained at any cluster, and the F-score of the entire clustering is the size-weighted sum of F-scores of all classes. We report performance over five random restarts for all models.
Results are presented in Table 2. The text-only baseline improves results over a random categorization algorithm. Our multi-modal model grounded in visual input improves over its unimodal variant. Our model is competitive with BERT and is surpassed by word2vec and CLIP. However, considering the small model and training data we used (see Table 1), our results are competitive.
Model	F-Score
Random	0.15 ± 0.0032
Text-only	0.26 ± 0.0098
Word2vec	0.40 ± 0.0172
BERTBASE	0.33 ± 0.011
CLIP	0.38 ± 0.0142
Ours	0.33 ± 0.0109
Table 2: Taxonomic categorization results, comparing a random baseline (top) against text-only (center) and multi-modal models (bottom), as mean F-score and std over 5 runs. Word2vec (bold) achieves the highest score.
Model	MAS
Taxonomic	5.72
Random	4.23 ± 1.88
Text-only	5.47 ± 0.25
Word2Vec	6.65 ± 0.16
BERTBASE	5.75 ± 0.23
CLIP	7.08 ± 0.41
Ours	7.45 ± 0.33
Table 3: Sytagmatic categorization results on the same models as in Table 2, reporting mean association strength (MAS) of pairs of clustered concepts in the SWOW dataset, over 5 random initializations. Taxonomic refers to the taxonomic gold categories. Our model (bold) achieves the highest score.
Syntagmatic categorization. We quantify the syntagmatic nature of a clustering by the mean association strength (MAS) of pairs of words in the SWOW dataset, where association strength of a pair of words  $(w_{1}, w_{2})$  is again number of participants who responded with w2 to cue word w1. MAS is computed across all word pairs from the taxonomic dataset in which both words were assigned the same cluster by this clustering solution.
Results are presented in Table 3. The multimodal models (ours and CLIP) outperform all unimodal models, an indication of the impact of multimodality on category learning: multimodal word learning shifts the learner towards syntagmatic relations more significantly than unimodal word learning. To our knowledge, this is the first computational result to support this hypothesis, shown empirically in human studies with infants (Elbers and van Loon-Vervoorn, 1999; Mikolajczak-Matyja, 2015).
Qualitative analysis. Table 4 shows four of the
<sup>6</sup>We omit Silberer and Lapata (2014) since we were unable to obtain their model.
<sup>7</sup>It was not clear how to design co-occurrence based vision-only baselines, as images do not naturally factorize into concepts/regions, unlike text which is divided into words.
8We also tried BERT and CLIP feeding as input the target word only. Results for CLIP slightly decreased, while results for BERT decreased significantly.
Ours 1 skis; axe; sled; parka; sleigh; pants; gloves
Ours 2 sailboat; canoe; swan; raft; boat; yacht; duck; willow; ship; drum
Ours 3 train; bullet; subway; tack; bridge; trolley
Ours 4 bedroom; rocker; drapes; bed; dresser; sofa; couch; piano; curtains; cushion; lamp; chair; fan; bureau; stool; cabin; book
W2V cluster avocado, walnut, pineapple, grapefruit, coconut, olive, lime, lemon
Table 4: Four clusters induced by our model (Ours 1, 2, 3, 4), sorted by  $P(c|w)$ , and one cluster induced by word2vec. Our clusters are syntagmatic, while the W2V cluster is taxonomic. Words highlighted by the same color belong to the same taxonomic category.
clusters created by our model and one cluster created by word2vec for the taxonomic categorization dataset. The clusters formed by our algorithm are syntagmatic, associating words frequently observed together (e.g., tokens in cluster 1 are related to snow activity, while cluster 2 broadly relates to water). The cluster formed by word2vec embeddings is taxonomic (all tokens are food products). Our results provide initial evidence that syntagmatic clusters emerge from an unsupervised training algorithm drawing on simple joint clustering of words and images.
5.2 Concreteness Estimation
Background. Fisher et al. (1994) suggest that the number of nouns in a sentence is among the earliest syntactic cues that children pick up. Consequently, noun identification is assumed to be one of the first syntactic tasks learned by infants. We approximate noun identification as concreteness estimation, since words representing concrete entities are mostly nouns.[10] Chang and Bergen (2021) show that while children acquire concrete words first, neural text-based models show no such effect, suggesting that multimodality impacts the learning process.
Setting. We evaluate concreteness estimation using the dataset by Brysbaert et al. (2013), which contains concreteness ratings for 40K English words averaged over multiple human annotated ratings on a scale of 1 to 5. We estimate the concreteness of a word as the maximum probability with which it was assigned to any cluster. For evaluation, we follow Charbonnier and Wartena (2019) and compute the Pearson correlation coefficient of our predictions with the ground-truth values. In addition, we investigate the impact of word frequency on our model's predictions by evaluating the model on subsets of words in the Brysbaert data of increasing minimum frequency in MSCOCO.