SlowGuess/NAACL_2022_OCR huggingface dataset:
Chenyu Zhang
Zhuowan Li*
Elias Stengel-Eskin*
{czhan105, vandurme, zli110, alias}@jhu.edu
Our commonsense knowledge about objects includes their typical visual attributes; we know that bananas are typically yellow or green, and not purple. Text and image corpora, being subject to reporting bias, represent this world-knowledge to varying degrees of faithfulness. In this paper, we investigate to what degree unimodal (language-only) and multimodal (image and language) models capture a broad range of visually salient attributes. To that end, we create the Visual Commonsense Tests (ViComTe) dataset covering 5 property types (color, shape, material, size, and visual co-occurrence) for over 5000 subjects. We validate this dataset by showing that our grounded color data correlates much better than ungrounded text-only data with crowdsourced color judgments provided by Paik et al. (2021). We then use our dataset to evaluate pretrained unimodal models and multimodal models. Our results indicate that multimodal models better reconstruct attribute distributions, but are still subject to reporting bias. Moreover, increasing model size does not enhance performance, suggesting that the key to visual commonsense lies in the data.
The observation that human language understanding happens in a rich multimodal environment has led to an increased focus on visual grounding in natural language processing (NLP) (Baltrusaitis et al., 2019; Bisk et al., 2020), driving comparisons between traditional unimodal text-only models and multimodal models which take both text and image inputs. In this work, we explore to what extent unimodal and multimodal models are able to capture commonsense visual concepts across five types of relations: color, shape, material, size, and visual co-occurrence (cf. Fig. 1). We further explore how this
ability is influenced by reporting bias (Gordon and Van Durme, 2013), the tendency of large corpora to over- or under-report events. We define visual commonsense as knowledge about generic visual concepts, e.g. "knobs are usually round", and we measure this knowledge via frequency distributions over potential properties (e.g. round, square, etc). A visually-informed language model should be able to capture such properties. Our color, shape, material, and co-occurrence data are mined from Visual Genome (Krishna et al., 2016), and our size data are created from object lists. They contain a large number of examples of per-object attribute distributions and "object-attribute" pairs.
Paik et al. (2021) evaluate language models' color perception using a human-annotated color dataset (CoDa), finding that reporting bias negatively influences model performance and that multimodal training can mitigate those effects. In this work, we confirm those findings while extending the evaluation to a broader range of visually salient properties, resulting in a more comprehensive metric for visual commonsense. In order to elicit visual commonsense from language models, we utilize soft prompt tuning (Qin and Eisner, 2021), which trains optimal templates by gradient descent for each model and relation type that we explore. We also utilize knowledge distillation to enhance a text-only model's visual commonsense ability, where the vision-language model serves as the teacher.
The major contributions of this work are: (1) we design a comprehensive analytic dataset, ViComTe, for probing English visual commonsense, that is applicable to any language model; (2) we use ViComTe to study models' ability to capture empirical distributions of visually salient properties. We examine unimodal language models, multimodal vision-language (VL) models, and a knowledge-distilled version of a VL model; and (3) we analyze the effects of reporting bias on the visually-grounded vs. ungrounded datasets and models.
*Joint Advising
1The dataset and code is available at https://github. com/ChenyuHeidiZhang/VL-commonsense.
Human Language Technologies, pages 5321 - 5335
Figure 1: We compare unimodal and multimodal models' abilities to capture visual commonsense knowledge. The commonsense knowledge is evaluated on five relation types: color, shape, material, size, and visual co-occurrence. We compare the model outputs with the gold distribution from ViComTe, which is mined from Visual Genome.
2.1 Vision-Language Modeling
Recent advances in vision-language (VL) modeling have led to increased success on benchmark tasks. Most VL models learn joint image and text representations from cross-modal training of transformers with self-attention, including LXMERT (Tan and Bansal, 2019), ViLBERT (Lu et al., 2019), VisualBERT (Li et al., 2019), UNITER (Chen et al., 2020), etc. Oscar (Li et al., 2020) additionally uses object tags in images as anchor points to facilitate the learning of image-text alignments and VinVL (Zhang et al., 2021) presents an improved object detection model. CLIP (Radford et al., 2021) learns by predicting caption-image alignment from a large internet corpus of (image, text) pairs.
While our work uses textual prompt tuning techniques, there have also been work on visual prompt engineering to enhance the performance of pretrained vision-language models. Zhou et al. (2021) model context in prompts as continuous representations and learn to optimize that context. Yao et al. (2021) develop a cross-modal prompt tuning framework that reformulates visual grounding as a fill-in-the-blank problem for both image and text.
2.2 Visual Commonsense
In one of the early attempts at learning visual commonsense, Vedantam et al. (2015) measure the plausibility of a commonsense assertion in the form of (obj1, relation, obj2) based on its similarity
to known plausible assertions, using both visual scenes and accompanying text. Zellers et al. (2021) learn physical commonsense via interaction, and use this knowledge to ground language. Frank et al. (2021) probe whether VL models have learned to construct cross-modal representations from both modalities via cross-modal input ablation.
Note that our definition of visual commonsense differs from that of Zellers et al. (2019), where the model is required to perform commonsense reasoning based on an image. Our definition of visual commonsense is more similar to the idea of stereotypic tacit assumptions (Prince, 1978) – the propositional beliefs that humans hold about generic concepts, such as “dogs have to be walked”. Weir et al. (2020) probe neural language models for such human tacit assumptions and demonstrate the models’ success. We extend this intuition to visual concepts and explore how visual information may help language models to capture such assumptions.
There has also been earlier work on the McRae feature norms (McRae et al., 2005), in which human annotators wrote down attributes that describe the meaning of words. For instance, "car" can be labeled as "has four wheels" and "apple" can be labeled as "is green". Silberer et al. (2013) expand the McRae dataset into a set of images and their visual attributes and construct visually grounded distributional models that can represent image features with visual attributes.
Zhu et al. (2020) examine the "language prior" problem in Visual Question Answering models,
where models tend to answer based on word frequencies in the data, ignoring the image contents. In this work, we explore to what extent such a language prior is recruited absent a visual input.
2.3 Reporting Bias
Pretrained language models such as BERT (Devlin et al., 2019) are trained on billions of tokens of text, capturing statistical regularities present in the training corpora. However, their textual training data can suffer from reporting bias, where the frequency distribution of specific events and properties in text may not reflect the real-world distribution of such properties (Gordon and Van Durme, 2013). For example, while grass is typically green, this may be under-reported in web corpora (as it is assumed to be true), and while motorcycle crashes may be more common in the real world, plane crashes are mentioned far more in news text (Gordon and Van Durme, 2013). Misra et al. (2016) highlight the reporting bias in "human-centric" image annotations and find that the noise in annotations exhibits a structure that can be modeled.
3 Dataset: ViComTe
3.1 Dataset Mining
For each relation color, shape, material, size, and object co-occurrence, our data take the form of (subject, object) tuples extracted from object distributions per subject. The goal is to predict the object and its distribution from the subject and relation. Table 1 summarizes the number of classes and subject-object pairs for each relation.2
Color, Shape, Material For color, shape, and material, the subject is a noun and the object is the color, shape, or material property of the noun, mined from attributes of Visual Genome (VG) (Krishna et al., 2016).<sup>3</sup> We manually create a list of single-word attributes for each relation, and only VG subjects that are matched with a specific attribute for more than a threshold number of times are recorded, in order to avoid noise in the dataset. The thresholds for color, material, and shape are 5, 2, and 1, respectively, chosen based on the availability of attributes of each relation in VG. VG attributes are filtered with the following steps: (1) attribute “Y colored / made / shaped” is treated as “Y”; (2) select only the last word for compound
attributes (e.g. treat "forest green" as "green"); (3) similar attributes are merged into a main attribute class (e.g. "maroon" and "crimson" become "red").
The above procedure produces a distribution over the set of attributes for each subject noun. From that distribution, a (subject, object) data instance is generated for each subject where the object is the attribute that associates with it the most. See the first three rows of Table 1 for examples.
Size Size is separated into size_smaller and size_larger, where the subject is a noun and the object is another noun that is smaller or larger, respectively, than the subject. To form the size dataset, we obtain a set of concrete nouns that appears in VG, which we manually classify into 5 size categories (tiny, small, medium, large, and huge). Typical objects in each category include pill, book, table, lion, mountain, respectively. We randomly pick two nouns from different categories to form a (subject, object) pair.
Visual Co-occurrence The visual co-occurrence dataset is generated in a similar way to the color, shape, and material datasets. Co-occurrence distribution is extracted from Visual Genome where two objects that occur in the same scene graph together for more than 8 times are recorded, and a (subject, object) instance is generated for each subject, where the object is the noun that co-occurs with the subject the most.
3.2 Data Grouping
Following Paik et al. (2021), we split the color, shape, and material datasets each into three groups: SINGLE, MULTI, and ANY. The SINGLE group is for subjects whose most common attribute covers more than  $80\%$  of the probability, e.g., the color of snow is almost always white. The MULTI group is defined as subjects not in the SINGLE group where more than  $90\%$  of the probability falls in the top 4 attribute classes, e.g., the color of a penguin in Fig. 1. The rest of the subjects are in the ANY group. Lower model performance for the SINGLE group would indicate the influence of reporting bias. For example, if the model is unable to correctly capture the distribution of the color of snow, it is likely because the color of snow has low probability of being reported in the training corpus, as people know it is white by default.
2See Appendix A.1 for more information on the object classes.
3Licensed under CC-BY 4.0.
Relation	# Classes	# (subj, obj) Pairs	Ex Template	Ex (subj, obj) Pair
color	12	2877	[subj] can be of color [obj]	(sky, blue)
shape	12	706	[subj] has shape [obj] .	(egg, oval)
material	18	1423	[subj] is made of [obj] .	(sofa, cloth)
size (smaller)	107	2000	[subj] is smaller than [obj] .	(book, elephant)
size (larger)	107	2000	[subj] is larger than [obj] .	(face, spoon)
co-occurrence	5939	2108	[subj] co-occurs with [obj] .	(fence, horse)
Table 1: Summary of the ViComTe dataset and the manual templates, including the number of classes, (subject, object) pairs, and an example pair for each relation.
Source	Group	Spearman ρ	# Subjs	Avg # Occ	Top5 # Occ	Btm5 # Occ	Acc@1
VG	All	64.3 ± 23.9	355	1252.6	64.6	308.6
SINGLE	62.2 ± 24.0	131	494.9	64.6	1181.6	80.2
MULTI	69.3 ± 20.7	136	1156.1	2062.2	347.0
ANY	58.4 ± 27.1	88	2529.6	8452.4	1213.4
Wikipedia	All	33.4 ± 30.6	302	543.6	1758.0	49.8
SINGLE	29.6 ± 29.9	110	352.2	345.8	35.0	35.5
MULTI	33.9 ± 30.9	119	500.8	1242.0	27.6
ANY	38.2 ± 30.4	73	902.0	3000.2	161.2
Table 2: Evaluation of ViComTe (mined from VG) and Wikipedia-mined color datasets by comparing with the human-annotated dataset CoDa. Reported are the average Spearman correlation  $(\times 100)$ , number of common subjects, average number of occurrences of the common subjects, average number of occurrences of subjects with top- and bottom-5 Spearman correlations, and the percentage of top-1 attributes being matched for the single group. ViComTe has higher correlations with human annotations.
3.3 Templates
In order to elicit model response and extract target objects and distributions from text, we manually design a set of templates for each relation. There are 7 templates for color, shape, and material each, 8 for size, and 4 for visual co-occurrence. See Table 1 for example templates.
3.4 Wikipedia Data
In order to compare text-based and visually-grounded data, we mine the color, shape, and material datasets from Wikipedia data, which is typically used in model pretraining. To mine these text-based datasets, we combine the sets of subjects in VG, take the manual list of attributes as objects again, and extract (subject, object) pairs if the pair matches any of the pre-defined templates. In Section 3.5 we will show the advantages of the VG-mined dataset over this text-based dataset.
3.5 Dataset Evaluation
To ensure the validity of ViComTe, we compare our color dataset with the human-annotated CoDa dataset (Paik et al., 2021), which we assume is close to real-world color distributions and has minimal reporting bias. We see a reasonably strong correlation with CoDa, indicating that the ViComTe dataset is a good and cost-effective approximation to human annotations.
Metrics We report the Spearman's rank-order correlation between the two distributions in comparison, averaged across all subjects. The Spearman correlation is used instead of the Pearson correlation since for our purpose the rank of the object distributions is more important than the exact values, which may change due to data variability. The top-1 accuracy (Acc@1) is the percentage of the objects with the highest probability in the source distributions matching those in the target distributions. These two metrics are also used in later sections when evaluating model distributions.
Analysis Table 2 shows the detailed results of the evaluation of the ViComTe and Wikipedia color datasets by comparing with the human-annotated dataset, CoDa. We can see that ViComTe has much higher Spearman correlation with CoDa, as well as substantially higher top-1 accuracy for the SINGLE group. The correlation is expected to be low for the ANY group, because objects in the ANY group can have many possible colors.
Reporting bias is present in both datasets, as the average number of occurrences of SINGLE group subjects are much fewer than that of the MULTI and ANY group subjects. Counter-intuitively, for ViComTe, the highly-correlated SINGLE group subjects have fewer average occurrences than the ones with low correlations. This is contrary to our expectation that more frequent objects would better
reflect the human-perceived distribution and can be explained by SINGLE subjects being easier to represent even without a large amount of data.
One example where the Wikipedia distribution diverges from the CoDa distribution is "penguin", whose most likely color in CoDa is black, followed by white and gray; however, its top color in Wikipedia is blue, because "blue penguin" is a specific species with an entry in Wikipedia, even if it is not as common as black and white penguins. One example where the VG distributions diverge from CoDa is "mouse", because in VG, most occurrences of "mouse" are computer mice, which are most commonly black, whereas when asked about the word "mouse", human annotators typically think about the animal, so that the most likely colors in CoDa are white and gray.[4]
3.6 Dataset splits
Each of the color, shape, material, size, and co-occurrence datasets is split into  $80\%$  training data and  $20\%$  test data. All evaluation metrics are reported on the test set. The training set is used for the logistic regression and the soft prompt tuning algorithm (Section 4.2).
4 Probing Visual Commonsense
We examine 7 pretrained transformer-based models and 2 variations of them, trained on a variety of data. BERT (Devlin et al., 2019), ALBERT (Lan et al., 2020), and RoBERTa (Liu et al., 2019) are trained on text only using a masked language modeling objective (MLM). Oscar (Li et al., 2020) is a vision-language model based on the BERT architecture, trained with an combined MLM and contrastive loss on text-image pairs. VisualBERT (Li et al., 2019) is another vision-language model based on BERT that learns joint representation of images and text. Tan and Bansal (2020) introduce the "vokenization" method, which aligns language tokens to their related images, mitigating the shortcomings of models trained on visually-grounded datasets in text-only tasks. Since our task is purely text-based, we also experiment with a pretrained vokenization model (BERT + VLM on Wiki). Finally, we use representations from CLIP (ViT-B/32) (Radford et al., 2021), which is trained with a contrastive image-caption matching loss.
Distilled Oscar As our experiments involve exclusively textual inputs, we develop a knowledge-distilled version of Oscar ("Distilled") which corrects for the lack of image input in our task. Knowledge distillation (Hinton et al., 2015; Sanh et al., 2019) is the process of transferring knowledge from one model to another, where the student model is trained to produce the output of the teacher model. Here, we use Oscar as the teacher and BERT as the student. The training data is part of the Oscar pretraining corpus: COCO (Lin et al., 2014), Flickr30k (Young et al., 2014), and GQA (Hudson and Manning, 2019), and the Distilled Oscar model has access to the text data only. We use the Kullback-Leibler loss to measure the divergence between the output logits of BERT and Oscar, and optimize the pretrained BERT on that loss to match the outputs of Oscar. Configurable parameters are set the same as for Oscar pretraining.
CaptionBERT Since VL models are trained largely on caption data, it could be that the differences between a text-only model and a VL model come not from a difference in modalities – text vs. images and text – but from a difference in domain – webtext vs. image captions. In order to disentangle the effects of the domain difference from those of visual inputs, we train a BERT model from scratch (“CaptionBERT”) on Oscar’s caption-based text data (the same data as for the Distilled model). If CaptionBERT, which does not have exposure to visual inputs, performs better than BERT and similarly to VL models (which are trained with visual inputs), it would suggest that the training domain matters more than the modality. If, on the other hand, CaptionBERT performs worse than VL models, it would highlight the importance of modality.
4.2 Evaluation Methods
We compare the visual commonsense abilities of pretrained unimodal and multimodal models. Given a list of prompts and a subject word, each model outputs the distribution of the target word. Following Paik et al. (2021), we apply zero-shot probes to models that are trained on a language modeling objective, and conduct representation probes for those that are not. We report the prediction accuracy and the Spearman correlation of the output distribution with the true distribution.
We use models trained with an MLM objective (BERT, Distilled, etc) directly for zero-shot predic
4 Additional examples are provided in Appendix A.3.
tion of masked tokens. $^{5}$  For Oscar we add a word-prediction head on top of it. The results across templates are aggregated in two modes. In the "best template" mode, for each example, the highest Spearman correlation among all templates is reported, and the top-1 result is regarded as correct if the true target object is the same as the top-1 result of any of the templates. In the "average template" mode, the output distribution is the mean of the distributions across all templates.
Since CLIP is not trained on a token-prediction objective, we implement logistic regression on top of the frozen encoder output, to predict the target attribute or object. The input is each of the templates with the subject [X] filled with an input in the dataset. Like Paik et al. (2021), to give the model ample chance of success, we take the template that results in the best test accuracy score, report that accuracy and the Spearman correlation associated with that template. For the classification head, we use the Scikit-Learn implementation of Logistic Regression (random_state=0, C=0.316, max_iter=2000) (Pedregosa et al., 2011).
Soft prompt tuning In order to overcome the limitation of self-designed prompts, we incorporate prompt tuning technique that learns soft prompts by gradient descent, from Qin and Eisner (2021). The algorithm minimizes the log loss:
$$
\sum_ {(x, y) \in E _ {r}} - \log \sum_ {\mathbf {t} \in T _ {r}} p (y | \mathbf {t}, x)
$$
for a set of example pairs  $E_r$  and template set  $T_r$ .
4.3 Size Evaluation
The size dataset differs from the other datasets in that we use relative sizes (X is larger/smaller than Y), as absolute size information is hard to obtain. Thus, we use two evaluation strategies for size.
Rank partition First, as in the previous prediction task, given a template such as "[X] is larger than [Y]" and an object [X], we ask the model to predict the distribution of [Y], taking only the distribution  $D$  of nouns in the size dataset. For the current object [X], we take the nouns in size categories that are smaller than the category of [X] ( $N_{sm}$ ), and those that are in larger categories ( $N_{lg}$ ).
Let the length of  $N_{sm}$  be  $m$  and the length of  $N_{lg}$  be  $n$ . Then for the "larger" templates, we compute the average percentage of overlap between the top  $n$  objects in  $D$  and  $N_{lg}$  and that between the bottom  $m$  objects in  $D$  and and  $N_{sm}$ . For the "smaller" templates, the "top" and "bottom" are reversed.
Adjective projection The second approach follows that of van Paridon et al. (2021), which projects the word to be evaluated onto an adjective scale. In this case, we compute the word embeddings of the adjectives "small" and "large" and the nouns from models, so the scale is large - small and the projection is calculated by cosine similarity. For instance, for the example noun "bear", the projection score is given by:
$$
\cos \_ s i m (\overrightarrow {\text {l a r g e}} - \overrightarrow {\text {s m a l l}}, \overrightarrow {\text {b e a r}})
$$
With good word embeddings, larger nouns are expected to have higher projection scores. The validity of the adjective scales from word representations is shown by Kim and de Marneffe (2013).
4.4 Measuring Model Reporting Bias
We measure the reporting bias of our models by comparing model performance on datasets with different levels of reporting bias and on the SINGLE, MULTI, ANY groups of the ViComTe dataset.
We assume that CoDa contains no reporting bias, in which case we can interpret Table 2 as showing that ViComTe contains a relatively small amount of it, and Wikipedia contains a relatively large amount. Thus, a larger correlation of model outputs with ViComTe and a smaller one with Wikipedia would indicate less model reporting bias.