SlowGuess/NAACL_2022_OCR huggingface dataset:
Also, since the SINGLE group subjects are those whose attribute distribution concentrates on a single attribute, these subject-attribute pairs are less likely to be reported in text corpora or even image annotations. Therefore, lower model correlation on the SINGLE group than the MULTI and the ANY groups would be a sign of model reporting bias.
The experimental results show that multimodal models outperform text-only models, suggesting their advantage in capturing visual commonsense. However, all models are subject to the influence of reporting bias, as they correlate better with the distributions from Wikipedia than those from CoDa
5For the target words that contain more than one subword tokens, we use the first token as the target. 6https://github.com/hiaoxui/ soft-prompts
Tune	Model	Color	Shape	Material	Cooccur
Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ
No	BERTb	26.1 ± 31.0*	11.7	38.7 ± 15.1	6.7	33.7 ± 19.6	30.0	4.7 ± 3.5
Oscarb	26.4 ± 30.7*	24.0	45.9 ± 14.1	53.0	38.6 ± 17.5	43.3	9.8 ± 6.9
Distilled	34.8 ± 27.3	27.5	46.2 ± 14.2	37.3	36.1 ± 20.2	37.7	10.1 ± 7.5
BERTl	37.6 ± 30.3	30.3	42.7 ± 17.1	28.4	36.6 ± 19.1	35.7	5.2 ± 3.8
Oscarl	31.8 ± 28.3	17.1	40.0 ± 16.9	38.1	39.2 ± 17.1	40.5	9.7 ± 6.7
Yes	BERTb	48.0 ± 22.9	47.4	49.2 ± 12.7*	76.1	41.2 ± 15.3	45.2	11.3 ± 7.9
Oscarb	58.1 ± 21.1	67.9	50.4 ± 11.5*	81.3	45.3 ± 14.3	66.2	12.7 ± 9.3
Distilled	57.1 ± 21.9	64.6	50.5 ± 12.3	82.8	45.4 ± 14.8	66.2	13.0 ± 10.1
BERTl	37.6 ± 30.3	30.3	49.2 ± 12.6	78.4	43.7 ± 15.1	53.3	11.4 ± 8.0
Oscarl	57.6 ± 21.6	65.3	50.1 ± 12.2	81.3	45.2 ± 15.2	65.8	12.8 ± 9.6
Table 3: Spearman correlation and top-1 accuracy (both × 100) of zero shot probing, before and after soft prompt tuning ("N" and "Y" for the "Tune" column). This is the "average template" case where the output distribution is the mean of distributions across all templates. The Spearman correlation reported is the mean across all subjects ± standard deviation, comparing the output distribution and the Visual Genome distribution. The subscripts  $b$  and  $l$  indicate the size of the model, and Distilled is the BERT model after distilling from Oscar. Asterisk indicates where there is no significant difference between  $\mathrm{BERT}_b$  and  $\mathrm{Oscar}_b$  (t-test p-value > 0.05).
and ViComTe. Prompt tuning and knowledge distillation substantially enhance model performance, while increasing model size does not.
5.1 Results with MLM Objective
Color, Shape, Material The resulting model performance for the "average template" mode is shown in Table 3. Prompt tuning is done in this mode only. Note that because the top-1 accuracy is taken among all possible classes of each relation, it should be interpreted together with the number of classes (Table 1).
We can see from Table 3 that Oscar does better than BERT in almost all cases. Significant difference between Oscar (base) and BERT (base) is seen in most cases. Also, after soft prompt tuning, both the Spearman correlation and the accuracy substantially improved. Although there is considerable variation of the Spearman correlations, we find consistent improvement per example with both prompt tuning and multimodal pretraining (Appendix A.2).
Table 3 also shows that knowledge distillation helps improve the performance of BERT in all cases, and the distilled model can sometimes even outperform the teacher model, Oscar. Moreover, the large version of each model does not always outperform its base counterpart, suggesting that increasing the size of the model does not enhance the model's ability to understand visual commonsense. Instead, training with visually grounded data does.
Fig. 2 illustrates the Spearman correlations of different models with the color distributions from CoDa, ViComTe and Wikipedia, under the "best template" mode. All models correlate moderately
Figure 2: Spearman correlations  $(\times 100)$  for color, under the "best template" case, for base models on CoDa, VG, and Wikipedia. While all models correlate the best with Wikipedia, BERT is the most biased.
with all three datasets, with the highest correlations to Wikipedia, indicating text-based reporting bias in all model types. BERT has the largest correlation gap between Wikipedia and CoDa, whereas the visually-grounded models have smaller gaps, indicating less reporting bias in VL models.
Visual Co-occurrence Table 3 also contains the results on visual co-occurrence before and after prompt tuning. Only the Spearman correlations are reported, because the top-1 accuracy is meaningless due to the large number of possible co-occurring objects with any noun.
Before prompt tuning, BERT has small Spearman correlations, suggesting that it may contain little knowledge about the visual co-occurrence relationship. Oscar demonstrates more such knowledge under the zero-shot setting. After prompt tuning, all model performances improve.
7 Appendix A.2 contains further details.
Model	Color	Shape	Material	Co-occur
Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ
BERTb	48.0 ± 21.6	51.4	53.2 ± 13.4	78.4	41.3 ± 15.6	51.1	30.2 ± 11.7
Oscarb	52.5 ± 20.8	63.1	54.4 ± 14.8	80.6	43.2 ± 14.4	63.0	31.2 ± 12.1
CLIP	51.9 ± 20.8	63.8	54.5 ± 13.9	79.9	42.9 ± 15.0	63.0	31.3 ± 11.6
Table 4: Spearman correlation and top-1 accuracy (both  $\times 100$ ) with a logistic regression head on model encoder outputs. Oscar and CLIP have comparable performance, both slightly better than BERT.
Group	Model	Color	Shape	Material
Spearman ρ	Acc@1	Spearman ρ	Acc@1	Spearman ρ	Acc@1
SINGLE	BERTb	36.8 ± 19.0	54.8	48.3 ± 12.3	83.0	35.9 ± 14.3	51.6
Oscarb	39.9 ± 15.3	60.3	49.3 ± 11.6	87.0	38.5 ± 12.8	65.1
CLIP	41.0 ± 15.2	66.3	49.2 ± 14.5	90.0	38.1 ± 12.8	64.1
MULTI	BERTb	49.7 ± 21.2	42.3	65.9 ± 16.9	59.5	53.8 ± 16.2	51.3
Oscarb	51.2 ± 19.9	50.6	65.2 ± 17.4	64.9	56.2 ± 13.0	53.9
CLIP	50.5 ± 21.1	55.4	64.6 ± 18.9	67.6	56.2 ± 14.3	59.2
ANY	BERTb	56.5 ± 19.5	46.1	100.0 ± 0	-	58.7 ± 15.2	35.7
Oscarb	62.5 ± 18.9	58.4	100.0 ± 0	-	60.4 ± 17.1	35.7
CLIP	60.3 ± 18.2	55.8	100.0 ± 0	-	63.5 ± 20.5	21.4
5.2 Results with Classification Head
Table 4 shows the results of BERT, CLIP, and Oscar when topped with a classification head. We observe that Oscar and CLIP achieve similar performance and both outperform BERT. Note that, while Visual Genome is part of Oscar's pretraining corpus and one might suspect that that gives it an advantage, CLIP is trained on a large corpus from web search that is unrelated to Visual Genome. Therefore, we can conclude that multimodal models pretrained on both images and text outperform text-only models.
Table 5 breaks down the results in Table 4 into three subject groups. Oscar and CLIP outperform BERT in almost all cases. The top-1 accuracy is higher for the SINGLE group than for the MULTI and ANY groups, perhaps because the SINGLE group subjects have only one most likely target attribute, which may be easier to predict. Note that the Spearman correlations for all three models become higher from group SINGLE to MULTI to ANY. Paik et al. (2021) argue that higher correlation for the ANY and MULTI groups is a sign of model reporting bias, as objects in those two groups are more often reported. Thus, the results here indicate that reporting bias is still present in multimodal models.
5.3 Results: Size Relation
Table 6 shows results of the rank partition method (Section 4.3), before and after prompt tuning. Sur
Table 5: Per-group Spearman correlation and top-1 accuracy (both × 100) with a logistic regression head on model encoder outputs. Note that the ANY group for shape only has one example, so the accuracy is less meaningful and is omitted. All models have higher correlations in the MULTI and ANY groups than the SINGLE group, which is a sign of reporting bias.
Tune	Model	Larger	Smaller
N	BERTb	80.0	67.1
Oscarb	79.5	67.7
Distilled	84.6	60.7
BERTl	80.9	66.1
Oscarl	79.4	70.7
Y	BERTb	69.9	55.7
Oscarb	70.6	57.3
Distilled	70.6	57.3
BERTl	70.0	55.7
Oscarl	70.6	57.3
Table 6: Percent correct for size relation, for "larger" and "smaller" templates, before and after soft prompt tuning. Interestingly, tuning does not help with size.
prisingly, prompt tuning does not help in this case. Moreover, the performance for the "larger" templates is higher than that of the "smaller" templates, suggesting that the models contain inherent preference towards the "larger" templates.
Fig. 3 shows the results of the adjective projection method. For BERT and Oscar, we use the average embedding of the subword tokens of the nouns projected onto that of the adjectives "large" and "small". For CLIP, we take the textual encoder outputs as the embeddings, resulting in a different score range from that of BERT and Oscar. The results show the following trend: larger objects are projected onto the "large" end of the spectrum, although the trend is sometimes broken towards the "huge" end. This may be due to the "huge" group
8 Appendix A.2 contains per-object plot for BERT vs Oscar.
Figure 3: The size projection scores, where the x-axis indicates the object groups. Outliers are omitted. All three models perform reasonably well, as larger objects have higher cosine similarities in general.
including nouns such as "pool" and "house" which can be modified by a relative size indicator "small".
5.4 Analysis and Limitations
In Table 3, the accuracy of BERT for shape is particularly low (only  $6.7\%$ ), despite that shape has only 12 classes. We hypothesize that this is due to reporting bias on shape in the text corpora that BERT is trained on. This hypothesis is supported by mining sentences from Wikipedia that contain (noun, attribute) pairs, where we see that the relation shape has fewer number of occurrences than material and color (Appendix A.3).
We also investigate whether the advantage of the visually-grounded models over pure-language models comes from the domain difference between web corpora and image captions, or the presence of actual visual input. Although its teacher is trained with visual inputs, the Distilled model is trained only on captions data and its performance matches that of Oscar, so we hypothesize that grounded training data enhance models' ability to capture visual commonsense. The CaptionBERT results support the hypothesis in favor of domain difference, since it performs better than BERT in both CoDa and VG (Fig. 2). Nevertheless, the visual inputs also have an effect, as Oscar has a higher correlation than CaptionBERT on CoDa. Thus, it seems that both domain and modality affect the ultimate model performance.
Finally, although multimodal models show improvement on the task, sometimes the improvement is not significant and the resulting correlations are still weak. Further work is needed to enhance the visual commonsense abilities of the models and
mitigate reporting bias, and our datasets can serve as an evaluation method.
In this paper, we probe knowledge about visually salient properties from pretrained neural networks. We automatically extract dataset of five visual relations: color, shape, material, size, and co-occurrence, and show that our ViComTe dataset has a much higher correlation with human perception data for color than data mined from Wikipedia. We then apply several probing techniques and discover that visually-supervised models perform better than pure language models, which indicates that they can better capture such visual properties. Distilling the knowledge from a visually-supervised model into a pure language model results in comparable performance with the teacher model.
We also observe less reporting bias in both visually-grounded text (VG-mined datasets) than Wikipedia text and visually-grounded models (Oscar, DistilledOscar, VisualBERT, and CLIP) than pure language models. However, visually-grounded models are still subject to the influence of reporting bias, as seen in the per-group analysis, where both types of models perform better for the MULTI group than the SINGLE group.
We would like to thank the reviewers for their comments and suggestions. Chenyu Zhang is supported by the Pistritto Research Fellowship. Elias Stengel-Eskin is supported by an NSF Graduate Research Fellowship. Zhuowan Li is supported by NSF 1763705.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2019. Multimodal machine learning: A survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell., 41(2):423-443.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online. Association for Computational Linguistics.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: Universal image-text representation learning. In European Conference on Computer Vision (ECCV).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Stella Frank, Emanuele Bugliarello, and Desmond Elliott. 2021. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
Drew A. Hudson and Christopher D. Manning. 2019. GQA: A new dataset for compositional question answering over real-world images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (ICCV), page 6700-6709.
Joo-Kyung Kim and Marie-Catherine de Marneffe. 2013. Deriving adjectival scales from continuous space word representations. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1625-1630, Seattle, Washington, USA. Association for Computational Linguistics.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. 2016. Visual
Genome: Connecting language and vision using crowdsourced dense image annotations.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR).
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. In Arxiv.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision (ECCV).
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Annual Conference on Neural Information Processing Systems (NeurIPS).
Ken McRae, George S. Cree, Mark S. Seidenberg, and Chris McNorgan. 2005. Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37:547-559.
Ishan Misra, C. Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. 2016. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2930-2939.
Cory Paik, Stephane Aroca-Ouellette, Alessandro Roncone, and Katharina Kann. 2021. The world of an octopus: How reporting bias influences a language model's perception of color. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 823-835, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jeroen van Paridon, Qiawen Liu, and Gary Lupyan. 2021. How do blind people know that blue is cold? distributional semantics encode color-adjective associations. In Proceedings of the Annual Meeting of the Cognitive Science Society.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Ellen F. Prince. 1978. On the function of existential presupposition in discourse. In Chicago Linguistic Society (Vol. 14, pp. 362-376).
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page 5203-5212, Online. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML).
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
Carina Silberer, Vittorio Ferrari, and Mirella Lapata. 2013. Models of semantic representation with visual attributes. In ACL.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Hao Tan and Mohit Bansal. 2020. Vokenization: Improving language understanding with contextualized, visual-grounded supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2066-2080, Online. Association for Computational Linguistics.
Ramakrishna Vedantam, Xiao Lin, Tanmay Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Learning common sense through visual abstraction. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2542-2550.
Nathaniel Weir, Adam Poliak, and Benjamin Van Durme. 2020. On the existence of tacit assumptions in contextualized language models. Proceedings of the Annual Meeting of the Cognitive Science Society.
Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2021. CPT: colorful prompt tuning for pre-trained vision-language models. ArXiv, abs/2109.11797.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. In Transactions of the Association for Computational Linguistics, volume 2, pages 67-78.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Rowan Zellers, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. 2021. SIGLeT: Language grounding through neuro-symbolic interaction in a 3D world. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2040-2050.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. VinVL: Making visual representations matter in vision-language models. In 2021 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2021. Learning to prompt for vision-language models. arXiv preprint arXiv:2109.01134.
Xi Zhu, Zhendong Mao, Chunxiao Liu, Peng Zhang, Bin Wang, and Yongdong Zhang. 2020. Overcoming language priors with self-supervised learning for visual question answering. In International Joint Conference on Artificial Intelligence (IJCAI), page 1083-1089.
A.1 List of Objects
Table 7 shows the list of all possible attributes for relations color, shape, and material. Table 8 shows the list of objects in the five categories of relation size. Visual co-occurrence has a large number of objects that are not listed here for space reasons.
Relation	Classes
Color	black, blue (aquac, azure, cyan, indigo, navy), brown (khaki, tan), gray (grey), green (turquoise), orange (amber), pink (magenta), purple (lavender, violet), red (burgundy, crimson, maroon, scarlet), silver, white (beige), yellow (blond, gold, golden)
Shape	cross, heart, octagon, oval, polygon (heptagon, hexagon, pentagon), rectangle, rhombus (diamond), round (circle), semicircle, square, star, triangle
Material	bronze (copper), ceramic, cloth, concrete, cotton, denim, glass, gold, iron, jade, leather, metal, paper, plastic, rubber, stone (cobblestone, slate), tin (pewter), wood (wooden)
Table 7: List of all objects for relation color, shape, and material. Inside the parentheses are the attributes that are grouped into the object class.
Size	Objects
Tiny	ant, leaf, earring, candle, lip, ear, eye, nose, pebble, shrimp, pendant, spoon, dirt, pill, bee
Small	bird, tomato, pizza, purse, bowl, cup, mug, tape, plate, potato, bottle, faucet, pot, knob, dish, book, laptop, menu, flower, pillow, clock, teapot, lobster, duck, balloon, helmet, hand, face, lemon, microphone, foot, towel, shoe
Medium	human, door, dog, cat, window, lamp, chair, tire, tv, table, desk, sink, guitar, bicycle, umbrella, printer, scooter, pumpkin, monitor, bag, coat, vase, deer, horse, kite
Large	elephant, car, tree, suv, pillar, stairway, bed, minivan, fireplace, bus, boat, cheetah, wall, balcony, bear, lion
Huge	building, airplane, plane, clocktower, tower, earth, pool, mountain, sky, road, house, hotel, tank, town, city, dinosaur, whale, school
Table 8: List of objects in five size categories.
A.2 Additional Probing
Best template mode Table 9 contains zero-shot results under the "best template" mode, for BERT (base), Oscar (base), BERT distilled from Oscar, RoBERTa (base), ALBERT (base), Vokenization, and VisualBERT (base). These results demonstrate
similar trends as the ones in the "average template" mode.
Figure 4: Spearman correlation per object class for BERT and CLIP with the logistic regression head, for color, shape, and material. The error margins are the standard deviations.
Per-object analysis Fig. 4 illustrates the fine-grained Spearman correlation  $\pm$  standard deviation per object group for BERT and CLIP.
Figure 5: The size projection scores from BERT and Oscar, where each point is one object. Cosine similarities are correlated between Oscar and BERT.
Size per-object Fig. 5 shows how the per-object projection scores on the size spectrum from BERT and Oscar are correlated.