Linearly Mapping from Image to Text Space
Authors: Jack Merullo
Louis Castricato
Carsten Eickhoff
Ellie Pavlick
Venue: ICLR 2023 poster (2023)
Keywords: representation learning
deep learning
grounded language learning
nlp
dl
image
image captioning
language grounding
grounded

Abstract: The extent to which text-only language models (LMs)  learn to represent the physical, non-linguistic world is an open question. Prior work has shown that pretrained LMs can be taught to ``understand'' visual inputs when the models' parameters are updated on image captioning tasks. We test a stronger hypothesis: that the conceptual representations learned by text-only models are functionally equivalent (up to a linear transformation) to those learned by models trained on vision tasks. Specifically, we show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single linear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We compare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNET (lexical category information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property information to the language model (e.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode category information (e.g., distinguishing hippo vs.\ elephant) and thus perform significantly better on benchmark language-and-vision tasks. Our results indicate that LMs encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images.

Published as a conference paper at ICLR 2023
LINEARLY MAPPING FROM IMAGE TO TEXT SPACE
Jack Merullo, Louis Castricato, Carsten Eickhoff, Ellie Pavlick Department of Computer Science
Brown University
Providence, RI, USA
fjackmmerullo, louis_castricato, earter,t ott.::/.....aaaa.........aaaaaea
ABSTRACT
The extent to which text-only language models (LMs) learn to represent features of the non-linguistic world is an open question. Prior work has shown that pre- trained LMs can be taught to caption images when a vision model's parameters are optimized to encode images in the language space. We test stronger hy- pothesis: that the conceptual representations learned by frozen text-only models and vision-only models are similar enough that this can be achieved with lin- ear map. We show that the image representations from vision models can be transferred continuous prompts frozen LMs by training only single lin- ear projection. Using these to prompt the LM achieves competitive performance on captioning and visual question answering tasks compared to models that tune both the image encoder and text decoder (such as the MAGMA model). We com- pare three image encoders with increasing amounts of linguistic supervision seen during pretraining: BEIT (no linguistic information), NF-ResNED (lexical cate- gory information), and CLIP (full natural language descriptions). We find that all three encoders perform equally well at transferring visual property informa- tion to the language model (c.g., whether an animal is large or small), but that image encoders pretrained with linguistic supervision more saliently encode cat- egory information (e.g., distinguishing hippo vs. elephant) and thus perform sig- nificantly better benchmark language-and-vision tasks. Our results indicate that LMS encode conceptual information structurally similarly to vision-based models, even those that are solely trained on images. Code is available here: https:// htt///gtttu.. com jmeru 1o/1 mber
INTRODUCTION
Much recent work in NLP has revolved around studying the limits on representational capacity in- curred by training on form-only text data, as discussed in Bender & Koller (2020). Tied this argument is the idea that without explicit grounding, language models are not inclined to learn con- ceptual representations of language that reflect the rich conceptual knowledge that humans gain from interacting with the physical, non-linguistic world. Despite this, there have been remarkable find- ings in large language models' abilities to generalize to and reason about non-lingen longlingiittic phenomena (Timpouuelli al. 2021 Eichenberg et al. 2021 Li et al. [02 Patel & lickick] 2022). Thus, an open question in the field is to what extent (if at all) a language model trained on text-only data can learn aspects of the physical world. In this paper, test specific hypothesis about the relationship
frozen image encoder encodes an image feature map
linear projection is tuned to project from image space to text space
The image projections fed as soft prompts into generative LM
"A picture of dog skateboard"
Image Encoder
Linear Proj.
Text Decoder
We train linear projections from image representations into the input space of language model to produce captions describing images. We find that LMs can describe the contents of most image representations, but performance varies based on the type of image encoder used.
Figure 1:

---

Published as a conference paper at ICLR 2023
between language model and image encoder representations: that these conceptual representations can be approximately mapped to one another through a linear transformation. To do this, we train single linear layer to project from the representation space of images into the language space of generative LM without tuning any other model parameters, which we call L MBeR; Linearly Map- ping Between Representation spaces. That is, we linearly transform "soft prompts" -vector())))) in the embedding space that do not correspond to discrete language tokens (Lester et al., 2021). The weights of this linear projection are tuned for an image captioning task (illustrated in Figure D. We can then evaluate its performance on vision-language (VL) tasks at test time by exploring the text the LM generates. Because of the simplicity of the linear transformation, we would expect that if the conceptual representation spaces of the two models are structured sim- ilarly, this transfer will be successful and the LM will have little trouble describing the contents of images.
image representation into
We use three different image encoders with increasing levels of linguistic supervision in pretraining: BEIT (Bao al. 2021). Normalizer Free Resnet50 (NFRN50) (Brock et al. 2021). and CLIP (Rad-) Ford et al. 202T to train different projections into the LM. By linguistic supervision, we refer to the extent which the image encoder was exposed to language data during its pretraining, thus influ- encing the expected representational similarity between it and an LM. While CLIP was pretrained to align images with full natural language captions in shared image-text representation space, BEIT had no exposure to language and was trained by predicting the contents of masked out sections of images. NFRN50 falls somewhere between these extremes: having been pretrained on an image classification task for identifying the subject of an image over the set of classes in ImageNetIk Rus- sakovsky et al. 2013). Although there no natural language in this task, the pretraining objective encourages the model to map visual features along lexical categorical concepts (the image classes) derived from the WordNet hierarchy (Miller 1995).
We show that prompting an LM with any of the three image encoders effectively transfers semantic content the image that the LM describes with natural language. However, performance also appears proportional to the strength of the linguistic supervision the image encoder had. While CLIP and NFRN5O perform competitively with tuning the models freely (e.g.. Tsimpoukelli et al. (202I), Eichenberg al. (22II). BEIT appears to transfer mostly coarse-grained visual properties and struggles with encouraging the LM to generate exact lexical categories. We interpret this as evidence that models trained either language similar to each other, but that the exact degree of similarity depends on the type of supervision the image encoder receives. In summary, we show: (1) that visual semantic information can be linearly mapped to language models in the form of soft prompts without tuning any model parameters. (2) That this mapping allows generative models to describe images and answer questions about images at a level that is comparable to what is achieved by multimodal models which tune image and language representations jointly. And (3) by training our prompting pipeline with different image encoder backbones, we demonstrate that linguistic supervision in pretraining plays key role in concept formation in models and thus, the transferability of visual features from vision to text spaces.
vision data learn conceptual spaces that are structurally
RELATED WORK
Our approach takes inspiration from recent work in adapting pretrained language models for ac-
cepting representations of images as inputs. Particularly. the Frozen and MAGMA models Tsim-
poukelli et al. 2021 Eichenberg et al. 202T), as well as (ung et al. (2022;; Alayrac et al. (2022):
Mokady et al. 2021; Luo et al. (2022) y Lin et al. (202T): Zhai et al. (202). which show that pre-
trained image and text networks can be tuned together on an image captioning task and applied to downstream vision-langual (VL.) tasks. These approaches either fine-tune the pretrained models, or train non-linear MLP projection/fusion networks between modalities, making interpretation of the representations difficult compared to our approach. SScialom et al. (2020) show a learned linear transformation is sufficient for BERT to encode image region representations which are then fed to text decoder to generate questions about the image, but it is not well understood what abstractions LMs are able to transfer from a transformation of this type, or if a text decoder can operate on linear transformations of visual encodings directly.
2

---

Published as a conference paper at ICLR 2023
Pretrained/from scratch LMs have typically been used in the past for image captioning applications in which an image representation is fed into the LM as input (Desai & Johnson 2021| Shen et al. 2021 Devlin et al.] 2013). Gui et al. (2022) : Yuan et al. pretrain vision-lann language models from scratch using image-caption data. ZZen et al. (022)) et al. (002); ; Wang et al. (2022) augment multimodal performance by feeding text prompts derived from VL. models into LM, in order to incorporate knowledge learned by LM training. These show LMs can interface with visual inputs described in text; our work questions whether the visual input can be fed directly into the LM, with- out bridging through language first. The success of aforementioned models on VL tasks indicates there is a representational similarity learned by text and image models independently, which we investigate in this paper.
Our work also highly related to the idea of model "stitching" <eenc Vedaldi 2015) in which two different models are attached at certain layer. L MBeR can be described as stitching the output of an image encoder to the input of an LMM in the form of soft prompts (Lester et al., (02T). Stitching offers distinct advantages in evaluating the representational similarity between two models, as de- scribed in Bansal et al. (202T), D. over other conventional methods like RSA and CKA (Kriegeskortc et al. 2008 Kornblith et al. 2019). For example, LiMBeR allows to show not just that CLIP en- codings are more similar to text encodings than BEIT representations, but that BEIT representations are nevertheless able to transfer visual property information the LM ($5.3)
There has been considerable interest in recent work in establishing if LMs model aspects of the non- linguistic world in order to model language. [Lu al. (2021) show that the weights of pretrained LM can generalize to tasks with different modalities. Hao et al. (0022) similarly show that LMs can act interfaces for multiple modalities. Li et al. (2021) show that models of entities and situations can be derived from contextual word representations. Patel & Pavlick (2022) show that very large LMs (GPT-3 scale (Brown et al., (020)) can learn in-context non non-linguisttic conceptual domains depicted in text. Our work differs from these in that we have an LM interface directly with non-text data without changing model weights and show that, although fundamentally different, the representation space of text-only LM shares non-trivial similarities to that of several vision-based models.
3 METHOD: LINEARLY MAPPING FROM IMAGE TO TEXT REPRESENTATIONS
While previous work has shown success in mapping images to language model soft prompts as a method for multimodal pretraining (e.g., Frozen, Magma; see Section 2 there have been no attempts to restrict the mechanism behind this mapping and understand how it works. Our basic approach is to train a single linear layer P to project from the hidden size h, of a pretrained image encoder into the input space e,. of a generative language model for an image captioning task. The projected inputs do not correspond to discrete language tokens, and can be thought of soft prompts (Lester et al. 2021) representing the image. For brevity, we refer to training P as Linearly Mapping Between Representation spaces (i.e., LiMBeR Our approach can also be viewed as paring down the method used in Tsimpoukelli et al. (2021) and Eichenberg et al. (2021), such that the only trained parameters reside in the projection P. By freezing the image encoder E and LM on either side of the projection, we can examine the similarities between the representation spaces of the two as a function of the ability of the LM to describe an image ínput or perform some task relating to it. We expect that, if a language model represents visual conceptual information structurally similarly to that learned by a vision encoder, then a simple linear transformation to the language space is all that is required to transfer visual features into the language model. Before describing the training procedure, we will describe the basic components of the model, and the variations we chose.
Language Model LM & Image Encoders We hypothesize that the conceptual representations learned by an LM are equivalent, up to linear transformation, to the representations from an image encoder E. The language model used is the 6 billion parameter decoder-only GPT-J model (Wang & Komatsuzzaii 2021). P is trained to project from h to the input space ez 4096 of the LM. We train several models with different E's s to determine the compatibility between encodings from
We avoid specifying images or text in our backronym because one could linearly map between any two representation spaces of any modalities (e.g. video-to-text text-to-text)
3

---

Published as a conference paper at ICLR 2023
E and the LM. We also test how the choice of E influences performance on this task, specifically, with regards to the degree of linguistic supervision E saw in pretraining, as described Section !
From E we extract an image encoding of dimensionality h representing the image. We then project that encoding to a e,, * sequence of soft prompts, which we hereafter refer to as image prompts. k is determined by the architecture of the E. For example, for consistency with the MAGMA model, we use the 12x12x3072d feature map before pooling from CLIP, which we flatten to k 12 * 12 144. The encoders experiment with are (1) CLIP RN50x16 (Radford et al. 2021). k = 144, hr = 3072. Because CLIP is trained to learn multimodal image-text embeddings. we expect that it will be easier for the model leam projection into language space than vision only encoder. (2) NFRN50 (Brock et al. 2021). k = 2, h4 = 2048. We train three variants using NF- Resnet50: one pretrained and frozen during caption training (NFRN50), one tuned during caption training (NFRN50 Tuned; note that the LM is still frozen), and one randomly initialized (NFRN50 Random). The NFRN5O models are pretrained on an image classification task on data that labeled according to WordNet hypo/hypernym structure. This signal trains the model to separate object classes according to these words. For this reason, we consider it to have indirect access to linguistic supervision. (3) BEIT- (Bao et al. 2021). k 196, h, 1024. BEIT is pretrained using a self-super masked visual token modeling task and does not have access to any labeled data which may give the model an inductive bias towards linguistic structure. We the 16-pixel patch version that was pretrained only on ImageNe122k. We additionally test two variants of this model: BEIT Random, randomly initialized, and BEIT FT. which was pretrained on the same task and then finetuned for image classification on the same dataset. We use this model to show that it is indeed the linguistic supervision of the pretraining objective which induces better performance in the captioning task.
3.1 TRAINING PROCEDURE
Following the MAGMA and Frozen models (Eichenberg al. 2021 Tsimpoukelli et al. 202 we train a projection on an image captioning task so that we can learn to align the representation spaces of E and the LM. All models are trained with the same basic hyperparameters and settings as described in the MAGMA paper (see Appendix [A] for details) on the Conceptual Captions 3M dataset (CC3M, Sharma al. (2088) for 15,000 training steps.
Baselines As baselines, we use NFRN50 Random, NFRN50 Tuned, and train our own instance of MAGMA Please note that NFRN50 Tuned is a stand-in for the Frozen model: it is architec- turally the same, but differs in that we use the hyperparameters used to train the MAGMA model. NFRN50 Random allows us to test the efficacy of L.eR when the image encoder backbone has not learned any useful visual features. The MAGMA we train uses the CLIP RN50x 16 image en- coder (Radford et al. 2021), GPT-J as the LM, and adapters in sequence in the attention blocks with a downsample factor of 4.
3.2 LIMITATIONS
Due to computational constraints, we did not control for the prompt length (k) for each image encoder. Tsimpoukelli al. (202I) experiment with small range for the value of k for the Frozen model and show that while there are some differences, k is mostly factor in hyperparameter tuning and should not strongly affect the comparison between models. We use much higher values of & for CLIP and BEIT, and this is therefore is not strongly controlled for in our study.
We consider LM runoff another potential confound. In some cases, if the LM recognizes and gen- erates a relevant word for one concept (e.g.. "the beach"", it might continue generating relevant information due to a strong linguistic prior for that info showing up (e.g.. "building a sandcastle"), giving the illusion it is recognizing every element in an image (even if never saw "the sandcastle"). Regardless, the scope of this problem is very limited, and across multiple large datasets our results show that recovery of any image information is still possible, even if the full and precise extent of which is impossible to know. We also include a 'blind' model in visual question answering analysis to further control for this.
4

---

Published as a conference paper at ICLR 2023
Image Captioning
giraffe in the lobby of the buildding
CLIP
J.P.orgaa
CLIP
tennis player in action
tennis player at the tennis toumament.
NFRN50
the giraffe the zoo.
NFRN50
tennis player during a tennis match.
BEIT
peacock the garden
BEIT
NFRN50
a man and woman in field of flowers
NFRN50 Random
new logo for the team
Random
Visual Question Answering
CLIP
He is surfing wave.
CLIP
A tennis racket
NFRN50
is surfing the waves.
NFRN50
tennis racket
He is jumping into the water.
BEIT
BEIT
A baseball bat.
NFRN50 Random
He swimming the pool.
NFRN50
tree
Random
Q:
What is the person doing? surfing
Q: What is the person holding?
A:
A: tennis racket
Figure 2: Curated examples of captioning and zero-shot VQA illustrating the ability of each model to transfer information to the LM without tuning either model. We use these examples also illustrate common failure modes for BEIT prompts of sometimes generating incorrect but conceptually related captions/answers.
PERFORMANCE ON VISION-LANGUACE TASKS
We first verify that image representations that are linearly projected into the input space of the LM carry semantic information about the content of the image that the LM can make sense of. Since we only tune a single projection between the image encoder and text decoder, the prompt tokens in the LM are equivalent to the image representation up to that linear transformation. If LMS are learning a conceptual space that reflects that of the non-linguistic, purely visually grounded space of the image encoder, the LM should be able to capture the image information and describe it in text.
Data We evaluate on image prompts generated by each image encoder multiple image captioning datasets: MSCOCO (Lin et al. 2014) and NoCaps (Agrawal et al.] 2019), as well as the VQA2 (Goyal et al., 2017) visual pnestion-answeree
dataset. Following convention from SimVLM and MAGMA, we input the prefix "A picture of" after every image to prompt the model. Like in previous work, find that this is favorable prompt which tends to increase performance.
Metrics For image captioning, we report CIDEr-D (Vedantam et al. 2013). CLIPScore, and Ref- CLIPScore (Hessel et al. 2 CIDEr-D rewards generating accurate words which are more likely to be visually informative, and CLIPScore can evaluate similarity between an image and caption without references, which helps us give credit for captions that vary greatly from the ground truth, but similar in semantic content (e.g. describing a pool as a lake). We report additional captioning metrics in Appendix B For visual question answering, we follow the few-shot procedure used in Eichenberg et al. (2021) in which we prompt the models with the "[image] Q: [q] A:" format. We take the first word of the generation and, like in the MAGMA paper, truncate to the length of the longest ground truth answer. We also use the normalization procedure and accuracy metric described in the VQA ?
Results Our main results can be seen in Table I As evidenced by comparing MAGMA and CLIP, and NFRN50 tuned and frozen, we find that there is relatively little benefit in training parameters in either encoder or decoder. Note that the MAGMA model we implemented is identical to the frozen CLIP model, with the exceptions that MAGMA tunes the image encoder and LM. On captioning and VQA tasks, performance of the jointly-tuner models (MAGMA, NFRN50 Tuned) is not con- sistently better, and is often worse, than just training the projection with frozen models. This trend persists across over 10 automatic captioning metrics, which are described in Appendix B Our results indicate that there is in fact a relationship between the linguistic supervision of the pretraining task
5

---

Published as a conference paper at ICLR 2023
Image Captioning
NoCaps
CIDEr-D
NoCaps (Al))
CoCo CIDEr-D
CoCo
In
Out
Near
All
CLIP-S
Ref-S
CLIP-S
Ref-S
ANFRN50 Tuned
20.9
30.8
25.3
27.3
66.5
72.5
35.3
69.7
74.8
MAGMA (released)
18.0
12.7
18.4
16.9
63.2
68.8
52.1
76.7
79.4
MAGMA (ours)
30.4
43.4
36.7
38.7
74.3
78.7
47.5
75.3
79.6
BETT Random
5.5
3.6
4.1
4.4
46.8
55.1
5.2
48.8
56.2
NERN5O Random
5.4
4.0
4.9
5.0
47.5
55.7
4.8
49.5
57.1
BEIT
20.3
16.3
18.9
18.9
62.0
69.1
22.3
63.6
70.0
NFRN50
21.3
31.2
26.9
28.5
65.6
71.8
36.2
68.9
74.1
BEIT FT.
38.5
48.8
43.1
45.3
73.0
78.1
51.0
74.2
78.9
@ &CLIP
34.3
48.4
41.6
43.9
74.7
79.4
54.9
76.2
80.4
VQA n-shots
2
Blind
20.60
35.11
36.17
36.99
NNFR550 Tuned
27.15
37.47
38.48
39.18
MMAGMA (ours)
24.62
39.27
40.58
41.51
MAGMA (reported)
32.7
40.2
42.5
43.8
NFRN50 Random
25.34
36.15
36.79
37.43
BEIT
24.92
34.35
34.70
31.72
NFRN50
27.63
37.51
38.58
39.17
CLIP
33.33
39.93
40.82
40.34
Table 1: Captioning Performance and Visual Question Answering (VQA) accuracy for all variations on model architecture and image encoders used. On captioning, we see consistent increasing trend in performance that correlates with increase in linguistic supervision. However BEIT (the only vision-only model), performs
above a randomly initialized NFRN50 model and is on par with the other models CLIPScore (CLIP-S) and RefCLIP Score (Ref-S) (Hessel et al., (021). We see that BEIT performs at the level of our random baselines on VQA, suggesting there is deficiency in complex visual-linguistic reasoning tasks
relating visual information
and performance on transferring to the LM. That is, CLIP outperforms NFRN50, which outperforms BEIT. To confirm this, we apply LiMBeR to BEIT model finetuned image classification (BEIT FT.), and find that this model improves performance drastically, even outperforming CLIP on No- Caps, and improving the importance of the linguistic supervision in the pretraining task, rather than perhaps architecture, the important factor for successful transfer.
BEIT on all metrics, including CLIP-Scom by 9-10 points. This suggests
Notably, we find that even vanilla BEIT, which has no linguistic supervision in pretraining, still transfers well the LM for captioning, far outperforming random NFRN50 across the board, which had no pretraining to learn visual features. We do find that BEIT captions using vaguer language, and/or semantically related-but-incorreed descriptions of objects (Figure 2 more examples in Ap- pendix B). We see this reflected in the CLIPScores of the captions as well, which reward semantic similanty rather than precise lexical overlap with a reference caption. BEIT captions score 62 and 63.6 for NoCaps and COCO respectively; of random NFRN50. Perhaps we see the greatest failure BETT prompts in the inability transfer details that the LM can to answer questions about images (At 4-shot VQA. BEIT scores 3 3..72% while a 'blind' LM with no image imput scores 36.99%). We hypothesize this is because BEIT rep- resentations do not encode visual information that corresponds well to lexical categories. In Section 5, we provide evidence favor of this hypothesis, and investigate the granularity of detail prompts from each frozen encoder transfer to the LMM
average only 4.5 points behind NFRN50 but 14.3 ahead
TRANSFER OF VISUAL CONCEPTS
Examining the conditions that cause image prompt succeed or fail to transfer to the LM can help us understand the differences between the text and image representation spaces. Doing so can also help understand why BEIT prompts perform so poorly for VQA despite performing decently for captioning. Section 5.1 we analyze the ability to accurately generate specific lexical categories in captions when they appear in images (e.g., mentioning "squirrel" when given picture of one).

---

Published as a conference paper at ICLR 2023
Recall and Palmerr Similarity Model
100
Recall Wu-Palmer Similarity each Vehicles
Recall W. Smilaity (Sryy)
krssso Random
Rocall Paltirr Similuityy (Gray)
NERROO fanden
BEIT
80
80
60
40
Environment
Figure 3: On average. recall of nouns in generated captions follows the standard pattern (CLIP>NERN5O- BEIT), However, judging by Wu-Palmer similarity, BEIT performs nearly the same or better than NFRN50 and CLIP on 4/5 of the noun categories. This indicates that although BEIT struggles to transfer the exact correct concept, it is transferring related one based on visual similarity. On the right we show this effect for individual vehicle words. BEIT may have never learned to distinguish the 'bus" concept, but the LM still understands to generate highly related concept, i.e., another vehicle. Average random Wu-Palmer similarity is around consistently.
Following that, in Section 5.3 we focus on mistakes the models make: when the LM generates a bad caption, does it generate caption that describes entities with similar visual properties? For example, a caption generated from an image of a "small", "woodland", and "furry" animal might mention the actual animal depicted (c.g., a squirrel); but does instead mention different but similar furry animal (e.g., rabbitj? We find that only linguistically informed image encoders (NFRN50, CLIP) tend strongly encode concepts aligning to lexical categories, but all pretrained models including BEIT encode property information approximately equally well, and far better than a randomly initialized image encoder baseline.
5.1 TRANSFER OF LEXICAL CATEGORICAL CONCEPTS
Using the COCO validation set, count the top 50 nouns, modifiers (e.g., adjectives), and relations (e.g., verbs, prepositional phrases) that appear in the ground truth captions and calculate how often they appear in the generated captions that were used to calculate the scores in Table
Metries We calculate the precision/recall/FI for each word, broken down along conceptual cate- gories. To test our hypothesis that BEIT transfers coarser information, also report the Wu-Palmer similarity (Wup) (Wu & Palmer the generated caption. The Wup score works by calculating the distance between the ground truth word and the generated word in the WordNet taxonomy, offering way to measure 'how close' word the correct answer.
1994) between the ground truth word and the most similar word in
Results Figure we show that BETT's recall for nouns in categories like 'people', 'environment', 'vehicles', and 'objects" is lower than NFRN50 or CLIP, but is comparable terms of Wup sim- iliarity in many categories. Unlike NFRN50 and CLIP's pretraining, BEIT's pretraining does not encourage learn conceptual differences between similar looking objects that different words. Compared to prompts from randomly initialized NFRN50, for which very few consistent patterns emerge, the LM can still extract the broad conceptual meaning behind BEIT prompts, as evidenced by high Wup similarity (and CLIPScore results in Table D. We interpret these results as supporting the hypothesis that BEIT prompts transfer conceptual information from the purely visual to purely text space, but only in terms of coarse-grained conceptual information corresponding to visual properties, not lexical categories. Our full analysis, including additional metrics and results for each individual word from the top 50 nouns, modifiers, and relations can be found in Appendix
5.2 PROBING
To rule out the possibility that BEIT representations encoding lexical concept information, but are merely unable to linearly transfer it to the LM due to representational differences, we train linear probes several datasets for image classification. We find that BEIT typically does not encode fine-

---

Published as a conference paper at ICLR 2023
grained information well as NFRN50 CLIP, though it far outperforms the randomly initialized NFRN50 baseline. We discuss training details and results in Appendix
5.3 TRANSFER OF COARSE-GRAIDG PERCEPTUAL CONCEPTS
To better understand what BEIT encodes, if not word category information, we further investigate where emrors arise, and how the structures of the embedding spaces for each frozen image encoder differ. For the sake of this analysis, we constrain the task generating captions for pictures of animals. The reason for this narrower scope is that the captions are easier to analyze: the caption describing picture an animal should virtually always mention the name of that animal, and the word used to describe the animal is mostly unambiguous.
Average Wu-Palmer Similarity over Wrong Predictions
Overlap of Mistaken Animal Properties w/ Ground Truth
of Shared Properties with GT Animal
100
B
Wu-Palmer Simlarity
80
0.8
0.6
20-
000
NFRN50 Random
BEIT
NFRN5S
CLP
NERN50 Random BEIT
NFRN50
CLIP
Wu-Palmer Similarity for captions in which the models don't mention the animal show that BEIT, NFRN50, CLIP are all similarly close, meaning even they predict the wrong animal. is on average very taxonomically similar. Right: When the model mistakes one animal for another the dataset, how similar are the AWA properties for the truc animal and the one it mistakes it most for? The average number of overlapping properties show that animals predicted from BEIT at least similar the real animal as NFRN50 and CL.P. Median shown the solid orange line while the dashed green line shows the mean.
(a) Left:
NFRN50 Random
BEIT
NFRN50
CLIP
Land
Land
Ocear
(b) UMAP projections of AWA images:Whit NERN50 and CLIP cluster tightly along lexical categories (color coded by animal), BEIT clusters the most distinctly along animals that live water/the occan; the randomly initialized NFRN50 mostly randomly overlap one cluster.
Figure 4
Data For this task we use the Animals With Attributes 2 (AWA) dataset (Xian et al. 2019) which contains 37k total images covering 50 animal classes. Each animal class also comes with annota- tions for 85 properties describing the animals (e.g., 'claws', 'stripes', 'jungle"), which allow us to analyze if prompts from certain encoders consistently make mistakes along any these dimensions.
Metrics When an image prompt produces caption, can measure the similarity of any animals mentioned to the WordNet synset of the ground truth animal label. We can also measure similarity using the annotated properties provided by the AWA dataset. For a given animal (c.g.. "squirrel"), we can look at the other animal the dataset that it is most often mistaken for (c.g., "rabbit") and compare the proportion of properties that they share.
Results We generate captions for each image using prompts from each frozen image encoder. We consider caption to be 'correct' if it contains the name of the animal the image depicts. CLIP and NERN50 are correct most often: 59% and 43% the time respectively. BEIT and the randomly initialized NFRN50 only achieve 13% and 0.4% accuracy, respectively. This aligns with previous
8

---

Published as a conference paper at ICLR 2023
observations that BEIT struggles with encoding fine- fine-grained lexical level concepts. By looking failure cases for each model, we can establish whether each model is predicting the presence of a similar animal or not. In Figure 4al we show that when captions generated from each model mistake one animal for another, the mistaken animals are highly similar to the ground truth animal when measuring both Wu-Palmer similarity (Averages: BEIT: 0.8, NFRN50: 0.81, CLIP: 0.8) and overlap of AWA properties (Averages: BEIT: 0.62, NFRN50: 0.68, CLIP: 0.59). Although BEIT prompts do not transfer the exact animal concept to the LM, the coarse grained perceptual information is transferred and 'understood' by the LM. In Figure 4b we create UMAP projections of the encodings for each image in AWA and indeed find that NFRN50 and CLIP cluster according to tight lexical categories (the animal types), BEIT clusters most tightly by perceptual features, such as habitat, having flippers, etc.
6 DISCUSSION & FUTURE WORK
We connect image representations as inputs to an LM with an extremely simple transformation: a linear projection. We interpret the success of generating text relevant an image with this approach as indicative of an underexplored representational similarity between language and vision repre- sentations. Depending on the linguistic guidance in the pretraining task used pretrain the image encoder, we see varying performance. Using a vision-only encoder (BEIT) leads to generations from the LM that are often incorrect but close under measures of perceptual relatedness. Unless finetuned with image classification BEIT has no inductive bias in pretraining to distinguish concepts that we might normally distinguish with language, especially when they are perceptually very similar. The fact that only image encoders trained with linguistic supervision can do this suggests interesting future work on the role of language category formation. Despite strong performance with linear transformation, the representation spaces of these models seem to contain differences that cannot be approximated in the language space, Multimodal models, ideally, will learn richer representations by taking advantage of these differences. It is useful to think about how current multimodal pretrain- ing objectives succeed and/or fail at doing this. LiMBeR can serve as a strong baseline for future multimodal models, as it provides point of comparison for minimal mapping between vision and language.
We L. M.eR useful tool for understanding how representations trained from different modal- ities can be similar or different. For concepts that are represented similarly, can we take advantage of this fact to reduce the amount of data required for learning good text representations? Where they are different, can multimodal models learn richer representations by incorporating information from both? For example, vision data may help with reporting bias in text corpora (Paik al. 2021). Answering these questions can help us understand the limits of text-only pretraining, as well as how to better ground LMs to non-linguistic data.
7 CONCLUSION
In this paper, we test how similar pretrained image and text representations are by training a lin- map to project image representations into the input space of a language model (LM), such that the LM can accurately describe the contents of the images. We show that models trained through LiMBeR (Linearly Mapping Between Representation spaces) are competitive image captioning and visual question answering benchmarks with similar models like MAGMA that tune both image and text networks. However, we also find that such transfer is highly dependant the amount of linguistic supervision the image encoder backbone had during its pretraining phase. BEIT, which vision-only image encoder underperforms compared to a Resnet model trained on image clas- sification. which in turn underperforms compared to CLIP, which was pretrained with natural lan- guage captions. We explore what conceptual information transfers successfully, and find through analysis of generated text, clustering, and probing that the representational similarity between LMS and vision-only image representations is mostly restricted to coarse-graines concepts of perceptual features, while linguistically supervised vision models can transfer lexical concepts. Our findings indicate that LMs and vision models learn conceptually similar representation spaces, such that a minimal linear transformation is an adequate approximation for transferring information about an image. The extent of this representational similarity is not well understood and is an interesting direction for future work.

---

Published as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
We are committed to making all of our results reproducible. Code for training all models, as well weights of the linear projections limber We also release the weights of the linear projections that were trained for LiMBeR. Because we froze the image and text models attached to the projection, the weights can be used to quickly reproduce our results with the corresponding off-the-shelf pretrained models with no other tuning necessary. We the default data splits for all datasets used for all tasks that require generation from the LM in our codebase as well. For the AWA tasks that require matching Wordnet synsets, we document the exact animal synsets that we used as the "gronnd truth' for the animal labels in Appendix D Table 6
be found here: [bttps://g giggiittgn thub. com/ jmeru 1o/
used and release the random seeds
9 ACKNOWLEDGMENTS
We would like to thank Aaron Traylor and Nihal Nayak for thoughtful discussions and feedback on this work, well Stability for donating compute resources for model training. This research is supported in part by ODNI and IARPA via the BETTER program views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government.
19000000040))) The
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8948-8957, 2019.
Jean-Bac Mann-Baptitte Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204.1411 2022.
Yamini Bansal, Prectum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems, 34:225-236, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185-5199 Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020.aa maiin..43.. URL http:::/aclaatholloo.o orq/ 2020.acl-10 in. 463.
Andy Brock, Soham De, Samuel Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization. In International Conference on Machine Learning, pp. 1059-1071. PMLR, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
11162- 11173, 2021.
Jacob Devlin, Hao Cheng, Hao Fang, Saurabh Gupta, Li Deng, Xiaodong He, Geoffrey Zweig, and Margaret Mitchell. Language models for image captioning: The quirks and what works. In Proceedings of the 53rd Annual Meeting the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers),
pp. 100-105, 2015.
10

---

Published as a conference paper at ICLR 2023
Constantin Eichenberg, Sid Black, Samuel Weinbach, Letitia Parealabescu, and Anette Frank. Magma- multimodal augmentation of generative models through adapter- -aased finetuning. ArXiv, abs/2112.05253, 2021.
Yash Goyal, Tejas Khot, Douglas Summers-Ster Stayy Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering, In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
Liangke Gui, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. Training vision- language transformers from captions alone. arXiv preprint arXiv:2205.0922 2022.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neu- ral network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 3519-3529. PMLR, 09-15 Jun 2019. URL https:
//proceedings http:/eedwn.g..ll..pes//////////
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, pp. 4, 2008.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Lan- guage Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November Associatio for Computational Linguisties. doi: 10. 18653vv 1021665311/2221.com w2022/.cmmppppmmii.....mmmm 2433. URL 22tp://hho.oooy..o22211111111.1 emnlp ma iicainn.o 243
Belinda Z Li. Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1813-1827, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Xudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang, Devi Parikh, and Lorenzo Torresani. Vx2text: End-to-end learning of video-based text generation from multimodal inputs. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7005-701111 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2018.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as univer- sal computation engines. CoRR, abs/2103.0522 2021. URL htl.httss///eexii..rre//
2103. 05247
Ziyang Luo, Yadong Xi, Rongsheng Zhang, and Jing Ma. A frustratingly simple approach for end- to-end image captioning, 2022. URL httptttts:///arat wttp://wwwww.oa/p////////// 12723
11

---

Published as a conference paper at ICLR 2023
George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39 41, 1995.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.091 11.0934,, 2021.
Cory Paik, Stéphane troca-Ouellette, Alessandro Roncone, and Katharina Kann. The world of an octopus: How reporting bias influences language model's perception of color. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
2021.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In Interna- tional Conference Learning Representations, 2022. URL httpp:://pemeeeveew..nes.
Corum?1d=gJcEMB sXHK,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-87888 8763.873..1166 PMLR, 2021.
S Rajbhandari, J Rasley, O Ruwase, and Y He. Zero: memory optimization towards training trillion parameter models. arxiv e-prints arxiv: 11910.02054 (2019), 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1113))2 2015. doi: 10.1007/s1 12677/51123------08116
Thomas Scialom, Patrick Bordes, Paul-Alexis Dray, Jacopo Staiano, and Patrick Gallinari. What bert sees: Cross-modal transfer for visual question generation. In Proceedings of the 13th Inter- national Conference on Natural Language Generation, pp. 327-337, 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemnyme, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565. Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v 11118/5//1/188--. 1238. URL httpt:t/:///. aalaacaannholoogooo www.aatooow....ooo.oeaaeeee. wwwp/o/g///p/...
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-lange and-language tasks? In International Conference Learning Representations, 2021.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for rision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. $227-5237, 2022.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:// com wttpc/o///oooooooooooooooooooo/ ttrto///eeoeet---tatorreeeeeeee
May 2021.
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, et al. Language models with image descriptors are strong few-shot language learners. arXiv preprint arXiv:2205. 10747, 2022.
Zhibiao Wu and Martha Palmer. Verbs semantics and lexical selection. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pp. 133-138, 1994.
12

---

Published as a conference paper at ICLR 2023
Yonggin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning com- prehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2251 11222111225.,2 2019. doi: 10.1109/PAAM..20188877777
Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, and Michael Zeng. Visual clues: Bridging vision and language foundations for image paragraph captioning. arXiv preprint arXiv:2206.01844 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveck Puro- hit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. 2022.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18123-18133, 2022.
A LTMBER TRAINING DETAILS
A.1 TRAINING DETAILS
We mimic the MAGMA pretraining process for each of our models as outlined in Eichenberg et al. (002T). As described above, the training task caption generation. For each training example, an image and caption pair (z. y) is fed into the model. The image encoder E encodes the image into a .1...... &g of dimensionality h and length For the CLIP Encoder, for example, we extract 12.3072) feature patches, which resize to (144,3072) and feed into the projection layer P. The output of P fed as tokens representing the image into the language model LM. The caption y is tokenized f1,..., fm where is the variable length of the caption. LM is given the encoded image tokens and starting with 1 learns to minimize the next token log probability of t, for timestep i conditioned on 11 ..... 4 and -.
During training, we minimize the loss with the A (Loshchilov & Hutter 2018) optimizer per mini-batch, with the help of 7eRO stage 2 (Rajbhandari 2019). We use dropout probability of 0.1, 1,1,a weight decay 0, betas = (0.9, 0.95), and gradient clipping 1.0. All models are trained for 15,000 training steps across 16 A000 GPUs for approximately 1.75 days. Our effective batch size was 2048, We use a learning rate of 8 10 4 for the projection layer P. For models where we tune E well, we tune its parameters with learning rate of * 10-6. 6.
Model
Prompt Length (k)
Hidden size (hz)
Pretraining
Contrastive Image-caption
CLIP RN50x16
144
3072
matching
NFRN50 BEIT
2048
Image Classification
196
1024
Self-supervised
Table 2: Summary of image encoders used for pretraining. Prompt length refers to the number of the tokens fed into the language model representing the image.
B CAPTIONING PERFORMANCE
To get a better idea of the kinds of captions LiMBeR produces, see Figure which includes 15 images that randomly selected from the COCO validation set (2017) and the generated cap- tions for all models we test. We include a greater range captioning metric results in Table 3 for the COCO dataset. Overall, find the same trends that report in the main paper, which
13

---

Published as a conference paper at ICLR 2023
RSA Similarities for Animal Images/Captions
GPT-J
0.037
0.015
CLIP
0.037
0.25
0.048
0.015
NFRN
0.015
BEIT
0.013
0.048
0.29
0.51
Rand.
0.0025
0.015
0.3
0.51
GPT-J
CLIP
NFRN
BEIT
Rand.
Figure 5: RSA similarity scores between representations of images of animals and the ground truth captions describing them. GPT-J is has very low correlation with image encoder representations
that 1.) the greater amount of linguistic supervision that an image encoder has, the better its cap- tioning performance and 2.) unfreezing the image encoder does not seem lead consistently significant improvements. We also include the breakdown of the SPICE metric across the associ- ated subcategories such as relations, attributes, and objects. Of the LiMBeR models, we find that CLIP based models do the best across the board [ for CLIP vs. 9.28 for NFRN50), Besides the random baseline, BEIT performs the worst overall except in the color category (0.45 vs. 0.42 for NFRN50). We also include heatmaps with recall/precision/FNNW http://erriiio////uuauaaaa /almere similarity metrics com- paring the captions generated by each model and the top 50 nouns (objects), modifiers, and relations from the ground truth captions from the COCO validation set (Figures
We also use COCO images and captions to measure the similarity between vision and text encodings of the same concepts. That is, if there is structural similarity between image and text encodings, do typical representational similarity metrics reflect this? For this experiment we want to encode large number of images and captions that depict a of concepts, and compare the relative repre- sentational similarity between the concepts for the image and text representations. The intuition is that if image and text models are representing concepts in similar structure, then two concepts that are represented similarly in an image model should be similar within text representations as well. We sample a small subset of images depicting 10 different animals from the COCO dataset, and encode each image with each of our four image encoders (without LiMBeR projection), and the ground truth captions with GPT-J. We use the last hidden state of the last token as the representation of the caption. We choose animals because is easy for humans intuitively compare how similar two animals are. In total, we collect 939 images and captions for each animal class and compare the representational similarity of the subset of encodings using the representational similarity analysis (RSA) technique from Kriegeskorte et al. (2008). The similarity matrix between each representation for each set calculated, and then the upper triangular matrix for each similarity matrix is used to calculate the Pearson correlation. The results of this can be seen in Figure We zero out the diago- nal for visibility of the other values, since the diagonal is always equal to I. RSA does seem to capture similarity in a way that reflects the transfer performance to the LMM. For example, similarity between the BEIT and randomly initialized NFRN50 model are unusually high given differences in performance in our experiments. Further analysis on the geometry of these representation spaces is required to make stronger statements about how (dis)similar they are. leave this for future work.
C VISUAL QUESTION ANSWERING PERFORMANCE
The VQA accuracies are broken down by question type and model for the 4-shot case in Table 5 We don't notice any significant patterns in the per question type breakdown. It does seem perhaps
14

---

Published as conference paper at ICLR 2023
Model
BLEU-T
BLBU-2
BLEU-3
BLEU-4
METEOR
ROUGE
CIDER
SPICE
CLIPScore
ReICLIPSeore
NFRNSO
0.375
0.229
0.127
0.316
MAGMA (Ours)
0.309
0.216
0.145
0.097
0.146
0.341
0.475
0.110
0.753
0.796
MAGMA (Released)
0.432
0.300
0.203
0.137
0.159
0.376
0.521
0.117
0.767
0.794
BEIT Random
0.258
0.132
0.068
0.232
0.052
0,022
0,488
0,502
NENRN Random
0.261
0.115
0.044
0.018
0.062
0.208
0.048
0,021
0.495
0.571
BEIT
0.064
0.697
NFRN5O
0.409
0.254
0.149
0.088
0.132
0.534
0.362
0.093
0.689
0.741
BEIT FT. 0.420
0.288
0.182
0.116
0.155
0.367
0.510
0.116
0.742
0.789
Table 3: All of the caption metrics for the models. For all scores, higher is better
Model
SPICE (All)
Relations
Cardinality
Attributes
Size
Color
Object
NFRN50 tuned
9.1
1.12
0.02
1.64
0.33
0.40
19.9
MAGMA (ours)
11.0
1.37
0.00
3.94
0.36
1.31
22.6
MAGMA (released)
11.7
1.99
1.30
3.95
0.50
1.32
23.8
NFRN50 random
2.14
0,06
0.00
0.22
0.00
0.12
4.93
BEIT
6.4
0.61
0.0
1.26
0.21
0.45
14.1
NFRN50
9.28
1.28
0.02
1.70
0.27
0.42
20.2
CLIP
12.1
1.83
0.08
3.76
0.39
0.75
25.08
Table F-scores (x100) for each fine-grained category of the SPICE metric, evaluated on the 2017 COCO validation dataset. The top and bottom divide separates models where the image encoder is either tuned of frozen, respectively. Models that use CLIP as their image encoder show a large jump in improvement over other models (even compared to tuned ResNet), especially in the Attributes (e.g. adjectives) and Object (e.g. nouns) categories.
NFRN50 is better at questions that require counting objects in an image. Future work is needed to determine if this is just noise significant trend.
Type how many is the what what color is the 14061
# Q's
Blind
RN Tune
MAGMA
RN Rand
BEIT
RN
CLIP
20462
33.6
34.7
32
33
34.8
17265
64.1
64.7
65.3
64.8
58.9
64.6
61.1
15897
17.4
19.9
22.4
16.7
8.9
20.3
26.9
30.5
33.7
36.7
33.1
28.9
34
40.3
what is the none of the above
11353
15.5
18.7
23.2
16.1
18.7
29.5
8550
38.5
40.1
36
38.7
39.9
is this is this
7841
62.4
64.1
65.8
64.2
57.7
63.7
57.1
7492
63.3
63.8
67.6
63.4
59.5
63.5
58.5
what is
6328
9.3
12.9
21.2
10
6.6
13.2
24.1
what kind of are the is there
5840
15
22.5
29.5
15.8
22.5
36.9
5264
63
64.2
64.3
63.9
59.4
63.8
58.9
4679
60.6
61.6
63.8
61.7
60.2
61.5
58.6
what type of
4040
16.4
23
30.7
17.2
12.8
23
38.2
where the
3716
2.4
0.9
1.3
3566
63.6
60.3
62.3
60
60.4
what are the does the
3282
13.4
17.3
23.1
14.7
9.8
18
28.6
3183
66.6
67.6
66
67.4
63.0
67.6
64.6
is there what color are the
3169
63.8
63.4
65.0
63.1
56.6
62.6
58.5
3120
62.4
62.3
64.6
61.7
63.1
3118
32
36.3
35.8
34.6
29.4
36
39.7
are these are there
2839
62,3
65.1
66.8
64.1
60.3
64.9
56.6
2771
60.1
60.2
62.8
60.1
57.5
60.1
55.4
what is the
2663
10
16.7
30.1
11.4
10.3
15.6
34.7
is man
2511
61.9
61.7
64.6
62.2
58.9
61.5
59.2
which
2448
23.8
26
27.9
22.7
7.1
26.1
26
how
2422
12.2
10.2
7.5
9.7
4.0
10.8
9.8
2359
60.6
62.9
63.5
62.4
$7.1
62.5
56.8
does this what is on the
2227
66.3
67.2
66.2
67.6
62.7
67.7
61.3
2174
11.5
14.9
17.8
13.2
7.3
15.2
23.0
how many people are
2005
27.2
28.0
26.7
27
10.6
27.7
18.7
what does the what time what in the
1970
7
7.6
12
6.8
2.7
7.4
14.2
1746
15.4
18.9
13.4
16.6
16.0
19.1
22.4
1733
9.8
13.7
23.8
10.8
13.8
15

---

Published as a conference paper at ICLR 2023
Type
# #Qsss
Blind
RN Tune
MAGMA
RN Rand
BEIT
RN
CLI
what this
1696
9.6
21.7
35.2
9.6
11.6
21.8
41.3
what are
1556
9.2
16.8
28.6
10.4
8.7
17.4
36.0
do why what color
1503
68.8
70.1
65.9
70.2
61.4
69.7
64.9
1438
3.2
2
2.8
1.5
0.6
1.9
2.8
1428
26.9
28.5
32.1
28.4
23.8
27.8
35.0
are they what color is there any
1335
60.8
61.2
65.8
62.5
59.8
61.4
59,4
1335
25,4
26.5
35.8
26.1
29.1
26.6
36.4
1330
59.1
58.3
61.6
58.7
54.4
58
52.7
where are the is he what sport is who is
1313
2.8
2.4
1.2
1.7
3.3
1087
61.8
62.2
64.2
61.6
59.8
63.1
61.2
1086
22
37.1
63.4
24.9
36.6
36.7
67.6
1070
13.6
14.4
14.3
13.7
5.3
14.8
9.6
is the has what hrand how many people are in
992
63.2
62.2
66.6
61
59.3
61.5
60.9
946
64
65.3
65.5
65.5
62.7
66.3
935
20.0
16.5
4.4
17.8
6.3
17.8
12.4
905
23,9
28.3
29.0
25.8
15.8
26.9
17.4
what is the person
900
8.2
16.3
32.2
9.2
12.4
17.7
38.0
65.1
67.6
68.2
67.7
61.4
67.4
can you what is the woman
872
59.5
58.9
62.3
59.9
58.8
59.6
59.5
853
8.5
15
28.5
10.2
6.7
14.6
32.6
what animal is
833
10.2
28.5
56.1
10.8
16.8
28.6
63.6
what is the color of the
826
35.4
40.4
39.3
36.8
34.3
39.3
44.7
was the person
818
62.6
63.8
62.1
59.9
60.9
794
61.9
61.7
61.3
62.4
57.5
62
57.1
what is the name what room is this person do you is that what number is could why the
780
3.4
4.4
8
3.8
1.5
4,5
9.0
762
15
43.4
62.9
19.9
24.0
39.3
65.9
734
62
61.3
63.3
62.5
61.9
$7.5
724
61.7
62.1
60.1
60.8
55.6
62
714
60.9
63.1
62.7
61.5
57.5
62.4
59.6
673
8.8
7.6
12.6
7.5
0.5
8.9
10.2
618
72.3
71.8
67.5
73.6
67.4
71.5
58.9
514
1.6
2.5
1.3
2.6
Table 5: Average 4-shot accuracies of models on every question type from the VQA2.0 dataset. Note that NFRN50, NFRN Random, and NFRN Tuned are renamed to save space. MAGMA refers to our version of the model.
D ANIMALS WITH ATTRIBUTES
AWA2 The Animals with Attributes Dataset contains 37k images of 50 animal classes annotated with 85 properties (e.g. 'stripes', 'tough'), For each image encoder generate captions using image projections into the LM for each image in the dataset and identify any animals mentioned using overlapping WordNet synsets with the ground truth labels. We report the animal synsets that we used as the "ground truth' synsets. For example, to calculate accuracy, etc. we split generated captions white space and check the possible synsets of each word and check if there is any overlap with our list. Each label synset is listed in Table
We report AWA experiments here: (1) the first appears in the main paper, and investigates how similar the most similar words mentioned in the generated captions are to the ground truth animal in terms of Wu-Palmer similarity for the generated captions in which the ground truth animal does appear-i.e. i.e., mistakes. The per animal results can be found in Table (2) We also report the average precision (AP) of the average animal properties vector for the animals predicted the ground truth for each model. This experiment is similar to the properties-overlap experiment reported in the paper but a measure of how often the predicted animal properties are similar vs. very different from the ground truth. If the LM mentions animal in the dataset, we add that animal's property vector to a running list. If the LM does not mention an animal the dataset, we ignore that caption. For each image-caption pair, take the average of the properties vectors that correspond to the animals mentioned in the captions, and calculate the AP against the ground truth binary properties vector. Results per animal class are in Table We exclude any entries for which model makes fewer than 50 mistakes for an animal class. For example, for all of the images labeled as depicting a tiger in the AWA dataset, CLIP based captions fail mention tiger single time.
16

---

Published as a conference paper at ICLR 2023
Animal Name antelope
WordNet Synset
antelope.n.0 grizzly.n.01
grizzly+bear
whaler+
killer Giller.whale...01
beaver
beaver.n.07
dalmatian
dalmatian.n.02
persian+cat
persian cat.n.01
horse
horse.n.01
german+shepherd
german srinaan_shepher..n.
..001
blue+whale
blue_w Slue_whale...0
siamese+cat
Stamese.cat.n..
skunk
skunk.n.04
mole
mole.n.06
tiger
tiger.n.02
hippopotamus
hippopotamus.n.00
Icopard
Icopard.n.02
moose
clk.n.01
spider+monkey
ppider.monkey.n.00
whaee
humpback.n.03
elephant
elephant.n.01
gorilla
gorilla.n.01
ox
ox.n.02
fox
fox.n.01
sheep
sheep.n.01
seal
seal.n.09
chimpanzee
chimpanzee.n.01
hamster
hamster.n.01
squirrel
squirrel.n.01
rhinoceros
rhinoceros.n.0)
rabbit
rabbit.n.01
bat
bat.n.01
giraffe
giraffe.n.01
wolf
wolf.n.01
chihuahua
chihuahua.n.03
rat
rat.n.01
weasel
weasel.n.02
otter
otter.n.02
buffalo
american 1atrricanbbisonn..1
zebra
zebra.n.01
giant+ppada
giant.pandaan.00
deer
deer.n.01
bobcat
bobcat.n.01
pig
hog.n.03
lion
lion.n.01
mouse
mouse.n.01
polar+bear
ice (ce_bear.n.01
collie
collie.n.01
walrus
walrus.n.01
raccoon
raccoon.n.02
cow
cow.n.01
dolphin
dolphin.n.02
Table 6: To aid with reproducibility, report all animal synsets that used for experiments that require disambiguating words in captions to animal classes. This allows us to correctly count a mention of "tigress" in caption as mention of the "tiger" animal type without relying on unreliable string matching techniques.
17

---

Published as a conference paper at ICLR 2023
Random
NFRN50 Random
BETT
NFRN50
CLIP
polar+bear
0.79
0.60
0.86
0.95
0.96
buffalo
0.73
0.64
0.84
0.93
0.95
bobcat
0.72
0.71
0.83
0.90
0.94
grizzly+bear
0.76
0.70
0.88
0.95
0.94
blue+whale
0.73
0.43
0.69
0.76
0.87
dalmatian
0.77
0.72
0.79
0.83
0.85
chihuahua
0.74
0.70
0.84
0.85
0.85
leopard
0.77
0.70
0.82
0.81
0.85
collie
0.72
0.70
0.85
0.85
0.85
german+shepherd
0.72
0.67
0.84
0.85
0.85
stamese+cat
0.75
0.69
0.85
0.85
0.84
persian+cat
0.74
0.67
0.85
0.85
0.84
ox
0.73
0.69
0.82
0.84
0.84
seal
0.77
0.55
0.77
0.74
0.82
spider+monkey
0.74
0.65
0.81
0.80
0.81
antelope
0.75
0.72
0.78
0.80
0.81
skunk
0.79
0.65
0.84
0.79
0.81
mole
0.79
0.64
0.79
0.84
0.80
lion
0.77
0.74
0.82
0.80
0.79
0.80
0.67
0.79
0.85
0.78
hamster
0.80
0.66
0.79
0.80
0.78
walrus
0.78
0.56
0.72
0.76
0.78
wolf
0.80
0.69
0.85
0.83
0.77
killer+whale
0.71
0.46
0.71
0.73
0.77
whhpe
0.73
0.43
0.66
0.67
0.77
chimpanzee
0.72
0.64
0.85
0.74
0.76
moose
0.75
0.60
0.81
0.79
0.74
fox
0.80
0.72
0.84
0.84
0.74
weasel
0.79
0.68
0.81
0.83
0.73
hippopotamus
0.79
0.58
0.76
0.74
0.73
deer
0.77
0.66
0.81
0.80
0.73
otter
0.79
0.60
0.80
0.82
0.72
pig
0.76
0.67
0.83
0.75
0.71
cow
0.71
0,65
0.78
0.78
0.70
sheep
0.76
0.65
0.84
0.82
0.68
horse
0.75
0.62
0.77
0.75
0.59
giant+panda
0.79
0.70
0.87
0.90
0.52
beaver
0.80
0.55
0.77
0.76
tiger
0.77
0.74
0.82
elephant
0.79
0.64
0.79
gorilla
0.72
0.60
0.91
0.84
squirrel
0.80
0.67
0.80
0.83
rhinoceros
0.78
0.67
0.80
0.76
rabbit
0.76
0.69
0.82
0.81
bat
0.82
0.67
0.74
0.74
giraffe
0.78
0.70
0.79
0.76
zebra
0.76
0.68
0.78
mouse
0.79
0.64
0.75
0.83
raccoon
0.79
0.64
0.83
0.81
dolphin
0.73
0.49
0.72
0.68
Mean=
0.76
0.64
0.80
0.81
0.79
Table 7: Wu-Palmer Similarity for mistakes. Animals for which a model made fewer than 50 mis- takes are dashed out
18

---

Published as a conference paper at ICLR 2023
Animal/Model
Random
NFRN50 Random
BETT
NFRN50
CLIP
hamster
0.70
0.74
0.83
1.00
1.00
rat
0.79
0.83
0.81
0.99
1.00
squirrel
0.76
0.70
1.00
1.00
1.00
rhinoceros
0.48
0.41
0.81
0.96
1.00
rabbit
0.75
0.76
0.85
1.00
1.00
bat
0.59
0.58
0.58
0.99
1.00
giraffe
0.66
0.73
0.92
1.00
1.00
wolf
0.87
0,84
0.94
1.00
1.00
chihuahua
0.80
0.80
0.79
0.91
1.00
zebra
0.79
0.86
1.00
1.00
1.00
chimpanzee
0.86
0.84
0.83
1.00
1.00
giant+panda
0.71
0.74
0.76
1.00
1.00
deer
0.83
0.91
1.00
1.00
1.00
bobcat
0.72
0.79
0.80
1 1.00
1.00
lion
0.74
0.93
0.93
1.00
1.00
mouse
0.77
0.69
0.81
0.98
1.00
polar+bear
0.72
0.76
1.00
1.00
1.00
raccoon
0.76
0.77
0.85
1.00
1.00
grizzly+bear
0.70
0.68
1.00
1.00
1.00
dolphin
0.55
0.69
1.00
1.00
1.00
tiger
0.77
0.84
1.00
1 1.00
1.00
lcopard
0.79
0.86
0.95
1.00
1.00
horse
0.88
1.00
1.00
1 1.00
1.00
german+shepherd
0.87
0.86
0.76
1.00
1.00
blue+whale
0.42
0.34
1.00
1 1.00
1.00
siamese+cat
0.83
1.00
1.00
1.00
1.00
killer+whale
0.42
0.56
0.88
1.00
1.00
beaver
0.68
0.66
0.77
1.00
1.00
hippopotamus
0.47
0.41
0.80
1.00
1.00
persian+cat
0.75
1.00
1.00
1 1.00
1.00
moider+monkey
0.79
0.77
0.99
1.00
1.00
humpback+whale
0.43
0.82
0.97
1 1.00
1.00
elephant
0.61
0.61
1.00
1.00
1.00
gorilla
0.76
0.75
0.89
1 1.00
1.00
moose
0.77
0.82
0.88
1.00
1.00
cow
0.74
0.82
1.00
1 1.00
1.00
fox
0.80
0.77
0.96
1.00
1.00
seal
0.57
0.51
0.95
1 1.00
1.00
sheep
0.59
0.70
0.98
1.00
1.00
pig
0.72
0.70
0.98
1.00
1.00
otter
0.66
0.72
0.73
0.76
1.00
mole
0.72
0.68
0.88
0.82
1.00
walrus
0.39
0.38
0.70
0.72
0.99
antelope
0.81
0.85
0.94
0.97
0.98
ox
0.74
0.69
0.89
0.93
0.95
collie
0.84
0.87
0.80
0.93
0.91
weasel
0.85
0.77
0.86
0.90
0.90
buffalo
0.64
0.69
0.91
0.74
0.85
skunk
0.70
0.79
0.83
0.83
0.84
dalmatian
0.86
0.86
0.83
0.72
0.73
Mean=
0.71
0.74
0.89
0.96
0.98
19

---

Published as a conference paper at ICLR 2023
Animal/Model
Random
NFRN50 Random BEIT NFRN50 CLIP
Table 8: Per animal average precision (AP) for properties of mentioned animal in captions per model for the Animals with Attributes 2 (AWA2) dataset. BEIT, which tends to do worse at captioning and question answering consistently predicts animals which share similar properties, and considerably better than randomly initialized NFRN50 and randomly selecting animals as baselines. This sug- gests that BEIT representations encode similar animals into broad conceptual categories (e.g. large savanna animals) which are able to linearly transfer to the LMM Without linguistic supervision, BEIT does not naturally distinguish these by the words we use describe them, as NFRN50 and CLIP do.
Animal/Model
NERN50 Random
BEIT
NFRN50
CLIP
giraffe
0.00
0.16
0.96
1.00
zebra
0.00
0.77
1.00
1.00
tiger
0.00
0.75
0.98
1.00
elephant
0.00
0.37
0.96
0.99
dolphin
0.00
0.37
0.58
0,98
rabbit
0.00
0.13
0.85
0.98
rhinoceros
0.00
0.04
0.30
0.98
squirrel
0.00
0.41
0.88
0.97
gorilla
0.00
0.20
0.93
0.96
sheep
0.00
0.27
0,91
0,96
raccoon
0.00
0.00
0.73
0.95
bat
0.00
0.00
0.28
0.94
horse
0.09
0.85
0.94
0.92
huhpback+whale
0.00
0.21
0.56
0.92
hippopotamus
0.00
0.06
0.59
0.92
fox
0.00
0.15
0.91
0.91
cow
0.00
0.41
0.82
0.90
giant+-panda
0.00
0.00
0.76
0,90
moose
0.00
0.01
0.41
0.89
deer
0.02
0.41
0.79
0.87
pig
0.00
0.15
0.76
0.86
leopard
0.00
0.42
0.92
0.86
seal
0.00
0.27
0.73
0.86
wolf
0.00
0.03
0.76
0.85
hamster
0.00
0.00
0.69
0.83
mouse
0.00
0.01
0.25
0.82
beaver
0.00
0.01
0.49
0.80
rat
0.00
0.00
0.37
0.78
chimpanzee
0.00
0.00
0.44
0.76
lion
0.08
0.14
0.40
0.66
killer+whale
0.00
0.00
0.10
0.53
otter
0.00
0.03
0.06
0.48
mole
0.00
0.01
0.02
0.40
walrus
0.00
0.00
0.00
0.36
bobcat
0.00
0.00
0.16
0.33
+bea
0.00
0.00
0.06
0.15
chihuahua
0.00
0.00
0.04
0.14
antelope
0.00
0.00
0.07
0.05
collie
0.00
0.00
0.06
0.04
weasel
0.00
0.00
0.03
0.03
buffalo
0.00
0.04
0.02
0.02
dalmatian
0.00
0.00
0.00
0.02
ox
0.00
0.00
0.00
0.01
skunk
0.00
0.00
0.00
0.01
siamese+cat
0.00
0.00
0.00
0.00
german+shepherd
0.00
0.00
0.00
0.00
20

---

Published as a conference paper at ICLR 2023
Animal/Model
NFRN50 Random
BETT
NFRN50
CLIP
persian+cat
0.00
0.00
0.00
0.00
polar+bear
0.00
0.00
0.00
0.00
spider+monkey
0.00
0.00
0,00
0,00
blue+whale 0.00
0.00
0.00
0.00
Mean=
0.00
0.13
0.43
0.59
Table 9: Accuracy for each model and each animal class in Animals with Attributes 2. caption is considered correct if the animal name is mentioned in the caption for the image.
E PROBING VISUAL REPRESENTATIONS
We train probes on the image encodings from each of our image encoders to classify fine- fine-grained lexical and coarse grained categorical concepts several datasets: COCO (Lin et al. 2014), and CC3M (Sharma et al. 2018), and CIFAR-I00 (Krizhevsky et al. 2009), The architecture a single linear layer which takes an image encoding of dimension h, (see Table and projects to the number of classes for the classification task. For single label classification, we use a softmax activation on the logits and train with cross entropy the loss function. For multilabel classification tasks, use a sigmoid activation layer on top of the logits and train with a binary cross entropy loss function. We consider certain class predicted if the value of the class after the sigmoid is t0.5.
Hyperparameters For simplicity, all probes are trained with the same hyperparameters (with a few exceptions for the CC3M probes): learning rate: le-4, optimizer: AdamW (Loshchilov & Hutter 2018), betas=(0.9, 0,999), batch size: 48 for CC3M probes; 32 for all others, max epochs: 300 for CC3M probes; 300 for all others.
Object Supercategory
Objects within class
accessory
backpack, unbrella, handbag, tie, suitcase bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe microwave, oven, toaster, sink, refrigerator tv, laptop, mouse, remote, keyboard, cell phone
animal
appliance
electronic
banana, apple, sandwich, orunge, broccoli, carrot, hot dog,
food
pizza, donut, cake
furniture
chair, couch, potted plant, bed, dining table, toilet
indoor
book, clock, vase, scissors, teddy bear, hair drier; toothbrush
kitchen
bottle, wine glass, cup. knife, spoon, bowl traffic light, fire hydrant, stop sign, parking meter; bench person
outdoor
person
frisbee, skis, snowboard, sports ball, kite, baseball bat,
sports
baseball glove, skateboard, surfboard, tennis racket bicycle, car, motorcycle, airplane, bus, train, truck, boat
vehicle
Table 10: All individual object labels and supercategories they fall under as annotated in the COCO dataset.
E COCO OBJECT CLASSES PROBE
The COCO dataset contains labels for 80 objects in images, typically used for object detection or segmentation tasks. Because image can be labeled for multiple objects, we train the probe multi-label classification task.
In addition to having fine-grained labels for each type. COCO provides labels for each broad cate- gory that each object belongs to, called the supercategory. In Table I0, we show which supercate- gory each object label falls under.
21

---

Published as a conference paper at ICLR 2023
E.1.1 COARSE-GRAINED SUPERCATEGORY PROBES
We train a multilabel classification probe to classify the object category seen in a given image. We report F1 for each Li LiMBeR model in Figure II We find that BEIT does bit worse than NFRN50 and CLIP overall, but is able to classify some categories (images with accessories, vehicles, and "outdoor' objects) well.
1.1.2 FINE-GRAINED OBJECT LABEL PROBES
Next, we look at the probe results for probes trained to identify individual objects by type. Our results can be found in Figure 12d We find the same pattern emerges, and that BEIT does not seem to be significantly closer to the other pretrained models in terms of F1 on the coarse-graing vs. fine-grained patterns as we might expect. However, we do show that BEIT encodes strong, but weaker lexical concept categories than the other two models, and that the finding that BEIT transfers coarser grained information is due irreconcilable representational differences between BEIT space and the LM space.
E.2 CC3M PROBES
We also train the same set of probes on image data from CC3M, but evaluate on the same validation set from COCO (i.e., the same evaluation as used in Section E. E.1.2. The purpose this experiment is to create setting for linear probe that better matches the LiMBeR setup use in the main paper. If there are concepts that the probe has no trouble with, but rarely appear in captions, that
could be an indicator that the IMM and the image encoder represent that concept very differently in representation space.
Data To align the CC3M images with the object labels COCO, we create labels by looking for exact string matches of the object label words (e.g. "teddy bear") in CC3M captions. Of the 80 object classes, we cut out any that have fewer than 1000 images. This leaves us with 782,794 training images for the probes across 53 object classes; fewer than the CC3M dataset used to train LiMBeR, but far more than the previous datasets we used for probing.
Results Because, in this setting, we train our probes on the same distribution as the Lii MReR models, we compare the Fl of the probes identifying objects in images to the Fl of those concepts appearing in the generated captions. Our results can be seen Figure 13 appears that if the image encoder encodes the lexical concept, it generally also transfers to the LM with LiMBeR. Limitations of this approach are that (1) the BEIT probe appears much worse at the domain shift from CC3M to COCO (e.g., F1 for animals drops 0.3 compared to when the probes are trained on COCO) and (2) some words in the label space are often substituted for more common words in generated captions (e.g. "person" could be generated "man", "woman", etc.). This makes it difficult to recognize cases where the probe succeeds but the transfer fails. An interesting problem for future work is better understanding which concepts encoded in image encoder's 's representations, but do not transfer well with a linear map to the L.M.
E.3 CIFAR-100 PROBES
100 (Krizhevsky et al. 2009) is a dataset of 60,000 32x32 images balanced across 100 object classes. Like COCO, the 100 object labels are also annotated for coarse-grained object categories including 'aquatic mammals' and "household furniture'. For CIFAR data, we train a classifier which classifies an image for a single object label. Like with COCO, we train a set of probes for the fine and coarse labels. We were surprised to find that for CIFAR images, BEIT representations tended to better than NFRN50 in terms of average Fl (for the fine-grained probe, BEIT: 0.57, NFRN50: 0.47); a first for any of the experiments we ran. Given the majority of evidence shows NFRN50 encodes lexical category information stronger than BEIT, we hypothesize this is not because of BEIT encoding lexical concepts more strongly, but due the small resolution of images: because BEIT uses visual tokens, it may be more robust to extremely blurry images, which are out distribution for both NFRN50 and BEIT.
22

---

Published as a conference paper at ICLR 2023
E.3.1 COARSE-GRAINED PROBES
In Figure 14 we show the Fl results for the coarse-grained image labels probes trained image encodings from each model. Each coarse-grained class in the testing set has 500 images each.
E.3.2 FINE-GRAINED PROBES
In Figure 13d we show the per object class Fl results for the image encodings from each model. Each class in the testing set has 100 images each. Similar the COCO probe do not see significant intra-model changes between the coarse-grained and fine-grained probe results.
23

---

Published as a conference paper at ICLR 2023
Ground Truth:
is standing bed Ln
NFRN50 Tuned
ny bike with basket,
MAGMA (Ours)
ny bike
MAGMA (Re.R.
.ir............................. vintage bike vintage bike
................................
NFRN Random:
blwe shict zed hat bieyele BEIT: the bike.
NFRN50: my bicycle
CLIP: bike, which ix fixed bike.
Ground Truth: black
front
NERN50 Tuned my motorcycle
MAGMA (Qurs) my
(RAGMA((ReL))) ................................ .ow.ee
1973 Honda CB750
NERN Random: the car the road
BEIT: the bike."
NFRN50: matorcycle on a matorcycle.
CLIP my motorcycle.
Ground Truth: van riding wave on top of vhite surfboard.
NFRN50 Tuned surfar in the water
MAGMA (Ours)
riding
MAGMA (Rel.)... surfex riding vave
NFRN Random: the view from the top of the mountain
BEIT surfor riding
NFRN50:
the
CLIP: surfer riding vave.
Ground Truth:
standing
beach flying kite.
NFRN50 Tuned man standing on the beach with a surfboard.
MAGMA (Ours) kite on beach.
NERN Random: running
AAGMA.(Rel....
BEIT: man standing on rock in the sea.
NFRN50: man surfing on the beach.
CLIP: van flying kise
Ground Truth: 1ady axplaine the procees at milking coe.
NFRN50 Tuned : horve and rider at the foit.
MAGMA (Ours)
with kide in audfance.
MAGMA (Bel)...
NFRN Random: w............................... trx.rsention. coneereeeeeeene wew.eeeeeettt................... coneetee
BEIT: the ccavd at acent.
NFRN50: ocoed of people at festival
CLIP: the dairy cOMe.
Ground Truth: voman is taking notes in front of her laptop
NFRN50 Tuned the mun voo vas azot and killed in the ioboy building.
MAGMA (Ours)
studying.
MAGMA (Rel.)... student working
w...............................
NFRN Random: the man who was killed is the attack.
BEIT: the room where the neeting took place.
NFRN50: the room where the newting took place.
CLIP student working bis laptop.
Ground Truth: couple
together.
NFRN50 Tuned
the tous befoce autch
MAGMA (Ours) Leon playing gase of sgoct,
MAGMA (Rel.) men't lscrocen game at: the Univereity of Yeu England
NFRN Random:
BEIT:
player
pi player leringg bosining
NFRN50: the tawe bafore the game.
CLIP: the tawe daring prsation sacsion.
Figure 6: 15 randomly selected images from the COCO 2017 validation dataset and the generated captions from all models.
24

---

Published as a conference paper at ICLR 2023
Ground Truth: The vegetablee are Taid out mostly at the bakde,
NFRN50 Tuned
Vegatablee and Eraice for sale at nackat in the city
MAGMA (Ours) Freah produce
MAGMA (Rel).
NFRN Random:
BEIT:
NFRN50:
CLIP:
AMERICA
Ground Truth:
NFRN50 Tuned
WOBAD VALD DOX 1000 tar teaU.
MAGMA (Ours)
fost to the Noseless.
MAGMA.(Rel.)
NFRN Random: ciel With Flomer for arile fore
BEIT.
cocking kitaten.
NFRN50:
CLIP:
Ground Truth:
NFRN50 Tuned the beach with person lying on the send.
MAGMA (Ours)
(RGMM.(Ree)..... 2 300 and WORED playingg the beach,
NERN Random:
BEIT: a MaS and Wornn doing handstand on the beach.
NFRN50:
witting the beach,
CLIP: ma playing with
Ground Truth: The vony tey ptaying the Viving romm his tome.
NFRN50 Tuned : littie ViEE doll
MAGMA (Ours)
MAGMA
boy www.............................
NFRN Random: the secist with zon.
BEIT: x buy in cinectulatee
NFRN50: a LiMATo girl with tuy airplon
CL paceos wish his Dew bay.
Ground Truth:
uhite toflet kitting
of
NFRN50 Tuned
the toilet in the hathroon.
MAGMA (Ours) actor and tailet
MAGMA (Rel.)
in
NFRN Random: the bride and groen the wedding
BEIT; à white and black cat on black background,
NFRN50:
public restroon.
CLIP: the toilet
Ground Truth: Rovoral
computece on Seak.
NFRN50 Tuned jay
MAGMA (Ours) Ty
Sapoop various
MAGMA (Rel.)... ny desk
NFRN Random;
witting bench
BEIT:
NFRN50: xy corputer deek at MOrK.
CLIP: deak looked Actor few daya af vorking.
Ground Truth: A living roon
with furniture and table.
NFRN50 Tuned the living FoOm.
MAGMA (Ours) the living Xoom
MAGMA (Rel.)... ttee 1iving
NERN Random: the living room house
BEIT: the living zoon.
NFRN50: the 1iving room 1n the house.
CLIP: the 1iving
Ground Truth:
standing bebind
NFRN50 Tuned
pen with
MAGMA (Ours) dairr covs dairy Carn.
MAGMA. (Rel.. aa dairy farn
NFRN Random: the interior of the church
BEIT: a horse being through the atceets.
NFRN50:
CLIP:
Figure 6: 15 randomly selected images from the first random seed taken from the COCO 2017 validation dataset with the generated captions from all models.
25

---

Published as a conference paper at ICLR 2023
Times Geserabed for Objects
ba 62
52
Pep Aeg
Avsual
Vise
(a) times generated / of times in ground truth
Prectation Cirecte in OT exptions showing Predictions (e300)
2
TAE BE M A
133 28C
Poogle
Antrol
Entrommett
wehede
Object
(b) precision
Recii GT eaptions showlin
Predictions Jenay)
11
La Tae
pu
THE
Peogfe
Envermment
Wehde
(c) recall
Cordion
Soa 41 2a7 seee sraaaa staipsa
di
R0 TA BAGU STARE
(d) The Wu-Palmer Similarity between the ground truth word and the most similar word in the generated captions.
Obleda cuptiaee showtny Predictions
cas
szu aer ina tuaiia ta
tza uoura
30r
459 25
HC K.2300 94
14
27
389
(c)
Figure 7: The top 50 nouns that appear in the ground truth captions of the COCO validation set and how often each model generates them
26

---

Published as a conference paper at ICLR 2023
Generntd Modian
-
-
2
AUR
(a) times generated / of times in ground truth
14
14
x
precision
Rosal Of
Prsdecions (e100)
M
-
uniey
Colo
Marirtal
Number
Sow/Max
Texters
(c) recall
Medi custion
Prediciooa 16
38
0
38
-
1
Colo
Masirall
Nurber
Aga
Texters
Stat
(d) fl
Figure 8: The top 50 modifiers that appear in the ground truth captions of the COCO validation set and how often each model generates them
27

---

Published as a conference paper at ICLR 2023
Retefirs
M
69
(0
34
62
69
-
2
(a) times generated of times in ground truth
154
precision
Rocall Rebrions
chowing Prdieziion
18
Fundtte
Action
Mosion Ver
(c) recall
Nateiion GT eupticn shonling Predicioon TEE
to
38
17
Punkee
Poshen
Autson
Moltaan
(d) fl
Figure 9: The top 50 relations that appear in the ground truth captions of the COCO validation set and how often each model generates them
28

---

Published as a conference paper at ICLR 2023
How much pickle will they eat?
Answer: 1/3
NFRN50 Tuned
Bnough to make then sick.
MAGMA (Ours)
enooggh
MAGMA (Rel.)...
Nnoough to make then sick.
NFRN Random: They will waz much da they vunt.
BEIT: They
cup pickle,
NFRN50;
will much want.
CLIP: X11 of it!
What is the thing above the stove called?
Answer: nicrovave
NERN50 Tuned
MAGMA (Ours) nicrovave
MAGMA/(RA))
vent: hood.
www.............................
NFRN Random: hob.
BEIT: stove.
NFRN50: atove.
CLIP:
Answer; cloudy
NFRN50 Tuned Clear
How are the skies?
MAGMA (Ours)
MAGMA (Rel...: the akies are claudy.
NFRN Random: They are clear and blue.
BEIT: The aky is clear and the sun is shining.
NFRN50:
cloudy
cloudy and MAGMA (Ours) Sw's blus.
CLIP:
strong.
What color dog on the table?
Answer:
NFRN50 Tuned
The dog is brown.
(AAMMAIRRl). www.eaa......................... black.
dog.
NFRN Random:
BEIT: It's brown dog.
NFRN50: is is black and white dog.
CLIP: it's
dog.
this picture taken inside outside?
Answer: outside
NFRN50 Tuned It's ousside.
MAGMA (Ours)
MAGMA (Rel)...
NFRN Random: It is picture paraon misting on banch
BEIT: It's
NFRN50: Inside
CLIP: re's cuceide.
Is he his way to corporate job?
Answer: no
NFRN50 Tuned
warring nilt sle,
MAGMA (Ours) be probably just street perforser.
MAGMA MAGMA.(Rel)...... wiw.............................
way
NFRN Random:
quy.
BEIT: Mo, he his way to 1ob interview
NFRN50: Be is on his way to corporate job
CLIP- Boo
way skate paxk
Is the street a two-way street?
Answer:
NFRN50 Tuned yea, it io.
MAGMA (Ours) no.
MAGMA (Rel.)
NFRN Random: Kes, Exo-way atreet.
BEIT: Vol,
NFRN50: Xes, it is.
CLI
Figure 10: 15 randomly selected images from the VQA2 validation set and the generated answers from all models.
29

---

Published as a conference paper at ICLR 2023
Does the zebra look happy?
Answer:
NFRN50 Tuned
MAGMA (Ours)
MAGMA. (Rel)...
to alwapp
NERN Random:
the aonakine.
BEIT: Yea, bappy.
NFRN50: Pewt
CLIP: sea, be is enjoying the grAaR.
What activity caused him to fall?
Answer: ski ing
NFRN50 Tuned
that in okling
MAGMA (Ours)
(AGMMA(Rel.)
NERN Random: No was zarber of the wrentling teaz.
BEIT: fee was coying to catch bell.
NFRN50: No was okiing on clope chat was covered in los
CLIP. lle
cheic lift.
Answer: mn
NFRN50 Tuned
MAGMA (Ours) jAt teck.
MAGMA (Rel.)
NERN Random: it teck zer about 10 ninutes to aake the kite fly
How many minutes did take for the little girl to make her kite fly?
BEIT:
NFRN50:
IP
What brand shirt man wearing?
Answer: Polo
NFRN50 Tuned : re's brand called.
MAGMA (Ours) orand.
MAGMA (Rel.) CPOMON
NFRN Random:
BEIT:
NFRN50:
CLIP:
SUBATAC-
What the zebra doing?
Answer: eating
NFRN50 Tuned 2t eating geass.
MAGMA (Ours)
MAGMA (Rel.) It 1s eating grass.
NFRN Random: It is trying to get out of the water.
BEIT:
eating graae.
NFRN50: Rating
CLIP; Bating graze.
Is the player wearing helmet?
Answer: no
NFRN50 Tuned No, be 14 nec.
MAGMA (Ours)
MAGMA MAG)A.RRe))...
NFRN Random: yes, be wearing Selvot.
BEIT:
NFRN50:
CLIP: Nl.
What is the cat sitting on?
Answer:
NFRN50 Tuned
book.
MAGMA (Ours)
suitcase.
MAGMA (Rel.)
suitease.
NFRN Random:
BEIT: It'u cut.
NFRN50: zt'u logtupt
CLIP: EattiCaee.
Are there people the photo?
Answer: no
NFRN50 Tuned yes, there are people in the photo.
MAGMA (Ours)
yes
MAGMA (Rel.).
there people the photo.
Aerereeeiieeeee ieneeneecen coneeieeeen ceneeneeeeenee ieeeee..........................
NFRN Random: Yea, there aze people the photo.
BEIT: yea,
people photo.
NFRN50:
CLIP- there people the photo.
Figure 10: 15 randomly selected images from the VQA2 validation set and the generated answers from all models.
30

---

Published as a conference paper at ICLR 2023
NFRN50 Random Probe F1 COCO Supercategory
BEIT Probe F1 per COCO Supercategory
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
food
animal
kitchen
accessory
appliance
sports
outdoor
furniture
electronic
indoor
vehicle
person
accessory
vehicle
outdoor
indoor
animal
person
appliance
electronic
sports
furniture
kitchen
food
NFRN50 Random
(b) BEIT
NFRN5C Probe F1 per COCO Supercatogory
CUP Probe F1 per COCO Supercategory
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
appliance
vehicle
kitchen
indoor
animal
electronic
food
accessory
sports
furniture
person
outdoor
sports
animal
person
vehicle
food
electronic
appliance
furniture
kitchen
indoor
outdoor
accessory
NFRN50
(d) CLIP
Figure 11: Probes trained on COCO images to classify the supercategories of the objects in the
images
31

---

Published as a conference paper at ICLR 2023
1.0
NFRN50 Random Proce I'1 per COCO Object Class
0.6
0.4
(a) NERN50 Random Per COCO object label probe Fl results
1.0
BEIT Probe F1 per COCO Object Cless
0.4
0.2
(b) BEIT Per COCO object label probe Fl results NFRN50 Probe F1 per COCO Object Class
1.0
0.8
0.6
0.4
0.2
0.0
Kasup
(c) NFRN50 Per COCO object label probe Fl results CUP Proce F1 per COGO Object Cless
1.0
0.6
0.4
0.2
0.0
Orty
REn
pessun
(d) CLIP Per COCO object label probe Fl results
32

---

Published as a conference paper at ICLR 2023
F1 of Probe vs F1 of Concept Appearing in Generated Captions for NFRN50 Random
1.0
Probe
0.8
Caption
0.6
0.4
0.2
0.0
person
boat
car
bicycle
airplone
bus
train
truck
banch
zabra
bird
cat
dog
elophant
bear
grafffe
backpack
umbreila
tie
suitcase
kite
bottle
cup
fork
spoon
bowl
piza
banaaaa
carres
cake
chair
couch
bed
table
toilet
laptop
mouse
remote
keyboard
oven
sink
book
clock
vase
teddy bear
motorcyce
dining
Probe vs of Concept Appearing Generated Captions BEIT
1.0
Probe
0.8
Coptton
0.6
0.4
0.2
0.0
person
bost
metorcycl
train
bitycle
birglane
car
bus
truck
bench
zebra elephant horse sheep
cow
dog
cat bird
backgack umbrella
Ei
suitcase
skateboard bottle
cup
fork
spoon
bowl
pizza
calee
orange
apple
carrot
bamana
bed
toilet
couch
chair
dining table
remote
keyboard
sink
oven
dockk
vase
book
teddy bear
F1 of Probe F1 of Concept Appearing in Generated Captions for NFRN50
1.0
Probe
0.8
Caption
0.6
0.4
0.2
0.0
person
train
bost
bus
motorcycle airplane
bicycle
car
truck
bench zebra eleghant giraffe sheep
dog
bird cow bear
umbrella suitcaso baackpad packpackk
tie
skateboard
bow
cup
fork
bottle
spoon
pizza
bantana
cake
carrot
agple
tollet
bed
chair
couch
dining table
keyboard
nemote
sink
oven
clock
teddy bear
vase
book
F1 of Probe vs F1 of Concept Appearing in Generated Captions for CLIP
1.0
Probe
Caption
0.6
0.4
0.2
0.0
person
frain
bus boat
airplene motorcydle bicydde
car
bench
elephant zebra
giraffe
sheep
dog
bear
cow
bird
umbrelle suitcase
tie
backpack
bowl
cup
bottle
spoon
pieza
banaña
carrot
apple
tollet
bed
couch
chair
table
laptop
tv
keyboard
mouse
remote
sink
oven
clock
teddy bear
vase
book
dining
Figure 13: Fl of image encoder probes trained CC3M and evaluated COCO. We find that Fl of captions by object category tend to follow those of probe performance. Notably the BETT probe much worse at transferring from CC3M COCO, and the captioning tends to be consistently higher which makes it difficult to draw conclusions for this model. Generally, it appears the abil- ity to encode lexical information into the image representation entails being able to transfer that information to the LM with linear map.
33

---

Published as a conference paper at ICLR 2023
NFRN50 Random Probe per CIFAR Coarse Label
BEIT Probe F1 CIFAR Coarse Label
0.0
0.0
trees
flowers
large man--maaaaoo www.oooddoooooo thinga fruit. vegetab
squatic mammuuls
household electrical deviices vehicles_
household fumiture
fish food, containers
Insacts
mediume ynammall
lagg, sonnvorre, Lnd herbivores
large, carnivores
vehicles 2
small mammals
repbiles
Invertabrates
people
flowers
fruit and vegelables
large man-made outtoor things
household. Saauehhol_fuumme fumiiture
vehic I
containars
insects
large conivooooe and herbivores
carnivores
household electrical devices
vehicles, 2
squatic mammalls
fish
small mammals
medium mammalls moninnezt invertedrates reptles
natural outdoor,
large natural putdoor
(a) NERN50 Random
(b) BEIT
NFRN50 Probe F1 CIFAR Coarse Label
CLIP Probe F1 CIFAR Coarse Label
0.8
0.6
trons
large natural outdoor soanes
flowers
fruit and vegelables
people
lagg poi pran-miin wee-inaaeeee outdoor thingg
household. furniture
food contalners
devices
vehiclee_ 1
insects
vehicles. 2
fish
sennwore and hebitoorss
aquatic, marnete
large caamivorree
Inverobratte reptiles
medium, mamamals
mammais
people
bese
flowee
househald furniture
containers
fruit and vegetables
large Marge/nin man--mdde. outdooy things
hourehold electrical devioes
large. natural, cutdoor scenes
vehiclee_
insects
vehicles_ 2
fish
large- comnvvree herbavores
lurge cariivores
non-isses Invertabrates
equalic mamimals
repélls
medium, mammals
small mammals
noniineet
househod
NFRN50
(d) CLIIP
Figure 14: Probes trained on CIFAR images to classify the coarse labels of the objects in the images
34

---

Published as a conference paper at ICLR 2023
NFRNSO Randon Proto F1 OIFAR Orjpct
1.0
84
(a) NFRNSO Random Per CIFAR object label prohe Fl results
per Ofject Clsss
64
(b) BEIT Per CIFAR object label probe Fl results
NYroed Propa per CIPAI Objed Claes
64
(c) NFRN50 Per CIFAR object label probe F1 results
1.
P Prote F1 per OFFR Objest Cles
(d) CLIP Per CIFAR object label probe Fl results
35


Reviews:
[{"review_id": "qTY5u4kDXy", "reviewer": "Reviewer_kxWt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple idea to test a clearly-defined hypothesis: whether representations learned by a vision model (trained on images only vs. trained on images including limited degrees of linguistic supervision) are functionally equivalent to those learned by a language model (up to a linear transformation). The authors report on different experiments and show that the degree of linguistic supervision used in the (frozen) image encoder matters, and the more linguistic supervision leads to better results on captioning and VQA. In turn, this is interpreted as the extent to which visual representations are functionally equivalent to textual representations (up to the linear transformation).", "summary_of_the_review": "Comments and questions:\n\n- Figure 4 has no number.\n\n- Why did you not have a BEIT-Large model random initialised as a baseline? That would have added much information to the NFRN50 random baseline, since it would allow you to compare both BEIT-Large and BEIT-Large random in a similar fashion to what you did with NFRN50.\n\n- I suggest clearly defining what is meant by conceptual/visual properties vs. lexical categories already at the introduction. You state early on in the paper that you find that all three models distinguish the former, but the more textual supervision models have during pretraining, the better models distinguish the latter. It would be clarifying to have one- or two-sentence explanations of what is meant by these properties/categories, perhaps including one or more examples of each.\n\n- Page 9: \"By looking at failure cases for each model, we can establish whether each model is predicting the presence of a similar animal or not. In Figure 4a, we show that when captions generated from each model mistake one animal for another, the mistaken animals are highly similar to the ground truth animal when measuring both\nWu-Palmer similarity (Averages: BEIT: 0.8, NFRN50: 0.81, CLIP: 0.8) and overlap of AWA properties (Averages: BEIT: 0.62, NFRN50: 0.68, CLIP: 0.59)\" -> There is an implicit 'if' statement in this part of the text that troubles me. You say that \"when captions mistake one animal for another\" something happens. But how often do models make a mistake that is of a different nature, e.g., generate an spurious caption? Does BEIT do that more often than NFRN50 and CLIP? I would like to see these numbers.\n\nTypos and presentation:\n\n- \"k is determined by the architecture of [the] E\"\n\n- In the Appendix, some of your Tables should be resized.\n\n- Page 19: \"All probes are [trianed]\"\n\n- The last Figure in your Appendix has no number.\n\n- In your appendix, you include many figures. Perhaps it may be a good idea to try to be more concise?\n", "strength_and_weaknesses": "Strengths: \n- Paper has a clear hypothesis and story.\n- Paper's results are compelling and substantiated by well-thought experiments, which cover the important sources of variation that affect the hypothesis laid out.\n- Good balance between empiricial decisions and theories to support those decisions.\n\n\nWeaknesses:\n- None that I can think of.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pCDOik936v2", "reviewer": "Reviewer_Sxvp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed to learn a linear mapping from vision embedding space to text embedding space of a pretrained language model. The authors provided in-depth analysis and comparison between vision-only pretrained models and (text-)supervised vision pretrained models. The effectiveness is shown through baseline comparisons.", "summary_of_the_review": "Overall, the reviewer thinks the analysis provided in this paper could be possibly helpful for the community but currently several cons need to be resolved towards a publication at ICLR.", "strength_and_weaknesses": "Pros:\n1. The idea of linear transforming is interesting.\n2. The comparisons between BEIT and other (text-)supervised models are interesting.\nCons:\n1. Significantly lack of discussion with relevant work. These works are highly relevant and even somewhat similar to the proposed method. Therefore, it is necessary to provide more comprehensive literature review to clarify the distinction with these work.\na. [r1,r2,r3,r4] reports a line of alternative method of leveraging pretrained (text-)supervised models for visual recognition: directly using text words to represent the information in visual input, which is a highly relevant baseline to the proposed method: they all use very few parameters or even no new parameters to bridge vision and text.\nb. [r5] reports the possibility of reading out the visual information from any pretrained vision only models by tuning a text encoder with the image model fixed through contrastive learning.\n\n2. The core assumption \"structural similarity between the two spaces\" is not directly tested. This could be checked by simply compare the vision similarity graph of objects from a few categories and the text similarity graph of the text embeddings of those categories.\n\n3. The authors assumed the linearity of the transformation from vision space to textual space, which is certainly not the case in the real world as the reviewer believe simple contradictions could be found: especially when there are non-visual correlation between two words, the distance between vision space could be non-correlated with the distance in text space. For example, blue and a depressed person. \nTherefore, the review takes the proposed idea as an approximation of the real world. But the author didn't provide any proper-upper bound for this.\na. For example, the linear transforming could be progressively added to a MLP. \nb. Tuned MAGMA is not considered as a proper upper-bound because this is essentially a trade-off between non-linearity and the number of examples that could be used to train the non-linear mapping.\nc. MAGMA also reports results on more datasets, which should be ideally also provided for the proposed method.\n\n\n\nMinor:\nFigure 5, the font size and font seem to be inconsistent.\n\n[r1] VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs, CVPR 2021\n\n[r2] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language\n\n[r3] Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning\n\n[r4] Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners\n\n[r5] LiT : Zero-Shot Transfer with Locked-image text Tuning", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "XVFf3rXbweW", "reviewer": "Reviewer_N4fG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies to what extent models trained on image-caption-only data can be frozen, passed through a linear projection, and then provided to a generative LM to generate captions for images. This builds off some earlier models/ideas like Frozen (Tsimpoukelli et al 2021) and Magma (Eichenberg et al 2021),  except here, only one linear projection is learned -- mapping an image encoder to the vocabulary space of a language model. In contrast, Frozen finetunes the whole image encoder, and Magma uses adapter layers to minimally finetune the image encoder.\n\nThe paper studies how well various image encoder learning paradigms can be used here -- BEIT (which was trained only on image patches), imagenet supervised training, and CLIP. When CLIP is plugged in here, CLIP performs well across the board on VQA and image captioning; even outperforming the authors' tuned implementation of Magma in a 0shot setting on VQA.\n\nThe paper dives further and probes how well concepts transfer (e.g. nouns, modifier, relations on COCO; perceptual concepts from Animals with Attributes). ", "summary_of_the_review": "Overall, to this reviewer, this paper seems impactful and I'd vote to accept it -- would still be curious as to how well models of different sizes do under this evaluation.", "strength_and_weaknesses": "Strengths:\n* This paper makes a simple yet important extension to the Magma / Frozen works -- adding the simplest possible adaption layer from an image-only encoder to a language model. The simplicity also means that there are plenty of opportunities to probe what the underlying image encoder learns during pretraining. For that reason, I think this paper could be informative to researchers who work in a variety of different communities. The model also could be a great baseline for further vision-language models.\n* The analysis about what fine-grained concepts are learned by the various models seems very detailed and interesting to this reviewer. The methodology could be useful for other works that want to probe these models, and to perhaps make them better.\n\nWeaknesses:\n* To this reviewer, the paper could be improved if it considered different model sizes (e.g. larger or smaller CLIP models or supervised resnets or BEITs). An interesting question is whether the probing results get better or plateau with model size (of both the encoder and perhaps the decoder; which here is GPT-J (6B parameters).\n\nMiscellaneous: I think there is something weird with the font in this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GSeuAkMH9h", "reviewer": "Reviewer_s1Pu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies whether learning a linear layer is good enough to map image features to text space. Three different image encoders are tested and it's found CLIP performs best on tasks requiring fine-grained category visual information.", "summary_of_the_review": "Overall, I lean toward rejection because of limited novelty and low performance.\n\n`After Rebuttal:`\n\nI agree that it's an interesting finding that people should know. But it doesn't surprise me because as I said, many papers have explored this direction, and this finding shares similar intuition as [Shen, et al.]. Overall, I would raise my score to weak reject but I am also okay with accepting it.", "strength_and_weaknesses": "Strength:\n1. How to bridge pre-trained image models and pre-trained LM is quite an interesting and intriguing problem. And it can enable desirable downstream tasks such as few/zero-shot VQA utilizing stored knowledge from both worlds. \n2. This paper verified a strong hypothesis: training only a linear layer is enough for mapping visual pre-trained knowledge to text space.\n3. In experiments, this paper goes through recent popular visual pre-trained models and brings a relatively comprehensive analysis.\n \nWeakness:\n1. A major concern is the novelty of the paper. As admitted in this paper, Frozen also adds and tunes a linear mapping layer to process image encoder output features for LM input. The only technical difference against Frozen is that this paper doesn't update visual encoders. It's a good finding but not enough for publication at ICLR.\n2. Performance vs MAGMA. In Tab1, when reporting image captioning results, only MAGMA(released) and MAGMA(ours) are compared while MAGMA(reported) is missing. If compared with MAGMA(reported), the proposed method in this paper would perform worse in NoCaps on four metrics. On the other hand, for VQA, the best result of MAGMA is from RN50x4. But in this paper, author(s) compares results on RN50x16 and the proposed method still underperforms on 1-2-4 shot VQA. \n3. It's already found in other works that CLIP is better than other non-text-guided visual pre-trained models in language/multimodal understanding [1]. What this paper found echoes similar findings before.\n\nRef:\n[1] Shen, Sheng, et al. \"How Much Can CLIP Benefit Vision-and-Language Tasks?.\" ICLR 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster