Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed to learn a linear mapping from vision embedding space to text embedding space of a pretrained language model. The authors provided in-depth analysis and comparison between vision-only pretrained models and (text-)supervised vision pretrained models. The effectiveness is shown through baseline comparisons.
strength_and_weaknesses: Pros:
1. The idea of linear transforming is interesting.
2. The comparisons between BEIT and other (text-)supervised models are interesting.
Cons:
1. Significantly lack of discussion with relevant work. These works are highly relevant and even somewhat similar to the proposed method. Therefore, it is necessary to provide more comprehensive literature review to clarify the distinction with these work.
a. [r1,r2,r3,r4] reports a line of alternative method of leveraging pretrained (text-)supervised models for visual recognition: directly using text words to represent the information in visual input, which is a highly relevant baseline to the proposed method: they all use very few parameters or even no new parameters to bridge vision and text.
b. [r5] reports the possibility of reading out the visual information from any pretrained vision only models by tuning a text encoder with the image model fixed through contrastive learning.

2. The core assumption "structural similarity between the two spaces" is not directly tested. This could be checked by simply compare the vision similarity graph of objects from a few categories and the text similarity graph of the text embeddings of those categories.

3. The authors assumed the linearity of the transformation from vision space to textual space, which is certainly not the case in the real world as the reviewer believe simple contradictions could be found: especially when there are non-visual correlation between two words, the distance between vision space could be non-correlated with the distance in text space. For example, blue and a depressed person. 
Therefore, the review takes the proposed idea as an approximation of the real world. But the author didn't provide any proper-upper bound for this.
a. For example, the linear transforming could be progressively added to a MLP. 
b. Tuned MAGMA is not considered as a proper upper-bound because this is essentially a trade-off between non-linearity and the number of examples that could be used to train the non-linear mapping.
c. MAGMA also reports results on more datasets, which should be ideally also provided for the proposed method.



Minor:
Figure 5, the font size and font seem to be inconsistent.

[r1] VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs, CVPR 2021

[r2] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

[r3] Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning

[r4] Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners

[r5] LiT : Zero-Shot Transfer with Locked-image text Tuning
clarity,_quality,_novelty_and_reproducibility: Clarity, quality and reproducibility look overall good. The novelty is rather limited.
summary_of_the_review: Overall, the reviewer thinks the analysis provided in this paper could be possibly helpful for the community but currently several cons need to be resolved towards a publication at ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies whether learning a linear layer is good enough to map image features to text space. Three different image encoders are tested and it's found CLIP performs best on tasks requiring fine-grained category visual information.
strength_and_weaknesses: Strength:
1. How to bridge pre-trained image models and pre-trained LM is quite an interesting and intriguing problem. And it can enable desirable downstream tasks such as few/zero-shot VQA utilizing stored knowledge from both worlds. 
2. This paper verified a strong hypothesis: training only a linear layer is enough for mapping visual pre-trained knowledge to text space.
3. In experiments, this paper goes through recent popular visual pre-trained models and brings a relatively comprehensive analysis.
 
Weakness:
1. A major concern is the novelty of the paper. As admitted in this paper, Frozen also adds and tunes a linear mapping layer to process image encoder output features for LM input. The only technical difference against Frozen is that this paper doesn't update visual encoders. It's a good finding but not enough for publication at ICLR.
2. Performance vs MAGMA. In Tab1, when reporting image captioning results, only MAGMA(released) and MAGMA(ours) are compared while MAGMA(reported) is missing. If compared with MAGMA(reported), the proposed method in this paper would perform worse in NoCaps on four metrics. On the other hand, for VQA, the best result of MAGMA is from RN50x4. But in this paper, author(s) compares results on RN50x16 and the proposed method still underperforms on 1-2-4 shot VQA. 
3. It's already found in other works that CLIP is better than other non-text-guided visual pre-trained models in language/multimodal understanding [1]. What this paper found echoes similar findings before.

Ref:
[1] Shen, Sheng, et al. "How Much Can CLIP Benefit Vision-and-Language Tasks?." ICLR 2022.
clarity,_quality,_novelty_and_reproducibility: The writing is good. The code should be easy to reproduce and the analysis/experiments are solid. But the novelty is limited based on the weaknesses I mentioned above.
summary_of_the_review: Overall, I lean toward rejection because of limited novelty and low performance.

`After Rebuttal:`

I agree that it's an interesting finding that people should know. But it doesn't surprise me because as I said, many papers have explored this direction, and this finding shares similar intuition as [Shen, et al.]. Overall, I would raise my score to weak reject but I am also okay with accepting it.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a simple idea to test a clearly-defined hypothesis: whether representations learned by a vision model (trained on images only vs. trained on images including limited degrees of linguistic supervision) are functionally equivalent to those learned by a language model (up to a linear transformation). The authors report on different experiments and show that the degree of linguistic supervision used in the (frozen) image encoder matters, and the more linguistic supervision leads to better results on captioning and VQA. In turn, this is interpreted as the extent to which visual representations are functionally equivalent to textual representations (up to the linear transformation).
strength_and_weaknesses: Strengths: 
- Paper has a clear hypothesis and story.
- Paper's results are compelling and substantiated by well-thought experiments, which cover the important sources of variation that affect the hypothesis laid out.
- Good balance between empiricial decisions and theories to support those decisions.


Weaknesses:
- None that I can think of.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and clear, and I find the experiments to be appropriate to answer the hypothesis. Reproducibility is addressed by sharing code to reproduce experiments, and in terms of novelty the paper contributes a small data point to a series of works in frozen / soft prompts in vision and language models. Even though it is a small data point, I still find it valuable.
summary_of_the_review: Comments and questions:

- Figure 4 has no number.

- Why did you not have a BEIT-Large model random initialised as a baseline? That would have added much information to the NFRN50 random baseline, since it would allow you to compare both BEIT-Large and BEIT-Large random in a similar fashion to what you did with NFRN50.

- I suggest clearly defining what is meant by conceptual/visual properties vs. lexical categories already at the introduction. You state early on in the paper that you find that all three models distinguish the former, but the more textual supervision models have during pretraining, the better models distinguish the latter. It would be clarifying to have one- or two-sentence explanations of what is meant by these properties/categories, perhaps including one or more examples of each.

- Page 9: "By looking at failure cases for each model, we can establish whether each model is predicting the presence of a similar animal or not. In Figure 4a, we show that when captions generated from each model mistake one animal for another, the mistaken animals are highly similar to the ground truth animal when measuring both
Wu-Palmer similarity (Averages: BEIT: 0.8, NFRN50: 0.81, CLIP: 0.8) and overlap of AWA properties (Averages: BEIT: 0.62, NFRN50: 0.68, CLIP: 0.59)" -> There is an implicit 'if' statement in this part of the text that troubles me. You say that "when captions mistake one animal for another" something happens. But how often do models make a mistake that is of a different nature, e.g., generate an spurious caption? Does BEIT do that more often than NFRN50 and CLIP? I would like to see these numbers.

Typos and presentation:

- "k is determined by the architecture of [the] E"

- In the Appendix, some of your Tables should be resized.

- Page 19: "All probes are [trianed]"

- The last Figure in your Appendix has no number.

- In your appendix, you include many figures. Perhaps it may be a good idea to try to be more concise?
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies to what extent models trained on image-caption-only data can be frozen, passed through a linear projection, and then provided to a generative LM to generate captions for images. This builds off some earlier models/ideas like Frozen (Tsimpoukelli et al 2021) and Magma (Eichenberg et al 2021),  except here, only one linear projection is learned -- mapping an image encoder to the vocabulary space of a language model. In contrast, Frozen finetunes the whole image encoder, and Magma uses adapter layers to minimally finetune the image encoder.

The paper studies how well various image encoder learning paradigms can be used here -- BEIT (which was trained only on image patches), imagenet supervised training, and CLIP. When CLIP is plugged in here, CLIP performs well across the board on VQA and image captioning; even outperforming the authors' tuned implementation of Magma in a 0shot setting on VQA.

The paper dives further and probes how well concepts transfer (e.g. nouns, modifier, relations on COCO; perceptual concepts from Animals with Attributes). 
strength_and_weaknesses: Strengths:
* This paper makes a simple yet important extension to the Magma / Frozen works -- adding the simplest possible adaption layer from an image-only encoder to a language model. The simplicity also means that there are plenty of opportunities to probe what the underlying image encoder learns during pretraining. For that reason, I think this paper could be informative to researchers who work in a variety of different communities. The model also could be a great baseline for further vision-language models.
* The analysis about what fine-grained concepts are learned by the various models seems very detailed and interesting to this reviewer. The methodology could be useful for other works that want to probe these models, and to perhaps make them better.

Weaknesses:
* To this reviewer, the paper could be improved if it considered different model sizes (e.g. larger or smaller CLIP models or supervised resnets or BEITs). An interesting question is whether the probing results get better or plateau with model size (of both the encoder and perhaps the decoder; which here is GPT-J (6B parameters).

Miscellaneous: I think there is something weird with the font in this paper.
clarity,_quality,_novelty_and_reproducibility: Seems clear and novel to this reviewer.
summary_of_the_review: Overall, to this reviewer, this paper seems impactful and I'd vote to accept it -- would still be curious as to how well models of different sizes do under this evaluation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:  
In this paper, the authors propose an approach to learn a mapping projection from a pretrained vision encoder to a text decoder (pretrained language model) while the parameters of those encoders and decoders are frozen. Although the technique itself might seem trivial from an engineering viewpoint, this paper deeply addresses a clearly-defined scientific hypothesis: whether representations learned by a vision model are functionally equivalent to those independently learned by a language model (up to a linear transformation). The paper presents a somewhat surprising results that just learning a single linear layer is as good as the previous methods based on fine-tuning or non-linear adapters in terms of the aligning visual features to text space. The authors demonstrated that this simplicity (mapping approach) also enables that there are plenty of opportunities to probe what the underlying image encoder learns during pretraining and how they can be transferred. Where, the authors conducted systematic experiments and show that the degree of linguistic supervision used in the (frozen) image encoder matters, and the more linguistic supervision leads to better results on captioning and VQA. 

Strengths:  
1. This paper investigates an intriguing approach that learns a simple mapping to bridge pre-trained image models and pre-trained LM.
2.  This approach enables to deeply analyze what kind of features are learned by an individually-trained visual encoders and how they can be transferred, as demonstrated in the paper. 
3.  The surprisingly good result with a linear projection is thought-provoking, and can be a firm baseline in vision-and-language studies. 
4.  The experiments are well-organized and comprehensive. The authors tested vision encoders with increasing levels of linguistic supervision in pretraining, which is reasonable and interesting. 
5. In particular, the methodology of evaluating what kind of (fine-grained) concepts are learned and transferred by the various models seems useful for other works that want to probe pretrained models.
6. Reproducibility is addressed by sharing code to reproduce experiments.

Weaknesses:  
1. From a technical viewpoint, the novelty of the method is little since it can be viewed as a simplified version of the standard methods like fine-tuning or adapters, although the simplicity does not directly mean bad.
2. Direct evaluation regarding the structural similarity between the vision and language spaces can be enhanced more. 
3. The writing of the paper is a little bit misleading in its claims.




justification_for_why_not_higher_score: This paper, while excellent in its scientific approach, does not propose something fundamentally new. In addition, given the score, a poster presentation would be appropriate.
(Having said that, spotlight wouldn't be that bad as the message of the paper is catchy.)

justification_for_why_not_lower_score: This paper received split scores of 3,6,8,8. All reviewers expressed interest in the linear mapping approach and its excellent performance. On the other hand, concerns were raised about the lack of novelty of the method, lack of comparison with other studies, and the obviousness of the results. To address these issues, the authors added explanations and some new results in the response, but the focus was still on novelty.
In the AC’s opinion, the novelty and contribution of this paper do not lie in the technique itself of learning linear mapping and its good performance, but in the approach of bridging independently learned vision encoders and language decoders with mapping only, and the new benefits it brings to the study of representation learning are well demonstrated by experiments as highlighted by two reviewers. Linear projection is its simplest form and would be an excellent baseline in this approach.
On the other hand, in relation to this point, the AC thinks the current paper is somewhat misleadingly written, as if it asserts the linear relationship between vision and language spaces. The paper should clearly convey the motivation to find out how far linear projection as the simplest form of mapping can go (and the results have been very good empirically).
Overall, the AC believes that the above problems are relatively easy to correct and that this paper will bring new insights and methodologies that researchers in the fields of vision-and-language and representation learning ought to know. In this sense, the pros of the paper outweigh the cons well, thus the AC would like to recommend acceptance.
Decision: Accept: poster