Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to use external knowledge for learning multi-modal representation from text and images. The authors extract a query given a text description attached to an image. They use the query to extract more informative descriptions from the freely available external knowledge. The externally obtained textual description is then paired with the image to enhance the jointly learned visual representation. 
strengths_and_weaknesses: This paper's idea is simple, effective, and properly presented. The paper is technically sound, and the experiments match NeurIPS standards.

The only weakness I see in this paper is that the authors are content with reporting the final performance to validate the proposed ideas. They show more detailed analyses in the appendix. However, I believe conducting a set of ablation studies will strengthen the paper. See below for more details.
questions: As I mentioned above, the authors could perform some interesting experiments to elaborate more on performance. For instance, which classes benefit more from exploring external knowledge? Can it be somehow quantified based on a rareness score? According to the main idea proposed in the introduction, the rare classes should benefit more from external knowledge. You already have a measure to determine the rareness of noun phrases used in the query extracting process. Can you use the same measure to see if there is a correlation between the rareness of a given class and the obtained boost after using external knowledge? 
limitations: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes K-LITE which leverages external knowledge as a more structured source of supervision signals for building transferable visual models with knowledge-augmented language-image pre-training. External knowledge can enrich the natural language supervision by explaining rare concepts with more common concepts thus increasing the concept coverage. K-LITE proposes systematic steps to construct queries to get knowledge from external sources such as WordNet or Wiktionary to facilitate the pre-training and evaluation for image classification and object detection tasks. To bridge the gap between pre-training and evaluation, the paper utilizes adapters to further modularize the original language encoder with external knowledge data. With extensive experiments conducted, the paper demonstrates K-LITE’s great generality for downstream datasets and data effectiveness on both image classification and object detection tasks.
strengths_and_weaknesses: Strengths:
1. The paper is well-written and easy to follow. The motivation of leveraging external knowledge to help language-image pre-training is natural and clear.
2. With knowledge-augmented training and evaluation, K-LITE achieves superior task-level transfer performances on 20 image classification and 13 object downstream tasks. In addition, K-LITE can get comparable performance as UniCL with only half of the pre-training data, showing great data efficiency.
3. The paper provides detailed and persuasive analyses and discussions to confirm the design choices of K-LITE. The analysis of train-evaluation concept overlap (increased from 13.26% to 51.24% with knowledge) well explained why external knowledge can facilitate language-image pre-training and task-level transfer to downstream IC and OD tasks.

Weaknesses:
1. While the motivation is natural, it is not surprising. Since many prior works have leveraged external knowledge for language-only and vision-language tasks before. Even for know-augmented visual recognition, there is at least a similar idea proposed [1], but the K-LITE paper didn’t cite [1]. Therefore, the technical novelty seems relatively limited.  *[1] Tian C, Wang W, Zhu X, et al. VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition[J]. arXiv preprint arXiv:2111.13579, 2021.*

2. Table 1 reveals that training/test inconsistency in terms of involving knowledge can dramatically degrade the model performance. Thus, to ensure higher performance, during evaluation, detailed explanations for noun phrases always need to be concatenated as knowledge prompts. This seems to be not aligned with the motivation of the paper and the spirit of utilizing external knowledge. For example, if we already what is “sashimi”, we don’t need to find the definition every time when we see the word later. I think this paper didn’t solve this motivation-reality gap very well. Even though applying K-Adapter can alleviate the problem, it didn’t bring any improvement in terms of accuracy compared with knowledge-free pre-training according to Table 1.
3. Since detailed explanations are appended to noun phrases, the length of the input sequence will be much longer than the previous knowledge-free category name or caption. This will definitively result in longer inference time and larger training costs. Although K-LITE enjoys better data efficiency, the author should also provide the training and inference budgets to show whether K-LITE possesses another aspect of efficiency.
4. From Figure 4 in the main paper and Figure 5 in the appendix, we can find that the performance and improvement of K-LITE largely depend on the quality of the external knowledge base. The method itself cannot guarantee an improved performance at the cost of external organized/structured data under some circumstances (just as the Limitations subsubsection stated).
questions: 1. In Section 3.1, how to compute the rare level of a noun phrase? Note that in Line 153, the paper said that the most *rare* noun-phrase over the corpus is used as a query for the sentence.
2. For the example sentence in Line 156, will all the three noun phrases “boxer”, “professional boxer”, and “the crowd” be used as queries? If the answer is yes, this is not aligned with “the most *rare* noun-phrase” in Line 153.
3. Typo in Line 279: “therefor” should be “therefore”, “mudularized” should be “modularized”.
limitations: The paper has reported limitations and offered satisfactory explanations. No obvious negative societal impact was found.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a knowledge base augmented image-language model. The main architecture is an image-text contrastive training like CLIP. The key idea is for the text input, a "keyword" (a noun) is extracted and used as a query to search in multiple knowledge bases to get the word hierarchy or definition. This additional textual information is also input to the image-language model as a language prompt. The model is evaluated on multiple downstream datasets on image classification and object detection tasks. Results show incorporating an external knowledge base is effective. 

strengths_and_weaknesses: The paper acknowledges related works (L111) that incorporate knowledge base in the NLP or vision-language models. The paper focuses on the transfer ability (transfer to specialized datasets and rare objects) of such a knowledge base augmented model, which is novel in the field. 

The quality of the paper is very good. The ablation studies are well conducted. The experimental results are strong. 

In general, the paper is clearly written and well organized. 

The experimental results show the proposed method works well on a wide range of specialized datasets for both image classification and object detection tasks. This method of incorporating a knowledge base can be applied to other research.

questions: 1. In Table 3, the bold number 21.3 is confusing. There is a 21.4 in the same row under LVIS tab. What's the meaning of bold font here?
2. In the query construction section (L144) the paper chooses a single noun as a query to search in the knowledge base. Are there more discussions on this design choice? For example, if it's helpful to query `adj+noun` for a more accurate definition, or sometimes `verb+noun` if the text input describes an action.

limitations: There is no major negative societal impact of the work. The paper also briefly discussed it in the appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper tackles the poor transfer learning performance of current visual models. It presents a systematic approach to leveraging external knowledge databases to train more transferable and sample-efficient visual models. The paper presents sufficient evidence that external knowledge can benefit large-scale task-level transfer for two core vision problems, image classification, and object detection. The model can achieve similar zero-shot performance to previous methods when using only half of the pre-training image-text pairs in some scenarios.
strengths_and_weaknesses: Strengths:
1. The motivation is clear. Leveraging external knowledge is promising to increase the concept coverage of vision-language models.
2. Novel solution. Two external knowledge sources including WordNet and Wiktionary are explored to enrich the natural language supervision, which is then used in both the language-image pre-training stage and the prompt-based evaluation stage.
3. Sufficient experiments. The experiments on two downstream tasks with several datasets support the main claims of the paper.

Weaknesses:
1. For knowledge acquisition, when multiple meanings exist for a given query, the method simply considers the first definition. Does the authors analyze the proportion of polysemy and the correct rate of the first-word sense?
2. In order to train more transferable and sample-efficient visual models, the paper instantiates K-LITE with two tasks to demonstrate the generalization ability. Have the authors tested on more tasks, like image-text retrieval?
3. In Table 2, why is the performance of 2-branch lower than 1-branch from scratch?
4. Why not consider other types of external knowledge sources like ConceptNet and knowledge graphs?  What are the advantages of the currently used knowledge sources over others?
questions: Please see the weaknesses.
limitations: Limitations and potential negative societal impacts are adequately addressed.
review_title: null

Meta-review: metareview: This paper presents a simple yet effective technique for leveraging external knowledge to learn multi-modal representation. The performance on image classification and zero-shot object detection dataset is surprisingly great. All reviewers give positive comments. After the discussion phase, the authors revise the paper accordingly and provide zero-shot retrieval comparisons, more quantitative analysis, reevaluate the checkpoint. The rebuttal satisfied the reviewers. The meta-reviewers thus recommend accepting it.
confidence: Certain
Decision: Accept