Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a new benchmark to evaluate transferability of models that learn visual representations from language supervision consisting of 20 image classification (IC) and 35 object detection (OC) datasets. They include an evaluation toolkit with a simple hyperparameter optimization method and define metrics for sample-efficiency and parameter-efficiency. The benchmark additionally includes a collected external knowledgebase for each dataset. The overall goal is the evaluate task-level transfer across domains, the ability of pretrained models to adapt to new tasks.

The authors evaluate 7 IC / 4 OD pretrained models and propose new model adaption methods, i.e. new methods to evaluate a model like CLIP. Extensive evaluation of the models on the new benchmark is done with regards to usefulness of language pretraining, choice of language-augmented initialization method, sample efficiency, parameter efficiency, external knowledge.
strengths: The benchmark is useful to evaluate task-level transfer. It contains many datasets with two task types (IC and OD) instead of only one. It should help achieve reproducibility in this field. Adding knowledgebases to the datasets is very helpful to standardize evaluation there, as otherwise every work would build its own knowledgebase.

In the general the work is exhaustive in all dimensions, there are enough tested models, ablated evaluation strategies, metrics to give a complete picture of both the quality of the benchmark as well as the performance of the models tested on that benchmark.

The toolkit code and the datasets are all released publicly so no information is missing to the reader.
weaknesses: This work has a lot of content with results distributed over various tables and figures in both the paper and supplementary, so it can be hard to understand. To elaborate:

In table 2 you list that you evaluated 7 IC and 4 OD, models but some of the results seem to be missing, e.g. I could not find the K-LITE performance for OD. Maybe you could create a table in the appendix in the style of table 2 where you list the results for each model so the reader doesn't have to collect this information from various places.

It seems you have improved over the original CLIP performance (line 191 "superior empirical performance") with the new language initialization method. This is hard to confirm with so many tables, methods, evaluation strategies and so on and also since you used only a small CLIP (B32). What would be interesting would be to compare with the original CLIP paper numbers directly, e.g. in their paper appendix table 10 row 8 model "L/14-336px" on ImageNet with linear probe achieves 85.4% accuracy, can you improve over this with your new language init methods?

- Figure 2 has some weaknesses that make it hard to understand:
    - |B| is not defined anywhere, this is probably the batchsize / dataset size. If it is not necessary it could be removed, i.e. having the figure and text talk about a single image x instead of many images. Otherwise it should be defined.
    - Line 170 describes projection layers W_u for image and W_v for text, however in figure 2b) its called W_t instead (probably a typo).
    - In figure 2b) the output of text encoder is not shown, as I understand it's shape DxK (class text prompts encoded as text), this should be added explicitly.
    - Having W with shape DxK in figure 2a), 2e) but with shape PxK in 2c) is confusing, if the shape changes it should probably get a different name.
    - Text uses lowercase u, v and figure uses uppercase U, V, without explicitly defining the difference. It can be assumed that lowercase are single datapoints and U, V are multiple datapoints, but this should be made explicit.
    - If I understand it correctly, methods 2b) and 2d) end up being mathematically the same during zero-shot but different during finetuning. If this is true, this information should be added to chapter 4 instead of having the reader deduct it.

Table 3: You are comparing the methods in figure 2c,d,e (Random, Lang-S, Lang-M). Why not compare also the original CLIP setting 2b)? Is the comparison unfair, if yes, why?
    
Table 3: Why is no result given for row 2 (CLIP, LP, Language-S)? I understand that with random linear heads zeroshot doesn't work, but with Language-Init (Separate) as in Fig 2d) it should work.

In line 239 the claim "CLIP outperforms Supervised ViT consistently" is not clear from table 3 as the Supervised ViT FT matches CLIP performance with enough data (50-shot and full). This contrasts your finding in line 48: "Leveraging ... text and vision ... consistently yields better performance" so you should try to explain why here this is not true and vision-only matches the performance of vision-text instead of vision-text yielding better performance.

The prompts are not described at all, this could be done in the appendix. e.g. are the prompts actually sentences like "This is an image of a dog." or only a word like "dog".

It is not stated exactly how the external knowledge is added, the reader has to infer from the hints in line 191 and 292 that the knowledge is added to the class prompt before feeding everything into the text encoder f. Details are missing, e.g. with GPT-3 there are 5 knowledge sentences generated, do you concatenate all 5 to the prompt, or generate multiple prompts, or etc. Also, which special tokens (CLS / SEP) or similar are used when constructing the input to the text encoder?

The hyperparameter tuning is very simple with grid search and only the 2 hyperparemeters learning rate and weight decay. Why was this approach chosen? Why not a more sophisticated method like BOHB (2018), DEHB (2021) or others?

correctness: Everything can be checked publicly and seems correct
clarity: See weaknesses
relation to prior work: Yes
additional feedback: No
review_title: New benchmark for task-level transfer of visual models pretrained with natural language supervision.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents dataset and benchmark tools for image classification and object detection. The authors combined 55 datasets and created benchmark and evaluation tools. They augmented datasets with Wordnet, Wiktionary and GPT3 language inputs and created baselines for the combined datasets.
strengths: There is plenty of work done to merge the datasets and create automatic annotations. There are detailed baselines for all combination of training methods and there is also analysis of the results.
weaknesses: The paper is very hard to read. The structure of paper is very scattered and it is not easy to follow central topic of the paper. The reader is overwhelmed with the details, but as they are not explained it is hard to understand them. Also the language quality is very low that makes the paper almost unreadable.
correctness: Yes
clarity: The paper needs to be rewritten to be suitable for NEURIPS conference. 
relation to prior work: Yes
additional feedback: Please restructure the paper and polish the English.
review_title: Official review of paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Introduce ELEVATER, a benchmark and toolkit for evaluating (pre-trained) language-augmented visual models. 
(i) Datasets. As downstream evaluation suites, it consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. 
(ii) Toolkit. An automatic hyper-parameter tuning toolkit is developed to facilitate model evaluation on downstream tasks. 
(iii) Metrics. A variety of evaluation metrics are used to measure sample-efficiency (zero-shot and few-shot) and parameter-efficiency (linear probing and full model fine-tuning).
strengths: 1, V-L pretraining is very hot these days.
2, The benchmarking includes settings such as zero-shot, few-shot, and full-shot settings. 
3, Reproducible Toolkit & Language-augmented Adaptation Methods.
4， A lot of OD datasets. Better than just using LVIS.


weaknesses: 1, Some existing conclusions and findings in the original CLIP paper, e.g. CLIP is better than some self-supervised methods such as MoCov3, MAE; CLIP sample efficiency is better than supervised ViT. Not surprising.
2, Limited comparison in the Image classification parts (only  CLIP) in Industry Track. How about BLIP, FILIP, DECLIP? The authors should compare more methods to have a more complete benchmarking.
3, How about fully supervised results for object detection without language aids in Table 4. Not consistent with Table 3.
4. More comparisons can be added in Table 4 such as X-DETR and so on.
5. How about the automatic hyperparameter tuning toolkit? Do you have a comparison of the benefit of using it?
correctness: Yes
clarity: Yes
relation to prior work: Yes
additional feedback: None
review_title: A benchmark for some visual-language models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces ELEVATER, a benchmark to evaluate the transferability of vision models to new datasets and tasks. The benchmark collates a large range of image classification and object detection datasets, that pre-trained vision models (such as CLIP and ViT) can be transferred to. The benchmark also contains evaluation protocols for measuring transferability in different n-shot settings, and also suggests language and knowledge augmentation techniques to add the cross-task transfer. The authors perform extensive experiments, using both language-free and language-augmented vision models, to evaluate how these models transfer to new vision tasks in terms of both sample efficiency and tuneable parameter efficiency.
strengths: 1. The benchmark is well motivated. It is clearly described how it is different from previous work.
2. The  novel industry and academic tracks in the benchmark are a really nice addition, as they make for a more equitable evaluation of vision models.
3. The proposed methods for task adaptation via language initialization are well motivated and show convincing results.
4. The experiments are very thoroughly designed, and the results are convincing. The results are mostly consistent with the hypotheses.
5. The experiments for integrating knowledge into task transfer are well designed, and the use of knowledge is also well motivated.
weaknesses: 1. The choice of aggregating scores across tasks, while standard, is questionable, since equal improvements on different tasks are not equivalent. Aggregating scores does not give us insights into what tasks the improvements are actually seen on. For instance, a 1% improvement on a task with 80% accuracy is not the same as 1% improvement on a 50% accuracy task. Further, tasks with different number of output labels have different baselines, and aggregating the scores does not account for the variance in those tasks' baseline scores.
correctness: The aggregation of scores is arguably not wrong, but does not accurately reflect model improvements in my opinion.
clarity: The paper is clearly written, and easy to follow.
relation to prior work: The paper clearly compares its benchmark against several zero-shot and few-shot image classification datasets/benchmarks, but only compares against a single object detection dataset. More discussion of object detection datasets and benchmarks would be useful.
additional feedback: -
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: ELEVATER is a benchmark toolkit for evaluating visual and text models. For image classification, it is an aggregation of 20 datasets with 1151 concepts and 3 text external knowledge bases. It allows for full, zero-shot, and few-shot evaluation. For object detection, it is an aggregation of 35 datasets and 314 concepts with 3 text external knowledge bases. Authors claim to be the first public
benchmark to standardize the zero-shot task-level transfer setting. They also provide a hyperparameter tuning toolkit to help evaluation. Using the benchmark, they have shown that it is not enough to just transfer from the visual encoder but the language-encoder should also be utilized. In summary, the authors provide a toolkit that automates hyperparameter tuning for downstream image classification and object detection. They evaluate 1) how language-visual models perform in comparison to visual-only models 2) how (I think) pre-training with additional language information improves downstream performance 3) and how different downstream approaches like FT, LP and using language in addition to visual change performance. I would consider their major contributions are their "language-init" approach and their comprehensive model comparisons as shown in the Supplementary for paper content and the toolkit for overall.
strengths: The authors propose a way to use language information called "language-init" for downstream learning. This approach showed that using language+visual information downstream improves performance on few-shot and zero-shot learning. This is an interesting approach that does not rely on prompt engineering.

The authors run comprehensive evaluation on a variety of language-based models and some visual-only models. They also explore pre-training strategies on the language side. They provide some interesting insights on how language can be used in downstream learning for visual-only tasks. 

weaknesses: While Figure 2 is sufficient, it would be useful to have an overall visual depiction of the evaluation pipeline. While the paper is well written, it is still a little difficult to follow and a visualization of the overall process may help. 

It is not clear to me how the external knowledge is used for evaluation. I understand you are using it to generate descriptions of a visual object for example, but it is not clear how that is used in the evaluation. For example, in Section 5.4 "We find that leveraging external knowledge improves upon the knowledge-free pre-training counterparts (UniCL and GLIP)."  in lines 286-287 but there is no clear explanation of how you are leveraging external knowledge. Is this just during pre-training? And if it is used in pre-training, it is not clear how.

The paper leads you to believe that there would also be an including of text analysis in this benchmark. You ask how using language improves visual learning, but why not the reverse? 

There is a lack of introducing prompting. Even though I am very familiar with prompting for zero-shot learning, I did not truly understand some of the details of the prompting strategy used until I explored the repositories related to this paper. 

Overall, my major concern is the lack of clarity on the approaches. The prompting strategy used is extremely unclear in both the paper and the appendix, there is too strong an assumption that the reader should have full understanding of CLIP and prompting. The pre-training strategies are also confusing although more clear in the appendix. 
correctness: This is the first toolkit to my knowledge. Others are related to visual reasoning tasks for visual-text models. 

The evaluation techniques and experimental designs were performed correctly and explained in great detail. 
clarity: The paper is very well written but the content can be quite confusing. As mentioned earlier, some of the confusing items are:
1) How external knowledge is used was not clear in neither the paper. Is it for prompting? Is it for pre-training? I know it is in more detail in the appendix, but it should be clearer in the main paper as well. 
2) That the focus was on downstream visual tasks not joint visual-text evaluation. 
3) Whether they were using existing pre-trained models, pre-training themselves via the toolkit or some combination of. 
4) How is prompting used, is external knowledge included in the engineering or are you using existing methods for zero-shot prompting? 

relation to prior work: There was no reference to any previous benchmarks that include the text domain of visual question answering, video captioning, etc. I think it would be beneficial to discuss these in isolation and how this work is different or addresses limitations to existing ones. 

I also think prompting should be mentioned for zero-shot learning in the related works. 
additional feedback: Table 1 should be more clear on what IC and OD are in case someone is skimming for information, especially because the table provides a great deal of information readers will refer to.

Figure 3 should be more specific to what LP and FT are for skimming.

In section 5, you propose two "research questions", but they are not actual research questions, more like statements of what you are addressing. 

As mentioned earlier, my main complaint is the clarity of the benchmark in relation to prompting, pre-training and the use of external knowledge.
review_title: Review of ELEVATER : A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

Meta-review: metareview: The paper presents a new exciting benchmark for evaluating pre-trained vision+language models. It was reviewed by five experts, who are uniformly supportive. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept