Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper performs an exhaustive empirical study to propose model patching where the goal is to improve accuracy for open-vocabulary models on specific tasks (i.e., patching tasks) without degrading accuracy on  tasks where performance is already adequate. Model patching refers to interpolation of model weights between a finetuned model and the original model. Among other experiments, the paper shows results of model patching on nine tasks where zero-shot CLIP performs poorly, and obtains improvement of over 15-60 percentage points while not losing performance on ImageNet itself. The paper also talks about broad transfer across multiple tasks, and notes that the proposed approach becomes more effective with increasing scale of datasets.
strengths_and_weaknesses: Strengths:
—-----------
+ The methodology tested and proposed is very simple, and easy-to-implement.
+ The results are interesting, and of broad relevance to the community, especially to those in large-scale ML practice.
+ The results are comprehensive, and cover experiments across various settings, including ones such as typographic attacks, counting and visual question-answering.
+ The Appendix is loaded with even more results, which make this an elaborate empirical effort.

Weaknesses:
—---------------
- The primary weakness of the paper is the limitation mentioned in the paper itself (L315-316): “....our method provides no guarantees on which data the model performance might change…”. This limits the robustness of the takeaways from the paper, and where one may be able to use them – especially considering the contributions are largely empirical.

- Considering \alpha is the key hyperparameter for the interpolation, it would have been nice to see how the performance changes with different \alphas. This, to me, is another weakness of the paper - amid the large number of results, it lacks a clearer perspective on how a reader can take away lessons that can be used in practice (especially considering practice is the focus of this work).

- In continuation to the above comment, there is no evident trend in the results (or a summary of it in the paper) to see when this method works best. 

- Considering the large number of results presented in the work (including the Appendix), the paper definitely needs a summary or discussion summarizing the lessons and takeaways for the work to be useful to the readers. It becomes the reader’s burden otherwise to sift and find the takeaways.

- The paper doesn’t compare with other papers that linearly interpolate neural network weights as mentioned in the baselines section. 

questions: * What is the summary of takeaways from the paper? The conclusion states “...it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy”. But a succinct summary of *how* this can be achieved is missing.

* One could argue that interpolating between model weights and finetuned weights is like parameter regularization techniques in continual learning, which impose a certain constraint on how far the weights can be from the original weights. This comparison is glaringly lacking in the main paper. While the Appendix presents a result with Elastic Weight Consolidation, it mentions that one needs access to the pre-training data for a complete experiment. However, there are many continual learning methods that do not need replay buffers (e.g. Gradient Projection Memory for Continual Learning, ICLR 2021 – this is only a representational example), and it may have been useful to see how such a method compares against them.

* Considering the recent focus on OOD generalization performance and robustness to corruption, how would this method work on OOD corruption benchmarks? It may be interesting to see the performance.

I will be happy to revisit the rating after the discussion with/response from authors.
limitations: Please see the weaknesses listed above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this submission, the authors attempt to improve the CLIP model on the tasks that it performs poorly. They propose to fine-tune pre-trained CLIP model on the target task with frozen classification layer and derive the final model with a linear interpolation between pre-trained and fine-tuned model. The mixing coefficient is decided by validation. This task-patching procedure could be adopted for patching multiple tasks. In the experiments, the authors show that the proposed method retains good performance on the task where the pre-trained model is already good at while improves the performance on the target tasks. In addition to classification tasks, the authors also show that their patching approach could improve CLIP model on 1) typographic attacks, 2) object counting, and 3) visual question answering.
strengths_and_weaknesses: Strengths
- Improving CLIP model is an active research problem. The authors propose a method to improve the model performance on wider range of tasks for open-vocabulary classification.
- The proposed approach is simple yet effective. To improve CLIP model on poorly performed task, the authors propose to freeze the classifier weights derived fro text encoder and fine-tune the model on given tasks. This preserve the ability of open-vocabulary classification while improve model performance on new tasks.
- The experiments show decent improvement (1% ~ 20%) over the tasks for which the model is fine-tuned while preserving the performance for the task that CLIP model is already good at.
- In addition to classification tasks, the authors show that their patching approach can improve object counting and VQA comparing to the pre-trained CLIP model


Weaknesses
- The writing could improve. The design of proposed method to retain open-vocabulary ability is based on freezing classification layer derived from text encoder. This only mentioned in one sentence in experimental setup. It was pretty confusing to me how the proposed method retain open vocabulary ability until I locate that single statement in experimental setup. I would recommend the authors to make it more clear through out the paper. 
- The proposed method is simple. However, in practice, it requires non-trivial effort. To patch the model, the proposed method requires supervised annotations and fine-tuning model with hyper-parameter search and validating the mixing coefficients. The procedure may not be simpler than adding downstream data to the pre-training dataset and then re-train image-text model. 
- The authors only compare the performance against pre-trained CLIP model. I am curious how the proposed method compare with a simple baseline: adding downstream data to the CLIP training dataset and re-train image-text model.
questions: Would appreciate if authors can address the questions mentioned in the weaknesses.
limitations: The authors discussed the limitations in the submission.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a simple yet effective way to do model patching, by interpolating the weights before and after fine-tuning. Experiments are performed with the image encoder from a recent dual-stream vision-language model CLIP. Empirical results show that the proposed method improves performance on other tasks while preserving the performance on ImageNet. Moreover, the proposed method can make CLIP more robust and more powerful for tasks like VQA.
strengths_and_weaknesses: The paper is overall well structured and easy to follow. The method is simple and effective.

One of my main concerns is about the motivation of "patching models on a single new task". More discussions on the real applications of this scenario would be necessary to better show the motivation of this research. Yet another concern is that this work keeps saying the "open-vocabulary" model. However, it is not clear why the proposed method only works for open-vocabulary models, or whether can be extended to other open-vocabulary models beyond CLIP. 

The experiments also need some more clarifications. Though the authors describe some connections and differences with some continual learning methods in the related work section, it would better show the efficacy of the proposed method by empirically comparing with them. In particular, in lines 264-266, the authors said that "in contrast to regularization or replay-based methods, patching requires no extra computational cost during training". Since the proposed method also requires training (i.e., fine-tuning on the data 
of the target task), it is not clear to me what the "training" means here. Can the authors provide an exact comparison of the performance VS training time/data between the proposed method and regularization methods like EWC and EWC++?
questions: Please address the questions above.
limitations: Yes, there is slight discussion about the limitation at the end of the paper.
review_title: null

Meta-review: metareview: The reviewers had some concerns about clarity of motivation and baselines.  My own opinion is that this work is valuable for the community because of the simplicity of the method and depth of experiments.
confidence: Less certain
Decision: Accept