Patching open-vocabulary models by interpolating weights
Authors: Gabriel Ilharco
Mitchell Wortsman
Samir Yitzhak Gadre
Shuran Song
Hannaneh Hajishirzi
Simon Kornblith
Ali Farhadi
Ludwig Schmidt
Venue: NeurIPS 2022 Accept (2022)
Keywords: zero-shot models
open-vocabulary models
model editing
model patching
vision-and-language
image-text models
CLIP

Abstract: Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot performance is far from optimal. We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate. Towards this goal, we introduce PAINT, a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched. On nine tasks where zero-shot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model. PAINT also allows a single model to be patched on multiple tasks and improves with model scale. Furthermore, we identify cases of broad transfer, where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes. Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on CLIP. Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.

Patching open-vocabulary models by interpolating weights
Gabriel Ilharco"!
Mitchell Wortsman"
Samir Yitzhak Gadre* Shuran Song? Ali Farhadi
Hannanch Hajishirzi Simon Kornblith
Ludwig Schmidt 1.3
University of Washington
"Columbia University
AA12 'Google Research, Brain Team
Abstract
Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks. However, there are still settings where their zero-shot perfor- mance is far from optimal. We study model patching, where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate. Towards this goal, introduce PAINT, patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning shot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy ImageNet within one percentage point of the zero-shot model. PAINT also allows a single model to be patched on multiple tasks and improves with model scale. Furthermore, transfer, where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes. Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on CLIP. Our findings demonstrate that it is possible to expand the of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.
task be patched. On nine tasks where zero-
identify cases of broad
1 Introduction
Open-vocabulary models are characterized by their ability to perform any image classification task based on text descriptions of the classes [56]. Thanks advances in large-scale pre- pre-training, recent examples of open-voccbylary models such as CLIP and BASIC have reached parity with or surpassed important task-specific baselines, even when the open-vocabulan
models are not fine-tuned on task-specific data (i.e., in zero-shot setting) [57, 31, 56, 88, 1, 86). For instance, the largest CLIP model from Radford et al. [57] used zero-shot setting matches the ImageNet accuracy of ResNet-50 trained on 1.2 million ImageNet images [14, 24].
Nevertheless, current open-vocabulary models still face challenges. The same CLIP model that matches a ResNet-50 on ImageNet has lower MNIST accuracy than simple logistic regression in pixel space [57]. Moreover, even when zero-shot models achieve good performance, they are usually still worse than models trained or fine-tuned on specific downstream tasks.
To address these issues, several authors have proposed methods for adapting zero-shot models task of interest using labeled data [82, 91, 21, 89, 37, 73]. common practice is to mun- the zero-shot model the task of interest [82, 56]. However, fine-tuned models can suffer from catastrophic forgetting [48, 76, 20, 33), performing poorly on tasks where the zero-shot model initially performed well [2, 82, 56]. Additionally, fine-tuning typically produces task-specific classification head, sacrificing the flexible text-based API that makes open-vocabulary models so appealing. Whereas an open- model can perform any classification task in a zero-shot fashion, fine-tuned
Equal contribution. Code available at https://githhe cwm/ml htt///ffounmddd www.ooontnnnnntttaaoooeeos Correspon- dance (gamaga, mitchnw, schmidt]@ca Ota
http://c...aaaattaaooooeoa waahington. edu, syOca. columbia. edu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1: Patching open-vocabulary models by lin- early interpolating weights. We wish improve ac- curacy on tasks where model performs poorly (patching tasks), without degrading performance on tasks where accuracy is already adequate (supported tasks). When interpolating weights of fine-tuned models and zero- shot (unpatched) modcls, there are intermediate solu- tions where accuracy improves on the patching task without reducing accuracy on supported tasks. Results are shown for CLIP models [57], averaged over nine patching tasks (Stanford Cars, DTD, EuroSAT, GTSRB, KITTI distance, MNIST, RESISC45, SUN397 and SVHN [35, 11, 25, 71, 22, 39, 7, 12, 84, 53)) and five supported tasks (ImageNet, CIFAR-10, CIFAR- 100, STL- and Food101 [14, 36, 12, 5)). We apply PAINT separately each patching task and average results across experi- ments. The dashed lines illustrate vertical movement from the unpatched models and horizontal movement from the fine-tuned models.
fine-tuned models
Average accuracy on patching tasks
0.9
weight interpolations
0.7
0.6
VIT-B/32
0.5
VIT-B/16
VIT-L/14
zero shat models
0.65
0.70
Average accuracy on supported tasks
model with a task-specific head can only process the specific task that it was fine-tuned on. This specialization can prevent knowledge obtained by fine-tuning on one task from transferring to other related tasks with different classes.
Another approach to adapting zero-shot models would be to add data from the downstream task to the pre-training dataset and train new open-vocabulary model from scratch. The resulting model could still perform any classification task, and zero-shot performance may improve on related tasks. However, training large image-text models from scratch can require hundreds of thousands of GPU hours [57, 56, 86], which makes this approach practically infeasible in most settings.
In this paper, we study patching open-vocabulary models, where the goal is to increase accuracy on new target tasks while maintaining the fiexibility of the model and its accuracy on other tasks. Patching aims to combine the benefits of fine-tuning and re-training from scratch: improved perfor- mance on the task of interest, maintaining the flexibility of an open vocabulary, transfer between tasks, and fast adaptation time. Motivated by these goals, we extend existing fine-tuning techniques [82] to open-vocabulary settings, where the class space not fixed. We introduce Patching with Interpolation (PAINT), a simple, two-step procedure for patching models: first, fine-tune the model on the patching task without introducing any sask-specific parameters; then, linearly interpolate between the weights of the model before and after fine-tuning. Linearly interpolating neural network weights
[52, 19, 54] has been previously used improve accuracy on a single task [28, 81] or robustness to distribution shift [82]. Indeed, averaging network weights has been explored in continual learning contexts, although for closed-vocabulayy models [40].
With PAINT, accuracy can improve on new tasks without degrading accuracy on unrelated tasks, as illustrated in Figure 1. For instance, applying PAINT to CLIP ViT-L/14 [57] independently on nine image classification tasks [35, 11, 25, 71, 22, 39, 7, 84, 53] improves accuracy by 15 to 60 percentage points compared to the unpatched model, while accuracy on ImageNet [14] decreases by less than one percentage point. We also observe promising trend: patching becomes more effective with
model scale (Section 4.1).
Beyond single tasks, we show that models can be patched on multiple tasks (Section 5). When patch- ing on nine image classification tasks simultaneously, a single CLIP ViT-L/14 model is competitive with using one specialized model for each task-the average accuracy difference is less than 0.5 percentage points.
Moreover, PAINT enables broad transfer (Section 6): accuracy on related tasks can increase, even when the class space changes. For instance, we partition EuroSAT [25]. a satellite image dataset, into two halves with disjoint labels. Patching VIT-L/14 model the first half improves accuracy on the second half by 7.3 percentage points, even though the classes are unseen during patching.
'The term patching is borrowed from software development terminology, drawing inspiration from recent work which conceptualizes developing machine learning models like open-source software [58, 62, 47, 72].

---

Finally, we investigate PAINT on case studies including typographic attacks [23], counting [32], and visual question answering [4] (Section 7). For instance, applying PAINT using synthetic typographic attacks leads to model that is less susceptible to typographic attacks in the real world, improving its accuracy by 41 percentage points.
In summary:
Even the best pre-trained models are not perfect. We introduce PAINT, method designed to improve accuracy on new tasks without harming accuracy elsewhere.
PAINT incurs no extra computational cost compared to standard fine-tuning, neither during fine-tuning itself nor at inference time.
PAINT can also be applied with multiple tasks, providing single model that is competitive with many specialized models.
Applying PAINT with one task can improve accuracy on related task, even when they do not share the same classes.
PAINT improves with model scale, indicating promising trend for future models.
2 Patching with interpolation (PAINT)
This section details our method for patching models single and multiple tasks.
Patching on a single task. Given an open-vocabulary model with weights 0zs and patching task Paach- our goal is to produce new model paach which achieves high accuracy D patch without decreasing model performance on tasks where accuracy is already acceptable. We let Dsupp denote a representative supported task where model performance is adequate, and later show that the method stable under different choices of Duupp (Section 4.2). The two-step procedure we explore for producing Punch is given below.
Step Fine-tune Oz on training data from Pacch to produce a model with weights On- [0, 1]. linearly interpolate between 8z and On to produce Paacch = (1 - a). 8/s + a On. The mixing coefficient is determined via held-out validation sets for D supp and D puuch-- We refer to the resulting model Paach-
Step 2. For mixing coefficient
In our experiments, we do not introduce any additional task-specific parameters when fine-tuning, as discussed in Section 3 and Appendices B and C.
Patching on a multiple tasks. In practice, we often want to improve model accuracy on multiple
patching tasks ((2)
patch which be accomplished with straightforward modifications to the procedure above. We explore three alternatives and examine their relative trade-offs in Section 5:
patch ..., (D())
Joint patching, where we merge all the patching tasks D (()) patch into a single task Ppacch before
running the patching procedure;
Sequential patching, where we iteratively repeat the patching procedure above on each new task patch and let 0z + Opachh after each completed iteration;
((0)
Parallel patching, where we apply the first step on each task in parallel to produce fine-tuned
models with weights o(2) " Then, we search for mixing coefficients a; produce
0patch (1-E a.) Ozs + E a; 0(..)
For joint and parallel patching we assume access to held-out validation sets for all tasks, while in sequential patching we only assume access to held-out validation sets from the tasks seen so far. Unless mentioned otherwise, we pick the mixing coefficient a that optimizes average accuracy on held-out validation sets from the supported and patching tasks.
Experimental setup
Tasks. We consider a diverse set of image classification tasks from Radford et al. [57]. most experiments, we use ImageNet [14] representative supported task, although we explore other supported tasks Section 4.2. We categorize tasks into patching tasks or supported tasks based on the accuracy difference between the zero-shot model and a model specialized to the task. A large

---

accuracy difference indicates that the task is a relevant target for patching because the zero-shot model is still far from optimal. Specifically, we consider subset tasks from Radford al. [57], categorizing tasks where the linear probes outperform the zero-shot model by over 10 percentage points as patching tasks: Cars [35], DTD [11], EuroSAT [25], GTSRB [71]. KITTI [22], MNIST [39], RESISC45 [7], SUN397 [84], and SVHN [53]. We use the remaining tasks as supported tasks:
CIFARIO [36], CIFAR 100 [36], Food101 [5]. ImageNet [14], and STLI0 [12]. We investigate
additional patching tasks case studies in Section 7 and provide further details in Appendix A.
Models. We primarily use CLIP [57] pre-trained vision transformer (ViT) models [15]. Unless other- wise mentioned our experiments are with the ViT-L/14 model, while Section 4.2 studies ResNets [24].
Fine-tuning on patching tasks. Unless otherwise mentioned, we fine-tune with a batch size of 128 for 2000 iterations using learning rate le-5 with 200 warm-up steps with cosine annealing learning rate schedule and the Adam V optimizer [43, 55] (weight decay 0.1). When fine-tuning, we use the frozen final classification layer output by CLIP's text tower that we do not introduce additional learnable parameters. This design decision keeps the model -ecabuauyy and does not harm accuracy, discussed in in Appendices B and C.
Evaluation. We use accuracy as the evaluation metric unless otherwise stated. We refer to the average of the mean accuracy the patching tasks and the mean accuracy on the supported tasks as combined accuracy.
4 Patching models on a single new task
As shown in Figure 1, when patching model on single task, we interpolate the weights of the zero-shot and fine-tuned model, producing a model that achieves high accuracy on both the patching task and the supported task. On the nine tasks, PAINT improves the accuracy of ViT-L/14 by 15 to 60 percentage points, while accuracy on ImageNet decreases by less than one percentage point. PAINT also allows practitioners to control the accuracy trade-off on the patching and supported tasks without re-training new model, by varying the mixing coefficient a.
4.1
The effect of scale
We consistently observe that PAINT is more effective for larger models. Our findings are aligned with those of Ramasesh et al. [59], who observed that larger models are less susceptible catastrophic forgetting. This section formalizes and provides insights for these observations.
Measuring the effectiveness of patching. We measure the effectiveness of patching via the accuracy difference between the single patched model and two specialized models with the same architecture and initialization. For both the supported task and patching task, we take specialized models that maximize performance on the task, considering the set of all interpolations between the zero-shot and fine-tuned models. We refer to this measure as accuracy distance to optimal. Formally, accuracy distance to optimal is given by
max Acc(Oa, Psupp) + max Acc(Oa., Ppanc)) max (Acc(fa. Psupp) + Acc(Oa. Ppacch) (1)
where Acc(O, D) represents the accuracy of model 0 on task D. In Figure (left), we show that accuracy distance to optimal decreases with scale, indicating that patching becomes more effective for larger models.
Model similarity. Fine-tuning modifies overparameterized models less [9], which provides insights on why larger models are casier to patch: less movement required to fit new data. We demonstrate this by evaluating representational similarity using Centered Kernel Alignment (CKA) [34] (see Appendix for details). As shown in Figure (center), the representations of the unpatched and fine-tuned models become more similar as models grow larger, indicated by larger CKA values. Moreover, Figure 2 (right) shows that the cosine similarity between the weights of the unpatched and fine-tuned models, cos(Oz,, 0R) (Ozs, 0n) 20100=) [[0=]]), increases with scale.
'In other words, (L [Acc(O, Dimp)]] + L D AAcc(0, D)una)))) /2, where Acc(O, Deup)) and Acc(O, Donar) accuracies on supported tasks and patching tasks, respectively.

---

Accuracy distance to optimal
Accuracy distance optimal
Centered Karnel Alignment
Centered Kerne Alignment
Cosine similarity
0.03
0.9
Cosine arity
0. 0.99990
0.02
0.8
0.99985
0.01
0.99980
10
Model size (GMACS, log scale)
Model size (GMACs, log scale)
Model size (GMACs, log scale)
Cars
EuroSAT
KITTI
RESISC45 SUN397
SVHN
DTD
GTSRB
MNIST
Average across datasets
Figure 2: Larger models are easier to patch (left). For larger models, the unpatched and fine-tuned model are more similar with respect to their representations (center) and weights (right). Model scale measured in Giga Multiply-Accmmula operations (GMACs).
Patching SUN397
Patching EuroSAT
Patching MNIST
1.0
SUN397 Accuracy
0.80
EuroSAT Accuracy
0.9
MNIST Accuracy
0.75
0.8
0.7
0.65
0.7
0.4
0.6 Accurray Accuracy
ImageNet Accuracy
ImageNet Accuracy
Interpolation (VIT-8/32)
Interpolation (VIT-U/44
Various hyperparameters
Interpolation (VIT-8/16)
Early stopping
L2 regularization to initialization
Figure 3: The frontier of accuracy trade-offs can be recovered by linearly interpolating weights. Interpolating the unpatched and fine-tuned models recovers the accuracy trade-off of early stopping, regularization towards the initialization, and changes in hyperparameters. Additional details and comparisons can be found in Appendix E.
4.2 Baselines and ablations
Baselines. There are many alternatives which enable a trade-off between accuracy on the supported and patching tasks. These methods include early stopping during fine-tuning, applying regularization term which penalizes movement from initialization, or training with different hyperparameters including smaller learning rate. Unlike interpolation, these methods do not enable navigating the accuracy trade-off without fine-tuning the model again many times. Moreover, Figure 3 demonstrates that the accuracy trade-off frontier for early stopping, regularization, or varying hyperparameters can be recovered by interpolating weights with different mixing coefficients. Appendix E provides additional baselines and discussion, including EMA [74], EWC [33], LwF [41], J. re-training a model with data from the patching task, and mixing the pre-training and fine-tuning objectives.
Additional supported tasks. In Figure we use ImageNet representative supported task. This section demonstrates that PAINT is stable under different choices of the supported task. Instead of ImageNet, we use CIFARIO, CIFAR100 100, Food101 and STL10. Figure 4 displays representative results, where performance is averaged over the nine patching tasks (see Appendix F for additional results). We observe consistent results across supported tasks, and that the optimal mixing coefficients are stable across different choices of supported tasks (Figure 4, right).
Additional models. In addition to the CLIP ViTs used in the majority of our experiments, we study four ResNet models [24] from Radford et al. [57] in Appendix G. We find that patching is less effective for ResNets compared to ViT' of similar size, which corroborates the findings Ramasesh et al. [59] that ResNets are generally more susceptible to catastrophic forgetting. However, similarly ViT's, we still observe improvements with scale. Finally, we show that patching also effective for closed-vord vocabulary models in Appendix H.
5

---

Accuracy loss using mix. another supported dalaset.
CIFAR100
Food101
0.9
ImageNet-
0.000
0.001
0.002
0.000
0.002
Average accuracy
Lusks
CIFARLO
0.002
0.000
0.001
0.001
0.005
0.7
0.7
CIFARI0O
0.005
0.002
0.000
0.004
0.033
0.6
VIT
0.6
Food101
0.001
0.001
0.002
0.000
0.003
B/16
VIT-U14
STLAD
0.002
0.002
0.005
0.002
0,000
0.4
Imsgeehed
Footlo
57300
CIFARIOD accuracy
Food101 accuracy
Figure 4: Results are consistent across supported tasks. For multiple supported tasks, we observe similar accuracy improvements on patching tasks, without substantially decreasing supported task accuracy. Additional results for the supported tasks F0od101, STL10 and ImageNet are in Appendix F. Moreover, choosing the mixing coefficients using different supported task does not substantially decrease combined accuracy on patching and supported tasks (right).
5 Patching models on multiple tasks
This section details experimenta results for patching on multiple datasets. Recall from Section that there various strategies for extending PAINT multiple datasets, which we briefly revisit. For joint patching we merge all the datasets into a single fine-tuning task and apply our patching procedure as before. For sequential patching we iteratively perform our procedure once per task, using the patched model at each step as the initialization for the next step. We also explore parallel patching, for which we have an unpatched model Oz and independently fine-tune on each of the tasks in parallel. We then search for mixing coefficients to combine the resulting models. For tasks 1..., k, let " search over each ag we instead search over a one-dimensional scalar a E [0, 1], which interpolates
((*))
denote the fine-tuned models for each task. Since it is impractical to exhaustively
between Oz and the average of all fine-tuned solutions k of Appendix J provides further experimental details.
These methods have various trade-offss and may be applicable for different scenarios. Joint patching is only possible when data from all tasks you wish to patch is available. On the other hand, sequential patching appropriate when the tasks are observed one after another. Finally, parallel patching can leverage distributed hardware.
Figure displays experimental results when patching on all nine tasks from Section 4. We observe that joint patching is the best-performing method on average. This is perhaps unsurprising since joint patching has simultancous access all patching datasets, unlike other patching strategies. Nevertheless, it is still interesting that for ViT-L/14, joint patching yields single model with only 0.5 percentage points worse combined accuracy than using multiple specialized models.? Joint patching also achieves 15.8 percentage points improvement over the unpatched model. Moreover, patching a ViT-B/32 model with the joint strategy achieves a combined accuracy 6.1 percentage points higher than VIT-L/14 unpatched model, which requires 12x more GMACs.
The accuracy of sequential patching approaches that of joint patching, especially for larger models. Note that, unlike in joint patching, forgetting can compound since the patching procedure is applied multiple times in sequence. In sequential patching, weight interpolations do not completely cradicate forgetting, but greatly mitigate it. This is most noticeable for smaller models: sequentially fine-tuning a ViT-B/32 without interpolation reduces the combined accuracy by 4.6 percentage points compared to the unpatched model, as shown in Appendix J. This is compared to combined accuracy increase of 11 percentage points when using sequential patching. Additional results, including experiments on SplitCIFAR [61], can be found in Appendix J.
Finally, parallel patching underperforms other patching strategies. Like sequential patching, parallel patching is in the challenging setting where data from all patching tasks is not available simultaneously.
'The results are averaged over three random seeds that control the order in which tasks are seen. 'We also explored adaptive black- box optimization algorithms to choose the mixing coefficients ou [60]. but observed little improvement (0.3 to 0.4 percentage points on average).
Recall from Section 3 that combined accuracy weight patching and supported tasks equally.

---

Accuracies for VIT-B/32
Accuracies for ViT-L/14
ImageNet
ImageNet
Cars
SVHN
Cars
SVHN
0.8
0.8
0.6
0.4
SUN397
SUN397
EuroSAT
RESISC45
EuroSAT
RESISC45
GTSRB
MNIST
GTSRB
MNIST
KITTI
KITTI
Sequential patching
Parallel patching
Multiple independently fine-tuned models
Joint patching
Unpatched model
Figure 5: Contrasting various strategies for patching on multiple tasks. On all experiments, ImageNet is used the supported task while the other nine datasets are used for patching. When data from all patching tasks available, joint patching yields a single model that is competitive with using ten different specialized models. Weight interpolations greatly mitigate catastrophic forgetting on the sequential case, but do not completely eradicate it. Finally, parallel patching underperforms other patching strategies, but still provides improvements over the unpatched model.
Cars
DTD
EuroSAT
GTSRB
KITTI MNIST RESISC45
SUN397
SVHN
Unpatched accuracy
86.2
64.9
79.9
51.7
43.4
82.6
73.4
76.9
72.8
87.0
66.1
87.2
71.1
60.4
91.3
74.2
88.9
Patched accuracy
(40.8)
(+1.2)
(47.3)
(+19.4)
(+17.0)
(48.7)
(40.8)
(+2.4)
(+16.1)
Table 1: PAINT generalize to unseen classes. We randomly partition each dataset into tasks A and with disjoint class spaces of roughly equal size. This table reports how patching on task A affects accuracy task B for the ViT-L/14 model. all cases, accuracy on task B improves when patching on task even though the classes are unseen during patching.
Moreover, unlike in joint sequential patching, model is optimized on data from all patching tasks. Using black box optimization algorithm for finding the mixing coefficients did not yield large improvements over using the same mixing coefficient for all models. However, it is possible that more sophisticated search methods could yield better results. In Appendix we present additional experiments for subset of the tasks where exhaustively searching the space of mixing coefficients is tractable, finding headroom for improvement most cases.
6 Broad transfer
An alternative to our patching approach to introduce parameters which are specific to each new task. By contrast, PAINT always maintains single model. This section describes additional advantage the single model approach: patching the model on task A can improve accuracy on task B, even when task A and B do not share the same classes. We refer this phenomenon broad transfer. Note that we are able to study this phenomenon because the single patched model remains open-vocabulary throughout the patching procedure. This is key advantage of PAINT compared to maintaining a collection of task-specific models.
We now describe two experiments measure the effects task B when patching the model task A. First, we explore broad transfer by randomly partitioning datasets into disjoint sets with no class overlap. For dataset ' partition the class space y into disjoint sets of roughly equal size VA and Vg- We build task with the examples (x. y) E 'D where y belongs to A. and task B with examples (æ, where y belongs to VB. Table shows how patching model task A affects the accuracy on task for nine datasets D. The accuracy improvements on task B range from 0.8 to 19.4 percentage points, even though the classes from task B are not seen during patching.
To further understand transfer, we consider additional task pairs and B, which are now different datasets. While some pairs A. B share classes, there still instances of broad transfer. Concretely,

---

Task A
MNIST SVHN EuroSAT RESISC45
MNIST SVHN MNIST RESISC45 EuroSAT FashionMNIST
FashionMNIIT GTSRB MISD
Task F
MNIST
MTSD GTSRB
Unpatched accuracy
58.6
76.4
71.0
60.2
67.7
76.4
19.3
50.6
68.9
93.2
69.7
70.4
70.8
77.5
30.8
69.8
Patched accuracy
(+10.3)
(+16.8)
(-1.3)
(+10.2)
(+3.1)
(+1.1)
(+11.5)
(+19.2)
Table 2:
Patching on task A can improve accuracy on a related task B. For pair of tasks A and B, report accuracy of the ViT-L/14 on task B, after patching task A, finding improvements seven out of eight cases.
(a) Real-world typographic attack
(b) SUN397 synthetic typographic attack
(c) Acc. on real-world typographic attacks
(d) Acc. on SUN397 synthetic typographic attacks
Atlacked real-eaaaae real-world acc.
Attacked SUN397 acc.
0.8
0.8
0.6
sky
0.6-
0.4
DOG
0.5
0.2
0.75
ImageNet acc.
ImageNet acc.
Figure 6: Guarding against real-world typographic attacks by patching synthetic data. (a) A sample from our real-world typographic attacks test set. A CLIP ViT-L/14 is "tricked" into classifying this image dog instead of cat. (b) Sample of synthetic typographic attack data. (c) Performance on real-world data with unseen classes after patching on only synthetie typographic attacks (curves produced by interpolating between the unpatched and fine-tuned model). (d) Analogous curves for the test set of the synthetic data used for patching.
Table 2 examines i) MNIST and SVHN, two digit recognition tasks with shared classes; ii) EuroSAT and RESISC45, two satellite imagery recognition tasks where there are unshared classes but some overlap; iii) GTSRB and MTSD [17], two traffic sign recognition datasets where there are unshared classes but some overlap; and iv) MNIST and MNSST [83], which do not share any classes but appear visually similar. In seven out of eight experiments, patching on task A improves accuracy by 1.1 to 19.2 percentage points on task B. The exception is when EuroSAT and RESISC45, where accuracy decreases by 1.3 percentage points.
In all experiments, when patching on task A we choose the mixing coefficient by optimizing the held-out validation accuracy on task A and supported task (in this experiment we use ImageNet). While it is possible for a method that introduces new parameters for each task to exhibit broad transfer to new data, this also requires knowing which parameters apply for the new data. This is not necessary in the single model approach.
7
Case studies
We further examine the performance of PAINT in three additional settings, which highlight weak- nesses of the zero-shot CLIP model and showcase broad transfer (Section 6).
Typographic attacks. Goh et al. [23] find that CLIP models are susceptible to typographic attacks, where text superimposed text on the pink note saying "dog" leads a CLIP to misclassify the image of a cat as a dog. To fix this vulnerability, class names to SUN397 [84], as seen in Figure 6 (b). We then collect a test set of 110 real world images by placing notes objects and taking photos. After applying PAINT using the synthetic data, we evaluate on the real-world images (Figure 6 (c)) and synthetic test set (Figure 6 (d)). We observe that while larger models more susceptible to typographic attacks, they also more amenable to patching. Furthermore, we see an example of broad transfer between the synthetic and real-world data: when patching ViT-L/14 synthetic data, its accuracy on real-world typographic attacks improves 41 percentage points even though the real-world classes are unseen. The cost is
image leads to misclassification. For example, in Figure 6 (a), the
procedurally generate typographic attack data by adding text with incorrect
'Dta available at http://///gttaub..oo/////oooooe/

---

a reduction of less than 1 percentage point on ImageNet. We present details on the task and data collection in Appendix K.
Counting. Radford et al. [57] find that CLIP models struggle to count the number of objects in CLEVR [32]. Here, the task is to choose an integer between 3 and 10 for each image, corresponding to the number of visible objects. While straightforward way to patch such a task to fine-tune on directly, we investigate if applying PAINT using a subset of the classes allows the patched model to generalize to other numbers. Specifically, we patch on images with 4, 5, 6, 8, or 9 objects. To evaluate broad transfer, we test on images with 3, 7, and 10 objects (7 for understanding interpolation and 3 and 10 for extrapolation). We find that PAINT improves accuracy from 59% to over 99% on unseen classes with less than half a percentage point decrease in ImageNet accuracy. For more details see Appendix L.
Visual question answering. As shown by Shen et al. [68], zero-shot CLIP models perform poorly on visual question answering [4]. Using CLIP for VQA typically involves additional parameters-foo instance, Shen et al. [68] trains a transformer [77] on CLIP features. contrast, our procedure for patching CLIP on VQA does not introduce new parameters. Following Shen et al. [68], we contrast images with a series of text prompts, where each prompt corresponds to an option in multiple-choice VQA, formed by both the question and a candidate answer using the following template: "Question: [question text] Answer: [answer text]". We evaluate on multiple-choice VQA v1 [4], where each question is associated with 18 candidate answers. Our results, further detailed in Appendix M, show that patching is effective for visual question answering: PAINT improves the accuracy of a ViT-L/14 model by 18 percentage points, while accuracy drops by less than one percentage point on ImageNet.
Related work
Continual learning and catastrophic forgetting, Learning tasks sequentially remains a challenge for neural networks. When a neural network learns a new task, the accuracy on other tasks often decreases, a phenomenon known as catastrophic forgetring [48, 76, 20, 33]. While forgetting in neural networks may actually aid learning [90], researchers have proposed various approaches for approaches such as elastic weight consolidation (EWC) [33] and synaptic intelligence (SI) [87] which penalize the movement of parameters and are related to weight-interpolation by Lubana et al. [44]; ii) Replay methods [61, 69, 42, 6, 64, 50]. which incorporate data or gradient information from previous tasks when learning new task; and iii) Introducing sask-specific parameters [65, 85, 46, 8, 78, 80].
alleviating catastrophic forgetting, including: i) Regularization-basse
In contrast to these approaches, PAINT requires no modification to the standard fine-tuning process besides the later weight interpolation step. Moreover, unlike regularization or replay based methods, PAINT requires no extra computational cost during training. In contrast to methods with task specific parameters, we maintain a single model. Having a single model is beneficial when there is new data which is similar to one of the tasks which have already been patched. Even without explicitly knowing which task the new data is similar to, we can observe accuracy improvements (see Scction 6).
Similar to our work is that of Mirzadeh et al. [50], who observe high accuracy on task A on the linear path between model which achieves high accuracy on task and model which is fine-tuned jointly on task A and B. Moreover, they observe high accuracy on task B on the linear path between a model fine-tuned on task B, and the jointly fine-tuned model. Therefore, there exists a path between a model which achieves good performance on task A and model fine-tuned on task B along which accuracy high on both tasks. However, in Mirzadeh et al. [50] this combined path can be non-linear, leading them to propose regularization and replay based method. In our work, we find that examining models on a linear path between the unpatched model (which has high accuracy on task A) and the model fine-tuned on task B is often sufficient for obtaining a model which achieves high accuracy on both tasks (Figure 1). We speculate that this is due to scale and model architecture: in contrast to Mirzadeh al. [50], initialize with a model pre-trained
large dataset consisting of 400 million images [57], and primarily use vision transformers [15]. As shown in Section 4.2, our method performs substantially worse with ResNets [24], which are used by Mirzadeh et al. [50].
Finally, Ramasesh et al. [59] and Mehta et al. [49] also observed that catastrophic forgetting less problematic for large and pre-trained models. In addition, Ramasesh et al. [59] found-similar to results-the vision transformers are less susceptible to forgetting than ResNets of the same size.
Linear mode connectivity and robust fine-tuning. Linearly interpolating neural network weights is a key step in PAINT. Because of the many nonlinear activations in neural network, it is not clear

---

a priori that linearly interpolating between two sets of weights can result in a high accuracy solution. However, researchers have observed that interpolating neural network weights can achieve high accuracy when training on MNIST from common initialization [52] or when part of the optimization trajectory shared [19, 28, 54, 18, 82, 47, 16, 81, 10]. The term linear mode connectivity was coined by Frankle et al. [19]: two networks exhibit linearly mode connectivity if the accuracy does not decrease when using weights on the linear path between them [52, 19J. Weight averaging for continual learning has also been studied by Lee et al. [40] for closed-vocabulary models.
While Nagarajan and Kolter [52] and Frankle et al. [19] focused on accuracy on a single task, Worts- man et al. [82] use linear mode connectivity to fine-tune models while preserving their robustness to natural distribution shifts. By interpolating the weights of a zero-shot and fine-tuned model, they find a solution which performs well both on the fine-tuning task and under distribution shift. In contrast to Wortsman et al. [82], we do not modify any task-specific parameters when fine- tinetunnng preserving the open-vocabulary nature of the models we patch. Unlike Wortsman et al. [82], we examine accuracy trade-offs across different tasks with little or no class overlap and adapt a model to multiple tasks.
In addition, closely related to our work is that of Matena and Raffel [47], who use Fisher-weightn averaging of language models before and after fine-tumi fine-tuning on downstream tasks. Unlike Fisher- weighted averaging of Matena and Raffel [47], we do not use different mixing coefficients for each parameter, and thus require no extra compute when patching. Moreover, we explore new strategies for patching multiple tasks (see Section 5), and focus on open-vocabulary image classifiers.
Interventions to change the behavior of a trained model. Several authors have studied the problem of updating a model to locally alter its behavior on certain inputs without external disruptions on other inputs [70, 13, 51, 66, 63, 62]. Previous literature uses various terms to refer to this process, including model editing, patching or debugging. A popular use case is to update trained language models to reflect changes in the world (for instance, facts like who is the current president of Brazil) [29, 45, 38, 30J. Moreover, inspired by software engineering practice, previous work explored "debugging" language models through user interaction [63, 62]. including providing corrective feedback to the models via natural language [3]. Mitchell et al. [51], De Cao et al. [13] propose training auxiliary networks to perform local edits on pre-trained models. Santurkar et al. [66] introduce a method for rewriting the prediction rules of a classifier, focusing on specific failure modes such as reliance on spurious correlations. In contrast with previous literature, our work explores patching models at the task level, aiming to systemically improve accuracy on a dataset-for instance, enabling a model to recognize dozens of satellite imagery classes with single patch.
9 Limitations and conclusion
I Limitations. When applying PAINT, accuracy on supported tasks can still decrease, especially for smaller models. This limitation is perhaps best reflected in the case of sequential patching: patched models underperform using multiple specialized models when many tasks are added sequentially. Using larger models and weight interpolations can alleviate this issue, but do not completely resolve it. Finally, better understanding on which datasets patching is more effective is an exciting direction for future research.
Conclusion. In this work, we explore several techniques for patching open-vocabulary models with the goal of improving accuracy on new tasks without decreasing accuracy elsewhere. PAINT is effective in several scenarios, ranging from classifying digits to defending against typographic attacks. PAINT becomes more effective with scale, and can be applied multiple tasks sequentially or simultancously. Our findings demonstrate that in many cireumstances it is possible to expand the set of tasks which models achieve high accuracy, without introducing new parameters, without re-training them from scratch, and without catastrophic forgetting.
Acknowledgments
We thank Akari Asai, Alex Fang. David Fleet, Huy Ha, Ari Holtzman, Pieter-Jan Kindermans, Marco Tulio Ribeiro, Ofir Press, Sarah Pratt, Sewon Min, Thao Nguyen and Tim Dettmers for helpful discussions and feedback, and Hyak at UW for computing support. This work is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML), Open Philanthropy, NSF IIS 1652052, NSF IIS 17303166, NSF IIS 2044660, NSF IIS 2132519, ONR N00014-188 1--1---8-- DARPA N66001-19-2-403.. DARPA 191 the Sloan Fellowship and gifts from Allen Institute for AI.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, 2022. https://arxiv wwrg/abs//20000 14198.
[2] Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs, The evolution of out-of-distributioo robustness throughout fine-tuning, 2021. botps://arxion hawpeoorg//gba. /bs/ 2106.15831
[3] Anonymous. Fixing model bugs with natural language patches, 2022. https://openrevviev.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitiick, and Devi Parikh. VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV), 2015. https: /ttp//aaxvvvv//a///a//////aaaava org/ abs 000//1155500.0...
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- mood-101-mining discriminative components with random forests. In European Conference on Computer Vision (ECCV), 2014. https://link: springer www.coocc/aaoooe cttpter/10.1000/ 3111:/9997--33----9 11t.:/100000//7---111-11113--330 --1---999-----9999-999999
[6] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. In International Conference on Learning Representations (ICLR), 2019. attps://arxiv org/ h/tg/////////e 111//1112.21 00420.
[7] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Institute of Electrical and Electronics Engi- neers (IEEE), 2017. https://ieeexploree ieee. .eee.org//eeeeeeeeeeee.ee....ooe abbstract//ocuueett//aeee
[8] Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In Advances in Neural Information Process-
ing Systems (NeurIPS), 2019. https: //tp:////p/cceingss.. ppg.oeeeiings.nnuripp....s neurips...... cetrriss.ccpaaape wttpscccpppprreee hctpppper//099/fiae/ (tt/a//6663220002200999902222200
Paper. pdf.
[9] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in dif-
ferentiable programming. Advances in Neural Information Processing Systems (NeurIPS), 2019.
https: sttps://proceeding httpp/ooeeinnnn.iiiir.o.oo nwuripe.cc./oooooooooeeeeoo poaripps.co/ppprr/11////11 Paper. .df.
[10] Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining, 2022. https://arxte: org.lv...rrraaaeeee htsp:/bbs2222// 03044.
[11] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR), 2014, SetcriibingTeettueeee ii___innnn11 2014_00222200_0 2111__11P___app______ pap.p..tel.....................m html.
thecvf. com/ content__vvr__000 2ai_22014/tttt wwtpm///iiiiiiit /Cimpoi_
[12] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsu- pervised feature learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
http://prrceeen...eeee////o mlr. pww.srressseee crres/vv5// vv5/ coates 11a a. html.
[13] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP). 2021. https: //arxiv. wwwp/a///////eo.ooo...oe.eeeeee/ 2104. 08164.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. https: /eieexpeeeeeeeeeee ieee, org/ abstract/////////// document/5206e0
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL. betps://openrevee wttp://ww..eeoommm/ddooooooooooa

---

[16] Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations (ICLR), 2022. https://arrai http://aaxivv.oraabaa////sas
.02966.
[17] Christian Ertler, Jerneja Mislej, Tobias Ollmann, Lorenzo Porzi, Gerhard Neuhold, and Yubin Kuang. The mapillary traffic sign dataset for detection and classification on a global scale. In European Conference Computer Vision (ECCV), 2020. https://arxis /arpii///xxivv///a org/ /t///////// abs/1999. 04422.
[18] Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kemel. In Advances in Neural Information Processing Systems (NeurIPS), 2020. https://arxiii org/ abs 2010 15110.
[19] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning (ICML), 2020. https: httpa/////ccceennegee mlr.
www...tw...ee...e.. prwss/viiii htt11199frfa1e00000m html..
[20] Robert M French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 1999. https://www.scienceeee
$136466139901294..
[21] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters, 2021. petts://arrae org/ abt//1bb/111... 04544.
[22] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. attps://ieeexplr
ieee.
[23] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021.
neurons.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. https:
/ //arxiv. www./ogg///////ooooo/...oe.ee/// aa/aaba///51 abbs/112..e. 03385.
[25] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
org/abs/ 11tp.///....00 00029.
[26] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. https: //axxiv////////a//a org/ hbtp//b///te ab50//0/. 02531.
[27] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. https://gitae
ccin_clip.
[28) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better gencralization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018. https: //atpi://rxiv..../ org/ aaaaa/////110 1803. 05407.
[29] Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang. Joongbo Shin, Janghoo Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models, 2022. attps://anes org/ abs//244..2222/////////// 14211.
[30] Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Conference on Learning Representations (ICLR), 2022. https: //arxiv. owgg/bs////// abs//110. 03215.
12

---

[31] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le. Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and lasnon-anggaage representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021. https://arxiv org/ abs/2102. 219.10918..
[32] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
attps://arxiv. org/ aba/aasss/..///// 1612.06890......
[33] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Traabka-Barwinnske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017. https: //tpxi////xxvv..///////a/ hts//bbs/11122 00796.
[34] Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Machine Learning (ICML), 2019. /tttps://arxiv.orr abs/ 1905. 00414.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In International Conference on Computer Vision Workshops (ICML), 2013. https: /tttps:/www..co cv- founatioon wwtp:pppaaccsss/////ooooopscssap conteettccnotttteeeeeeeeeeeeeeee _ecvttt_________eee wecv_wwosshops..... 2013/W19hhtml/Kranee _a_paauuuuse____eeeeeeeeee_ee_e BD__DD__eeeeeeee acttett_oo.eeeeeeeecceeee htw..DDDiiect..appeeataatttaoaaa 2013 wttp:ooossoooo1_____e_eee_ee___ _cperp html.
[36] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images,
2009. https://wwes cs. toronto..... wwrrontt.eduukrrii//// edw/"krrz/eearninnn features- 2009-0000922-- TR. pd.
[37] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Confer- ence on Learning Representations (ICLR), 2022. https://openrevieet nwt. wwtp:/wumteeot/meeeee.ee.. form??d= UYneF2XSJWh.
[38] Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez. Cyprien de Masson d' Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems (NeurIPS), 2021. https:
[39] Yann LeCun. The mnist database of handwritten digits, 1998. http:://yani http:://yyannneee....mm com/ exdb/mnistt
[40] Sang-Woo Lee, Jin- Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in Neural Information Processing Systems (NeurIPS), 2017. attps://arrie wttpaawwwwwaa.aaaaa//eeeaerleeee /as///ba///a 1701108444111
Saa
[41] Zhizhong Li and Derek Hoiem. Learning without forgetting. forgenn.. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
09282.
[42] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017. https://arxs org/ org/abs/ ///abb///1/ taabs/106....... 08840.
[43] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https: http://pppnrrriiee..teffo/oooom id-Bkg6RicqY7.
[44] Ekdeep Singh Lubana, Puja Trivedi, Danai Koutra, and Robert Diek. How do quadratic regularizers prevent catastrophic forgetting: The role of interpolation, 2021. https://arxix: org/abs/2102. 02805.
[45] Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. Time waits for no one! analysis and challenges temporal misalignment, 2021. https:
// //arxiv. wwg/abs//////eeooooot..otceaeere htsp//b..///////////////
13

---

[46] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network multiple tasks by learning to mask weights. European Conference on Computer Vision (ECCV), 2018. https://arxie orgo /bs///ba///a 1801.06519
[47] Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging, 2021.
[48] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Morivation. Elsevier, 1989, https: / /www:/wwww../ sciencedirect. wwt.cooo/ccccc. www/sccince////oooccccccccoccccc httpe//atticleaeeaeaeeee prrrcc////aa/pp///09077711066660
[49] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning, 2021. https://arxiv, www.aaiww..iiaaaaaae..ooo oog.ooggg/a... rrg//ba///
2112.09153.
[50] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity multitask and continual learning. In Interna- tional Conference on Learning Representations (ICLR), 2021. https: org abs/ 2010. 2410.04495.
[51] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D) Manning. Fast model editing at scale. In International Conference on Learning Representations (ICLR), 2021.
https:// //axivv.//xi... wwtp/iiv.or///ooooooo.o aa//ab///21111 11309.
[52] Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In NeurlPS, 2019. https: http:e///procceeings www.sseeeripsccmm cc/ paper/2019/file//00 (155667722225666c111100666666666 Pa.....d..........p............p
[53] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural In- formation Processing Systems (NeurIPS) Workshops, 2011. attps://storase /toss/trae..oollaaaapgoeooooooee googleapis. tolpppu--oolssuubbb wwwpblicppubicationdaaaaaaeee /ttp///ff333//8///// /37648 pdf.
[54] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems (NeurIPS), 2020. https: //arxiv. org/
oogg/ba////e abs/abas///// 11687.
[55] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style,
deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 2019. https: .rgg abs/ /9tp://1..... 01703.
[56] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Luong, Mingxing Tan, and Quoe V Le. Combined scaling for zero-shot transfer learning, 2021.
org/
10050.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Inter- national Conference on Machine Learning (ICML), 2021. attps::/arre www.iiiw.w/.vbbb..aooa org abg/003/2103 00020.
[58] Colin Raffel. source software. 2021. a- to- sooo-buildmmdeess-ee models-like weikke-eebbildd-ee soapeen-suuuee.om source- software html.
A call to build models
like we
build open-
https://colinraffell cttpbllccc/////////////////
[59] Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations (ICLR), 2021.
https:t/ttppptee http::////p/.eeeeeeeeeeeee wwt.s/eeeeeettoomiddttataaa. _Peea..
[60] Rapin and O. Teytaud. Nevergrad A gradient optimization platform. https://GitHu: com com//Bccloolileeerrc///vvrrgggge 2018.
14

---

[61] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Conference on Computer Vision and Pultern Recognition (CVPR), 2017. https: http::ss////xvv/ov//o//o/o
1611.0777
[62] Marco Tulio Ribeiro and Scott Lundberg, Adaptive testing and debugging nlp models. In Asso- ciation for Computational Linguistics (AC (ACL), 2022. https://www www.:softtoooooooo com/ enn /enuus/n
[63] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Association for Computational Linguisties (ACL), 2020. https: http:///acaaathhhho
org/ /2200.0002.2 acl 1am-mmain... 442.
[64] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Expe- rience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. htapps///arxiv..o/ www.o/rg///bb//eeeeeeeeee 1811. 11682.
[65] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks, 2016.
https://arxia: org/ aaaaaaaa1111 /106/ 04671.
[66] Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing a classifier by rewriting its prediction rules. In Advances in Neural Information Processing Systems (NeurIPS), 2021. http::://axxi..og/abbb....
[67] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs, 2021. https://arxiv: http//aaiiv..rgga////aaa 02114.
[68] Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Chang, Zhewei Yao, and Kurt Keutzer. How much clip benefit vision-and-language tasks? In International Conference on Learning Representations (ICLR), 2022. https:/ //arxi//aaii////aa org/ abs/2107 /es//2077.6683.
[69] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems (NeurIPS), 2017. https://arxiv.orr org/ abs/aaa// 08690..
[70] Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations (ICLR), 2020. https://arxiv wwwp:/www.wrraa/////e///// 20000///000000455.
[71] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In International Joint Con- ference on Neural Networks (IJCNN), 2011. https://ieeexplooo ieee org/ document/ 6033395.
[72] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems (NeurIPS), 2021. https: //arxiv org/ abs/2111..990
[73] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision-and lan--uugge tasks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https://arxiv org/ abs/ 06825.
(74) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Conference on Computer Vision and Pattern Recognition, 2016. https: http///axvvv..v// org/ wogg/ba//// abs//122. 00567v3.
[75] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 100m The new data in multimedia research. Communications of the ACM, 2016. https:/ sttps://arxiv. org/abs/ 100/.//53...0 01817.
15

---

[76] Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, 1998. https://link. springer. , www//hapter///.////// 1007/978 1-4615-6529-2.25
[77]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017. https://axxiv. /tapaxxxvvvorr...//a abs/ 1706.03762.
[78] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations' (ICLR), 2020. http://ppaarrreeeeeeeeferorrrrar
[79] Ross Wightman. Pytorch image models. 2019.
[80] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Raste- gari, Jason Yosinski, and Ali Farhadi. Supermasks superposition. In Advances in Neural Information Processing Systems (NeurIPS), 2020. https://arxiv org/ httg/bb//2000 14769.
[81] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Komblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML), 2022.
https://arra org/ abs/2203 05482.
[82] Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https:/ //apxiv.
trrgabbs/2100 01903.
[83] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017. https://arxiv wwwpeiiiivorroeoooooeoeee hts///bss////// /t////778000777 07747.
[84] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), 2016. https: htaps://1inn springer. wwt.coo/aaaaaao aataa/rticcee/ 1007 / 111123-011--777 014-078--y
[85] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations (ICLR), 2018
org/ abs/ 1708. 01547.
[86] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022. https://aaat org/abs/22001 01917.
[87] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), 2017. https:// Texiv.org/ www.///////////////...o/eeeeeee/ absaaa////// 172.00200..00.....
[88] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https://arxiv. htg/abs/2211..0 07991.
[89] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free frre clip-adapter for better language modeling, 2021. https: : //tp//aavvvv//////////ee///aavv/ org/
argg/bss/2e aa/aabb///11 111111100011 03930.
[90] Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. International Conference on Learning Representations (ICLR), 2022.
https://arxiv. org/ 2202/ 00155.
[91] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https: // /arxivvv org/ absaabs///22 2203. 05557.
16


Reviews:
[{"review_id": "iyoUBwlOKns", "reviewer": "Reviewer_i385", "summary": "This paper proposes a simple yet effective way to do model patching, by interpolating the weights before and after fine-tuning. Experiments are performed with the image encoder from a recent dual-stream vision-language model CLIP. Empirical results show that the proposed method improves performance on other tasks while preserving the performance on ImageNet. Moreover, the proposed method can make CLIP more robust and more powerful for tasks like VQA.", "questions": "Please address the questions above.", "limitations": "Yes, there is slight discussion about the limitation at the end of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is overall well structured and easy to follow. The method is simple and effective.\n\nOne of my main concerns is about the motivation of \"patching models on a single new task\". More discussions on the real applications of this scenario would be necessary to better show the motivation of this research. Yet another concern is that this work keeps saying the \"open-vocabulary\" model. However, it is not clear why the proposed method only works for open-vocabulary models, or whether can be extended to other open-vocabulary models beyond CLIP. \n\nThe experiments also need some more clarifications. Though the authors describe some connections and differences with some continual learning methods in the related work section, it would better show the efficacy of the proposed method by empirically comparing with them. In particular, in lines 264-266, the authors said that \"in contrast to regularization or replay-based methods, patching requires no extra computational cost during training\". Since the proposed method also requires training (i.e., fine-tuning on the data \nof the target task), it is not clear to me what the \"training\" means here. Can the authors provide an exact comparison of the performance VS training time/data between the proposed method and regularization methods like EWC and EWC++?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "hyfOaL7ado0", "reviewer": "Reviewer_G7Ms", "summary": "The paper performs an exhaustive empirical study to propose model patching where the goal is to improve accuracy for open-vocabulary models on specific tasks (i.e., patching tasks) without degrading accuracy on  tasks where performance is already adequate. Model patching refers to interpolation of model weights between a finetuned model and the original model. Among other experiments, the paper shows results of model patching on nine tasks where zero-shot CLIP performs poorly, and obtains improvement of over 15-60 percentage points while not losing performance on ImageNet itself. The paper also talks about broad transfer across multiple tasks, and notes that the proposed approach becomes more effective with increasing scale of datasets.", "questions": "* What is the summary of takeaways from the paper? The conclusion states “...it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy”. But a succinct summary of *how* this can be achieved is missing.\n\n* One could argue that interpolating between model weights and finetuned weights is like parameter regularization techniques in continual learning, which impose a certain constraint on how far the weights can be from the original weights. This comparison is glaringly lacking in the main paper. While the Appendix presents a result with Elastic Weight Consolidation, it mentions that one needs access to the pre-training data for a complete experiment. However, there are many continual learning methods that do not need replay buffers (e.g. Gradient Projection Memory for Continual Learning, ICLR 2021 – this is only a representational example), and it may have been useful to see how such a method compares against them.\n\n* Considering the recent focus on OOD generalization performance and robustness to corruption, how would this method work on OOD corruption benchmarks? It may be interesting to see the performance.\n\nI will be happy to revisit the rating after the discussion with/response from authors.", "limitations": "Please see the weaknesses listed above.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n—-----------\n+ The methodology tested and proposed is very simple, and easy-to-implement.\n+ The results are interesting, and of broad relevance to the community, especially to those in large-scale ML practice.\n+ The results are comprehensive, and cover experiments across various settings, including ones such as typographic attacks, counting and visual question-answering.\n+ The Appendix is loaded with even more results, which make this an elaborate empirical effort.\n\nWeaknesses:\n—---------------\n- The primary weakness of the paper is the limitation mentioned in the paper itself (L315-316): “....our method provides no guarantees on which data the model performance might change…”. This limits the robustness of the takeaways from the paper, and where one may be able to use them – especially considering the contributions are largely empirical.\n\n- Considering \\alpha is the key hyperparameter for the interpolation, it would have been nice to see how the performance changes with different \\alphas. This, to me, is another weakness of the paper - amid the large number of results, it lacks a clearer perspective on how a reader can take away lessons that can be used in practice (especially considering practice is the focus of this work).\n\n- In continuation to the above comment, there is no evident trend in the results (or a summary of it in the paper) to see when this method works best. \n\n- Considering the large number of results presented in the work (including the Appendix), the paper definitely needs a summary or discussion summarizing the lessons and takeaways for the work to be useful to the readers. It becomes the reader’s burden otherwise to sift and find the takeaways.\n\n- The paper doesn’t compare with other papers that linearly interpolate neural network weights as mentioned in the baselines section. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ds-6JUcAax", "reviewer": "Reviewer_xjCv", "summary": "In this submission, the authors attempt to improve the CLIP model on the tasks that it performs poorly. They propose to fine-tune pre-trained CLIP model on the target task with frozen classification layer and derive the final model with a linear interpolation between pre-trained and fine-tuned model. The mixing coefficient is decided by validation. This task-patching procedure could be adopted for patching multiple tasks. In the experiments, the authors show that the proposed method retains good performance on the task where the pre-trained model is already good at while improves the performance on the target tasks. In addition to classification tasks, the authors also show that their patching approach could improve CLIP model on 1) typographic attacks, 2) object counting, and 3) visual question answering.", "questions": "Would appreciate if authors can address the questions mentioned in the weaknesses.", "limitations": "The authors discussed the limitations in the submission.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Improving CLIP model is an active research problem. The authors propose a method to improve the model performance on wider range of tasks for open-vocabulary classification.\n- The proposed approach is simple yet effective. To improve CLIP model on poorly performed task, the authors propose to freeze the classifier weights derived fro text encoder and fine-tune the model on given tasks. This preserve the ability of open-vocabulary classification while improve model performance on new tasks.\n- The experiments show decent improvement (1% ~ 20%) over the tasks for which the model is fine-tuned while preserving the performance for the task that CLIP model is already good at.\n- In addition to classification tasks, the authors show that their patching approach can improve object counting and VQA comparing to the pre-trained CLIP model\n\n\nWeaknesses\n- The writing could improve. The design of proposed method to retain open-vocabulary ability is based on freezing classification layer derived from text encoder. This only mentioned in one sentence in experimental setup. It was pretty confusing to me how the proposed method retain open vocabulary ability until I locate that single statement in experimental setup. I would recommend the authors to make it more clear through out the paper. \n- The proposed method is simple. However, in practice, it requires non-trivial effort. To patch the model, the proposed method requires supervised annotations and fine-tuning model with hyper-parameter search and validating the mixing coefficients. The procedure may not be simpler than adding downstream data to the pre-training dataset and then re-train image-text model. \n- The authors only compare the performance against pre-trained CLIP model. I am curious how the proposed method compare with a simple baseline: adding downstream data to the CLIP training dataset and re-train image-text model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept