Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
Authors: Shuo Chen
Jindong Gu
Zhen Han
Yunpu Ma
Philip Torr
Volker Tresp
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Robustness Benchmark
Multimodal Foundation Models
Model Adaptation
TL;DR: This paper presents a benchmark and analysis of the robustness of vision-language adaptation methods.

Abstract: Various adaptation methods, such as LoRA, prompts, and adapters, have been proposed to enhance the performance of pre-trained vision-language models in specific domains. As test samples in real-world applications usually differ from adaptation data, the robustness of these adaptation methods against distribution shifts are essential. In this study, we assess the robustness of 11 widely-used adaptation methods across 4 vision-language datasets under multimodal corruptions. Concretely, we introduce 7 benchmark datasets, including 96 visual and 87 textual corruptions, to investigate the robustness of different adaptation methods, the impact of available adaptation examples, and the influence of trainable parameter size during adaptation. Our analysis reveals that: 1) Adaptation methods are more sensitive to text corruptions than visual corruptions. 2) Full fine-tuning does not consistently provide the highest robustness; instead, adapters can achieve better robustness with comparable clean performance. 3) Contrary to expectations, our findings indicate that increasing the number of adaptation data and parameters does not guarantee enhanced robustness; instead, it results in even lower robustness. We hope this study could benefit future research in the development of robust multimodal adaptation methods. The benchmark, code, and dataset used in this study can be accessed at https://adarobustness.github.io.

Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
Shuo Chen 1.3- Jindong Gu Zhen Han Yunpu Ma Philip Torr² Volker Tresp 1.4
Institute of Informatics, LMU Munich *Department of Engineering Science, University of Oxford Siemens AG "Munich Center for Machine Learning (MCML) shuo shuo. chwnocampuscpss.coomm 1mu. de, d jindong. gußeng. ox. ac.uk, hanzhen021110163.
Abstract
Various adaptation methods, such as LoRA, prompts, and adapters, have been proposed to enhance the performance of pre- pre-trained vision-languaan models in specific domains. As test samples real-world applications usually differ from adaptation data, studying the robustness of these adaptation methods against distri- bution shifts is essential. In this study, we assess the robustness of widely-used adaptation methods across vision-language datasets under multimodal corrup- tions. Concretely, we introduce 7 benchmark datasets, including 96 visual and 87 textual corruptions, investigate the robustness of different adaptation methods, the impact available adaptation examples, and the influence of trainable parame- ter size during adaptation. Our analysis reveals that: 1) Adaptation methods more sensitive to text corruptions than visual corruptions. 2) Full fine-tur tuning does not consistently provide the highest robustness; instead, adapters
achieve better robustness with comparable clean performance. 3) Contrary to expectations, our findings indicate that increasing the number of adaptation data and parameters does not guarantee enhanced robustness; instead, results in even lower robustness. We hope this study could benefit future research in developing robust multimodal adaptation methods. The benchmark, code, and dataset used in this study can be accessed at hittt:t///// /tttts:////aaaeee htat:::://adrrobutteeesss http///arrbustttssiiiituuuubbbb github.io
Introduction
Employing large-scale pre- pre-trainn training dision-language (VL) models has become the standard for work
on VL tasks [38] [70 39] [71, (70 22 (7, (77). These models are typically trained in a self-supervised
manner on unlabeled web-scale datasets general domain [54] [.] To address the comain-speciffi challenges and improve performance downstream tasks, various model adaptation methods have
been proposed
29
36
Although adaptation methods can achieve promising results on various VL benchmark datasets, real-world applications often introduce various distribution shifts that differ from the conditions encountered during model adaptation [44). For instance, these shifts can manifest as variations in lighting conditions in images and typos texts. Therefore, it critical ensure model robustness against distribution shifts, particularly in safety-critical applications where unexpected wrong deci- sions can have severe consequences. such as self-driving systems (67 56 44) and clinical diagnostics 35 47). However, robustness research for multimodal models is still rare, leaving many essential questions unanswered: Which adaptation methods perform better on which tasks in terms of both performance and robustness? How robust are the various multimodal adaptation methods against.
"cqual contribution corresponning author
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Original Image
Gaussian Noise
Defocus Blur
Snow
A looking reflection in the mirror.
couple cats sitting on top a bathtub.
Two dogs looking thersselves mirror.
couple of standing next to each other
Original Text
Typos
Random Char Insertion
Drop Nouns
What is the starring mirror
What the cut Starri ng a7? What iss tche starrióng at? [UNK] starring
nothing
yes
cat
Figure 1: Multimodal adaptation methods are sensitive to image and text corruptions. The two rows show image captioning and visual question answering predicted by Adapter [29], respectively. Blue boxes contain the original image and query text. Orange boxes present the corrupted images, texts and model output.
visual corruptions, language corruptions, or both? Will more examples or more trainable parameters assure better robustness?
To this end, this work investigates the robustness of various adaptation methods on VL models to answer the above research questions. Concretely, we introduce a diverse set of 96 visual corruptions. including impulse noise, snow etc., and 87 textual corruptions encompassing text addition, back translation, etc. Moreover, extensive experiments have been conducted on adaptation methods across 4 VI. datasets, including VQAv2 [20]. GQA 31, NLVR² [6]] and MSCOCO Caption [5]. While several studies have explored the robustness of VI. models, our work represents significant advancement it provides the first large-scale benchmark robustness analysis of existing adaptation methods on VL models. We limit the models and tasks to those related to images, specifically, multimodal-to-tex models, e.g., CLIP-BART (62). Video-language models are outside the scope of this research.
Our analysis reveals several interesting findings: 1) Adaptation methods demonstrate higher degree of sensitivity towards text corruptions compared visual corruptions. 2) Full fine-tuning does not consistently yield the best relative robustness, whereas adapter can achieve better robustness with comparable performance. 3) Surprisingly, our experiments reveal that large quantity of adaptation data and model parameters do not guarantee improved robustness. In fact, increasing the amount of adaptation data might even lead decreased robustness. 4) There is no single adaptation method that surpasses others across all tasks and corruptions. To summarize, our contributions are as follows:
We construct a suite of large-scale robustness benchmark datasets including 96 visual corruptions and 87 textual corruption methods.
We evaluate the robustness of adaption methods on VL models based on massive experi- ments.
We release the benchmark, code, as well as a leaderboard the community to facilitate future research on the robustness of multimodal adaptation methods.
2 Related Work
have shown out- standing performance various VL tasks. Some use contrastive learning align visual features with language representations and achieve surprising zero-shot performance 42 54 However, contrastive dearning-based methods are limited to close-ended tasks and are inflexible. Another line of work follows BERT's [11 seetrainnttee--iiee
Vision-language Models. Pre-trained VL models [4] 60 7
45 40 64, 13, 79]
paradigm 41 60 7 64]. They treat object features
extracted using pre-trained object detectors [18 as visual words sent to language models [[]. For example, VL--ARTT 7 uses BART 337 or T5 (55] as the text encoder and Faster-RCNI [18 as the visual backbone. Unlike other methods, VL-BART unifies VI. tasks via a single text generation task. CLIP-BART
62 follows the same idea VL-BART but adopts the CLIP [54) image encoder

---

to extract pixel-level features. Recent approaches [66 14 follow such unified view and freeze large language models (LLMs) to utilize the in-context learning ability of LLMs. However, as shown in [62] LM fine-tuning is still crucial to achieve competitive performance on various downstream VL tasks. In this study, we follow the work in [2] that benchmarks model adaptation methods. CLIP-BART [62] is selected as our VL. model, given its generation flexibility and unified architecture.
Model Adaptation Methods. To enhance the performance of pre-trained VL models on downstream tasks and avoid infeasible computation, various adaptation methods have been proposed. Existing methods can be classified into three categories [2]: (1) adding a few trainable parameters while
freezing other model parts (29 46 43
3
36 ; (2) updating few model parameters sparsely [25 63
76]: and (3) low-rank factorization of parameters to be updated, such as in LoRA B0) Adapters (9
belong to the first category and have been widely used in vision, language, and multimodal models
29 78 6. Other representative methods in the first category include Hyperformers [46], Compacters 33 and prompt-tuning (36] 31. Although numerous adaptation methods have been proposed and widely adopted, their robustness against distribution shifts remains understudied.
Natural Robustness. The robustness of deep learning models against distribution shifts is critical Regarding vision robustness, researchers have investigated
for world applications [7
26
16
image classification models 26
48.. video classification 75,, and transformer- semantic segmentation [32
16
28 57
21
22
24] object detection
architectures 12 51
52
23 73
In the field of
59 72 and various methods [15] 10 50, [ are available. The robustness investigation on multimodal models is gaining
natural language processing (NLP), many robustness analysis toolboxes [58
more attention but related studies are lacking. The literature includes the robustness of multimodal
fusion models 744 audio-visual models [65] text-to-image generative models g text-to-video
retrieval models
well image-text models (53]
In contrast to all the works above, our study focuses on the robustness of adaptation methods integrated into large pre-trained vision-language models. Understanding their robustness on different VL tasks will facilitate the design of more robust adaptation methods for multimodal models.
3 Preliminaries of Model Adaptation Methods
The pretrain- seetain-then-finne fientffietune paradigm on large models has shown dominant performance on multimodal tasks [4] [T [54] yet the prohibitive costs of full fine-tuning have spurred intensive research efforts
46 33 As the transformer architecture (68) is used for most state-of-the-an large pre-trained models, adaptation methods mainly focus on tweaking the input or the intermediate layers of the attention layers inside the large models. Formally, given a pre-trained large-scale model F' parameterized by 0, we need to adapt F on a task-specific dataset D, e.g., a VQA dataset, to get the adapted model F", Then, we can obtain the output y = [x 0) by providing an input x= {Ii,,... } with tokens from D. Adaptation methods differ in how they interact with F f(x; 0) (Fig. 2). In general, full fine-tuning updates all O.
towards developing parameter-efficinn
adaptation methods 662 30 36
29
Prompt (36] concatenates the input x with an extra prefix. LoRA 30 introduces modifications to
the update mechanism of the model parameters o and adapters 29 46 33 modify the intermediate output and input of O.
Full fine-tuning directly updates the whole 0 on D and becomes prohibitive due to the rapidly growing model size. Therefore, the following adaptation methods have been developed achieve comparable performance while optimizing only a few parameters.
rraset adaptation concatenates the input x with either trainable prefix (soft prompt) (36) or a manually designed prefix (3] For the given input x = f=i,..., ) with n tokens, the pre-trained model will first form an embedding matrix X E 2axd where d is the dimension of the embedding space. Soft-prompts [36] are then represented as a learnable parameter P E IPxd where p is the length of the prompt. Next, P is concatenated with the original embedded input X to form a new single matrix defined as [P: X] E R((+n)xd During adaptation, the model is trained to maximize the probability of the desired output while only updating P.
LoRA 30 utilizes low-rank decomposition matrices to update parameters. For intermediate model
parameters On E dR such as the parameters from self-attention module in the transformer archi- tecture, its update A0, is represented by low-rank decomposition A = BA, B E Rdxr, A e grxk, r < min(d, k). During adaptation, 0, is frozen while B and A are updated.
3

---

Full fine-tuning
Prompt
LoRA
Adapter
Adaptor
Generated Texts
Generated Texts
Generated Texts
Generated Texts
Hyperformer
Pre-trained Model
Pre-trained Model
Attention
B
Adaptar
Iyporformn
A
Attention
Compacter
AuB:
[ embterel
Solt Prompt
embiigg embreal
[embing embura
[embinn embertt]
Compaccior 4g.00
Figure 2: Illustration of adaptation methods used in our study. Green areas indicate trainable parameters whereas frozen parameters are in blue. The input the concatenation of image and text embedding (embing embteat) and the output is the generated text.
Adapter-based adaptation inserts sub-networks with a few learnable parameters into the large model. Adapter [29] consists of a pair of downsampling and upsampling layers as well as residual connection. Suppose the original input to an intermediate layer Ou in model F is Xy E R adapters insert a downsampling layer 0² E RRd xd1 and an upsampling layer l² E R. xio,, where do, di are dimensions of the hidden embeddings, respectively. The output after injecting adapters is defined as h far (a (So (xo))) + Xu. where for denotes function parameterized by g and (-) an activation function such as GELU [27]. To further reduce redundant parameters in adapters, Compacter 333 decomposes parameter matrices. It introduce parameterized hypercomplex
multiplication (PHM) layers 0 = > A, @ B,, A; E R B, E IR 2 x + which decompose the layer in the adapter by Kronecker products. Compacter also shares the parameter of A, across all layers and decomposes B; even further with low-rank decomposition. However, as found in [62], such sharing and further decomposition severely decreases the VL performance. In our study, we only use PHM layers. Hyperformer [46] relies on a hyper-netword shared across tasks to generate the weights in adapters given task index and layer index. The hyperformer maintains learnable embeddings for each task and each layer. For Ny tasks and Nr. layers, the learnable embeddings in the hyperformer denoted f1..... byT E Rdd and 4 1..... , IN, E Rdd, respectively. The hyperformer consists of ...
:-11
task projector 0" E IR (d, and a hyper-network 0 x +d, +d, and generates an adapter's weights in the 7th layer for the the jth task following
0D, O² fox (fer ([tj,4)))
Adaptation shared over tasks [2] further reduces redundant parameters by exploiting similar information shared across multiple tasks. In multi-VL-task setting, intuitive way is to train the adaptation modules per task using: Multiple Adapters, Multiple Compacters, Multiple LoRA, and Multiple Prompts. Additionally, We can train only one set of adapter layers for all tasks, and we have Single Adatter, Single Compacter, Single LoRA, and Single Prompt. Besides, Half-shared adapter [62] only shares the upsampling layers downsampling layers across different tasks. Detailed information is presented in Appendix B.2
4 Corruption Methods
A blank method is also added, which is used to examine the importance of visual information to VI models. Blank corruption turns the original image into blank picture by setting all pixel values 255. All image corruptions can be categorized into five groups: noise, blur, weather, digital, and extra. Specifically, we use 20 image corruption methods, (1) noise: impulse noise, Gaussian noise, shot noise, speckle noise; (2) blur: zoom blur, defocus blur, motion blur, frosted glass blur, Gaussian blur; (3) digital: JPEG compression, contrast, elastic, spatter, saturate, pixelate; (4) weather: snow, frost, fog, brightness; and (5) extra: blank. We follow the severity convention in ImageNet-C [26] and define 5 levels of severity for each method, except for blank corruption. total, we have 96 types of visual corruption and we leave the details in the Appendix A. By applying all image corruptions to 4 datasets used in this study, i.e., VQAv2 (0), GQA 31 NLVR2 [1 and MSCOCO Caption (5).
Image Corruptions. We the corruption methods from ImageNet-C (26] and 4 (53
construct 4 out-of out-of-distribution (OOD) benchmark datasets.

---

Table Dataset Statisties.
VQAv2
GQA
NLVR
MSCOCO Caption
The Number of
Images
QA pairs
Images 72.1K
QA pairs
Images 103.2K
QA pairs
Images
Captions
Training
113.2K
605.
943.0K
86,4K
113.2K
566.8K
Validation set
5.0K
26.7K
10.2K
132.1K
8.1K
7.0K
5.0K
5.0K
Test set 5.0K
26.3K
398
12.6K
8.1K
7.0K
5.0K
5.0K
Text Corruptions. In addition to visual feature shifts, text corruptions are also essential for evaluating the robustness of aasion-language models [ [3).. We have incorporated total of 35 corruption methods, inspired by the approaches presented in 772 53 4. These methods can be categorized into three groups based on the level of corruption: character-level, word-level, and sentence-level. Furthermore, they are further subdivided into six sub-categories, namely character modification, text style modification, text addition, dropping text based on POS, positional drop, and text swap. To name a few examples, the category text style transforms sentences to desired styles such as passive, formal, or double negative. Text addition inserts extra words, like adverbs in InserIAdv. Text drop modifies words based on POS tagging. dropping nouns (DropNN) or verbs (DropVB), For detailed information, please refer to Appendix A.2 In addition the above corruptions, Qiu et al. proposed in [53] that we should ensure that the corrupted text has the same semantics as the original one to make sure the image-text pairs remain meaningful. We follow this setting and use the same fidelity guarantee mechanism as [53) Various severity levels for text corruptions are also introduced the benchmark, including 5 severity levels on character-level corruptions and some word-level corruptions. For sentence-level corruptions, only one perturbation is available. In total, we have 87 different perturbations. After applying all text corruptions on VQAv2 [20), GQA (31), and NLVR? [61], we construct another 3 OOD benchmark datasets. In the end, we have constructed 7 OOD robustness benchmark datasets to fully investigate the robustness of adaptation methods on vision-language models.
5 Experimental Settings
Tasks and Datasets.
The popular representative VI tasks (visual question answering. visual rea- soning, and image captioning) and well-known VI. datasets are applied in this work. For visual question answering. VQAv2 [20) and GQA [3]] are adopted. Additionally, we incorporate NLVR² [61] for visual reasoning and MSCOCO
5 for image captioning. The statistics are shown in Table [
Models. CLIP-BART (T5) 62 serves as our base model. Because the model adaptation on VL models mainly on language model components and the encoder-decoder architecture can tackle VL tasks via unified text-generation task (62.. CLIP-BART (T5) utilizes a single-stream fusion scheme, where the language model takes the concatenation of visual representations and text representations input. The single-stream approach enables the model to effectively integrate visual and textual information, leveraging the complementary strengths of both modalities. Specifically,
LeP-RReNNe1101 [54] is the vision encoder that receives resized 224 x 224 images, and representations
from the last convolutional layer are extracted as the visual features. BART base [37 and T5 55 deal with the downstream text generation task.
Model Adaptation Methods. We investigate the robustness of four mainstream adaptation methods: full fine-tuning, soft prompt (36), LoRA A (30], and adapter-based methods, including Adapter [29], Hyperformer [46). and Compacter [33]. To better understand their robustness, shared adaptation methods are also investigated. Specifically, for soft prompt, i.c., LoRA, and Compacters, conduct experiments in both single and multiple manners and the half-shared manner for Adapter (Section
See Appendix B.2 for detailed information, e.g., training strategy and hyperparameters.
VL Task Evaluation Metrics. Accuracy on the Karpathy-test split evaluated for VQAv2. For GQA, accuracy on the test-dev split is evaluated, and accuracy on the test-P split is used for NLVR". In image captioning, we use CIDEr (69] on the Karpathy-test split.
The model performance Pr D, (i.e., in-distribution test datasets) and Po on Do (i.e., out-of-distribution test datasets) are first evaluated, where P the corresponding evaluation metric for each task, such CIDEr [69] for image caption. Then, the
Robustness Evaluation Protocol.
5

---

Table 2:
Clean performance and relative robustness (RR) of adaptation methods based on CLIP- BART against image (up) and text (down) corruptions. RR and the corresponding standard deviation are averaged and calculated over all image or text corruption methods. The percentage of trainable parameters for each adaptation method is also reported. We strike out those high RRs with quite low performance. The best RR for each column is in bold.
Adaptation method Updated
VQAv2
GQA
NLVR
MSCOCO Caption
Image Corruptions
Params Acc (%)
RR (%)
Ace (%)
RR (9)
Acc (%)
RR (%)
CIDEr
RR (%)
Full Fine-tuning
100%
66.75
84.8645.17
55.04
89.20
73.01
90.34+004
115.03
68 68.40
Multiple Adapters
12.22%
65.30
85.33:1450
53.39
86.162004
69.41
92.02.1044
114.47
68. 18.722..14
Half-shared Adapters
8.36%
65.20
85.18
52.96
89.37
70.03
91.72:1044
I 114.50
68.45
Single Adapter
4.18%
65.35
85.7615532
54.14
82.49
73.89
90004
115.04
68 68
Hyperformer
5.79%
65.38
85.38+434
52.52
90.05=0044
72.21
10.13
114.89
68.7440.14
Multiple Compacters
7.05%
64.91
85.65+481
52.75
88.89
69.45
91.33
115.6
68.67
Single Compacter
2.70%
64.47
85.477+496
52.90
82.62
69.94
92.04
113.06
69.92=a.13
Multiple LoRA
17.72%
65.44
84.78+446
52.05
91.15
51.32
115.41
68.47
Single LoRA
5.93%
65.34
84.7814.81
53.19
82.58
73.58
90.05
114.54
69.26
Multiple Prompts
4.53%
46.81
34.01
49.87
108.62
67.70
Single Prompt
2.00%
44.00
37.54
51.95
103.70
68.56
Adaptation method
Updated Params
VQAv2
GQA
NLVR
Text Corruptions
Acc (%)
RR (%)
Ace (%)
RR (%)
Acc (%)
RR (%)
Full Fine-tuning
100%
66.75
73.65
55.04
66.92:22.14
73.01
17.06.+1100
Multiple Adapters
12.22%
65.30
26.62+20.6
53.39
66.93-422.43
69.41
90.14+10.19
Half-shared Adapters
8.36%
65.20
76.78
52.96
68.20+
70.03
89. 19.16:11012
Single Adapter
4.18%
65.35
77.64
54.14
67.47:120003
73.89
88.4941087
Hyperformer 5.79%
65.38
75.06
52.52
70.30
72.21
87.27
Multiple Compacters
7.05%
64.91
27.10
52.75
47.39
69.45
90.00
Single Compacter
2.70%
64.47
77.17
52.90
67. 20.33
69.94
90. 10
Multiple LoRA
17.72%
65.44
74.04
52.05
68.77
51.32
Single LoRA
5.93%
65.34
74.50=2142
53.19
63.94412.99
73.58
87.64
Multiple Prompts
4.53%
46.81
34.01
49.87
Single Prompt
2.00%
44.00
37.54
51.95
Relative Robustness RR = 1 AP/P, 53 4 is computed based on the clean performance Pa and corrupted performance Po. where AP- = (Pr Po). RR is a score ranging from 0 to 1, where RR indicates that F is totally robust and R 0 means that F is not robust all. The RR with severity 5 is reported across the main paper; detailed scores on others are in Supplementary.
6 Results and Analysis
Sec. 6.11 examines the robustness of each adaptation method and tries to answer the first question: Which adaptation methods perform better which tasks with respect both performance and robustness? Sec. compares the robustness sensitivity on image and text corruptions and looks for the answer to how robust the various multimodal adaptation methods against visual corruptions, language corruptions, or both? In Section 66.3 we analyze the influence on robustness given different sizes of adaptation data and trainable parameters. Especially, aim answer will more examples or more parameters ensure better adaptation robustness?
6.1 Robustness of Multimodal Adaptation Methods
Full fine-tuning, prompt-tuning. LoRA, and adapter-based methods are four types of adaptation methods investigated in this study, and their relative robustness against image and text corruptions are presented in Table ,The reported relative robustness is the average value across all images or .The
text corruption methods. Although full fine-tuning generally achieves higher clean performance, our analysis reveals that its robustness is comparatively weaker than other adaptation methods. In many cases, the adapter and hyperformer achieve better robustness with much fewer parameters and comparable clean performance, For instance, full fine-tuning's against text corruptions on the v VQAv2 dataset is the smallest, for both CLIP-BART and CLIP-T5. Prompt tuning, despite exhibiting

---

high robustness, fails to perform well on the clean test dataset. The same conclusions can be drawn on corrupted data with different corruption levels, as shown in Supplementary. Please note that we have excluded robustness scores associated with very low task performance in Table 2
Single Adapter vs Full Fine-tuning. Previous research [6] has shown that a single adapter can achieve comparable performance on the four tasks with significantly fewer parameters than full fine-tuning. When it comes to robustness, a single adapter comparable to or slightly better than full fine- fine-tuning on VQAv2, NLVR?, and MSCOCO Caption given image corruptions. The same goes for text corruption. For example, as shown in the 4th row and 1st row in Table 2 (lower panel), single adapter on CLIP-BART achieves an average RR of 77. 74.4 against text corruptions on VQAv2, while full fine-tuning's s RR is 73.65%. However, on GQA, a single adapter less robust than full fine-tuning. Full fine-tuning achieves an average RR of 89.20% against image corruptions, while the RR of a single adapter is only 82.49%. Also, a single adapter's 's clean accuracy on GQA is lower than full fine-tuning's 's 55. 04%. In contrast, multiple and half-shared adapters have more parameters but achieve better robustness on the four tasks than a single adapter. In conclusion, a single adapter can achieve similar or better robustness on VQAv2, NLVRS, and MSCOCO Caption compared to full fine-tuning. On GQA, multiple and half-shared adapters are better.
Adapter-basse based Methods. Although training multiple tasks with one set of adapter layers has the least number parameters, such a single setting might hinder the robustness on certain tusks. For instance, Single Adapter's robustness that of the half-shared (89.37 37%% and multiple settings 16%)..) This also applies to Single LoRA and Single Compacter. An explanation could be that the half-shared mechanism does not only learn more general representation across tasks; it also maintains task-specific knowledge. On GQA, Single LoRA's robustness against image corruptions is lower by 8.57% compared to Multiple LoRA's, and the robustness against text corruptions is lower by 4.83%. However, compared with multiple settings of LoRA and Adapter, the Hyperformer has relatively fewer parameters but achieves comparable or better robusiness. Hyperformer on CLIP-T5 also obtains the best robustness results against image corruptions all four tasks.
GQA against image corruptions (82.4%%) is lower than
Robustness against Natu- ral Dataset Distribution Shift. To provide a more re- alistic assessment of the bustness, an additional natu- ral distribution shift corrup- tion is included in our work. VQA-RAD 34 is man- ually constructed dataset where clinicians asked nat- urally occurring questions about radiology images and provided reference answers. The images in VQA-RAD are shown neither in model pre-training nor in adaptation and can be seen as natural out-of- distribution dataset. Results in Fig. 3 are relatively low but adapters perform relatively well, such as the Single Adapter in CLIP-BART and Multiple Adapters in CLIP-T5. Full finetuning fails to generalize well in CLIP-BART compared to other adaptation methods whereas the performance of CLIP-T5 with full finetuning is the second best.
VOGHAD acouracy (CUPAAAII)
VIMARAD accurac JELINSS
6.23
2-0.0
602
0.22
6:2
0.2
620
0.2
B sangigt stador
Mustllc Sopein Ausgee
Figure 3: The accuracy results on VQA-RAD 34 which include 11 adaptation methods on CLIP-BART (left) and on CLIP-T5 (right).
Vision-language Tasks. Among all datasets, MSCOCO Caption the most vulnerable one against image corruptions, where all adaptation methods have decreased on average more than 30%. This is plausible as only relies on visual information, whereas other tasks provide both visual and language information. Besides, GQA is the task with the lowest robustness performance against text corruptions. Moreover, on GQA, the extreme single-module setting fails to achieve good robustness, such as Single Adapter, Single LoRA, and Single Compacter. This indicates that information sharing with the other two datasets may hinder the robustness on GQA. To overcome such an issue, one can adopt the multiple-mool moople manner or Hyperformer. Adaptation methods show better robustness on NLVR2 compared to the other three tasks on both corruptions.
7

---

CLIRBART
MSCOCO Caption
Full Fineturing
Relative Robustness
Multiple Ads pters
Acapters
Single Adepter
Hypertomer
60-
23
Multiple Compacters
Single Compacter
Single LoRTA
VOA+2
0GA
CLP-BART
CLIPITS
Figure 4: Relative robustness (%) of adaptation methods based on CLIP-BART (left) and CLIP-TS (middle) against blank corruption. We group MSCOCO Caption results from CLIP-BART and CLIP-T5 together in the right sub-figure. We omit two bars in NLVR from the middle figure as multiple adapters and multiple compacters did not perform well.
6.2 Robustness Sensitivity to Image Corruptions and Text Corruptions
This work introduces both image and text corruptions to examine the robustness of various adaptation methods. Our experimental findings suggest a potential vulnerability of adaptation methods on multimodal VL models text corruptions, particularly those the character level. Across all three tasks, the udaptation methods exhibit lower robustness indicators against text corruptions. For instance, Single Adapter based on CLIPP-BAR RT has the best robustness result 85.76% against image corruptions on VQAv2. However, although it is still the most robust adaptation method against text corruptions, the relative robustness 77.64% Among image corruptions, zoom blur drops the robusiness the most, and within text corruptions, char-level methods are most challenging to these VL adaptation methods. Detailed analysis is in Supplementary Section 6.
Blank Image Corruption. Blank corruption evaluates the influence on the robustness of visual information in VL tasks. All datasets used in this study rely on visual information and only the MSCOCO Caption contains no language information. In blank corruption, we set the pixel values of testing image data to 255, i.e. transforming the original image into a blank image. The results are shown in Fig. 4 The relative robustness on MSCOCO Caption is the lowest among all four datasets. This is plausible since image captioning relies only on visual information and is not supposed to perform well given blank image. Apart from image captioning, adaptation methods on all three datasets could secure relative robustness exceeding 50% the absence of useful visual information. Several questions within the VL tasks can be accurately answered without relying visual information, suggesting that language information plays a more significant role than visual information. This also explains the higher sensitivity to text perturbations compared to the sensitivity to image corruptions.
Compounding Effects of Multimodal Corrup- tions. To assess the impact of compounding distribution shifts both the visual and text modalities, we selected a subset of corruption methods from each category and presented the results in Fig. 5 Specifically, for text corruption methods, we selected ocr the character level, swap syn word embu at the word level, and back translation the sentence level. For visual cor- ruption methods, we selected Gaussian noise in the noise category, zoom blur in the blur cat- egory, JPEG in the digital category, and snow in the weather category. We tested the full fine- tuning and single adapter on CLIP-BARY and CLIP-T5. The results demonstrate that combin- ing corruptions from two modalities can lead to a greater drop robustness. Additionally, the re- sults show similar trends as the single-modal cor- ruptions. Character-level corruptions still lead to the most severe performance drop compared
ROET But Pretacile
CLP KONT
2:18
21
9:13
2118
20.0
2:21
Syerrym
21:1
21
2043
21/5
23
nonce Grop
CUP113 AAgie
Prai
0.2
2G
2031
M:M
31.
3,41
20:11
34.44
U4X
12.4
Kober
Bhrr
Nome
Blrr
Figure 5: The average performance drop given both visual and text corruptions. A darker color indicates a severe performance drop.
8

---

Vigh?2
NSCOCO Copton
1.70
Accuracy (%)
0.00
$3.93
312
0.005
316-
-3073
300
25%
47%
6th: 66h:
109%
47%
60%
100%
30%
67%
06%
100%
300
256
40%
50%
2ix
1076
Rebusiness (1)
VQJr22
G044
NEVR-2
0.22
MSCOCC Coption
0.00
Sigs Alspar
0.31
04.5
Hepepetteeer
Myiple Compoctess
04.5
Figure 6: The first row represents the clean performance and relative robustness of full fine-tuning and single adapter on CLIP-BART given the different sizes of the adaptation dataset. Green lines stand for performance in each task and the orange is robustness. The second row is relative robustness given the different sizes the adaptation dataset. The X-axis shows the random subset ratio of the training dataset during adaptation, ranging from 20% to 100%.
to sentence- and word-level corruptions. Zoom blur still drops the robustness the most among image corruptions.
6.3 The Influence of Adaptation Data Size and Parameter Size on Robustness
Adaptation Data Size. Adaptation methods are gaining more attention due to their efficient fine- tuning manner compared to full fine-tuning. We take a further step and evaluate their performance and robustness given different sizes of training data during adaptation. Fig. compares the performance and robustness of full fine-tuning and single adapter. Given more adaptation data, performance in all tasks has a steady increase, and most cases, the performance of full fine-tuning surpasses single adapter's 's performance, while the latter achieve comparable performance given the whole adaptation data. Only on MSCOCO Caption, single adapter outperforms full fine-tuning given a subset of adaptation data. Single adapter achieves better RR compared to full fine-tuning against both image and text corruptions but has a robustness drop on GQA. Fig. 6 also demonstrates the robustness of other adaptation methods. All lines present steady declining tendency, which indicates that increasing the size of the adaptation data does not consistently enhance relative robustness. Besides, the blue full fine-tuning, lines take the lowest position on all three datasets given text corruptions and on VQAv2, NLVR² and MSCOCO Caption given image corruptions. In comparison other adaptation methods, full, fine-tuning has relatively lower relative robustness, despite having the most trainable parameters. The last conclusion is that there is no single adaptation method that surpasses others across all tasks and corruptions and all methods share similar robustness fluctuation given different size of the adaptation dataset.
Adaptation Parameter Size. Fig. 7 presents RR and clean performance of prompt-tuning given different soft prompt lengths added to the concatenated embeddings. There is a steady increase in the performance on four tasks with longer prompt lengths which proves that prompt methods perform better given more parameters. Regarding relative robustness, such a steady increase does not apply to all tasks, and longer soft prompts do not ensure better relative robustness. Fig. also shows the experimental results from the other 4 adaptation methods given different sizes of trainable parameters. The results demonstrate that more parameters do not ensure enhanced robustness and some even reduce it, such as the single compacter and single adapter on GQA.
7 Discussion and Conclusion
This study focuses on the robustness of adaptation methods on pre-trained lision-language models and provides 7 benchmark datasets containing 96 visual and 87 textual corruptions. We systematically inspect the robustness of 11 adaptation methods on popular VL datasets and conclude that: 1) these adaptation methods are more sensitive towards text corruptions compared to visual corruptions, 2) full fine-tuning does not achieve the best robustness; instead, adapters demonstrate better robustness while maintaining comparable performance, 3) surprisingly, more adaptation data and more parameters do

---

Vigha2
GQA
NEVR--2
MSCOCO Caption
Accurecy
7133
Druk Nore
Natgla hangge
Sngl
40.07
14 Ooglan
V06-7
0.
Capson
v0h22 N
Ad0con Capton
30
M002
MODDED
Figure 7: The top row shows the clean performance and relative robustness from prompt adaptations with different prompt lengths CLIP-BART. Blue lines stand for performance on each task and purple lines represent relative robustness. The bottom row shows the relative robustness given the different number of parameters in adaptation methods. Different colors stand for different embedding sizes and larger numbers are with more parameters.
not ensure better robustness. In fact, it can even lead to worse robustness, 4) there is currently no adaptation method achieving both the best performance and best robustness across all corruptions and all tasks. The main limitation of this work is that the analysis is limited number of multimodal models due to the availability of usable code, model weights, and massive experiments. Potential future work includes investigating more diverse pre-trained VL models, designing more robust adaptation methods, and integrating future model adaptation methods to make our benchmark up-to-date.
Acknowledgments and Disclosure of Funding
This work is partially supported by the UKRI grant: Turing AI Fellowship EP/W002981// and EPSRC/MURI grant: EP/N019474/L. We would also like thank the Royal Academy of Engineering and FiveAI. We also gratefully acknowledge the computational and data resources provided by the Leibniz Supercomputing Centre (www.lrz.de).
References
[1] Jean- Bant-Baettte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237366 2022.
Hangho Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Suhhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mfxture-of ooo.oalltoommm----ooooooo experts. Advances in Neural Information Processing Systems, 35:32897-32912 2022.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, ct al. Language models are few-shot leamerrs.
Advances in neural information processing systems, 33:1877-1901, 2020.
[4] Madeline Chantry, Shruti Vyas, Hamid Palangi. Yogesh Rawat, and Vibhav Vineet. Robustness analysis of video-language models against visual and language perturbations. Advances in Neural Information Processing Systems, 35:34405 34420, 2022.
[5] Xinlei Chen, Hao Fang. Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and
C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 10Xiv:1504.00:255 2015.
[6] Zhe Chen, Yuchen Duan, Wenhai Wang. Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08522 2022.
[7] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-Bel and-aanuugee tasks via text generation. In International Conference Machine Learning, pages 1931-1942. PMLR, 2021.
10

---

[8] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arXiv preprint arXiv:2202.04053, 2022.
[9] Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120- 100-128.000 2019.
[10] Dorottya Demszky, Devyani Sharma, Jonathan H Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. Learning to recognize dialect features. arXiv preprint arXiv:2010./2700 2020.
[11] Jacob Devlin, Ming-Way Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18Xiv: 04805, 2018.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[13] Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. arXiv preprint arXiv:2202.109) 2022.
[14] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.
Magma-multimodal augmentation of generative models through adapter-hased finetuning. arXiv preprint arXiv:2112.05253. 2021.
[15] Matt Gardner, Yoav Atzzi Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models' local decision boundaries via contrast sets. arXiv preprint urXiv:2004.000 02709, 2020.
[16] Robert Geirhos, Patricia Rubisch, Claudio Michaclis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.1233: 2018
[17] Robert Geirhos, Carlos RM Temme. Jonas Rauber, Heiko H Schüit, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
[18] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[19] Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. Robustness gym: Unifying the nip evaluation landscape. arXiv preprint Stti:22011008400 2021.
[20] Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9904-6913, 2017.
[21] Jindong Gu and Volker Tresp. Improving the robustness capsule networks to image affine transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7285- 7293, 2020.
[22] Jindong Gu, Volker Tresp, and Han Hu. Capsule network is not more robust than convolutional network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14309- 14317, 2021.
[23] Jindong Gu. Volker Tresp, and Yao Qin. Are vision transformers robust to patch perturbations? In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI1, pages 404-221. Springer, 2022.
[24] Jindong Gu, Hengshuang Zhao, Volker Tresp, and Philip HS Torr. Segrgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Isruel, October 23-27, 2022, Proceedings, Part XXIX, pages 308-325. Springer, 2022.
[25] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463, 2020.
[26] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.1226. 2019.
[27] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606. 0016600008415... 3, 2016.
[28] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
11

---

[29] Neil Houlsby. Andrei Ciurgiu. Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for alp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 106.09685, 2021.
[31] Drew A Hudson and Christopher D Manning. Gqa: A dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 77000 2019.
[32] Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8828-8838,
2020.
[33] Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022-D1 100222003,200 2021.
[34] Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 1-1), 2018
[35] Joonsang Lee, Angela Steimmann, Yao Ding, Hannah Lee, Constance Owens, Jihong Wang, Jinzhong Yang. David Followill, Rachel Ger. Dennis MacKin. et al. Radiomics feature robustness as measured using an mri phantom. Scientific reports, 11(1))3973, 2021.
[36] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. arXiv preprint arXiv:2104.... 086611 2021.
[37] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising generation, translation, and comprehension. arXiv preprint 1110.1910.134611 2019.
pre-training for natural language
[38] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang. Ming Yan, Bin Bi, Jiabo Yc, Hehong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint arXiv:2205.12000 2022.
[39] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301 2tt.::39.1.11997,, 2023.
[40] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
[41] Liunian Harold Li. Mark Yatskar, Da Yin. Cho-Jui Hsieh, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprini arXiv: 908.003557,2200 2019.
[42] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. arXiv preprint arXiv:2012. 15409, 2020.
[43] Xiang Lisa Li and Percy Liang. Prefix- trefix-tuning: Optimizing continuous prompts for generation. arXiv preprint Xiv: 000900000000...00000000000000000 2021.
[44] 7e Liu, Yingfeng Cai,, Hai Wang, Long Chen, Hongho Gao, Yunyi Jia, and Yicheng Li. Robust target recognition and tracking of self-driving cars with radar and camera information fusion under severe weather conditions. JEEE Transactions on Intelligent Transportation Systems, 23(7):64006655 2021.
[45] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and -nd--and--laan language tasks. Advances in neural information processing systems, 32, 2019.
[46] Raheeh Karimi Mahabadi, Sehastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernttwrrs.. arXiv preprint arXiv: (06,049996 2021.
[47] Joscha Maier. Elias Eulig, Tim Vöth, Michael Knaup, Jan Kuntz, Stefan Sawall, and Mare KachelrieB. Real-time scatter estimation for medical et using the deep scatter estimation: Method and robustness analysis with respect to different anatomics, dose levels, tube voltages, and data truncation. Medical physics, (6(1):238-249 2019.
[48] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Mauhias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint 19740074474 2019.
[49] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(1 1):39 41. 1995.
12

---

[50] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In International Conference on Machine Learning, pages 690S-6916. PMLR, 2020.
[51] Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. Conference on Artificial Intelligence, volume 36, pages 2071-2088 2022.
In Proceedings of the AAAI
[52] Francesco Pinto, Philip HS Torr. and Puneet K. Dokania. An impartial take to the can vs transformer robustness contest. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XIII, pages 466 480. Springer, 2022.
[53] Jiclin Qiu, Yi Zhu, Xingjian Shi, Florian Wenzel, Zhiqiang Tang, Ding Zhao, Bo Li, and Mu Li. Are multimodal models robust to image and text perturbations? arXiv preprint arXiv:2212. 2022.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI.R 2021.
[55]
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2 21(5))5485-5551 2020.
[56] Julien Rebut, Andrei Bursuc, and Patrick Pérez. Styleless layer: Improving robustness for real-world driving. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8992-899 99998999. IEEE, 2021.
[57] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5440 5400. PMLR, 2019.
[58] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing nlp models with checklist. arXiv preprint arXiv:2005.00 04.18, 2020.
[59] Barbara Rychalska, Dominika Basaj. Alicja Gosiewska, and Przemyslaw Biecek. Models in the wild: On corruption robustness of neural nip systems. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part III 26, pages 235-247. Springer, 2019.
[60] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and lifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: 08530, 2019.
[61] Alane Suhr, Stephanie Zhou, Ally Zhang, Tris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint 1xXx::8111.0099) 2018.
[62] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision- and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5227-5237, 2022.
[63] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, Advances in Neural Information Processing Systems, 01224193-22200 2021.
[64] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint 1rXiv: 07490, 2019.
[65] Yapeng Tian and Chenliang Xu. Can audio-visuu visual integration strengthen robustness under multimodal attacks? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5601 1601-5611, 2021.
[66] Maria Tsimpoukelli. Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
[67] James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Eilyan Bitar, Ersin Yumer, and Raquel Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving. arXiv preprint arXiv:2101.0675 2021.
[68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[69] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4577 2015.
[70] Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to to- sequence learning framework. arXiv preprint 220.:2200000022... 2022.
13

---

[71] Wenhui Wang, Hangbo Bao. Li Dong. Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed. Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for all vision and vision-language tasks, arXiv preprint arXiv:2208. 10442. 2022.
[72] Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, et al. Textflint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joins Conference on Natural Language Processing: System Demonstrations, pages 347-355, Online, aug 2021. Association for Computational I. Linguisties.
[73] Boxi Wu, Jindong Gu, Zhifeng Li, Deng Cai, Xiaofei Hc, and Wei Liu. Towards efficient adversarial training on vision transformers. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIII. pages 307-325. Springer, 2022.
[74] Karren Yang. Wan-Yr Yi Lin, Manash Barman, Filipe Condessa, and Zico Kolter. Defending multimodal fusion models against single-source adversaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3340-3340 3349, 2021.
[75] Chenyu Yi, Siyuan Yang. Haoliang Li. Yap- peng Tan, and Alex Kot. Benchmarking the robustness of spatial-temporal models against corruptions. arXiv preprint arXiv:2110.06513, 2021.
[76] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameler-efficiet fine-tuning for transformer-basee masked language-modee models. arXiv preprint arXiv:2106. 10199, 2021.
[77] Yan Zeng. Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, and Wangchunshu Zhou. X 2.ylm: 2022.
All-in-one pre-trained model for vision-language tasks. arXiv preprint arXiv:2211.1244
[78] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.0300 2021.
[79] Ce Zhou, Qian Li. Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji. Qiben Yan, Lifang He, al. A comprehensive survey pretrained foundation models: A history from hert to chatgpt. arXiv preprint arXiv:2302.0 09419, 2023.
14

---

A Corruption Details
A.1 Image Corruption Methods
We have incorporated 20 image corruption methods (Fig. 8 following [26] 4 531 which can be categorized into 5 categories noise, blur, weather, digital, and extra. We also introduce 5 levels of severity (Fig. 9 to image corruption methods except for blank corruption.
Noise There are 4 different noises used in this study, namely Impulse, Gaussian, Shot, and Speckle. Impulse noise mimics the corruption caused by bit errors by introducing a mixture of salt and pepper noise, with varying intensities such 0.03, 0.06, 0.09, 0.17, and 0.27. Gaussian noise simulates corruptions due to low-lighting conditions by pixel value normalization and normal noise addition. The intensity of this noise is scaled according severity, with values of 0.08, 0.12, 0.18, 0.26, and 0.38. Shot noise simulates electronic noise caused by discrete light and is also called Poisson noise. Speckle noise is additive noise where larger noise will be added if the original pixel value is larger.
Blur Zoom blur is observed when the camera swiftly moves toward an object. It blurs toward the center of the frame. Defocus blur occurs when the image is out of focus and its severity is defined by the disk radius convolved over the image ranging from (3, 0.1), (4, 0.5), (6, 0.5), (8, 0.5). (10, 0.5). Motion blur occurs when the camera is moving quickly. The blurring effect is created by kernel with different radius and sigma ranging from (10, 3). (15, 5). (15, 8). (15, 12), (20, 15). Frosted Glass blur occurs when the glass of windows panels frosts. Gaussian blur generates blurred pixels by weighted average of its neighbors. The farther the neighbors are, the lower the weight in the average is.
Digital JPEG is lossy image compression format that converts the original picture to JPEG format with quality ranging from 25, 18, 15, 10, 7 given different severity. Contrast simulates corruptions caused by the lighting conditions and object's color. Elastic stretches small image regions for stretching effects. Spatter appears when the lens is occluded by rain or mud. Pixelate transforms the original images into a small number of large pixels.
Weather Snow mimics the visual obstruction caused by precipitation. Frost occurs when lenses are covered by crystals. Fog obscures the object and is rendered by the diamond- square algorithm. Brighiness adds bright effect to the image simulating daylight.
Extra we introduce new corruption method called blank that sets all pixel values to 255, i.e. turning the original image into blank picture.
A.2 Text Corruption Methods
We have included 35 text corruptions methods following 72 53
which corrupt on 3 levels (c.g. character, word, and sentence level) and can be categorized into 3 main categories, 6 sub-categories.
Character Modification. Character modification simulates common mistakes during typing and corrupts the text on a character level and contains 9 methods, namely OCR, Punct, Typos, Keyboard, Spelling Error, char random insert, char random replace, char random swap, char random delete. OCR replaces character based on common Optical Character Recognition (OCR) errors. Keyboard substitutes original characters based on keyboard distance.
Text Style Transformation. Methods in the text style category modify the text on a sentence level and transform the sentence style to target one, such turning the original sentence to passive, formal, causual, to double negative, and changing the tense.
Text Addition. Text addition inserts additional words to the original text. InsertAdv adds adverb before each verb. Appendlrr adds irrelevant phrases to the original texts. Random Insert randomly inserts token [UNK] to the original texts.
15

---

Impulse Noise
Gaussian Noise
Shot Noise
Speckle Noise
Zoom Blur
Defocus Blur
Motion Blur
Glass
Gaussian Blur
JPEG Compression
Contrust
Elustic
Pixelate
Snow
Frost
Fog
Brightness
Spatter
Saturate
Figure 8: We introduce corruption methods to image data following the methods in (26 Except for the blank corruption, each type of corruption has five levels of severity. In total, there are 96 different corruptions, as shown Appendix
Original Image
Zoom Blur
Severity
Severity
Original text: Does the grass below the zebra appear be green and short?
Keyboard: Doeq the belo! the zebra agpear be green amd short/
OCR: Does the glass below the zobla appear to de green and shorl? insert: Does [UNK] the [UNK] gruss below [UNK] zebra appear to green short?
Figure 9: Examples of image and text corruptions, The top row shows an original image from GQA and images corrupted by zoom blur with levels of severity. The second row presents text corruptions on the original texts where the red sign indicates the corrupted parts. More examples are shown in supplementary material.
Text Drop based on POS. Text drop transforms the text on a word level based on POS tagging There are methods dropping nouns (DropNN), verbs (DropVB), dropping both nouns and verbs (Drop VB & NN), dropping random nous and verbs (Drop Rand NN. Drop Rand VB), keeping only nouns (Only NN), keeping only verbs (Only VB), keeping both (Only NN & VB).
Text Drop based on Position. According to the word position, there are methods of dropping the first (Drop First), dropping the last (Drop Last), dropping both the first and the last (Drop First and Last), shuffling word order (Shuffle Order), and randomly removing words (Random Delete).
Text Swap. Under the category text swap, we can replace words randomly or with synonyms by word embedding (SwapSyn Word Embd) and WordNet [49 (SwapSyn WordNer). We also deploy back translation (Back Trans) from [72], which first translates English into French and then translates it back to English. Randwom Swap randomly swaps words position.
16

---

Table Image Corruption Methods
Category
Corruption Method Severity
Impulse
5
Gaussian
5
Noise
Shot
Speckle
Zoom
Defocus
Blur
Motion
5
Frosted Glass
5
Gaussian Blur
JPEG
5
Contrast
Elastic
Digital
Saturate
5
Spatter
Pixelate
Snow
5
Frost
Weather
Fog
5
Brightness
5
Extra
Blank
Category
20 Methods
96 Severity
In addition the above corruptions, Qiu al. proposed in [ that should ensure the corrupted text has the same semantics as the original one to make sure the image-text pairs remain meaningful. We follow this setting and use the same fidelity guarantee mechanism as [53].
B Model Implementations
B.1 CLIP-BARI/TS
CLIP-BART/TS 62 combination CLIP and BART/T5, follows VL-T5 (7] which unifies VL tasks to text- generation problem. Specifically, given pair of an image x and a sentence z e.g. picture and corresponding question texts in VQA, as the input the model, CLIP-BART/TS aims to maximize the agreement between the prediction and text label of M tokens y (V1, 3/...., language model, parameterized by 04, Visual
The primary generative model an encoder-decodd representation from input images is extracted from a CLIP and visual projection layer, parameterized by 0 and VL -L respectively. The concatenation of visual representation and sentence representation is fed into the encoder-decoder language model. The training goal is to minimize the cross-entropy loss 66]:
I (x. x² y: 0: 0" 0V--))
CE (for (WV-LL xx0),0 .2)
(1)
M
Σ Di log (Sor VL x)))
where fo means a function parameterized by 8, x V- is the projected visual representation.
The unified structure is beneficial to multi-task training where universal dataset D from N VL datasets is constructed and used train the VI. model. Under such a scenario, the parameters are optimized by minimizing the averaging loss D [62]:
17

---

Table Text Corruption Methods
Category
Sub Category
Level
Corruption Method
Severity
character
OCR
character
Punct
character
Typos
character
Keyboard
Char Modification
character
Spelling Error
character
char random insert
character
char random replace
character
char random swap char random delete
character
Natural
sentence
Passive
sentence
Tense
sentence
Formal
Text Style
sentence
Casual
sentence
Active
sentence
Double Neg
word word
InsertAdv
Text Addition
Appendlrr
word
Random Insert
word
Drop NN
word
Drop Rand NN
word
DropVB
word
Drop VB NN
Drop Text based on POS tag
word
Only NN
word
Only VB
Synthetic
word
Only NN & VB
word
Drop Rand VB
word
Drop First
word
Drop Last
Positional Drop
word
Drop First and Last
sent
Shufflc Order
word
Random Delete
word
SwapSyn Word Embd
word
SwapSyn WordNet Back Trans
Machine
Text Swap
sentence word
Random Swap
35 methods
levels of severity
C (D; 0%, 0r ,0
Σ
(x S
yy:0 0L
,0 -V-L)
D
(ar La WDED
B.2 Adaptation Methods
Full fine-tuning directly updates the whole 0 on D and becomes prohibitive due to the rapidly growing model size. For instance, simply loading a GPT-3 language model with 175B parameters as the VL model backbone would require 700GB of memory Therefore, the following more efficient adaptation methods are developed to achieve comparable performance while optimizing only a few parameters.
Prompt-based adaptation modifies the input x to the model F by either concatenating trainable prefix (Soft Prompt) (36] or manually designed prefix [ For the given input x = (x1,..., with n tokens, the pre-trained model will first form an embedding matrix X E 1gaxd where d is the
(175 x 10º) 4(bytes) 1 700GB
18

---

Table 5: We deploy eleven distinct adaptation methods in total.
Type
Method
Full Fine-tuning
Full Fine-tuning
Single Adapter
Adapter
20
Half-shared Adapters
Multiple Adapters
Single Compacter
Compacter
33
Multiple Compacter
Hyperformer
46
Hyperformer
Single LoRA
LoRA 30
Multiple LoRA
Single Prompt
Soft Prompt
36
Multiple Prompts
dimension of the embedding space. P E Rexd where p is the length of the prompt. Next, P is concatenated with the original embedded input X to form new single matrix defined as [P; X| E IR (dd+n))xd_ During adaptation, the model is trained to maximize the probability of the desired output while only updating P.
36 are then represented as learnable parameter
LoRA 30 also freezes the pre-trained model parameters 0, but it utilizes low-rank decomposition matrices to update gradients. For an intermediate model parameter 6g E Rdxk which can be the parameters from one self- attention module in the transformer architecture, its update is represented by a low-rank decomposition as shown in Formula 3 6g is frozen, whereas B and contain trainable parameters while adapting.
0u A0 = Ou + BA, E A RRRK r < (d).
Adapter- bdapter-based adaptation inserts small modules between parameters in 0 and modifies the interme- diate learning process. Variants of adapter-based methods differ in the insertion manner. Adapters [29] consists of a pair of downsampling and upsampling layers and residual connection. Suppose the original input to an intermediate layer 0g in model θ is Xo E IRdoo adapters insert downsampling layer @ dodo and an upsampling layer e E d x where do. dj are dimensions of the hidden embeddings. The output after injecting adapters defined in Formula where (-)) is an activation
function such as GELU 22
h fou (a (foz (xoo)) + Xg
based on the mechanism of adapters 29 but it utilizes matrix decomposition and parameter sharing to further reduce redundant parameters in adapters. It introduces parameterized hypercomplex mulriplication (PHM) layers (Formula which decompose the layer the adapter by Kronecker products. Compacter also shares the parameter of A, across all layers and decomposes B, even further with low-rank decomposition. However, as found in [2]. such sharing and further decomposition severely decreases the VL performance. In our study, we only use PHM layers.
Compacter 33
0D
Σ A 2 Bi, Ai E Rick B, x
Hyperformer [46] also aims to reduce redundant parameters in adapters. It relies on hyper- network that is shared across tasks to generate the weights in adapters given a task index and a layer index. The hyper-network maintains learnable embeddings for each task and each layer. For NT tasks and Ny, layers, the dy-dimensional embeddings can be denoted as l,...., LNy E Rd:; 11....,IN: ,NN, RRd The Hyperformer consists of task projector H E R(de+dd)xd, and a hyper- network 0 E IRd, x(dxxd1 +di xdo), and generates an adapter's s weights the 7th layer for the jth task following Formula 6
[00, [0".00 0"1 for (for ([t,,4))) 41))
(6)
19

---

Adaptation shared over tasks is inspired by Hyperformer and proposed in [62] which aims to exploit similar information shared across multiple tasks and to reduce redundant parameters. For vanilla adapters in a multi-task setting with Nr tasks, the collection of all inserted adapter modules can be denoted as = {@D, @²} where 6 (e") stands for the subset of downsampling (upsampling) layers in adapters. The straightforward application is to train the adaptation modules per task, so we have independent !O," 657 for the zth task, dubbed as Multiple Adapters. The same goes for other adaptation methods. By training one prompt, low-rank weights, and compacter layer for each task, we obtain Multiple Prompts, Multiple LoRA, and Multiple Compacter respectively, We can also train only one set of adapter layers for all tasks, and we have Single Adapter where 6 0
and @ 0y i a j. Also, if we use the same prompts, low-rank weights, and compacter layers, we have Single Prompt, Single LoRA, and Single Compacter respectively. For adapters, we can make parts of the weights shareable by making ? = 0 i + j and. In this way, the task-specific information of 7th can still be learned by the rest @. Such sharing mechanism is called half-shared adapter. Lastly, Hyperformer already shares information from multiple tasks and therefore does not have such extensions.
B.3 Training and Eyaluation
We follow the same experimental and hyperparameter settings [62 vision encoder that takes the resized 224 x 224 images as input. The 7 x 7 gird features in the last convolutional layer are extracted visual features and are downsampled to 6 x 6 by adaptive maximum pooling. BART base and Tu are both studied in this work as encoder-decoder language models. During training, AdamW is the optimizer along with linear decay scheduler. Models are trained for 20 epochs, and the learning rate increases from 0 to the highest learning rate in the first 2 epochs. Training batch sizes are set 500 and 250 for CLIP-BART and CLIP-T5 respectively. Models are trained in multi-task setting where the training dataset includes all training split from 4 VL datasets. After training, we first evaluate the clean performance. Specifically, accuracy on the Karpathy-test test split is evaluated for VQAv2. For GQA, accuracy on the test-dev split is evaluated, and accuracy on the test-P split used for NIVR2. In image captioning, we use CIDEr [6) on the Karpathy-test split. Then, we evaluate the corrupted performance on the corresponding corrupted test split from each dataset and calculate the relative robustness. All the training and evaluations are conducted on LRZ AI Systems at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities.
FLIP-ResNetlON is the
C Limitations
The limitations of this work mainly include 1) the adoption of simulated noise instead of real- real-world data due to the difficulty of obtaining real-world corruptions, 2) the analysis is on a limited number of multimodal models due to the availability of usable code, model weights, and the massive experiments, 3) the experiments require heavy GPU usage to pre-train vision-language models.
D Licensing
All the models and datasets used in this study are publicly available. The code for model VL- T5(BART) and (5(BAR)( have the MIT License and the code for Hyperformer and Soft Prompt has the Apache 2.0 License. Our benchmark datasets are built upon 4 publicly available VI. datasets, namely VQAv2 200, GQA (3I] NLVR2 [l], and MSCOCO [5] We also publicize our benchmark datasets, corruption codes, and benchmark codes under MIT License.
and
E
Impact
From our understanding, there are negative societal impacts of study. This study aims to investigate the robustness of multimodal adaptation methods and facilitate future research and real-world applications.
20


Reviews:
[{"review_id": "wXkhQt2RNY", "reviewer": "Reviewer_Npuc", "summary": "", "questions": "", "limitations": "The limitations are present in the supplementary material and (with the limitations listed above) looks complete.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The analysis seems relatively thorough. I'm convinced by the large array of tasks.\n2. The tasks of choice and the models chosen (modern V+L models) are relevant to the community. These results demonstrate robustness on axes that people care about (e.g. robustness to typos).\n3. Considering corruptions on both the vision and language sides are interesting and their connection to efficient fine-tuning is quite useful.\n\nIn summary, given the proliferation of large models and the difficulties in training one from scratch, the relevance of adaptation methods for such models is very high. Robustness is a very useful quality to study in this domain, and I believe this paper is an important step towards this.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "CNknlJj7yY", "reviewer": "Reviewer_Ssg9", "summary": "", "questions": "", "limitations": "The authors include a section discussing limitations in the appendix. The authors acknowledge my concern number 3 from above.\n\nThe authors claim there are no major concerns about societal impact, which I tend to agree with.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper addresses a very important topic today: the robustness of visual-language model to perturbations. This topic will only grow in importance as these models become more and more ubiquitous and also larger and larger.\n2. According to the authors—and as far as I know this is true—no other work addresses the robustness of this category of methods: methods that adapt pretrained visual-language models per task.\n3. The paper is very well written and easy to follow. This is one of the biggest strengths of this work.\n4. The paper covers a large number of obfuscation attacks, both on the text side and the image side.\n5. The paper draws several insightful conclusions about the robustness of different adaptation methods to both image and text corruption that I think future work will find relevant. These are supported by an extensive amount of experiments.\n6. The authors release code for reproducing the experiments and recomputing the image and text obfuscations mentioned in this paper.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zh7xNfyCpt", "reviewer": "Reviewer_UNok", "summary": "", "questions": "", "limitations": "1. The human study is missing.\n2. The attacking strategy is limited. More adversarial attacks can be considered.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper presents a novel benchmark for the robustness of adaptation methods.\n2. The benchmark is comprehensive, covering different models, different types of corruption, and different datasets.\n3. The paper is well-written and easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "DLk3rKKQ1b", "reviewer": "Reviewer_rME1", "summary": "", "questions": "", "limitations": "I think there is no potential negative societal impact in the submission.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1)\tThis work is the first to discuss the multimodal robustness in the VL adaptation task.\n\n(2)\tAbundant experiments including 96 visual and 87 textual corruptions are provided.\n\n(3)\tThe conclusions are intriguing.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JKCPg6TZkn", "reviewer": "Reviewer_gdTB", "summary": "", "questions": "", "limitations": "(1) It may be unclear whether the selected baseline model is enough to support the claim of the findings. \n\n(2) Most of the perturbation methods are adopted from existing works directly, the novelty contribution seems trivial.\n\n(3) The evaluation metric is adopted from previous works as well, no new evaluation metric is proposed.\n", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The paper is well-written and easy to follow.\n\n(2) The paper also provided findings based on the benchmark results, like adaptation methods are more\nsensitive to text corruptions than visual corruptions, which could be useful for future research.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper aims to evaluate the robustness of adaptation methods against distribution shifts. Authors proposed both vision and language datasets and then conducted some analysis. The findings are interesting: the sensitivity of adaptation methods, the benefit of full training, and the increment of parameter size and data size, etc. 

AC read the paper carefully. Together with the review comments, the paper has the following strength:
- Important and timely problem. The adaptation of vision language models in the era of large language models is important. This benchmark could be of interest and significance to the community.
- Comprehensive and insightful analysis. The findings are of immense importance to the community to further study the factors that affects the adaptation performance of downstream tasks.
- The paper draws several insightful conclusions about the robustness of different adaptation methods to both image and text corruption that I think future work will find relevant. These are supported by an extensive number of experiments.
- The authors release code for reproducing the experiments and recomputing the image and text obfuscations mentioned in this paper.

There are also weaknesses as proposed by other reviewers, including statistical tests, related work discussions, etc. Authors tried their best to reply, and their response was acknowledged by AC, but it is a bit unlucky that not all reviewers responded, thus the score is not changed. The paper received mixed ratings (4-7) with an average score of 5.8, which is obviously low. AC does not only look at the scores, but the review comment and the author response. In the meantime, AC has also noticed that some reviewers posted unprofessional feedback with strong focus on discussion of work that is not even related, plus not responding to the authors. After checking the paper details carefully, AC finds that some reviewers are not responsible, and their review quality is not high. The final decision is acceptance. However, acceptance is not the terminal. AC still recommends that the authors carefully read all the reviews and make the paper better.
Decision: Accept (Poster)