SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CLIP-BART/T5 [62], a combination of CLIP and BART/T5, follows VL-T5 [7] which unifies VL tasks to a text-generation problem. Specifically, given a pair of an image  $x^{I}$  and a sentence  $x^{S}$ , e.g., a picture and corresponding question texts in VQA, as the input to the model, CLIP-BART/T5 aims to maximize the agreement between the prediction and text label of  $M$  tokens  $\mathbf{y} = (y_{1}, y_{2}, \ldots, y_{M})$ . The primary generative model is an encoder-decoder language model, parameterized by  $\theta^{L}$ . Visual representation from input images is extracted from a CLIP and a visual projection layer, parameterized by  $\theta^{V}$  and  $\theta^{V \rightarrow L}$  respectively. The concatenation of visual representation and sentence representation is fed into the encoder-decoder language model. The training goal is to minimize the cross-entropy loss [62]:
$$
\begin{array}{l} l \left(\boldsymbol {x} ^ {I}, \boldsymbol {x} ^ {S}, \boldsymbol {y}; \theta^ {L}, \theta^ {V}, \theta^ {V \rightarrow L}\right) \\ = \operatorname {C E} _ {M} \left(f _ {\theta^ {L}} \left(\boldsymbol {x} ^ {V \rightarrow L}, \boldsymbol {x} ^ {S}\right), \boldsymbol {y}\right) \tag {1} \\ = - \sum_ {i = 1} ^ {M} y _ {i} \log \left(f _ {\theta^ {L}} \left(\boldsymbol {x} ^ {V \rightarrow L}, \boldsymbol {x} ^ {S}\right) _ {i}\right), \\ \end{array}
$$
where  $f_{\theta}$  means a function parameterized by  $\theta$ ,  $\pmb{x}^{V\rightarrow L}$  is the projected visual representation.
The unified structure is beneficial to multi-task training where a universal dataset  $\mathcal{D}$  from  $N$  VL datasets is constructed and used to train the VL model. Under such a scenario, the parameters are optimized by minimizing the averaging loss on  $\mathcal{D}$  [62]:
Table 4: Text Corruption Methods
Category	Sub Category	Level	Corruption Method	Severity
Natural	Char Modification	character	OCR	5
character	Punct	1
character	Typos	5
character	Keyboard	5
character	Spelling Error	5
character	char random insert	5
character	char random replace	5
character	char random swap	5
character	char random delete	5
Text Style	sentence	Passive	1
sentence	Tense	1
sentence	Formal	1
sentence	Casual	1
sentence	Active	1
sentence	Double Neg	1
Text Addition	word	InsertAdv	1
word	Appendlrr	1
word	Random Insert	5
Synthetic	Drop Text based on POS tag	word	Drop NN	1
word	Drop Rand NN	1
word	DropVB	1
word	Drop VB & NN	1
word	Only NN	1
word	Only VB	1
word	Only NN & VB	1
word	Drop Rand VB	1
Positional Drop	word	Drop First	1
word	Drop Last	1
word	Drop First and Last	1
sent	Shuffle Order	1
word	Random Delete	5
Machine	Text Swap	word	SwapSyn Word Embd	5
word	SwapSyn WordNet	5
sentence	Back Trans	1
word	Random Swap	5
35 methods	87 levels of severity
$$
\begin{array}{l} \mathcal {L} \left(\mathcal {D}; \theta^ {L}, \theta^ {V}, \theta^ {V \rightarrow L}\right) \\ = \frac {1}{| \mathcal {D} |} \sum_ {\left(\boldsymbol {x} ^ {I}, \boldsymbol {x} ^ {S}, \boldsymbol {y}\right) \in \mathcal {D}} l \left(\boldsymbol {x} ^ {I}, \boldsymbol {x} ^ {S}, \boldsymbol {y}; \theta^ {L}, \theta^ {V}, \theta^ {V \rightarrow L}\right) \tag {2} \\ \end{array}
$$
B.2 Adaptation Methods
Full fine-tuning directly updates the whole  $\theta$  on  $\mathcal{D}$  and becomes prohibitive due to the rapidly growing model size. For instance, simply loading a GPT-3 language model with 175B parameters as the VL model backbone would require 700GB of memory<sup>3</sup>. Therefore, the following more efficient adaptation methods are developed to achieve comparable performance while optimizing only a few parameters.
Prompt-based adaptation modifies the input  $\mathbf{x}$  to the model  $F$  by either concatenating a trainable prefix (Soft Prompt) [36] or a manually designed prefix [3]. For the given input  $\mathbf{x} = \{x_1, \ldots, x_n\}$  with  $n$  tokens, the pre-trained model will first form an embedding matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$  where  $d$  is the
$^{3}(175\times 10^{9})\times 4(\mathrm{bytes})\times 10^{-9} = 700\mathrm{GB}$
Table 5: We deploy eleven distinct adaptation methods in total.
Type	Method
Full Fine-tuning	Full Fine-tuning
Adapter [29]	Single Adapter
Half-shared Adapters
Multiple Adapters
Compacter [33]	Single Compacter
Multiple Compacter
Hyperformer [46]	Hyperformer
LoRA [30]	Single LoRA
Multiple LoRA
Soft Prompt [36]	Single Prompt
Multiple Prompts
dimension of the embedding space. Soft-prompts [36] are then represented as a learnable parameter  $\mathbf{P} \in \mathbb{R}^{p \times d}$  where  $p$  is the length of the prompt. Next,  $\mathbf{P}$  is concatenated with the original embedded input  $\mathbf{X}$  to form a new single matrix defined as  $[\mathbf{P}; \mathbf{X}] \in \mathbb{R}^{(p + n) \times d}$ . During adaptation, the model is trained to maximize the probability of the desired output while only updating  $\mathbf{P}$ .
LoRA [30] also freezes the pre-trained model parameters  $\theta$ , but it utilizes low-rank decomposition matrices to update gradients. For an intermediate model parameter  $\theta_0 \in \mathbb{R}^{d \times k}$ , which can be the parameters from one self-attention module in the transformer architecture, its update is represented by a low-rank decomposition as shown in Formula 3.  $\theta_0$  is frozen, whereas  $B$  and  $A$  contain trainable parameters while adapting.
$$
\theta_ {0} + \Delta \theta = \theta_ {0} + B A, B \in \mathbb {R} ^ {d \times r}, A \in \mathbb {R} ^ {r \times k}, r \ll \min  (d, k). \tag {3}
$$
Adapter-based adaptation inserts small modules between parameters in  $\theta$  and modifies the intermediate learning process. Variants of adapter-based methods differ in the insertion manner. Adapters[29] consists of a pair of downsampling and upsampling layers and a residual connection. Suppose the original input to an intermediate layer  $\theta_0$  in model  $\theta$  is  $\mathbf{x}_0\in \mathbb{R}^{d_0}$ , adapters insert a downsampling layer  $\theta^D\in \mathbb{R}^{d_0\times d_1}$  and an upsampling layer  $\theta^U\in \mathbb{R}^{d_1\times d_0}$  where  $d_0,d_1$  are dimensions of the hidden embeddings. The output after injecting adapters is defined in Formula 4 where  $\sigma (\cdot)$  is an activation function such as GELU [27].
$$
h = f _ {\theta^ {U}} \left(\sigma \left(f _ {\theta^ {D}} \left(\mathbf {x} _ {0}\right)\right)\right) + \mathbf {x} _ {0} \tag {4}
$$
Compacter [33] is based on the mechanism of adapters [29], but it utilizes matrix decomposition and parameter sharing to further reduce redundant parameters in adapters. It introduces parameterized hypercomplex multiplication (PHM) layers (Formula 5), which decompose the layer in the adapter by Kronecker products. Compacter also shares the parameter of  $A_{i}$  across all layers and decomposes  $B_{i}$  even further with low-rank decomposition. However, as found in [62], such sharing and further decomposition severely decreases the VL performance. In our study, we only use PHM layers.
$$
\theta^ {D} = \sum_ {i = 1} ^ {k} A _ {i} \otimes B _ {i}, A _ {i} \in \mathbb {R} ^ {k \times k}, B _ {i} \in \mathbb {R} ^ {\frac {d _ {0}}{k} \times \frac {d _ {1}}{k}} \tag {5}
$$
Hyperformer [46] also aims to reduce redundant parameters in adapters. It relies on a hypernetwork that is shared across tasks to generate the weights in adapters given a task index and a layer index. The hyper-network maintains learnable embeddings for each task and each layer. For  $N_{T}$  tasks and  $N_{L}$  layers, the  $d_{e}$ -dimensional embeddings can be denoted as  $\pmb{t}_{1},\dots,\pmb{t}_{N_{T}}\in \mathbb{R}^{d_{e}}$ ;  $l_{1},\ldots ,l_{N_{L}}\in \mathbb{R}^{d_{e}}$ . The Hyperformer consists of a task projector  $\theta^T\in \mathbb{R}^{(d_e + d_e)\times d_p}$  and a hypernetwork  $\theta^H\in \mathbb{R}^{d_p\times (d_0\times d_1 + d_1\times d_0)}$ , and generates an adapter's weights in the  $i^{th}$  layer for the  $j^{th}$  task following Formula 6.
$$
\left[ \theta^ {D}, \theta^ {U} \right] = f _ {\theta^ {H}} \left(f _ {\theta^ {T}} \left(\left[ \boldsymbol {t} _ {j}, \boldsymbol {l} _ {i} \right]\right)\right) \tag {6}
$$
Adaptation shared over tasks is inspired by Hyperformer and proposed in [62] which aims to exploit similar information shared across multiple tasks and to reduce redundant parameters. For vanilla adapters in a multi-task setting with  $N_{T}$  tasks, the collection of all inserted adapter modules can be denoted as  $\Theta = \{\Theta^{D},\Theta^{U}\}$  where  $\Theta^D (\Theta^U)$  stands for the subset of downsampling (upsampling) layers in adapters. The straightforward application is to train the adaptation modules per task, so we have independent  $\{\Theta_i^D,\Theta_i^U\}$  for the  $i^{th}$  task, dubbed as Multiple Adapters. The same goes for other adaptation methods. By training one prompt, low-rank weights, and compacter layer for each task, we obtain Multiple Prompts, Multiple LoRA, and Multiple Compacter respectively. We can also train only one set of adapter layers for all tasks, and we have Single Adapter where  $\Theta_i^D = \Theta_j^D$  and  $\Theta_i^U = \Theta_j^U$ ,  $i\neq j$ . Also, if we use the same prompts, low-rank weights, and compacter layers, we have Single Prompt, Single LoRA, and Single Compacter respectively. For adapters, we can make parts of the weights shareable by making  $\Theta_i^D = \Theta_j^U$ ,  $i\neq j$  and. In this way, the task-specific information of  $i^{th}$  can still be learned by the rest  $\Theta_i^U$ . Such sharing mechanism is called half-shared adapter. Lastly, Hyperformer already shares information from multiple tasks and therefore does not have such extensions.
B.3 Training and Evaluation
We follow the same experimental and hyperparameter settings as [62]. CLIP-ResNet101 is the vision encoder that takes the resized  $224 \times 224$  images as input. The  $7 \times 7$  gird features in the last convolutional layer are extracted as visual features and are downsampled to  $6 \times 6$  by adaptive maximum pooling. BART<sub>base</sub> and T5<sub>base</sub> are both studied in this work as encoder-decoder language models. During training, AdamW is the optimizer along with a linear decay scheduler. Models are trained for 20 epochs, and the learning rate increases from 0 to the highest learning rate in the first 2 epochs. Training batch sizes are set as 500 and 250 for CLIP-BART and CLIP-T5 respectively. Models are trained in a multi-task setting where the training dataset includes all training split from 4 VL datasets. After training, we first evaluate the clean performance. Specifically, accuracy on the Karpathy-test split is evaluated for VQAv2. For GQA, accuracy on the test-dev split is evaluated, and accuracy on the test-P split is used for NLVR<sup>2</sup>. In image captioning, we use CIDEr [69] on the Karpathy-test split. Then, we evaluate the corrupted performance on the corresponding corrupted test split from each dataset and calculate the relative robustness. All the training and evaluations are conducted on LRZ AI Systems at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities.
C Limitations
The limitations of this work mainly include 1) the adoption of simulated noise instead of real-world data due to the difficulty of obtaining real-world corruptions, 2) the analysis is on a limited number of multimodal models due to the availability of usable code, model weights, and the massive experiments, 3) the experiments require heavy GPU usage to pre-train vision-language models.
D Licensing
All the models and datasets used in this study are publicly available. The code for model VL-T5(BART) and CLIP-T5(BART) have the MIT License and the code for Hyperformer and Soft Prompt has the Apache 2.0 License. Our benchmark datasets are built upon 4 well-known and publicly available VL datasets, namely VQAv2 [20], GQA [31], NLVR² [61], and MSCOCO [5]. We also publicize our benchmark datasets, corruption codes, and benchmark codes under MIT License.
E Impact
From our understanding, there are no negative societal impacts of our study. This study aims to investigate the robustness of multimodal adaptation methods and facilitate future research and real-world applications.