- ViLT. We first select ViLT [1] as the encoder-only VL model because of its succinct structure and prominent performance on multiple downstream tasks. Given an input image  $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$  and a sentence  $\mathbf{T}$ , ViLT yields  $M$  image tokens using a linear transformation on the flattened image patches, where each token is a 1D vector and  $M = \frac{HW}{P^2}$  for a given patch resolution  $(P, P)$ . Word tokens are encoded through a Byte-Pair Encoder (BPE) [44] and a word-vector linear projection. Then, tokens of two modalities and a special learnable token  $\langle cls \rangle$  are concatenated. By attending visual and text tokens and a special token  $\langle cls \rangle$  in a Transformer encoder with twelve layers, the output feature from the  $\langle cls \rangle$  token is fed into a task-specific classification head for the final output. Taking the VQA task as an example, the VQA classifier adopts a linear layer to output a vector with  $H_s$  elements, where  $H_s$  is the number of all possible choices in the closed answer set of the VQA task. The final output is obtained through the element with the highest response in the vector.
- BLIP. The BLIP model also adopts an encoder-only structure. Specifically, BLIP first encodes an image through a twelve-layer vision transformer ViT-B/16 [42], and generates word token embeddings through the BPE and a linear layer. Next, the image features and word token embeddings are then fed into a multimodal encoder. The structure of the multimodal encoder is the same as a twelve-layer transformer decoder, where each layer contains a self-attention module, a cross-attention module and a feed-forward module. In each layer, the multimodal encoder first accepts the word token embeddings as input and attends to them through the self-attention module. Then, the updated embedding will fuse with image features through the cross-attention module. Finally, the output from the multimodal encoder will be fed into different projection heads for downstream tasks, just like the ViLT model.
- Unitab. Unitab adopts an encoder-decoder framework. It first embeds text  $\mathbf{T}$  via RoBERT $_a$  [45] and flats features after encoding image  $\mathbf{I}$  through ResNet [43]. The attached visual and text token features are then fed into a standard Transformer network [46] with six encoder layers and six decoder layers. Finally, the sequence predictions  $[\langle ans_1\rangle, \langle ans_2\rangle, \dots, \langle end\rangle]$  are obtained autoregressively through a projection head. The network stops regressing when an end token  $\langle end\rangle$  appears. For different tasks, the output tokens may come from different pre-defined vocabularies. Given the REC task as an example, four tokens  $[( \langle loc x_1 \rangle, \langle loc x_2 \rangle), (\langle loc x_3 \rangle, \langle loc x_4 \rangle)]$  will be selected from the location vocabulary, which forms the coordinate of a bounding box. As a result, these models can handle both text and grounding tasks.
- OFA. OFA also adopts an encoder-decoder structure. Different from Unitab, it adopts the BPE to encode text and extends the linguistic vocabulary by adding image quantization tokens [58]  $\langle img\rangle$  for synthesis tasks. Note that the main difference between OFA and Unitab lies in their pre-training and fine-tuning strategies rather than the model structure. For example, in the pre-training process, Unitab focuses on learning alignments between predicted words and boxes through grounding tasks, while OFA captures more general representations through multi-task joint training that includes both single-modal and multimodal tasks. Overall, OFA outperforms Unitab in terms of performance improvement.