Bridging Vision and Language Spaces with Assignment Prediction
Authors: Jungin Park
Jiyoung Lee
Kwanghoon Sohn
Venue: ICLR 2024 poster (2024)
Keywords: Multimodal learning
vision-language tasks
frozen LLMs
optimal transport
assignment prediction
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This paper presents to bridge frozen image encoders and large language models (LLMs) for grounding LLMs to images.

Abstract: This paper introduces VLAP, a novel approach that bridges pretrained vision models and large language models (LLMs) to make frozen LLMs understand the visual world. VLAP transforms the embedding space of pretrained vision models into the LLMs' word embedding space using a single linear layer for efficient and general-purpose visual and language understanding. Specifically, we harness well-established word embeddings to bridge two modality embedding spaces. The visual and text representations are simultaneously assigned to a set of word embeddings within pretrained LLMs by formulating the assigning procedure as an optimal transport problem. We predict the assignment of one modality from the representation of another modality data, enforcing consistent assignments for paired multimodal data. This allows vision and language representations to contain the same information, grounding the frozen LLMs' word embedding space in visual data. Moreover, a robust semantic taxonomy of LLMs can be preserved with visual data since the LLMs interpret and reason linguistic information from correlations between word embeddings. Experimental results show that VLAP achieves substantial improvements over the previous linear transformation-based approaches across a range of vision-language tasks, including image captioning, visual question answering, and cross-modal retrieval. We also demonstrate the learned visual representations hold a semantic taxonomy of LLMs, making visual semantic arithmetic possible.

Published as a conference paper at ICLR 2024
BRIDGING VISION AND LANGUAGE SPACES WITH ASSIGNMENT PREDICTION
Jungin Park Jiyoung Lee Kwanghoon Sohn 1,3+
Yonsei University "NAVER Al Lab {newrun, kuschn)eyonsel.. ac.
"Korea Institute of Science and Technology (KIST)
1ee. www.vjevverrrr..oo com
ABSTRACT
This paper introduces VLAP, a novel approach that bridges pretrained vision mod- els and large language models (LLMs) to make frozen LLMss understand the vi- sual world. VLAP transforms the embedding space of pretrained vision models into the LLMs' word embedding space using single linear layer for efficient and general-purpose visual and language understanding. Specifically, we har- ness well-established word embeddings bridge modality embedding spaces. The visual and text representations are simultaneously assigned to a set of word embeddings within pretrained LLMs by formulating the assigning procedure an optimal transport problem. We predict the assignment of one modality from the representation of another modality data, enforeing consistent assignments for paired multimodal data. This allows vision and language representations to con- tain the same information, grounding the frozen L.LMs' word embedding space in visual data. Moreover, a robust semantic taxonomy of LLMs can be preserved with visual data since the L.MSs interpret and reason linguistic information from correlations between word embeddings. Experimental results show that VLAP achieves substantial improvements over the previous linear transformation-bes approaches across range of vision- vision-language tasks, including image captioning, visual question answering, and cross modal retrieval. We also demonstrate the learned visual representations hold semantic taxonomy of LLMs, making visual semantic arithmetic possible.
INTRODUCTION
Vision-language models (VLMs) have achieved significant progress in demonstrating remarkable transfer and zero-shot capabilities on vision-language downstream tasks (Tan & Bansal 2019 Lu
al. 2019 Chen et al. 2020 Huang al. 2020 Radford al. 202T Jia et al. al. 2021). Most cutting- edge VLMs have been developed with progressively scaled-up up foundation models and datasets.
However, this trend underscores the demand for substantial computational resources and an exten- sive collection of image-text pairs. Concurrently, pretrained unimodal foundation models, such as vision transformers (Dosovitskiy al. 2021 Caron et al. 202T Bao et al.) 2022) and large lan- guage models (LLMs) (Brown al.) 2020: Zhang al.) 2022 Raffel et al. 2020 Touvron et al. 2023 Chiang et al. 2002),, y have been developed with self-
renewing the state-of-the-an
vision and language downstream tasks respectively. This raises the interesting question: Can pretrained unimodal models extend their capabilities beyond the modality of pretraining data?
Recently, intriguing attempts have been made to assemble pretrained vision models and LLMs to
seem VLMs (Mokady et al. [021]] Tsimpoukelli et al., 2021 Alayrac et al.] 2022 Patel & Pavlick
2022 Eichenberg al. 2022 Li et al. 2022 2023 Guo al. 2023: Liu al. 2023a). Most of these attempts suggested ways to train relatively small-scal connection modules while freezing pre-
trained parameters. By doing so, it is possible to efficiently make VLMs that leverage discriminative
visual representations from pretrained vision models, coupled with powerful language modeling and zero-shot capabilities of pretrained LLMs. While the burden of training costs those methods has significantly decreased compared to pretraining VLMs from scratch, they still require high compu- tations (both time and memory), which limits applicability (e.g. Flamingo (Alayrac et al. 2022)
"Corresponding authors.

---

Published as a conference paper at ICLR 2024
dog wearing glasses and
Image Encoder
Linear
LLM Encoder
LLM Decoder
holding book
<Prompt>
Input Image
Image Embedding 7.
Output Text
Q"
A dog wearing glasses and holding book
Assignment Prediction
map
Lcap
+ LLM Encoder
Q
Text Embedding z
Frozen
Input Text
LLM Vocab.
Train
Figure I: Overview of VLAP. We train a single linear layer following two learning objectives: Assignments
prediction to bridge the modality gap between the visual and text representations; and image captioning to yield the generative capability of frozen LLMs.
requires 15 days with 1535 TPUs to train 10.2B parameters, BLIP-2 (Li at. 2023) requires 9 days with 16 A100 GPUs to train 188M parameters).
Linear transformation bbadod approaches (Merullo et al. 2023 Koh ct al. 2023) have facilitated effi- cient modal alignment with cost-efficient linear transformation between vision and language spaces. Merullo et al. (2023) has shown that visual representations can be linearly transformed into LLMs' input embedding soft prompts, demonstrating that well-learned visual and text represen- tations are functionally equivalent. Similarly, Koh et al. (2023) has proposed to generate free-form text interleaved with images by learning additional trainable tokens and linear layers. These linear transformation-base
cross-modal alignments have been achieved by the image captioning objec- tive (Merullo et al., 2023) and an additional retrieval objective (Koh et al. 2203)) which directly compares the representations of image-text pairs and enforces instance discrimination as contrastive learning (Chopra et al. 2005). However, the inherent modality gap between different modality data leads to the inconsistency between the isolated embedding spaces (as their pretraining has been solely on unimodal data) and makes cross-modal alignment for frozen unimodal models difficult with the following challenges: (1) it relies heavily on the amount of linguistic supervision used in the pretraining of vision models, showing limited performance in image-only trained vision models (e.g. CLIP BEiT) (Merullo et al. (023). ). (2) the contrastive objective (such InfoNCE (van den
Oord et al. 2018) in Koh et al. (2023) is insufficient to bridge complex embedding spaces because inherently encourages the existence of the modality gap (Liang et al.) 2022).
In this paper, we propose a novel linear transformation based VLM, which we call VLAP, bridging vision and language models with assignment prediction. Our VLAP learns linear feature map- ping through a novel optimal transport-based assignment prediction that maximizes the similarity between intermediate assignments for image and text data instead of directly comparing their rep- resentations. Specifically, visual and text representations are assigned to the most relevant word embeddings within pretrained LLMs. Since there is no supervision for word assignments for visual information, we cast this assignment procedure as an optimal transport problem. The assignment for one modality is predicted from the other modality representation, making two different modal- ity representations contain the same linguistic information. We exploit the readily available word embedding space, following two main advantages: (1) it does not require additional learnable em- bedding space (e.g. prototypes in Asano et al.] (2020b): Caron et al. (2020)) and (2) it allows visual representations to keep abundant linguistic contextual information and inherit a semantic taxonomy of LLMs. Our main contributions four-fold:
We present VLAP that efficiently bridges pretrained vision models and LLMs VLAP employs optimal transport-based assignment prediction objective to make visual and text representations compatible with each other.
VLAP leverages the word embeddings of pretrained LLMs as a fixed central space for optimal transport. This allows us to casily bridge two pretrained frozen unimodal models by exploiting the fundamental components of L.LMs.
Mapping visual data to LLMs' word embeddings results in learned visual representations that hold a semantic taxonomy of LLMs, facilitating the operation in a textual way for visual data, e.g. visual semantic arithmetic operation.
2

---

Published as a conference paper at ICLR 2024
VLAP substantially outperforms existing linear Transformation-aaeee methods (Merullo ct al. 2023] Koh et al. (022)) in a range of vision-language tasks, while also demonstrating
high computational and memory efficiency.
RELATED WORK
Pretraining vision-languee language models (VLMs) Tan & Bansal 2019 Lu et al. (019); Chen et al. (020);: Huang et al. (200)) Jia et al. (2021); Radford et al. (2021) has received tremendous inter- est thanks to their robust zero-shot capability for vision- language tasks such image captioning, visual question answering, and cross mooda retrieval. Concurrently, in the past few years, large lan- guage models (LLMs) have achieved signifieant success in generating human-like language content with growth in model size and datasets at scale (Devlin et al. 2019; Raaford et al. 2018 2019)
Brown al.) 2020 Raffel et al.) 2020; Chung et al.) 2022). To conjugate this ability of LLMs for vision-laaneeeeeee language rescarch, recent works have attempted to transport the visual embeddings as
prompts of LLMs (Timpoukelli al., [202T] Eichenberg et al. 2022 Mokady al.) [02T]] Alayrac
et al. 2022: Sung et al., [022 Li et al., 2023). They accomplished this by tuning the visual en- coder (Tiipouuelll
al. 22221) 1 introducing additional lightweight modules Moliles et al. (202T);
Alayrac et al. (2022); Sung al. ): |Cohen et al. (0022
Li al. (2023), employing both
approaches (Eichenberg et al. 2022).
Our work is highly related to LLaVA (Liu et al. 2023a) LiMBeR (Merullo et al. 2023), and FRO- MAGe (Koh et al.) 2023). Different from the modular-batt methods, which require relatively a high computational cost, they learned linear mapping to project visual embeddings into the text embedding space. They demonstrated that vision models and LLMs share non-trivial similar in- formation even though two unimodal models are independently trained. However, their learning objectives to directly compare the visual and text representations are restrictive solutions to bridge the modality gap (Liang al. 2022). Unlike the previous methods, compare the intermediate assignments between image and text data by formulating an optimal transport-based assignment pre- diction with word embeddings of LLMs. We demonstrate that the modality gap between pretrained vision models and LLMs can be effectively bridged while preserving high computational efficiency with the proposed assignment prediction.
METHOD
Assignment Prediction
Our primary goal is make pretrained LLMs compre- hend visual inputs with minimum training for general- purpose languan tasks. More precisely, we aim to bridge pretrained vision models and LLMs by learning a linear layer only (i.e., keeping the original parameters frozen) while preserving the representative power of pretrained vision models and the general- ization ability of pretrained LLMs. To this end, proposed a novel linear transformation-bart VLAP: bridging vision and language models with as- signment prediction. The key component to bridge pretrained vision and language models mapping in- formation between image and text representations to make frozen LLMs interpret visual inputs as linguistic components. The previous methods (Merullo et al, 2023 Koh et al.) 2023) linearly transform the visual representation from pretrained vision model into L.MS'' embedding space, and directly compare representations of paired multimodal data. In contrast, we resolve the linear cross-modal alignment using the assignment prediction ob- jective (Asano et al., 2020bb Caron et al., 2020) that enforces consistent assignments to multimodal data. VLAP learns linear layer by optimizing two objectives simultaneously: (1) optimal transport- based assignment prediction to enforce the visual and text representations contain the equivalent information and (2) image-to-tee generation to incorporate the visual representation into the gener- ative ability of LLMs. This section introduces each component in detail, and the overview of VLAP is shown in Figure
Z
Q"
Image Embed.
Prediction
00
Prediction
Word Embed.
Z!
Q
OT
Text Embed.
2 Assigned Word Embeddings
method
Figure 2: Assignment prediction. The modal- ity gap can be relaxed by predicting the word assignments of one modality from the other modality representations.
3

---

Published as a conference paper at ICLR 2024
3.1 ASSIGNMENT PREDICTION
We assign visual and text representations into word embeddings by defining optimal transport as if each data is clustered into the word embedding space of pretrained LLMs. The proposed assignment prediction objective aims predict each other's assignments. We notice that recent works for the assignment prediction problem formulate set of learnable embeddings (i.e., prototypes) to cluster the given data (Caron al., 2020) Xu et al. 2020 Liu et al.] 2023b Duan et al. 2022 requiring additional memory and gradient computation. Contrary to this, VLAP utilizes readily available word embeddings of LLMs fixed central space into which visual representations mapped. Therefore, our training process is more stable and memory-efficient.
Visual and text representations from pretrained models. Given set of images, we ex- tract the last hidden state from pretrained image encoder as the visual representations X (XI,..., 3B), where the representation for each image contains du-diimensionll N, feature vec- tors. i.e., X, [x)_... Xn" N² E IR,, Similarly, we feed set of B captions corresponding to the images into pretrained LLM to obtain the text representations Z' = ( ..., Zz where Zn 6.1 ..... z,TT] } E RTTxx xd Tn is the number words in the n-th caption, and is the embed- ding dimension of LLM.
Word assignment. We project the visual representations X into the same dimension as the em- bedding space of LLMs using linear layer 9(-):
z" (Z E xnd xd Z" g (xn), 1, B).
We assign the image and text to a set of words by mapping the averaged visual and text representa- tions to the frozen word embeddings W (W..... WK] } LLM. where is a vocabulary size. Similar to the prior works (Asano et al. 2020b Caron et al. 2020). we simultaneously optimize the assignment Q" {qi ..... 9 and Q qq..... Us to maximize the similarity between the representations for each modality and the word embeddings:
max Tr(QuT W 2n) =(I(Q")
max Tr(QU W Z') +cH((Q)),
Q'EQ
Q'E
(2)
where Z" 2
No Σ 2.S ž'= [z) ,.., 1
z To Σ
BBI:
We denote the trace matrix as Tr(-), entropy function as H(Q) - Ens Qis log Qij- and smooth- ness parameter of the mapping as The previous methods assume that all samples are equally assigned to each cluster by constraining the matrix Q to belong to the transportation polytope in the whole dataset (Asano al.) (2200b) or the minibatch (Caron et al.) 2020). However, the equiparti- tion assumption can lead to impractical solutions in our work since the numbers of words are not equally distributed practice. Therefore, we restrict the transportation polytope with the following constraints:
Q = == e R B QQ1 = pw. ww, 1-k====------0-
B 1B).
(3)
where 1g denotes the vector of ones in dimension B and μa is the marginal distribution of words, N₁ i.e, Mw (k) No where N₂ denotes the total number of the k-th word in the dataset. Formally,
ww can be represented as a vector with the length of LLM's vocabulary size (c.g.. 50,272 for OPT (Zhang et al.. 2022). 128 for T5 (Raffel et al. 2020)). ). By satisfying the constraints, the word embeddings are selected according to their frequency in the dataset. The optimized assignments Q8e and Q* can be obtained over the set Q as the form of normalized exponential matrix (Cuturi
2013:
W Z
W Z
Q R* Diag(ww) exp(
Diag (c,
Q" Diag( exp
Diag(c),
(4)
where c is a re-normalized vector in I2 obtained using the iterative Sinkhorn- Knopp algorithm Cu- turi 2013)
4

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparisons between MAGMA [Eichenberg al. 2022). LiMBeR (Merullo al.
2023). and VLAP for zero-shot image captioning on NoCaps and MSCOCO datasets. We report
architectures of visual and language models, CIDEr-D (Vedantam et al. 2015), CLIP-Score, and RefCLIP Score (Hessel ct al. )
NoCaps (CIDEr-D)
NoCaps (All)
MSCOCO
Method
Vision Encoder
LM
Out
Near
All
CLIP-S
Ref-S
CIDEr-D
CLIP-S
Ref-S
MAGMA
CLIP RN50x16
GPT-J
30.4
43.4
36.7
38.7
74.3
78.7
47.5
LiMBeR
BEÎT
GPT-J
20.3
16.3
26,9
28.5
62.0
69.1
22.3
63.6
70.0
LiMBeR
CLIP RN50x16
GPT-J
34.3
48.4
41.6
43.9
74.7
79.4
54.9
76.2
80.4
VLAP
BEÎT
OPT
31.1
45.4
40.6
42.2
72.1
77.3
50.7
73.7
76.4
VLAP
BEIT
T5
31.6
46.3
41.9
43.4
72.6
78.9
51.6
74.2
78.5
VLAP
CLIP ViT-B/32
OPT
48.2
62.7
59.3
61.3
84.8
88.5
69.9
86.7
91.8
VLAP
CLIP ViT-B/32
T5
48.3
62.7
59.6
61.6
85.1
88.7
69.4
87.6
92.0
VLAP
CLIP RN50x16
GPT-J
53.8
67.5
65.7
64.5
88.3
90.1
75.3
90.6
92.2
Relaxing the modality gap with assignment prediction. The objective of our method is to pre- dict the assignment of one modality from the other modality representation, i.e., predicting Q" from Z' and Q' from Z". The assignment prediction can be formulated with the cross-entropy loss between the assignment and probability that corresponding modality data belongs to each word:
mmap
Σ
One log Pis
Qii log Pi
(5)
exp(z,, Wx/T)
where
P nk
m E (v,4)
ΣN
ke exp( 2. /T)
where 7 a temperature parameter eW al. 2018). This loss function makes two different modal- ity representations contain the same information.
3.2 IMAGE CAPTIONING WITH FROZEN L.MSS
To connect the (projected) visual representations to frozen LLM to yield the general capability of I.MM (i.e., generative ability), we train the linear projection layer g(-) with the image captioning objective. Specifically, the visual representations Z, are prepended to the input text prompt (e.g. "A
photo of"), being active as soft prompts of LLM. Following the previous works (Merullo et al. 2023
Koh et al. 2023), we employ the prefix language modeling loss the objective:
Ceap
log fe(s, Zy 1p.
81,-.11
B
where N2 is the number of words in the caption, [prefix] is the input text prompt, and is a text token of the f-th word.
The final objective function is a weighted sum of the assignment prediction loss and captioning loss:
Crap + Seap-cap:
where Amup and Aeap control the importance of each objective. By minimizing the final objective, we simultaneously make the visual and text representations contain the same information while keeping capability LLM, allowing the frozen image encoder and LLM can be effectively connected with only the linear transformation.
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETTINGS
We previously defined the transport polytope with the word distribution of a given dataset. This assumption poses possible problem: VLAP often fails when the word distribution of training

---

Published as a conference paper at ICLR 2024
Table 2: Performance comparisons between Frozen Tsimpoukelli et al. 2021 MAGMA (Eichenberg et al.
2022). LiMBeR (Merullo al.) 2023). and VLAP zero- and few-shot visual question answering on the VQA2 datasets. We report the architectures of visual and language models and accuracy (%)
n--hott
Method
Vision Encoder
LM
2
Frozen
NFRN50
GPT-2
29.5
35.7
38.2
MAGMA
CLIP RN50x16
GPT-J
32.7
40.2
42.5
43.8
LiMBeR
BEiT
GPT-J
24.9
34.4
34.7
31.7
LiMBeR
CLIP RN50x16
GPT-J
33.3
39.9
40.8
40.3
VLAP
BEiT
OPT 1,
34.5
44.2
45.1
45.7
VLAP
BEIT
B
34.7
43.3
45.4
45.5
VLAP
CLIP ViT-B/32
OPT
40.4
52.6
53,8
54.7
VLAP
CLIP ViT-B/32
T5
41.1
51.3
51.9
52.6
Table 3: Performance comparisons between ViLBERT (Lu et al.
2019).
CLIP (Radford al. 2021 ES-
PER Yu et al. 2022). FROMAGe (Koh et al. 2023). and VLAP for zero-shot mmage-and-lext-too and lext-to-image (T2I) retrieval on the Visual Dialog datasets. Following Koh et al. (2033
(IT2T)
report Normal-
ized Discounted Cumulative Gain (NDCG), Mean Reciprocal Recall (MRR), R R and R@ 10 for IT2T
retrieval and R@I R@5, and R@ T21 retrieval.
Train. Params.
Training Data
IT2T
T21
Method
NDCG
MRR
R@D
R@5
R@ 10
R@!
R@5
R@I
ViLBERT
114M
3.1M
11.6
6.9
2.6
7.2
11.3
CLIP VIT-L/14
300M
400M
10.9
8.5
3.1
8.7
15.9
17.7
38.9
50.2
ESPER
4M
0.5M
22.3
25.7
14.6
Incapable
FROMAGe
5.5M
3.1M
16.5
22.0
17.6
20.1
25.1
20.8
44.9
56.0
VLAP w/CLIP-T5Bus
0.6M
3.1M
20.5
24.9
21.1
23.7
29.6
26.9
55.3
69.1
data cannot cover real-world scenarios. To prevent this issue, preserve the soft assignments as in Caron et al. (2020) instead of strictly assigning representations to words included in the given caption data. In addition, carefully argue that large dataset (e.g. CC3M (Sharma et al. 2018) sufficient to cover the real-world scenario. To validate this, we mainly investigate rision-language tasks with the zero-shot setting, i.e., training and test data come from different datasets.
Datasets. We evaluate VLAP zero-shot image captioning, visual question answering (VQA), and modal retrieval. We first train the model on the CC3M (Sharma et al.) (018)) and evaluate the performance the following datasets for each task. For zero-shot image captioning,
evaluate
the performance on MSCOCO (Lin et al. 2014) and NoCaps (Agrawal et al.) 2019). following (Merullo et al. 2023). For visual question answering, we evaluate the model the VQA2 (Goyal) ct al. (2017) dataset from zero-shot to 4-shot settings. In cross-modal retrieval, we use the Visual Dialog (Das et al.) (017) dataset for the comparability to previous work (Koh 2023 The input text prompt for each task is presented in Section
Model architecture. For the frozen image encoders, we employ two vision transformer models according to their pretraining data configuration: BEiT (Bao al. (2022) pretrained on image-only data and CLIP VIT/B-32 (Radford et al., 2021) jointly learned with the text encoder on image-text data. For the frozen language models, we explore two types of LLMss according to their architectural configuration: OPT 1.3 (Zhang et al. (022) as secoder-basedd LLMs and T5 (Raffel et al.) 2020) as encoder--decoder--aass LLMs.
4.2 ZERO-SHOT IMAGE CAPTIONING
We provide performance comparisons between Frozen TTimpoukelli et al., 2021 I), MAGMA (Eichenberg al. 2022), LiMBeR (Merullo al. 2023), and VLAP for zero-shot image captioning in Table We mainly report CIDEr-D (Vedantam et al. 2015), CLIPScore, and Ref-CLIPSCcre (Hessel et al. 2021). following (Merullo et al. 2023). The results show that

---

Published as a conference paper at ICLR 2024
Input
Groundtruth
Input
Groundtruth
mon rising on the back motorcycle
Deveroge th gloss on tobie next laptop
Prediction
Prediction
BEIT-OPT:
loptop
mon sitting
gratsy grass
BETT-T: loptop sitting de
CLIP-OPT.A man on a roud riding a bike
CLIPOPTT glass noxt to laptop
photo of>
CLIP-TS:A man in red snirt rusng matorcycle road
photo of>
LUP-TS.WA loptap with gless of wine
Input
Groundtruth
Input
Groundtruth
A dion riding ski down snow covered stope
gentioman riding foorse down city strect
Prediction
Prediction
BEIT-OPT-AL group of people
mon Bylng CLI-OPPE mon on a on snowy siope
BEIT-TS: mar is sithing hore road
CLI-OPT: mon the middl of mad rillig horse CUP-TS: mon noing a horse in the midtle of street
photo
CLIP-S: person riding ska on snowy slope surfoce
photo of>
(a) Zero-shot image captioning
Input Image
Question:
Input Image
Question:
this?
the image?
Answen:
Answer: BEIT-OPT:
OPT.4 BEIT-TS:
BEIT-T5
OPT. CUP-TS: zeora
CLIPTT:: bear
Input
Question: the
Question: person doing?
Answer:
BEIT-OPT. A water
Answer:
BEIT TS. mon
BEIT-OPT: sumning
CLIP-OPT OP--OPT:AA
BEIT-TS: riding
suring CUP-TS: sorfing
(b) Visual question answering
Input Image and Dialogg
many people
Question:
male
Question: color
yes
Answer:
Answer: he?s smiling would say yes
man happy
Question: Prediction: looking at girafFa
doing
(c) Visual dialog (image-and-text to text (IT2T) retrieval)
Dialogs
Selected (Top
Question: Answer: yes
you any
Answer:
Question: is it day time
Question: about how old the san
Answer: yes
Answer: raybe in his 38s
Question: what color is his hair
Question: what color is his shirt
Answer: brown
Answer: baby blue
Questiion: is he wearing shorts ts Answer:
Question: what color are his shorts
Answer: black
Questtion: can his shoes
Questiio: what color are his shoes
yes
white
(d) Text-to-image retrieval
Figure 3: Selected examples from VIAP for vision-language tasks, including (a) zero-shot image captioning, (b) visual question answering (VQA), (c) visual dialog, and (d) text-to-image (T2I) retrieval.
VLAP consistently outperforms the previous linear transformation-base approach (Merullo et al. 2023 with large margins. Especially, VLAP with the BEiT with T5 achieves 14.9% and 29.3% CIDEr-D improvements on each dataset. It achieves comparable performance to LiMBeR with the CLIP image encoder, despite BEiT lacking linguistic supervision its pretraining. VLAP also attains significantly higher CLIPScores, which represent how the image and generated caption are semantically similar, improving the CLIP-S by 10.4%, 11.4%, and the RefCLIP-S by 9.3%, 11.6% NoCaps and MSCOCO, respectively. In addition, parameters LLMs (T5 has 220M parameters, while GPT-J has 6B parameters), demonstrating the effectiveness of the proposed method in grounding LLMs visual data. In practice, VLAP with the CNN-based vision model (i.e., CLIP RN50x16 (Radford et al.] (20TT) and the larger LLM (i.e., GPT-J (Wang & Komatsuzaki 2021) significantly outperforms MAGMA and LiMBeR with
use much smaller number of

---

Published as a conference paper at ICLR 2024
Two people riding bikes on boardwaal with beots
A beoutful view of the waterr
The people
rifing hiles
Input
Generated Caption
Input
Caption
Input
Caption
Visual Semantic Subtraction
group of people sitting red table roare
group of people eating pizza tuble
group of people cating sandwich
tabée
Input
Caption
Input
Caption
Input
Caption
Visual Semantic Summation
Figure 4: Selected examples for visual semantic arithmetic.
the same backbones by large margins across all metrics. The performance evaluated with additional captioning metrics, including BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie 2005).
ROUGE (Lnn 2004), and SPICE (Anderson et al. 2016), are shown in Appendix
Qualitative examples are illustrated Figure 3a The results show impressive performance of VLAP across all vision-language models. VLAP with the BETT consistently generates apposite captions, even occasionally produces relatively simplified captions and provides some wrong words (e.g.
'snow' 'white field' or 'air' in the example).
4.3 VISUAL QUESTION ANSWERING
We evaluate the performance with zero-shot and few-shot settings on visual question answering (VQA). In n-shot settings, we randomly prepend n complete examples before each question as de- scribed in Tsimpoukelli et al. (202);; Eichenberg al. (2022). We provide the comparison between VLAP and three baselines (Tsimpoukeli et al. 2021] Eichenberg et al. 2022 Mcrullo et al.) 2023), shown in Table VLAP outperforms the previous methods with large margins, showing the effectiveness of the proposed method. Surprisingly, VLAP with the BEiT and T5 which the most challenging condition (in terms of pretraining data and the number of parameters), achieves higher performance than the previous methods for all evaluation settings.
In Figure 3b provide the qualitative examples for zero-shot and 4-shot VQA. For zero-shot VQA, the results show that while VLAP generally infers the correct answers with all image encoders and language models, some failure cases are shown with the BEiT models.
4.4 CROSS-MODAL RETRIEVAL
For cross-modal retrieval, compare the performance ViLBERT Lu al. 2019 CLIP (Rad- ford et al. 2021). ESPER (Yu et al. 2022), FROGMAGe Koh et al. 2023). and VLAP evaluated on Visual Dialog DDss al. 2017) for two types of retrieval tasks: (1) visual dialog referred as mmagge-ndd-tex--tee
(IT2T) retrieval to reason the correct answer from given an image, ques- tion, and a dialog about image. We report the Normalized Discounted Cumulative Gain (NDCG), Mean Reciprocal Recall (MRR), and Recall@k, following Yu et al. (2022); Koh al. 2023). (2) text-to-image (T2I) retrieval to find the image from a given dialog. We report the Recall perfor- mance for T2I retrieval. Since dialog in the dataset consists of question-answer pairs for images, we use the text prompt used in the VQA task. We provide the comparisons for cross-modee retrieval in Table
Image-and-text to text (IT2T) retrieval. For IT2T retrieval, we compute the loss (i.e. cross- entropy loss) between the prediction given answer candidates, and select the answer with the lowest loss. VLAP outperforms the previous methods (Lu et al. 2019 Radford et al. 202 Yu et al. Koh et al. 2023), achieving 3,5%, 3.6%, and 4.5% improvements on R 1, 5, and 10, respec-
2022
We follow the procedure of the official VQA 2.0 repo: https: whu.thttbbbcc. com / wtt.://oo/oo-iiioni.aa.... l.bb
VQA
8

---

Published as a conference paper at ICLR 2024
tively. While VLAP shows slightly lower NDCG and MRR performance than ESPER (Yu et al., 2022), they train the model on the superset of Visual Dialog (i.e. MSCOCO), already containing in- context information of the dataset. The comparison between VLAP and FROMAGe demonstrates the effectiveness of our assignment prediction objective as the main difference between the two methods is the learning objective. VLAP consistently outperforms FROMAGe (Koh et al. 2023) even with fewer trainable parameters, improving 4.0% and 2.9% on NDGC and MRR, respectively.
Text-to-image (T2I) retrieval. As VLAP minimizes the modality gap between image and text data, the visual and text representations can be directly used to retrieve each other. For T21 retrieval, we first extract the text and visual representations for the given dialog and all images in the dataset. We directly measure the similarity between the text and visual representations and select the image with the highest similarity. VLAP substantially outperforms the prior works (Radford et al., 2021 Koh et al., 2023) over all evaluation metrics, achieving 9.2% and 6.1 % improvements on R over CLIP (Radford et al. 2021) and FROMAGe.
The experiments on cross-modal retrieval demonstrate the flexibility of VLAP for vision-language tasks. We emphasize that VLAP provides competitive results even on the retrieval task, showing that VLAP can be applied as a general model without being limited to tasks.
4.5 VISUAL SEMANTIC ARITHMETIC
The pretrained LLMs capture task-agnostic linguistic structures and lexical semantics (Jawahar iet al., 2019 Liu et al. 2019; Tenney et al. 2019; Hewitt & Manning 2019 Vulić et al. 2019). Recent work TTwel
al. 2022) has shown this capability persists in VLMs so that the visual embeddings can be expressed in textual way, which is called visual semantic arithmetic. VLAP could perform this task because the visual representations are learned to hold semantic taxonomy of L.M.s. For example, the semantic direction between two images can be expressed by subtract- ing one visual representation from another representation. Similarly, the summation of two visual representations allows for guidance for text generation. To demonstrate the arithmetical ability of VLAP, we provide an analysis of visual semantic arithmetic (i.c., subtraction and summation oper- ations) in Figure 4 In subtraction, the conceptual direction between two images can be obtained. For example, "A beautiful view of the water" and "Two people riding bikes at sunset" are obtained from the same image of "Two people riding bikes on a boardwalk with boats" by subtracting images containing "two people" and "view of the ocean," " respectively. Meanwhile, the visual concepts can be guided by other visual semantics with summation operations. For example, scene of people sitting around a table can be guided by the image of "pizza" " and "sandwich", generating a caption of "A group of people eating pizza (or sandwich) on table". These results demonstrate that the visual representations from VLAP contain the semantic taxonomy of LLMs, allowing the VLAP's visual representations be expressed in a textual way.
CONCLUSION AND FUTURE WORK
We propose VLAP bridge pretrained vision encoders and LLMs using single linear layer for vision-language tasks. VLAP efficiently and effectively learns the linear mapping between image and text data by formulating assignment prediction using LLMs' word embeddings as the optimal transport. Experimental findings corroborate the effectiveness of VLAP, demonstrating significant performance compared to the previous linear transformation-basee methods on spectrum of vision- language tasks. Furthermore, we demonstrate that VIAP holds a semantic taxonomy of LLMs, highlighting the emerging flexibility and applicability of the proposed approach.
While linear transformation- based methods excel in computation and memory efficiency. there is still a substantial performance gap against modular-based methods, such as Flamingo and BLIP-2. We attribute such gap to a lot of trainable parameters in their methods (10.2B in Flamingo, 188M in BLIP-2) and larger training data (1.8B image-text pair in Flamingo, 129M image-text pair in BLIP- 2). Since the optimal transport-based based assignment prediction be easily moved to the modular- based methods, scaling VLAP with the modular-based models and training on larger multimodal datasets promising directions for future work.

---

Published as a conference paper at ICLR 2024
6 REPRODUCIBILITY STATEMENT
VLAP employs the pretrained BETT and CLIP image encoder as vision models and the pretrained 1.38 and Base as LLMs, which can be accessed by anyone. We provide PyTorch implemen- tation for VLAP at https: / /ig hub, com/ com/park haanmparr--anninn jungin/vlall enable researchers to reproduce the results demonstrated in the paper as well as conduct additional analysis for VLAP with other vision models and LLMs.
The implementations will
ACKNOWLEDGEMENT
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) IA22220011122220077000
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In ICCV, 2019.
SannBBppistte Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sehastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic proposi- tional image caption evaluation. In ECCV, 2016.
Yuki M. Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. In NeurIPS, 2020a.
Yuki M. Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In JCLR, 2020b.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshop. 2005.
Hangbo Bao, Dong, Songhao Piao, and Furu We. Beit: Bert pertaining of image transformers. In ICLR, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Ven-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
10

---

Published as a conference paper at ICLR 2024
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. 2023.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In CVPR, 2005.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finetunedd language models. In arXiv preprint arXiv: 2210.11416 2022.
Niv Cohen, Rinon Gal, Eli A. Meirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language representations. In ECCV, 2022.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS, 2013.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. CVPR, 2017.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghnn,, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang. Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi- modal alignment using representation codebook. In CVPR, 2022.
Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabeseu, and Anette Frank. Magma-multimdaa augmentation of generative models through adapter-based finetuning. In EMNLP, 2022.
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Jiaxian Guo, Junnan Li. Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven C.H. Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. In CVPR, 2023.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
John Hewitt and Christopher D. Manning, A structural probe for finding syntax in word representa- tions. In NAACL, 2019.
Zhicheng Huang. Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. In CoRR, 2020.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL, 2019.
Chao Jia Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yunhsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and laniun-language representation learning with noisy text supervision. In ICML, 2021.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In ICMI., 2023.

---

Published as a conference paper at ICLR 2024
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision- vision-language understanding and generation. In /CML, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: un- derstanding the modality gap in multi-modal contrastive representation learning. In NeurIPS, 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, Visual instruction tuning. NeurIPS, 2023a.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In NAACL, 2019.
Yang Liu, Zhipeng Zhou, and Baigui Sun. Cot: Unsupervised domain adaptation with clustering and optimal transport. CVPR, 2023b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. In NeurlPS, 2019.
Jack Merullo, Louis Castricato, Carsten Fickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In ICLR, 2023.
Ron Mokady, Amir Hertz, and Amit H. Bermano. Clipcap: Clip prefix for image captioning. In arXiv preprint arXiv: 2111.09734, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In ACI., 2002.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In ICLR, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. Technical report, OpenAI, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1-67, 2020.
Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for repre- sentation and semi-supervised learning. Neural Computation, 33(11), 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI, 2018.
12

---

Published as a conference paper at ICLR 2024
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efftcient transfer learning for vision-and-langee and-language tasks. CVPR, 2022.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In EMNLP, 2019.
lan Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nip pipeline. ACL, 2019.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text genera- tion for visual-semantic arithmetic. In CVPR, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziére, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. In arXiv preprint arXiv:2302.13971 2023.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NeurlPS, 2021.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. In arXiv preprint arXiv: 1807.03748, 2018.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015.
Ivan Vulić, Edoardo M. Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. Probing pre- trained language models for lexical semantics. In ACL, 2019.
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model. 2021. URL
com/
htt.:/mmeestttaaaeaeeeeerrrr jax
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Renjun Xu, Pelen Liu, Liyan Wang, Chao Chen, and Jindong Wang. Reliable weighted optimal transport for unsupervised domain adaptation. In CVPR, 2020.
Youngjac Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Rowan Zellers, Prithviraj Ammanabrolu, Ronan Le Bras, Gunhee Kim, and Yejin Choi. Multimodal knowledge alignment with reinforcement learning. In arXiv preprint ar XrXiv:2205. /2630, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer Opt: Open praitrdi transformer language models. In arXiv preprint arXiv: 2205.0 01068, 2022.
13

---

Published as a conference paper at ICLR 2024
Table 4: Hyperparameters for training VLAP corresponding each image encoders and LLMs.
Model
Hyperparameter
BEiT-OPTi.s
BEiT- 15TT-
CLIP-OPTi3B
CLIP-T5B.
Warmup steps
1,5K
3K
1.5K
3K
Learning rate
le-4
5e-3
le-4
5e-3
Batch size
128
256
128
256
Total steps
30K
15K
30K
15K
Final learning
0
Adam W /
(0.9, 0.999)
Text prompt
A photo of
A TRAINING DETAILS
In Table we provide hyperparameters used in training.
B INFERENCE DETAILS
As demonstrated in Section 4 VLAP presents the emerging flexibility and applicability to vari- ous vision-language tasks. While VLAP is trained on unified training procedure, the inference provides output with slightly different scheme with respect to the task. In Figure we provide il- lustrations for inference corresponding to each task to prevent confusion and help clearly understand the inference procedure of VLAP.
B.1
IMAGE CAPTIONING
For image captioning, VLAP provides the text description for the given image. As shown in Fig- 5a use "A photo of" as the input text prompt.
B.2 VISUAL QUESTION ANSWERING
For VQA, we use the text prompt "Question: O Answer:" for the OPT model and "Question: O Short Answer:" for the TS model, following Li al. (2023).
B.3 CROSS-MODAL MODAL RETRIEVAL
Visual dialog (IT2T retrieval). In visual dialog (IT2T retrieval), the image and the sequence of question-answw dialog are given as inputs. We prepend the visual representations text represen- tations of the dialog sequence and feed them into LLMs. As shown in Figure 5c we measure the cross-entropy loss between the generated output and the answer candidates, and select the one with the lowest loss as the prediction.
Text-fo-image (T2I) retrieval. Different from the other tasks, VLAP does not leverage the gener- ative ability of L.LSs for T21 retrieval. As shown in Figure [dd] we extract the visual representations for all images in the dataset. We also extract the text representations for the input dialog and directly measure the similarity between the text representations and all visual representations. We select the target image with the highest similarity.
C ADDITIONAL RESULTS
C.1 IMAGE CAPTIONING
We provide the zero-shot image captioning performance the NoCaps (Agrawal al. 2019) and MSCOCO (Lin et al. 2014) datasets with additional evaluation metrics, including BLEU (Papineni
al. 2002). METHOR (Banerjee & Laviee 2005), ROUGE (Lin. 2004), and SPICE (Anderson
14

---

Published as a conference paper at ICLR 2024
A man riding a skateboard in a park Output
Image Encoder
Linear
* LLM Encoder
Decoder
photo
Input Image
Img. Embed. Ze
(a) Image captioning
Image Encoder
Riding skoteboard
Linear
+ LLM Encoder
# Decoder
Question: what is the man doing? Short answer:
Input Image
Img. Embed. Ze
Answer
(b) Visual question answering
H Image
CCE
/ Linear
S LLM Encoder
LLM Decoder
White
Encoder
Input Image
Img. Embed. Zμ
Question: is this outside
White
Short answer: yes
Yes
Question:
do you see any clouds Short answer: no it's too zoomed in
No
Con't
Blue
Question: is it day time Short answer: yes
Answer Candidates
Question: what color are his shoes Short answer:
(c) Visual dialog (image-and-text text (IT2T) retrieval)
g Image Encoder
LLM Encoder
Matching
Linear
Question: is this outside Short answer: yes
Question: do you any clouds Short answer:
it's too zoomed in
Question:
day Short answer: yes
Question:
what color
his shoes Short answer: white
Input Dialog
Image Candidates
(d) Text-lo-image retrieval
Figure 5: Illustrations for the inference on (a) image captioning, (b) visual question answering, (c) visual dialog. and (d) text-to-image retrieval.
ct al. 2016). As shown in Table 5 VLAP with CLIP ViT-B/32 RRafford et al.) 2021 and two LLMs (Zhang et al. 2022 Raffel et al. 2020) persistently outperform the previous methods, in- cluding MAGMA (Eichenberg et al. 2022). LiMBeR (Merullo et al. 2023). and FROMAGe (Koh) et al. 2023). In addition, VLAP with BEiT (Bao al. 2022) achieves comparable better per- formance on several evaluation metrics than the previous approaches with CLIP image encoders, demonstrating the effectiveness of the proposed assignment prediction objective.
15

---

Published as conference paper at ICLR 2024
Table 5: Performance comparisons between MAGMA [Eichenberg al. 2022). LiMBeR (Merullo al. 2023 FROMAGe [Koh al. 2023), VLAP zero-shot image captioning MSCOCO dataset.
report the architectures of Visual and language models, BLEU (Papineni et al. 2002), METEOR (Banerjee & Lavic
2005), ROUGE (Lin 2004), and SPICE (Anderson et al. 2016).
Method
Vision Encoder
LM
BLEU-1
BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE SPICE
MAGMA
CLIP RN50x16
GPT-J
0,432
0.300
0.203
0.137
0.159
0.376
0.117
LiMBeR
BEiT
GPT-J
0.319
0.187
0.105
0.060
0.106
0.299
0.093
LiMBeR
CLIP RN50x16
GPT-J
0.400
0.278
0.187
0.126
0.161
0.376
0.121
FROMAGe
CLIP VIT-L/14
OPTs2
0.477
0.293
0.172
0.102
0.287
VLAP
BETT BEIT
OPT. 0.449
0.348
0.225
0.174
0.280
0.369
0.118
VLAP
B5ese
0,458
0.357
0.266
0.176
0.287
0.382
0.120
VLAP
CLIP VIT-B/32
OPT
0.567
0.429
0.343
0,291
0.310
0.482
0.135
VLAP
CLIP ViT-B/32
T5
0.571
0.437
0.348
0.297
0.311
0.487
0.137
Output
Input
Output
(a) Building-~Country
(b) Capital
facebook
Microsoft
Japan
Input
Output
Input
Output
(c) CEO -Company
(d) Food Mond-+Country
Figure 6: Illustrations visual relation benchmark.
C.2 VISUAL SEMANTIC ARITHMETIC
Performance on the visual relation benchmark. To further explore the capability in visual se- mantic arithmetic of VLAP, we evaluate VLAP on the visual relation (VR) benchmark (Tewel et al. 2022). The VR benchmark consists of total of 320 relations, including building-xcunntyy country-xcapital CEO CEO->company, food->countr, and leader >country Since the relation of lcader-ycuuntry includes out-of-date information (e.g. 'Obama'- experiment. We illustrate examples for the VR benchmark in Figure 6
exclude it from this Table 6 we compare the
performance between VLAP, ClipCap (Mokady et al.) 2021}, and ZeroCap (Tewel al., 2022) in terms of BLEU-1 Papineni et al. 2002), R and CLIP-Score (Hessel et al. 2021), following Tewel al. (0022). For fair comparisons,
CLIP VTT-B/32 (Radford et 2021) as the image encoder and GPT-2 (Radford et al., 2019) as the LLM. The results show that VLAP outperforms the previous methods by large margins. In particular, we attain high correlations of over 75% in all relations on the semantic distance-base based metric, i.e., CLIP-Score.
identically
Visualization. We provide UMAP visualization (Sainburg et al. 2021) of visual representations analyze the visual arithmetic analogy VLAP in feature space, as shown Figure The blue and green circles are the real pairs and the red circles denote the embeddings obtaied by the visual arithmetic operation. For example, the images of 'the Eiffel Tower'- "FFrncee" and 'the Leaning Tower of Pisa'-'Italy in Figure 7a have "building-country 'building' to 'country" can obtained through a subtraction operation between visual features of 'Ialy' and 'the Leaning Tower of Pisa.' The summation of this direction with the embedding of 'the Eiffel Tower' allows us to derive the embedding corresponding to the same relation as 'building- country,' i..., 'France.' In practice, the embedding obtained by visual arithmetic operations is close to the image of 'France" in the feature space and the LLM generates the caption of 'A photo of France' with the embedding obtained by visual arithmetic operations. Similarly, the direction for
relation. The direction from
16

---

Published as a conference paper at ICLR 2024
Table 6: Performance comparisons between ClipCap (Mokady et al.
2021). ZeroCap Tewel et al. 2022 and
VLAP visual semantic arithmetic on the Visual Relation dataset. Following Tewel (222). we report BLEU-1, R@S, and CLIP-Scort
Building
Country
Country
Capital
CEO
Company
Food
Country
Method
B@1
R@5
CLIP-S B@1 1 R@5
CLIP-S
B@ 1 R@S
CLIP-S
B@I
R@5 CLIP-S
ClipCap
0.003
0.035
0.24
0.0
0.0
0.22
0.004
0,005
0.18
0.0
0.0
0.24
ZeroCap
0.1
0.32
0.7
0.14
0.32
0.68
0.1
0.3
0.64
0.03
0.33
0.66
VLAP
0.3
0.53
0.87
0.46
0.62
0.89
0.25
0.5
0.75
0.17
0.5
0.81
A photo of France
7.00
5.0-
6.75-
UMAPP
UMAP
A photo
6.50
London
3.5-
6.25-
1.2
1.4
1.6
2.0
13.75
14:00
14.25
14.50
14.75
UMAPo
UMAPo
(a) Building Soallinng--ountty Country
(b) Capital
Figure 7: UMAP visualization Sainburg et al. 2021 of selected examples from visual relation benchmark.
relation 'country-capital" can be obtained by subtracting image 'USA' from the image of 'Washington.' The LLM generates the caption 'A photo of London' from the embedding of the summation of the direction and the image 'England.'
D ABALATION STUDY
D.1 LOSS FUNCTION
Table 7: Zero-shot image captioning performance on MSCOCO (Lin et al. 2014) corresponding to (a) different ratios between the halancing parameters (i.c., Ampp Aeap) and (h) combinations of learning objectives.
Amap
Aeap
CIDEr-D
CLIP-S
Ref-S
Leap
LUM
Luee
Lmpp
CIDEr-D
CLIP-S
Ref-S
51.7
72.1
76.3
62.3
80.3
$1.7
72.1
76.3
0.4
0.6
68.1
84.8
89,2
58.4
78.2
81.7
0.5
0.5
69.6
86.3
91.6
61.8
79,0
82.3
0.6
0.4
69.9
86.7
91.8
65.7
82.5
85.8
0.8
0.2
64.5
83.1
87.8
69.9
86.7
91.8
49.6
72.4
76.1
(b) Combinations of training objectives
(a) Different ratios between Amap and Amap
To validate the effectiveness of each component in VLAP, we evaluate the zero-shot image captioning performance of VLAP trained with various combinations of learning objectives on MSCOCO (Lin et al., 2014). All experiments are conducted with CLIP ViT-B/32 (Radford et al.) 2021 and 1.T (/hang et al. 2022) as the vision model and L.M,, respectively.
Effectiveness of >amap and Aeap- The balancing parameters in equation 7 control the importance of Cmup and Lesp- In Table 7a we report the performance corresponding to different ratios between Amap and Acap while keeping the sum of two terms as 1. In the first row (i.c., Amap Acap 0 1), the final objective is Leaps which is equivalent to the language modeling objective of LiM- BeR (Merullo al. 2023) and the first stage of LLaVA (Liu et al. 2023a). With this objective only,
17

---

Published as a conference paper at ICLR 2024
Image Text
Image Text
15.0
Image Text
UMAP
UMAF
UMAPt
UMAPP
||||
|A
|
|
10
UMAPo
UMAPo
UMAP
UMAAPo
(a) Before training
(b) w/Ceap
(c) w/ + Lum +
Laee
(d) w/ &map
Figure 8: UMAP visualization Sainburg al. combinations of objectives.
2021 of visual and text representations corresponding to the
VLAP performs significantly poorly, attaining 51.7 of CIDEr-D. While VLAP with only assignment prediction (i.e.. L = mmpp) performs worse than with only language modeling, Cmpp improves the performance even by a small portion, showing performance improvement of 10.6 on the CIDEr-D score. Increasing the ratio of Amap gradually improves the performance, achieving the best perfor- mance of 69.9 with Amap : Acap 0.6 0.4. Thus, the results demonstrate the effectiveness of the proposed assignment prediction objective while emphasizing the necessity of the language modeling objective to leverage the generative ability of L.LMs.
With additional objectives. Image-text contrastive learning (ITC) and image-text matching (ITM) objectives, which directly compare image and text representations, have been widely used in cross-modal alignment (Li et al. 2021 2022, 2023). We use the ITC and ITTM objectives to eval- uate the performance of VLAP trained with these objectives. For ITC, we contrast the similarity between averaged image and text representations (i.e., Z" and Z') of positive pair against those of negative pairs batch. The ITC objective Lirc is defined as the cross-entropy loss between pairwise similarities and the groundtruth one-hot similarities, where the positive pair has 1 and the negative pairs have O. For ITM, we learn an additional linear layer as a binary classifier to determine whether image-text pair is matched or not. We concatenate the averaged image and text repre- sentations and feed them into the classifier with the softmax activation. Following Li et al.] (2021) 2022 2023), we also employ the hard negative sampling strategy based on the pairwise similarity. The ITM objective LITM is defined as the binary cross-entropy loss.
We evaluate VLAP trained with different combinations of learning objectives: (1) Leap' (2) Leap + LITMi (3) Leap + LITM + Lire: (4) Leap + CrTM + LiTc + Lmap: and (5) the original ob- jective of VLAP, Leup Lmap- As shown in Table 7b LITM and Luce contribute to performance improvement with Leup, achieving 58.4 in (2) and 61.8 in (3). However, we notice that the ITM and ITC objectives hurt overall performance, as shown in the performance comparison between (4) and (5). We argue that ITM and ITC solely focus direct pairwise comparisons without considering the semantic correlation between predefined two embedding spaces. Namely, they overly enforce instance discrimination encourage the existence of the modality gap (Liang al. 2022).
To further analyze the impact of each objective, we also provide UMAP visualization (Sainburg al. T02T) with the modality gap distance, shown in Figure 8 Note that the blue and red circles represent image and text embeddings and the gray lines refer to image-text pairs, respectively. Formally, the modality gap A is defined as the difference between the center of visual and text representations (Liang et al. 2022):
Z
(8)
π
where n the total number of image-text pairs, z and o are the i-th averaged visual and text representations. As shown in Figure 8a we can observe the obvious modality gap before training, which separates two modality embeddings, and the default gap distance between CLIP ViT-B/32 and Tasee is || 1.378. Although the image captioning objective Leap (Figure (b) and the addi- tional objectives (Figure Sc reduce the modality gap distance to ||||| = 0.972 and || = 0.870, respectively. the modality gap still exists. In Figure 8d the modality gap between
modality
embeddings is significantly relaxed and the gap distance is further reduced to |||| = 0.725 with assignment prediction objective.
18

---

Published as a conference paper at ICLR 2024
Table 8: Ablation study for the components in assignment prediction. We evaluate the zero-shot image cap- tioning performance on MSCOCO (Lin et al. 2014) report the CIDEr-D score.
MSCOCO (CIDEr-D)
Method
0.1
0.05
0.01
0.005
(i)
VLAP w/o word embeddings
20.4
43.8
38,9
(ii)
VLAP w/o word distribution
50.7
57.9
57.1
46.2
(iii)
VLAP
61.8
68.0
69.9
67.7
Table 9: Performance comparisons VLAP according to the training dataset for several zero-shot vision-
language tasks.
Training Data
NoCaps (CIDEr-D)
MSCOCO
VQA2
IT2T
T21 (R@1 1)
Vision Encoder
LM
(CIDEr-D)
(n = 0)
(R@ 1)
CC3M CLIP VIT-B/32
BS
61.6 62.4
69.4
21.1 23.0
CC12M CLIP ViT-B/32
T5Bue
70.1
41.3
28.2
D.2 OPTIMAL TRANSPORT
The assignment prediction VLAP mainly differs from the previous optimal transport- (or clustering-basede approaches (Asano al. 2020b Caron et al.] 2020 Asano al. 2020a Duan et al. 2022) in the following aspects: (1) defining tixed central space instead of learnable central space and (2) defining the transportation polytope with the word distribution of the training dataset instead of the equipartition assumption. We conduct additional experiments to validate the effec- tiveness of each component in our assignment prediction. Table [8 evaluate each model for zero-shot image captioning (CIDEr-D) on MSCOCO corresponding to the varying entropy parame- ter c. (i) "VLAP w/o word embeddings", we train VLAP with learnable prototypes as in [Asano et al. (2020b); Caron et al. (2020)] Asano et al. (2020a); Duan et al. (2022). In this setting, define 3K prototypes following Caron et al. (2020) and apply equipartition assumption. In (ii) "VLAP w/o word distribution", we also verify the effectiveness of the word distribution. In this setting, we use the word embeddings as a fixed central space and perform the optimal transport with the equipartition assumption. The comparison between (i) and (ii) shows that the word embedding achieves substantial performance improvement, demonstrating the effectiveness of the word embed- ding. The comparison between (ii) and (iii) original VLAP shows that the transportation polytope with the word distribution achieves additional performance improvement and provides more robust performance regarding
D.3 TRAINING DATASETS
To verify the impact of the training dataset scale the zero-shot capability, we train VLAP on the CC12M (Changpinyo al. 202T) dataset, which contains about 4x more images than CC3M. We report the performance marginally improved across all tasks despite the considerable increase in the amount of training data. Since VLAP defines transportation polytope with marginal word distribution, simply increasing the number of training data while keeping similar word distribution could not derive a significant performance improvement. Namely, VLAP has training efficiency for relatively small amount of data that might cover a word distribution.
zero-shot vision-language tasks in Table The performance of VLAP is
19


Reviews:
[{"review_id": "Vjq2iwj6Mu", "reviewer": "Reviewer_m8gW", "summary": "The paper aims to align the LLMs (encoder/decoder or just decoder) with image encoders such that the LLMs can comprehend visual input better. It further restricts the design space to freeze the original LLM and visual encoder, just relying on a cheap learned linear transformation. To adapt such a transformation, the paper presents two learning objectives -- assignment prediction and image captioning. Empirical results are presented on 3 different tasks -- image captioning, VQA and cross-modal retrieval (I2T, T2I).", "questions": "-- Can the authors show ablation studies for the L_map and L_cap objectives to develop better understanding of each component?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The problem is well motivated with wide applications.\n\n* The paper is mostly well written and explained.\n\n* The empirical results show a big delta which demonstrates the effectiveness of the approach. The studies are also conducted on wide range of problem settings.", "weaknesses": "* The motivation for restricting the learned parameter space to just linear layers is unclear -- it would have been more interesting to see more analysis around different learned parameter space including non-linear layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3CIkpDJh7", "reviewer": "Reviewer_oDtB", "summary": "The paper proposed to bridge the vision and language modalities by predicting the assignment between LLM word embeddings and those two modalities. Specifically, the optimal transport is employed to decide the assignment between LLM word embeddings and image/caption contextualized embeddings, and then the model is required to predict the assignment of one modality from the other modality. Experiments are conducted on multiple tasks/datasets to prove the effectiveness of the proposed method.", "questions": "1. Does the LLM word embedding have to be from the same LLM as used in language encoding?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Demanding one modality's representation to predict the assignment between the other modality and common feature space (LLM word embedding) is an interesting idea to bridge two modalities. \n2. Evaluations on different tasks show a better performance than previous work.", "weaknesses": "1. Comprehensive ablation of w/ and wo/ assignment prediction on the same vision/language backbones is missing. \n2. Comparison with other baselines that are designed for alignment is missing. For example, contrastive alignment in ALBEF, BLIP, and the first-stage alignment by BLIP2 which includes image-text matching, and image-grounded text generation.\n3. In experiments, the pre-training data is CC3M which is too small in terms of scale. Whether this method can be generalized to larger scale is not validated.\n4. In Tab1,2,3, when compared with previous works, the vision/language backbone is always different. I wonder if using the same backbones as previous works, will the proposed method still outperform them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NE4BzEScxi", "reviewer": "Reviewer_5A8P", "summary": "This paper introduces VLAP bridges vision encoders and language models through assignment prediction and the use of word embeddings to map visual representations into language space. \nAn optimal transport-based training objective is proposed to enforce the consistency of word assignments for paired multimodal data. This allows frozen LLMs to ground their word embedding space in visual data and use their robust semantic taxonomy visually. \nThe experiments demonstrate that VLAP outperforms the linear transformation-based approaches in a variety of vision-language tasks, such as image captioning, visual question answering, and cross-modal retrieval.\nIt also shows that the visual representations that have been acquired contain a semantic taxonomy of LLMs, thus making it possible to do visual semantic arithmetic.", "questions": "Either additional ablation, justification, or additional baseline can elaborate the concern in the weakness (2).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written and easy to follow.\nThe work proposed a straightforward way of learning the linear projection layer for visual modality to learn multimodal representation, which accommodates the LLM generation.\nThe visualization shows an impressive semantic arithmetic ability to combine multimodality understanding in LLM generation.", "weaknesses": "(1) The main concern of this work is the methodology is relatively incremental without new concepts or findings.\nConcept-wise and architecture-wise, it is similar to Asano et al. (2020) Selavi, which performs optimal transport across modalities with similar pipelines. Mathematics using the Sinkhorn clustering Swav as Caron et al. (2020).\n(2) The main difference lies in 3 parts: word embedding as fixed center space, different distribution assumptions (polytope), and LLM application.\nThe first two are the most interesting part, which will be different from previous Sinkhorn-based work.\nHowever, there is no ablation study on these two components, which leads the readers to question whether borrowing existing Selavi and Swav will also work.\n(3) Also, there is no ablation on different objectives, such as existing next-word prediction on learning visual projection on LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7KGG0WrJCW", "reviewer": "Reviewer_dYxb", "summary": "In vision-language modeling, a significant challenge persists: bridging the modality gap between pretrained vision and language models. This gap arises primarily due to the models' pretraining exclusively on unimodal data, leading to inconsistencies in their embedding spaces. Motivated by this limitation and the computational costs of previous methods, this work introduces VLAP, a novel linear transformation-based approach that employs assignment prediction to connect vision encoders and large language models (LLMs). By harnessing the established word embeddings of LLMs and introducing an optimal transport-based assignment prediction objective, VLAP maps visual data representations to LLM's word embeddings, aiming for consistent modality representation. This not only results in visual data representations with the semantic richness of LLMs but also surpasses prior methods in computational and memory efficiency across various vision-language tasks.", "questions": "Please also refer to the previous section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The limitations from SOA mentioned in the paper exist, and the motivation is valid.\n2. Resolving the modality gap problem with the cross-modal assignment prediction using word embeddings of LLMs is a better solution than previous methods.", "weaknesses": "1.  A better alignment (reducing the gap) in multi-modality is the essential contribution of this work. However, it lacks studies or results, apart from the overall performance, to validate that the gap reduction is achieved by the current predicted assignments rather than the linear layers from previous works.\n2. The authors mentioned, ``Mapping visual data to LLM’s word embeddings results in learned visual representations that hold a semantic taxonomy of LLMs.'' However, there's a lack of quantitative/qualitative results to validate that this allows visual representations to inherit a semantic taxonomy of LLMs.\n3. The final objectives are influenced by the assignment prediction loss and captioning loss. However, there's a lack of study on these hyperparameters. Also, which part contributes more to the learning remains a question.\n4. For the probability that the corresponding modality data belongs to each word, $P_{nk}$, what does $P_{nk}^{v}$ in the visual modality signify? Does this ``word'' refer to the single word token in the class label of that visual region?\n5. There's a lack of formal definitions for the terms/operations appearing in equations, i.e., $Tr(\\cdot)$, $[prefix]$.\n\n[Summary] The current limitations and motivations are valid, and the claimed contribution is significant. However, in the paper's delivery, there's a concern about how this performance is achieved by the proposed architecture and mechanism. Additionally, the paper lacks a depth of study beyond introducing a novel architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "85dFa83C5l", "reviewer": "Reviewer_zAaX", "summary": "The paper proposes a method to align the visual representations of pretrained visual encoders into the input space of pretrained language models, using a linear projection layer. The linear layer is the only trainable part in the system, which is supervised by two losses: (a) assignment consistency - the visual features and text features are assigned to the word, and a similarity loss between the assignment results is applied (b) an image captioning objective. Using this method, experiments are done on 3 tasks, including image captioning, VQA, image-text retrieval to show that the method outperforms existing methods. Different variations of visual and text models are studied in the experiments. Additionally, some qualitative visual semantic arithmetic results are provided.", "questions": "1. Could abalations with and without the assignment loss be provided to show its effectiveness?\n2. Could visualizations (e.g. t-SNE) over the feature space with and without the assignment loss be provided, to show its effects in aligning the features?\n3. The difference/contribution over LLaVA or MAGMA can be more clearly discussed.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The method is simple and clear - train a linear layer with two losses including the newly proposed assignment prediction loss. \n2. Intensive experiments are provided on 3 tasks using different visual backbones (CLIP, BeiT) and text backbones (OPT-1.3B and T5-base), where results consistently outperforms existing methods.\n3. The paper is well-written and easy to follow.", "weaknesses": "My major concern is (a) the lack of ablations and feature space visualizations to show the effectiveness of the proposed loss and (b) the contribution over existing works like MAGMA is not enough. \n1. The paper is an extension of MAGMA (Merullo et. al. Linearly mapping from image to text space. In ICLR, 2023.). While MAGMA is discussed in the paper, the difference is that this paper with MAGMA is the proposed assignment prediction loss. However, the effectiveness of the proposed loss is not shown clearly in the paper.\n2. No ablation results are provided to show the effectiveness of the proposed loss - this is related to weakness-1. Since the major contribution lies in this loss, an ablation to show the contribution of this loss in the final results is very critical.\n3. The finding that a linear layer can transform visual representations into language models is not surprising, given existing works LLaVA (“Visual Instruction Tuning”, as in its first training stage), which is not discussed in this paper, and MAGMA as discussed. Therefore, the contribution of this work is weakened.\n4. The authors motivate the work by criticizing the “distance-preserving nature of the linear layer”. However, the proposed method is still a linear layer, which doesn’t solve this problem. While Fig-4 provides several examples to show the visual semantic arithmetic, a visualization of feature space would be preferred to show the effects of the assignment loss\n5. The paper would be easier to read if the method names (abbreviations) in the results tables come with citations next to them, or are described in texts to show which is which.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)