Unified Generative and Discriminative Training for Multi-modal Large Language Models
Authors: Wei Chow
Juncheng Li
Qifan Yu
Kaihang Pan
Hao Fei
Zhiqi Ge
Shuai Yang
Siliang Tang
Hanwang Zhang
Qianru Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language
multi-modal understanding
Primary area: machine_vision
TL;DR: We propose an structure-induced approach to unify generative and discriminative paradigms.

Abstract: In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucinations and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM’s hidden state. This approach enhances the MLLM’s ability to capture global semantics and distinguish fine-grained semantics. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating a novel kernel for fine-grained semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine-grained retrieval tasks. By employing a retrieval-augmented generation strategy, our approach further enhances performance in some generative tasks within one model, offering a promising direction for future research in vision-language modeling.

Unified Generative and Diseriminative Training for Multi-modal Large Language Models
Wei Chow!
Juncheng Li Qifan Yu'
Kaihang Pan'
Hao Fei
Zhiqi Ge
Shuai Yang' Siliang Tang' Hanwang Zhang
Qianru Sun*
'Zhejiang University Natiooal University of Singapore
Nanyang Technological University 'Singapore Management University
(xieqiao, junchengli, yuqifan, (aihangpan)$zzj edu.
{zhiqige, syang, siliang/ezju. edu. cn haofei37enus edw.. sg, hanwangzhang@nte edu. adu qiamrusun@smu edu. sg
Abstract
recent times, Vision-Language Models (VLMs) have been trained under two pre- dominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucina- tions and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-griined semantic differentia- tion. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM's hidden state. This approach enhances the MILM's ability to capture global semantics and distinguish fine-grained semanties. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating novel kernel for fine- aneeggriindd semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the of-the-art results multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine- fine-grained retrieval tasks. By employing a retrieval-augmente generation strat- egy, our approach further enhances performance in some generative tasks within one model, offering promising direction for future research in vision-language modeling. The project repository is here.
Introduction
In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms: generative training and discriminative training. Generative Training has achieved remarkable success in enabling Multimodal Large Language Models (MLLMs) [1, 55, 86] to develop wide range of powerful capabilities that handle various complex tasks (e.g.. open-world visual question-answering, image caption generation, etc.) within a single model. However, challenges such hallucinations and weak image object discrimination abilities [7, 89] persist. Discrininative Training, exemplified by CLIP [73], exhibits remarkable representation capabilities for zero-shot image-text classification and retrieval. Nonetheless, encounters difficulties in processing complex scenarios (i.e., retrieving multi-mon modal documents with interlcaved images and texts) [53, 54] and exhibits a limited ability to discern detailed semantic differences [79, 85].
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The disparity between these two paradigms has sparked recent studies aimed at imparting discrim- inative ability to generative pre-trained MLLMs. However, certain aspects of performance still pose limitations (e.g., singular discriminative tasks [89], weak discriminative task performance [40], weak generalization [59], etc.), while others entail compromising the model's 's original generative capabilities [8].
Overall, the reason generative paradigms struggle with performing discriminative tasks like retrieval is due to overlooking two crucial abilities:
(i) Comprehensively capturing the global semanties. Recent studies have revealed that causal LLMs tend to exhibit a bias towards capturing global information from the input samples, often resulting in tendency to overlook information located in the middle. especially for long sequences [15, 57]. As illustrated in Figure (a), we chose 500 samples from WebQA [10], where the task is to find answer.
and reason about the right image-text pair among five distractors to produce a yes We conducted experiments using VILA [52], a MILMM with state-of-the-ar interleaved image-text comprehension ability, alongside our model. When placing the relevant pair in different positions, the performance of MLLMs followed a "U" shape, indicating a bias in capturing global semantic information. Consequently, MLLMS encounter difficulties in forming comprehensive representations that encompass global semantics for retrieval tasks.
(ii) Keenly differentiating the detailed semantics. Some research [47, 82] has found that the existing generative training framework cannot fully distinguish input semantics in certain contexts, causing MILMs to struggle with tasks requiring fine-grained semanties [46, 98]. As depicted in Figure 1(b), we noticed that MILLMs face challenges in choosing the right description for two similar images in the MMVP-VLM benchmark [81]. This indicates that MILMs struggle to effectively differentiate the detailed semantics of input samples, naturally leading to difficulties in forming effective queries for retrieval.
VILA
Ours
65
reduralsmm?
12
cheice meets the
Accurayff
55-
TENt
image:
image:
secund
batterly cdnge closed.
butterfy spen.
4
Related Pair Index
A case that related pair indox 3
45
A
case for State Ø
WebQA
MMVP-VLM
Figure 1: (a) In WebQA [10], the accuracy roughly forms a "U" shape curve when the relevant image-text pair for a question appears at different positions. While our model also shows similar trends, it tends to be more stable overall. (b) The accuracy of various types of questions in MMVP- VLM [81]. it can be observed that our model's performance improves on such tasks after introducing the discriminative training. Details can be seen in Appendix E.3
In this paper, we argue that the current separated paradigms possess the potential for achieving syner- gistic gains. We propose Sugar: Structure-induced approach to unify generative and discriminative paradigms (shown in Figure 2), leveraging discriminative training to acquire the two abilities above while harnessing the potential of generative training in complex discriminative tasks like image-text interleaved retrieval and fine-gre fin-grained retrieval. Specifically, we explicitly impose the semantic rela- tionships between different input samples as an induced structural constraint on the hidden state of MLLMs. We consider the interleaved image-text sequence as the general format of input samples, and then formulate the relationship between any two samples as dynamic sequence alignment problem within the Dynamic Time Warping framework [67,33]. 33]. In this way, we can explicitly modulate the hidden states of the MLLM by leveraging the semantic relationships between interleaved input sequences, thereby encouraging the MLLM to fully capture the global semantics of the inputs.
To further enhance the ability to differentiate fine-grained semantics, we integrate a novel kernel into the Dynamic Time Warping framework. Leveraging the strengths of various discriminative pre-trained models, it performs dynamic sequence alignment for diverse embeddings tailored to specific contexts, thus addressing the inherent limitations fully utilizing input semantics. Through this explicit structure-induced constraint, our framework enables MLLMs to capture the global semantics and fine- fine-grained details of the input multimodal sequence more effectively, thus bridging the gap between generative and discriminative training paradigms.
2

---

Dynamic Sequence Alignment
Discriminative Loss Ld
lackual generated
simiarlyy between input sequence
prodicted simliarty
Generative Loss Lg
Tokes
generalle lood genteaain
output hidden states
interieaved sequences input
Figure 2: Our structure- generative and discriminative training joint training strategy.
Our method effectively balances both discriminative and generative tasks, demonstrating synergistic benefits. (i) Large-scale generative pre-trained models possess semantic-rich hidden states [41, 91, 231. which facilitate discriminative tasks like retrieval. Moreover, harnessing the capabilities of MLLM is crucial for complex discriminative tasks, such as interleaved image-text retrieval and fine-grained retrieval. (ii) By integrating discriminative tasks, the model's effectiveness in generative tasks, particularly within tasks requiring cognitive and discrimination abilities, is enhanced, thereby mitigating certain occurrences of hallucinations. (iii) We can employ Sugar to realize retrieval- augmented generation [2], eliminating the need for an off-the-shelf retrieval module [75], thereby amplifying the performance of various generative tasks. The usage of off-the-shelf retrieval presents a challenge wherein the retriever's performance affects the generator's 's final output [62]. This necessitates independent optimization of both components, posing a dilemma in selecting optimal configurations. However, our approach circumvents such optimization challenges.
Through extensive experimentation, generative tasks, Sugar establishes new ttate-of-the-an results on the tasks for complicated multi- modal comprehension tasks (i.e., DEMON [47]), fine-grained semantic distinctions (i.e., VizWiz [28], MME [95], object hallucinations detection (i.e., POPE [51]) (Section 4.2 and Section 4.3). For discriminative tasks, we achieved competitive results in image-text retrieval compared, and signifi- cantly surpassed CLIP in interleaved retrieval and fine-grained retrieval (Section 4.4). Furthermore, employing the strieval-augmentt generation (RAG) strategy led further improvements in a series of generative tasks (Section 4.5).
have demonstrated the effectiveness of our approach. For
2 Related Work
Multi-modal Large Language Models. Flamingo [3] and BLIP-2 [49] integrate LLMs with visual encoders, showcasing impressive zero-shot capabilities by aligning visual features with language representations. Building upon the advancements of LLa [55], subsequent stud- ies [103, 19, 94, 6, 42, 72, 95, 98, 45] propose fine-tuning MLLMs with multimodal instruction tuning data [102]. Recently, there has been surge in research [52, 80, 22, 21, 48] dedicated to enhancing the capacity of MLLMS to process interleaved image-text inputs effectively. However, these models primarily focus on generative tasks, overlooking the importance of introducing discrimi- native constraints. In this paper, we propose a couucture-nnduce joint training strategy for unifying generative and discriminative tasks, further enhancing the capabilities of MLLMs, especially those requiring cognitive and discriminative abilities.
Vision-Languagg Pre-training. Vision- Lanouage Pre- fre-training primarily come in two forms: single- stream and dual-stream. In singlc-stream models, the embeddings for the image and text modalities are concatenated and jointly encoded [39, 50], while in dual-stream models, they are encoded by separate mondality-speciffi encoders with optional cross- modality fusion [73, 31, 5]. These models have shown effectiveness in tasks such as classification and retrieval. However, they face challenges including difficulty in processing complex composed sequences [53, 54] and limited ability to discern detailed semantic differences [81, 79]. Recent attempts to utilize generative MLLMs for discriminative tasks have faced limitations, such as singular discriminative tasks [89]. weak discriminative task performance [40], poor generalization [59], and compromised generative capabilities [8].
LLMs for Retrieval. Early models for retrieval primarily focused on word representations [16, 64, 74], with minimal generative capabilities. Some recent works have endeavored to fine-tune generative pre-trained LLMs to generate discriminative embeddings, albeit at the expense of compromising the model's 's original generative capabilities [44, 70, 65, 63, 24, 71J. GRIT [66] integrates generative and discriminative tasks in NLP and demonstrates mutual benefits between them. However, its training cost is prohibitively high compared to individual tasks. Moreover, due to its specialized attention mechanism, the model can only be trained from scratch.
Generation. Retrieval-Augmennter Generation (RAG) 69], which har- nesses the advanced inference capabilities of L.MMs along with external knowledge, has the potential to significantly mitigate issues related to long-tail tail entities and reduce the occurrence of hallucina-
Retrieval-Augmente
3

---

tory responses [29, 36, 101, 77, 90, 92, 97]. Recently, there have also been related studies in the multimodal domain attempting to utilize retrieval augmentation [93, 96]. These methods typically require an additional retrieval module (e.g., CLIP), leading to component optimization challenges where the overall model performance is affected by the performance of the retrieval model, as well as concerns regarding the compatibility between the retrieval model and the MIL.Ms. Furthermore, retrieval modules like CLIP struggle to handle compositional or fine-grained scenarios, posing certain challenges for retrieval.
Method
As illustrated in Figure 3, initially introduce the problem formulation and offer overview of our structure-induced joint training strategy in Section 3.1. Subsequently, we delve into the specifics of dynamic sequence alignment algorithm in Section 3.2. Finally, we further introduce the Triple Kernel to aid in discriminating detailed semantics in Section 3.3.
3.1 Problem Formulation and Architecture Overview
interisseed sequence
image
Intiriaeeed Sequinon
coont
MLP
The Mao sky
projectorgl
projectond
wisue kokknn
loos Bio
VIT
Toduel bokkns
Sequee Sequencee
the prodoos
(a) Dynamic Sequence Alignment
below?
(b) Sugar Framework
Figure 3: (a) Dynamic Sequence Alignment. Semantically matched slices are connected with blue dashed line. The arrows indicate the direction of the ordered temporal alignment path. With these alignments, we can obtain the similarity between two interleaved inputs for training. (b) Sugar Framework. Sugar supports both multi-modal generation and retrieval simultaneously.
We view the interleaved image-text sequence the general format for input samples, where im- ages and textual data are alternately arranged. Typically, Multimodal Large Language Models (MILMs) [55, 52, 11, 6] tailored to generate text based on such input sequences, and it is conventionally optimized using self-regressive loss Lg. A special scenario arises when the input comprises only one image and question, prompting the MLLMs to generate an answer accordingly.
While intuitive, this optimization objective solely supervises text generation and lacks constraints on the hidden states of the entire interleaved sequence input. Additionally, the existing generative training framework struggles to fully distinguish imput semantics in certain contexts, such as discerning fine-grained object details. Consequently, it fails to adequately capture the global information or distinguish detailed semantics of the input samples.
Hence, we introduce a structure-induced constraint La (see in Figure 2), which explicitly imposes the semantic relationships between different input samples as an induced structural constraint on the hidden states of MLLMs, facilitating the model in capturing global semanties. We conceptualize the derivation semantic relationships between input samples as a Dynamic Sequence Alignment problem [67]. Additionally, we straightforwarddy select token in the hidden state of the MLLM to encompass all preceding input information, eliminating the need for training any specialized tokens.
To further effectively distinguish detailed semantics, we integrate a novel kernel into the Dynamic Time Warping framework. Leveraging the strengths of various discriminative pre-trained models. Combined with this newly proposed loss with hyperparameter a, the training objective can be formulated as:
L Ly aLs
3.2 Dynamic Sequence Alignment
We formulate the computation relationships within input interleaved sequences as a dynamic sequence alignment problem, and solve it by global alignment kernel. For two interleaved image-text sequence, each consisting of n and m images/sentences in total respectively (which we'll refer to as slices later on). We encode and normalize each slice, resulting in two sequences (æ1,..., Zx,) and

---

y= (y1,... (yi....., ym) all of which take values state space X. that is two elements of X def - X². In our setting, X' simply Rd, d refers to the feature dimension. We define the global alignment kernel as follows, and it has been proved to be positive-defintt under mild conditions and may prove more robust to quantify the similarity of two sequences [73, 68]:
k(x,y)= = E II rEA(xy)
e t E (0,
(2)
Following the suggestion by [18], we let Po I0E (IxL(E)) Mma(1)) + log(2 (1x Peatt) 20
is standard deviation, and can be calculated by 8v M for Ii, M/ in x,y, is fixed pre-defined hyperparameter and yp (i)) Mrz(()) is the distance between slice (i) and (ra(1) for an alignment (details for the definition of alignment can be seen in Appendix D.2).
/
Due to the causal attention mechanism, the token in hidden state of MILM can encapsulate infor- mation from preceding tokens in the sequence. Therefore, we directly utilize the last token d, of a sequence from the MLLM's hidden state and map it to the using an MLP to represent the entire in-context sequence. During training, we obtain a set of (r1, r2...., Tn) and their corresponding input sequence embedding set X2........ X,,) It is noteworthy that T; and rj (x; and xj) may originate from the same sequence but occupy different positions, thus enabling our method to utilize samples more efficiently.
Leveraging the GAK, we can derive the similarity matrix of (ri, r2,...,, Tn) and (x1, X2,..., Xn)
distinctively, denoted as M .M's R"*a For imposing the semantic relationships between different input samples as an induced structural constraint on the hidden state of MLLMs, we employ Mean Squared Error (MSE) loss aligned M with the label matrix M This approach eliminates the need for pre-defined label (i.e., positive and negative candidates) during training, allowing seamless integration into the aforementioned training framework (for specific training templates, please refer to Appendix E) Thus, we have the discriminative loss La:
L
(ml mly)²
i=1
Additionally, when both x and y contain only one slice, the computed result of the formula is mono- tonically increasing with the directly calculated cosine similarity (proof can be seen in Appendix 3). Therefore, in such cases, we simplify the computation by directly using cosine similarity. If r; and rj comes from the same input interleaved sample, we manually set mly = 1.
3.3 Detailed Semanties Modeling
To further effectively distinguish detailed semantics, we further propose the Triple Kernel (TK), positive definite kernel compatible with the previous framework. The TK leverages representations from diverse pre-trained discriminative models across uni-modal and cross-modal settings, harnessing their respective strengths. The definition is as below:
For two slice 6 E Rd, meets (i) la| = |6] = 2, d = d1 +d2 a = concat(a; (1, a2), 6 = concat(b,, b2).
bi E dd 02, b2 E IRda and |ai| [a2] = |b,| b2l = 1; or (ii) |a| = |b| = 1. We define tripe
kernel follows:
||a:
b, ||2
|a| [6] = 2 and b in uni-modal
p(a, b) =
||o
by |
|a| = [b] = 2 and a, b in cross- cross-modal else
(4)
|la b1
We prove triple kernel yp is conditionally positive-definite kernel defined on X x X R (Ap- pendix 2), aligning with the kernel definition in [18], thereby possessing its properties.
In practice, we let the feature dimension d = d1 + dy. For images, we employ DINOv2-base [68] and CLIP ViT-L/14 [73] for encoding, then concatenate the embeddings after normalization. For sentences, we utilize BGE--base [87] and CLIP ViT-L/144 keeping the dimension unchanged. By utilizing the Triple Kernel, we can fully leverage the strengths of these three models, effectively distinguishing detailed semantics.
5

---

penson kisses young person
Sugar
smashed
Sugar
Sugar
into
water and close the sand
socuut parson kisses an old person
sar serased into IERE
pemon water and in sand
Sensirivity Detailed Semanties
They are ougeteer on Halloween
Tell me the andraakk of Singapore
m
Martra Rey Sends oe
integrated resect
Sugge
A sirt the Middle Ages
Diing Marina Bay Singapoce.
World Knowledge
change
girl
pampkits
bey and girl
girl beids her feot
Cib Sugar
p Sugar
SuQUa
Sugar
promptt
what if weekling 1950u?
Indian wedling
kins
cutting the cake
9.0
SugAr
Supar
prompt
Multimodal Concept Composition
9.
went to really wenderf füt wedding
the bride and her brtalemmaidi looked wheolutely gorgeous
was pectins when
be goving ibair apporent
they made thair
wodkend
[nevic]
ond ticket
be has only
the porumn chroe waiting in the croual c exitemely excited.
she nibce holding
course was his muma
His mother poide
P.
aralussio
for someone
join lüm.
diploma..
Retrieval and Dialog
you deseribe editing process
image langet image??
Image:
Targol Emage:
add filter add contrast to everything change his eyes from blue to reddish brown
Fine-grained Image Discrimination
dancing.
the group was dressed and rawdy for the waiddig recepttion
during reseption, the best mon roasted the groom.
party into the might
groem on the floor for the first dance.
A mun read out wedding ceremony, and and bride exchanged rings.
P.. Sigar
Sugar
Retrieval
Figure 4: Selected examples for various image-text tasks. The pink background indicates retrieval results, while the blue background indicates generated results. More examples are provided in the Appendix F.2.

---

Method
L.M
Res.
VQA
GQA
VizWiz
SQA'
VQA"
POPE
MME
MME
MMB
LL
MM
BLIP-2 [49]
Vicuna-133 3B
224
41.0
41
42.5
85.3
1293.8
290.0
29.1
22.4
InstuatB11IP [1]]
Vicuma-7B
224
49.2
34.5
60.5
50.1
36
26.2
InstructBLIPP [19]
Vicuna-13B 3B
49.5
33.4
50.7
1212.8
291.9
25.6
Shikra [12]
Vicuna- 3B
224
77.4*
58.8
IDEFCCS 9B [42]]
LLaMA-7B 7B
224
50.9
38.4
35.5
25.9
48.2
IDEFICS-8BB [42]
LLaMA-65B
224
60.0
45.2
36.0
30.9
54.5
Owen [6]
Qwen 7B
448
78.8"
59.3*
35.2
67.1
63.8
38.2
Qwen- Qwen-VL-Chall VL-Chat [6]
Owen-7B
448
78.2"
57.5"
38.9
68.2
61.5
1487.5
360.7
60.6
LLaVA-1.5 (.54-155555) [55]
Vicuna-7B
336
78.5"
62.0"
50.0
66.8
58.2
85.9
1510.7
64.3
49.0
30.5
VII.A-7B [52]
1lama-2-78
79.9"
296.1
70.0
Sugar
Vicuna 7B
76.0"
58.7" 60.4
69.4
57.5
86.6
1550.8
300.0 64.9
75.6
31.3
Table 1: Comparison with state-of-the-an methods on 11 visual-Ianguage benchmarks. We mark the best performance bold and the second-best underlined. Benchmark names are abbreviated due to space limits. VQA-v2 [27]; GQA [35]; VizWiz [28]; SQA': ScienceQA-IMG [61]; VQAT: TextVQA [76]: POPE [51]; MME", MMEC: MME Perception, MME Cognition [95]; MMB: Bench(In-thee Wnnthn-Wilddd---eeeeeee Wild)-Detail [56]; MM-Vel Vet [99]. indicates the training images the datasets are observed during training.
MMBench [58]; VL Wa,
Experiments
To assess Sugar's generative ubility, we conduct a comprehensive comparison with state-of-the-ar models on 11 commonly used vanual-language benchmarks in Section 4.2. Furthermore, we evaluate more complicated multimodal comprehension tasks on DEMON with 29 datasets in Section 4.3. For discriminative tasks, we compare performance across three different retrieval tasks: image-text retrieval, interleaved retrieval, and fine-grained retrieval in Section 4.4. Subsequently, we leverage Sugar's discriminative ability for reurieeal-uummented generation compared with common used retrieve module in Section 4.5, Finally, we conduct ablation experiments to analyze the effectiveness of our method in Section 4.6.
4.1 Setup
We apply our method to VILA [52], recent state-of thetrr-rtt MLLM supporting interleaved input. We further fine-tune VILA using LoRA [30]. Details about the experiments setting, datasets and the instruction examples, please check in Appendix E.
4.2 Multimodal Comprehension on 11 Benchmarks
We conduct a comprehensive comparison with sfate-of-the-art models 11 commonly used bench- marks, as shown in Table 1. Compared to existing models, Sugar achieves remarkable improvements over the second-best performing model on tasks requiring fine-grained semantics (i.e., LLaVA [56], VizWiz [28], SQA [61] improve by 8%, 4.5%, 1.8% respectively) and benchmarks for detecting hallucinations (i.e. POPE [51]), while maintaining competitive results in other tasks. Notably, Sugar excels in discriminative tasks and still achieves 5 state-of-thhe-s results and 3 second-best results on 11 benchmarks for generative tasks, even outperforming some models larger than 7B. Our results demonstrate the benefits of incorporating the discriminative loss, aiding in fine- fine-grained semantic tasks and reducing hallucinations.
4.3 Complicated Multimodal Comprehension DEMON
Table 2 demonstrates the superior performance of Sugar on the DEMON benchmark, which comprises 7 categories and a total of 29 sub-tasks. These tasks are considerably more complex than the previously used 11 common benchmarks. DEMON is tailored to evaluate the capacity of models and systems to understand demonstrative instructions that include multiple, interleaved, and multimodal contexts, presenting the essential information needed to complete a task. Sugar surpasses the previous state- of-the-art model on the DEMON benchmark, VPG-C [47], across of 7 categories. For example, we achieve performance improvements of 36.1% in Tixt-Rich Images QA (TRQA) tasks and 17.2% in Visual Relation Inference (VRI) tasks, both of which require detailed semantics, compared to the second-ber best performing model. This underscores advanced ability to associate interleaved
7

---

LTM
MMD
VST
VRI
MMC
KGQA
TRQA
MMR
OpenFlamingo
MPT-7B
16.9
24.2
13.9
21.7
32.0
30.6
41.6
BLIP-2 [49]
Vicuna-13R
26.1
21.3
10.7
17.9
39.2
33.5
39.7
InstrucBBLIP [19]
Vicuna 7B
33.6
24.4
11.5
21.2
47.4
44,4
48.6
MiniGiPT-4 [103]
Vicuna-7B
13.7
17.1
8.0
16.6
30.3
26.4
43.5
LLaVA [56]
Vicuna: 7B
7.8
10.7
8.3
15.9
36.2
28.3
41.5
(oPlug-OO]]94
LLaMA-7B
12.7
19.3
5.4
16.3
33.3
32.5
42.5
VPG-C [47]
Vicuna-7B
37.5
25.2
25.9
22.2
48.6
44.9
50.3
VILA-7B [52]
Vicuna-78
47.8
25.8
13.2
17.2
60.1
50.5
Sugar
Vicuna-78 78
51.8
34.3
32.3
16.8
64.4
68.9
51.7
Table 2: Comparision with ofate-of-the-ar method on DEMON [47] benchmark.
VIST
(a) MSCOCO
Model
Inputs
Rø
R@s
R 10
to Winarrunnd
Model
R611 RAP5
R@10
CLIP VIT-L/M
Sc
5.9
19.5
28.0
FROMAGRe())
23.4
Image
59.0
Sugar
FROMAGe
Sc
Sc
11.9
10.1
23.8
26.3
31.7
UNITER
35.0
18.5
2.5
10.3
14.5
1.3
FROMAGerg&d)
23.4
47.2
58.0
MLLASs
37.0
13.25
10.0
Sugar
22.0
49.1
63.1
BLIP
Sc
6.2
16.8
23.4
933
CLIP VIE-L/14
Sc
8.8
22.3
29.8
FROMAGo(d)
Sugar
FROMACK(SH)
Image + Test
2688
26.4
25.6
52.3
52.4
63.6
63.4
Sugar
FROMAGE
FROMAGe
VIT-L/14
Sc441
Sca4i
5c
Sc
13.2
11.0
18.2
27.3
28.5
42.7
51.8
36.7
37.0
FLANA/TA
Sugar
MI BERTERT
CLIP VIE
12.3
25.5
30.8
11.5
20.5
10.5
27.1
14,3
90
4.8
5.0
Sugar"
Sc44i
21.9
46.7
59.2
Table 3: Retrieval results compared with previous models, reported by Recall@k for (a)(b) and Accuracy (%) for (c). (a) MSCOCO for image-text retrieval: FROMAGe(d) indicates the FROMAGe model pre-trained only with discriminative loss, and FROMAGeeg++d indicates joint training with both discriminative and generative losses. (b) VIST for interleaved retrieval: indicates retrieval over images not previously seen in the story sequence. "5c+4i" is shorthand for captions and images, and "5c" is shorthand for 5 captions. (c) Winoground for fnneggraieed retrieval.
text-image inputs for stronger in-context understanding, and Sugar's strong capability capture global semantics in interleaved sequences, facilitated by joint training with discriminative loss.
4.4 Zero-shot Cross-modal Information Retrieval
Image-text Retrieval. We evaluated the performance of Sugar the widely adopted MSCOCO [38] dataset in the context of a standard image-text retrieval task. Sugar demonstrated comparable performance to FROMAGe [40] in R @1 and surpassed it in R@5 and R@ 10, highlighting Sugar's
superiority in normal retrieval tasks.
Interleaved Retrieval. To assess the proficiency of Sugar in processing multimodal contextual information, we evaluated its performance in retrieving relevant images conditioned on sequences of interleaved image-text inputs from the Visual Storytelling (VIST) dataset [32]. We conducted evaluations across several experimental configurations, following the same setup as FROMAGe [40] (see Appendix F.1). Our results show that Sugar outperforms FROMAGe in most settings, particularly achieving 20.3% improvement in the 5c+4i configuration, significantly surpassing both CLIP and BLIP-2. This demonstrates that method effectively leverages MILMs' ability to handle complex interleaved sequence inputs, thereby achieving superior retrieval performance.
Fine-grained Retrieval. We tested fine-grained retrieval using the Winoground dataset [79], which evaluates the ability to perform vision-linguistic compositional reasoning. Surprisingly, Sugar outperformed all discriminative pre-trained models (both single-stream and dual-stream encoder architectures), achieving improvements of 5.3%. 77.1%, and 86.2% over the second-best model in the Text, Image, and Group dimensions, respectively. This demonstrates Sugar's 's strong capability to distinguish detailed semantics and performing compositional reasoning.
4.5 Retrieval-Augmeente Generation
Due to Sugar's dual capabilities in both discrimination and generation, we can achieve retrieval augmentation without the need for an additional retrieval module. For performing retrieval-augmeened generation (RAG), we selected two tasks, namely VizWiz and SQA!, as they offer held-in data that were not seen during model training. We utilized a mixed set comprising the widely-used VA-1.5 SFT subset and the held-in datasets of the two tasks as the knowledge base and employed different

---

(a) Retrieval--uueee -utieeeall-ugmeetteed Generation
(h) Ablation Study
VizWiz
SQA'
Generative Tasks
Discriminative Tasks
1.3
50.0
06.8
SQA'
POPE
KGQA
MSCOCO
VIST
Winoground
+CLIP 12.66-4.88)
62.0
(-8.6%)
+BLIP2
14.0%)
62.5
(-64%)
Sugar
72.6
$6.6
64.4
49.1
46.7
36.3
VILA
578
61.1
""llo dati,
72.8
"55.3
61.7"..::
49.6
"""31
+CLIP
(-14.7%)
65.7
(+0.650
wio data,
68.0
86.4
62.5
46.0
40.7
33.5
+BLIP2
19.5 (- 14.2%%
66.1
2+206%)
w/o GAK
72.1
86.0
61.1
48.2
33.5
29.3
Sugar
60.4
69.4
71.6
IRAG
61.9 (+2.55)) 71.9 (+33665)
w/o ArgPool
72.4
86.2
64.6
39.7
38.1
31.5
Table 4: (a) Retrieval-Augmeenter Generation. (b) Ablation Study. For MSCOCO, we report the R R@5 in text-to-image retrieval. For VIST, we report the @@5 of retrieving an image given 5 captions and images. For Winoground, we report the Image score. For other tasks, we report Accuracy (%).
retrieval modules to retrieve relevant knowledge for the MILM. The results are as follows: (i) We observed drop in performance for LLaVA-1.5 with RAG in all tasks. This may be because L. designed solely for single-image input, without the ability to utilize in-context external knowledge. (ii) Compared to VILA, Sugar's performance improved in both tasks, whereas VILA improved in SQA' but decreased in WizWiz. These findings suggest that Sugar's 's retrieved knowledge is more beneficial, while the knowledge retrieved by CLIP and BLIP-2 may hinder performance.
4.6 Ablations
Importance of Both Tasks, (1) w/o data,: As shown in Table 4, when we reduce the amount of data for discriminative tasks (Row 2), there are performance drops of 1.5% in hallucination detection tasks (i.e., POPE) and 4.2% in interleaved multi-modal comprehension tasks (i.e., KGQA). (2) w/o data,;: Similarly, reducing the data for generative tasks (Row 3), the performance on generative tasks declines with a 10.1% decrease in VIST, which requires global semantics capturing, and a 4. % decrease in Winoground, which necessitates fine- one-grained semantic understanding. This indicates that generative and discriminative training can mutually benefit each other.
When we exclude the Global Alignment Kernel (GAK) (Row 4) and resort to using the average similarity for the slices, a notable decrease in performance is observed across several interleaved image-tex tasks (i.e., a 5.1% decrease in KQGA and a 28.3% decrease in VIST). This underscores the fundamental role of GAK in aiding Sugar to capture global semantics effectively. (2) w/o TK: Upon removal of the Triple Kernel (TK) (Row 5) and utilization of CLIP for encoding the input sequence instead, a dramatic performance decline evident in Winoground, with 43.5% decrease. This underscores the significant role of TK in facilitating the distinction of detailed semantics. (3) w/o AvgPool: When solely using the last token for retrieval, general decline performance is observed across discriminative tasks, with decreases of 19.1% % for MSCOCO, 22.6% for VIST, and 13.2% for Winoground. This phenomenon may be attributed to the last token of an image often corresponding a pooling token, containing relatively weaker semantic information. Utilizing all image tokens and performing AvgPooling tends to yield greater improvements in retrieval tasks.
Effectiveness of Individual Components. (1) w/o GAK
5 Conclusion
Visiua-Language Models (VLMs) have been trained using both generative and discriminative paradigms, each with distinct advantages and limitations. To bridge this gap, we introduce Structure- induced approach to unify generative and discriminative paradigms, which imposes semantic rela- tionships between input samples, thereby enhancing the MILM's ability to capture global semantics and distinguish fine-gra ane-grained details. This approach effectively balances generative and discriminative tasks, yielding synergistic benefits. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-at results in multiple generative tasks, particularly those requiring cognitive and discrimination abilities, while also demonstrating competitive performance in discrim- inative tasks such as image-text retrieval and achieving state-of- oh-thh--rrt results in interleaved and fine-grained retrieval. Furthermore, employing retrieval-augmmenee generation strategy within single model leads to additional improvements, offering a promising direction for future research.
Acknowledgment. This work has been supported in part by the Key Research and Development Projects in Zhejiang Province (No. 2024C01 106, 2024C01028), the NSFC (No. 62272411) I), the National Key Research and Development Project of China (201 2018AAA0101900), and Ant Group.

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Allenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint MaXi::3333008774,, 2023.
[2] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022. 3
[3] Yean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[4] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.011 200223008.0990, 31390, 2023.
[5] Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Gun- Chen, Xiangtai Li, Zhen Dong. Lei Zhu, and Shuicheng Yan. Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis. arXiv preprint arXiv:2410.08261, 2024. 3
[6] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile lision-aanguage model for understanding, localization, text reading, and beyond. arXiv preprint, 2023. 3, 4,
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey, arXiv preprint 2rXiv:2404.189330 2024.
[8] Oriol Barbany, Michael Huang, Xinliang Zhu, and Arnab Dhua. Leveraging large language models for multimodal search. arXiv preprint arXiv:2404. 15790, 2024. 2,
[9] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. Coyo-700m: Image-text pair dataset, 2022. 20
[10] Yinghsan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. WebQA: Multihop and Multimodal QA. 2021. 20,30 30
[11] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghura- man Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt- v2: Large language model as a unified interface for vision-language multi-task learning. arXiv:2310.09478, 2023. 19
[12] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint 2306522306.55195, 2023. 7
[13] Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Pan, Zonggang Yuan, and Huajun Chen. Zero- shot visual question answering using knowledge graph. In The Semantic Web-ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24-28, 2021, Proceedings 20, pages 146-162 166-162. Springer, 2021. 30
[14] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 19
[15] João Coelho, Bruno Martins, João Magalhaes, Jamie Callan, and Chenyan Xiong. Dwell in the beginning: How language models embed long documents for dense retrieval. arXiv preprint 4XXv:2444.041633 2024. 2
10

---

[16] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Su- pervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv: 1ae::/005.0000 02364, 2017. 3
[17] Marco Cuturi and Mathieu Blondel. Soft-dtw: differentiable loss function for time-series. In International conference on machine learning, pages 894-903, PMLR, 2017. 17
[18] Marco Cuturi, Jean- ShanPPhllippe Vert, Oystein Birkenes, and Tomoko Matsui. A kernel for time series based on global alignments. In 2007 IEEE International Conference on Acoustics, Speech and Signal
volume 2, pages 11-413. IEEE, 2007. 5, 17
[19] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024. 7.
[20] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint wrXiv:2309..7445 2023. 22
[21] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu, fitho-of-though:: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning, 2024. 3
[22] Hao Fei, Shengqiong Wu, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision I1m for understanding, generating, segmenting, editing. 2024. 3
[23] Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024. 3
[24] Minghe Gao, Juncheng Li, Hao Fei, Wei Ji, Guoming Wang, Wenqiao Zhang, Siliang Tang, and Yucting Zhuang. De-fine: Decomposing and refining visual programs with auto-feedback. arXiv preprint arXiv:2311.12881
2023.
[25] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jía, Jinliu Pan, Yuxi Bi. Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmeent generation for large language models: A survey, arXiv preprint arXiv:2312.10997, 2023.
[26] Zhiqi Ge, Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang, Siliang Tang, and Yueting Zhuang. Worldgpt: Empowering Ilm multimodal world model. arXiv preprint arXiv:2404. 18202, 2024. 24
[27] Yash Goyal, Tejas Khot, Douglas Summers- Stay. Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2017.1 7, 7,20
[28] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018. 3, 7,20
[29] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929-3938. PMIR, 2020. 4
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: /06.02200000685, 2021. 7,20 20
[31] Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with unified transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1449, 2021.

---

[32] Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadch, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He. Pushmeet Kohli, Dhruv Batra, al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1233-1239, 2016. 8, 20
[33] Xiaohua Huang, Abhinav Dhall, Roland Goecke, Matti Pietikainen, and Guoying Zhao. A global alignment kernel based approach for group-level happiness intensity estimation. arXiv preprint arXiv: / 1109/80903333 03313, 2018. 2
[34] Xuanwen Huang, Wei Chow, Yang Wang, Ziwei Chai, Chunping Wang, Lei Chen, and Yang Yang. One graph model for cross-domain dynamic link prediction. arXiv preprint
arXiv:2402.02168. 2024. 19
[35] Drew Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019. 7, 20
[36] Wei Ji, Renjie Liang, Zhedong Zheng, Wenqiao Zhang, Shengyu Zhang, Juncheng Li, Mengze Li, and Tat-seng Chua. Are binary annotations sufficient? video moment retrieval via hier- archical uncertainty-baase active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2023. 4
[37] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 7(3):535-547, 2019. 23
[38] Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image de- scriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137, 2015, 8, 20, 23
[39] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Laainn-and-annguage transformer without convolution region supervision. In International conference machine learning, pages 5583- PMLR, 2021. 3
[40] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models images for multimodal generation. arXiv preprint 1xXiv:2301.18822 2, 2023. 2, 3, 8, 23, 24
[41] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint 1.X.::2338000022.. 2023.
[42] Hugo Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang. Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, al. Intro- ducing idefics: An open reproduction of state-of-the-are visual language model, 2023. URL https//huggingfuce co/blog/idefics. Accessed, pages 09-18, 2023. 3,7
[43] Hector Levesque, Emest Davis, and Leora Morgenstem. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. 29
[44] Chaofan Li, Zheng Liu, Shitao Xiao, and Yingxia Shao. Making large language models better foundation for dense retrieval. arXiv preprint arXiv:2312.15503, 2023.
[45] Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. Gradient-regulated meta-prompt learning for generalizable vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2551-25631 2023. 3
[46] Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems, 35:7290-7303, 2022. 2
[47] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal ulss to follow zero-shot demonstrative instructions. In The Twelfth International Conference on Learning Representations, 2023. 2, 3, 7, 8, 20,
12

---

[48] Juncheng Li, Siliang Tang, Linchao Zhu, Wenqiao Zhang, Yi Yang, Tat-Seng Chua, Fei Wu, and Yueting Zhuang. Variational cross ross-graph reasoning and adaptive structured semantics learning for compositional temporal grounding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12001-122 2023.
[49] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023. 3, 7
[50] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021. 3
[51] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint urXiv22305.10355 2023, 3, 7, 20
[52] Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
2, 3, 4, 7, 8, 19, 20, 22
[53] Weizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca, and Bill Byrne. Fine-graine grained late-interaction multi-modal retrieval for retrieval augmented visual question answering. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 1,3
[54] Weizhe Lin, Jingbiao Mei, Jinghong Chen, and Bill Byme. Prefimr: Scaling up fine-grained late- iite-interaction multi-modal retrievers. arXiv preprint, (aXXiv:4022.08327) 2024. 1,3
[55] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023. 1, 3, 4, 7, 19,20 20
[56] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 7, 8, 20
[57] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
[58] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06220
2023. 7,20
[59] Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Rar: Retrieving and ranking augmented milms for visual recognition. arXiv preprint 2rXiv:2403.13800 2024. 2,
[60] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05100 2017. 20
[61] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022. 7, 20
Chun Zhu, Oyvind
[62] Yang Luo, Zangwei Zheng, Zirui Zhu, and Yang You. How does the textual information affect the retrieval of multimodal in-context learning? arXiv preprint 2xXi::2404.12666 2024. 3
[63] Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. arXiv preprint arXiv:2110.083/9, 2023. 3
[64] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013. 3
13

---

[65] Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904 2022.
[66] Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.0990 2024. 3
[67] Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pages 69 84, 2007. 2, 4, 17
[68] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07799 2023. 5
[69] Kaihang Pan, Zhaoyu Fan, Juncheng Li, Qifan Yu, Hao Fei, Siliang Tang, Richang Hong, Han- wang Zhang, and Qianru Sun. Towards unified multimodal editing with enhanced knowledge 2024.
collaboration. arXiv preprint arXiv:2409.
[70] Kaihang Pan, Juncheng Li, Hongye Song, Hao Fei, Wei Ji, Shuo Zhang, Jun Lin, Xiaozhong Liu, and Siliang Tang. Controlretriever: Harnessing the power of instructions for controllable retrieval. arXiv preprint arXiv:2308. 10025, 2023. 3
[71] Kaihang Pan, Juncheng Li, Wenjie Wang. Hao Fei, Hongye Song, Wei Ji, Jun Lin, Xiaozhong Liu, Tat-Seng Chua, and Siliang Tang. 13: I ntent-i ntrospective retrieval conditioned on i nstructions. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1839-1849, 2024.
[72] Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-tokens for multimodal 11m. arXiv preprint arXiv:2405.. 01926, 2024. 3
[73] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021. 1, 5, 19, 22
[74] Nils Reimers and Iryna Gurevych. Sentence-bert, Sentence embeddings using siamese bert- networks. arXiv preprint arXiv: 10084, 2019. 3
[75] Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrieval-augmente black-box language models. arXiv preprint arXiv:2301. 12652, 2023.
[76] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang. Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8322 2019. 20
[77] Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, and Mike Bendersky. Quill: Query intent with large language models using retrieval augmentation and multi-stage distillation. arXiv preprint arXiv:22 20. 18718, 2022.
[78] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023. 22
[79] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022. 1, 8, 20, 24
[80] Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang. Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, Hongsheng Li, Yu Qiao, and Jifeng Dai. Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint S.ti:22011.12000 2024.
14

---

[81] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. arXiv preprint arXiv:2401.06209, 2024. 2, 3, 21
[82] Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305. 2023.
2
[83] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence, 40(10):2411 2427, 2017. 30
[84] Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. Uniir: Training and benchmarking universal multimodal information retrievers. arXiv preprint arXiv:2311. 171166, 2023. 23
[85] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization in multimodal 1lm. arXiv preprint arXiv:2406.055 05127, 2024.
[86] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. In Proceedings of the International Conference on Machine Learning, 2024.
[87] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[88] Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309. 16671, 2023. 22
[89] Chenglin Yang, Siyuan Qiao, Yuan Cao, Yu Zhang, Tao Zhu, Alan Yuille, and Jiahui Yu. Ig captioner: Information gain captioners are strong zero-shot classifiers. arXiv preprint arXiv:2311.17022 2023. 2,
[90] Nan Yang. Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv:2304.0444 2023. 4
[91] Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.172440 2023.
[92] Zhuolin Yang. Wei Ping. Zihan Liu, Vijay Korthikanti, Weili Nie, De-An Huang, Linxi Fan, Zhiding Yu, Shiyi Lan, Bo Li, et al. Re-vilm: Retrieval-augmmnted visual language model for zero and few-shot image captioning. arXiv preprint arXiv:2302.04858, 2023.
[93] Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovee, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmente multimodal language modeling. arXiv preprint arXiv:2211.125611 2022.
[94] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023. 3,8
[95] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. arXiv preprint arXiv:2306.13549 2023. 3, 7, 20
[96] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.0059// 2(3), 2023. 4
15

---

[97] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang. Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[98] Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang, Wei Ji, and Yueting Zhuang. language model for fine-graines scene graph generation in open world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21560-21571, 2023. 2,
[99] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin. Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabili- ties. arXiv preprini arXiv:2308. 02490,, 2023. 7, 20
[100] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-tra vre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1975-11986, 2023. 22
[101] Wenqiao Zhang, Jiannan Guo, Mengze Li, Haochen Shi, Shengyu Zhang, Juncheng Li, Siliang Tang, and Yueting Zhuang. Boss: Bottom-up cross-modal semantic composition with hybrid counterfactual training for robust content-based image retrieval. arXiv preprint arXiv:2207.04221, 2022. 4
[102] Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, et al. Hyperllava: Dynamic visual and language expert tuning for multimodal large language models. arXiv preprint arXiv:2403. 13447, 2024.
[103] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023. 3, 8
[104) Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024. 20
16

---

A
Broader Impact
The broader impact of Sugar, carries both potential benefits and risks upon deployment and release. Some considerations unique due their visual nature, while others mirror existing instruction- following Large Language Models (LLMs). Built upon Vicuna, CLIP. DINOv2, and BGE, Sugar inherits issues associated with LLMs and vision encoders. Below, outline risks and mitigation strategies for its release.
Hallucination.
Similar to other MLLMs, Sugar may generate outputs detached from facts or input data, posing concerns, especially in critical applications like medicine and the field related security.
Biases. Bias from base models can transfer Sugar, originating from both the vision encoder (CLIP)
and the language decoder (Vicuna), potentially leading to biased outcomes or unfair representations.
Ethical Impacts. This study doesn't 't raise ethical concerns, as it doesn't It involve subjective assess- ments or private data, only utilizing publicly available datasets.
Expected Societal Implications. significant societal concern lies in potential misuse, such as fabricating unauthorized texts leading to misinformation, privacy breaches, and other damaging consequences. Strong ethical standards and ongoing surveillance are essential for mitigation.
These issues aren't 't unique to our method but are prevalent across different techniques for multi- concept customization. Despite the risks, believe the benefits outweigh the potential harm, allowing ongoing investigation and improvement of the model while engaging the community in developing better mitigation strategies. Moreover, its release foster new applications and research directions, contributing to the progress and responsible deployment of foundation models in vision- language tasks.
B Limitations
(i) Our method, while effective, may inherit limitations from the underlying models, such as halluci- nation in generating outputs detached from facts or input data and potential biases originating from the model we used. (ii) The training data might inevitably contain mismatched image and text, which could adversely affect training.
C Mathematical Proof
Theorem 1. The alignment kernel K can be computed in quadratic complexity, namely in O(mnd?)
iterations. where m, n denotes the length of two sequence and their hidden dimension all is m, m,n, d E R.
Proof. Given x = (æ1...., In) and y (VI...., Vm) two sequences of X*, we set the double- subscripted series Ma, as M4.0 0 for 1, 1,... n, Mo., 0 for j 1,..., m. and Mo,0 1. Computing recursively for (i.j) E {L,..., n} x {1, m} the terms
Mi (M.j-1 M1-18-1 + M6-1,))(ZE v,)
obtain that ((x,y) M Mn The result be proved by recursion and is intuitively an equivalent of the Dynamic Time Warping(DTW) [67, 17] algorithm where the max-sum algebra is simply replaced by the sum-product one [18].
Theorem 2. triple kernel a conditionally symmetric positive-deffnite kernel [18] defined on X x X R.
Proof. (i) for two slice a, b Rd, meets |a| |b| 2, + d2, concat(a az), concat(bi, 62), ar bi dd 0g by E Id and [a] |ag]
[b2 1:

---

let a' concat a2 d2), a' - concat(b/ 82 b)): when a'(b") is from text modal, we let a' concat(a, a2, (b concat(b), b2. 0)), and a' concat(a,, 0, a2) concat(b), b2)) for image modal. _d we can unify the Equation 4 first and second case in:
pla, b) ,p(
(1a2116g1 [dgl1b)111a - K11l2 + (1 (221 b2112
+ +1 (ag1---11661111 dp - k/l1
2 (122116g bΔه/И/Л11 - 1/11 +0+0
20
As (1 | (62 2 and (1 la's ([(] 2 0. for any family a1, a2,, X' and 6₁, C2, On R,
we have that
Σ (.cp((m). = Σ ccyp(as, 20
(ii) for two slice meets have that
|a| = |b| = 1: for any family on, On E and cı, 0,, Cn. E R, we
Σ Cecp(())) = ccylla Eeaglla xyl1
Additionally, it's evident that for both (i) and (ii), ((a, b) (b, a). Therefore, triple kernel y is conditionally symmetric -eitive-definite kernel defined on X' x X - R
Theorem 3. when both x and y contain only one slice, GAK is monotonically increasing with the directly calculated cosine similarity.
Proof. let x = (x),, = (v), and cosine similarity and is cos = cos < 2. y >. we can get
o = 81 M N
81 2 8
and we have ((a, |a|2 + 16/2 2cos < a, >= 2(1 - cos < a, a, >), thus:
( (1 20
Po 202 (x)1(6) + +og
2 -
v (1)211)))-2(0)) (+2(())
(Im Min(()) + log
2 -
262
28
cos a.
+ log - ousge,bb2
8²
Letting
I-ens<a.b> 82
and substituting the result of po = t +ogg22 - e-1) into Equation 2, we
obtain:
K(x,y) Σ
tEAlxy)
e - do
e -tooo 2-
-t
e-
Letting et, we can further obtain:
8 K K(x,y) 2 - 8
18

---

As (as b) E 11,,1], strictly increases with cos(a, b) and 8 E [e- 8 when the hyperparameter is fixed. Derivative K y) can be obtained:
y)x 2-st8
2
0
s2-
s2
Overall, when both x and y contain only one slice, GAK is monotonically increasing with the directly calculated cosine similarity.
D Method Details
D.1 Architecture Details
We adopt the manifold multimodal model architecture [55, 52, 11, 6, 34], formulated as follows:
Visual Representation. We first process Zimg subject to a visual representation backbone V. that outputs sequence of features Ping R Lx hvsioo where Pung be the patch features output by Vision Transformer.
As an example, Ping might V (Famg).
Vision-Language Projector. Next, we map Ping to a sequence of embeddings Cimg E R via a learned projector Fwb, where Cing F& (Pimg).
Language Model. Finally, we concatenate the sequence eime with the text prompt embeddings Cprompt embed (uome(upropp). prommp)), passing the result the language model. Generally, we have the interleaved
image-text input Timput by concatting all the Eprompt and Eing- The language model generates output text Ugen LMg (Muppu)
Retrieval Projector. For discriminative tasks, we select the token de from MLLM's hidden state and map it to т₃ via learned projector FG..
In Implementation, we utilize CLIP VIT-L/14 [73] the visual encoder, and Vieuna 1.5 [14] the language model.
D.2 Sequence Alignment
An alignment ı of length |x] p between two sequences and y is pair of increasing p-tuples (T1, x2) such that
m -.. (p)) n, क2 (1) 5 = m,
(5)
We write A(x. y) for the set of all possible alignments between and y. Intuitively, an alignment π between and describes a way associate each element sequence x to one or possibly more elements in y, and vice versa. Such alignments can be conveniently represented by paths in the n x m grid displayed in the left of Figure:
with unitary increments and no simultaneous repetitions, that is v1 <isp-1
T (i + ())) +1,
m2( +1
(T (i +1 - (T2(1 (1) 2 1.
(6)
The score on path is defined as:
|=|
S(T)
p (1) (i))11))))))
(7)
19

---

E Experimental Details
E.1 Datasets
Training Data. Our vision-language task datasets are a subset of VILA [52], including MMC4 [104], COYO [9], LLaVA-1.5 SFT dataset [55].
We use a prompt template formatted as (system-message a system prompt from Vicuna, and the following messages all have the same meaning.):
(system-message)) USER: <image>\n {question). ASSISTANT: (answer).
For interleaved vision-language datasets, the template formatted
(system-messagg)) USER: (interleaving question). ASSISTANT: {answer).
Training Strategy. To jointly train the discriminative loss and generative loss, we calculate the loss as follows. Since the last token of an image is often a padding token, we take all 576 hidden state tokens before the LM head for images and apply average pooling to obtain a single token. For text, we directly take the last toke in the hidden state of MILM.
During training, We calculate the discriminative loss using the last token from either the end of the text or the image in the MLLM's hidden state. Notably, in an interleaved input sequence with multiple texts images, randomly select multiple last tokens from the same sequence more efficiently utilize the samples.
Evaluation Data. For generative tasks, we first evaluate on wide range of question-ant question-answwriigg tasks and some MLLM-oriented comprehension benchmarks, including VQA-v2 [27], GQA [35], VizWiz [28], ScienceQA-IMG [6 IJ. Tex [76], POPE [5 1],, MME [95], MMBench [58], LLaVA- Bench (In-the-Wiil [56], MM-Vet [99] and. The split of test sets and the evaluation metrics are aligned with those described in VILA[52] and LLaVA [55].
To test the generative ability in interleaving tasks, we use DEMON [47], a comprehensive benchmark that demonstrative instruction following ability, including wide variety of multi-modal datasets from different fields and scenarios.
For generation tasks, our evaluation encompasses MSCOCO [38] for image-text retrieval, Visual Storytelling (VIST) [32] for interleaved retrieval and Winoground [79] for fine-grained retricval.
F.2 Training
We train the parameters for both the LLM and the MLP for embedding the MLLM's hidden state, initializing from VILA [52]. To enhance efficiency for the LLM, we employ LoRA tuming [30] on the W, and W, matrices using low-rank adaptation. In our implementation, we set the rank r = 128 and 256. We utilize the AdamW optimizer [60] in conjunction with a cosine learning rate scheduler.
The hyperparameters for the AdamW optimizer are configured with warm-up ratio of 0.03 and maximum learning rate of le Training is conducted on 8 A800 GPUs for approximately 12 hours.
E.3 Introduction Experiment Details
(a) WebQA. The original WebQA contains two types of questions: "Qcate": "text" (open-ended questions) and "Quate": "YesNo" (binary judgment questions). For ease of evaluation, we only used the second type. We selected 500 samples of the "YesNo" question type from WebQA [10], each containing one relevant image-text pair and five unrelated image-text pairs. Since the original data provides responses in declarative sentences, we modified the answers of these samples to be either "yes" or "no" by prompting with "please answer the question in Yes or No."
20

---

We transformed this dataset into prestion-answerie format. Each question takes the following form (Due to display problems, have performed line breaks, the same below.):
(system-message)) USER: {qustion} please answer the question Yes or No. ASSISTANT: (answer).
pairs}. An
Here case for one sample in Figure 5:
g
(sysiem-mossago).
professional basketball
during
scason?n
Absolute Hody Control Blackfield 2015 a3 Belgian project Absolute Body Control at the Blackfield festival 2015 in Gelsenkirchen'' Germany'n
Miranda Cosgrove, Steve Carell, Australian premiere, Despicable
Minanaa Cosgrove and Steve Carell Despicable Me red curptt movie premiere at Event Cinemas, Boedi Junction, Sydney, Australia,
mike scort 2015 Mike Scott of the Hawkss, making a lay-up against the Washington Wizards at the Verizon
Madonaa Rebel Heart Tour Rebel Heart Tourin
useful
Canniball Corpse Rockharz The American death metal band Cannibal Corpe at 25th Rockharz Open
David Allan Coe David Allan Coe, performing 2008
Germanylss
Yes
Sugar
Figure 5: A Case for WebQA. The index for the usefull pair is three.
In WebQA, the accuracy roughly forms "U" shape curve when the relevant image-text pair for question appears at different positions. While Sugar also shows similar trends, it tends to be more stable overall. Specific numerical results can be found in Table 5.
Index
VILA
50.0
49.0
47.4
44.4
44.4
49.0
Sugar
63.8
60.4
59.6
59.2
60.4
61.0
Table 5: Specific accuracy (%) values displayed WebQA. The index indicates the position of the useful image-text pair, denoting which position it occupies in the sequence.
(b) MMVP-VLM Benchmark. MMVP-VLM [81] contains 30 carefully annotated images in each dimension of capability, with pairs of images being highly similar each other (as indicated by their high similarity scores in CLIP). To evaluate the discriminative ability of generative models on these finely nuanced images, transformed this dataset into a question-ansee answering format. Each question takes the following form:
Esystem-measagee) USER: First Image: <image> InScond <inge>>>magg>>
which choice meets the first image:
["Statteentt2")) ("aaa A. (dataaa ("data("Stttmment""]] (data
please answer in A or
ASSISTANT:
{answer).
Among them, both Statement and Statement 2. as well as Image and Image 2, are highly similar, with only subtle differences. Furthermore, there corresponding relationship between Image and Statement 1, and between Image 2 and Statement 2. We employed random seed to ensure that the correct answer is equally distributed between option and option B. The specific values for the experiment are provided in Table 6.
21

---

Image
Q
3
1
Average
Size
OpenAl VIT-L-144
224
13.3
13.3
20.0
20.0
13.3
53.3
20.0
6.7
13.3
19.3
OpenAI VIT-L-14477
336
0.0
20.0
40.0
20.0
6.7
20.0
33.3
6.7
33.3
20.0
SigLIP VIT-S0-14 [100]
224
26.7
20.0
53.3
40.0
20.0
66.7
40.0
20.0
53.3
37.8
SigLIP VIT-SO-14 [100]
384
20.0
26.7
60.0
33.3
13.3
66.7
33.3
26.7
53.3
37.0
DEN VIT-H-14 [20]
224
20.0
26.7
73.3
26.7
26.7
66.7
46.7
13.3
53.3
39.3
DEN VIT-H-14 [20]
378
13.3
20.0
53.3
33.3
26.7
66.7
40.0
20.0
40.0
34.8
MetaCLIP VrT-L-14
224
13.3
6.7
66.7
6.7
33.3
46.7
20.0
6.7
13.3
23.7
MetaCLIP ViT-H--14 [88]
224
6.7
13.3
60.0
13.3
6.7
53.3
26.7
13.3
33.3
25.2
EVA01 ViT-g-14
224
6.7
26.7
40.0
6.7
13.3
66.7
13.3
13.3
20.0
23.0
EVA02 ViT-bigE-14+ [78]
224
20.0
66.7
26.7
26.7
66.7
26.7
20.0
33.3
33.3
VII.A-7B [52]]
3362
46.7
53.3
43.3
50.0
50.0
48.5
Sugar'
336
56.7
50.0
63.3
50.0
60.0
66.7
56.7
63.3
53.3
57.8
Table 6: Performance Comparison of VILA and Various CL.I--Based Models on Different Visual Patterns in MMVP-VL Benchmark. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. Sugar achieves state-of-the-an performance on the majority of tasks, demonstrating its powerful discriminative ability. We use symbols for visual patterns due to space limit: Orientation and Direction, Q: Presence of Specific Features, S State and Condition. to: Quantity and Count, Positional and Relational Context, P Color and Appearance, t Structural and Physical Characteristics, A: Texts, Viewpoint and Perspective. indicates that we question-answering the test method, instead of dot product.
E,4 Retrieval-Augmente Generation.
For performing retrieval-auue augmeeeed generation (RAG), we selected two tasks, namely VizWiz and SQA', as they provide held-in data not seen during model training. We did not use VQA"2, GQA, and VQAT because their held-in data is subset the widely-used VL. SFT. Benchmarks like POPE, MMB, and others lack held-in data. Therefore, we focused on VizWiz and SQA! for our experiments. We utilized a mixed set comprising the widely-used LLaVA-1.5 SFT subset and the held-in dataset of the two tasks as the knowledge base and employed different retrieval modules to retrieve relevant knowledge for the MILM. Similar to common practice, we average the similarity scores for CLIP (We choose CLIP ViT-L/14 @ 336px). For BLIP-2, we compute the similarity using its multimodal token's CLS token. Figure shows some specific retrieval results on test data, demonstrating that Sugar can better integrate information from both images and text, retrieving more similar data as external knowledge.
Flares
What's in this box please? thank lahhen the provided information insufficient, respond Usanswerable' inAnwer question using single or phrase.
type this?'n provided information insufficient, respond with Unanswerable: lnAnswer the question using single word phrase. (chocolate baystacks)
OUTS
What's most likely in the box7nA. rapkins'n more foodlaC' gasinD, wnterinnnnwer with the option's from given choices directly. (B)
CLIP
What this item"nWhen the provided information insufficient, respond with 'Unanweeabll inAnewer the question single word or phrase. (sunflower seads)
BUIKS
Figure 6: Selected examples from do retrieval auggeente generation. Sugar can retrieve more useful knowledge compared with CLIP and BLIP-2. Inside the parentheses the answers, note that the When retrieving, we will only retrieve the questions, not the answers, which are shown here for convenience only.
22

---

What is the brand of this canura"iRelfrene OCR token: DAKOTA, DIGITAL, SingleUUs
Camera Pire, AigitalnAnswer the question using a single word or phmase.
e.g
What brand thisPrnnsseer the question using single phmse. (Nokia)
What is the megapixels" the question using single word or phrase.
CLIP
might placement (The placement of the portraii of'a woman hanging over the toilet in the resirom can unusual and incomfortable atmosphere toiler not a common sight or typical bathroom dul0eecor, and some people may feel unessy, they being watched invaded upon space where they expeet
almosphere restroom?
portmitt honging ahove
privacy. This unintentioal effect on the atmosphere might lead visiluns louling restroom design odd, intrusive, somewhat inpppropritte)
What is the vehicle that is in fronr of the nnnwer the question using single word phrase.
What vehicle is to the left of the man'nAnswer the question using single or phrase. (Car)
What of vehicle is purkedFFAnnower the question using a single word phrase. (Train)
CLIP
How many wheels dees vebicle have nomaalyy"a the question using single word phrase. (18)
happens when people agree exchange goods services. People give up something something else. Sometimes people harter, or directly exchange one good or service for another InC lama
Hazel open
the school cafeteria.
gut everything
they
wamled. The table below shows which ilems they each wanked: Ln'n the images of their lunches.
Then the question below InClar's lunch Hazel's lunch'nWhatt Clara and Hazel trade each
get what they want?aaA Clara can trade somaloess Hazel's brocvoli 'nB. Hazel cam track her almonds for Clara's tomatoes InC. Clam trade her tomatocs for Hazel's camots inD Hazel trade Clam's omnges.
option's letter the given choices directly.
(A)
What is this woman going to cat?laA. surriteiaB. Iaco'nC, sandwichinD, Annswer with the oprion's letter from the given choices directly..
table? (There are various fruits and vegetables on rable, including pears, peaches, plums, grapes, green peppers, and bananas.)
types
vegetables
CLIPP
what brand of phome"akeferenee OCR token: 09, JUN, THU, MOHOROLA, inilaton. Invitation: JB.., Room. Sales, 12:56, The The'AAmwer the question using single word or phrase. (motorolaa
BLIKZ
Figure 7: Selected examples from do retrieval-augmentte generation (continued for Figure 6).
F
More Results
F1 Details of Retrieval
Image-text Retrieval. FROMAGe [40] was evaluated on the 5K validation set of MSCOCO 2017. Due to the split method confusion in FROMAGe, we report our image-text retrieval results on MSCOCO val2014's 5K val set following UnilR [84] and the Karpathy split [38]. What's more we then utilize FAISS [37], powerful library for efficient similarity searches in dense vector spaces, to
23

---

index and retrieve candidates. Therefore, the results may exhibit slight differences when compared under identical settings. The results in Table 3(a) are provided for reference only.
Interleaved Retrieval. We conduct evaluations across several experimental configurations, following the same setup FROMAGe [40]. The settings are as follows:
Retrieval of the last image given the descriptions of the preceding images. This evaluates models'
ability to condition temporally dependent language.
Retrieval of the last image given the descriptions of the preceding 5 images and the 4 preceding images. This assesses models' capability to process interleaved image-and-text context.
Fine-grained Retrieval. Winoground [79] is designed to evaluate the abilitty of vision and language models to perform vision-linguistic compositional reasoning. The task involves matching two images with captions, where both captions contain identical set of words/morphemes arranged in different orders. This dataset, meticulously hand-curated by expert annotators, includes a rich set of fine-grained tags to facilitate detailed performance analysis.
F.2
Quality Results
To analyze Sugar's s emergent behaviors and observed weaknesses, we present additional qualitative samples that were not included in the main paper due space constraints. Please note that for brevity, we have omitted the system prompts and the line breaks after the images for all the quality examples.
We hope these additional results and observations showcase the potential of Sugar in various applica- tion areas. In future work, it is important to investigate these emergent behaviors more thoroughly and to understand the underlying mechanisms that enable Sugar to demonstrate such generalization abilities. This will pave the way towards building better MLLMs, including enhancing robustness, re- ducing biases, and improving the alignment and scope of the learned vision-language representations.
World Knowledge: We observe that Sugar can leverage the world knowledge [26] embedded within the LLM to enhance performance on multimodal tasks. For example, as shown in Figure 4, the model understands that during Halloween, people typically dress up in various ways to portray scary, funny, or creative characters, such ghosts and skeletons.
Retrieval the Same Sequence at Different Place: One interesting emergent behavior of Sugar is its ability to retrieve sequences from different positions within the input interleaved sequence, demonstrating flexibility and high sample efficiency, as shown in Figure 4. Unlike CLIP, which requires encoding each sample separately. Sugar can encode sequences of varying lengths for the same moltl document a single forward pass.
What's 's more, Sugar is capable of both retrieval and generation tasks. Below in Figure 8 are some examples from the VIST dataset.
some pictures from my family 's biyy last wekkend.
we made hamburgers into litile staa stapox
my wifl is ready for the attuack
my dad sytng food
the sunset siferwards besuriful..
we hod much fin all the aill
axtived the station
cume advertisens ni fir four wheelers.
then they
finally when they get they came a group of polise
the police asked them where they and how help them.
P.. Sugar
jfly
down to the
sarikon
for location.
O
starte with
panale
wase
that
theme wan large red line inack the
finally theme cool the parade.
the local
cowns fommy bikes.
downs
deep impression on te
junie loam
Figure 8: Selected examples for various image-text tasks. The pink background indicates retrieval
results, while the blue background indicates generated results.
24

---

Fine-grained Image Discrimination: As shown in Figure 9, Sugar excels at accurately discerning subtle differences between images and identifying detailed objects and their attributes. VILA, on the other hand, tends to describe the content of the images without pinpointing the precise differences between them. contrast, Sugar provides more concise and direct answers.
g
What distinguishes these two images?
Difference:
The main difference between the two images the perspective, In the first image, the view is from a higher angle. giving locusing on the entrance of the building. This differencee in perspective allows for more detailed view of the entrance
parking lot building. contrast, second image from angle, and the surrounding
VILA
sccond image.
the two people that were standing on the sidewalk gone.
which one of following meets the image:'n facing back.
B.a car facing front.In please answer
with
B.a car facing front.
VILA
p Suger
A.a car facing back.
which one of fellowing meclss the imageia painting the wall behind the bed.vn
Emplly on the wall behind the bed/'n the ohe choice.
please
Empty on
A. A painting on the wall behind the bed.
one of following meets image'n
A points ("I?"'
points ("80"""m please answer in A or B with the content of the choice.
A.points 1'0
B.points (BB0""
g
Source Image:
Targel Imagee:
Please give editing Request describe the transformation from the source image
image.
The depicts young with bloade laais, saniling and looking directly at the camera. The image has been edited remsove the background, focusing solely on the girl. The buckground has been replaced with solid color, which is specified in the image description, The girl's position relative to house his not boen allered, she standing in front The image does not contain any additional objects. The girl's smile and direct
purple shirt, standing front of white bouse. The girl
the main focus of the image.
Change the girls face Zoom it in slightly.
Siapar
Figure Selected examples. Sugar excels at accurately discerning subtle differences between images and identifying detailed objects and their attributes.
25

---

Style Following: Sugar exhibits certain degree of in cn-texte style following capability. As shown in Figure 10, with the aid of external knowledge, Sugar partially adopts the style of retrieved results, resulting in more accurate and detailed answers compared to scenarios without retrieval augmentation. g
weather, season, day of visiting this place? time photo taken a good
time to visit this place!
Question
p. Sugse
Yes.
What ard things should be cauticus about when I visit hore?
0.9
pier salety: JEAU
visiting
lakeside location, cautious
following'''nt.
T-shaped wooden pier sturdy and safe to walk on, and be mindfill of any loose planks or slippery surfhees inan2. Swimming conditions: If you plan swim the lake, check for potential hazards such submenged objects strong underwater currents, and mware of the water temperature inin3. Wildlife: Some lakes may have local wildlife, such snakes alligatons, which could pose threar. Remain cautious and observant, especially near the water's odge, m/d4, Weather conditions: The sky cloudy, so peepared for potential rainfall changes in lemperature, and monitor weather forecasts before your visit. lains, Hiking the surrounding explore the green trees mountains
you plan to background, cautious potential hazards steep and slippery terrain unstable mail conditions. Carry appropeiate hiking and wear sturdy footweerr inin6, Protect the environment: respectfal natural beauty of disturbing wildlife.
littering. damaging any trees plants,
Retrieved External Knowledge
The best time to visit this screne lakeside loaatoo daring day, the weather clexr and the lighting good photography The ploto making good
taken
weather
Answer
What is unusual about this image?
Question
Lub Sugar
A nun ironing clothes on car.
Answer
What is the problem this city might be facing? What are some possible solutions?
p
+RAG
The city might facing traffic congestion issucs, particularly the four-lan lane sections of the highway since most have their brake lights indicaling slow slopped traffic. Possible solutions could include/ainl. Implementing munagement strategies, such ramp meters control highway optimizing traflic signal timings improve traffic flow, in2, Encouraging the use of public trinspottttio,, carpooling ride-sharing to reduce number cars road.nn In3, Expanding or adding more lanes case traffic congestion the lane sections of the highway''n4 Exploring altemative routes to divert congested arens, In5, Improving walking and biking infmustucture ciry, thas promoting more sustainable and non non-metorized modes of transport.
cars enter the
Retrieved External Know ledge
unusual aspect this image presence standing pocentially be dengemuus.
car imcing clothes,
right
Figure 10: Selected examples from WAA--eeeh(I--tteeeeeeee Using external knowledge, Sugar partially follows the style of retrieved results, providing more accurate and detailed answers compared to not using retrieval augmentation.
Interleaved Comprehension: As demonstrated by results from DEMON [47], Sugar exhibits superior interleaved comprehension capabilities compared to VILA, particularly in tasks requiring fine-grained analysis and an understanding of global context. For instance, in the third example of Figure 1, VILA confuses character names, whereas Sugar maintains narrative coherence while adhering the style of the preceding text. Similarly, in the third example of Figure 12, VILA provides an irrelevant response, while Sugar delivers a more contextually appropriate answer. Additionally, Figure 13 demonstrates Sugar's s ability effectively capture global information, identifying the relevant images and text within the sequence to provide accurate responses.
26

---

Given two images and a senience ouilining their relation, your responsibility is to judge whether the scntence correctly describes the connection them.'a
Relation: The left image shows least rear-facing standing in froat larger standing animals with borns.
True
VILA
False
Sugar
Having images, can you informationn presented answer following question?'.
Image Caption #1: TtadiumTrack Track inside Olympic Stadium for the Monster Spectacular. 'n
Image Caption Olympique Moniccll match Montrcal Impaci Oympic Stadium Montreal.
Question: Is of Olympic Stadium of Montreal always the same?
The yes.
No, the center of the Olympic Stadium of Montreal is not always the same.
Sugar
Given the progression of the story with the first few images, can write fitting end considering the last image?Un
Captiont Betty sitring on living speakes Betty
and cries into hundkerchief.n
Captionk2 Betly sitling living room. Wilma crying wiping
handkk
while Herly speaks to her, In
Caption'3 Wilmm and Hetty are sitting on the couch the living room in floet of the Wilma holding handkerchiel and talking Betty/n
Captionf44FFee and Barney are in car. Barney looking at Fred and holding the steering wheel. Fred speaks, then Barney speaks.
Captiont5:
++
Wilma sitting and talking to Hetry.
living front the television holding handkerchiic
0.9
Pred and Barney Bamey driving while Fred talking Barney while look behind them.
Figure 11: Selected examples for Interleaved Comprehension.
27

---

g
Utiizing the information, including images, lext, and tables that I provide, could you provide a correct answer to the following
DINBLEND
Global Table:
Context:
Question: How many people the diablero?
There are five people the the image.
four people image.
Suge
Provided with scries of diagrams from lextbook, your responsibility correclly answer the following question in
Question: How many layers of the atmosphere is identified the diagram?
There layers simosshhre identified dingram.
There are five layers the atmosshere identified in the diagram.
stories
pictures, would you climux picture??
Captiond He was inspired by the ducks, in
Caption#2:HHe liked the way the sand that had washed
looked.
Caption83:AA lunch drew sketchim
Ca would hang painting the
other window like this oncin
CaptioniS:
was inspired by the ducks.
VILA
Then he would take picture of the beach put in the other window.
Sugar
Figure 12: Selected examples for Interleaved Comprehension (continued for Figure 11).
28

---

g
Did the Cadillac Series 61 Fastback four door sodan only?'
Image Caption #1: 1949 Cadillac Series 61 Fastback Flickr- uxlondy (1
Image Ception
exfordy
Image Caption
Series 51 Codillac Limousine.'n
Cuption #4; 1949 Cadillac Series Convertibee fvr
Image Caption 1949
Image Caption
1949 Cadillac Series Convertibla frr2: the question Yos No.
VILA
2.
No.
Sugar
Are there any frees near the HSBC Hong Kong Headquariers building which are taller than the building""
Image Caption #1: HSBC Hong Kong headquarters building. Hong Kong, Mar 06'n
Image
Caption Headquarteess Building, Hong Kong, lep showing canmons.
Imagge Caption Hong
Kong HSBC headquariers building IMG 5378 HSBC Hong Kong headquarters building, Hong Kong
Image
Caption #4: Hong Kong HSBC headquariess building IMG 5377 HSBC Hong Kong headquasters building, Hong Kong.)
mage Caption #5: HSBC Hong Headquartes.
Caption n6: HK HSBC Main Building 2008 Facade
Hong Kreg hendquaters building, plense answer the question
VILA
No.
Sugar
Figure 13: Selected examples for Interleaved Comprehension (continued for Figure 12).
Sensitivity with Detailed Semantics: Sugar can address various examples inspired by the Winograd schema [43]. These examples consist of multiple sentences that differ only by single word, leading to different resolutions of ambiguity. Sugar can accurately match images and text, demonstrating its minor changes in input prompts. Figure 14 showcases some that align with the Winograd schema from Winoground.
sensitivity
0
P.
The dog wears a hat hut person
the wearing neutral colors poses and the person wearing brighter colors takes a picture
0. Suge
O Sugar
porsoa wears lut the dog doesn't
the wearing brighter colors and persen wearing ocuiral colors takes a picture
the plant sating the bug
an inflatable
head in the clouds
flamingo on
0.0
0.0 Supe
O
bug is cating the plant
clouds in the head
porson inflatable flamingo
Sugu
Figure 14: Selected examples from Winoground. Sugar is Sensitivity with Detailed Semantics
29

---

G Retrieval for Knowledge-based VQA
In this section, we use FVQA [83] and WebQA [10], two knowledge-based VQA datasets, to verify Sugar's 's effectiveness of combining retrieval and comprehension abilities in a single model, thereby avoiding compatibility issues and suboptimal performance.
Historically, solving FVQA has relied on modeling the knowledge database using a Knowledge Graph [13]. For WebQA, each question is associated with 10-20 knowledge bases, but only one relevant to the image and caption. FVQA knowledge is textual, whereas WebQA knowledge consists of both text and pictures.
Implement Details. In this experiment, we used CLIP ViT-L/14 336px, and both experiments report the ROUGE-L Score.
For FVQA, answers originate from two sources: directly from the image or from the knowledge base To minimize interference, we only tested questions requiring the knowledge base. We used the following prompt for FVQA: "Please answer the questions based on the pictures. the reference information is useful, please use it. Otherwise, please ignore the reference information. Reference information: retrieved knowledge <image> question." " The baseline without retrieval means we did not search for knowledge, but directly input the image and question for the model to answer. CLIP image means using the image to retrieve knowledge, + CLIP text means using the text to retrieve knowledge, and CLIP average means using the average annotations of both image and text to retrieve knowledge. For our model, sugar+rag indicates the average result obtained using both image and text to retrieve knowledge.
For WebQA, each question has 10-20 negative captions and images. Due to context length limitations in LLaVA and VILA, we could not input all the data, necessitating a retrieval model to extract relevant knowledge. Due to the large dataset size, we randomly selected 1000 samples. For WebQA, +CLIP image means providing the positive image and using it to retrieve the most relevant text from the knowledge base, which then used as input for the model to answer the question. Conversely, CIJP text uses the text to retrieve relevant images. For model, sugar+rag indicates the result obtained using the average similarity score of the aforementioned methods.
FVQA
WehQA
MaVA--15-7B
5,9
LLaVA-1.5-7B CLIP image
6.8
81.8
LLaVA-1.5-7B- CLIP text
7.1
79.2
LLaVA-1.5-7B- CLIP (average)
7.9
/
VII.A-75
6.4
/
VILA-7B + CLIP image
9,0
80.0
VILA-7B CLIP text
10.2
71.2
VIIAA-B + (average)
11.0 /
Sugar
6.5
/
Sugar + rag
20.7
88.7
Table 7: Comparison between the independent generator + retriever and Sugar on knowledge-basse
VQA. ''' indicates not applicable.
Results. Based on Table 7, we can observe that while MILM can answer a small portion of FVQA questions using its internal knowledge, it still requires the support of retriever for enhanced accuracy. However, the impact of retrieval strategies on the results inconsistent. For instance, using text retrieval often outperforms image retrieval in FVQA, whereas in WebQA, image retrieval is more effective, Additionally, there are compatibility issues between retrieval strategies and models. For example, in WebQA, VIIA is more sensitive to CLIP's retrieval strategy, with fluctuations 3.4 times greater than those of I.L. VA.155 Our integrated retriever and generator model, however, does not require an additional retriever and avoids the aforementioned optimization and selection issues.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction provide accurate representation the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification:
We give the proof in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give experimental setup and implementation details in Section 4.1 and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [NA]
Justification:
The codes will come soon and all the data is public to access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt /onee ssssssolicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: we have provided necessary implementation details our method in Ap- pendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We followed the baseline settings on the evaluation benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We give the statements of experiments compute resources in Appendix E.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //teer/nneuipeee httpp/////errrpp.copoeiiileeeeee wwtpppccccpcccciccceeeecceccaeee ctw.:/eeeeiiiiiiieieeeeeeieee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss Broader Impacts Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have already cited all the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "zUbbsHOgJl", "reviewer": "Reviewer_4VP3", "summary": "This paper proposed a method for unifying generative training and discriminative training of multi-modal LLMs. Generative training mainly uses auto-regressive formulation while discriminative training mainly performs contrastive representation matching. The goal of this paper is trying to use discriminative training to improve multi-modal LLMs.\n\nThe paper unifies generative training and discriminative training by introducing a Dynamic Sequence Alignment module which aligns similar text and image data on the hidden states of a multi-modal LLM. In addition, Detailed Semantics Modeling is proposed to effectively distinguish detailed semantics.\n\nThe paper conducts evaluation on a wide range of benchmarks.", "questions": "What is the training recipe of the proposed method?", "limitations": "-", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The motivation is clear and the paper is easy to follow.\n\nThe concept of unifying generative training and discriminative training is interesting.", "weaknesses": "It's unclear what is dynamic time warping framework.\n\nThe performance improvement of the proposed method sugar is not significant. Compared with some baselines, such as VILA and LLaVA-1.5, Sugar performs worse than them on many tasks, as shown in Table 1. This raises concerns about the effectiveness of the proposed method. \n\nThis could be meaningless to align a visual token and a text token in an MLLM model since the LLM is trained with next-token prediction instead of contrastive learning like CLIP. The current token is conditioned on previous tokens. I can't think of a reasonable explanation for this mechanism. It **could be** meaningful to align visual tokens. In addition, the experiment results also suggest that this method is not effective as expected. \n\nWhat is the evaluation protocol?  Does Sugar train on each benchmark first then evaluate or directly zero-shot evaluation? In the former case, will Sugar lose generative ability after training with discriminative task data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and insightful comments. We will explain your concerns point by point.\n\nFirst, we must clarify that we only used **a very limited amount of data** (see **Table A of the Rebuttal PDF**). Despite this, we have effectively integrated the generative and discriminative paradigms, alleviating the challenges remain in generative paradigms and collaterally enabling our model to tackle many other tasks. Such as complex retrieval tasks (Section 4.4), multi-modal retrieval-augmented generation (Section 4.5), and knowledge-based VQA in one model. \n\n> **Q1**: It's unclear what is dynamic time warping framework.\n\n**A1:**  Thank you for your question!  We apologize for not clearly explaining our method and the dynamic time warping framework. Below, we will elaborate on DTW and our contributions.\n\n**(1.1)** The Dynamic Time Warping (DTW) framework is a dynamic programming-based method for calculating the similarity of unequal length sequences, commonly used in text information retrieval *[Müller M., 2007]* *[Cuturi M, Blondel M., 2017]* and sequence matching problems *[Cao et al., 2020]*. \n\n**(1.2)** To enable the current separated generative and discriminative paradigms to achieve synergistic gains, we primarily face two challenges: **(a)** comprehensively capturing the global semantics for interleaved sequences of unequal length; **(b)** keenly differentiating the detailed semantics.\n\n**(1.3)** We explicitly designed two corresponding modules in Sugar to tackle these challenges: **(a)** formulate the relationship between interleaved sequences as a dynamic sequence alignment problem within the DTW framework;  **(b)** integrate a novel kernel into the framework to leverage the strengths of various discriminative pre-trained models. The ablation experiments validate the effectiveness of our approach (Section 4.5 and Table G).\n\n*[Müller, 2007]* Dynamic Time Warping. Information Retrieval for Music and Motion.\n\n*[Cuturi & Blondel, 2017]* Soft-DTW: A Differentiable Loss Function for Time-Series. ICML 2017.\n\n*[Cao et al., 2020]* Few-Shot Video Classification via Temporal Alignment. CVPR 2020.\n\n> **Q2**: The performance improvement of the proposed method Sugar is not significant. Compared with some baselines, such as VILA and LLaVA-1.5, Sugar performs worse than them on many tasks, as shown in Table 1. This raises concerns about the effectiveness of the proposed method.\n\n**A2:** Thank you for raising an important concern. We will address your question from the following three aspects:\n\n**(2.1)** We used a very **limited amount of data** compared to VILA. Despite this, we achieved competitive results and gained additional retrieval and retrieval-augmented capabilities. By using an equitable data volume (incorporating 300K SFT data used by VILA), we observed enhancements in 11 comprehension benchmarks of Table 1 (**Table B,C**), demonstrating our model's scalability.\n\n**(2.2)** In the original manuscript Table 1, we primarily used the single-image QA benchmarks. To provide a more comprehensive assessment of our model, we have supplemented our method with 5 new, **more challenging** benchmarks, showcasing our superiority (see **Table E**). To answer **multi-image QA**, the model must clearly distinguish between different images and understand their shared global information. For **in-context interaction**, it must fully comprehend the preceding multiple image-text dialogue to provide reasonable responses. For **visual prompt comprehension**, the model needs to accurately identify visual cues in the images to answer questions. Our model achieved the best results across all three task types.\n\n> **Q3**: This could be meaningless to align a visual token and a text token in an MLLM model since the LLM is trained with next-token prediction instead ... (omitted)\n\n**A3:**  We apologize if our manuscript gave the impression that the method of Sugar was unclear.\n\n**(3.1)**  Simply using contrastive learning methods like CLIP can only calculate the similarity between single images and texts. However, we **consider the interleaved image-text sequence as the general format** of input samples. This means that the number of tokens in different sequences is of unequal length. If we simply use contrastive learning like CLIP, the similarity of interleaved sequences would lose global information.\n\n**(3.2)**  Hence, we select the final token of a sequence to represent this input interleaved sequence. As you mentioned, *'the current token is conditioned on previous tokens*' , the final token can integrate global information from the preceding content. In this way, we can modulate the hidden states of the MLLM by leveraging the semantic relationships between interleaved input sequences, thereby ensuring consistency between hidden states and encouraging the MLLM to capture the global semantics. Additionally, we introduced a novel kernel to provide more detailed signals, enhancing the ability to differentiate fine-grained semantics. \n\n**(3.3)** The effectiveness of our method is demonstrated through experiments on 11 comprehension benchmarks and others VILA can't do (**Table B,C,D,E**).\n\n> **Q4:** What is the evaluation protocol? Does Sugar train on each benchmark first then evaluate or directly zero-shot evaluation? In the former case, will Sugar lose generative ability after training with discriminative task data?\n\n**A4:** All the experiments in our paper are **zero-shot**. We have merely introduced a structure-induced training strategy within the generative paradigm and subsequently conducted zero-shot evaluations for each task.\n\n> **Q5:** What is the training recipe of the proposed method?\n\n**A5:**  We apologize if our manuscript suggested that Sugar's training method was unclear. The specific data we used is detailed in **Table A of the Rebuttal PDF**.\n\nWe hope the above discussion addresses your concerns. The discussion remains open, and we always welcome your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDWkdiz0qr", "reviewer": "Reviewer_Lcdg", "summary": "The paper addresses the limitations of Vision-Language Models (VLMs) by proposing a unified approach that combines generative and discriminative training paradigms. This new method leverages interleaved image-text sequences and introduces a structure-induced training strategy. It aims to enhance the MLLM's ability to capture global semantics and fine-grained details, effectively balancing generative and discriminative tasks. The approach uses dynamic sequence alignment within the Dynamic Time Warping framework and integrates a novel kernel for fine-grained semantic differentiation. Extensive experiments demonstrate that this method achieves state-of-the-art results in various generative and discriminative tasks.", "questions": "- Can you provide more detailed ablation studies to understand the contribution of each component of the proposed method, such as the dynamic sequence alignment and the GAK?", "limitations": "- Considering the connection with real-world scenarios is necessary. The authors need to discuss: what requirements might the tasks introduced in the paper have in real-world scenarios?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel method that successfully integrates generative and discriminative training paradigms, addressing the weaknesses inherent in each when used independently.\n- The authors clearly articulate the challenges faced by existing VLMs and provide a well-defined solution.", "weaknesses": "- While the paper shows impressive results, there is limited discussion on the potential limitations and areas where the model might underperform.\n- The paper primarily focuses on specific benchmarks. It would be beneficial to discuss how well the proposed method generalizes to other types of vision-language tasks not covered in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments and we will explain your concern as follows.\n\n> **Q1:** While the paper shows impressive results, there is limited discussion on the potential limitations and areas where the model might underperform.\n\n**A1**: Thank you very much for your valuable question! Our potential limitations are as follows:\n\n**(1.1)** Firstly, the context length limitation of Vicuna, capped at 4096 tokens, poses a significant challenge. This restricts Sugar's ability to handle very long multimedia documents.\n\n**(1.2)** Moreover, real-world applications often present varied and dynamic data, which might not align perfectly with the model's training data. This discrepancy could lead to suboptimal performance in diverse and unforeseen contexts.\n\n**(1.3)** The limitation of our model lies in its reliance on the existing Vicuna and CLIP encoder,  which is suboptimal and somewhat limits the comprehension ability *[Karamcheti et al., 2024]*. For example, there are some hallucinations.\n\n*[Karamcheti et al., 2024]* Prismatic VLMS: Investigating the Design Space of Visually-Conditioned Language Models. ICML, 2024.\n\n> **Q2:** The paper primarily focuses on specific benchmarks. It would be beneficial to discuss how well the proposed method generalizes to other types of vision-language tasks not covered in the experiments.\n\n**A2:** Thank you for your suggestion. In the original manuscript, we primarily used the single-image QA benchmarks from LLaVA-1.5 to evaluate our model's multi-modal comprehension capabilities. To provide a more comprehensive evaluation of our model, we have supplemented our method with 5 benchmarks in **Table E of the Rebuttal PDF**.\n\nTo answer **multi-image QA**, the model must clearly distinguish between different images and understand their common global information. In **in-context interaction**, the model must fully comprehend the preceding image-text dialogue to provide reasonable responses (Visual Dialog and VIST-SIS, involve 10 and 5 rounds of dialogue respectively). For **visual prompt comprehension**, the model must meticulously identify visual cues in the images to answer questions. Our model achieved the best results across all three task types.\n\n> **Q3:** Can you provide more detailed ablation studies to understand the contribution of each component of the proposed method, such as the dynamic sequence alignment and the GAK?\n\n**A3:** To test the effectiveness of our method, we introduced two variants of GAK: GMin and GMax, and examined the impact of the hyperparameter $\\delta$. To avoid interference from additional data, we explored the impact of different parameters without using any extra task-specific data. We tested two variants of the kernel function: \n\nGMin kernel: $\nK_{GMin}(\\mathbf{x}, \\mathbf{y}) =\\min_{\\pi \\in \\mathcal{A}(\\mathbf{x}, \\mathbf{y})} \\prod_{i=1}^{|\\pi|} e^{-\\phi_\\sigma}  \\in (0, 1]\n$, \n\nGMax kernel$: \nK_{GMax}(\\mathbf{x}, \\mathbf{y}) =\\max_{\\pi \\in \\mathcal{A}(\\mathbf{x}, \\mathbf{y})} \\prod_{i=1}^{|\\pi|} e^{-\\phi_\\sigma}  \\in (0, 1]\n$. \n\nBased on the experimental results in **Table G**, we found that:\n\n1. GAK is relatively stable and not very sensitive to the hyperparameter δ.\n2. GAK generally performs better compared to GMin and GMax.\n3. Tasks requiring global information, such as VIST and KGQA, and those needing detailed semantics, like Winground, are the most sensitive to changes in the method.\n\n> **Q4:** Considering the connection with real-world scenarios is necessary. The authors need to discuss: what requirements might the tasks introduced in the paper have in real-world scenarios?\n\n**A4:**  Thank you for your insightful question. Below, we outline two valuable real-world applications:\n\n**(4.1) Complex information retrieval.** Real-world information retrieval requires models to search through **interleaved multi-modal sequences** , as articles, websites, and books are often composed of extensive interwoven image and text sequences. Current retrievers like CLIP and Sentence BERT are designed to handle single images or texts and struggle with complex interleaved retrieval tasks. In Section 4.4, we tested complex retrieval scenarios and achieved promising results. Adapting to more complex scenarios is an important direction for future research.\n\n**(4.2) Complex comprehension tasks**, such as the 5 new tasks mentioned in Q2, which respectively require capturing global semantics and detailed semantic differentiation.\n\n**(4.3) Knowledge retrieval for answering questions.** In reality, many questions require external knowledge for accurate answers. For tasks such as FVQA and WebQA, traditional methods first require the retriever to find relevant knowledge in a knowledge base, and then the generator to formulate the answer. The final performance, therefore, depends on both the retriever and generator models, highlighting **compatibility issues** and **sub-optimal issues** between them. Based on the **Table D**, we can observe the following: \n\n**(4.3.1)** MLLM can answer a small portion of FVQA questions using internal knowledge, but it still requires some knowledge from a retriever for accurate responses. \n\n**(4.3.2)** The impact of retrieval strategies on the results is inconsistent. For instance, text-based retrieval strategies often outperform image-based ones in FVQA, whereas in WebQA, image-based retrieval is more effective.\n\n**(4.3.3)** There are also compatibility issues between retriever and models. For example, in WebQA, VILA is more sensitive to retrieval strategy, exhibiting fluctuations 3.4 times greater than those of LLaVA-1.5.\n\n**(4.3.4)** Our integrated retriever and generator model does not require additional retrievers, thereby avoiding the optimization and selection issues mentioned above.\n\nWe hope this clarifies your concerns. We are committed to thoroughly incorporating your suggestions in the next version of the paper. Thank you once again for your excellent feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zn1ZqJQEkp", "reviewer": "Reviewer_DDMF", "summary": "This paper proposes a novel learning paradigm to learn MLLMs based on interleaved image-text corpora. \nIt introduces a structure-induced training strategy that imposes semantic relationships between input samples and\n the MLLM’s hidden state.\nThis work apply the dynamic time warping framework to calculate the semantic similarity between different image-text sequences.\nThen, a discriminative loss is applied to sequence similarity matrices calculated based on raw inputs and MLLM hidden states. \nThe framework can also leverage the capabilities of multiple vision and language encoders to more accurately calculate the similarity matrices.\nExperiment results show that the new learning paradigm demonstrates good performance on basic multimodal comprehension benchmarks, \ncomplicated multimodal comprehension benchmark DEMON, cross-model information retrieval, and retrieval-augmented generation.", "questions": "1. Can sugar better leverage the multi-modal in-context examples or better understand interleaved image-text content, is there any evaluation for that?\n2. What is exactly the amount of interleaved image-text sequences (from MMC4) and image-text pairs (from other datasets) used to train Sugar. \n3. What is the context window size of Sugar?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow. \n2. This paper proposes a novel learning paradigm based on interleaved image-text corpora.", "weaknesses": "1. This paper did not discussed the impact of including interleaved image-text pairs in MLLM learning. For example, how will it affect the performance on basic visual-language benchmarks (Table 1) and image-text retrieval. Will there be any negative effects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your comprehensive comments and constructive advice. We will explain your concern as follows.\n\n> **Q1:** This paper did not discussed the impact of including interleaved image-text pairs in MLLM learning. For example, how will it affect the performance on basic visual-language benchmarks (Table 1) and image-text retrieval. Will there be any negative effects?\n\n**A1:** Thank you for your insightful question. We conducted additional ablation experiments (**Table F of the Rebuttal PDF**) by gradually increasing the MMC4 sampling ratio in basic visual-language benchmarks (Table 1) and image-text retrieval. The results demonstrated the following effects of interleaved image-text pairs on MLLM learning.\n\n**(1.1)** MMC4 does not directly contribute to performance improvement for basic visual-language benchmarks. This is because its long sequence undermines the model's interleaved ability (e.g., VQA dropped from 79.9 to 60.5 when MMC4 radio increased). Additionally, without Sugar, MMC4 is unable to assist in image-text retrieval.\n\n**(1.2)** (a) Delightfully, using a synergistic framework in Sugar can significantly enhance the capability of interleaved text  while maximizing the retention of basic visual-language capabilities as the MMC4 data increases. Moreover, it will facilitate certain tasks that require distinguishing between multiple images or texts, such as POPE (85.5 to 86.4）and multi-image VQA(from 38.4 to 39.5). (b) Only Sugar can simultaneously achieve image-text retrieval. With the increase in MMC4 data, it can significantly improve its performance in complex retrieval tasks (increasing from 14.4 to 23.2) while maintaining competitive performance in general retrieval tasks.\n\n*Please refer to Table F of the Rebuttal PDF for the Results. The table below is the same as in the PDF*\n\n> **Q2:** Can Sugar better leverage the multi-modal in-context examples or better understand interleaved image-text content, is there any evaluation for that?\n\n**A2:** Thank you for your constructive suggestions. Following your recommendations, we have conducted additional experiments on 5 benchmarks across 3 task types to further evaluate the effectiveness of our method in **Table E**.\n\nTo effectively answer **multi-image QA**, the model must clearly distinguish between different images and understand their common global information. In **in-context interaction**, the model must fully comprehend the preceding image-text dialogue to provide reasonable responses. For **visual prompt comprehension**, the model needs to meticulously identify visual cues in the images to answer questions. Our model achieved the best results across all three task types, especially in VIST-SIS, which involves 5 rounds of interleaved image-text dialogue.\n\n*Table E Comparison with baseline in 5 challenging tasks*\n\n|              | **Multi-image VQA** |          | **In-context Interaction** |          | **Visual Prompt** |\n| ------------ | ------------------- | -------- | -------------------------- | -------- | ----------------- |\n|              | SEED                | Mantis   | Visual Dialog              | VIST-SIS | BLINK             |\n| LLaVA-1.5-7B | 58.6                | 31.3     | 53.7                       | 14.1     | 37.1              |\n| VILA-7B      | 61.1                | 38.4     | 58.5                       | 17.4     | 39.2              |\n| Sugar        | **63.6**            | **41.0** | **62.5**                   | **28.2** | **42.2**          |\n\nWe also present the results in Table E of the Rebuttal PDF.\n\n> **Q3:** What is exactly the amount of interleaved image-text sequences (from MMC4) and image-text pairs (from other datasets) used to train Sugar.\n\n**A3:**  Thank you for your question. We apologize if our manuscript gave the impression that the training recipe for Sugar was unclear. The specific data usage and comparisons are detailed in the **Table A**. We did **not** use any additional data beyond what baseline used.\n\nWe utilized a **limited dataset** in comparison to VILA and LLaVA-1.5. Nonetheless, we achieved competitive outcomes and acquired additional retrieval and retrieval-augmented capabilities. By employing an equivalent data volume and integrating the 300K SFT data used by VILA， we observed improvements in 12 multi-modal comprehension capabilities (**Table B,C**), showcasing the model's scalability.\n\n> **Q4:** What is the context window size of Sugar?\n\n**A4:**  Thank you for your insightful query! Due to the context length limitation of Vicuna-7B, which is **4096** tokens. So, Sugar cannot handle very long multimedia documents.  However, we believe that the mutual enhancement of generative and discriminative capabilities demonstrated by Sugar, as well as the exploration of integrating retrieval and generation within a single model, are beneficial. Your question is valuable and will guide our future improvements and research directions.\n\nThank you once again for your recognition and constructive suggestions, which have been instrumental in enhancing the quality of our research!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their insightful and valuable comments! 

We thank all the reviewers for agreeing that this paper presents a very interesting idea of **addressing the limitations of the original generative paradigm** in comprehensively capturing global information and keenly discerning fine-grained semantic details through the introduction of discriminative supervision. Collaterally, our method leverages the strengths of MLLMs to handle **complex information retrieval** tasks that cannot be effectively solved by discriminative models such as CLIP. Our paradigm also realizes **retrieval-augmented generation** and addresses **knowledge-based** VQA problems within a **single model**, showing great promise for future work.

Overall, we are encouraged that they find that:

- The motivation and novelty for Sugar are clear, reasonable, and meaningful. (all Reviewers)
- Clearly articulate the challenges faced by existing MLLMs and provide a well-defined solution. (all Reviewers)
- The experiments are thorough, clearly showing that the proposed framework can excel well at both comprehension and discrimination tasks. (Reviewers Lcdg and Reviewers DDMF)
- The effect of combining our retriever and generator is interesting, which can inspire fellow works (Reviewer Lcdg).

To address the concerns raised by the reviewers, overall we have conducted several additional experiments and analyses:

- We further validated Sugar's effectiveness using more data equivalent to the baseline, as shown in **Tables B and C of the Rebuttal PDF**. 
- We present the performance across 5 tasks of 3 new types (in-context, multi-image, visual prompt comprehension) to more comprehensively evaluate our model's capabilities in **Table E of the Rebuttal PDF**.
- We conducted a more detailed dataset and method ablations to verify the stability of Sugar and the effectiveness of each component in Sugar in **Table G of the Rebuttal PDF**.
- We validated the 2 tasks requiring external knowledge in **Table D of the Rebuttal PDF** to demonstrate the benefit of combining retrieval and comprehension abilities in a single model, thereby avoiding compatibility issues and sub-optimal performance.

Next, we address each reviewer's detailed concerns point by point. We sincerely thank all reviewers for their recognition of our work and the valuable suggestions provided! And discussions are always open. Thank you!

Meta-review: This paper proposes a novel method for unifying generative and discriminative training paradigms in multi-modal large language models (MLLMs). The approach aims to address the limitations of existing Vision-Language Models (VLMs) by integrating generative training, which typically struggles with issues like hallucinations, and discriminative training, which often falls short in complex scenarios requiring fine-grained semantic differentiation. The authors introduce a structure-induced training strategy that leverages dynamic sequence alignment within the Dynamic Time Warping (DTW) framework, enhancing the model's ability to capture both global and fine-grained semantics.
After the rebuttal, this paper received all positive recommendations (two weak accepts and one borderline accept).  The major concern from reviewers was about the Sugar's effectiveness, which has been addressed in the rebuttal. The AC agrees with the reviewer’s assessments and recommends accepting the paper.
Decision: Accept (poster)