Unified Generative and Discriminative Training for Multi-modal Large Language Models

Authors: wei  chow
juncheng  li
qifan  yu
kaihang  pan
hao  fei
zhiqi  ge
shuai  yang
siliang  tang
hanwang  zhang
qianru  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 01:10:33

Abstract: In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucinations and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM’s hidden state. This approach enhances the MLLM’s ability to capture global semantics and distinguish fine-grained semantics. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating a novel kernel for fine-grained semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine-grained retrieval tasks. By employing a retrieval-augmented generation strategy, our approach further enhances performance in some generative tasks within one model, offering a promising direction for future research in vision-language modeling.

Main paper: 
 introduction: 
 In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms: generative training and discriminative training. Generative Training has achieved remarkable success in enabling Multimodal Large Language Models (MLLMs) [1, 55, 86] to develop a wide range of powerful capabilities that can handle various complex tasks (e.g., open-world visual question-answering, image caption generation, etc.) within a single model. However, challenges such as hallucinations and weak image object discrimination abilities [7, 89] persist. Discriminative Training, exemplified by CLIP [73] , exhibits remarkable representation capabilities for zero-shot image-text classification and retrieval. Nonetheless, it encounters difficulties in processing complex scenarios (i.e., , retrieving multi-modal documents with interleaved images and texts) [53, 54] and exhibits a limited ability to discern detailed semantic differences [79, 85] .The disparity between these two paradigms has sparked recent studies aimed at imparting discriminative ability to generative pre-trained MLLMs. However, certain aspects of performance still pose limitations (e.g., singular discriminative tasks [89] , weak discriminative task performance [40] , weak generalization [59] , etc.), while others entail compromising the model's original generative capabilities [8] .Overall, the reason generative paradigms struggle with performing discriminative tasks like retrieval is due to overlooking two crucial abilities:(i) Comprehensively capturing the global semantics. Recent studies have revealed that causal LLMs tend to exhibit a bias towards capturing global information from the input samples, often resulting in a tendency to overlook information located in the middle, especially for long sequences [15, 57] . As illustrated in Figure 1 (a), we chose 500 samples from WebQA [10] , where the task is to find and reason about the right image-text pair among five distractors to produce a yes or no answer. We conducted experiments using VILA [52] , a MLLM with state-of-the-art interleaved image-text comprehension ability, alongside our model. When placing the relevant pair in different positions, the performance of MLLMs followed a 'U' shape, indicating a bias in capturing global semantic information. Consequently, MLLMs encounter difficulties in forming comprehensive representations that encompass global semantics for retrieval tasks.(ii) Keenly differentiating the detailed semantics. Some research [47, 82] has found that the existing generative training framework cannot fully distinguish input semantics in certain contexts, causing MLLMs to struggle with tasks requiring fine-grained semantics [46, 98] . As depicted in Figure 1 (b), we noticed that MLLMs face challenges in choosing the right description for two similar images in the MMVP-VLM benchmark [81] . This indicates that MLLMs struggle to effectively differentiate the detailed semantics of input samples, naturally leading to difficulties in forming effective queries for retrieval. [10] , the accuracy roughly forms a "U" shape curve when the relevant image-text pair for a question appears at different positions. While our model also shows similar trends, it tends to be more stable overall. (b) The accuracy of various types of questions in MMVP-VLM [81] , it can be observed that our model's performance improves on such tasks after introducing the discriminative training. Details can be seen in Appendix E. 3 In this paper, we argue that the current separated paradigms possess the potential for achieving synergistic gains. We propose Sugar: Structure-induced approach to unify generative and discriminative paradigms (shown in Figure 2 ), leveraging discriminative training to acquire the two abilities above while harnessing the potential of generative training in complex discriminative tasks like image-text interleaved retrieval and fine-grained retrieval. Specifically, we explicitly impose the semantic relationships between different input samples as an induced structural constraint on the hidden state of MLLMs. We consider the interleaved image-text sequence as the general format of input samples, and then formulate the relationship between any two samples as a dynamic sequence alignment problem within the Dynamic Time Warping framework [67, 33] . In this way, we can explicitly modulate the hidden states of the MLLM by leveraging the semantic relationships between interleaved input sequences, thereby encouraging the MLLM to fully capture the global semantics of the inputs.To further enhance the ability to differentiate fine-grained semantics, we integrate a novel kernel into the Dynamic Time Warping framework. Leveraging the strengths of various discriminative pre-trained models, it performs dynamic sequence alignment for diverse embeddings tailored to specific contexts, thus addressing the inherent limitations in fully utilizing input semantics. Through this explicit structure-induced constraint, our framework enables MLLMs to capture the global semantics and fine-grained details of the input multimodal sequence more effectively, thus bridging the gap between generative and discriminative training paradigms. Our method effectively balances both discriminative and generative tasks, demonstrating synergistic benefits. (i) Large-scale generative pre-trained models possess semantic-rich hidden states [41, 91, 23] , which facilitate discriminative tasks like retrieval. Moreover, harnessing the capabilities of MLLM is crucial for complex discriminative tasks, such as interleaved image-text retrieval and fine-grained retrieval. (ii) By integrating discriminative tasks, the model's effectiveness in generative tasks, particularly within tasks requiring cognitive and discrimination abilities, is enhanced, thereby mitigating certain occurrences of hallucinations. (iii) We can employ Sugar to realize retrievalaugmented generation [2] , eliminating the need for an off-the-shelf retrieval module [75] , thereby amplifying the performance of various generative tasks. The usage of off-the-shelf retrieval presents a challenge wherein the retriever's performance affects the generator's final output [62] . This necessitates independent optimization of both components, posing a dilemma in selecting optimal configurations. However, our approach circumvents such optimization challenges.Through extensive experimentation, we have demonstrated the effectiveness of our approach. For generative tasks, Sugar establishes new state-of-the-art results on the tasks for complicated multimodal comprehension tasks (i.e., DEMON [47] ), fine-grained semantic distinctions (i.e., VizWiz [28] , MME [95] ), object hallucinations detection (i.e., POPE [51] ) (Section 4.2 and Section 4.3). For discriminative tasks, we achieved competitive results in image-text retrieval compared, and significantly surpassed CLIP in interleaved retrieval and fine-grained retrieval (Section 4.4). Furthermore, employing the retrieval-augmented generation (RAG) strategy led to further improvements in a series of generative tasks (Section 4.5). 
related work: 
 Multi-modal Large Language Models. Flamingo [3] and BLIP-2 [49] integrate LLMs with visual encoders, showcasing impressive zero-shot capabilities by aligning visual features with language representations. Building upon the advancements of LLaVA-1.5 [55] , subsequent studies [103, 19, 94, 6, 42, 72, 95, 98, 45] propose fine-tuning MLLMs with multimodal instruction tuning data [102] . Recently, there has been a surge in research [52, 80, 22, 21, 48] dedicated to enhancing the capacity of MLLMs to process interleaved image-text inputs effectively. However, these models primarily focus on generative tasks, overlooking the importance of introducing discriminative constraints. In this paper, we propose a structure-induced joint training strategy for unifying generative and discriminative tasks, further enhancing the capabilities of MLLMs, especially those requiring cognitive and discriminative abilities.Vision-Language Pre-training. Vision-Language Pre-training primarily come in two forms: singlestream and dual-stream. In single-stream models, the embeddings for the image and text modalities are concatenated and jointly encoded [39, 50] , while in dual-stream models, they are encoded by separate modality-specific encoders with optional cross-modality fusion [73, 31, 5] . These models have shown effectiveness in tasks such as classification and retrieval. However, they face challenges including difficulty in processing complex composed sequences [53, 54] and limited ability to discern detailed semantic differences [81, 79] . Recent attempts to utilize generative MLLMs for discriminative tasks have faced limitations, such as singular discriminative tasks [89] , weak discriminative task performance [40] , poor generalization [59] , and compromised generative capabilities [8] .LLMs for Retrieval. Early models for retrieval primarily focused on word representations [16, 64, 74] , with minimal generative capabilities. Some recent works have endeavored to fine-tune generative pre-trained LLMs to generate discriminative embeddings, albeit at the expense of compromising the model's original generative capabilities [44, 70, 65, 63, 24, 71] . GRIT [66] integrates generative and discriminative tasks in NLP and demonstrates mutual benefits between them. However, its training cost is prohibitively high compared to individual tasks. Moreover, due to its specialized attention mechanism, the model can only be trained from scratch.Retrieval-Augmented Generation. Retrieval-Augmented Generation (RAG) [25, 69] , which harnesses the advanced inference capabilities of LLMs along with external knowledge, has the potential to significantly mitigate issues related to long-tail entities and reduce the occurrence of hallucina-tory responses [29, 36, 101, 77, 90, 92, 97] . Recently, there have also been related studies in the multimodal domain attempting to utilize retrieval augmentation [93, 96] . These methods typically require an additional retrieval module (e.g., CLIP), leading to component optimization challenges where the overall model performance is affected by the performance of the retrieval model, as well as concerns regarding the compatibility between the retrieval model and the MLLMs. Furthermore, retrieval modules like CLIP struggle to handle compositional or fine-grained scenarios, posing certain challenges for retrieval. 
method: 
 As illustrated in Figure 3 , we initially introduce the problem formulation and offer an overview of our structure-induced joint training strategy in Section 3.1. Subsequently, we delve into the specifics of dynamic sequence alignment algorithm in Section 3.2. Finally, we further introduce the Triple Kernel to aid in discriminating detailed semantics in Section 3.3. We view the interleaved image-text sequence as the general format for input samples, where images and textual data are alternately arranged. Typically, Multimodal Large Language Models (MLLMs) [55, 52, 11, 6] are tailored to generate text based on such input sequences, and it is conventionally optimized using self-regressive loss L g . A special scenario arises when the input comprises only one image and a question, prompting the MLLMs to generate an answer accordingly.While intuitive, this optimization objective solely supervises text generation and lacks constraints on the hidden states of the entire interleaved sequence input. Additionally, the existing generative training framework struggles to fully distinguish input semantics in certain contexts, such as discerning fine-grained object details. Consequently, it fails to adequately capture the global information or distinguish detailed semantics of the input samples.Hence, we introduce a structure-induced constraint L d (see in Figure 2 ), which explicitly imposes the semantic relationships between different input samples as an induced structural constraint on the hidden states of MLLMs, facilitating the model in capturing global semantics. We conceptualize the derivation of semantic relationships between input samples as a Dynamic Sequence Alignment problem [67] . Additionally, we straightforwardly select a token in the hidden state of the MLLM to encompass all preceding input information, eliminating the need for training any specialized tokens.To further effectively distinguish detailed semantics, we integrate a novel kernel into the Dynamic Time Warping framework. Leveraging the strengths of various discriminative pre-trained models.Combined with this newly proposed loss with a hyperparameter α, the training objective can be formulated as:EQUATIONWe formulate the computation of relationships within input interleaved sequences as a dynamic sequence alignment problem, and solve it by global alignment kernel. For two interleaved image-text sequence, each consisting of n and m images/sentences in total respectively (which we'll refer to as slices later on). We encode and normalize each slice, resulting in two sequences x = (x 1 , . . . , x n ) and y = (y 1 , . . . , y m ) all of which take values in a state space X , that is two elements of X ⋆ def = ∞ i=1 X i . In our setting, X is simply R d , d refers to the feature dimension. We define the global alignment kernel as follows, and it has been proved to be positive-definite under mild conditions and may prove more robust to quantify the similarity of two sequences [73, 68] :EQUATIONFollowing the suggestion by [18] , we letEQUATION), σ is standard deviation, and it can be calculated by σ = δ M +N 2 for x i , y i in x, y. δ is a fixed pre-defined hyperparameter and φ x π1(i) , y π2(i) is the distance between slice x π1(i) and y π2(i) for an alignment (details for the definition of alignment can be seen in Appendix D.2).Due to the causal attention mechanism, the token in hidden state of MLLM can encapsulate information from preceding tokens in the sequence. Therefore, we directly utilize the last token d i of a sequence from the MLLM's hidden state and map it to the r i using an MLP to represent the entire in-context sequence. During training, we obtain a set of (r 1 , r 2 , . . . , r n ) and their corresponding input sequence embedding set (x 1 , x 2 , . . . , x n ). It is noteworthy that r i and r j (x i and x j ) may originate from the same sequence but occupy different positions, thus enabling our method to utilize samples more efficiently.Leveraging the GAK, we can derive the similarity matrix of (r 1 , r 2 , . . . , r n ) and (x 1 , x 2 , . . . , x n ) distinctively, denoted as M r , M l ∈ R n×n . For imposing the semantic relationships between different input samples as an induced structural constraint on the hidden state of MLLMs, we employ Mean Squared Error (MSE) loss aligned M r with the label matrix M l . This approach eliminates the need for pre-defined label (i.e., positive and negative candidates) during training, allowing seamless integration into the aforementioned training framework (for specific training templates, please refer to Appendix E.1). Thus, we have the discriminative loss L d :EQUATIONAdditionally, when both x and y contain only one slice, the computed result of the formula is monotonically increasing with the directly calculated cosine similarity (proof can be seen in Appendix 3). Therefore, in such cases, we simplify the computation by directly using cosine similarity. If r i and r j comes from the same input interleaved sample, we manually set m l ij = 1.To further effectively distinguish detailed semantics, we further propose the Triple Kernel (TK), a positive definite kernel compatible with the previous framework. The TK leverages representations from diverse pre-trained discriminative models across uni-modal and cross-modal settings, harnessing their respective strengths. The definition is as below:For two slice a, b ∈ R d , meets (i) |a| = |b| = 2 , d = d 1 +d 2 , a = concat(a 1 , a 2 ), b = concat(b 1 , b 2 ), a 1 , b 1 ∈ R d1 , a 2 , b 2 ∈ R d2 and |a 1 | = |a 2 | = |b 1 | = |b 2 | = 1; or (ii) |a| = |b| = 1.We define tripe kernel as follows:φ(a, b) =    ||a 1 -b 1 || 2 |a| = |b| = 2 and a, b in uni-modal ||a 2 -b 2 || 2 |a| = |b| = 2 and a, b in cross-modal ||a -b|| 2 else (4)We prove triple kernel φ is a conditionally positive-definite kernel defined on X × X → R (Appendix 2), aligning with the kernel definition in [18] , thereby possessing its properties.In practice, we let the feature dimensiond = d 1 + d 2 .For images, we employ DINOv2-base [68] and CLIP ViT-L/14 [73] for encoding, then concatenate the embeddings after normalization. For sentences, we utilize BGE-base [87] and CLIP ViT-L/14, keeping the dimension unchanged. By utilizing the Triple Kernel, we can fully leverage the strengths of these three models, effectively distinguishing detailed semantics. They are together on HalloweenTell me the landmark of Singapore.Sands is an integrated resort facing Marina Bay in Singapore.A girl in the Middle Ages. the bride and her bridesmaids looked absolutely gorgeous .Sugarit was so precious when they cut the cake together .they are going to be such a great couple . their love was so apparent .[male] is at his graduatio n .he has only one ticket for someone to join him .the person he chose is waiting in the crowd , extremely excited .she takes a picture of him holding his diploma .of course it was his mom ! they made their vows to each other .His mother feels that he is her pride A man read out the wedding ceremony, and the groom and bride exchanged rings. Table 1 : Comparison with state-of-the-art methods on 11 visual-language benchmarks. We mark the best performance bold and the second-best underlined. Benchmark names are abbreviated due to space limits. VQA-v2 [27] ; GQA [35] ; VizWiz [28] ; SQA I : ScienceQA-IMG [61] ; VQA T : TextVQA [76] ; POPE [51] ; MME P , MME C : MME Perception, MME Cognition [95] ; MMB: MMBench [58] ; LLaVA Wd : LLaVA-Bench(In-the-Wild)-Detail [56] ; MM-Vet [99] . * indicates the training images of the datasets are observed during training.Sugar SugarSugar Sugar Sugar Sugar 
experiments: 
 To assess Sugar's generative ability, we conduct a comprehensive comparison with state-of-the-art models on 11 commonly used visual-language benchmarks in Section 4.2. Furthermore, we evaluate more complicated multimodal comprehension tasks on DEMON with 29 datasets in Section 4.3.For discriminative tasks, we compare performance across three different retrieval tasks: image-text retrieval, interleaved retrieval, and fine-grained retrieval in Section 4.4. Subsequently, we leverage Sugar's discriminative ability for retrieval-augmented generation compared with common used retrieve module in Section 4.5. Finally, we conduct ablation experiments to analyze the effectiveness of our method in Section 4.6.We apply our method to VILA [52] , a recent state-of-the-art MLLM supporting interleaved input. We further fine-tune VILA using LoRA [30] . Details about the experiments setting, datasets and the instruction examples, please check in Appendix E.We conduct a comprehensive comparison with state-of-the-art models on 11 commonly used benchmarks, as shown in Table 1 . Compared to existing models, Sugar achieves remarkable improvements over the second-best performing model on tasks requiring fine-grained semantics (i.e., LLaVA Wd [56] , VizWiz [28] , SQA [61] improve by 8%, 4.5%, 1.8% respectively) and benchmarks for detecting hallucinations (i.e., POPE [51] ), while maintaining competitive results in other tasks. Notably, Sugar excels in discriminative tasks and still achieves 5 state-of-the-art results and 3 second-best results on 11 benchmarks for generative tasks, even outperforming some models larger than 7B. Our results demonstrate the benefits of incorporating the discriminative loss, aiding in fine-grained semantic tasks and reducing hallucinations. Image-text Retrieval. We evaluated the performance of Sugar on the widely adopted MSCOCO [38] dataset in the context of a standard image-text retrieval task. Sugar demonstrated comparable performance to FROMAGe [40] in R@1 and surpassed it in R@5 and R@10, highlighting Sugar's superiority in normal retrieval tasks.Interleaved Retrieval. To assess the proficiency of Sugar in processing multimodal contextual information, we evaluated its performance in retrieving relevant images conditioned on sequences of interleaved image-text inputs from the Visual Storytelling (VIST) dataset [32] . We conducted evaluations across several experimental configurations, following the same setup as FROMAGe [40] (see Appendix F.1). Our results show that Sugar outperforms FROMAGe in most settings, particularly achieving a 20.3% improvement in the 5c+4i configuration, significantly surpassing both CLIP and BLIP-2. This demonstrates that our method effectively leverages MLLMs' ability to handle complex interleaved sequence inputs, thereby achieving superior retrieval performance.Fine-grained Retrieval. We tested fine-grained retrieval using the Winoground dataset [79] , which evaluates the ability to perform vision-linguistic compositional reasoning. Surprisingly, Sugar outperformed all discriminative pre-trained models (both single-stream and dual-stream encoder architectures), achieving improvements of 5.3%, 77.1%, and 86.2% over the second-best model in the Text, Image, and Group dimensions, respectively. This demonstrates Sugar's strong capability to distinguish detailed semantics and performing compositional reasoning.Due to Sugar's dual capabilities in both discrimination and generation, we can achieve retrieval augmentation without the need for an additional retrieval module. For performing retrieval-augmented generation (RAG), we selected two tasks, namely VizWiz and SQA I , as they offer held-in data that were not seen during model training. We utilized a mixed set comprising the widely-used LLaVA-1.5 SFT subset and the held-in datasets of the two tasks as the knowledge base and employed different For MSCOCO, we report the R@5 in text-to-image retrieval. For VIST, we report the R@5 of retrieving an image given 5 captions and 4 images. For Winoground, we report the Image score. For other tasks, we report Accuracy (%).retrieval modules to retrieve relevant knowledge for the MLLM. The results are as follows: (i) We observed a drop in performance for LLaVA-1.5 with RAG in all tasks. This may be because LLaVA is designed solely for single-image input, without the ability to utilize in-context external knowledge.(ii) Compared to VILA, Sugar's performance improved in both tasks, whereas VILA improved in SQA I but decreased in VizWiz. These findings suggest that Sugar's retrieved knowledge is more beneficial, while the knowledge retrieved by CLIP and BLIP-2 may hinder performance.Importance of Both Tasks. (1) w/o data g : As shown in 5) and utilization of CLIP for encoding the input sequence instead, a dramatic performance decline is evident in Winoground, with a 43.5% decrease. This underscores the significant role of TK in facilitating the distinction of detailed semantics.(3) w/o AvgPool: When solely using the last token for retrieval, a general decline in performance is observed across discriminative tasks, with decreases of 19.1% for MSCOCO, 22.6% for VIST, and 13.2% for Winoground. This phenomenon may be attributed to the last token of an image often corresponding to a pooling token, containing relatively weaker semantic information. Utilizing all image tokens and performing AvgPooling tends to yield greater improvements in retrieval tasks. 
conclusion: 
 Vision-Language Models (VLMs) have been trained using both generative and discriminative paradigms, each with distinct advantages and limitations. To bridge this gap, we introduce Structureinduced approach to unify generative and discriminative paradigms, which imposes semantic relationships between input samples, thereby enhancing the MLLM's ability to capture global semantics and distinguish fine-grained details. This approach effectively balances generative and discriminative tasks, yielding synergistic benefits. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, particularly those requiring cognitive and discrimination abilities, while also demonstrating competitive performance in discriminative tasks such as image-text retrieval and achieving state-of-the-art results in interleaved and fine-grained retrieval. Furthermore, employing a retrieval-augmented generation strategy within a single model leads to additional improvements, offering a promising direction for future research. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We give the statements of experiments compute resources in Appendix E.2.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the Broader Impacts in Appendix A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have already cited all the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The broader impact of Sugar, carries both potential benefits and risks upon deployment and release. Some considerations are unique due to their visual nature, while others mirror existing instructionfollowing Large Language Models (LLMs). Built upon Vicuna, CLIP, DINOv2, and BGE, Sugar inherits issues associated with LLMs and vision encoders. Below, we outline risks and mitigation strategies for its release.Hallucination. Similar to other MLLMs, Sugar may generate outputs detached from facts or input data, posing concerns, especially in critical applications like medicine and the field related to security.Biases. Bias from base models can transfer to Sugar, originating from both the vision encoder (CLIP) and the language decoder (Vicuna), potentially leading to biased outcomes or unfair representations.Ethical Impacts. This study doesn't raise ethical concerns, as it doesn't involve subjective assessments or private data, only utilizing publicly available datasets.Expected Societal Implications. A significant societal concern lies in potential misuse, such as fabricating unauthorized texts leading to misinformation, privacy breaches, and other damaging consequences. Strong ethical standards and ongoing surveillance are essential for mitigation.These issues aren't unique to our method but are prevalent across different techniques for multiconcept customization. Despite the risks, we believe the benefits outweigh the potential harm, allowing ongoing investigation and improvement of the model while engaging the community in developing better mitigation strategies. Moreover, its release can foster new applications and research directions, contributing to the progress and responsible deployment of foundation models in visionlanguage tasks.(i) Our method, while effective, may inherit limitations from the underlying models, such as hallucination in generating outputs detached from facts or input data and potential biases originating from the model we used. (ii) The training data might inevitably contain mismatched image and text, which could adversely affect training.Theorem 1. The alignment kernel K can be computed in quadratic complexity, namely in O(mnd 2 ) iterations. where m, n denotes the length of two sequence and their hidden dimension all is d, m, n, d ∈ R.Proof. Given x = (x 1 , . . . , x n ) and y = (y 1 , . . . , y m ) two sequences of X ⋆ , we set the doublesubscripted series M i,j as M i,0 = 0 for i = 1, ..., n, M 0,j = 0 for j = 1, ..., m, and M 0,0 = 1.Computing recursively for (i, j) ∈ {1, ..., n} × {1, ..., m} the termsM i,j = (M i,j-1 + M i-1,j-1 + M i-1,j )k(x i , y j )we obtain that K(x, y) = M n,m The result can be proved by recursion and is intuitively an equivalent of the Dynamic Time Warping(DTW) [67, 17] algorithm where the max-sum algebra is simply replaced by the sum-product one [18] .Theorem 2. triple kernel φ is a conditionally symmetric positive-definite kernel [18] defined on X × X → R.Proof. (i) for two slice a, b ∈ R d , meets |a| = |b| = 2 , d = d 1 + d 2 , a = concat(a 1 , a 2 ), b = concat(b 1 , b 2 ), a 1 , b 1 ∈ R d1 , a 2 , b 2 ∈ R d2 and |a 1 | = |a 2 | = |b 1 | = |b 2 | = 1: let a ′ = concat(a ′ 1 , a ′ 2 , a ′ 3 ), a ′ = concat(b ′ 1 , b ′ 2 , b ′ 3 ): when a ′ (b ′ ) is from text modal, we let a ′ = concat(a 1 , a 2 , 0)(b ′ = concat(b 1 , b 2 , 0)), and a ′ = concat(a 1 , 0, a 2 )(b ′ = concat(b 1 , 0, b 2 )) for image modal. 0 ∈ R d2 . we can unify the Equation 4 first and second case in:φ(a, b) = φ(a ′ , b ′ ) = (|a ′ 2 ||b ′ 2 | + |a ′ 3 ||b ′ 3 |)||a ′ 1 -b ′ 1 || 2 + (1 -|a ′ 2 ||b ′ 2 |)||a ′ 2 -b ′ 2 || 2 + (1 -|a ′ 3 ||b ′ 3 |)||a ′ 3 -b ′ 3 || 2 ≥ (|a ′ 2 ||b ′ 2 | + |a ′ 3 ||b ′ 3 |)||a ′ 1 -b ′ 1 || 2 + 0 + 0 ≥ 0 As (1 -|a ′ 2 ||b ′ 2 |) ≥ 0 and (1 -|a ′ 3 ||b ′ 3 |) ≥ 0.for any family α 1 , α 2 , ...α n ∈ X and c 1 , c 2 , ..., c n ∈ R, we have that i,jc i c j φ(x i , x j ) = i,j c i c j φ(x ′ i , x ′ j ) ≥ 0(ii) for two slice meets |a| = |b| = 1: for any family α 1 , α 2 , ...α n ∈ X and c 1 , c 2 , ..., c n ∈ R, we have that i,jc i c j φ(x i , x j ) = i,j c i c j ||x i -x j || 2 ≥ 0Additionally, it's evident that for both (i) and (ii), φ(a, b) = φ(b, a). Therefore, triple kernel φ is a conditionally symmetric positive-definite kernel defined on X × X → R Theorem 3. when both x and y contain only one slice, GAK is monotonically increasing with the directly calculated cosine similarity.Proof. let x = (x), y = (y), and cosine similarity of x and y is cos = cos < x, y >. we can get σ = δ M + N 2 = δ 1 + 1 2 = δφ σ = 1 2σ 2 φ x π1(i) , y π2(i) + log 2 -e - φ ( x π 1 (i) ,y π 2 (i)) 2σ 2 = 1 2δ 2 φ x π1(i) , y π2(i) + log 2 -e - φ ( x π 1 (i) ,y π 2 (i)) 2δ 2 = 1 -cos < a, b > δ 2 + log 2 -e -1-cos<a,b> δ 2 Letting t = 1-cos<a,b> δ 2and substituting the result of φ σ = t + log(2e -t ) into Equation 2, we obtain:K(x, y) = π∈A(x,y) |π| i=1 e -ϕσ = e -ϕσ = e -t+log( 1 2-e -t ) = e -t 2 -e -tLetting s = e -t , we can further obtain:K(x, y) = s 2 -sAs cos⟨a, b⟩ ∈ [-1, 1], s strictly increases with cos⟨a, b⟩ and s ∈ [e -2 δ 2 , 1] when the hyperparameter δ is fixed. Derivative of K(x, y) can be obtained:K(x, y) ′ = 2 -s + s (2 -s) 2 = 2 (2 -s) 2 > 0Overall, when both x and y contain only one slice, GAK is monotonically increasing with the directly calculated cosine similarity.We adopt the manifold multimodal model architecture [55, 52, 11, 6, 34] , formulated as follows:Visual Representation. We first process x img subject to a visual representation backbone V ω that outputs a sequence of features p img ∈ R L×hvision where p img = V ω (x img ). As an example, p img might be the patch features output by a Vision Transformer.Vision-Language Projector. Next, we map p img to a of embeddings e img ∈ R L×htext via a learned projector F ψ , where e img = F ψ (p img ).Language Model. Finally, we concatenate the sequence e img with the text prompt embeddings e prompt = embed(u prompt ), passing the result to the language model. Generally, we have the interleaved image-text input x input by concatting all the e prompt and e img . The language model generates output text u gen = LM θ (x input ).Retrieval Projector. For discriminative tasks, we select the token d i from MLLM's hidden state and map it to r i via a learned projector F φ .In Implementation, we utilize CLIP ViT-L/14 [73] as the visual encoder, and Vicuna 1.5 [14] as the language model.An alignment π of length |π| = p between two sequences x and y is a pair of increasing p-tuples (π 1 , π 2 ) such thatEQUATIONWe write A(x, y) for the set of all possible alignments between x and y. Intuitively, an alignment π between x and y describes a way to associate each element of a sequence x to one or possibly more elements in y, and vice versa. Such alignments can be conveniently represented by paths in the n × m grid displayed in the left of Figure 3 .with unitary increments and no simultaneous repetitions, that isEQUATIONThe score on a path is defined as:EQUATIONTraining Data. Our vision-language task datasets are a subset of VILA [52] , including MMC4 [104] , COYO [9] , LLaVA-1.5 SFT dataset [55] .We use a prompt template formatted as (system-message is a system prompt from Vicuna, and the following messages all have the same meaning.):{system-message}. USER: <image>\n {question}. ASSISTANT: {answer}.For interleaved vision-language datasets, the template is formatted as:{system-message}. USER: {interleaving question}. ASSISTANT: {answer}.Training Strategy. To jointly train the discriminative loss and generative loss, we calculate the loss as follows. Since the last token of an image is often a padding token, we take all 576 hidden state tokens before the LM head for images and apply average pooling to obtain a single token. For text, we directly take the last toke in the hidden state of MLLM.During training, We calculate the discriminative loss using the last token from either the end of the text or the image in the MLLM's hidden state. Notably, in an interleaved input sequence with multiple texts or images, we randomly select multiple last tokens from the same sequence to more efficiently utilize the samples.Evaluation Data. For generative tasks, we first evaluate on a wide range of question-answering tasks and some MLLM-oriented comprehension benchmarks, including VQA-v2 [27] , GQA [35] , VizWiz [28] , ScienceQA-IMG [61] , TextVQA [76] , POPE [51] , MME [95] , MMBench [58] , LLaVA-Bench (In-the-Wild) [56] , MM-Vet [99] and. The split of test sets and the evaluation metrics are aligned with those described in VILA [52] and LLaVA [55] .To test the generative ability in interleaving tasks, we use DEMON [47] , a comprehensive benchmark that demonstrative instruction following ability, including a wide variety of multi-modal datasets from different fields and scenarios.For generation tasks, our evaluation encompasses MSCOCO [38] for image-text retrieval, Visual Storytelling (VIST) [32] for interleaved retrieval and Winoground [79] for fine-grained retrieval.We train the parameters for both the LLM and the MLP for embedding the MLLM's hidden state, initializing from VILA [52] . To enhance efficiency for the LLM, we employ LoRA tuning [30] on the W q and W v matrices using low-rank adaptation. In our implementation, we set the rank r = 128 and α = 256. We utilize the AdamW optimizer [60] in conjunction with a cosine learning rate scheduler.The hyperparameters for the AdamW optimizer are configured with a warm-up ratio of 0.03 and a maximum learning rate of 1e -4. Training is conducted on 8 x A800 GPUs for approximately 12 hours.(a) WebQA. The original WebQA contains two types of questions: "Qcate": "text" (open-ended questions) and "Qcate": "YesNo" (binary judgment questions). For ease of evaluation, we only used the second type. We selected 500 samples of the "YesNo" question type from WebQA [10] , each containing one relevant image-text pair and five unrelated image-text pairs. Since the original data provides responses in declarative sentences, we modified the answers of these samples to be either "yes" or "no" by prompting with "please answer the question in Yes or No."We transformed this dataset into a question-answering format. Each question takes the following form (Due to display problems, we have performed line breaks, the same below.):{system-message}. USER: {qustion}\n{image-text pairs}.\n please answer the question in Yes or No.\n ASSISTANT: {answer}.Here is a case for one sample in Figure 5 : In WebQA, the accuracy roughly forms a "U" shape curve when the relevant image-text pair for a question appears at different positions. While Sugar also shows similar trends, it tends to be more stable overall. Specific numerical results can be found in Table 5 . Among them, both Statement 1 and Statement 2, as well as Image 1 and Image 2, are highly similar, with only subtle differences. Furthermore, there is a corresponding relationship between Image 1 and Statement 1, and between Image 2 and Statement 2. We employed a random seed to ensure that the correct answer is equally distributed between option A and option B. The specific values for the experiment are provided in Table 6 . For performing retrieval-augmented generation (RAG), we selected two tasks, namely VizWiz and SQA I , as they provide held-in data not seen during model training. We did not use VQA v2 , GQA, and VQA T because their held-in data is a subset of the widely-used LLaVA-1.5 SFT. Benchmarks like POPE, MMB, and others lack held-in data. Therefore, we focused on VizWiz and SQA I for our experiments. We utilized a mixed set comprising the widely-used LLaVA-1.5 SFT subset and the held-in dataset of the two tasks as the knowledge base and employed different retrieval modules to retrieve relevant knowledge for the MLLM. Similar to common practice, we average the similarity scores for CLIP (We choose CLIP ViT-L/14@336px). For BLIP-2, we compute the similarity using its multimodal token's CLS token. Figure 7 shows some specific retrieval results on test data, demonstrating that Sugar can better integrate information from both images and text, retrieving more similar data as external knowledge.What's in this box please? thank you. \nWhen the provided information is insufficient, respond with 'Unanswerable'.\nAnswer the question using a single word or phrase. ours What type of snack is this?\nWhen the provided information is insufficient, respond with 'Unanswerable'.\nAnswer the question using a single word or phrase. index and retrieve candidates. Therefore, the results may exhibit slight differences when compared under identical settings. The results in Table 3 (a) are provided for reference only.Interleaved Retrieval. We conduct evaluations across several experimental configurations, following the same setup as FROMAGe [40] . The settings are as follows:1. Retrieval of the last image given the descriptions of the preceding 5 images. This evaluates models' ability to condition on temporally dependent language.2. Retrieval of the last image given the descriptions of the preceding 5 images and the 4 preceding images. This assesses models' capability to process interleaved image-and-text context.Fine-grained Retrieval. Winoground [79] is designed to evaluate the ability of vision and language models to perform vision-linguistic compositional reasoning. The task involves matching two images with two captions, where captions contain an identical set of words/morphemes arranged in different orders. This dataset, meticulously hand-curated by expert annotators, includes a rich set of fine-grained tags to facilitate detailed performance analysis.To analyze Sugar's emergent behaviors and observed weaknesses, we present additional qualitative samples that were not included in the main paper due to space constraints. Please note that for brevity, we have omitted the system prompts and the line breaks after the images for all the quality examples.We hope these additional results and observations showcase the potential of Sugar in various application areas. In future work, it is important to investigate these emergent behaviors more thoroughly and to understand the underlying mechanisms that enable Sugar to demonstrate such generalization abilities. This will pave the way towards building better MLLMs, including enhancing robustness, reducing biases, and improving the alignment and scope of the learned vision-language representations. World Knowledge: We observe that Sugar can leverage the world knowledge [26] embedded within the LLM to enhance performance on multimodal tasks. For example, as shown in Figure 4 , the model understands that during Halloween, people typically dress up in various ways to portray scary, funny, or creative characters, such as ghosts and skeletons.Retrieval the Same Sequence at Different Place: One interesting emergent behavior of Sugar is its ability to retrieve sequences from different positions within the input interleaved sequence, demonstrating flexibility and high sample efficiency, as shown in Figure 4 . Unlike CLIP, which requires encoding each sample separately, Sugar can encode sequences of varying lengths for the same multi-modal document in a single forward pass.What's more, Sugar is capable of both retrieval and generation tasks. Below in Figure 8 are some examples from the VIST dataset. Fine-grained Image Discrimination: As shown in Figure 9 , Sugar excels at accurately discerning subtle differences between images and identifying detailed objects and their attributes. VILA, on the other hand, tends to describe the content of the images without pinpointing the precise differences between them. In contrast, Sugar provides more concise and direct answers.Style Following: Sugar exhibits a certain degree of in-context style following capability. As shown in Figure 10 , with the aid of external knowledge, Sugar partially adopts the style of retrieved results, resulting in more accurate and detailed answers compared to scenarios without retrieval augmentation.What's the best weather, season, time of the day of visiting this place? Is the time when this photo was taken a good time to visit this place? Yes. +RAG When visiting this serene lakeside location, be cautious about the following:\n\n1. Wooden pier safety: Ensure the T-shaped wooden pier is sturdy and safe to walk on, and be mindful of any loose planks or slippery surfaces.\n\n2. Swimming conditions: If you plan to swim in the lake, check for potential hazards such as submerged objects or strong underwater currents, and be aware of the water temperature.\n\n3. Wildlife: Some lakes may have local wildlife, such as snakes or alligators, which could pose a threat. Remain cautious and observant, especially near the water's edge.\n\n4. Weather conditions: The sky is cloudy, so be prepared for potential rainfall or changes in temperature, and monitor weather forecasts before your visit.\n\n5. Hiking in the surrounding area: If you plan to explore the green trees and mountains in the background, be cautious of potential hazards like steep and slippery terrain or unstable trail conditions. Carry appropriate hiking gear and wear sturdy footwear.\n\n6. Protect the environment: Be respectful of the natural beauty of the area, and avoid littering, damaging any trees or plants, or disturbing wildlife.What are the things I should be cautious about when I visit here?The best time to visit this serene lakeside location is during the day, as the weather is clear and the lighting is good for photography. The photo was taken during the day, and the weather appears to be clear, making it a good time to visit.What is unusual about this image?A man ironing clothes on car.The city might be facing traffic congestion issues, particularly in the four-lane sections of the highway since most cars have their brake lights on, indicating slow or stopped traffic. Possible solutions could include:\n\n1. Implementing traffic management strategies, such as ramp meters to control the rate at which cars enter the highway or optimizing traffic signal timings to improve traffic flow.\n2. Encouraging the use of public transportation, carpooling or ride-sharing to reduce the number of cars on the road.\n3. Expanding or adding more lanes to ease traffic congestion in the four-lane sections of the highway.\n4. Exploring alternative routes to divert traffic from the congested areas.\n5. Improving walking and biking infrastructure in the city, thus promoting more sustainable and non-motorized modes of transport.What is the problem this city might be facing? What are some possible solutions?The unusual aspect of this image is the presence of a man standing on the hood of a car while ironing clothes. This is not a common sight and could potentially be dangerous. Interleaved Comprehension: As demonstrated by results from DEMON [47] , Sugar exhibits superior interleaved comprehension capabilities compared to VILA, particularly in tasks requiring fine-grained analysis and an understanding of global context. For instance, in the third example of Figure 11 , VILA confuses character names, whereas Sugar maintains narrative coherence while adhering to the style of the preceding text. Similarly, in the third example of Figure 12 , VILA provides an irrelevant response, while Sugar delivers a more contextually appropriate answer. Additionally, Figure 13 demonstrates Sugar's ability to effectively capture global information, identifying the relevant images and text within the sequence to provide accurate responses.Style Following Sugar• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: The research conducted in the paper conforms, in every respect, to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper proposes a novel method for unifying generative and discriminative training paradigms in multi-modal large language models (MLLMs). The approach aims to address the limitations of existing Vision-Language Models (VLMs) by integrating generative training, which typically struggles with issues like hallucinations, and discriminative training, which often falls short in complex scenarios requiring fine-grained semantic differentiation. The authors introduce a structure-induced training strategy that leverages dynamic sequence alignment within the Dynamic Time Warping (DTW) framework, enhancing the model's ability to capture both global and fine-grained semantics.
After the rebuttal, this paper received all positive recommendations (two weak accepts and one borderline accept).  The major concern from reviewers was about the Sugar's effectiveness, which has been addressed in the rebuttal. The AC agrees with the reviewer’s assessments and recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper proposed a method for unifying generative training and discriminative training of multi-modal LLMs. Generative training mainly uses auto-regressive formulation while discriminative training mainly performs contrastive representation matching. The goal of this paper is trying to use discriminative training to improve multi-modal LLMs.

The paper unifies generative training and discriminative training by introducing a Dynamic Sequence Alignment module which aligns similar text and image data on the hidden states of a multi-modal LLM. In addition, Detailed Semantics Modeling is proposed to effectively distinguish detailed semantics.

The paper conducts evaluation on a wide range of benchmarks.
strength_weakness: strength_weakness: It's unclear what is dynamic time warping framework.

The performance improvement of the proposed method sugar is not significant. Compared with some baselines, such as VILA and LLaVA-1.5, Sugar performs worse than them on many tasks, as shown in Table 1. This raises concerns about the effectiveness of the proposed method. 

This could be meaningless to align a visual token and a text token in an MLLM model since the LLM is trained with next-token prediction instead of contrastive learning like CLIP. The current token is conditioned on previous tokens. I can't think of a reasonable explanation for this mechanism. It **could be** meaningful to align visual tokens. In addition, the experiment results also suggest that this method is not effective as expected. 

What is the evaluation protocol?  Does Sugar train on each benchmark first then evaluate or directly zero-shot evaluation? In the former case, will Sugar lose generative ability after training with discriminative task data?
questions: questions: What is the training recipe of the proposed method?
limitations: limitations: -
paper_summary: paper_summary: The paper addresses the limitations of Vision-Language Models (VLMs) by proposing a unified approach that combines generative and discriminative training paradigms. This new method leverages interleaved image-text sequences and introduces a structure-induced training strategy. It aims to enhance the MLLM's ability to capture global semantics and fine-grained details, effectively balancing generative and discriminative tasks. The approach uses dynamic sequence alignment within the Dynamic Time Warping framework and integrates a novel kernel for fine-grained semantic differentiation. Extensive experiments demonstrate that this method achieves state-of-the-art results in various generative and discriminative tasks.
strength_weakness: strength_weakness: - While the paper shows impressive results, there is limited discussion on the potential limitations and areas where the model might underperform.
- The paper primarily focuses on specific benchmarks. It would be beneficial to discuss how well the proposed method generalizes to other types of vision-language tasks not covered in the experiments.
questions: questions: - Can you provide more detailed ablation studies to understand the contribution of each component of the proposed method, such as the dynamic sequence alignment and the GAK?
limitations: limitations: - Considering the connection with real-world scenarios is necessary. The authors need to discuss: what requirements might the tasks introduced in the paper have in real-world scenarios?
paper_summary: paper_summary: This paper proposes a novel learning paradigm to learn MLLMs based on interleaved image-text corpora. 
It introduces a structure-induced training strategy that imposes semantic relationships between input samples and
 the MLLM’s hidden state.
This work apply the dynamic time warping framework to calculate the semantic similarity between different image-text sequences.
Then, a discriminative loss is applied to sequence similarity matrices calculated based on raw inputs and MLLM hidden states. 
The framework can also leverage the capabilities of multiple vision and language encoders to more accurately calculate the similarity matrices.
Experiment results show that the new learning paradigm demonstrates good performance on basic multimodal comprehension benchmarks, 
complicated multimodal comprehension benchmark DEMON, cross-model information retrieval, and retrieval-augmented generation.
strength_weakness: strength_weakness: 1. This paper did not discussed the impact of including interleaved image-text pairs in MLLM learning. For example, how will it affect the performance on basic visual-language benchmarks (Table 1) and image-text retrieval. Will there be any negative effects?
questions: questions: 1. Can sugar better leverage the multi-modal in-context examples or better understand interleaved image-text content, is there any evaluation for that?
2. What is exactly the amount of interleaved image-text sequences (from MMC4) and image-text pairs (from other datasets) used to train Sugar. 
3. What is the context window size of Sugar?
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Awaiting Your Feedback
Official Comment by Authors
Response to rebuttal
Awaiting Your Feedback
Looking Forward to Your Reply
Looking Forward to Your Reply
Table F
Table E
Table E
Table D
Table B
Table B
Table E
Table D
Table C
Table B
comment: Dear Reviewer Lcdg,

Thank you once again for reviewing our submission. We are deeply grateful for your recognition of the value of our work, particularly our proposal of the unified approach that combines generative and discriminative training paradigms to address the challenges of global semantic understanding and detailed semantic capture in existing VLMs. In response to your thoughtful suggestions regarding performance across various vision-language task types and real-world applications, we have conducted additional experiments on 5 more challenging benchmarks, as well as a knowledge-based VQA task of practical significance. These efforts further demonstrate the effectiveness and practical value of our method.

Over the past few days, we have carefully addressed the concerns of other reviewers. They have kindly raised their scores and agreed that our approach not only addresses the limitations of VLMs and enhances comprehension capabilities, but also offers emergent insights into complex retrieval tasks and the unified nature of generative and retrieval abilities, which could inspire future work.

As the author-reviewer discussion period for NeurIPS 2024 is drawing to a close, please feel free to let us know if you have any remaining concerns or questions. We greatly appreciate your feedback and look forward to any further insights you may have.
Thank you for your support of our work. Your valuable feedback has made our work better!
My concerns are well resolved in the rebuttal, and I would like to raise my rating to 6.
Thank you for increasing the score. Your valuable suggestions greatly contribute to the quality of our manuscript. Thank you again for your precious time and valuable suggestions!
Thanks for authors' rebuttal, I think they have addressed most of my concerns, so I will raise my score.
Dear Reviewer 4VP3,

Thank you again for reviewing our submission. As the author-reviewer discussion period for NeurIPS 2024 is nearly over, please let us know if any further information or clarification is needed. We are ready to engage in any further discussions with you！
Looking forward to your further feedback!
Dear Reviewer DDMF,

Thank you for the time and effort you have dedicated to reviewing our submission. We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work. As the author-reviewer discussion period for NeurIPS 2024 will be over soon, please let us know if you require any additional information or clarification from our end. We are open to engage in further discussions to enhance our submission. Thank you!
Dear Reviewer 4VP3,

Thank you for the time and effort you have dedicated to reviewing our submission. We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work. As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you require any additional information or clarification from our end. We are open to engage in further discussions to enhance our submission.
Table F: Ablation study on the effects of MMC4. We evaluated basic visual-language benchmarks, including VQA and GQA. For image-text retrieval, we reported MSCOCO R@5 for both text-to-image(t2i) and image-to-text (i2t). Additionally, we included Mantis for multi-image VQA and VIST R@5 for complex retrieval tasks.
|                             |                   | basic visual-language benchmarks |      |        |      |      |      |      | image-text retrieval |                | multi-image VQA | complex retrieval |
| --------------------------- | ----------------- | -------------------------------- | ---- | ------ | ---- | ---- | ---- | ---- | -------------------- | -------------- | --------------- | ----------------- |
|                             | MMC4 sample ratio | VQA                              | GQA  | VizWiz | SQAI | VQAT | POPE | MMB  | t2i    | i2t | Mantis          | VIST      |
| VILA-7B                     |                   | 79.9                             | 62.3 | 57.8   | 68.2 | 64.4 | 85.5 | 68.9 | /                    | /              | 38.4            | /                 |
| + directly fine-tune        | 25%               | 72.3                             | 59.9 | 53.5   | 64.1 | 58.7 | 84.8 | 63.4 | /                    | /              | 38.5            | /                 |
|                             | 50%               | 67.8                             | 54.7 | 49.1   | 58.9 | 51.8 | 82.8 | 59.7 | /                    | /              | 38.7            | /                 |
|                             | 75%               | 60.5                             | 49.1 | 39.3   | 43.4 | 43.2 | 81.7 | 53.2 | /                    | /              | 38.2            | /                 |
| + fine-tune with our method | 25%               | 78.7                             | 60.2 | 60.6   | 70.1 | 62.1 | 85.9 | 66.1 | 40.3                 | 36.2           | 38.7            | 14.4              |
|                             | 50%               | 75.9                             | 57.9 | 59.2   | 65.9 | 57.0 | 86.4 | 62.3 | 40.7                 | 36.3           | 39.0            | 19.1              |
|                             | 75%               | 73.7                             | 55.1 | 57.1   | 62.3 | 51.7 | 85.7 | 59.7 | 40.2                 | 36.0           | 39.5            | 23.2              |

*We also present the results in Table F of the Rebuttal PDF.*
Table E: Comparison with baseline in 5 challenging tasks.
|                     | **Multi-image VQA** |          | **In-context Interaction** |          | **Visual Prompt** |
| ------------------- | ------------------- | -------- | -------------------------- | -------- | ------------------ |
|                     | SEED                | Mantis   | Visual Dialog              | VIST-SIS | BLINK              |
| LLaVA-1.5-7B        | 58.6                | 31.3     | 53.7                       | 14.1     | 37.1               |
| VILA-7B             | 61.1                | 38.4     | 58.5                     | 17.4     | 39.2               |
| Sugar | **63.6**            | **41.0** | **62.5**                 | **28.2** | **42.2**           |

*We also present the results in Table E of the Rebuttal PDF.*
Table E: Comparison with baseline in 5 challenging tasks.
|                     | **Multi-image VQA** |          | **In-context Interaction** |          | **Visual Prompt** |
| ------------------- | ------------------- | -------- | -------------------------- | -------- | ------------------ |
|                     | SEED                | Mantis   | Visual Dialog              | VIST-SIS | BLINK              |
| LLaVA-1.5-7B        | 58.6                | 31.3     | 53.7                       | 14.1     | 37.1               |
| VILA-7B             | 61.1                | 38.4     | 58.5                     | 17.4     | 39.2               |
| Sugar | **63.6**            | **41.0** | **62.5**                 | **28.2** | **42.2**           |

*We also present the results in Table E of the Rebuttal PDF.*
Table D: Knowledge-based VQA.
|                               | FVQA     | WebQA    |
| ----------------------------- | -------- | -------- |
| LLaVA-1.5-7B                  | 5.9      | /        |
| LLaVA-1.5-7B + CLIP image     | 6.8      | 81.8     |
| LLaVA-1.5-7B + CLIP text      | 7.1      | 79.2     |
| LLaVA-1.5-7B + CLIP (average) | 7.9      | /        |
| VILA-7B                       | 6.4      | /        |
| VILA-7B + CLIP image          | 9.0      | 80.0     |
| VILA-7B + CLIP text           | 10.2     | 71.2     |
| VILA-7B + CLIP (average)      | 11.0     | /        |
| **Sugar**                     | **6.5**  | /        |
| **Sugar + rag**               | **20.7** | **88.7** |

*We also present the results in Table D of the Rebuttal PDF.*
Table B: Comparison with baseline on 11 visual-language benchmark with equitable data volume.

|                     | VQA       | GQA        | VizWiz   | SQAI     | VQAT     | POPE     | MMEP       | MMEC      | MMB      | LLaVAWd  | MM-Vet   |
| ------------------- | --------- | ---------- | -------- | -------- | -------- | -------- | ---------- | --------- | -------- | -------- | -------- |
| LLaVA-1.5-7B        | 78.5     | 62.0      | 50.0     | 66.8     | 58.2     | 85.9     | 1510.7     | –         | 64.3     | 49.0     | 30.5     |
| VILA-7B             | 79.9    | 62.3      | 57.8     | 68.2     | 64.4     | 85.5     | 1533.0     | 296.1     | 68.9     | 70.0     | 34.9     |
| Sugar               | 76.0    | 58.7     | 60.4     | 69.4     | 57.5     | 86.6     | 1550.8     | 300.0     | 64.9     | 75.6     | 31.3     |
| with equitable data volume | **80.2** | **63.1** | **61.0** | **72.1** | **65.1** | **87.2** | **1550.0** | **309.0** | **69.3** | **76.4** | **36.8** |

*We also present the results in Table B of the Rebuttal PDF.*
Table B: Comparison with baseline on 11 visual-language benchmark with equitable data volume.

|                     | VQA       | GQA        | VizWiz   | SQAI     | VQAT     | POPE     | MMEP       | MMEC      | MMB      | LLaVAWd  | MM-Vet   |
| ------------------- | --------- | ---------- | -------- | -------- | -------- | -------- | ---------- | --------- | -------- | -------- | -------- |
| LLaVA-1.5-7B        | 78.5     | 62.0      | 50.0     | 66.8     | 58.2     | 85.9     | 1510.7     | –         | 64.3     | 49.0     | 30.5     |
| VILA-7B             | 79.9    | 62.3      | 57.8     | 68.2     | 64.4     | 85.5     | 1533.0     | 296.1     | 68.9     | 70.0     | 34.9     |
| Sugar               | 76.0    | 58.7     | 60.4     | 69.4     | 57.5     | 86.6     | 1550.8     | 300.0     | 64.9     | 75.6     | 31.3     |
| with equitable data volume | **80.2** | **63.1** | **61.0** | **72.1** | **65.1** | **87.2** | **1550.0** | **309.0** | **69.3** | **76.4** | **36.8** |

*We also present the results in Table B of the Rebuttal PDF.*
Table E: Comparison with baseline in 5 challenging tasks.
|                     | **Multi-image VQA** |          | **In-context Interaction** |          | **Visual Prompt** |
| ------------------- | ------------------- | -------- | -------------------------- | -------- | ------------------ |
|                     | SEED                | Mantis   | Visual Dialog              | VIST-SIS | BLINK              |
| LLaVA-1.5-7B        | 58.6                | 31.3     | 53.7                       | 14.1     | 37.1               |
| VILA-7B             | 61.1                | 38.4     | 58.5                     | 17.4     | 39.2               |
| Sugar | **63.6**            | **41.0** | **62.5**                 | **28.2** | **42.2**           |

*We also present the results in Table E of the Rebuttal PDF.*
Table D: Knowledge-based VQA.
|                               | FVQA     | WebQA    |
| ----------------------------- | -------- | -------- |
| LLaVA-1.5-7B                  | 5.9      | /        |
| LLaVA-1.5-7B + CLIP image     | 6.8      | 81.8     |
| LLaVA-1.5-7B + CLIP text      | 7.1      | 79.2     |
| LLaVA-1.5-7B + CLIP (average) | 7.9      | /        |
| VILA-7B                       | 6.4      | /        |
| VILA-7B + CLIP image          | 9.0      | 80.0     |
| VILA-7B + CLIP text           | 10.2     | 71.2     |
| VILA-7B + CLIP (average)      | 11.0     | /        |
| **Sugar**                     | **6.5**  | /        |
| **Sugar + rag**               | **20.7** | **88.7** |

*We also present the results in Table D of the Rebuttal PDF.*
Table C: DEMON benchmark with equitable data volume.
|                            | MMD      | VST      | VRI      | MMC      | KGQA     | TRQA     | MMR      |
| -------------------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| LLaVA-1.5-7B               | 37.5     | 25.2     | 25.9     | 22.2     | 48.6     | 44.9     | 50.3     |
| VILA-7B                    | 47.8     | 25.8     | 13.2     | 17.2     | 60.1     | 42.1     | 50.5     |
| Sugar                      | 51.8     | 34.3     | 32.3     | 16.8     | 64.4     | 65.9     | 51.7     |
| with equitable data volume | **54.1** | **34.7** | **34.2** | **23.1** | **65.0** | **68.1** | **53.2** |

*We also present the results in Table C of the Rebuttal PDF.*
Table B: Comparison with baseline on 11 visual-language benchmark with equitable data volume.

|                     | VQA       | GQA        | VizWiz   | SQAI     | VQAT     | POPE     | MMEP       | MMEC      | MMB      | LLaVAWd  | MM-Vet   |
| ------------------- | --------- | ---------- | -------- | -------- | -------- | -------- | ---------- | --------- | -------- | -------- | -------- |
| LLaVA-1.5-7B        | 78.5     | 62.0      | 50.0     | 66.8     | 58.2     | 85.9     | 1510.7     | –         | 64.3     | 49.0     | 30.5     |
| VILA-7B             | 79.9    | 62.3      | 57.8     | 68.2     | 64.4     | 85.5     | 1533.0     | 296.1     | 68.9     | 70.0     | 34.9     |
| Sugar               | 76.0    | 58.7     | 60.4     | 69.4     | 57.5     | 86.6     | 1550.8     | 300.0     | 64.9     | 75.6     | 31.3     |
| with equitable data volume | **80.2** | **63.1** | **61.0** | **72.1** | **65.1** | **87.2** | **1550.0** | **309.0** | **69.3** | **76.4** | **36.8** |

*We also present the results in Table B of the Rebuttal PDF.*