Title: Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Multimodal Models; Shuffle Learning; Directed Tokens
Abstract: Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these models still suffer from some fundamental limitations related to robustness and generalization due to the alignment and correlation between visual and textual features. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's pre-training and fine-tuning phases. In addition,  we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to reconstruct the correct order of visual inputs. Then, we introduce a new Image-to-Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims at improving the vision-language alignment of Large Multimodal Models (LMMs). It starts by ascertaining that the popular LMM LLaVA 1.5 pays little attention to image inputs, as its VQA performance is just moderately decreased if the visual input is removed. Inspired by the jigsaw puzzle pre-training scheme, the paper proposes to add three different losses in the pre-training and visual instruction tuning stages of LLaVA: the first loss tasks the model to reconstruct the order of the input image where the patches have been permuted; similarly, the second loss tasks the model to reconstruct the order of the permuted response. This is only applied during pre-training; finally, the third Image-to-Response Guided Learning loss encourages the model to pay more attention to visual tokens. The reconstruction of the image in the right order is feasible thanks to a set of Directed Tokens that are additionally fed to the LMM and whose representation at the output layer serves to predict the index of the k-th image patch.\nExperiments are done on the same dataset employed by LLaVA and demonstrate consistent improvements over LLaVA 1.5.\n\nSTRENGTHS AND WEAKNESSES:\n**STRENGTHS:**\n\nStrong results. While the idea of shuffle learning is not new, its application to image and text, along with the Image-to-Response loss, brings consistent and solid improvements over LLaVA across a multitude of benchmarks, given the same training data. The effectiveness of the proposed method remains persistent even when varying the Large Language Models.\n\n\n**WEAKNESSES:**\n\n1. Ablation studies in Table 6 lack a fundamental experiment to isolate the role of the Image-to-Response Guided Loss. The last row of the table does show its effect when paired with the shuffle learning objectives, but we do not know what would be the results of training LLaVA 1.5 only with the Image-to-Response Guided Loss, in addition to the usual next token prediction loss. Please add this during the rebuttal.\n\n2. Computational cost. The authors experiment with 32 A100 GPUs, while LLaVA 1.5 only requires 8 A100 GPUs (assuming 80GB of VRAM for each A100). I imagine this is because each training sample must be forwarded to the model three times: one for the next token prediction with original image and text, one for image shuffle learning with the permuted image, and one for text shuffle learning with the permuted text. Consequently, to keep the same batch size as LLaVA, it is necessary to increase GPUs. One benefit of this method is to improve LLaVA without having access to new training data. Yet, computational resources are important too. It would be good to include the results of Direct-LLaVA trained with the same amount of GPUs as LLaVA 1.5.\n\n3. The choice of the shuffling strategy (L276-287) is theoretically justified, but it lacks empirical ablation. What does it happen if we just use purely random permutations?\n\n4. Presentation. I feel that figures, and tables in particular, have a small font size, making it difficult to read numbers. Moreover, I do not see the point of including Table 4, as it reports the very same results as Tables 2 and 3.\n\n5. It would be interesting to repeat the experiment from Table 1 with Direct-LLaVA too.\n\nQUESTIONS:\nSee **WEAKNESSES** above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 1, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Directed Tokens, a simple and effective approach to enhance multimodal alignment in LMMs through shuffle learning tasks. Specifically, it introduces two ordering tasks for the image and text, respectively, with a directed token to strengthen multimodal understanding alignment. To better utilize visual information during response generation, an Image-to-Response Guided Loss is further designed. Experiments show that the proposed method achieves superior results on multiple benchmarks. However, I have some concerns about this paper.\n\nSTRENGTHS AND WEAKNESSES:\n**Positive points**\n1. The paper introduces a novel shuffle learning paradigm coupled with directed-token mechanisms, effectively enhancing visual-textual alignment in LMMs by forcing models to reconstruct correct modality orders during both pre-training and fine-tuning phases.\n2. The proposed Image-to-Response Guided loss explicitly enhances visual feature utilization during inference through attention-layer optimization, significantly reducing language-preference bias while maintaining the original autoregressive modeling framework.\n\n**Negative points**\n1. The core idea of this paper is simple and intuitive, but the overall writing quality appears rushed and needs further polishing.\na)Several figures are currently placed in narrow side columns with reduced size. Given the single-column template, reformatting them as full-width figures would improve clarity. In addition, the visual design of some figures and tables looks somewhat rough and could be refined for better presentation quality.\nb)Figure and table references are currently written in plain text rather than using cross-referencing.\nc)There are also several other issues in equations factual correctness. Please refer to the Minor issues for details.\n2. The Related Work section is overly generic regarding large multimodal models. It would benefit from a more structured discussion of representative LMM alignment paradigms, such as cross-attention (Flamingo, CogVLM[A]), Q-former (BLIP-2, MiniGPT-4), and MLP (LLaVA, Qwen2-VL[B]), as well as recent research on auxiliary task design[B] and attention mechanism[C] for improving vision-language alignment.\n3. The image permutation reconstruction task lacks sufficient implementation detail. The paper would benefit from a clearer explanation of how the permutation index $k$ is represented and predicted, including the design of the order projection layer and related hyperparameters.\n4. The design of the permutation space appears somewhat heuristic. While selecting 10,000 permutations from a large combinatorial space is effective, it would be helpful to include ablations on the number of permutations and patch size.\n5. If I understand correctly, the two shuffle learning tasks are designed to be independent, with each forward-backward pass performing either image order reconstruction or text order reconstruction. However, in Equation 13, these two loss terms appear simultaneously, which seems inconsistent with this setup and introduces ambiguity.\n6. The paper could benefit from an analysis of failure cases or potential trade-offs. For example, the Image-to-Response Guided Loss may lead the model to over-rely on visual cues, possibly affecting its instruction-following or text-only capabilities.\n7. Some experimental results (rows 4, 5, and 6) in Table 3 appear incomplete or too limited in scope. These entries could either be expanded with additional metrics or removed if full results are unavailable.\n\n\n**Minor Issues:**\n1. In Eq. 2, the notation for the image input is denoted as $x$ initially and later as $I$, which is inconsistent.\n2. In Eq. 8, $[x, p, \\overline{p}]$ should be $[x, q, \\overline{p}]$.\n3. Eq.4 appears to be redundant with Eq.2.\n4. The symbol $L$ is used to represent both the number of layers (in Eq. 12) and the sequence length (in Eq. 2), which may cause confusion.\n5. In line 133, Page 4, the phrase “two stages of the instruction-tuning procedure” is inaccurate, as instruction-tuning itself is the second stage.\n6. In line 236, Page 6, the expression $x_a^T = [x_a^T, \\mathbf{drt}]$ seems problematic due to self-referencing on both sides of the equation.\n7. In line 256, Page 6, “the attention score from the visual token $v$ to the response token $r$”may cause confusion, as it suggests the visual token attends to the response token, which contradicts the autoregressive nature of decoding.\n\n\n[A] Cogvlm: Visual expert for pretrained language models. NeurIPS 2024.  \n[B] Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. Arxiv 2024.  \n[C] Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS 2021.  \n[D] Visual grounding via accumulated attention. CVPR 2018.\n\nQUESTIONS:\nRefer to Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Direct-LLaVA, a framework designed to improve the alignment between visual and textual modalities in large multimodal models (LMMs). The authors introduce two novel shuffle learning tasks: image order reconstruction and text order reconstruction, to be used during both pre-training and fine-tuning. The key contribution is the directed token, which is learnable and is appended to the input sequence to capture cross-modal information. The paper also introduces an Image-to-Response Guided Loss to enforce attention from visual inputs to textual outputs. Experimental results demonstrate state-of-the-art performance across multiple academic and instruction-following benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The proposed “Directed Token” provides a simple yet elegant solution to cross-modal information integration under autoregressive modeling. The token captures both visual and textual cues more effectively than traditional visual tokens, as confirmed through comprehensive ablation studies.\n\n2. This paper is well motivated. The authors demonstrate that existing models like LLaVA often ignore visual inputs, producing nearly identical outputs even with blank images. \n\n3. The model achieves state-of-the-art results across a wide range of academic and instruction-following benchmarks, including VQAv2, GQA, SciQA-IMG, TextVQA, POPE, MMMU, and MME.\n\nCons:\n\n1. The paper's structure and formatting can be improved. There is no need to put everything in the main context, it would be easier to follow if the authors put some content as the appendix. For example, Table 2 and Figure 5 are too small to be well understood. \n\n2. While the Directed Token is empirically effective, the paper lacks deeper theoretical justification or analysis regarding why it facilitates better cross-modal alignment. Its design is intuitive, but the underlying learning dynamics remain unexplored.\n\nQUESTIONS:\nQuestiones: \n\n1. I suggest the authors to reformat the paper, and put some experiments into the appendix. \n\n2. It would be better to have a complete algorithm box. \n\n3. How is the balance among multiple losses (CE, image/text order, and attention-guided loss) tuned? Are these loss weights fixed or adaptively adjusted during training?\n\nLIMITATIONS:\nYes, but the limitation is not sufficient. And it is attached as the supplementary, very difficult to find.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Direct-LLaVA, a new method that introduces directed tokens and shuffle learning objectives (reconstructing image and text order) into the training of vision-language models (LMMs) to improve multimodal alignment and robustness. The authors further incorporate an Image-to-Response Guided Loss to strengthen the influence of visual inputs in textual outputs. Extensive experiments on academic and instruction-following benchmarks demonstrate consistent performance gains over LLaVA-1.5 and other baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- Introduces two new learning objectives—reconstructing image order and text order—to directly improve cross-modal alignment and reduce over-reliance on the language modality.\n- Proposes a novel “directed token” mechanism to aggregate visual and textual information for permutation prediction, showing consistent improvements over baseline designs.\n- Achieves SoTA results across multiple benchmarks (e.g., VQAv2, GQA, VizWiz, SciQA-IMG, POPE, SEED-Bench, MMMU-Val) under different LLM backbones (Vicuna, LLaMA, Qwen).\n- Presents thorough ablation studies on model components (token types, shuffle strategies, LLMs), demonstrating robustness and effectiveness across different configurations.\n- Provides clear implementation details and fair comparisons using consistent training data, model sizes, and evaluation protocols.\n\n### Weaknesses\n\n- The design choice of using a single directed token at the end of the sequence lacks deeper analysis; alternative placements or token designs are not explored.\n- There is a lack of analysis of failure cases where multimodal misalignment still occurs; visualizations are limited and do not systematically reveal limitations.\n- The shuffle learning tasks are somewhat synthetic and may not fully represent the types of semantic misalignment encountered in real-world scenarios.\n- The main components (e.g., permutation prediction, guided attention, special tokens) are individually incremental, and the novelty lies more in the combination than in fundamental algorithmic innovation.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nLimitations such as scale constraints and lack of hyperparameter sensitivity exploration are briefly discussed in Appendix C, though more analysis could still be provided.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}