Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
Authors: Thanh-Dat Truong
Huu-Thien Tran
Tran Thai Son
Bhiksha Raj
Khoa Luu
Venue: NeurIPS 2025 poster ()
Keywords: Large Multimodal Models; Shuffle Learning; Directed Tokens
Primary area: applications
TL;DR: This paper introduces a new simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems.

Abstract: Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these models still suffer from some fundamental limitations related to robustness and generalization due to the alignment and correlation between visual and textual features. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's pre-training and fine-tuning phases. In addition,  we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to reconstruct the correct order of visual inputs. Then, we introduce a new Image-to-Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks.

Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language- Vision Models
Thanh-Dat Truong', Huu-Thien Tran', Thai-Son Tran", Bhiksha Raj", Khoa Luu'
CCVIU Lab, University Arkansas, USA
Vietnam National University, Ho Chi Minh City University of Science, Vietnam Carnegie Mellon University, USA
(tt032, ht035, khoaluu) edu. ttson0fit hemus edu. vn, bhiksha@cs. cmu. edu https: /t/p//aakvviuuuu github. http///ppopeccc///reeeeeeeeeeee
Abstract
Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability various understanding tasks. However, these mod- els still suffer from some fundamental limitations related to robustness and gen- eralization due the alignment and correlation between visual and textual fea- tures. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding. and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's 'ss pre- training and fine-tuning phases. addition, we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to construct the correct order visual inputs. Then, we introduce a new Image-to- Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs academic task-oriented and instruction -folowiinnn LMM benchmarks.
1 Introduction
Large multimodal models (LMMs) have gained more attention recently due to their outstanding ca- pabilities in general-purpose assistants. By training via visual instruction tuning [39, 37, 38, 3, 29, 28), these LMMs, e.g., L.LVVA [39), InstructBIII [29, 28), MiniGPT-4 [69], Qwen-VI. [4], have shown impressive performance on cndtruction-oollowiin and visual reasoning tasks. Then, these LMMs have been further developed for specific scientific tasks, e.g., LLaVA-Med [26]. DriveGPT4 [56], PMC-LLaMA [53], etc. To measure the performance and capabilities of LMMs, several bench- marks [63, 58] have been introduced to evaluate the L.MSs in different aspects, e.g., art, business, health and medicine, science, tech and engineering, and other fields. Recent studies further improve the performance of LMMs by scaling the training data [37, 38, 66, 65]. using better visual encoders [5, 28, 28, 51] large language models [25], improving objective learning [50, 31, 60, 68), using multimodal preference data [54, 45, 61], extending to other modalities [35, 34, 33] (e.g., videos [35, 34, 43], graphs [33]).
Motivation of this Work. While recent efforts in improving the performance of LMMs focus on scaling data and models [37, 38, 25, 65, 5, 28, 28, 51, 42, 49], this work studies the pitfalls of these LMMs in new simple but efficient aspect (Fig. 1). In particular, LMMs are usually biased language preferences since the large language model (LLM) has been pre-trained on the exascale
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

data. Meanwhile, due to the data complexity, the LMMs tend to overlook the information of visual inputs. For example, an LMM can achieve similar results using only languages [50].
How does visual information impact LMM's
responses? To demonstrate this point, we con- duct experiments by evaluating the LMM, i.c., LLaVA v1.5 7B [37], on the AciG [41] and MMMU [63] benchmarks. We remove the visual information by using the black image as an input instead of the original image of the benchmark. As in Table 1, the performance of the models is still maintained without the vi- sual information. As in Fig. LLaVA [37] an- swers similarly for two cases even though the important information of the images in the sec- ond case was blacked out. The model cannot learn a well-alignment between visual and tex- tual features. Thus, the answers produced by the LLaVA model are dominated by the lan- guage model with low consideration of visual inputs. This problem indicates the multimodal alignment in current LMMs has not been well learned yet, leading to prioritizing language preferences and overlooking the visual infor- mation. As a result, the LMMs will produce less informative outputs even hallucinate the
VDAv2
B1.6
SEED SMnch-IMG
Vatte
16/22
Quen-dk-Cher HRrE
90.4
Liiiieet 2-18
LLWieet
surt boleteg skite
Largo Natreddaaaeeeeoee
Decetill
indeconng
Wearder 24 settencs 13 Waln
inteedin
19 cery 1111
My
Figure I: Our Proposed Direct-LLaVA achieves ttat performance on various LMM benchmarks (Top) with new shuffle learning ap- proaches of Image (Bottom Left) and Text (Bot- tom Right) during pre-training and fine-tuning phases.
results, leading to low model performance.
In this paper, we, therefore, address two funda- mental questions for the current LMMs. For the first question, given an image whose patches are shuffled, will the LMM be able to recon- struct the original image matched with language representations? If this is not the case, the LMM relies on the language encoder, which ignores the visual information. Then, in the second ques- tion, given shuffled textual description of images, will the LMM be able reconstruct the textual sentence to represent the visual information in the image?
If the LMM cannot do that, it means the align- ment between visual and textual features has not been well represented by the LMM. Finally, by addressing these two questions, we intro- duced novel learning approach to improve the robustness of the LMM models.
Contributions of this Work. This paper presents a novel learning approach improv- ing the alignment between visual and textual features in the LMM via solving the shuffle problems. Our approach adopts simple but ef- ficient learning strategy to learn the right order of visual information and textual descriptions, thus improving their alignment. Our contribu- tions can be summarized as follows. First, introduce the problem of ordering the visual and textual information in LMM. In particular, we introduce a new shuffle learning method in the pre- training and fine-tuning phase, forcing the model to reconstruct the right order of images and textual descriptions for better alignment. Second, to support shuffle learning, we propose a new directed- token approach to capture visual and textual correlations in the LMM, thus improving the capability of reconstructing the right order of visual inputs. Third, to improve the mutual information between visual and textual features, we introduce new maae-to-RResonnse Guided loss based on the atten- tion layers to enhance the information flow of the visual inputs to the LMM's 's responses. Finally, our ablation studies have shown the effectiveness of different aspects of our approach. Through inten-
Figure 2: The Example of Answer Produced by LLaVA v1.5 [37]. The first two rows are samples selected from ScienceQA-IMG, and the last two rows are sampled from MMMU.
2

---

sive experiments, our approach has achieved state-of-the performance on academic -riento and instruction-follee followingg LMM benchmarks.
2 Related Work
Large Multimodal Models. Thanks the re- markable advancements in LLMS [34, 10, 1, 46, 3], has promoted the development of LMMs. Numerous LMMs have been devel- oped to enhance effectiveness in handling such data, which can be classified into large vision- language models [39, 37J, large video-language models [52, 67, 35, 30), and large audio-language models [20, 15J. Early work by [2] effectively bridged vision and language modalities in a few- shot learning setting, followed by [28], which enhanced inter- modaltty connectivity via Q-Former. This further developed into instruction- aware model within the vision- -anguage instruction- tuning framework [39]. LA [39] established a streamlined visual-to-language space projection using linear layer, later refined by [37] with MLP and AnyRes, technique adept at handling high-resolution images. Subsequent studies [38, 24, 64, 23, 27] contributed further improvements, culminating in a robust model [25] capable of handling diverse vision tasks, including video com- prehension. Later, [26] extended LLaVA to a biomedical model through cost-effective curriculum learning approach. [7] leveraged multimodal federated learning enhance LMM training. Mean- while, (57, 55, 36] introduced LMMs for 3D point cloud understanding tasks.
Table 1: Performance of LLaVA v1.5 [37] With
and Without Visual Information.
LMM
Image
SciQA-IMG
MMMU-Val
LLaVA-v1.5-70 LLAVA-v1.57B
Origin
66.8
35.3
Black
64.1
32.4
Shuffle Learning. Decomposing image into smaller patches, shuffling these patches, and then reconstructing their original order forms a classical pattern recognition problem known as the jig- saw puzzle. The early work in classification [14] solved the jigsaw puzzle by predicting the spatial position of each patch. Later, it has proven highly useful in self-supervised learning for feature rep- resentation. Indeed, [6] incorporates jigsaw puzzle challenge alongside classification improve visual information generalization across domains. [12] employs jigsaw puzzle generator to create varying levels of granularity for fine- fine-grained classification. Chen al. [9] extended shuffle learning to transformers to enhance performance in visual recognition tasks. Truong et al. [48] developed a robust video understanding model by solving the shuffled temporal frames via the directed attention mechanism. This body of work suggests that using the jigsaw puzzle as a pretext task is broadly effective for generalizing spatial information within images. Building on this, we investigate the impact of reconstructing the order of elements not only in images but also in the texts, analyzing the interplay between these two modalities in relation each other's permutated version.
The Proposed Directed Token Approach to LMM
Inspired by LLavA [39, 37J, we develop a new LMM, namely Direct-LLaVA (Fig. 3), by adopting the design of LLaVA v1.5. In particular, our Direct-LLaVA model consists of the vision encoder and the large language model. The visual features will undergo the languael (VL) connector to align with the textual features. Formally, given the image and a multi-turn conversation data (xx). x xi. x². Xyy x²") where is the number of turns, following the standard protocol of
IBI
m
Largo MadimsdelMeees
Lango Luttmmodal Modela
Tight Siee Star destere
Tuaras
ablle
Truwiieeerd Erbeeddnng
CLConeech
Teieeber and Emooddingg
vertirg canti My
Figure 3: The Proposed LLaV Framework. (A) Reconstructing Image Order Task. (B) Reconstructing Text Order Task. (C) Directed-Tokee Modeling Approach in Image Order Recon- struction. (D) Autoregressive Modeling Approach in Text Order Reconstruction.
3

---

[39, 37], the instruction data x
instruct in the tt turn is formed as in Eqn. (1).
Randomly Choose [xg x or [x, xx] If 1
xinstruct
Ift 1
Then, learning LMM can be formed as an auto-regressive training objective as in Eqn. (2).
0* arg max Exa, Mantrect log p(xx x, Xinsmmss)
arg max Ex Minnaas
log pe (2, Ximmuacc, Xaci)
where is the sequence length of the answer X = [x1 222 .... #L), is the parameters of the LMM model, and Xinstruct<cá and Na<c are the tokens of instructions and answers in all turns before token 2;- Similar to LLaVA [39, 37], Direct-LLaVA follows two stages of the instruction-tumi tutiiig procedure, Le., pre-training and fine-tuning.
The success of the current late-of-the-an LMMs [39, 37, 38] relies on auto-regressive modeling. Indeed, the form of auto-regression naturally matches the nature of the data, where each input token in the sequence depends on the previous ones. This modeling approach can capture the complex dependencies and correlations within the image and language and maintain consistency and coher- ence data. As a result, the order of data, i.e., visual image patches or textual sentences, plays an important role since the LMM auto-regressivell models multimodal inputs conditioning on all pre- vious elements to maintain context and coherence. If the data order is incorrect, the LMM loses its ability to model correct contexts and produce logically consistent predictions. Under this modeling principle, we aim to improve the LMMs by addressing two fundamental questions. First, given a random shuffled image, we aim develop LMM capable of reconstructing the original image that is represented by the current textual descriptions (Fig. 3(A)). This learning mechanism encourages the LMM to learn the strong correlation between visual and textual features, thus providing better understanding of visual information. Second, given an original image and a shuffled textual descrip- tion, our LMM aims to predict the correct textual description that represents the visual information of the image (Fig. 3(B). The learning objective allows the model to improve correlation across modalities further, thus enhancing the important role of visual information in the LMM's responses. In the next sections, we will describe our approach to developing these two learning objectives in the pre-training and fine-tuning phases of the LMMs.
3.1 The Shuffle Learning In Pre-training Phase
The primary goal of the pre- training phase is to learn the alignment between the visual and the language spaces. The traditional LMM [39, 37] train the alignment based on the single-turn conver- sations (Fig. 4(A)) where the instruction data is formed as in Eqn. (3).
Xinstract
Randomly Choose (x,, (x, x,]
Xa
where X, is randomly sampled from set of questions designed to ask for the content of an im- age, and p is the textual description of the image x. Then, learning the LMM with single-turn conversation can be formed as in Eqn. (4).
arg max Ex.
log p(x., Xenstruct)
(4)
Reconstructing Image Order Task. To improve the visual understanding of the LMM, we in- troduce a new image order reconstructing task during the pre-training phase. particular, during training, for each image x, we will randomly shuffle image patches, followed by reconstructing the original order of the image described via the textual description p via the LMM. Formally, let x = P(x, k) be the shuffled image where P is the permutation method that shuffles the patches of the image (we use the patch size of the vision encoder) and k is the indexing associated with the per- mutation. In this learning task, the textual description of the image is crucial for reconstructing the original image from the shuffled version. It provides additional reference knowledge that supports the LMM in learning to correct the image order. As shown in (Fig. 4(B)), while both predictions

---

are meaningful images. only one of them is correct and matches the description. Therefore, the instruction data in this image ordering task can be formed as follows,
Instrutt =
Randomly Choose [p. x] or [x, p]
(5)
Then, learning to reconstruct the order of the image via the LMM model can be formulated as in Eqn. (6).
0* arg max E. Linge & (Misnn
log x, X, (nstrnet)
(6)
Directed Tokens. To predict the permutation index k from the shuffled image, we propose to design a visual order projection head via a lin- car layer. We can adopt the last hidden-stage features of the visual tokens as the input of this projection head. However, this approach in- efficient for two reasons. First, the visual to- kens are designed to contain the general knowl- edge of the visual inputs. Second, if the vi- sual tokens are placed at the beginning of the instruction input (i.e., p.). the textual infor- mation from the description is not captured by the visual tokens via attention mechanisms due to the autoregressive modeling of the LMM (in this case, the texts are future tokens of the vi- sual tokens). Therefore, to address this prob- lem, introduce new learnable directed token drt placed at the end of the input to- ken sequence (Fig. 3(C)). The term "directed tokens" pays tribute to concepts of the order of the input tokens in shuffle learning tasks.
suenery picture
Lorge Mustiicaa
chasing
Madda
riald covered by yellow Lewres
chastag
Large
Musiicccl
Hudil
represest
trage:
field cavered in Leaves chaxing yallow
cut is chesing dog cevered
Lorgs Nuttrrida
dog is coverie by yellow leaves
cat field
Figure 4: (A) Traditional Single-Turn Conversa- tion Task. (B) Reconstructing Image Ordering Task. (C) Reconstructing Text Ordering Task.
Then, the permutation index k can be predicted by using the hidden-stage features of drt as = ProjectLN(drtan) where drt, is the last hidden-stage feature of the directed token, and ProjectLN is the order projection layer. The instruction data with the directed token is formed as,
Xinstruct RRandolly Choose [p. X, drt] or [x. drr]
Since the directed token drt is placed at the end of instruction data, it can efficiently capture the knowledge from both visual and textual features via the self-attention mechanism with auto- regressive modeling. Therefore, the directed token can provide better information from visual and textual knowledge to predict the permutation index than the visual tokens of the image.
Reconstructing Text Order Task. Learning LMM requires deep understanding of the correla- tion between visual and textual features. To further improve the visual understanding and reduce the dominance of the language modality of the LMM, we present a new reconstructing text order task during the pre- pre-training phase. Formally, given original image and a shuffled word-order description p. the goal of this task is to encourage the LMM to predict the back of the original de- scription matched with the visual information represented in the image. Although this task can be easily accomplished using the large language model of the LMM without visual features, visual in- formation plays an important role improving the visual understanding of the LMM in this learning paradigm. For example, as shown in Fig. 4(C), while both reconstructed sentences predicted by the LMM are meaningful, only one is correct with the context represented by the visual information. In this context, the image plays a sigmificant role since it provides an additional reference that aids the LMM accurately re-order the shuffled descriptions. Therefore, the LMM must incorporate and understand the visual features to correctly reconstruct the text order, as shown in Fig. 3(D). In this task, the instruction data can be formulated as in Eqn. (8).
lest
Randomly Choose (q. P. x] or [.
(8) where q the prompt that instructs the reconstructing text order task, i.e., q = re-order the sentence to represent the information of the image: It should be noted that although
x
Jest
[p. drt]
5

---

the directed token drt may not contribute directly to text order predictions, drt is placed at the end of the answers to ensure consistency in the prompting format with the prior reconstructing image order task. Then, the text reconstructing order task via the LMM model can be formulated as follows,
0r arg max Eyges ix log p(x 18xt
x, X Xinstruct)
Learning Objective of the Pre-training Phase. To summarize, our LMM model is optimized with three learning objectives, including the traditional single-turn conversation, the reconstructing image order task, and the reconstructing text order task. Formally, the learning objective of the pre-training phase can be formed as in Eqn. (10).
aig
Est Porarret log p( (x, Xintua) F.P. k. suuge log p[k[xx Simmer E. Exteo (a (x iistree
log pl [x, x
(10)
3.2 The Shuffle Learning In Fine-tuning Phase
The goal of the fine-tuning phase is to enable visual and language understanding and a general- purpose visual assistant of the LMM through instruction peeructtio-tuiing with multi-round conversation. The training process of the LMM can be defined as in Eqn. (2) with the instruction data formed in Eqn. (1). This fine-tuning procedure helps the model improve the alignment between visual and textual modalities and enables language-driven visual reasoning. Then, to further improve the visual understanding and reasoning skills of the LMM, we adopt the Reconstructing Image Order Task presented in the previous section into the fine-tuning phase. This objective helps to improve the reasoning skills and understanding of the LMM by learning to reconstruct the image based on the content of the multi-round conversation instruction data. In this learning task, the directed token drt is placed at the end of the last answer to comprehensively capture the context of the entire conversation for the reconstructing image order task. Formally, the last answer of instruction data can be structured as x = xx" drt). Then, the learning objective of the fine-tuning phase with the reconstructing image order task can be formed as,
arg max Ex. x Niuttoctt log p(xa X., Xinstrua) + log p(k|X, Xunsuue))
(11)
where x = P(xx, k) is the shuffled version of the original image and k is the permutation index. While the first objective enables the reasoning capability of the LMM, the second objective will improve the visual understanding of the LMM by reconstructing the correct order of the image.
should be noted that we do not perform the reconstructing text order task during the fine-tuning phase, since the purpose of the fine-tuning task is to enable reasoning based on the multi-round conversation. The shuffled texts could destroy the context of the conversation and the reasonability of the LMM, leading to suboptimal performance. Therefore, we only adopt the reconstructing image order task during the fine-tuning phase to ensure the reasoning and visual understanding capability of the LMM.
3.3 Image-to- -oagge-t-Responee Guided Learning
Enhancing the visual knowledge in the response produced by the LMM is important since it will help the model capture more visual features, therefore enhancing the robustness of the answers. To further improve the visual understanding of the model and reduce the burden of the LMM when learning to capture visual and textual correlations, we propose new mmage-to-Responee Guided loss to enforce the attention learning from the prior knowledge of image and response. Formally, Guided Loss can be formulated as in Eqn. (12).
the proposed Image-to-Hee
ΣΣΣ (1- LIVIR 1=1 REVR
L.
(12)
where v is the position of the visual token, where r is the position of the response token, V is the list of visual tokens, R. is the list of textual response tokens, a is the attention score from the visual token v to the response token r at the 1th layer of the LMM, and L is the number of layers in the LMM. Minimizing the the visual tokens to the response tokens produced by the LMM, i.e , Therefore, it will help to indicate the attention learning to enhance the impact of the visual features in its textual responses.
Reseponse Guided loss will increase the attention score from

---

Table Comparison with prior methods on academic-task-oriee benchmarks.
Method
LLM
Image Size
Data Size
VQAv2 GQA
VizWiz SciQA-IMG TextVQA
Pretrain
Finetune
BLIP-2 [28]
Vicuna- 13B
224 x 224
129M
65.0
41.0
19.6
61.0
42.5
InstructBLIP [11]
Vicuna-7B 7B
224 x 224
129M
1.2M
49.2
34.5
60.5
50.1
InstructBLIP [11]
Vicuna- 13B
224 x 224
129M
1.2M
49.5
33.4
63.1
50.7
Shikra [8]
Vicuna-13B
224 224
600K
5.5M
77.4
IDEFICS-9B [19]
LLaMA-7B
224 x 224
353M
1M
50.9
38.4
35.5
25.9
IDEFICS-80B [19]
LLaMA-65B
224 224
353M
1M
60.0
45.2
36.0
30.9
Qwen-V1. [4]
Qwen-7B
448 x 448
14B
50M
78.8
59.3
35.2
67.1
63.8
Qwen-VI-Chat [4]
Qwen-7B
448 448
1.48
50M
78.2
57.5
38.9
68.2
61.5
LLaVA-1.5-7B [37]
Vicuna-7B
336 336
558K
665K
78.5
62.0
50.0
66.8
58.2
LLaVA-1.5-13B [37]
Vicuna-13B
336
336
558K
665K
80.0
63.3
53.6
71.6
61.3
Divv
Vicuna-7B
336 336
558K
663K
79.0
63.2
52.3
74.3
63.2
Direct-LLaVA
Qwen-7B
336
336
558K
665K
79.9
63.3
$4.3
75.7
65.1
Direct-LLaVA
LLaMA-8B
336
336
558K
665K
80.0
63.9
$4.4
75.8
64.9
Direct-LLaVA
Vicuma-1 13B
336 336
558K
665K
81.6
65.3
54.9
77.3
65.4
Finally, the Image-to-Response Guided loss will be integrated into the learning objective of the pre-training (Eqn. (10)) and fine-tuning phase (Eqn. (11)). In particular, the learning loss of the pre-training task Cpretrain can be formed as,
Leretrain =
LCE + Limuge-Order + Lext-Order + LI-RR
(13)
Similarly, the learning objective of the fine-trainin traitrain task Limetune can be formulated as Eqn. (14).
Liinetuee LCE + Limuge Onder + LI
(14)
where Ler is the typical loss the LMM model [37, 39], Limage Order and Lext Order entropy losses of the reconstructing image and text order tasks.
Onder are the cross-
4 Experiments
4.1 Implementation and Benchmarks
Implementation. Our framework adopts the implementation of LLaVA v1.5 [375 We use the CLIP- ViT-L-14 (336?) encoder for the vision tower, and four different LLMs, i.e., Vicuna 7B [10], Vicuna 13B [10]. Qwen 7B [3], and LLaMA3 8B [13]. We adopt the multi- perception [37] for the VL connector. To ensure the consistency of our implementation, the directed token drt is placed at the end of the sequence. We use 32 NVIDIA A1 in our experiments. For fair comparisons, the learning hyper-parameters of LLaVA v1.5 in our training. We use the training data LLaVA v1.5 in our experiments. We also include additional ablation studies to analyze the impact of the different data size and other benchmarks in the supplementary.
adopt
Shuffling Strategy. Our image permutation function P shuffled the patches of the image where the patch size is 14 x 14. Therefore, will be Np! permutations of the image patches where Np 14x 576 is the number of image patches. Learning with all permutations remains inef- ficient due the its large variation. The choice of permutations plays an important role in improving visual understanding. In particular, if the two permutations are very far apart, the LMM may easily predict the image order since they have significant differences. Meanwhile, when all the permuta- tions are close to each other, learning reconstruct the order becomes a challenging problem since the two different permutations have minor differences. Therefore, to effectively develop a set of per- mutations, we randomly select 10, 000 permutations from Np 576! so that the Hamming distance between two permutations is as close as possible. Meanwhile, the large language model can model complex semantic sentences. Thus, for the text shuffling, randomly permute the word positions in the text sentences.
3362
Benchmarks. Following the standard protocol [37], we evaluate our models on two sets of bench- marks, i.e., Academic Trik-erie and task-oriented task includes five benchmarks: Visual Question Answering V2 (VQAv2) [16]. Ques- tion Answering on Image Scene Graphs (GQA) [18], Answer Visual Questions from People Who Are Blind (VizWiz) [17], Science Question Answering (SciQA-IMG) [41], and Visual Reason- ing based on Text in Images (TextVQA) [47]. The CnstuuctionFoolloviii L.M has five bench- marks: Polling-base based Object Probing Evaluation for Object Hallucination (POPE) [32], Multimodal
LMM Benchmarks. The academic
7

---

Table 3: Comparison with prior methods benchmarks for cessructiionfollowiii LMMs.
Image
POPE
Betch
LLAVA-WSI
MMMMU-VI
Linetune
pop
DLIP22
InsreetiLiP
Vicuna-111
1212.8
IDERICSOR
1JaMa-78
1M
44.5
IDERICS-008 (19)
L1aMA-658
1M
Qwan-VL
Owen-78
50M
56.3
Ow
50M
56.2
1487.5
35.9
LLaVA 7B [39]
Vicusa-BB
595K
155K
33.5
SU9.6
LLaWA-1.5 1.5-7 (37)
Vicusa-78 7B
665K
58.6
1510.7
35.3
LLaW
Vicuso
3335
558K
665K
61.6
1551.5
26.4
LL2VA
70
1355
353K
83.9
85.0
03.3
693
1321.9
097
36.1
Direct LLaVA
Oween
3335
558K
665K
39.0
88.3
87.0
64.0
70.4
1538.1
71.0
36.0
36,9
Dines-ULav LL
LLMMA-8B 8B
3396
558K
665K
59.2
87.0
83.4
64.2
20.5
1555.1
21.1
36.0
37.6
Dinos-ULaVA
Vicuma-130 1369
335
558K
665K
90.5
89.2
87.8
686
71.5
43.
1872.1
72.3
38.0
LLMss with Generative Comprehension Benchmark (SEED-Bench) [22], Comprehensive Evaluation Benchmark of LMM (MME) [59], LLaVV Benchmark in the Wild (LLaVA-Wild) [39], Integrated Capability Benchmark (MM- Vet) [62], and Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU-Val [63].
4.2 Comparision with State-of-the-Art Methods
Comparison with SoTA methods on Academic-task-orientee
Benchmarks. Table 2 highlights the performance of our approach, evaluated with different LLM versions on various academic-focused bench- marks against previous SoTA methods. The results underscore that our approach surpasses prior benchmarks accuracy across these tasks. Specifically, Direct-LLaVA, when integrated with Vicuna-13B [10], consistently outperforms the previous SoTA, LLaVA-1.5, which also employs Vicuna-13B 13B. Notable achievements include 77.3% and 65.4% accuracy ScienceQA and TextVQA benchmarks, respectively, surpassing VLa results of 71.6% and 61.3%. Additionally, Direct-LLaVA shows considerable accuracy improvements on VQAv2, GQA, and VizWiz, with increases of 1.6%, 2%, and 1.3%
Figure 5: Comparison of Response in Multiple- Choice Questions Between Direct-LLaVA-7B and LLaVA-vI.5-7B on MMMU-Val.
Comparison with SoTA methods on cntrruction-Followinn LMM Benchmark. As in Table 3, our method demonstrates superior performance on several instruction-following benchmarks relative to previous approaches. Our model achieves Fl scores of 90.5%, 89.2%. and 87. $7.8% on the POPE benchmark's ' random, popular, and adversarial settings, surpassing LLaVA- 1.5 [37] with Vicuna- 13B, which achieved 87.1%. %, 86.2%, and 84.5%, respectively. Additionally, Direct-LLaVa enhances SEED-Bend Bench performance, achieving an accuracy of 65.5% compared to LLa VA-1.5's 61.6%. Our model also records score of 1572.1 on the MME benchmark, 41. % MM- [62] and 38% overall accuracy on MMMU-Val [63], underlining its efficacy across complex understanding tasks.
Visualization. Fig. 5 illustrates the response of our Direct-LLaVA on MMMU-Val compared to LLaVA v1.5. Our responses are correct and consistent with the information represented by the visual image. In addition, to further illus- trate the robustness of our approach, Fig. visualize the conversation about the image be- tween our Direct-LLaVA model and humans with set of queries. As shown in this conversa- tion, the Direct-LLa' VA model can better under- stand visual information and answer correctly and consistently. Meanwhile, LLaVA tends produce hallucinated answers produced by L.M that are inconsistent with the images.
Slove
Figure 6: Comparison of Response in Conver- sation Between Direct-LLaVA-7B and LLaVA- v1.5-7B on In-the-Witt Samples.
8

---

4.3 Ablation Study
Effectiveness of Large Language Models. Ta- ble 4 provides an in-depth analysis of how vari- ous LLMs impact performance when integrated with different LMMs, i.e., Vicuna-7B, Vicuna- 13B, LLaMA-8B [13], and Qw [3]. In the case of LL, the larger 13B model consis- tently demonstrates superior results compared to the smaller 7B model across both academic- oriented and mnstruction-follwwiii increase, other benchmarks exhibit notable improvements with the Vicuna- 13B version. This trend holds during the evaluation of these Vicuna versions within Direct-LLaVA. For instance, Direct- LLaVA integrated with the larger Vicuna model increases the performance on ScienceQA and MME from 74.3% to 77.3% and from 1524..9 to 1572.1, respectively. In addition, with larger model size, LLaMA outperforms compared to Qwen in most benchmarks, with a few exceptions. For exam- ple, Qwen slightly surpasses LLaMA with accuracy of 65. 1% compared to LLaMA's 64.9% in TeXtVQA evaluations. Overall, utilizing Vicuna-13B as the LLM within our Direct-LLaVA frame- work yields the SoTA performance.
Table 4: Effectiveness of Different Large Lan- guage Models.
Method
LLM
SeQA Test
VOA
rand pop
MME
SELD
img
TlaVA-e13 Vicua 75
06.8
582
87.3
36
1310
58.6 66 373
LLaV Vicuna 13B
71.6
6133
87.1 36.2 84.5
1551,3
61.6
68.2 42.7
Dices-LaaA Vicuna 76
74.3
6522
883 833 W611
61.3 60.. 35.
Liaves-LaaV Qwen
75.7
65.1
59.0 58.3
67.0
1538.1
61.0
70.4 40.0
DirecsLLaVA LLMMA
75.8
04.9
19.2
57.0
K3
1555.1
64.2
70.5 40.1
Dar Vicuna 138
77.3
65.4
91.5
59.2
87.8
1572.
65.6
71,5 43.3
benchmarks. Although the POPE Fl score shows only a minor
Effectiveness of Directed Tokens. We analyze the impact of directed tokens on the performance of our Direct-LLaVA compared using only visual tokens. As in Table 5, our results indicate that with directed tokens, the model consistently achieves higher scores across both seademic-orientee and instruction- -estuuctionffolloinng benchmarks, regardless of the language model used. BEED-Bench, for in- stance, is witnessed a significant boost in both image and video evaluation settings, with the accuracy metric increasing from 68.2% and 42.7% to 71.5% and 43.3%. Furthermore, while the performance of Direct-LLaVA with visual tokens alone is not as strong as with directed tokens, it still outper- forms the original LLaVA. Notably, the POPE F1 scores across random, popular, and adversarial split each increase by approximately 2%, alongside a substantial improvement in the MME bench- mark, with scores increasing from 1531.3 to 1541.7. This experiment, therefore, confirms that our Direct-LLaVA not only suraasess the baseline LLaVA model on these complex tasks but that the directed token usage further enhances its effectiveness.
Effectiveness of Shuffle Learning in Pre- Training Phase. To assess the impact of the shuffling technique, we conduct experiments under three configurations: shuffling images only, shuffling text only, and shuffling both text and images during the pre-training phase. As portrayed in Table 6, integrating directed to- kens address the image ordering reconstruc- tion problem slightly outperforms their use in solving text ordering, indicating that shuffling images proves more effective than shuffling text. Fur- thermore, combining both image and text shuffling during pre-training yields improvements across all benchmarks. Notably, in this setup, the MME score reaches 1519.3, surpassing scores of 1513.5 and 1516.3 achieved with single-modality shuffling. Additionally, our Direct-LLa tently outperforms the prior LLaVA-7B model across all scenarios, confirming that leveraging di- rected tokens to tackle both text and image shuffling enhances competitive performance overall.
Table 5: Effectiveness of Token Selection, i.e., Vi- sual Tokens (vis) and Directed Tokens (drt) For Reconstructing Image Order Task.
Method
Token
SaQA
IMG
VQA
Text
rand pop adv MMC
POPE
all Img vid
SEED Dench
LLaVA v13-11 7B
66.5
582
87.5 16.1 84.2 1510.7
58.6 66.1 37.3
Dicos-L1a9--7
via
21.5
61. 88.1 87.4 85.8 1520.4
61.1 67.1 38.5
Dirett11-aAA-7
drt
74.3
632
88.8
88.9 86.0 1524.9
63.3
69.7 38.8
LLVVe-13-133
91.1
11.
1
vis
26.4
88.8
Direct LLVV-1LLV
90.5
43.3
consis-
Effectiveness of Shuffle Learning in Fine- tuning Phase. We investigate the impact of shuffling images during the fine-tuning stage. As shown in Table 6, incorporating im- age shuffling leads to notable improvements in performance across both academic-oriented and instruction-followinn benchmarks. Re- markably, accuracy scores in ScienceQA and TeXtVQA increase from 71.0% and 61.3% to 72.3% and 62.3%, respectively. Additionally, the JEED-Bench accuracy for both image and video comprehension tasks shows a 1% improvement. These findings underscore the ad advantages of applying image shuffling in the fine-tuning process.
Table 6: Effectiveness of Shuffle Learning and Image-to-Response Guided Learning Ap- proaches. Pretraining (Pret.), Finetuning (Fine.), Image (I) and Text (T),
Mathed
Fiot File G...
SCKON Tear
VOA
mml
POPE
pop
MMI:
SEED Beach
573
59.9
38
86.
151.5
41.1
Doer-1 L
09:1
68.
681
964
64
1516
61.1
41.
DoerLLyva 78
71.0
61.3
58.4
87.2
843
15193
61.7
66.9
42.1
Dina- 1aV
23.1
62.
18.5
87
1531
61.5
67.6
42.5
Diave TB /
74.3
65.2
8888
55.9
86
15349
63.5 08.7 38.8

---

Effectiveness of Image-to-Response Guided Loss. We evaluate the impact of attention information loss on model performance by comparing outcomes with and without such loss. As in Table 6, the application of Imagetoo-Reponss Guided Loss (LI-R) boosts the model's performance across various benchmarks. In particular, results for two academie-oriented benchmarks improve from 72.3% and 62.3% to 74.3% and 63.2%, respectively. The POPE F1 Score also increases by 0.3%, 1.4%, and 0.8% on random, popular, and adversarial splits. Additionally, the MME benchmark score climbs by 3.5, while SEED-Bert -eBhhhhh accuracy advances from 62.3% to 63.3%. This indicates that our loss objective contributes positively to the overall performance.
5 Conclusions
Our paper has presented a new shuffling learning approach to improving the robustness and align- ment between visual and textual features. In particular, we have introduced two new learning tasks, i.e., reconstructing image order and reconstructing text order, into the pre-training and fine-tuning phases. Our new learning tasks have significantly improved visual understanding and cross-mes modality alignment of the LMM. To effectively support the reconstructing image order task, we have intro- duced a new directed token approach to capture both visual and textual features effectively. Then, to further enhance the correlation between visual tokens and the LMM's 's responses, the new Image- so-Response Guided loss has been introduced. Through intensive experiments on various academic task-oriented and instruction-followin SoTA performance. Our study explores the effectiveness of proposed learning tasks and losses for improving LMM performance under selected hyperparameters and model scales. However, it still has limitations related to objective balancing. Our detailed discussion of limitations provided in the Appendix.
LMM benchmarks, our proposed approach has achieved
Acknowledgment. This work is partly supported by NSF CAREER (No. 2442295), NSF SCH (No. 2501021), NSF E-RISE (No. 2445877). NSF SBIR Phase 2 (No. 2247237) and USDA/NIFA Award. We also acknowledge the Arkansas High-Performe Computing Center (HPC) for GPU servers.
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] J.-B. Alayrac, J. Donahue, Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Mil- lican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 227366, 2022.
[3] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. /6660 2023.
[4] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.1 12966, 1(2):3, 2023.
[5] Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, 7. Chen, P. Chu, X. Dong. H. Duan, Q. Fan, Z. Fei, Y. Gao, J. Ge, C. Gu, Y. Gu, T. Gui, A. Guo, Q. Guo, He,
Y. Hu, T. Huang, T. Jiang, P. Jiao, Z. Jin, Z. Lei, J. Li, J. Li, L. Li, S. Li, W. Li, Y. Li, H. Liu,
J. Liu, J. Hong, K. Liu, K. Liu, X. Liu, C. Lv, H. Lv, K. Lv, L. Ma, R. Ma, Z. Ma, W. Ning,
L. Ouyang. J. Qiu, Y. Qu, F. Shang, Y. Shao, D. Song. Z. Song, Z. Sui, P. Sun, Y. Sun, H. Tang,
B. Wang, G. Wang, J. Wang, Wang, R. Wang. Y. Wang. Z. Wang. X. Wei, Q. Weng, F. Wu,
Y. Xiong, C. Xu, R. Xu, H. Yan. Y. Yan, X. Yang, H. Ye, H. Ying, J. Yu, J. Yu, Y. Zang, C. Zhang, L. Zhang, P. Zhang, P. Zhang, R. Zhang. S. Zhang, S. Zhang, W. Zhang, W. Zhang,
X. Zhang, X. Zhang, H. Zhao, Q. Zhao, X. Zhao, F. Zhou, Z. Zhou, J. Zhuo, Y. Zou, X. Qiu,
Y. Qiao, and D. Lin. Internim2 technical report, 2024.
[6] F. M. Carlucci, A. D'Innocente, S. Bucci, B. Caputo, and T. Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages
2019.
10

---

[7] J. Chen and A. Zhang. FedMBridge: Bridgeable multimodal federated learning. In Forty-first International Conference on Machine Learning, 2024.
[8] K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. 15/95, 2023.
[9] Y. Chen, X. Shen, Y. Liu, Q. Tao, and J. A. Suykens. Jigsaw-vit: Learning jigsaw puzzles in vision transformer. Pattern Recognition Letters, 166:53-60, 60, 2023.
[10] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang. L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
[11] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang. B. Li, P. Fung, and S. Hoi. InstructBLIIP. Towards general-purpont vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[12] R. Du, D. Chang, A. K. Bhunia, Xie, Z. Ma. Y.ZZ Song, and J. Guo. Fine-grained vi- sual classification via progressive multi-granularity training of jigsaw patches. In European Conference on Computer Vision, pages 168. Springer, 2020.
[13] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[14] A. C. Gallagher. Jigsaw puzzles with pieces of unknown orientation. In 2012 IEEE Conference on computer vision and pattern recognition, pages 382-389, IEEE, 2012.
[15] D. Ghosal, N. Majumder, A. Mehrish, and S. Poria. Text-to-audio generation using instruction- tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304.1377 2023.
[16] Y. Goyal, T. Khot, D. Summers-Star Stay, D. Batra, and D. Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[17] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018.
[18] D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[19] Huggingface Team. Introducing idefics: An open reproduction of state-of-the-art visual lan- guage model. http:///wwggingacceee.ooooo/////
2023.
[20] A. S. Hussain, S. Liu, C. Sun, and Y. Shan. M2ugen: dulti-modal music understanding and generation with the power of large language models. arXiv preprint arXiv:2311.11255, 2023.
[21] A. Kembhavi, M. Salvato, E. Kolve, M. Sco, H. Hajishirzi, and A. Farhadi. A diagram is worth
a dozen images, 2016.
[22] B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal Imms with generative comprehension. arXiv preprint arXiv:2307.16112, 2023.
[23] B. Li, H. Zhang, K. Zhang, D. Guo, Y. Zhang. R. Zhang, F. Li, Z. Liu, and C. Li. Llava-next: What else influences visual instruction tuning beyond data?. May 2024.
[24] B. Li, K. Zhang, H. Zhang, D. Guo, R. Zhang, Li, Zhang, Z. Liu, and C. Li. Llava-next: Stronger lims supercharge multimodal capabilities in the wild, May 2024.
[25] B. Li, Y. Zhang, D. Guo, R. Zhang, Li, Zhang, K. Zhang, Li, Liu, and C. Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
11

---

[26] C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. Llava-med: Training large language-and-Ve -iviinioon assistant for biomedicine in one day. Ad- vances in Neural Information Processing Systems, 36, 2024.
[27] F. Li, R. Zhang, H. Zhang, Y. Zhang, B. Li, W. Li. Z. Ma, and C. Li. Llava-next: Tackling multi imatt video, and 3d in large multimodal models, June 2024.
[28] Li, D. Li, S. Savarese, and S. Hoi. 22ip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[29] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. In International Conference on Machine Learning, pages 1 PMLR, 2022.
[30] K. Li, Y. He, Y. Wang. Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao. Videochat: Chat-centric video understanding. arXiv preprint 1rXiv:2305.0635: 2023.
[31] L. Li, Z. Xie, M. Li, S. Chen, P. Wang, L. Chen, Y. Yang, B. Wang. and L. Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665 2023.
[32] Y. Li, Y. Du, K. Zhou, J. Wang. W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint 20355 2023.
[33] Y. Li. B. Hu, H. Shi, W. Wang, L. Wang, and M. Zhang. Visiongraph: Leveraging large multi- modal models for graph theory problems visual context. arXiv preprini urXiv:2405.044: 049500 2024.
[34] Y. Li, C. Wang, and J. Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2025.
[35] B. Lin, Yc. B. Zhu, J. Cui, M. Ning, Jin, and L. Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint ar3iv:231 10122, 2023.
[36] D. Liu, X. Huang, Y. Hou, Z. Wang, Z. fei Yin, Y. Gong, P. Gao, and W. Ouyang. Uni3d-llo Unifying point cloud perception, generation and editing with large language models. ArXiv, abs/2402.03327, 2024.
[37] H. Liu, C. Li, Y. Li, and J. Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[38] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[39] H. Liu, C. Li. Q. Wu, and Y. J. Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
[40] Y. Liu, H. Duan, Y. Zhang, Li S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, K. Chen, and D. Lin. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06881, 2023.
[41] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[42] H.-Q. Nguyen, T.-D. Truong, X. B. Nguyen, A. Dowling, X. Li, and K. Luu. Insect-foundation: A foundation model and large-scale Im dataset for visual insect understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21945- 21955, 2024.
[43] T.-T. Nguyen, P. Jguyen, Cothren, A. Yilmaz, and K. Luu, Hyperglm: Hypergraph for video scene graph generation and anticipation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
12

---

[44] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69-84. Springer, 2016.
[45] Sarkar, S. Ebrahimi, A. Etemad, A. Beirami, S. Ó. Ank, and T. Pfister. Mitigating object hallucination via data augmented contrastive tuning. arXiv preprini arXiv:2405 18654, 2024.
[46] M. Siino. Merock at semeval-2024 task 4: Mistral 7b for multilingual detection of persua- sion techniques in memes. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2200 20244), pages 53-59, 2024.
[47] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17-8326, 2019.
[48] T.-D. Truong, Q--h. Bui, C. N. Duong, H.-S. Seo, S. L.. Phung, X. Li, and K. Luu. Direcformer: A directed attention in transformer approach to robust action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
[49] T.-D. Truong, H.-Q. Nguyen, X.-B. Nguyen, A. Dowling, X. Li, and K. Luu. Insect-foundation: foundation model and large multimodal dataset for vision-language insect understanding. International Journal of Computer Vision, pages 1-26, 2025.
[50] Wang, W. Zhou, Y. Huang, N. Xu, S. Zhang, H. Poon, and M. Chen. mdpo: Con- ditional preference optimization for multimodal large language models. arXiv preprint arXiv:2406. 1839, 2024.
[51] X. Wang, D. Song, S. Chen, C. Zhang, and B. Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via hybrid architecture. arXiv preprint 2ef:::2009002889, 2024.
[52] Y. Weng, M. Han, H. He, X. Chang, and B. Zhuang. Longvlm: Efficient long video under- standing via large language models. arXiv preprint 2xXiv:2404003384, 2024.
[53] C. Wu, W. Lin, X. Zhang, Y. Zhang, W. Xie, and Y. Wang. Pme-Ilama: toward building open-source language models for medicine. Journal of the American Medical Informatics Association, page ocac045, 2024.
[54] W. Xiao, Z. Huang, L. Gan, W. He, H. Li, Z. Yu, H. Jiang, F. Wu, and L. Zhu. Detecting and mitigating hallucination in large vision language models via fine-graine tne-grained-- ai feedback. arXiv preprint arXiv:2404. 14233, 2024.
[55] R. Xu, X. Wang, T. Wang. Y. Chen, J. Pang. and D. Lin. Pointllm Empowering large language models to understand point clouds. In ECCV, 2024.
[56] Z. Xu, Y. Zhang, F. Xie, Z. Zhao, Y. Guo, K.-Y. K. Wong, 7. Li, and H. Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
[57] S. Yang, J. Liu, R. Zhang, M. Pan, 7. Guo, X. Li, 7. Chen, P. Gao, Y. Guo, and S. Zhang- Lidar-Ilm: Exploring the potential of large language models for 3d lidar understanding. arXiv 2023.
preprint arXiv:2312.14001
[58] W. Ye. G. Zheng, Y. Ma. X. Cao, B. Lai, J. M. Rehg. and A. Zhang, Mm-spubench: Towards better understanding of spurious biases in multimodal 1lms. arXiv preprint 11XXi:24006.1226. 2024.
[59] S. Yin, C. Fu. S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
[60] T. Yu, Y. Yao, H. Zhang, T. He, Y. Han, G. Cui, J. Hu, Z. Liu, H.-T. Zheng, M. Sun, et al. Rihf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional hu- man feedback. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 13807-13816 2024.
13

---

[61] Т. Yu, H. Zhang, Y. Yao, Y. Dang, D. Chen, X. Lu, G. Cui, T. He, Z. Liu, T.-S. Chua, et al. Rlaif-v: Aligning milms through open-source feedback for super gpt-4v trustworthiness. arXiv preprint arXiv: 17220, 2024.
[62] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02 02490, 2023.
[63] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng. Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9556-9567, 2024.
[64] Y. Zhang, B. Li, h. Liu, j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li. Llava-next: A strong zero-shot video understanding model, April 2024.
[65] Y. Zhang, Zhang, Gu, Zhou, N. Lipka, D. Yang, and Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306. 17107, 2023.
[66] B. Zhao, B. Wu, M. He, and T. Huang. Svit: preprint arXiv:2307.04066 2023.
Scaling up visual instruction tuning. arXiv
[67] Y. Zhao, I. Misra, P. Krähenbühh, and R. Girdhar. Learning video representations from large language models. In Proceedingy of the IEEE/CVF Conference Computer Vision and Pal- tern Recognition, pages 6586-6597, 2023.
[68] Z. Zhao, B. Wang, L. Ouyang. X. Dong, J. Wang, and C. He. Beyond hallucinations: En- hancing Iylms through hallucination-awer direct preference optimization. arXiv preprint arXiv:2311 16839, 2023.
[69] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023,
14

---

Appendices
A Benchmarks
Following the standard benchmarks of LLaVa v1.5, we evaluate our models on two sets bench- marks, i.e., Academic Task-oriented Benchmarks and Instruction-Folloiing LMM Benchmarks. For the academic task-oriented benchmarks, we adopt five different benchmarks, including Visual Question Answering V2 (VQAv2) [16]. Question Answering on Image Scene Graphs (GQA) [18], Answer Visual Questions from People Who Are Blind (VizWiz) [17], Science Question Answer- ing (SciQA-IMG) [41], and Visual Reasoning based on Text in Images (TextVQA) [47]. While VQAv2 and GQA focus on evaluating the visual understanding based on the open-ended short an- swers, VizWiz evaluates the generalization of the model based the visual questions raised by impaired people. On the other hand, SciQA-IMG benchmarks will measure the performance of the LMM on scientific questions via multiple-choice questions. The TextVQA benchmark evalu- ates the capability of models in reading and reasoning about text in images. Meanwhile, we use six EsstuuctionFFolowvvii based Object Probing Evaluation for Object Hallucination (POPE) [32], Multimodal L.LMs with Generative Comprehension Benchmark (SEED-Bench) [22]. Comprehensive Evaluation Bench- mark of LMM (MME) [59]. LLaVA Benchmark in the Wild (LLaVA-Will) [39], Integrated Ca- pability Benehmark (MM- Vet) [62], and Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark (MMMU-VA Val) [63]. While the POPE benchmark evaluates the hallucination of the model based on the tree subset, i.e., random (rand), common (pop adv), and adversarial (adv), SEED-Bench measures the generative comprehension of the LMM on both images and videos with multiple-choice questions. For the video evaluation, we adopt the middle frame of the video as a vi- sual input. The MME perception benchmark evaluates visual understanding of the LMM via binary (yes/no) questions. Meanwhile, the L.LlVA-Wild and MM-Vet benchmarks measure the capabili- ties of the LMM in engaging in visual conversations. The MMMU benchmark evaluates LMMs on massive multi-discipline tasks demanding high-level subject knowledge and reasoning.
LMM Benchmarks evaluate our proposed approach, including Polling-
B Additional Ablation Study
Effectiveness of Data Size. Table 7 presents comparative analysis of LLaVA-7B and Direct- LLaVA-7B performance across varying data sizes during pre-training and tne-tuning, evaluated on range of LLM benchmarks. We the data training size of LLaVA v1 [39] and LLaVA v1.5 [37] for our experiments. Overall, both models demonstrate enhanced outcomes with larger fine-tuning datasets, with Direct-LLaVA consistently surpassing LLaVA. Notably, performance on instruction-followiin benchmarks, such as POPE, MME, and SEED-Bench, improves substantially as fine-tuning data increases. Specifically, the F1 score on the POPE benchmark rises by approxi- mately 12.5% across different configurations. For the MME benchmark, the scores of LLaVA and Laae increase markedly from 809.6 and 1 102.11 to 1510.7 and 1524.9, respectively. The accuracy gain on SEED-Bench is also significant, with larger datasets nearly doubling the mod- els' performance. Furthermore, Direct-LLaVA-7B consistently outperforms LLaVA across both academic-oriented and cnstruction-oolloiin benchmarks. For instance, on academic benchmarks, Direct-LLaVA-7B achieves 7.5% and 5% higher accuracy in ScienceQA and TextVQA, respec- tively. These results underscore the robustness of our proposed method.
Table 7: Effectiveness of Pre-training and Fine-tuning Data Size.
Data Size
SciQA
Text
POPE
SEED-Bench
Method
MME
Pretrain Finetune
IMG
VQA rand pop adv
all img vid
LLaVA-7B
595K
158K
46.9
26.1
76.3 72.2 70.1
809.6
33.5 37.0 23,8
Direct-LLaVA. VA-7B
595K
158K
54.6
29.3
79.5 76.2 74.3
1102.1
36.5 40.3 27.7
LLaVA-v1.5-7B
558K
665K
66.8
58.2
87.3
86.1 84.2
1510.7
58.6
66.1 37.3
Direct-LLaV--BB
558K
665K
74.3
63.2
88.8
88.9 86.0
1524.9
63.3 69.7 38.8
Scalability to Larger Data and Benchmarks. To illustrate our scalability to larger data and other benchmarks, conduct experiments on LLaVA-One Vision data [25] with LLaVA and Qwen2.5-
15

---

0.5B. We report our results on MME, MMMU, SeedBench-IMG, A12D [21], and MMBench [40]. As shown in Table 8, when data is scaling up. our proposed approach still maintains effectiveness and significantly improves the performance of LMM on various benchmarks.
Table 8: Effectiveness of Direct-LLaVA on Large Dataset.
% Samples for Pretext
MME
MMMU
SeedBench- IMG
A12D
MMBench
LL
1238
31.4
65.5
57.1
52.1
Direct-I. LaVA
50%
1351
32.9
67.5
62.6
55.4
Direct-LLaVA
100%
1494
34.5
68.4
69.4
58.7
Effectiveness of Data In Pretext. To understand the effectiveness of our proposed approach on the ratio of data used, we conducted an experiment using only 50% of data for our pretext tasks. As shown in Table 8, our approach can improve the performance of the LMMs. The results have further confirmed the effectiveness of our proposed learning approach.
Visualization of Shuffle Predictions. We provide the real-world images to illustrate our effective- ness of the shuffling learning mechanism. As shown in Figure for Direct-LLa with cls (no text in reconstruction), the LMM predicts the image order well but shows noticeable differences from the originals. In contrast, Direct-LLaVA VA with drt (text included) better aligns with the original images since information of language and visuals are well captured in drt token. To highlight the impact of text in reconstruction, we altered the description. Although image patches became inconsistent, the images were adapted to match the semantic meaning of the text.
Direct-LLavA
Direct-LLavA Using drt token
Direct-LLavA Using drt token
Origin Image
Using cls token
ALI MARE
Prompt: The cap is detached Pronpt: The cap is detached from the bottle and laid on fron the bottle and laid on the table, the Left side the table, on the right of the bottle, front front of the aliemaare box the atiemware box
of bottle,
Prompt: The dog is chasing the cat in rield covered yellow
Pronpt: The cat is chasing the dog in field covered yellow leaves.
Figure 7: Additional Visualization (Better in 2x zoom for details).
The choice to place single directed token drt the end of the input sequence is to ensure attends the full context from both visual and textual modalities under the autoregressive modeling constraint. As shown in Section 3.1, placing directed token after other tokens allows aggregate multimodal information effectively, which would not be possible if positioned earlier due to the causal attention mask. Our ablation study in Table validates this design: using directed tokens yields consistent performance gains over visual tokens across all benchmarks. In addition, to further justify this design, conducted
Ablation study on the placement of the directed token.
16

---

Table 10: Ablation study number of permuta- tions
Table 9; Ablation study on difference place- ment of directed token drt
# Permutations
Science IMG
Tex
Placement
Science IMG
TextVQA
5000
72.5
62.1
Beginning 70.9
61.4
10000
74.3
63.2
End
74.3
63.2
15000
74.5
64.0
an additional experiment comparing placements at the beginning and end of the sequence. Results show that placing the token at the end yields superior performance, supporting our choice.
Ablations the number of permutations. Our selection of 10,000 permutations follows the well-establiseed practice in self-supervised vision learning, particularly works on jigsaw-based tasks [6, 44, 48]. Specifically, between them, ensuring sufficient diversity while avoiding overly trivial or degenerate cases. To val- idate this choice, increasing from 5,000 to 10,000 leads to improved performance on ScienceQA-IMG and TextVQA, confirming that a richer permutation space helps the model better learn spatial relationships. How- ever, increasing the number to 15,000 leads to only slight improvements, indicating that 10,000 well-chosen permutations already provide sufficient diversity for effective learning, and additional permutations may have limited impact.
sample permutations maintain balanced Hamming distance
conducted ablation varying the number of permutations. As shown below,
Guided loss LI-R- To isolate the effect of the Image- to Response Guided Loss, we conducted an additional experiment by training LLaVA-115 with only the next-token prediction loss and the Image-to-Responee shuffle learning tasks. This setting allows us to directly assess the contribution of the guided loss visual-textual alignment. As shown in Table 11, Image-to-Response Guided loss consistently improves performance across all benchmarks, demonstrating its effectiveness in reinforcing visual- textual alignment, yet remains lower than our full Direct-LLaVA model. This result suggests that while the loss improves visual grounding, the Image-to-Resoone loss alone may not be sufficient for robust multimodal reasoning.
Ablation study the mmage-to-Respone
Guided Loss, excluding our proposed
Table 11: Ablation study Image-to-Response Guided loss
POPE
BEED-Bench
Model
Science IMG
TextVQa
MME
rand
pop
adv
all
img
vid
LLaVA LLaVA with LI-.R
66.8
58.2
1510.7
87.3
86.1
84.2
58.6
66.1
37.3
69.9
60.2
1518.4
87.9
87.8
85.1
59.9
67.4
38.4
Direct-LLaVA
74.3
63.2
1524.9
88.8
88.9
86.0
63.3
69.7
38.8
C Pseudocode for the training procedure of our framework
The Algorithm elucidates the framework of our proposed Direct-LLaVA during two-stage training process.
D Discussion of Limitations
Our paper has adopted a specific set of parrae-prraeters and learning methods to support our hy- pothesis. However, work could contain several limitations. Our work investigated the effec- tiveness of our proposed learning tasks and losses in improving the LMM's performance. Thus, the investigation of balance weights among learning objectives has not been fully exploited, and we leave this experiment as our future work. Due to computation limitations, our experiments are limited to the standard language model size (i.e., Vicuna 13B, Vicuna 7B, I.AMA3 8B, and Qwen 7B) and data scale (LLaVV v1.5). Nevertheless, we hypothesize that our proposed approaches can generalize to larger-scale language models and data settings due to their fundamental theories.
17

---

Algorithm Training Step Two-Stage Training Procedure of Direct-LLavA
Input: Initial model parameters 0, training dataset D = (x, {1 x )
Output: Updated model parameters 6*
1: for each iteration i do
2: (I,p, p. {x < <d)))=n>= + ()t
3: Task 1: Autoregressive with Cross-Entropy and Guided Attention Losses
4:
Tinstruct <-
b Equ. (3)
5:
Ta x
6:
LCE f CrossEntropyyLoss(( Tinstructs Ta)
b Eqn. (4)
7:
LI-R Image
Ta)
Eqn. (12)
8:
&- A - Vo(LCE + L1-n)
9:
Task 2: Image Order Reconstruction
I Instrucc
image & Geelnstructioon(Imaggl
b Eqn. (7)
11:
F f- P(x, k)
b Shuffle the image
12:
Limuge-Onde Order PeeediccPermutaaioonn (ö, iinstruct
> Eqn. (6)
13:
4-0 B Vo(Limage-Caal
14:
Task 3: Text Order Reconstruction (Pre-training Only)
15:
TeXt
< Get IntlnnnuuttoonTeessse Task')
b Eqn. (8)
instrnet
Rext concat(p, drt)
17:
LText-Onder 4- ReorderPreding
b Eqn. (9)
a+ 0 V (e(CText-Order)
19: end for
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims declared in the abstract match with the contributions, experimental results, and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the paper are discussed in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [es]]
19

---

Justification:
The description of the formula is provided in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The details of datasets and implementations are presented in the experimental sections.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification:
The code will be published may the paper be accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: (https://nipp /ttp:////ppcc//////
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips... ///ipscc////// public/guiden Seweedeeeeeeemmmmeeeeee onPolicy) for more details.
cc/
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The details of training and testing are presented in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Following the standard evaluation of large multimodal models, we evaluate our model by the standard metrics of the evaluation benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g,, Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational resources used our experiments are presented in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ate:://neuuippee http:r/p.....ppppet.etcccccceeee
Answer: [Yes]
Justification:
The content of the paper and datasets strictly follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The paper does not have negative societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitiga- tion strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not have risk. The released models will be available may the paper be accepted.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper provides all the references to code, data, and models used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, haperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce the new dataset. The code of the paper will be published may the paper be accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The research in this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research in this paper does not involve crowdsourcing nor research with human subjects. Thus, there is no requirement for IRB.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not utilize the LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy futtps://neur/: / for what should or should not be described.
25


Reviews:
[{"review_id": "hBspBqjAQD", "reviewer": "Reviewer_oM4U", "summary": "This paper proposes Direct-LLaVA, a new method that introduces directed tokens and shuffle learning objectives (reconstructing image and text order) into the training of vision-language models (LMMs) to improve multimodal alignment and robustness. The authors further incorporate an Image-to-Response Guided Loss to strengthen the influence of visual inputs in textual outputs. Extensive experiments on academic and instruction-following benchmarks demonstrate consistent performance gains over LLaVA-1.5 and other baselines.", "questions": "see weakness", "limitations": "Limitations such as scale constraints and lack of hyperparameter sensitivity exploration are briefly discussed in Appendix C, though more analysis could still be provided.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n- Introduces two new learning objectives—reconstructing image order and text order—to directly improve cross-modal alignment and reduce over-reliance on the language modality.\n- Proposes a novel “directed token” mechanism to aggregate visual and textual information for permutation prediction, showing consistent improvements over baseline designs.\n- Achieves SoTA results across multiple benchmarks (e.g., VQAv2, GQA, VizWiz, SciQA-IMG, POPE, SEED-Bench, MMMU-Val) under different LLM backbones (Vicuna, LLaMA, Qwen).\n- Presents thorough ablation studies on model components (token types, shuffle strategies, LLMs), demonstrating robustness and effectiveness across different configurations.\n- Provides clear implementation details and fair comparisons using consistent training data, model sizes, and evaluation protocols.\n\n### Weaknesses\n\n- The design choice of using a single directed token at the end of the sequence lacks deeper analysis; alternative placements or token designs are not explored.\n- There is a lack of analysis of failure cases where multimodal misalignment still occurs; visualizations are limited and do not systematically reveal limitations.\n- The shuffle learning tasks are somewhat synthetic and may not fully represent the types of semantic misalignment encountered in real-world scenarios.\n- The main components (e.g., permutation prediction, guided attention, special tokens) are individually incremental, and the novelty lies more in the combination than in fundamental algorithmic innovation.", "was_revised": true, "final_justification": "Thank for authors' rebuttal, which addressed my concerns. I will maintain my positive attitude and score.", "rebuttal": "Dear Reviewer oM4U,\n\n\nWe sincerely appreciate your positive feedback. We are glad you appreciated our novel learning objectives and directed token design, and the robustness of our results across different benchmarks. We are also grateful that the clarity of our methodology and thorough ablation studies were particularly notable. Your comments are very much appreciated, and we would like to address your points as follows.\n\n\n[Q1] **Alternative placements of the directed token**\n\n\n[A1] We chose to place a single directed token at the end of the input sequence to ensure it attends to the full context from both visual and textual modalities under the autoregressive modeling constraint. As shown in Section 3.1, placing the directed token after all other tokens allows it to aggregate multimodal information effectively, which would not be possible if positioned earlier due to the causal attention mask. Our ablation study in Table 5 validates this design: using directed tokens yields consistent performance gains over visual tokens across all benchmarks. In addition, to further justify this design, we conducted an additional experiment comparing placements at the beginning and end of the sequence. Results show that placing the token at the end yields superior performance, supporting our choice.\n\n\n| Place     | Science IMG | TextVQA |\n| --------- | ----------- | ------- |\n| Beginning | 70.9        | 61.4    |\n| End       | **74.3**        | **63.2**    |\n\n\n[Q2] **Visualization of multimodal misalignment**\n\n\n[A2] While visualization is restricted by the rebuttal policy, we have analyzed failure cases through systematic ablations and benchmark results. As shown in Table 1, LLaVA-v1.5 exhibits minimal performance drop when visual input is replaced with a black image, highlighting misalignment where the model overly relies on language priors. Our ablation studies in Table 6 further indicated that removing the proposed Image-to-Response Guided loss or shuffle learning leads to performance degradation across vision-intensive benchmarks (e.g., SciQA drops from 74.3\\% to 72.3\\%), indicating residual multimodal gaps. We will incorporate more visualization in our revised versions.\n\n\n[Q3] **The shuffle learning tasks may not fully represent semantic misalignment encountered in real-world scenarios.**\n\n\n[A3] While the tasks are constructed in a controlled manner, they are inspired by well-established techniques in vision learning, where solving shuffled inputs (e.g., jigsaw puzzles) has been widely used to improve spatial reasoning and feature generalization [1, 2, 3]. Our adaptation of this paradigm to multimodal settings aims to explicitly address real-world misalignment issues, as shown in Table 1, where LLaVA performs similarly with or without visual input. By forcing the model to reconstruct the correct order of image patches or textual segments, our method strengthens the visual-textual grounding. This is further supported by the performance improvement on real-world benchmarks (Table 3), indicating the effectiveness of shuffle learning beyond synthetic settings.\n\n\n[Q4] **The main components are individually incremental, and the novelty lies more in the combination than in fundamental algorithmic innovation.**\n\n\n[A4] We respectfully but strongly disagree with the reviewer’s assessment. The novelty of our work lies in the synergistic integration of shuffle learning, directed tokens, and guided attention into a unified framework that addresses the underexplored challenge of modality misalignment in LMMs. Our approach spans both image and text modalities through image order reconstruction and text order reconstruction tasks during pre-training. It continues with image order reconstruction during fine-tuning, enabling consistent cross-modal alignment across training stages. The proposed Image-to-Response Guided Loss further enforces visual grounding via attention supervision, a mechanism not explored in prior LMMs. As reflected in performance gains across 10+ real-world benchmarks (Tables 2–3), this integrated design delivers practical impact beyond incremental additions. Moreover, Reviewer 2 highlights the elegance of the directed token under causal attention; Reviewer 3 notes the effectiveness of coupling it with shuffle learning; and Reviewer 4 affirms that our combination of tasks and losses yields consistent improvements, even when model backbones vary.\n\n\n**References**\n\n\n[1] Carlucci, et al. Domain generalization by solving jigsaw puzzles. CVPR, 2019.\n\n\n[2] Noroozi, et al. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016.\n\n\n[3] Truong, et al. DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition. CVPR, 2022."}, {"review_id": "Qe3Oewn5QM", "reviewer": "Reviewer_Xcbu", "summary": "This paper proposes Direct-LLaVA, a framework designed to improve the alignment between visual and textual modalities in large multimodal models (LMMs). The authors introduce two novel shuffle learning tasks: image order reconstruction and text order reconstruction, to be used during both pre-training and fine-tuning. The key contribution is the directed token, which is learnable and is appended to the input sequence to capture cross-modal information. The paper also introduces an Image-to-Response Guided Loss to enforce attention from visual inputs to textual outputs. Experimental results demonstrate state-of-the-art performance across multiple academic and instruction-following benchmarks.", "questions": "Questiones: \n\n1. I suggest the authors to reformat the paper, and put some experiments into the appendix. \n\n2. It would be better to have a complete algorithm box. \n\n3. How is the balance among multiple losses (CE, image/text order, and attention-guided loss) tuned? Are these loss weights fixed or adaptively adjusted during training?", "limitations": "Yes, but the limitation is not sufficient. And it is attached as the supplementary, very difficult to find.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The proposed “Directed Token” provides a simple yet elegant solution to cross-modal information integration under autoregressive modeling. The token captures both visual and textual cues more effectively than traditional visual tokens, as confirmed through comprehensive ablation studies.\n\n2. This paper is well motivated. The authors demonstrate that existing models like LLaVA often ignore visual inputs, producing nearly identical outputs even with blank images. \n\n3. The model achieves state-of-the-art results across a wide range of academic and instruction-following benchmarks, including VQAv2, GQA, SciQA-IMG, TextVQA, POPE, MMMU, and MME.\n\nCons:\n\n1. The paper's structure and formatting can be improved. There is no need to put everything in the main context, it would be easier to follow if the authors put some content as the appendix. For example, Table 2 and Figure 5 are too small to be well understood. \n\n2. While the Directed Token is empirically effective, the paper lacks deeper theoretical justification or analysis regarding why it facilitates better cross-modal alignment. Its design is intuitive, but the underlying learning dynamics remain unexplored.", "was_revised": true, "final_justification": "My concerns have been well addressed, and I will recommend this paper as \"accept.\"", "rebuttal": "Dear Reviewer Xcbu,\n\n\nWe sincerely appreciate your thoughtful and encouraging feedback. We are pleased that you found our proposed method a simple yet effective approach to cross-modal integration, with strong motivation and compelling empirical support. Your recognition of our experimental rigor and comprehensive benchmarking is truly appreciated. We value your comments and would like to respond to your points as follows.\n\n\n[Q1] **The paper's structure and formatting can be improved.**\n\n\n[A1] We agree that some structural improvements could enhance readability. In the revised version, we will restructure our paper as your suggestion to improve the presentation of the paper.\n\n\n[Q2] **The paper lacks deeper theoretical justification or analysis regarding why the Directed Token facilitates better cross-modal alignment.**\n\n\n[A2] We thank the reviewer for the thoughtful comment. The directed token is theoretically motivated by the causal attention constraint in autoregressive LMMs, where earlier tokens (e.g., visual inputs) cannot attend to later ones (e.g., text). By placing a learnable token at the end of the sequence, it has full access to both modalities and functions as a global integrator of cross-modal information. From an information-theoretic view, it acts as a bottleneck token, summarizing joint semantics for downstream tasks. This design aligns with our experimental findings. As shown in Table 5, using the directed token significantly improves performance over standard visual tokens.\n\n\n[Q3] **It would be better to have a complete algorithm box.**\n\n\n[A3] We appreciate your suggestion. The subsequent section elucidates the algorithm employed during the two-stage training process.\n\n\n### **Algorithm: Training Step in the Two-Stage Training Procedure of Direct-LLaVA**\n\n\n**Input**: \n- Initial model parameters: $\\theta$ \n- Training dataset $\\mathcal{D} = \\{(x, p, \\\\{(x_q^t, x_a^t)\\\\}_{t=1}^T)\\}$ \n- Number of training iterations: $N$\n\n\n**Output**: \n- Updated model parameters: $\\theta^*$\n\n\n---\n\n\n**for** *each iteration* **do** \n1. Sample a training instance $(x, p, \\\\{(x_q^t, x_a^t)\\\\}_{t=1}^T)$ from $\\mathcal{D}$\n\n\n2. **Autoregressive Task with Cross-Entropy and Guided Attention Loss**   \n  a. Construct instruction input:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$x\\_{\\text{instruct}} \\leftarrow \\text{GetInstruction}(\\text{`ARTask'})$ (Eq. 3)   \n  &nbsp;&nbsp;&nbsp;&nbsp;$x_a \\leftarrow p$   \n  b. Compute losses:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\mathcal{L}\\_{\\text{CE}} \\leftarrow \\text{CrossEntropy}(x, x\\_{\\text{instruct}}, x\\_a)$ (Eq. 4)  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\mathcal{L}\\_{\\text{I→R}} \\leftarrow \\text{ImageToResponseLoss}(x, x\\_a)$ (Eq. 12)  \n  c. Update model:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\theta \\leftarrow \\theta - \\nabla\\_\\theta (\\mathcal{L}\\_{\\text{CE}} + \\mathcal{L}\\_{\\text{I→R}})$\n\n\n3. **Image Order Reconstruction Task**  \n  a. Construct instruction:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$x\\_{\\text{instruct}}^{\\text{image}} \\leftarrow \\text{GetInstruction}(\\text{`ImageTask'})$ (Eq. 7)  \n  b. Shuffle image:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\bar{x} \\leftarrow \\mathcal{P}(x, k)$  \n  c. Compute loss:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\mathcal{L}\\_{\\text{Image-Order}} \\leftarrow \\text{PredictPermutation}(\\bar{x}, x\\_{\\text{instruct}}^{\\text{image}})$ (Eq. 6)  \n  d. Update model:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\theta \\leftarrow \\theta - \\nabla\\_\\theta (\\mathcal{L}\\_{\\text{Image-Order}})$  \n\n\n4. **Text Order Reconstruction Task (Pre-training Only)**  \n  a. Construct instruction:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$x\\_{\\text{instruct}}^{\\text{text}} \\leftarrow \\text{GetInstruction}(\\text{`TextTask'})$ (Eq. 8)  \n  b. Construct answer:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$x\\_a^{\\text{text}} \\leftarrow \\texttt{concat}(p, \\mathbf{drt})$  \n  c. Compute loss:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\mathcal{L}\\_{\\text{Text-Order}} \\leftarrow \\text{ReorderPrediction}(x, x\\_{\\text{instruct}}^{\\text{text}}, x\\_a^{\\text{text}})$ (Eq. 9)  \n  d. Update model:  \n  &nbsp;&nbsp;&nbsp;&nbsp;$\\theta \\leftarrow \\theta - \\nabla\\_\\theta (\\mathcal{L}\\_{\\text{Text-Order}})$ \n\n\n**end for**\n\n[Q4] **How is the balance among multiple losses tuned? Are these loss weights fixed or adaptively adjusted during training?**\n\n\n[A4] In our current implementation, we assign equal weighting to all loss components (cross-entropy, image/text order reconstruction, and attention-guided loss) for simplicity and fairness in evaluating their contributions. As noted in our limitations, we have not yet explored loss weighting strategies. Despite this, our model still achieves consistent performance gains across all benchmarks (Tables 2–3), indicating the robustness of the proposed objectives even under uniform weighting. We reserve the investigation of loss weighting for future work."}, {"review_id": "7gPVtDlp7A", "reviewer": "Reviewer_kMkc", "summary": "This paper proposes Directed Tokens, a simple and effective approach to enhance multimodal alignment in LMMs through shuffle learning tasks. Specifically, it introduces two ordering tasks for the image and text, respectively, with a directed token to strengthen multimodal understanding alignment. To better utilize visual information during response generation, an Image-to-Response Guided Loss is further designed. Experiments show that the proposed method achieves superior results on multiple benchmarks. However, I have some concerns about this paper.", "questions": "Refer to Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Positive points**\n1. The paper introduces a novel shuffle learning paradigm coupled with directed-token mechanisms, effectively enhancing visual-textual alignment in LMMs by forcing models to reconstruct correct modality orders during both pre-training and fine-tuning phases.\n2. The proposed Image-to-Response Guided loss explicitly enhances visual feature utilization during inference through attention-layer optimization, significantly reducing language-preference bias while maintaining the original autoregressive modeling framework.\n\n**Negative points**\n1. The core idea of this paper is simple and intuitive, but the overall writing quality appears rushed and needs further polishing.\na)Several figures are currently placed in narrow side columns with reduced size. Given the single-column template, reformatting them as full-width figures would improve clarity. In addition, the visual design of some figures and tables looks somewhat rough and could be refined for better presentation quality.\nb)Figure and table references are currently written in plain text rather than using cross-referencing.\nc)There are also several other issues in equations factual correctness. Please refer to the Minor issues for details.\n2. The Related Work section is overly generic regarding large multimodal models. It would benefit from a more structured discussion of representative LMM alignment paradigms, such as cross-attention (Flamingo, CogVLM[A]), Q-former (BLIP-2, MiniGPT-4), and MLP (LLaVA, Qwen2-VL[B]), as well as recent research on auxiliary task design[B] and attention mechanism[C] for improving vision-language alignment.\n3. The image permutation reconstruction task lacks sufficient implementation detail. The paper would benefit from a clearer explanation of how the permutation index $k$ is represented and predicted, including the design of the order projection layer and related hyperparameters.\n4. The design of the permutation space appears somewhat heuristic. While selecting 10,000 permutations from a large combinatorial space is effective, it would be helpful to include ablations on the number of permutations and patch size.\n5. If I understand correctly, the two shuffle learning tasks are designed to be independent, with each forward-backward pass performing either image order reconstruction or text order reconstruction. However, in Equation 13, these two loss terms appear simultaneously, which seems inconsistent with this setup and introduces ambiguity.\n6. The paper could benefit from an analysis of failure cases or potential trade-offs. For example, the Image-to-Response Guided Loss may lead the model to over-rely on visual cues, possibly affecting its instruction-following or text-only capabilities.\n7. Some experimental results (rows 4, 5, and 6) in Table 3 appear incomplete or too limited in scope. These entries could either be expanded with additional metrics or removed if full results are unavailable.\n\n\n**Minor Issues:**\n1. In Eq. 2, the notation for the image input is denoted as $x$ initially and later as $I$, which is inconsistent.\n2. In Eq. 8, $[x, p, \\overline{p}]$ should be $[x, q, \\overline{p}]$.\n3. Eq.4 appears to be redundant with Eq.2.\n4. The symbol $L$ is used to represent both the number of layers (in Eq. 12) and the sequence length (in Eq. 2), which may cause confusion.\n5. In line 133, Page 4, the phrase “two stages of the instruction-tuning procedure” is inaccurate, as instruction-tuning itself is the second stage.\n6. In line 236, Page 6, the expression $x_a^T = [x_a^T, \\mathbf{drt}]$ seems problematic due to self-referencing on both sides of the equation.\n7. In line 256, Page 6, “the attention score from the visual token $v$ to the response token $r$”may cause confusion, as it suggests the visual token attends to the response token, which contradicts the autoregressive nature of decoding.\n\n\n[A] Cogvlm: Visual expert for pretrained language models. NeurIPS 2024.  \n[B] Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. Arxiv 2024.  \n[C] Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS 2021.  \n[D] Visual grounding via accumulated attention. CVPR 2018.", "was_revised": true, "final_justification": "The response has addressed my main concerns about the manuscript, and the alignment of vision and language is of great significance in the MLLM field. Therefore, I adjust my rating to borderline accept, acknowledging the potential contribution of this work to the community.", "rebuttal": "Dear Reviewer kMkc,\n\n\nWe sincerely appreciate your thoughtful and encouraging feedback. We are pleased that you appreciate the novelty of our modality-shuffling framework and the effectiveness of our guided loss in improving visual-text alignment and reducing language bias. Your comments are deeply valued, and we address your specific points below.\n\n\n[Q1] **The core idea of this paper is simple and intuitive, but the overall writing quality appears rushed and needs further polishing.**\n\n\n[A1] Thank you for the constructive feedback. In the revised version, we will carefully reformat all figures and tables as suggested to ensure better readability. We appreciate the reviewer’s attention to these details and will ensure a polished revision.\n\n\n[Q2] **The Related Work section is overly generic regarding large multimodal models.**\n\n\n[A2] We thank the reviewer for the valuable suggestions [4, 5, 6, 7]. In the revised version, we have incorporated a more structured discussion of alignment paradigms in LMMs, including cross-attention mechanisms (e.g., Flamingo, CogVLM), Q-former-based alignment (e.g., BLIP-2, MiniGPT-4), and MLP-based projections (e.g., LLaVA, Qwen2-VL). These additions help position our work more clearly within the broader landscape of multimodal learning.\n\n\n[Q3] **The image permutation reconstruction task lacks sufficient implementation detail.**\n\n\n[A3] We have detailed our implementation in L163–165. We denote the permutation index as $k$, which represents the specific ordering of image patches applied via a permutation function $P(x, k)$. To predict $k$, we introduce a visual order projection head, implemented as a linear layer applied to the last hidden-state representation of the directed token. The model learns to regress the permutation index $k$ from this representation.\n\n\n[Q4] **Ablations on the number of permutations and patch size.**\n\n\n[A4] Our selection of 10,000 permutations follows the well-established practice in self-supervised vision learning, particularly works on jigsaw-based tasks [1, 2, 3]. Specifically, we sample permutations to maintain a balanced Hamming distance between them, ensuring sufficient diversity while avoiding overly trivial or degenerate cases. To validate this choice, we conducted an ablation varying the number of permutations. As shown below, increasing from 5,000 to 10,000 leads to improved performance on ScienceQA-IMG and TextVQA, confirming that a richer permutation space helps the model better learn spatial relationships. However, increasing the number to 15,000 leads to only slight improvements, indicating that 10,000 well-chosen permutations already provide sufficient diversity for effective learning, and additional permutations may have limited impact.\n\n\n| # Permutations      | Science IMG | TextVQA |\n| ----- | ----------- | ------- |\n| 5000  | 72.5        | 62.1    |\n| 10000 | 74.3        | 63.2    |\n| 15000 | **74.5**        | **64.0**    |\n\n\n[Q5] **Two loss terms appearing simultaneously in Equation 13 introduce ambiguity.**\n\n\n[A5] Eqn (13) is not ambiguous. We clarify that although Eqn. (13) presents the three all loss components in a single expression for conciseness, they are not applied simultaneously within a single forward-backward pass. As clarified in Algorithm 1 (included in the revised version), each training sample is assigned to only one task per step: either autoregressive prediction, image order reconstruction, or text order reconstruction. These tasks are handled in independent forward-backward passes, and the corresponding loss is computed based on the selected task. Eqn. (13) is intended to summarize the overall training objective across the dataset, not a per-step computation. We will update the text surrounding Eqn. (13) to better reflect this and avoid potential confusion.\n\n\n[Q6] **Analysis of failure cases or potential trade-offs.**\n\n\n[A6] We conducted an additional experiment using LLaVA trained with only the Image-to-Response Loss, without incorporating our shuffle learning tasks. The resulting performance on SciQA-IMG and TextVQA is lower than that of our full Direct-LLaVA model, but still demonstrates performance improvement over the LLaVA baseline. This result suggests that while the loss improves visual grounding, the Image-to-Response loss alone may not be sufficient for robust multimodal reasoning.\n\n\n|                        | Science IMG | TextVQA |\n| ---------------------- | ----------- | ------- |\n| LLaVA                  | 66.8        | 58.2    |\n| LLaVA with I-to-R Loss | 69.9        | 60.2    |\n| Direct-LLaVA           | **74.3**        | **63.2**    |\n\n\n[Q7] **Some experimental results in Table 3 appear incomplete or too limited in scope.**\n\n\n[A7] We acknowledge that rows 4, 5, and 6 in Table 3 appear limited in scope due to missing metrics. These entries correspond to baseline models whose full results are not publicly disclosed in the original papers; thus, we have included the available results for completeness and reference. In the revised version, we will clarify this in the table caption and main text. We are open to removing these rows if the limited reporting causes confusion or detracts from clarity.\n\n\n[Q8] **Minor issues**\n\n\n[A8] We would like to thank the reviewer for the suggestion. We have revised and updated the minor issues to improve the presentation of our paper.\n\n\n\n\n\n\n**References**\n\n\n[1] Carlucci, et al. Domain generalization by solving jigsaw puzzles. CVPR, 2019.\n\n\n[2] Noroozi, et al. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016.\n\n\n[3] Truong, et al. DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition. CVPR, 2022.\n\n[4] Wang et al. Cogvlm: Visual expert for pretrained language models. NeurIPS, 2024.\n\n[5] Wang et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv, 2024.\n\n[6] Li et al. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 2021.\n\n[7] Deng et al. Visual grounding via accumulated attention. CVPR, 2018."}, {"review_id": "Ffz3bxitnf", "reviewer": "Reviewer_5PqY", "summary": "This paper aims at improving the vision-language alignment of Large Multimodal Models (LMMs). It starts by ascertaining that the popular LMM LLaVA 1.5 pays little attention to image inputs, as its VQA performance is just moderately decreased if the visual input is removed. Inspired by the jigsaw puzzle pre-training scheme, the paper proposes to add three different losses in the pre-training and visual instruction tuning stages of LLaVA: the first loss tasks the model to reconstruct the order of the input image where the patches have been permuted; similarly, the second loss tasks the model to reconstruct the order of the permuted response. This is only applied during pre-training; finally, the third Image-to-Response Guided Learning loss encourages the model to pay more attention to visual tokens. The reconstruction of the image in the right order is feasible thanks to a set of Directed Tokens that are additionally fed to the LMM and whose representation at the output layer serves to predict the index of the k-th image patch.\nExperiments are done on the same dataset employed by LLaVA and demonstrate consistent improvements over LLaVA 1.5.", "questions": "See **WEAKNESSES** above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 1, "strengths_and_weaknesses": "**STRENGTHS:**\n\nStrong results. While the idea of shuffle learning is not new, its application to image and text, along with the Image-to-Response loss, brings consistent and solid improvements over LLaVA across a multitude of benchmarks, given the same training data. The effectiveness of the proposed method remains persistent even when varying the Large Language Models.\n\n\n**WEAKNESSES:**\n\n1. Ablation studies in Table 6 lack a fundamental experiment to isolate the role of the Image-to-Response Guided Loss. The last row of the table does show its effect when paired with the shuffle learning objectives, but we do not know what would be the results of training LLaVA 1.5 only with the Image-to-Response Guided Loss, in addition to the usual next token prediction loss. Please add this during the rebuttal.\n\n2. Computational cost. The authors experiment with 32 A100 GPUs, while LLaVA 1.5 only requires 8 A100 GPUs (assuming 80GB of VRAM for each A100). I imagine this is because each training sample must be forwarded to the model three times: one for the next token prediction with original image and text, one for image shuffle learning with the permuted image, and one for text shuffle learning with the permuted text. Consequently, to keep the same batch size as LLaVA, it is necessary to increase GPUs. One benefit of this method is to improve LLaVA without having access to new training data. Yet, computational resources are important too. It would be good to include the results of Direct-LLaVA trained with the same amount of GPUs as LLaVA 1.5.\n\n3. The choice of the shuffling strategy (L276-287) is theoretically justified, but it lacks empirical ablation. What does it happen if we just use purely random permutations?\n\n4. Presentation. I feel that figures, and tables in particular, have a small font size, making it difficult to read numbers. Moreover, I do not see the point of including Table 4, as it reports the very same results as Tables 2 and 3.\n\n5. It would be interesting to repeat the experiment from Table 1 with Direct-LLaVA too.", "was_revised": true, "final_justification": "After reading the rebuttal, my concerns have been solved, and I think that the paper now meets the acceptance bar.", "rebuttal": "Dear Reviewer 5PqY,\n\n\nWe sincerely appreciate your constructive feedback. We are pleased that you found our method to yield robust and consistent improvements across benchmarks and LLMs. We would like to address your points in detail below.\n\n\n[Q1] **Ablation studies in Table 6 lack a fundamental experiment to isolate the role of the Image-to-Response Guided Loss.**\n\n\n[A1] To isolate the effect of the Image-to-Response Guided Loss, we conducted an additional experiment by training LLaVA-1.5 with only the next-token prediction loss and the Image-to-Response Guided Loss, excluding our proposed shuffle learning tasks. This setting allows us to directly assess the contribution of the guided loss to visual-textual alignment. The resulting performance on ScienceQA-IMG and TextVQA shows improvement over the LLaVA-1.5 baseline, but remains lower than our full Direct-LLaVA model. This result suggests that while the loss improves visual grounding, the Image-to-Response loss alone may not be sufficient for robust multimodal reasoning.\n\n\n|                        | Science IMG | TextVQA |\n| ---------------------- | ----------- | ------- |\n| LLaVA                  | 66.8        | 58.2    |\n| LLaVA with I-to-R Loss | 69.9        | 60.2    |\n| Direct-LLaVA           | **74.3**        | **63.2**    |\n\n\n[Q2] **Computational cost: Direct-LLaVA trained with the same number of GPUs as LLaVA 1.5.**\n\n\n[A2] We clarify that our experiments were conducted using A100 GPUs with 40GB of VRAM, not 80GB. We employed 32 GPUs primarily to accelerate training and support the breadth of ablation studies, not to increase batch size or model capacity. Importantly, despite the additional compute, we strictly maintained the same training hyperparameters as LLaVA-1.5, including batch size, learning rate, optimizer, and number of epochs, to ensure a fair and controlled comparison.\n\n\n\n\n[Q3] **The use of purely random permutations**\n\n\n[A3] In the early development phase, we have already experimented with purely random permutations sampled from the full permutation space. However, the model failed to converge under this setting. This is primarily because visual random permutations are extremely hard to predict, especially when the image patches are rearranged into highly disordered configurations with no spatial continuity. This randomness creates an enormous and unstructured search space for the permutation prediction task, resulting in unstable and ineffective learning objectives. Moreover, many of these permutations are either too trivial or too chaotic, providing inconsistent or uninformative supervision. To address this, we adopted a more structured strategy by sampling 10,000 permutations with controlled Hamming distances, following established practices in self-supervised learning (e.g., [1, 2, 3]). This design ensures a more learnable yet sufficiently diverse permutation space, enabling better training stability and performance. To validate this choice, we conducted an ablation varying the number of permutations. As shown below, increasing from 5,000 to 10,000 leads to improved performance on ScienceQA-IMG and TextVQA, confirming that a richer permutation space helps the model better learn spatial relationships. However, increasing the number to 15,000 leads to only slight improvements, indicating that 10,000 well-chosen permutations already provide sufficient diversity for effective learning, and additional permutations may have limited impact.\n\n\n| # Permutations      | Science IMG | TextVQA |\n| ----- | ----------- | ------- |\n| 5000  | 72.5        | 62.1    |\n| 10000 | 74.3        | 63.2    |\n| 15000 | **74.5**        | **64.0**    |\n\n\n[Q4] **Presentation of some tables and figures**\n\n\n[A4] We will revise the figures and tables to ensure that all numerical values are clearly legible in the camera-ready version. Regarding Table 4, its purpose is to highlight the impact of different large language models (LLMs) under a unified setting (Direct-LLaVA and LLaVA baselines) to isolate the effect of the LLM backbone. While some values overlap with Tables 2 and 3, Table 4 provides a focused comparison across models using the same architecture and training setup, which is not the emphasis of Tables 2 and 3.\n\n\n[Q5] **It would be interesting to repeat the experiment from Table 1 with Direct-LLaVA too.**\n\n\n[A5] We conducted the same experiment as in Table 1 by replacing the visual input with a black image and evaluating Direct-LLaVA. However, unlike LLaVA, our model is explicitly trained to depend on meaningful visual input through image order reconstruction and Image-to-Response Guided Loss. As a result, when presented with blank images, Direct-LLaVA consistently refuses to generate an answer or outputs an empty response, leading to 0\\% accuracy across test samples. This behavior reflects the model's learned reliance on visual grounding and its sensitivity to missing or meaningless visual inputs. For this reason, we report the result as N/A to indicate that the model does not produce valid outputs under such conditions, rather than interpreting the result as typical task performance. We will clarify this behavior in the revised version.\n\n\n| LMM         | Image  | SciQA-IMG | MMMU-Val |\n| ----------- | ------ | --------- | -------- |\n| LLaVA v1.5  | Origin | 66.8      | 35.3     |\n| LLaVA v1.5  | Black  | 64.1      | 32.4     |\n| DirectLLaVA | Origin | **74.3**      | **36.1**     |\n| DirectLLaVA | Black  | N/A       | N/A      |\n\n\n\n\n**References**\n\n\n[1] Carlucci, et al. Domain generalization by solving jigsaw puzzles. CVPR, 2019.\n\n\n[2] Noroozi, et al. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016.\n\n\n[3] Truong, et al. DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition. CVPR, 2022."}]

Author rebuttal:

Meta-review: This submission received all four positive scores from the reviewers (4xBA). The reviewers especially appreciated simplicity and clear motivation of the proposed approach, together with thorough evaluations and state-of-the art results across a broad set of metrics. The remaining questions were addressed in the rebuttal. The final recommendation is therefore to accept.
Decision: Accept (poster)