DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs
Authors: Lingchen Meng
Jianwei Yang
Rui Tian
Xiyang Dai
Zuxuan Wu
Jianfeng Gao
Yu-Gang Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Multi-modal Models
High-resolution Visual Token
Primary area: machine_vision
TL;DR: In this work, we present DeepStack, a simple yet effective way to connect vision and language in the context of LMMs.

Abstract: Most large multimodal models (LMMs) are implemented by feeding visual tokens as a sequence into the first layer of a large language model (LLM). 
The resulting architecture is simple but significantly increases computation and memory costs, as it has to handle a large number of additional tokens in its input layer. 
This paper presents a new architecture *DeepStack* for LMMs. 
Considering $N$ layers in the language and vision transformer of LMMs, we stack the visual tokens into $N$ groups and feed each group to its aligned transformer layer from bottom to top. Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost. We apply *DeepStack* to both language and vision transformer in LMMs, and 
validate the effectiveness of *DeepStack* LMMs with extensive empirical results. Using the same context length, our DeepStack 7B and 13B parameters surpass their counterparts by 2.7 and 2.9 on average across 9 benchmarks, respectively. Using only one-fifth of the context length, DeepStack rivals closely to the counterparts that use the full context length. These gains are particularly pronounced on high-resolution tasks, *e.g.*, 4.2, 11.0, and 4.0 improvements on TextVQA, DocVQA, and InfoVQA compared to LLaVA-1.5-7B, respectively. We further apply *DeepStack* to vision transformer layers, which brings us a similar amount of improvements, 3.8 on average compared with LLaVA-1.5-7B.

DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs
Lingehen Meng 1:2 Jianwei Yang Rui Tian 1.2
Xiyang Dai :21
Zuxuan Wu Jianfeng Gao
Yu-Gang Jiang"
Shanghai Key Lab of Intell. Info. Processing, School CS, Fudan University
Shanghai Collaborative Innovation Center of Intelligent Visual Computing
Microsoft Corporation
https://deepstacce vl. gi ichbbioo/
VQ.w2
Sequence LMMs
DeepStack LMMs
Stack tokens from bottom top
POPE
GQ4
Transformer Layer
Transformer Layers xL
KG
Transformer Layer
SKED
TestVQA
Downscale
Transformer Layer to
InfoVQA
DocYQA
Transformer Layers XL
Transformer Layer fa
Sequence
knn-770)
Supirne (vo, ud-22880, ets 1cs-2990)
DeepStrckV tok exx Jen
DespStarbell (vic 3ab-3830 eix, (mn-576)
Figure 1:
Left: Conventional large multimodal models (LMMs) string all visual tokens into a
sequence for high- and low-resolution images. Middle: Our DeepStack LMMs stack the tokens into t t)
grid and infuse them into the first and middle transformer layers from bottom to top simply using residual connection. With no architecture modification and context length increasing, our model can handle multiple times more visual tokens as inputs. Right: We apply DeepStack separately to Vicuna-7B (DeepStack-L) and CLIP ViT-L (DeepStack-V), Our models can take 4x more visual tokens, and significantly outperforms the sequence LMM with same context length and rival the one using a much longer context, over wide range of benchmarks.
Abstract
Most large multimodal models (LMMs) are implemented by feeding visual tokens as a sequence into the first layer of a large language model (LLM). The resulting architecture is simple but significantly increases computation and memory costs, as it has to handle a large number of additional tokens in its input layer. This paper presents new architecture DeepStack for LMMs. Considering N layers in the language and vision transformer of LMMs, we stack the visual tokens into N groups and feed each group to its aligned transformer layer from bottom to top, as illustrated in Fig. 1. Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost. We apply DeepStack to both language and vision transformer in LMMs, and validate the effectiveness
DeepStack LMMs with extensive empirical results. Using the same context length, our DeepStack 7B and 13B parameters surpass their counterparts by 2.7 and 2.9 on average across benchmarks, respectively. Using only fifth of the context length, DeepStack rivals closely to the counterparts
Equal contributions; Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that use the full context length. These gains are particularly pronounced on high- resolution tasks, e.g., 4.2, 11.0, and 4.0 improvements on TextVQA. DocVQA, and Info compared to L.aVA-1.5-77, respectively. We further apply DeepStack to vision transformer layers, which brings us a similar amount of improvements, 3.8 on average compared with LLa V..
Introduction
With the tremendous advancements large language models (LLMs) [62, 63, 87, 6, 6, 65, 59), we have witnessed surge of efforts of developing large multimodal models (LMMss) [51, 88). To connect vision and language models for LMMs, a conventional way is transforming images into a number of visual features using pretrained vision encoders (e.g.. CLIP [61]) and flattening them to a sequence of "language tokens" which are then fed into an LLM. With sufficient alignment and instruction tuning, the entire system can demonstrate broad conversational capability for multimodal inputs [51].
To incorporate visual inputs, it usually requires the LMMs to handle a large number of visual tokens as the prefix tokens in addition to the original language prompts. This inevitably introduces a tremendous memory and compute overhead into the L.LMs, which is particularly significant when comes to high-resolution images and multi-frame videos. Several previous works attempt to mitigate this issue by proposing various token compression strategies. A straightforward way is to reduce the number of tokens with spatial grouping [70, 47J. Instead of pooling vision tokens, few work instead to concatenate local tokens along the feature dimension to preserve visual information [11, 48]. Moreover, other works seek more sophisticated token resampling, such as Q-Former [43], Perceiver [4] and Abstractor [8], etc. In MM1 [57], the researchers performed an extensive analysis of these approaches and found no significant discrepancies among them. Despite the huge effort, all these works inherently sacrifice fine-grate ginengrained visual information to reach the trade-off between the compute overhead and the information flow into LLMs, which is arguably problematic for high-resolution images and videos. Most recently, few works [22, 48, 50, 19, 20] proposed multi-crop strategies and string several times more visual tokens to support high-resolution scenarios, while at the cost of substantial overhead.
All current efforts to wire vision with LLMs follow the routine in which visual tokens are always rolled together a Id sequence, and fed into the first layer of LLMs as inputs. In this work, step outside the box and question whether we can find a better strategy to handle the large number of visual tokens regarding both efficacy and efficiency. Instead of examining the LLMs in traditional left-to-right orientation, we adopt a novel bottom-to-top perspective, revealing that they constitute a hierarchical arrangement of transformer layers. Based on this observation, we propose DeepStack, simple, yet novel way of feeding visual tokens into L.M.s. As shown in Fig. 1, instead of putting the long sequence of visual tokens from left to right, where each layer of the stack is connected to one layer in the LLMs by simple residual connection. As a result, with the context length unchanged, we can feed into LLMs several times more visual tokens to handle complex visual inputs. Meanwhile, the combination of per-layer parallel attention and myyer-b-llayer progression can effectively leverage the LLMs' capacity for modeling the dependencies of visual tokens.
restructure the visual tokens into a layered stack,
To examine the effectiveness of our method, we apply it to representative LMMs, LLaVA- [51] and I.LaVA-Next [50]. Extensive empirical results demonstrate the effectiveness of our method. More specifically, with the same setting of I.L our model can achieve significant performance gain across a wide range of benchmarks. In particular, our model brings 4.2, 1 11.0, and 4.0 performance gains on TextVQA, VQA, and InA compared to LLa respectively. To summarize, our main contributions are three-fold:
We propose a simple yet effective DeepStack strategy for connecting vision and language in the context of LMMs. This new strategy introduces architecture change while significantly increasing the number of tokens LLMS can take.
With the DeepStack strategy, we present our new model DeepStack, and compare it with LMMs across wide range of multimodal tasks. Our model demonstrates consistent improvement over the baseline methods, in particular for high-resolution tasks.

---

We further conduct comprehensive ablation studies on different aspects of our proposed method, which provide useful guidance and insights behind the design choices.
Finally, although we only demonstrate the effectiveness of our proposed method in the context of LMMs, we note that this simple strategy could be generalized to any models or tasks built on top of transformer layers. We hope this new design could shield new lights and open up new exploratory directions regarding how to wire vision encoders and LLMs in large multimodal models.
2
Related Works
Large Language Models (LLMs). Recently, natural language processing (NLP) has witnessed significant progress, particularly with the advent of large language models (LLMs) [74, 87, 64, 6]. Building the foundational architecture of Transformers [75], language models [18, 74, 87, 64, 6, 39] have demonstrated strong scalability through the pretraining- seaenng-then-nee ween-finetuning paradigm. Specifically, BERT [18] utilizes the transformer encoder and introduces masked language modeling task to pre- mre-train the model on vast unlabelled data, showing excellent performance after fine-tuning on downstream tasks. Other follow-ups [39, 36] continue along the lines of BERT, constantly refining and optimizing its performance. The T5 [64] series further unifies different NLP tasks within an encoder-decoder architecture, demonstrating effectiveness across dozens of language understanding tasks. Meanwhile, the GPT [62, 63, 4] series employs simple decoder-only transformers to pretrain the language model using unified next-token prediction paradigm. This approach shows remarkable scalability in terms of both model size and data scale. To enhance instruction-follon followiig abilities, InstructGPT [59] and ChatGPT emphasize the importance of instruction tuning and Reinforcement Learning from Human Feedback (RLHF). These models exhibit excellent capabilities in open- domain conversation tasks, ranging from text generation question answering. In response to ChatGPT, recent works [74, 15, 38] have made significant efforts in developing an open-source LLMs community. Building on the success of the LLaMA [74] series foundation model, Alpaca [71], Vicuna [15], and GPT-4-LLM [60] showcase the improvements brought by higher-quality instruction datasets. Other works [24, 27, 1, 86] take different approach, aiming to achieve comparable performance with much smaller set of parameters. The Phi [24, 27, 1] series revisits the importance of the pre-training corpus and achieves success with models containing around 3 billion parameters. In this paper, we develop our model based on Vicuna [15] and Phi-3 [1], aiming to equip the well-trained LLMs with informative visual tokens and a relatively small training effect.
Large Multi-modal Models (LMMs). The success of CLIP [61] and its follow-ups [66, 28, 77] demonstrates the effectiveness of aligning vision and language modalities into a unified semantic space. showcasing promising capabilities in zero-shot classification tasks. More recently. Flamingo [3] and BLIP [44] have utilized visual perceivers [26] to resample visual tokens from image features as inputs for language models through cross-attention. BLIP-2 [42] and Instruct-BL.. [16] further incorporate this mechanism into large language models for tasks such as visual captioning and question-answering Although visual perceivers can translate image features into fixed set of visual tokens, they face constraints related to convergence costs and data requirements. In parallel, LLaVA and its follow-ups [13, 76, 47, 50, 49] achieved success in connecting vision and language using a simple projection module. It greatly simplifies the difficulties of alignment tasks and even achieves better performance with less training effort. However, due to the rigorous input resolution of pre-trained models, these directions meet difficulties on downstream tasks requiring finer-grained visual information, e.g. tasks relevant to OCR and documents. To alleviate this problem, recent works [48, 22, 21, 73, 89] utilize a mixture of experts (MOE) schemes to leverage different pre- trained vision models, typically assembling the visual tokens along the feature dimension. Other attempts [85, 19, 50] split high-resolution images into multi-crop patches and merge them into a longer sequence, which significantly increases the training and evaluation cost. In this work, we conduct experiments on the projector- based connection framework and revisit the connection scheme that utilizes projected visual tokens for the input layer of LLMs. We find that the early layers of LLMs can also well process visual token inputs. Besides that, we propose a DeepStack scheme to stack finer-grained visual tokens to the early layers of L.Mss, enhancing visual capabilities without introducing extra input tokens.
3

---

High Res. Image
High Res. Image
LLM
Connector
LLM Block
VIT Block
Vision Encoder
Connedto dlor
Patch Embed
VT Black
VIT Block
LLM Block
ViT Block
Low Res. Image
Low Res. Image
Vis. lokens Txl. tokens
DeepStack-L
DeepStack-V
Figure 2: Architecture of DeepStack. The main innovation
the DeepStack strategy infuses visual tokens into different layers. Left: DeepStack for LLMs. Given an input image, we feed the tokens extracted from the low resolution version the input layer of LLM. Considering the 2D nature of images, the neighbors from the high-resolution version and reorganize them into DeepStack, which are then fed to the consequent layers LLMs. Right: DtepStack ViTs, We apply similar sampling strategy but feed the visual tokens into the VIT layers of vision encoder.
extra
3 DeepStack
DeepStack is a versatile strategy that provides finer- grainer visual information without increasing the visual context length for LMMs. It achieves this by dividing image feature extraction into streams: a global-view stream that captures global information, and a high-resolution stream that enhances the global information by stacking dilated high-resolution image features across different layers of the I.MMs. This dual-stream approach offers LMMs detailed visual features while maintaining efficiency. By leveraging this simple yet effective method, we build DeepStack, which significantly improves the ability of LMMs to process and comprehend fine-grained visual details. We illustrate DeepStack in Fig. and propose pseudo-code implementation in Algorithm.
3.1 Preliminary: Large Multimodal Model
Large Language Models (LLMss. LLMs [2, 11, 70, 74] are typically pre-trained on huge amount of unlabeled text corpus using a transformer decoder-only architecture. The primary pre-training task next-token prediction driving their learning process. Formally, the learning objective can be formulated as:
N Σ log Po (11))
(1)
where P represents the large language model and the trainable parameters of the model, with the training objective to maximize the probability of 2++1 as the next token, given the previous tokens
Ti:t .i..... .......2.. Ft.
Language Multi-modal Models (LMMs). LMMs extend pre-trained LLMs to generate responses conditioned on input images. This achieved by using visual tokens as a prefix:
N
Σ
log Po (1 P1a.X)
where X E Iglxe represents the sequence of visual tokens [43, 51, 4J, with being the squence length and the hidden dimension of the LLM.
Image Tokenization. Previous works [45, 43, 51] widely explored how to encode input images into visual tokens. The tokenization schemes usually leverage vision-language pre-trained image encoder F. egg. CLIP [61], to extract image features fY from an input image I. Then, the image features are converted into visual tokens using connection module M as follows:
X X= f F*(1

---

Algorithm 1: DeepStack PyTorch pseudocode.
# Hot Input enbeddinga for LGM (Original inputa for traditional LM00; # vis_pos: the location visual tokens;
X, Xatack; Driginal visual tokena, Extra high-resolution visual taken 1iat;
# letares ni Index of starting layer, and layer interval for stacking,
daf forward(Ho, Xstack, Starts ", wis_pos):
Hu
for (idz, Tremeformerayeer) enunerate layeera):
# DeapStack:
idw >- heart (ide Lecare)% 0: H[jvi__oo|| += X"tack[(ide [start//n]]
# Original Transforner: TranstornerLaaarr(H
The connection module .M can take various forms, mainly divided into projection modules [51, 49] and perceiver resamplers [4, 43]. In the former, M is implemented as either a single-layer linear projection [51] or a multi-layer MLP [49], directly projecting dense image features into the hidden space of the LLM. In the latter, M utilizes a cross-attention mechanism with a set of fixed-length learnable queries to extract image features, similar to the approach in [7]. They transform dense image features into sparse image queries, which are then used as input tokens for the language model. However, the resamplers- baaede methods easily struggle with hallucinations on spatial reasoning tasks [17]. In this paper, we mainly focus on the projection-based connection module for its efficiency and effectiveness.
3.2 DeepStack for Improved Image Tokenization
Now that we obtain the visual tokens for LMMs using projection-based connection module, the following challenge is how to provide informative visual tokens while keeping the multi-modal processing effective.
Scaling Visual Tokens. Based on the projection-based connection module, many follow-up attempts to increase the visual capability by introducing multiple image crops [50, 73] for scaling up the resolution involving multiple vision encoders to serve mixture of visual experts [89, 73, 21]. For these approaches. the visual tokens from different image crops or vision encoders are concatenated together along the axis of the sequence or the dimension before projection.
DeepStack Strategy. order to incorporate ane-grained image information while maintaining efficiency, we enhance the input visual tokens X by stacking high-resolution visual tokens into different LLM decoder layers. In practice, we first upsample the input image according to its aspect ratio and simultaneously tokenize it obtain high-resolution visual tokens. To prepare the tokens for hierarchy stacking, we split the righuresolution visual tokens into different token sets sttnck with spatial dilation [80, 14]. This sampling approach ensures that the visual tokens Xstack' have the same length as the global visual tokens X. Additionally, token Xstack' corresponds to the nearest
neighbor of X in spatial.
staack
ststkkk xsttcc2
Xstack5
Sampling2D (M(F* (Ihires)))
As shown in Fig. 2, given an LLM of L decoder layers, the LLM is first split into different blocks. Specifically, DeepStack split the early layers of LLM P into a set of deepstack blocks B {pV', pvv ...., p!" } for stacking visual tokens, and the later layers into plain block pl. for original prefix sequential modeling. We denote that each deepstack block pV ends at the v -th layer of P. while the plain block P ends at the last layer. We use H' to represent the hidden states of visual tokens after the i-th transformer decoder layer, with H being the visual hidden states after the final decoder layer. Formally, the output of each block can be formulated as follows:
v =pv (X) + staakk
H V2 pV²
(HV))
+ xatack2
(5)
H² = pl pL(HH (H (H²*)
5

---

Specifically, we divide the layers into equally sized deepstack blocks, with the block length of by default.
DeepStack for Vision Transformers (ViTs), Our DeepStack can be also applied to ViTs for better feature extraction and image tokenization as illustrated in Fig. (DeepStuck- V). In contrast LMM, and the first several ViT encoder layers for
we use the patch embedding layers PatchFmmeddie tokenization and the reset ViT encoder layers for DeepStack. Formally, we replace the F and in Eq. (4) with the Patch Embedding Layers and the first several encoder layers, and utilize the rest of encoders layers as P in Eq. (5). Please refer to Sec. 4.3 for more details.
Comparison with Other Visual Token Enhancement Strategies. To provide a deeper understanding of the DeepStack mechanism, we compare our strategy with previous visual token enhancement strategies by examining the hidden states of visual tokens after the final LLM decoder layer, denoted as H Previous methods can be broadly categorized into two approaches: Sequence Concatenation and Dimension Concatenation.
As for the former, visual tokens from the entire image and local crops are concatenated sequentially, significantly increasing the overall sequence length the computation cost. The LLM decoder processes these concatenated visual tokens as longer visual prefix, directly modeling the extended sequence.
H P (SeeCCat(X) x=tack))
As for the latter, visual tokens are concatenated along the feature dimension, keeping the sequence length constant. When using a projection module as the connection module, the enhanced visual tokens can be viewed as the sum of features from two individual projection modules.
H P (MM(DimCat(ff, phires])) ~ ((M + +M² M (((riiies)
In our DeepStack, we employ a unique approach where enhancement occurs from bottom to top layer by layer. The processing of H in DeepStack unfolds in two phases. In the early layers of the decoder, the layers function similarly to an encoder, recurrently enhancing the input visual tokens by adding high-resolution visual tokens residually; In the later layers, the decoder performs plain sequence modeling as usual. This dual-phase processing fully leverages the LLM's capabilities by combining both encoding and sequence modeling. By integrating high-resolution visual information at multiple layers, DtacStack effectively enhances visual token representation without increasing visual context length, demonstrating its superiority
previous methods.
Deep layers for LLM sequence modeling
H
PL
DV"
(...(pr))
X
Xstack!) + Xstackk)... + Xstack
(8)
Early layers for visual tokens encoding
4 Experiments
4.1
Implementation Details
We mainly follow the training recipe of Llava [51], of which the training pipeline consists of two stages, iee pre-training (PT) stage and supervised-finetunin
(SFT) stage. We utilize pre-trained CLIP-large-336 [61] as our default image encoder. To obtain high-resolution feature maps, we split the high-resolution image into patches to comply with the resolution requirement and mosaic the image feature together as whole-image features.
Pre-training dataset. We utilize LCS-558k [51] as pre-training data for both experiments based on L.L and LLAVV-Nexx, which contain 558k samples from LAAONN [66], CC [9] and SBU [84], captioned by BLIP [45].
Fine-tuning datasets. We utilize LLA L.aVA-mixed-665k [51] as instruction-followiin data for both experiments based on LLa However, the SFT dataset used in Llava-Next is not publicly

---

Gourral VQA VQA GQA
Tesz-orieted VQA
JAM benelmarkss
Method
LLM
Eff
Vis. Tol.
Car.
PT
SPT
Text
Doc
Info
SEED
POPE
MM
MM
Res.
Lon.
VQA!
VQA!
VQA
(all)
(2ll)
ML
Vet
BLIP-2(43)
Vicuna 13B
224
32
32
129M
41.0
41.0
42.5
46.4
85.3
InstruciBLIP [16]
Vicusa-7B
32
53.4
Intr [16]
Vicuma-13BB
224
32
32
129M
1.2M
49.5
50.7
78.9
Shikra [12]
Vicura-138
224
600K
5.5M
77.4"
IDEPICS-9B [37]
LLaMA-7B
224
353M
1M
50.9
38.4
IDEFICS-SD S0B [37]
LLMMA-65B
224
35.M
1M
60.0
45.2
Owes-
Qwea- 7B
256
50M
78.8*
56.1
Owen-VL VL-Chat [5]
Qwen-7B
148
2.56
256
1.1B
50M
78.2*
57.50
61.5
58.2
VILA [47]
Llamm22-76
335
576
576
50M
1M
79.9
62.3*
64.4
61.1
85.5
34,9
VILA [47]
Lama2-13B
336
.576
576
5UM
1M
80188
63.3*
66.6
62.8
84.3
58.8
ELaVA-1:5
Vicuna-78
336
558K
603K
78-57
62.0"
28.1
25,8
58.6
85.9
35.3
30.5
LL.VA-1.55 [49]
Vicuna-133
576
576
558K
661K
80.0"
28.
35.4
LLaYA-Next [50]
Vicuna-BB
672
2880
2880
558K
765K
81.89
64.2"
64.9
74.5*
37.1"
64.7
36.5
35.1
44.1
LLaVA- (50)
Vicuma 7B
672
2880
2680
558K
765K
82.8"
65.4"
66.9
77.5*
44.5*
65.6
86.2
35.9
49.1
DeepSuck-V
Vicuna-78
672
2880
576
558K
665K
%0..4
64.1"
63.5
41.0
30.0
62.3
87.6
14.9
33.0
DeepStik-V
558K
$1.1
31,1
DeepStnckL.
Vicuna-7B 7B
672
2830
576
558K
665K
79.59
63.1"
62.4
39.1
29.8
60.6
86.7
35.7
29,9
DeepStackLL
Vicuma 13B
672
2550
576
558K
665K
80.9*
61.2*
64.6
41.5
33.0
63.5
87.7
35.2
35,9
DeypStack---HD
Vicuna-7B 7B
1344
14400
2580
558K
748K
82.0*
65. 24
66.7
78.8*
41.22
63.6
86.5
35.6
57.5
DeepSack-L-HDi
Skuna-138
2880
55KK
45.24
33.4
39.3
Table 1: Comparison with other LMMs on benchmarks. Eff. Rex. indicates the effective image resolution
taken by each method. Vis. Tok. indicates the number of visual tokens used for LLMs (not only for the input layers); Cxt. Len. indicates the input visual context length of LLMs. Previous methods feed the visual tokens the input embeddings, thus the Vis. Tok. Cxt. Len. all the time. For comparison with I.LVA--Nex since 765K instruction tuning data is not available, our model is fine-tuned on our 748K data. t indicates that our model is fine-tune tuned from LLaVA VA-Next. * The training images of the datasets are observed during training. denotes we validation sets, We unfreeze the vision encoder in DeepStack-V V and DeepStack-L--HID
report the performance while freezing it in DeepStack-L for fair comparison with previous methods. We fine-tuning the vision encoder can bring further improvement on DeepStack-L. (please refer to Sec. 4.3 and Supp.)
available, we thus combine an SFT dataset of 748K samples following the guidance [50]. In contrast, we do not involve the user images uploaded to their website.
Training configuration. We train our model with only the projection model tuned in the PT stage. In SFT stage, we unfreeze LLM. For Experiments on DeepStack-V V and DeepStack-HD. we tune the image encoder with learning rate of le-6 following [50]. Otherwise, we freeze our vision encoder for a fair comparison. We use 16x V100 for experiments with Phi-3 [1] and 8x H100 for experiments with Vicuna [15]. Please refer to supplementary material for more detailed training hyper- parameerra.etrrs..
4.2 Quantitive Results
We evaluate DeepStack range of benchmarks, encompassing both academic task-oriented evalua- tions and recent large multi-modal language model (LMM) benchmarks. Specifically, we focus on text-oriented datasets, including ChartVQA [54], Voc [56], InfoVQA [55]. MultiDocVQA [72), TextVQA [69], to demonstrate effectiveness in high-resolution scenarios. Additionally, we perform zero-shot evaluations of DeepStack on commonly used video understanding benchmarks to assess its performance on finer-grained tasks.
General VQA and LMM benchmarks. We assess DeepStack two classic general VQA bench- marks, VQAv2 [23] and GQA [25], as well as five recent LMM benchmarks: SEED [40], POPE [46], MMMU [83], and MM-Vet [81]. As presented in Tab. 1, DeepStack outperforms its direct baseline model, LLaVA, on both VQAv2 and GQA, showcasing state-of-ther
performance in traditional VQA tasks. Furthermore, DeepStack consistently surpasses other methods on the recent LMM benchmarks. DeepStack achieves comparable performance on MM-V Vet on the experiments based on LLaVA-1.5. However, due we lack of fancy instruction- folloowing data used in -.AVAAmmix-7755 our experiments with VAeVe-Next lag behind the LLaVA-Next. Notably, the significant performance boost on the POPE benchmark suggests that our DeepStack strategy effectively alleviates visual hallucination by providing rich and detailed visual information for visual understanding.
Text-Oriented benchmarks. To further validate the effectiveness of DeepStack, we evaluate on more text-oriented benchmarks, including ChartQA [54], DocVQA [56], InfoVQA [55]. Multi- DocVQA [72], and TextVQA [69]. These benchmarks contain high-resolution images and typically require the model to answer questions based on fine-graines visual inputs. As shown in Tab. 2,
7

---

Method
LLM
Vis.
Cxt.
PT
SFT
Churt
Doc VQx*
Info
MultDoo VQA*
Text
QAT
VQA
VQA
LLaVA-L.5 [49]
Vicuna-7B
576
576
558K
665K
18.2
28.1
25.8
16.7/72
58.2*
LLaVA-L55 [49]
Vicuna-138
576
576
558K
665K
18.2
30.3
29.4
18.3/80
61.2*
LLaVA-Nexi [50]
Vicuna-78
2880
2880
558K
765k
54.8
74.4
37.1
44.4/313
61.9
LLaVA [50]
Vicuna 13B
558K
77.5
46.3//32.6
DeepStack-V V
Vicuna-78
2880
576
558K
665K
20.6
41.0
30.0
23.0/
63.5*
DeepStack-V
Vicuna-138
2880
576
558K
665K
20.2
41.7
33.1
23.5/11.2
63.9*
DeepStack-L
Vicuna-78
2880
576
558K
665K
19.2
39.1
29.8
22.2/ 10.5
62.4*
DeepStack-L.
Vicuna 13B
2880
558K
41.5
22.6/
DoepStack-HD
Vicuna-7B
144400
2880
558K
748K
56.3
78.8
41.2
/
66.7
DeepSttcc-IIDi
Vicuna- 13B
14400
2880
558K
748K
64.0
81.0
45.2
49.4 39.1
68.7
Table Results on Text-Oriented benchmarks, where high resolution is essential. denotes we use OCR
tokens for TexIVQA following LLaVA-1.5 [49]. denotes report the performance validation sets.
equipping our model with DeepStack results in consistent gains across all benchmarks. This strongly demonstrates that DeepStack enhances visual token even without increasing sequence length.
Zero-shot performance on Video QA benchmarks. We also conduct zero-shot evaluations on video QA benchmarks, including EgoSchema [52] and Next-QA [78] for multiple-choice VQA, and MSVD-QA [10, 79] and ActivityNet-Q2 [82] for open-ended VQA. Inspired by [33], we sample frames from each video uniformly and mosaic the frames into images to adapt video QA tasks the image domain. Thanks to the higher effective resolution brought by refined visual tokens, DeepStack effectively handles zero-shot video QA tasks even without being fine-tuned on any video data.
Multi-choice VQA
Open-ender VQA
Next-QA
MSVD
ActivityNet Acc.
Method
EgoSchema
Cus.
Des.
Tem.
Acc
Acc.
Score
Score
LLaVA LLaVA-L5--B 1.5.BB
35.4
59.5
68.9
55.5
59.6
75.5
4.0
48.6
3,2
DeepStack-11-78
38.4
61.9
69.4
55.5
61.0
76.0
4.0
49.3
3.1
Table 3: Zero-shot evaluation on Video QA benchmarks. We collate 6 frames uniformly sampled from
each video into x grid and resize the resulting image sauare. Our model clearly outperforms the baseline because more visual information is included with the same context length. We mark the best performance bold.
4.3 Model Inspection
We further conduct sufficient experiments give in-depth inspiration on the mechanism of DeepStack. In this section, we experiment with phi-3 [1] as the language backbone for the training efficiency. We report the performance on 7 benchmarks, including I general VQA (GQA), 2 multi-modal benchmarks (POPE and SEED), and 4 text-oriented VQA (TextVQA, DocVQA, ChartQA and InforVQA). We can evaluate the model performance by comparing the average scores over the benchmarks.
(8) Starting layer to Insert visual tokens
(b) Layer interval for DeepStack
(c) Number layers for DeepStack
Layer Idx
N-Layers
Moan uore
Monn soore
Moun soore
65
48
012 4
16
24
2 Interval
Storting Layer
N-Layurs
Figure 3: Analysis using LLM layers to process visual tokens. (a) We insert the visual tokens into different starting layers and initialize correspondence input embeddings as zero; (b) We fix the first layer to insert global visual tokens and ablation on the interval s for stacking high-resolution tokens; (c) We ablation number of layers for token stacking.
LLMs can well process visual tokens in the early decoder layers. To understand why earlier layers of LLMs are suitable for processing visual tokens, we conducted an experiment the insertion layer for visual tokens. Traditionally, visual tokens are inserted at the input layer, e.g. O-th layer. We progressively insert them deeper, initializing the corresponding input embeddings to
8

---

zero. As shown in Fig. 3 (a), inserting visual tokens before the 8th of 32 decoder layers in Phi-3 results in acceptable performance variations. However, inserting them beyond the midpoint leads to significant performance drop. This confirms that earlier layers efficiently handle initial visual information integration. We also explore the impact of inserting visual tokens at non-consecutive layers. In Fig. 3 (b), we fixed global visual tokens at the input layer and varied the interval between two decoder layers for stacking high-resolution tokens. All stacking settings consistently improved performance. Finally,
explored the number of layers used for stacking high-resolution tokens. As shown in Fig. 3 (c), increasing the layers for stacking consistently enhances overall performance, with the best results achieved using four layers.
StepStack can also boost Vision Transformers (VIT). To further explore the potential of DeepStack for vision transformers, utilize the DeepStack on ViT. Specifically, we use the patch embedding layers and the first N VIT encoder layers to extract visual tokens, including the original tokens and 4x extra high-resolution tokens, and then stack the high-resolution tokens into the next 4 encoder layers, respectively. We need to unfreeze the vision encoder to adapt the pre- pre-trained encoder to our DeepStack. As shown in Tab. 4 and Sec, 4.3, when using the first 16 ViT encoder layers (total 24 layers for our ViT-Large) to extract visual tokens before DeepStack, DeepStack-V surpass the baseline model. And the performance keeps increasing when using more encoder layers before DeepStuck.
StecFFrrme-
51.3
49.1
Baseline
Meanssore
12 16
No N-EncLayerr For DeepStack
Enhance Layers before DeepStack
Enc.
GQA
POPE
SEED
TexVVA
DocVQA
ChariQA
InfoVQA
| AVG
None
Nome
62.5
85.5
63.5
56.7
28.3
None
None
62.4
85.8
64.0
56.1
27.5
15.3
28.3
48.5
DeepStuct-V
PuschEmbed-O Enc. Layers
56.9
80.8
54.9
44.4
13.7
12.3
25.3
41.2
DeepStuct-V
PuschEmbed-4 Enc. Layers
58.7
83.1
57.4
48.2
17.0
13.2
26.1
43.4
DtepStuckV
8 Fac. Layers
60.4
84.2
59.7
DrepStuct-V
PuschEmbedr-1 Enc. Layers
61.8
85.5
62.1
55.5
29.3
16.0
26.2
48.1
DeepStnct-V
PaschEmbed-16 6 Enc. Layers
62.9
86.3
63.9
59.1
36.9
18.2
29.3
50.9
2ucchFibed-220 Fnc. Layers
60.1
Table 4: Ablations the number of VIT encoder layers for DeepStack-V. V.
Better spatial consistency leads to better performance. Different sampling strategies may lead to different results. In Tab. 5, we compare our default strategy with two other variants for organizing the visual tokens. As shown in Fig. 2d Grid use each of the local crop as a layer and 1d Sequence simply flatten the visual tokens to oneddimensional and then re- shape them into a layer stack. Accordingly, keeping the spatial coherence, i.e. 2d Spatial, as in our default setting could achieve the best result.
2d Spatior
Sequential
2d Givl
Figure 4: Visualization of three pling methods for DeepStack.
Consistent
Sampling
POPE
SEED
TeaVQA
DoeVQA
ChartA
InfoVQA
| AVG
None
None
62.5
85.5
63.5
31.7
28.3
49.1
X
2d Spanial
62.2
85.1
62.3
58.1
35.1
16.4
30.1
49.9
2d Spatial
63.0
86.4
62.9
58.8
38.7
30.8
51.1
2d Grid
86.2
61.2
33.2
49.9
Id Sequential
61.9
33.1
49.3
Table 5: Ablations on image consistency and sampling method. We apply the Resize transformation both the original image and the high-resolution image for consistency. original image and Pad-Resize on the nigh-resolution image. 2d Spatial refers to sampling based on spatial locations, such as using 4-neighbor method. 2d Grid means the visual tokens are divided into 2d grids, with each grid stacked per layer. Id Sequential indicates that the high resolutoot visual tokens are first flattened into sequence and then uni uniformly sampled for each layer. Please refer to Fig. 4 for better understanding.
inconsistency, we use Resize on the
DeepStack boosts LMMs from high-resolution tokens, not residual connections. We experiment to assess the impact of high-resolution images and residual connections in DeepStack by stacking original visual tokens into different layers. As shown in Tab. 6, stacking repeated original tokens (dummy tokens) does not improve performance. This indicates that the performance boost in DeepStack comes from the high-resolution tokens, not from the residual connections.

---

Tok. Enhance
Stack Tok.
GQA
POPE
SEED
TexVQQ
DocVQA
ChartQA
InfoVQA
AVG
None
None
62.5
85.5
63.5
56.7
31.7
15.8
28.3
49.1
DeepStock
Dummy
62.2
85.3
31.2
28.8
49.1
DrepSiock
Hi-Res
63.0
86.4
62.9
58.8
38.7
17.2
30.8
51.1
Table 6: Ablations on high-resolution visual tokens for stacking, Dummy refers to repeating the original visual tokens for token stacking; Hi-Res is our default setting that uses high-rcsolution visual tokens for stacking.
DeepStack achieves better trade-off between performance and effectiveness. We compare Deep- Stack with other token enhancement strategies, including dimension wise concatenation, sequence- wise with high-resolution visual tokens, and string both global visual and high-resolution tokens. As shown in Tab. 7, although string-base methods can bring significant improvement some benchmarks, they increase the number of tokens at the same time, which will increase the training and inference cost. Meanwhile, DeepStack achieves the best trade-off between performance and effectiveness without introducing extra visual tokens.
Tok. Enhanze
N Tok.
Eit. Tok.
GQA
POPE
SEED
TeaVQQ
DocVQA
ChartQA
InfoVQA
AVG
None
576
576
62,5
85.5
63.5
56.7
31.7
15.8
28.3
49,1
Dimension Concat
576
2880
59.5
86.3
62.9
56.4
35.9
16.4
28.5
49.4
String
2304
61.8
50.7
Global+ Hi Res String
2880
2880
62.3
86.4
62.6
54.7
43.3
16.7
31.2
$1.0
Stapktack
576
2880
63.0
86.4
62.9
58.8
38.7
17.2
30.8
51.1
Table 7: Ablations on different token enhancement strategies. Dimension Concat refers to concatenate X and Xstack via the channel of features hidden space; Hi-Res String and xstack and [X, xstack] via sequence, respectively.
String refers to string
Stack unleashes the power after fine-tuning the image encoder. We further experiment with how DeepSttac compared coporated with fine-tuning backbones. As shown in Tab. 4, DeepStack achieves the best performance when fine-tuning the backbone. It is worth noticing that when fine- tuning the backbone without DeepStack, the improvement is limited. After combining backbone fietuning with DeepStack, the performance significantly increases among different benchmarks. is because of the deep interaction between visual tokens and the LLM decoder.
Tok. Enhance
Ft Enc.
GQA
POPE
SEED
TeaVQA
DocVQA
CharQA
InloVQA
AVG
Nome
62.5
63.5
56.7
15.8
28.3
None
62.4
64.0
56.1
15.3
28.3
48.5
DeepStock
63.0
86.4
62.9
58.8
38.7
17.2
30.8
51.1
DeepStack
63.1
86.8
63.9
61.1
41.2
18.9
31.5
52.4
Table 8: Ablations on fine-tuning vision encoder. DeepStack achieves best performance after fine-tuning vision encoder.
5 Conclusion
In this work, we had presented DeepStack, a simple yet effective way to connect vision and language in the context of LMMs. Unlike previous works that always string (compressed) visual tokens into a sequence, we alternatively introduced a new perspective on transformer decoder layers in LLMs, and proposed a DeepStack strategy feed different visual tokens into different layers of LLMs. This strategy significantly mitigates the efficiency overhead introduced by visual tokens and makes it possible to convey more visual information to L.Mss. As result, our DeepStack demonstrated consistent improvements over two baseline models across a wide range of benchmarks. The benefits are particularly significant on tasks that inherently require more tokens, such high-resolution image understanding. We hope this new DeepStack strategy could open up new ideas on how to connect vision and language for faster and better multimodal models in the regime of LMMs.
Limitation and Future Works. DeepStack simply inserts the visual tokens into middle LLMs layers via residual connection in heuristic manner. Though it already exhibits promising results, may find a more powerful way to infuse the visual information, e.g., through gated function or layer-wise positional embeddings. Meanwhile, how systematically decide the starting layer and number of layers also deserves more study. We leave these as promising directions.
A Acknowledgement This project was supported by NSFC under Grant No. 62102092.
10

---

References
[1] M. Abdin, S. Jacobs, A. A. Awan. Aneja, A. Awadallah, H. Awadalla. N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al. Phi-3 technical report: highly capable language modcl locally on your phone. arXiv preprint Xiv:24404 14219, 2024.
[2] Achiam, S. Adler, S. Agarwal, L. Ahmad, Akkaya, L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint XXXX:23333008772 2023.
[3] J.-B, Alayrac, Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. NewrIPS, 2022.
[4] J.-B. Alayrae, Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, 7. Gong, S. Samangooei, M. Monteiro, J. T.. Menick, S. Borgeaud, A. Brock, A. Nematzadeh. S. Sharifzadeh. M. Binikowki, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: visual language model for few-shot learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Adv. Neural Inform. Process. Syst.. volume 35, pages 23716-23736 23736. Curran Associates, Inc., 2022.
[5] Bai, S. Bai, Yang. S. Wang. S. Tan, Wang. Lin, C. Zhou, and J. Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint (tt.:2308.19666 2023.
[6] T. Brown, B. Mann, Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
[7] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and 7agoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[8] J. Cha, W. Kang, J. Mun, and B. Roh. Honeybee: Locality-enhanced projector for multimodal Ilm, 2024,
[9] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In CVPR, 2021.
[10] D. Chen and W. B. Dolan, Collecting highly parallel data for paraphrase evaluation. ACL, 2011.
[11] Chen, D. Zhu, X. Shen, X. Li, Z. Liu, Zhang, Krishnamoorthi, V. Chandra, Xiong, and M. Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, 2023.
[12] K. Chen, Z. Zhang, W. Zeng, R. Zhang, Zhu, and R. Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.1511 2023.
[13] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311. 12799, 2023.
[14] L.-C. Chen, G. Papandreou. I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI. 2017.
[15] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang. J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[16] W. Dai, Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P.N. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tningg NewrIPS, 2024.
[17] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. 7hao, W. Wang, B. A. Li, P. Fung, and S. C. H. Hoi. Instructhlip: Towards general-purpose vision-language models with instruction tuming. ArXiv, abs/2305.06500. 2023.
[18] J. Devlin, M.-W. Chang, K. Lec, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
[19] X. Dong, P. Zhang, Y. Zang. Y. Cao, B. Wang, L. Ouyang. S. Zhang, H. Duan. W. Zhang, Y. Li, et al. pioneering large language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.006 06510, 2024.
[20] X. Dong, Zhang, Zang. Y. Cao, B. Wang. L. Ouyang. S. Zhang. Duan, W. Zhang, Y. Li, H. Yan.
Y. Gao, Z. Chen, X. Zhang, W. Li. J. Li, W. Wang. K. Chen, C. He, X. Zhang, J. Dai, Y. Qiao, D. Lin, and
J. Wang. socnull-xcomposr22-4iic A pioneering large dision-languag model handling resolutions from
336 pixels to 4k hd, 2024.

---

[21] X. Fan, T. Ji, C. Jiang, S. Li. S. Jin, S. Song, J. Wang. B. Hong, L. Chen. G. Zheng, et al. Mousi: -elall vision-language models. arXiv preprint 1xXiv:2401.122211 2024.
[22] P. Gao, R. Zhang, C. Liu, I.. Qiu, S. Huang. W. Lin, S. Zhao, S. Geng, Z. Lin, P. Jin, et Sphinx-x: Scaling data and parameters for family of multi-modal large language models. arXiv preprint arXiv:2402.05933. 2024.
[23] Y. Goyal, T. Khot, D. Summers-Stay D. Batra, and D. Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
[24] S. Gunasekar, Y. Zhang, Aneja, C. C. T. Mendes, A. Del Giomo, S. Gopi, M. Javaheripi, P. Kauffmann,
G. de Rosa, O. Saarikivi, et al. Texthooks are all you need, arXiv preprint ArXiv:2306. 11644, 2023.
[25] D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
[26] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
[27] M. Javaheripi, S. Bubeck, M. Abdin, J. Aneja, Bubeck, C. C. T. Mendes, W. Chen, A. Del Giorno, R. Eldan, S. Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog. 2023.
[28] C. Jia, Y. Yang. Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, 2021.
[29] K. Kafle, B. Price, S. Cohen, and C. Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018.
[30] S. Kazemzadeh, V. Ordonez, M. Matten, and Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014.
[31] A. Kembhavi, M. Salvato, E. Kolve, M. Seo. H. Hajishirzi, and A. Farhadi. A diagram is worth a dozen images. In ECCV, 2016.
[32] G. Kim, T. Hong, M. Yim, J. Nam. J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park. Ocr-free document understanding transformer. In ECCV, 2022.
[33] W. Kim, C. Choi, W. Lee, and W. Rhee. An image grid can be worth a video: Zero-shot video question answering using vlm. arXiv preprint 24t.::403..84066 2024.
[34] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC, 2017.
[35] LAION-4V. https:/hhuggingfacc... co//aatsset///iio/ggdd--yoooo
2023.
[36] Z. Lan, M. Chen, S. Goodman, K. Gimpel. P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised lcarning of language representations. arXiv preprint arXiv: 1942, 2019.
[37] H. Laurençon, D. Strien, S. Bekman, L. Tronchon, L. Saulnier, T. Wang. S. Karamcheti, A. Singh, G. Pistilli, Y. Jernite, et al. Introducing ideties: An open reproduction of of visual language model, 2023. URL http://huggingfuce co/blog//defics. Accessed, 2023.
[38] T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilie, D. Hesslow, R. Castagné, A. S. Luccioni, Yvon, M. Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023.
[39] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and I.. Zettlemoyer. Bart: Denoising sequence-to- to-squeee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910.1..91003141111 2019.
[40] B. Li, R. Wang, G. Wang. Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint 231iv:2307.16122 2023.
[41] B. Li, P. Zhang, K. Zhang, F. Pu, X. Du, Y. Dong, H. Liu, Y. Zhang, G. Zhang, C. Li, and Z. Liu. Lmms-eval: Accelerating the development of large multimoal models, 2024.
[42] J. Li, D. Li. S. Savarese, and S. Hoi. Blip-2: 2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023.
12

---

[43] Li, D. Li. S. Savarese, and S. C. H. Hoi. Blip-2: Bootstrapping language-inage pre-training with frozen image encoders and large language models. In ICML, 2023.
[44] Li, D. Li. C. Xiong, and S. Hoi. Blip: Bootstrapping language-iman pre- me-ttrinnig for unitied vision- language understanding and generation. In ICML, 2022.
[45] J. Li, D. Li, C. Xiong, and S. C. H. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[46] Y. Li, Y. Du, K. Zhou, J. Wang. W. X. Zhao, and J.-R. Wen. Evaluating object hallucination in large vision-language models. arXiv preprint 2335 10355, 2023.
[47] J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023.
[48] Z. Lin, C. Liu, R. Zhang, P. Gao, L. Qiu, H. Xiao, H. Qiu, C. Lin, W. Shao, K. Chen, et al. Sphinx: The joint mixing of weights. tasks, and visual embeddings for multi-modal large language models. arXiv preprint arX1v:2311.0755,
2023. 2023.
[49] H. Liu, Li, Y. Li, and Y.J. Lee. Improved baselines with visual instruction tuning. ArXiv, abs/2310.03744, 2023.
[50] H. Liu, C. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Lee. Llava-next Improved reasoning, ocr, and world knowledge, 2024.
[51] H. Liu, C. Li, Q. Wu, and Y. Loe. Visual instruction tuning. In NeurlPS, 2023.
[52] K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. NewrlPS, 2024.
[53] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
[54] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning, arXiv preprint arXiv:2203. 10244, 2022.
[55] Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar. Infographicvga. In WACV, 2022.
[56] Mathew, Karatzas, and C, Jawahar. Docyqa: A dataset for vqa on document images. In WACV, 2021.
[57] B. McKinzie, Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng, F. Weers, A. Belyi, H. Zhang, K. Singh, D. Kang, A. Jain, H. Hè, M. Schwarzer, T. Guntor, X. Kong, A. Zhang, J. Wang, C. Wang, N. Du, T. Lei, S. Wiseman, G. Yin, M. Lee, Z. Wang, R. Pang, P. Grasch, A. Toshey, and Y. Yang. Mmi: Methods, analysis & insights from multimodal Ilm pre-training, 2024.
[58] A. Mishra, S. Shekhar, A. K. Singh, and Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[59] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
[60] B. Peng, C. P. He, M. Galley, and J. Gao. Instruction tuning with gpt arXiv preprint 23X20:3304.03277. 2023.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[62] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, al. Improving language understanding by generative pre-training. 2018
[63] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[64] C. Raffel, N. Shazeer, A. Roberts, K. Lee. S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unitied text-to-text transformer. Journal of machine learning research. 2020.
13

---

[65] T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Iff', D. Hesslow, R. Castagn'e e. A. S. Luccioni, F. Yvon. and M. G. et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 2022.
[66] C. Schuhmann. R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, Coombes, Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, I.. Schmidt, R. Kaczmarezyk, and J. Jitsev. LAION-5h: An open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
[67] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, 2022.
[68] ShareGPT. htttps://shaaeee
com/ 2023.
[69] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In CVPR, 2019.
[70] Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang. Generative multimodal models are in-context learners, 2024.
[71] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpaca:
A strong, replicable instruction-folloiin model. Stanford Center for Research on Foundation Models. http:://rrfm. stanford. edw/2023/03/1 3/lpaca. html, 2023.
[72] R. Tito, D. Karatzas, and E. Valveny. Hierarchical multimodal transformers for multipage docvqa. Puttern Recognition, 2023.
[73] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1ms.. arXiv preprint arXiv:2401.00200 2024.
[74] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, F. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:230.133971 13971, 2023.
[75] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. NeurlPS, 2017.
[76] J. Wang, L. Meng, Z. Weng, B. He, Z. Wu, and Y.G. Jian. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.0755 2023.
[77] Z. Wu, Z. Weng, W. Peng, X. Yang, A. Li, L. S. Davis, and Y.-G. Jiang. Building open-vocabulary video clip model with better architectures, optimization and data. TPAMI, 2024.
[78] J. Xiao, X. Shang. A. Yao, and T.-S. Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In CVPR, 2021.
[79] D. Xu, Zhao, Xiao, Wu, H. Zhang. X. He, and Zhuang. Video question answering via gradually refined attention over appearance and motion. In MM, 2017.
[80] F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv: 1,00122,, 2015.
[81] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02443 2023.
[82] Z. Yu, D. Xu, J. Yu, T. Yu. Z. Zhao, Y. Zhuang, and D. Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
[83] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint ar Xrx: I. 16502, 2023.
[84] K. Yun, J. Honorio, D. Chattopadhyay, L. Berg, and D. Samaras. Two-person interaction detection using body-pose features and multiple instance learning. In CVPR, 2012.
[85] P. Zhang, X. D. B. Wang, Y. Cao. C. Xu, L. Ouyang, Z. Zhao, S. Ding. S. Zhang, H. Duan, H. Yan. et al. Internim-xcom xcomposer: A vision- language large model for advanced text- image comprehension and composition. arXiv preprint arXiv:2309. 15112, 2023.
14

---

[86] P. Zhang. G. Zeng. T. Wang, and W. Lu. Tmnyllaa: An open-source small language model. arXiv preprint urXiv:2401.03385, 2024.
[87] S. Zhang. S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00 01068, 2022.
[88] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, abs/2304. 10592, 2023.
[89] 7. Zong, B. Ma, D. Shen, G. Song, H. Shao, D. Jiang, H. Li, and Y. Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint urXiv:2404.33446 2024.
15

---

A Training Details
A.1 Custom Supervised Finetuning Dataset
We follow LLaVA-Next [50] to combine a custom data mixture containing 748K SFT data shown
Tab. 9. Following [51, 50], our 748K training data mixture contains (1) LLM instruction fol-
lowing data, e.g. ShareGPT [68]; (2) GPT4/GPT4V generated data, e.g. LLaVA-instruct [51],
SPT4eGPT4V [13], LAION-GPT4V [35]; (3) srademie-task-orientee data, e.g. VQAv2 [23]. GQA [25], etc.
Dataset
Size
Task Prompt
ShareGPT [68]
40K
LLaVA-instruct [5]]
158K
ShareGPTaY [13]
39K
LAION-GPT4V [35]
11K
VQAv2 [23]
83K
GQA [25]
72K
OKVQA [53]
9K
OCRVQA [58]
80K
CharQA [54]
7K
"Answer the question using a single word or phrase."
DVQA [29]
DocVQA [56]
10K
AI2D [31]
SyatDoo-EN [32]
20K
A-OKYQA [67]
"Answer option's letter the given directly."
RefCOCO [30]
48K
"Provide shoet description this region."
[G4
86K
"Provide bounding box coordinate the region this sentence describes"
Table 9: Data combination of our 748K SFT data.
A.2 Detailed Training Configuration
We list the detailed training for evaluation several benchmarks.
as follows. For evaluation, we utilize LLMs-Eval [41]
Hypter-param
PT
DeepStack SFT
DeepStack-V V SFT
DeepStack-HD SFT
global batch size
128
128
128
Ir
1c-3
2c-5
20-5
2c-5
buckhome
freeze
freeze
2e-6
2e-6
Ir scheslule
casine decny
cosine decay
cosine decay
cosine decay
Ir warmup ratio
0.03
0.03
0.03
0.03
epoch
optimizer
Adam
Adamn
Adamn
Adam
Table 10: Training hyper-parameters.
B More Experiments
B.1
Improved DeepStack-L with Fintuning Vision Encoder
As shown in Tab. 11, after finetuning the vision encoder, our DeepStack-L achieves further improve- ment. This further demonstrates the effectiveness and the potential of our DeepStack strategy.
Generol VQA" GQA
Text-oriented VQA
LMM benchmarks
Method
LLM
Edf
Vis
Cat. Lem.
PT
SPT
Text VQA!
Doc
Info
SEED
POPE
MM
MM
Res.
Tok:
VQA!
VQAT
(all)
(all)
MUT
Ver
LLaVA-1.5 (49)
Vicuna-7B
58.6
DeepStnck-L.
Vicuna-TD
672
2880
576
558K
665K
79.5e
63.1
62.4
39.1
29.8
60.6
86.7
35.7
29.9
DeepStack-L+ L+
Vicuna-7B
672
2880
576
S58K
665K
81.1*
63.9*
61.5
39.3
30.]
63.3
86.7
37.1
29.8
LLAVA-15 [49)
Vicuno-13R
336
$76
558K
665K
80.0
63.39
61.3
30.3
28.4
61.6
85.9
34.8
38.4
DeepSStcc-L
Vicuna-138
672
2880
576
558K
80.9
64.2
64.6
33.0
63.5
87.7
35.2
35.9
DeepStuck-1A4 x
Vicuna-138
13
672
288D
576
558K
665
82.1*
65.14
65.2
43.1
34.0
64.
86.6
34.7
36.2
Table 11: Improved DeepStack-L with fintuning vision encoder. indicates that the vision encoder fine-tuned.
16

---

WILCOME WORDISTON
Q: What does say in the bettom right corner?
Q: How many stars can you see on the white board?
Q: What candy bar down there bottom?
Q: What brand is the mobile phone?
Congratuuasiioss
Honey DeepStack: Henshey's
Samsung
DeepStack: Postoode'
DeepStack:
DeepStack:
Q: Please describe in as much detail as possible what each person the picture is daing.
LLaVA-1.5:
group people gathered around cow grassy weman standing next to the oow, possibly petting it or lending to it. Another woman standing nearby, possibly abserving the scend or walling for her turn interact with the cow. There other people others are further away. few individuals are weering liee, which suggests a more formal or professional setting. In the scene, there are also a couple of chairs and a dining table, indicating this gathering might be
image,
background,
standing
meeting. handbag attendees.
placed ground, possibly belanging
DeepStack:
In the image, there group people gathered around a brown and white cow, woman is standing next to the cow, haiding its haltor, while a man is standing nearty, possibly observing scene. Other people also present in the erea. some standing closer the cow
away. near the center and two others closer to the right side. A truck can be seen the background. possibly indicaling that the event taking place at fair or similar outdoor venue.
VQAv2
Q: Please describe the image as detailed as possible
LLaVA-1.5:
GQA
The Image diaplays graph with central circle that represents VODA (Voics over QAM) system. The circle is divided into four quadrants, each representing a different aspect the system. The quadrants are labeled with the acronyms VOQA, POP GADA, and TEX. The graph includes
K5,i
62.1
which related the VOGA system as well. These smaller circles are posilioned at various distances from the main circle, indicating the different levels of the system.
soattered
66
28
DeepStack:
25.
TA
The image features circular diagram various divided into different sections, each with own set of labe's and numbers. The central section of the diagram labaled "VDA2" while the surrounding sections are laboled "Pope," "GOA, "Seed," "InfoVAA, "DocVOA, diagram, with some located in the central section and others scattered throughout the different sections. The numbors are likely used to represent data or information related the labels and sections the diagram.
diagram
There
InfoVQA
DocVQA
Figure 5: Visualization. Both LLaVA-1.5 and DeepStack use S76 visual context length for a fair comparison. Top: We mark the area corresponding to each question with red circle. DeepStack can well answer the questions which need high-resolution and fine-grained understanding. Bottom: DeepStack demonstrates accurate visual understanding detailed visual captioning.
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We properly claim our contribution in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper has discussed the limitations.
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper is not include thcoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided all the details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We release it to the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /Cod/Soodemiiicceee Sooddeeumisssoooooo
/ //nips... /tippscc/ppuit wttp:cccpppppppcccccceo/ococccca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The training and test details have been reported in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The cost of training required to report error hars is excessively high,
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the computation resources in this paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tt:r//nn...pee cc/ cctppblic//eeeeeeccccecccccceeee httpo/lic//hiiieeeee ettiichuuellnnes
Answer: [Yes]
Justification: The paper conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided the broader impacts in our paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper did not release data or models for the main contribution.
Guidelines:
The answer NA means that the paper poses such risks.
22

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper follows the CC-BY 4.0 license in experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper did not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing and research with human subjects in paper.
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing experiments in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "sXOU4Iwd62", "reviewer": "Reviewer_QJM9", "summary": "This paper proposes a method to add more visual information to a MM-LLM without increasing the number of tokens processed by the model. The idea is simple, just add visual tokens to the existing hidden representation between each layer of the transformer. The approach is evaluated on many tasks and shows good results.", "questions": "None", "limitations": "na", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written, the approach would be reproducible from the given descriptions. The idea is novel and simple and effective. The experiments are thorough.", "weaknesses": "Overall the paper is well done. The experiments are thorough, the idea is well explained, and the method is reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We are pleased that you find our work novel and simple! If you have any additional questions, please feel free to add detailed comments; we are happy to answer them and sincerely hope to address your concerns if any.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aHbxIwd4t7", "reviewer": "Reviewer_SQxC", "summary": "This paper proposes a new visual token organization method. Specifically, it proposes to stack visual tokens instead of the commonly used stringing. Experiments show that the proposed StackFormer can improve performance on TextVQA, DocVQA, and InfoVQA.", "questions": "- In table 1, what do † and * mean?\n\n- Could you explain why StackFormer cannot improve LLaVA-NeXt on MLLM benchmarks?", "limitations": "the authors have discussed the limitations in the paper. StackFormer cannot significantly improve larger model LLaVA-NeXt on MLLM benchmarks. It may be another limitation.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The proposed method is novel, different from the commomly-used stringing method. \n- Experiments show that the proposed method StackFormer can significantly improve performance on some datasets, especially the traditional VQA datasets.", "weaknesses": "- There are many typos in the paper. The authors need to improve their writing and polish the paper. Like Line 18 \"StackFormeruses\" --> \"StackFormer uses\"; Line 134: Multi-modal Language Models (LLMs). Sometimes it uses LMMs and sometimes MLLMs. Both are ok, but please use only one in the same paper.\n\n- There is no significant improvement for LLaVA-Next on MLLM benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## q1: Typo\nThank you for pointing it out, we have polished the representation and will update it in the next version.\n\n## q2: Improvement for LLaVA-Next on MLMM benchmarks \nThank you for your valuable comments! We suspect that two main factors contribute to the results: (1) the image resolution, and (2) the quality of the SFT dataset (please refer to L227-L228 in our main paper).\n\n\nLLaVA-Next scales up the effective resolution from $336 \\times 336$ to {$672\\times 672$, $336\\times 1344$, $1344 \\times 336$} by using dynamic multi-image crops and stringing visual tokens [1]. This approach allows the model to handle most resolutions in MLLM benchmarks as mentioned. Quantitatively, we analyzed the average width and height of images from each benchmark. As shown in the table below, the average resolution of multi-modal benchmarks is significantly smaller than that of text-oriented benchmarks. Consequently, the multi-image crops already suffice to cover most of those benchmarks in LLaVA-Next. Further upsampling the image to a higher resolution using bilinear interpolation does not introduce additional information. As a result, the proposed layer stacking of visual tokens extracted from upsampled images does not necessarily show gains on them as significant as the originally high-resolution text-oriented benchmarks.\n\n|Benchmark | average resolution (height $\\times$ width) |\n| ----| ---- | \nTextVQA | 819.4 $\\times$ 952.3\nDocVQA | 2098.6  $\\times$ 1782.8\nInfoVQA | 3002.4  $\\times$ 1161.5\nSEED | 899.2  $\\times$ 1090.0\nPOPE | 478.8  $\\times$ 584.7\nMMMU | 488.2  $\\times$ 723.0\nMMVet | 797.4  $\\times$ 1059.5\n\nAs discussed in our main submission, the SFT-768K data used in LLaVA-Next includes a portion of private data collected from the LLaVA demo. This part of the data is used to generate responses with GPT-4V to obtain high-quality instruction-following data. In contrast, our SFT-748K dataset lacks this component, leading to limitations on the GPT4-evaluated benchmark, *i.e.* MM-Vet.\n\nFurthermore, we observe that MLLM benchmarks emphasize evaluating reasoning, hallucination, *etc.*, while text-oriented benchmarks prioritize fine-grained perception and understanding (please refer to Fig. 3 in **our rebuttal PDF**). Additionally, previous works [2] indicate that MLLM benchmarks, such as MMMU, benefit more from LLM capabilities rather than visual representations. We suggest these differences may contribute to that StackFormer for LLaVA-Next does not obtain significant gains compared to text-oriented benchmarks.\n\n\n## q3: In table 1, what do † and * mean?\nIn Table 1, † indicates that the model is continuously fine-tuned based on the LLaVA-Next checkpoint. Since the training codes and the 765K instruction tuning data used in LLaVA-Next are not publicly available, we mix a 748K SFT dataset based on information from the LLaVA-Next blog. However, we do not have access to the private data used in LLaVA-Next or the exact combination ratios among the datasets. Consequently, we fine-tuned our model using our 748K dataset and LLaVA-Next to ensure a fair comparison.\n\nAdditionally, * denotes that the training images from the downstream benchmark are observed during the SFT stage. Besides, we mark MMMU* to indicate that we are reporting the validation results for MMMU. We will clarify these details in the next version of our paper.\n\n[1] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge   \n[2] LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MtJd2vZUEr", "reviewer": "Reviewer_REjR", "summary": "The authors present a new architecture StackFormer - an effective and simple way to infuse fine-grained visual tokens from CLIP vision transformer to the early layers of  LLaVA-1.5 and LLaVA-Next language models, without increasing the sequence length of visual tokens for LLMs. It doesn't require architecture change while significantly increases the number of tokens LLMs can take, so it improves accuracy especially for high-resolution images and videos.", "questions": "Can you show in more detail in the Figures 1 and 2:\n- how exactly you split the high-resolution image into patches?\n- and how exactly you split the high-resolution visual tokens into different token sets with spatial dilation?\n\nCan you provide numerical indicators (flops, params, latency, memory consumption) that Stackformer achieves the best trade-off between performance and effectiveness compared to other VLMs?", "limitations": "Limitation and Future Work: \nThe paper presents limited options for processing high-resolution images by naively splitting the image into many patches and applying VIT-CLIP to them separately. While there are many approaches to processing high-resolution images using transformers: ViTDet [1], SwinV2 [2], Patch-Fusion [3], ... or simply naively resize the ViT-CLIP model to the required image resolution, using 2D interpolation of the pre-trained position embeddings [4].\n\n\n1. Exploring Plain Vision Transformer Backbones for Object Detection, Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He, 2022\n2. Swin Transformer V2: Scaling Up Capacity and Resolution, Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo, 2021\n3. PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation, Zhenyu Li, Shariq Farooq Bhat, Peter Wonka, 2023\n4. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, 2020", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The authors propose an effective and simple way to increase the resolution of the visual part of VLM to increase the accuracy of the VLM, splitting the images into patches, separately applying ViT-CLIP to them, collecting (mosaic) the feature maps into a single high-resolution feature map as whole-image feature, and using residual connections to embed this feature map into the LLM.\nStackFormer outperforms its baseline model LlaVA on both VQAv2, GQA, POPE  as well as on Text-Oriented and Zero-shot Video QA benchmarks.\nStackFormer achieves best performance when the backbone is fine-tuned, while when fine-tuning a backbone without a StackFormer, the improvement is limited.", "weaknesses": "In Fig. 1 and 2 are missing details about the implementation of StackFormer:\n- how exactly you split the high-resolution image into patches\n- and how exactly you split the high-resolution visual tokens into different token sets with spatial dilation\n\nWhile you write that StackFormer achieves the best trade-off between performance and effectiveness without introducing extra visual tokens, specific indicators of overhead costs when using Stackformer are not provided. Flops, parameters, latency, memory consumption, accuracy are not provided in a single table to compare Stackformer with other VLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We provide a detailed architecture figure in our **rebuttal pdf**. We recommend referring to Figure. 1 and Figure. 2 in our **rebuttal pdf** for a better understanding of the high-resolution token processing.\n\n## q1-2: How to split high-resolution images into patch crops; and how to split high-resolution visual tokens into different sets.\n\nPlease refer to **global author rebuttal**, and Fig.1 and Fig.2 in our **rebuttal pdf**.\n\n## q3: Comparison of flops, params, latency\n\nThanks for the great suggestion! We compare our StackFormer with LLaVA-1.5 and another representative work VILA [1]. As shown in the table, our model improves the baseline by introducing a ~3% increase in FLOPs without significantly increasing the iteration time during training. Additionally, our StackFormer does not require extra data or the intermediate stage training used in VILA while achieving comparable performance, which is of great training efficiency. We will include the analysis and comparison in our revision.\n\n**Comparison on 7B models** \n\n|Method | Training cost | Training Data (PT+SFT) | iter time |TFLOPs | Params |SEED | TextVQA | DocVQA | ChartQA | InfoVQA |\n| ----| ---- | ---- |---- |---- |---- | ---- |---- |---- |---- |---- | \nLLaVA-1.5 | ~0.1k GPU hours | 558K+665K | 19.0 | 26.7 | 7.06B | 58.6 | 58.2* | 28.1 | 18.2 | 25.8\nVILA-1.5  $\\star$ |~5k GPU hours | 50M+1M| 17.2 | 26.7 |  7.06B | 61.1 | 64.4* | 44.5* | 25.7 | 32.5 | \nLLaVA-1.5+StackFormer | ~0.1k GPU hours| 558K+665K  | 17.9 | 27.5 | 7.06B |  60.6 | 62.4* | 39.1 | 19.2 | 29.8 | \nLLaVA-1.5+StackFormer $\\star$ | ~0.1k GPU hours | 558K+665K | 18.3| 27.5 |7.06B | 63.3 | 64.5* | 39.3 | 21.0 | 30.1 | \n\n*For iter time, we average the time cost of 100 training iterations under the same 8xA100 machine*.  \n*$\\star$ indicates that the vision encoder is fine-tuned during the SFT stage*.  \n*\\* indicates that the images of the benchmark training set are observed during SFT stage*.  \n\n[1] VILA: On Pre-training for Visual Language Models\n\n## q4: Other approaches to processing high-resolution images\n\nThanks for sharing different works in the related domain. In this work, our primary goal is to efficiently and effectively process image tokens extracted from high-resolution images for **large multimodal models (LMMs)**, rather than the techniques on how to extract high-resolution features. Therefore, we employ a simple and widely used approach, *i.e.*, adaptive multi-crop, for high-resolution feature extraction to ensure a fair comparison with LlaVA and other works.\n\nTo further comprehend our findings, we follow the suggestion and conduct experiments using other approaches to process high-resolution images, as follows. \n\n1. [*Our default setting*] Multi-crop sub-images + StackFormer-LLM (freeze ViT).\n\n2. Multi-crop sub-images + StackFormer-LLM (unfreeze ViT): We unfreeze the parameters of ViT during the SFT stage, building on the approach in #1.\n\n3. Multi-crop sub-images + StackFormer-ViT (unfreeze ViT): We use the first 20 layers of ViT to extract multi-crop high-resolution features and the last 4 layers to stack these features. The stacked features are then fed into a projection module as visual token inputs for LMMs.\n\n4. whole image, 2D interpolation on ViT + StackFormer-LLM (unfreeze ViT): We directly interpolate the positional embedding of ViT to process high-resolution images. StackFormer is then used to stack the extracted features. ViT needs to be unfrozen because the input resolution differs from the pre-training vision encoder.\n\n5. whole image, ViT-det + StackFormer-LLM (unfreeze ViT): We apply techniques from ViTDet, incorporating global and local attention in transformer blocks and discarding the CLS token. StackFormer is used for token stacking in the LLM.\n\nWe conduct ablation experiments using Phi-3 (3B) as our language model. As shown in the table below, our StackFormer on LLM/ViT (#1, #2, #3) achieves significant performance gains compared to the baseline model. However, other methods, such as 2D positional interpolation and ViT-Det, result in performance drops due to inconsistencies between image pre-training and the LMM SFT stage. These techniques commonly used on detection often require longer training schedules (*e.g.*, 100 epochs on COCO for ViTDet), which may be unsuitable for the one-epoch SFT pipeline used in LMMs.\n\n#### Ablations on processing high-resolution images with phi-3 (3b LLM)\n| # |Method | AVG | SEED | TextVQA | DocVQA | ChartQA | InfoVQA\n|---- | ---- |---- |---- |---- | ---- |---- |---- |\n|0 |Baseline | 38.1 | 62.6 | 55.7 | 28.1 | 15.8 | 28.3\n|1 |Multi-crop, Stackformer-LLM | 40.9 | 62.9 | 58.4 | 37.8 | 16.6 | 28.9 | \n|2 |Multi-crop, Stackformer-LLM (unfreeze ViT) | 42.5 | 63.9 | 60.3 | 39.0 | 19.1 | 30.1 \n|3 |Multi-crop, Stackformer-ViT | 42.0 | 64.0 | 60.1 | 38.4 | 17.1 | 30.6\n|4 |Whole image, 2d pos-interpolation, Stackformer-LLM| 36.1 |62.7 | 53.4 | 25.1 | 14.8 | 24.5 \n|5 |Whole image, ViT-Det style, Stackformer-LLM  | 34.1 | 60.3 | 48.6 | 22.6 | 14.2 | 25.0 \n\nFurthermore, we conduct additional experiments to utilize StackFormer on LLM/ViT (#1, #2, #3) with 7/13B LLM models, which further demonstrate the effectiveness of our StackFormer. Please refer to part 3 (Additional Results on vicuna-1.5) in **global author rebuttal**.\n\nWe believe that other techniques for extracting high-resolution features with vision transformers, such as patch-fusion [1] and swin-attention [2], may have the potential to obtain higher performance than the commonly-used naive multi-crop approach. We leave these explorations in the future work.\n\n[1] PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation.  \n[2] Swin Transformer V2: Scaling Up Capacity and Resolution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First of all, we sincerely appreciate all your valuable comments and suggestions.

In this work, we proposed a new model called Stackformer to handle the long sequence of visual tokens in large multimodal models (LMMs). Unlike all previous works that string visual tokens into a long sequence, we instead stack them layer by layer and feed them into the large language models using a simple residual connection. As demonstrated by our extensive empirical studies, the proposed method significantly improves the LMMs' ability to handle high-resolution images while keeping the context length unchanged for LLMs.

We are encouraged that all reviewers gave positive ratings to our work and recognize the merits of our work including:

* **Novelty**: "The idea is novel" (R-QJM9), "The proposed method is novel, different from the commonly-used stringing method" (R-SQxC).
* **Simplicity**: "simple and effective" (QJM9), "The authors propose an effective and simple way" (REjR).
* **Effectiveness**: "simple and effective" (QJM9), "can significantly improve performance on some datasets" (SQxC), "The authors propose an effective and simple way" (REjR).

We carefully read the comments by all reviewers and attempted to provide comprehensive responses accordingly. Please find the rebuttal below each official review. We hope the responses could answer the questions raised by reviewers and address any concerns about our work.

Thanks again to all reviewers for the time and effort!
 
  
***
   

## Author Global Responses

We provide a detailed architecture figure in our **rebuttal pdf**. We recommend referring to Figure. 1 and Figure. 2 in our **rebuttal pdf** for a better understanding of the high-resolution token processing.

### 1. How to split high-resolution images into patch crops
We split high-resolution images into sub-image crops using dynamic high-resolution techniques [1,2,3]. Given the grid pinpoints template $A$={($a^{h}_i, a^{w}_i$)} and the input resolution of vision encoder $\mathrm{r}$, the resolution candidates are calculated as $R$={($r \cdot a^{h}_i, r \cdot a^{w}_i$)}. For each image $I$, we first select the best-fitting resolution from $R$ to resize $I$. The resized image is then split into fixed-size sub-images of $r \times r$ accordingly.

For example, we use CLIP-ViT-336 as the image encoder and define the grid pinpoints templates $A$ = {(1,2), (1,3), (1,4), (2,1), (2,2), (3,1), (4,1)} to ensure at most 4 sets of high-resolution tokens for stacking by default. This setup allows us to obtain the resolution candidates $R$ = {(336, 336), (336, 672), (336, 1008), (336, 1344), (672, 336), (672, 672), (1008, 336), (1344, 336)}. For each input image, we first select the best-fitting resolution. The image is then resized and split into $336 \times 336$ sub-images accordingly. Thus, the high-resolution sub-images can be directly encoded with the image encoder.

Please refer to Fig. 1 in our **rebuttal pdf** for a clearer understanding.

[1] SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models.   
[2] SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models.   
[3] LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.   

### 2.  How to split high-resolution visual tokens into different sets
Given the high-resolution visual tokens $\mathbf{X}^{hires} \in \mathbb{R}^{(a^h \cdot h) \times (a^w \cdot w) \times C}$, we apply 2D sampling to divide the tokens into different sets. Here, $h$ and $w$ represent the image feature shape of the vision encoder, while $a^h$ and $a^w$ denote the aspect ratio of the resized image. The 2D sampled token sets for StackFormer are calculated as $\mathbf{X}^{stack} = \{\mathbf{X}^{hires}[i::a^{h}, j::a^{w}]\}$, where $i$ = {$0, 1, \ldots, a^{h} - 1$} and $j$ = {$0, 1, \ldots, a^{w} - 1 $}.

Please refer to Fig.2 in our **rebuttal pdf** for a better understanding.


### 3. Additional Results on vicuna-1.5 (7b/13b LLM) 
Inspired by Reviewer REjR, we further conduct experiments on 7b and 13b LMMs. For StackFormer-ViT, we use the first 20 layers of ViT to extract multi-crop high-resolution features and the last 4 layers to stack these features. The stacked features are then fed into a projection module as visual token inputs for LMMs, without creasing the visual context length.

The results demonstrate that our StackFormer can be effectively utilized for both LLM and ViT. 

*$\star$ indicates that the vision encoder is fine-tuned during SFT stage.*
| # | Method |  VQAv2 | GQA | TextVQA | DocVQA | InfoVQA | SEED | POPE | MMMU | MM-Vet |
|---- |---- | ---- |---- |---- |---- |---- |---- |---- |---- |---- |
|0 | Llava-1.5-7b |  78.5 | 62.0 | 58.2 | 28.1 | 25.8 | 58.6 | 85.9 | 35.3 | 30.5
|1 | Stackformer-LLM-7b   | 79.5 | 63.1 | 62.4 | 39.1 | 29.8 | 60.6 | 86.7 | 35.7 | 29.9
|2 | Stackformer-LLM-7b $\star$   | 81.1 | 63.9 | 64.5 | 39.3 | 30.1  | 63.3 | 86.7 | 37.1 | 29.8
|3 | Stackformer-ViT-7b $\star$  | 80.4 | 64.1 | 63.5 | 41.0 | 30.0 | 62.3 | 87.6 | 34.9 | 33.0

| #| Method | VQAv2 | GQA | TextVQA | DocVQA | InfoVQA | SEED | POPE | MMMU | MM-Vet |
---- |---- | ---- |---- |---- |---- |---- |---- |---- |---- |---- |
0 | Llava-1.5-13b  | 80.0 | 63.3 | 61.3 | 30.3 | 28.4 | 61.6 | 85.9 | 35.3 | 30.5
1 |Stackformer-LLM-13b   | 80.9 | 64.2 | 64.6 | 41.5 | 33.0 | 63.5 | 87.7 | 35.2 | 35.9 
2 |Stackformer-LLM-13b $\star$  | 82.1 | 65.1 | 65.2 | 43.1 | 34.0 | 64.4 | 86.6 | 34.7 | 36.2
3 |Stackformer-ViT-13b $\star$  | 81.1 | 64.2 | 63.9 | 41.7 | 33.1 | 63.0 | 86.6 | 34.7 | 31.1

Meta-review: This paper has received consistent feedback from three reviewers, all of whom are in favor of acceptance. The paper proposes a simple yet effective layer stacking strategy to significantly increase the number of tokens in large language models (LLMs). The experiments are comprehensive, and the results are solid. Therefore,  the AC has decided to accept this paper.
Decision: Accept (poster)