DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs

Authors: lingchen  meng
jianwei  yang
rui  tian
xiyang  dai
zuxuan  wu
jianfeng  gao
yu-gang  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 02:11:37

Abstract: Most large multimodal models (LMMs) are implemented by feeding visual tokens as a sequence into the first layer of a large language model (LLM). 
The resulting architecture is simple but significantly increases computation and memory costs, as it has to handle a large number of additional tokens in its input layer. 
This paper presents a new architecture *DeepStack* for LMMs. 
Considering $N$ layers in the language and vision transformer of LMMs, we stack the visual tokens into $N$ groups and feed each group to its aligned transformer layer from bottom to top. Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost. We apply *DeepStack* to both language and vision transformer in LMMs, and 
validate the effectiveness of *DeepStack* LMMs with extensive empirical results. Using the same context length, our DeepStack 7B and 13B parameters surpass their counterparts by 2.7 and 2.9 on average across 9 benchmarks, respectively. Using only one-fifth of the context length, DeepStack rivals closely to the counterparts that use the full context length. These gains are particularly pronounced on high-resolution tasks, *e.g.*, 4.2, 11.0, and 4.0 improvements on TextVQA, DocVQA, and InfoVQA compared to LLaVA-1.5-7B, respectively. We further apply *DeepStack* to vision transformer layers, which brings us a similar amount of improvements, 3.8 on average compared with LLaVA-1.5-7B.

Main paper: 
 introduction: 
 With the tremendous advancements in large language models (LLMs) [62, 63, 87, 6, 6, 65, 59] , we have witnessed a surge of efforts of developing large multimodal models (LMMs) [51, 88] . To connect vision and language models for LMMs, a conventional way is transforming images into a number of visual features using pretrained vision encoders (e.g., CLIP [61] ), and flattening them to a sequence of "language tokens" which are then fed into an LLM. With sufficient alignment and instruction tuning, the entire system can demonstrate a broad conversational capability for multimodal inputs [51] .To incorporate visual inputs, it usually requires the LMMs to handle a large number of visual tokens as the prefix tokens in addition to the original language prompts. This inevitably introduces a tremendous memory and compute overhead into the LLMs, which is particularly significant when it comes to high-resolution images and multi-frame videos. Several previous works attempt to mitigate this issue by proposing various token compression strategies. A straightforward way is to reduce the number of tokens with spatial grouping [70, 47] . Instead of pooling vision tokens, a few work instead to concatenate local tokens along the feature dimension to preserve visual information [11, 48] . Moreover, other works seek more sophisticated token resampling, such as Q-Former [43] , Perceiver [4] and Abstractor [8] , etc. In MM1 [57] , the researchers performed an extensive analysis of these approaches and found no significant discrepancies among them. Despite the huge effort, all these works inherently sacrifice fine-grained visual information to reach the trade-off between the compute overhead and the information flow into LLMs, which is arguably problematic for high-resolution images and videos. Most recently, a few works [22, 48, 50, 19, 20] proposed multi-crop strategies and string several times more visual tokens to support high-resolution scenarios, while at the cost of substantial overhead.All current efforts to wire vision with LLMs follow the routine in which visual tokens are always rolled together as a 1d sequence, and fed into the first layer of LLMs as inputs. In this work, we step outside the box and question whether we can find a better strategy to handle the large number of visual tokens regarding both efficacy and efficiency. Instead of examining the LLMs in a traditional left-to-right orientation, we adopt a novel bottom-to-top perspective, revealing that they constitute a hierarchical arrangement of transformer layers. Based on this observation, we propose DeepStack, a simple, yet novel way of feeding visual tokens into LLMs. As shown in Fig. 1 , instead of putting the long sequence of visual tokens from left to right, we restructure the visual tokens into a layered stack, where each layer of the stack is connected to one layer in the LLMs by simple residual connection. As a result, with the context length unchanged, we can feed into LLMs several times more visual tokens to handle complex visual inputs. Meanwhile, the combination of per-layer parallel attention and layer-by-layer progression can effectively leverage the LLMs' capacity for modeling the dependencies of visual tokens.To examine the effectiveness of our method, we apply it to two representative LMMs, LLaVA-1.5 [51] and LLaVA-Next [50] . Extensive empirical results demonstrate the effectiveness of our method. More specifically, with the same setting of LLaVA-1.5, our model can achieve significant performance gain across a wide range of benchmarks. In particular, our model brings 4.2, 11.0, and 4.0 performance gains on TextVQA, DocVQA, and InfoVQA compared to LLaVA-1.5-7B, respectively. To summarize, our main contributions are three-fold:• We propose a simple yet effective DeepStack strategy for connecting vision and language in the context of LMMs. This new strategy introduces no architecture change while significantly increasing the number of tokens LLMs can take.• With the DeepStack strategy, we present our new model DeepStack, and compare it with LMMs across a wide range of multimodal tasks. Our model demonstrates consistent improvement over the baseline methods, in particular for high-resolution tasks.Large Multi-modal Models (LMMs). The success of CLIP [61] and its follow-ups [66, 28, 77] demonstrates the effectiveness of aligning vision and language modalities into a unified semantic space, showcasing promising capabilities in zero-shot classification tasks. More recently, Flamingo [3] and BLIP [44] have utilized visual perceivers [26] to resample visual tokens from image features as inputs for language models through cross-attention. BLIP-2 [42] and Instruct-BLIP [16] further incorporate this mechanism into large language models for tasks such as visual captioning and question-answering. Although visual perceivers can translate image features into a fixed set of visual tokens, they face constraints related to convergence costs and data requirements. In parallel, LLaVA and its follow-ups [13, 76, 47, 50, 49] achieved success in connecting vision and language using a simple projection module. It greatly simplifies the difficulties of alignment tasks and even achieves better performance with less training effort. However, due to the rigorous input resolution of pre-trained models, these directions meet difficulties on downstream tasks requiring finer-grained visual information, e.g. tasks relevant to OCR and documents. To alleviate this problem, recent works [48, 22, 21, 73, 89] utilize a mixture of experts (MOE) schemes to leverage different pretrained vision models, typically assembling the visual tokens along the feature dimension. Other attempts [85, 19, 50] split high-resolution images into multi-crop patches and merge them into a longer sequence, which significantly increases the training and evaluation cost. In this work, we conduct experiments on the projector-based connection framework and revisit the connection scheme that utilizes projected visual tokens for the input layer of LLMs. We find that the early layers of LLMs can also well process visual token inputs. Besides that, we propose a DeepStack scheme to stack finer-grained visual tokens to the early layers of LLMs, enhancing visual capabilities without introducing extra input tokens.  
deepstack: 
 DeepStack is a versatile strategy that provides finer-grained visual information without increasing the visual context length for LMMs. It achieves this by dividing image feature extraction into two streams: a global-view stream that captures global information, and a high-resolution stream that enhances the global information by stacking dilated high-resolution image features across different layers of the LLMs. This dual-stream approach offers LMMs detailed visual features while maintaining efficiency. By leveraging this simple yet effective method, we build DeepStack, which significantly improves the ability of LMMs to process and comprehend fine-grained visual details. We illustrate DeepStack in Fig. 2 and propose a pseudo-code implementation in Algorithm. 1.Large Language Models (LLMs). LLMs [2, 11, 70, 74] are typically pre-trained on a huge amount of unlabeled text corpus using a transformer decoder-only architecture. The primary pre-training task is next-token prediction driving their learning process. Formally, the learning objective can be formulated as:EQUATIONwhere P represents the large language model and θ is the trainable parameters of the model, with the training objective to maximize the probability of x t+1 as the next token, given the previous tokensx 1:t = x 1 , . . . , x t .Language Multi-modal Models (LMMs). LMMs extend pre-trained LLMs to generate responses conditioned on input images. This is achieved by using visual tokens as a prefix:EQUATIONwhere X ∈ R l×c represents the sequence of visual tokens [43, 51, 4] , with l being the squence length and c the hidden dimension of the LLM.Image Tokenization. Previous works [45, 43, 51] widely explored how to encode input images into visual tokens. The tokenization schemes usually leverage a vision-language pre-trained image encoder F v , e.g. CLIP [61] , to extract image features f v from an input image I. Then, the image features are converted into visual tokens using a connection module M as follows: The connection module M can take various forms, mainly divided into projection modules [51, 49] and perceiver resamplers [4, 43] . In the former, M is implemented as either a single-layer linear projection [51] or a multi-layer MLP [49] , directly projecting dense image features into the hidden space of the LLM. In the latter, M utilizes a cross-attention mechanism with a set of fixed-length learnable queries to extract image features, similar to the approach in [7] . They transform dense image features into sparse image queries, which are then used as input tokens for the language model. However, the resamplers-based methods easily struggle with hallucinations on spatial reasoning tasks [17] . In this paper, we mainly focus on the projection-based connection module for its efficiency and effectiveness.EQUATIONNow that we obtain the visual tokens for LMMs using a projection-based connection module, the following challenge is how to provide informative visual tokens while keeping the multi-modal processing effective.Scaling Visual Tokens. Based on the projection-based connection module, many follow-up attempts to increase the visual capability by introducing multiple image crops [50, 73] for scaling up the resolution or involving multiple vision encoders to serve as a mixture of visual experts [89, 73, 21] . For these approaches, the visual tokens from different image crops or vision encoders are concatenated together along the axis of the sequence or the dimension before projection.DeepStack Strategy. In order to incorporate fine-grained image information while maintaining efficiency, we enhance the input visual tokens X by stacking high-resolution visual tokens into different LLM decoder layers. In practice, we first upsample the input image according to its aspect ratio and simultaneously tokenize it to obtain high-resolution visual tokens. To prepare the tokens for hierarchy stacking, we split the high-resolution visual tokens into different token sets X stack i with spatial dilation [80, 14] . This sampling approach ensures that the visual tokens X stack i have the same length as the global visual tokens X. Additionally, token X stack i corresponds to the nearest neighbor of X in spatial.EQUATIONAs shown in Fig. 2 , given an LLM of L decoder layers, the LLM is first split into different blocks. Specifically, DeepStack split the early layers of LLM P into a set of deepstack blocksB V = {P V 1 , P V 2 , ..., P V n } for stacking visual tokens, and the later layers into a plain block P L for original prefix sequential modeling. We denote that each deepstack block P V i ends at the N V i -th layer of P, while the plain block P L ends at the last layer. We use H i to represent the hidden states of visual tokens after the i-th transformer decoder layer, with H L being the visual hidden states after the final decoder layer. Formally, the output of each block can be formulated as follows:EQUATIONSpecifically, we divide the layers into equally sized deepstack blocks, with the block length of 1 by default.DeepStack for Vision Transformers (ViTs). Our DeepStack can be also applied to ViTs for better feature extraction and image tokenization as illustrated in Fig. 2 (DeepStack-V). In contrast to LMM, we use the patch embedding layers PatchEmbedding and the first several ViT encoder layers for tokenization and the reset ViT encoder layers for DeepStack. Formally, we replace the F and M in Eq. ( 4) with the Patch Embedding Layers and the first several encoder layers, and utilize the rest of encoders layers as P in Eq. ( 5). Please refer to Sec. 4.3 for more details.Comparison with Other Visual Token Enhancement Strategies. To provide a deeper understanding of the DeepStack mechanism, we compare our strategy with previous visual token enhancement strategies by examining the hidden states of visual tokens after the final LLM decoder layer, denoted as H L . Previous methods can be broadly categorized into two approaches: Sequence Concatenation and Dimension Concatenation.As for the former, visual tokens from the entire image and local crops are concatenated sequentially, significantly increasing the overall sequence length the computation cost. The LLM decoder processes these concatenated visual tokens as a longer visual prefix, directly modeling the extended sequence.EQUATIONAs for the latter, visual tokens are concatenated along the feature dimension, keeping the sequence length constant. When using a projection module as the connection module, the enhanced visual tokens can be viewed as the sum of features from two individual projection modules.EQUATIONIn our DeepStack, we employ a unique approach where enhancement occurs from bottom to top layer by layer. The processing of H L in DeepStack unfolds in two phases. In the early layers of the decoder, the layers function similarly to an encoder, recurrently enhancing the input visual tokens by adding high-resolution visual tokens residually; In the later layers, the decoder performs plain sequence modeling as usual. This dual-phase processing fully leverages the LLM's capabilities by combining both encoding and sequence modeling. By integrating high-resolution visual information at multiple layers, DeepStack effectively enhances visual token representation without increasing visual context length, demonstrating its superiority over previous methods.EQUATIONEarly layers for visual tokens encoding 4 Experiments 
implementation details: 
 We mainly follow the training recipe of Llava [51] , of which the training pipeline consists of two stages, i.e. pre-training (PT) stage and supervised-finetuning (SFT) stage. We utilize pre-trained CLIP-large-336 [61] as our default image encoder. To obtain high-resolution feature maps, we split the high-resolution image into patches to comply with the resolution requirement and mosaic the image feature together as whole-image features.Pre-training dataset. We utilize LCS-558k [51] as pre-training data for both experiments based on LLaVA-1.5 and LLaVA-Next, which contain 558k samples from LAION [66] , CC [9] and SBU [84] , captioned by BLIP [45] .Fine-tuning datasets. We utilize LLaVA-mixed-665k [51] as instruction-following data for both experiments based on LLaVA-1.5. However, the SFT dataset used in Llava-Next is not publicly available, we thus combine an SFT dataset of 748K samples following the guidance [50] . In contrast, we do not involve the user images uploaded to their website.Training configuration. We train our model with only the projection model tuned in the PT stage.In SFT stage, we unfreeze LLM. For Experiments on DeepStack-V and DeepStack-HD, we tune the image encoder with a learning rate of 1e-6 following [50] . Otherwise, we freeze our vision encoder for a fair comparison. We use 16× V100 for experiments with Phi-3 [1] and 8× H100 for experiments with Vicuna [15] . Please refer to our supplementary material for more detailed training hyper-parameters.We evaluate DeepStack on a range of benchmarks, encompassing both academic task-oriented evaluations and recent large multi-modal language model (LMM) benchmarks. Specifically, we focus on text-oriented datasets, including ChartVQA [54] , DocVQA [56] , InfoVQA [55] , MultiDocVQA [72] ,TextVQA [69] , to demonstrate effectiveness in high-resolution scenarios. Additionally, we perform zero-shot evaluations of DeepStack on commonly used video understanding benchmarks to assess its performance on finer-grained tasks.General VQA and LMM benchmarks. We assess DeepStack on two classic general VQA benchmarks, VQAv2 [23] and GQA [25] , as well as five recent LMM benchmarks: SEED [40] , POPE [46] , MMMU [83] , and MM-Vet [81] . As presented in Tab. 1, DeepStack outperforms its direct baseline model, LLaVA, on both VQAv2 and GQA, showcasing state-of-the-art performance in traditional VQA tasks. Furthermore, DeepStack consistently surpasses other methods on the recent LMM benchmarks. DeepStack achieves comparable performance on MM-Vet on the experiments based on LLaVA-1.5. However, due to we lack of fancy instruction-following data used in LLaVA-mix-765K, our experiments with LLaVA-Next lag behind the LLaVA-Next. Notably, the significant performance boost on the POPE benchmark suggests that our DeepStack strategy effectively alleviates visual hallucination by providing rich and detailed visual information for visual understanding.Text-Oriented benchmarks. To further validate the effectiveness of DeepStack, we evaluate it on more text-oriented benchmarks, including ChartQA [54] , DocVQA [56] , InfoVQA [55] , Multi-DocVQA [72] , and TextVQA [69] . These benchmarks contain high-resolution images and typically require the model to answer questions based on fine-grained visual inputs. As shown in Tab. equipping our model with DeepStack results in consistent gains across all benchmarks. This strongly demonstrates that DeepStack enhances visual token even without increasing sequence length.Zero-shot performance on Video QA benchmarks. We also conduct zero-shot evaluations on video QA benchmarks, including EgoSchema [52] and Next-QA [78] for multiple-choice VQA, and MSVD-QA [10, 79] and ActivityNet-QA [82] for open-ended VQA. Inspired by [33] , we sample frames from each video uniformly and mosaic the frames into images to adapt video QA tasks to the image domain. Thanks to the higher effective resolution brought by refined visual tokens, DeepStack effectively handles zero-shot video QA tasks even without being fine-tuned on any video data. We further conduct sufficient experiments to give in-depth inspiration on the mechanism of DeepStack.In this section, we experiment with phi-3 [1] as the language backbone for the training efficiency. We report the performance on 7 benchmarks, including 1 general VQA (GQA), 2 multi-modal benchmarks (POPE and SEED), and 4 text-oriented VQA (TextVQA, DocVQA, ChartQA and InforVQA). We can evaluate the model performance by comparing the average scores over the 7 benchmarks. LLMs can well process visual tokens in the early decoder layers. To understand why earlier layers of LLMs are suitable for processing visual tokens, we conducted an experiment on the insertion layer for visual tokens. Traditionally, visual tokens are inserted at the input layer, e.g. 0-th layer. We progressively insert them deeper, initializing the corresponding input embeddings to zero. As shown in Fig. 3 (a), inserting visual tokens before the 8th of 32 decoder layers in Phi-3 results in acceptable performance variations. However, inserting them beyond the midpoint leads to a significant performance drop. This confirms that earlier layers efficiently handle initial visual information integration. We also explore the impact of inserting visual tokens at non-consecutive layers. In Fig. 3 (b), we fixed global visual tokens at the input layer and varied the interval between two decoder layers for stacking high-resolution tokens. All stacking settings consistently improved performance. Finally, we explored the number of layers used for stacking high-resolution tokens. As shown in Fig. 3 (c), increasing the layers for stacking consistently enhances overall performance, with the best results achieved using four layers.DeepStack can also boost Vision Transformers (ViT). To further explore the potential of DeepStack for vision transformers, we utilize the DeepStack on ViT. Specifically, we use the patch embedding layers and the first N ViT encoder layers to extract visual tokens, including the original tokens and 4× extra high-resolution tokens, and then stack the high-resolution tokens into the next 4 encoder layers, respectively. We need to unfreeze the vision encoder to adapt the pre-trained encoder to our DeepStack. As shown in Tab. 4 and Sec. 4.3, when using the first 16 ViT encoder layers (total 24 layers for our ViT-Large) to extract visual tokens before DeepStack, DeepStack-V surpass the baseline model. And the performance keeps increasing when using more encoder layers before DeepStack. Better spatial consistency leads to better performance. Different sampling strategies may lead to different results. In Tab. 5, we compare our default strategy with two other variants for organizing the visual tokens. As shown in Fig. 4 , 2d Grid use each of the local crop as a layer and 1d Sequence simply flatten the visual tokens to one-dimensional and then reshape them into a layer stack. Accordingly, keeping the spatial coherence, i.e. 2d Spatial, as in our default setting could achieve the best result.  
conclusion: 
 In this work, we had presented DeepStack, a simple yet effective way to connect vision and language in the context of LMMs. Unlike previous works that always string (compressed) visual tokens into a sequence, we alternatively introduced a new perspective on transformer decoder layers in LLMs, and proposed a DeepStack strategy to feed different visual tokens into different layers of LLMs. This strategy significantly mitigates the efficiency overhead introduced by visual tokens and makes it possible to convey more visual information to LLMs. As a result, our DeepStack demonstrated consistent improvements over two baseline models across a wide range of benchmarks. The benefits are particularly significant on tasks that inherently require more tokens, such as high-resolution image understanding. We hope this new DeepStack strategy could open up new ideas on how to connect vision and language for faster and better multimodal models in the regime of LMMs.Limitation and Future Works. DeepStack simply inserts the visual tokens into middle LLMs layers via a residual connection in a heuristic manner. Though it already exhibits promising results, we may find a more powerful way to infuse the visual information, e.g., through gated function or layer-wise positional embeddings. Meanwhile, how to systematically decide the starting layer and number of layers also deserves more study. We leave these as promising directions.Acknowledgement This project was supported by NSFC under Grant No. 62102092.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: We release it to the public.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The training and test details have been reported in this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The cost of training required to report error bars is excessively high.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provided the computation resources in this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: This paper follows the CC-BY 4.0 license in experiments.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper did not introduce new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing and research with human subjects in our paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
stack tokens from bottom to top: 
 Figure 1 : Left: Conventional large multimodal models (LMMs) string all visual tokens into a sequence for high-and low-resolution images. Middle: Our DeepStack LMMs stack the tokens into a grid and infuse them into the first and middle transformer layers from bottom to top (■ ↑ ■ ↑ ■ ↑ ) simply using a residual connection. With no architecture modification and context length increasing, our model can handle multiple times more visual tokens as inputs. Right: We apply DeepStack separately to Vicuna-7B (DeepStack-L) and CLIP ViT-L (DeepStack-V). Our models can take 4× more visual tokens, and significantly outperforms the sequence LMM with same context length and rival the one using a much longer context, over a wide range of benchmarks.Most large multimodal models (LMMs) are implemented by feeding visual tokens as a sequence into the first layer of a large language model (LLM). The resulting architecture is simple but significantly increases computation and memory costs, as it has to handle a large number of additional tokens in its input layer. This paper presents a new architecture DeepStack for LMMs. Considering N layers in the language and vision transformer of LMMs, we stack the visual tokens into N groups and feed each group to its aligned transformer layer from bottom to top, as illustrated in Fig. 1 . Surprisingly, this simple method greatly enhances the power of LMMs to model interactions among visual tokens across layers but with minimal additional cost. We apply DeepStack to both language and vision transformer in LMMs, and validate the effectiveness of DeepStack LMMs with extensive empirical results. Using the same context length, our DeepStack 7B and 13B parameters surpass their counterparts by 2.7 and 2.9 on average across 9 benchmarks, respectively. Using only one-fifth of the context length, DeepStack rivals closely to the counterparts 
 
 Appendix: A.1 Custom Supervised Finetuning DatasetWe follow LLaVA-Next [50] to combine a custom data mixture containing 748K SFT data shown in Tab. 9. Following [51, 50] , our 748K training data mixture contains (1) LLM instruction following data, e.g. ShareGPT [68] ; (2) GPT4/GPT4V generated data, e.g. LLaVA-instruct [51] , ShareGPT4V [13] , LAION-GPT4V [35] ; (3) academic-task-oriented data, e.g. VQAv2 [23] , GQA [25] , etc.ShareGPT [68] 40KLLaVA-instruct [51] 158K ShareGPT4V [13] 39K LAION-GPT4V [35] 11KVQAv2 [23] 83K"Answer the question using a single word or phrase."GQA [25] 72K OKVQA [53] 9K OCRVQA [58] 80K ChartQA [54] 7K DVQA [29] 16K DocVQA [56] 10K AI2D [31] 2K SynthDog-EN [32] 20KA-OKVQA [67] 66K "Answer with the option's letter from the given choices directly."RefCOCO [30] 48K "Provide a short description for this region." "Provide the bounding box coordinate of the region this sentence describes" VG [34] 86KTable 9 : Data combination of our 748K SFT data.We list the detailed training hyper-parameters as follows. For evaluation, we utilize LLMs-Eval [41] for evaluation on several benchmarks. In the image, there is a group of people gathered around a brown and white cow. A woman is standing next to the cow, holding its halter, while a man is standing nearby, possibly observing the scene. Other people are also present in the area, some standing closer to the cow and others further away. There are a few chairs scattered around the scene, with one near the center and two others closer to the right side. A truck can be seen in the background, possibly indicating that the event is taking place at a fair or similar outdoor venue.In the image, there is a group of people gathered around a cow in a grassy field. A woman is standing next to the cow, possibly petting it or tending to it. Another woman is standing nearby, possibly observing the scene or waiting for her turn to interact with the cow.There are several other people in the background, some of whom are standing closer to the cow, while others are further away. A few individuals are wearing ties, which suggests a more formal or professional setting. In the scene, there are also a couple of chairs and a dining table, indicating that this gathering might be a social event or a meeting. A handbag can be seen placed on the ground, possibly belonging to one of the attendees. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Top: We mark the area corresponding to each question with a red circle. DeepStack can well answer the questions which need high-resolution and fine-grained understanding. Bottom: DeepStack demonstrates a more accurate visual understanding in detailed visual captioning. 


Decision: This paper has received consistent feedback from three reviewers, all of whom are in favor of acceptance. The paper proposes a simple yet effective layer stacking strategy to significantly increase the number of tokens in large language models (LLMs). The experiments are comprehensive, and the results are solid. Therefore,  the AC has decided to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a method to add more visual information to a MM-LLM without increasing the number of tokens processed by the model. The idea is simple, just add visual tokens to the existing hidden representation between each layer of the transformer. The approach is evaluated on many tasks and shows good results.
strength_weakness: strength_weakness: Overall the paper is well done. The experiments are thorough, the idea is well explained, and the method is reproducible.
questions: questions: None
limitations: limitations: na
paper_summary: paper_summary: This paper proposes a new visual token organization method. Specifically, it proposes to stack visual tokens instead of the commonly used stringing. Experiments show that the proposed StackFormer can improve performance on TextVQA, DocVQA, and InfoVQA.
strength_weakness: strength_weakness: - There are many typos in the paper. The authors need to improve their writing and polish the paper. Like Line 18 "StackFormeruses" --> "StackFormer uses"; Line 134: Multi-modal Language Models (LLMs). Sometimes it uses LMMs and sometimes MLLMs. Both are ok, but please use only one in the same paper.

- There is no significant improvement for LLaVA-Next on MLLM benchmarks.
questions: questions: - In table 1, what do † and * mean?

- Could you explain why StackFormer cannot improve LLaVA-NeXt on MLLM benchmarks?
limitations: limitations: the authors have discussed the limitations in the paper. StackFormer cannot significantly improve larger model LLaVA-NeXt on MLLM benchmarks. It may be another limitation.
paper_summary: paper_summary: The authors present a new architecture StackFormer - an effective and simple way to infuse fine-grained visual tokens from CLIP vision transformer to the early layers of  LLaVA-1.5 and LLaVA-Next language models, without increasing the sequence length of visual tokens for LLMs. It doesn't require architecture change while significantly increases the number of tokens LLMs can take, so it improves accuracy especially for high-resolution images and videos.
strength_weakness: strength_weakness: In Fig. 1 and 2 are missing details about the implementation of StackFormer:
- how exactly you split the high-resolution image into patches
- and how exactly you split the high-resolution visual tokens into different token sets with spatial dilation

While you write that StackFormer achieves the best trade-off between performance and effectiveness without introducing extra visual tokens, specific indicators of overhead costs when using Stackformer are not provided. Flops, parameters, latency, memory consumption, accuracy are not provided in a single table to compare Stackformer with other VLMs.
questions: questions: Can you show in more detail in the Figures 1 and 2:
- how exactly you split the high-resolution image into patches?
- and how exactly you split the high-resolution visual tokens into different token sets with spatial dilation?

Can you provide numerical indicators (flops, params, latency, memory consumption) that Stackformer achieves the best trade-off between performance and effectiveness compared to other VLMs?
limitations: limitations: Limitation and Future Work: 
The paper presents limited options for processing high-resolution images by naively splitting the image into many patches and applying VIT-CLIP to them separately. While there are many approaches to processing high-resolution images using transformers: ViTDet [1], SwinV2 [2], Patch-Fusion [3], ... or simply naively resize the ViT-CLIP model to the required image resolution, using 2D interpolation of the pre-trained position embeddings [4].


1. Exploring Plain Vision Transformer Backbones for Object Detection, Yanghao Li, Hanzi Mao, Ross Girshick, Kaiming He, 2022
2. Swin Transformer V2: Scaling Up Capacity and Resolution, Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, Baining Guo, 2021
3. PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation, Zhenyu Li, Shariq Farooq Bhat, Peter Wonka, 2023
4. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby, 2020

Comments: comment: Thanks for the rebuttal. I would like to keep my score of 7: Accept.
Thanks to the authors for the answers, detailed schemes and additional results. This makes the approach clearer. Based on this, I increase the rating of the paper to "6: Weak Accept"