Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight

Authors: ziyuan  huang
kaixiang  ji
biao  gong
zhiwu  qing
qinglong  zhang
kecheng  zheng
jian  wang
jingdong  chen
ming  yang
ant  group
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:28:18

Abstract: This paper introduces Chain-of-Sight, a vision-language bridge module that accelerates the pre-training of Multimodal Large Language Models (MLLMs). 
Our approach employs a sequence of visual resamplers that capture visual details at various spacial scales.
This architecture not only leverages global and local visual contexts effectively, but also facilitates the flexible extension of visual tokens through a compound token scaling strategy, allowing up to a 16x increase in the token count post pre-training.
Consequently, Chain-of-Sight requires significantly fewer visual tokens in the pre-training phase compared to the fine-tuning phase. 
This intentional reduction of visual tokens during pre-training notably accelerates the pre-training process, cutting down the wall-clock training time by $\sim$73\%.
Empirical results on a series of vision-language benchmarks reveal that the pre-train acceleration through Chain-of-Sight is achieved without sacrificing performance, matching or surpassing the standard pipeline of utilizing all visual tokens throughout the entire training process. 
Further scaling up the number of visual tokens for pre-training leads to stronger performances, competitive to existing approaches in a series of benchmarks.

Main paper: 
 introduction: 
 Recently, Large Language Models [70, 6, 80, 5, 3] have received unprecedented attention, owing to their remarkable capabilities in text comprehension and generation. Riding on the success of LLMs, Multimodal Large Language Models (MLLMs) [74, 90, 63, 56, 26, 88, 73] demonstrate impressive zero-shot transferability across a wide range of vision-language tasks, such as image captioning, visual question answering, and visual grounding.The exceptional generalization ability exhibited by the contemporary MLLMs can be largely attributed to their extensive pre-training on a massive amount of data [16, 14, 69, 31, 18] . However, as the volume of data escalates, so does the wall-clock training time, which has become a major obstackle in further explorations. According to [63] , 60,000 GPU hours are needed for training a 7B model on just 96 million image-text pairs. This intensive computational demand is not only prohibitive to many researchers, but also leads to a significant carbon footprint.One of the key reasons for the prolonged training time is the extensive length of visual tokens Typically, the image-text pairs in the pre-training phase involve around 23 text tokens (see Table 1 ). In contrast, most MLLMs handle substantially more visual tokens during pre-training, e.g., 144 [10, 11] , 256 [4, 50, 97] , or even higher [63, 16, 55, 56, 64, 48] . Reducing the number of visual tokens presents a straightforward way to speed up training, as it allows for an increase in batch size and a concurrent decrease in step time. Meanwhile, the reduced memory consumption allows for better optimization stages [81] , further reducing time requirements. However, training with fewer visual tokens often results in compromised performance for existing vision-language models. Recent current MLLMs maintain a constant set of visual tokens in both pre-training and fine-tuning. These tokens typically represent visual contents at a single visual scale. In contrast, our Chain-of-Sight approach leverages the idea of visual hierarchy, producing multi-scale visual tokens. Moreover, the token scaling strategy enabled by our multi-scale visual resamplers allow us to start with a small pool of visual tokens for pre-training, before increasing the number of tokens during fine-tuning. This considerably accelerates the pre-training phase.To resolve this dilemma, this work introduces Chain-of-Sight, a vision-language bridging module for efficient pre-training of MLLMs. Unlike existing approaches that maintain a constant token count throughout both pre-training and fine-tuning, Chain-of-Sight allows for a marked increase in the number of tokens after the pre-training stage, thereby reducing the tokens needed during pre-training. The core mechanism is our multi-scale visual resampler, which produces visual tokens of multiple visual scales. Inspired by the classical concept of multi-scale feature hierarchy in visual understanding [106, 42, 33, 107, 83, 53] , we partition the visual features produced by the visual backbone using windows of multiple sizes. For each window size, a visual resampler is implemented to produce a specified number of visual tokens per window. Subsequently, the visual tokens from various window sizes are gathered and linked in a global-to-local manner, forming a chain of reasoning steps from coarse views gradually to fine-grained perspectives.On top of this, we propose a post-pretrain token scaling strategy, which compounds the elements of input resolution and window size manipulation to enable a significant escalation in the token count for our Chain-of-Sight, reaching up to 16× increase during fine-tuning. Such adaptability allows for the fine-tuning of the model with a flexible granularity or complexity as required, without the the necessity for an additional pre-training phase.By intentionally reducing the number of visual token by ∼90% in the pre-training, a 2.5× batch size is allowed with a step time reduction of 30%, leading to a 3.7× faster pre-training in terms of wall-clock time (∼73% less) for the same amount of data, when compared with using all visual tokens during pre-training. Meanwhile, our observations indicate that this acceleration does not come at the expense of performance. The results achieved by our Chain-of-Sight model pre-trained with 32 tokens match or surpass those obtained using 336 visual tokens throughout the training process, when both models use the same tokens during fine-tuning. Further scaling up the tokens in the fine-tuning stage leads to enhanced performance at small additional training costs. This scaling showcases the potential of Chain-of-Sight to capitalize on the initial efficiency gains and adapt its framework to achieve even greater levels of accuracy and effectiveness in visual understanding for MLLMs. 
method: 
 Our objective is to accelerate the pre-training of MLLMs. To this end, we resort to reducing the number of visual tokens inputted into the language model. To mitigate the performance drop associated with fewer visual tokens, we introduce a versatile bridge module within our framework, named Chain-of-Sight. This module is designed to enable the increase in the token count on demand after pre-training. With this capability, we are able to substantially lower the number of visual tokens during the pre-training phase, while retaining the ability to scale up and capture a rich level of visual detail during fine-tuning. The concept of Chain-of-Sight is illustrated in Fig. 1 .Modern MLLMs are typically constructed by three core components: (1) a visual encoder, (2) a vision-language bridge module, and (3) a language model. Given that the language models often have a much larger size than the visual encoder, they account for the majority of computation during pre-training [90, 4, 73, 19, 58] . Consequently, the number of input tokens processed by the language model is a crucial factor determining the total computational workload. Through partitioning visual features into windows and restricting cross-attention to the windowed features associated with the learnable tokens, our Chain-of-Sight approach produces visual tokens that encompass multiple scales. Thanks to the post-pretrain token scaling strategy, Chain-of-Sight reduces the required number of visual tokens in pre-training, thus accelerating the process. In contrast, the number of visual tokens remains constant in resampler-based methods [44, 2, 4, 99] for pre-training and fine-tuning, and the linearlayer [56, 63, 97, 15] produce a large number of visual tokens, incurring a high cost for pre-training.As detailed in Table 1 , the pre-training data predominantly comprise image-text pairs that contain fewer than 50 text tokens. In contrast, existing MLLMs are designed to handle 2× more visual tokens, often requiring such as 144 [10, 11] , 256 [4, 50, 97] , or even more visual tokens [63, 16, 55, 56, 64, 48] .The imbalance between the visual tokens and text tokens means that processing these visual tokens has become the main efficiency bottleneck in MLLM pre-training. This prompts our exploration for more efficient vision-language bridging structures, which is capable of reducing the number of visual tokens in pre-training without compromising performance.The multi-scale visual resamplers serve as the foundational mechanism enabling the flexible extension of visual tokens after the pre-training phase. This subsection focuses on the architectural details of the multi-scale visual resamplers, as visualized in Fig. 2 (b), while the the extension of visual tokens is discussed in the subsequent subsection.Essentially, the idea of exploiting multi-scale or pyramid structures to handle natural hierarchy of visual contents has been long established as a standard practice [30, 42] , proving effective in countless visual tasks [33, 61, 83, 53] . Despite this, the potential for harnessing multi-scale visual hierarchies remains under-explored in the context of MLLMs.Visual resampler. Visual resampler is a Perceiver [36] -like structure that introduces a set of learnable queries and uses cross-attention to condense visual knowledge into a predetermined set of visual tokens [4, 2, 95, 99] . We construct Chain-of-Sight with visual resamplers due to their flexibility in selecting the token count for a specified feature, independent of the features' dimensionality.Multi-scale visual resamplers. One of the effective strategies for building multi-scale features within a network involves combining operations that spans diverse fields of views [13, 45, 101, 21] . Given that the resampler structure inherently gathers visual cues on a global scale across the entire feature map, our strategy focuses on enhancing the perception of the fine-details in the image.To this end, we partition the visual features into non-overlapping local windows of various sizes. More precisely, given a visual feature X ∈ R L×L×C extracted by the visual encoder, where L and C denote the feature size and channel, respectively, we define a set of window sizes, denoted as W = [W 1 , ..., W m ]. This setup leads to a collection of windowed visual featuresX win = [X 1 , ..., X m ].Each X i represent a set of L 2 /W 2 i windowed features obtained by applying the partition operation on the original visual feature maps with a corresponding window size W i . This naturally forms features of multiple visual scales. At every scale level, each windowed feature is allocated with N i learnable queries. These learnable queries are then utilized within the visual resampler to perform cross-attention solely on their corresponding windowed feature. This yields N tokens, where the N can be calculated as follows:EQUATIONThe learnable queries within the same scale share the parameters of the visual resampler, despite their different spatial locations. However, because the queries at various scales are intended to capture features from varying fields of view, distinct sets of parameters are used for each scale. This results in a group of visual resamplers operating across multiple scales. On top of this, we enable the resamplers to aggregate visual features from multiple feature levels, as in [8] (see Appendix for details).Coarse-to-fine integration. Upon acquiring a series of multi-scale visual tokens from the multi-scale visual resamplers, our method integrates these prompts in a structured coarse-to-fine fashion. The final token sequence fed to the language model begins with tokens derived from larger windows, which presents an overall view of the image, and proceeds with tokens obtained from smaller windows that contains fine-grained details. Our priliminary experiments reveal a substantial difference in the overall performance between the coarse-to-fine and the reversed order.Reducing the number of visual tokens can effectively accelerate pre-training, but typically at the expense of performance. To address this dilemma, we enhance the token count after pre-training, which allows accelerated pre-training with fewer visual tokens, while a subsequent increase in tokens ensures the final performance after fine-tuning, as demonstrated in Fig. 2(c ). Specifically, based on the multi-scale visual resamplers, the increase in the token count is accomplished via our compound token scaling strategy that integrates two core mechanisms: resolution scaling and window scaling.Resolution scaling. Enhancing the input resolution stands as the most direct way to augment the number of visual tokens. At the cost of additional computation overhead in the visual backbone, it allows for a quadratic rise of the token count with the resolution enhancement. The concept of resolution scaling is investigated in many existing approaches based on linear projectors [55, 69, 26] or visual resamplers [50] . They can broadly be viewed as particular instances within our Chain-of-Sight framework, which regards the window size as a fixed factor. In this context, linear projectors use the smallest possible window size for visual token generation, whereas visual resamplers employ the resolution in the pre-training phase as their window size.Window scaling. The windowing mechanism in our multi-scale visual resamplers enable scaling up token numbers by manipulating the window sizes. As in Eq. 1, reducing the window sizes can further produce a quadratic increase in the number of visual tokens on top of the resolution enhancement.Compound scaling. Combining the above token scaling strategies, our compound scaling is capable of producing a 16× increase in the tokens during fine-tuning, as in Fig. 3 . This allows us to fine-tune the scale at which visual features are represented and sampled, improving the model's capability of leveraging varying levels of detail and abstraction inherent in the visual content. Consequently, the Chain-of-Sight framework significantly boosts the visual comprehension capability of the model during the fine-tuning stage, effectively compensating for the performance drop incurred by the low number of visual tokens during pre-training.Initialization. Inspired by [9] , we initialize the parameters of the newly introduced visual resamplers by simply inflating the pre-trained parameters, as in Fig. 2 . As for the new visual queries, we apply a nearest neighbor strategy to initialize them based on the pre-trained queries. 2 . Our method achieves a pre-train acceleration of 73% without compromising performance.In this section, we provide our experimental setup, empirical results, and the comparisons with existing methods.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations are provided at the end of the manuscript. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental setup: 
 Model details. We instantiate our MLLM with CLIP-ViT-L/14 [78] as the visual encoder and Vicuna [20] as the language model. For efficiency, we adapt Vicuna with LoRA [34] Training settings. The training of Chain-of-Sight is divided into two stages. For the first stage, we sample around 65M image-text data involving multiple tasks, as detailed in Table 1 . The multiscale visual resamplers and the LoRA paramters [34] are unlockded for training. For the first 120,000 iterations, we use the input resolution of 224 and unlock the resamplers and the LoRA parameters [34] for training. During the last 30,000 iterations, the input resolution is raised to 448, where the parameters in the visual backbone is further activated and the tokens are scaled up through our compound scaling. The second stage of the Chain-of-Sight model is supervised fine-tuning, where we remove all the captioning datasets except for COCO.Evaluation benchmarks. The evaluation of our approach involves various tasks including image captioning, visual question answering, text recognition, as well as the tasks defined in popular vision-language benchmarks. Details can be seen in Table A2 .We first ablate the Chain-of-Sight (CoS) design for accelerating the pre-training of MLLMs. For the ablations, we omit the high resolution tuning in the first stage unless otherwise specified.Pre-train acceleration by Chain-of-Sight. Fig. 4 shows the cost for pre-training and supervisedfinetuning with various number of visual tokens, as well as the corresponding average performance over 12 benchmarks. We make several key findings. Compared to the linear projection, we fine-tune the visual backbone at a resolution of 224x224. Pre-trained with 80 tokens, CoS slightly falls behind when we use 80 tokens for fine-tuning, while achieving stronger performance when we scale up the tokens to 336. In this case, the total time for pre-training can be saved by 50% when compared to using linear projections with 256 visual tokens.Compared with visual resamplers, when a high number of visual tokens are used during pre-training, Chain-of-Sight performs similarly. However, when pre-trained with fewer visual tokens, Chainof-Sight have notable advantages on captioning and text recognition for 80 tokens, outperforming resamplers by 1.8 for captioning and 10.1 for text recognition. Further scaling up tokens pushes our model to have a performance stronger than the model pre-trained with 336 tokens. In addition, our compound scaling proves more effective than existing resolution scaling methods [57, 50, 69] .We also try further reducing the number of tokens for pre-training. The performance drop is mainly observed in vision-language benchmarks and question answering capabilities, with minor affect on the captioning task. Nevertheless, we observe that our model pre-trained with 32 tokens achieve a similar or stronger overall performance with resamplers pre-trained with 336 tokens, while taking only 0.27× wall-clock training time (achieving a 3.7× pre-train acceleration).Vision-language benchmarks. We find the performances on vision-language benchmarks more heaviliy affected by the number of tokens used in the supervised fine-tuning stages than the ones used in the pre-training stage, especially for the question answering task with multiple choices, as in Table 2 . This provides a strong support for using a small set of visual tokens in the pre-training stage for acceleration, and a large set of visual tokens for optimal performance. Visual grounding. Table 3 shows the comparison between Chain-of-Sight and its baselines on referring expression comprehension (REC). The conclusions are consistent with the above findings. Since Chain-of-Sight incorporates both global and local contexts, using our method notably boost the performance of visual resamplers, achieving an improvement of 3.82 and 3.49 on the average performance when using 80 and 336 visual tokens, respectively. Notably, for the REC task, when compared with Chain-of-Sight model pretrained with 336 tokens, we can achieve at most a 2.85× acceleration (reducing the wall-clock training time to 0.35× using 48 tokens for pre-training) without performance loss. Against the standard visual resampler pre-trained with 336 tokens, our 32-tokenvariant perform favourably, while reducing the training cost by 73%.Post-pre-train compound scaling of visual tokens. In Table 4 , we ablate our token scaling strategy. The experiment share the same pre-training, where the baselines are models fine-tuned with 80 visual tokens at 224 2 and 448 2 resolutions. For scaling up the number of tokens, we separate the ablations on global and local tokens. Notably, we find scaling up the global tokens alone has negligible effects on the average performances, while scaling up local tokens brings notable improvement on various benchmarks. Combining both further brings slight improvements over the model with up-scaled local tokens on almost all benchmarks. Hence, we use 1296 tokens for fine-tuning our final model. Efficient model pre-training. As the model size consistently expands, the efficiency of training large models has become increasingly important. Beyond efforts in the system optimizations [81, 82, 24, 23] , the pre-training of large models can be accelerated by sparse computation, such as masking [46, 77] or mixture of experts [28, 51] . Our approach presents a novel perspective for accelerating pre-training for MLLMs by reducing visual tokens required.Multi-scale hierarchy in vision. Multi-scale hierarchy is a fundamental property in vision, which has led to the introduction and evolution of convolutional networks [30, 42, 40, 33] as well as its application in various vision problems [61, 83, 53, 13] . Recently, transformers are also shown to benefit from multi-scale hierarchy [98, 60, 27, 47] . This work extends multi-scale hierarchy to language models for stronger visual capabilities and higher training efficiency.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: We do not have theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have provided details for training our model in description and using tables.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: Even though our approach uses LoRA for fine-tuning, the whole training process is still computational intensive. Hence, it is difficult to provide error bars for the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
discussions: 
 Limitations. Despite the strong performance and the notable acceleration achieved by Chain-of-Sight, our approach leverages parameter efficient fine-tuning (PEFT) for adapting language models. Hence, the generality of the final model might be limited, compared to approaches that fine-tunes the whole language model during supervised fine-tuning process [57, 10, 52] or even the pre-training stage [63, 4, 26] . This is mainly due to the limited training resources and is exactly what motivates us to explore efficient pre-training methods. We believe the pre-train acceleration achieved by the presented approach has stronger potentials beyond our results. The Chain-of-Sight model employs off-the-shelf pre-trained language models. As such, all CoS models inherits the shortcomings from the language model, including certain model biases or hallucinations. While to some extent CoS enhances the visual capabilities of language models, proper assessment and safety precautions are still required before deploying our model.Our multi-scale visual resamplers consider visual hierarchy in two aspects. In addition to the multiple spatial scales detailed in the manuscript, we also enable the visual resamplers to aggregate from multiple feature levels in the visual backbone, which is useful in a wide range of visual tasks [47, 53] but often neglected in current MLLMs.In Fig. A1 , we demonstrate the structure for the multi-level feature aggregation. Essentially, we exploit features from multiple layers in the visual backbone, and the learnable queries aggregate sequentially from lower level to higher level features. Here, we provide details on the training data and the evaluation benchmarks. [103] RefCOCOg [67] Table A1 shows per-dataset length distributions of our training data. The majority of the data used in the pre-training stage have an average length lower than 25, which provides strong motivation for our approach that reduces the number of visual tokens for pre-training acceleration.Table A2 shows the detailed description on the benchmarks we use to evaluate our model. Table A2 : Summary of the evaluation benchmarks.Dataset Description Split MetricsNoCaps [1] Captioning of natural images. val CIDEr (↑) Flickr [76] Captioning of natural images. karpathy-test CIDEr (↑) COCO [54] Captioning of natural images. karpathy-test CIDEr (↑)General VQAVQAv2 [32] VQA on natural images. test-dev VQA Score(↑) GQA [35] VQA on scene understanding and reasoning test-balanced VQA Score (↑) OK-VQA [68] VQA on natural images requiring outside knowledge. val VQA Score (↑) ScienceQA-Img [65] Multi-choice VQA on a diverse set of science topics test Accuracy (↑)Text-rich benchmarks TextVQA [87] VQA on natural images containing text. val VQA Score (↑) TextCaps [86] Captioning of natural images containing text. test CIDEr (↑) LVLM Benchmarks SEED-Bench [43] Multi-choice VQA on a diverse set of topics IMG Accuracy (↑) MMBench [59] Multi-choice VQA on a diverse set of topics test Accuracy (↑) MME [29] Open-ended VL Benchmark by yes/no questions Perception & Cognition Accuracy (↑) POPE [49] Multi-choice VQA for testing hallucinations overall F1-Score (↑) MMMU [104] VQA on a diverse set of topics val Accuracy (↑)RefCOCO [38] Refer grounding on natural images. overall Accuracy (↑) RefCOCO+ [103] Refer grounding on natural images. overall Accuracy (↑) RefCOCOg [67] Refer grounding on natural images. overall Accuracy (↑) Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provided that in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Definitely.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed broader impacts in our appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: We do not plan to release any data. For the model, we currently do not have safeguards for releasing it. We will make sure that the guidelines and instructions are in place when we release the model. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, we have. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: This work leverages multi-scale tokens to reduce the number of visual tokens in pre-training VLMs, speeding up the pre-training of VLMs by ~2.8x (including SFT) -- 3.7x (excluding SFT), while maintaining/improving the performance. The approach is validated for a number of models, tasks, and datasets. 

The reviewers agreed that this work is clearly presented, and appreciate the gains in pre-training efficiency.
There are concerns regarding no results with bigger models, however, 7B models used in the experiments are of respectable size.

I can recommend this paper for publication given the scope for adoption of the proposed approach due to pre-training efficiency gains.

Reviews: review: paper_summary: paper_summary: 1. This work proposes Chain-of-Sight, a training method of MLLM that leverages global and local visual contexts effectively.
2. To boost efficiency in the pretraining stage, the authors propose a post-pretrain token scaling strategy. During the pretraining state, it requires significantly fewer visual tokens and cuts down the wall-clock training time by 73% in the pretraining stage. To effectively increase tokens during fine-tuning stage to enhance performance, they propose a compound strategy by manipulating input resolution and window size at the same time.
3. When both use the same number of tokens during fine-tuning, the results achieved by the Chain-of-Sight model pre-trained with 32 tokens match or surpass those obtained using 336 visual tokens throughout the training process.
strength_weakness: strength_weakness: 1. The method of hierarchical (multi-scale) visual input has already been explored in many former works, which is contradictory to the expression in L91-92 "Despite this, the potential for harnessing multi-scale visual hierarchies remains under-explored in the context of MLLMs". For example, LLaVA-NeXT [1] and InternLM-XComposer2-4KHD [2] both adopted multi-scale strategies and proved their effectiveness. 

2. There are mistakes in the table of experiment results. For example, in Table 3, RefCOCO+, test-A, the second-best one should be 90.57. 



[1] Liu, Haotian, et al.  "LLaVA-NeXT: Improved reasoning, OCR, and world knowledge." https://llava-vl.github.io/blog/2024-01-30-llava-next/

[2] Dong, Xiaoyi, et al. "Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd." *arXiv preprint arXiv:2404.06512* (2024).
questions: questions: 1. In Table 2 and Table 3, why are many of the results of "CoS, pre-train 80, FT 336" better than those of "CoS, pre-train 336, FT 336"? Analysis or explanations are suggested. 
2. The settings of ablation experiments are not explained. Is that the same as sec 3.1 training settings? If so, why do the results in Table 5 not agree with those in Table 2&3&4?
limitations: limitations: The experiment is only conducted on 7B-level model with PEFT. Larger scale model with full-parameter training is suggested to prove its effectiveness and extensibility in modern pretraining.
paper_summary: paper_summary: This paper proposed a post-pretrain token scaling strategy, Chain-of-Sight, to accelerate the pre-training of Multimodal Large Language Models (MLLMs).
Through the proposed method, the authors were able to achieve a significant improvement in pre-training speed.
The authors confirmed through an ablation study that the proposed method exhibits performance similar to existing approaches.
strength_weakness: strength_weakness: The paper only includes experimental results using CLIP-ViT-L/14 as the visual encoder and Vicuna. Experimental results based on other models are needed.

The explanation of how the number of tokens during pre-training and fine-tuning (e.g., PT:80 FT: 1296) was determined is insufficient.
questions: questions: see weakness.
limitations: limitations: The authors addressed both the limitations and potential negative social impacts in Section 5 (Conclusion) and Section A (Broader impact).
paper_summary: paper_summary: This paper introduces Chain-of-Sight (CoS), a novel vision-language bridge module designed to accelerate the pre-training of Multimodal Large Language Models (MLLMs). The key innovation is a sequence of visual resamplers that capture visual details at various spatial scales, allowing for a significant reduction in visual tokens during pre-training while maintaining or improving performance. The authors conduct sufficient experiments, including the evaluation on diverse benchmarks and three scaling experiments, to justify their assumptions and model designs.
strength_weakness: strength_weakness: One minor weakness is that the authors have not discussed the potential impact of the scale of training data. Multiple compared MLLMs (e.g. Qwen, VILA) in Tables 5 and 6 are probably pre-trained on unequal sizes of data from diverse sources. It will help to clarify how the proposed Chain-of-Sight improves the performance on various tasks if the authors rule out the impact of the differences in the pre-training data.
questions: questions: Q1: See the Weakness above.

Q2: Will there be differences in the convergence speed of different compound scaling strategies? Which combination of window scale, resolution scale, and compound scale exhibits the fastest convergence?

Q3: Do the authors interpolate the positional embeddings when increasing the input resolution mentioned in Line 175?

Q4: Do the authors mean to refer to Table 3 instead of Table 6 in Line 222?
limitations: limitations: None.

Comments: title: Response to the authors' rebuttal
comment: Thanks for your reply, my concerns have been partially solved. However, I'm still concerned about the limited technical novelty (Q1), and the effect on larger-scale model or full-parameter training. Therefore, I would raise my score a bit but still be on the borderline (Please consider my score to be 4.5).
Thank you for providing the detailed answers. My concerns have been resolved and the rating of 7 is maintained. Good luck!