Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

Authors: alex jinpeng wang
linjie  li
yiqi  lin
min  li
lijuan  wang
mike zheng shou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-04 10:48:12

Abstract: Training models with longer in-context lengths is a significant challenge for multimodal machine learning due to substantial GPU memory and computational costs. This exploratory study does not present state-of-the-art models; rather, it introduces an innovative method designed to increase in-context text length in multi-modality large language models (MLLMs) efficiently. We present \ModelFullName (\ModelName), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs). For instance, our method expands the pre-training in-context length from 256 to 2048 tokens with fewer FLOPs for a 56 billion parameter MOE model. Experimental results demonstrate that \ModelName enhances OCR capabilities and delivers superior performance on common downstream benchmarks for in-context few-shot evaluation. Additionally, \ModelName proves effective for long context inference, achieving results comparable to full text input while maintaining computational efficiency.

Main paper: 
 introduction: 
 Large Language Models (LLMs), such as OPT, Mistral, and LLaMA-2 [4, 5, 6] , have significantly advanced the field of Natural Language Processing (NLP). These advancements are partly due to the increased capability of LLMs to process long contexts, from 512 tokens [7] up to 16K tokens [6] . Building on these developments, recent multi-modal learning research [1, 8, 9, 10] has shifted focus from simple image-text pairs, like those in CC3M [11] and LAION-400M [12] , to more complex and lengthy interleaved document datasets. Examples include web corpora like MMC4 [13] and the OBELICS [14] dataset, as well as PDF corpora like DocVQA [15] . However, training models on these complex datasets presents significant challenges due to the increased GPU memory and computational demands of extended contexts. For instance, while processing just 5M data items from MMC4 and 10M from the OBELICS dataset, OpenFlamingo-9B [9] resorted to sub-sampling text and processing only 256 tokens at a time, yet it still requires 32 80GB A100 GPUs for over three days. This highlights the need for more computation-efficient methods to handle long context lengths effectively.In the domain of LLMs, two popular methods to extend context length are the use of memorizing banks [16] and novel self-attention mechanisms [17, 18] . These methods have inspired advancements in the multi-modality domain as well. For example, the Large World Model [19] introduces Ring Attention [18] , and MA-LMM [20] employs memory banks to process long video understanding tasks. While these techniques have shown promise, our approach aims to increase in-context text length For our method, we incorporate VisInContext after 128 text tokens. We implement PyTorch Flamingo [1] models with different in-context length during pre-training. The language model is a 56B MOE [2] model loaded with 4-bit quantization and the batch size on each GPU is 32 with FP16. We train the model with DeepSpeed [3] Zero-2.by leveraging the strengths of visual encoders in MLLMs. We first observe that existing MLLMs usually exploit a much lighter visual encoders, compared to its text decoders. For instance, Flamingo-9B consists of a 304.4M ViT-L/16 [21] as image encoder, and a 7.1B Chinchilla [1] model as the text decoder. Additionally, previous works [22, 23] have demonstrated that visual encoders trained on paired image-text data also exhibit emergent OCR capabilities.Motivated by these observations, we propose Visualized In-Context Text Processing (VisInContext), a method that uses visual tokens to process extended textual contexts, which is complementary of existing methods in extending context length. Specifically, we convert long textual content into images and use the visual encoders to extract textual representations. In this way, we can efficiently and effectively enable models with much longer text contexts, as shown in Figure 1 . With VisInContext, we show that the in-context text length can be increased by 7 times over the competing baseline. Additionally, we observe almost the same overall computation FLOPs even as in-context length extends significantly. Our extensive experiments will also show that VisInContext renders superior model performance on conventional in-context few-shot evaluations and document understanding, with much lower computational cost.Contributions. In summary, our contributions are as follows: i. We introduce Visualized In-Context Text Processing (VisInContext), a novel method that increases in-context text length using visual tokens. VisInContext directly compresses text context at input-level, which is complementary to existing techniques with improved self-attention or memory banks. ii. We demonstrate that VisInContext is effective for both training and inference stage with much lower computational cost.iii. With extended text context brought by VisInContext, our model improves the average in-context few-shot performance from 55.8% to 57.8% over the competing baseline. iv. As a byproduct, our method also shows great potential in document understanding on popular document QA tasks and our newly proposed sequential document retrieval task.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Reflected in Section1 contribution.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
method: 
 The goal of VisInContext is to process in-context text using visual tokens so that the model can handle long text context more efficiently. We primarily base our study on Flamingo-based architecture [1, 9, 14] , as it has shown success in improving a model's ability to learn from long multimodal context that contains arbitrarily interleaved text and images.Before diving into model details, we define the following terms: With VisInContext, the In-context Text Length is greater than the text token length, as part of the text is represented using visual tokens.The implementation and architecture of VisInContext are shown in Figure 2 . It is based on a dual-stream encoder model that integrates both visual and textual data. To effectively handle long interleaved data, we use a pre-sampling strategy as in Flamingo-style works [1, 9, 14] . We sample m images I 1 , I 2 , . . . , I m ∈ I with corresponding texts T 1 , T 2 , . . . , T m ∈ T . Tokens are concatenated in the form < visual 1 >< text 1 > . . . < visual m >< text m >, where < visual > is a single placeholder token. A random 256-token sequence is then sampled. However, since the overall length of a web document is generally much longer than 256 tokens (In-context Text Length ≥ Text Token Length), this sampling approach can lead to the omission of a lot of related text context.To address this issue, we convert these omitted text context into visual signals by rendering them into images. We first concatenate all omitted text segments and divide them into K parts to render text images, namedT ′ 1 , T ′ 2 , . . . , T ′ m ∈ T ′ .Both the original images and the text-rendered images are then processed through a shared frozen vision encoder. Then, we employ two learnable resamplers to extract a fixed number of tokens from both the raw and text-rendered image features, respectively. To facilitate the model to learn from rendered text images, we introduce two novel model designs, Token Masking mechanism and Text-Centric Contrastive Learning (TCCL). Token Masking allows the model to only read from text image tokens by masking the raw image tokens with masking ratio 1, which ensures that the model won't simply be ignoring the text images during training, hence can learn the association between the rendered text images {T ′ i } and the text tokens {T i }. TCCL aligns the visual text representation from the resampler with the embeddings extracted from text tokenizers in LLM, which reduces the gap between our visual text tokens and the text tokens the LLM is trained to perceive. With these designs, VisInContext not only reduces computational demands-as evidenced by a reduction in flops and inference time-but also improves the OCR ability, as we will show in our experiments.This module converts textual data into a visually rich RGB format, specifically rendering the text into an image size of p h × np w , where n is the number of patches. We employ the HERSHEY font at a size of 10px. On average, one 16x16 patch accommodates approximately 1.5 OPT text tokens. A 224x224 text image contains about 294 text tokens. Consequently, a visual encoder operating on this rendered text image requires only 1/3 of tokens to encode an equivalent amount of text, compared to the text tokenizer in language models. The vision encoder is quite lightweight ViT-L (340M) compared to language model MOE (56B), which makes the processing of rendered text images significantly more efficient than directly inputting the text into a language model.In our initial experiments, we find that combining tokens from raw images and text images directly led to the network disregarding the text-image input. To address this issue, we introduce a Token Masking strategy to force the model to learn text semantics from visual inputs. During pretraining, the raw image and text image are first encoded into the same number of tokens after resampler, and then we mask the raw image tokens with a pre-defined probability. When masking out the raw image tokens, the model can focus on learning the association between rendered text images and the complementary text tokens. At inference time, we add the text-image tokens and image tokens together, to allow the model effectively leverage information from both sources.Motivation. Given that the vision encoder, typically a frozen Vision Transformer (ViT) [24] , never observes rendered text images during pretraining, it may struggle to derive text semantics from pixels. To mitigate this issue, we introduce a new training objective, Text-Centric Contrastive Loss (TCCL). This objective aims to guide the resampler on rendered text images to interpret visual representations of text with a proficiency comparable to traditional text tokenizers, so that the textual semantics can be effective extracted from the rendered text images.Mechanism. TCCL utilizes raw text token embeddings from the text tokenizer as soft supervision signals to supervise the resampler to learn text-centric representation. To reduce the global semantic gap between text image embeddings and text token embeddings, we first aggregate these embeddings with average pooling and then align them with TCCL. Intuitively, TCCL is designed to turn the joint of the vision encoder and resampler into a "visual" text tokenizer, as it promotes the text image embeddings to share a similar global semantic as the text token embeddings. The core of TCCL is formulated as a contrastive loss:EQUATIONWhere L ij denotes the contrastive loss for comparing the i th text image against the j th text, f vi and f tj represent the feature embeddings of the i th text image and j th text, respectively. τ is a parameter that control the sharpness of the output distribution. Note that f vi and f ti are different features extracted from the same text, as the i th text image is a direct rendering of the i th text.3 ExperimentQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitation in conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental setup: 
 Pretraining. We validate VisInContext with Open-Flamingo [9] and CosMo [25] . To enhance computational efficiency, all models utilize float16 precision. For the 56B MOE [2] model, we employ DeepSpeed's [3] Zero-2 stage with CPU offloading and further optimize the model by quantizing it to 4-bit precision1 . We also use Flash Attention [17] to further improve memory efficiency. For all other experiments, we train the model using DeepSpeed Zero-2 without CPU off-loading. The Open-Flamingo 9B baseline is based on Mistral7B [5] .Our pretraining dataset includes a 180M subset of DataComp1B [26] , MMC4 [13] , the OBELICS [14] dataset, and OCR Rendered Text [27] . (More details are provided in the Appendix B. Downstream Evaluation. Our objective is to demonstrate that in-context length can be extended using visual tokens, thereby enhancing the understanding of complex multimodal documents. Consequently, we focus primarily on tasks related to long-context understanding.To evaluate the long-context understanding ability, we adopt the few-shot evaluation setting in Flamingo [1] . We report answer accuracy on the OK-VQA [28] , TextVQA [29] , VizWiz [30] , and VQAV2 [31] . Additionally, we assess performance on captioning tasks using COCO [32] and Flickr30K [33] . Moreover, we also propose a setting named text-only in context few-shots to explore text-only in-context evaluation. For this setting, we use in-context sampling without visual input to generate long-context inputs and the visual input is not observed by the model.In order to illustrate the impact of having long in-context text, we evaluate the model for document understanding on DocVQA [15] and OCR VQA [34] . Lastly, we introduce a new task, sequential multimodal document retrieval. This dataset is based on the existing interleaved OBELICS [14] dataset. Further details are provided in the Sec. D of the appendix.Impact of Extended In-Context Text Length. Interleaved document datasets typically contain long texts. For instance, the OBELICS [14] dataset has an average token length of 815 tokens per document. Due to GPU memory constraints, Flamingo-like models [14, 9] only sub-sample 256 tokens during pretraining, which leads to a significant loss of context information. We compare the baseline model pre-trained with 256 tokens, against our method with an increasing In-context Text Length to 2048 tokens. Comparison on Inference Cost. We then analyze the inference cost of VisInContext and compare to the baseline. Both models are based on a 56B MOE LLM with a batch size of one to explore the maximum manageable In-context Text Length. The results, shown in Figure 4 , demonstrate that the In-context Text Length can be extended up to 9192 tokens for the 56B MOE model on 80GB H100 GPUs with our method at inference stage. This result Figure 5 : Sequential multi-modal retrieval example. The input sequence is I1, T1, R1, I2, T2, R2 that from interleaved document in OBELICS [14] We report the sequence retrieval result on OBELICS-Hybrid6.highlights the efficiency and advantages of VisInContext, also show its potential in understanding very long document.In this section, we evaluate the model on document understanding tasks. Unlike common visionlanguage tasks that usually short-form pairs, this task requires comprehension of long and complex document data. We evaluate our model on DocVQA and OCRVQA. All document images are of size 384 × 384. Following Pix2Struct [35] , we finetune the model on DocVQA train data and report performance on the average normalized Levenshtein similarity (ANLS) metric.Results in Table 3 show that our method significantly outperforms the baseline. For instance, we achieve a 6.9% improvement on OCRVQA. To further analyze why our method enhances document understanding, we present the validation accuracy of the LLM on the Rendered Text [27] dataset during pretraining in Figure 3 . We observe a substantial improvement in next word prediction accuracy, with top-1 accuracy increasing from 67.37% to 85.25% (a 16% improvement) and top-5 accuracy rising from 80.76% to 93.38%. These findings indicate that the LLM can effectively understand text embedded in visual signals with VisInContext.In order to further analyze the benefit of having long text context in multimodal modeling, we propose a new task -Sequential Multimodal Retrieval (SMR), based on document data from interleaved OBELICS [14] dataset. The document is composed of interleaved data, consisting of images and texts arranged in a meaningful sequence.We show one sample in Figure 5 and define the input and output of this task as below: Input: Given a pair of content items, an image and a corresponding text We sample the first 1K documents that contain data like I 1 , T 1 , R 1 , I 2 , T 2 , R 2 from OBELICS [14] and named it as OBELICS-Hybrid6, which have at least three frames and three texts. (See Sec. E in appendix for more details.) This task encourages the model to leverage the contextual and semantic relationship in interleaved sequences to effectively predict and retrieve the subsequent pair.EQUATIONTo enable our model with retrieval, we follow CosMo [25] to add a simple contrastive head between visual embedding and language embedding from the middle layers. Recall that visual embeddings are either from raw images or rendered images or the addition of the two in our method. reports the results from our model with several input variants. We observe taking surrounding text input as rendered text image performs much better on the Sequence to image retrieval, while on par on Sequence to text retrieval, when compared with taking surrounding text input as raw text. These results further support the designs of VisInContext in the context of document understanding.Beyond utilizing the visual encoder, some works [36, 8] also employ linear embedding to extract visual features directly from raw images. To show the generality of our method, we also explore FuYu [8] model as a baseline and integrate VisInContext into the model. (See Sec. A in the appendix for more details.) As indicated in Table 5 , our method is successful in improving the performances on DocVQA dataset that require long-context understanding. Ablations on Model Design. We conduct ablation studies on the following modeling components to demonstrate their effectiveness: Text Image, TCCL, and Token Masking. Results are detailed in Table 6 , which reveal two findings: 1. Token Masking is crucial for the model to learn from rendered text images. Without Token Masking, the model can only perform comparably to the baseline. Forcing the model to learn text semantics from rendered text images via token masking significantly improves model performance. 2. Utilizing TCCL with Token Masking yields better performance than using Token Masking alone.Ablations on Font Size and Interval Threshold. As shown in Table 7 , optimal performance varies with changes in font size. We found that adjusting the font size impacts performance similarly to altering the patch size-both methods effectively increase the contextual information within each patch. We prefer modifying the font size over the patch size because it allows for more intuitive adjustments. Our findings indicate that the model does not need a highly detailed understanding of each word to perform effectively.Another important factor is the font interval threshold. As shown in Table 8 , we observed that a too-large interval leads to inferior results. This is intuitive because a larger threshold results in fewer texts in the rendered text image.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
related work: 
 Multimodal Language Models. Current mainstream Multimodal Large Language Models (MLLMs) [37, 38, 22, 39, 40, 41] leverage the capabilities of Large Language Models (LLMs) [42, 6] due to their strong reasoning abilities, as demonstrated by recent advancements. These models typically adopt one of two primary designs for integrating visual information. The first approach involves the effective adaptation of visual representations, which are acquired via a separate visual encoder, into the text-based LLM framework like CLIP, GIT, and BLIP2 [22, 43, 37] . The representative method in this category incorporates visual representations into the language model using cross-attention, as seen in the Flamingo series models [1, 9, 14] . Along this line, recently some works like LLaVA [40] , EMU2 [44] , InternVL [45] , DeepSeeker [10] , and QWen [41] lead to superior results on multi-modality tasks with supervised finetuning on high-quality data. The second approach uses visual embeddings directly as input "tokens" for the LLMs, bypassing the traditional use of a separate visual encoder. This method processes visual patches with a linear layer and uses the resulting embeddings as direct inputs to the LLM, as implemented in models like ViLT [36] and FuYu [8]. This strategy omits the need for an additional visual encoder and simplifies the architecture.In this work, we adopt the Flamingo [1] architecture as our main baseline for the following reasons: First, the Flamingo model emphasizes in-context few-shot learning ability and designs comprehensive few-shot evaluation strategies. Second, our focus is on extending the in-context text length during pre-training rather than on supervised fine-tuning.Enhancing Text Understanding through Visual Inputs. Traditional text tokenization processes raw text efficiently, but it faces challenges such as vulnerability to spelling errors and limited crosslingual transferability [46, 47] . These issues have prompted the exploration of tokenizer-free models, which aim to improve robustness and facilitate better cross-language applicability. For instance, a single spelling error can lead to entirely different tokens using traditional tokenization methods, impacting model performance.Recent developments have seen innovative approaches like the Pixel model [46] , which proposes processing text as an image using both an image encoder and an image decoder. This approach has sparked a series of studies that process not only textual data but also images, charts, and tables through a unified visual input system [35, 46, 48, 47] . These models are trained on a diverse array of visual data, such as webpage screenshots and user interface images, sourced extensively from the internet. They are specifically designed to handle visually-situated text in an end-to-end manner, offering the potential to support a wide range of applications.Long Context Modeling. The challenge of incorporating more tokens into LLMs is an active area of research [49, 50] . Common approaches involve novel self-attention mechanisms [18, 51, 52] , compressed token [53, 54, 55] or memory banks [16] . Some works [56] exploit tensor parallelism or sequence parallelism to reduce memory costs. There also have some works focus on position embedding [57, 58] . In multi-modality research, closed-source models like Gemini [59] and GPT-4V [60] support long context inference up to millions of tokens. Open-source models such as MA-LMM for Long-Term Video Understanding [20] can process up to one hour of video using a long memory bank. The most relevant work Large World Model [19] extends token length using Ring Attention.In contrast to these methods, our method utilizes off-the-shelf LLMs and compresses text tokens into visual tokens for efficient processing. Our method is complementary to these existing techniques and can be integrated with them to achieve lower computational cost and longer context length. 
conclusion and limitations: 
 This paper centers on multi-modality learning and addresses the in-context length limitations presented by heavy computational cost of LLMs in MLLMs. Our contribution is a novel and efficient method named VisInContext, which enables the model to perceive long text context as rendered text images.Comprehensive experiments show that VisInContext is effective on conventional in-context few-shot evaluations and document understanding, while being much more efficient.One limitation of our method is, currently our method requires processing a fixed size image even for brief texts. In future work, we plan to dynamically reduce token counts with variable image sizes by retaining only non-empty tokens during pre-training. We aim to expand this method to additional tasks and encourage the community to further explore this direction. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We discuss the details in the first subsection of experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impact in Sec G of appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Multimodal Large Language Model (MLLM), besides using visual encoders, there are several works [36, 8] that utilize only linear embeddings to encode visual information. To demonstrate the generality of our method, we first extend it to the FuYu [8] model in this section. We evaluated our methodology using the FuYu [8] architecture, a prominent model that leverages visual information through simple linear embeddings. This approach utilizes a large language model framework without the need for a visual encoder, instead employing linear embedding to process visual information.Our system architecture, based on FuYu [8], incorporates a single decoder where both visual and textual inputs are converted into token embeddings and processed by the same Transformer structure.Inputs are divided into two segments: the first is a sequence of image patches forming a screenshot and image patches from text image, and the second a sequence of textual tokens that contextualize the screenshot. The configuration of this uni-modal approach is depicted in Figure 6 . Input Format. The model inputs are formatted such that a text sequence of m tokens is divided into screenshot segment (m s ) tokens and text segment (m t ) tokens, each comprising 256 tokens.The screenshot dimensions are defined as p h × p w pixels. Special tokens are integrated to guide the model's understanding of segment differentiation.The rendering strategies are consistent with those employed in our visual encoder-based methods.Architecture. The architecture follows the FuYu model [8], a widely used framework among visual encoder-free multi-modal language models (LMs), utilizing a 9 billion parameter model. Image patches are transformed into embeddings through a linear projection, while textual inputs utilize corresponding word embeddings. These embeddings are subsequently processed together in the Transformer blocks. To keep contrastive loss, we compute the average token metric after 3rd layer.The primary objective of this experiment is to assess whether autoregressive screenshot language models (LMs) can accurately interpret text within screenshots using only linear embedding for the rendered text image. In this setup, the screenshot LM processes 256 text tokens derived from the screenshot context along with an additional 25 text tokens.Our training setup follows the Flamingo model, using image-text data sourced from DataComp [26] .For interleaved data, since FuYu does not support interleaved input, we sample one frame and a fixed length of text during pretraining. The model is trained with DeepSpeed Zero 2 optimization [3] and uses fp16 data type. We initialize the Language Model from Persimmon-8B weight.The associated data statistics for pretraining, presented comprehensively in Table 9 , mainly include Datacomp [26] subset, MMC4 [13] , Obelics [14] and Rendered Text [27] . Our research extends beyond the Mistral model, incorporating other language models such as OPT and MoE. The comparative results are summarized in Table 12 . We noted marked improvements in performance across all models with the use of more in-context examples. This indicates that VisInContext's effectiveness is not highly dependent on the specific language model used, showcasing the broad applicability and robustness of our methodology.In this experiment, we present examples to demonstrate our method. As shown in Figure 7 , we provide samples from the validation sets of DocumentVQA [15] and ChartQA [62] .Using VisInContext, we observe that the method answers questions more accurately, even when the font is unclear. For instance, consider the first pdf image have low resolution.Data Collection We retrieve 1,000 samples from the OBELICS dataset, each sample consisting of six segments in the fixed order: I 1 , T 1 , T 2 , I 2 , T 3 , T 4 . Each image has one matching text segment and additional surrounding text. We use relative positioning to indicate which text is matched and which is surrounding.Retrieval Details To perform the retrieval task, we incorporate contrastive loss during pretraining, following the approach of CosMo [25] . We add a contrastive head for the uni-modality text and vision embeddings. Using the mean of the text and image features as a query, we retrieve the next image or text segment. This task tests the model's ability to handle long in-context text.We compute the dot product similarity and rank the scores to determine the final result. When processing surrounding text as "Raw Text," we concatenate T 1 and the surrounding text R 1 directly to the LLM to obtain the text embedding. We then use the mean of this text embedding and the image embedding of I 1 to retrieve the next image or text.For our method, we render the surrounding text R 1 into an image and use the sum of two resampler outputs as the image embedding. We compute the mean of this image embedding and the T 1 embedding and use this mean vector to retrieve the next image or text segment. In this work, we primarily follow the methods from the Flamingo series [1, 14, 9] , as these provide comprehensive support for in-context pretraining.For zero-shot evaluation, the input sequence is formatted as follows:<Visual><Question><Answer> For few-shot evaluation, such as a two-shot evaluation, the input includes two in-context examples. The sequence then becomes:<Visual1><Question1><Answer1><Visual2><Question2><Answer2>For the text-only input sequence in a text-to-image few-shot setting, the format is:<Question1><Answer1><Question2><Answer2> <Visual><Question><Answer> Note that we remove all visual tokens to create a longer input sequence, which can then be rendered into a text image for text-to-image few-shot evaluation.One approach involves activating the visual encoder during pretraining, allowing the model to independently learn visual OCR information within the vision encoder.As shown in Table 13 , this method significantly enhances performance in document understanding tasks. However, it also introduces considerable instability during pretraining and requires extended iterations (from 200k to 500k) for convergence. Additionally, it decreases performance in classification tasks. Therefore, we use the frozen visual encoder by default, as the token resampler alone suffices to develop document understanding capabilities.This work introduces VisInContext, a method to enhance token efficiency in multi-modality large language models (MLLMs) by using visual tokens to process extended textual contexts.Positive Impacts: VisInContext can democratize access to advanced NLP technologies by reducing computational resources required for long text sequences. This improvement promotes sustainable AI practices by lowering energy consumption and allows researchers with limited resources to utilize powerful MLLMs for applications in education, healthcare, and content generation.Negative Impacts: Potential negative impacts include the misuse of efficient text processing for spreading misinformation or creating deepfake content. Additionally, reliance on visual tokens may introduce biases if training data is not diverse.Mitigation Strategies: To mitigate these risks, we recommend implementing content moderation, developing ethical AI usage guidelines, and ensuring diverse and balanced training datasets. Continuous monitoring and auditing of AI systems using VisInContext can also help address unintended consequences.In summary, VisInContext offers significant advancements in token efficiency and computational sustainability, but it is essential to consider and address its broader societal impacts responsibly.NeurIPS Paper ChecklistJustification: We have reviewed and followed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).The implementation is from https://github.com/TimDettmers/bitsandbytes. 


Decision: The paper originally received mixed reviews, with concerns including:

1. clarity of presentation
2. motivation and applicability in practice
3. potential limitations and trade-offs due to occluding parts of the image
4. method complexity
5. comparison to additional baselines

Following rebuttal, observing the author-reviewer discussions, the AC got the impression most concerns were addressed, with only some concerns of clarity of presentation remaining for reviewer iV3t who also suggested a constructive solution to resolve the clarity issue in the revised manuscript.

In light of the above, the AC recommends accepting the paper and urges the authors to incorporate all the discussions into the revised manuscript. Especially the constructive suggestion by iV3t.

Reviews: review: paper_summary: paper_summary: The paper introduces Visualized In-Context Text Processing (VisInContext), a novel technique designed to enhance multi-modal learning models by efficiently expanding their in-context text length. This method transforms extensive text into visual tokens, substantially lowering GPU memory consumption and computational requirements during training and inference. Models utilizing VisInContext demonstrate superior performance on standard downstream benchmarks for in-context few-shot evaluation compared to conventional approaches. They also show improved document comprehension, particularly in document QA and sequential document retrieval tasks. An additional advantage of VisInContext is its compatibility with current context-extending techniques, allowing for potential combined applications.
strength_weakness: strength_weakness: - The motivation of “text-only in-context few-shots experiment” is not clear. These experiments appear tailored specifically to validate the proposed method rather than addressing practical applications. In particular, the use of text-only versions of visual question answering (VQA) or image captioning tasks for in-context learning seems questionable. The relevance and applicability of such text-only adaptations of inherently visual tasks in this context require further justification.
- There are some unconvincing parts about token masking. In Line 87~88, the paper says the masking ratio of raw image tokens is 1.0. Then the model does not observe the raw image at all. Or, is the model initialized with OpenFlamingo Model, especially for cross-attention layer and resampler? If so, how does the token masking probability affect the model’s ability to learn text semantics from visual inputs? For my intuition, it seems there could be some trade-off of partly observing (masking) raw-image and learning text-image semantics from rendered text images at the same time.
- Minor :
    - Reference, Figures needs compile check. There are some errors. (Line 159, 521, etc. )
    - The formats of Table 1, 2 need to be more polished
questions: questions: - Table 1 exhibits an interesting trend where the baseline model occasionally shows improved performance with an increased number of shots in certain scenarios. This pattern might be attributed to specific characteristics of the datasets or peculiarities of the classification task at hand. What explanation do the authors propose for this counterintuitive observation?
limitations: limitations: The paper addresses thier limitation.
paper_summary: paper_summary: This paper introduces a method called Visualized In-Context Text Processing (VisInContext) to address the challenge of processing long in-context texts in multimodal learning, which arises due to the high GPU memory and computational costs associated with lengthy textual content. VisInContext converts long textual content into images and uses a visual encoder to extract textual representations, thereby increasing the in-context text length that can be processed effectively. The method is based on a dual-stream encoder model that employs Token Masking and Text-Centric Contrastive Learning (TCCL) to improve the model's ability to learn from the rendered text images, and the paper demonstrates the effectiveness of VisInContext through experiments on various tasks, showing that it outperforms the baseline in terms of performance and inference cost, while also improving the optical character recognition (OCR) ability of the model.
strength_weakness: strength_weakness: 1. The proposed method involves multiple steps, including text rendering, token masking, and contrastive learning, which might add complexity to the implementation. This could be a barrier to adoption for some practitioners.

2. From my perspective, the pipeline of "text -> text-rendering -> LLM" seems somewhat derivative from the initial purpose of LLMs (Large Language Models). That is, one may question whether we truly need such a complicated paradigm. If the answer is yes, then this approach appears more akin to an engineering exercise. Further, the performance may be affected by the selected font type, font size, and even the rendered canvas. 

3. The process of rendering text into images and then processing these images through a vision encoder may introduce new bottlenecks, particularly as the text length increases. The paper could explore these potential limitations or trade-offs in greater detail.
questions: questions: refer to weaknesses
limitations: limitations: My primary concern is the necessity of the "text -> text-rendering" process. Perhaps it would be beneficial to explore a new perspective that focuses on aligning visual and textual information more effectively. However, as we know, image space typically contains much more redundancy than semantic space. For the current version, I am not fully convinced that projecting text into the image space is the optimal approach. Furthermore, rendering text may introduce several additional complications that warrant careful consideration.
paper_summary: paper_summary: The paper proposes a method to increase the context size of multi-modal large language models. The goal is to increase the context with minimal GPU memory for both training and inference as well as floating pointing operations. Finally, the authors show that the method obtains good results on in-context learning.
strength_weakness: strength_weakness: I think that the paper is hard to follow and some parts hard to understand. Please see below for some questions

My other concern is related to the experimental part, that I find a bit weak. The comparison is made against only two existing models and I don't clearly understand the numbers. Firstly, one concern is that there seems to be a degradation in performance with 0 shots in some cases.

Also, probably I am missing something, but let's look at Tab. 1. Wouldn't an increase context size (ICL 256 vs 2048) would allow for more shots to be used? I don't understand what is different in terms of the theoretical context between 0 shot Open-Flamingo and VisInContext. Isn't the same information fed to the model? If yes, then I don't see how the experiment measures the impact of the context size. Or is it just way to feed more information from a particular document as opposed to sub-sampling it? In the later case, I would consider this to be a slightly unfair comparison and I don't really understand what are the benefits of the model if in the zero-shot setup where you have access to a larger portion from the document (as opposed to subsampling) the performance drops quite a lot in some cases (okvqa, vizwiz). This shows that the way that the context is processed is not ideal.
questions: questions: Questions related to lines 75-76. 
1. Both images and text are concatenated to result in a sequence of 256 tokens?
2. Is m chosen so that the concatenation is 256 tokens?
3. What does the corresponding text represent? Based on Fig. 2, `I_i` seems to be an actual image and not an image representing the text, so I am confused. Or the bear image has nothing to do with `I_i`? If it doesn't where do the image tokens `I_i` are used in Fig 2?

Questions lines 81-82.
4. Is this the same m as above? If yes, can you elaborate what the m represents? I assume no, since it's `M` vs `m`, but I think other letters can be used to make this more clear
limitations: limitations: The limitations are briefly discussed
paper_summary: paper_summary: Extending context length is a fundamental challenge for LLMs. Unlike previous approaches that focus on efficiently handling text tokens, this paper introduces a novel method: encoding lengthy text information into image renderings. These image tokens effectively increase the context length and enhance the performance of LLMs across various downstream tasks.
strength_weakness: strength_weakness: 1. Information loss from image rendering

The current approach has two potential sources of information loss: 1) rendering long texts into an image, and 2) encoding an image into MLLM embeddings. This loss is not thoroughly investigated in the paper. For instance, Table 2 shows that using the original lengthy 426 tokens outperforms the proposed rendered image, albeit at higher computational costs. While it is acceptable to trade some performance for efficiency, the trade-off should be clearly demonstrated in the paper.

---
2. Optimal compression rate

Due to the potential information loss, rendering very long texts into a single high-resolution image might not be optimal. For a context of, say, 2048 tokens, what is the best approach: a single image with 2048 words, or 32 images with 64 words each? The current ablation study only examines the rendering aspects like font size and font interval threshold. However, the trade-off between the number of images and the number of words per image is a crucial study that should be included in the paper.

---
3. Comparison with other long context methods

A significant weakness of the paper is the lack of comparison with other long context methods. Specifically, the paper must compare the proposed compression-into-image approach with compression-into-text approaches, such as [1-3]. Indeed, text compression loses the original semantics, while image rendering retains all previous words. Therefore, combining both approaches—compressing the semantics first and then rendering them into an image—could potentially offer the best of both worlds.

In addition to the text compression approach, it would be beneficial to discuss the pros and cons of this work in comparison with other long context methods. The current paper only discusses classic efficient self-attention models like Longformer. However, there are more recent and diverse approaches, such as handling long sequences by dividing them into multiple chunks [4-5] or using interpolation positional encodings [6-7], among others.

[1] Mu et al. Learning to Compress Prompts with Gist Tokens. NeurIPS 2023.\
[2] Chevalier et al. Adapting Language Models to Compress Contexts. EMNLP 2023.\
[3] Ge et al. In-context Autoencoder for Context Compression in a Large Language Model. ICLR 2024.\
[4] Bertsch et al. Unlimiformer: Long-Range Transformers with Unlimited Length Input. NeurIPS 2023.\
[5] Song et al. Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs. ICLR 2024.\
[6] Chen et al. Extending Context Window of Large Language Models via Positional Interpolation. arXiv 2023.\
[7] Li et al. Functional Interpolation for Relative Positions improves Long Context Transformers. ICLR 2024.
questions: questions: This paper claims that long texts can be converted into images. If that's the case, do we need text tokens at all? Can we replace all the text with images, instead of just some prefixes as is currently done?
limitations: limitations: Discussed, but preliminary. The paper only addresses a minor technical limitation regarding static vs. dynamic tokenization of images. However, there are many more potential limitations to consider. For example, I wonder if the current approach would also be effective for larger models, such as Llama-3, which has a longer context length of 8192.

Comments: title: Final rating
Response to the rebuttal
Response to the Rebuttal
Rebuttal answer
comment: Thank you very much for your thoughtful review and for taking the time to reassess our paper during the rebuttal!
Thanks for the prompt response.  I suppose authors made a good rebuttal and all my concerns have been solved properly. I thus raise my score to WEAK ACCEPT.
Contrastive learning in our method is designed to make the vision encoder and resampler work together as a `"visual text tokenizer."` This means it encourages the embeddings of rendered text images to align with those of regular text tokens, allowing them to capture similar overall meanings. With contrastive learning, we observe:

**Improved Next Word Prediction Accuracy:** We observed that models trained with contrastive learning `perform better in predicting the next word on the Rendered Text Image dataset`. This improvement indicates that the model has a better understanding of text and stronger OCR (Optical Character Recognition) capabilities. For example, in Figure 3 the validation OCR accuracy (val_ocr_average_acc1) drops significantly from 85.25 to 74.67 when contrastive learning is removed.

**Enhanced Performance on TextVQA:** According to Table 6, the VisInContext model with contrastive learning shows a significant improvement on the TextVQA dataset, with accuracy increasing from 18.3 to 21.8. This dataset requires the model to `read and understand text within images to answer questions`, highlighting the model's enhanced ability in text-based reasoning when contrastive learning is applied. 

One potential limitation of the model with contrastive loss is its tendency to struggle in scenarios `where the image contains a significant amount of irrelevant or misleading text`, which can lead to incorrect interpretations. To illustrate this, we conducted a simple experiment using images with false text from the Typographic Attack dataset, as discussed in Section 7.2 of [1]. In this experiment, we posed a basic QA task asking, 'What is in this image?' The results showed that **the model with contrastive loss exhibited a higher language model (LM) loss**. For example, when presented with an image of a cup labeled with the misleading text 'iPad,' the model ` incorrectly responded with 'A blue iPad.'`

Considering we are unable to provide figures at this stage to aid understanding, we believe these examples effectively demonstrate that contrastive learning is particularly beneficial in cases where understanding text within images is essential.

[1]. Joanna et al, Disentangling visual and written concepts in CLIP,  CVPR'22
Ok, I got the context. Nevertheless, I think the manuscript should be clearer. Overall, I still think the idea of using rendered text is interesting and underexplored, despite the debate on the necessity or complexity of using rendered texts. Therefore, I will keep my initial rating.
Thank you for the explanation. I have reviewed the response and comments raised by other reviewers. In my initial comments, the term "derivative" was actually meant to be "deviated". Apologies for the typos.

I still have some confusion about the exact helpful clues or information captured by contrastive learning. From the results, I can observe that it truly brings improvement. However, I am wondering what kinds of cases can be improved with and without the TCCL approach. It would be helpful if you could provide some comparison examples, such as cases where TCCL leads to significant improvements versus cases where it does not provide as much benefit. Concrete examples illustrating the strengths and limitations of the TCCL method would give me a clearer understanding.
Dear reviewer zUCF,

the discussion period draws to a close, could you please check and reply to the authors' response? Please also revise your score accordingly if needed.

Sincerely,
your AC.
The masking ratio of 1.0 mentioned in Lines 87-88 `differs from` the predefined probability of 50% referenced in Lines 107-110. 

The predefined probability of 50% in Lines 107-110 indicates that, during pre-training, there is a **50% chance that the raw image tokens will be masked**. 

The masking ratio of 1.0 in Lines 87-88 refers to the `specific implementation of masking`, where all raw image tokens are masked.

The purpose of Lines 87-88 is to address the issue where combining tokens from raw images and text images directly caused the network to overlook the text-image input (as mentioned in Lines 105-107).
I still don't get it. The masking ratio is set to 0.5 for pretraining, and no tokens are masked during downstream evaluation. Then, why do we need lines 87~88, where the masking ratio is set to 1.0, if we don't face the situation where "the model is not supposed to see them at all"?
Hey! Sorry for not being more clear. I am referring to the original questions where I think for most of them, parts of the response from the rebuttal need to be included in the revised paper for added clarity. For example the explanation around Table 1, parameter m and the other parts.
Thanks for your timely response! The term "Training" in lines 87-88 refers to the **application of token masking exclusively during the pre-training stage**.
During the pre-training stage, token masking is applied to ensure the LLM sees only the raw image 50% of the time. For the remaining 50%, the LLM processes both the rendered text image and the raw image together.
This approach _prevents the model from adopting a trivial solution that completely disregards the rendered text image_.

During `downstream evaluation tasks, no tokens are masked`. For instance, in the in-context text-only few-shot setting in Table 2, where rendered text images are available, the raw image token remains unmasked. 
The vision feature input to the cross-attention model in this scenario is the sum of the raw image token and the rendered text image token. 

For zero-shot evaluation in Table 1, where no rendered text image is present, only the grey-shaded components in Fig. 2 are retained.

I hope this clears up any confusion. Please let me know if further clarification is needed.
Thanks for the response. I carefully read through the authors' responses and the discussions among the other reviewers. While most of my initial concerns have been addressed, I still have an unclear part, which is related to the token masking. In lines 87-88, the paper states that "which ensures that the model won’t simply be ignoring the text images during training." However, according to the rebuttal, "a 1.0 masking ratio" is needed for the situation when the model is not supposed to see the raw image tokens, such as text-only in-context few-shot, or zero-shot evaluation. The word "training" in line 88 confuses me. Could you elaborate on this more, please?
Thanks for your timely and positive comment!

We want to emphasize that the idea of **challenging the common belief about image redundancy** is `not our primary claim` but rather an interpretation by the other reviewer.  Our discussion is focused solely on rendered text images, which consist of white backgrounds and dark text, rather than real-world images.

Also, we `do not claim to replace text tokenizers entirely`. Instead, our work explores how rendered text images can be utilized within the scope of multimodal large language models (MLLMs) to increase in-context text length in novel ways.
Thank you for your prompt feedback and for raising the score! We are glad that our rebuttal brings more clarity and answers all your questions.  We would love to resolve any remaining clarity problems, if you could elaborate more about “there might be a problem of clarity”.
Thank you for the rebuttal. As other reviewers noted, this paper challenges the common belief that "image space typically contains more redundancy than semantic space" and argues that images can serve as better tokenizers than texts. Given the ambitious nature of this claim, it may be more challenging to convince people.

However, the proposed concept of rendering images as text opens up several intriguing research directions. For example: 1) What is the optimal way to tokenize the combination of images and text? 2) How can we effectively train large models with these tokens, considering the challenges in next-token prediction?

For these reasons, I believe this paper could make a valuable contribution to the conference, sparking new ideas among attendees, and I am inclined to maintain my original rating of acceptance.
Thank you for the rebuttal! I confirm I have read the rebuttal and I currently don't have other questions. The rebuttal brings more clarity and answers my questions, but I still think there might be a problem of clarity. Hence, I slightly raise my score.