Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning
Authors: Alex Jinpeng Wang
Linjie Li
Yiqi Lin
Min Li
Lijuan Wang
Mike Zheng Shou
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modality
In-context Learning
Vision-Language
Large Language Model
Primary area: machine_vision
TL;DR: We present Visualized In-Context Text Processing(VisInContext), which processes long in-context text using visual tokens.

Abstract: Training models with longer in-context lengths is a significant challenge for multimodal machine learning due to substantial GPU memory and computational costs. This exploratory study does not present state-of-the-art models; rather, it introduces an innovative method designed to increase in-context text length in multi-modality large language models (MLLMs) efficiently. We present \ModelFullName (\ModelName), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs). For instance, our method expands the pre-training in-context length from 256 to 2048 tokens with fewer FLOPs for a 56 billion parameter MOE model. Experimental results demonstrate that \ModelName enhances OCR capabilities and delivers superior performance on common downstream benchmarks for in-context few-shot evaluation. Additionally, \ModelName proves effective for long context inference, achieving results comparable to full text input while maintaining computational efficiency.

Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning
Alex Jinpeng Wang', Linjie Li', Yiqi Lin', Min Li Lijuan Wang", and Mike Zheng Shou 2 Show Lab, National University of Singapore Miicrosoft *Central South University
Abstract
Training models with longer in-context lengths is significant challenge for multi- modal machine learning due substantial GPU memory and computational costs. This exploratory study does not present state-of-the-art models; rather, it introduces innovative method designed increase in-context text length in multi-modality large language models (MLLMs) efficiently. We present Visualized In-Context Text Processing (VisInContext), which processes long in-context text using visual tokens. This technique significantly reduces GPU memory usage and floating point operations (FLOPs) for both training and inferenceing stage. For instance, our method expands the pre-training in-context text length from 256 to 2048 tokens with nearly same FLOPs for a 56 billion parameter MOE model. Experimental results demonstrate that model trained with VisInContext delivers superior perfor- mance on common downstream benchmarks for in-context few-shot evaluation. Additionally, VisInContext is complementary to existing methods for increasing in-context text length and enhances document understanding capabilities, showing great potential in document QA tasks and sequential document retrieval. The code is available at https:/ / github. com/ showlab/ aaallla///iiiiii InContext..
Introduction
Large Language Models (LLMs), such as OPT, Mistral, and LLaMA-2 [4, 5, 6], have significantly advanced the field of Natural Language Processing (NLP). These advancements are partly due the increased capability of LLMs to process long contexts, from 512 tokens [7] to 16K tokens [6]. Building on these developments, recent multi-modal learning research [1, 8, 9, 10] has shifted focus from simple image-text pairs, like those in CC3M [11] and LAION-400M [12]. to more complex and lengthy interleaved document datasets. Examples include web corpora like MMC4 [13] and the OBELICS [14] dataset, as well PDF corpora like Doc [15].
However, training models on these complex datasets presents significant challenges due to the increased GPU memory and computational demands of extended contexts. For instance, while processing just 5M data items from MMC4 and 10M from the OBELICS dataset, OpenFlamingo- 9B [9] resorted to sub-sampling text and processing only 256 tokens at time, yet it still requires 32 80GB A100 GPUs for over three days. This highlights the need for more computation-effin efffcciin methods to handle long context lengths effectively.
In the domain of LLMs, popular methods to extend context length are the use of memorizing banks [16] and novel self-attention mechanisms [17, 18]. These methods have inspired advancements in the multi-modality domain well. For example, the Large World Model [19] introduces Ring Attention [18], and MA-LMM [20] employs memory banks to process long video understanding tasks. While these techniques have shown promise, our approach aims increase in-context text length
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

GPU Memory Usage In-Context Text Length
TFlops vs. In-Context Text Length
Original
11
Original Method
In-Conest Text Length tlog? scale)
Our Method
In-Context Text Length (log2 scalel
Our Method
Nvidia H100 GPU Memory
VisInConted
VistnContext
GPU Memory Usage (GB)
Floos (TFLOPS)
(a) GPU memory consumption.
(b) The flops comparison.
Figure VisInContext significantly increases the in-context text length from to 2048 during pre-
training on NVIDIA H100 GPU. For our method, we incorporate VislnContext after 128 text tokens. We implement PyTorch Flamingo [1] models with different in-context length during pre-training The language model is 56B MOE [2] model loaded with 4-bit quantization and the batch size each GPU 32 with FP16. We train the model with DeepSpeed [3] Zero-2.
by leveraging the strengths of visual encoders in MLLMs. We first observe that existing MLLMs usually exploit a much lighter visual encoders, compared its text decoders. For instance, Flamingo-99 9BB consists of a 304.4M ViT-L/16 [21] as image encoder, and 7.1B Chinchilla [1] model as the text decoder. Additionally, previous works [22, 23] have demonstrated that visual encoders trained on paired image-text data also exhibit emergent OCR capabilities.
Motivated by these observations, we propose Visualized In-Context Text Processing (VislnContext), a method that uses visual tokens to process extended textual contexts, which is complementary of existing methods in extending context length. Specifically, we convert long textual content into images and use the visual encoders to extract textual representations. In this way, we can efficiently and effectively enable models with much longer contexts, shown in Figure 1. With VisInContext, we show that the in-context text length can be increased by 7 times over the competing baseline. Additionally, we observe almost the same overall computation FLOPs even as in-context length extends significantly. Our extensive experiments will also show that VisInContext renders superior model performance conventional in-context few-shot evaluations and document understanding, with much lower computational cost.
follows: i. We introduce Visualized In- Context Text Processing (VisInContexi), novel method that increases in-context text length using visual tokens. VisInContext directly compresses text context input-level, which is complementary to existing techniques with improved self-attention or memory banks. ii. We demonstrate that VisInContext effective for both training and inference stage with much lower computational cost, iii. With extended text context brought by VisInContext, our model improves the average in-context few-shot performance from 55.8% to 57.8% over the competing baseline. As byproduct, our method also shows great potential in document understanding on popular document QA tasks and our newly proposed sequential document retrieval task.
Contributions. In summary, contributions
2
Method
The goal of VisInContext is to process in-context text using visual tokens that the model can handle long text context more efficiently. We primarily base our study on Flamingo-based architecture [1, 9, 14], it has shown success in improving model's ability learn from long multimodal context that contains arbitrarily interleaved text and images.
2.1 Terminology
Before diving into model details, define the following terms:

---

Generation:
Cross Attention Layer
Token Masking
Large Language Model
Re-sampler
Re-sampler
TCCL
LM Block
Pooling
Pooling
Vision Encoder
Visson Encoder
Tokenization && Embodding
d
car
<Visual 1>
Vissual
Vissual
Image Rendering
Figure 2: VisInContext Pipeline. The VislnContext pipeline builds upon the Flamingo model for in-context few-shot modeling (represented in gray). VisInContext processes interlcaved image-text data by rendering portions of the in-context text into images. This approach maintains the Text Token Length of the model while allowing for significantly extended In-context Text Length.
In-context Text Length:
The actual length of text tokens observed by the model within document.
Text Token Length: The length of the text sequence input directly to the LLM, corresponding to the token count of this sequence.
With VisInContext, the In-context Text Length is greater than the text token length, as part of the text is represented using visual tokens.
2.2 Overall Architecture
The implementation and architecture of VisInContext are shown in Figure 2. It is based on dual-stream encoder model that integrates both visual and textual data. To effectively handle long interleaved data, we use pre-sampling strategy in Flamingo-style works [1. 9, 14]. We sample m images 41, .2....: I' E with corresponding texts T2,T2,... Tm E T. Tokens are concatenated in the form < visual, >< text, > ... visual, >< textm >, where visual > is a single placeholder token. A random 256-token sequence is then sampled. However, since the overall length of web document is generally much longer than 256 tokens (In-context Text Length Text Token Length), this sampling approach can lead to the omission of a lot of related text context.
To address this issue, we convert these omitted text context into visual signals by rendering them into images. We first concatenate all omitted text segments and divide them into parts to render text images, named T", Ti...,, I' E T'. Both the original images and the text-rendered images are then processed through a shared frozen vision encoder. Then, employ two learnable resamplers to extract a fixed number of tokens from both the raw and text-rendered image features, respectively. To facilitate the model to learn from rendered text images, we introduce two novel model designs, Token Masking mechanism and Text-Centric Contrastive Learning (TCCL). Token Masking allows the model to only read from text image tokens by masking the raw image tokens with masking ratio 1, which ensures that the model won't simply be ignoring the text images during training, hence can learn the association between the rendered text images {T, and the text tokens (Ti). TCCL aligns the visual text representation from the resampler with the embeddings extracted from text tokenizers in LLM, which reduces the gap between our visual text tokens and the text tokens the LLM is trained to perceive. With these designs, VisInContext not only reduces computational demands as evidenced by reduction in flops and inference show in our experiments.
also improves the OCR ability, will
2.3 Text Rendering
This module converts textual data into a visually rich RGB format, specifically rendering the text into an image size of pa x npaa, where n the number patches. We employ the HERSHEY font at a size of 10px. On average, one 16x16 patch accommodates approximately 1.5 OPT text tokens. A 224x224 text image contains about 294 text tokens. Consequently, a visual encoder operating on this rendered text image requires only 1/3 of tokens to encode equivalent amount of text, compared
3

---

to the text tokenizer in language models. The vision encoder is quite lightweight ViT-L (340M) compared language model MOE (56B), which makes the processing of rendered text images significantly more efficient than directly inputting the text into language model.
2.4 Token Masking
In our initial experiments, we find that combining tokens from raw images and text images directly led to the network disregarding the text-image input. To address this issue, we introduce Token Masking strategy to force the model to learn text semantics from visual inputs. During pretraining, the raw image and text image are first encoded into the same number of tokens after resampler, and then we mask the raw image tokens with pre-defined probability. When masking out the raw image tokens, the model can focus on learning the association between rendered text images and the complementary text tokens. At inference time, we add the exxte tokens and image tokens together, to allow the model effectively leverage information from both sources.
2.5 Text-Centric Contrastive Loss (TCCL)
Given that the vision encoder, typically a frozen Vision Transformer (ViT) [24], never observes rendered text images during pretraining, may struggle derive text semantics from pixels. To mitigate this issue, we introduce a new training objective, Text-Centric Contrastive Loss (TCCL). This objective aims to guide the resampler rendered text images interpret visual representations of text with a proficiency comparable to traditional text tokenizers, so that the textual semantics can be effective extracted from the rendered text images.
Motivation.
Mechanism. TCCL utilizes raw text token embeddings from the text tokenizer as soft supervision signals to supervise the resampler to learn text-centric representation. To reduce the global semantic gap between text image embeddings and text token embeddings, we first aggregate these embeddings with average pooling and then align them with TCCL. Intuitively, TCCL is designed to turn the joint the vision encoder and resampler into a "visual" text tokenizer, as it promotes the text image embeddings share similar global semantic as the text token embeddings. The core of TCCL is formulated as contrastive loss:
(f. Ek=1 exp(sim(fo,) Swx)/T)
Lij = - -logg
Where Lig denotes the contrastive loss for comparing the ith text image against the j text, fv. and ft, represent the feature embeddings of the jth text image and jth text, respectively. T is a parameter that control the sharpness of the output distribution. Note that fu, and fú, are different features extracted from the same text, as the 7t text image is direct rendering of the ith text.
3 Experiment
3.1 Experimental Setup
Pretraining. We validate VisInContext with Open- Olem-Flamingo [9] and CosMo [25]. To enhance computational efficiency, all models utilize floatl6 precision. For the 56B MOE [2] model, we employ DeepSpeed's [3] Zero-2 stage with CPU offloading and further optimize the model by quantizing to 4-bit precision? 1. We also use Flash Attention [17] to further improve memory efficiency. For all other experiments, we train the model using DeepSpeed Zero-2 without CPU off-loading. The Deen-Flamingo 9B bascline is based on Mistral7B [5].
Our pretraining dataset includes a 180M subset of DataComp IB [26]. MMC4 [13], the OBELICS [14]
dataset, and OCR Rendered Text [27]. (More details are provided in the Appendix B. For each input document or image-text pair, we render a text sequence into an image with fixed size of 16x8 192 (512 patches) by default, with pt. = Pw 16.
'The implementation is from https://githuu.comm//ii

---

Method
Text
ICL Tokenst
Shots
VQA
Caption
Classi. Mean
okvqu
lexivqu
vizwiz
vqav2
coco
flickr
HM
Open-
40.2
21.3
23.3
47.8
82.3
59.4
60.4
47.8
Flamingo
Raw Text
256
42.5
22.2
32.2
49.8
90.5
63.5
63.8
52.1
MOE [9]
46.8
23.2
40.5
49.9
98.2
66.2
66.0
55.8
39.5
26.4
26.3
48.5
84,4
60.5
62.2
49.7
+Rendered Image
+VisInContext
2048
4
44.3
28.9
32.0
50.3
94.2
65.3
65.5
54.4
32
46.3
31.2
41.2
51.0
101.3
68.4
65.2
57.8
Table I: Increasing in-context text length with VisInContext significantly improves performance on multi-modality downstream tasks. The model is pre-trained with 56B MOE model. ICL stands for in-context text length. HM is short for hatefulmemes. With VisInContext, we increase the ICL from 256 to 2048, lcading clear improvements over the baseline. indicates our implementation.
Method
Text
Text
T-Shots
VQA
Caption
Mean
Source
Tokens
okvqa
lextvqa
vizwiz
vquv2
coco
flickr
Open- Flamingo9B
10
18.1
14.8
21.5
26.5
40.1
32.1
25.5
Raw
62
23.8
18.1
23.7
40.5
57.5
35.3
33.2(7.71)
Text
Baseline [9]
426
32
25.2
16.4
25.5
34.6
66.1
38.5
34.4(8.91)
10
0
16.2
16.8
15.4
30.6
42.3
33.5
25.8
Rendered
+VisInContext
10
4
17.2
21.8
19.7
35.2
52.4
35.2
30.3(4.51)
Image
10
32
21.3
22.6
21.5
38.8
60.3
37.0
33.6(7.81)
Table 2: VisInContext effectively incorporates in-context text with visual tokens, demonstrating significant performance improvements with consistent token usage. Here, T-shots refer to text-only in-context examples. Tokens indicate the length of the input to the LLM. Text source describes the preprocessing method for in-context examples. denotes our implementation on 180M pretraining data.
Downstream Evaluation. Our objective is demonstrate that in-context length can be extended using visual tokens, thereby enhancing the understanding of complex multimodal documents. Conse- quently, focus primarily tasks related to long-context understanding.
To evaluate the long-context understanding ability, we adopt the few-shot evaluation setting in Flamingo [1]. We report answer accuracy the OK-VQA [28], TexiVQA [29]. VizWiz [30], and VQAV2 [31]. Additionally, we assess performance on captioning tasks using COCO [32] and Flickr30K [33]. Moreover, also propose setting named text-only in context few-shots to explore text-only in-context evaluation. For this setting, we use in-context sampling without visual input to generate long-context inputs and the visual input is not observed by the model.
In order to illustrate the impact of having long in-context text, we evaluate the model for document understanding Voc [15] and OCR VQA [34]. Lastly, we introduce new task, sequential multimodal document retrieval. This dataset is based on the existing interleaved OBELICS [14] dataset. Further details are provided in the Sec. D of the appendix.
3.2 In-context Few-shot Evaluation
Impact of Extended In-Context Text Length. Interleaved document datasets typically contain long texts. For instance, the OBELICS [14] dataset has an average token length of 815 tokens per document. Due to GPU memory constraints, Flamingo-like models [14, 9] only sub-sample 256 tokens during pretraining, which leads to a significant loss of context information. We compare the baseline model pre-trained with 256 tokens, against our method with an increasing In-context VisInContext. For example, on
Text Length to 2048 tokens. Table shows clear advantage TextVQA, accuracy improves from 23.2% to 31.2% with 32-shot Similarly, the average model performance across all datasets show an increase from 55.8% to 57.8%. These findings demonstrate that ContnContext effectively increases the In-context Text Length to improve multi-modality understanding.
Few-shot Evaluation with Text-only In-context Examples. downstream tasks often differ in format from pretraining data, several works [1, 9, 14] have tested the few-shot abilities of models
5

---

Method
Text Source
Doc
OCR VQA
val
test
Open-Plamingo-99 Baseline
Raw Text
45.3
48.2
51.5
+VislnContext
Rendered Image
48.5
52.2(4.01)
58.4(6.91)
Table 3: VisInContext clearly boosting the baseline on document understanding tasks.
67:37
18.76
Odde research
we TislCContett
Trowulan has e
concentraa
rEM9
305
50%
85335
100, 00 Swargg 2015
93.38
won So
ferter soseeiing lonk uyo was The HA
VislnContes
34
100
(4). Rendered Text
(b). Token Prediction Accuracy on Validation
Figure 3: VisInContext significantly improves the OCR ability of LLM. We present the Rendered Text [27] images and the corresponding next-wa word prediction accuracy on the validation set. Using same pre- ur-itrannig steps, VislnContext achieves significantly better results in predicting words visual images, even when the fonts are difficult to recognize.
using in-context examples. For instance, in the VQA dataset, few question-and and-answer pairs provided in-context examples with visual signals. However, for zero-shot evaluation, two question-and-ane pairs are added as in-context examples without visual signals in [1, 9, 14]. Follow the zero-shot setting, we examine the effect of having text-only in-context examples and extend it to multi-shot setting, by leaving out the corresponding images (See Appendix .E for more details). We compare model performance of the baseline Open-Flamingo 9B and our method under the same setting, where the differences in how these text-only in-context examples are processed. Specifically, Open-Flaming directly takes in them as text tokens, while VisInContext takes in the corresponding rendered text images.
Inference GPU Memory Usage vs. In In-Context Text Length Table 2 summarizes the results across four VQA bench-
13
marks and two captioning benchmarks. Notably, compared to the text-only O-shot setting, our VislnContext with 32- shot significantly improved on all VQA and captioning benchmarks considered. Though the 32-shot performance VisInContext is slightly lower than the competing base- line, cut down the input tokens to the LLM from 426 to only Text Token Length, which lead to significant reduc- tion in the inference cost. These outcomes highlight two key points: i. VisInContext can effectively understand text rendered in images. if. Text rendered images be comparably effective as raw text, when used as text-only in-context examples.
Nindia Memory
In-Content Langih
30
GPU Momory Usage (0B)
VisInContext extends the in- context text length MOE based MLLM from 1k to 9k at inference stage.
Figure 4:
Comparison on Inference Cost. We then analyze the inference cost of VisInContext and compare to the baseline. Both models are based on 56B MOE LLM with batch size of one to explore the maximum manageable In-contexi Text Length. The results, shown in Figure 4, demonstrate that the In-context Text Length can extended up to 9192 tokens for the 56B MOE model on 80GB H100 GPUs with our method at inference stage. This result

---

Yj
Hight NOX
MSA 1048 preece an amare tythohhcti proenr Looneel sodecille mendrre frame HOS 122 enchd projits
The
off surfos entis invieed rooed repicaaton and
The
pecuugig cptiaation ge
R,
Vaioo Structure kik
The mae 20 belevs the fethrr forhhess coredir in cieccrad appndtentl
on 25-8 ler
hes menbrree defied bloyer
right
Repeodaciive Dydee
The baltee ay 200) kiluss tnee farinuse Fist the staches bos mamroas respie seutrip and for
eeechcce hes viev meed ura lutta
tompoeed rovers ooneat spece chor diery
roallin siffooed Anserile
R2.
Figure 5: Sequential multi-modal retrieval example. The input sequence is T1, Ri 12, T2, Ra that from
interleaved document in OBELICS [14] dataset.
Visual Input
Text Input
Surrounding Text Input
Seq-I
Seq-T
Raw Image
Raw Text
16.3
64.8
Raw Image
Raw Text
Raw Text
18.9
67.5
Raw Image
Raw Text
Rendered Text Image
22.7
66.5
Table 4: The model pretrain with VisInContext significantly improves sequence understanding ability. We report the sequence retrieval result on OBELICS-Hyybrdd..
highlights the efficiency and advantages of VislnContext, also show its potential in understanding very long document.
3.3 Document understanding
In this section, we evaluate the model on document understanding tasks. Unlike common vision- language tasks that usually short-form pairs, this task requires comprehension of long and complex document data. We evaluate our model on Doc and OCRVQA. All document images are of size 384 x 384. Following Pix2Struct [35], we finetune the model on DocVQA train data and report performance on the average normalized Levenshtein similarity (ANLS) metric.
Results in Table show that our method significantly outperforms the baseline. For instance, we achieve 6.9% improvement on OCRVQA. To further analyze why our method enhances document understanding, we present the validation accuracy of the L.LM on the Rendered Text [27] dataset during pretraining in Figure 3. We observe substantial improvement in next word prediction accuracy, with top-1 accuracy increasing from 67.37% to 85.25% (a 16% improvement) and top-5 accuracy rising from 80.76% 93.38%. These findings indicate that the LLM can effectively understand text embedded in visual signals with VisInContext.
3.4 Sequential Multi-modal Retrieval
In order to further analyze the benefit of having long text context in multimodal modeling, we propose a new task- Sequential Multimodal Retrieval (SMR), based on document data from interleaved OBELICS [14] dataset. The document composed of interleaved data, consisting of images and texts arranged in meaningful sequence.
We show one sample in Figure 5 and define the input and output of this task below: Input: Given
a pair of content items, an image and a corresponding text (7, Ti, R2, I2, T2, R2), from a document D. I Image, T' is the matched text and R is the surrounding text. Output: The task is retrieve the next image 12 and the next text T2 in the sequence. Named as Seq-I and Seq-T, correspondingly.
We sample the first IK documents that contain data like 4, Tı, R1, 12, T2, R2 from OBELICS [14] and named it as OBELICS -ybbidd, which have at least three frames and three texts. (See Sec. E in appendix for more details.) This task encourages the model to leverage the contextual and semantic relationship in interleaved sequences to effectively predict and retrieve the subsequent pair.
To enable our model with retrieval, we follow CosMo [25] to add simple contrastive head between visual embedding and language embedding from the middle layers. Recall that visual embeddings either from raw images rendered images the addition of the two in method. Table 4
7

---

Method
Pretrain Text Source
Task DucVQA-val
Yu9B 18li
Raw-Tex: Texl
42.3
VisenContext
+Rendered Image
44.5 (2.21)
Table 5: Pretraining with VisInContext helps long-context understanding task for FuYu
model. means our implementation on 180M data.
reports the results from our model with several input variants. We observe taking surrounding text input as rendered text image performs much better on the Sequence to image retrieval, while on par on Sequence to text retrieval, when compared with taking surrounding text input text. These results further support the designs of VisInContext in the context of document understanding.
3.5 Extension MLLM with Linear Embedding
Beyond utilizing the visual encoder, some works [36, 8] also employ linear embedding to extract visual features directly from raw images. To show the generality our method, we also explore Fu [8] model baseline and integrate VisInContext into the model. (See Sec. A in the appendix for more details.) As indicated in Table our method is successful improving the performances on VQA dataset that require long-conted understanding.
Text Image
Token Masking
TCCL
Ok- VQA
TextVqa
VizWiz
V2a
11.5
15.3
8.7
24.2
11.3
15.0
9.4
30.1
17.8
18.3
15.3
33,5
13.5
15.3
10.3
30.9
17.2
21.8
19.7
35,2
Table Ablation study of the component in our pipeline for text-only 4-shot example.
Font Size
4
6
8
10
12
Dataset
16
32
TexVQQA
15.4
17.2
18.5
21.8
20.3
TexVVQA DocVQA
21.8
20.5
21.3
VQA
39.8
42.5
45.6
44.3
36.2
Table Font ablation. We report the result on DocVQA val dataset.
Table 8: Font interval thresh ablation. Larger thresh leads to few texts general.
3.6 Ablation Study
Ablations on Model Design. We conduct ablation studies on the following modeling components to demonstrate their effectiveness: Text Image, TCCL, and Token Masking. Results are detailed in Table 6, which reveal two findings: 1. Token Masking crucial for the model to learn from rendered text images. Without Token Masking, the model can only perform comparably to the baseline. Forcing the model to learn text semantics from rendered text images via token masking significantly improves model performance. 2. Utilizing TCCL with Token Masking yields better performance than using Token Masking alone.
Ablations on Font Size and Interval Threshold. As shown in Table 7, optimal performance varies with changes in font size. We found that adjusting the font size impacts performance similarly to altering the patch size-bothh methods effectively increase the contextual information within each patch. We prefer modifying the font size over the patch size because it allows for more intuitive adjustments. Our findings indicate that the model does not need highly detailed understanding of each word to perform effectively.
Another important factor is the font interval threshold. As shown in Table 8, we observed that too-large interval leads to inferior results. This is intuitive because larger threshold results in fewer texts in the rendered text image.

---

4 Related Work
Multimodal Language Models. Current mainstream Multimodal Large Language Models (MLLMs) [37, 38, 22, 39, 40, 41] leverage the capabilitiees of Large Language Models (LLMs) [42. 6] due their strong reasoning abilities, as demonstrated by recent advancements. These models typically adopt one of two primary designs for integrating visual information. The first approach involves the effective adaptation of visual representations, which are acquired via a separate visual encoder, into the text-based LLM framework like CLIP, GIT, and BLIP2 [22, 43, 37]. The repre- sentative method in this category incorporates visual representations into the language model using cross-altention, as seen in the Flamingo series models [1, 9, 14]. Along this line, recently some works like LLa VA [40], EMU2 [44], InternVL [45], DeepSeeker [10], and QWen [41] lead to superior results on multi-modality tasks with supervised finetuning on high-quality data. The second approach uses visual embeddings directly as input "tokens" for the LLMs, bypassing the traditional use of separate visual encoder. This method processes visual patches with linear layer and uses the resulting embeddings as direct inputs to the LLM, as implemented in models like ViLT [36] and Yu [8]. This strategy omits the need for additional visual encoder and simplifies the architecture.
In this work, we adopt the Flamingo [1] architecture as our main baseline for the following reasons: First, the Flamingo model emphasizes in in-context few-shot learning ability and designs comprehensive few-shot evaluation strategies. Second, our focus on extending the in-context text length during pre-training rather than on supervised fine-tuning.
Enhancing Text Understanding through Visual Inputs. Traditional text tokenization processes raw text efficiently, but it faces challenges such as vulnerability to spelling errors and limited cross- lingual transferability [46,47) These issues have prompted the exploration of tokenizer-free models, which aim to improve robustness and facilitate better cross-language applicability. For instance, single spelling error can lead to entirely different tokens using traditional tokenization methods, impacting model performance.
Recent developments have seen innovative approaches like the Pixel model [46], which proposes processing text as an image using both an image encoder and an image decoder. This approach has sparked series of studies that process not only textual data but also images, charts, and tables through a unified visual input system [35, 46, 48, 47]. These models are trained on a diverse array of visual data, such webpage screenshots and user interface images, sourced extensively from the internet. They are specifically designed to handle risually-situate text in an end-to-end manner, offering the potential to support wide range of applications.
Long Context Modeling. The challenge of incorporating more tokens into LLMs is active area of research [49, 50]. Common approaches involve novel self-attention mechanisms [18, 51, 52]. compressed token [53, 54, 55] or memory banks [16]. Some works [56] exploit tensor parallelism or sequence parallelism to reduce memory costs. There also have some works focus on position embedding [57, 58]. In multi-modality research, closed-source models like Gemini [59] and GPT- 4V [60] support long context inference up to millions of tokens. Open-source models such as MA-LMM for Long-Term Video Understanding [20] can process up to one hour of video using a long memory bank. The most relevant work Large World Model [19] extends token length using Ring Attention.
In contrast to these methods, our method utilizes off-the-shelf L.MSs and compresses text tokens into visual tokens for efficient processing. Our method is complementary these existing techniques and can be integrated with them to achieve lower computational cost and longer context length.
5 Conclusion and Limitations
This paper centers on multi-modalle learning and addresses the in- in-context length limitations presented by heavy computational cost of LLMS in MLLMs. Our contribution is novel and efficient method named VislnContext, which enables the model to perceive long text context as rendered text images. Comprehensive experiments show that VisInContext is effective on conventional in-context few-shot evaluations and document understanding, while being much more efficient.

---

One limitation of our method is, currently our method requires processing fixed size image even for brief texts. In future work, we plan to dynamically reduce token counts retaining only non-empty tokens during pre-training. We aim to expand this method to additional tasks and encourage the community to further explore this direction.
with variable image sizes by
Acknowledgement
This research is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG3-RP--022-000)
References
[1] Dean-Baptiste Alayrae, Jeff Donahuc, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-237736, 2022.
[2] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford. Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiiv2401.04000 040888, 2024.
[3] Jeff Raslcy, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 3505-3506, 2020.
[4] Susan Zhang, Stephen Roller. Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2022.
[5] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint
2023.
[6] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0925: 2023.
[7] Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACI-HIL (I), 2019.
[8] Adept AI. Fuyu-8B. https: //yps:///ww... adept ai/blog//uuuu 8b, n.d. Accessed: finsert date here].
[9] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision language models. arXiv preprint arXiv:2308. 01390, 2023.
[10] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepscck-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525, 2024.
[11] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 2556-2565 2018.
[12] Christoph Schuhmann, Richard Vencu, Romain Beaumont, and Kaczmarezyk. Laion 400m: Open dataset of clip filterieeedddee 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
[13] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023.
[14] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander M Rush, Douwe Kiela, et al. Obelies: An open web-scale filtered dataset of interleaved image-text text documents. In seirth- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
10

---

[15] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209. 2021.
[16] Yuhuai Wu, Markus N Rahe, Del.esley Hutchins, and Christian Szegedy. Memorizing transformers. arXiv preprint (xXx::203.0891:1 2022.
[17] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-cfficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 18116644-16333 2022.
[18] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.0188) 2023.
[19] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringatiention. arXiv preprint 2Xxx:2402.08268 2024.
[20] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao. Ashish Shah. Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: cemorr-augmentee large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.0577 05726, 2024.
[21] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. In European Conference on Computer Vision, pages 497-515. Springer. 2022.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR,
2021.
[23] Yiqi Lin, Conghui He, Alex Jinpeng Wang, Bin Wang. Weijia Li, and Mike Zheng Shou. Parrot captions teach clip to spot text. arXiv preprint 2:22222112.14222, 2023.
[24] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[25] Alex Jinpeng Wang. Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. Cosmo: Contrastive streamlined multimodal model with interleaved pre-training. arXiv preprint S.kiv:240110084499 2024.
[26] Alex Fang Samir Yitzhak Gadre, Gabriel IIharco. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108 2023.
2023.
[27] Wendler, Chris. Renderedtext dataset. RenderedText, 2023. Accessed: 2023-05-05.
https://hugginggaaee https///uugggggaoooooooottttssse
[28] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEElevf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[29] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 13266 2019.
[30] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P Bigham. Vizwiz, grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3608-3617, 2018
[31] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-69133 6913, 2017.
[32] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-1 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.

---

[33] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting casiiontt--hraase correspondences for richer image-to-senlence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649. 2015.
[34] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirhan Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
[35] Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming- Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pages 18893-18912. PMLR, 2023.
[36] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: ssioo-and-langueee transformer without convolution or region supervision. In International conference on machine learning, pages 5583-5594, PMLR, 2021.
[37] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-inage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[38] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lchter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, ct al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
[39] Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 2022.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[41] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.
[42] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Fric Sigler, Mateusz I Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, lyya Sutskever, and Dario Amodci. I Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[43] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022.
[44] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint 2rXiv:2337.05222. 2023.
[45] Zhe Chen, Jiannan Wu. Wenhai Wang. Weijie Su. Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic lisuul-linguistic tasks. arXiv preprint arXiv:2312.142:: 2023.
[46] Phillip Rust, Jonas F Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliont. Language modelling with pixels. International Conference on Learning Representations, 2023.
[47] Tianyu Gao, Zirui Wang, Adithya Bhaskar, and Danqi Chen. Improving language understanding from screenshots. arXiv preprint arXiv:2402. 14073, 2024.
[48] Michael Tschannen, Basil Mustafa, and Neil Houlsby. Clippo: amage-andd-laguagg understanding from pixels only. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11006-11000 11017, 2023.
[49] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint IrXiv:1944.10000 2019.
[50] Iz Beltagy, Matthew E Peters. and Arman Cohan. Longformer: The long- documett transformer. arXiv preprint arXiv:2004.05 2020.
12

---

[51] Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unlimiformer: Long-range trans- formers with unlimited length input. Advances in Neural Information Processing Systems, 36, 2024.
[52] Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jachyung Kim, Sukmin Yun, Jung-Woo Woo Ha, and Jinwoo Shin. Hierarchical context merging: Better long context understanding for pre-trained Ilms. arXiv preprint arXiv:2404. 10308.. 2024.
[53] Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression a large language model. arXiv preprint arXiv:2307.0694 06944, 2023.
[54] Jesse Mu, Xiang Li. and Noah Goodman. Learning compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024.
[55] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models compress contexts, arXiv preprint arXiv:2305.14755 14788 2023.
[56] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Moham- mad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5, 2023.
[57] Shanda Li. Chong You, Guru Guruganesh, Joshua Ainslie. Santiago Ontanon. Manzil Zaheer. Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint 2rXiv:2310.04411. 2023.
[58] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023.
[59] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai. Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
[60] OpenAL. Gptv system card, 2023. Accessed: 2024-05-22.
[61] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212. 12017, 2022.
[62] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question ans weriinn about charts with visual and logical reasoning. arXiv preprint arXiv: 10244, 2022.
13

---

A Extending VisInContext to MLLM Using Only Linear Embedding
In Multimodal Large Language Model (MLLM), besides using visual encoders, there several works [36, 8] that utilize only linear embeddings to encode visual information. To demonstrate the generality of our method, first extend it to the FuYu [8] model in this section. We evaluated methodology using the FuYu [8] architecture, a prominent model that leverages visual information through simple linear embeddings. This approach utilizes large language model framework without the need for visual encoder, instead employing linear embedding to process visual information.
A.1 Methodology
Our system architecture, based on FuYu [8], incorporates single decoder where both visual and textual inputs are converted into token embeddings and processed by the same Transformer structure. Inputs are divided into two segments: the first sequence of image patches forming screenshot and image patches from text image, and the second sequence of textual tokens that contextualize the screenshot. The configuration of this uni-modal approach is depicted in Figure 6.
LA
7.
Large Language Model
Figure 6: The main pipeline is based on Fuyu [8]. What's 's different is introduce additional text image. During pre-training. the rendered text image and original image is also alternatively. The DCSE is preserved. show one image-text pair here for simplicity.
Tokescatton & Limbedding
Bendered ista Image
Input Format. The model inputs are formatted such that a text sequence of m tokens is divided into sereenshot segment (m,) tokens and text segment (m;) tokens, each comprising 256 tokens. The screenshot dimensions are defined as ps. Pw pixels. Special tokens are integrated to guide the model's s understanding of segment differentiation.
The rendering strategies are consistent with those employed in our visual encoder-based methods.
Architecture. The architecture follows the Fu model [8], widely used framework among visual encoder-free multi-modal language models (LMs), utilizing 9 billion parameter model. Image patches transformed into embeddings through linear projection, while textual inputs utilize corresponding word embeddings. These embeddings are subsequently processed together in the Transformer blocks. To keep contrastive loss, compute the average token metric after 3rd layer.
A.2 Evaluation Settings
The primary objective of this experiment assess whether autoregressive screenshot language models (LMs) can accurately interpret text within screenshots using only linear embedding for the rendered text image. In this setup, the screenshot LM processes 256 text tokens derived from the screenshot context along with an additional 25 text tokens.
A.3 Training Settings
Our training setup follows the Flamingo model, using image-text data sourced from DataComp [26]. For interleaved data, since Yu does not support interleaved input, we sample one frame and fixed length of text during pretraining. The model is trained with DeepSpeed Zero 2 optimization [3] and uses fp16 data type. We initialize the Language Model from Persimmon-8B weight.
B Pretraining & Downstream Task Evaluation details
B.1 Pretraining Data Details
The associated data statistics for pretraining, presented comprehensively in Table 9, mainly include Datacomp [26] subset, MMC4 [13], Obelics [14] and Rendered Text [27].
14

---

Data Type
Dataset
Sample
Image-Text
DataCompl [26]
108M
MMC4 [13]
30M
Interleaved Image-Text
Obelics [14]
30M
Synthetic Data
Rendered Text [27]
12M
Total
180M
Table 9: Statisties of the Pre-training Dataset: Subsets random sampling.
The comparison between our method and Open-flaming baseline, utilizing equivalent-scale pre- training data, consistently demonstrates the superior performance of our approach across diverse tasks,
VinInCantext
VinInContext
VinInContext
3B
Language Model
OPT-IML-L.BB [61|
Mistral-78 7B
MOE
Model
Vision Model Backbone
openai/clip
spenai/clip
1a1on/CLIP-VI1
-vit-large
-vit-large
-B-16-1sion28
-patch14
-patchid
-e32B-b79K
Crusss Layer Interval
4
Test Sequence Lengh
128
128
128
Training
ICL Text Length
2048
Effective
1536
768
Max Training Steps
200K
200K
500K
Weight Decay
0.1
0.1
0.1
Optimizet
adamw((.99 0.999)
adamw((099. 0.999)
adamw(0.9, 0.999)
Gradient Chippng
1.0
1.0
1.0
Initial Max
5e-5
3e-5
3e-5
Learning Rate
Decay Schedude
Cosine
Cosine
Cosine
Linear warmup Steps
5000
5000
5000
Table 10: The hyperparameters used in pre-training for three distinct VisInContext variations. The learning rate and batch size is smaller for sine the GPU memory limitation 32GB.
B.2 Hyperparameter Configuration
In this subsection, we outline the essential training details required for reproducibility. Our exper- iments included three different model sizes, with larger models requiring smaller batch sizes due to GPU memory limitations. We employed DeepSpeed ZeRO-2 optimization with fp16 precision and adjusted gradient accumulation steps to match the data type count. Comprehensive results are presented in Table 10.
The text input sequence length is 128 as default. Since we adopt VisInContext to increase the in-context length, the ICL length can be increased to 2048, around 15 text images. The 7 for TCCL 0.07.
B.3 Parameter Details
The Flamingo [1] baseline include Resampler, Language Model, Visual Encoder, Cross-attention. Both the Language Model and Visual Encoder are frozen during pretraining. We mainly train the Gated Cross Attention layer and Resampler layer.
15

---

Model
Language
Vision
Gated Cross Attention
Resampler
Flamingo-9B
7.1B
428M
0.8B
194M
Flamingo-9B Baseline
7B
307M
0.35B
194M
MOE Basclinet
56B
307M
0.5B
194M
Table 11: Parameter counts for each component in MLLM. means our implementation.
Method
Text Source
Parameter
Text Tokens
T-Shots
VQA
Mean
ok-vqa
textvqa
vizwiz
vgav2
11.2
15.8
5.4
33.6
16.5
OРT- 1.3B [4]
Rendered Image
1.3B
10
17.2
21.8
7.8
33.2
20.0 (3.51)
10
32
21.3
22.6
11.5
35.8
22.8(6.31)
10
0
28.5
23.2
24.4
37.7
28.5
Rendered
MPT
30.1
28.4
40.3
30.5(2.01)
Image
32
32.5
25.4
30.3
41.8
32.5(4.0f)
Table 12: VisInContext performs well over different Language Models.
B.4 Extension to Other Language Models
Our research extends beyond the Mistral model, incorporating other language models such as OPT and MoE. The comparative results are summarized in Table 12. We noted marked improvements in performance across all models with the use more in-context examples. This indicates that VisInContext's effectiveness is not highly dependent the specific language model used, showcasing the broad applicability and robustness of our methodology.
C Document Understanding Example
In this experiment, present examples demonstrate our method. As shown in Figure 7, provide samples from the validation sets of DocumentVQA [15] and ChartQA [62].
Using VisInContext, observe that the method answers questions more accurately, even when the font unclear. For instance, consider the first pdf image have low resolution.
Sequential Multi-modal Retrieval Details
Data Collection We retrieve 1,000 samples from the OBELICS dataset, each sample consisting of six segments in the fixed order: 41. Ti. T2, 12, T3. T4. Each image has one matching text segment and
additional surrounding text. We use relative positioning indicate which matched and which is surrounding.
To perform the retrieval task, incorporate contrastive loss during pretraining, following the approach CosMo [25]. We add a contrastive head for the uni-modality text and vision embeddings. Using the mean of the text and image features as query, retrieve the next image or text segment. This task tests the model's 's ability to handle long in-context text.
Retrieval Details
We compute the dot product similarity and rank the scores to determine the final result. When processing surrounding text as "Raw Text," we concatenate Tj and the surrounding text R₁ directly to the LLM to obtain the text embedding. We then the mean of this text embedding and the image embedding of I, to retrieve the next image or text.
For our method, we render the surrounding text RS into an image and use the sum of two resampler outputs as the image embedding. We compute the mean of this image embedding and the Ti embedding and use this mean vector retrieve the next image or text segment.
16

---

Consumer Dynamics
Q: What percentage share the 21-25 segmen!?
cor
Baseline: 64 #VisInContext:
Seaoed prange addonigee
Condact aulier
coweeppent nukertion ceoeee
aechesniceeees mosee yodke anteriee
Your Cumpar now has 100 kutals curss lccations in tidka,
opnorig og undor buance ICC kitte loe end.
Weiccm/eeeteee' herilage
Q: What is the brand hamee of the five star catogory hotels?
Q: Which country is represented by red line?
Baseline: The rod line Peru
Baseline: WelcomHeeriagee +VislnContext: Welcoml
+VisInContext: Kenya
Figure The document understanding example of our method.
E In-Context Few-Shot Example
In this work, we primarily follow the methods from the Flamingo series [1, 14, 9], as these provide comprehensive support for in-context pretraining.
For zero-shot evaluation, the input sequence is formatted as follows:
WVisual)><Queeccioo>><sssee
For few-shot evaluation, such as a two-shot evaluation, the input includes two in-context examples. The sequence then becomes:
off.aaaiieeeetttoooeeeeeeeliiilt Viisual>><qusstioox><<aeeer
For the text-only input sequence in text-to-image few-shot setting, the format is:
ocuettioatb.ccaarrrrrxxsssssssa2 <Tisual><Qquetttion><Aaeee
Note that we remove all visual tokens create a longer input sequence, which can then be rendered into a text image for text-to-image few-shot evaluation.
F Activating the Visual Encoder
One approach involves activating the visual encoder during pretraining, allowing the model to independently learn visual OCR information within the vision encoder.
As shown in Table 13, this method significantly enhances performance in document understanding tasks. However, it also introduces considerable instability during pretraining and requires extended
17

---

Method
DocVQA
OCR VQA
Classification Hatefulmems
test
Frozen
48.5
52.2
58.4
61.3
Learnable
50.3(1.11)
54.0(1.81)
59.0(0.61)
59.4(1.94)
Table 13: The impact of opening visual encoder during pre- me-training.
iterations (from 200k 500k) for convergence. Additionally, it decreases performance classifi- cation tasks. Therefore, we use the frozen visual encoder by default, as the token resampler alone suffices to develop document understanding capabilities.
G Boarder Impact
This work introduces VisInContext, a method to enhance token efficiency in multi-modality large language models (MLLMs) by using visual tokens to process extended textual contexts.
VisInContext can democratize access to advanced NLP technologies by reducing computational resources required for long text sequences. This improvement promotes sustainable Al practices by lowering energy consumption and allows researchers with limited resources to utilize powerful MLLMs for applications education, healthcare, and content generation.
Positive Impacts:
Negative Impacts: Potential negative impacts include the misuse of efficient text processing for spreading misinformation or creating deepfake content. Additionally, reliance on visual tokens may introduce biases if training data is not diverse.
Mitigation Strategies: To mitigate these risks, recommend implementing content moderation, developing ethical Al usage guidelines, and ensuring diverse and balanced training datasets. Contin- monitoring and auditing of AI systems using VislnContext also help address unintended consequences.
In summary, VisInContext offers significant advancements in token efficiency and computational sustainability, but essential consider and address its broader societal impacts responsibly.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Reflected in Section contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This paper do not propose new theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The reproduce details are already included in the experiment sections and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We provide the code in supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wttpccccpppcccaeeccccccccccccclt Sondei/eeadeeeeeeeuubiiiiiii ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The details are reported in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not include error bars the experiments computationally expensive, and existing literature do not have the convention to report error bras.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss the details in the first subsection of experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : htt////eerrppp.eee wwwpuripp.ccppuulllaaeee htte/publcc/Eeei wewtiiisseeeelnnenen ines?
Answer: [Yes]
Justification:
We have reviewed and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in Sec G of appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We expect our model to have similar risk as the pre-trained language model and Flamingo model. We discuss potential mitigation strategies in Appendix G.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cited all code, data, and models used in this paper.
Guidelines:
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Included code in supplementary material.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work do not include human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "5rRIqFlW9A", "reviewer": "Reviewer_Vzob", "summary": "The paper introduces Visualized In-Context Text Processing (VisInContext), a novel technique designed to enhance multi-modal learning models by efficiently expanding their in-context text length. This method transforms extensive text into visual tokens, substantially lowering GPU memory consumption and computational requirements during training and inference. Models utilizing VisInContext demonstrate superior performance on standard downstream benchmarks for in-context few-shot evaluation compared to conventional approaches. They also show improved document comprehension, particularly in document QA and sequential document retrieval tasks. An additional advantage of VisInContext is its compatibility with current context-extending techniques, allowing for potential combined applications.", "questions": "- Table 1 exhibits an interesting trend where the baseline model occasionally shows improved performance with an increased number of shots in certain scenarios. This pattern might be attributed to specific characteristics of the datasets or peculiarities of the classification task at hand. What explanation do the authors propose for this counterintuitive observation?", "limitations": "The paper addresses thier limitation.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The suggested method presents an innovative and intriguing concept: converting text into visual representations to decrease computational expenses.", "weaknesses": "- The motivation of “text-only in-context few-shots experiment” is not clear. These experiments appear tailored specifically to validate the proposed method rather than addressing practical applications. In particular, the use of text-only versions of visual question answering (VQA) or image captioning tasks for in-context learning seems questionable. The relevance and applicability of such text-only adaptations of inherently visual tasks in this context require further justification.\n- There are some unconvincing parts about token masking. In Line 87~88, the paper says the masking ratio of raw image tokens is 1.0. Then the model does not observe the raw image at all. Or, is the model initialized with OpenFlamingo Model, especially for cross-attention layer and resampler? If so, how does the token masking probability affect the model’s ability to learn text semantics from visual inputs? For my intuition, it seems there could be some trade-off of partly observing (masking) raw-image and learning text-image semantics from rendered text images at the same time.\n- Minor :\n    - Reference, Figures needs compile check. There are some errors. (Line 159, 521, etc. )\n    - The formats of Table 1, 2 need to be more polished", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The motivation of “text-only in-context few-shots experiment” is not clear:**\n\nThe motivation behind the “text-only in-context few-shots experiment” aligns with the common practices in mainstream few-shot learning architectures like Flamingo, IDEFICS, and EMU2[1]. These models often include `text-only prompts for zero-shot evaluation`.\n\nFor example, in a typical zero-shot evaluation setting, the input to the model is structured as __<text 1><text 2><text 0><image 0>__, where \"0-shot\" actually involves using 2-shot text-only prompts (Lines 170-173). This design helps bootstrap the output format, enabling the model to `produce answers that follow the style of the prompts`. Follow this line, we test the longer prompt beyond 2-shot text data.\n\nSo the primary reasons for using text-only prompts in these experiments are:\n1. **Testing Prompt Understanding**: This setup tests the ability of model to clearly understand the prompt and follow instructions.\n2. **Practicality**: It provides a practical advantage when prompt images are difficult to obtain, ensuring the model can still perform well in the absence of visual input.\n\nMotivated by these, we propose to evaluate text-only in-context few-shot and demonstrate that text rendered as images can be comparably effective as raw text.\n\n**Q2. The details about token masking in Lines 87-88:**\n\nThere seems to be a misunderstanding. The vision feature input to the cross-attention model is a sum of raw image tokens and rendered text image tokens. As depicted in Figure 2.\n\nIn our implementation, the `raw image tokens are masked with a pre-defined probability`, which is 50% in our case (details are provided in Lines 107-110). This means the model only observes the raw image tokens half of the time. The reference to a 1.0 masking ratio in lines 87-88 indicates that we mask all the raw image tokens when the model is not supposed to see them at all.\nHowever, the rendered text image tokens remain intact and still observed by the model.\n\nThe rationale behind this approach is that we experimentally found that models _tend to overly rely on raw pixel images, often ignoring the information from rendered text images_. By masking the raw image tokens, we encourage the model to pay attention to the rendered text images, thereby learning text-image semantics more effectively. For inference, we sum raw image token and rendered text image token (Line111-112).\n\n**Q3. Some other minor typos:**\n\nThank you for pointing out these minor issues. We will fix the reference and figure compilation errors (Lines 159, 521) and polish the formats of Tables 1 and 2 to ensure they meet the required standards.\n\n**Q4. Explanation for Baseline Model Occasionally Performing Better:**\n\nAttention to Visual Tokens: For classification tasks, the model might prioritize distinguishing visual tokens over textual information. This could lead to better performance in scenarios where visual cues are more prominent.\n\n` More shots do not always result in better performance.` As seen in related works like Flamingo (Table 1 in this work), it's not uncommon for more shots to occasionally result in worse performance. This variability can be due to the _random selection of example shots from the support sets_. If the selected examples are quite different from the query image-text pair, the model's performance might drop. Therefore, it is normal that the baseline outperforms our work occasionally and we `mainly focus on mean accuracy over all datasets.`\n\nSome works, like RICES [2], analyze this phenomenon and focus on prompt ensembling by retrieving similar samples from the query set to highly improve multi-shot performance, which is not our focus.\n\n[1]. Sun Q, Cui Y, Zhang X, et al. Generative multimodal models are in-context learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14398-14409.\n\n[2]. Yang Z, Gan Z, Wang J, et al. An empirical study of GPT-3 for few-shot knowledge-based VQA. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(3): 3081-3089.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UsZtsYaj0l", "reviewer": "Reviewer_zUCF", "summary": "This paper introduces a method called Visualized In-Context Text Processing (VisInContext) to address the challenge of processing long in-context texts in multimodal learning, which arises due to the high GPU memory and computational costs associated with lengthy textual content. VisInContext converts long textual content into images and uses a visual encoder to extract textual representations, thereby increasing the in-context text length that can be processed effectively. The method is based on a dual-stream encoder model that employs Token Masking and Text-Centric Contrastive Learning (TCCL) to improve the model's ability to learn from the rendered text images, and the paper demonstrates the effectiveness of VisInContext through experiments on various tasks, showing that it outperforms the baseline in terms of performance and inference cost, while also improving the optical character recognition (OCR) ability of the model.", "questions": "refer to weaknesses", "limitations": "My primary concern is the necessity of the \"text -> text-rendering\" process. Perhaps it would be beneficial to explore a new perspective that focuses on aligning visual and textual information more effectively. However, as we know, image space typically contains much more redundancy than semantic space. For the current version, I am not fully convinced that projecting text into the image space is the optimal approach. Furthermore, rendering text may introduce several additional complications that warrant careful consideration.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is in well-written, which makes it easy to understand.\n\n2. VisInContext can reduce GPU memory usage and FLOPs for both training and inference, allowing the model to handle much longer text contexts with lower computational cost.\n\n3. The model trained with VisInContext delivers better performance on common downstream benchmarks for in-context few-shot evaluation and document understanding. \n\n4. The method shows potential in document understanding tasks, as evidenced by the improvements on DocVQA and OCR VQA datasets, and the enhanced next-word prediction accuracy of the LLM on the Rendered Text dataset.", "weaknesses": "1. The proposed method involves multiple steps, including text rendering, token masking, and contrastive learning, which might add complexity to the implementation. This could be a barrier to adoption for some practitioners.\n\n2. From my perspective, the pipeline of \"text -> text-rendering -> LLM\" seems somewhat derivative from the initial purpose of LLMs (Large Language Models). That is, one may question whether we truly need such a complicated paradigm. If the answer is yes, then this approach appears more akin to an engineering exercise. Further, the performance may be affected by the selected font type, font size, and even the rendered canvas. \n\n3. The process of rendering text into images and then processing these images through a vision encoder may introduce new bottlenecks, particularly as the text length increases. The paper could explore these potential limitations or trade-offs in greater detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Complexity of Implementation and Barrier to Adoption:**\n\nIt is not true, and we would like to emphasize that our implementation is not only simple but also easy to adopt for future works.\n\n**i. Text Rendering:** This step is performed during the preprocessing phase on the CPU and can be implemented in OpenCV with just a few lines of code. Specifically, our implementation requires only 5 lines of code.\n\n**ii. Token Masking:** This is a straightforward selection process that can be implemented in a single line of code.\n\n**iii. Contrastive Loss:** The computation of the contrastive loss is performed on the average token representations, which is a standard and easily implemented technique.\n\nOverall, our method is designed to be simple and straightforward, ensuring it is accessible to practitioners without adding undue complexity.\n\n\n**Q2. Derivation from the initial purpose of LLMs; Need for such a complex paradigm; Engineering exercise:**\n\nWe disagree with your assessment. Here are the key points:\n\n**1.Clarification on Purpose:** It is not clear what you mean by the \"initial purpose\" of LLMs. Our focus is on `Multi-modality Large Language Models (MLLMs)`, not solely LLMs. Our VisInContext method significantly increases the in-context text length for MLLMs during both the pre-training and inference stages.\n\n**2. Clarification on why we need such complex paradigm:**  Increasing the in-context text length is a crucial challenge, particularly during pre-training, while most works on LLMs require fine-tuning or post-tuning to support longer contexts. There is no prior work addressing this topic in MLLMs. It is widely recognized that vision encoders are significantly smaller than LLMs. Therefore, processing longer but not highly related text with a visual encoder at a very modest computational cost is an efficient approach. Actually, compared to rendered text images, `text tokenizers in LLM require numerous human-defined preprocessing steps` such as lowercasing, punctuation removal, stop words removal, tokenization, and more—generally involving almost ten steps. In this context, `rendered text images offer a much simpler paradigm` for processing text.\n\n**3. Clarification on engineering exercise:** We use a fixed font and a simple white background. While font size does affect performance, we have already discussed this in our experiments. Our method significantly increases the in-context text length for MLLMs, representing a substantial improvement. This is not merely an engineering exercise but a `strategic enhancement to the model's ability to process long text`. Additionally, it is unclear what you mean by \"engineering exercise\".\n\nIn conclusion, our method addresses a critical need in MLLMs and provides tangible benefits, demonstrating its value beyond a simple engineering exercise.  For MOE based MLLM, we increase the in-context text length from 256 to 2048 during pre-training stage.\n\n**Q3. Potential Bottlenecks of Using a Vision Encoder and Discussion of Limitations:**\n\nWe argue that both the LLM and vision encoder have limitations in terms of processing long text sequence, especially for the purpose of extended context for multimodal understanding. Below we summarize the pros and cons of using vision encoder to process long text sequence.\n\n**1. Size Comparison:** The vision encoder in an MLLM is typically much smaller compared to the LLM itself. \nFor example, the vision encoder is 340M but the LLM is 56B for MOE-based MLLM.\nThis size difference suggests that the vision encoder is a more economical way to process text information.\n\n**2. Efficiency:** Rendered text images can encode more text within the same number of tokens (Lines 98-99), making this method efficient for handling longer contexts.\n\n**3. Complementary Technique:** Our method is complementary to existing techniques for increasing in-context text length in LLMs. It provides an additional means to enhance model performance without replacing current methods. \n\n**4. Limitations and Trade-offs:** We acknowledge potential limitations, such as dynamic token handling, which are discussed in Lines 302-304. However, our experimental results show that the benefits of our approach outweigh this limitation.\n\n**Q4. Concerns about redundancy in image space and the necessity of the \"Text -> Text-Rendering\" Process:**\n\nWe disagree with the notion that projecting text into image space is suboptimal due to redundancy.\n\n**1.Redundancy in Real-World Images vs. Rendered Text Images:** While real-world images often contain redundancy because adjacent patches look similar, include colors, textures,  background noise, which can be redundant and not directly related to the underlying semantic meaning.  But for rendered text image,  since the image is primarily composed of meaningful text with minimal background details, it `closely mirrors the semantic density of the text` itself and significantly `reduces the typical redundancy` found in image space.\n\nIn addition, process text with image have following advantages:\n\n**1. Efficiency of Image Tokens:** In our method, the number of image tokens is fewer than the equivalent number of text tokens for the same length of input (Line 98-99). This demonstrates that processing text in image space can be more efficient than using text tokens alone.\n\n**2. Economic Efficiency:** Our approach shows that preprocessing text into image space is more economically friendly compared to relying solely on text models. For example, flops in Figure 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KKKJVMYjdH", "reviewer": "Reviewer_iV3t", "summary": "The paper proposes a method to increase the context size of multi-modal large language models. The goal is to increase the context with minimal GPU memory for both training and inference as well as floating pointing operations. Finally, the authors show that the method obtains good results on in-context learning.", "questions": "Questions related to lines 75-76. \n1. Both images and text are concatenated to result in a sequence of 256 tokens?\n2. Is m chosen so that the concatenation is 256 tokens?\n3. What does the corresponding text represent? Based on Fig. 2, `I_i` seems to be an actual image and not an image representing the text, so I am confused. Or the bear image has nothing to do with `I_i`? If it doesn't where do the image tokens `I_i` are used in Fig 2?\n\nQuestions lines 81-82.\n4. Is this the same m as above? If yes, can you elaborate what the m represents? I assume no, since it's `M` vs `m`, but I think other letters can be used to make this more clear", "limitations": "The limitations are briefly discussed", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tackles an important problem and can be of interest to the research community. It seems to achieve promising results and the idea seems interesting, but there are parts that are unclear or need better studying.", "weaknesses": "I think that the paper is hard to follow and some parts hard to understand. Please see below for some questions\n\nMy other concern is related to the experimental part, that I find a bit weak. The comparison is made against only two existing models and I don't clearly understand the numbers. Firstly, one concern is that there seems to be a degradation in performance with 0 shots in some cases.\n\nAlso, probably I am missing something, but let's look at Tab. 1. Wouldn't an increase context size (ICL 256 vs 2048) would allow for more shots to be used? I don't understand what is different in terms of the theoretical context between 0 shot Open-Flamingo and VisInContext. Isn't the same information fed to the model? If yes, then I don't see how the experiment measures the impact of the context size. Or is it just way to feed more information from a particular document as opposed to sub-sampling it? In the later case, I would consider this to be a slightly unfair comparison and I don't really understand what are the benefits of the model if in the zero-shot setup where you have access to a larger portion from the document (as opposed to subsampling) the performance drops quite a lot in some cases (okvqa, vizwiz). This shows that the way that the context is processed is not ideal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Questions related to line 75-76.**\n\n_i. Clarification on Token Concatenation:_\nNotice that <visualx> is used as `a placeholder to indicate the position of an image` within the token sequence. It has a `token length of 1`. For example, in a sequence of 256 tokens, the structure could be <visual1><text1><visual2><text2><visual3><text3>, where the visual placeholders are interleaved with text tokens. In this example, 253 tokens out of the 256 are text tokens. If the text exceeds the length limit, it will be truncated during preprocessing.\n\n_ii. Pre-defined Hyperparameter $m$:_\nThe hyperparameter $m$ is predefined and set to a specific value, which is 3 in this work. It indicates the number of images included in a sequence. If there are not enough images to meet this value, zero tensors are used as padding to ensure the sequence length remains consistent at 3.\n\n_iii. Explanation of Corresponding Text and Image Tokens:_\nThe interleaved dataset, such as MMC4, includes multiple images and their corresponding texts. The \"corresponding text\" refers to the text paired with a specific image. In our notation, $I_i$ represents the $i$-th raw image. In Figure 2, visual information from both raw images and rendered text images is integrated into the LLM using a `cross-attention mechanism`. In this mechanism, `the text tokens act as queries, while the keys and values are the visual tokens derived from the images, after applying token masking`. Notice that $I_i$ only does self-attention with $T_i$ according placeholder <Visual i>.\n\n\n\n**Q2. Questions about Line 81-82:**\n\nThe $M$ in these lines represents the number of rendered text images used. \nIt is different from $m$, which is used to denote the number of raw images. \nWe will use a different letter in the revised version.\n\n\n**Q3. Do not understand Table 1:**\n\n_1. The details of ICL:_ The evaluation pipeline for MLLMs typically consists of a pre-training stage followed by a fine-tuning or zero-shot evaluation stage on downstream tasks. The \"In-context Text Length\" mentioned in Table 1 refers to the `pre-training stage (Lines 160-162)`. For downstream tasks, we ensure a fair comparison by maintaining the same number of shots and input settings across all models. This means that the settings for `both the baseline model and our method are identical during downstream evaluation`, allowing us to accurately assess the benefits of longer context pre-training. Our method `naturally supports longer shots during inference (Figure 4)`; however, comparing results between significantly different shot numbers, such as 128 shots versus fewer shots, would be unfair as more shots leads to better result in general. Therefore, we do not include these comparisons in Table 1.\n\n_2. Seems to be a degradation in performance with 0 shots in some cases, okvqa and vizwiz drops a lot :_ Each dataset has its own biases and evaluates different capabilities of the models. It is common for results to be slightly below the baseline in some datasets, but the overall mean performance is significantly better than the baseline. Please refer to Q4 for additional information on RVzob as a supplementary resource regarding the instability associated with increased shots.\n\n**Q4. Comparison against only two models:**\n\nFlamingo and Fuyu are two strong and representative methods: Flamingo represents models with visual encoders, while Fuyu represents models with only linear embeddings. All MLLMs fall into one of these two categories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8PKjvdljrt", "reviewer": "Reviewer_QgLn", "summary": "Extending context length is a fundamental challenge for LLMs. Unlike previous approaches that focus on efficiently handling text tokens, this paper introduces a novel method: encoding lengthy text information into image renderings. These image tokens effectively increase the context length and enhance the performance of LLMs across various downstream tasks.", "questions": "This paper claims that long texts can be converted into images. If that's the case, do we need text tokens at all? Can we replace all the text with images, instead of just some prefixes as is currently done?", "limitations": "Discussed, but preliminary. The paper only addresses a minor technical limitation regarding static vs. dynamic tokenization of images. However, there are many more potential limitations to consider. For example, I wonder if the current approach would also be effective for larger models, such as Llama-3, which has a longer context length of 8192.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The idea is simple yet surprisingly effective. Previous works like PIXEL and CLIPPO suggested treating text as images to eliminate the need for tokenizers and unify various languages into a single image format. In contrast, this paper uses image rendering to encode lengthy texts, employing a PIXEL-like method to enhance long-context understanding.", "weaknesses": "1. Information loss from image rendering\n\nThe current approach has two potential sources of information loss: 1) rendering long texts into an image, and 2) encoding an image into MLLM embeddings. This loss is not thoroughly investigated in the paper. For instance, Table 2 shows that using the original lengthy 426 tokens outperforms the proposed rendered image, albeit at higher computational costs. While it is acceptable to trade some performance for efficiency, the trade-off should be clearly demonstrated in the paper.\n\n---\n2. Optimal compression rate\n\nDue to the potential information loss, rendering very long texts into a single high-resolution image might not be optimal. For a context of, say, 2048 tokens, what is the best approach: a single image with 2048 words, or 32 images with 64 words each? The current ablation study only examines the rendering aspects like font size and font interval threshold. However, the trade-off between the number of images and the number of words per image is a crucial study that should be included in the paper.\n\n---\n3. Comparison with other long context methods\n\nA significant weakness of the paper is the lack of comparison with other long context methods. Specifically, the paper must compare the proposed compression-into-image approach with compression-into-text approaches, such as [1-3]. Indeed, text compression loses the original semantics, while image rendering retains all previous words. Therefore, combining both approaches—compressing the semantics first and then rendering them into an image—could potentially offer the best of both worlds.\n\nIn addition to the text compression approach, it would be beneficial to discuss the pros and cons of this work in comparison with other long context methods. The current paper only discusses classic efficient self-attention models like Longformer. However, there are more recent and diverse approaches, such as handling long sequences by dividing them into multiple chunks [4-5] or using interpolation positional encodings [6-7], among others.\n\n[1] Mu et al. Learning to Compress Prompts with Gist Tokens. NeurIPS 2023.\\\n[2] Chevalier et al. Adapting Language Models to Compress Contexts. EMNLP 2023.\\\n[3] Ge et al. In-context Autoencoder for Context Compression in a Large Language Model. ICLR 2024.\\\n[4] Bertsch et al. Unlimiformer: Long-Range Transformers with Unlimited Length Input. NeurIPS 2023.\\\n[5] Song et al. Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs. ICLR 2024.\\\n[6] Chen et al. Extending Context Window of Large Language Models via Positional Interpolation. arXiv 2023.\\\n[7] Li et al. Functional Interpolation for Relative Positions improves Long Context Transformers. ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Information loss from image rendering and trade-off between performance and computation cost:**\n\n**1.Information Preservation in Rendering:**\nThe rendering process preserves all the words in the text image, ensuring that no textual information is lost during this step.\n\n**2.Information Encoding:**\nEvaluating potential information loss during the embedding step is challenging as it operates at the feature level. \n\n**3. Performance vs. Computational Cost Trade-off:**\nWe acknowledge the trade-off between performance and computational efficiency.  Using the original lengthy 426 tokens can outperform the rendered image for inference, but at a higher computational cost. To further investigate this trade-off, we conducted additional pre-training experiments using the smaller Mistral-7B model. We show the result below and find that with the same in-context text length and batch size, our method shows a slight decrease in performance (e.g., from 25.4 to 24.6 on average). However, our approach allows for `longer in-context lengths and larger batch sizes with the same computational resources`. When utilizing these advantages, the mean accuracy of our method clearly outperforms the baseline.\n\n| **Method**       | **ICL Length** | **BSZ** | **(okvqa)** | **(textvqa)** | **(vizwiz)** | **(vqav2)** | **Caption (coco)** | **Caption (flickr)** | **Mean** |\n|------------------|----------------|---------|------------------|-------------------|------------------|-----------------|--------------------|----------------------|----------|\n| **9B Baseline**  | 512            | 16      | 17.1             | 14.8              | 21.5             | 26.5            | 40.1               | 32.1                 | 25.4     |\n| **+VisInContext**  | 512            | 16      | 16.3             | 15.1              | 20.3             | 25.4            | 39.1               | 31.5                 | 24.6     |\n| **+VisInContext**  | 512            | **64**      | **18.5**             | 17.4              | 22.3             | 27.0            | 41.2               | 31.8                 | 26.4     |\n| **+VisInContext**  | **4096**           | 16      | 18.3             | **19.3**              | **22.5**             | **28.4**            | **42.3**               | **34.8**                 | **27.6**     |\n\n> Caption: We compare the zero-shot evaluation of the OpenFlamingo-9B baseline model with our VisInContext. In this context, BSZ stands for batch size, and the prefix text token length is 128. With the same GPU memory, VisInContext supports a larger batch size or a longer in-context learning (ICL) length.\n\nDue to time limit of rebuttal phase, we only conducted one analysis and will include more detailed discussion on this trade-off in the revised version of our paper.\n\n**Q2. Optimal compression rate:**\nOur current implementation is already multiple text image inputs for longer texts by splitting the text into several images. This allows us to process texts longer than what a single image can accommodate. In Figure 2, we depicted only one image for simplicity and clarity.\n\nThe changes in `font size and font interval directly correspond to the number of text images` required. For example, using a smaller font size results in fewer images needed to render the same amount of text. This relationship indicates that adjusting font size and interval settings effectively controls the number of images used.\n\n**Q3. Comparison with other long context methods:**\n\nThank you for insight comment.\n\n[1]: This method involves an additional fine-tuning stage and modifies the cross-attention mask of the transformer. \n[2]: This method reuses the language model multiple times to obtain a summary vector for each segment and the training pipeline is very different.\n[3]: This approach requires an additional instruction fine-tuning stage on specific instruction data to produce the desired output.\n\nThese methods focus on re-use the pre-trained model by fine-tune on specific learning objective[1,2] / instruction data[3]. While these methods are not directly compatible with our pre-train (also increase ICL in this stage) and few-shot evaluation pipeline, these works have inspired us significantly.  We are particularly interested in adopting the strategy from [2] during the inference stage for further increase the ICL. However, the code is hard to modify under our codebase and we are still working on it.\n\nRegarding the pros and cons of other long context methods, we will expand our discussion in the revised manuscript to include:\n[4] and [5]: We will discuss these techniques in lines 287-289. For [6] and [7], we will cover these approaches in lines 290-292, which use positional interpolation to extend the context window.\n\n**Q4: Do we need text tokens?**\nAt the current stage, our method still relies on partial text input to perform `next-word prediction and uses autoregressive language modeling loss to optimize the model`. In this way, we can test downstream tasks directly by predicting next word. Completely replacing text tokens with images would require developing a unified learning target beyond the current contrastive loss. This is a complex challenge and an area of ongoing research.\n\n**Q5: If current method suit for larger models such as Llama-3:**\n\nThe current approach is indeed suitable for larger models like Llama-3 405B, which has an in-context length of 8192 during pre-training (enabled by techniques like pipeline parallelism on 16 H100 GPUs and low bit quantization). `These techniques are applicable to both vision and language models`, allowing vision tokens to scale similarly.\n\nFurthermore, the parameters of the vision model are only a fraction of the overall model parameters, making it feasible to include more visual tokens without significant overhead. For instance, the Llama-3 405B vision-language model incorporates a ViT-H vision encoder with 630M parameters, demonstrating the compatibility of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper originally received mixed reviews, with concerns including:

1. clarity of presentation
2. motivation and applicability in practice
3. potential limitations and trade-offs due to occluding parts of the image
4. method complexity
5. comparison to additional baselines

Following rebuttal, observing the author-reviewer discussions, the AC got the impression most concerns were addressed, with only some concerns of clarity of presentation remaining for reviewer iV3t who also suggested a constructive solution to resolve the clarity issue in the revised manuscript.

In light of the above, the AC recommends accepting the paper and urges the authors to incorporate all the discussions into the revised manuscript. Especially the constructive suggestion by iV3t.
Decision: Accept (poster)