From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
Authors: Wanpeng Zhang
Zilong Xie
Yicheng Feng
Yijiang Li
Xingrun Xing
Sipeng Zheng
Zongqing Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Image Tokenizer
Token Merge
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce a novel BPE Image Tokenizer that applies BPE tokenization to visual data, enabling more effective integration of visual information in MLLMs and improving their performance.

Abstract: Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models. For further details, visit our website https://github.com/BeingBeyond/Being-VL-0.

Published as a conference paper at ICLR 2025
FROM PIXELS TO TOKENS: BYTE-PAIR ENCODING ON QUANTIZED VISUAL MODALITIES
Wanpeng Zhang'; Zilong Xie², Yicheng Feng', Yijiang Li Xingrun Xing Sipeng Zheng', Zongqing Lu
1 Schooo of Computer Science, Peking University
"The Chinese University of Hong Kong
3 University of California, San Diego
"Beijing Academy of Artificial Intelligence
"Institute of Automation, Chinese Academy of Sciences
"BeingBeyond
ABSTRACT
Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conven- tional approaches that rely on separate visual encoders, our method directly incor- porates structural prior information into image tokens, mirroring the successful tokenization strategies used text-only Large Language Models. This innova- tive approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that BPE Image Tokenizer significantly enhances MLLMs' mul- timodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-S formance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models. For further details, visit our website.
model that demonstrates superior per-
INTRODUCTION
The development of Multimodal Large Language Models (MLLMs) has made significant progress (Yin et al., 2023; Team et al., 2023; Liu et al., 2024b). However, these multimodal foundation models often model different modalities separately, incorporating many modality-specifie designs such as specialized encoders and decoders (Liu et al., 2024b; Zhang et al., 2024; Jin et al., 2023). While this approach allows training data to align well with these -odality-peecific designs, it often struggles to achieve unified understanding of multimodal information (Team, 2024). The primary reason for this limitation could be that while encoders of other modalities can learn rich information, without the assistance of the corresponding decoders, LLMs cannot fully comprehend the complex patterns contained within the embeddings provided by the encoder. other words, LLMs need to learn to interpret the token embeddings again, which is the job of the decoders of other modalities, leading to difficulties in aligning with these modalities (Baltrušaitis et al., 2018).
Recent research has begun explore unified token-based representations for MLLMs (Team, 2024; Lu et al., 2024; Zheng et al., 2024), attempting to achieve better capabilities for multimodal infor- mation processing by moving away from modality-specifff designs. This approach offers two main advantages: first, it can achieve unified information understanding, and second, it enables unified generation that can be decoded into different modalities. However, these methods simply convert various modal information into unified representations and then directly feed them into Transformer models (Vaswani, 2017), hoping learn the data just with massive computing resources. Although
Wangpeng Zhang, Yicheng Feng, and Yijiang Li work as interns at BAAI Correspondence Zongqing Lu rxonggingllu@ppuu.ddd.

---

Published as a conference paper at ICLR 2025
Ing Yocab
Codabook
<paioh
Transformer
PT
5
26
53
235
643
146
53
16
Quantize
Tokenize
2,,,,,,,,,,,3,,,,,,,, 26,15,,5,,99 93,166
[235,643,146. 546,33166
"a white cat and an orange cal on the ground." "a white cat and an orange cat on the ground.
Figure 1: Illustration of our BPE Image Tokenizer. The overall process begins with quantizing the image into initial token IDs. The BPE Image Tokenizer then combines these tokens based on learned patterns, similar to text tokenizers. This combination results in tokens that inherently contain more semantic information. The final tokenized sequence thus incorporates structural prior information from the image, enabling the Transformer model to deeper comprehend the alignment between visual and textual information during training. This approach facilitates more effective integration of visual data into MLLMs, enhancing their multimodal understanding capabilities.
this approach can be effective to some extent, it consumes enormous computing resources and time while failing to achieve significant improvements in performance.
In comparison to the widely adopted training paradigm of text-only LLMs (Touvron et al., 2023; Achiam al., 2023), observe that current token-based MILMs often overlook crucial compo- nent: the tokenizer that explicitly merges data sources. In text-only LLMs, the most widely adopted tokenizer is the Byte-Pair Encoding (BPE) tokenizer (Sennrich, 2015; Radford al., 2019), which learns to merge characters into tokens of varying lengths based on the word frequency in the training corpus before providing them to the Transformer for learning. While many believe this approach solely for conserving computational resources, some recent theoretical and experimental analyses suggest that using merged tokens plays vital role in the Transformer's learning of sequential data (Rajaraman al., 2024; Makkuva et al., 2024; Merrill et al., 2024). In fact, if raw sequence data is provided directly, Transformer models may even fail to learn how to predict entirely.
In this paper, we reveal that Transformer models cannot effectively learn certain types of two- dimensional sequence data. However, when we apply operations similar to BPE tokenizers to the two-dimensional data, i.e., merging tokens based on the frequeecyy of training data, the learning ef- fectiveness can improve significantly. We believe the benefit comes from the addition of necessary structural prior information former models to be aware of important relational information in the data during learning. Our theoretical analysis proves that tokenizing sequence data certain types of two-dimensional data can indeed achieve smaller losses, which we further validate experimentally.
the tokens during the merging process, making it easier for Trans-
Based on the insights provided by our theoretical analysis and experimental validation, propose a new learning paradigm for MLLMs. By tokenizing the unified representation of multimodal data using a novel BPE Image Tokenizer, allowing MLLMs built from text-only LLMs to have good multimodal capabilities. As illustrated in Figure after quantizing the image into token IDs, the BPE Image Tokenizer further combines these token IDs according to its learned patterns. Similar to text tokenizers, our BPE Image To- kenizer ensures that the tokens being fed to the language model decoder inherently contain more semantically meaningful information in the statistical sense. Intuitively, the tokenized IDs, which have incorporated structural prior information from the image, could provide the Transformer model with better comprehension of the specific alignment between text and image in the training data. Pre- liminary results further validate the effectiveness of explicit tokenization for multimodal data. We hope that the new paradigm for learning MLLMs proposed in this paper will provide better guidance for subsequent scaling-up efforts, leading to the building of more powerful MLLMs.
enable Transformer models to better understand image data,
In summary, we make the following key contributions:
2

---

Published as a conference paper at ICLR 2025
We are the first to propose new MLLM learning paradigm that explicitly tokenizes mul- timodal data like text-only LLMs, centered around our novel BPE Image Tokenizer.
We theoretically analyze why this learning paradigm can bring benefits and further provide corresponding experimental validation.
We design an algorithm for training the BPE Image Tokenizer and train the Being-VI-O with this tokenizer. The performance evaluation further validates the capability enhance- ments this learning paradigm brings to MILMs.
2 NOTATIONS AND FORMULATION
Before delving into the theoretical analysis of our proposed paradigm, we introduce key notations and concepts used throughout this paper.
Image Representation and Quantization. We represent an image as a set patches {Xij his.jsm. where is the number of patches per row/column, and X;z denotes the patch at position (i.j). We employ Vector Quantization (VQ) to discretize these patches, using a codebook with size C je]. and quantization function VQ R2 C.
BPE Image Tokenizer. Our proposed BPE Image Tokenizer converts a quantized image into a sequence of token IDs, defined
X T", where T the set of tokens and is the set of all finite sequences of tokens. special case of encoding is flattening, flat - em² which arranges image patches into sequence row by row.
Unigram Model. For unigram model Q E I-gramm given a token sequence (11 11)) the probability is defined as:
Q(t) = Q# ([t)) II (tok(t,),
where Q# is distribution over N representing the probability of the sequence length, and Quak is distribution over the dictionary tokens 'J.
Multimodal Large Language Model (MLLM), We define MLLM as probabilistic model Q over token sequences, capable processing both text and image data as follows:
1. Input Processing: t (xeetize(XMxx)) timage enc e (X(aageeee))
2. Sequence Modeling: P(t. ..., f4-1) Q(t,/tz.).
3. Generation: ~ P(ts|yi,..., 1i
The training objective for an MLLM is to minimize the cross-entropy loss:
ly log P(ylyeei
-(xx)
where D is dataset of input-output pairs (). The goal is to find the optimal parameters 0* that minimize this loss: f" = arg ming (0).
THEORETICAL ANALYSIS
Previous studies have demonstrated that Transformers struggle effectively learn certain one- dimensional sequence data (Rajaraman et al., 2024; Makkuva et al., 2024). In this section, we extend this concept to two-dimensional image data, considering simplified model where the im- age data-generating distribution follows a 2D 1tth order Markov process. This simplified model is defined as follows:
Definition (2D ooder Markov process). For each variable X with probability 2. it depends
only on Xj-k.j- ie., Pr(X,, = 1[X;-k, = 0) = and Pr(X,,; = 1[X = 11 = qi With probability J. it depends only on X i.e., Pr(X,, 1|X = 0) and Pr(X,,; =
Xijjk 1) q. Figure 2(a) illustrates this process.

---

Published as a conference paper at ICLR 2025
Trensfommrr m/ varimuu Optinal
kyyeee?
Cross-Entrapy
Crocs-E Entroyy
byers=
g
22
3.
Optimal lows
Optimaal CE loss (unigram)
1.5
9.2
10-
50
100
150
200
250
0
50
100 150
200
250
Itarations
Iterationn
(a) 2D kth--oder Markov process. Sequence data transformed based on horizontal or vertical k* order Markoy conditional probabilities.
(b) Without tokenizer, Transformer fails to learn 2D k -oderr Markov se- quence data, only achicving the per- formance of a unigram model (dotted line). The gray area represents Trans- former under various parameters
(c) With BPE tokenizer, even Trans- former models with only few layers (less parameters) can casily learn the 2D 4*h -order Markov sequence data, achieving optimal cross-entropy loss (dashed line).
Figure 2: Definition of 2D h*h -order Markov sequence data, and the performance of Transformer in learning such sequence data with or without tokenizer. For details of the hyperparameters used in the experiments, please refer to section B.
This simplification is intuitive, as pixels an image often exhibit conditional dependencies with other pixels at specific horizontal and vertical distances. Consequently, real-world image data can be viewed as a composite of multiple such simplified processes. When attempting learn such se- quence data directly using a Transformer model, an interesting phenomenon occurs: the Transformer fails to surpass the performance of the stationary unigram model, regardless of various hyperparame- ter choices. As shown in Figure 2(b), the Transformer fails to improve the cross-entropy loss beyond that of the optimal unigram model on the training data (indicated by the dotted line). However, as shown in Figure 2(c), when encoding the sequences in each direction separately using BPE Tok- enizer trained on this distribution, even a minimal Transformer model with only a few layers can readily achieve optimal loss. This finding suggests that BPE Tokenizer could play crucial role in the Transformer's 's learning process for the defined two-dimensional sequence data.
To formally analyze this phenomenon in more general setting, first let m be the number of patches per row in an square image, and let Xi denote the patch at position (i,j) for 1 S i, S m. An image can be represented by the set of m patches {Xiş Si.jjmm which we abbreviate as X for simplicity. Each patch can be encoded into index with VQ-GAN model's codebook, denoted by C, with C = [e] representing the size of the codebook.
We then consider two scenarios for image generation:
Scenario 1. Every column in the image X is generated independently by an ergodic Markov pro- cess supported C, proceeding from top bottom, with the same transition kernel P(- and stationary distribution π.
Scenario Every row in the image X is generated independently by an ergodic Markov process supported on C, proceeding from left to right, with the same transition kernel P 1())-) and stationary distribution
Given a model Q on tokens and an encoding function enc(-) that encodes an image into a sequence of tokens, we define the cross entropy loss as
Lm (Q enc)
Eloog [[[llo@(((ee()))]]
m²
where the subseript m indicates the dependence on size m. We also introduce the trivial encoding function flat(:), which merely flattens an image into sequence of patches arranged row by row from left to right:
(x) = (X11, X12,... X21. X22, X2m Xm1 Xm2 Xmm)..
Using this notation, - (Q o flat) represents the cross entropy loss for model Q along with the encoding function fla For simplicity, we abbreviate it as L. (Q) throughout this paper. We
4

---

Published as a conference paper at ICLR 2025
define (p) the Shannon entropy for any discrete distribution p. p(y) log)
(p)
Σ
yEsupp(p)
Now, we present our main theoretical results:
Proposition 1. For data generating processes described in either Scenario I or Scenario 2, as - xc, the optimal cross-entropy loss among unigram model family Q 21-gram satisfies
lim inf min
(Q) 2 H(T) = Σ (o)(())(((a)))
(3)
m-000 QER1-gram
aEe
In contrast, the optimal unconstrained cross entropy loss satisfies
lim min L. (Q) = Ho _ Σ Σ m-
T(a) log (P(d' aa)).
(4)
acC d'Ee
Here provide sketch of the proof of Proposition 1. For the full proof, please Appendix A.2.
Sketch of Proof. For the optimal cross-entropy loss among Q 1-ggram. observe that under both scenar- ios, the marginal distribution of each patch is precisely the stationary distribution т. When applying a unigram model to flat(X and leveraging its inherent independence, the optimal loss closely approximates (z), with discrepancy that vanishes as m - oo. Regarding the optimal uncon- strained cross-entropy loss, note that the best model is simply the generating model of flat(X) with its cross-entropy loss converging to Hos as m - XC.
Remark 1. Consider the 2D k Markov process in Figure 2(a). Let 2
1 6. For this process, limm-+00 Tn (P) log( ( (1 8) log( F-z). However, the unigram stationary distribution is
which means switching between 0 and with equal probability
(2, } and (x) log(2). The ratio limm-xo H
H(a)
approaches do as 5 + 0. This
example shows thut the gap between (xx) and Hos could be arbitrarily large.
The implications of Proposition and Remark are profound for understanding the limitations of simple models in capturing the structure of two-dimensional image data. This result shows signif- icant performance gap between unigram models and the optimal unconstrained model for such data. be arbitrarily large. This find-
This gap, represented by the difference between H (T) and H ing underscores a critical limitation of simple unigram models, which Transformers could default earlier experimental results. These models
to when processing sequences, according inherently restricted in their ability to capture the true structure of image data, suggesting that more sophisticated tokenization methods necessary for effectively processing two-dimensional image data with Transformer models. We further discuss the potential information loss of our tokenizer in Appendix A.I.
Proposition 2. For data generating processes described in either Scenario or Scenario 2, assume that å å mina,a'e P(a'la) > 0. Then there exists a tokenizer with a dictionary containing at most D tokens, along with an encoding function enc(.) applied to X, such that
lim sup min m-+0o QeAl-gram
Dm enc(-))
Hoc.
(5)
where log(1/))) log( D)) and D N an arbitrary constant that is sufficiently large.
Here we provide a sketch of the proof of Proposition 2. For the full proof, please see Appendix A.2.
Sketch of Proof. The proof leverages the fact that in both scenarios, the generated image com- posed of m parallel Markov sequences, each of length m, sharing the same transition kernel and stationary distribution. We first apply the result from the one-dimensional case (Lemma A. to each column (or row) separately. Then, we construct tokenizer for the entire image by concate- nating the tokens from each column (or row). By carefully bounding the cross-entropy loss this constructed tokenizer, the additional complexity introduced by the two-dimensional structure while maintaining the bound derived from the one-dimensional case.
show that it satisfies the desired inequality. The key step is to handle
5

---

Published as a conference paper at ICLR 2025
Algorithm 4.1 BPE Image Tokenizer training procedure.
Input vo, m, D.
vo: initial vocab size, m: new vocab size, D: training data
2: e < to
>> et current vocab size A: adjacency matrix
3: A < zeros(0 V x v))
4: V <-g 9
b V: extended vocabulary
5: for i +- m do
6:
A < UpdateMatrix(D)
7:
(p: f) < MaxFreqPair( A)
D pt best pair, f: frequency
8:
if' = then break
9:
end
10:
V (-U(((e)))) -
11:
for each d E D do d' + Replace p with v in d + {
13:
15:
end for
D+D D'
17:
++1+1
D set next id for new token
end for
19: return V
Proposition offers insights into the potential benefits of using a tokenizer for image data. This result demonstrates that with an appropriate tokenizer, we can achieve a loss arbitrarily close to the optimal unconstrained loss Hoc. significantly outperforming the unigram model bound of H These theoretical results motivate the design of our BPE Image Tokenizer. In the following sections, we will describe the implementation of our algorithm and present corresponding experimental results.
PRELIMINARY IMPLEMENTATION
To validate the effectiveness of our proposed BPE Image Tokenizer and the training paradigm in en- hancing models' multimodal understanding capabilities, we implemented the entire procedure and conducted series of experiments. Given the scope of this paper, we emphasize that this imple- mentation for training an MLLM is preliminary and does not involve extensive scaling in terms of dataset size or computational resources directly compare with state-of- ofte-off-the-rr foundation models.
4.1 BPE IMAGE TOKENIZER TRAINING
Dataset Preparation: We constructed a diverse image dataset comprising 2.78 million images from ImageNet (Deng al., 2009), CC (Sharma al., 2018). LAION (Schuhmann al., 2022), and SBU (Ordonez et al., 201 1). We applied filtering mechanism similar to LLaVA (Liu et al., 2024b), selecting images based on their class labels or coun-phrase statistics derived from captions keep the concept diversity.
Image Preprocessing: We used pretrained VQ-GAN model released by Chameleon (Team, 2024) to preprocess the images. This VQ-GAN model uses codebook size of 8192 and quantizes images of any size into a 1024-dimensional ID tensor.
Tokenizer Training Algorithm: We developed the BPE image tokenizer training algorithm, in- spired by the text BPF algorithm. The main procedure of this algorithm presented in Algorithm 4. with supporting functions detailed in Algorithms B. and B.2 (see Appendix B). Key designs of our algorithm include: 1) Use an adjacency matrix to count token co-occurrences; 2) Simulta- neous consideration of horizontal and vertical token combinations; 3) Ability to merge tokens into arbitrary shapes through iterative merging.
Extended Vocabulary: The training process results in extended vocabulary that merges tokens based on learned patterns. To analyze the impact of vocabulary size on performance, we trained multiple tokenizer versions with extended vocabulary sizes ranging from 1024 to 16384.

---

Published as a conference paper at ICLR 2025
4.2 MLLM TRAINING
Base Model: We used the Llama- -3.1-85 model (Dubey et al., 2024) as our base text LLM.
Token Embedding Expansion: Before training the MILM, first expanded the token embedding layers of the base model to accommodate the new image token IDs. For instance, with 8192 VQ- GAN token IDs and a BPE Image Tokenizer vocabulary of size 4096, we expanded the embedding layers from n X m to (n + 8192 + 4096) x m, where n is the original text embedding dimension and m the number of embedding layers. We also added extra special tokens mark the beginning and end positions of images.
Training Process: The training process consisted of two stages:
Stage 1: Image Understanding Pretraining (PT). In this stage, we froze the original text token embeddings and trained only the new image token embeddings using image caption data. This stage used 595K images from CC-3M (Sharma et al., 2018) and 558K from LCS (Liu al., 2024b).
Stage 2: Supervised Fine-Tuning (SFT). After stage 1, we unfroze all weights and per- formed full parammeeer tne-ttunig using complex conversation data with image inputs. This stage used I 1.27 million entries from the LLaVA-One Vision Dataset (Li et al., 2024), includ- ing General QA (504K), Doe & Chart (249K), Reasoning (343K), and OCR (180K) tasks.
The core distinction between our proposed training pradigm and existing widely used MLLM train- ing approaches (Liu et al., 2024b:a) mainly lies in the first stage of pretraining. The conventional approaches typically align pretrained CLIP encoder (Radford et al., 2021) with text-based LLM, where the image understanding capability is largely derived from the CLIP encoder, which was trained on 400 million image-text pairs (Radford et al., 2021), far exceeding the amount of data we provide to our model for image understanding. Though our proposed paradigm also involves pretraining an image tokenizer, it does not directly provide image comprehension capability to the tokenizer. Instead, the MLLM acquires its entire image comprehension capability after the first stage of training. Intuitively, this approach allows for more direct fusion of the image modality with the text-based LLM, potentially mitigating biases introduced by separate training and subsequent align- ment. Our experimental results also demonstrate that even with significantly limited amount of data compared to that used in pretraining the CLJIP encoder, we can still endow text-based L.Mss with good image understanding capabilities.
5 EXPERIMENTAL RESULTS AND ANALYSIS
Our experiments aimed to evaluate the effectiveness of the proposed BPE Image Tokenizer and its impact on MILMM performance. We analyze the results from three main perspectives: the impact of the BPE Image Tokenizer, the effect of additional data scaling, and the influence of extended vocabulary. Table 5.1 summarizes our main results, while Figures 3(a) and 3(b) provide insights into the impact of the BPE vocabulary and its token usage patterns.
5.1 EXPERIMENTS SETUP
Benchmarks: We evaluated our model using multiple benchmarks:
VQAv2 (Goyal et al., 2017): Visual Question Answering
MMBench (Liu al., 2023): Multimodal Understanding
MME (Fu et al., 2023): Multimodal Evaluation (separate tests for perception MMEP and cognition MME3)
POPE (Li et al., 2023): Object Hallucination Evaluation
VizWiz (Gurari et al., 2018): Visual Question Answering for Visually Impaired Users
Data Scaling Experiments: To further investigate the impact of dataset size on performance, we
gradually augment training data with RefCOCO (50.6K) (Kazemzadeh et al., 2014), AOKVQA (66.2K) (Schwenk et al., 2022), ShareGPT4o (57.3K) (Chen et al., 2023), and ALLa VA Inst (70K) (Chen al., 2024).
7

---

Published as a conference paper at ICLR 2025
Table 5.1: Performance comparison of different settings and training strategies across multiple benchmarks. Consistent improvements across all benchmarks demonstrate the benefit from the BPE Image Tokenizer and the effectiveness of the designed training strategy. Further performance gains are observed with incremental data additions, highlighting the potential scalability of our approach. Specifically, the 'LLM' represents the Llama-3, 1 1-BB backbone model we used. MME" and MME represent MME-perception and MME-cognition, respectively.
Training type
VQAv2
MMBench
MME
MME
POPE
VizWiz
SFT
51.1
35.9
972.3
231.8
73.8
43.1
LLM+VQ
PTTful))SST
37.0
261.4
PT(F
55.4
37.6
1054.5
277.0
76.0
45.3
SFT
52.2
35.4
1029.7
269.6
76.3
45.3
LLM+VQ+BPE (Being-VL-O)
PT
56.5
38.6
1144.6
284.3
77.3
45.8
PT(freeze)+SFT
57.1
40.9
1223.5
307.1
79.0
46.0
Additional sculing (PT)
#ReiCOCOK50.6K
58.6
42.3
1257.4
314.3
47.1
+AOKVQA (66.2K)
59.6
43.1
1288.1
321.4
80.4
47.5
Additional scaling (SFT)
*ShareGPT4o (57.3K)
60.2
43.7
1304.5
327.7
80.9
47.8
+ALLaVA Inst (70K)
60.6
44.0
1316.2
331,0
81.3
48.2
5.2 IMPACT OF THE BPE IMAGE TOKENIZER
Importance of Two-Stage Training: Our results clearly demonstrate the necessity of the two-stage training process. Models trained only with SFT show notably lower performance compared those that underwent both PT and SFT. This finding underscores the importance of the pretraining stage in guiding the expanded token embedding layers to learn meaningful visual representations.
We observe a consistent performance advantage when freezing the text token embeddings during pretraining, i.e., PT(freeze), compared to training all embeddings, i.e., PT(full). For instance, in the 'LLM+VQ-BPE' setting, PT(freeze)+SFT outperforms SFT across all benchmarks, with notable improvements in MME" (1223.5 vs. 1 1144.6) and MME (307.1 vs. 284.3). This suggests that allowing the text embeddings update during pretraining may inter- fere with the model's ability to focus on learning visual representations.
Freezing vs. Full Pretraining:
The integration of our BPE Image Tokenizer improves model performance across all benchmarks, as evidenced by comparing the 'LLM+VQ' and "LLM+VQ+BPE' rows in Table 5.1. These gains indicate that the BPE Image Tokenizer provides our Being-vL-O model with better multimodal understanding. The improvements is consistent across different training methods, also highlighting the robustness of our approach.
Performance Gains:
5.3 IMPACT OF ADDITIONAL DATA SCALING
To investigate the scalability of our approach, we incrementally add more datasets to both the pre- training and SFT phases. The results, shown in the lower part of Table 5.1, reveal clear trend of performance improvement with increased data volume.
Pretraining Data Scaling: Adding RefCOCO (50.6K) and AOKVQA (66.2K) to the pretraining phase leads to consistent improvements across all benchmarks. For instance, VQAv2 scores increase from 57.1 to 59.6, and MMBench scores rise from 40.9 to 43.1. This suggests that our model benefits from exposure to wider range of visual concepts and question-answering patterns during pretraining.
SFT Data Scaling: Further improvements observed when adding ShareGPT4o (57.3K) and ALLaVA Inst (70K) to the SFT phase. The consistent improvement across different benchmarks indicates that approach effectively leverages additional data to enhance multimodal understand- ing.
Scalability Potential: The continuous performance improvements with data scaling suggest that our training paradigm has not yet reached its upper limit. This finding is encouraging, as it implies that further performance gains could be achieved with larger datasets more diverse data sources.
8

---

Published as a conference paper at ICLR 2025
0.230
VCA M02p FOr 1221Z
0.835
Normalized Score
00.90
1.64
2.04
6.53
be a La Llbttd a Hil
Extended Vacabuary
(a) The performance trend various datasets as the BPE vocabulary size changes. To intuitively compare the trends across dif- ferent datasets, we normalize the scores each dataset to the score obtained when us- ing vocabulary size of IK. The results show that for most datasets, performance reaches its optimum when the vocabulary size is 8K.
(b) Visualization of token embedding weights with different vocabulary sizes. From top to bottom, they respectively rep- resent the 4K/8K/16K versions. The vertical axis represents the mean absolute value of weights across all layers. The hori- zontal axis represents the range of corresponding image token IDs. The left side of the red line represents the range VQ- GAN's codebook (8K). The right area represents the range correspondingly expanded BPE vocabulary (4K/8K/16K).
Figure 3: (Left) The relationship between model performance and the size of the BPE vocabulary. (Right) The visualization of model weights for tokens usage under different vocabularies.
5.4 IMPACT OF BPE VOCABULARY AND TOKEN USAGE PATTERNS
We conduct detailed analysis of how the vocabulary of the BPE Image Tokenizer affects model performance and token usage patterns. Figure 3(a) illustrates the relationship between vocabulary size and normalized performance scores across different benchmarks. We observed that when the vocabulary size is lower than 8K (which equals the codebook size of the VQ-GAN model), the performance improves with the increase in vocabulary size. However, when the vocabulary size reaches 16K, all models on datasets except POPE show performance decline. This trend suggests an optimal vocabulary size should balances efficiency and model complexity.
To further verify this finding and gain deeper insights into how models utilize different vocabularies, visualize the token embedding weights of models trained with different vocabulary sizes. Figure 3(b) shows the absolute values of these weights, averaged across all layers. We observe three distinct patterns: 1) With smaller extended vocabulary (4K), the model tends utilize more extended tokens from the BPE image tokenizer, 2) While with larger extended vocabulary (16K), the model uses more of the original token IDs covered by the VQ-GAN's codebook; 3) Only when using a balanced extended vocabulary size (8k) does the model's token selection become more uniform, achieving the best performance.
We hypothesize that this phenomenon occurs because the original token IDs covered by the VQ- GAN codebook contain fine-grained visual information, while the combined token IDs provided by the BPE Image Tokenizer capture higher-level semantic concepts. A balanced utilization of these two types of information appears to be most conducive to the model's overall learning and performance. These findings may provide valuable insights for future MLLM training design.
6 RELATED WORK
Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remark- able capabilities in various tasks, including visual question answering, image captioning, and cross- modal retrieval (Yin et al., 2023). The evolution of MLLMs be broadly categorized into two main approaches: late-fusion models with specialized encoders and carly-fusion token-based mod- els.
Traditional late-fusion MILMS typically employ distinct specialized designs for different modalities (Team, 2024). In this approach, speeffic modules trained separately using specialized data, followed by additional alignment steps to achieve late-fusion different modalities. For in- stance, in the image modality, CLiP-bbsed visual encoders (Radford et al., 2021; Fang et al., 2023)

---

Published as a conference paper at ICLR 2025
first pre-trained on extensive image-caption datasets, then aligned with text-only LLM backbones (Touvron et al., 2023; Peng et al., 2023) using additional data. Notable examples of this approach include Flamingo (Alayrac al., 2022), LLaVA (Liu et al., 2024b;a). IDEFICS (Laurençon et al., 2024), and Emu (Sun et al., 2023). However, these models often face challenges in modality align- ment, leading to issues such hallucination in MILMMs (Bai et al., 2024).
To address these challenges, recent research has explored early-fusion approaches through unified representations, proposing token-based methods for multimodal learning (Team, 2024; Lu et al., 2024; Zheng et al., 2024). These methods typically utilize Vector Quantization (VQ) models (Van Den Oord et al., 2017: Razavi et al., 2019; Esser al., 2021) convert images into discrete tokens. The concept of token- based multimodal learning was initially explored in studies such as BEiT (Bao et al., 2021), which introduced a self-supervised method for acquiring visual representations based on tokenized image patches. This idea was further developed in works like Cm3 (Aghajanyan et al., 2022) and CM3Leon (Yu al., 2023), which enabled joint reasoning across modalities and scaled up to autoregressive text-to-image generation. More recent models like Gemini (Team et al., 2023), Unicode (Zheng et al., 2024), and Chameleon (Team, 2024) have adopted end-to-end token-based approaches for multimodal learning.
While these token-based MLLMs demonstrate enhanced reasoning and generation capabilities across various modalities without requiring -ppelity components, they still face challenges in representation learning and alignment (Baltrušaitis et al., 2018). Our proposed BPE Image To- kenizer paradigm addresses these challenges by adhering more closely to the learning method of text-only LLMs. Unlike current token-bass approaches, our method directly incorporates crucial structural prior information into the tokens through explicit merging. This approach enables Trans- former models to learn input data more effectively, as supported by both our theoretical analysis and experimental results.
The key distinction of our work lies in its focus on optimizing the tokenization process itself, rather than relying solely on pre-trained visual encoders or simple quantization. While traditional visual encoders can achieve high compression rates, their encoded embeddings often depend on specialized decoders for interpretation. In contrast, our BPE Image Tokenizer creates tokens that are directly meaningful to the Transformer model, facilitating more effective learning of visual information with- out the need for specialized decoders. This approach bridges the gap between visual and textual modalities more seamlessly, potentially leading more robust and versatile MLLMs.
7 CONCLUSIONS, LIMITATIONS AND FUTURE WORK
In this paper, proposed a novel paradigm for MLLM training, demonstrating improvements in image understanding capabilities across various benchmarks, even with limited training data. Our approach centers the novel BPE Image Tokenizer, which effectively combines image token IDs to facilitate better integration of visual information into MILMs. We provided theoretical insights into the importance of tokenization for learning 2D sequence data with Transformer models, supporting our empirical findings and offering a new perspective on processing visual information in language models. Our traind Being- VL-0 model not only showcased the effectiveness of the BPE Image Tokenizer but also demonstrated the scalability of our approach.
While our results are promising, we acknowledge several limitations of our current work. The experiments were conducted with limited computational resources and training data compared to state-of-the-aar MILMMs, potentially understating the full potential of our approach. Additionally, the simplified 2D Markov process used in our theoretical analysis, while instructive, may not fully capture the complexities of real-world image data.
Based on the findings and limitations, we propose several directions for future work. Scaling up the training data and model size would allow us to fully explore the potential of our BPE Image Tokenizer approach in large-scale MILMss Investigating the applicability of our method to other visual tasks, including video understanding, could significantly broaden its impact. Exploring more sophisticated tokenization strategies that better capture the complex dependencies in rorl-world multimodal data is another promising avenue.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by NSFC under Grant 62450001 and 62476008. The authors would like to thank the anonymous reviewers for their valuable comments and advice.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multi- modal model of the internet. arXiv preprint M.Xi:2201.0052222 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongho Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint 2rXiv:2404.18953 2024.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41() 423-443, 2018.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 1XX1:22006.02254. 2021.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing syt4v-synthesized data for a lite vision-language model, 2024.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799. 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Ieee, 2009.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun. Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19358-199 19369, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Yash Goyal. Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
11

---

Published as a conference paper at ICLR 2025
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018.
Yang Jin, Kun Xu. Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, al. Unified language-visse vision pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309. 04669, 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03.3 03326, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen, Evaluating object hallucination in large vision-language models. In The 2023 Conference Empirical Methods in Natural Language Processing, 2023. URL https:/ xo22wOXZXF.
/ http///pppeeee.ee//e/e/e net/ forum? did
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 377.0:2307.06281, 2023.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 26439-26455, 2024.
Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Martin Jaggi, Hyeji Kim, and Michael Gastpar. Attention with markov: A framework for principled analysis of transformers via markov chains. arXiv preprint arXiv:2402.04116 2024.
Mike A Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series. arXiv preprint 1xXiv:244.111577 2024.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Baolin Peng, Chunyuan Li, Pengeheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint 2xXiv:2334400277 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Nived Rajaraman, Jiantao Jiao, and Kannan Ramchandran. Toward theory of tokenization in Ilms. arXiv preprint arXiv:2404.08335, 2024.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings OCI, 2018.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222 2023.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 2980:22005.09818. 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint a306v: 13549, 2023.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2(3), 2023.
Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle atten- tion. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, and Zongqing La. Unicode: Learning unified codebook for multimodal large language models. arXiv preprint arXiv:2403.09072, 2024.
13

---

Published as a conference paper at ICLR 2025
A THEORETICAL DISCUSSIONS
A.1 INFORMATION Loss OF THE TOKENIZER
Let X denote the original image, V denote the VQ-tokenized sequence, and T denote the final BPE-tokenized sequence. The information loss introduced by the BPE tokenization process can be quantified using conditional entropy:
Lope H - H(XV).
(6)
This quantity represents the increase in uncertainty about the original image X when using BPE tokens T compared to using VQ tokens V. Since
Lope H T)) - (V) = [H
H(T) (H V) H(V),,
(7)
and recall that mutual information can be expressed as:
T(X;V) H(X) + H(V) H(X,V)
(8)
((;T) = ((X) H H(),T).
Substituting back into equation 7 and we get
Lope + (10
I(X H(V)
= (X(X) - I(X;T) -
- I(X;
= =((;; -((;TT).
Therefore, the information loss can be expressed the reduction mutual information between the original image and its tokenized representation. This formulation is particularly useful as it directly quantifies how much information about the original image preserved through tokenization process.
Next, we need to analyze how each BPE merge operation affects the mutual information I(X:T) For any given BPE token sequence T, we can decompose H (X based on individual tokens:
H(I[T) Σ P
(10)
During a merge operation that combines tokens (ti,tz) into tms the change in conditional entropy be expressed as:
AH P( (m) (P( 44)
(11)
For the whole token distribution, the increase in conditional entropy can quantified using the KL divergence:
( H 14, (y) + (L
(12)
where KT.(.) represents the KI. divergence. In this equation,
(X(t,,t) ty) represents the original uncertainty about X given the separate tokens
KL(P(X 14, 4(X (m)) represents the extra uncertainty introduced by merging the
tokens
The equation shows that no other sources of information loss exist beyond these two terms.
This leads an initial upper bound on the total information loss:
Lipe S Σ
(13)
merges
14

---

Published as a conference paper at ICLR 2025
The summation here represents the over all merges during the BPE process. Considering that BPE performs merges based on frequency patterns and with minimum merge frequency threshold Pmnin we can relax the inequality to get more interpretable bound:
Lbpe I
Dipe Degl)) x (-P (-Pinin log (og(Pii))))
(14)
Here, Dogl is the size of the VQ codebook, and Dipel the size the vocabulary after BPE extension.
To put this bound in perspective, consider a typical configuration:
Dall 8192 (VQ codebook size)
Dipel 8192 + 8192 (vocabulary size after BPE extension)
Pmin 0.01 (minimum merge frequency)
The upper bound on information loss for the whole vocabulary would be:
Lope 3 (8192 8192 - 8192) (-0.01 x log(0.01)) ~ 377.3 bits.
(15)
For a single image, the original VQ tokens (32 32 patches) contain 32 32 x log, 8192
13312 bits information, and the per token loss of the extended BPE vocabulary is Lépe/ / Dbpel (Dayl) ~ 0.046 bits. We can calculate that the max loss ratio of the single image is only 32 x 32 x 0.046/13312 ss 0.35%, which is relatively small information loss. Considering the benefits brought by BPE tokenization as discussed earlier, we believe this loss is acceptable.
A.2 FULL PROOFS
We begin by proving equation 3. For any unigram model Q E Qi-gram on the tokens, there exist distributions Q# and Qtok. supported on N and Dict (dictionary of the tokenizer) respectively, such that for all token sequence = (e......)))
Proof of Proposition 1.
[e]
Q(t) Q+ (( II (t,).
r=1
Consequently, for every m N have
1
Lm (Q) =
[log Q# ((flat(X)))))
log Qtok(t)
m²
m
(Eflat(X)
E
[log 2+
E [log Qiok (Xij)]
m
m²
m
376
Llg Qtok (Xx)]
m
Σ T(a) log (or(QQok(a))
2 (T)),
where in (i) we apply the definition of flat()), and in (Hi we use the fact that π is the stationary distribution for each column/row as assumed in Scenario I/Scenario 2. Thus, equation 3 is proved.
Next, we proceed to prove equation 4. For an arbitrary model Q, we have
Lm (Q)
E log(((flat(X
= Ellog( P (flat (X)) ((a)/Q((l((())))) X))) E[oog( P* (flat X))
DRI. (Pth) (Pt)),
2 (.
15

---

Published as a conference paper at ICLR 2025
where Pn, denotes the actual joint distribution of (Xa and PKL(Pm IQQ) denotes the KL diver- gence between Pm and Q. Under Scenario 1, leveraging the independence between columns we derive
H(Pm)
m
Ellog T (X1,)] +
Σ Eloog P(Xi)) X(-1))|
Σ (a) log( (m 1) Σ Σ
T(a) a) log (P(d' ()))
aEE
aEE a'ER
=mH(1) (T) + m(m 30
This result is also applicable under Scenario 2 by switching the roles of i and j. As result, in both scenarios we have
lim inf min (Q)= 2 lim -
1 H(P:)) Ho
m. +00 m
m m=- ?
Conversely, it is evident that
1
min L. (Q) S Lm (P // H(Pm).
m²
Hence,
lim sup min (Q) S lim (m())))
Ho
m-+0c Q
m=+00
This finishes the proof of equation
Proof of Proposition 2. In the one-dimensional case, where the data string generated from an ergodic Markov process, authors in (Rajaraman et al., 2024) have proved the following result:
Lemma A.1 (Theorem 4.1 in (Rajaraman et al., 2024)). Suppose a string of length m is generated from an ergodic data source supported on a funite set A, with a transition kernel P(- -) and
stationary distribution Assume the transition kernel satisfies mina.a'AA P(a' aa) A 8 o.
Then, there exists a tokenizer with a dictionary containing at most D tokens, along with an encoding function enc(-) applied 8, such that
min 1 E[]lo (E(ee((L)Q(encss)))
limsup sup
17
(16)
$ - Σ Σ ((a) a) (1/P(a' aa))
aE a'eA
where log(1/8)/ log(()))9)))))) (D) and D) E M is arbitrary constant that sufficiently large.
Building on the results of Lemma A. we extend these findings to images generated under Scenario or
In both scenarios, the generated image is composed of m parallel Markov sequences, each of length m, sharing the same transition kernel P- and stationary distribution π.
Consider Scenario 1. According to Lemma A. and equation 16, there exists tokenizer equipped with a dictionary, denoted as Dict where Dict < D, and an encoding function, denoted as enccoo(+), and there also exists unigram model Qm Qi-gram for each N, such that
lim sup - (ellog(1/mmmmel (X Xx))))
m-+cc
<
1
Σ
Σ
T(a)P(a' log (I/P(a' a))
1 -
a'ea
1
Ho..
16

---

Published as a conference paper at ICLR 2025
Here X, represents the j-th column of image X and j E {1...., m) is arbitrary. For the unigram model Qm there exist distributions Q and Qmk, supported on N and Dict respectively, such that for all token sequence = (f1,..,, }
[el
Qm = Q (1e)) II
Qtak (t,).
Consequently, is straightforward to derive:
lim sup 1 E m-+00 77
1
log(L/Qmak(I) (t))
1l.c
(17)
(Eencen((X)))
To construct qualified tokenizer for image X, we can simply use the dictionary Dict, and define the encoding function as
enc X) (enCool (X) 1), en (xx))
which concatenates the tokens returned by each column. Subsequently, let Qm E Q1.gram be the unigram model defined by
(t) = Quiif (1t) II (t,) (t,))
Here Quaif is the uniform distribution over {1, 2, m² and Q is previously defined. It then follows that:
6m (2m enc)
1 m
Ellog(1 1/ Q |nce (x)c(()))) +
E
log(1/Qtok(HH) (t))
m
(Een)cal(X..)
2 log)m)
1
E
Σ
log(1 log(1/Qtok(1))
m2
(Eenecui(X
Finally, by equation 17
lim
min m=+00 QEQi-gram
Lm enc)
lim sup P' (Qm enc) S
1 1- Ho:
1
m-
thereby proving equation 5.
The proof Scenario is quite similar and omitted here.
17

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
B.1 ADDITIONAL PSEUDO CODES
The additional functions used in Algorithm 4.1 are shown in Algorithm B.1 and B.2.
Algorithm B.1 Update Adjacency Matrix
1:
procedure UPDATEMATRIX(()
2: A (eros(u, (V, v)
3:
for each d E D do if d is 1D then
4:
5:
for j - 1 to |d] 1 do Aldj,dj+i +- Aldj, dj+1] +
6:
7:
end for
8:
else
9:
for each (ar, y) in d do
10:
Update for neighbors of (x. y)
11:
end for
12:
end
end for
14: return A
end procedure
Algorithm B.2 Find Most Frequent Pair
1: procedure MAXFREQPAIR(AA)
2: U + UpperTri(A)
3: if UU- = 0 then
4:
return (null, 0)
5:
end if
6: (i,j) & arg max U
7:
return ((i,))
8:
end procedure
B.2 HYPERPARAMETERS FOR FIGURE 2(B) AND 2(C)
In the experiment shown in Figure 2(b), we experimented all parameter combinations listed in Table B. for the Transformer model. For the experiment in Figure 2(c) that used BPE tokenizer, except for the layers shown, all other parameters used the minimum values from Table B.1.
Table B.l: Hyperparameters for Transformer models
Hyperparameter
Value / Range
BPE dictionary size
10
batch size
{8, 16, 32}
gradient acc. steps
learning rate weight decay scheduler
2e-3
1e-3
cosine adamw
optimizer dropout number of heads
O
{1. 4, 8, 16)
number of layers
{2, 4, 6, 8}
embedding dimension
(10, 20, 30, 40)
18

---

Published as a conference paper at ICLR 2025
B.3 HYPERPARAMETERS FOR THE VQ-GAN MODEL
The hyperparameters for the VQ-GAN model used in our experiments are shown in Table B.2.
Table B.2: Hyperparameters for the VQ-GAN model
Hyperparameter
Value
embedding dimension
256
codebook size
8192
z. channels
256
resolution
512
dropout
B.4 HYPERPARAMETERS FOR TRAINING MLLM
The hyperparameters for pre-training/ supervised enservvsed--ine-tuuee fine-tuning MLLM are shown in Table B.3.
Table B.3: Hyperparameters for training MLLM
Hyperparameter
PT
SFT
batch size gradient accumulation
2
4
learning rate learning schedule
le-3
3e-5
cosine
cosine
warmup ratio weight decay
0.03
0.03
0
0
epoch optimizer
3
2
AdamW 2
Adam 2
deepspeed stage
B.5 DETAILS OF THE PIPELINE
In this section, we provide more detailed descriptions our pipelines as below.
B.5.1 LLM+VQ+BPE PIPELINE
Our complete pipeline (LLM+VQ+BPE) consists three main components:
Image Quantization: We employ a pretrained VQ-GAN model (with a codebook size of 8192) quantize input images into sequence of token IDs. This process converts images of any size into a 1024-dimensional
tensor.
BPE Image Tokenizer: This component, which distinguishes our approach from previous methods, processes the quantized token IDs using our trained BPE tokenizer. The tokenizer learns to merge frequently co-occurring tokens, creating a new vocabulary that captures higher-level visual patterns.
Multimodal Learning: The processed image token IDs are combined with text token IDs and fed into the LLM (Llama-3.1 1-8B in our experiments). Before training, we expand the token embedding layers of the base model to accommodate both the VQ-GAN token IDs (8192) and the BPE vocabulary (e.g., 4096), along with special tokens for marking image positions.
19

---

Published as a conference paper at ICLR 2025
B.5.2 LL BASELINE
The LLM+VQ variant serves as our baseline, following simpler approach:
It uses the same VQ-GAN model for image quantization as in LLM+VQ+BPE.
The quantized token IDs directly combined with text token IDs, bypassing the BPE tokenizer stage.
The token embedding layers are only expanded to accommodate the VQ-GAN token IDs and special tokens.
B.5.3 TRAINING PROCESS
Both variants undergo the same two-stage training process:
Image Understanding Pretraining (PT): During this stage, train the expanded token embeddings using image caption data. the PT(freeze) setting, we freeze the original text token embeddings, while in PT(full), all embeddings are updated.
Supervised Fine-Tuning (SFT): All parameters are unfrozen and fine-tuned using SFT data.
C ADDITIONAL RESULTS
C.I ADDITIONAL TRAINING WITH LLAMA2 VERSION / FREEZING DURING SFT
To further investigate the effects different base models and varying component freezing strategies, we conducted additional experiments. The results are shown in Table C.I
Table C.I: Full results with additional experiments. The red parts indicate using Llama2 as the base LLM for training. The blue parts indicate freezing the embedding corresponding to the visual token during the SFT stage.
Training type
VQAv2
MMBench
MME"
MMR
POPE
VizWiz
SFT
$1.1
35.9
972.3
231.8
73.8
43.1
Llama3.1+V0
PTiful]+SFT
53.7
37.0
1037.2
261.4
75.3
44.2
PT(freeze 1EX
55.4
37.6
1054.5
277.0
76.0
45.3
Llama2+VQ
PTifreeze 1ex++SFT
$4.0
35.7
991.9
254.2
75.1
44.4
SFT
52.2
35.4
1029.7
269.6
76.3
45.3
SFFfur))+FTT(reeze visual)
31.7
17.8
624.1
171.9
46.4
29.5
Llama3, VVQQ+P
SFF
56.5
38.6
1144.6
284.3
77.3
45.8
PT(freeze (eftree visual)
488.7
35.2
PT(freeze (ex))+FFT
57.1
40.9
1223.5
307.1
79.0
46.0
Uluma2+VQQBB
PT(freeze teX
56.5
38.1
1112.2
277.8
77.9
44.9
Additional scaling (PT)
-Rc/COCO/50.6K)
58.6
42.3
1257.4
314.3
79.8
47.1
+AOKVQA (66.2K)
1288.1
321.4
80.4
47.5
Additional scaling (SFT)
ShuneGPT40
60.2
43.7
1304.5
327.7
80.9
47.8
+ALLaVA Inst (70K)
60.6
44.0
1316.2
331.0
81.3
48.2
20

---

Published as a conference paper at ICLR 2025
C.2 ADDITIONAL EVALUATION
To assess our method's s effectiveness across various capabilities, we performed comprehensive eval- uations analyzing performance across distinct sub-categories. The results shown in Table C.2 and C.3
Table C.2: Comparison of LLM+VQ+BPE and LLM+VQ performance across different categories MME benchmark.
Category
I.M++Q+BPEE
LLM+VQ
Existence
145.00
113.33
Count
120.00
110.00
Position
106.67
103.33
Color
148.33
120.00
Posters
136.24
121.08
Perception
Celebrity
111.76
89.51
Scene
125.00
101.75
Landmark
130.25
110.00
Artwork
112.75
90.50
OCR
87.50
95.00
Commonsense Reasoning
107.14
89.57
Numerical Calculation
62.50
50.00
Cognition
Text Translation
95.00
102.50
Code Reasoning
42.50
35.00
Table C.3: Comparison of LLM+VQ+BPE and LLM+VQ performance across different categories in MLLM-bench The numbers in the table represent the quantity of answers judged be better for each respective model. "Tie" indicates the number of answers where there is no significant difference between the two models' answers.
Category
LLM+VQ+BPE LLM+VQ
Tie
perception
26
34
10
understanding
52
33
25
Applying
27
14
19
Analyzing
49
40
31
Evaluation
12
19
9
Creation
7
9
4
Total
173
149
98
Table C.4: Evaluation results the Nocaps and Flickr30k benchmarks. For Nocaps, chose the validation split. For Flickr30k. we selected a Ik-image split. We compared the impact of adding BPE on the performance of image captioning tasks that require detailed understanding. Additionally, investigated versions of LLM+VQ+BPE (+SF7), which used additional scaling as described earlier, to observe how more instruction tuning affects performance in image captioning tasks.
Nocaps METEOR
Flickr30k METEOR
Model
CIDEr
SPICE
ROUGE-L
CIDEr
SPICE
ROUGE-L
L.L++VQ
76.4
17.2
24.4
51.3
76.4
16.2
24.4
$1.3
LLM+VQ+BPE
75.7
16.9
24.1
50.8
75.7
15.9
24.1
50.8
BLE (+SF7)
80.7
18.2
25.0
52.6
80.7
17.2
25.0
52.6
21

---

Published as a conference paper at ICLR 2025
D ADDITIONAL DISCUSSION ABOUT FURTHER SCALING
Our methods with BPE Image Tokenizer represent a departure from conventional CLIP-based visual encoding methods, introducing both challenges and opportunities that warrant careful discussion.
A critical aspect of evaluating our approach is understanding the implicit data scale disparity be- tween our method and existing CLIP-based MILMs. Traditional visual encoders typically leverage massive pretraining
CLIP's 's original implementation utilized 400M image-text pairs (Rad- ford al., 2021), while subsequent models expanded to datasets like LAION-2B, processing billions of samples across multiple epochs. In contrast, our BPE Image Tokenizer achieves competitive per- formance using approximately 2.78M images, representing roughly 0. of the training data used by encoders like
This substantial difference in pre-training data requirements highlights a fundamental trade-off in MLLM development: while CLIP-based approaches benefit from extensive pre-training, our method demonstrates remarkable efficiency learning visual representations from limited data. The com- parable performance achieved with significantly fewer resources suggests that our approach may offer a more scalable path forward for MLLM development, particularly in scenarios where massive datasets computational resources are not available.
While our current implementation may not achieve sfte-oofthh--ee performance compared to heavily optimized CLIP-based MLLMs, this limitation should be viewed within the broader context of our research objectives. The primary contribution of this work lies not in establishing a new performance benchmark, but rather in introducing and validating novel training paradigm for MLLMs. The theoretical foundations and experimental results presented here serve as proof-of-concept for an alternative approach visual-language modeling.
We acknowledge that achieving state-of-the-arr performance would require:
Substantial expansion of training data
Significant computational resources for large-scale training
Extensive engineering optimizations
These requirements extend beyond the scope of our current investigation, which focuses on estab- lishing the theoretical and practical viability of our approach.
The promising results obtained with limited resources suggest exciting potential for future research. We believe work lays the groundwork for direction in MILMM research, demonstrating that alternative approaches to visual-language modeling can achieve competitive performance even with limited resources. As the field continues evolve, the principles and methods introduced in this paper may contribute to the development of more efficient and scalable MLLMs. Our future work will focus scaling up these initial findings while preserving the core benefits of our approach.
22

---

Published as a conference paper at ICLR 2025
E EXAMPLES
Table E.I: Output examples models trained with and without using BPE tokenizer.
The text
marked in red has obvious hallucinations.
Case
LLM+VQ+BPE
LLM+VQ
Q: Give a brief description of the image.
The image appears be taken from the perspective of someone riding bicycle or motorcycle on dirt road mountain trail. We see handlebars the fore- ground, suggesting this is a first-person view from the rider.
The image features torcyclist riding on dirt road, wearing a helmet and black jacket. They are sur- rounded by group of peo- ple who are taking pictures of the rider as they pass by. The motorcyclist appears be enjoying the moment and capturing the memory.
Q: What in the photo?
This photo shows workspace or home of- fice setup. There are two computer monitors on a desk. The desk is posi- tioned in front of a window. The desk appears to be an L-shaped corner desk, with clean and minimalist design.
In the photo, desk is up with laptop positioned the center, surrounded by a few other laptops and mon- empty black chair placed front of the desk. A potted plant is located on the left side of the scene, creating lively atmosphere.
itors. There
As shown in Table E.1, in these two cases, the version without using the BPE tokenizer, which directly uses VQ-quantized tokens for Llama-3. 1-8B training, frequently exhibits hallucination issues. In contrast, the model trained with the BPE tokenizer information in the images, showcasing better image comprehension capabilities.
accurately describe the
F BROADER IMPACT
The development of our proposed novel paradigm for training MILMs has the potential to advance artificial intelligence and its applications across various domains, including medical imaging and autonomous systems. However, this progress also raises important ethical considerations, such as privacy concerns and the potential for bias in AI systems. As this technology evolves, it crucial to balance innovation with responsible development, necessitating collaboration between researchers, ethicists, and policymakers. Future work should not only focus on technical improvements but also on ensuring that these advancements benefit society while mitigating potential risks. This research thus opens new avenues for capabilities while underscoring the importance ethical considera- tions in technological progress.
23

---

Published as a conference paper at ICLR 2025
G COMPUTING RESOURCES
We list the hardware resources used Table G.I.
Table G.I: Computing resources for our experiments.
CPU
GPU
RAM
Intel 3GHz 64
Nvidia A800 (80GB) 8
1024GB
H LICENSES
In our code, we have used the following libraries which are covered by the corresponding licenses:
Numpy (BSD-3-Clan) license)
PyTorch (BSD-3-Clause license)
Transformers (Apache license)
Numba (BSD-2-CClase license)
24


Reviews:
[{"review_id": "qznLoXDBnR", "reviewer": "Reviewer_ZNHN", "summary": "This paper tried to use BPE for image tokenization. From the results shown to us, there is some improvement.", "questions": "1. Why is there a 0.99 in L770 and L253? Is this made up?\n2. For Figure 2 (b) (c), all settings converge after ~150 iterations. I don't think they make any difference.\n3. Why vocabulary changes from 8k-> 16k, there is a performance drop? I can not find any evidence in the proof that can demonstrate this point.\n4. The proposed BPE is very similar to super pixel or clustering algorithm. Authors should discuss the difference and compare the performance. \n5. In table 5.1, authors can add another classical setting: During SFT, visual encoder is frozen.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. From the results shown to us, there is some improvement.\n2. Experiment settings are clear", "weaknesses": "1. Performance is poor compared to any CLIP style or even DINO style MLLM as the visual encoder.\n2. There is no projector in the experiments. This could be an extreme unfair setting compared to classical pipeline.\n3. I do not think proofs are helpful to understand what is going on in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1dUpKi993n", "reviewer": "Reviewer_e25B", "summary": "This paper proposes to apply Byte-Pair Encoding to visual data, which first encode image into discrete token IDs, and then train the BPE image tokenize to get image tokens with semantic prior (e.g., previously image tokens for 'a white cat' is separated in the sequence of image tokens, after BPE, one token is representing 'a white cat'). The experiments are mainly based on applying BPE image tokenizer training to LLaMA-3.1 and compare on VQAv2, MMBench, etc.", "questions": "1. The LLM+VQ+BPE is also supervised finetuned on LLaVA-One-Vision, etc data, however, is far behind LLaVA-OneVision and other models that trained on these data. Then what's the benefit of this VQ+BPE compared with previous MLLM practices?\n2. Is this VQ+BPE applied to other LLM beyond LLaMA-3.1-8B and get similar observations?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This BPE image tokenization approach is novel and that potentially help the transformer better understand alignment between text and image with a semantic image token.\n2. There is a theoretical analysis on how BPE tokenize benefits transformers learning in Section 3.\n3. The scaling of BPE is reflected in that the model improves when adding larger scale of data such as ShareGPT4, etc.", "weaknesses": "1. The experimental evidences are kind of weak. First, it's far behind current MLLMs SOTA on public benchmarks. For example, the best presented number of proposed model is LLM+VQ+BPE with Additional scaling (SFT) , which achieves 60.6 on VQAv2, 44.0 on MMBench, and 48.2 on VizWiz, which is far behind similar size 7B LLaMA-based MLLMs.\n2. Second, the ablation is not sufficient to show the benefit of BPE image tokenizer. Only one Table results compare LLM+VQ and LLM+VQ+BPE. The details of these two models are not illustrated, e.g., what is exactly implemented for LLM+VQ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1jGRlkjJsr", "reviewer": "Reviewer_UsVN", "summary": "This paper presents a novel BPE image tokenizer that brings byte-pair encoding (BPE) to image tokenization, enhancing multimodal large language models (MLLMs) in aligning visual and textual information. By embedding structural priors into image tokens, the method shows promise in cross-modal tasks and scalability, offering a new approach to unified visual and text processing.", "questions": "1.Applicability of the Theoretical Model: How does a simplified 2D Markov process adequately capture the complex structure of real-world image data?\n\n2.Sensitivity to Information Loss: How is the potential impact of information loss in tokenization, especially for detail-sensitive tasks, theoretically assessed?\n\n3.Task Representativeness and Generalization: How can results on VQA-like tasks ensure performance on precision-demanding tasks like image caption?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.This paper creatively adapts byte-pair encoding (BPE) for images, aiming to make visual data work more seamlessly with text in multimodal models.\n\n2.The approach integrates structural information directly into image tokens, which could help models better understand and align visuals with text, showing solid potential in cross-modal tasks.", "weaknesses": "1.Theoretical framework has several notable limitations:\n  1.1Lack of Multimodal Fusion Analysis: The paper’s theoretical analysis is focused on 2D image data alone and does not delve into how the BPE tokenizer facilitates the fusion of visual and textual information. Multimodal tasks typically require deep semantic and structural alignment across modalities, which is not sufficiently addressed in this analysis. This omission limits the theoretical support for the tokenizer’s efficacy in a multimodal model context.\n  1.2Absence of Analysis on Information Loss in Tokenization: The paper lacks a theoretical exploration of the potential information loss from BPE tokenization, such as the simplification of high-frequency visual details. There is no quantification of how the loss of these details might impact overall model performance. This gap in the analysis leaves the question of how well the BPE tokenizer preserves image details unanswered.\n\n2.A notable limitation of this paper is its focus on evaluating the BPE image tokenizer primarily through VQA-like tasks, which generally require only broad semantic alignment across modalities. While effective for assessing general multimodal comprehension, these tasks may not fully capture the demands of applications like image segmentation or image caption, where finer-grained visual detail and spatial relationships are crucial. Without evaluation on these more intricate tasks, it remains unclear how well the method handles scenarios that require detailed visual representation, potentially limiting its applicability to real-world multimodal use cases that demand high visual fidelity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)