From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities

Authors: Wanpeng Zhang, Zilong Xie, Yicheng Feng, Yijiang Li, Xingrun Xing, Sipeng Zheng, Zongqing Lu
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal Large Language Models, Image Tokenizer, Token Merge

Abstract:
Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models. For further details, visit our website https://github.com/BeingBeyond/Being-VL-0.

Decision: accept (poster)
Reviewer scores: 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730120730779

thread_type:
official_review

<thread_object>
creator: reviewer_usvn

<content>
summary:
This paper presents a novel BPE image tokenizer that brings byte-pair encoding (BPE) to image tokenization, enhancing multimodal large language models (MLLMs) in aligning visual and textual information. By embedding structural priors into image tokens, the method shows promise in cross-modal tasks and scalability, offering a new approach to unified visual and text processing.

strengths:
1.This paper creatively adapts byte-pair encoding (BPE) for images, aiming to make visual data work more seamlessly with text in multimodal models.

2.The approach integrates structural information directly into image tokens, which could help models better understand and align visuals with text, showing solid potential in cross-modal tasks.

weaknesses:
1.Theoretical framework has several notable limitations:
  1.1Lack of Multimodal Fusion Analysis: The paper’s theoretical analysis is focused on 2D image data alone and does not delve into how the BPE tokenizer facilitates the fusion of visual and textual information. Multimodal tasks typically require deep semantic and structural alignment across modalities, which is not sufficiently addressed in this analysis. This omission limits the theoretical support for the tokenizer’s efficacy in a multimodal model context.
  1.2Absence of Analysis on Information Loss in Tokenization: The paper lacks a theoretical exploration of the potential information loss from BPE tokenization, such as the simplification of high-frequency visual details. There is no quantification of how the loss of these details might impact overall model performance. This gap in the analysis leaves the question of how well the BPE tokenizer preserves image details unanswered.

2.A notable limitation of this paper is its focus on evaluating the BPE image tokenizer primarily through VQA-like tasks, which generally require only broad semantic alignment across modalities. While effective for assessing general multimodal comprehension, these tasks may not fully capture the demands of applications like image segmentation or image caption, where finer-grained visual detail and spatial relationships are crucial. Without evaluation on these more intricate tasks, it remains unclear how well the method handles scenarios that require detailed visual representation, potentially limiting its applicability to real-world multimodal use cases that demand high visual fidelity.

questions:
1.Applicability of the Theoretical Model: How does a simplified 2D Markov process adequately capture the complex structure of real-world image data?

2.Sensitivity to Information Loss: How is the potential impact of information loss in tokenization, especially for detail-sensitive tasks, theoretically assessed?

3.Task Representativeness and Generalization: How can results on VQA-like tasks ensure performance on precision-demanding tasks like image caption?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730528560487

thread_type:
official_review

<thread_object>
creator: reviewer_e25b

<content>
summary:
This paper proposes to apply Byte-Pair Encoding to visual data, which first encode image into discrete token IDs, and then train the BPE image tokenize to get image tokens with semantic prior (e.g., previously image tokens for 'a white cat' is separated in the sequence of image tokens, after BPE, one token is representing 'a white cat'). The experiments are mainly based on applying BPE image tokenizer training to LLaMA-3.1 and compare on VQAv2, MMBench, etc.

strengths:
1. This BPE image tokenization approach is novel and that potentially help the transformer better understand alignment between text and image with a semantic image token.
2. There is a theoretical analysis on how BPE tokenize benefits transformers learning in Section 3.
3. The scaling of BPE is reflected in that the model improves when adding larger scale of data such as ShareGPT4, etc.

weaknesses:
1. The experimental evidences are kind of weak. First, it's far behind current MLLMs SOTA on public benchmarks. For example, the best presented number of proposed model is LLM+VQ+BPE with Additional scaling (SFT) , which achieves 60.6 on VQAv2, 44.0 on MMBench, and 48.2 on VizWiz, which is far behind similar size 7B LLaMA-based MLLMs.
2. Second, the ablation is not sufficient to show the benefit of BPE image tokenizer. Only one Table results compare LLM+VQ and LLM+VQ+BPE. The details of these two models are not illustrated, e.g., what is exactly implemented for LLM+VQ.

questions:
1. The LLM+VQ+BPE is also supervised finetuned on LLaVA-One-Vision, etc data, however, is far behind LLaVA-OneVision and other models that trained on these data. Then what's the benefit of this VQ+BPE compared with previous MLLM practices?
2. Is this VQ+BPE applied to other LLM beyond LLaMA-3.1-8B and get similar observations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697138437

thread_type:
official_review

<thread_object>
creator: reviewer_znhn

<content>
summary:
This paper tried to use BPE for image tokenization. From the results shown to us, there is some improvement.

strengths:
1. From the results shown to us, there is some improvement.
2. Experiment settings are clear

weaknesses:
1. Performance is poor compared to any CLIP style or even DINO style MLLM as the visual encoder.
2. There is no projector in the experiments. This could be an extreme unfair setting compared to classical pipeline.
3. I do not think proofs are helpful to understand what is going on in the experiments.

questions:
1. Why is there a 0.99 in L770 and L253? Is this made up?
2. For Figure 2 (b) (c), all settings converge after ~150 iterations. I don't think they make any difference.
3. Why vocabulary changes from 8k-> 16k, there is a performance drop? I can not find any evidence in the proof that can demonstrate this point.
4. The proposed BPE is very similar to super pixel or clustering algorithm. Authors should discuss the difference and compare the performance. 
5. In table 5.1, authors can add another classical setting: During SFT, visual encoder is frozen.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732290423948

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response to all reviewers

comment:
We thank all the reviewers for the efforts in reviewing our paper and providing insightful suggestions! This has been of great help in improving our paper. In response to the concerns raised by the reviewers, we have provided detailed replies in the responses below. 

We have also conducted additional experiments and made revisions to the paper based on the reviewers' suggestions. The newly added content is highlighted in blue text within the revised version. Specifically, we've made the listed revisions:

- Adjusted the y-axis scale in Figure 2(c) to better align with Figure 2(b), preventing potential misunderstanding of the two figures.
- Following reviewer UsVN's suggestion, added Section A.2 to discuss the information loss of our BPE image tokenizer.
- Following reviewer e25B's suggestion, added Section B.5 to provide more detailed descriptions of both LLM+VQ+BPE and LLM+VQ pipelines.
- Following reviewer ZNHN's suggestion, included experiments on freezing visual tokens during the SFT stage, with results presented in Section G.1.
- Following reviewer e25B's suggestion, included experiments using different base LLM (Llama 2), with results presented in Section G.1. Additionally, supplemented extra evaluation results analyzing our method's performance in handling detailed information, with results presented in Section G.2.

Feel free to check the updated PDF paper. If there are still questions, please let us know. We are looking forward to further discussion!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732474205625

thread_type:
official_comment

<thread_object>
creator: area_chair_bgwd

<content>
comment:
Dear Reviewers,

This is a friendly reminder that the discussion period will end on Nov 26th (Anywhere on Earth). If you have not already, please take a careful look at the other reviews and author responses, and comment on whether your original rating stands. Thank you.

Best, AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732809535719

thread_type:
official_comment

<thread_object>
creator: area_chair_bgwd

<content>
comment:
Dear reviewers,

This is a friendly reminder that the discussion period has been extended until December 2nd. If you haven’t yet, we kindly encourage you to review the authors' rebuttal and messages at your earliest convenience and confirm whether your comments have been adequately addressed.

We greatly appreciate your service to this process.

Best, AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733199637798

thread_type:
official_comment

<thread_object>
creator: area_chair_bgwd

<content>
comment:
Dear reviewers ZNHN & e25B,

Could you please review the authors' rebuttal and messages and confirm whether your comments have been adequately addressed?  There are only a few hours left in the discussion period, and your input would be much appreciated.  Thank you.

Best, 
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733234305117

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the rebuttal

comment:
Dear AC and reviewers,

We sincerely thank you for the time and effort you have dedicated to reviewing our work. We greatly appreciate all reviewers' insightful comments and constructive suggestions. In particular, we would like to express our gratitude to reviewer UsVN for the valuable suggestion regarding information loss. This inspired us to supplement our revision with additional theoretical proofs and discussions on information loss, which have further strengthened the theoretical support of our work. We are also grateful to all reviewers for their meticulous questions, which helped us identify potential confusions or misunderstandings for readers. The revisions made based on these reviews have improved the quality of our work.

The core motivation of our research lies in our attempt to achieve unified representations of cross-modal information through BPE-style processing, incorporating structural priors into tokens to enable early-fusion of modal information. We believe this design represents an innovative step forward from traditional MLLM training pipelines, and we are confident that it will spark interest among researchers and inspire follow-up work in this direction.

While we understand that some reviewers may have been too busy during the rebuttal period, which resulted in limited discussion during the rebuttal period, we still **encourage more discussion during the subsequent AC-reviewer discussion phase** to confirm whether our rebuttal has adequately addressed reviewers' questions and concerns.

For your convenience, **to help the AC and reviewers more easily grasp the key points of the entire rebuttal, we provide a summary here**, hoping everyone can have a better understanding.

---

# Our work in brief

We propose a novel BPE Image Tokenizer that applies byte-pair encoding (BPE) principles to visual data, enabling better integration of visual information into MLLMs. Unlike conventional approaches using separate visual encoders, our method directly incorporates structural prior information into image tokens. We provide theoretical analysis showing why this paradigm benefits transformers' learning of 2D sequence data, and validate it through comprehensive experiments.

---

# Reviewers' positive recognitions

The reviewers have recognized several aspects of our work: 

- **From reviewer ZNHN:**
	- Clear experimental settings
	- Observable improvements in the presented results
- **From reviewer e25B:**
	- The novelty of the BPE image tokenization approach that could help transformers better understand text-image alignment
	- The theoretical analysis demonstrating BPE's benefits for transformer learning
	- The demonstrated model improvements when scaling with larger training data
- **From reviewer UsVN:**
	- An innovative approach to adapting byte-pair encoding (BPE) for images
	- The method's potential in facilitating text-image alignment through structural information integration
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new image tokenizer, BPE Image Tokenizer, which merges image token IDs to enhance the incorporation of visual information into MLLMs.  The paper initially received scores of 5,5,6.  Strengths include novel approach, theoretical analysis, and some promising results.  Weaknesses include relatively weak performance compared to state-of-the-art, some limitations in the theoretical framework, and issues with ablation study.  The rebuttal addressed several of these concerns, and the final score was 5,5,8.  Despite multiple requests by the AC, there was little discussion provided by some of the reviewers, including no participation from one of the 5 reviewers.  The other 5 reviewer, during the AC-reviewer discussion phase, notified to the AC that their score is between a 5 and 6 and would not be surprised if the paper is accepted.  The AC carefully reviewed the paper, rebuttal, messages, and feel that despite some shortcomings in the empirical results, the approach is interesting and can bring a novel perspective to the MLLM literature.  Overall, the AC feels that the strengths outweigh the weaknesses, and recommends accept.  Please incorporate all of the promised revisions into the final version.