Analyzing the Language of Visual Tokens

Authors: David Chan, Rodolfo Corona, Joonyong Park, Cheol Jun Cho, Yutong Bai, Trevor Darrell
Primary area: foundation or frontier models, including LLMs
Keywords: Tokenization, Analysis, Statistical Methods, Vision and Language

Abstract:
With the introduction of transformer-based models for vision and language tasks, such as LLaVA and Chameleon, there has been renewed interest in the discrete tokenized representation of images. These models often treat image patches as discrete tokens, analogous to words in natural language, learning joint alignments between visual and human languages. However, little is known about the statistical behavior of these visual languages—whether they follow similar frequency distributions, grammatical structures, or topological alignments as natural languages. In this paper, we take a natural-language-centric approach to analyzing discrete visual languages and uncover striking similarities and fundamental differences. We demonstrate that, although visual languages adhere to Zipfian distributions, higher token innovation drives greater entropy and lower compression, with tokens predominantly representing object parts, indicating intermediate granularity. We also show that visual languages lack cohesive grammatical structures, leading to higher perplexity and weaker hierarchical organization compared to natural languages. Finally, we demonstrate that, while vision models align more closely with natural languages than other models, this alignment remains significantly weaker than the cohesion found within natural languages. Through these experiments, we demonstrate how understanding the statistical properties of discrete visual languages can inform the design of more effective computer vision models.

Decision: reject
Reviewer scores: 6, 5, 3, 8

Reviews and discussion:
<thread>
creation date:
1729864960701

thread_type:
official_review

<thread_object>
creator: reviewer_umeh

<content>
summary:
The paper studies the language expressed through “visual tokens” – discrete tokens representing images, that are obtained from an image tokenizer followed by linearization. Such tokens are used jointly with language tokens in multi-modal transformer-based models, such as Chameleon. The paper analyzes various statistical properties of the language induced by visual tokens, such as its entropy, naturality, and topology, in comparison to natural human language and language generated by language models. The paper shows multiple interesting findings highlighting similarities and differences between “visual languages” and natural languages. The authors also include several discussions regarding the potential implications of their findings, suggesting ways in which they can inform the design of more effective vision models.

strengths:
* Multi-modal models, especially those involving vision and language, are of growing usage and interest. The paper takes a unique stance on analyzing these models, viewing visual tokens as language and applying well-established empirical statistics methods.

* I found the analysis very comprehensive, covering 5 different tokenizers, multiple datasets, and more than 6 methods. It also tackles fundamental research questions.

* The paper is clearly written and easy to read, balancing well between concrete results and intuitive interpretations.

weaknesses:
* The main drawback of the paper is that while it discusses multiple potential implications of the reported findings, it does not demonstrate them, leaving practical exploration of these implications for future work. I find the observations interesting and important by themselves, but the paper could be made much stronger with such experiments.

* I may have overlooked it, but there is very little discussion on the differences across visual tokenizers and datasets. If it implies that current tokenizers largely behave similarly (regardless of hyperparameters, etc) then this should be emphasized.

* Not a major weakness, but a point for consideration is that the paper focuses on *discrete* visual tokens, while recent performant models like LLaVA rely on *continuous* tokens. The authors acknowledge that, noting that some of the analyses could be applied to continuous tokens, but they leave the exploration of this for future work. However, I think the paper could be made more impactful by applying its unique analysis to continuous tokens as well and comparing them to the discrete tokens.

questions:
Typos:
* L73 – “we aim to show through these experiments show that”
* L79 – “The first, question that we examine“
</content>
</thread_object>

</thread>
<thread>
creation date:
1730448499905

thread_type:
official_review

<thread_object>
creator: reviewer_1eu7

<content>
summary:
This paper tries to investigate the similarity between language tokens and visual tokens. More specifically, the authors inspect the equivalence of visual and natural languages through an empirical analysis of token distributions, segmentation granularity, and syntactic and semantic structures.

strengths:
1. By examining visual tokens through the lens of linguistic principles, the authors provide a novel framework for analyzing multimodal models, which enriches the discourse surrounding the integration of vision and language.
2. The comparison of visual languages to natural languages, especially in terms of grammatical structure and co-occurrence patterns, yields interesting conclusions about the nature of visual representation and its implications for model design.
3. The paper writing is clear. This clarity aids in understanding the complex interactions between visual and linguistic modalities.

weaknesses:
1. The paper primarily borrows conclusions and formulas from the realm of language models and applies them to visual tokens without offering substantial new insights specific to visual representations. The analysis and discussion often appear superficial, failing to yield novel conclusions regarding the unique characteristics of visual tokens.
2. The study lacks original analytical frameworks or targeted statistical experiments designed specifically for visual tokens. As a result, the manuscript reads more like an experimental record rather than a comprehensive exploration that provides meaningful inspiration or deeper understanding for the reader.
3. The conclusions drawn regarding model design and their implications for improving multimodal task performance are notably unclear. The recommendations do not provide concrete guidance on how to effectively implement these insights in practical model development, leaving the reader uncertain about their applicability in real-world scenarios.

questions:
1. How does your analysis of visual tokens extend beyond existing conclusions from the NLP domain? What specific novel insights can you offer regarding the unique properties of visual tokens?
2. What additional analyses or experiments could be conducted to deepen the understanding of visual tokens? Are there specific hypotheses or exploratory questions that you believe warrant further investigation?
3. Can you clarify how the conclusions drawn in your paper can be practically applied to improve the design of multimodal models? What specific recommendations do you have for researchers looking to leverage your findings in their model development?
4. In your comparisons of visual languages and natural languages, what specific aspects do you believe are most critical to consider when developing models that integrate both modalities? How do these aspects influence the training strategies employed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730613350704

thread_type:
official_review

<thread_object>
creator: reviewer_ok1p

<content>
summary:
The paper analyzes the discrete visual tokens learned with VQ-VAEs and investigates several of their properties, such as their frequency distributions, grammatical structures, and topological alignments with natural languages. Some of their findings include: 1) 1-gram and 2-gram visual tokens do not follow Zipf's law, but longer n-grams match it better. 2) visual tokenizers are good at capturing part-level representations. 3) Context-free grammar may not as accurately represent "visual languages" as natural languages. 4) "vision languages" align less with each other than with natural languages.

strengths:
1. The idea of analyzing the visual tokens learned by VQ-VAE models and investigating their properties using natural language tools is interesting and could be inspiring.
2. The writing is clear, with each research question and findings clearly stated in each of the sections.

weaknesses:
1. The paper presents various findings and suggests potential implications and directions for future work. However, it lacks follow-up experiments to support these hypotheses, making the claims less convincing and raising doubts about the practical value of the findings.
2. The approach of applying language tools directly to visual tokens is questionable, especially as the findings suggest that visual tokens may not inherently exhibit natural language structures. Additionally, it remains uncertain whether applying 1D-based methods to 2D visual tokens is appropriate.
3. Overall, it seems to me that the results presented do not convincingly demonstrate that using language tools on visual tokens is a valid method for analysis or clarifying the implications of their similarity or dissimilarity to natural language structures. The paper provides broad analyses, but a deeper focus on fewer points with more experiments would strengthen the claims.

questions:
1. How do you define n-grams for visual tokens obtained from 2D images?
2. Is it possible that "vision languages" actually follow natural language properties, but require vision-specific analysis tools or adaptations of existing tools to reveal them?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730937911444

thread_type:
official_review

<thread_object>
creator: reviewer_cqku

<content>
summary:
This paper looks at the statistical properties of "visual languages," where images are broken into discrete tokens like words in a sentence, used in multimodal models such as transformers. The authors explore whether these visual tokens behave similarly to natural language in terms of frequency distributions, grammatical structures, and alignment. Their key findings are that visual tokens follow a Zipf-like distribution with higher entropy, lack cohesive grammar, and mainly represent parts of objects, aligning only partially with natural language. These results suggest that visual languages have unique characteristics, which might benefit from specialized model designs to handle them effectively.

strengths:
1. This paper is well-oragnized and offers a fresh perspective by treating visual tokens as discrete elements analogous to words in natural language.
2. The experiments are well-executed, with thorough empirical analysis across several datasets and tokenization methods.
3. The work has significant implications for multimodal model design, suggesting that unique features of visual tokens may require new model designs for better performance in vision-language tasks.

weaknesses:
1. While the paper evaluates various tokenization methods (e.g., VQ-VAE, Chameleon), it could benefit from exploring alternative tokenization strategies, especially non-discrete or hybrid methods.
2. The study primarily relies on commonly used datasets (e.g., MS-COCO, ImageNet) that may not fully capture the diversity and complexity of visual scenes in real-world multimodal applications. Including more varied datasets with richer visual and contextual details.

questions:
The authors can consider to include more varied datasets with richer visual and contextual details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732319786913

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We're following up on the rebuttal for our paper, "Analyzing the Language of Visual Tokens." We appreciate the time and effort you've invested in reviewing our work.

In our rebuttal and the updated paper, we hope that we've thoroughly addressed the concerns and suggestions raised in the initial reviews. We are happy to discuss any of these points further, and we are eager to engage in more discussion during the official ICLR discussion period. If you find that all your questions have been resolved, we kindly ask you to consider reflecting this in the scores.

Thank you once again for your thoughtful feedback and consideration!

Best regards,
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper tackles an interesting problem: that of analyzing the "structure" within sequential vision models, from a linguistic point of view. The main finding of the statistical evaluation is that visual tokens follow a Zipf distribution with higher entropy and lack cohesive grammar. While the study is interesting, the work only speculates on the implications of this finding, and the finding by itself is very un-surprising. The paper received very mixed reviews, but the highest-rating reviewer shared concerns regarding the practical implications of this work. Given that I also share this major concer, I recommend this paper for rejection.