Glyph: Scaling Context Windows via Visual-Text Compression
Authors: name: Jiale Cheng; name: Yusen Liu; name: Xinyu Zhang; name: Yulin Fei; name: Wenyi Hong; name: Ruiliang Lyu; name: Weihan Wang; name: Zhe Su; name: Xiaotao Gu; name: Xiao Liu; name: Yushi Bai; name: Jie Tang; name: Hongning Wang; name: Minlie Huang
Published: 2025-10-20 17:58:56

Abstract: Large language models (LLMs) increasingly rely on long-context modeling for
tasks such as document understanding, code analysis, and multi-step reasoning.
However, scaling context windows to the million-token level brings prohibitive
computational and memory costs, limiting the practicality of long-context LLMs.
In this work, we take a different perspective-visual context scaling-to tackle
this challenge. Instead of extending token-based sequences, we propose Glyph, a
framework that renders long texts into images and processes them with
vision-language models (VLMs). This approach substantially compresses textual
input while preserving semantic information, and we further design an
LLM-driven genetic search to identify optimal visual rendering configurations
for balancing accuracy and compression. Through extensive experiments, we
demonstrate that our method achieves 3-4x token compression while maintaining
accuracy comparable to leading LLMs such as Qwen3-8B on various long-context
benchmarks. This compression also leads to around 4x faster prefilling and
decoding, and approximately 2x faster SFT training. Furthermore, under extreme
compression, a 128K-context VLM could scale to handle 1M-token-level text
tasks. In addition, the rendered text data benefits real-world multimodal
tasks, such as document understanding. Our code and model are released at
https://github.com/thu-coai/Glyph.