PixelWorld: Towards Perceiving Everything as Pixels
Authors: name: Zhiheng Lyu; name: Xueguang Ma; name: Wenhu Chen
Published: 2025-01-31 17:39:21

Abstract: Existing foundation models typically process visual input as pixels and
textual input as tokens, a paradigm that contrasts with human perception, where
both modalities are processed in a unified manner. With the rise of embodied
and agentic AI, where inputs primarily come from camera pixels, the need for a
unified perception framework becomes increasingly evident. In this paper, we
propose to unify all modalities (text, tables, code, diagrams, images, etc) as
pixel inputs, i.e. "Perceive Everything as Pixels" (PEAP). We introduce
PixelWorld, a novel evaluation suite that unifies all the mentioned modalities
into pixel space to gauge the existing models' performance. Our findings show
that (1) PEAP outperforms baseline with token-based input in multimodal
datasets, benefiting from unified input for better disambiguation, (2)
significant declines in reasoning and coding capabilities across all models
when processing pixel-based input, underscoring the need to enhance foundation
models' perceptual abilities, (3) larger models can maintain strong performance
on non-reasoning tasks under PEAP, while smaller models like Phi-3.5-V suffer
significant performance degradation, (4) the attention pattern of PEAP is
highly aligned with text token input, (5) PEAP can be accelerated significantly
by exploiting the spatial sparsity. We conclude that the existing frontier
models are competent in pixel perception, however, there is still headroom for
improvement. Our code, dataset will be released upon acceptance.