JPEG-LM: LLMs as Image Generators with Canonical Codec Representations

## Abstract

Abstract Recent work in image and video generation has been adopting the autoregressive LLM architecture due to its generality and potentially easy integration into multi-modal systems. The crux of applying autoregressive training in language generation to visual generation is discretization—representing continuous data like images and videos as discrete tokens. Common methods of discretizing images and videos include modeling raw pixel values, which are prohibitively lengthy, or vector quantization, which requires convoluted pre-hoc training.
In this work, we propose to directly model images and videos as compressed files saved on computers via canonical codecs (e.g., JPEG, AVC/H.264). Using the default Llama architecture without any vision-specific modifications, we pretrain Jpeg-LM from scratch to generate images (and Avc-LM to generate videos as a proof of concept), by directly outputting compressed file bytes in JPEG and AVC formats. Evaluation of image generation shows that this simple and straightforward approach is more effective than pixel-based modeling and sophisticated vector quantization baselines (on which our method yields a 31% reduction in FID). Our analysis shows that Jpeg-LM has an especial advantage over vector quantization models in generating long-tail visual elements. Overall, we show that using canonical codec representations can help lower the barriers between language generation and visual generation, facilitating future research on multi-modal language/image/video LLMs.

## 1 Introduction

With large language models (LLMs) the field of NLP has shifted to multi-task processing (e.g., machine translation, code generation, action planning) using a single LLM with little data needed for adaptation (Ouyang et al., ). We envision that future research will continue shifting to multi-modal multi-task processing, where text and visual data are mixed.
However, current paradigms of generating images and videos differ substantially from text generation, requiring specialized and complicated training and representations (Van Den Oord et al., ; Rombach et al., ; Peebles and Xie, ).
In this work, we simplify the task of image and video generation by using the exact autoregressive transformer architecture as in mainstream LLMs (Radford et al., ), over canonical and universal codecs: JPEG for images (Wallace, ), and AVC/H.264 for videos (Wiegand et al., ).

The key obstacle to training autoregressive models for image and video generation is *discretization*, as continuous data like images and videos need to be represented as discrete tokens.
Current generative vision models that follow autoregressive language modeling objectives (Bengio et al., ) often adopt vector quantization (VQ) to encode images or videos to some learned latent codes and then apply language models (Van Den Oord et al., ; Ramesh et al., ; Yu et al., ; Yan et al., ; Yu et al., ).(^1^11The other major line of generative vision models are diffusion models, a score-based, non-autoregressive method (Song and Ermon, ; Ho et al., ; Rombach et al., ; Peebles and Xie, ). Since the diffusion objectives are drastically different from the language modeling objective, it is challenging to integrate them in a multi-modal setup (e.g., with regular language models). While not a main focus of this work, we include comparisons with diffusion models in our later experiments as a secondary evaluation.)
However, VQ methods often demand sophisticated tokenizer training that requires a careful hyperparameter selection for vision-specific modules (e.g., downsampling factor in convolutions) and balancing across several losses (Van Den Oord et al., ; Esser et al., ).
VQ also involves a two-stage, non-end-to-end learning process (first the neural tokenizer, then the latent code LM).
This makes downstream adaptation of the models less flexible (e.g., tuning the VQ tokenizer interferes with the learned latent code LM).
Overall, the use of conventional LLM architectures (end-to-end autoregressive sequence modeling) as generative vision models is not yet straightforward.

The seminal work of ImageGPT (Chen et al., ) attempted to bridge this gap by using a regular GPT architecture to model pixels sequentially.
They have shown a small-scale success at a very low resolution of 32x32 pixels.
More realistic images at a size of 256x256 would require modeling a prohibitive amount of tokens in each sequence (65K or 196K tokens depending on color modes), not to mention videos. This hinders the method’s wider adoption by the field.

In this work, we tackle the problem of training LLM architectures for image and video generation where the essential discretization neither adds significant complications to the pipeline like VQ methods, nor is computationally prohibitively expensive like ImageGPT.
Specifically, we use canonical file encodings/codecs—JPEG for images (Wallace, ), and AVC/H.264 for videos (Wiegand et al., )—as non-neural preprocessors that discretize data. We show that codec-based representations greatly mitigate the sequence length limitation while being simple and effective.
This design enables us to train a vanilla transformer with the conventional language modeling objective for image and video generation in a realistic setup.

We pretrain two 7B models with a Llama-2 architecture (Touvron et al., ), named Jpeg-LM and Avc-LM, that can generate 256x256 images and 256x144 videos with 15 frames, with an average context length of 5K and 15K, respectively.
In our main image modeling/generation evaluations, we show that Jpeg-LM surpasses strong VQ-based models in generation quality (an average of 31% FID reduction) and produces surprisingly realistic qualitative examples.
Our results also show Avc-LM can generate videos with realistic movements.
Furthermore, we analyze in which aspects Jpeg-LM is particularly stronger than VQ models and discover that our non-neural, training-free codec representations are more competent in capturing long-tail elements in images (e.g., human faces/eyes and text characters in small sizes).

Overall, this work presents how conventional LLM architectures can be used as generalized models towards visual generation.
Our approach using canonical codecs does not incur vision-specific complications in the pipeline or suffer from sequence length infeasibility seen in prior work.
Compared to the baselines, our models are much simpler to train and more effective.
Following the previous efforts in unifying detached language-based tasks, our method helps pave the way to a unification of multiple modalities, facilitating the exploration of porting LLM techniques (e.g., alignment, scaling, efficiency, security, etc.) to all modalities.

## 2 Background

In this work, we explore autoregressive image generation as a straightforward extension of prominent LLM setups (Radford et al., ).(^2^22As a proof of concept, we mainly explore autoregressive modeling in visual generation only (images and videos, without text-conditioning), while future work may explore more diverse multi-modal setups.)
Conventional language modeling (Bengio et al., ) models the likelihood of sequential data autoregressively.
Specifically, given a sequence of discrete tokens $x_{1},x_{2},\cdots,x_{N}$ (or $\boldsymbol{x}_{1:N}$), a language model models $p(\boldsymbol{x}_{1:N})=\prod_{i=1}^{N}p(x_{i}\mid\boldsymbol{x}_{1:i-1})$, an objective used in most mainstream LLMs.
The key of applying language modeling to visual generation is how to discretize continuous data $\boldsymbol{x}$ like images and videos to discrete tokens $\boldsymbol{x}_{1:N}$ like in language.
Below we give an overview of two prominent approaches to the discretization of images.

### 2.1 Pixel values: ImageGPT

ImageGPT (Chen et al., ) is an image generation model based on a conventional LLM architecture (GPT-2).
The images are discretized as a sequence of pixel values (integers 0–255) from the upper-left to the bottom-right pixel (raster scan).
Since there are three channels of colors for each pixel, to reduce the number of tokens in each pixel sequence, ImageGPT clusters pixel colors to 512 distinctive clusters (i.e., for each pixel, three values from 0 to 255 are converted to one value from 0 to 511).

ImageGPT models the probability of pixel sequences autoregressively: $p(\operatorname{pixel-value}(\boldsymbol{x})_{i}\mid\operatorname{pixel-value}
(\boldsymbol{x})_{1:i-1})$.
This is an expensive process, and ImageGPT only models and generates 32x32 images.
Images with a more realistic resolution like 256x256 would require 65K tokens for each image (or 196K tokens without color clustering), a prohibitive sequence length for LLMs.

### 2.2 Latent codes: Vector-Quantization models

Vector-quantization (VQ) operates as a two-stage process, tokenizer training and language model training (Esser et al., ; Ramesh et al., ).
We take VQ-VAE as our example tokenizer which discretizes continuous images (Van Den Oord et al., ).
The tokenizer first learns an encoder $E$ to project an image $\boldsymbol{x}$ to spatial features $E(\boldsymbol{x})$.
Then for each feature $\boldsymbol{e}$ in $E(\boldsymbol{x})$, it is quantized to $\boldsymbol{\hat{z}}$ by looking up the nearest neighbor in a learned codebook $\mathcal{Z}$: $\boldsymbol{\hat{z}}=\operatorname{quantize}(E(\boldsymbol{x}))=[\operatorname
{argmin}_{\boldsymbol{z}_{k}\in\mathcal{Z}}\lVert\boldsymbol{e}-\boldsymbol{z}
_{k}\rVert_{2}^{2}]_{\boldsymbol{e}\in E(\boldsymbol{x})}$.
The index $k$ of the nearest entry in codebook $\mathcal{Z}$ for each spatial feature forms the sequence of VQ latent codes. A decoder $G$ is then learned to reconstruct the original image from the quantized representations.
Overall, VQ-VAE learns an encoder $E$, decoder $G$, and codebook $\mathcal{Z}$, with three distinct losses: reconstruction loss, codebook loss, and commitment loss.
$L_{\text{VQ-VAE}}=\lVert\boldsymbol{x}-G(\boldsymbol{\hat{z}})\rVert_{1}+
\lVert\operatorname{sg}[E(\boldsymbol{x})]-\boldsymbol{\hat{z}}\rVert_{2}^{2}+
\beta\lVert\operatorname{sg}[\boldsymbol{\hat{z}}]-E(\boldsymbol{x})\rVert_{2}
^{2}$.
An effective VQ tokenizer needs a large amount of training data, proper hyperparameters for the vision-specific modules (e.g., downsampling factor in convolutional encoder $E(\cdot)$), and a careful balance between the different losses (e.g., in $L_{\text{VQ-VAE}}$), which add significant complications to the pipeline.

A language model architecture can then be trained over the VQ latent codes (a sequence of index $k$ above) as a generative vision model:
$p(\operatorname{VQ-code}(\boldsymbol{x})_{i}\mid\operatorname{VQ-code}(
\boldsymbol{x})_{1:i-1})$.
Notably, since the training of language model comes after and depends on the VQ tokenizer, a post-hoc update to the VQ tokenizer is challenging since it would lead to a non-trivial retraining or adaptation of the trained language model.
Indeed in §5.3 we find that the VQ tokenizer, though trained with a large amount of data, still struggles with long-tail elements in the images and is hard to be optimized once and for all.

For simplicity and end-to-end adaptability, we propose to discretize continuous image and video data via canonical codecs.

Figure: Figure 1: Jpeg-LM and Avc-LM are simple autoregressive transformers that directly model and generate canonical file encodings.
Refer to caption: extracted/5803872/resources/jpeglm_2408_fig.png

## 3 Jpeg-LM and Avc-LM

Though images and videos are continuous data and naturally have 2D or 3D data structures, they are stored as files on computers efficiently via compression/codecs, which leads to a discrete 1D representation.
We aim to explore whether standard
LLM architectures can directly learn to model and generate canonical vision file encodings, which can subsequently be read/opened as generated images or videos.
Generation in this paradigm would greatly mitigate the sequence length infeasibility in ImageGPT while being simple and end-to-end trainable compared to VQ methods.
Moreover, canonical file encodings/codecs are often non-neural and training-free and are robust to distributional shifts (§5.3).
In this work, we choose the most popular and established file encodings/codecs for images and videos, JPEG (Wallace, ) and AVC/H.264 (Wiegand et al., ), respectively.(^3^33For images, PNG is also a common format. However, unlike the lossy JPEG, PNG is a lossless compression method (similar to ZIP) and often results in less effective compression and much longer sequences than JPEG.)

### 3.1 Canonical codecs: JPEG and AVC/H.264

Canonical non-neural codecs like JPEG and AVC have a high-level intuition to compress signals that are less perceptible to human eyes more aggressively.
JPEG has three main steps to encode each image: discrete cosine transform (DCT), quantization, and entropy coding.
DCT converts each image patch to a weighted combination of a preset of patches containing low- and high-frequency patterns.
Quantization zeroes out some high-frequency patterns from the weighted combination, since human eye is not good at perceiving them.
Entropy encoding such as Huffman coding is then used to reduce the total numbers/bits representing the patches/images.(^4^44A further intuitive and interactive description can be found at [https://parametric.press/issue-01/unraveling-the-jpeg/](https://parametric.press/issue-01/unraveling-the-jpeg/) (Shehata and Conlen, ).)

AVC (H.264) operates on patches (macroblocks) of video frames.
Each patch can be encoded using blocks of pixels that are
already encoded within the current frame (intra-frame prediction) or using blocks of pixels encoded in other frames (inter-frame prediction with motion estimation).
The prediction is then subtracted from the current patch to form a residual.
The residual then goes through a process similar to JPEG, involving DCT, quantization, and bitstream encoding.
The encoded content is a crucial part to the subsequent container files like MP4.

Both codecs have been used widely for decades and substantially compress the data (and thus sequence length) compared to raw pixel modeling (in our setup 40x in JPEG and 110x in AVC).
Our focus is to use these canonical codecs as off-the-shelf tools to convert images and videos to sequences of discrete bytes efficiently.(^5^55Both codecs operate at bits level at the core (due to entropy encoding), but modeling at bytes level is effective according to our experiments.)
We wish to fit an LLM to implicitly learn the grammars and semantics of the canonical codecs.

### 3.2 Jpeg-LM and Avc-LM

JPEG and AVC convert images and videos to bytes.
Most of these bytes represent the image and video content after entropy encoding. However, there are also metadata and special patch/macroblock separators that are invariant across images or videos and use up multiple bytes.
To address them along with other unknown frequent byte combinations that are compressed suboptimally by entropy encoding (e.g., by JPEG’s standard, fixed Huffman tables),
we further extend the default byte vocabulary (256 discrete values) *slightly* with byte-pair encoding (BPE), a standard preprocessing scheme in LLMs, which merges bytes appearing together frequently to a new single token.(^6^66More precisely, for the metadata/headers in the byte sequence that are well-known to be redundant across examples (e.g., JPEG quantization and Huffman tables), we remove them in the preprocessing and later add them back to the generated bytes from the model. For more complicated codecs like AVC, we let BPE handle such metadata.)
Since JPEG and AVC produce sequences of variable lengths based on the content of images and videos, special beginning-of-sequence and end-of-sequence tokens are also added to the vocabulary.
The entries in the vocabularies are considered as our JPEG/AVC tokens.

Given an image $\boldsymbol{x}$, we propose Jpeg-LM to model $p(\operatorname{JPEG-token}(\boldsymbol{x})_{i}\mid\operatorname{JPEG-token}(
\boldsymbol{x})_{1:i-1})$.
Given a video $\boldsymbol{x}$, we propose Avc-LM to model $p(\operatorname{AVC-token}(\boldsymbol{x})_{i}\mid\operatorname{AVC-token}(
\boldsymbol{x})_{1:i-1})$.
We use conventional LLM architectures (autoregressive transformers) without any vision-specific modifications (no convolutions, no 2D positional embeddings) to maximize the models’ generality.

Figure: (a) Prompt
Refer to caption: extracted/5803872/resources/tmp2_main_show/rainier_partial.jpg

## 4 Experimental Setup

### 4.1 Jpeg-LM

We pretrain a 7B Llama-2 model (Touvron et al., ) from scratch using 23M 256x256 images.
JPEG encodes each image with a quality factor of 25 (qualitative illustration in §5.3).(^7^77[https://pillow.readthedocs.io/](https://pillow.readthedocs.io/))
We first use 10K images to derive 320 BPE tokens as our vocabulary entries.
On average, each image in our training data leads to 5K tokens.
For batching efficiency, we concatenate all sequences in the dataset and chunk in sequences of length 12K.
In total, we have 9.5M sequences and thus 114B JPEG tokens (for each epoch).
The model is trained approximately for two epochs with a maximum learning rate of 3e-4.

### 4.2 Avc-LM

As a proof of concept that canonical video codecs can be used for video generation as well,
similar to Jpeg-LM, a 7B Llama-2 model is pretrained from scratch as Avc-LM using 2M 256x144 videos subsampled from Bain et al. ().
Due to the scope of experiments, we only keep the first 5 seconds of each video with 3 frame-per-second (thus 15 frames in total). The video is then processed with AVC/H.264 codec with a constant quantization parameter 37.(^8^88[https://ffmpeg.org/](https://ffmpeg.org/))
We use 10K videos to derive 1024 BPE tokens as the vocabulary entries.
On average, each video in our training data has 15K tokens.
We perform data concatenation and chunk in context lengths of 32K for efficient batching.
In total, we have 1.3M sequences and thus 42B AVC tokens.

### 4.3 Image generation baselines

#### VQ transformer

We use a pretrained VQ tokenizer from Tang et al. (), which used 200M images (ITHQ-200M, closed source dataset) to train a VQ-VAE model.
This VQ tokenizer processes each image in the 23M image training set for our Jpeg-LM (vocabulary size 4096, sequence length 1024).
We then train a 7B Llama-2 transformer with the same configuration as in Jpeg-LM.
We use this VQ model as a main comparison to our Jpeg-LM throughout this work.

#### ImageGPT + super-resolution

ImageGPT uses GPT-2 XL as its underlying architecture.
The pretrained model in (Chen et al., ) is trained over 14M 32x32 images from ImageNet.
For a comparable evaluation, we use a super-resolution model (Rombach et al., ) over ImageGPT’s output.(^9^99The pretrained model provides 4x super-resolution. In our pilot study, we find performing a 4x super-resolution, followed by a 0.5x downsample, then another 4x super-resolution yields the best result for the 32^2-to-256^2 conversion.)

#### Diffusion

Though not a focus of this work, we include two variants of diffusion models in the baselines, Stable Diffusion (inpainting optimized) (Rombach et al., ) and VQ diffusion (Gu et al., ; Tang et al., ). Both diffusion models can take partial images (through masking) and generate completed images, a setup we use across models in later evaluations.
These baseline diffusion models are smaller in model size (~1B) but consume orders of magnitude more training data (200M–5B).
They only serve as a secondary reference, and our focus is on comparing autoregressive image generation models under mainstream LLM paradigms.

**Table 1: Zero-shot, partial-image-conditioned, FID evaluation on ImageNet-1K (lower is better). $r_{\text{prompt}}$ indicates the ratio of the image passed to the model as prompt. Best results among the autoregressive models are in bold fonts (reference diffusion results are italicized if better).**
|  | $r_{\text{prompt}}=0.25$ | $r_{\text{prompt}}=0.5$ | $r_{\text{prompt}}=0.75$ |
| --- | --- | --- | --- |
| Stable Diffusion (inpaint) | 266.71 ($\pm$1.67) | 132.98 ($\pm$0.53) | 58.17 ($\pm$0.10) |
| VQ Diffusion | 252.42 ($\pm$0.20) | 125.16 ($\pm$0.26) | 57.49 ($\pm$0.25) |
| ImageGPT (super-resolution) | 289.48 ($\pm$0.61) | 262.76 ($\pm$0.48) | 258.11 ($\pm$0.69) |
| VQ Transformer | 302.92 ($\pm$0.29) | 172.73 ($\pm$0.21) | 71.88 ($\pm$0.19) |
| Jpeg-LM | 272.12 ($\pm$1.24) | 123.09 ($\pm$0.28) | 34.21 ($\pm$0.21) |

## 5 Results

In works of language modeling, a fundamental evaluation is to collect a set of validation data, use the prefixes of data as prompts to the pretrained language model, and sample from the language model for a completion (Holtzman et al., ; Meister et al., ).
The completions are then evaluated for their quality against the gold validation data through distance metrics like Mauve score (Pillutla et al., ).

In this work, since we focus on vision-modality-only models with LLM architectures, we retain partial images (and later partial videos) as prompts to our models and evaluate their completions.
Given a prompt ratio $r_{\text{prompt}}$, the autoregressive image generation models condition on $\operatorname{discretization}(\boldsymbol{x})_{1:(r_{\text{prompt}}\times N_{
\text{tokens}})}$ for the generation.(^10^1010More specifically, the fixed-length VQ transformer and ImageGPT condition on $\operatorname{discretization}(\boldsymbol{x})_{1:(r_{\text{prompt}}\times N_{ \text{tokens}})}$ and generate $\operatorname{discretization}(\boldsymbol{x})_{(r_{\text{prompt}}\times N_{ \text{tokens}}):N_{\text{tokens}}}$. Variable-length Jpeg-LM conditions on $\operatorname{discretization}(\boldsymbol{x})_{1:\operatorname{patch-position} (r_{\text{prompt}}\times N_{\text{patches}})}$ and generates until a EOS token is produced. Throughout the work, sampling from autoregressive transformers is by default with $\text{top-}p=\{0.9,1.0\}$ and $\text{top-}k=\{40,80\}$.)
Throughout the evaluations, the comparison between Jpeg-LM and VQ transformer would be the most direct, as they share the same paradigm, model size, and training data (except that VQ transformer uses substantially more data in the tokenizer training stage).

Figure: (a) Original
Refer to caption: extracted/5803872/resources/landscape_a.png

### 5.1 Qualitative analysis

In Figure 2, we show the generation samples from Jpeg-LM along with baseline models over independently sourced data outside existing training sets.
We observe that by directly outputting JPEG file bytes, Jpeg-LM can generate surprisingly realistic facial expressions (especially the eyes, compared to the strong VQ transformer), landscape, common objects, texts in image forms, etc.
Figure 6 and Figure 7 show further examples of Jpeg-LM and VQ transformer on unconditional generation.

### 5.2 Quantitative results

In Table 3, we show prompting Jpeg-LM, VQ transformer, and other baselines with different levels of partial images in ImageNet-1K (Russakovsky et al., ).
The FID evaluation (Heusel et al., ) contains 5000 randomly sampled images from ImageNet-1K’s validation set.
This is *zero-shot* generation (w.r.t. models’ training sets) and without class-conditioning.
Experiments were done three times with different seeds.
Jpeg-LM consistently outperforms the VQ transformer in all prompting ratios. It mostly surpasses diffusion baselines with inpainting capabilities as well.

In Table 3, we show prompting the models with partial images in FFHQ (Karras et al., ).
This is also a *zero-shot* setup without training to the FFHQ distribution and is evaluated on 1000 randomly sampled FFHQ images.
Jpeg-LM consistently outperforms the VQ transformer and other baselines.

In Table 3,
we further validate our findings on fully unconditional generation with Jpeg-LM and VQ transformer. Since they were trained on the same training data, we can compare their FID of unconditional generation w.r.t. our held-out, i.i.d. evaluation set.
We again observe that Jpeg-LM achieves a better FID.

These findings show Jpeg-LM’s overall competence as a image generation model with a pure LLM architecture modeling canonical file encodings.

### 5.3 Why Jpeg-LM ? A case study over long-tail elements in images

To further explore in which aspects our Jpeg-LM excels compared to the baselines, especially the VQ transformer, we first compare how data is processed/compressed before being trained in transformers in Jpeg-LM and VQ models.

#### JPEG vs. VQ compression

Jpeg-LM and VQ transformers can both be interpreted as first performing compression and then autoregressive modeling.
The VQ model, unlike the non-neural JPEG compression, trained its VQ-VAE quantizer with a large amount of data (200M images in our case).
In Figure 4, we observe that both compression methods are relatively successful in compressing and decompressing general scenes like nature/landscape backgrounds.
However, we find VQ suffers in small but highly perceptible elements in the images, like human faces or eyes.
For images that contain small text characters, we observe the image degradation in VQ also happens in a non-predictable way, generating seemingly clear but uninterpretable text characters.
On the other hand, the image degradation due to the non-neural, training-free JPEG compression happens in a predictable manner, arguably more preferrable, especially when images contain long-tail elements with important meanings.

**Table 4: Zero-shot, partial-image-conditioned, FID evaluation on downscaled FFHQ (for both FID and $\Delta$, lower is better). An increased gap between Jpeg-LM and the VQ transformer shows Jpeg-LM is more robust to small but meaningful long-tail elements.**
|  | $r_{\text{prompt}}=0.375$ | $r_{\text{prompt}}=0.5$ |
| --- | --- | --- |
| Stable Diffusion (inpaint) | 136.28 ($\pm$2.48) | 120.54 ($\pm$6.46) |
| $\Delta_{\text{downscaled}-\text{original}}$ | $+$20.98 | $+$30.72 |
| VQ Diffusion | 83.63 ($\pm$1.16) | 47.90 ($\pm$1.12) |
| $\Delta_{\text{downscaled}-\text{original}}$ | $+$22.75 | $+$7.32 |
| ImageGPT (super-resolution) | 46.67 ($\pm$0.62) | 40.46 ($\pm$0.70) |
| $\Delta_{\text{downscaled}-\text{original}}$ | $-$15.06 | $-$14.82 |
| VQ Transformer | 56.33 ($\pm$0.86) | 47.94 ($\pm$0.21) |
| $\Delta_{\text{downscaled}-\text{original}}$ | $+$3.08 | $+$6.79 |
| Jpeg-LM | 35.80 ($\pm$0.17) | 26.25 ($\pm$0.45) |
| $\Delta_{\text{downscaled}-\text{original}}$ | $-$0.35 | $-$0.90 |

#### Quantitative analyses on long-tail elements

In Figure 4, we first show the per-class FID in our ImageNet-1K generation experiments.
For each class of images, we calculate the difference between their FID with Jpeg-LM generations and FID with the VQ transformer generations.
We also estimate the frequency/coverage of each class of images on the internet by querying Google image search and logging the total number of returned results.
We observe a statistically significant correlation between the per-class FID difference and the class frequency.
The more advantage we observe in Jpeg-LM over the VQ model, the less frequent the corresponding class is.
In other words, Jpeg-LM excels relatively more in long-tail sub-distributions.

In Table 4, we further intervene on
the FFHQ images by downsizing them (to 0.5x, while padding the images with black background to keep the overall size), aiming to test different models’ performance on smaller
visual concepts (e.g., small human faces).
Such concepts, though small in size, can still be highly perceptible by humans and contain important meanings.
We thus want the models to be robust on them.
We perform similar prompted image generations with Jpeg-LM, VQ transformer, and other baseline models.(^11^1111The FID is measured on the active proportion of the images, excluding the black paddings.)
We find that Jpeg-LM still consistently outperforms the VQ transformer (and other baselines as well).
Especially, Jpeg-LM achieves slightly better performance while VQ transformer becomes worse compared to the experiments with original image size. These deltas in opposite directions highlights the robustness of Jpeg-LM.

These findings show that Jpeg-LM not only has a promising performance overall, but specially strong with long-tail visual elements in the images.

Refer to caption: extracted/5803872/resources/example_vid_gen/vid1.png

### 5.4 Proof-of-concept video generation

One advantage of using canonical file encodings in LLM paradigms for vision generation is simplicity.
From Jpeg-LM that generates images, we naturally take one step further and train a video generation model, Avc-LM, that models canonical video codecs (AVC/H.264) with autoregressive transformers.

As a proof of concept, we prompt Avc-LM with partial videos (i.e., frames) from a held-out set from our training data and investigate the model completions.
In Figure 5 (along with §C), we show qualitative examples generated by Avc-LM. We observe that Avc-LM can capture the motion of moving objects reasonably.

## 6 Related Work

Current image and video generation models often adopt an autoregressive or diffusion approach. The autoregressive approach can build upon pixel-based representations as explored in Van Den Oord et al. (); Van den Oord et al. (); Chen et al. (). These methods suffer from prohibitively long sequences and only operate on low-resolution images. The autoregressive approach can also build upon vector quantization, which involves a sophisticated pre-hoc tokenizer training in addition to the autoregressive model (Van Den Oord et al., ; Esser et al., ; Ramesh et al., ; Yu et al., ; Yan et al., ; Yu et al., ; Mentzer et al., ; Lu et al., ; Liu et al., ).
Diffusion models generate images or videos by an iterative denoising process, and they have specialized objectives and architectures that are challenging to be incorporated to regular LLM paradigms to form multi-modal systems (Song and Ermon, ; Ho et al., ; Rombach et al., ; Ho et al., ; Gu et al., ; Tang et al., ; Gu et al., ; Peebles and Xie, ; Crowson et al., ).
For example, performing simple tasks outside visual generation like classification with diffusion architectures is already not straightforward (Li et al., ).
In this work, we propose to model canonical codecs (JPEG and AVC/H.264) with conventional language model architectures for visual generation.
Horton et al. () and Wu et al. () are independent work that also process file bytes data, but they both focus on visual understanding (instead of generation) and use specialized modules to handle the byte sequences (whereas we use a general Llama-2 model).
Perez et al. () concurrently discover that JPEG formats can be used with language models in file anomaly handling and generation (on low-resolution images).
As a universal codec, JPEG is a novel form of data encoding for efficient image understanding (Park and Johnson, ).
Kang et al. () explore an image generation model that performs generation and JPEG compression in one system with GANs.
JPEG artifacts can also be corrected by learning a restoration model (Kawar et al., ), which is potentially helpful to the generations from our Jpeg-LM for improving image quality.
Compressive codecs are also a rising topic in language. Jiang et al. () use canonical compressors as feature extractors for texts. Lester et al. () train language models to generate compressed texts.

## 7 Conclusion

In this work, we propose Jpeg-LM and Avc-LM that generate images and videos using mainstream LLM architectures (autoregressive transformers) with canonical codec representations (JPEG for images, AVC/H.264 for videos).
Our approach greatly mitigates the length infeasibility of pixel-based sequence modeling while enabling simple, flexible, and end-to-end training compared to sophisticated vector quantization methods. Our image generation evaluation shows Jpeg-LM achieves better results than the baselines, with an especial advantage in generating long-tail visual elements.
Our work contributes to a unifying paradigm of language generation and visual generation, facilitating future research to port successful LLM techniques (e.g., alignment, efficiency, etc.) to all modalities.
Though our focus in this work does not involve visual understanding tasks or analyses of context efficiency, future work may explore these aspects based on our paradigm.

One notable significance of this work is to show that vanilla autoregressive modeling with canonical codecs is *indeed possible* in visual generation.
This is an approach almost void of prior work, likely because there are many potential, assumed challenges with the method.
For example, both JPEG and AVC operate at bits level due to the entropy coding.(^12^1212One could potentially train models over sequences of bits but it would be substantially more expensive.)
The bytes in the files do not have consistent meanings and would depend on their context and the implicit Huffman tables. For generality, our models also do not use any vision-specific modules like convolutions or 2D positional embeddings, potentially making the task more challenging.
However, we observe that conventional, vanilla language modeling surprisingly conquers these challenges without special designs as training goes (e.g., Jpeg-LM generates realistic images barely with any corrupted JPEG patches).
Based on the findings of this work, future work may continue to investigate the scaling aspect of this family of models (similar to the LLM literature) or better architectures for canonical codecs without loss of generality for other modalities.
An extended discussion can be found in §A.

#### Acknowledgements

The authors would like to thank
Yuejun Han, Igor Tsvetkov, Omer Levy, Chunting Zhou, Lili Yu, Luke Zettlemoyer, Jingwei Ma, Zeyu Liu, Lijun Yu, Jianing Yang, Alisa Liu, Jiacheng Liu, Weijia Shi, Sachin Kumar, Orevaoghene Ahia, and members of UW Tsvetshop and Koh-lab for helpful discussions. Insights from Wortsman et al. () are helpful for debugging instabilities during the training of our models.

## Appendix A Continued Discussion

Our work focuses on the challenging task of visual generation (e.g., outputting images) rather than visual understanding (e.g., inputting images, outputting classes or texts). In the field of visual understanding, the encoding of images has less restricted forms. For example, Bavishi et al. () and El-Nouby et al. () linearly project image patches as inputs to the transformers, Liu et al. () pass CLIP embeddings (Radford et al., ) to language models, etc. However, these image encoding formulations are not applicable to image generation.
Though not a focus in this work, future work may extend our Jpeg-LM and Avc-LM that share the same underlying architectures with regular language models to image and video understanding scenarios.

Compared to raw pixel modeling that would represent a 256x256 image with 65K or 196K tokens (depending on color modes), using canonical codecs like JPEG substantially reduces the sequence length to 5K on average. In terms of sequence length, the VQ transformers are usually more aggressive, representing each image with 1K tokens.
It is notable that this an ideal hyperparameter discovered in prior work that helps model global structures—increasing the number of tokens in VQ (thus reducing the downsampling patch size) may lead to degenerated results rather than helping the model learn with more capacity (Esser et al., ).
Our work proposes to model canonical codecs as a proof of concept, and future work may compare with more VQ setups or further improve the context efficiency of Jpeg-LM.

#### Limitations

Machine learning models that generate images, especially the models using natural language as convenient controls or even deepfakes that are maliciously trained to swap faces, lead to risks of generating unsafe and harmful content (Nguyen et al., ; Qu et al., ).
Though we mitigate such risks in our model by not including texts for conditioning and not processing multiple images/videos for any types of synthesis, the use cases of the model still require extensive care.
The purpose of this work is purely scientific—to explore a fundamental algorithm for general visual generation. Our approach helps lower the barriers of porting LLM techniques to visual generation, and we plan on adopting advances in LLMs (e.g., alignment and watermarking) to further enhance safety in future work (Ganguli et al., ; Kirchenbauer et al., ).
In this work, we pretrain a 7B model. Even with our moderate-scale data, we estimate a full training of Jpeg-LM to take a month on 32 Nvidia A100 GPUs. As our model shares the same architecture as regular LLMs, we plan on exploring techniques in LLM efficiency to reduce compute footprint in future work.

Figure: Figure 6: Unconditional generation by Jpeg-LM.
Refer to caption: extracted/5803872/resources/uncond_examples_jpeglm/1.jpg

Figure: Figure 7: Unconditional generation by VQ transformer.
Refer to caption: extracted/5803872/resources/uncond_examples_vq/1.jpg

## Appendix B More Qualitative Examples from Jpeg-LM

Figure 6 and Figure 7 show further examples of Jpeg-LM and VQ transformer on unconditional generation.

## Appendix C More Qualitative Examples from Avc-LM

More generations from Avc-LM can be found in
Figure 8, Figure 9, Figure 10, Figure 11, and Figure 12.
Similar to Figure 5, we observe realistic object movements (e.g., flag, clouds, clock, cars on the street, and camera movement towards a building).

Refer to caption: extracted/5803872/resources/sample_video_e/00001.png

Refer to caption: extracted/5803872/resources/sample_video_d/00001.png

Refer to caption: extracted/5803872/resources/sample_video_c/00001.png

Refer to caption: extracted/5803872/resources/sample_video_b/00001.png

Refer to caption: extracted/5803872/resources/sample_video_a/00001.png

## Appendix D Detailed Configurations for the Canonical Codecs

Our JPEG encoding uses the pillow package. We specifically encode each image with: image.save(format=’JPEG’, quality=25, subsampling="4:2:0", streamtype=2, restart_marker_blocks=1). More details about these arguments can be found at [https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#jpeg-saving](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#jpeg-saving).
Our AVC/H.264 encoding uses the ffmpeg package. Specifically, the configurations/commands we used are: ffmpeg -vf "fps=3,scale=256:144:force_original_aspect_ratio=decrease, pad=256:144:(ow-iw)/2:(oh-ih)/2" -t 5 -c:v libx264 -pix_fmt yuv420p -profile:v baseline -qp 37 -bf 0 -an -sn -x264opts "slice-max-mbs=1" -trellis 0 -me_method dia -threads 1 -subq 0 -psy 0 -mixed-refs 0 -fast-pskip 0 -partitions none -refs 3 -bsf:v h264_mp4toannexb. More details about these flags can be found at [https://ffmpeg.org/ffmpeg.html](https://ffmpeg.org/ffmpeg.html).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Why wasn’t a comparison made with Llamagen and VAR?

2. Why wasn’t Llama 3 used?

3. The metrics seem unreasonable—why wasn’t there conditional image generation included?
soundness: 3
strengths: 1. This article is well-organized and easy to understand.

2. I trust the authors' experiments; they look very interesting and demonstrate that JPEG compression, along with other compression algorithms, can be applied to image compression, decompression, and image/video generation.

3. The article shows substantial innovation. I believe it’s a good paper.
summary: The article uses compression algorithms like JPEG to encode image data into discrete tokens, then leverages current large language models to learn from these encoded tokens. The advantage of this approach is that it avoids artifacts and distortions often introduced by some existing image tokenizers during reconstruction. I find this innovation compelling, yet I have significant concerns that make me cautious about these issues.

Firstly, JPEG has a limited compression ratio under visually lossless conditions. Meanwhile, DNN-based tokenizers are continuously improving, with better reconstruction quality and increasing compression ratios. Additionally, I didn’t find any convincing comparative experiments in the article, such as comparisons with works like Llamagen or VAR. I’ve given it a score of 6, but if you can address my concerns, I will revise my score.
weaknesses: 1. This article lacks many necessary experiments, such as comparisons with VAR and Llamagen.

2. I have doubts about the scalability of this approach, as JPEG has a limited compression ratio. Even with a 4-10x compression ratio (where higher compression ratios often lead to low-quality generated content), it may still be insufficient to support long-duration video generation.

3. Generating high-quality images may require more time.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. How does JPEG-LM handle artifacts introduced by JPEG compression, and would incorporating a post-processing step improve image quality?

2. Could this codec-based approach extend effectively to other file formats, such as PNG for lossless compression or HEVC for higher-definition video?

3. Have you considered integrating JPEG-LM with diffusion techniques to leverage the best of both methods for higher fidelity in complex textures?
soundness: 4
strengths: 1. Efficiency and Simplicity: JPEG-LM and AVC-LM bypass the need for complex tokenization and preprocessing by using JPEG and AVC codecs, which simplifies training compared to VQ-based methods.

2. Performance on Long-Tail Elements: The approach excels in generating visually complex, less frequent elements (like small text or detailed faces) where VQ models often degrade in quality.

3. Versatility in Modeling: Using a standard LLM architecture without vision-specific adjustments, JPEG-LM extends the utility of language models to image and video generation effectively, aligning with LLMs for text.

4. Robustness in Out-of-Distribution Scenarios: The codec-based approach is less sensitive to distributional shifts, potentially due to its reliance on stable, non-neural compression methods.
summary: The paper introduces JPEG-LM and AVC-LM, two autoregressive language models that generate images and videos using standard codecs (JPEG for images and AVC/H.264 for videos) instead of specialized, vision-specific tokenization or pixel-based sequences. By directly modeling compressed byte sequences, these models avoid complex pre-processing or training processes typical of vector quantization (VQ) models. JPEG-LM, trained on JPEG image bytes, consistently outperforms VQ-based models in generating realistic visuals and demonstrates particular robustness in long-tail visual elements, such as small facial features and text characters.
weaknesses: 1. Limited Improvements Over Diffusion Models: While JPEG-LM shows promise, its performance does not match the leading results from diffusion-based models, especially in complex or higher-resolution tasks.

2. Dependency on JPEG Compression Constraints: Although effective, relying on JPEG and AVC formats may limit the granularity of generated details, especially in nuanced textures or high-frequency patterns.

3. Scalability for Video Generation: While AVC-LM is a compelling proof of concept, its application is limited to low frame rates and resolutions, indicating potential constraints in real-world, high-resolution video synthesis.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Model architecture
  - Does "no 2D positional embedding" indicate that the proposed model simply adopts the standard rotary positional embedding from the default setting of LLaMA2?

- Experiments
  - Is there any possibility that the proposed model generate collapsed binary data (which is impossible to decode)? If yes, how did the author treat such data in the experiments?
  - Were the test images used for prompts and FID computation also encoded by JPEG with quality=25? The results of VQ and ImageGPT shown in Fig. 2 seem to contain some ringing artifacts and block noise, which should not occur without JPEG encoding. Could you clarify the reason for these artifacts?
soundness: 3
strengths: - The main research question is quite simple but really interesting: can LLM-like models directly generate compressed binary data in an efficient and effective manner? The experimental results shown in this paper provide a positive answer to this question, which would be worth reporting in the community.

- The design of the proposed model is also simple. It follows LLaMA-2 model architecture, and its vocabulary is basically a simple byte one slightly extended by byte-pair encoding to handle a few frequently appearing sequences in metadata and headers. This simplicity enhances reproducibility of the method and is suitable as a baseline for future studies.

- The experimental results show that the proposed model outperforms VQ Transformer using learning-based discrete representations obtained by VQ-VAE, which follows a popular design in recent studies. They would let us rethink how to effectively compress data for generative models.
summary: In this paper, the authors propose a new auto-regressive model that directly generates compressed binary data in JPEG or AVC formats. As such canonical codec provides compressed discrete representation of data, it allows the model to simply follow the standard architecture of language models without any vision-specific modification. The experimental results show that the proposed model outperforms the baseline method that adopts VQ-VAE for tokenization, especially when less common contents appear in the image.
weaknesses: - The JPEG images for training were compressed with a very low quality factor of 25 (compared with 70-90 in standard usage), which causes many artifacts, such as ringing and block noise, in the generated images as shown in Section 5 and appendix. I conjecture that this choice aims to substantially reduce the number of tokens processed by the proposed method. It would be beneficial if the authors could provide some justification on this setting.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: * I really like the idea of this paper, but concerned about the experiments. If the above concerns can be addressed, I am willing to raise my score.
* I am not familiar with JEPG representation. Can it keep the spatial shape of an image? The paper claims not to use 2d RoPE, then how can it generate images/videos with different shapes or lengths?
soundness: 2
strengths: Overall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. I like the novel idea of using canonical codecs to transform continuous images and videos into discrete tokens without the need for training VQ-VAEs. This opens up a promising direction of unified multimodal modeling using a joint transformer decoder.
summary: This paper introduces JPEG-LM, a novel autoregressive model for image and video generation using canonical codecs as representations. The authors aim to simplify the task of visual generation by eliminating the need for vision-specific modifications or complex discretization processes such as vector quantization (VQ). Their key contribution is the use of JPEG and AVC codecs to transform continuous visual data into discrete tokens, significantly reducing sequence length while maintaining generation quality. The models, based on a Llama-2 architecture, achieve state-of-the-art performance in generating realistic images and videos, especially outperforming VQ-based approaches in handling long-tail visual elements. Experiments demonstrate that JPEG-LM can generate high-quality visual content with fewer complications and show robustness in generating fine details like human faces or text.
weaknesses: * The method section of this paper lacks more technical details or background knowledge of the proposed method, especially for readers without much knowledge of canonical codecs. (Although I know the core idea of this paper is quite straightforward.)
* The biggest problem is that this paper only shows the inpainting results of image generation and preliminary results of video generation. It lacks standard results on benchmarks like ImageNet and COCO (Text-to-Image) to fully demonstrate the potential of the proposed framework. The only inpainting results are restricted to the "auto-regressive" manner, which deviates from conventional inpainting with arbitrary shapes and makes the comparison with diffusion models unfair. The paper claims to remove all visual-specific design to achieve unified framework with language models, but conducts no multimode experiments such as text-to-image generation, even though it uses data from LAION-5B.
* There is a similar work [1], which uses the same idea to model molecules.

[1] Flam-Shepherd, Daniel, and Alán Aspuru-Guzik. "Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files." arXiv preprint arXiv:2305.05708 (2023).

Meta-review: This paper proposes to apply an autoregressive Transformer to directly generate the bytes in the JPEG representation of images and videos. The main motivation is to replace the role of trained VQVAEs which simplifies the modeling pipeline. Reviewers find the idea interesting, and most ratings are positive except for reviewer VnZm. After reviewing the paper and discussions, the AC believe's that the concerns raised by VnZm is not addressed. The fact that the paper mainly relies on image completion tasks as an evaluation makes it difficult to support the claim that the JPEG codes are indeed superior than VQVAEs. Although Table 3 includes uncondtional sampling evaluations, their absolute numbers seem very high for both VQVAE baseline and the proposed method. I would strongly suggest the authors to consider standard image generation benchmarks and evaluation protocols, such as class conditional/unconditional sampling with standard FID evaluation protocols. The other issue is that I find the use of 'LLM' misleading -- one would have assumed that it utilizes a pretrained language model, whereas in reality it's just an autoregressive Transformer trained from scratch. I would suggest clarifying this by either changing the title or explaining this explicitly early on. Based on these considerations, I don't feel like this paper is ready to be accepted, but I hope the authors can keep improving it.

Decision: Reject