ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correction

## Abstract

Abstract Autoregressive models (ARMs) and diffusion models (DMs) represent two leading paradigms in generative modeling, each excelling in distinct areas: ARMs in global context modeling and long-sequence generation, and DMs in generating high-quality local contexts, especially for continuous data such as images and short videos. However, ARMs often suffer from exponential error accumulation over long sequences, leading to physically implausible results, while DMs are limited by their local context generation capabilities. In this work, we introduce Autoregressive Coherent multimodal generation with Diffusion Correction (ACDC), a zero-shot approach that combines the strengths of both ARMs and DMs at the inference stage without the need for additional fine-tuning. ACDC leverages ARMs for global context generation and memory-conditioned DMs for local correction, ensuring high-quality outputs by correcting artifacts in generated multimodal tokens. In particular, we propose a memory module based on large language models (LLMs) that dynamically adjusts the conditioning texts for the DMs, preserving crucial global context information. Our experiments on multimodal tasks, including coherent multi-frame story generation and autoregressive video generation, demonstrate that ACDC effectively mitigates the accumulation of errors and significantly enhances the quality of generated outputs, achieving superior performance while remaining agnostic to specific ARM and DM architectures. Project page: https://acdc2025.github.io/

## 1 Introduction

Autoregressive models (ARM) (Brown, ) and diffusion models (DM) (Ho et al., ) are the two prominent paradigms in modern generative modeling, each with its strengths, weaknesses, and areas where they excel. ARMs excel at modeling discrete token sequences, leading to their domination of the language domain (Dubey et al., ; Abdin et al., ), and more recently, multimodal modeling (Kondratyuk et al., ; Lu et al., ; Xie et al., ; Jin et al., ), as vector quantized (VQ) autoencoder architectures become advanced (Esser et al., ; Yu et al., ). ARMs are useful for 1) modeling the global context, as it attends causally to all previous tokens, and 2) long-sequence generation, as they are not limited to fixed-length tokens (Liu et al., ; ). These advantages render ARMs excellent for high-level long-term planners (Du et al., ; Huang et al., ) and world models (Ha & Schmidhuber, ; Hao et al., ; Ge et al., ). However, the “plans” generated by ARMs cannot be corrected, often leading to physically implausible results (Kambhampati et al., ) and low quality images/videos. To mitigate this issue, one often needs an external verifier to locally correct for the mistakes (Kambhampati et al., ) and rely on vision-specific decoder fine-tuning (Ge et al., ; ). The inability to correct for the errors exacerbate as the length of the generated sequences becomes longer, inherently due to the exponential error accumulating nature of these models. The accumulated errors are especially noticeable in autoregressive video generation, with videos often diverging when generating over a few tens of frames (Liu et al., ).

On the other hand, DMs excel at modeling continuous vectors, especially capable of generating high-quality images (Rombach et al., ; Saharia et al., ) and short video clips (Brooks et al., ).
The generative process in diffusion models, which transitions from noise to the data space, inherently creates a Gaussian scale-space representation of the data distribution. This approach allows the model to iteratively refine the generated image in a coarse-to-fine manner, capturing both global structure and fine-grained details effectively.
In other words, DMs are excellent local context generators. Another intriguing feature of diffusion models (DMs) is their high degree of controllability. A pre-trained DM can be repurposed for a variety of tasks, such as solving inverse problems (Chung et al., ), image editing (Meng et al., ; Kim et al., ), adversarial purification (Nie et al., ), and more, by reconstructing the inference path and applying appropriate guidance. Since DMs function as score estimators, they implicitly serve as priors for the data distribution (Song & Ermon, ), ensuring that generated or modified samples remain close to the high-density regions of the distribution. However, DMs typically generate fixed-length vectors, which limits their suitability for tasks involving long sequence generation or planning.

Combining the advantages of ARMs and DMs, researchers have investigated ways to combine the two to by using ARM as the global model and DM as the local vision generator. This led to the construction of multimodal generators (Ge et al., ; Li et al., ), robot planners (Du et al., ), and world models (Xiang et al., ). However, all these models require re-purposing of the base models, and require expensive modality-specific fine-tuning or instruction tuning. Moreover, to the best of our knowledge, using DMs as a way to avoid exponential error accumulation of ARMs has not been investigated in the literature.

To fill in this gap, in this work, leveraging the intriguing strengths of both pre-trained ARMs and DMs, we present Autoregressive coherent multimodal generation with Diffusion Correction (ACDC), a flexible zero-shot combination method of existing models applicable at inference stage, regardless of the specific design of the ARMs and DMs.
In ACDC, the ARM is the main workhorse, responsible for generating multimodal tokens that respect the global context. For every chunk of vision tokens generated (e.g. a single image, 16 video frames), the tokens are decoded into the continuous space with the visual detokenizer, and locally corrected with a memory-conditioned DM through the SDEdit (Meng et al., ) process. Specifically, to distill the global context of the generated sequence into the DM without any fine-tuning, we additionally propose a memory module implemented with a large language model (LLM), which causally corrects the conditioning texts of the DM.

Through extensive experiments with various multimodal ARMs including Large World Models (Liu et al., ), Unified-IO-2 (Lu et al., ), and Show-o (Xie et al., ) on two distinct tasks: coherent multi-frame story generation and autoregressive video generation, we show that ACDC significantly improves the quality of the generated results and can correct for the exponential accumulation in errors (See Fig. 1 for representative results), achieving all this while being training-free and agnostic to the model class.

## 2 Related works

### 2.1 Unifying autoregressive models with diffusion models

#### Diffusion models as vision decoder for autoregressive models

Instead of the standard way of using VQGAN decoder as the visual detokenizer, works such as SEED-X (Ge et al., ) and Mini-Gemini (Li et al., ) propose to leverage Stable Diffusion XL (SDXL) (Podell et al., ) as the visual decoder by fine-tuning a pre-trained SDXL to take in the visual tokens as the condition through attention. While the image generation quality of these methods exceed the ones that only use ARMs since the pre-trained diffusion model guarantees high-quality outputs, consistency is hard to control. Pandora (Xiang et al., ) is a work that shares a goal that is similar to ours, which uses two Q-formers (Li et al., ) as adapters to connect ARMs and DMs, repurposing them as a world model.

#### Proposals in unified architecture

For methods that use DMs as vision decoders, the ARM component is still responsible for generating the multimodal tokens, and hence the methods can be considered as late-fusion methods. Recently, several methods have been proposed to combine the two model classes in the earlier pre-training stage in an early-fusion manner. Transfusion (Zhou et al., ) trains the LMM with the next token prediction loss for language and with the diffusion (i.e. denoising) loss for image patches. The sampling modes are switched between next token prediction and continuous diffusion when the special token is sampled. Show-o (Xie et al., ) is similar to Transfusion in that it uses both next token prediction and diffusion, but differs in that the discrete tokens are generated through discrete diffusion (Austin et al., ; Yu et al., ). Diffusion forcing (Chen et al., ) generalizes ARMs so that it can take perform noisy autoregression, as well as iteratively refine the current token by denoising it.

While promising, all of the methods discussed require altering the architecture of the DM(^1^11For instance, conditioning the diffusion model on the previous frame (Ge et al., ) or video (Xiang et al., )) or the ARM, as well as extensive fine-tuning and aligning. Our goal, on the other hand, is to leverage the pre-trained model components as is, without any fine-tuning. For example, later we show that Show-o also benefits from the use of the proposed ACDC.

### 2.2 Diffusion models for image/video sequence generation

#### Diffusion models as world models

Recently, several works aimed to repurpose diffusion models as world models (Ha & Schmidhuber, ), which estimate the next state (i.e. image or video) given the current state and action. GenHowto (Damen et al., ) fine-tuned a DM to predict the next image frame on the previous frame and a text instruction on a targetted dataset (Souček et al., ). SEED-story (Yang et al., ) achieves a similar goal on another targetted story dataset but on a longer sequence. Genie (Bruce et al., ) and GameNGen (Valevski et al., ) take latent actions (e.g. control signal) as input to output the game frame. Going further, world models that can generate chunks of video frames (Xing et al., ; Xiang et al., ) were also proposed. While promising as world models, these models do not possess the understanding and planning abilities of LMM, and again, deviate from a general-purpose DM.

#### Autoregressive generation with diffusion models

Standard video diffusion models (Ho et al., ; Blattmann et al., ; Brooks et al., ) generate data of fixed length. One can repurpose video DMs for arbitrary-length video generation by using diagonal denoising proposed in FIFO-diffusion (Kim et al., ), or train a video DM from scratch using a similar strategy (Ruhe et al., ). Unfortunately, even the models that are repurposed for long sequence generation can only attend to a local context, hampering the consistency of the generated videos.

## 3 Autoregressive coherent modeling with diffusion correction

Figure: Figure 2: Illustration of the proposed ACDC method. For each chunk of image token ${\textnormal{v}}^{i}$ decoded into a a frame ${\boldsymbol{x}}_{0}^{i}$, 1) LLM memory module causally summarizes the previous prompts ${\boldsymbol{y}}^{1:i-1}$ to provide the text input ${\boldsymbol{y}}^{i}$, 2) Diffusion correction is applied by perturbing ${\boldsymbol{x}}_{0}^{i}$ with forward diffusion, then running reverse diffusion conditioned on ${\boldsymbol{y}}^{i}$. {B,E}o{T,V} indicates beginning, end of text, vision tokens, respectively.
Refer to caption: x2.png

In this section, we explore the use of Multimodal ARMs as world models used in the task of coherent story and video generation, where the key lies in the fidelity of the model’s predictions, as well as controlling the exponential accumulation of errors.
Specifically, we assume a Partially Observable Markov Decision Process (POMDP) setup (Sutton, ) where the actions are described by texts and the states are rendered as image frames.

#### Autoregressive multimodal modeling for world models

For predictive planning (Chang et al., ; Zhao et al., ; Damen et al., ), given the initial state and the goal, both a policy and a world model is required. The policy is responsible for predicting the next action from the current state. The world model, on the other hand, should be capable of modeling the next state from the current state and action pair.

Figure: Figure 3: Before (left) and after (right) correction through the proposed LLM memory module. Key global context is distilled into the local prompts.
Refer to caption: x3.png

Let us define ${\mathbf{u}}^{i}$ as a chunk of $i$-th text tokens with sequence length $m$, and ${\mathbf{v}}^{i}$ as a chunk of image tokens with sequence length $n$. In a multimodal multi-turn generation setup, at the $i$-th turn of generation of the image keyframe, as illustrated in Fig. 2, the base ARM will sample from

$$ $\displaystyle{\mathbf{v}}^{i}\sim p_{\phi}({\mathbf{v}}^{i}|{\mathbf{v}}^{1:i- 1},{\mathbf{u}}^{1:i}),\quad\mbox{where}\quad{\mathbf{v}}^{1:i-1}:=[{\mathbf{v }}^{1},{\mathbf{v}}^{2},\cdots,{\mathbf{v}}^{i-1}],{\mathbf{u}}^{1:i}:=[{ \mathbf{u}}^{1},{\mathbf{u}}^{2},\cdots,{\mathbf{u}}^{i}].$ (1) $$

where the ARM is parameterized with $\phi$. For the image tokens, the chunk can be decoded autoregressively in a patch-wise raster scan order, or through a discrete diffusion process (Yu et al., ).
Here, ${\mathbf{u}}^{i},{\mathbf{v}}^{i}$ are discrete tokens. To visualize, the discrete image tokens are decoded to the continuous pixel space with a visual detokenizer (e.g. decoder of a VQ-GAN (Esser et al., )) ${\boldsymbol{x}}_{0}^{i}={\mathcal{D}}({\mathbf{v}}^{i})$. To make a distinction between the continuous decoded variable and the discrete token variable, note that we use bold italics for the decoded variables. Here, the subscript in ${\boldsymbol{x}}_{t}^{i}$ denotes the diffusion time(^2^22We review diffusion models in Appendix A.), and the superscript denotes the physical time (index).

#### Diffusion correction

The decoded $i$-th image ${\boldsymbol{x}}^{i}_{0}$ often contains visual artifacts. The problem is less significant in the initial frame but exacerbates for later predicted frames. To correct the visual artifacts, our goal is to (1) bring ${\boldsymbol{x}}^{i}_{0}$ closer to the clean data manifold, yet (2) proximal to the starting point. This can be effectively implemented with a text-conditional SDEdit (Meng et al., ), where one can use any text-to-image (T2I) DM, such as stable diffusion (Rombach et al., ) that operates in the latent space, or DeepFloyd IF (Team, ) that operates in the pixel space. As illustrated in Fig. 2, the process reads

$$ $\displaystyle\tilde{\boldsymbol{x}}_{0}^{i}\sim p_{\theta}({\boldsymbol{x}}_{0 }|{\boldsymbol{x}}_{t^{\prime}}^{i},{\boldsymbol{y}}),\quad{\boldsymbol{x}}_{t }^{i}\sim p({\boldsymbol{x}}_{t}^{i}|{\boldsymbol{x}}_{0}^{i}),$ (2) $$

where $\theta$ is the DM parameters, $t^{\prime}$ is a hyperparameter that chooses the degree of perturbation, and ${\boldsymbol{y}}$ is the text condition. Typically, we choose a moderate scale of $t^{\prime}\in[0.4,0.5]$ such that the correction does not alter the content of the image, but only make local corrections.
After obtaining $\tilde{\boldsymbol{x}}_{0}^{i}$, if this is not the terminal state, we can re-encode it back to the image tokens, i.e. $\tilde{\mathbf{v}}^{i}={\mathcal{E}}(\tilde{\boldsymbol{x}}_{0}^{i})$, where ${\mathcal{E}}$ is the image encoder of the ARM, and the sampling proceeds with the swapped tokens. In Appendix B, we theoretically show that our diffusion correction algorithm meets both criteria (1)—Theorem 1 and (2)—Theorem 2,3.

#### Large Language Models as memory module

Care must be taken when choosing the text condition as input for our diffusion correction module $p_{\theta}$ due to two factors. First, the DM is a local image correction module that should only be conditioned for the current frame, and not the spurious information from the past states. However, it should also take into account the key information from the previous states, such as the example given in Fig. 3. Looking at the second frame only, it is impossible to deduce that “His” in the local context is referring to “Benny, the golden retriever”. While one could aim to design and train a separate memory module specified for this task, we take a simpler approach, where we take a large language model (LLM) to causally generate new prompts conditioned on the previous prompts keeping the key information. Concretely, with an LLM parametrized with $\varphi$ and the system/user prompt ${\mathbf{d}}$ for the memory module, we change Eq. (2) to

$$ $\displaystyle\tilde{\boldsymbol{x}}_{0}^{i}\sim p_{\theta}({\boldsymbol{x}}_{0 }|{\boldsymbol{x}}_{t^{\prime}}^{i},{\boldsymbol{y}}^{i}),\quad{\boldsymbol{x} }_{t^{\prime}}^{i}\sim p({\boldsymbol{x}}_{t^{\prime}}^{i}|{\boldsymbol{x}}_{0 }^{i}),\quad{\boldsymbol{y}}^{i}\sim p_{\varphi}({\boldsymbol{y}}^{i}|{ \boldsymbol{y}}^{1:i-1},{\mathbf{d}}),$ (3) $$

where ${\boldsymbol{y}}^{i}$ denotes the $i-$th input prompt to the DM that is refined by the LLM memory module by summarizing ${\boldsymbol{y}}^{1:i-1}$, i.e. the previous prompts used.

Our algorithm is illustrated in Fig. 2. The importance of memory module can be seen theoretically in Theorem 3. We provide the implementation details of the memory module in Appendix D.

#### Incorporating physical constraints

When generating images with multimodal ARMs, the degradation in the quality of images is not the only issue. Specifically, we observe that the ARMs are less sensitive to the physical feasibility of the rendered image. For instance, we often observe cases where rabbits have three ears, or where there is more than one moon or sun in the background, as can be seen in Fig. 5. Fortunately, incorporating physical constraints to diffusion models has been widely explored in the context of inverse problems (Chung et al., ; Yuan et al., ) and conditional diffusion models (Zhang et al., ).
Thanks to the versatility of diffusion inference, one can additionally incorporate user feedback (e.g. inpainting masks, depth constraints) to explicitly correct for the errors, or automatically project the image to the manifold of feasible data (Gillman et al., ).

#### Extension to autoregressive video generation

We consider an extension to autoregressive video generation from a single text prompt using LWM (Liu et al., ). Note that as we use a multimodal ARM as our base model, extension to multi-prompt input between the frames is trivial. In an autoregressive text-to-video generation, sampling is done similar to Eq. (1)

$$ $\displaystyle{\boldsymbol{x}}_{0}^{i}={\mathcal{D}}({\mathbf{v}}^{i}),\quad{ \mathbf{v}}^{i}\sim p_{\phi}({\mathbf{v}}^{i}|{\mathbf{v}}^{1:i-1},{\mathbf{u} }),$ (4) $$

where ${\mathbf{u}}$ is the prompt, and the video frames are decoded independently with a VQ-GAN decoder. When generating videos using AR models, visual artifacts often appear in the generated frames, and inconsistencies in motion between these frames may occur due to the independent decoding.
To resolve these issues, we propose to use our diffusion correction scheme analogous to Eq. (2) with text-to-video (T2V) DMs, such as AnimateDiff (Guo et al., ) or VideoCrafter2 (Chen et al., ). Out of $N$ total frames to be generated with the ARM, let $L<N$ be the number of frames that T2V models take as input. After sequentially sampling the video frames to get ${\bm{X}}_{0}^{1:L}=[{\boldsymbol{x}}_{0}^{1},\cdots,{\boldsymbol{x}}_{0}^{L}]$ we apply SDEdit similar to Eq. (2)

$$ $\displaystyle\tilde{\bm{X}}_{0}^{1:L}\sim p_{\theta}({\bm{X}}_{0}|{\bm{X}}_{t^ {\prime}}^{1:L},{\boldsymbol{y}}),\quad{\bm{X}}_{t^{\prime}}^{1:L}\sim p({\bm{ X}}_{t^{\prime}}^{1:L}|{\bm{X}}_{0}^{1:L}).$ (5) $$

Then, the corrected $\tilde{\bm{X}}_{0}^{1:L}$ can be re-encoded back to resume sampling with an ARM

$$ $\displaystyle{\mathbf{v}}^{j}\sim p_{\phi}({\mathbf{v}}^{j}|[\tilde{\mathbf{v} }^{1:L},{\mathbf{v}}^{L+1:j-1}],{\mathbf{u}})\quad\tilde{\mathbf{v}}^{1:L}={ \mathcal{E}}(\tilde{\bm{X}}_{0}^{1:L}).$ (6) $$

Unlike standard image diffusion models (Rombach et al., ), which are limited in their ability to model temporal dynamics, video diffusion models provide a more robust mechanism for ensuring temporal coherence throughout the video. When utilizing diffusion correction, selecting the time step $t^{\prime}\in[0.4,0.6]$ allows for control over which aspects of the video are prioritized during refinement. Since global temporal motion is largely established during earlier time steps, higher values such as $t^{\prime}=0.6$ are effective for refining temporal motion, while lower values like $t^{\prime}=0.4$ are more suitable for addressing local visual artifacts. This approach ensures that both temporal consistency and visual quality are maintained throughout the video generation process.

Figure: Figure 4: Qualitative comparison of the story generation task.“Golden retriever in sunlit meadow with butterflies (frame 1) … jumped (frame 3) … chasing butterfly (frame 4)”
Refer to caption: x4.png

**Table 1: Quantitative evaluation of the Story generation task. Best, second best.**
| Method | Frame consistency ($\uparrow$) | CLIP-sim ($\uparrow$) | ImageReward ($\uparrow$) | FID ($\downarrow$) |
| --- | --- | --- | --- | --- |
| Stable Diffusion v1.5 (Rombach et al., ) | $0.6822$ | $28.91$ | $-0.8455$ | $\bm{32.11}$ |
| Show-o (Xie et al., ) | $0.8211$ | $28.76$ | $-0.5752$ | $60.50$ |
| Show-o (Xie et al., ) + ACDC (ours) | $\bm{0.9062}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 10.4\%}$ | $\underline{30.82}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor<br>}{rgb}{0.91,0.33,0.5}\scriptstyle\blacktriangle 7.16\%}$ | $-\bm{0.0003}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb<br>}{0.91,0.33,0.5}\scriptstyle\blacktriangle 0.574}$ | $\underline{56.36}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor<br>}{rgb}{0.91,0.33,0.5}\scriptstyle\blacktriangle 7.34\%}$ |
| UIO-2XXL (Lu et al., ) | $0.8833$ | $29.46$ | $-0.8354$ | $67.12$ |
| UIO-2XXL (Lu et al., ) + ACDC (ours) | $\underline{0.8962}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.91,0.33,0.5}\scriptstyle\blacktriangle 1.46\%}$ | $\bm{31.09}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.91,0.33,0.5}\scriptstyle\blacktriangle 5.53\%}$ | $-\underline{0.2624}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{<br>pgfstrokecolor}{rgb}{0.91,0.33,0.5}\scriptstyle\blacktriangle 0.574}$ | $57.80{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.91,0.33,0.5}\scriptstyle\blacktriangle 16.1\%}$ |

## 4 Experiments

In this section, we test our hypothesis on two distinct experiments: story generation and autoregressive video generation, by applying ACDC to the sampling steps. For all diffusion correction, we employ DDIM sampling (Song et al., ) with classifier free guidance (CFG) (Ho & Salimans, ) scale set to 7.5. For the hyperparameters of the base ARM sampling, we use the default settings advised in the original work (Liu et al., ; Xie et al., ; Lu et al., ), unless specified otherwise.

### 4.1 Story generation

#### Benchmark dataset generation

The goal of this section is to study whether ACDC can truly reduce error accumulation and improve the coherent story generation quality. As the base ARM, we choose UIO-2XXL (Lu et al., ) and Show-o (Xie et al., ), as they are two representative open-sourced models. As the base DM, we choose Stable Diffusion v1.5 (Rombach et al., ). We note that other choices such as DeepFloyd IF for DM can also be utilized, without any modifications to the algorithm.
To test our hypothesis, it is crucial to select the set of examples that are approximately in-distributed to both the base ARM and DM. We found that existing open-sourced benchmarks such as ChangeIT (Souček et al., ) does not meet this criterion, and hence, we decided to generate 1k stories with an LLM. Following the practices in Alpaca (Taori et al., ) and self-instruct (Wang et al., ), we manually construct 10 examples of the stories and randomly select 3 examples when querying gpt-4o-mini to generate another synthetic data.
Our stories consist of 6 consecutive prompts, with the first prompt describing the name and the species of the main character and the background it is surrounded in. The following prompts mainly consist of the change in the character’s motion and background but is constructed such that a single prompt does not fully describe the context. Further details and examples are provided in Appendix C. The full prompts used to generate the images and videos in this work are gathered in Appendix F.

#### Evaluation on the benchmark

We compare the results of applying ACDC to two baseline ARMs, using 4 different metrics: frame consistency (Esser et al., ), CLIP similarity, ImageReward (Xu et al., ), and FID. Frame consistency is computed as the sum of cosine similarities in the consecutive image frames to capture the consistency of the story. The other three metrics measure the quality of the generated images, as well as the faithfulness to the given text prompt frame-wise.
To compute FID against in-distribution images, we generate pseudo-ground truth images using SDXL-lightning (Lin et al., ) with 6 NFE.

In Tab. 1, we observe that ACDC improves the baseline ARMs by significant margins in both frame consistency and image quality metrics. Although the FID score lags behind SD v1.5, this can be partly attributed to the fact that we consider SDXL-generated images as ground truth. In Fig. 1 and Fig. 4, we see that ACDC is capable of generating coherent content, with the same main character with changing background and motion. In contrast, Show-o generates images with artifacts starting from the first frame, which quickly exacerbates as the error accumulates in multiple frame generation. Throughout the generation results, we observe that the generation results from Show-o degrade significantly after the fourth frame. SD generates high quality images frames, but is incapable of understanding the global context and generating a consistent story.

Figure: Figure 5: Incorporating user constraints to correct for physical errors in the generated image frames.
Refer to caption: x5.png

#### Correcting physical errors through user constraints

During the generation process of the base ARM, we often observe physically incorrect images, as illustrated in Fig. 5. Using standard ARM inference, we observe that for such cases, the error accumulates faster, where the generated frames with the same physical error that propagated from the previous frame. In contrast, we observe that after the correction with SD inpainting, the physical error no longer persists, and one can prevent the propagation of physical errors. More examples can be seen in Fig. 9.

**Table 2: Ablation study in story generation task with 100 test stories.**
| Components | Metrics |  |  |  |
| --- | --- | --- | --- | --- |
| ACDC # | LLM memory | F.Con. ($\uparrow$) | CLIP $\uparrow$ | FID $\downarrow$ |
| 0 | ✗ | $0.822$ | $28.60$ | $112.5$ |
| 2 | ✓ | $\bm{0.888}$ | $28.58$ | $109.6$ |
| 4 | ✓ | $0.843$ | $28.79$ | $107.6$ |
| 6 | ✓ | $\underline{0.853}$ | $\bm{30.85}$ | $\bm{101.4}$ |
| 6 | ✗ | $0.835$ | $\underline{29.43}$ | $\underline{101.6}$ |

#### Ablation studies

On top of SDEdit correction, we have another crucial component of ACDC: LLM memory. We investigate the effect of the memory component in Fig. 4 and Tab. 2. From Fig. 4, we observe that ACDC without memory is as effective as ACDC in the earlier frames, but the performance degrades in the later frames, where without the global context, it is hard to correct for the errors from the ARM, as can be seen in the prompt example in Fig. 3. In Tab. 2, we again see the significant gap of ACDC with and without the memory component.

Further, we study the effect of ACDC by controlling the number of frames that the correction is applied. ACDC # 2 refers to the case where we apply the correction to the first two frames and the following frames are generated only through ARM. Two facts are notable: 1) ACDC enhances the result even when we do not correct for all the frames, showing that one can reduce the exponential error accumulation, 2) Correcting for all the frames performs the best, although the frame consistency is slightly compromised compared to ACDC # 2.

### 4.2 Autoregressive video generation

**Table 3: Quantitative evaluation of the autoregressive video generation task.**
| Method | Subject consistency ($\uparrow$) | Background Consistency ($\uparrow$) | Motion Smoothness ($\uparrow$) | Dynamic Degree ($\uparrow$) | Aesthetic Quality ($\uparrow$) | Imaging Quality ($\uparrow$) |
| --- | --- | --- | --- | --- | --- | --- |
| LWM (Liu et al., ) | $0.7369$ | $0.8695$ | $0.9479$ | $0.7018$ | $0.4105$ | $0.5127$ |
| LWM (Liu et al., ) + ACDC (ours) | $\bm{0.7622}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 3.43\%}$ | $\bm{0.8821}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 1.45\%}$ | $\bm{0.9501}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 0.23\%}$ | $\bm{0.7099}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 1.15\%}$ | $\bm{0.4406}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 7.33\%}$ | $\bm{0.5164}{\color[rgb]{0.91,0.33,0.5}\definecolor[named]{pgfstrokecolor}{rgb}<br>{0.91,0.33,0.5}\scriptstyle\blacktriangle 0.72\%}$ |

Figure: Figure 6: Qualitative comparison of the autoregressive video generation task. White dotted box refers to the first 16 frames, which are corrected by ACDC. The remaining frames are left uncorrected. Magenta arrows indicate regions where visual artifacts or inconsistencies with preceding frames occur in the original video.
Refer to caption: x6.png

#### Experimental settings

We conducted experiments using the Large World Model (LWM) (Liu et al., )—a large multimodal model capable of generating videos frame-by-frame—to assess the impact of ACDC on reducing error propagation in video generation task. We observed that LWM frequently exhibits visual artifacts and content inconsistency when generating sequences longer than 16 frames. To investigate the effectiveness of ACDC, we compared two generation pipelines: (1) directly generating all $N=32$ frames using LWM and (2) generating the first $L=16$ frames with LWM, applying ACDC using AnimateDiff (Guo et al., ) to correct the frames, and subsequently generating the remaining 16 frames. For both pipelines, we introduced diversity in the initial frame by setting the CFG scale to $5.0$ and utilizing top-$k$ sampling with $k=8192$. The subsequent frames were generated with a reduced CFG scale of $1.0$ and top-$k$ sampling with $k=1000$ to ensure consistency. All experiments were performed on 4 NVIDIA H100 GPUs. These experiments allowed us to evaluate the role of ACDC in mitigating error propagation and preserving video quality across extended sequences.

#### Dataset and Evaluation on the benchmark

To evaluate the video generation and correction capabilities across a diverse range of subjects and motions, we utilized 800 prompts from VBench (Huang et al., ), spanning multiple categories. In addition, we assessed both the spatial and temporal quality of the generated videos using 2 spatial and 4 temporal metrics, leveraging the tools provided by VBench for accurate measurement. For spatial quality, we employed the LAION aesthetic predictor (LAION-AI, ) to assess frame-wise aesthetic quality, and the Multi-Scale Image Quality Transformer (MUSIQ) (Ke et al., ) to quantify imaging quality, specifically evaluating noise and blur levels in each frame. For temporal quality assessment, we measured subject consistency and background consistency between frames using DINO (Caron et al., ) feature similarity and CLIP similarity, respectively. To mitigate the potential risk of improving consistency at the expense of dynamic motion, we employed RAFT (Teed & Deng, ) to measure the dynamic degree and verify whether the observed consistency enhancements were achieved without hindering the dynamic quality of motion. Lastly, to assess the smoothness of generated motion, we compared the generated motion against the predicted motion from a video frame interpolation model (Li et al., ), allowing us to quantify the smoothness of motion transitions between frames.

In Tab. 3, we observed that ACDC improves both subject and background consistency compared to the baseline LWM, with a notably large margin. Importantly, the dynamic degree did not decrease, indicating that the improvement in consistency was achieved without the video becoming overly static. Furthermore, both imaging quality and aesthetic quality showed improvements, with aesthetic quality exhibiting a substantial increase, likely driven by the influence of the video diffusion model. As demonstrated in Fig. 1 and Fig. 6, LWM exhibits minor visual artifacts and content inconsistencies in the initial 16 frames (first 3 figures in each row), which, if left uncorrected, progressively worsen in subsequent frames, resulting in more pronounced artifacts and content drift. However, by applying ACDC to correct the first 16 frames, these issues are effectively mitigated, enabling the consistency in later frames to remain aligned with the corrected early frames, thus preventing the propagation of errors throughout the sequence.

## 5 Conclusion

We presented Autoregressive Coherent multimodal generation with Diffusion Correction (ACDC), a flexible, zero-shot approach that leverages the complementary strengths of autoregressive models (ARMs) and diffusion models (DMs) for multimodal generation tasks. By employing DMs as local correctors through SDEdit and incorporating a memory module with large language models (LLMs), ACDC addresses the exponential error accumulation inherent in ARMs, resulting in consistent and high-quality outputs across a range of tasks. Our experiments on coherent story generation and autoregressive video generation validate that ACDC effectively integrates pre-trained ARMs and DMs without requiring additional fine-tuning or architectural modifications, thus establishing a robust and scalable framework for multimodal generation. Future work could explore further enhancements in the memory module and investigate the application of ACDC to other complex multimodal tasks, pushing the boundaries of generative modeling.

## Appendix A Overview of diffusion models

In diffusion models (Ho et al., ; Song et al., ), we first define the forward noising process with a stochastic differential equation (SDE).
The variance preserving (VP) forward diffusion trajectory is given by the following SDE (Song et al., ; Ho et al., )

$$ $\displaystyle d{\boldsymbol{x}}=-\frac{1}{2}\beta(t){\boldsymbol{x}}dt+\sqrt{ \beta(t)}d{\mathbf{w}},$ (7) $$

with $\beta(t)=\beta_{min}+(\beta_{max}-\beta_{min})t$ ,${\boldsymbol{x}}(t)\in{\mathbb{R}}^{d}$, and $t\in[0,1]$. Here, ${\mathbf{w}}$ is a standard $d-$dimensional Wiener process. An intriguing property of the forward diffusion is that the forward perturbation kernel is given in a closed form

$$ $\displaystyle p({\boldsymbol{x}}(t)|{\boldsymbol{x}}(0))={\mathcal{N}}({ \boldsymbol{x}}(t);\sqrt{\bar{\alpha}(t)}{\boldsymbol{x}}(0),(1-\bar{\alpha}(t )){\bm{I}}),$ (8) $$

where $\bar{\alpha}(t)={\rm exp}\left(\int_{0}^{t}\frac{1}{2}\beta(u)\,du\right)$. Here, it is important to note that the parameters $\beta_{min}$ and $\beta_{max}$ are chosen such that $\bar{\alpha}(0)\approx 1$ and $\bar{\alpha}(1)\approx 0$ such that as the forward diffusion approaches the terminal state $t\rightarrow 1$, $p_{1}({\boldsymbol{x}})$ approximates standard normal distribution. The reverse SDE of Eq. (7) reads

$$ $\displaystyle d{\boldsymbol{x}}=\left[-\frac{1}{2}\beta(t){\boldsymbol{x}}- \beta(t)\nabla_{{\boldsymbol{x}}}\log p_{t}({\boldsymbol{x}})\right]dt+\sqrt{ \beta(t)}{\mathbf{w}}.$ (9) $$

Sampling data from noise ${\boldsymbol{x}}\sim p_{data}({\boldsymbol{x}})$ amounts to solving Eq. (9) from $t=1$ to $t=0$, which is governed by the score function $\nabla_{{\boldsymbol{x}}}\log p_{t}({\boldsymbol{x}})$. The score function is trained with the denoising score matching (DSM) objective

$$ $\displaystyle\min_{\boldsymbol{\theta}}{\mathbb{E}}_{t,{\boldsymbol{x}}(t)\sim p ({\boldsymbol{x}}(t)|{\boldsymbol{x}}(0)),{\boldsymbol{x}}(0)\sim p({ \boldsymbol{x}})}\left[\lambda(t)\|s_{\boldsymbol{\theta}}({\boldsymbol{x}}(t) ,t)-\nabla_{{\boldsymbol{x}}_{t}}\log p({\boldsymbol{x}}(t)|{\boldsymbol{x}}(0 ))\|_{2}^{2}\right],$ (10) $$

where $\lambda(t)$ is a weighting function. The score function can be shown to be equivalent to denoising autoencoders (Karras et al., ) through Tweedie’s formula (Efron, ), and also equivalent to epsilon matching (Ho et al., ). When plugging in the trained score function to Eq. (9), we have the empirical reverse SDE

$$ $\displaystyle d{\boldsymbol{x}}=\left[-\frac{1}{2}\beta(t){\boldsymbol{x}}- \beta(t)s_{\boldsymbol{\theta}}({\boldsymbol{x}}(t),t)\right]dt+\sqrt{\beta(t) }{\mathbf{w}}.$ (11) $$

Running Eq. (11) will sample from ${\boldsymbol{x}}\sim p_{\theta}({\boldsymbol{x}})$, with $p_{\theta}({\boldsymbol{x}})=p_{data}({\boldsymbol{x}})$ if $\nabla_{{\boldsymbol{x}}}\log p_{t}({\boldsymbol{x}})\equiv s_{\boldsymbol{
\theta}}({\boldsymbol{x}}(t),t)$ for all $t\in[0,1]$ (Song et al., ).
It is notable that the time-marginal distributions of $Eq.~{}(\ref{eq:reverse_sde})$ can also be retrieved by running the so-called probability-flow ODE (PF-ODE)

$$ $\displaystyle d{\boldsymbol{x}}=-\frac{1}{2}\beta(t)\left[{\boldsymbol{x}}+ \nabla_{{\boldsymbol{x}}}\log p_{t}({\boldsymbol{x}})\right]dt.$ (12) $$

We can again use a plug-in estimate for the score term, achieving the empirical PF-ODE

$$ $\displaystyle d{\boldsymbol{x}}=-\frac{1}{2}\beta(t)\left[{\boldsymbol{x}}+s_{ {\bm{\theta}}}({\boldsymbol{x}},t)\right]dt.$ (13) $$

Modern foundational diffusion models are typically text-conditioned (Rombach et al., ; Team, ), so that the user can control the content to be generated. Denoting ${\boldsymbol{y}}$ as the conditioning text, the task of text-to-image or text-to-video generation can be considered sampling from the posterior $p({\boldsymbol{x}}|{\boldsymbol{y}})$. In order to do this, we train a conditional score function with paired (image, text) data

$$ $\displaystyle\min_{\boldsymbol{\theta}}{\mathbb{E}}_{t,{\boldsymbol{x}}(t)\sim p ({\boldsymbol{x}}(t)|{\boldsymbol{x}}(0)),({\boldsymbol{x}}(0),{\boldsymbol{y} })\sim p({\boldsymbol{x}},{\boldsymbol{y}})}\left[\lambda(t)\|s_{\boldsymbol{ \theta}}({\boldsymbol{x}}(t),{\boldsymbol{y}},t)-\nabla_{{\boldsymbol{x}}_{t}} \log p({\boldsymbol{x}}(t)|{\boldsymbol{x}}(0))\|_{2}^{2}\right],$ (14) $$

where we denote the empirical joint distribution as $p({\boldsymbol{x}},{\boldsymbol{y}})$. In the optimal case, we hae $\nabla_{{\boldsymbol{x}}}\log p_{t}({\boldsymbol{x}}|{\boldsymbol{y}})\equiv s
_{\boldsymbol{\theta}}({\boldsymbol{x}}(t),{\boldsymbol{y}},t)$. Sampling from the posterior can be achieved by plugging in the conditional score function to Eq. (9) or Eq. (12).

## Appendix B Theoretical Justification

Our diffusion correction algorithm starts from an initial point ${\boldsymbol{x}}$, which was obtained by detokenizing the sampled output of an ARM. We first perturb it with forward diffusion

$$ $\displaystyle{\boldsymbol{x}}(t^{\prime})=\sqrt{\bar{\alpha}(t^{\prime})}{ \boldsymbol{x}}+\sqrt{1-\bar{\alpha}(t^{\prime})}{\boldsymbol{\epsilon}},\quad {\boldsymbol{\epsilon}}\sim{\mathcal{N}}(0,{\bm{I}}),$ (15) $$

then recovers it through the conditional reverse PF-ODE from time $t^{\prime}$ to 0 0

$$ $\displaystyle\tilde{\boldsymbol{x}}(0)=\int_{t^{\prime}}^{0}-\frac{1}{2}\beta( t)[{\boldsymbol{x}}+s_{\bm{\theta}}({\boldsymbol{x}},{\boldsymbol{y}},t)]dt,$ (16) $$

where ${\boldsymbol{y}}$ is the text condition corrected through our LLM memory module. Let $p_{t}({\boldsymbol{x}})$ the distribution of forward-diffused clean data obtained through Eq. (7), and $q_{t}({\boldsymbol{x}})$ be the distribution of the forward-diffused corrupted data from the ARM sampling, again obtained through
Eq. (7).

###### Theorem 1 ( nie2022diffusion ) .

The KL divergence between $p_{t}$ and $q_{t}$ monotonically decreases through forward diffusion, i.e.

$$ $\displaystyle\frac{\partial D_{KL}(p_{t}||q_{t})}{\partial t}\leq 0.$ (17) $$

Adding Gaussian noise to a random variable as in Eq. (7) leads to convolution with a Gaussian blur kernel for distributions, and hence $p_{t},q_{t}$ are blurred distributions. Theorem 1 states by blurring the distribution, the two become close together, and there must exist $t^{\prime}\in[0,1]$ with $D_{KL}(p_{t^{\prime}}||q_{t^{\prime}})<\varepsilon$. Recovering with Eq. (16) will guarantee that we pull the sample closer toward the desired distribution.

However, care must be taken since we should also keep the corrected image close to the starting point. To see if this hold, we start from a simplified case without considering the text condition (i.e. null conditioning). Let us define $\hat{\boldsymbol{x}}(0)$ as follows

$$ $\displaystyle\hat{\boldsymbol{x}}(0)=\int_{t^{\prime}}^{0}-\frac{1}{2}\beta(t) [{\boldsymbol{x}}+s_{\bm{\theta}}({\boldsymbol{x}},t)]dt.$ (18) $$

We have the following result

###### Theorem 2 .

Assume that the score function is globally bounded by $\|s_{{\bm{\theta}}}({\boldsymbol{x}},t)\|\leq C$. Then,

$$ $\displaystyle{\mathbb{E}}[\|\hat{\boldsymbol{x}}(0)-{\boldsymbol{x}}\|]\leq(1- \sqrt{\bar{\alpha}(t^{\prime})})\left(\eta+\frac{C}{\bar{\alpha}(t^{\prime})} \right)+\frac{\sqrt{1-\bar{\alpha}(t^{\prime})}\sqrt{d}}{\sqrt{\bar{\alpha}(t^ {\prime})}}$ (19) $$

where $\eta={\mathbb{E}}[\|{\boldsymbol{x}}\|]$.

###### Proof.

By reparametrization, the forward diffusion to $t^{\prime}$ in Eq. (7) reads

$$ $\displaystyle{\boldsymbol{x}}(t^{\prime})=\sqrt{\bar{\alpha}(t^{\prime})}{ \boldsymbol{x}}+\sqrt{1-\bar{\alpha}(t^{\prime})}{\boldsymbol{\epsilon}},\quad {\boldsymbol{\epsilon}}\sim{\mathcal{N}}(0,{\bm{I}}).$ (20) $$

Let $\mu(t)$ be the integrating factor

$$ $\displaystyle\mu(t)={\rm exp}\left(\int_{t^{\prime}}^{t}\frac{1}{2}\beta(u)\, du,\right)$ (21) $$

where by definition, $\mu(t^{\prime})=1$ and $\mu(0)=1/\sqrt{\bar{\alpha}(t^{\prime})}$.
Multiplying $\mu(t)$ on both sides of Eq. (18),

$$ $\displaystyle\frac{d}{dt}(\mu(t){\boldsymbol{x}}(t))=-\frac{1}{2}\beta(t)\mu(t )s_{\boldsymbol{\theta}}({\boldsymbol{x}},t).$ (22) $$

Integrating both sides from $t^{\prime}$ to 0,

$$ $\displaystyle\mu(0)\hat{\boldsymbol{x}}(0)-\mu(t^{\prime}){\boldsymbol{x}}(t^{ \prime})=-\frac{1}{2}\int_{t^{\prime}}^{0}\beta(t)\mu(t)s_{\boldsymbol{\theta} }({\boldsymbol{x}},t)dt.$ (23) $$

Rearranging,

$$ $\displaystyle\hat{\boldsymbol{x}}(0)$ $\displaystyle=\frac{1}{\mu(0)}{\boldsymbol{x}}(t^{\prime})-\frac{1}{2\mu(0)} \int_{t^{\prime}}^{0}\beta(u)\mu(u)s_{\boldsymbol{\theta}}({\boldsymbol{x}}(u) ,u)\,du$ (24) $\displaystyle=\sqrt{\bar{\alpha}(t^{\prime})}{\boldsymbol{x}}+\frac{\sqrt{1- \bar{\alpha}(t^{\prime})}}{\sqrt{\bar{\alpha}(t^{\prime})}}{\boldsymbol{ \epsilon}}-\frac{1}{2\sqrt{\bar{\alpha}(t^{\prime})}}\int_{t^{\prime}}^{0} \beta(u)\mu(u)s_{\boldsymbol{\theta}}({\boldsymbol{x}}(u),u)\,du$ (25) $$

Hence,

$$ $\displaystyle\hat{\boldsymbol{x}}(0)-{\boldsymbol{x}}=(\sqrt{\bar{\alpha}(t^{ \prime})}-1){\boldsymbol{x}}+\frac{\sqrt{1-\bar{\alpha}(t^{\prime})}}{\sqrt{ \bar{\alpha}(t^{\prime})}}{\boldsymbol{\epsilon}}-\frac{1}{2\sqrt{\bar{\alpha} (t^{\prime})}}\int_{t^{\prime}}^{0}\beta(u)\mu(u)s_{\boldsymbol{\theta}}({ \boldsymbol{x}}(u),u)\,du.$ (26) $$

Taking expectation on both sides and leveraging triangle inequality, we have

$$ $\displaystyle{\mathbb{E}}[\|\hat{\boldsymbol{x}}(0)-{\boldsymbol{x}}\|]$ $\displaystyle\leq\underbrace{{\mathbb{E}}[\|(1-\sqrt{\bar{\alpha}(t^{\prime})} ){\boldsymbol{x}}\|]}_{T_{1}}$ (27) $\displaystyle+\underbrace{{\mathbb{E}}\left[\left\|\frac{\sqrt{1-\bar{\alpha}( t^{\prime})}}{\sqrt{\bar{\alpha}(t^{\prime})}}{\boldsymbol{\epsilon}}\right\| \right]}_{T_{2}}$ (28) $\displaystyle+\underbrace{{\mathbb{E}}\left[\frac{1}{2\sqrt{\bar{\alpha}(t^{ \prime})}}\left\|\int_{t^{\prime}}^{0}\beta(u)\mu(u)s_{\boldsymbol{\theta}}({ \boldsymbol{x}}(u),u)\,du\right\|\right]}_{T_{3}}$ (29) $$

$T_{1}=(1-\sqrt{\bar{\alpha}(t^{\prime})}){\mathbb{E}}[\|{\boldsymbol{x}}\|]$. From ${\mathbb{E}}[\|{\boldsymbol{\epsilon}}\|]\leq\sqrt{d}$, $T_{2}\leq\frac{\sqrt{1-\bar{\alpha}(t^{\prime})}\sqrt{d}}{\sqrt{\bar{\alpha}(t
^{\prime})}}$. Finally, from the boundedness assumption of $s_{\boldsymbol{\theta}}({\boldsymbol{x}},t)$,

$$ $\displaystyle\left\|\int_{t^{\prime}}^{0}\beta(u)\mu(u)s_{\boldsymbol{\theta}} ({\boldsymbol{x}}(u),u)\,du\right\|$ $\displaystyle\leq C\int_{t^{\prime}}^{0}\beta(u)\mu(u)\,du$ (30) $\displaystyle=2C\int_{t^{\prime}}^{0}\frac{d\mu(u)}{du}du$ (31) $\displaystyle=2C(\mu(0)-\mu(t^{\prime}))$ (32) $\displaystyle=2C\left(\frac{1-\sqrt{\bar{\alpha}(t^{\prime})}}{\sqrt{\bar{ \alpha}(t^{\prime})}}\right),$ (33) $$

so $T_{3}=C\frac{1-\sqrt{\bar{\alpha}(t^{\prime})}}{\bar{\alpha}(t^{\prime})}$. Summing up,

$$ $\displaystyle{\mathbb{E}}[\|\hat{\boldsymbol{x}}(0)-{\boldsymbol{x}}\|]\leq(1- \sqrt{\bar{\alpha}(t^{\prime})})\left(\eta+\frac{C}{\bar{\alpha}(t^{\prime})} \right)+\frac{\sqrt{1-\bar{\alpha}(t^{\prime})}\sqrt{d}}{\sqrt{\bar{\alpha}(t^ {\prime})}}$ (34) $$

∎

Recall that $\beta(t)$ is designed such that $\bar{\alpha}(0)\approx 1$ and $\bar{\alpha}(1)\approx 0$. Hence, Theorem 2 implies that the deviation from ${\boldsymbol{x}}$ after SDEdit, if $t^{\prime}$ is not too large, is bounded. Next, we further consider the case of $\tilde{\boldsymbol{x}}(0)$ in Eq. (16)

###### Theorem 3 .

Assume that $\|s_{\boldsymbol{\theta}}({\boldsymbol{x}}_{t},{\boldsymbol{y}},t)-s_{
\boldsymbol{\theta}}({\boldsymbol{x}}_{t},\tilde{\boldsymbol{y}},t)\|\leq Kd({
\boldsymbol{y}},\tilde{\boldsymbol{y}})$, where $d(\cdot,\cdot)$ measures the feature distance between the text conditions ${\boldsymbol{y}}$ and $\tilde{\boldsymbol{y}}$. Let $\tilde{\boldsymbol{x}}(0)$ be the SDEdit reconstruction using time $t^{\prime}$ and the conditional score $s_{\boldsymbol{\theta}}({\boldsymbol{x}}_{t},{\boldsymbol{y}},t)$ for solving the PF-ODE. Then,

$$ $\displaystyle{\mathbb{E}}[\|\tilde{\boldsymbol{x}}(0)-{\boldsymbol{x}}\|]\leq( 1-\sqrt{\bar{\alpha}(t^{\prime})})\left(\eta+\frac{C}{\bar{\alpha}(t^{\prime}) }+K\bar{\alpha}(t^{\prime})d({\boldsymbol{y}},\tilde{\boldsymbol{y}})\right)+ \frac{\sqrt{1-\bar{\alpha}(t^{\prime})}\sqrt{d}}{\sqrt{\bar{\alpha}(t^{\prime} )}}.$ (35) $$

###### Proof.

$$ $\displaystyle\|\tilde{\boldsymbol{x}}(0)-{\boldsymbol{x}}\|$ $\displaystyle=\|\tilde{\boldsymbol{x}}(0)-\hat{\boldsymbol{x}}(0)+\hat{ \boldsymbol{x}}(0)-{\boldsymbol{x}}\|$ (36) $\displaystyle\leq\|\tilde{\boldsymbol{x}}(0)-\hat{\boldsymbol{x}}(0)\|+\|\hat{ \boldsymbol{x}}(0)-{\boldsymbol{x}}\|$ (37) $$

by triangle inequality. The second term in the RHS is given by Theorem 2. Regarding the first term of the RHS,

$$ $\displaystyle\|\tilde{\boldsymbol{x}}(0)-\hat{\boldsymbol{x}}(0)\|$ $\displaystyle\leq\frac{Kd({\boldsymbol{y}},\tilde{\boldsymbol{y}})}{2\mu(0)} \int_{t^{\prime}}^{0}\beta(u)\mu(u)\,du$ (38) $\displaystyle=K\frac{1-\sqrt{\bar{\alpha}(t^{\prime})}}{\bar{\alpha}(t^{\prime })}d({\boldsymbol{y}},\tilde{\boldsymbol{y}}).$ (39) $$

Thus,

$$ $\displaystyle\|\tilde{\boldsymbol{x}}(0)-{\boldsymbol{x}}\|$ $\displaystyle=\leq(1-\sqrt{\bar{\alpha}(t^{\prime})})\left(\eta+\frac{C}{\bar{ \alpha}(t^{\prime})}+K\bar{\alpha}(t^{\prime})d({\boldsymbol{y}},\tilde{ \boldsymbol{y}})\right)+\frac{\sqrt{1-\bar{\alpha}(t^{\prime})}\sqrt{d}}{\sqrt {\bar{\alpha}(t^{\prime})}}.$ (40) $$

∎

Theorem 3 shows that for the conditional case, in order to control the deviation from ${\boldsymbol{x}}$, an additional factor of the correctness of the text prompt should be considered. Only when ${\boldsymbol{y}}=\tilde{\boldsymbol{y}}$, the bound matches the unconditional case. Otherwise, there is larger deviation. Note that due to the discrepancy between how the text conditioning is used in ARMs and DMs, we do not know the underlying “ground truth” ${\boldsymbol{y}}$. Hence, Theroem 3 implies that it is important that we use the memory module to align the text condition used in the SDEdit process using the memory module.

Table: Table 4: System, user prompt used to generate new stories.

Table: Table 5: Example stories that is fed to GPT-4o to generate new stories.

Table: Table 6: Generated stories with GPT-4o.

Figure: Algorithm 1 GPT-4o Story Generation with Random Examples.

###### Theorem 1 ( nie2022diffusion ) .

###### Theorem 2 .

###### Proof.

###### Theorem 3 .

###### Proof.

## Appendix C Data generation

As stated in Sec. 4, we first manually construct 10 example stories to be fed to the LLM. The examples are shown in Tab. 1. As shown in Tab. 4 and Alg. 1, we aim to sample diverse stories with the LLM by using 1.0 for the temperature, and prompting the model to be creative in its choice. Some examples of the generated stories are presented in Tab. 6.

## Appendix D LLM memory module

Table: Table 7: LLM memory module system and user prompt

We explored two design choices for the LLM memory module. First, one could query the LLM for every image ${\boldsymbol{x}}_{0}^{(i)}$ to be generated. Alternatively, one could query the LLM one time with all the story prompts, and ask it to causally refine it. As we did not observe much difference in the quality, we decided to choose the latter. The prompt design used for the LLM memory module is provided in Tab. 7. For the LLM, we used gpt-4o-mini for our experiments but found open-sourced models such as gemma-9b-it (Team et al., ) also produced similar results.

## Appendix E Further experimental results

Figure: Figure 7: Further results of ACDC applied to Show-o.
Refer to caption: x7.png

Figure: Figure 8: Further results of ACDC applied to Unified-IO-2.
Refer to caption: x8.png

Figure: Figure 9: Incorporating user constraints to correct for physical errors in the generated image frames.
Refer to caption: x9.png

Figure: Figure 10: Further results of ACDC applied to LWM.
Refer to caption: x10.png

## Appendix F Prompts for results

Fig 1

- •
Row 1-2.
–
Prompt 1: "Luna, the fox, stood at the edge of a moonlit forest vibrant with glowing fireflies."
–
Prompt 2: "With a flick of her bushy tail, she slipped stealthily through the underbrush, her paws barely making a sound."
–
Prompt 3: "Luna darted between the trees, her amber eyes glinting in the dim light as she chased a fluttering moth."
–
Prompt 4: "Suddenly, she stopped in her tracks, her ears perked up at the rustle of leaves nearby."
–
Prompt 5: "With a playful bounce, Luna leaped over a fallen log, determined not to lose her target."
- •
Row 3-4:
–
Prompt 1: "Benny the beagle sat in a vibrant garden filled with blooming flowers and buzzing bees under a clear blue sky."
–
Prompt 2: "His nose twitched as he picked up the sweet scent of roses and lavender."
–
Prompt 3: "Benny jumped up and trotted toward the colorful flowerbeds, tail wagging with excitement."
–
Prompt 4: "He playfully pawed at the petals, sending a few fluttering to the ground."
–
Prompt 5: "The beagle barked joyfully at a bumblebee that dared to dance close to his snout."
- •
Row 5-6: "A male interviewer listening to a person talking."

Fig 4

- •
Prompt 1: "Baxter, the adventurous golden retriever, stood at the edge of a sunlit meadow dotted with wildflowers and fluttering butterflies."
- •
Prompt 2: "With a wagging tail, he bounded forward, the soft grass giving way beneath his paws."
- •
Prompt 3: "Baxter jumped playfully, leaping over a small brook that shimmered in the sunlight."
- •
Prompt 4: "He chased after a bright yellow butterfly, his golden coat glistening as he ran."
- •
Prompt 5: "Baxter skidded to a halt, nose to the ground, investigating the sweet scent of blooming clover."

Fig 5

- •
Prompt 1: "The curious rabbit hopped through a vibrant meadow, wildflowers swaying gently in the summer breeze."
- •
Prompt 2: "Ruby, the rabbit, bounded playfully between patches of colorful blooms."
- •
Prompt 3: "With a swift turn, Ruby, the rabbit, darted towards a butterfly fluttering nearby, her ears perked with excitement."
- •
Prompt 4: "Ruby, the rabbit, leaped gracefully, her paws barely brushing the ground as she chased the delicate creature."
- •
Prompt 5: "Suddenly, Ruby, the rabbit, skidded to a halt, captivated by a glimmering stream sparkling in the sunlight."

Fig 6

- •
Row 1-2: "A boat sailing on a stormy ocean."
- •
Row 3-4: "A wooden house overseeing the lake"
- •
Row 5-6: "An elegant ceramic plant pot and hanging plant on indoor"

Fig 10

- •
Row 1-2: "Aerial shot of interior of the warehouse."
- •
Row 3-4: "Aerial video of Stuttgart tv tower in Germany."
- •
Row 5-6: "A chef holding and checking kitchen utensils."
- •
Row 7-8: "A scenic view of the golden hour at Norway."
- •
Row 9-10: "Close up view of lighted candles."

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - The FID column in Table 1 should have downside red triangle.

- It's better to show the generation result corresponding to the text prompt example in Figure. 3 etc.
soundness: 3
strengths: - The visualizations present well surpassing generative quality of the proposed framework.
summary: This paper proposes to leverage pre-trained text-to-image diffusion models to guide the inference process of autoregressive models for story and video generation tasks. Additional large language model is also used to refine the text prompt with clearer details. The proposed framework is able to fit in any base model to enhances them, and reaches leading performance on multi-frame story and video generation benchmarks.
weaknesses: - How is LLM applied to (short) video generation as few frames won't have too much content changed?

- How about the size, memory and computation cost for the proposed model when it's being additionally added to the baseline? Comparisons with similar total parameters or inference time should be also conducted as this would be crucial for end users.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: - Are there any qualitative examples to show the transition consistency of adjacent video clips (i.e., a video version of Figure 4) ? 
- In Eq. (2), It seems that the $x_t^i$ in the second term should be $x_{t'}^i$.
soundness: 2
strengths: - This paper for the first time proposes a flexible zero-shot combination of multi-modal autoregressive models (ARMs) and diffusion models (DMs). With the help of DMs' correction conditioned on LLM-refined text prompts, the model effectively improves the generation quality of multi-modal ARMs.
- This paper is clearly written and easy to understand.
summary: This paper introduces a tuning-free method that combines multi-modal autoregressive models (ARMs) and diffusion models. The proposed method, termed ACDC, uses diffusion models to correct the visual outputs of multi-modal ARMs, conditioned on the LLM refined text-prompts. As demonstrated by the experiments, ACDC achieves more coherent generation results in terms of both multi-frame story generation and autoregressive video generation.
weaknesses: - The proposed method, i.e., ACDC, is technically trivial.
  - This method can be considered as a simple application of using diffusion techniques (e.g., SDEdit, DDIM) to correct the frame output of multi-modal autoregressive models (ARMs). I don't see any challenges (and how does this paper address these challenges) to do so. 
  - For the so-called "Incorporating user constraints", it is very travail. Basically,  anyone can apply SD-inpainting to correct the generated frames of any ARMs (or world models).

- Technically, ACDC can not assure the temporal consistency of adjacent video clips, since they are corrected (re-generated) from perturbed frames and conditioned on different LLM-refined text prompts. It may even break the original temporal consistency of adjacent video clips generated by the multi-modal ARMs.
  - Since the paper only provides several frames of different text prompts, e.g., figure 4, it's unclear whether these scenes transition smoothly. 
  - From my perspective, since the diffusion model's corrections are based on different perturbations, the consistency of consecutive video clips might be worse than some existing multi-text video generation methods, e.g., Gen-L-Video [1], which is also tuning-free.


- This method seems to have little correlation with multi-modal ARMs. I.e., if one wants to design methods that use LLMs and diffusion models to correct/refine the per-chunk video generation result, why it must be applied on multi-modal ARMs? There are many multi-text video generation models, e.g., [2],[3], which are capable of generating consecutive video clips described by sequential text prompts. This paper fails to compare the proposed ACDC to these methods (or apply ACDC on top of these methods), and apparently the generation results (as shown in its [project page](https://acdc2025.github.io/)) of ACDC are inferior to these methods in terms of both visual quality and temporal consistency.

[1] Wang, Fu-Yun, et al. "Gen-l-video: Multi-text to long video generation via temporal co-denoising." *arXiv preprint arXiv:2305.18264* (2023). ( https://github.com/G-U-N/Gen-L-Video )

[2] Villegas, Ruben, et al. "Phenaki: Variable length video generation from open domain textual descriptions." *International Conference on Learning Representations*. 2022.

[3] Oh, Gyeongrok, et al. "MEVG: Multi-event Video Generation with Text-to-Video Models." *European Conference on Computer Vision*. Springer, Cham, 2024.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: My major concern lies in the depth of the main idea and the experiments.
soundness: 2
strengths: - This paper is well-structured and easy to follow, with clear explanations of the proposed concepts. Figure 2 effectively illustrates the main architecture.
- One of the key advantages of the proposed approach is that it requires no additional training, which makes it highly efficient and adaptable.
- The method can be applied to a wide range of tasks, including story generation and video generation.
summary: The authors try to combine ARMs and DMs into a multi-modal model. They find ARMs suffer from error accumulation, and DMs are limited by their local focus. Their ACDC addresses these issues by combining ARMs for global context and memory-conditioned DMs for local corrections, ensuring high-quality outputs. Experiments demonstrate ACDC's ability to reduce errors and improve performance across multimodal tasks, without additional fine-tuning.
weaknesses: - This paper introduces diffusion correction to address visual artifacts in multimodal ARMs generation. However, state-of-the-art models, such as Emu2 (CVPR 2024), have already largely resolved this issue, which largely weakening the contribution of this work.
- The SDEdit-like method adds extra computational overhead, which may not be justified given the current advancements.
- The paper lacks sufficient experiments and comparisons. More baselines are needed to validate the effectiveness of the proposed ACDC method. For video generation, only LWM is compared, which is insufficient. For story generation, the paper should include comparisons with more works, such as ARMs-based models (StoryGPT-V, SEED-Story) and DMs-based models (ConsiStory, StoryGen).
- The improvements in video generation results are minimal, as evident from the provided page.
- In my view, the introduction lacks depth and insights. There should be a more thorough analysis of the challenges faced by both DMs and ARMs. For instance, as the paper claims, "ARMs often suffer from exponential error accumulation over long sequences, leading to physically implausible results, while DMs are limited by their local context generation capabilities."

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see weaknesses.
soundness: 3
strengths: 1. The motivation of this paper is clearly introduced through analyzing the strengths and weaknesses of autoregressive models (ARMs) and diffusion models (DMs).
2. The correction for the exponential accumulation in errors of autoregressive models for visual generation is a significant issue. The authors propose a training-free and model-agnostic approach to address this issue, and achieve improved results of coherent multi-frame story generation and autoregressive video generation.
3. The authors propose a multimodal story generation benchmark for the evaluation of multi-frame story generation.
summary: This paper proposes Autoregressive coherent multimodal generation with Diffusion Correction (ACDC) to levarage the  intriguing strengths of both pre-trained autoregressive models and diffusion models in a training-free manner for coherent multi-frame story generation and autoregressive video generation. The authors utilize diffusion models for local correction through SDEdit and incorporate a memory module with LLMs to address  the exponential error accumulation inherent in autoregressive models.
weaknesses: 1. The core technical contribution of this method is a memory-conditioned diffusion model through SDEdit, which, in my opinion, seems to lack sufficient novelty. Of course, I would like to see the authors' response and the opinions of other reviewers to assess whether my evaluation is reasonable.
2. Since the authors utilized an additional LLM memory module and diffusion correction, they should provide information on the extra inference overhead compared to the baseline model.
3. For the quantitative evaluation of the Story generation task, only the beseline model without ACDC and Stable Diffusion are evaluated. Some models specifically designed for story generation should be taken into account for evaluation, such as MM-Interleaved[1].

[1] MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer