Story-Adapter: A Training-free Iterative Framework for Long Story Visualization

## Abstract

Abstract Story visualization, the task of generating coherent images based on a narrative, has seen significant advancements with the emergence of text-to-image models, particularly diffusion models.
However, maintaining semantic consistency, generating high-quality fine-grained interactions, and ensuring computational feasibility remain challenging, especially in long story visualization ( i.e. , up to 100 frames).
In this work, we propose a training-free and computationally efficient framework, termed Story-Adapter , to enhance the generative capability of long stories.
Specifically, we propose an iterative paradigm to refine each generated image, leveraging both the text prompt and all generated images from the previous iteration.
Central to our framework is a training-free global reference cross-attention module, which aggregates all generated images from the previous iteration to preserve semantic consistency across the entire story, while minimizing computational costs with global embeddings. This iterative process progressively optimizes image generation by repeatedly incorporating text constraints, resulting in more precise and fine-grained interactions.
Extensive experiments validate the superiority of Story-Adapter in improving both semantic consistency and generative capability for fine-grained interactions, particularly in long story scenarios.
The project page and associated code can be accessed via this https URL .

## 1 Introduction

Story visualization aims to generate a sequence of coherent images from text prompts, reflecting the narrative’s progression and enabling users, even without an artistic background, to visually present their stories (Li et al., ; Maharana & Bansal, ; Chen et al., ).
Recent advancements in text-to-image models, particularly diffusion models, have significantly improved the quality of generated visuals, producing high-quality, creative, and aesthetically pleasing images (Saharia et al., ; Rombach et al., ; Kang et al., ). These models greatly outperform earlier approaches such as generative adversarial networks (Brock, ) in terms of image quality.

However, story visualization remains challenging, particularly in maintaining semantic consistency and capturing complex interactions as the story length increases.
Two main paradigms have emerged in this domain.
The Auto-Regressive paradigm (Fig. 2A), which generates frames sequentially (Pan et al., ; Liu et al., ), often struggles with semantic consistency due to error accumulation and the inability to reference future frames, leading to inconsistencies in the overall narrative.
Although techniques like Consistent Self-Attention (CSA) (Zhou et al., ) can help mitigate these inconsistencies, their reliance on intermediate denoising features results in high memory consumption, limiting scalability for longer stories.
To address these challenges, Zhou et al. () further propose the Reference-Image paradigm, which employs fixed reference images to guide the visualization process. However, as shown in Fig. 2B, while using only the initial frames as reference images alleviate scalability issues, it fails to provide the global semantic coherence necessary for long-story visualization, ultimately resulting in the propagation of errors from the reference images to subsequent frames.
As such, both paradigms experience quality degradation when visualizing long stories. Additionally, they inherit the limitations from Stable Diffusion (SD) (Rombach et al., ), particularly in generating fine-grained interactions (as shown in Fig. 1).

To address these limitations, we present Story-Adapter, an iterative framework that adapts pre-trained SD models for long story visualization.
Unlike existing methods that generate images auto-regressively or rely on static reference images (Fig. 2 A&B), our approach prioritizes semantic consistency by incorporating all generated images from previous iterations into the current one. This process offers two key advantages.
1) It offers a comprehensive view of the entire narrative, thereby reducing error accumulation and mitigating the propagation of flaws from reference images.
2) By continuously engaging with text prompts, Story-Adapter optimizes generative quality for details based on insights from earlier iterations.
As illustrated in Fig.1, our framework enhances both semantic consistency and the quality of fine-grained interactions across iterations, resulting in more coherent and higher-quality visualizations. For example, the image depicting complex character interactions, such as “the snowman greeting the fox” demonstrates substantial improvement over iterations compared to previous methods(Liu et al., ; Zhou et al., ).

During initialization, only text prompts of the story are utilized to generate reference images. In subsequent iterations, the global embeddings of all images generated in the previous round, along with the text embeddings, collaboratively guide the image generation process.
To implement the iterative paradigm efficiently, we propose a plug-and-play Global Reference Cross-Attention (GRCA), where all global image embeddings act as keys and values. This significantly reduces computational costs, as global embeddings operate at a lower dimensionality than the intermediate denoising features used in CSA.
Additionally, to strike a balance between visual consistency and text controllability, we introduce a linear weighting strategy in the iterative paradigm to fuse both modalities.

Extensive experiments demonstrate that Story-Adapter consistently outperforms existing methods for visualizing both regular-length and long stories (up to 100 frames). Specifically, in the context of regular-length story visualization using the StorySalon benchmark dataset (Liu et al., ), Story-Adapter exceeds the baseline model, StoryGen (Liu et al., ), achieving a 9.4% improvement in average Character-Character Similarity (aCCS) (Cheng et al., ) and a 21.71 reduction in average Fréchet Inception Distance (aFID) (Cheng et al., ).
For long story visualization, Story-Adapter also demonstrates solid advancements, achieving gains of 3.4% in aCCS and 8.14 in aFID compared to StoryDiffusion (Zhou et al., ), demonstrating the superior generative quality of Story-Adapter, particularly in terms of semantic consistency and fine-grained interactions.

Figure: Figure 2: Comparison of paradigms for long story visualization: (A) Auto-Regressive (AR): generates frames sequentially referencing on previous finite frames (e.g. the previous three frames); (B) Reference-Image (RI): employs fixed reference images (e.g. the beginning four frames) as reference images; (C) Iterative Paradigm: leverages all frames from the previous iteration as reference images.
Refer to caption: extracted/5908139/fig/Fig_paradigm-simple.jpg

## 2 Related Work

### 2.1 Diffusion Models

Diffusion models (Ho et al., ; Song et al., ; Sohl-Dickstein et al., ; Nichol & Dhariwal, ) have emerged as powerful tools for data distribution modeling through iterative denoising.
Recent advancements in sampling techniques (Xiao et al., ; Song et al., ; Luo et al., ), backbone architectures (Peebles & Xie, ; Lu et al., ), and latent space denoising (Rombach et al., ; Podell et al., ) have led to their widespread adoption in various generative tasks, including video (Esser et al., ; Yang et al., ), 3D (Luo & Hu, ; Xu et al., ), audio (Ruan et al., ; Huang et al., ), and human motion generation (Zhang et al., ; Karunratanakul et al., ).
While Text-to-Image diffusion models (Saharia et al., ; Zhang et al., ; Rombach et al., ; Podell et al., ) have gained significant attention, challenges persist in generating coherent image sequences for tasks like story visualization due to the inherent randomness and fine-grained interaction generation.

### 2.2 Story Visualization

Story visualization (Chen et al., ; Li, ) has evolved from GAN-based approaches like StoryGAN (Li et al., ) to more advanced techniques.
Recent developments leverage diffusion models (Shen et al., ; Tao et al., ) and combine them with auto-regressive paradigm, as seen in AR-LDM (Pan et al., ) and StoryGen (Liu et al., ).
These methods have improved coherence in image sequences and extended to open-ended story visualization.
However, challenges remain in maintaining semantic consistency for the whole story and avoiding error accumulation, especially for longer narratives (Wang et al., ; Zhou et al., ; Liu et al., ).

### 2.3 Subject-Consistent Image Generation

The consistency of the generated subjects is critical for tasks such as story visualization and video generation.
Recent advancements in subject-consistent image generation have focused on reducing computational resources while maintaining consistency.
Early approaches like Gal et al. (); Ruiz et al. () require extensive fine-tuning, prompting more efficient methods (Ryu, ; Han et al., ; Kumari et al., ; Yuan et al., ).
Notable progress includes IP-Adapter (Ye et al., ) with its decoupled cross-attention design and technique like PhotoMaker (Li et al., ) that accelerates generation using identity images.
Recently, StoryDiffusion (Zhou et al., ) introduced Consistent Self-Attention (CSA) to boost the frame-wise subject consistency but still faces limitations in long image sequences.
In contrast, Story-Adapter maintains image semantic consistency in long image sequences by using cross-attention on global embeddings from all generated images of the previous iteration and the corresponding text features.
Along with our iteration paradigm, the whole generations are gradually improved w.r.t semantic consistency and generative quality for fine-grained interactions.

## 3 Method

Figure: Figure 3: Illustration of the proposed iterative paradigm, which consists of initialization, iterations in Story-Adapter, and implementation of Global Reference Cross-Attention.
Refer to caption: extracted/5908139/fig/Fig_2.jpg

Compared to regular-length stories, long stories contain more characters and more complex interactions, leading to higher requirements for semantic consistency and fine-grained interaction generation.
To address the above challenges,
we resort to an iterative paradigm that progressively refines all the generated images, w.r.t. semantic consistency and visual details in multiple rounds.
We instantiate the iterative paradigm by equipping a fixed Stable-Diffusion (SD) model with a cross-attention mechanism, termed Story-Adapter.
The pipeline is demonstrated in Fig. 3.

### 3.1 Initialization

To build the initialization for iteration, we only employ text prompt $T_{k}$ for the $k_{th}$ image in the story to guide the fixed ${\rm{SD}}(z,T_{k})$ in generating the initial images, where $z$ is the random noise.
All generated images from the initial step will be stored as reference images for the first iteration.
We denote $i=0$ as the initialization of Story-Adapter.
Thus, the whole initialization process can be represented as:

$$ $\begin{split}&{x_{k}^{i=0}}={\rm{SD}}(z,T_{k}),k\in[1,B],\\ &{x_{1,\cdots,B}^{i=0}}=[x_{1}^{i=0},x_{2}^{i=0},\cdots,x_{k}^{i=0},\cdots,x_{ B-1}^{i=0},x_{B}^{i=0}],\end{split}$ (1) $$

where $B$ denotes the length of the story.
Compared to subject-consistent image generation methods (Ye et al., ) that introduce reference image guidance, initialization which relies only on text prompts more faithfully visualizes the corresponding content in the story.
The following iterations benefit from the rich visual content provided by the initialization of the reference images.

### 3.2 Story-Adapter

This subsection demonstrates how each image is updated within an iteration in Story-Adapter.
Formally, for the $i_{th}$ iteration, we use all visualizations from the previous iteration $x_{1,\cdots,B}^{i-1}$ as the reference images $R$ to refine the generated images in the current round.
For the generation of the $k_{th}$ image of a long story, we define a function ${\rm{SD_{GRCA}}}(z,T_{k},x_{1,\cdots,B}^{i-1})$ to represent the whole denoising process with our Global Reference Cross-Attention (Sec. 3.3).

Thus the $i_{th}$ iteration can be expressed as:

$$ $\begin{split}&{x_{k}^{i}}={\rm{SD_{GRCA}}}(z,T_{k},x_{1,\cdots,B}^{i-1}),k\in[ 1,B],\\ &{x_{1,\cdots,B}^{i}}=[x_{1}^{i},x_{2}^{i},\cdots,x_{k}^{i},\cdots,x_{B-1}^{i} ,x_{B}^{i}],\\ \end{split}$ (2) $$

As iterations proceed, the reference images evolve to be more coherent,
as Story-Adapter consistently improves the semantic consistency in a global view.
Additionally, generative quality for fine-grained interactions is also constantly optimized as Story-Adapter repeatedly engages text prompt constraints during iterations.

Figure: Algorithm 1 Pseudo-Code of Story-Adapter.

### 3.3 Global Reference Cross-Attention

Although incorporating image context or reference images extends text-to-image generation to character-consistent image sequences,
existing AR paradigms (Pan et al., ; Liu et al., ) suffer from error accumulation over long stories,
while RI paradigms (Zhou et al., )
may propagate flaws from the reference images.

In contrast, we propose an efficient plug-and-play augmentation module to equip SD models, called Global Reference Cross-Attention (GRCA).
We utilize a pre-trained CLIP (Radford et al., ) image encoder to extract a global embedding $c$ for each reference image from the previous round, effectively preserving the semantics of reference images using only a few tokens.
The token simplification allows GRCA to incorporate all reference images as guidance in the cross-attention process without incurring significant computational overhead.

In the $i_{th}$ iteration of Story-Adapter, given all the reference images in the previous round $x_{1,\cdots,B}^{i-1}\in{\mathbb{R}^{{B}\times{h}\times{w}\times\rm{3}}}$, ${h}$, ${w}$ denote reference image resolution.
We define a function ${\rm{Attention}}(Q,K,V)$ to indicate the attention calculation, where $Q$, $K$, and $V$ represent the query, key, and value in the attention, respectively.
GRCA in the visualization for the $k_{th}$ image can be specified as:

$$ $\begin{split}&{c_{1,\cdots,B}^{i}}={\rm{CLIP}}(x_{1,\cdots,B}^{i-1}),{c_{1, \cdots,B}^{i}}\in{\mathbb{R}^{{B}\times{d}}},\\ &{c_{1,\cdots,B}^{i}}={c_{1,\cdots,B}^{i}}{W_{c}},{W_{c}}\in{\mathbb{R}^{{d} \times{ne}}},\\ &{c_{1,\cdots,B}^{i}}={\rm{flatten}}({c_{1,\cdots,B}^{i}}),{c_{1,\cdots,B}^{i} }\in{\mathbb{R}^{{1}\times{Bn}\times{e}}},\\ &{{Q}_{k}^{i}}={{I_{k}}{W_{q}}},{{K_{k}^{i}}}={{c_{1,\cdots,B}^{i}}{W_{k}}},{{ V}_{k}^{i}}={{c_{1,\cdots,B}^{i}}{W_{v}}},\\ &{{\rm{GRCA}}({I_{k}},{x_{1,\cdots,B}^{i-1}})}={{\rm{Attention}}({{Q}_{k}^{i}} ,{{K_{k}^{i}}},{{V}_{k}^{i}})},\end{split}$ (3) $$

Where $W_{c}$ is the projection matrix of global embeddings transformed into reference tokens. $d$ and $e$ denote the embedded dimension of global embeddings and the projection dimension of the projection matrix, respectively.
$n$ indicates the number of reference tokens for a single reference image, $n=4$ if not specified.
$\rm{flatten(.)}$ represents a flatten operation for vectors.
$W_{q}$ is the mapping weight matrix for the intermediate denoising feature $I$ in SD.
$W_{k}$, $W_{v}$ are the mapping weight matrices of the reference tokens.

Eventually, we merge the outputs from GRCA with the outputs from text cross-attention, to guide the visualization of $k_{th}$ image in the story.
In particular, with corresponding text prompt $T_{k}$ and all reference images $x_{1,\cdots,B}^{i-1}$, the intermediate denoising feature ${I}_{k}^{i}$ is obtained as follows:

$$ $\begin{split}&{{I}_{k}^{i}}={{\rm{Attention}}({I}_{k}^{i},{T}_{k},{T}_{k})}+ \lambda{{\rm{GRCA}}({I_{k}^{i}},{x_{1,\cdots,B}^{i-1}})}.\\ \end{split}$ (4) $$

where $\rm{\lambda}$ is a balance factor for controlling the influence of GRCA on the visualization results.
We propose a linear weighting strategy to adjust the weight factor for each iteration, where the weight factor increases linearly with a low value to trade off visual consistency and text alignment in the iterative paradigm.
Since the existing diffusion models contain a cross-attention design associated with the reference image,
our GRCA could be directly plugged in and reuse the cross-attention weights without training.
We demonstrate the procedure of Story-Adapter, along with the linear weighting strategy in Algo. 1.

## 4 Experiments

In this section, we first introduce the datasets, the evaluation metrics, and implementation details.
Then we compare Story-Adapter with previous AR-based and RI-based methods for visualization of both regular-length and long stories.
Finally, we validate the effectiveness of the proposed iterative paradigm and Global Reference Cross-Attention (GRCA) through extensive ablations.
Additional experimental results, comparison on subject-consistent generation, and human evaluation can be found in the Appendix.

### 4.1 Dataset and Evaluation

We use the StorySalon dataset (Liu et al., ) to benchmark performance for regular-length story visualization.
For long story visualization, we curate multiple long stories using GPT-4o (OpenAI, ).
To evaluate the efficacy of Story-Adapter, we report CLIP text-image similarity (CLIP-T) (Radford et al., ), average Fréchet Inception Distance (aFID) (Cheng et al., ), and Character-Character Similarity (aCCS) (Cheng et al., ).
CLIP-T is to measure image-text alignment, both aFID and aCCS are used to evaluate semantic consistency among generated images.

Figure: Figure 4: Qualitative comparisons for regular-length story visualization. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/Fig_4.jpg

### 4.2 Implementation Details

To ensure a fair comparison, we used the weights of IP-Adapter (Ye et al., ) and IP-AdapterXL (Ye et al., ), respectively, resulting in two models: Story-Adapter and Story-AdapterXL.
We utilized DDIM (Song et al., ) for 50-step sampling with an unclassified classifier guidance score set to 7.5.
For the hyperparameters in our iterative paradigm, we set the number of story iterations to 10 by default.
The weight factor $\lambda$ is set to 0.3 for the initial iteration and 0.5 for the final iteration, with linearly interpolated values for the intermediate iterations by our linear weighting strategy.

### 4.3 Regular-length Story Visualization

Based on the standard setup on StorySalon dataset (Liu et al., ), we compare with existing story visualization methods and Stable Diffusion Model (SDM) baselines, including StoryDiffusion (Zhou et al., ), StoryGen (Liu et al., ), AR-LDM (Pan et al., ), SDM (Rombach et al., ), Finetuned-SDM (fine-tuned on StorySalon), and Prompt-SDM.
For Prompt-SDM, we use prompts of “cartoon-style images”.
To adhere to copyright restrictions and ensure fair comparisons, we exclusively utilize text prompts from the open-source subset of the StorySalon test set for evaluation. This subset comprises 6,026 prompts, with an average of 14 frames per story and the longest story containing up to 44 frames.

**Table 1: Quantitative comparison for regular-length story visualization.**
| Method | CLIP-T $\uparrow$ | aCCS $\uparrow$ | aFID $\downarrow$ |
| --- | --- | --- | --- |
| SDM (Rombach et al., ) | 0.323 | 0.662 | 23.10 |
| Prompt-SDM (Rombach et al., ) | 0.289 | 0.699 | 18.18 |
| Finetuned-SDM (Rombach et al., ) | 0.309 | 0.639 | 23.05 |
| AR-LDM (Pan et al., ) | 0.237 | 0.683 | 40.25 |
| StoryGen (Liu et al., ) | 0.255 | 0.724 | 36.34 |
| Story-Adapter (Ours) | 0.305 | 0.760 | 16.52 |
| StoryDiffusion (Zhou et al., ) | 0.311 | 0.765 | 14.84 |
| Story-AdapterXL (Ours) | 0.310 | 0.818 | 14.63 |

#### Quantitative Evaluation.

CLIP-T results in Tab. 2 show that Story-Adapter and StoryDiffusion (Zhou et al., ) visualize content more aligned to the text prompt than previous story visualization models (AR-LDM and StoryGen).
Meanwhile, since neither Story-Adapter nor most baselines are trained on the StorySalon dataset, we introduce aFID and aCCS metrics for a fair evaluation of the character consistency among generated story images.
Results of aFID and aCCS in Tab. 2 illustrate that Story-Adapter achieves higher semantic consistency of the generated images compared to StoryDiffusion.
Such results validate the effectiveness of our design for coherent image sequence visualization.

#### Qualitative Evaluation.

In Fig. 4, we provide the qualitative comparison results of the open-ended story visualization.
Although AR-LDM and StoryGen generate coherent image sequences based on story prompts, the quality of the generated images degrades when story length increases due to the error accumulation issue of the AR paradigm.
Results of StoryDiffusion (Zhou et al., ) and Story-Adapter show satisfactory story visualization performance.
However, StoryDiffusion cannot maintain consistency between certain subjects due to lacking global story comprehension (e.g., “cat” in Fig. 4).
Additionally, since StoryDiffusion requires the first few generated images as references, the visualization results are affected by the reference image flaws (e.g., “closed-eye issue” in Fig. 4).
In comparison, Story-Adapter performs better in regular-length story visualization benefited from the global features engaged in GRCA.

Figure: Figure 5: Qualitative comparisons for long story visualization. The image sequences in orange and blue boxes are generated by StoryDiffusion and Story-Adapter, respectively. Story-Adapter shows advantages in generating semantic consistency and character interactions. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/Fig_5.png

Figure: Figure 6: Ablation study of iterative paradigm: the effect of the iterative paradigm and the impact of different fixing $\lambda$. Zoom in for a better view. See Appendix for results with more iterations.
Refer to caption: extracted/5908139/fig/Fig_6.jpg

### 4.4 Long Story Visualization

To better evaluate generative quality for long story visualization (i.e., up to 100 frames), we compare to subject-consistent image generation model IP-Adapter (Ye et al., ) in addition to existing story visualization methods.
SDM baselines are not included in comparison as they are not suitable to generate long consistent content.
We use GPT-4o (OpenAI, ) to generate 20 long story cases of ten 50-sentence descriptions and ten 100-sentence descriptions.

#### Quantitative Evaluation.

The quantitative results in Tab. 2 show that our Story-Adapter significantly improves the semantic consistency and the generative coherence for fine-grained interactions for long story visualization compared to existing models.
Notably, IP-Adapter employs the same guidance image that leads to less aFID. In contrast, our method improves visual consistency without the need to fix the same reference image.

#### Qualitative Evaluation.

Fig. 5 shows the visualization results for long stories, indicating that Story-Adapter can generate high-quality, thematically consistent long image sequences based on the text prompts.
In particular, StoryDiffusion cannot convey interactions between multiple characters correctly (e.g., “turtle lifting the fishbone trophy” in the 34-th frame and “rabbit running past the camel” in the 46-th frame), whereas Story-Adapter visualizes the interactions between the characters accurately while maintaining subject consistency.

#### Computational Cost Comparison.

We evaluate the computational cost of single-image generation using CSA in StoryDiffusion (Zhou et al., ) and the proposed GRCA with varying numbers of reference images,
under the base attention setting for fair comparison.
FLOPs are calculated within the diffusion model UNet.
As shown in Fig. 8, as the number of reference images increases, StoryDiffusion experiences a significant rise in computation in terms of FLOPs, while Story-Adapter and Story-AdapterXL are slightly affected.
This demonstrates the potential of modeling on global embeddings as in GRCA to efficiently sustain global story semantics for long story visualization.

Figure: Figure 7: Qualitative ablation studies of initialization and GRCA. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/Fig_8.jpg

### 4.5 Ablation Study

#### Global Reference Cross-Attention.

We ablate the effect of global semantics modeling by GRCA for long story visualization.
Specifically, for each image visualization in the sequence, we only use the single reference image at the corresponding index during the iteration as guidance.
By establishing a global comprehension of the story for the diffusion model, Story-Adapter maintains the semantic consistency in the generated image sequence (Tab. 3 and Fig. 4.4).

#### Iterative Paradigm.

We conduct ablation experiments to evaluate the effect of the proposed iterative paradigm for long story visualization and to validate our linear weighting strategy compared to the fixed weight factors.
As shown in Tab. 3 and Fig. 6, the iterative paradigm improves generation quality for fine-grained interactions and semantic consistency.
This is mainly because the iterative paradigm offers a global view of the
entire story, thus reducing error accumulation and alleviating the propagation of the reference image flaws.
A fixed weight factor of 0.3 minimally impacts visualization during iteration, while a fixed factor of 0.5 leads to excessive consistency in the image sequence.
This enables flexibility within the iterative paradigm.

#### Initialization.

To ablate the effect of the proposed initialization, we use a sequence of images consisting of the characters as reference images (i.e., w/o initialization).
Tab. 3 shows that when removing the proposed initialization, there is a significant decrease in the image-text alignment of Story-Adapter in terms of CLIP-T.
Fig. 4.4 illustrates that without initialization, the diffusion model fails to generate the required objects according to text prompts, e.g., “nightingale” and “robot”.

#### GRCA vs CSA.

We investigate GRCA and CSA in Tab. 3 and Fig. 4.4, using the outputs of the first iteration from Story-Adapter and StoryDiffusion, respectively.
Though GRCA generates less visual consistency during the first iteration than CSA in terms of aCCS and aFID in Tab. 3,
GRCA’s global comprehension improves the consistency of multiple characters throughout stories shown in Fig. 4.4.
For example, GRCA effectively preserves the consistency of emerging characters (e.g., “the character 1900”) while CSA fails.

## 5 Conclusions and Discussions

We introduce Story-Adapter, an iterative framework that adapts pre-trained Stable Diffusion models for long story visualization.
By using the generated images from previous iterations as references, our method maintains semantic consistency and enhances generative quality for fine-grained interactions throughout the story, effectively reducing error accumulation and avoiding the propagation of flaws.
For efficiency, we propose a plug-and-play Global Reference Cross-Attention (GRCA) module, which utilizes global image embeddings to reduce computational costs while preserving essential image information flow.
Extensive experiments demonstrate that Story-Adapter outperforms existing methods on the regular-length story visualization dataset, and shows strong results in long story visualization.
These findings highlight the potential of our iterative paradigm to advance the quality and coherence of text-to-image story visualization.

## Appendix A Paradigms

Existing story visualization methods usually employ the Auto-Regressive (AR) or Reference-Image (RI) paradigms.
In this work, we propose a novel iterative paradigm for story visualization.
Next, we will discuss different story visualization paradigms in detail.

Figure: Figure 9: Different paradigms for story visualization. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/Fig_paradigm.jpg

### A.1 Auto-Regressive Paradigm

#### Setting.

As shown in Fig. 9, AR paradigm-based methods typically use a limited number of previous frames and the corresponding text prompt of the current frame to guide current image generation.
This helps the methods maintain semantic consistency between consecutive frames.

#### Discussion.

However, the AR paradigm cannot consider future frames when synthesizing the current image, which makes the AR paradigm only maintain semantic consistency in neighboring frames but not throughout the story.
Besides, the AR paradigm easily suffers from error accumulation.
Therefore, the image quality of the AR paradigm gets worse as the length of the story increases.

### A.2 Reference-Image Paradigm

#### Setting.

RI paradigm-based methods employ the beginning visualized frames as reference images to guide the visualization of the rest of the story when performing long story visualization (see Fig. 9).
Bootstrapping based on fixed reference images helps the methods to effectively maintain identity consistency in long story visualizations.

#### Discussion.

However, such a setup ignores the consistency of emerging characters in the story, and all visualizations are affected by flaws in the reference images.
Both issues affect the quality of long story visualizations with the RI paradigm.

### A.3 Iterative Paradigm

#### Setting.

To address the aforementioned limitations, we propose an iterative paradigm in Story-Adapter (Fig. 9).
We constantly consider all generated images in the previous iteration with an iterative mechanism and model on the global embeddings.
Specifically, when generating for the $k_{th}$ image, we propose to implement Global Reference Cross-Attention (GRCA) on global embeddings from all generated images in the previous iteration.

#### Discussion.

By using all generated images from the previous iteration as reference images to guide the current generation, we effectively maintain semantic consistency throughout the story.
Moreover, all the generated images as references are updated through each iteration.
Taken together, the iterative paradigm effectively avoids the influence of defects in some reference images.

## Appendix B Subject-Consistent Generation Comparison

In the evaluation phase, we employ GPT-4o (OpenAI, ) according to the settings of StoryDiffusion (Zhou et al., ) to generate 20 character descriptions and 100 specific activity descriptions, respectively.
We combine them as 2000 test descriptions, to compare Story-Adapter and subject-consistent image generation baselines,
including IP-Adapter (Ye et al., ), PhotoMaker (Li et al., ), and StoryDiffusion (Zhou et al., ).

**Table 4: Quantitative comparison with subject-consistent image generation methods.**
| Method | CLIP-T $\uparrow$ | CLIP-I $\uparrow$ |
| --- | --- | --- |
| IP-Adapter (Ye et al., ) | 0.307 | 0.872 |
| Story-Adapter (Ours) | 0.326 | 0.877 |
| IP-AdapterXL (Ye et al., ) | 0.312 | 0.879 |
| PhotoMaker (Li et al., ) | 0.317 | 0.880 |
| StoryDiffusion (Zhou et al., ) | 0.330 | 0.882 |
| Story-AdapterXL (Ours) | 0.332 | 0.884 |

#### Quantitative Evaluation.

For quantitative comparisons on subject-consistent image generation, we employ CLIP text-to-image similarity (CLIP-T) and image-image similarity (CLIP-I) to measure consistency between the character images and generated images.
Tab. 4 shows that Story-Adapter achieves SoTA performance in terms of both quantitative metrics, which demonstrates Story-Adapter’s ability to generate subject-consistent image sequences based on text prompts or image prompts.

#### Qualitative Evaluation.

Fig. 10 shows the qualitative comparison results.
Story-Adapter generates higher-quality images in subject-consistency and detailed interactions.
In contrast, IP-Adapter fails to generate correctly, e.g., “paper”, “whiteboard”, and “chainsaw”.
PhotoMaker cannot generate images consistently, e.g., maintaining details of the attire.
Despite accurately generating content according to text prompts with visual consistency, StoryDiffusion suffers from visualizing complex details due to lacking global story comprehension.
By incorporating a global story view in our iterative paradigm, Story-Adapter can maintain visual consistency, especially in details throughout the story.

Figure: Figure 10: Qualitative comparison of subject-consistent image generation methods.
Refer to caption: extracted/5908139/fig/Fig_3.jpg

## Appendix C Human Evaluation

#### Setting.

To complement the evaluation metrics to accurately reflect the quality of the generated stories, we involve human evaluation to further compare Story-Adapter and baselines.
Referring to the setting in StoryGen (Liu et al., ), we invite participants to rate various aspects: text-image alignment (Align.), character interaction (Inter.), content consistency (Cons.), image quality (Qual.), and preference (Pref.) on a scale from 1 to 5.
The higher the better.

**Table 5: Human evaluation comparison of subject-consistent image generation, regular-length story visualization, and long story visualization. The best is highlighted in red.**
| Subject-Consistent Image Generation |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | Align. $\uparrow$ | Inter. $\uparrow$ | Cons. $\uparrow$ | Qual. $\uparrow$ | Pref. $\uparrow$ |
| IP-Adapter (Ye et al., ) | 2.51 | 3.27 | 4.58 | 4.33 | 4.19 |
| IP-AdapterXL (Ye et al., ) | 2.66 | 3.36 | 4.72 | 4.51 | 4.26 |
| PhotoMaker (Li et al., ) | 3.79 | 4.18 | 4.25 | 4.47 | 4.11 |
| StoryDiffusion (Zhou et al., ) | 4.15 | 4.28 | 4.50 | 4.54 | 4.48 |
| Story-Adapter | 4.02 | 4.20 | 4.41 | 4.39 | 4.33 |
| Story-AdapterXL | 4.20 | 4.35 | 4.58 | 4.61 | 4.54 |
| Regular-Length Story Visualization |  |  |  |  |  |
| SDM (Rombach et al., ) | 4.11 | 2.37 | 2.01 | 4.17 | 1.14 |
| Prompt-SDM (Rombach et al., ) | 4.03 | 3.49 | 1.99 | 4.40 | 1.26 |
| Finetuned-SDM (Rombach et al., ) | 3.35 | 3.82 | 2.15 | 3.41 | 1.60 |
| AR-LDM (Pan et al., ) | 3.08 | 3.64 | 2.90 | 2.64 | 2.05 |
| StoryGen (Liu et al., ) | 3.72 | 4.17 | 3.83 | 3.79 | 3.39 |
| StoryDiffusion (Zhou et al., ) | 3.96 | 4.48 | 4.52 | 4.24 | 4.37 |
| Story-Adapter | 3.89 | 4.21 | 4.36 | 4.09 | 4.10 |
| Story-AdapterXL | 4.06 | 4.60 | 4.74 | 4.53 | 4.62 |
| Long Story Visualization |  |  |  |  |  |
| AR-LDM (Pan et al., ) | 3.30 | 3.68 | 3.42 | 2.15 | 3.27 |
| StoryGen (Liu et al., ) | 3.51 | 4.06 | 3.88 | 2.72 | 3.51 |
| IP-Adapter (Ye et al., ) | 3.79 | 4.27 | 4.30 | 4.19 | 4.06 |
| IP-AdapterXL (Ye et al., ) | 3.83 | 4.23 | 4.61 | 4.47 | 4.11 |
| StoryDiffusion (Zhou et al., ) | 4.16 | 4.30 | 4.53 | 4.33 | 4.35 |
| Story-Adapter | 3.97 | 4.15 | 4.42 | 4.19 | 4.29 |
| Story-AdapterXL | 4.35 | 4.47 | 4.70 | 4.62 | 4.65 |

#### Results.

Tab. 5 shows that our Story-Adapter receives more preference from the participants.
It is worth noting that although IP-Adapter receives higher scores for consistency in the subject-consistent image generation task, Story-Adapter is more favored in text-image alignment and generating character interactions.
For regular-length and long story visualization, Story-Adapter is more preferred compared to baselines in most evaluation aspects, especially visual consistency and capability to generate character interactions.
This is aligned with the quantitative measurement.

## Appendix D More Iterations

#### Setting.

In this section, we compare results on different iterations in the iterative paradigm and investigate the impact of longer iterations on story visualization.
Specifically, we study the visualization results in the initialization, $1_{st}$, $5_{th}$, $10_{th}$, and $15_{th}$ iterations, respectively.

Figure: Figure 11: Story visualization results from different iterations by Story-Adapter. Accurate interactions are denoted in green, and wrong or missing ones are in red.
Refer to caption: extracted/5908139/fig/Fig_more_iter.jpg

#### Results.

Tab. 6 shows that as iteration increases, Story-Adapter achieves significant improvement in visual consistency (aCCs and aFID) while text-image alignment (CLIP-T) drops slightly.
This further demonstrates the contribution of the iterative paradigm to the semantic consistency of the overall story.
However, we also note that a further increase in iterations harms text-image alignment, with limited gain in visual content consistency.
This indicates that while Global Reference Cross-Attention (GRCA) effectively improves the content consistency of the long story, the increasing weighting factor of GRCA during the iterations poses a challenge to aligning the text prompts.

**Table 6: Quantitative comparison of multiple iterations.**
| Iteration | CLIP-T $\uparrow$ | aCCS $\uparrow$ | aFID $\uparrow$ |
| --- | --- | --- | --- |
| initialization | 0.330 | 0.502 | 214.94 |
| $1_{th}$ iteration | 0.322 | 0.757 | 105.17 |
| $5_{th}$ iteration | 0.319 | 0.783 | 100.81 |
| $10_{th}$ iteration | 0.306 | 0.840 | 91.35 |
| $15_{th}$ iteration | 0.297 | 0.848 | 90.62 |

Fig. 11 demonstrates a significant improvement in generative quality for fine-grained interactions as the iteration proceeds.
The iterative paradigm effectively alleviates the diffusion model’s limitations on complex interaction generation by continuously creating input channels for text prompts.
But more iterations wouldn’t improve the generation quality further.
Therefore, 10 iterations in the iterative paradigm is an optimal choice based on the quantitative and qualitative experiments.

## Appendix E More Visualization Results

In this section, we provide more visualization results from Story-Adapter and the baselines.

### E.1 Visual Comparison

We compare the long story visualization results of representative work with AR-based, RI-based, and iterative paradigms, respectively.
Specifically, Fig. 12, Fig. 13, and Fig. 14 show the generated results of the same “Pianist” story from the proposed Story-Adapter (iterative), StoryGen (Liu et al., ) (AR-based), and StoryDiffusion (Zhou et al., ) (RI-based), respectively.

#### Results.

Fig. 12 shows that the visualization quality from StoryGen constantly gets worse as the length of the story increases.
In Fig. 13, StoryDiffusion maintains high visual quality throughout the story, but it suffers from the flaw in the beginning frame that serves as the reference image, e.g., “closed-eye”.
In addition, the subject “the character 1900” is not consistently generated as baby, kid, and adult.
In contrast, our Story-Adapter effectively achieves high-quality story visualization and addresses the aforementioned limitations (see Fig. 14).

Figure: Figure 12: Visualization results of StoryGen for the “Pianist” story. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/xagen.png

Figure: Figure 13: Visualization results of StoryDiffusion for the “Pianist” story. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/xastory.png

Figure: Figure 14: Visualization results of our Story-Adapter for “Pianist”. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/xaour.png

### E.2 Longer Story Visualization Results

In Fig. 15, we show the visualization results of the long story (up to 100 frames).

Figure: Figure 15: Our long story visualization results for “Winnie the Pooh”. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/xcomic24.png

### E.3 Different Style

We provide the long story visualization results from Story-Adapter in a realistic style in Fig. 16.
The experiment results suggest that Story-Adapter can be applied to different visual styles as well.

Figure: Figure 16: Our realistic style story visualization results for “loyal dog”. Zoom in for a better view.
Refer to caption: extracted/5908139/fig/xcomic22.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See above weaknesses.
soundness: 3
strengths: 1. The proposed method is training-free and requires few computational resources to generate long stories.

2. The introduced reference cross-attention module is general and could potentially be adopted in other existing story visualization models.

3. A human evaluation was conducted to further assess the effectiveness of the proposed method.
summary: The paper proposes a training-free method for the story visualization task, where an iterative paradigm is implemented to continuously refine previously generated images with the help of a proposed global reference cross-attention module. Both qualitative and quantitative results show that the proposed method effectively improves consistency between frames in a story.
weaknesses: 1. I am concerned about the effectiveness of the proposed method. As shown in the paper, the method is training-free, and the consistency between different frames in a story relies primarily on the proposed global reference cross-attention (GRCA) module. According to Eq. (1), there is no connection between frames during initialization, and each image frame is generated independently, as x^i=0_k is generated solely from its corresponding text T_k. It is unclear how the GRCA can significantly adjust each image frame to achieve consistent characters and background images. It might be better to include the initialisation images in the paper, and also some discussion about the such powerful capabilities of GRCA.

2. The idea of using global reference cross-attention to incorporate global features into generation does not appear entirely novel. It utilizes global features, which primarily address global consistency between image frames but have limited impact on regional details. Consequently, issues with regional consistency, such as closed-eye issue (L 421), may also be a limitation of the proposed method.

3. How did the authors choose the lambda values in Eq. (4)? Table 3 suggests a tradeoff between text-image alignment and image consistency when adjusting the lambda value.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to the weakness part.
soundness: 3
strengths: 1. The plug-and-play nature of Story-Adapter allows it to integrate seamlessly with existing pre-trained Stable Diffusion models, offering a practical solution for enhancing model performance without extensive retraining.

2. The framework's iterative approach provides a mechanism to enhance narrative consistency and visual quality over successive iterations, improving fine-grained interactions and maintaining semantic consistency.

3. GRCA reduces computational overhead by leveraging global embeddings, making Story-Adapter suitable for long story visualizations with minimal additional memory usage.
summary: This paper introduces the Story-Adapter framework, a training-free approach aimed at enhancing long story visualization in computer vision. The proposed method focuses on achieving high semantic consistency and generating intricate character interactions across extended narratives (up to 100 frames) without additional training. The core components include an iterative paradigm and a Global Reference Cross-Attention (GRCA) module. These features enable the model to maintain coherence across all frames by iteratively refining generated images, with GRCA ensuring that each new frame adheres to the global semantic context established by preceding frames.
weaknesses: 1. The novelty of this paper is somewhat limited.

2. The paper is overall hard-to-follow. For example, Eq. 3 is so confusing and lacks clarity in terms of implementation details. The left-hand variables in the first three lines are identical, which raises questions about potential typographical errors.

3. In generating each frame of the story, is the reference image derived from the one frame from the previous round only, or from all images generated in the previous round? If the former, it remains unclear how multiple rounds of using the same image can maintain character consistency across the story. If the latter, as the story progresses and new characters are introduced, how is it ensured that characters who appear less frequently are not overlooked?

4. In Fig. 3, different colors denote different characters. Is a masking mechanism applied for each character to distinguish them? If not, how is it ensured that information about each character can be accurately extracted using global reference cross-attention?

5. Although Fig. 5 highlights the advantages of this method compared to StoryDiffusion, these advantages are less pronounced in Figure 14. For instance, in Figure 14, the male characters appear highly similar. For example, the crew member in the third frame, the captain in the fourth frame, and the adult character 1900 in subsequent frames look nearly identical.

6. Adding a discussion section on the limitations of the proposed method would strengthen the paper's argument.

7. There are some training-free methods that deserve to be discussed.

	a) Cao M, Wang X, Qi Z, et al. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 22560-22570.

	b) Tewel Y, Kaduri O, Gal R, et al. Training-free consistent text-to-image generation[J]. ACM Transactions on Graphics (TOG), 2024, 43(4): 1-18.

	c) He H, Yang H, Tuo Z, et al. DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion[J]. arXiv preprint arXiv:2407.12899, 2024.

	d) Cheng J, Lu X, Li H, et al. AutoStudio: Crafting Consistent Subjects in Multi-turn Interactive Image Generation[J]. arXiv preprint arXiv:2406.01388, 2024.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 4
questions: - Since the framework is basically the same as IP-Adapter, it is valuable to discuss how to adaptively select the reference image.
- Also, it would be interesting to discuss what the real obstacles to achieving consistent generation are, is a CLIP representation enough for detail consistency? If not, if some other representations like DINO-V2 help?
soundness: 3
strengths: - Consistent generation, or story generation is a good task with many real-world applications.
- This approach is training-free and can be easily applied into existing framework.
summary: This paper proposes a training-free framework Story-Adapter to achieve consistent image generation. Instead of relying on static reference images, this work incorporates all images generated in previous iterations (in a dynamic manner) into the current iteration, which is achieved through global reference cross-attention. The method adopts an iterative paradigm. For initialization, only the corresponding prompts are used to generate images, just like regular text-to-image generation. Then, all images generated in the previous round are used to update the current generation. Specifically, all images in the previous round are projected into the image tag and used as K and V to calculate a new set of intermediate features, which are finally weighted and summed into the final features.
weaknesses: - The novelty is quite limited, this is exactly IP-Adapter, but just wrap it up as a storytelling story. The only difference is that it uses all previous images as references.
- It does not make sense to use all previous images as references, if some images are irrelevant to the current generation, it may become a noisy reference and harm the current generation. There is no discussion on how to select reference images adaptively.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: In addition to the weaknesses mentioned, I have another one question.

Your method depends on initialization to generate a fixed character. How does the choice of initialization affect the robustness of the method? For instance, if the initialization produces significantly different images, can Story-Adapter still converge to a consistent character in a relatively short time?

I hope the authors can make up for the weaknesses mentioned and address these questions.
soundness: 3
strengths: 1. Iterative Enhancement: Unlike previous works that primarily focus on direct generation of consistent results, Story-Adapter leverages iterations as a key dimension for enhancing quality.
2. Detailed Description of Differences: The paper thoroughly analyzes previous works, highlighting their drawbacks and clearly articulating the distinctions between their contributions and those of others.
3. Clear and Concise Expression: The paper articulates complex ideas with clarity, supported by detailed images and diagrams that enhance understanding and reinforce the technical explanations.
4. Comprehensive Experiments and Ablation Studies: The paper includes thorough experimental evaluations and well-structured ablation studies for each module, effectively validating the system's design and performance.
5. Detailed Formulas and Algorithms: The paper provides clear formulas and algorithms to eliminate ambiguity.
summary: This paper introduces a framework aimed at improving the quality and consistency of visualizations for stories, especially long stories, using diffusion models. Traditional models often struggle with maintaining semantic consistency and intricate interactions as story length increases. To address this, the authors propose Story-Adapter as a training-free iterative approach.

Story-Adapter features a unique Global Reference Cross-Attention (GRCA) mechanism, which helps maintain consistency across images by using global embeddings from all previously generated frames. Story-Adapter updates each frame iteratively by considering all previous results, performing better in consistency and coherence compared with previous methods.

Experiments show that Story-Adapter outperforms existing methods like StoryGen and StoryDiffusion in both standard and longer story tasks when using CLIP-T, aCSS and aFID as the benchmark.
weaknesses: 1. Comparison with Baseline for Regular-length Stories: The paper utilizes IP-Adapter as a key component of the method, making it crucial to include IP-Adapter as a baseline for comparison. While the authors do provide this comparison for long story visualizations in Table 2, it is notably absent for regular-length stories in Table 1.
2. Analysis of Iterative Generation: Iterative generation appears to be a significant differentiator between Story-Adapter and IP-Adapter. However, there is no curve of the performance changes over iteration epochs. The authors should consider including this as part of their experimental evaluation.
3. Insufficient Analysis of Computational Costs: The authors analyze the computational cost between Story-Adapter and StoryDiffusion, yet they do not account for the multiple epochs required for Story-Adapter's generation in their cost calculations. It would be beneficial to present the time consumption explicitly as an additional column in Tables 1 and 2.
4. Ambiguity in Figure 3: The presence of the white and purple masks in the bottom right corner of Figure 3 may create confusion regarding the existence of attention masks.