StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
Authors: Yupeng Zhou
Daquan Zhou
Ming-Ming Cheng
Jiashi Feng
Qibin Hou
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Consistent character generation
Diffusion model
Image generation
Video generation
Transition prediction
Primary area: diffusion_based_models
TL;DR: Generating long-range image and video with consistent characters, based on Consistent Self-Attention and Motion Predictor.

Abstract: For recent diffusion-based generative models, maintaining consistent content across a series of generated images, especially those containing subjects and complex details, presents a significant challenge. In this paper, we propose a simple but effective self-attention mechanism, termed Consistent Self-Attention, that boosts the consistency between the generated images. It can be used to augment pre-trained diffusion-based text-to-image models in a zero-shot manner. Based on the images with consistent content, we further show that our method can be extended to long range video generation by introducing a semantic space temporal motion prediction module, named Semantic Motion Predictor. It is trained to estimate the motion conditions between two provided images in the semantic spaces. This module converts the generated sequence of images into videos with smooth transitions and consistent subjects that are more stable than the modules based on latent spaces only, especially in the context of long video generation. By merging these two novel components, our framework, referred to as StoryDiffusion, can describe a text-based story with consistent images or videos encompassing a rich variety of contents. The proposed StoryDiffusion encompasses pioneering explorations in visual story generation with the presentation of images and videos, which we hope could inspire more research from the aspect of architectural modifications.

StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
Yupeng Zhou!
Daquan Zhou 25t Ming-Ming Cheng
Jiashi Feng²
Qibin Hou 1:3ft
VCIP & TMCC, CS, Nankai University 2 ByteDance Inc.
NKIARI, Futian, Shenzhen
https://Storppiffusionn wit.on..itttuuii..g
Abstract
For recent diffusion-based generative models, maintaining consistent content across series of generated images, especially those containing subjects and complex details, presents a significant challenge. In this paper, we propose a simple but effective self-attention mechanism, termed Consistent Self-Attention, that boosts the consistency between the generated images. It can be used to augment pre-trained diffusion-based text-to-image models in zero-shot manner. Based the images with consistent content, we further show that our method can be extended to long- range video generation by introducing semantic space temporal motion prediction module, named Semantic Motion Predictor. It is trained to estimate the motion conditions between two provided images in the semantic spaces. This module converts the generated sequence of images into videos with smooth transitions and consistent subjects that are more stable than the modules based on latent spaces only, especially in the context of long video generation. By merging these two novel components, our framework, referred to as StoryDiffusion can describe a text-based story with consistent images videos encompassing a rich variety of contents. The proposed Story Diffusion encompasses pioneering explorations visual story generation with the presentation of images and videos, which we hope could inspire more research from the aspect architectural modifications.
Introduction
With extensive pre-training and advanced architectures, diffusion models have shown superior performance in generating very high-quality images and videos over previous generative-adversaniy network (GAN) based methods [5]. However, generating subject-consistent (e.g. characters with consistent identity and attire) images and videos to describe a story is still challenging for existing models. The commonly used IP-Adapter [55] taking an image reference could be used guide the diffusion process to generate images similar to it. However, due to the strong guidance, the controllability over the generated content of the text prompts reduced. On the other hand, recent -tate-of-the-art identity preservation methods, such as InstantID [47] and PhotoMaker [26], focus on identity controllability
the consistency of the attires and the scenarios cannot guaranteed. Hence, in this paper, we aim to find method that can generate images and videos with consistent characters in terms of both identity and attire while maximizing the controllability of the user via text prompts.
common approach preserve the consistency between different images (or frames in the context of video generation) is to use a temporal module [15, 4]. However, this requires extensive computational resources and data. Differently, we target to explore lightweight method with minimum data and computational cost, or even in a zero-shot manner.
DDuiing the internship at ByteDance Inc. Prooect lead. 'Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Consistent images generated by StoryDiffusion
"Jungle Adventure" TM
"The Moon Exploration by Lecun" "
Ref Image
newepaper there was treesure in the wooden house the Sorest
Lecun the momming
Lecun was told he moon NASA
He drove to the Eorest
siger sppenced!
frishteed and run
rooket
Lecun reached the moon surface
Suddemly Im found walkoed inside
There bugo treastre in bouse!
Lecun completed successfully
on moon
(a)
Transition Videos generated by StoryDiffusion
"Video Clips"
LLon-RRagee Video"
Figure 1: Images and videos generated by our StoryDiffusion, (a) Comic generated by StoryDiffusion telling the story of man who discovers treasure while exploring the jungle. (b) Comic generated by StoryDiffusion Diffusionn describing the expedition to the moon by Lecun, with a reference image control [26] same as Fig. 7(b). (c) Videos generated by our StoryDiffusion. Click the image play the video. Best viewed with Acrobat Reader. More generated videos can be found in the uploaded supplementary file.
As evidenced by previous works [45, 19, 6], self-attention is one of the most important modules for modeling the overall structure of the generated visual content. Our main motivation is that we could use some shared reference image information to guide the self-attention calculation, the consistency between generated images supposed to be improved clearly. As the self-attention weights are input-dependent, model training or fine-tuning might not be required. Following this idea, we propose Consistent Altf-Atttntion, the core of our StoryDiffusion, which can be inserted into the diffusion backbone to replace the original self-attention in zero-shot manner.
Different from the standard self-attention that operates on the tokens representing single image, Consistent Self-Attention incorporates reference tokens sampled from reference images during the token similarity matrix calculation and token merging. The sampled tokens use the same set of Q-K-V weights and thus no extra training is required. As shown in Fig. 1, the generated images using Consistent Self-Attention successfully preserve the consistency in both identity and attire, which is vital for storytelling. Intuitively, Consistent Self-Attention builds correlations across images in the batch, generating consistent character images in terms of identity and attire, such as clothes. This enables to generate subject-consistent images for storytelling.

---

For any given story text, we begin by dividing it into several prompis, with each prompt corresponding to an individual image. Then our method could generate highly consistent images that effectively narrate story. To support long story generation, we also implement Consistent Self- Seff-Attention together with a sliding window across the generated consistent images. This removes the peak memory consumption's 8 dependency on the input text length, making it possible to generate long stories. To stream the generated story frames into videos, we further propose Semantic Motion Predictor that can predict transitions between two images in the semantic spaces. We empirically found that predicting motions in the semantic space generates more stable results than the predictions in the image latent spaces. Combined with the pre-trained motion module [13], Semantic Motion Predictor can generate smooth video frames that are notably better than recent conditional video generation methods, such SEINE [7] and SparseCtrl [12].
Our contributions are summarized below:
We propose a training-free and hot- hot-pluggable attention module, termed Consistent Self- Attention. It can maintain the consistency of characters in sequence of generated images for storytelling with high text controllability.
We propose a new motion prediction module that can predict transitions between two images in the semantic space, termed Semantic Motion Predictor. It can generate more stable long-range video frames that can be casily upscaled to minutes than recent popular image conditioning methods, such SEINE [7] and SparseCtrl [12].
We demonstrate that our approach could generate long image sequences or videos based on pre-defined text-based story with the proposed Consistent Self-Attention and Semantic Motion Predictor with motions specified by text prompts. We term the new framework as StoryDiffusion
2 Related work
2.1 Controllable text-to-image diffusion generation
As important sub-field of diffusion model applications [40, 17, 36, 38, 27, 52, 41], text-to-image generation [37, 33, 34], has attracted considerable attention recently. In addition, to enhance the controllability of text-to-image generation, multitude of methods emerged well. Among them, ControlNet [57] and T21-Adapter [31] introduce control conditions, such as depth maps, pose images, or sketches, direct the generation of images. MaskDiffusion [61] and StructureDiffusion [9] focus on enhancing the text controllability. There are also some works [30, 28] controlling the layout of generated images.
Preservation, which is expected to generate images with a specified ID, is also a hot topic. According to whether test-time fine-tuning is required, these works can be divided into two major categories. The first one only requires fine-tuning a part of the model with given image, such as Textual Inversion [10], DreamBooth [39], and Custom Diffusion [25]. The other one, exemplified by IPAdapter [55] and PhotoMaker [26], leverages models that have undergone pre-training on large datasets, allowing the direct use of given image control image generation. Different from both of the two types, we focus on maintaining the subject consistency in multiple images, to narrate story, Our Consistent Self-Attention is training-free and pluggable and can build connections across images within a batch to generate multiple subject-consistent images.
2.2 Video generation
Due to the success of diffusion models in the field of image generation [37, 17], the exploration in the domain of video generation [13, 23, 42, 49, 54, 56] is also becoming popular. VDM [15] is among the first that extends the 2D U-Net from image diffusion models to 3D U-Net to achieve text-based generation. Later works, such as MagicVider [60] and Mindscope [46], introduce ID temporal attention mechanisms, reducing computations by building upon latent diffusion models. Following Imagen, Imagen Video [16] employs a cascaded sampling pipeline that generates videos through multiple stages.
In addition to traditional end-to-end text-to-video (T2V) generation, video generation using other conditions is also an important direction. This type of methods generates videos with other auxiliary
3

---

( Teat-tt--Magg Piffusion with Corsistout Self- Artratioo
(+) Consisiont Solf-AtH Block (CAS)
[e] Consistent Sol--Att Layer (CAL)
Roseetllccs
Ranellach
Cross Atterton
CAL
Nor
CSA
Norta
CAB
CAB
Xi
Xc-1
Boost ComtisteMEN Consistent Self-Ar Self-AAten Hion
Piffissoo UINGT
Split the Story into promptts lyy GFFT and Geverate in Batthh Story;
gating side, then torns liis ltsad look straightt ained and contiincing hii wolk: Along Hhs spott lyy the roadside, bronds down piek up, He
sovin mow
Sowpling
drest
forward, loaghting joyffilly
ToCoNS
(S+N)=C
Softrwek
Conteet
Genorated Consistont Images Storgtellng
Nx(S NN)
Linoat
N=C
N-C
(4) Consistart Self-Attortion
Figure 2: The Pipeline of StoryDiffusion to generating subject- cobbect-consiteent images. To create subject- consistent images to describe story, we incorporate our Consistent Self-Attention into the pre-trained text-to-image diffusion model. We split story text into several prompts and generate images using these prompts batch. Consistent Self- ooff-Atteniion builds connections among multiple images in batch for subject consistency.
controls, such as depth maps [12, 14], pose maps [53, 21, 48, 29], RGB images [3, 7, 7,3,, or other guided motion videos [59, 51].
Our video generation method focuses on transition video generation, which is expected generate videos with given start frame and an end frame. Typical related works are SEINE [7] and SparseC- trl [12]. SEINE randomly masks video sequences the initial input of the video diffusion models in training to enable the predictions of the transition between two frames. SparseCtrl introduces a sparse control network synthesize the corresponding control information for each frame using sparse control data, thereby directing the generation of videos. However, the aforementioned transition video generation methods rely solely on temporal networks image latent space for the predictions of intermediate content. Thus, these methods often perform poorly on complex transitions, such as large-scale movements of characters. Our StoryDiffusion aims to perform predictions in image semantic spaces to achieve better performance and can handle larger movements, which will show in our experiment section.
Method
Our method can be divided into two stages, as shown in Fig. and Fig. 3. In the first stage, StoryDiffusion utilizes Consistent Self-Attentin to generate subject-consistent images in training- free manner. These consistent images can be directly applied to storytelling and can also serve as input for the second stage. In the second stage, our StoryDiffusion create consistent transition videos based on these consistent images.
3.1 Training-free consistent images generation
The key to addressing the above issues lies in how to maintain consistency of characters within batch of images. This means we need to establish connections between images within batch during generation. The previous image and video editing methods [6, 50] use DDIM inversion and insert additional keys and values in attention calculation [40] to keep similarity. Unlike existing methods [24, 20, 11] applied single images or highly similar video clips, we aim to generate set of images where the character remains consistent, yet each image portrays different scenes and actions, for use in anime production dory-boardinng Therefore, aim to share intermediary tokens within

---

batch of images to enable mutual interaction through self self-attention computation, thereby preserving consistency. We obtain these intermediary tokens by randomly sampling some pixels from the image batch before attention calculation, enabling plug-and-play capability and eliminating the need for training. We name this operation as Consistent Self- Attf-Atttntion and insert it into the location of the original self-attention in the existing U-Net architecture of image generation models and reuse the original self-attention weights.
Formally, given a batch of image features I E R where B, N, and C are the batch size, number of tokens in each image, and channel number, respectively. we define function Attention [Xx. Xg. Xx) ) to calculate self-attention. Xx. Xg. and X₂ stand for the query, key, and value used in attention calculation, respectively. The original self-attention performed within each image feature I; in I independently. The feature I; is projected to Qi K,, Vi and sent into the attention function, yielding:
O, Attention (Q:, Kg V₁).
To build interactions among the images within a batch to keep subject consistency, our Consistent Self-Attention samples some tokens S; from other image features in the batch:
S RandSample (11, 1,, 1-11, 1, 1+41, IB-1,
where RandSample denotes the random sampling function. After sampling, we pair the sampled tokens S; and the image feature to form on P. to generate the new key Kpt and value Vp, for Consistent Belf-Attention. Here, the original query Q; is not changed. Finally, we compute the self-attention as follows:
of tokens P.- We then perform linear projections
O Attention (Qi, Kpi, Vp.).
Given the paired tokens, our method performs the self-attention across batch of images, facilitating interactions among features of different images. This type of interaction promotes the model to the convergence of characters, faces, and attires during the generation process. Despite the simple and training-free free manner, our Consistent Self-Attention which we will demonstrate detail our experiments. These images serve as illustrations to narrate complex story shown in Fig. 2. To make it clearer, also show the pseudo code in Algorithm ?? in the Appendix.
efficiently generate subject-consisters images,
Sugrols for Frame
Predictor
Samantic Space
2- 2-H
Worion Prodictor
Imaage Ensoder
Senorated Consishent
Timages
2*NxC
Semwsrtic Space Interpolation
P-N-C
Transformer Block
Diffirsion
PANSC
Figure 3: The pipeline of our method for generating transition videos for obtaining subject-consistent images, described in Sec. 3.1. To effectively model the character's large motions, encode the conditional images into the image semantic space for encoding spatial information and predict the transition embeddings. These predicted embeddings then decoded using the video generation model, with the embeddings serving as control signals in cross-attention to guide the generation of each frame.
3.2 Semantic Motion Predictor for video generation
Illustrated in Fig. 3, the sequence of the generated character-consistent images can be further refined videos by inserting frames between each pair of adjacent images. This can be regarded as a video generation task with known start and end frames conditions. However, empirically observed
5

---

that recent methods, such as SparseCtrl [12] and SEINE [7], cannot join two condition images stably when the difference between the two images large. We argue that this limitation stems from their sole reliance on temporal modules to predict intermediate frames, which may be not enough to handle the large state gap between the image pair. The temporal module operates within pixels on each spatial location independently, therefore, there may be insufficient consideration of spatial information when inferring intermediate frames. This makes it difficult to model the long-distance and physically meaningful motion.
To address this issue, we propose Semantic Motion Predictor, which encodes the image into the image semantic space to capture the spatial information, achieving more accurate motion prediction from a given start frame and an end frame. More specifically, in our Semantic Motion Predictor, we first use a function E' to establish a mapping from the RGB images to vectors in the image semantic space, encoding the spatial information. Instead of directly using linear layers as E,, we utilize a pre-trained CLIP image encoder as E to leverage its zero-shot capabilities for enhancing performance. Using E, the given start frame Fs and end frame F. are compressed image semantic space vectors Ks, Ke-
K2. K. E (F,, Fe).
(4)
Subsequently, in the image semantic space, we train transformer-basee structure predictor to perform predictions of each intermediate frame. The predictor first performs linear interpolation to expand the two frames K₃ and Ke into sequence K1, .... ... where L is the required video length. Then, the sequence K1, K2. .... K is sent into a series of transformer blocks B to predict the transition frames:
P2 P = B (K1, K2...,,KK Ki).
Next, we need to decode these predicted frames in the image semantic space into the final transition video. Inspired by the image prompt methods [55], we position these image semantic embeddings P2.,,... Pr. as control signals, and the video diffusion model as the decoder leverage the generative ability of the video diffusion model. We also insert additional linear layers to project these embeddings into keys and values, involving into cross-attention of U-Net.
Formally, during the diffusion process, for each video frame feature V,, we concatenate the text embeddings T and the predicted image semantic embeddings P.. The cross-attention is computed as follows:
V, = Autesttttention (V,, CT, Pt), concat P.))
(6)
Similar to previous video generation approaches, we optimize our model by calculating the MSE loss between L frames predicted transition video O (O₁, O2. OL) and L frame ground truth G = (G1, GL)::
Loss MSE (G.
(7)
By encoding images into an image semantic space for integrating spatial positional relationships, our Semantic Motion Predictor could better model motion information, enabling the generation of smooth transition videos with large motion. The results and comparisons that showcase the notable improvements can be observed in Fig. I and Fig. 6.
Experiments
4.1
Implementation details
For the generation of subject-consistent images, due to the training-free and pluggable property, we implement our method on both Stable Diffusion XI. [34] and Stable Diffusion 1.5 [37]. To align with the comparison models, we conduct comparisons on the Stable-XL model using the same pre-trained weights. All comparison models utilize 50-step DDIM sampling[43], and the classifier-free free guidance score [18] is consistently set to 5.
For the generation of consistent videos, we implement our method based on the Stable Diffusion 1.5 pertained model and incorporate a pretrained temporal module [13] to enable video generation. All comparison models adopt a 7.5 classifier-free guidance score and 50-step DDIM sampling. Following the previous methods [12, 7], we use the Webvid10M [2] dataset to train our transition video model. For training our transition video model, we utilize the Animate V2 motion module [13] as our initial weights of the temporal module and and fine-tune the module. We then set our learning rate at le-4 and conduct training 100k iterations for Semantic Motion Predictor on 8 A100

---

old fisherman cable-koitt sweater and boots
photo of beaming gardener in overalls with
straw hat
Lovinng out pióvic solo
Roviing boat at davin
Starggazing with telescope
Polog loundry in the basemenn
Unweropping a birthday gift
Folding origami paper Into shopos
If St-hdapter
Photollakkt akoor
StoryPiffhsion
Teat Controlaicllty
Fase Consiotrwny
Attre Coincsion
Telt Contralabilty
TiMe Comsirooyy
Attre Cohssion
IT-Adaptor
X
IT-Adeptor
X
SX
PkotooMaker
PhotolMokor
StorypHffusion
StaryPiffusion
Figure 4: Comparison of consistent image generation with recent methods.
GPUs. To encode the conditional images into the image semantic space, we utilize the OpenCLIP ViT-H-14 [35, 8] pre-trained model. Our Semantic Motion Predictor incorporates 8 transformer layers, with a hidden dimension of 1024 and 12 attention heads.
4.2 Comparisons of consistent image generation
We mainly evaluate our method for generating subject-consistent images by comparing it with the two most recent ID preservation methods, IP-Adapter [55] and Photo Maker [55]. To the performance, we use GPT-4 to generate twenty character prompts and one hundred activity prompts to describe specific activities. The format of our character prompts is "[adjective] [group or profession] [wearing clothing]" and the format of activity prompts is "[action] [location or object]". We combine character prompts with activity prompts to obtain groups of test prompts. For each test case, we use the three comparison methods generate group of images that depict person engaging in different activities to test the model's s consistency. Since IP-Adapter and PhotoMaker require an additional image to control the ID of the generated images, we first generate image of character to serve as the control image. We conduct both qualitative and quantitative comparisons to comprehensively evaluate the performance of these methods consistent image generation.
The Chosen One
ConsiStory
Zero-shot coherent storybook
Storyuiffusion
Figure 5: Additional comparison of our StoryDiffusion with recent storybook generation methods, The Chosen One [1]. ConsiStory [44] and Zero-shot coherent storybook [22].
7

---

Table 1: Quantitative comparisons of consistent image generation. Our StoryDiffusion achieves better text similarity and subject similarity even without any training.
Metric
I--Adptter [55]
Photo Maker [26]
StoryDiffusion (ours)
Imae Similarity
0.6129
0.6541
0.6586
Character Similarity
0.8802
0.8924
0.8950
Qualitative comparisons. The qualitative result is shown in Fig. 4. Our StoryDiffusion generate highly consistent images, whereas other methods, -d-Aapteer and PhotoMaker, may produce images with inconsistent attire diminished text controllability. For the first example, the IP-Adapter method generates an image lost "telescope" with the text prompt "Stargazing with a telescope". PhotoMaker generates images matching the text prompt. but there notable discrepancies in the attire across the three generated images. The third-row images generated by our StoryDiffusion exhibit consistent faces and attire with better text controllability. For the last example "A focused gamer wearing oversized headphones", IP-Adapter loses the "dog" in the second image and the "cards" in the third image. The images generated by PhotoMaker could not maintain the attire. Our Story Difouuiffsioon still generates subject-consisten images, with the same face, and same attire, and conforms to the description in the prompt. To further demonstrate the effectiveness of our method, we compare our method with concurrent or recent storybook generation works, including The Chosen One [1], ConsiStory [44], and Zero-shot Coherent Storybook [22] in Fig. 5. Our approach not only outperforms these methods but also offers greater flexibility and faster inference times. By contrast, The Chosen One requires time-consse consuming LoRA self trainini for each sample; Zero-shot Coherent Storybook necessitates a two-step process, first generating images and then embedding with Iterative Coherent Identity Injection; and ConsiStory involves iterative segmentation mask calculations during diffusion to maintain consistency.
Quantitative comparisons. We evaluate the quantitative comparison and show the results in Tab. 1. We evaluate two metrics, the first one is text-image similarity, which calculates the CLIP Score between the prompts and the corresponding images. The second aspect is character similarity, measured by the CLIP Scores of character images after using the background removal method RMBG- 1.4. Our StoryDiffusion achieves the best performance on both quantitative metrics, which shows our method's 's robustness in maintaining character meanwhile conforming to prompt descriptions.
4.3 Comparisons of transition videos generation
In transition video generation, we conduct comparisons with the two state-of-the-art methods, SparseCtrl [12] and SEINE [7], evaluate our performance. We randomly sample around 1000 videos as the test dataset. We employ the three comparison models to predict the intermediate frames of transition video, given the start and end frames, in order assess their performance.
SEINS
Spacnodrr)
Stary)itfseinn
Figure 6: Comparisons of transition video generation with the recent wate-of-the--a methods.
Qualitative comparisons. We conduct the qualitative comparison of transition video generation and show the results Fig. Our Diffuuufn notably outperforms SEINE [7] and SparseCtrl [12], generating transition videos that are smooth and physically plausible. For the first example, two

---

Table 2: Quantitative comparisons with state-of-the-ar transition video generation models.
Methods
LPIPS-/ (4)
LPIPS-a (4)
CLIPSIM-/ (1)
CLIPSIM-a (1)
FVD (4)
FID(4)
SEINE
0.4332
0.2220
0.9259
0.9736
321
140
SparseCtrl
0.4913
0.1768
0.9032
0,9756
429
Ours
0.3794
0.1635
0.9606
0.9870
271
109
Ablation Study of Sampling Rate
ID Image
Consistent Images with Specified
0.3
0.5
0.3
0.9
(a)
(b)
Figure 7: Ablation study. (a) Evaluations of the impact of different sampling rates in Consistent Self-Attention. (b) We explore the introduction of external control IDs govern the generation of characters. Our StoryDiffusion can generate consistent images that conform to the ID images.
people kissing underwater, the intermediate frames generated by SEINE are corrupted, and there is direct jump to the final frame. SparseCtrl generates results with slightly better continuity, but the intermediate frames still contain corrupted images, with numerous hands appearing. However, our Stffyuiffusion succeeds in generating videos with very smooth motion without corrupted intermediate frames. For the second example, the intermediate frames generated by SEINE have corrupted arms. SparseCtrl, on the other hand, fails to maintain consistency generates consistent videos with excellent continuity. For the last example, the video generate adheres to physical spatial relationships, unlike SEINE and SparseCtrl, which only change the appearance in the transition. More visual examples can be found in the Sec. A.
appearance. Our StoryDiffusion
Quantitative comparisons. Following previous works [12, 58], compare our method with SEINE and SparseCtrl with four quantitative metrics, including LPIPS-f, LPIPS-a, CLIPSIM-f, and CLIPSIM-a, as shown in Tab. LPIPS-f and CLIPSIM- measure the similarities between the first frame and other frames, which reflect the overall continuity of the video. LPIPS-a and CLIPSIM-a measure the average similarities between consecutive frames, which reflect the continuity between frames. FVD and FID are also computed to evaluate generation quality. Our model outperforms the other two methods across all four quantitative metrics. These quantitative experimental results demonstrate the strong performance of our method in generating consistent and seamless transition videos.
4.4 Ablation study
User-specified ID generation. We conduct an ablation study to test the performance of generating consistent images with user-specified ID. Since our Consistent Self-Attention pluggable and training-free, free, we combine Consistent Self-Attention with PhotoMaker, giving images to control

---

Table 3: Ablation study on different random sampling ratios for both random sampling and grid sampling.
Sampling Method
Rand 0.3
Rand 0.5
Rand 0.7
Grid 0.5
Character Similarity
86.39%
88.37%
89.26%
89.29%
CLIP Score
57.14%
57.1 1%
56.96%
56.53%
Table 4: User study on subject-consistent image generation and transition video generation.
Consistent Images Generation
IP-Adapter
PhotoMaker
StoryDiffusion (ours)
User Preference
20.8 %
10.9 %
68.3 %
Transition Video Generation
SEINE
SparseCull
StoryDiffusion (ours)
User Preference
5.9
9.6 %
84.5
the characters for consistent image generation. The results are shown in Fig. 7. With the control of the ID image, our StoryDiffusion can still generate consistent images conformed to the given control ID, which strongly indicates the scalability and plug-and and and-play capability of our method.
Sampling Rate of Consistent Self-Attention. Our Consistent Self-Attention approach samples tokens from other images within a batch, incorporating them into keys and values during self-attention. The original intent of random sampling is to maintain consistency while avoiding excessive structural information, thereby preventing the weakening of text control and maintaining diversity in poses. We conducted an ablation study find the optimal sampling rate (results in Fig. 7). sampling rate of 0.3 does not maintain subject consistency, as seen in the third column of images on the left in Fig. 7, whereas higher rates do. Quantitatively, higher sampling rates can over-correlate images and reduce text control, while lower rates weaken character consistency. We also added a quantitative comparison with the grid sampling method (Tab. 3). While grid sampling better preserves character consistency, it sacrifices text prompt controllability, making our adjustable sampling ratio ideal for balancing both factors. practice, we set the sampling rate to 0.5, balancing consistency with minimal diffusion process impact.
4.5
User study
We conduct a user study with 79 people. Each user is assigned 50 questions to evaluate the effective- ness of our subject-consistent image generation method and transition video generation method. For subject-consistent image generation, we compare with the recent sate-of-the-an methods IP-Adapter and PhotoMaker. In transition video generation, we compare with recent state-of-the-ar methods SparseCtrl and SEINE, For fairness, the order of the results randomized, and users are not informed about which generation model corresponds to each result. As shown in Tab. 4, whether for subject- consistent image generation or transition video generation, our model demonstrates an overwhelming advantage.
Conclusions
In this paper, we propose StoryDiffusion, novel method that can generate consistent images in a training-frree manner for storytelling and transition these consistent images into vidcos. Our Consistent Self-Attention builds connections among multiple images efficiently generate images with consistent faces and clothing. We further propose the Semantic Motion Predictor to transition these images into videos and better narrate the story. We hope that our StoryDiffusiee can inspire future controllable image and video generation endeavors.
Broader impact. Our StoryDiffusion can generate high-quality character-consiste pictures and videos. Certainly, similar to the previous image and video generation methods, our method may encounter some ethical issues. The generated portraits and videos may be used improperly, such as for fabricating false information. We strongly hope that the use of relevant technologies has clear responsibilities and strengthens legal and technical supervision to promote proper use.
10

---

Acknowledgement. This work is supported by the National Science Fund of China (No. 62225604) and the Fundamental Research Funds for the Central Universities (Nankai University, 070-63223049). We also acknowledge "Science and Technology Yongjiang 2035" key technology breakthrough plan project (20247.10)..
References
[1] Omri Avrahami, Amir Hertz, Yacl Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. The chosen one: Consistent characters in text-to-image diffusion models. In ACM SIGGRAPH 2024 conference papers, 2024. 7,8
[2] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 6, 19
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint, 2023. 4
[4] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 11096 2018.
[6] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023. 2,
[7] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. arXiv preprint, 2023. 3, 4, 6, 8, 17, 22
[8] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In CVPR, 2023. 7, 19
[9] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In ICLR, 2023.
[10] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint, 2022. 3
[11] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint, 2023. 4
[12] Yuwei Guo. Ceyuan Yang, Anyi Rao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint, 2023. 3, 4, 6, 8, 9, 17, 22
[13] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. ICLR, 2024. 6,6
[14] Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, et al. Animate-a-story: Storytelling with retrieval-augmentedd video generation. arXiv preprint, 2023. 4
[15] J Ho, T Salimans, A Gritsenko, W Chan, M Norouzi, and DJ Fleet. Video diffusion models. arxiv 2022, arXiv preprint, 2022. 1,3

---

[16] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint, 2022. 3
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 3
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint, 2022. 6
[19] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. In ICCV, 2023. 2
[20] Susung Hong. Junyoung Seo, Heeseong Shin, Sunghwan Hong, and Seungryong Kim. Direct2v: Large language models are frame-level directors for zero-shot text-to-video generation. arXiv preprint, 2023. 4
[21] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-vice synthesis for character animation. arXiv preprint. 2023. 4
[22] Hyeonho Jeong, Gihyun Kwon, and Jong Chul Ye. Zero-shot generation of coherent storybook from plain text story using diffusion models. arXiv preprint, 2023. 7,8
[23] Yumingg Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2performer: Text-driven human video generation. arXiv preprint, 2023. 3
[24] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2023.
[25] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In CVPR, 2023. 3
[26] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding. arXiv preprint, 2023. 1, 3, 8
[27] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
3
[28] Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung. Directed diffusion: Direct control object placement through attention guidance. arXiv preprint, 2023. 3
[29] Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided text-to-video generation using pose-free videos. arXiv preprint, 2023.
[30] Jiafeng Mao and Xueting Wang. Training-free location-aware text-to-image synthesis. arXiv preprint, 2023. 3
[31] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint, 2023. 3
[32] Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In CVPR, 2023. 4
[33] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint, 2022. 3
[34] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint, 2023. 3, 6, 19
12

---

[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In ICMI,, 2021. 7, 19
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text-conditional image generation with clip latents. arXiv preprint, 2022. 3
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 3, 6
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015, 2015. 3
[39] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023. 3
[40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 2022. 3, 4
[41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE TPAMI, 2023. 3
[42] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint, 2022. 3
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint, 2020. 6
[44] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consistent text-to-image generation. ACM Transactions on Graphics (TOG), 2024. 7,
[45] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco. Zsolt Kira, and Mar Gonzalez-Franco. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. arXiv preprint, 2023. 2
[46] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang, Modelscope text-to-video technical report. arXiv preprint, 2023.
[47] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preservinn generation in seconds. arXiv preprint, 2024. 1
[48] Tan Wang, Linjie Li, Kevin Lin, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for referring human dance generation in real world. arXiv preprint, 2023. 4
[49] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint, 2023. 3
[50] Jay Zhangjie WU, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning image diffusion models for text-to-video generation. In ICCV, 2023. 4
[51] Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn motion pattern for few-shot-based video generation. arXiv preprint, 2023. 4
[52] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint, 2022. 3
13

---

[53] Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia- Wei Liu, Chenxu Zhang, Jiashi Feng. and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint, 2023. 4
[54] Mengjiao Yang, Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Pieter Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint, 2023. 3
[55] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint, 2023. 1, 6, 7, 8, 19
[56] David Junhao Zhang, Jay Zhangjic Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-l: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint, 2023.
[57] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-imaae diffusion models. In ICCV, 2023. 3, 17
[58] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[59] Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint, 2023. 4
[60] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint, 2023. 3
[61] Yupeng Zhou, Daquan Zhou, Zuo-Liang Zhu, Yaxing Wang, Qibin Hou, and Jiashi Feng. Maskdiffusion: Boosting text-to-image consistency with conditional mask. arXiv preprint, 2023.
3
14

---

Appendix overview
We provide an overview to clearly display the contents of the appendix, we have also uploaded video folder.
In Sec. A, we provide additional experimental results, including generated video and comics, and additional comparisons of image and video generation.
In Sec. B, we list the limitations and the future work.
In Sec. C, provide the license of dataset and pre-trained model use.
A Additional experimental results
We show video results in the upload files and more comic results in Fig. 8. We present additional comparison results here as a supplement to those discussed in the main text, as shown in Fig. 9 and Fig. 10.
A.1 Video results
We have also uploaded video files in the supplementary materials, which contain videos generated by our method. The files have uploaded include 20 videos, which, based on our Semantic Motion Predictor, turn keyframes generated by Consistent Self-Attention or from the video into videos. The video we uploaded contains two types: longer ones with a lot of movement and the second with shorter, less active ones. Together, they show the model can handle different styles. Due to the upload size limit 100MB, we compressed some of the video files.
A.2 Comic results
We show comic examples in Fig. 8, which narrate two compelling stories, serving complement to Fig. These examples collectively demonstrate the practical value our method in artistic creation.
Consistent images generated by StoryDiffusion "Lessons of the Heart: A Reacher-Student Bond"
"Your Name Life Swap"
Minuba Uniag Vakyo.
Taabi is tunter scheol stadess living is Nars Japa
One day, Suat dextinion veee nagisally
Mary aa consciantios high stbool teecher
Jock studeal tought Mary.
allignso, devenice
Jack oftec bucumcee engrossed In stadying usail late the night.
body.
beaetf oa boy.
Dras
other's
outsnading
realing bosss,
Over cultivated geamistves of anadents gradually sgine.
Years later, succesffi entrggemnns, returns to visit the ciderly Mary.
renlallone
refloollng the magical
(a)
(b)
Figure 8: More comics results generated by our StoryDiffusion.
15

---

A scholar in tweed jacket
A focused gamer wearing oversized headphones
Making a wish
Photographing under surnec
Teosing a Gisbee in the
Spinning a globe
Playing with a dog Shafflng a dock of cards
on dtar
rack
IP-Adipter
PhotoMaker
SteryDiffusion
Face Camistency
Cohcston
Cantrdisbilit
Consituncy
Cabesioe
IP-Adapier
IP-Adapter
PhotoMaker
PhotoMaker
StoryDiffusonn
StoryDiffesion
A cartoon rabbit with a black shirt
A yellow dog wearing black collar
rummages through a
swiftly dodging the playful butterflies
levens itself over small fence
on porch. head
up near
pile intirn leave
beween pawlic
firepaace
WomTN
Ph
Story Difiasionn
Test Cantroltabilit Charneter Consisteacy Attire Colesisa
Test Controlibility Character Consitency Atrre Cabesion
IP-Adapter
V
IP-Adapter
X
PhotoMaker
X
PhotoMaker
SteryDiffasion
SturyDiffusion
Figure 9: Additional visual comparison of consistent image generation.
A.3 Consistent image generation
We showcase additional results of consistent character image generation in Fig. The results reveal better text consistency of Consistent Self Allf-Attention than Addpeere For example, in the case of 'A scholar in tweed jacket,' IP-Adapter failed generate a star. In 'A cartoon rabbit with a black shirt,' the action of flipping through the fence is not correctly generated in the result generated by the IP-Adapter. In 'A yellow dog wearing a black collar,' it fails to generate woman and has incorrect positional relationships. PhotoMaker does not diminish text consistency but is unable to preserve the clothing of generated characters, as seen with non-human characters, as illustrated by the latter two examples. Unlike them, our Consistent Self-Attention generates results with high text consistency and enhanced character coherence. The experimental results further attest to the efficacy of our Consistent Self-Attention
first two examples. Moreover, its performance drops
16

---

A.4 Transition video generation
We present an additional comparison with SparseCtrl [12] and SEINE [7] transition video gen- eration. As illustrated in Fig. 10, our Semantic Motion Predictor can produce more coherent and smoother intermediate frames compared SparseCurl and SEINE, thus further demonstrating the advantages of Semantic Motion Predictor.
Start Frame
Generated Frame
End Frame
SEINS EN
StoryDiffusion Sparse Crr
SEINE
StoryDifiusion SparseCtrl
SEINE
StoryDiffusion CmurseCin
Figure 10: Additional visual comparison of transition video generation. The red box represents the frames input into the model.
A.5 Consistent images generation with ControlNet
Given that our Consistent Self-Autention is training-free and pluggable, we further explore integration with ControlNet [57] to introduce pose control in the generation of subject-consisten images. The

---

Pose Control Map
Results with Consistent Self-Attention
Results without Consistent Self-Attention
Results with Consistent Self-Attention
Results without Consistent Self-Attention
Results with Consistent Self-Attention
Results without Consistent Self-Attention
Figure 11: Generation results of our Consistent Self-Attention combined with Con
18

---

results of combining our Consistent Self-Attention with ControlNet are displayed in Fig. 11. Our approach is also capable of generating subject-consisten images under the guidance of ControlNet.
A.6 Experiments Plug-and-Play capability
We have additionally implemented our StoryDiffusion on SD 1.5 and SD 2.1, and put the result compared SDXL into Fig. 12. Our method maintains good performance when integrated into different models, which demonstrates the "plug-and and-play" characteristics.
StoryDiffusion based SD 1.5
StoryDiffusion based SD 2.1
based SDXL
Figure 12: The pluggable capability of our StoryDiffusion several popular diffusion models, Consistent Self-Attention works well across multiple models.
B Limitations and future work
The first limitation arises in our subject-consistent image generation. Similar to current state-of-the- art methods [55], there may exist inconsistencies in some minor clothing details, such ties. In this case, our Consistent Self-Attention may require more detailed prompts to maintain consistency across images. The second limitation is in our transition video generation. Although one can utilize StoryDiffusiin to generate longer videos by sequentially connecting the generated consistent images, it becomes challenging to stitch two images when there is significant difference between them. Consequently, our method is not yet perfect for generating very long videos due the absence of global information exchange. We will further explore long video generation in future work.
C Dataset and model licenses
Webvid-10m 10M: Webvid-10M, [2] is a large-scale video dataset featuring 10 million video clips with associated textual descriptions, designed for training and evaluating machine learning models video understanding and generation tasks.
URL: www www.www.w.............o.......ww robots.bot..oo...... ac. "...uk/vvg' http://gggeeerrrc/rroooeeee---ee
Stable XL [34] is diffusion-basee text-to-image generative model provided by Stability AI, which is capable of generating high-quality images based on the given text.
Stable XL:
Licenses: CreativeML Open RAIL++MM License. URL: stability
image
OpenCLIP: OpenCLIP [8] is an open source implementation of OpenAl's CLIP [35] (Contrastive
Language-Image Pre-training).
URL: github. httpa//..con..oommmaiii//o/////
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
We confirm the main claims reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in Sec. B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
We did not include theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have described the model details in the implementation details in Sec. 4.1 and carefully described the experimental evaluation the experimental chapter. The information we provide is sufficient and detailed for replication purposes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have made a detailed description of the implementation details ensure that they are repeatable and use publicly available datasets. However, we intend to make our code publicly available following the paper's acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Saweee/oooouuumisiiiioo policy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctppblpcccccccccccccccccccccccc geiiii/ggge/////ieeeeeeiiicissee /ooe//oooeeeeseeeeeeeeeeee wwwsssssiilloooooeoooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetalls
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have carried out detailed narration in the implementation details in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Based involved in this work is high, with results that are replicable and stable, rather than simply reporting the highest outcomes. Additionally, previous related work [7, 12] has also not reported error bars. We thus do not run the statistical significance test.
experimental experience, the reproducibility of the experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state this detailed information of computer resources in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttps://neurr nwwrips.....p// cc/ cc/public// cehiceellllleeees
Answer: [Yes]
Justification: We confirm that the research involved in the article complies with the NeurIPS Code of Ethics in all respects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper have conducted discussion of broader impacts Sec. 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: At present, there is no relevant description we will set up safeguards when we release the model of StoryDiffusion.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We state dataset and model license in Sec. C.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "M4oFywGc3P", "reviewer": "Reviewer_SSyn", "summary": "This paper proposes story diffusion which is a video generation model encapsulating two contributions:\n* First, the authors show how to generate a sequence of image that are self-consistent (e.g. consistent appearance for characters, consistent attire, etc) but obey different prompts.  This is done using ordinary image diffusion models by generating these images in a batch but having tokens attend to randomly selected tokens in othe rimages in the batch.  The authors show that their approach does not require any additional training.\n* Second, these images (which might be generated following a “story” or sequence of text prompts) can be filled in with higher frequency video frames.  The approach is to train a Stable Diffusion model with an AnimateDiff temporal module to use CLIP features of these keyframes (as well as linearly interpolated ones) as control signals.\n\nThe authors show in evaluations that their approach beats previous papers like IP-Adapter and Photo Maker on character consistency and outperforms SEINE and SparseCtrl on video generation.", "questions": "* There is some detail provided for the semantic motion predictor in the Appendix that I recommend moving to the main paper (it reads somewhat too abstractly without this detail).\n* What does plug-and-play / hot-pluggable mean and how is it different from training-free?", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I very much enjoyed reading this paper.  It has strong and compelling results and attacks a timely/important problem.  The consistent generation solution is particularly simple conceptually and does not require any training and the second part of the paper builds on top of AnimateDiff2 which content creators should be able to use very easily.", "weaknesses": "I do not see any weaknesses that would cause me to vote against acceptance.  That being said, the sampling ablation is not very serious (just a handful of cherry picked examples that don’t clearly show any differences relative to the parameter being ablated).   More generally it is not so clear why the authors prefer to sample tokens rather than use all of them.  Is this for computational reasons?  If I am right, the proposed attention should scale linearly in the batch size (so it is unclear why this would be problematic and necessitate sampling).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we are extremely grateful that the reviewer recognizes our efforts and contributions. At the same time, we also appreciate the reviewer's constructive feedback and would like to respond specifically to the following aspects.\n\n**Q1: Improve ablation experiments of Random Sampling.**\n\nSpecial thanks for your insightful feedback. We also agree that a more systematic quantitative evaluation of the sampling strategy would make the paper better.  Thus, we have run some quantitative evaluations as shown below:\n\n| Random Sample Rate  |  0.3    |  0.5    | 0.7    |\n|---------------------|--------|--------|--------|\n| Character Similarly | 86.39% | 88.37% | **89.26%** |\n| CLIP Score          | **57.14%** | 57.11% | 56.96% |\n\n**Insights & Analysis:** The initial reason for using random sampling is to avoid reinforcing fixed characters' poses across different images, which is probably reducing the controllability of the text prompts over the generated images, visualized in Figure 3 in the rebuttal PDF. This trend is also reflected in the newly added experiment shown in the above table. The introduction of the sampling ratio is to achieve the best trade-off between character consistency and text controllability. Besides, a lower sampling ratio indeed effectively reduces the computational cost of self-attention, reducing GPU memory consumption, but this is not the initial purpose of doing so.  There are also many engineering implementations to address memory consumption, such as the tiling method demonstrated in supplementary materials, or saving sampled tokens for reuse when inferring other images.\n\n\n**Q2: Transferring semantic motion predictor details from the appendix to the main paper.**\n\nWe are very thankful for the reviewer's suggestions. We will adjust the structure of the article by moving the implementation details section to the first subsection of the experiments chapter to improve the readability of the paper in the revision.\n\n\n**Q3: Meaning of the word \"plug-and-play\".**\n\nThanks to the reviewer for pointing out this issue. The difference between plug-and-play and training-free is that plug-and-play implies compatibility with multiple different models, whereas training-free merely emphasizes that no training is required, without indicating whether the method can be extended.\nWe state in the paper that our consistent self-attention (SA) is plug-and-play because our method is implemented based on the properties of self-attention. \n\nIn our experiments with consistent self-attention, we found that consistent SA works effectively across multiple models, such as Stable Diffusion 1.5, Stable Diffusion 2.1, and Stable Diffusion XL.  Therefore, we believe that SD is plug-and-play. We have presented the experimental results in Figure 1 of the upload  Rebuttal PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQWc19n384", "reviewer": "Reviewer_S3nw", "summary": "Aiming to generate a story-based images or videos, this paper introduces StoryDiffusion, and proposes to use the following methods:\n1. Consistent self-attention, which is a training-free way that modifies existing self-attention to maintain the between frames.\n2. Semantic motion predictor, which is additional module that predicts accurate motions from generating video.", "questions": "1. I'm curious if the consistent self-attention is robust across random seeds, since it picks tokens randomly. Again, it would be great to see comparisons with other token schemes.\n2. The Semantic Motion Predictor seems to require video datasets. What is the main motivation for not fully fine-tuning the model with the datasets but only the module?", "limitations": "The authors have addressed the limitations but they haven't discussed the potential social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper tackles maintaining consistency and similarity between batches, which is necessary when it comes to application to real-world applications.\n- It maintains consistency in a simple but effective way, with consistent self-attention working without training.", "weaknesses": "- The random selection of tokens in consistent self-attention is not well-explained. What are advantages over other baselines despite the randomness?\n\n- There are some attention mechanisms that achieve consistency across frames within batches. In particular, some works (e.g., [A], [B], [C], and [D]) heuristically pick frames to attend to (e.g., the first frame, the frame before or after a given frame, or frames in a rotational manner), possibly due to the complexity. It seems that this paper lacks such discussion or comparison.\n\n- The novelty seems questionable. It appears to be a post-processing technique or trick for Stable Diffusion that increases consistency between frames.\n\n- Typo: L129 has a \".\" in the middle of the sentence.\n\n[A] Wu, Jay Zhangjie, et al. \"Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[B] Khachatryan, Levon, et al. \"Text2video-zero: Text-to-image diffusion models are zero-shot video generators.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[C] Hong, Susung, et al. \"DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation.\" arXiv preprint arXiv:2305.14330 (2023).\n\n[D] Geyer, Michal, et al. \"Tokenflow: Consistent diffusion features for consistent video editing.\" arXiv preprint arXiv:2307.10373 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we sincerely thank the reviewer for the thorough review and constructive feedback. We also appreciate the positive rating provided by the reviewer. After carefully reading the comments, we hope to address the reviewer's concerns in the following:\n\n**Q1:  \"(whether) consistent self-attention is robust across random seeds, and  comparisons with other token schemes.\"**\n\nWe are very thankful for the feedback and hope to address the reviewer's concern from three aspects: intuition and insights, the robustness of the method, and the quantitative evaluation of the method as below:\n\n(1) The insight behind random sampling is to avoid reinforcing fixed characters' poses across different images, which is probably reducing the controllability of the text prompts over the generated images. This trend is also reflected in the newly added experiment shown in the table in part (2). The introduction of the sampling ratio is to achieve the best trade-off between character consistency and text controllability. Besides, a lower sampling ratio indeed effectively reduces the computational cost of self-attention, reducing GPU memory consumption.\n\n(2) To further address the reviewer's concerns, we add a quantitative evaluation for the impact of the random sampling ratio and the comparison with the Image Grid method. As shown in the table, while the non-random GridSampling method indeed results in better character consistency, the controllability of the text prompt is reduced. More visualization results are also shown in Figure 3 in the uploaded file in the rebuttal. In summary, the introduction of the sampling ratio is to achieve the best trade-off between character consistency and text controllability.\n\n|                     | RandSample (Rate 0.3) | RandSample  (Rate 0.5) | RandSample  (Rate 0.7) | GridSample (Rate 0.5) |\n|---------------------|-----------------------|-------------------------|-------------------------|-----------------------|\n| Character Similarly | 86.39%                | 88.37%                  | 89.26%                  | 89.29%                |\n| CLIP Score          | 57.14%                | 57.11%                  | 56.96%                  | 56.53%                |\n\n\n(3) All reported results in the paper are average values over three different random seeds. We will add more details in the revision to make this clearer. Under this setting, our proposed method still achieves consistently higher performance than the selected baseline methods.\n\n**Q2: \"Novelty, ... (Consistent Self-Attention) appears to be a post-processing technique\"**\n\nWe respectively remind the reviewer that our contributions are two-fold: (1) a new block to generate consistent sequence of images and (2) a novel semantic motion predictor to convert the sequence of images into dynamic videos with infinite length. The novelty of each point is explained below for your reference:\n\n(1) We hope to clarify that Consistent Self-Attention is not a post-processing trick. It is used to modify the UNet architecture and operate together with the generation process. Even though there are some self-attention mechanisms in video editing tasks, the purpose and focus of our consistent self-attention are different as mentioned in Q3.\n\n(2) We propose a new framework for long video generation with high dynamics. We modify an image model with CSA in a training-free manner and the semantic motion predictor transforms the sequence of images into videos with meaningful and dynamic motions. Compared to traditional text-to-video approaches that treat the entire process as a black box,  our two-stage method,  offers better controllability. Because it allows dynamic adjustment of keyframes to modify video results, aligning with the demands of film production.\n\nBased on the above justification, we hope the reviewer could re-assess the contributions of the proposed framework. \n\n**Q3: \"Discussion with other attention mechanisms\"**\n\nWe thank the reviewer for raising those extra works. Some of the papers have been discussed in the main paper (E.g. Tune-A-Video). The major differences between StoryDiffusion and the rest of the works are summarized below and will be added in the revision:\nThe attention mechanisms used in the mentioned papers are designed for highly similar consecutive video frames, and adopt specific methods for selecting keys and values in the attention mechanism to enhance coherence of video: \"Tune a video\"  uses the first frame and the previous frame as references for selection and requires additional fine-tuning of the \"q linear\"  to align queries; \"Text2video-zero\" uses the first frame as keys and values; \"DirecT2V\" uses a temporal rotational selection of keys and values; Tokenflow uses a frame set with fixed frame intervals as keys and values.  \n\nDifferently, our method targets to generate consistent characters with large dynamics that allow strong text controllability. The generated images could vary from each other to a large extent without direct motion continuity between adjacent images. In this case, these methods for selecting keys and values are not applicable. Differently, consistent self-attention emphasizes the mutual interaction between images within a batch to enhance consistency, rather than simply selecting some frames as key and value for unidirectional interaction.\n\n**Q4: What is the main motivation for not fully fine-tuning the model with the datasets but only the module?**\n\nThe main motivation for freezing the spatially related model parameters is to save the computational cost. As we have the first frame and the last frame as prior knowledge, this practice significantly saves the training cost with acceptable video generation performance.\n\n**Q5: haven't discussed the potential social impact.**\n\nWe kindly remind the reviewer that the potential social impact has been included in the Broader Impact paragraph in Sec. 5. (Line 275-279) We will try to make it clearer in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3T819tfED", "reviewer": "Reviewer_fV1e", "summary": "The paper proposes a diffusion-based generative model designed to create subject-consistent images and videos that correspond to given story texts. The paper presents both qualitative and quantitative results showing enhanced subject consistency in generated images and videos compared to existing work.", "questions": "- According to Algorithm 1, a tile size W is used for solving the OOM issue. I’m curious whether reducing the sampling rate while maintaining the tile size across the entire batch can be considered an alternative approach? Have the authors experimented with such a configuration? If so, why the current implementation with a specific tile size was preferred?\n- In L217, how is the CLIP score for character similarity computed? Are any segmentation or bounding boxes applied for computing the Image-to-Image CLIP score?\n- According to Appendix B, the paper implements upon stable diffusion, which uses VAE for encoding the images into the latents of shape BxHxWxC. In this context, how does the paper define an image token?\n- In Equation 5, $l$ -> $L$.", "limitations": "The limitation section has adequately stated most of the limitations. However, the time efficiency of video generation could be discussed more. Generating transitions independently for every two frames could significantly increase generation time. How does this approach compare to other baseline methods like SparseCtrl and SEINE in terms of overall video generation time?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a training-free and pluggable consistent self-attention module that enhances the consistency of characters in a sequence of generated images.\n- To effectively model the large movements of characters, the paper proposes a semantic Motion Predictor for predicting the transition between two images in semantic space.\n- Experimental results show that the proposed model surpasses existing methods in generating subject-consistent images and videos.", "weaknesses": "- The Method section lacks clarity:\n    * The process of splitting the story text into several prompts as mentioned in Figure 2 is not described in the main text.\n    * In Section 4.1, the formation of story text from character prompts and activity prompts also lacks clarity.\n    - In L182 and Equation 6, it is unclear whether the text embedding T represent the text associated with frame $F_s$ or $F_e$ or something else.\n- According to sec 3.2, it seems like the proposed model independently generates transition videos for every two consecutive frames, which could be time-intensive and potentially result in less smooth transitions.\n- The main paper should include the necessary/essential training details instead of leaving all implementation details in the appendix.\n- In Figure 6, for the ablation study on sampling rates, it is hard to tell the effect of different sampling rates. It seems for all sampling rates, the model failed to maintain consistency? In the second and third columns, the dog no longer wears the blue vest. Also, according to Algorithm 1, in the actual implementation, the module uses a title size W when sampling the other tokens. However, the value of title size is not specified or discussed. The impact of varying the tile size \"W\" on consistency is also not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to thank the reviewer for acknowledging our efforts and contributions. Additionally, we deeply appreciate the reviewer's constructive feedback and would like to reply to the question raised by the reviewer in detail.\n\n**Q1: \"The Method section lacks clarity\"**\n\nWe are very thankful to the reviewer for pointing out these detailed issues. We will revise our paper to address the clarity issues pointed out by the reviewer to enhance clarity:\n\na. We will add the submodule to Figure 2 to illustrate how we use GPT-4 to split the story text into several prompts.\n\nb. We will add the format of prompts to Section 4.1 for clarity. The format of our character prompts is \"[adjective] [group or profession] [wearing clothing]\" and the format of activity prompts is “[action] [location or object]”.\n\nc. We will add this statement to clarify that T in Line 182 and Equation 6 refer to the prompt corresponding to the previous image.\n\n\n**Q2: \"could be time-intensive and potentially result in less smooth transitions.\"**\n\n We would like to address the reviewer's concerns through the following aspects:\n\n(1) For the time consumed, we compared our method with comparable methods of the average generation time for a 16-frame 512x512 video.  Our StoryDiffusion's average inference time is comparable to SEINE and significantly better than SparseCtrl, demonstrating that our method is not time-consuming compared to previous approaches. \n\n|                | SparseCtl | SEINE | StoryDiffusion |\n|:--------------:|:---------:|:-----:|:--------------:|\n| Inference Time | 94s       | 20s   | 27s            |\n\n(2) For the smooth transition, although our method generates transitions for every two consecutive keyframes, as demonstrated in the videos uploaded in the supplementary materials, our method can produce smooth long videos. The reason may be that our method learns from a large number of video clips, making each video clip coherent on its own. Additionally, we acknowledge that long video generation is a very challenging problem in the \"Limitations\" section of the paper, which may not be completely resolved within a single work. Since the video files in our supplementary materials have already demonstrated effectiveness, we will further explore how to establish connections between multiple short videos to make the transition smoother in our future work.\n\n**Q3:  \"the dog no longer wears the blue vest.\"**\n\nWe sincerely thank the reviewer for pointing out the ambiguity. The \"blue vest\" appears only in the first image just because the corresponding prompt contains \"blue vest,\" while the other prompts do not. We will add the prompts to Figure 6 (a) of the main paper to prevent misunderstanding.\n\n**Q4: \"Why is the current implementation with a specific tile size preferred?\"**\n\nWe sincerely thank the reviewer for the feedback on the tile size. Overall, we use a fixed tile size for stability and robustness. To better illustrate this, we conducted an additional experiment, which we present below. Increasing the tile size and reducing the sampling rate within certain limits indeed enhances consistency. However, beyond a certain point, further increases result in decreased character consistency.  Additionally, we may not need such a large number of images at once in most cases. Therefore, we chose to use a slightly smaller tile size and a larger sampling rate after considering multiple factors. We appreciate the feedback and will make revisions in the method section to better help readers understand this point.\n\n|          | Sample:0.3, Tile:3 | Sample:0.5,Tile:3 | Sample:0.7, Tile:3 | Sample:0.2,Tile:8 | Sample:0.1, Tile:16 |\n|:-----------:|:----------:|:----------:|:----------:|:-----------:|:-------------:|\n| Character Similarity | 86.39      | 88.37             | 89.26         | 89.50           | 86.35          |\n\n**Q5: \"Are any segmentation or bounding boxes applied?\"** \n\nYes, in the process of calculating character similarity, we first use the SOTA background removal method RMBG-1.4 to eliminate the background. Then, we use only the images of the foreground characters for calculating character similarity.  We sincerely thank the reviewer for pointing this out and will add the relevant details to our paper for clarity.\n\n**Q6: \"How does the paper define an image token?\"**\n\n We greatly appreciate the reviewer for pointing out this issue.  In the attention modules of the UNet, the 3-dimensional HxWxC latent feature in VAE space is flattened into a 2-dimensional sequence of HWxC. We refer to the 1xC vector in this HxW sequence as an image token.  We will add the definition of \"image token\" at the beginning of the Methods section to avoid any confusion", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6NaObuAJpm", "reviewer": "Reviewer_uTuy", "summary": "This paper presents StoryDiffusion, a framework based on diffusion models designed to generate consistent images or videos. StoryDiffusion comprises two key components. The first is a novel self-attention mechanism, named Consistent Self-Attention, which enhances the consistency between generated images and can augment pre-trained diffusion-based text-to-image models without additional training. The second component is a Semantic Motion Predictor module, which predicts motion conditions in semantic space to generate videos with smooth transitions and stable subjects. The combination of these two components in StoryDiffusion offers a lightweight, training-free solution to the problem of content consistency in generated images and videos, while maintaining high controllability over the generated content.", "questions": "The motivation and method introduction are very clear. Here are some experiment-related questions and suggestions corresponding to the weaknesses that need the authors' responses:\n1. Could you provide a qualitative comparison with newer story generation tasks such as [1], [2], [3]? These works also rely solely on text prompts rather than input images like IP-Adapter, making them more relevant to your method. Additionally, since [2] also modifies self-attention to maintain content consistency, please explain the differences between Consistent Self-Attention and their approach.\n2. Given that the paper repeatedly emphasizes that the proposed modules are plug-and-play, there should be relevant experiments in the main text to substantiate this claim.\n3. It would be beneficial to have a clear comparison demonstrating the superiority of StoryDiffusion in terms of inference time and space overhead.\n4. For objective metrics, consider including FVD. For the user study, increasing the number of participants would help avoid the collected samples having a cluster bias.", "limitations": "The authors have addressed limitations and societal impacts well. They acknowledge ethical concerns, noting that StoryDiffusion, like other generative methods, could be misused to create false information. They call for clear responsibilities and stronger legal and technical supervision to ensure proper use.\n\nIn the appendix, they identify two main limitations: minor inconsistencies in subject details, such as clothing, which may require detailed prompts, and challenges in generating very long videos due to difficulties in stitching images with significant differences. Future work will explore these areas further.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The article is logically structured and easy to understand.\n2. The authors provide clear code examples, making the proposed methods easy to reproduce.\n3. The Consistent Self-Attention mechanism can augment pre-trained diffusion-based models without requiring additional training. \n4. The Semantic Motion Predictor module extends the method to address video consistency issues, ensuring smooth transitions and stable subjects in generated videos.\n5. The paper includes extensive experimental results demonstrating the effectiveness of StoryDiffusion.\n6. The framework offers a lightweight and efficient approach to generating consistent visual content.", "weaknesses": "1. Although the authors introduce methods like IP-Adapter in the related work and introduction sections, the paper's focus should be on story generation. Therefore, it lacks comparisons with similar story generation works, such as [1], [2], [3].\n2. The paper repeatedly emphasizes that the proposed modules are plug-and-play. However, the main text does not provide corresponding ablation studies to offer more substantial examples.\n3. The paper claims that StoryDiffusion is a lightweight method with minimal data and computational cost, but it lacks a detailed analysis of time and space overhead.\n4. The quantitative metrics are limited, mostly based on the CLIP score. Additionally, the user study involves a small number of participants and lacks detailed explanations of the setup.\n\n[1] Avrahami, Omri, et al. \"The Chosen One: Consistent Characters in Text-to-Image Diffusion Models.\" arXiv preprint arXiv:2311.10093 (2023).\n\n[2] Tewel, Yoad, et al. \"Training-Free Consistent Text-to-Image Generation.\" arXiv preprint arXiv:2402.03286 (2024).\n\n[3] Jeong, Hyeonho, Gihyun Kwon, and Jong Chul Ye. \"Zero-shot generation of coherent storybook from plain text story using diffusion models.\" arXiv preprint arXiv:2302.03900 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to express our sincere gratitude to the reviewer for the thorough review and constructive feedback. \nFollowing the suggestion of the reviewer, we provide detailed responses in the hope of addressing the reviewer's concerns.\n\n**Q1: Additional Comparison Methods.**\n\n We are thankful to the reviewer for the feedback. We have run extra  experiments and the visualization results are included in Figure 2 of the uploaded PDF in the rebuttal. We will cite these three papers and include comparisons with them in the revised paper.  Compared with them, our method performs better and is more flexible with less inference time as it requires only a single generation pass. In contrast, \"The Chosen One\" requires time-consuming LoRA self-training for each example; \"Zero-shot Generation of Coherent Storybook\" requires first generating the images and then performing Iterative Coherent Identity Injection embedding;  \"Consistory\" iteratively calculates the segmentation mask during the diffusion process to improve consistency and can also be considered concurrent work with ours due to the close timing.\n\n**Q2: Experiments about the \"plug-and-play\" characteristics.**\n\nWe are very appreciative of the reviewer's feedback. We have additionally implemented our StoryDiffusion on SD1.5 and SD2.1, and put the result compared SDXL into Figure 1  of the rebuttal PDF.   Our method maintains good performance  when integrated into different models, which demonstrates the  \"plug-and-play\" characteristics.\n\n**Q3: Efficient analysis.**\n\nBased on the reviewer's suggestions, we performed an efficient analysis of consistent image generation and transition video generation:\n \n(1) Our consistent self-attention is training-free, which means we are more efficient than the models needed to train at large datasets (IP-Adaper or PhotoMaker) or tune models at inference time (The chosen one).\nWe have listed the average time required to generate a 1024x1024 image, training parameters, as well as the amount of training data used below. Our inference time is similar to that of the comparison methods, but it does not require extensive additional pre-training, which demonstrates that our method is efficient.\n\n|                     | IP-adapter | PhotoMaker | StoryDiffusion |\n|:-------------------:|:----------:|:----------:|:--------------:|\n| Inference Time      | 6s         | 5s         | 8s             |\n| Trainable Parameter | 22M        | 110.3M     | 0              |\n| Finetune Dataset    | 10M        | 0.3M       | 0              |\n\n(2) For video, our inference time and parameter count are comparable to other methods; Meanwhile, we achieve significantly better performance.\n\n|                 | SEINE | SparseCtrl | StoryDiffusion |\n|:---------------:|:-----:|:----------:|:--------------:|\n| Inference Time | 20s   | 94s        | 27s            |\n| Parameter       | 0.9B  | 1.9B       | 2.0B           |\n\n**Q4: “For objective metrics, consider including FVD. For the user study, increasing the number of participants”**\n\n(1) We sincerely thank the reviewer for the feedback. We have expanded the quantitative experiments accordingly following the reviewer's suggestions. We first have supplemented the FVD comparison results:\n\n| SparseCtrl | SEINE  | StoryDiffusion |\n|:-----------:|:-----------:|:--------------:|\n| 429         | 321         | 271            |\n\n(2) Additionally, to further address the reviewer's concerns, we have also added the FID results calculated on the frames:\n\n| SparseCtrl | SEINE  | StoryDiffusion |\n|:-----------:|:-----------:|:--------------:|\n| 181         | 140         | 109            |\n\n(3) Furthermore, for the user study, following the reviewer's suggestions, we have  increased the number of participants and organized a study with 79 participants. Our survey also included participants from a diverse range of professions and knowledge backgrounds to avoid cluster bias.\n\n| SparseCtrl | SEINE      | StoryDiffusion |\n|:-----------:|:-----------:|:--------------:|\n| 5.9%       | 9.6%       | 84.5%          |\n\n| IP-adapter | PhotoMaker | StoryDiffusion |\n|:-----------:|:-----------:|:--------------:|\n| 20.8%      | 10.9%      | 68.3%          |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to sincerely thank all the reviewers for their thorough review, and we are deeply grateful for their recognition of our efforts in this work.  After carefully reading their review comments, We are very pleased to reply to the reviewer regarding the issues they are concerned about. 

We also uploaded a PDF file containing three figures to visually support our rebuttal explanations,  as mentioned in the corresponding responses.

Meta-review: The paper received one Strong Accept, two Accepts, and one Weak Accept. The reviewers appreciated the problem setting as well as the technical approach (in particular the proposed Consistent Self-Attention). While there were concerns regarding the comparisons and clarity about the method, these were addressed in the rebuttal stage and all reviewers recommend accept with varying degrees. I agree with the reviewers and also recommend Accept.

Based on the interest in the problem and the potential usefulness of the approach, I recommend this paper for a Spotlight. In the final paper, it would be good to include all the discussion from this rebuttal stage and having a more extensive qualitative or even quantitative comparison against the methods mentioned by Reviewer uTuy would be appreciated.
Decision: Accept (spotlight)