Title: Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion Models
Venue: NeurIPS 2023
Accepted: no
Keywords: Story Generation

Diffusion Model
Abstract: Benefiting from the impressive diffusion models, conditional generative models have exhibited exceptional capabilities in various generation tasks, for example, image or short video generation based on text description. In this work, we focus on the task of generating a series of coherent images based on a given storyline, denoted as open-ended visual storytelling. We make the following three contributions: (i) to fulfill the task of visual storytelling, we introduce two modules into a pre-trained stable diffusion model, and construct an auto-regressive image generator, termed as StoryGen, that enables to generate the current frame by conditioning on a text prompt and preceding frame; (ii) to train our proposed model, we collect paired image and text samples by sourcing from various online sources, such as videos, E-books, and establish a data processing pipeline for constructing a diverse dataset, named StorySalon, with a far larger vocabulary than existing animation-specific datasets; (iii) we adopt a three-stage curriculum training strategy, that enables style transfer, visual context conditioning, and human feedback alignment, respectively. Quantitative experiments and human evaluation have validated the superiority of our proposed model, in terms of image quality, style consistency, content consistency, and visual-language alignment. We will make the code, model, and dataset publicly available to the research community.

1

Abstract

Benefiting from the impressive diffusion models, conditional generative models have exhibited exceptional capabilities in various generation tasks, for example, image or short video generation based on text description. In this work, we focus on the task of generating a series of coherent images based on a given storyline, denoted as open-ended visual storytelling . We make the following three contributions: (i) to fulfill the task of visual storytelling, we introduce two modules into a pre-trained stable diffusion model, and construct an auto-regressive image generator, termed as StoryGen , that enables to generate the current frame by conditioning on a text prompt and preceding frame; (ii) to train our proposed model, we collect paired image and text samples by sourcing from various online sources, such as videos, E-books, and establish a data processing pipeline for constructing a diverse dataset, named StorySalon , with a far larger vocabulary than existing animation-specific datasets; (iii) we adopt a three-stage curriculum training strategy, that enables style transfer, visual context conditioning, and human feedback alignment, respectively. Quantitative experiments and human evaluation have validated the superiority of our proposed model, in terms of image quality, style consistency, content consistency, and visual-language alignment. We will make the code, model, and dataset publicly available to the research community.

1 Introduction

"Mirror, mirror, here I stand! Who is the fairest in the land?"

This paper explores an exciting yet challenging task of visual storytelling , with the goal of training a model that can effectively capture the relationship between visual elements in images and their corresponding language descriptions, to generate a sequence of images that tell a visual coherent story. The ultimate goal is to generate a sequence of images that tell a coherent story, provided in the form of natural language. The outcome of this task has significant potential for education, providing children with an engaging and interactive way to learn complex visual concepts, and develop imagination, creativity, emotional intelligence, and language skills, as evidenced by the research in psychology [4, 41].

In the recent literature, there has been significant progress in image generation, particularly with 30 the guidance of text, such as stable diffusion [35], DALL·E [33] and Imagen [9]. However, these 31 models are not sufficient for visual storytelling for two reasons: (i) existing models generate images 32 independently without considering the context of previous frames or the overall narrative, resulting 33 in visual inconsistencies and a lack of coherence in the visual story; (ii) generating images by only 34 conditioning on text can lead to ambiguities or require unnecessarily long descriptions, particularly 35

StoryGen

(1) Once upon a time,

there was a yellow dog,

with ears that flopped and

a tail that wagged…

(2) The yellow dog loved

to run and play, jumping

over fences and chasing

after butterflies…

(3) Curious, he followed

his nose and it led him to a

hidden cave in the

woods…

(4) Inside the cave, the

yellow dog found a

treasure chest filled with

shiny gold coins and

sparkling jewels...

(5) So, the yellow dog

took a few coins and

jewels and ran to the

nearby town…

(6) The mayor rewarded

the yellow dog with a

medal of honor and a

lifetime supply of bones…

(7) From that day on, the

yellow dog became a hero

in the town. People would

come to visit him…

ChatGPT

: OK. Here

is a story: Once upon

a time, there was a

{yellow dog}…

User

: Hey ChatGPT,

please give me a story

about a {yellow dog}.

when dealing with subtle differences, for example, distinguishing between animals from the same 36 category but different breeds. To address the limitations, we introduce a novel auto-regressive 37 architecture, termed as StoryGen , that builds upon pre-trained stable diffusion model, with two extra 38 modules serving for style transfer and visual context conditioning. At inference time, StoryGen takes 39 the preceding frames and text prompts as conditions to synthesize the current frame, i.e. , iteratively 40 creating visual sequences that are not only aligning to language description, but also coherent. 41

42

43

44

45

46

47

48

49

In practise, visual storytelling is faced with significant challenges due to the lack of high-quality image-text data. Most existing works are limited to train on a few specific animations, for example, StoryGAN [17], StoryDALL-E [22] and AR-LDM [28], resulting in a small and restricted vocabulary and characters. To overcome this limitation, we have created a dataset called StorySalon , that features a rich source of coherent images and stories, primarily comprising children's storybooks collected from three different sources: videos, E-books, and synthesized sample data from our StoryGen model with human verification. As a result, our dataset includes a diverse vocabulary with different characters, storylines, and artistic styles.

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

We follow a three-stage training procedure: Firstly , we insert a LoRA-like architecture into text conditioning module in stable diffusion model, i.e. , on top of the image-to-text cross-attention, in order to adapt the pre-trained diffusion model on our collected dataset; Secondly , we introduce a visual context module that enables the generation process to condition on preceding image; Lastly , we finetune the model with data after human verification, i.e. , leveraging the feedback to further align the model with human preference. As a result, the scale and diversity of our collected dataset enable the model to acquire the ability of open-vocabulary visual storytelling, by that we mean, our model can generate new image sequences that are not limited to pre-defined storylines, characters, or scenes. For example, we can prompt a large language model to create unique and engaging stories, then feed into StoryGen for generation, as shown in Figure 1.

To summarise, we make the following contributions in this paper: (i) we propose the task of openended visual storytelling , that involves generating engaging image sequence that is aligning to a given storyline, for example, written by a large language model; (ii) we develop a novel architecture based on stable diffusion, termed as StoryGen, which can generate image sequences in an autoregressive manner, taking both preceding image and text prompt of current frame as condition; (iii) we initiate a data collection pipeline and collect a large-scale, diverse datasets of storybooks, from online videos, E-books and synthesized samples, including paired image-text samples of a diverse vocabulary with different characters, storylines, and artistic styles; (iv) we adopt a three-stage curriculum training strategy, that enables style transfer, visual context conditioning, and human feedback alignment, respectively. Experimentally, we conduct both quantitative comparison and human evaluation, showing that the outputs from ou proposed model are more preferred, in terms of image quality, style consistency, and image coherence.

2 Related Works

Diffusion Models learn to model a data distribution via iterative denoising and are trained with 73 denoising score matching. Drawing from the principles of Langevin dynamics and physical diffusion 74 processes, diffusion models have undergone refinement through a multitude of works [39, 27, 45]. 75 Notably, DDPM [10] has demonstrated improved performance over other generative models, while 76 DDIM [40] has significantly boosted generation efficiency. In view of their superior generative 77

capabilities, diffusion models have found extensive utility in various downstream applications besides 78 image generation, such as video generation [46, 5, 12, 9, 38], image manipulation [2, 24, 14, 7], 79 grounded generation [18], 3D texturing [34], and image inpainting [47, 26, 19, 1]. 80

Text-to-image Generation involves the creation of images from textual descriptions. The task has 81 been tackled using various generative models, with Generative Adversarial Networks (GANs) [6] 82 being the first widely-used model. Several GAN-based architectures such as StackGAN [50], 83 StackGAN++ [51], and AttnGAN [48] have achieved notable success in this area. Additionally, 84 pre-trained auto-regressive transformers [43] such as DALL·E [33] have demonstrated the ability 85 to generate high-quality images in response to textual prompts. Recently, diffusion models have 86 emerged as a popular approach to text-to-image generation. New images can be sampled under text 87 guidance from the data distribution learned by diffusion models with iterative denoising process. 88 DALL·E 2 [32] leverages CLIP [29] features to achieve well-aligned text-image latent space, while 89 Imagen [37] relies on large language models like T5 [31] to encode text. Stable Diffusion (or Latent 90 Diffusion) Model [35] performs diffusion process in latent space, and it can generate impressive 91 images after pre-training on a large-scale text-image datasets. 92

93

94

95

96

97

98

99

100

101

102

103

Story Synthesis is first introduced as the task of story visualization (SV) by StoryGAN [17], which presents a GAN-based framework and the inaugural dataset named Pororo, derived from cartoons. Subsequently, some other works also follow the GAN-based framework, such as DUCOStoryGAN [21] and VLC-StoryGAN [20]. In the case of word-level SV [16], more emphasis is placed on the representation of text, whereas VP-CSV [3] employs VQ-VAE [42] and a transformer-based language model to conserve character appearance and enhance visual quality. StoryDALL-E [22] extends the story visualization task to story continuation with the initial image given and recommends using a pre-trained DALL·E generative model [33] to produce coherent images. AR-LDM [28] introduces an auto-regressive latent diffusion model that can generate highly realistic images, but with only a limited vocabulary. NUWA-XL [49] is a concurrent work that exploits hierarchical diffusion model to synthesize long videos, with the keyframes generated first, followed by frame interpolation.

104

105

106

107

108

109

110

111

112

Existing models are mostly developed on specific scenes, which limits their ability for generating image sequences for diverse stories. In this paper, we target for more ambitious applications, to develop an open-ended visual storytelling model, that can digest stories of arbitrary length and diverse topics, and synthesize a sequence of coherent images in terms of both style and semantic.

3 Method

To be self-contained, we first present a brief overview to diffusion model in Section 3.1; then, we detail our proposed model for storybook generation in Section 3.2, starting from problem formulation, then architecture details, and lastly on training details.

3.1 Preliminaries on Diffusion Models

Diffusion models are a type of generative models that undergo a denoising process, converting input 113 noise into meaningful data samples. Diffusion models comprise a forward diffusion process that 114 incorporates Gaussian noise into an image sample x 0 , accomplished via a Markovian process over 115 T steps. If we denote the noisy image at step t as x t , the transition function q ( x t | x t -1 ) connecting 116 x t -1 and x t can be expressed as follows: 117

where β t ∈ (0 , 1) is the variance schedule controlling the step size. 118

Using Gaussian distribution property and reparametrization, if we define α t = 1 -β t and ¯ α t = 119 ∏ t i =1 α i , we can write equation 1 as follows: 120

Diffusion models also comprise a reverse diffusion process that learns to restore the initial image 121 sample from noise. A UNet-based model [36] is utilized in the diffusion model to learn the reverse 122

128

129

130

131

132

3.2 StoryGen Model

In this section, we start by defining the problem for visual storytelling, then we introduce the core components in our proposed architecture, namely, Style Transfer Module and Visual Context Module, lastly, we present details for training the model with a curriculum learning regime.

3.2.1 Problem Formulation

In visual storytelling, our goal is to generate a sequence of coherent and consistent images that corre133 spond to a given story in the form of natural language. To achieve this, we propose an auto-regressive 134 image generation model, called StoryGen , that generates the current frame I k by conditioning on 135 both current text description T k and the previous frame I k -1 , as illustrated in Figure 2. The model is 136 formulated as follows: 137

Here, {T 1 , T 2 , . . . , T L } represents the given story in text sentences, { ˆ I 1 , ˆ I 2 , . . . , ˆ I L | ˆ I i ∈ R H × W × 3 } 138 denotes the generated storybook, H,W refer to the width and height, respectively. ψ SDM ( · ) refers 139

:ConvLayer

Previous Images

：FrozenParameters

:TrainableParameters

:Image Embedding

:Text Embedding

Image Encoder

TextConditioningModule

VisualContextModule)

Cross

Self

attn

Self

Cross

FFN

FFN

attn

LoRA

attn

attn

TextEncoder

Text

Image

Encoder

Encoder

- It's probably a horse trying to keep its head dry in the rain.

Current

Previous

Text

Images

Current Text

diffusion process p θ . The process p θ can be expressed using the following equation. 123

where µ θ is the predicted Gaussian distribution mean value. 124

As we compute the loss function by taking the mean absolute error of the noise term ϵ θ into account, 125 we can express the mean value µ θ in terms of the noise term ϵ θ as follows: 126

Therefore, the objective can be written as: 127

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

to a semi-frozen stable diffusion model (SDM), with a small number of newly-added trainable parameters ( θ ). It takes randomly sampled gaussian noise, text description and preceding image as input, and generate coherent image sequence that align with the story's narrative. In the following sections, we present the architecture detail for one-step generation conditioned on text and image.

3.2.2 Architecture Details

Generally speaking, our model is built upon the foundation of a pre-trained stable diffusion model (SDM), that has been pre-trained on large number of paired image-caption samples, to gradually transform the noisy latent into an image. To tackle the problem of open-ended storytelling, we introduce two computational modules, namely, Style Transfer Module, and Visual Context Module, that enables the model to condition on not only text descriptions, but also the preceding RGB image, as shown in Figure 2. Formally, we can express the generation procedure as:

where x , ϕ text ( · ) and ϕ vis ( · ) denote the noisy latent, encoded text description and preceding image.

Style Transfer Module. To steer a pre-trained stable diffusion model towards the style of children's storybooks, we propose to insert a lightweight, LoRA-like [13] architecture into the text conditioning module, effectively acting as style transfer. This can be expressed as:

W Q , W K and W V denote the projection matrices, adopted from the text conditioning module in pre-trained stable diffusion model. ϕ text ( · ) and C text k refer to the pre-trained CLIP text encoder and extracted text embedding respectively. ∆ Q, ∆ K and ∆ V are calculated by a learnable projection of x , C text k and C text k , respectively, resembling LoRA operations.

Visual Context Module. In order to generate visually coherent images, we insert a visual context module after the text conditioning, specifically, it is a transformer decoder comprising a self-attention layer, a cross-attention layer, and a feed-forward network, where the cross-attention layer employs a casual attention mechanism by using the noisy latent as query and the visual features of the previous frame as key and value, which can be formally denoted as:

W Q , W K and W V refer to three learnable projection matrices, ϕ vis ( · ) denotes the visual feature extracted by a pre-trained CLIP visual encoder. It is worth noting that the visual context module can also extend to multiple condition frames by concatenating their CLIP features as visual contexts.

Training Objective. At training stage, we randomly sample a triplet each time, i.e. , {I k , I k -1 , T k } , and the objective defined in Equation 5 can now be transformed into:

and as we adopt classifier-free guidance [11] in inference, the predicted noise can be expressed as:

where w is the guidance scale.

3.2.3 Curriculum Learning

In this section, we describe the three-stage training strategy, that includes single-frame pre-training, multiple-frame fine-tuning, and alignment with human feedback. This curriculum learning approach enables the model to gradually learn from simple to complex tasks, ultimately improving its ability to generate high-quality images that align with the given story narratives. Details for our proposed curriculum learning are presented below.

Single-frame Pre-training. We start by training the style transfer module, which has been inserted into the text conditioning module of a pre-trained stable diffusion model, in a single-frame manner. At this pre-training stage, we do not introduce the visual context module, and freeze all other parameters except for the LoRA-like plug-in. This training approach allows us to quickly adjust to the desired

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

visual style and character appearance in storybooks, while also maintaining the generation ability of the pre-trained stable diffusion model.

Multiple-frame Fine-tuning. Here, we fine-tune the visual context module while freezing other parameters of the model. Till this point, this allows the generation procedure to utilize information from either the text description or the preceding frame. To avoid over-fitting to the text descriptions, we adopt a technique inspired by BERT training [15], randomly dropping some words in the text with certain probability. The entire visual context module is fine-tuned during this stage.

Fine-tuning with Human Feedback. After multiple-frame fine-tuning, the model has developed basic storybook generation capabilities, to avoid from generating, potentially scary, toxic or biased content, we also propose to align the model with human preference. Specifically, we prompt ChatGPT to generate approximately 200 stories and use our model to synthesize images. After manually filtering around 100 high-quality storybooks from this corpus, we add them to the training set for further fine-tuning. As future work, we aim to add more books into this human feedback step.

Inference. With the three-stage training regime, we can streamline the entire inference process into a unified generation framework. As shown in the Figure 1, at inference time, we can prompt the ChatGPT to generate engaging, yet educational storylines, and synthesize the first image using a single-frame approach with only style transfer module involved; the previously synthesized frames, along with story description at current step, are treated as condition to generate image sequence in an auto-regressive manner. Experimentally, our proposed StoryGen is shown to generate images that align with the storyline, as well as maintaining consistency with previously generated frames.

4 Dataset Preparation

For training visual storytelling model, we collect a dataset called StorySalon , that contains approximately 2K storybooks and more than 30K well-aligned text-image pairs. This dataset is comprised of storybooks with potentially aligned text and image pairs, sourced from three different sources: video data, E-book data, and additional data from human feedback.

4.1 Image-text Data from Videos & E-books

Here, we elaborate the procedure of extracting paired image-text samples from YouTube videos and E-books (pdf and corresponding audios available).

Visual Frame Extraction. To begin with, we download a significant amount of videos and subtitles from YouTube, by querying keywords related to children story, for example, storytime . We then extract the keyframes from the videos, along with the corresponding subtitles and their timestamps. To remove duplicate frames, we extract ViT features for each frame using pre-trained DINO [23], for the image groups with high similarity score, we only keep one of them. Next, we use YOLOv7 [44] to segment and remove person frames and headshots, as they often correspond to the story-teller and are unrelated to the content of the storybook. Finally, we manually screen out frames that are entirely white or black. Similarly, we also acquire a number of electronic storybooks from the Internet, and extract images from E-book, except for those with extraneous information, for example, the authorship page. We acquire the corresponding text description with Whisper [30] from the audio file. For E-books that do not have corresponding audio files, we use OCR algorithms, to directly recognize the text on each page.

Visual-Language Alignment. Here, for each image, we acquire two types of text description, namely, story-level description, and visual description. This is based on our observation that there actually exists semantic gap between story narrative and descriptive text, for example, the same image can be well described as "The cat is isolated by others, sitting alone in front of a village." in story, or "A black cat sits in front of a number of houses." as visual description, therefore, directly finetuning stable diffusion model with story narrative maybe detrimental to its pre-trained text-image alignment.

In practise, to get story-level paired image-text samples, we align the subtitles with visual frames by using Dynamic Time Warping (DTW) algorithm [25]. To get visual descriptions, we use ChatCaptioner [52] to generate captions for each image, as shown in Figure 3. s At training time, this allows us to substitute the original story with more accurate and descriptive captions.

Video

E-book

Synthetic

Text

Caption 1

Caption 2

Caption 3

Unstructured

Metadada

Step 2

Visual-language

Alignment

Aligned

Image-text Pairs

Step 1

Visual Frame

Extraction

Step 3

Visual Frame

Post-processing

Text

: Santa waved down

with a cry of delight:

merry Christmas, my dears,

and good night. Let

YouTube know you're

interested in videos.

Story

: Santa waved down

with a cry of delight:

merry Christmas, my

dears, and good night.

Caption

: Two mice are

sleeping in the bed…

一

Visual Frame Post-processing. In practice, we discovered that books in the frames can potentially 231 interfere with our image generation model by having story texts printed on them. To address this, we 232 use an OCR detector to identify the text regions in the frames and an image inpainting model to fill in 233 the text and headshot regions. This process results in more precise image-text pairs that can be fed 234 into the diffusion model. 235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

4.2 Additional Data from Human Feedback

As outlined in Section 3.2.3, we use the model (trained after two stages) to generate a set of new storybooks, and incorporate human feedback into the fine-tuning process. Following a rigorous manual review, we carefully select the best pieces, and add them into the training dataset. This allows us to continually improve the quality of our model and ensure that it produces engaging, yet educational storybooks, that align with human's preference.

4.3 Discussion

Given the diversity of our data sources and data types, the StorySalon dataset exhibits a significantly broader range of visual styles and character appearances over other animation-specific datasets. Moreover, our dataset surpasses others in terms of vocabulary coverage by a substantial margin. Notably, our texts seamlessly integrate narrative storylines and descriptive visual prompts, ensuring the preservation of text-image alignment while adapting to the art style of storybooks.

5 Experiment

In this section, we start by describing our experimental settings, then compare with other models from three different perspectives: style, quality and coherence with quantitative and subjective human evaluation. Additionally, we present the results of our ablation experiments to prove the effectiveness of our proposed training regime.

5.1 Training Settings

Our model is based on publicly released stable diffusion checkpoints, with a learning rate of 1 × 10 -5 and a batch size of 512 . We begin with a single-frame pre-training stage, which involves 10,000 iterations on 8 NVIDIA RTX3090. In the multiple-frame fine-tuning stage, we fine-tune the model for 40,000 iterations using a single condition image. To improve the robustness of the training procedure, we apply a 10% ∼ 30% words dropout with a probability to the texts in the current frames. We also fine-tune the model with human feedback on the training set with 100 manually generated storybooks in addition for 5,000 iterations. During inference, we utilize DDIM sampling and classifier-free guidance with a weight of 6.0.

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

5.2 Quantitative Results

To evaluate the quality of our generated image sequence, we adopt the widely-used Fréchet Inception Distance (FID) score [8]. However, as there is no standardized metric for evaluating the consistency of images, we include human evaluation for comparison.

Fréchet Inception Distance (FID). We present a comparison of the FID scores between our model and other existing ones, including SDM and Propmt-SDM, which conditions on an additional cartoonstyle-directed prompt "A cartoon style image" . Specifically, we calculate the FID scores between the distribution of the generated results from these models and the distribution of the our proposed StorySalon testset. As shown in Table 1, we evaluate the generated image sequences from 100 storylines, obtained by prompting ChatGPT. Our StoryGen model outperforms the original stable diffusion models (SDM) and Prompt-SDM by a large margin, demonstrating the effectiveness of our model in generating high-quality coherent images.

Human Evaluation. We conduct two types of human evaluation experiments to assess the quality of our generated storybooks. In the first experiment, we randomly select an equal number of groundtruth storybooks, the results of our StoryGen and the generation results of SDM and Propmt SDM. Participants are then invited to rate these four categories of storybooks on a score ranging from 1 to 5, taking into account text-image alignment, style consistency, content consistency, and image quality, higher scores indicate better samples. In the second experiment, we prompt ChatGPT to produce a number of storylines and use our StoryGen along with the two variations of stable diffusion to generate corresponding image sequences. Participants are asked to choose the preferred results of each storyline. To mitigate bias, participants are unaware of the type of storybooks they are evaluating during these two human evaluation experiments. In both experiments, we have invited approximately 30 participants in total.

Table 1 presents the results of our human evaluation. As can be seen, our model has shown significant performance improvement in its overall score compared to stable diffusion models, especially in terms of consistency and alignment, indicating that it can generate images that are highly consistent with the given text prompts and visual contexts, thus better exploiting the contextual information.

Ablation Studies. Our study compares the performance of our model with and without fine-tuning using human feedback. We evaluate the FID score of between our generated image sequence and the test set of our StorySalon dataset. The results demonstrate that fine-tuning with human feedback slightly improves performance, we conjecture that this could be attributed to the fact that the number of human-verified samples are limited, due to a result of resource limitation. In future work, we intend to address this quantity constrain by continually augmenting the dataset with new samples and closely monitoring the progressive advancements.

5.3 Qualitative Results

In Figure 4, we present the visualization results, showing that our model can generate storybooks with a broad vocabulary, while maintaining coherence and consistency throughout the narrative. The generated images successfully maintain the consistency of the artistic style and character appearance, whereas the results from SDM and Prompt-SDM fail to do so. Moreover, style of the generated results from SDM's are also incongruent with the requirements of visual storytelling for childrens.

6 Conclusion

In this paper, we consider the exciting yet challenging task known as open-ended visual storytelling , which involves generating a sequence of consistent images that tell a coherent visual story based

1

1

1

2

3

4

5

6

7

2

3

4

5

6

7

2

3

4

5

6

7

1

1

1

2

3

4

5

6

7

2

3

4

5

6

7

2

3

4

5

6

7

on the given storyline. Our proposed StoryGen architecture can take input from the preceding 305 frame along with the text prompt to generate the current frame in an auto-regressive manner. A 306 three-stage curriculum training strategy has been introduced for effective training and alignment with 307 human preference. Due to the limitations of dataset in previous works, we have also collected a 308 large-scale, diverse dataset named StorySalon that includes paired image-text samples sourced from 309 storybook data from videos, e-books and synthesized samples. The StorySalon dataset possesses 310 a diverse vocabulary of storyline, character appearances and artistic styles. While comparing with 311 stable diffusion models with quantitative experiment and human evaluation, our proposed model 312 substantially outperform existing models, from the perspective of image quality, style consistency, 313 content consistency, and image-language alignment. 314

315

316

317

318

References

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18208-18218, 2022. 3

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing 319 instructions. arXiv preprint arXiv:2211.09800 , 2022. 3 320

Hong Chen, Rujun Han, Te-Lin Wu, Hideki Nakayama, and Nanyun Peng. Character-centric story 321 visualization via visual planning and token alignment. arXiv preprint arXiv:2210.08465 , 2022. 3 322

K. Dickinson David, A. Griffith Julie, Golinkoff Roberta, Michnick, and Hirsh-Pasek Kathy. How reading 323 books fosters language development around the world. Child Development Research , 2012. 1 324

325

326

327

Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 , 2023. 3

328

329

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 2020. 3

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to330 prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022. 3 331

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans 332 trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural 333 Information Processing Systems , 2017. 8 334

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P 335 Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video 336 generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022. 1, 3 337

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural 338 Information Processing Systems , 2020. 2 339

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022. 340 5 341

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. 342 Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022. 3 343

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and 344 Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 345 2021. 5 346

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal 347 Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276 , 348 2022. 3 349

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional 350 transformers for language understanding. In Proceedings of NAACL-HLT , pages 4171-4186, 2019. 6 351

Bowen Li. Word-level fine-grained story visualization. In Proceedings of the European Conference on 352 Computer Vision , 2022. 3 353

Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, 354 and Jianfeng Gao. Storygan: A sequential conditional gan for story visualization. In Proceedings of the 355 IEEE Conference on Computer Vision and Pattern Recognition , 2019. 2, 3 356

357

358

Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Guiding text-to-image diffusion model towards grounded generation. arXiv preprint arXiv:2301.05221 , 2023. 3

359

360

361

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022. 3

Adyasha Maharana and Mohit Bansal. Integrating visuospatial, linguistic, and commonsense structure into 362 story visualization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language 363 Processing , 2021. 3 364

Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Improving generation and evaluation of visual 365 stories via semantic consistency. In Proceedings of the 2021 Conference of the North American Chapter of 366 the Association for Computational Linguistics: Human Language Technologies , 2021. 3 367

368

369

370

Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Storydall-e: Adapting pretrained text-to-image transformers for story continuation. In Proceedings of the European Conference on Computer Vision , 2022. 2, 3

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

Caron Mathilde, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision , 2021. 6

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021. 3

Meinard Müller. Dynamic time warping. Information retrieval for music and motion , 2007. 6

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021. 3

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , 2021. 2

Xichen Pan, Pengda Qin, Yuhong Li, Hui Xue, and Wenhu Chen. Synthesizing coherent story with auto-regressive latent diffusion models. arXiv preprint arXiv:2211.10950 , 2022. 2, 3

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning , 2021. 3

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356 , 2022. 6

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research , 2020. 3

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022. 3

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , 2021. 1, 3

Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721 , 2023. 3

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022. 1, 3

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , 2015. 3

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 2022. 3

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022. 3

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning 414 using nonequilibrium thermodynamics. In International Conference on Machine Learning , 2015. 2 415

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 416 arXiv:2010.02502 , 2020. 2 417

Gabrielle A. Strouse, Angela Nyhout, and Patricia A. Ganea. The role of book features in young children's 418 transfer of information from picture books to real-world contexts. Frontiers in Psychology , 2018. 1 419

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural 420 information processing systems , 2017. 3 421

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz 422 Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 423 2017. 3 424

425

426

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696 , 2022. 6

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , 2011. 2

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565 , 2022. 3

Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.05034 , 2022. 3

Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018. 3

Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303.12346 , 2023. 3

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the International Conference on Computer Vision , 2017. 3

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence , 2018. 3

Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 446 Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint 447 arXiv:2303.06594 , 2023. 6 448

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aim to generate a series of coherent images given a series of text prompts resembling a visual storybook. To do so, the authors focus on two fronts: (1) leveraging the Stable Diffusion model to generate the series of images and (2) generating a diverse dataset used to train the model on a range of styles. For generating a set of coherent images, the authors condition the Stable Diffusion on both the text prompt and a set of previously generated frames, both encoded using a frozen CLIP encoder. The text conditioning is passed to the U-Net layers via the standard cross-attention module and LoRA. To insert the image conditioning, the authors introduce a Visual Context Model resembling the standard text conditioning module. The diffusion model is then partially trained to generate images that are both consistent with the text prompt and previously generated frames. To attain images ranging in style, the authors construct a new dataset, named StorySalon, consisting of Youtube videos and E-Books. These raw storybooks are filtered and re-captioned to better align with the visual content of the storybook. Qualitative results demonstrate the ability to generate new storybooks on prompts generated by ChatGPT while quantitative results demonstrate improvements over simple baselines.\n\nSTRENGTHS:\n- The authors focus on an important task of generating a series of coherent images that follow a given text prompt. This could potentially be useful beyond a simple storybook creation, e.g., for video generation. \n- The Visual Context Module as a means for injecting image-level details to the denoising network is simple and intuitive and could be useful in other tasks. For example, in image editing where the desired edit cannot be easily described using language.\n- The visual results generated by StoryGen are impressive in comparison to the evaluated baselines.\n\nWEAKNESSES:\n**Dataset Creation:**   \n- I have some reservations regarding the use of the term Human Feedback in Section 3.2.3 and Section 4.2. While ChatGPT was fine-tuned to align with human preference, I believe that using ChatGPT for generating additional prompts should not be considered Human Feedback. While this is an intriguing approach, I believe replacing Human with LLM is more reflective of what is actually done here.\n- Regarding the ablation study performed by the authors, it seems that “Human Feedback” leads to a quite negligible decrease in FID and I am therefore uncertain if this really contributes to the curriculum learning scheme. The authors mention that more stories can be added, but I would have expected to see a bigger improvement if this stage is truly important.\n\n**Evaluation:**   \n- There are numerous essential evaluations that are missing from the current submission. Among these, the most important is a thorough evaluation and comparison of StoryGAN, Story-DALL-E, and AR-LDM. All three have publicly available code so an evaluation is needed to understand the improvement realized by StoryGen. \n- I am not sure that FID is a particularly interesting metric here since all evaluated methods in Table 1 use Stable Diffusion to generate the images. Moreover, I do not believe that FID is a good metric when trying to measure how much the image captures a given style, as is the goal here. Maybe a CLIP-based metric using a prompt depicting a style would be more appropriate here? \n- There are numerous ablation studies that I believe are required to understand the contribution of both the proposed architecture and dataset. \n    - Architecture: \n        - An ablation study on the Visual Context Module and whether a simpler conditioning is possible (see my detailed question below). \n        - Was an ablation study performed on the BERT-like masking during the multi-frame fine-tuning? \n    - Dataset\n        - The impact of the visual-language alignment stage in preparing the dataset. The authors state that directly fine-tuning on the story narrative may be detrimental, but do not validate this claim. \n- Some additional evaluations could help validate the effectiveness of the method. \n    - First, the authors claim that the method can be used to generate stories of arbitrary lengths (Line 106). It would be great to quantify this by generating stories of varying lengths and validating whether there is a loss in quality after a certain length. \n- One particularly interesting component of the method is the Visual Context Module, so I would have liked to see far more evaluations performed on it. For example, the authors mention that multiple frames can be used for conditioning by concatenating their CLIP feature. Some interesting questions that could help strengthen the importance of the component include: \n    - How much was this evaluated? \n    - How much does conditioning on more frames assist in temporal consistency? \n    - How many previous frames can be concatenated without hindering performance?\n- Stating that the model achieves a significant improvement in the alternative models seems like a strong over-claiming when approximately 30 participants were used for the user study. A substantially larger pool of participants would be needed to truly quantify this improvement, especially since this is the only relevant metric used to evaluate the methods. Why are the reported FID metrics between Table 1 and Table 2 different? Is a different dataset used? I would expect only “without HF” to be different if both use the same dataset.\n\nQUESTIONS:\nI hope the authors can help clarify several questions that I have regarding both the method and the evaluations. \n\n**General Comments:**   \n- A reference to Rombach et al. is missing when discussing Stable Diffusion (e.g., in Line 38 where it is first used).\n- First stated contribution is that the authors propose the task of open-ended visual storytelling, but this was a previously studied problem (as also mentioned in Line 44). \n- When describing the style transfer module and visual context module, the authors reuse the same notation for the weight matrices \\(W^K\\) and \\(W^V\\). To make the writing a bit clearer, the authors should use a different notation for each case since these are separate modules if I understood correctly.\n\n**Method:**   \n- What happens if the image conditioning is simply done using the standard image-to-image technique where the noised latent is conditioned on the previous image? Adding an ablation study would help validate this design choice. If performing a thorough ablation study is difficult, providing intuition on the motivation for the Visual Context Module would help highlight the contribution to readers.\n- Where is the style transfer model illustrated in Figure 2? Is this the blocked labeled LoRA? The authors mention that this is a LoRA-like architecture. Could the authors kindly clarify what exactly the difference between the original LoRA design is and why this modification was made?\n- Regarding the Multi-Frame Conditioning: \n    - According to Figure 2, it seems like StoryGen receives several previous frames as conditioning while Line 136 indicates that only the previous frame is used for conditioning. Could the authors kindly clarify this? \n    - Similarly, in the general setting, it seems that multiple frames are used for conditioning. Therefore, would it be more accurate to revise the equation in Line 150 to indicate any number of \\ell frames as conditioning? \n    - In Section 5.1, the authors mention that the multiple-frame fine-tuning stage is done using a single conditioning image (Lines 256-257). Where do the multiple frames come into play? Is this really a multi-frame conditioning? \n\n**Small comments:**   \n- Line 70: ou -> our \n- Line 142: generate -> generates, align -> aligns\n- Line 276: Propmt -> Prompt\n\nLIMITATIONS:\nThe authors include a discussion on current limitations and potential societal impacts in the supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes the model StoryGen for the task of visual storytelling. Visual storytelling is a task to generate a sequence of consistent images given a story (several sentences). StoryGen is a diffusion model taking in both image and text as conditions, and outputs an image consistent with the conditions. The training process includes pre-training for single image, finetuning for multiple image and finetuning with human feedback. On top of the StoryGen model, this work also provides a dataset, called StorySalon, which consists of 2k story books (30k well aligned text-image pairs). \n\nThe overall structure of StoryGen model is simple. It is built upon existing well trained diffusion models and image/text encoders. To generate cartoon-like image, LoRA is adopted into the text conditioning module in a diffusion model. The author calls it the style-transfoer module. The parameters in LoRA are updated at this pre-training stage to give single cartoon image. Next is the multipe image fune-tuning. StoryGen conditions on both text and image, which is implemented by using two cross attention layers: One is noise input + text and the other is noise input + encoded previous generated image. After the second step, StoryGen if further finetuned on 100 high-quality stories. \n\nThe author also spends some efforts to collect the StorySalon dataset. To begin with, the author downloads a huge number of videos and subtitles from online web resources with potential stories. Then give story-level description and visual level description for each story. The story level description is obtained by using dynamic time warping algorithm using subtitles. The visual level description is derived from ChatCaptioner. Finally, OCR method is applied to get potential videos captions.\n\nThe experiment section shows that StoryGen model can give consistent and story-like output images, while other methods fail.\n\nSTRENGTHS:\nThe paper is clearly written. The collected StorySalon dataset could benefit the research community.\n\nWEAKNESSES:\nThere is not much technical novelty, and the experimental results are limited.\n\nQUESTIONS:\n- Line 192 mentions the 100 high quality books are added into the training set.  For the third stage training, is it still using the datasets from previous steps, but with 100 more samples? Would it make more sense to only use the 100 high quality books for further fine-tuning? Also, the name (fine-tuning with human feedback) is confusing since there is neither human feedback nor reward model. Is there any human involved to give preferences of different generated outputs from the StoryGen model?\n\n- The human feedback fine-tuning stage doesn’t seem to help a lot in the quantitative scores. It would be better to include visual examples without human feedback fine-tuning. If the difference is too small, it would be better to re-structure the paper.\n\n- For the visual examples given in Figure 4, I noticed the boy with yellow hair shows in both stories. Does this boy appear a lot in the training set? Also, it would be better if the author could provide more than one examples for each story using StoryGen. Otherwise, it feels like the model is overfitting to the training data.\n\nLIMITATIONS:\n- Since there is no human labeler involved in collecting StorySalon dataset and the total number of stories in StorySalon is only 2k, I’m concerned about its quality.\n\n- It is also unclear from the examples provided if the results is derived by just overfitting the training set.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work focuses on the application of image generation based on a given story. Specifically, the proposed model is conditioned on the current sentence and prior generated images to ensure the story is engaging and coherent. A progressive training strategy is proposed to achieve a good model. To improve the proposed method, a new dataset is collected, while a set of human-verified generative samples are also utilized to improve the generated images. \n\n--------------------------\nI acknowledge the author's effort in the rebuttal and have made changes to the review accordingly.\n\nSTRENGTHS:\n+ This work demonstrates the possibility of generating visual storytelling images conditioned on the given stories. \n+ A three-stage curriculum training strategy is proposed to train the proposed model. However, it would be great to demonstrate the limitation of training the model with multiple-frame (i.e., without single-frame pre-training)\n+ The authors collected a large-scale dataset to enable model training for storytelling purposes.\n\nWEAKNESSES:\n- the technical contribution of this work is limited. Most of the components are not novel and the key contributions are the way it is combined to generate a plausible output. It is unclear what the insights generated from this work that is not previously obvious to the community.\n- The description of the new StorySalon dataset is limited. Specifically, it is unclear if the collected dataset has obtained legal consensus and properly handled copyright issues. \n- The work lacks a comparison with existing work, such as those introduced in line 44 and line 93-102. The two baselines in Table 1 are too naive as both are inherently limited to generate a fair comparison with the proposed method. \n\n\nMinor:\n- Fig 2 should clearly state that the Image encoder only considers a single previous frame to generate the next frame.\n\nQUESTIONS:\n- Please justify what it means by \"open-ended\". Was the dataset divided in a manner that the training and validation/test dataset has distinct distribution or content?\n\n- The final stage of curriculum training strategy fine-tune the model with human feedback. Here, the stories are generated with ChatGPT and the images are synthesized with the same model. This creates a problem of bootstrapping machine learning and makes it unclear whether such an approach can benefit the model. A recent work [1] discussed the curse of recursion with generated data. I want the invite the author to discuss the problem of this. \n\nIn addition, the results in Table 2 show that with or without HF has a small difference. Can the author also explain what does 0.19 translate to the differences in the qualitative results?\n\n[1] Shumailov et al. THE CURSE OF RECURSION: TRAINING ON GENERATED DATA MAKES MODELS FORGET.  https://arxiv.org/pdf/2305.17493.pdf\n\n- In line 141, why is a small number of newly-added trainable parameters required and how does it impact the model performance/learnability?\n\n- In Table 1 and Two, why is the FID score in both table inconsistent?\n\nLIMITATIONS:\nThe paper (in supplementary) discusses that data bias is an issue that needs to address in this domain. Collecting a larger dataset for training is the solution discussed. This may be valid considering this work is still in the early stage of the research. \n\nI want to point out that the discussed approach is limited as (1) it is resource-consuming, and (2) it will face the problem of copyright in order to obtain a good dataset for training. The data ownership issue may be a major hindrance.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents StoryGen, an auto-regressive image generator that leverages text and image conditioning. StoryGen incorporates a style transfer module integrated into the text-conditioning module, along with a visual context module. The authors also constructed a substantial dataset called StorySalon, comprising 2K storybooks and 30K text-image pairs.\n\nSTRENGTHS:\n1. This paper constructs a new dataset StorySalon contains 2K storybooks and more than 30K well-aligned text-image pairs. The authors have invested significant effort into filtering the data, making it a valuable resource for advancing the field of story visualization.\n2. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The illustration does not align with the description provided. In line 135, the authors state that \"StoryGen generates the current frame $\\mathcal{I}_k$ by conditioning on both the current text description $\\mathcal{T}_k$ and the previous frame $\\mathcal{I}_{k-1}$, as illustrated in Figure 2.\" However, the left figure of Figure 2 shows the image conditioned on more than one previous image, which contradicts the mentioned conditioning approach.\n2. The improvement of Human feedback appears to be trivial, as indicated in Table 2. The 0.19 FID score gap could potentially be attributed to different training seeds, which raises doubts about the significance of the reported improvement. (I do not agree with the statement that 200 stories are too small since the model is trained using 2k stories overall. It appears to be sufficient for human alignment and does not require an extensive amount of data.)\n3. The FID score lacks precision in the test set, particularly with only 100 storylines. It is recommended that the authors expand the test set by including more stories to provide a more accurate evaluation.\n4. The baselines SDM and Prompt-SDM are too weak.  It is suggested that the authors compare StoryGen with finetuned or LoRA-finetuned SDM models using the same training settings to establish a more robust baseline for comparison.\n5. The auto-regressive generation approach employed by StoryGen has already been proposed by AR-LDM. Consequently, the architecture design itself lacks novelty.\n6. StoryGen is only conditioned on one previous image and does not utilize the corresponding caption of the previous image. In the depicted cases of Figure 1 and Figure 4, there is only one main recurring character. If multiple characters were present, StoryGen may struggle to ground the characters in the previous images. Furthermore, if a character does not exist in the previous image, StoryGen may face difficulties in maintaining consistency between frames.\n7. The language understanding capacity of StoryGen appears to be weak. For instance, in the second case of Figure 4, the rabbit appears small in the fourth frame, whereas it should be as big as it is in the fifth frame. Additionally, in the sixth frame, the boy's hair does not become brighter as described in the caption. Moreover, in the seventh frame, multiple other boys are depicted with the same yellow hair, which contradicts the previous story setting. This limitation may stem from StoryGen solely relying on the only one previous frame $\\mathcal{I}_{k-1}$ and not incorporating previous captions into its generation process.\n\nQUESTIONS:\n1. StoryGen generates the current frame $\\mathcal{I}_k$ by conditioning on current text description $\\mathcal{T}_k$ and the previous frame $\\mathcal{I}_{k-1}$.  I am wondering why the authors do not use all historical frames $\\mathcal{I}_0, \\cdots, \\mathcal{I}_{k-1}$ in this process.\n2. StoryGen introduces a style transfer module $\\phi_{text}$ and a single-frame pre-training stage. However, since the training set is constructed from multiple sources (as mentioned by the authors in Line 244), the style is inconsistent. Moreover, many LoRA style transfer plugins not only tune the cross-attn module but also the self-attn modules. Why not follow this setting and add LoRA layers in both self- and cross-attn layers? This way, StoryGen can also benefit from other style transfer LoRA developed by the community.\n3. In Line 186, the authors mention randomly dropping some words in the text with a certain probability following BERT. I have doubts about this technique because there is no masked language modeling task in StoryGen, and such a technique may not be helpful.\n4. I question whether the authors truly need to replace descriptive captions with story narrative text. The story in the storybook and the story generated by LLMs are both in the form of story narrative text, rather than descriptive captions. Using story narrative text to train the model may lead to implicit alignment with human performance.\n\nLIMITATIONS:\n1. The author should provide examples of the constructed dataset showcasing different visual styles and character appearances.\n2. The StorySalon dataset consists of 2K storybooks and over 30K well-aligned text-image pairs, which is smaller compared to datasets such as FlintstonesSV (24K stories and 123K image-caption pairs), PororoSV (14K stories and 74K image-caption pairs), and VIST (27K stories and 136K image-caption pairs). Despite the authors' claim that StoryGen can perform open-ended story generation, it remains unclear whether StoryGen can generate stories involving more complex scenarios with unusual entities.\n3. Apart from human feedback, the authors have not conducted any other ablation studies to evaluate the effectiveness of their proposed techniques, such as word dropout and curriculum learning.\n4. There are concerns regarding the legality of using web-crawled e-books. The authors should provide additional information about the sources of the e-books and clarify whether proper copyright guidelines were followed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose an approach for fine-tuning diffusion models for the task of story generation, where a model must generate frames for sentences in a story. To do so, they propose adding adaptors conditioned on both images and text into a pre-trained stable diffusion UNet. The authors also introduce a scraped dataset of 2k stories with 30k image-text pairs, which serves as the data foundation for their fine-tuning.\n\nSTRENGTHS:\nS1. The dataset of story text and images is a significant contribution, which can be useful for future work on visual storytelling.\n\nS2. The paper is generally well framed and motivated. Writing and presentation is in general strong and polished.\n\nS3. Simplicity of the method. The method uses off-the-shelf components and algorithms (LoRA, cross-attention, etc.) to enable new capabilities. I see this simplicity as a strength not a weakness. \n\nS4. Inclusion of human evaluation is a strength.\n\nWEAKNESSES:\nW1. Presentation. Figure 2 suggests that StoryGen is conditioned on all past frames. However, in reality StoryGen is conditioned on the most recent frame.\n\nW2. Results. It appears that the model is not able to preserve style and content as well as say DreamBooth. For example the stripes on the shirt in Figure 4 are not preserved. \n\nW3. Evaluation. It seems that the StoryGen model without human feedback model was not evaluated in the human evaluation (Table 1). Given that the ablation in Table 2 shows similar FID scores for StoryGen with and without human feedback, it is not clear if this step is really necessary. \n\nW4. Lack of baselines. StoryGen is specifically fine-tuned for the desired task, while stable diffusion is not. Given this, it is perhaps not surprising hat StoryGen greatly outperforms stable diffusion. Can some other baselines, perhaps based on the DreamBooth (with the subject in the first image representing the special [V] token) be used for a stronger baseline? This is just one idea; however, comparing against some prior work may help to elucidate the strength of the method.\n\nW5. Evaluation. Fine-tuning can sometimes hurt the generality of a model. How well are individual frames generated relative to the base SD model? Is it possible to do a human evaluation here or compute CLIP scores? Ideally, the story would be cohesive (frame-to-frame consistency) without degradation in quality of each frame.\n\nQUESTIONS:\nQ1. Can the authors provide further discussion of why StoryGen without human feedback was not evaluated in the human evaluation (W3)? Is it possible to add this evaluation? I believe this would help contextualize the importance of the human feedback in the proposed method.\n\nQ2. Can the other concerns related to evaluation be addressed (W2, W4, W5)?\n\nLIMITATIONS:\nThe paper does not directly address limitations in the main paper, which is an additional weakness. I suggest discussing failure cases or otherwise conducting a failure analysis.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}