CoIn: A Lightweight and Effective Framework for Story Visualization and Continuation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 09:52:06

Abstract: Story visualization aims to generate realistic and coherent images based on multi-sentence stories. 
However, current methods face challenges in achieving high-quality image generation while maintaining lightweight models and a fast generation speed. 
The main issue lies in the two existing frameworks.
The independent framework prioritizes speed but sacrifices image quality with the non-collaborative image generation process and basic GAN-based learning.
The autoregressive framework modifies the large pretrained text-to-image model in an auto-regressive manner with additional history modules, 
leading to large model size, resource-intensive requirements, and slow generation speed.
To address these issues, we propose a lightweight and effective framework, namely CoIn.
Specifically, we introduce a Context-aware Story Generator to predict shared context semantics for each image generator. 
Additionally, we propose an Intra-Story Interchange module that allows each image generator to exchange visual information with other image generators.
Furthermore, we incorporate DINOv2 into the story and image discriminators to assess the story image quality more accurately.
Extensive experiments show that our CoIn keeps the model size and generation speed of the independent framework, while achieving promising story image quality.

Main paper: 
 related work: 
 Text-to-Image Synthesis. Recent advancements in text-to-image synthesis have primarily focused on three main frameworks: Generative Adversarial Networks (GANs) [8] , autoregressive models [4, 5, 31] , and diffusion models [24, 30, 32, 47] . GANs, such as Stack-GAN [45, 46] , AttnGAN [42] , DM-GAN [48] , DF-GAN [40] , GALIP [39] , and StyleGAN-T [36] employ adversarial training strategies between generators and discriminators to generate high-quality images from text descriptions. Large pretrained autoregressive models, including DALL•E [31] , Make-A-Scene [6] , Parti [43] , and VAR [41] , have demonstrated scalability and proficiency in synthesizing images. These models generate images by sequentially autoregressively predicting pixel tokens based on previously generated tokens. Diffusion models [3, 12, 13, 23, 37] , such as VQ-Diffusion [9] , GLIDE [24] , DALL-E2 [30] , Latent Diffusion Models [32] , Imagen [35] , eDiff-I [1] , and SDXL [27] , have gained significant interest. These diffusion models address some of the challenges faced by GANs, such as mode collapse and training instability, resulting in the generation of diverse sets of images. These text-to-image generative models have a significant impact on story synthesis models and are often utilized as the generative backbone for generating story images based on story descriptions. Story Visualization and Continuation. StoryGAN [18] was the pioneering work that introduced the task of story visualization and proposed a GAN-based sequence generation model. It consisted of a deep RNN-based context encoder and two discriminators for images and stories. Subsequent works have built upon and refined this network. For example, CP-CSV [38] introduced a new foreground segmentation module to optimize the consistency between characters and backgrounds in the story. DUCO [21] and VLC [20] [17] enhanced semantic consistency through dual learning, with DUCO considering inter-image sequence consistency via copy-transform and VLC focusing on textual information by incorporating external common-sense knowledge. WL-SV [16] and Clustering GAN [17] simplified the two-stage GAN network of StoryGAN and further improved story quality through fine-grained word-level features and clustering learning, respectively. These GAN-based models adopt the independent framework. Conversely, recent approaches such as pre-trained DALL-E-based StoryDALL-E [22] , and pre-trained stable diffusion-based AR-LDM [26] have been developed for story visualization. They utilize large pre-trained text-to-image models and synthesize story images in an autoregressive manner. Notably, StoryDALL-E [22] also introduced a new task called story continuation, which involves generating unseen plots and characters based on a given source frame. AR-LDM [26] and Make-A-Story [29] can handle both the story visualization and story continuation tasks.Recently, there are some works focusing on open-ended story visualization. TaleCrafter [7] pioneered an interactive approach with sketch and layout controls for story visualization. CogCartoon [49] introduced a character-plugin generation to minimize data and storage requirements. Intelligent Grimm [19] curated a diverse open-ended story dataset from YouTube and e-books. The proposed CoIn differs greatly from the previous story visualization and continuation models. The CoIn shows a novel contextualize and interchange framework that is different from independent and autoregressive frameworks. It decomposes local image generation and context generation, which enables the story generator to synthesize more coherent story images. Furthermore, it adopts an Intra-Story Interchange module to exchange visual information between intra-story generators. Compared to previous models, our CoIn is a more effective and efficient framework for synthesizing high-quality and coherent story images. 
the proposed method: 
 In this work, we propose a novel framework for story visualization and continuation named CoIn. To synthesize high-quality story images while maintaining reasonable computational requirements and generation speed, we propose: (i) a Context-aware Story Generator (Context-aware SG) that explicitly captures and leverages global context information by sharing contextual features with all image generators throughout the synthesis process. (ii) an Intrastory Interchange module (Intra-SI) that facilitates the exchange of visual features among image generators within the story to improve visual consistency. (iii) a pair of DINO-based story and image Discriminators (DINO-baed D) that enhance the accuracy in assessing the quality of story images. In the following section, we first present the overall structure of our CoIn. Then, we introduce Context-aware SG, Intra-SI, and DINO-baed D in detail.As illustrated in Figure 3 , CoIn consists of CLIP Text and Image Encoders [28] , a Context-aware SG with a Contextual Story-Visual Fusion module (Contextual SVF), an Intra-SI module, a pair of DINObased story and image discriminators. The CLIP Text-Image Encoder encodes text descriptions and source story images into text and image vectors. The Contextual SVF takes text and image vectors, Gaussian noise, and a learnable context vector as input and outputs the fused story embedding and extracted context embedding. Afterward, the Context-aware SG utilizes the noise, fused story embedding, and extracted context embedding to synthesize story images. The story generator comprises two classes of subgenerators: image generator and context generator. The image generator employs fused story embedding, while the context generator utilizes extracted context embedding. In the image generator, the noise is passed through a Fully Connected (FC) layer and reshaped to (4, 4, 256) . Then, a series of upsampling blocks upsamples the image features and incorporates fused story embedding into the generation process. The Intra-SI collects synthesized image features from different image generators within the same story and performs cross-frame visual feature exchange. The contextual visual features synthesized by the context generator are added to the output features in the upsampling block. Finally, a convolution To enable feature interchange during the intra-story image generation process, we propose the Intra-SI module. As illustrated in Figure 3 and Figure 4 , the Intra-SI takes the fused story embeddings from the Contextual SVF and collects the image visual features from intra-story image generators. First, Intra-SI concatenates the story embeddings and visual features, respectively. The concatenated story embeddings are then fed into two FC layers to predict the vector 𝜶 ∈ R 𝐶 and 𝜷 ∈ R 𝐶 , where 𝐶 is the channel size of the concatenated visual features. Afterward, the vectors 𝜶 and 𝜷 are reshaped to (𝐶, 1) and (1, 𝐶). We perform matrix multiplication between the reshaped 𝜶 and 𝜷 to obtain the predicted 1 × 1 convolutional kernel weights. Finally, we conduct the 1 × 1 convolution process on concatenated visual features based on the predicted kernel weights.As part of the Shared Context Generator, the Intra-SI modules are applied only in the first four SF-BLKs to strike a balance between visual appearance consistency and variance. The proposed Intra-SI module in our CoIn framework facilitates interaction among image generators throughout the story generation process. By incorporating the Intra-SI module, the image generators have the opportunity to exchange visual features among themselves. This exchange of visual features enhances the generated story images, resulting in improved visual appearance consistency. The proposed Intra-SI enables our CoIn to interact across the image generators during the story generation process. As the Shared Context Generator, the Intra-SI modules are only applied in the first 4 SF-BLKs to balance visual appearance consistency and variance. The Intra-SI provides the chance for the image generator to exchange visual features among them and improves generated story images with better visual appearance consistency.To improve the efficiency of adversarial learning and the accuracy of story image quality assessment, we propose the DINO-based Discriminator. We first introduce the DINO-based Story Discriminator. As shown in Figure 5 , the story image sequence is first encoded by the frozen DINOv2-Small Image Encoder. The visual features from the 2 𝑛𝑑 , 7 𝑡ℎ , 9 𝑡ℎ layers in DINOv2-Small are extracted and further analyzed through two residual blocks. Then we concatenate the output visual features and encode them into a more compact visual feature. We also concatenate the text and image vectors and encode them into a more compact vector. Afterward, we replicate the encoded text-image vector and concatenate it with compact visual features. A story adversarial loss is predicted by two convolution layers to evaluate the story quality. Based on the visual understanding ability of pre-trained DINOv2 [25] , the DINO-based Story Discriminator can assess the story quality more accurately. The DINO-based Image Discriminator shares a similar structure but differs in how it processes the input features. Rather than concatenating the visual features of the same story as the whole story visual feature, it concatenates the visual features of each image with their corresponding textual features. This allows the discriminator to assess the quality of each image.The DINO-based Discriminator is partly inspired by GALIP [39] which employs the pretrained CLIP-ViT [28] to extract visual features in the image discriminator. In this work, we extend it and propose the DINO-based Story and Image Discriminator to assess the story quality. Compared with CLIP-ViT which only utilizes highlevel contrastive pretraining, DINOv2 incorporates an additional mask to reconstruct masked parts. It enables DINOv2 to extract richer and more detailed visual features. However, we observed that directly using the DINOv2 model resulted in the generation of patch-like patterns in the synthesized images, as shown in Figure 8 (a). These patch-like patterns were consistent with the size of the DINOv2 input (16, 16) . When we replaced DINOv2 with CLIP-ViT-B/16, the patch-like patterns disappeared. We suspect that this issue may be related to the masked pretraining employed by DINOv2, which might not provide sufficient supervision for smooth transitions between adjacent image patches. Consequently, the generator may have learned to generate adversarial features. To address this issue, we propose the Slight Shift trick. Before feeding the images into DINOv2, we slightly enlarge them and randomly crop an image corresponding to the resolution of DINOv2's input. This process helps prevent the generator from accurately identifying the patch intervals, thereby avoiding the generation of patch-like patterns.We follow previous text-to-image models [39, 40] and employ the hinge loss [44] and MA-GP [39, 40] to stabilize the adversarial training process. Finally, the whole formulation of our CoIn is shown as follows:EQUATIONwhere 𝑧 
experiments 4.1 datasets: 
 We evaluate our approach on two challenging datasets: Pororo-SV [14] and Flintstones-SV [10] . Our method is developed using PyTorch. The resolution of the output images on Pororo-SV and Flintstones-SV is 256 × 256. The network is trained 200 epochs and 150 epochs on Pororo-SV and Flintstones-SV. We use the Adam optimizer [15] with 𝛽 1 =0.0 and 𝛽 2 =0.9 to train our model. We set the learning rate 0.0001 for the generator and 0.0004 for the discriminator. The hyper-parameters of the generator 𝑀 and 𝑁 are set to 6 and 3. The hyper-parameters of the objective function 𝑘, 𝑝, 𝜆, 𝛾, are set to 2, 6, 0.5, and 0.5. All models were trained on 8 × A6000 GPUs. Following the previous story visualization works [16, 22, 26] , we adopt the Fréchet Inception Distance (FID) [11] to evaluate the image quality of synthesized story images, and adopt the Fréchet Story Distance (FSD) [16, 38] to evaluate the consistency of the story image sequence. To compare the efficiency of different models, we measure the generation speed and the number of parameters in the story generator. The generation speed is evaluated on a single A6000 GPU.To evaluate the performance of our proposed CoIn, we compare it with several state-of-the-art story visualization and continuation methods [16, 20, 22, 26, 38] in Table 1 and Table 2 . From Table 1 and Table 2 , we can observe that our CoIn achieves promising FID and FSD with fast generation speed and a small number of parameters. Compared with WL-SV [16] which is the best model based on the independent framework, our CoIn significantly improves the FID and FSD of Pororo-SV and Flintstones-SV. Compared with StoryDALL-E [22] , which utilizes large pre-trained DALL-E models and synthesizes story images in an autoregressive manner, our CoIn framework achieves superior results despite having significantly smaller generator parameters and faster synthesis speed. This demonstrates the effectiveness and efficiency of our approach in generating high-quality story images. Compared with AR-LDM [26] which is an autoregressive model based on the powerful stable diffusion [33] , our CoIn has a significantly smaller number of generator parameters but still achieves a competitive performance. models, respectively. As shown in Figure 6 , the characters synthesized by WL-SV [16] contain broken or incorrect shapes. However, the story images synthesized by our CoIn have the correct shapes and clear backgrounds. Compared with StoryDALL-E [22] in Figure 7 , our CoIn can synthesize a more consistent character appearance. Although our CoIn has achieved better image quality than StoryDALL-E, we have observed some limitations in terms of generalization compared to AR-LDM [26] . Due to the powerful Stable Diffusion with large-scale pretraining, AR-LDM can generate uncommon content not frequently seen in the training set. However, it is worth noting that our CoIn still achieves comparable results to AR-LDM in certain scenarios with ∼360×faster synthesis speed and 4.6% generator parameters.To verify the effectiveness of different components in the proposed CoIn, we conduct ablation studies on the story visualization and continuation tasks, respectively. The results of Pororo-SV and Flintstones-SV are shown in Intra-SI. We also compare the results of CLIP-based D and DINObased D. Our baseline is a modified text-to-image DF-GAN [40] adapted for story visualization tasks. The baseline fuses the story information through FC layers. From Table 3 , we can observe that our proposed DINO-based D achieves better performance than CLIP-based D on these two tasks. If we further introduce the Context G to decompose the generation task, we can observe a further improvement of FID and FSD. Armed with Contextual SVF, the model also decreases FID and FSD from 24.39 and 42.56 to 19.38 and 38.19 for the story visualization task on Pororo. The proposed Intra-SI further decreases FID and FSD on two datasets. The ablation studies demonstrate the effectiveness of our proposed modules in both story visualization and continuation tasks. Furthermore, we compare the differences in generated images between the DINO-based D with and without Slight Shift in Figure 8 . It is evident that Slight Shift effectively alleviates the issue of generating patch-like patterns. 
conclusion: 
 In this paper, we propose a lightweight and effective CoIn (Contextualize and Interchange) framework for story visualization and continuation. Moreover, we propose a Context-aware Story Generator that explicitly extracts the global context information and synthesizes context visual features for each image generator. Furthermore, we propose an Intra-story Interchange module to enable information exchange between intra-story image generators. Lastly, a pair of DINO-based discriminators is introduced to assess the story quality more accurately. Our CoIn achieves competitive synthesis quality with ∼360×faster synthesis speed and 4.6% generator parameters. Our CoIn also significantly reduces the required resources and training time. This is particularly crucial for tasks like story visualization that involve generating a large number of images. 
: 
 always employ an independent generator design. (b) Recent models [22, 26] employ large pretrained generative models and synthesize story images frame-by-frame. (c) Our CoIn proposes a novel contextualize and interchange story generator to share the global context visual semantics and interchange local visual features during the synthesis process.Existing story visualization models [2, 18, 20, 21, 26, 29, 38] have significantly advanced in generating story images. However, they still face difficulties in achieving both high-quality image generation and maintaining lightweight models with a fast generation speed. The existing frameworks for story visualization exhibit inherent limitations that hinder their effectiveness and efficiency. The independent framework [16, 18, 20, 21, 38] , which prioritizes generation speed, compromises image quality due to its reliance on independent generators and basic Generative Adversarial Networks (GANs). As depicted in Figure 1 (a), in the independent framework, the fusion of textual information for each frame occurs only before it is passed to the image generator. The image generator operates independently during the image generation process without any information exchange, thereby increasing the difficulty of synthesizing coherent story images. Conversely, the autoregressive framework [22, 26, 29] utilizes large pretrained text-to-image models [31, 32, 34] , transforming them into an auto-regressive approach for generating story images (see Figure 1(b) ). It incorporates an additional history module to retain and encode the generated historical frames. However, as shown in Figure 2 , this approach results in significantly larger model sizes, high resource requirements for training and inference, and the frame-by-frame design further slows the generation speed. Moreover, both independent and autoregressive frameworks do not distinguish between local story and global context information. Consequently, the image generator needs to implicitly decode the entangled local story and global context features and synthesize the corresponding visual features Figure 2 : Comparing with the state-of-the-art models on Pororo-SV [14] . Our CoIn achieves comparable image quality metrics, such as FID and FSD, to the large pretrained text-toimage model-based autoregressive framework (StoryDALL-E [22] , AR-LDM [26] ), while preserving the fast speed, small model parameters, and low resource requirements of the independent framework (WL-SV [16] ).simultaneously. This diminishes the model's learning efficiency and compromises the quality and consistency of the generated images.To achieve high-fidelity and coherent story image generation while maintaining reasonable computational requirements and generation speed, we propose the CoIn (Contextualize and Interchange) for story visualization and Continuation. As shown in Figure 2 , compared to WL-SV [16] which is based on the independent framework, our CoIn boosts the image quality significantly while keeping the generation speed and parameters of the story generator. Compared with StoryDALL-E [22] and AR-LDM [26] which adopt pretrained text-to-image model-based autoregressive framework, our CoIn achieves competitive results with ∼1000× and ∼360×faster synthesis speed and 5.0% and 4.6% generator parameters.With careful consideration of the characteristics of story visualization and continuation tasks, we develop the novel CoIn framework tailored to meet their specific requirements. Unlike previous models, our CoIn distinguishes between local story and global context to ensure the overall coherence of generated story images. As depicted in Figure 1 Overall, our contributions can be summarized as follows:• We introduce a fast, lightweight, and effective framework for story visualization and continuation to synthesize highquality and coherent story images.  
 
 Appendix:  


Decision: The paper introduces CoIn, a framework that successfully balances lightweight efficiency and high-quality image generation for story visualization and continuation.


Reasons to accept: 
The paper presents effective module design for improving generation efficiency. Quantitative and qualitative evaluation support the effectiveness of the proposed approach.


Reasons to reject:
One major concern is on the writing and presentation, detailed in Reviewer FPNu’s review and response to rebuttal.
The paper is recognized for its speed advantage, although its performance is comparable to or even inferior to existing methods like AR-LDM (in WACV24, 2024/01/04). (Reviewer xhrW)
Reviewers also express concerns on the framework impact and novelty as it is limited to improve the efficiency on the specific task. 


The paper receives the initial rating of weak accept, borderline reject, and reject. After rebuttal and extensive discussions, reviewers raise the rating to weak accept, borderline accept, and weak reject. After reading the paper, review comments, and discussion, AC agrees that the paper has merit and recommends accept, **with the condition that** authors incorporate the content from the rebuttal into the final version, especially the great suggestions from Reviewer FPNu on clarifying the paper presentation (in both original review and response to the rebuttal). The authors should not include the incorrect claims into the final version, such as [26] is published on 2024/04/09 while WACV24 is held on 2024/01/04.

Reviews: review: paper_summary: paper_summary: The paper presents CoIn, a lightweight and efficient framework for story visualization and continuation, designed to address the challenges of generating high-quality images while maintaining model efficiency. CoIn introduces three key components: a Context-aware Story Generator to capture global context information, an Intra-Story Interchange module for visual feature exchange among image generators, and DINO-based discriminators for accurate quality assessment of story images. CoIn achieves competitive image quality with significantly faster generation speeds and smaller model sizes compared to existing methods.
strength_weakness: strength_weakness: 1. Novelty: The introduction of the Context-aware Story Generator and the Intra-Story Interchange module are quite innovative. 
2. Solid Evaluation: The paper provides thorough quantitative and qualitative evaluations, including comparisons with state-of-the-art models and ablation studies that validate the effectiveness of each proposed component.
3. Promising Results: The result looks promising both in terms of generation quality and generation speed, implying the practicality of this work.
limitations: limitations: 1. Would be nice to include user study as story continuation and visualization are quite subjective tasks
paper_summary: paper_summary: This paper proposes a lightweight and effective framework for story visualization. It introduces a context-aware story generator, and incorporates DINOv2 as the discriminator to assess the image discriminators.
strength_weakness: strength_weakness: - The paper presents a lightweight framework with fast processing speed.
limitations: limitations: - The writing in the paper requires significant improvement. For instance, the sentences in Lines 89-95 convey the same meaning, leading to redundancy. Additionally, similarities at the beginning of the third and fourth paragraphs indicate repetition of content.
- There are clarity issues with some figures. For example, Lines 326-328 mention Figure 3 along with the Context-aware SG and Contextual SVF, but the placement of these modules in the figure is unclear.
- The names and abbreviations of certain modules can be confusing, such as Context-aware SG, Contextual SVF, Contextual Story-Visual Fusion module, Shared Context Generator, Context G.
- The distinction between the contextual story-visual fusion and the story fusion module is unclear. The contribution of the contextual story-visual fusion module is not clearly articulated and seems to involve the use of CLIP, which may not be novel.
- The meaning of the context information in Lines 423-425 and how it is extracted using Contextual SVF are not explicitly stated.
- The choice of baselines for presenting qualitative results may not be up-to-date.
- The ablation study lacks clarity. It should involve adding each sub-module separately. The interpretation of terms like "+DINO-based D" and "Contextual SVF" is ambiguous and may lead to confusion in understanding the experimental setup. Does “+DINO-based D” mean +CLIP-based D+DINO-based D? Does “Contextual SVF” mean +CLIP-based D+DINO-based D+ Contextual G+ Contextual SVF?
paper_summary: paper_summary: The article introduces a lightweight and effective framework, CoIn, for Story Visualization and Continuation. This framework utilizes a GAN-based approach, extracts global context information through a Context-aware Story Generator, and visual information through an Intra-story Interchange module. It also employs DINO-based discriminators to enhance GAN training. As the article demonstrates, this method can achieve impressive visual results while significantly accelerating compared to methods with similar performance.
strength_weakness: strength_weakness: 1. Motivation is clear.
2. The overall paper is easy to follow.
limitations: limitations: 1. Could you provide more visual results compared to AR-LDM? Their visual performance and diversity are obviously better than those of CoIn, e.g., in Fig.6 and Fig.7.
2.  Could you compare the results on the StorySalon dataset with StoryGen [19]? I believe that more experiments on various datasets can support the effectiveness of your method.

I will raise my score if there are more results presented to support the CoIn.