StoryGPT-V: Large Language Models as Consistent Story Visualizers

## Abstract

Abstract Recent generative models have demonstrated impressive capabilities in generating realistic and visually pleasing images grounded on textual prompts. Nevertheless, a significant challenge remains in applying these models for the more intricate task of story visualization. Since it requires resolving pronouns (he, she, they) in the frame descriptions, i.e., anaphora resolution, and ensuring consistent characters and background synthesis across frames.
Yet, the emerging Large Language Model (LLM) showcases robust reasoning abilities to navigate through ambiguous references and process extensive sequences. Therefore, we introduce StoryGPT-V , which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters grounded on given story descriptions.
First, we train a character-aware LDM, which takes character-augmented semantic embedding as input and includes the supervision of the cross-attention map using character segmentation masks, aiming to enhance character generation accuracy and faithfulness.
In the second stage, we enable an alignment between the output of LLM and the character-augmented embedding residing in the input space of the first-stage model. This harnesses the reasoning ability of LLM to address ambiguous references and the comprehension capability to memorize the context. We conduct comprehensive experiments on two visual story visualization benchmarks. Our model reports superior quantitative results and consistently generates accurate characters of remarkable quality with low memory consumption. Our code will be made publicly available 1 1 1 Please refer to the webpage for qualitative results. .

## 1 Introduction

Image generation algorithms have made significant strides and are on the verge of matching human-level proficiency.
Despite this progress, even a powerful image generator suffers from story visualization task, which involves generating a series of frames that maintain semantic coherence based on narrative descriptions [, , , ]. This challenge arises from the fact that captions for a single image are typically self-sufficient, lacking the continuity needed to capture the narrative of object interactions that unfold through multiple sentences over a sequence of frames. This poses a promising avenue for further research and exploration of story visualization. Such a task demands a model capable of producing high-quality characters and detailed environmental objects grounded on given text descriptions. Moreover, it requires the ability to disambiguate referential pronouns in the subsequent frame descriptions, e.g., “she, he, they”.

Prior studies [, , , , ] explore the realm of story visualization but do not take reference resolution [] (i.e., anaphora resolution in the context of natural language processing [, ]) into consideration. Story-LDM [] first extended story visualization benchmarks with referential text and devises an attention memory module that retains visual context throughout the series of generated frames. However, it still struggles to generate precise characters for referential text since the interaction between current descriptions and contextual information occurs within the CLIP [] semantic space, causing a loss in fine-grained language understanding and hindering referencing capabilities. Furthermore, the attention memory module requires maintaining all previous images in latent pixel space for attention calculations, significantly increasing memory demands with each additional frame in autoregressive generation.

The limitation of previous works leads us to rethink how to achieve accurate and efficient reference resolution toward consistent story visualization. Large Language Models (LLMs) [, , , ], trained on extensive text corpora, have exhibited impressive capabilities in deciphering contextual references in natural language descriptions. Prior works [, ] have demonstrated the effectiveness of harnessing LLMs for tasks involving image comprehension and generation, where the visual features are adapted within LLM’s token space rather than the pixel space. Hence, such a model could be utilized to efficiently address ambiguous references for story visualization tasks.

In this work, we aim at story visualization grounded on given co-referential frame descriptions, focusing on delivering high-quality and coherent portrayals of characters.
To achieve this, we leverage a powerful text-to-image model [] to generate high-quality characters and environmental objects grounded on given frame descriptions, coupled with the reasoning ability of Large Language Models (LLMs) to resolve ambiguous references and improve the cohesiveness of the context. To improve the generation of highly faithful characters, we enhance the pre-trained Latent Diffusion (LDM) towards character-aware training in the first stage. We first augment the token feature by incorporating the visual representation of the corresponding character. Additionally, we regulate the cross-attention map of the character token to highlight the interaction between the conditional token and specific latent pixels.

In addressing the challenge of ambiguous reference, which cannot be effectively handled by a robust text-to-image model alone, we leverage an LLM that takes interleaved images and co-referential frame descriptions as input, and aligns its visual output with the character-augmented embedding encoded by first-stage model. Such semantic guidance, along with LLM’s casual modeling, enables effective reference resolution and consistent generation. Furthermore, our approach efficiently preserves context by processing images as sequences of tokens in the LLM input space with low memory consumption.

Contributions. Our contributions are as follows:

- •
We first enhance the text representation by integrating the visual features of the corresponding characters, then refine a character-aware LDM for better character generation by directing cross-attention maps with character segmentation mask guidance.
- •
We adapt LLM by interlacing text and image inputs, empowering it to implicitly deduce references from previous contexts and produce visual responses that align with the input space of the first-stage Char-LDM. This leverages the LLM’s reasoning capacity for reference resolution and the synthesis of coherent characters and scenes.
- •
Our model is capable of visualizing stories featuring precise and coherent characters and backgrounds on story visualization benchmarks. Furthermore, we showcase the model’s proficiency in producing extensive (longer than 40 frames) visual stories with low memory consumption.

## 2 Related Work

Text-to-image synthesis. Numerous works [, , ] have demonstrated unprecedented performance on semantic generation. Recently, diffusion-based text-to-image models [, , , ] have shown significant advancements in enhancing image quality and diversity through the utilization of diffusion models. However, these text-to-image approaches primarily concentrate on aligning individual-generated images grounded on text descriptions and do not take into account the crucial aspects of character and scene consistency across multiple frames in the story visualization task. Additionally, they lack the capability to effectively resolve co-reference issues within a narrative description.

Multi-modal Large Language Models. Large Language Models (LLMs) wield an extensive repository of human knowledge and exhibit impressive reasoning capabilities. Recent studies [, , , ] utilize pre-trained language models to tackle vision-language tasks, and subsequent studies [, , , , , ] further enhance multi-modal abilities by aligning vision models with LLM input space. In addition to multi-modal comprehension, several works are dedicated to more challenging multi-modal generation tasks. FROMAGe [] appends a special retrieval token to LLM and maps the hidden representation of this token into a vector space for retrieving images. Several current works [, , ] learn a mapping from hidden embeddings of an LLM represents for additional visual outputs into the input space of a frozen pre-trained text-to-image generation model []. In this work, we fed multi-modal LLM with interleaved image and referential text descriptions as input and aligned the output with a character-aware fused embedding from our first-stage Char-LDM, guiding the LLM in implicitly deducing the references.

Story Visualization. StoryGAN [] pioneers the story generation task, which proposes a sequential conditional GAN framework with dual frame and story level discriminators to improve image quality and narrative coherence. DuCoStoryGAN [] introduces a dual-learning framework that utilizes video captioning to enhance semantic alignment between descriptions and generated images. VLCStoryGAN [] used video captioning for semantic alignment between text and frames. Recently, StoryDALL-E [] retrofits the cross-attention layers of the pre-trained text-to-image model to promote generalizability to unseen visual attributes of the generated story. These methods do not consider ambiguous references in text descriptions. Story-LDM [] first introduced reference resolution in story visualization tasks and proposed an autoregressive diffusion-based framework with a memory-attention module to resolve ambiguous references. Nevertheless, it struggled with accurately resolving references and was memory-intensive, as it required retaining all previous context in pixel space. In our work, we employ a powerful causal inference LLM for reference resolution, and it efficiently maintains context by mapping visual features into several token embeddings as LLM inputs rather than operating in latent pixel space.

Figure: Figure 2: (a) In the first stage, a fused embedding is created by integrating character visuals with text embeddings, serving as the Char-LDM’s conditional input, and the cross-attention maps of Char-LDM will be guided by corresponding character segmentation mask for accurate and high-quality character generation (Section [3.2](#S3.SS2)). (b) In the second stage, the LLM takes the interleaved image and text context as input and generates $R$ [IMG] tokens. These tokens are then projected by LDM Mapper into an intermediate output, which will be encouraged to align with fused embedding as Char-LDM’s input. The figure intuitively shows how the character-augmented fused embedding and the casual language modeling aid LLM for reference resolution (Section [3.3](#S3.SS3)).
Refer to caption: /html/2312.02252/assets/x2.png

## 3 Methods

The objective of story visualization is to transform a textual narrative, composed of a series of $N$ descriptions ${S_{1},...S_{N}}$, into a sequence of corresponding visual frames ${I_{1},...,I_{N}}$ that illustrate the story. We’ve developed a two-stage method aimed at generating temporally consistent visual stories with accurate and high-quality characters. First, we augment text representation with characters’ visual features and refine a character-aware LDM [] (Char-LDM) towards high-quality character generation. This is achieved by directing the cross-attention maps of specific tokens associated with the corresponding characters, using character segmentation mask supervision (Section [3.2](#S3.SS2)). Then, we leverage the reasoning ability of LLM to resolve ambiguous references by aligning the output of LLM with Char-LDM input space for temporal consistent story visualization (Section [3.3](#S3.SS3)).

### 3.1 Preliminaries

Cross-attention in text-conditioned Diffusion Models. In diffusion models [, ], each diffusion step $t$ involves predicting noise $\epsilon$ from the noise code $z_{t}\in\mathbb{R}^{(h\times w)\times d_{v}}$ conditioned on text embedding $\psi(S)\in\mathbb{R}^{L\times d_{c}}$ via U-shaped Network [], where $\psi$ is the text encoder, $h$ and $w$ are the latent spatial dimensions and $L$ is the sequence length. Within U-Net, the cross-attention layer accepts the spatial latent code $z$ and the text embeddings ${\psi}(S)$ as inputs, then projects them into $Q=W^{q}z$, $K=W^{k}{\psi}(S)$ and $V=W^{v}{\psi}(S)$, where $W^{q}\in\mathbb{R}^{d_{v}\times d^{\prime}}$, $W^{k},W^{v}\in\mathbb{R}^{d_{c}\times d^{\prime}}$. The attention scores is computed as $A=\mathrm{Softmax}(\frac{QK^{T}}{\sqrt{d^{\prime}}})\in\mathbb{R}^{(h\times w)\times L}$, where $A[i,j,k]$ represents the attention of $k$-th text token to the $(i,j)$ latent pixel. In this context, each entry $A[i,j,k]$ within the cross-attention map $A$ quantifies the magnitude of information propagation from the $k$-th text token to the latent pixel at position $(i,j)$. This feature of the interaction between semantic representation and latent pixels is harnessed in various tasks such as image editing [, ], video editing [], and fast adaptation [, , , ].

### 3.2 Character-aware LDM with attention control

Integrate visual features with text conditions. To achieve accurate and high-quality characters in story visualization, we augment text descriptions with visual features of corresponding characters and guide the attention of text conditions to focus more on corresponding character synthesis. Given a text description $S$, suppose there are $K$ characters that should be generated in image $I$, images of those characters {$I_{c}^{1},...,I_{c}^{K}$}, a list of token indices indicating each character name located in the description, denoted as $\{i_{c}^{1},...i_{c}^{K}\}$. Inspired by [, , ], we first utilize CLIP [] text encoder ${\psi}$ and image encoder $\phi$ to obtain text embedding and visual features of the characters appear in the image respectively. Then, we augment the text embedding if the token represents a character name. More specifically, we concatenate the token embedding and the visual features of the corresponding character and feed them into an MLP to obtain the augmented text embedding. Each augmented token embedding in the augmented embedding $c$ is formulated as below:

$$ $c^{k}=\textrm{MLP}\left(\mathrm{concat}\left(({\psi}(S[i_{c}^{k}]),\phi(I_{c}^{k})\right)\right)$ (1) $$

where $i_{c}^{k}$ refers to the index of the text token for character $k$, and $I_{c}^{k}$ the image corresponding to character $k$. The embeddings for tokens in $c$ that are unrelated to the character remain identical to the vanilla CLIP token embeddings. The enhanced embedding $c$ is then employed as supervision for the second-stage training, which will be further detailed in Section [3.3](#S3.SS3), where $c_{1},...c_{N}$ are the corresponding augmented embeddings for $S_{1},...,S_{N}$.

Controlling attention of text tokens. Previous work [] has demonstrated that the visual characteristics of generated images are influenced by the intricate interplay between latent pixels and text embedding through the diffusion process of LDM []. However, in vanilla LDM [], a single latent pixel can unrestrictedly engage with all text tokens. Therefore, we introduce a constraint to refine this behavior and strengthen the impact of the token representing the character’s name on certain pixels in the denoising process, as illustrated in Figure [2](#S2.F2) (a). First, we obtain an offline segmentation mask of corresponding characters denoted as $\{M_{1},...M_{K}\}$ as supervision signals via SAM []. We then encourage the cross-attention map $A_{k}$ for each character $k$ at the token index position $i_{c}^{k}$, to align with the binary segmentation mask $M_{k}$, whereas diverging from irrelevant regions $\bar{M}_{k}$, formulated as follows:

$$ $\mathcal{L}_{\textrm{reg}}=\frac{1}{K}\sum_{k=1}^{K}(A_{k}^{-}-A_{k}^{+})$ (2) $$

where

$$ $A_{k}^{-}=\frac{A_{k}\odot\bar{M}_{k}}{\sum_{i,j}(\bar{M}_{k})_{ij}},\,\,A_{k}^{+}=\frac{A_{k}\odot M_{k}}{\sum_{i,j}(M_{k})_{ij}}$ (3) $$

where $K$ is the number of characters to be generated in the image, $i_{c}^{k}$ is the index of text token representing character $k$ and $\odot$ is the Hadamard product.
By reducing the loss, it increases the attention of character tokens to the relevant pixels of their respective characters, while reducing their attention to irrelevant areas. Moreover, as the token embeddings are enriched with the visual features of the corresponding character, this attention control serves to deepen the connection between the augmented semantic space and latent pixel denoising, which can consequently enhance the quality of synthesized characters.

Our first stage Char-LDM focuses solely on the quality of image generation grounded on a single caption. Yet, there remain challenges that surpass the abilities of text-to-image generators in visualizing a sequence of stories. Firstly, story visualization demands character and background consistency, an aspect not covered by our first-stage enhancements. Moreover, the inherent nature of lengthy descriptions includes referential terms like he, she, or they, which presents a significant challenge for LDM in achieving accurate inference. In contrast, LLMs can adeptly infer the intended character to which the ambiguous text refers. Therefore, to address this issue, we harness the formidable reasoning capabilities of LLM to disambiguate such references.

### 3.3 Aligning LLM for reference resolution

To enable an LLM to autoregressively generate images conditioned on prior context and resolve ambiguous references, the model must be capable of (i) processing images; (ii) producing images; and (iii) implicitly deducing the subject of reference. The model could understand the image by learning a linear mapping from the visual feature to the LLM input space, and generate images by aligning the hidden states with conditional input required by LDM, which is the fused embedding encoded by first-stage Char-LDM’s text and visual encoder. It integrates the visual features of characters into the text embedding. This character-augmented embedding, along with the causal language modeling (CLM) [, , ] will direct the LLM to implicitly deduce and generate the correct character for the referential input, as depicted in Figure [2](#S2.F2) (b).

More specifically, the LLM input consists of interleaved co-referential text descriptions and story frames with flexible frame length $n$, in the order of $(I_{1},S_{1},...,I_{n-1},S_{n-1},S_{n})$, where $2\leq n\leq N$. We first extract visual embeddings $\phi(I_{i})\in\mathbb{R}^{d_{i}}$ with CLIP [] visual backbone, where $i\in[2,n]$, and learn $\texttt{Mapper}_{LLM}$ with trainable matrix $\mathbf{W}_{v2t}\in\mathbb{R}^{d_{i}\times me}$ which maps $\phi(I_{i})$ into $m$ $k$-dimensional embeddings reside within LLM input space [, , ], where $e$ is the dimension of LLM embedding space. Additionally, like recent works [, , ] in enabling LLM to generate images, we add additional $R$ tokens, denoted as [IMG1], …, [IMGR] to represent visual outputs and incorporate trainable matrix $\mathbf{W}_{gen}\in\mathbb{R}^{R\times e}$ into frozen LLM. The training objective is to minimize the negative log-likelihood of producing [IMG] tokens conditioned on previously interleaved image/text tokens $\mathcal{T}_{prev}$:

$$ $\mathcal{L}_{gen}=-\sum_{r=1}^{R}\log p(\texttt{[IMG${}_{r}$]}|\mathcal{T}_{prev},\texttt{[IMG${}_{<r}$]})$ (4) $$

where

$$ $\mathcal{T}_{prev}=\{\phi(I_{<i})^{T}\mathbf{W}_{v2t},\psi(S_{1:i})\}$ (5) $$

where $i\in[2,n]$ is the number of text descriptions of the current step.
To align [IMG] produced by LLM with LDM input space, we utilize a Transformer-based $\texttt{Mapper}_{LDM}$ to project [IMG] tokens to the input space of first-stage finetuned LDM with $L$ learnable query embeddings $(q_{1},...,q_{L})\in\mathbb{R}^{L\times d}$, where $L$ is the maximum input sequence length of the LDM, similar to BLIP-2 Q-Former []. The training objective is to minimize the distance between $\mathrm{Mapper}$’s output Gen Emb and the augmented conditional text representations of LDM, i.e., Fuse Emb introduced in Section [3.2](#S3.SS2), formulated as:

$$ $\mathcal{L}_{\textrm{align}}=||\texttt{Mapper}_{LDM}\left(h_{\texttt{[IMG${}_{1:R}$]}},q_{1},...q_{L}\right)-c_{i}||_{2}^{2}$ (6) $$

where $h_{[\texttt{IMG}_{1:R}]}$ denotes the last hidden states of LLM’s [IMG] tokens. Suppose we can get access to the original text without reference $S_{i}^{{}^{\prime}}$. Then, $c_{i}$ is the augmented text embedding of caption $S_{i}^{{}^{\prime}}$ encoded by the first-stage model’s text and visual encoder. For instance, if $S_{i}$ is ”They are talking to each other,” then $S_{i}^{{}^{\prime}}$ would be ”Fred and Wilma are talking to each other.” This non-referential text, augmented with character visual features, assists LLM in efficiently disambiguating references using casual language modeling.

Inference. During the inference process, the model sequentially visualizes stories grounded on text descriptions. It begins by processing the text description of the initial frame $S_{1}$. Focusing exclusively on frame generation, we constrain the LLM to generate only $R$ specific [IMG] tokens and then feed these token embeddings into the first-stage Char-LDM, resulting in the generation of the first frame $I_{1}^{gen}$. Subsequently, the LLM takes a contextual history that includes the text description of the first frame $S_{1}$, the generated first frame $I_{1}^{gen}$, and the text description of the second frame $S_{2}$ as input. This process is repeated to visualize the entire story progressively.

Table: Table 1: Main experiments on FlintStonesSV []. The top portion is evaluated on the dataset w/o extended referential text. The bottom half displays the results on the extended dataset with co-reference. ^†StoryDALL-E [] takes the source frame as additional input.

## 4 Experiments

### 4.1 Experimental Setups

Datasets.
Our experiments are conducted using two story visualization datasets: FlintstonesSV [] and PororoSV []. FlintstonesSV [] contains 20132-training, 2071-validation, and 2309-test stories with 7 main characters and 323 backgrounds, while PororoSV [] consists of 10,191 training samples, 2,334 for validation, and 2,208 for testing with 9 main characters. We follow [] to extend the datasets with referential text, by replacing the character names with references, i.e., he, she, or they, wherever applicable. Please refer to the supplementary for details.

Evaluation metrics.
To measure the accuracy of the characters and background in the generated stories, we consider the following evaluation metrics the same as previous story visualization literature [, , ]: Following [], we finetune Inception-v3 to measure the classification accuracy and F1-score of characters (Char-Acc, Char-F1) and background (BG-Acc, BG-F1) respectively. In addition, we consider the Frechet Inception Distance (FID) score, which compares the distribution between feature vectors from real and generated images for quality assessment.

When assessing text-image alignment, the CLIP [] score falls short in reliability since it cannot capture fine-grained details. Therefore we choose the powerful captioning model BLIP2 [] as the evaluation model and fine-tune it on the corresponding datasets. We then employ it as a captioner to predict 5 captions for generated images and 5 captions for ground truth images as a comparison to report the average BLEU4 [] and CIDEr [] score to assess text-image alignment.

Comparison Approaches. We compare our model with state-of-the-art approaches: VLCStoryGAN [], StoryDALL-E [], LDM [] and Story-LDM []. Following previous research [, ], we use 4 consecutive frames for evaluation. For StoryDALL-E [], which takes both story descriptions and the initial frame as input, we use the first frame of a 5-frame story and evaluate using the generated 4 frames. We finetune vanilla Stable Diffusion (LDM) on FlintStonesSV [] and PororoSV [] as a baseline. Since Story-LDM [] does not provide pre-trained checkpoint or cleaned training code, we initiate training from pre-trained LDM(^2^22[https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large](https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large)).

Implementation Details. For the first stage training, we freeze CLIP [] text encoder and fine-tune the remaining modules for 25k steps with a learning rate of 1e-5 and batch size of 32 on original non-referential text. To enhance inference time robustness and flexibility, we adopt a training strategy that includes $10\%$ unconditional training, i.e., classifier-free guidance [], $10\%$ text-only training, and $80\%$ character-augmented fuse training (Section [3.2](#S3.SS2)).

For the second stage training, we use OPT-6.7B(^3^33[https://huggingface.co/facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b)) model as the LLM backbone. To expedite the second stage alignment training, we first pre-compute non-referential fused embeddings residing in the input space of the first-stage Char-LDM. We map visual features into $m=4$ token embeddings as LLM input, set the max sequence length as 160 and the number of additional [IMG] tokens represents for LLM’s visual output as $R=8$, batch size as 64 training for 20k steps. Please refer to the supplementary for more details.

### 4.2 Visual Story Generation

Figure: Figure 3: Qualitative comparison on FlintStonesSV [] (left) and PororoSV [] (right) with co-reference descriptions.
Refer to caption: /html/2312.02252/assets/x3.png

Quantitative Results. (i) Generation with original descriptions. The upper half of Table [1](#S3.T1) shows the comparison results on original FlintStonesSV [] without referential text descriptions. Our first-stage Char-LDM exhibits superior performance in generating accurate characters (Char-Acc, Char-F1) and background scenes (BG-Acc, BG-F1), achieving high fidelity (FID), and exhibiting better alignment with given text descriptions (BLEU4 [], CIDEr []).
(ii) Generation with co-referenced descriptions. Table [1](#S3.T1) (bottom) and Table [2](#S4.T2) show the results on extended FlintStonesSV [] and PororoSV [] with co-referential text descriptions [] respectively. By harnessing the merit of reasoning and comprehension abilities of LLM, our model substantially boosts performance in reference resolution compared to baselines, while maintaining a strong text-image alignment grounded in the provided text descriptions.

Table: Table 2: Performance comparison on PororoSV [] with co-referenced descriptions. ^†StoryDALL-E [] takes the source frame as additional input.

Figure: Figure 4: Human evaluation results on FlintStonesSV [] w.r.t visual quality, text-image alignment, character accuracy and temporal consistency.
Refer to caption: /html/2312.02252/assets/x4.png

Qualitative Results. Figure [3](#S4.F3) demonstrates qualitative comparison on FlintStonesSV [] and PororoSV [] with co-reference descriptions. LDM [] could generate high-quality images but struggles to produce correct characters in the presence of reference in the captions. Story-LDM [], despite incorporating an attention-memory module to handle context, fails to produce accurate characters in some frames. In comparison, our model excels at generating frames with pleasing visuals, accurate characters, and maintaining temporal consistency in the background scenes.

Human Evaluation. In addition, we use Mechanical Turk to assess the quality of 100 stories produced by our methods or Story-LDM [] on FlintStonesSV []. Given a pair of stories generated by Story-LDM [] and our model, MTurkers are asked to decide which generated four-frame story is better w.r.t visual quality, text-image alignment, character accuracy, and temporal consistency. Each pair is evaluated by 3 unique workers. In Figure [4](#S4.F4), our model demonstrates significantly better story visualization quality with accurate and temporally coherent synthesis.

### 4.3 Ablation Studies

First stage ablation. We conducted an ablation study for the first stage and presented results in Table [3](#S4.T3). w/o $\mathcal{L}_{reg}$ indicates that we disabled the $\mathcal{L}_{reg}$ loss (Equation [2](#S3.E2)), i.e., the model underwent training without the influence of segmentation masks to direct the cross-attention maps. w/o augmented text signifies that the model’s conditional input during its training phase was the standard CLIP [] text embedding, rather than the fused embedding incorporating the character’s visual attributes as discussed in Section [3.2](#S3.SS2). freeze vis denotes the visual encoder remained frozen during training. Unless specified, the last two layers of the visual encoder are made adjustable. The final two rows employ our default training strategy and the only distinction lies in the inference phase. Default (w/o img) takes vanilla CLIP [] text embedding as input condition, whereas Default (w/ img) employs the fused embedding. As indicated by Table [3](#S4.T3), integrating character visual features during training significantly enhances the generation performance and the additional cross-attention control propels the model to achieve its peak on accurate character generation. Note that the FID score of Default (w/ img) is slightly higher than Default (w/o img). This is because, during inference, the reference images for corresponding characters in Default (w/ img) are obtained online, introducing a slight deviation from the original distribution.

Table: Table 3: Ablation study for the first stage finetuning LDM with cross-attention control.

Second stage ablation. As shown in Table [4](#S4.T4), we conducted an ablation study on (i) whether to align with the text embedding (Embtext) or the fused embedding (Embfuse) of our first stage model; (ii) whether the model’s input consists of a sequence of captions (Caption-) or utilizes interleaved training with both images and captions (Interleave-) (Equation [5](#S3.E5)). Experimental results shown in Table [4](#S4.T4) indicate that image-text interleave training can significantly enhance performance. It is intuitive that taking both images and corresponding captions as input provides a more profound comprehension of the characters and their interactions within the image than when provided with sole captions. This, in turn, amplifies its generative capabilities.

Table: Table 4: Second stage training strategy ablation. Input only caption or interleaved text and image. The output of LLM is aligned with our Char-LDM text embedding (Embtext) or character-augmented fused embedding (Embfuse).

### 4.4 Analysis

We further investigate the impact of first-stage finetuning with cross-attention control by visualizing averaged cross-attention maps in U-Net latent pixel space and interpolating them to match the size of the generated images. As illustrated in Figure [5](#S4.F5), vanilla LDM (top) finetune on FlintStonesSV [] w/o $\mathcal{L}_{reg}$ (Section [3.2](#S3.SS2)) fails to accurately focus on the corresponding characters for character tokens. Our model (bottom), which incorporates cross-attention guidance, is able to precisely direct attention to generated characters given corresponding character tokens.

Figure: Figure 5: Visualization of cross attention maps of corresponding character tokens.
Refer to caption: /html/2312.02252/assets/x5.png

### 4.5 Properties

Our model could generate longer stories featuring accurate characters, at a faster speed and with lower computational consumption. Our architecture allows our model to retain an extensive context requiring minimal computational resources by efficiently mapping visual features into tokens instead of operating in pixel space. Figure [6](#S4.F6) shows the comparison between our model and Story-LDM [] w.r.t GPU memory consumption and inference speed for longer-frames story generation. Our model is capable of producing sequences exceeding 50 frames with low memory usage, whereas Story-LDM [] encounters GPU memory limitations (80G A100) when generating 42 frames. This is because Story-LDM [] requires the retention of the entire context, e.g., $n$ frames in latent pixel space ($n\times h\times w\times d$), whereas our model processes visual features as four token embedding ($n\times 4\times d$) with the same dimensions as the text tokens in LLM. Table [5](#S4.T5) compares the accuracy of generated characters and FID score for long story visualizations between our model and Story-LDM []. The performance of Story-LDM [] significantly decreases when generating longer stories and reaches the memory limit before 50 frames. In contrast, by utilizing the capacity of LLM to retain extensive context, our model upholds accurate character consistency in visualizing lengthy narratives with co-referential text descriptions.

Figure: Figure 6: Compare inference speed and GPU memory consumption between our method and Story-LDM []. Story-LDM encounters the 80GB GPU limit when generating sequences exceeding 40 frames.
Refer to caption: /html/2312.02252/assets/x6.png

Table: Table 5: Longer-frames story visualization comparison on FlintStonesSV [] with referential text. Story-LDM reaches maximum GPU capacity when generating 50 frames.

Our design could be easily adapted to any LLMs. In our work, we experimented with OPT-6.7b(^4^44[https://huggingface.co/facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b)) and Llama2-7b-chat(^5^55[https://huggingface.co/meta-llama/Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat)) models. Our findings, as illustrated in Table [9](#A2.T9), indicate an improvement when changing from OPT [] to a more powerful model, Llama2 [].

Table: Table 6: Performance on FlintstonesSV [] dataset with referential text using different LLMs.

Our model, StoryGPT-V, is capable for multi-model generation. Owing to StoryGPT-V design leveraging the advanced capabilities of Large Language Models (LLMs), it exhibits a unique proficiency in that it can extend visual stories. StoryGPT-V is not merely limited to visualizing stories based on provided textual descriptions. Unlike existing models, it also possesses the innovative capacity to extend these narratives through continuous text generation. Concurrently, it progressively synthesizes images that align with the newly generated text segments.

Our model represents a notable advancement in story visualization, being the first of its kind to consistently produce both high-quality images and coherent narrative descriptions. This innovation opens avenues for AI-assisted technologies to accelerate visual storytelling creation experiences by exploring various visualized plot extensions as the story builds.

## 5 Conclusion

In this paper, we aim at high-quality and consistent character synthesis for story visualization grounded on co-referential text descriptions. To accomplish this, we utilize the strengths of the LDM for generating high-quality images, combined with the reasoning capability of LLM to comprehend extended contexts, resolve ambiguities, and ensure semantic consistency in the generation process. We first finetune LDM by guiding the cross-attention map of LDM with character segmentation masks, which improves the accuracy and faithfulness of character generation. Next, we facilitate a mapping from the output of LLM to align with the input space of the first stage LDM, thus allowing Multi-modal LLM to both process and produce images. This process leverages the LLM’s logical reasoning to clarify ambiguous references and its capacity to retain contextual information. Our model reports superior quantitative results and consistently generates characters with remarkable quality.

## Appendix

Figure: Figure 7: Our model StoryGPT-V extending stories in both language and vision: Gray part represents the text descriptions from datasets. Blue part corresponds to the frames and the continued written stories based on the previous captions generated by our model StoryGPT-V. This is the first model capable of story visualization and multi-modal story generation (continuation) by leveraging an LLM.
Refer to caption: /html/2312.02252/assets/x7.png

## Appendix A Multi-modal Story Generation

Owing to StoryGPT-V design leveraging the advanced capabilities of Large Language Models (LLMs), it exhibits a unique proficiency in that it can extend visual stories. StoryGPT-V is not merely limited to visualizing stories based on provided textual descriptions. Unlike existing models, it also possesses the innovative capacity to extend these narratives through continuous text generation. Concurrently, it progressively synthesizes images that align with the newly generated text segments.

Figure [7](#Ax1.F7) presents an example of a multi-modal story generation. Initially, the first four frames are created according to the text descriptions from the FlintstonesSV [] dataset (gray part). Subsequently, the model proceeds to write the description for the next frame (blue part), taking into account the captions provided earlier, and then creates a frame based on this new description (blue part). This method is employed iteratively to generate successive text descriptions and their corresponding frames.

Our model represents a notable advancement in story visualization, being the first of its kind to consistently produce both high-quality images and coherent narrative descriptions. This innovation opens avenues for AI-assisted technologies to accelerate visual storytelling creation experiences by exploring various visualized plot extensions as the story builds.

## Appendix B Ablation Studies

### B.1 Effect of first-stage design.

In Table [7](#A2.T7) lower half, we conducted an ablation study on how the stage-1 design contributes to the final performance. In the first line, the stage-2 LLM is aligned with vanilla LDM fine-tuned on FlintstonesSV []. The second line aligns the LLM output with our Char-LDM’s text embedding (Embtext), while the last line aligns with character-augmented fused embedding (Embfuse) of our Char-LDM. The first two lines align to the same text embedding encoded by the CLIP [] text encoder, however, our Char-LDM enhanced with cross-attention control ($\mathcal{L}_{reg}$) produces more precise characters. Different from Embtext, the last line is aligned with Embfuse, which is augmented with characters’ visual features. This visual guidance helps LLM to interpret references more effectively by linking “he, she, they” to the previous language and image context.

Table: Table 7: The output of our stage-2 model is aligned with conditional input of vanilla LDM [] (finetuned on FlintstonesSV []), our Char-LDM text embedding (Embtext) or character-augmented fused embedding (Embfuse).

### B.2 Number of [IMG] Tokens

We further examined the impact of the number of added [IMG] tokens. As indicated in Table [8](#A2.T8), aligning with the fused embedding and setting $R=8$ yields the best performance.

Table: Table 8: StoryGPT-V Ablations: Impact of $R$, the number of added [IMG] tokens. Embtext: the output of LLM is aligned with text embedding extracted from the text encoder; Embfuse: aligned with fused embedding Embfuse of first stage model.

### B.3 Different LLMs (OPT vs Llama2)

Table: Table 9: Performance on FlintstonesSV [] dataset with referential text using different LLMs.

Our primary contribution lies in leveraging Large Language Models (LLMs) for reference resolution for consistent story visualization. In our work, we experimented with OPT-6.7b(^6^66[https://huggingface.co/facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b)) and Llama2-7b-chat(^7^77[https://huggingface.co/meta-llama/Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat)) models. It’s important to note that the utilization of Llama2 was specifically to demonstrate its additional capability for multi-modal generation. The ablation study of different LLMs was not the main focus of our research.

Our findings, as illustrated in Table [9](#A2.T9), indicate only a slight improvement when changing from OPT [] to Llama2 []. This marginal difference is attributed to the evaluation metric’s emphasis on image-generation capabilities, which assesses whether the model’s visual output aligns well with first-stage Char-LDM’s conditional input space.

## Appendix C Evaluation

### C.1 Text-image alignment.

CLIP [] is trained on large-scale image-caption pairs to align visual and semantic space. However, a domain gap exists between pre-train data and the story visualization benchmark. Therefore, we finetune CLIP [] on the story visualization datasets. However, we found it still hard to capture fine-grained semantics, either text-image (T-I) similarity or image-image similarity (I-I), i.e., the similarity between visual features of generated images and corresponding ground truth images.

Upon this observation, we choose the powerful captioning model BLIP2 [] as the evaluation model. We finetune BLIP2 on FlintstonesSV [] and PororoSV [], respectively, and employ it as an image captioner for generated visual stories. We avoided direct comparisons to bridge the gap between BLIP2’s predictions and the actual ground truth captions. Instead, we used the fine-tuned BLIP2 to generate five captions for each ground truth image and one caption for each generated image. and report average BLEU4 [] or CIDEr [] score based on these comparisons.

Table: Table 10: Text-image alignment score for FlintstonesSV [] with referential text descriptions in terms of CLIP [] similarity, BLEU4 [] and CIDEr [].

### C.2 Human evaluation.

We use Mechanical Turk to assess the quality of 100 stories produced by our methods or Story-LDM [] on FlintStonesSV []. Given a pair of stories generated by Story-LDM [] and our model, people are asked to decide which generated four-frame story is better w.r.t visual quality, text-image alignment, character accuracy and temporal consistency. Each pair is evaluated by 3 unique workers. The human study interface is illustrated in Figure [8](#A3.F8).

Figure: Figure 8: Human study interface.
Refer to caption: /html/2312.02252/assets/x8.png

## Appendix D Implementation Details

### D.1 Data preparation

FlintstonesSV [] provides the bounding box location of each character in the image. We fed the bounding boxes into SAM [] to obtain the segmentation map of corresponding characters. This offline supervision from SAM [] is efficiently obtained without the need for manual labeling efforts. Furthermore, we enhance the original datasets from resolution of 128x128 to 512x512 via a super-resolution model(^8^88[https://huggingface.co/CompVis/ldm-super-resolution-4x-openimages](https://huggingface.co/CompVis/ldm-super-resolution-4x-openimages)) and then we proceed to train and evaluate all models on this enhanced dataset.

### D.2 Extending dataset with referential text

We follow Story-LDM [] to extend the datasets with referential text by replacing the character names with references, i.e., he, she, or they, wherever applicable as shown in Algorithm [1](#alg1). The statistics before and after the referentail extension are shown in Table [11](#A4.T11). Please refer to Story-LDM [] implementation(^9^99[https://github.com/ubc-vision/Make-A-Story/blob/main/ldm/data](https://github.com/ubc-vision/Make-A-Story/blob/main/ldm/data)) for more details on how the referential dataset is extended.

Table: Table 11: Dataset statistics of FlintstonesSV [] and PororoSV []

### D.3 First stage training

We built upon pre-trained Stable Diffusion [] v1-5(^10^1010[https://huggingface.co/runwayml/stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5)) and use CLIP [] ViT-L to extract characters’ visual features. We freeze the CLIP text encoder and fine-tune the remaining modules for 25,000 steps with a learning rate of 1e-5 and batch size of 32. The first stage utilizes solely the original text description without extended referential text. To enhance inference time robustness and flexibility, with or without reference images, we adopt a training strategy that includes $10\%$ unconditional training, i.e., classifier-free guidance [], $10\%$ text-only training, and $80\%$ augmented text training, which integrates visual features of characters with their corresponding token embeddings.

### D.4 Second stage training

We use OPT-6.7B(^11^1111[https://huggingface.co/facebook/opt-6.7b](https://huggingface.co/facebook/opt-6.7b)) model as the LLM backbone in all experiments in the main paper. To expedite the second stage alignment training, we first pre-compute non-referential fused embeddings residing in the input space of the first-stage Char-LDM. We map visual features into $m=4$ token embeddings as LLM input, set the max sequence length as 160 and the number of additional [IMG] tokens as $R=8$, batch size as 64 training for 20k steps. Llama2 is only trained for the experiments highlighted in the supplementary materials, demonstrating its capability for multi-modal generation and the ablation of different LLMs. The training configuration is almost the same as OPT, except for batch size 32. All experiments are executed on a single A100 GPU.

Please refer to all the details at the [source code](https://github.com/xiaoqian-shen/StoryGPT-V).

Figure: Algorithm 1 Character Replacement Algorithm

Figure: Figure 9: DALL-E 3 [] zero-shot inference on FlintstonesSV [] dataset.

## Appendix E Limitations

Our method demonstrates proficiency in resolving references and ensuring consistent character and background conditions in the context provided by guiding the output of a multi-modal Large Language Model (LLM) with character-augmented semantic embedding. However, several limitations remain. The process involves feeding the previously generated frame into the LLM to produce a visual output that aligns with the Latent Diffusion Model (LDM) input conditional space. This approach guarantees semantic consistency, enabling the generation of characters and environmental objects that resemble their originals. Nonetheless, there are minor discrepancies in detail. This is because the visual output from the Large Language Model (LLM) is aligned with the semantic embedding space rather than the pixel space, which hinders the complete reconstruction of all elements in the input image. However, the current most powerful multi-modal LLM, i.e., DALL-E 3 [], could not solve this exact appearance replication in the multi-round image generation task (Figure [9](#A4.F9)), indicating an area ripe for further exploration and research.

## Appendix F Qualitative Results

We provide more generated samples on FlintstonesSV [] and PororoSV [] with referential text as Figure [11](#A6.F11)-[19](#A6.F19) show.

Figure: Figure 10: Qualitative comparison on FlintstonesSV [] with co-reference descriptions.
Refer to caption: /html/2312.02252/assets/x10.png

Figure: Figure 12: Qualitative comparison on FlintstonesSV [] with co-reference descriptions.
Refer to caption: /html/2312.02252/assets/x12.png

Figure: Figure 14: Qualitative comparison on FlintstonesSV [] with co-reference descriptions.
Refer to caption: /html/2312.02252/assets/x14.png

Figure: Figure 16: Qualitative comparison on FlintstonesSV [] with co-reference descriptions.
Refer to caption: /html/2312.02252/assets/x16.png

Figure: Figure 18: Qualitative comparison on PororoSV [] with co-reference descriptions.
Refer to caption: /html/2312.02252/assets/x18.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See above weaknesses.
soundness: 3
strengths: 1. The paper proposes combining two powerful models, LLMs and LDMs, for the story visualization task, and experiments show promising results from this combination.

2. Experiments indicate that fusing visual features into text embeddings to refine an LDM can improve the quality of the output results.

3. Autoregressive generation, which continuously utilizes visual features from the previous frame, helps maintain consistency across different frames in a story and enables the model to generate longer narratives.
summary: This paper focuses on the task of story visualization and proposes combining the advantages of LLMs and LDMs to generate a consistent story from a given long narrative. To achieve this, the proposed model consists of two stages: the first stage introduces visual features into the text embedding to produce a fused embedding, which is then used in the second stage to generate results. Experimental results demonstrate that the proposed method effectively generates high-quality, consistent outputs.
weaknesses: 1. The strong performance of the proposed method primarily relies on the powerful capabilities of both LLMs and LDMs. Although the authors have introduced a way to combine both models for the story visualization task, the technical novelty may be limited.

2. I have concerns about the diversity of the proposed model. In the first stage, the authors incorporate visual features into text embeddings and use segmentation masks to supervise LDM training. In this setup, the output results of the refined LDMs for a given story may be constrained by the infused visual features. Based on this, I am curious about the diversity of outputs when the same story is provided as input.

3. The paper emphasizes that the proposed method can effectively address consistency issues, such as resolving pronouns in frame descriptions; however, the authors provide only qualitative results for this claim. Including quantitative results could better support the effectiveness of the approach.

4. It may be beneficial to include recent works such as StoryGen and StoryDiffusion, as well as more complex datasets like the StorySalon dataset.

StoryGen, StorySalon dataset: Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yanfeng Wang, and Weidi Xie. Intelli- gent grimm-open-ended visual storytelling via latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6190–6200, 2024.

Story-Diffusion: Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Con- sistent self-attention for long-range image and video generation. NeurIPS 2024, 2024.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Please check out the Weaknesses part.
soundness: 3
strengths: - Pronoun resolution is an interesting topic in storytelling. Introducing LLMs to address this problem is a very reasonable approach. The cross-attention map visualized on the anonymous webpage demonstrates that it can effectively assist LDMs in associating pronouns with characters.

- This paper conducts comprehensive ablation experiments to demonstrate the effectiveness of the proposed training strategy in the two-stage training process. Additionally, it shows better competitiveness across various metrics compared to the methods listed on the FlintStonesSV Benchmark.
summary: This paper proposes StoryGPT-V, a novel framework that integrates LLMs with LDMs to enhance the ability to resolve pronouns and consistent generation. It employs a two-stage training process: first, it trains a character-aware LDM to take character embeddings as input; second, it aligns the output of LLM with the input space of LDM. Results show that LLMs significantly enhance the model's ability to understand pronouns, and StoryGPT-V shows good capability to generate consistent stories.
weaknesses: My main concern is that the paper does not sufficiently discuss the latest methods.

- Before this paper, there were various storytelling papers(e.g., TaleCrafter[1] in SIGGRAPH Asia 2023, AR-LDM[2] in CVPR 2024) that integrated LLMs into diffusion models, and some works(ConsiStory[3] in TOG 2024) also manipulate attention modules for consistent story generation. A concurrent work, SEED-Story[4], also shares a similar idea that utilizes an LLM for multimodal story generation, and the diffusion model serves as a visual decoder. These works are highly relevant and deserve discussion. 
- Furthermore, some works have also conducted experiments on FlintStonesSV, such as AR-LDM and TaleCrafter. Adding comparisons with these works could potentially enhance the paper's credibility, if possible.

[1] Talecrafter: Interactive story visualization with multiple characters
[2] Synthesizing coherent story with auto-regressive latent diffusion models
[3] Training-free consistent text-to-image generation
[4] SEED-Story: Multimodal Long Story Generation with Large Language Model

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What is the benefit of the proposed multimodal alignment over previous methods? It'd make the work more solid by conducting an ablation study comparing the proposed alignment method to counterpart alignment approaches (Table 3), measuring performance on key metrics like character consistency or reference resolution accuracy.
soundness: 1
strengths: The work addresses an important problem in visual storytelling for enhanced multiple character consistency, leveraging LLM's comprehension and reasoning ability.
summary: This work targets to tackle the ambiguous subject reference challenge in text-to-image visual storytelling. The core idea is to leverage the comprehension and reasoning ability of Large Language Model. First, the authors train a character-aware Latent Diffusion Model. Then, the character-aware semantic embeddings from LDM and LLM output are aligned. Experiments are conducted on two story visualization benchmarks and show more consistent character generation.
weaknesses: 1. The paper writing needs improvements. For example, (1) the sentence "Since it requires resolving pronouns (he, she, they) in
the frame descriptions, i.e., anaphora resolution, and ensuring consistent characters and background synthesis across frames." is incomplete in the abstract L14. (2) The core contribution and benefit over previous MLLM alignment approaches is unclear. Suggest to elaborate the advantage of the proposed ''implicitly deduced reference'' explicitly (L109, L139).
2. Incomprehensive related work. For instance, StoryGen (CVPR'24), StoryDiffusion (NeurIPS'24), and SeedStory ('24) are recent work towards subject consistency in story visualization. In particular, SeedStory ('24) also leverages LLM's reasoning ability and aligns text-image.
3. Incomplete experiments. It misses important comparisons to the published StoryGen (CVPR'24) and StoryDiffusion (NeurIPS'24). Suggest to include in Table 1 and 2.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - Notation: what is m in line #282? Also is k the same as e in line #282-283?
- In Figure 6(b), how did you use longer stories? Did you extend existing stories with LLM?
- Please address weaknesses mentioned above.

Typo:
Line #014: visualization. Since -> visualization since
soundness: 2
strengths: - The proposed idea of using segmentation mask for guiding the cross-attention map of LDM makes sense, and using LLM for retaining contexts and reference resolution is interesting.
- In experimental results, it is shown that the proposed model outperforms the baselines in both quantitative and qualitative evaluations.
summary: This paper tackles the story visualization task and propose a two-stage approach. First, the authors propose Char-LDM, a finetuned LDM by guiding the cross-attention map of LDM with character segmentation mask. Second, they utilize Multi-modal LLM for retaining contexts and reference resolution by mapping from the output of LLM to align with the input space of the first stage LDM. In the experimental results, it is shown that the proposed approach outperforms baselines including Story-LDM with lower computational resources such as GPU memory  for longer image frames and inference speed.
weaknesses: - Novelty seems somewhat incremental as using segmentation mask of characters for generating them is not new (e.g., [1]), and also for LLM to understand and generate images, it uses the similar idea from prior work such as [2].
- In the experiments, there are points to raise concerns regarding the technical soundness of them. First, it is said: "Story-LDM (Rahman et al., 2023) does not provide pre-trained checkpoint or cleaned training code" (line #368), so it was not clear how it was implemented even though it is written: "we initiate training from pre-trained LDM". Furthermore, in Table 1 (bottom), the reported Char-acc and Char-F1 (77.23 & 88.26) in the paper are different from what are reported in Story-LDM paper (69.19 & 86.59). Second, in human evaluation, only three judges were used and it seems too few. 
- Writing needs to be clearer. e.g., it is not clear whether only first stage (Char-LDM) was used or both stages were used for the result in Table 1 (upper half) as it is written: "Our first-stage Char-LDM exhibits superior performance in generating accurate characters" (line #414). 
- The claim of using less GPU memory is questionable. My understanding is 2nd stage needs to load LLM to memory, so it seems more expensive compared to the baseline, i.e., Story-LDM, especially for the typical scenario for story visualization of generating five image sequence. This seems also confirmed in Figure 6(b) when the number of frames to generate is less than 10. 
- It lacks an important ablation study: using only first stage without second stage to see the effectiveness of LLM.

References

[1] Song et al., Character-Preserving Coherent Story Visualization, ECCV 2020

[2] Koh et al., Generating Images with Multimodal Language Models, NeurIPS 2023