SlowGuess/NAACL_2021_OCR huggingface dataset:
Adyasha Maharana Darryl Hannan Mohit Bansal
{adyasha, dhannan, mbansal}@cs.unc.edu
Story visualization is an underexplored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform text-to-image synthesis models on this task. However, there is room for improvement of generated images in terms of visual quality, coherence and relevance. We present a number of improvements to prior modeling approaches, including (1) the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, (2) a copy-transform mechanism for sequentially-consistent story visualization, and (3) MART-based transformers to model complex interactions between frames. We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative. Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not accurately reflect performance. Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images. We also present correlation experiments of our proposed automated metrics with human evaluations.<sup>1</sup>
While generative adversarial networks (GANs) have achieved impressive results on a variety of
$^{1}$ Code and data: https://github.com/adymaharana/StoryViz.
The car to carry freight trains to ride Pororo and friends starts on road. The car to carry freight trains is riding across the snow-covered field. Pororo is complaining to Crong on the field. Pororo asks the car on the snow-covered field. The car told on the snow-covered field.
Figure 1: Illustration of the Pororo-SV dataset (Captions & Ground Truth) and the corresponding images generated from our model (Generated).
image generation tasks (Zhu et al., 2019; Qiao et al., 2019), the task of story visualization (Li et al., 2019b) is a variation of image generation that is more challenging and underexplored. In this setting, there is a story which consists of a sequence of images along with captions describing the content of the images, e.g., a web comic. The goal of the task is to reproduce the images given the captions (Figure 1). The benefits of investigating this task are far reaching. It combines two interesting and challenging sub-areas: text-to-image synthesis and narrative understanding, providing an excellent test bed for exploring and developing multimodal modeling techniques. From an application perspective, such a system could be used to enhance existing textual narratives with visual scenes. This tool would be especially useful to comic artists, who are infamously overworked, allowing them to automatically generate initial drawings speeding up their workflow. Additionally, such a system would have many applications in an educational setting, allowing educators to cater to a more diverse set of learning styles by automatically generating visualizations for a given topic, such as the water cycle in a science lesson. Furthermore, the data in this domain is cartoon-style, meaning the generated im
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2427-2442 June 6-11, 2021. ©2021 Association for Computational Linguistics
ages avoid many of the ethical issues associated with real-world data. For a more detailed discussion, see Section 9.
The challenge of this task extends beyond tasks such as text-to-image or text-to-video synthesis. Namely, there is an explicit, narrative component to the data, which must first be accurately extracted from the text, and then consistently reproduced throughout the images. If the setting or a description of a character is provided in the first caption, this must be carried throughout the scene unless modified by a subsequent caption. Furthermore, the scenes in a single story can change drastically as the story progresses, requiring models to produce a greater variety of images than in a text-to-video task, which typically consists of short videos displaying a single action. To address these issues, we consider the task as proposed in Li et al. (2019b), which provides a baseline architecture, StoryGAN, along with datasets for the task. We introduce techniques that build on existing work and are focused on improving consistency across frames, resulting in images of higher visual quality.
First, we augment the model with Dual Learning via video redescription. The output images are fed through a video captioning model, which is trained to reproduce the ground truth story captions. This provides an additional learning signal to the model, forcing it to semantically align with the given narrative. Next, we add a Copy-Transform module that can take generated images from previous timesteps and copy the most relevant features of those images into the next generated frame, thus making the images more consistent in appearance. Finally, we propose the use of Memory-Augmented Recurrent Transformer (MART) (Lei et al., 2020) to model the correlation between word phrases in the input text and corresponding regions in the generated image. The recurrent nature of MART allows for the learning of sophisticated interactions between the image frames, yielding images that are more consistent in terms of character appearances and background imagery. We call the model architecture with the aforementioned additions DU(AL)-CO(PY)-STORYGAN or DUCo-STORYGAN.
Next, we focus on exploring alternative evaluation methods for story visualization models. While modeling improvements are crucial for progressing in this domain, evaluating these models is a challenge in itself. Like many other generative tasks, it is nontrivial to evaluate a story visualization model.
Human evaluation is the most reliable option, but its monetary and time costs make this ill-suited to be the only evaluation method. Most prior work relies upon standard GAN evaluation metrics, which may provide some insight into how well the images were reproduced, yet miss out on other aspects of the story visualization task, such as the visual consistency of the setting across frames and global semantic alignment. Therefore, we make evaluation another focal point of the paper, exploring a variety of automatic evaluation metrics, which capture various aspects of the task, e.g., evaluating the quality of the images, the relevance to the story, the diversity of the generated frames, and the model's ability to accurately represent the characters. We present results from our model and baseline models on all metrics along with qualitative results, demonstrating the improvements from our proposed techniques. Using these metrics, we also provide ablation analyses of our model.
1. For the story visualization task, we improve the semantic alignment of the generated images with the input story by introducing dual learning via video redescription.
2. We enable sequentially-consistent story visualization with the introduction of a copy-transform mechanism in the GAN framework.
3. We enhance prior modeling techniques in story visualization with the addition of Memory Augmented Recurrent Transformer, allowing the model to learn more sophisticated interactions between image frames.
4. We present a diverse set of automatic evaluation metrics that capture important aspects of the task and will provide insights for future work in this domain. We also conduct correlation experiments for these metrics with human evaluation.
Li et al. (2019b) introduced the task of story visualization and the StoryGAN architecture for sequential text-to-image generation. There have been a few other works that have attempted to improve upon the architectures presented in this paper. PororoGAN (Zeng et al., 2019) aims to improve the semantic relevance and overall quality of the images via a variety of textual alignment modules and a
patch-based image discriminator. Li et al. (2020) also improve upon the StoryGAN architecture by upgrading the story encoder, GRU network, and discriminators and adding Weighted Activation Degree (Wen et al., 2019). Song et al. (2020) is a more recent work which makes improvements to the StoryGAN architecture; the primary contribution is adding a figure-ground generator and discriminator, which segments the figures and the background of the image. Our model improvements of MART, dual learning, and copy-transform build upon more recent techniques and we support them with a detailed series of ablations.
Text-to-Image and Text-to-Video Generation. While story visualization is an underexplored task, there has been plenty of prior work in text-to-image synthesis. Most papers in this area can be traced back to StackGAN (Zhang et al., 2017). Subsequent work then made various modifications to this architecture, adding attention mechanisms, memory networks, and more (Xu et al., 2018; Zhu et al., 2019; Li et al., 2019a; Yi et al., 2017; Gao et al., 2019). Huang et al. (2018) and Qiao et al. (2019) are direct precursors of our work. Both of these works subject the generated output as an image captioning task which attempts to reproduce the original text. Our proposed dual learning approach is an expansion of this module, where we use a state-of-the-art video captioning model based upon the MART (Lei et al., 2020) architecture to provide an additional learning signal to the model and increase the semantic consistency across images.
In the domain of text-to-video synthesis, Li et al. (2018), Pan et al. (2017), Gupta et al. (2018) and Balaji et al. (2019) generate videos from single sentences. In contrast to videos, story visualization does not have the requirement that the frames flow continuously together. Therefore, it allows for more interesting interactions and story-level dynamics to be captured that would only be present in longer videos.
Interactive Image Editing. Another task related to story visualization is interactive image editing. In this setting, rather than going from purely text to image, the model is given an input image along with textual instructions/directions, and must produce an output image that modifies the input image according to the text. This can take the form of high level semantic changes to the image, such as color and shape, as in Liu et al. (2020), Nam
et al. (2018), and Chen et al. (2018), or this might take the form of Photoshop-style edits, as in Laput et al. (2013), Shi et al. (2020), and Manuvinakurike et al. (2018a). Alternatively, Cheng et al. (2020), Manuvinakurike et al. (2018b), and El-Nouby et al. (2019) are slightly closer to our task due to their sequential nature, where an image is modified repeatedly according to the textual feedback provided via a dialogue. However, unlike story visualization, these tasks do not have a narrative component. Furthermore, they involve repeatedly editing a single object at each timestep instead of generating diverse scenes with dynamic characters.
Formally, the task consists of a sequence of sentences  $S = [s_1, s_2, \dots, s_T]$  and a sequence of images  $X = [x_1, x_2, \dots, x_T]$ , where the sentence  $s_k$  describes the contents of the image  $x_k$ . The model receives  $S$  as input and produces a sequence of images  $\hat{X} = [\hat{x}_1, \hat{x}_2, \dots, \hat{x}_T]$ , attempting to accurately reproduce  $X$ . As detailed in Li et al. (2019b), there are two aspects of this task. The first is local consistency, which is concerned with the quality of individual pairs in the sequence; an example is locally consistent if image  $\hat{x}_k$  accurately represents the contents of sentence  $s_k$ . The second aspect is global consistency, which is concerned with the quality of the entire sequence. Namely, whether the sequence of images  $\hat{X}$  accurately captures the content of the sequence of sentences  $S$ .
The general approach to this task as followed by StoryGAN (Li et al., 2019b) is as follows: The story encoder creates the initial representation  $h_0$  of the story  $S$ . This is then passed to the context encoder, which is a recurrent model that takes a sentence  $s_k$  as input and forms a representation  $o_k$ . Each of these representations  $o_k$  are then fed to the image generator, which outputs an image  $\hat{x}_k$ . The generated images are passed to two discriminators, the image discriminator and story discriminator, which each evaluate the generated images  $\hat{x}_k$  in different ways and produce a learning signal that can be used to adjust the parameters of the network.
3.2 DUCO-STORYGAN
The framework of our model is based on the StoryGAN architecture. We improve upon the context encoder and expand the network with dual learning and copy-transform mechanisms. The image and
Figure 2: Illustration of DUCo-STORYGAN architecture. The story encoder is used to initialize memory module in MART context encoder, which encodes the captions for the image generator. The copy-transform mechanism copies features from images generated in previous timesteps to the image in current timestep. The generated images are passed to story and image discriminators, and dual learning video captioning model.
story discriminators, and the story encoder from the original model are retained in DUCo-STORYGAN; each contributes to a separate loss term i.e.  $\mathcal{L}_{img}$ ,  $\mathcal{L}_{story}$  and  $\mathcal{L}_{KL}$  respectively. See Appendix for details on the loss terms. An overview of our model architecture can be seen in Figure 2.
MART Context Encoder. One of the primary challenges of story visualization is maintaining consistent background imagery and character appearances throughout the story. This is addressed with a recurrent context encoder which has access to the global narrative while encoding the caption in each time-step. We use the Memory Augmented Recurrent Transformer (MART) (Lei et al., 2020), where the memory is initialized with the conditioning vector  $h_0$  from the story encoder. It takes word embeddings  $W_k = [w_{k1}, w_{k2}, \dots, w_{kL}]$  where  $w_{ij} \in \mathcal{R}^{1 \times d_w}$ , corresponding to the frame caption at each timestep and produces contextualized embeddings which are then pooled to a single weighted representation  $c_k$  using attention. This allows the context encoder to capture sophisticated interactions among the words which the image generator can then capitalize on:
$$
\begin{array}{l} [ m _ {k 1}, \dots . m _ {k L} ], h _ {k} = \operatorname {M A R T} ([ w _ {k 1}, \dots . w _ {k L} ], h _ {k - 1}) \\ c _ {k} = \sum_ {i = 1} ^ {L} \alpha_ {k i} m _ {k i}; \alpha_ {k i} = \frac {\exp (m _ {k i} ^ {T} u)}{\sum \exp (m _ {k i} ^ {T} u)} \\ \end{array}
$$
where  $u$  is a query vector learned during training. The Transformer encoder is followed by a layer of GRU cells that take the contextualized embedding as input along with isometric Gaussian noise,  $\epsilon_{k}$  and produce an output vector  $g_{k}$ . The outputs  $c_{k}$  and  $g_{k}$  are concatenated and transformed into fil
ters, and subjected to convolution with a projection of the sentence embedding  $s_k$ , resulting in output vector  $o_k$ . See Appendix for more details.
Image Generator. The image generator follows prior text-to-image generation approaches (Qiao et al., 2019; Xu et al., 2018; Zhang et al., 2017) and uses a two-stage approach. The first stage uses outputs  $o_k$ ; the resulting image is fed through a second stage, which aligns the contextualized word encodings  $m_k$  from MART with image subregions generated in first-stage and reuses weighted encodings for image refinement.
Dual Learning via Video Redescription. Dual learning provides the model with an additional learning signal by taking advantage of the duality of certain tasks, i.e., if X can be used to produce Y, then Y can be used to produce X. Here, our primary task is story visualization, and we consider the secondary task of video captioning. We refer to this process as video redescription. To execute the idea of learning via video redescription, we employ a video captioning network which takes the sequence of generated images and produces a corresponding sequence of captions. The video captioning network is based on a recurrent encoder-decoder framework  $(V_{enc}(.), V_{dec}(.)$  and is trained using a cross-entropy loss on the predicted probability distribution  $(p)$  over its vocabulary. Specifically,  $\mathcal{L}_{\text{dual}} = \sum_{k=1}^{T} \sum_{i=1}^{L} \log p_{ki}(w_{ki})$ . The hidden state in recurrent model helps the captioning network to identify narrative elements in the sequence of images and penalize the generative model for a lack of consistency in addition to semantic misalignment. We pretrain the video cap
tioning network using ground truth data and freeze its parameters while training the generative model. We also include a multiplier,  $\lambda_{\text{dual}}$ , which allows us to scale the loss. The implementation of the encoder-decoder framework can vary. For our primary model, we adapt the MART video captioning network (Lei et al., 2020) to accept a 2D matrix of features at each time step where each column corresponds to an image sub-region (see Sec. 5).
Sequentially-Consistent Story Visualization. While certain components, such as character positions, will change from frame to frame, there are other components like background and appearances which usually carry over to adjacent frames. To take advantage of this continuity, we augment the model with a copy-transform mechanism. This mechanism can take into consideration the generated image from previous timesteps, and reuse aspects of those prior images during the current timestep. The copy-transform module  $F^{\text{copy}}(.)$  performs attention-based semantic alignment (Xu et al., 2018) between word features  $m_{k} \in \mathcal{R}^{D_{w} \times L}$  in the current timestep and image features  $i_{k-1} \in \mathcal{R}^{D_{i} \times N}$  from previous step. Each column of  $i_{k-1}$  is a feature vector of a sub-region of the image. The word features are first projected into the same semantic space as image features i.e.  $m_{k}' = U m_{k}$ , where  $U \in \mathcal{R}^{D' \times D}$ . For the  $j^{th}$  image sub-region, the word-context vector is calculated as:
$$
c _ {j k} = \sum_ {i = 0} ^ {L} \beta_ {j i} m _ {i k} ^ {\prime}; \beta_ {j i k} = \frac {\exp (h _ {j} ^ {T} m _ {i k} ^ {\prime})}{\sum_ {i = 0} ^ {L} \exp (h _ {j} ^ {T} m _ {i k} ^ {\prime})}
$$
$\beta_{jik}$  indicates the weight assigned by the model to the  $i^{th}$  word when generating the  $j^{th}$  sub-region of the image. The weighted word-context matrix is then concatenated with the generative image features from the current timestep and sent for upsampling to the image generator.
Objective. Bringing it all together, the final objective function of the generative model is:
$$
\min _ {\theta_ {G}} \max _ {\theta_ {I}, \theta_ {S}} \mathcal {L} _ {K L} + \mathcal {L} _ {i m g} + \mathcal {L} _ {s t o r y} + \lambda_ {d u a l} \mathcal {L} _ {d u a l}
$$
where  $\theta_{G},\theta_{I}$  and  $\theta_{S}$  denote the parameters of the entire generator, and image and story discriminator respectively. See Appendix for more details.
Dataset. We utilize the Pororo-SV dataset from the original StoryGAN paper which has been
adapted from a video QA dataset based on animated series (Li et al., 2019b) $^2$ . Each sample in Pororo-SV contains 5 consecutive pairs of frames and captions. The original splits of Pororo-SV from Li et al. (2019b) contain only training and test splits with nearly  $80\%$  overlap in individual frames. For a more challenging evaluation, we use the test split proposed in (Li et al., 2019b) as validation split (2,334 samples) and carve out an "unseen" test split from the training examples. The resulting dataset contains 10191, 2334 and 2208 samples in training, validation and test splits respectively. In this version, there is  $58\%$  frame overlap between the validation and train splits and 517 samples in the validation split contain at least one frame which is not present in the training set. Conversely, the test split has zero overlap with the training split.
Experimental Settings. Our model is developed using PyTorch, building off of the original StoryGAN codebase. All models are trained on the proposed training split and evaluated on validation and test sets. We select the best checkpoints and tune hyperparameters by using the character classification F-Score on validation set (see Appendix).
5 Evaluation of Visual Story Generation
As with any task, evaluation is a critical component of story visualization; however, due to the complexity of the task and its generative nature, evaluation is nontrivial. For instance, characters are the focal point of any narrative and similarly should be the focus of a model when producing images for the story. Hence, Li et al. (2019b) measure the character classification accuracy within frames of generated visual stories in order to compare models. However, it is also important that the characters and background are consistent in appearance, and together form a cohesive story rather than an independent set of frames. Inspired by insights such as this, we explore an additional set of evaluation metrics that capture diverse aspects of a model's performance on visual story generation.
Character Classification. We finetune the pretrained Inception-v3 (Szegedy et al., 2016) with a multi-label classification loss to identify characters in the generated image. Most earlier work in story visualization report the image-level exact-match (EM) character classification accuracy. However,
2We opt to not use the CLEVR-SV dataset as we believe that this dataset lacks a narrative structure and is not suitable for story visualization.
we contend that the exact match accuracy is not sufficient to gauge the performance of generative models, and the micro-averaged F-score of character classification should also be reported. For example, if Model A generates one of two characters in a frame with better quality than Model B (which generates none), it results in the same EM accuracy as Model B but an improvement in the recall/F-Score of the model, making the latter more reliable as a metric for quality. Our conclusion is based on the observation of consistent improvement in character classification scores with increasing training epochs and manual evaluation of image quality (see Fig 4).
Video Captioning Accuracy. In order to measure global semantic alignment between captions and generated visualizations, we propose to use video captioning models which have been pretrained on ground truth data to identify narrative elements in a sequence of frames. We use the Memory-Augmented Recurrent Model proposed in Lei et al. (2020) and add a CNN encoder (Sharma et al., 2018) on top of the Transformer encoder to extract image embeddings. The final convolutional layer (Mixed_7c) in finetuned Inception-v3 is used to extract a local feature matrix  $f \in \mathcal{R}^{64 \times 2048}$  (reshaped from  $2048 \times 8 \times 8$ ) for each image in the story. We then use this trained video captioning model to caption the generated frames. The generated captions are compared to the ground truth captions via BLEU evaluation<sup>3</sup>, and this functions as our proposed metric for measuring global semantic-alignment between the captions and generated story. This pretrained model is also used as the video captioning dual network during training of DUCo-STORYGAN.