SlowGuess/NAACL_2021_OCR huggingface dataset:
Discriminative Evaluation. Generative metrics such as BLEU are known to be noisy and unreliable. Hence, we also develop a discriminative evaluation setup. In order to compute similarity between generated image and ground truth, we compare the feature representations from either images in this discriminative setup. The training dataset for story visualization may contain one or more frames with the exact set of characters that are referenced in captions in the evaluation data. When we are checking for the presence of these characters in a generated image, we do not want to reward the model for copying the exact same frame from the training set
instead of generating a frame suited to the input caption. In order to evaluate this consistency, we propose discriminative evaluation of the story visualization model. Using the character annotations for the final frame of each sample in the test splits, we extract a set of 4 negative frames which are taken from elsewhere in the video but contain those specific characters (see Fig. 7 in Appendix). The human evaluation accuracy on this dataset is  $89\%$ $(\kappa = 0.86)$  and is used as an upper bound when interpreting model accuracy performance. The cosine similarity between Inception-v3 features of final generated frame and candidate frames is computed and the frame with most similarity is selected as predicted frame. We report Top-1/2 accuracies.
R-Precision. Several prior works on text-to-image generation report the retrieval-based metric R-Precision (Xu et al., 2018) for quantifying the semantic alignment between the input text and generated image. If there are  $R$  relevant documents for a query, the top  $R$  ranked retrieval results of a system are examined; if  $r$  are relevant, the R-precision is  $r / R$ . In our task $^4$ ,  $R = 1$ . The encodings from a pretrained Deep Attention-based Multimodal Similarity Model (DAMSM) are used to compute cosine similarity and rank results. Since this model only evaluates a single text-image pair for similarity, it is not suitable for evaluating story visualization. Therefore, we train a new version of DAMSM to extract global representations for the story and sequence of images, referred to as Hierarchical DAMSM (H-DAMSM) (see Appendix).
The models used in the aforementioned evaluation metrics are trained independently of DuCoSTORYGAN on the proposed Pororo-SV splits and the pretrained weights are used for evaluation. See Appendix for other upper bounds.
6.1 Main Quantitative Results
The results for Pororo-SV validation set can be seen in Table 1. The first row contains the results using the original StoryGAN model (Li et al., 2019b) $^5$ . The second row functions as another
4The R-precision score is obtained from 10 runs with 99 randomly picked mismatched story candidates in each run.
We use a reduced training dataset as compared to the original StoryGAN paper (see Sec 4). However, we evaluate our StoryGAN code base on their exact splits and get  $26.1\%$  exact-match accuracy, which is approximately equivalent to the  $27\%$  reported in the original paper where they demonstrate that StoryGAN outperforms previous baselines such as ImageGAN, SVC, and SVFN.
<sup>3</sup>We use the nlg-eval package (Sharma et al., 2017) for BLEU evaluation.
Model	Char. F1	BLEU2/3	R-Precision	Frame Acc.	Top-1 Acc.	Top-2 Acc.
StoryGAN (Li et al., 2019b)	41.11	3.86 / 1.72	3.40 ± 0.01	21.90	22.42	45.40
StoryGAN + Transformer	42.45	3.92 / 1.73	4.03 ± 0.17	22.14	23.79	47.15
CP-CSV (Song et al., 2020)	43.79	3.96 / 1.73	3.97 ± 0.21	22.08	24.29	46.39
DUCo-STORYGAN	48.27	4.51 / 1.92	6.10 ± 0.07	22.71	25.62	47.39
Table 1: Results on validation split of Pororo-SV Dataset.
Model	Char. F1	BLEU2/3	R-Precision	Frame Acc.	Top-1 Acc.	Top-2 Acc.
StoryGAN (Li et al., 2019b)	18.59	3.24 / 1.22	1.51 ± 0.15	9.34	23.14	42.27
StoryGAN + Transformer	19.29	3.29 / 1.23	1.49 ± 0.07	9.58	23.31	42.29
CP-CSV (Song et al., 2020)	21.78	3.25 / 1.22	1.76 ± 0.04	10.03	22.23	41.86
DUCo-STORYGAN	38.01	3.68 / 1.34	3.56 ± 0.04	13.97	23.72	42.48
Table 2: Results on test split of Pororo-SV Dataset.
Figure 3: Sample results from StoryGAN and DUCoSTORYGAN on unseen test split.
baseline, where we replace the GRU-based context encoder in StoryGAN with a Bidirectional Transformer (Devlin et al., 2019). The conditioning augmentation vector is not used to initialize the context encoder in this model since a non-recurrent Transformer lacks a hidden state. We see  $1 - 2\%$  improvements in character classification and retrieval with this model over StoryGAN. The third row contains results from the more recent CP-CSV model (Song et al., 2020) which uses figure-ground segmentation as an auxiliary task for preserving character features. Consequently, it results in  $2.68\%$  improvement in character classification over StoryGAN and smaller improvements for other metrics. The final row contains results with DUcoSTORYGAN, which significantly outperforms previous models (including CP-CSV) across all metrics. The character classification F-Score improves by  $7.16\%$  suggesting that the characters generated in our images are of higher visual quality. Similarly, we see consistent improvements in BLEU as well as R-Precision with our model. As demon
strated in Sec 7.1, the improvement in BLEU can be attributed to the addition of dual learning, which directly optimizes the dual task of video captioning. The R-Precision indicates that our model learns better global semantic alignment between the captions and images. Lastly, the Top-1/2 accuracy scores show that our model is learning to generate diverse images, rather than copying scenes that feature the same characters from the training data.
DUCo-STORYGAN performs dramatically better than other models on the unseen test split (see Table 2). As can be seen in Fig 3, StoryGAN performs rather poorly on unseen samples compared to DUCo-STORYGAN. While the former produces images that are blurry and character shapes that are faint, the latter generates frames with sharp character features. This is reflected in the wide improvement margins on character classification scores in Table 2. Similar improvements are also observed for BLEU and R-Precision metrics, indicating that our model generates images which are more relevant to the input caption. When generating stories for the Pororo-SV test split, models tend to copy background elements from the samples seen in the training set, since the captions lack sufficient information about the setting. Hence, we observe little improvement over random chance in the discriminative accuracy scores for different models on test split. For instance, instead of generating the tinted background in ground truth in Fig. 3, the models produce a clear blue sky which is closer to samples seen in the training set. However, discriminative evaluation will be valuable for future work in this domain when inputs contain detailed information about the visual elements.
We also provide per character results for the Character F-Score. With DUco-STORYGAN, we see up to  $20\%$  improvement for less frequent char
Win %	Mean Rating
Attribute	Ours	StGAN	Ours	StGAN
Visual Quality	82%	3%	2.06	1.22
Consistency	78%	3%	2.94	1.78
Relevance	26%	2%	1.28	1.04
Table 3: Human evaluation on Likert Scale 1-5. Win% = % times stories from one model was preferred over the other (StGAN = StoryGAN). Tie% = % samples remaining after considering Win% of both models.
acters (see Table 6).
We conduct human evaluation on the generated images from DUCo-STORYGAN and StoryGAN, using the three evaluation criteria listed in Li et al. (2019b): visual quality, consistence, and relevance. Two annotators are presented with a caption and the generated sequence of images from both models, and asked to rate each sequence on a scale of 1-5. Results are presented in Table 3. With respect to pairwise evaluation, predictions from our model is nearly always preferred over those from StoryGAN (see Win% columns). Similarly, we see large improvements in mean rating of stories generated by DUCo-STORYGAN. However, we also see higher Tie% and low mean rating for the attribute Relevance, suggesting that much work remains to be done to improve understanding of captions.
Correlation Experiments: We also examine the correlation between our proposed metrics and human evaluation of generated images. We compute the Pearson's correlation coefficient between human ratings of 50 samples on three different attributes using the 1-5 Likert scale and their corresponding automated metric evaluation scores. Significant correlation  $(\rho = 0.586)$  was observed between our proposed Character F-Score metric and Visual Quality, lending strength to its use as automated metric for story visualization.
7.1 Ablations
Table 4 contains plus-one ablations for DUcoSTORYGAN. The first row is the StoryGAN baseline and the second row is the StoryGAN + Transformer model, as discussed in Section 6. We then iteratively add each of our contributions and observe the change in metrics<sup>6</sup>. First, we upgrade the
Figure 4: Progression of character classification scores (top) and generated images (bottom) with training.
Transformer encoder to MART, which brings about the largest improvements across all metrics. The use of word embeddings with access to global conditioning vector and attention-based semantic alignment proves important to the task of story generation. Next, we use the MART context encoder with our proposed dual learning and copy-transform improvements. With the addition of video captioning as a learning signal, we see  $0.20\%$ $(p = 0.071)$  improvement in character F-score and  $1.12\%$  improvement in R-Precision  $(p = 0.032)$  over MART. The highest improvements are observed for BLEU score, since the model is optimized on video captioning. Next, we evaluate the addition of the copy-transform mechanism where features from generated images in previous timesteps are copied to the image in current timestep. We observe  $1.04\%$  improvements for character classification and a slight drop in performance on video captioning. Similarly, there is  $1.14\%$  improvement in Top-1 accuracy for the discriminative dataset.
As discussed in Section 3, we explore a variety of implementations for the dual learning component of our model. While MART-based video captioning works the best, we provide a discussion of other approaches in the Appendix.
7.2 Qualitative Examples
Figure 5 contains two generated examples from the Pororo-SV dataset. The top row in each example contains the ground truth images, the middle row the images generated by StoryGAN, and the final row the images generated by our model. In
All our improvements in DUCo-STORYGAN are statistically significant, except for discriminative evaluation and frame accuracy scores for the dual learning module.
${}^{6}$  Statistical significance is computed with 100K samples using bootstrap (Noreen, 1989; Tibshirani and Efron, 1993).
Model	Char. F1	BLEU2/3	R-Precision	Frame Acc.	Top-1 Acc.	Top-2 Acc.
StoryGAN (Li et al., 2019b)	41.11	3.86 / 1.72	3.40 ± 0.01	21.90	22.42	45.40
StoryGAN + Transformer	42.45	3.92 / 1.73	4.03 ± 0.17	22.14	23.79	46.15
StoryGAN + MART	47.03	4.15 / 1.81	5.11 ± 0.12	22.25	24.48	46.42
+ Story Captioning	47.23	4.78 / 1.87	6.32 ± 0.08	22.30	24.53	47.41
+ Copy Transform	48.27	4.51 / 1.92	6.10 ± 0.07	22.71	25.62	47.39
Table 4: Ablation results on validation split of Pororo-SV dataset.
Figure 5: Comparative examples of generated images.
example A, we demonstrate the superior visual quality and consistency of the frames generated by DUCo-STORYGAN, as compared to StoryGAN. The MART encoder allows our model to comprehend long captions as well as attend to each word while generating images. The retention of native character features throughout the story during regeneration can be attributed to the copy-transform mechanism in our model. In contrast, we see that both models fail at generating defined characters in example B. This may be due to the fact that  $Poby$  is an infrequent character in the dataset and hence, both models fail to learn its features.
7.3 Linguistic Analysis
We perform visual analysis of the captions and predictions from DUCo-STORYGAN and observe two major recurring themes. First, the frequency of characters in the training data is a significant deciding factor for generated image quality. We looked at the samples that contained at least Pororo (most frequent character) and found that generated stories are better when there is only a single character in the frame's narrative as compared to multiple characters. This points to the inability of current story visualization models to align captions with multiple subjects/objects to the corresponding images. Second, generated images are poor for scenes containing infrequently occurring objects such as book/plate/boat/plane etc. in the caption. This behavior is expected since the model is unaware of real-world objects that do not already appear in the training set with sufficient frequency. Moreover, since the Pororo-SV dataset has been adapted from the annotations of a video QA dataset, the captions often contain information that can only span over multiple frames ("Pororo wakes up and then says what to Crong. Pororo stares at Pororo and throws a ball"), or cannot be visualized through images ("Poby gets the box and Eddy asks not to open it"). Hence, our results with metrics like BLEU and R-Precision which are supposed to capture the relevance between images and caption stay relatively low (see Tables 1 and 2).
In this paper, we investigate the underexplored task of story visualization. We improve upon prior modeling approaches and demonstrate the effectiveness of these new approaches by performing a robust set of ablation experiments. We also present a detailed set of novel evaluation methods, which we validate by demonstrating improvements across various baselines. Evaluation for story visualization is a challenging open research question in itself, and we hope that these methods will encourage more work in this domain.
9 Ethics/Broader Impacts
From an ethics standpoint, we provide a brief overview of the data that the model is trained on in Section 4 and a more detailed discussion in the Appendix. We provide some analyses of the data and refer the reader to the original StoryGAN paper, where the dataset was created, for further details. All of the language data consists of simple English sentences. Our experimental results are specific to the story visualization task. Pororo-SV is the most challenging story visualization task available; therefore, our results would likely generalize to other story visualization datasets. While story visualization is an exciting task with many potential future applications, the generated images still contain many obvious visual artifacts and therefore models trained on this task are still far from being deployed in any real world settings.
Story visualization minimizes many of the ethical issues associated with image and video generation. DeepFakes, which are algorithmically generated fake images, have become increasingly problematic (Nguyen et al., 2019). Oftentimes, these images are indistinguishable from real images, raising privacy concerns and providing a source of misinformation. The images that we generate here are not subject to this same issue, due to the fact that they are Cartoons, and are therefore unable to be confused with real images. The focus of the task is not on the realism of the images, but rather on the multimodal narrative. Therefore, cartoons are actually better suited for the task as real-world images only add additional visual complexity that is not relevant to the narrative.
We thank Peter Hase, Jaemin Cho, Hyounghun Kim, and the reviewers for their useful feedback. This work was supported by DARPA MCS Grant #N66001-19-2-4031, DARPA KAIROS Grant #FA8750-19-2-1004, ARO-YIP Award W911NF-18-1-0336, and a Google Focused Research Award. The views are those of the authors and not of the funding agency.
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. 2019. Conditional gan with discriminative filter generation for text-to-video synthesis. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial
Intelligence, IJCAI-19, pages 1995-2001. International Joint Conferences on Artificial Intelligence Organization.
Jianbo Chen, Yelong Shen, Jianfeng Gao, Jingjing Liu, and Xiaodong Liu. 2018. Language-based image editing with recurrent attentive models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8721-8729.
Yu Cheng, Zhe Gan, Yitong Li, Jingjing Liu, and Jianfeng Gao. 2020. Sequential attention gan for interactive image editing. In ACM MM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahhou, Yoshua Bengio, and Graham W Taylor. 2019. Tell, draw, and repeat: Generating and modifying images based on continual linguistic instruction. In Proceedings of the IEEE International Conference on Computer Vision, pages 10304-10312.
Lianli Gao, Daiyuan Chen, Jingkuan Song, Xing Xu, Dongxiang Zhang, and Heng Tao Shen. 2019. Perceptual pyramid adversarial networks for text-to-image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8312-8319.
Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. 2018. Imagine this! scripts to compositions to videos. In Proceedings of the European Conference on Computer Vision (ECCV), pages 598-613.
Qiuyuan Huang, Pengchuan Zhang, Dapeng Wu, and Lei Zhang. 2018. Turbo learning for captionbot and drawingbot. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6456-6466.
Kyung-Min Kim, Min-Oh Heo, Seong-Ho Choi, and Byoung-Tak Zhang. 2017. Deepstory: Video story qa by deep embedded memory networks. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2016-2022.
Gierad P. Laput, Mira Dontcheva, Gregg Wilensky, Walter Chang, Aseem Agarwala, Jason Linder, and Eytan Adar. 2013. Pixeltone: A multimodal interface for image editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '13, page 2185-2194, New York, NY, USA. Association for Computing Machinery.
Jie Lei, Liwei Wang, Yelong Shen, Dong Yu, Tamara L Berg, and Mohit Bansal. 2020. Mart: Memory-augmented recurrent transformer for coherent video paragraph captioning. arXiv preprint arXiv:2005.05402.
Chunye Li, Liya Kong, and Zhiping Zhou. 2020. Improved-storygan for sequential images visualization. Journal of Visual Communication and Image Representation, 73:102956.
Wenbo Li, Pengchuan Zhang, Lei Zhang, Qiuyuan Huang, Xiaodong He, Siwei Lyu, and Jianfeng Gao. 2019a. Object-driven text-to-image synthesis via adversarial training. In CVPR.
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. 2019b. Storygan: A sequential conditional gan for story visualization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6329-6338.
Yitong Li, Martin Renqiang Min, Dinghan Shen, David E Carlson, and Lawrence Carin. 2018. Video generation from text. In AAAI.
Yahui Liu, Marco De Nadai, Deng Cai, Huayang Li, and Bruno Lepri Xavier Alameda-Pineda and, Nicu Sebe and. 2020.Describe What to Change: a textguided unsupervised image-to-image translation approach. In ACM MM.
Ramesh Manuvinakurike, Jacqueline Brixey, Trung Bui, Walter Chang, Doo Soon Kim, Ron Artstein, and Kallirroi Georgila. 2018a. Edit me: A Corpus and a Framework for Understanding Natural Language Image Editing. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Ramesh Manuvinakurike, Trung Bui, Walter Chang, and Kallirroi Georgila. 2018b. Conversational image editing: Incremental intent identification in a new dialogue task. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 284-295.
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim. 2018. Text-adaptive generative adversarial networks: Manipulating images with natural language. In Advances in neural information processing systems, pages 42-51.
Thanh Thi Nguyen, Cuong M Nguyen, Dung Tien Nguyen, Duc Thanh Nguyen, and Saeid Nahavandi. 2019. Deep learning for deepfakes creation and detection. arXiv preprint arXiv:1909.11573, 1.
Eric W Noreen. 1989. Computer-intensive methods for testing hypotheses. Wiley New York.
Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. 2017. To create what you tell: Generating videos from captions. In ACM International Conference on Multimedia (ACM Multimedia).
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. 2019. Mirrorgan: Learning text-to-image generation by redescription. Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. CoRR, abs/1706.09799.
Jing Shi, Ning Xu, Trung Bui, Franck Dernoncourt, Zheng Wen, and Chenliang Xu. 2020. A benchmark and baseline for language-driven image editing. In ACCV.
Yun-Zhu Song, Zhi-Rui Tam, Hung-Jen Chen, Huiao-Han Lu, and Hong-Han Shuai. 2020. Character-preserving coherent story visualization. In Proceedings of the European Conference on Computer Vision (ECCV).
C Szegedy, V Vanhoucke, S Ioffe, J Shlens, and Z Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826.
Robert J Tibshirani and Bradley Efron. 1993. An introduction to the bootstrap. Monographs on statistics and applied probability, 57:1-436.
Ziteng Wen, Linbo Xie, Hongwei Feng, and Yong Tan. 2019. Robust fusion algorithm based on rbf neural network with ts fuzzy model and its application to infrared flame detection problem. Applied Soft Computing, 76:251 - 264.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. 2018. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR 2018.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. 2017. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pages 2849-2857.
Gangyan Zeng, Zhaohui Li, and Yuan Zhang. 2019. Pororogan: An improved story visualization model on pororo-sv dataset. In Proceedings of the 2019 3rd International Conference on Computer Science and Artificial Intelligence, pages 155-159.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. 2017. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. 2019. DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5802-5810.
A Methods
StoryGAN only uses pretrained sentence embeddings as a representation for the caption, while DUCo-STORYGAN uses a combination of sentence and word embeddings. The context encoder is responsible for encoding the captions and transforming them into image embeddings.