SlowGuess/NAACL_2022_OCR huggingface dataset:
sub-stories for each image, while the reward model is a CNN-based model designed to output the story reward. 2) GLACNet (Kim et al., 2018): combines both local and global attention. Image features are fed sequentially to a bi-LSTM where the output is a global representation of the entire story. This is concatenated with local image-specific features to create glocal vectors which are passed to a decoder for story generation. 3) KG-Story (Hsu et al., 2020): For each image, a word-form conceptual representation is created by predicting a set of terms which are then used to query Visual Genome (Krishna et al., 2017) and OpenIE (Pal et al., 2016) to identify links between sets of terms across images. Finally, a Transformer (Vaswani et al., 2017) takes in the term paths to decode the story. 4) MCSM+BART (Chen et al., 2021): image concepts and related concepts extracted from ConceptNet (Liu and Singh, 2004) are used as input for generating richer stories with BART (Lewis et al., 2020). To incorporate the most appropriate concepts, their Maximal Clique Selection Module model learns a correlation map, reflecting cooccurrence probabilities of all candidate concepts.
Evaluation Metrics To assess the performance for RoViST, we analyze its correlation with reliable human judgements by recruiting many responders (26) whereas related works (Guan and Huang, 2020; Hu et al., 2020) have used 3-7 annotators. In total, the 26 responders analysed 400 machine generated sentences across 80 stories and 4 models, including AREL, GLACNet, KG-Story and MCSM+BART. A Likert scale was used to score 3 different criteria for each story based on what we believe defines a good story - 1) the story is visually grounded, 2) sentences are natural sounding and topically coherent, and 3) there is no repeating plots within the story. Annotators were additionally asked to vote for which of the 4 models produced the best story relating to the visual prompt based on no particular criteria. We follow existing literature and report the Spearman's correlation  $\rho$ , Pearson's correlation  $r$  and Kendall's correlation  $\tau$ .
Baseline We select 11 baseline metrics to compare with our metric: BLEU-1,2,3,4 (Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR
AREL: I went on a boat trip to the lake . This is a picture of a lake. This is a picture of a field. The water was clear and the water was calm. The boats were docked in the water .	Grounding Score: 3.4 / 5 Coherence Score: 2.7 / 5 Non-redun. Score: 2.7 / 5 Proportion of votes: 4%
GLACNet: The cruise ship was getting ready to go. They were off to a great spot. The beach was beautiful. There was a lot of boats. It was a very nice day.	Grounding: 2.5 / 5 Coherence Score: 3.3 / 5 Non-redun. Score: 4.6 / 5 Proportion of votes: 12%
KG-Story: We had a great time at the lake. There were so many boats. It was very beautiful. I spent all day out on this field. And even saw one boat.	Grounding: 2.8 / 5 Coherence Score: 3.3 / 5 Non-redun. Score: 3.9 / 5 Proportion of votes: 24%
MCSM+BART: We had a nice summer in [location]. The fields were absolutely gorgeous. We also had a farm that looked like a real farm all around. There were even boat campsites. Some of the boats were out at night to camp.	Grounding: 3.7 / 5 Coherence Score: 3.6 / 5 Non-redun. Score: 4.1 / 5 Proportion of votes: 60%
Figure 3: Average human scores for an example story across 3 criteria for 4 different VST models. 'Proportion of votes' refers to the percentage of voters who voted that model's story as the best out of the 4. Blue highlighted words visually relate to the image.
(Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016), WMD (Kusner et al., 2015),  $F_{BERT}$  (F1-measure version of BERTScore) (Zhang et al., 2019) and TIGEr (Jiang et al., 2019).
6.1 Human Scores versus Story Ranking
We first investigate whether there is any correlation between the human scores for each 3 criteria and the model that was voted as the best for each photo sequence. For each photo sequence, we rank each of the 4 models' stories based on the proportion of votes that it received. The correlations were then calculated between the mean human scores for each criteria and the model rankings, and the average correlation coefficients were finally taken across the unique stories to obtain the values in Table 2. We also sum up the human scores across the 3 criteria and measure its correlation with the rankings to further analyze at an Overall level.
Interestingly, we find that sentence coherence plays the most significant role when ranking stories whereas non-redundancy and visual grounding are less important. Figure 3 provides an example of this case where our human annotators preferred KG-Story and GLACNet over AREL which was more visually grounding but less coherent-sounding. We observe even stronger correlation when we sum the 3 criteria scores, suggesting that all 3 aspects combined can give better guidance when judging a story as can be seen in Figure 3 where most of the votes went to  $\mathrm{MCSM + BART}$  which scored relatively well in all 3 areas.
2The implementation details can be found in the Appendix
Grounding	Coherence	Non-re Dun	Overall
ρ	r	τ	ρ	r	τ	ρ	r	τ	ρ	r	τ
BLEU-1	0.198	0.168	0.127	0.052	0.044	0.030	0.018	-0.044	0.019	0.080	0.064	0.051
BLEU-2	0.261	0.233	0.181	0.057	0.057	0.037	-0.028	-0.148	-0.006	0.066	0.035	0.049
BLEU-3	0.259	0.229	0.173	0.121	0.160	0.083	-0.073	-0.165	-0.053	0.065	0.062	0.043
BLEU-4	0.225	0.134	0.148	0.121	0.082	0.077	-0.075	-0.195	-0.058	0.051	-0.027	0.026
ROUGE-L	0.244	0.222	0.164	0.197	0.161	0.127	-0.039	-0.138	-0.021	0.109	0.075	0.077
METEOR	0.348	0.319	0.228	0.291	0.256	0.213	0.203	0.075	0.140	0.327	0.280	0.223
CIDEr	0.269	0.158	0.194	0.207	0.104	0.146	0.013	-0.190	0.005	0.182	-0.005	0.131
SPICE	0.311	0.301	0.214	0.052	0.069	0.031	0.018	-0.051	0.015	0.127	0.134	0.095
WMD	0.472	0.490	0.337	0.186	0.236	0.129	0.106	0.015	0.076	0.262	0.312	0.183
FBERT	0.180	0.175	0.149	0.287	0.320	0.202	0.088	0.038	0.061	0.199	0.218	0.128
TIGEr	0.519	0.504	0.354	-0.03	-0.089	-0.027	-0.224	-0.325	-0.147	0.010	-0.005	0.010
RoViST(-VG/C/NR)	0.509	0.460	0.365	0.446	0.456	0.308	0.531	0.736	0.397	0.554	0.579	0.387
Table 1: Criteria level Spearman's  $\rho$ , Pearson's  $r$  and Kendall's  $\tau$  correlations between automatic metrics and mean of human scores. Correlations for Grounding, Coherence, Non-re Dun and Overall are measured with RoViST-VG, RoViST-C, RoViST-NR and RoViST respectively.
ρ	r	τ
Grounding	0.423	0.434	0.400
Coherence	0.663	0.698	0.618
Non-re Dun	0.379	0.484	0.328
Overall	0.754	0.769	0.676
Table 2: Criteria level Spearman's  $\rho$  ,Pearson’s  $r$  and Kendall’s  $\tau$  between human scores and story ranking.
6.2 Correlation Analysis with Human Scores
Table 1 displays the correlation between the metrics and the mean human scores. The results were analyzed at a criteria level by examining correlations between each criteria's scores with our metric which targets that criteria. We also analyze the Overall scores by summing up the 3 criteria scores and measuring its correlation with RoViST which represents the sum of the scores produced by RoViST-VG, RoViST-C and RoViST-NR.
With the grounding correlations, RoViST-VG outperforms the baselines for Kendall's correlation. However, it is slightly outperformed by TIGEr when comparing Spearman's correlation and by TIGEr and WMD when comparing Pearson's correlation. We note that all baseline metrics are reference-based and therefore, a likely explanation for the moderate correlations for even simple metrics like METEOR is that human references can already provide a good guideline when assessing text-to-image relatedness. Moreover, we hypothesize that image captioning metrics will perform well for the visual grounding aspect in the case when the model happens to output a sentence that sounds like an image caption. However, unlike image captioning, we emphasize that just having high correlation between image objects and text descrip
Figure 4: Kendall (left) and Spearman (right) correlation vs. Number of References.
tions does not necessarily mean a good story as we highlighted in the previous section. Examining the coherence and non-redundancy aspect, we observe that a majority of the baselines correlate poorly. Conversely, our RoViST-C and RoViST-NR metric designed to specifically target these criteria generated significantly higher correlations. When comparing at the Overall level, we also achieved noticeably better results in terms of  $\rho$ ,  $r$  and  $\tau$ .
6.3 Changing Number of References
Figure 4 shows how the Spearman and Kendall correlations for some of the metrics vary with different number of human-written references versus our reference-less metric. The stories selected for our analysis each have a different number of reference stories ranging from 1 to 4. As there were not many stories with 4 references, we select those stories that had 3 references, resulting in 60 stories with 300 sentences for analysis. We then compute the correlations with the human judgement across the metrics using 1,2, and 3 references.
It is evident that the results from the reference-based metrics fluctuate significantly according to the number of references. However, the trend is unclear. Increasing the number of references from
Figure 5: Predicted coherence probabilities from RoViST-C for 4 VST models.
1 to 2 appears to improve the correlations for some of the metrics like CIDEr, SPICE and ROUGE-L. This may be because having more references can better capture allowable variations in storytelling compared to a single reference. However, increasing from 2 to 3 references actually worsens the performance for many of the metrics (BLEU-1/4, CIDEr, METEOR and ROUGE-L). A possible explanation could be that the additional reference added may have caused bias for some metrics. In particular,  $n$ -gram based metrics like BLEU and ROUGE focus on  $n$ -gram overlap. Thus, it is possible that the additional reference introduced may have a high  $n$ -gram overlap with the candidate but for unimportant filler words like 'the' or 'and'. Our metric on the other hand, alleviates this issue by first being a reference-free metric and secondly, by only focusing on important words (nouns) in the candidate story via RoViST-VG.
It is also noted that examining more amount of references could potentially reveal a better trend. However, this is challenging as the maximum amount of references in the VIST dataset is 5 with  $82.50\%$  of the stories having 3 or less. Moreover, collecting multiple human reference stories is an expensive process in most cases.
6.4 Qualitative Analysis
We conduct qualitative analysis on our visual grounding scorer (RoViST-VG) and coherence scorer (RoViST-C).
RoViST-VG Figure 6 in Appendix A displays an example gold story with noun mentions highlighted in blue, followed by the corresponding bounding box regions that gave the highest similarity score retrieved by our RoViST-VG model. We observe
that the model performs well at matching a majority of the nouns. However, words that are less visually grounding like 'corner' or intangible nouns such as 'visit' are extremely challenging to ground. Consequently, RoViST-VG can sometimes retrieve a region that is not closely related for these types of words. This also occurs for words that are mentioned in the story but not explicitly shown in the images like the word 'photos' in Example Story 2. A potential problem of this may be the presence of false positives if a story tends to mention many nonvisual entities. This could lead to a higher grounding score compared to a story that only mentions a few entities that are visually grounded. Nevertheless, our model still serves as guidance for analyzing how visually detailed a story is and can also reflect how many related entities are mentioned.
RoViST-C The qualitative results for 4 example machine stories is displayed in Figure 5. Noticeably, RoViST-C tends to assign higher probabilities to sentences that flow. These sentences do not necessarily need to be about the same topic. For instance, sentence 2 and 3 in AREL's story each have a different topic focus but the sentence transition is given a 0.90 coherence score as they follow a narrative style. Conversely, consecutive sentences with similar topics but are incoherent can be given low scores such as sentences 4-5 from KG-Story. It is clear that training ALBERT with sentence order prediction allows the model to capture inter-sentence coherence and is not just limited to modelling topic similarity across sentences.
We propose RoViST, a metric for evaluating VST tasks on 3 aspects: visual grounding, coherence and non-redundancy. RoViST correlates well with human judgement, outperforming other metrics when comparing the coherence and non-redundancy criteria as well as when combining all 3 criteria. While some existing metrics slightly outperform our method on visual grounding, we note that image-to-text similarity is just one aspect of VST and this aspect alone is insufficient in defining a good story. Unlike other metrics, RoViST is reference-free and hence, robust to the number of references which are costly to obtain for VST. It is also interpretable and can be used to gauge out where the model is underperforming. We hope that RoViST provides preliminary insight into future work on developing VST models and evaluations.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pages 382-398. Springer.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Hong Chen, Yifei Huang, Hiroya Takamura, and Hideki Nakayama. 2021. Commonsense knowledge aware concept selection for diverse and informative visual storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 999-1008.
Yin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge Belongie. 2018. Learning to evaluate image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5804-5812.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth  $16 \times 16$  words: Transformers for image recognition at scale.
Jian Guan and Minlie Huang. 2020. Union: An un-referenced metric for evaluating open-ended story generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9157-9166.
Chao-Chun Hsu, Zi-Yuan Chen, Chi-Yang Hsu, Chih-Chia Li, Tzu-Yuan Lin, Ting-Hao Huang, and Lun-Wei Ku. 2020. Knowledge-enriched visual storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7952-7960.
Ting-Yao Hsu, Chieh-Yang Huang, Yen-Chia Hsu, and Ting-Hao Huang. 2019. Visual story post-editing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6581-6586.
Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2020. What makes a good
story? designing composite rewards for visual storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7969-7976.
Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1233-1239.
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. 2019. Tiger: Text-to-image grounding for image caption evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2141-2152.
Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung-Wha Park, and Byoung-Tak Zhang. 2018. Glac net: Glacial attention cascading networks for multi-image cued story generation.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73.
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learning, pages 957-966. PMLR.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Hwanhee Lee, Seunghyun Yoon, Franck Dernoncourt, Doo Soon Kim, Trung Bui, and Kyomin Jung. 2020. *Vilbertscore: Evaluating image caption using vision- and language bert.* In *Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems*, pages 34–39.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pages 201-216.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart:
Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211-226.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13-23.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Harinder Pal et al. 2016. Demonyms and compound relational nouns in nominal open ie. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, pages 35-39.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer imageto-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91-99.
Naeha Sharif, Lyndon White, Mohammed Bennamoun, and Syed Afaq Ali Shah. 2018. Nneval: Neural network based evaluation metric for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 37-53.
Ritika Singh and Satwinder Singh. 2021. Text similarity measures in news articles by vector space model using nlp. Journal of The Institution of Engineers (India): Series B, 102(2):329-338.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575.
Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018. No metrics are perfect: Adversarial reward learning for visual storytelling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 899-909.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13041-13049.
A RoViST-VG Example Output
Figure 6 shows the retrieved regions from the RoViST-VG model for an example gold story (top) and a machine-story generated from the  $\mathrm{MCSM + BART}$  model (bottom). The blue highlighted words are the nouns while red highlighted words indicate words that do not explicitly appear in the image sequence or are less visually grounding words.
RoViST-VG We use the Adam optimizer (Kingma and Ba, 2014) with a 0.00001 weight decay. The learning rate was initially set to 0.00005 and was reduced by  $5\%$  with each consecutive epoch. For the ViT model, we use the 'vit-base-patch16-224' style configuration which outputs image features as a 768 dimensional vector. Further, the linear layer used to project the text and embedding features to the joint embedding space (of dimension 1024) uses a tanh activation function. No normalization of the image and text embeddings was done during the training process as we did not find any benefit from doing this. Finally, we set the mini-batch size to 64 and use early stopping to cease training
after the validation loss fails to improve for 3 consecutive epochs. We note that  $85\%$  and  $15\%$  of the data was used in the training and validation set respectively. The model converged in 3 epochs, taking approximately 12 hours with a Nvidia Tesla P100 GPU.