GROOViST: A Metric for Grounding Objects in Visual Storytelling
Authors: Aditya Kaushik Surikuchi
Sandro Pezzelle
Raquel Fernández
Venue: EMNLP 2023 Main (2023)
Keywords: visual storytelling
grounding
NLG evaluation

Abstract: A proper evaluation of stories generated for a sequence of images---the task commonly referred to as visual storytelling---must consider multiple aspects, such as coherence, grammatical correctness, and visual grounding. In this work, we focus on evaluating the degree of grounding, that is, the extent to which a story is about the entities shown in the images. We analyze current metrics, both designed for this purpose and for general vision-text alignment. Given their observed shortcomings, we propose a novel evaluation tool, GROOViST, that accounts for cross-modal dependencies, \textit{temporal misalignments} (the fact that the order in which entities appear in the story and the image sequence may not match), and human intuitions on visual grounding. An additional advantage of GROOViST is its modular design, where the contribution of each component can be assessed and interpreted individually.

GROOViST: A Metric for Grounding Objects in Visual Storytelling
Aditya K Surikuchi University of Amsterdam a k. sur surikuchi@uva.r nl
Sandro Pezzelle, Raquel Fernández ILLC, University of Amsterdam {s. pezzeezel,,, raquel. fernandez)@uva.nit
Abstract
proper evaluation of stories generated for sequence of images-thh the task commonly re- ferred to as visual storytelling-mus consider multiple aspects, such as coherence, grammat- ical correctness, and visual grounding. In this work, we focus on evaluating the degree of grounding, that is, the extent which story is about the entities shown in the images. We analyze current metrics, both designed for this purpose and for general vision-text alignment. Given their observed shorteomings, propose a novel evaluation tool, GROOViST, that ac- counts for cross modal dependencies, temporal misalignments (the fact that the order in which entities appear the story and the image se- quence may not match), and human intuitions on visual grounding. An additional advantage of GROOViST its modular design, where the contribution of each component can be assessed and interpreted individually.
there was lots to see and es the fostival including lisiening to unusual isstnimenss. 2) many stalss had hindmade clething and une even had dresses specifically for little girls. 3) part the festival grounds there were alsa numerous sculpcures that one could touch, stalls aderned with fandmade plass bottke .5) miklay thousands aseedancce
One story and corresponding image sequence from the VIST dataset. Noun phrases in green contribute positively to the grounding score by GROOViST; those in red contribute negatively. The GROOViST score for this sample 0.855, i.e., our metric considers as well- grounded (within range: [-1, 1J). Best viewed in color.
Figure I:
appropriate- they indeed poorly correlate with human judgments (Wang al., 2018). Morcover, proper evaluation must consider multiple aspects, such as coherence, grammaticality and, impor- tantly, visual grounding. Yet, most evaluation metrics proposed specifically for visual storytelling do not consider the images at all (Hu et al., 2020).
1 Introduction
In this paper, we focus on evaluating a story's degree of grounding, that is, the extent which story is about the entities shown in the images. To the best of our knowledge, there only one metric proposed to date for evaluating grounding in visual storytelling, the Visual Grounding scorer (Ro ViST-V)) by Wang et al. (2022). We carry out an extensive analysis of this metric and reveal that it has critical shortcomings. To overcome this, propose novel, modular evaluation tool, which we name GROOViST (grounding objects in visual storytelling). We show that GROOViST is robust to temporal misalignments, correlated with human intuitions about grounding. and easy to interpret. Our code is available at: https: http//gttttbbbb comm/akskucii/rrooiiiii
Generating textual story that is plausible given a sequence of images a challenging task involving aspects such as cross-modal interactions, temporal dependencies between linguistic and visual content, and causal reasoning. In the language-and-vision community, Huang et al. (2016) operationalized the task and released the Visual Storytelling Dataset (VIST), collection English stories created by speakers on top of 5-image visual sequences. Sev- eral models have been proposed for the task of generating plausible stories for a given sequence, ranging from RNNs (Kim et al., 2018) to Trans- formers, trained either end-to-end or leveraging additional knowledge-gnne graphs (Chen et al., 2021). Evaluating the quality of the automatically generated stories is extremely difficult: Given the creative nature of the task (many stories could be sensible for a given image sequence), reference- based metrics like METEOR (Banerjee and Lavie, 2005) or CIDEr (Vedantam et al., 2015) are not
Analyses of Existing Metries
To assess the level of visual grounding story in visual storytelling, Wang et al. (2022) proposed

---

RoViST-VG, This metric is the output of model pre-trained on the Flickr30K Entities dataset (Plum- mer et al., 2015) to learn the relationships between the nouns in a story and the regions of an image in a contrastive learning regime. For a given <image- sequence, story> pair, RoViST-VG extracts: from each image, the bounding boxes and correspond- ing visual features of its 10 most salient regions, using FasterRCNN (Ren al., 2015); from the story, the Glo (Pennington et al., 2014) represen- tations of each noun in The pre-trained model receives these extracted embeddings (from GL and Fa)terRCNN) and returns the final representa- tions T and I, respectively. The grounding score is then calculated using Eq. (1) as the maximum cosine similarity between T and I, weighted by inverse document frequencies (idf) of the nouns.
VIST ofiginel parr VI51 rendom pairs
AESOP ofiginal pairs AESOP random pairs
VIP original pairs VWP random pairs
1,000.00
944
RUAAIST-VG
CIROOVST
Ro VoV does not exhibit the expected pattern: it does not assign lower scores to the random <images, story> pairs. contrast, this case for CLIPScore and our proposed GROOViST metric.
Figure 2:
of movie shots, each including 5-10 images with corresponding sentences that make up their stories.
We compute RoViST-VG and CLIPScore on the original <image sequence, story> pairs in the test splits of these datasets, and compare these scores to the ones obtained a random setting where each image sequence is paired with five random stories (from the corresponding dataset); among these, we consider the pair that receives the highest score. We expect a metric that properly captures visual grounding to assign higher scores to the orig- inal stories than to the randomly paired stories.
RoViST-VG log
(xP max (cos(T.,s (I)
To analyze the suitability of RoViST-VG, we compare it to CLIPScore (Hessel et al., 2021).
CLIPScore has not been designed to evaluate visual storytelling. Here, we use it score each image- sentence pair independently in a story sequence. This approach not ideal as it cannot capture tem- poral misalignments between a text and a visual content (e.g., an early sentence may be 'they were getting ready to go to the circus' but the circus may only appear later). However, since CLIPScore has been designed for general vision-text alignment, we expect it to be reasonably effective at capturing visual grounding at the mage--enneence level. It cor- responds the cosine similarity between CLIP's (Radford et al., 2021) representations of a sentence e and an image v (with 2.5 as re-scaling factor).
Figure 2 shows the average scores of the metrics in both settings. Surprisingly, RoViST-VG scores are not higher the original setting than the ran- dom setting. In fact, on VIST, the random <image sequence, story> pairs receive higher RoViST-VG scores than the original ones. In contrast, CLIP- Score follows the expected pattern.
2.2 Correlation with Flickrök-Expert ratings
We assess the ability of the two metrics to capture general image-caption grounding using Flickr8k- Expert (Hodosh et al., 2013), publicly available dataset with human ratings for image-caption pairs. In particular, we consider the subset of 3391 sam- ples where all three annotators agree. 3 CLIPScore is designed for this purpose and therefore well- suited for the task. RoViST-VG is not meant for measuring image-caption grounding, although it should align with human ratings to some extent, given its purpose and pre-training. However, as we can see in Table 1, RoViST-VG shows correla- tion with human vatings-whin CLIPScore does.
Next we explore how good the above metrics are at capturing grounding in visual storytelling data.
2.1 Grounding in visual storytelling datasets We analyze the scores assigned by these metrics to the stories in three visual storytelling datasets: (1) VIST (Huang et al., 2016), that comprises se- quences of five natural images (from Flickr) and corresponding five-sentence stories; (2) AESOP (Ravi et al., 2021), that includes sequences of three synthetic images (created using entities from Ab- stract Scenes; Zitnick and Parikh, 2013) and cor- responding paraggaa long stories; (3) VWP (Hong et al., 2023), which comprises sequences
25055 samples for VIST and 991 for AESOP. Due to the lack of separate test split for VWP, we considered all 13843 samples in dataset.
More details on RoViST-VG are provided in Appendix C.
"Human annotators rated captions on a scale of 4.

---

RoViST-VG
CLIPScore
GROOViST
nouns and decrease the scores of some frequent- and-grounded nouns, adversely affecting the over- all score. Hence, after the penalization step, we use word concreteness ratings (Brysbaert et al., 2014) for weighting the resulting scores (instead of idf) and capture the fact that concrete NPs are more likely to be visible.
To (p-value) 0.019 (0.125) 0.556 (0.0) 0.414 (0.0)
Table I:
Correlation (Kendall re) between Flickr8k- Expert human ratings and the automatic metrics.
3 GROOViST
Normalization Finally, to obtain the GROOViST score of story, we aggregate the weighted scores of all its NPs and normalize the sum by the total number of NPs in the story, which results in value unaffected by story length (or more precisely, by the number of NPs in it):
Our analyses showed that RoViST-VG has some important limitations as a metric for assessing the degree of visual grounding- bothinggg--thhhhoe in stories and image captions. To overcome these issues, we pro- pose GROOViST, a modular metric consisting of various components informed by insights from both CLIPScore and RoViST-VG. These are:
E NP pos, PPPS
T NP meg.) (n + m) (2)
where n and m are the number of NPs with positive and negative scores, respectively. See Figure for how this facilitates interpretability. The pseudo- code and working example for GROOViST are provided in Algorithm and Figure 4, respectively. GROOViST scores are unboumded by default, but tanh can be used to map them to the [-1, 1] range.
Noun phrase (NP) extraction We process the story and extract all the NPs; this is similar to RoViST-VG but better because RoViST-VG only considers nouns and fails to handle compounds such as "parking lor'. Additionally, focusing NPs allows for the contribution of accompanying adjectives (e.g.. faces').
Role of GROOVIST Components
Vision- languaae alignment We compute align- ment scores between all the extracted bounding boxes and NPs and select the highest score for each NP. This step is similar to RoViST-VG but, instead of training a dedicated model, use the off-the- shelf CLIP (Radford al., 2021) model.
We test GROOViST on the same evaluation criteria used in Section 2. From Figure 2 and Table 1, we observe that GROOViST fares well both eval- uation criteria. First, it assigns higher grounding scores to original compared to random stories. Sec- ond, moderately correlates with human image- caption ratings. This indicates that GROOViST is more robust metric than RoViST-VG.
Penalizing poorly grounded NPs The previous steps result in a positive score for all the NPs in a story. Yet, some may in fact be poorly grounded (i.e., have low visual alignment score). Such NPs, therefore, should contribute negatively to the over- all degree of grounding of story. To operationalize this, we select the mean score over all NPs in the entire dataset tance of each NP's score from f. assigning negative values to NPs with scores below 0 (NP neg) while retaining the scores of NPs with values above (NP
To understand the impact of GROOViST's com- ponents the final grounding score, conduct several experiments by both ablating the compo- nents and replacing them with plausible alterna- tives.
threshold A and calculate the dis-
Ablations Penalizing poorly grounded NPs and Concreteness weighting are the two components of GROOViST that can be ablated from the metric.
Replacements The Concreteness weighting and Noun phrase (NP) extraction components of GROOViST can be replaced with idf weights and nouns, respectively.
Concreteness weighting RoViST-VG uses in- verse document frequencies (idf) for weighting the similarity scores of nouns handle abstract frequent words such as 'time". However, we ob- serve that idf weights tend to increase the simi- larity scores of some less-frequent non-grounded
In total, we consider six alternative versions of our metric, which we obtain by applying all pos- sible combinations of ablations and replacements.
'Examples are provided Appendix A. *98.7% of NPs in the VIST tost set contain words for which concreteness ratings are available.
'Using spaCy's 's English transformer pipeline for chunking: https://spacyy ht/models/////eeooottctsssclllss ht////eeeeeeeennnoo/o/////e/// www.eeenncreeweee..e

---

We test these versions on the same evaluation crite- ria used in Section 2. Table 2 reports how they fare with respect to the two criteria we consider.
indicates that most aspects mentioned in the story are depicted in the sequence of images"). We formulate two hypotheses about the strengths and weaknesses of GROOViST and CLIPScore and ex- perimentally test their validity using the human grounding judgments.
Criterion
Criterion 2
GROOViST
V
GROOVIST (-C)
GROOVIST (-P)
4
5.1 Temporal misalignment
GROOViST (-C-P)
V
Effective metrics for measuring grounding in vi- sual storytelling should account for possible tem- poral misalignments between the visual and tex- tual modality. That is, they should account for the fact that entities that are grounded in an image could be mentioned earlier or later in the story- not necessarily in the corresponding sentence. We hypothesize that GROOViST-sinee it takes into account the entire story ter with human judgments than CLIPScore sam- ples with high temporal misalignment. To test this hypothesis, define temporal misalignmenn t of sentence; in a sequence as the number of its NPs matching with visual entities in images (img; at other positions of the sequence, normalized by the total number of its NPs. The overall temporal misalignment T of story is then the average of its sentence- t values:
GROOVIST (-NPs +Ns)
GROOVIST (-C +iff)
V
+
GROOVIST (-C +idf -NPs +Ns)
Table Results of ablating and replacing different com- ponents of GROOViST. C and P refer to Concreteness weighting and Penalizing poorly grounded NPs compo- nents respectively. v indicates that the criteria are met; x indicates that the criteria are not met; J. for Criterion indicates a deterioration the ability the metric distinguish between original and random stories. for Criterion 2 indicates a decrease in the correlation of metric scores with FlickrSk- Frickrsk-Expert ratings.
bet-
We observe that ablating or replacing compo- nents from GROOViST results in scores that either do not meet at least one of the criteria do so much lower extent.? This is particularly ap- parent in the metric versions where the Penalizing poorly grounded NPs component ablated, which further confirms its importance. The GROOViST (-C +idf) version satisfics Critcrion 1, indicating that frequency-based information can be helpful as a heuristic. However, it may result in discrepancies shown in Appendix A, Figure 4. We consider concreteness to be more theoretically motivated notion than frequency to capture visual grounding. Its value is apparent with respect to Criterion 2: re- placing Concreteness weighting with idf weighting decreases the correlation of the metric scores with Flickr8k-Expert ratings.
((eentence)) # NNss matching imgjæs)
(3a)
# (NPs in sentence;)
T(story) = Σ ((sentence,) /
(3b)
where n is the number of sentences in a story. We consider a story to have high temporal mis- alignment if T 2 1.0, i.e., at least many the average number of NPs per sentence are misaligned. In the annotated data, T [0.16, 1.53) and 18% of the stories exhibit high temporal misalignment, indicating the prevalence of the phenomenon.
As can be seen in Figure 3, our hypothesis is confirmed: GROOViST exhibits a higher correla- tion with human ratings than CLIPScore on sam- ples with a high T, i.e., its scores are overall more aligned with human intuitions when in the presence of temporally misaligned entities. This confirms the ability of GROOViST handle non-trivial grounding dynamies in a story, different from CLIP- Score. At the same time, we notice that CLIPScore achieves a higher correlation than our metric in samples with low T. which confirms once again that the former is an effective tool for capturing grounding in well-aligned multimodal data.
5 Evaluation of GROOVIST
To further evaluate the extent to which GROO- ViST captures intuitions stories' degree of visual grounding, we compare our metric to human judg- ments. Since no previous work collected human data for this specific purpose, we run a small data collection by asking 5 participants to rate sample of the VIST data. In particular, we ask partici- pants to provide ratings for 100 randomly sampled VIST <image sequence, story> pairs, using 4- point Likert-like scale (instructions: "a score of
"The resulting values are provided in Appendix E.
# AApeendix B provides further details.

---

5.2 Proportion of noun phrases
GROONEST
GROOViST builds on noun phrases. As explained above, this has some obvious advantages, e.g., it al- lows to mcasure the individual contribution of cach NP toward the final score (see Figure 1), but also some possible limitations. For example, we hypoth- esize that GROOViST scores may be dependent on the number of NPs: for stories where grounding hinges mostly on NPs, we expect GROOViST to be well aligned with human intuitions; less so when hinges on verbs, for example, in which case CLIP- Score may be better. To test this hypothesis, we define proportion-of-NPt (P) of story as the frac- tion of NPs to all the words in the story:
Typothoss
0.54
typortesis
5.43
34-
0.37
0.23
-31
-212
Sang01 with
with low
with high
low
Figure 3: Kendall's 7 ((vrrannt= 'c") correlations of all grounding metrics with human scores for temporal mis- alignment (left) and noun phrase proportion (right).
GROOViST is modular, highly interpretable, and aligned with human intuitions on visual grounding Preliminary results indicate that GROOViST is a suitable tool for evaluating automatically generated stories. We plan to test this aspect extensively future work.
# #NPs in story)
P(story) # (all words in story)
We focus the subset of <image sequence, story> pairs with high human ratings, 9 to ensure our analysis genuinely explores the role of NPs in well-grounded stories without being influenced by other factors. We then compute P values for these sequences and bin them into two sets sots-low P and high P-using the distribution's 's mode (0.2325))." 10 The high P bin comprises 32.7% of the total num- ber of subset samples.
Limitations
In this section, we discuss the limitations specific to our metric and to the general reference-free evalua- tion paradigm. As discussed in Section 5.2, GROO- ViST is heavily dependent on noun phrases making it oblivious to other visually informative words, such verbs. For identifying poorly grounded NPs, GROOViST relies on threshold value, which determined based on the dataset of interest. This makes GROOViST vulnerable to the skew the dataset. Despite our preliminary analysis, GROO- ViST's evaluation of model-generated stories is yet to be fully tested. Also, in general, reference- free metries rely
In Figure 3, firmed. GROOViST scores turn out to be very well aligned with human intuitions-aat and indeed signif- icantly more correlated than CLIPScore-in the high P bin. In contrast, our metric lags behind CLIPScore in the low P bin, though the distance between the metrics is rather small, and the two metrics generally achieve very low correlations. Although the dependency of GROOViST the proportion of NPs in story might be seen limitation of the metric, argue that nouns and accompanying phrases tend to offer the most visual information (Wang et al., 2022). As for Ro V- VG, it achieves a very low correlation with human ratings in both analyses, which confirms its flaws.
that our hypothesis is con-
underlying pre-trained model, which often is stagnant in learning and might quire regular fine-tuning for prolonged future rele- vance. We would also like to underline that through- out this work, we only used and evaluated mod- els trained the English language text. However, given the modularity of GROOViST, it is possi- ble to switch to models such as multilingual-CIn CLIP (Carlsson et al., 2022).
6 Conclusion
Ethics Statement
We proposed GROOViST, novel reference-free metric for evaluating grounding in visual story- telling, an aspect that surprisingly often over- looked in this task. We showed that existing met- rics have serious shortcomings, and analyzed the strengths and limitations of our proposed metric.
For collecting human judgments, we recruited par- ticipants voluntary basis among collcagues of our institution. All data collected for this work is de-identified ensure the privacy and security of everyone involved. The authors of the VIST dataset (Huang et al., 2016) mention that all images used are CC-licensed.
"Human rating 2 on a scale to 4.
The same results also hold when using mean and median.

---

Acknowledgements
Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2020. What makes a good story? designing composite rewards for visual sto- rytelling. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):999--7978
We thank the participants of the human evaluation study and the members of the Dialogue Modelling Group for their vital feedback on the design and experiments. Furthermore, anonymous EMNLP reviewers for their helpful sug- gestions and for engaging in fruitful discussions. AKS is supported by the TIMELY project under EU H2020 grant 101017424. RF is supported by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and inno- vation programme (grant agreement No. 819455).
are grateful to the
Ting- Hao Kenneth Huang, Francis Ferraro. Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Push- meet Kohli, Dhruv Batra, C. Lawrence Zitmick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Mar- garet Mitchell. 2016. Visual storytelling. In Pro- ceedings of the 2016 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1233-1239 San Diego, California. Association for Computational Linguistics.
References
Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung- Wha Park, and Byoung-Tak Zhang. 2018. GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation. CoRR, 00//1805.19973
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Sumnarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empiricul Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Mare Brysbaert, Amy Beth Warriner, and Victor Ku- perman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior Research Methods, 46:904-911 911.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrann correspondences for richer image- to-sentence models. In Proceedings of the IEEE In- ternational Conference on Computer Vision (ICCV).
Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. 2022. looss-lingual and multilin- gual CLIP. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 6848-6854, Marseille, France. European Language Resources Association.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Hong Chen, Yifei Huang, Hiroya Takamura, and Hideki Nakayama. 2021. Commonsense knowledge aware concept selection for diverse and informative visual storytelling. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2):999-10) 1008.
Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. 2021. Acsop: Ab- stract encoding of stories, objects and pictures. In Proceedings of the IEEE/CVF International Confer- ence Computer Vision (ICCV), pages 2052-2063.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A reference-fre free evaluation metrie for image captioning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 7514-7528, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-enn: Towards real-time object detection with region proposal networks. In Ad- vances in Neural Information Processing Systems, volume 28, Curran Associates, Inc.
Micah Hodosh, Peter Young, and J. Hockenmaier. 2013, Framing image description as a ranking task: Data, models and evaluation metrics (extended abstract). J. Artif. Intell. Res., 47:853-899.
Ramakrishna Vedantam, C Lawrence Zitnick and Devi Parikh. 2015. CIDEr: Consensus based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4576-4575
Xudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demherg, and Bernt Schiele. 2023. Visual writing prompts: Character-groum curated image sequences. Transactions of the Associ- ation for Computational Linguistics, 11:565-581.
story generation with
Eileen Wang, Caren Han, and Josiah Poon. 2022. Ro- ViST: Learning robust metrics for visual storytelling.

---

In Findings the Association for Computational Linguistics: NAACL 2022. pages 2691-2702 Seattle, United States. Association for Computational Lin- guistics.
Algorithm 1 RoViST-VG pre-training
Input: 1)A mini-batch image regions i, with shape (m x 3 x 224 x 224) where is the batch size. 2)A mini-batch of matching noun pairs Tn with shape (m, 300) where 300 represents the di- mensions of the GLoVe vectors. Output: Symmet- ric loss for the mini-batch.
Xin Wang, Wenhu Chen, Yuan-Fang. Wang, and William Yang Wang. 2018. No metrics perfect: Adversarial reward learning for visual storytelling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 199-909, Melbourne, Australia. Association for Computational Linguistics.
Initialization: Pretrained VIT model with linear head for the image encoder, and single linear layer for the text encoder.
h2 VisioonTranfoommer(La)
L. Zitnick and Devi Parikh. 2013. Bringing seman- tics into focus using visual abstraction. In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR).
tan, b;) image embeddings:
Shape=[m,1022)
T. = tanh( + b,)
text embeddings;
(m,
Appendix
4: logits = Te Ier
b shape=lm, m] b shape=[m, m] shape=[m, m] labels (Jsim T/2 shape=[m, m]
5: Ism = Le x I
A Text concreteness example
6: Tsim Te T
Figure 4 shows that through idf weighting RoViST- VG penalizes the alignment score of the relatively frequent noun 'church' (0.266). Inversely, idf weighting increases the low alignment score of the abstract and relatively less-frequent noun 'pic" (0.232). This could have unintended effects on the overall score of the metric, resulting in several discrepancies discussed in Sections 2 and 5.
Limage: cross crros__etroppy.oo sosoeettoooyoossslllllssss logitss
Lteat = cross Lacrass_ettto_____ Josssaabsssss,,s, logits)
10: L Sxymmetric (Limage + 2
D GROOViST pseudocode
For given <image sequence, story> pair, the pseu- docode in Algorithm 2 outlines the steps involved in computing the GROOViST score.
Human evaluation
We recruited participants on voluntary basis among colleagues of our institution. We asked the participants for their consent through an informed consent form (see Figure 5). Participants who ex- pressed their consent, were provided access to scoring web interface with instructions. Each of the participants provided ratings for 100 <image sequence, story> pairs of the VIST data test set 4-point Likert-like scale.
Algorithm 2 GROOViST
Input: Image sequence bounding boxes (V/), cor- responding story NPs (T;), concreteness weights (W;), pre-trained CLIP model, score threshold (0) Output: unbounded GROOViST score G
Steps:
NP pos, NP
Nopos
NP nog <- }, ()
for k 1 to #(Ti) do b number of NPs
nPe. w <- (K))( W;[
C RoViST-VG
4: np, + 0.0
>> score of noun phrase mp
5: for each v E V do
Model The RoVIST-VG model comprises an image-encoder and text-encoder. The image- encoder encompasses pre-trained VIT model, an additional linear layer (W.), and tanh activation function for obtaining the image representations le- The text-encoder has a linear layer (W;) and a tanh activation function for encoding GLoVe vectors into text embeddings Te.
6:
+ CLIP(v)
nps <- max(nps. cos(npe, ve))
8: end for
if np, 2 0 then
10:
NP pas + npp
11: else
12: NP NP + 10 npps) x
13: end if
14: end for
Pre-training The procedure used for pre-training the Ro ViST-VG model is provided in Algorithm 1.
15: G = C NP
NPpos pos NP nege) (((T))

---

RoVIST-VG:
Nouns
wedding
church
bridesmaids
pic
kiss
groom
bride
guests
bouquet
IDF weighas
3.48
3.76
6.09
6,38
5.47
4.22
3.92
4.21
5.88
6.25
Model similarities
0.384
0.266
0.278
0.232
0.274
0.394
0,362
0.316
0.262
0.377
Final scores
1.339
1.002
1.894
1.478
1.499
1.883
1.421
1.330
1.544
2.359
CROOVIST:
Noun plaassss
the wedding
the church
the bridesmalds
a quick pic
quick kiss
groom
the bride
the guests
Joy
the bouquet
Model similarities
0.876
0.675
0.626
0.583
0.572
0.674
0.850
0.595
0.533
0.670
Penalisation
0.878
0.675
0.626
-0.033
-0.044
0.674
0.850
-0.021
-0.083
0.670
Concreteners rarings
2.675
3.165
2.920
2.175
2.943
4,540
3,030
2.630
2.370
3.085
Finol scores
1.829
-0.058
1) this is the church where the wedding held 2) the bridesmaids took quick pic together .3) 3) the bride and groom leaned forward for a quick kiss .4) the guests were overwhelmed with joy 5) the bouquet was beautiful.
Figure 4: Example showing discrepancies of RoViST-VG idf weighting along with GROOViST noun phrase contributions for comparison. The overall normalized GROOViST score for this sample is 0.846.
Hello!
this experiment, ask you judge sequences of images and short stories about these sequences. You will be asked to use a numeric score to rate how much the stories are a good fit for the images. This information will help the investigators to devise automatic evaluation metric to judge stories.
Only the scores you provide and your email address (for identifying your scores) are collected as data.
If necessary, only the scores would be made publicly available for future research.
you select the "I consent" option, you acknowledge that:
Your participation voluntary.
You are older than 18 years of age.
You aware that you may choose to terminate your participation at any time for any reason.
The data you provide (only the scores), if necessary, will be published and made available for future research.
consent
not consent
Thank you for your interest to help our research.
>>
Figure Informed consent form used for recruiting participants.

---

E
Ablation and replacement results
A on VIST
Flickrök-Expert
GROOViST
0.413
0.414
GROOViST (-C)
0.127
0.421
GROOViST (-P)
-0.071
0.289
GROOViST (-C -P)
0.025
0,461
GROOViST (-NPs +Ns)
0.260
0.410
GROOVIST (-C +idf')
0.890
0.379
GROOViST (-C +idf -Ns 4Ns)
0.348
0.341
Table 3: Results of ablating/replacing components of GROOViST. A refers to the difference between the scores obtained by the original and random <image sequence, story> pairs- the higher the better.


Reviews:
[{"review_id": "Qfip0k2ncY", "reviewer": "Reviewer_G7ME", "summary": "", "questions": "A. Attributing negative scores to NPs, seems a convoluted step and is not clear how much improvement does it really bring. Is it purely for interpretability? \n\nB. In Figure 1, you state that “[…] score for this sample is 1.278, i.e., our metric considers it as moderately grounded[…]”. Can you provide more details about the range of the metric? What is a bad score?\n\nC. Why is the normalization equation (eq 2), split into “positive” and “negative” NPs?\n\nD. What is exactly modular about GROOViST?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your detailed and insightful review.\n\nFor clarity reasons, we will start by answering your 4 questions and then address your concerns under Reasons to reject.\n\n**Responses to Questions:**\n\nA. Penalizing poorly grounded NPs based on a factor $\\theta$ is an essential step before aggregation. Summing up the individual NP scores without this penalization step would indeed lead to an overall score that is overinfluenced by highly grounded NPs. Given the goal of our metric, this is undesirable. Consider the case where we have two stories, a well-grounded one, and one mentioning some ill-grounded – if not unrelated – NPs. Suppose that the two stories receive the following sets of scores (based on their NPs): $\\lbrace 0.63, 0.605, 0.61 \\rbrace$ and $\\lbrace 0.33, 0.74, 0.8, 0.48, 0.62, 0.72 \\rbrace$ for the well-grounded and ill-grounded, respectively. Aggregating the scores without penalizing the NPs responsible for the ill-groundedness would lead to exactly the same GROOViST score for both stories: $1.845/3 = 3.69/6 = 0.615$. This signifies the importance of the penalization step in making the metric scores sensible. As a by-product of this step, our metric is also highly interpretable, as shown in Figures 1 and 4.\n\nB. The range of the metric is unbounded by default. In Fig 1, we consider the score to be moderately grounded based on the distribution of scores of the dataset. We will further clarify this aspect in the final version.\n\nC. Eq. 2 reflects the difference between positive and negative NPs, as explained in our response to question A above.\n\nD. GROOViST builds on 4 components or modules: NP extraction, vision-language alignment, concreteness weighting, and penalizing poorly grounded NPs; followed by a normalization step. All these components are key to the functioning of our metric. At the same time, each of them can be updated or replaced if needed (e.g., CLIP-based vision-alignment scores could be replaced with scores by a different model). As such, our metric is model-agnostic.\n\n**Responses for Reasons-to-Reject:**\n\n1. We believe we further clarified the modular nature of our metric in our answer to your question D. As for the need of an ablation analysis, this is a very important point and we thank you for raising it. Following your comment, we performed an ablation analysis by removing, one at a time, the modules that – by definition – can be removed from the metric without needing a replacement by some other alternative (e.g., the NPs module cannot be ablated, it can only be replaced, e.g., by nouns), namely, concreteness weighting and the penalization of poorly grounded NPs. By evaluating the ablated versions of the metric following the steps described in section 2, we found that neither of the two meet the desired criteria of differentiating original <image, story> pairs from random pairs. We observed the same when jointly ablating the two modules. This shows that both these modules are essential for the calculation of GROOViST. Profiting from the additional page, we will include this analysis and its results in the final version of the paper.\n\n2. The choice of using NPs over Ns is motivated by linguistic reasons. In fact, using simple Ns over NPs splits compound nouns such as “*parking lot*” or “*post office*” into two separate nouns, which is clearly undesirable. However, given the modular nature of our metric, we can still replace NPs with Ns and test the effect of it. We tested so by means of an analysis similar to the one reported above. As expected, this deteriorates the ability of the metric to distinguish between original and random stories, which validates our theoretical motivation.  \nAs for the use of concreteness weighting, this is motivated in Section 3. We observed that IDF has several flaws when used in a metric that aims at evaluating the degree of grounding of a text. Moreover, it is not informative of the degree of abstractness/concreteness of words, as we also highlight in Fig. 4 in the Appendix. However, following your comment, we did test the effect of replacing concreteness weighting with idf in our metric (using the IDF scores of the NPs). In this case, we did not observe a deterioration of the metric with respect to the criteria described in Section 2. We conjecture that this may be due to the effect of the other modules, making IDF’s shortcomings less impactful. We will report this analysis in the paper and discuss its implications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a new metric for evaluating grounding in generated visual stories. The idea is simple and clean, well presented and the paper is easy to read.\n\nGROOViST builds upon the weaknesses of previous metrics, generating in the end a better, more aligned with human judgment visual storytelling metric.", "reasons_to_reject": "I strongly believe ablation studies are needed in this work. The authors propose a few improvements, that work (the proposed metric seems to be better than previous metrics), but do not properly evaluate which parts are bringing this improvement and to what extent. \n“GROOViST is its modular design, where the contribution of each component can be assessed and interpreted individually” (from abstract). I find this exact interpretation and analysis missing from this work. \n\nHow, by how much, in which cases does the step from nouns to noun phrases aids performance? Similar questions should be answered for replacing idf with concreteness weighting.", "questions_to_authors": "A. Attributing negative scores to NPs, seems a convoluted step and is not clear how much improvement does it really bring. Is it purely for interpretability? \n\nB. In Figure 1, you state that “[…] score for this sample is 1.278, i.e., our metric considers it as moderately grounded[…]”. Can you provide more details about the range of the metric? What is a bad score?\n\nC. Why is the normalization equation (eq 2), split into “positive” and “negative” NPs?\n\nD. What is exactly modular about GROOViST?", "paper_topic_and_main_contributions": "This work introduces a new metric for evaluating grounding in the task of visual storytelling. GROOViST builds upon existing visual grounding metrics for visual storytelling, in the form of RoViST-VG and pre-trained vision-language models, in the form of CLIP (and the associated CLIPScore).\n\nGROOViST brings several improvements over previous metrics, such as considering noun phrases instead of only nouns, computing image-text alignment using CLIP (instead of training a specific model for this task), replacing idf weighting with “concreteness” weighting. Finally, for each NP they assign a score that can also be negative (probably for interpretability reasons), averaging over all these scores to obtain the final value of the metric.\n\nEvaluations on two datasets are performed and a couple of hypotheses about the strengths and weaknesses of the metrics are tested with good results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2LfgV68GSK", "reviewer": "Reviewer_sXd8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your review. \n\n**Responses for Reasons-to-Reject:**\n\n1. Thanks for asking us about the generalizability of our metric. \n\t- First, in our paper we showed that GROOViST can be used with (and is suitable for) different visual storytelling datasets, namely, VIST and AESOP – the two visual storytelling datasets available at the submission time. Since a new dataset, Visual Writing Prompts, was recently released by *Hong et al. (2023)*, we tested GROOViST on it. We found that GROOViST works well on this dataset, too, which provides further evidence for the generalizability of our metric. We will report the results of this additional analysis to the updated version of the paper. \n\t- Second, while we developed and validated GROOViST using stories produced by human speakers, we argue that it can be used to assess the degree of grounding of model-generated stories. Indeed, we have preliminary experiments on this, which show the potential of applying GROOViST to this domain. We will discuss this point in the revised version of the paper.\n\n2. Thanks for raising this important point. Please check our response to Reviewer G7ME for a detailed answer.\n\nReferences:  \nXudong Hong, Asad Sayeed, Khushboo Mehra, Vera Demberg, Bernt Schiele; Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequences. Transactions of the Association for Computational Linguistics 2023; 11 565–581. doi: https://doi.org/10.1162/tacl_a_00553", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper thoroughly examines the limitations of current metrics, providing valuable insights for the field.\n- Experimental results demonstrate the effectiveness of the proposed metric.\n- The paper is well written, with detailed problem descriptions.\n", "reasons_to_reject": "- Limited Generalization: The paper does not adequately address how the proposed metric applies to various kinds of visual storytelling methods. Further clarification is needed in this aspect to ensure broader applicability.\n- The paper lacks a detailed experiment on the modularity of the proposed metric.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The short paper proposes a metric to evaluate temporal misalignments in visual storytelling models. It offers a critical analysis of existing metrics for this task and includes experimental results to support its claims.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "afqMA8RODd", "reviewer": "Reviewer_Pc9w", "summary": "", "questions": "A- You explain your proposed metric in Section 3. When I first read it, I didn't find any specific way to address temporal alignment between sentences and images. However, in Section 4.1 you comment: \"We hypothesize that GROOViST—since it takes into account the entire story holistically—correlates better with human judgments than CLIPScore\". Don't you think you should mention this explicitly in Section 3? There are some mentions, like in lines 151-152: \"We process the story and extract all the NPs\", but maybe this should be highlighted better.\nB- Regarding Noun Phrase (NP) extraction, starting in line 151, you don't specify how you accomplish the task. What tool(s) are you using for NP extraction?\nC- A similar question for Vision-language alignment (line 158): how do you extract object bounding boxes? Faster R-CNN? Another object detector? And how do you use CLIP to compute the similarity of those object-based visual features and NPs? As far as I know, CLIP has been trained for entire images, using different visual backbones trained from scratch (ViT, Resnet...). If you use Faster R-CNN for object detection, I assume you have to use the same visual backbone as CLIP, at least. Please, explain this further. \nD- Your analysis shows that CLIPScore may be better for some circumstances (Figure 3, samples with low T and samples with low P). Can you provide a distribution of samples for high/low T and P for the datasets you use? This would be interesting, in order to see whether your metric is better \"only for marginal cases\" or instead, those are the most frequent cases. ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your review and questions.\n\n**Responses to Questions:**\n\nA. We agree that this point should also be mentioned earlier in the paper. We will update the camera-ready version accordingly.\n\nB. Thank you for underlining this. We experimented with 2 existing libraries – NLTK pos-tagging-chunking and Spacy noun chunks (based on its English transformer pipeline). While the two libraries lead to comparable performance, upon manual inspection, we found Spacy to be more consistent in identifying phrases. We will release the code for extracting noun-phrases using both libraries, while defaulting to Spacy.\n\nC. We extract bounding boxes using FasterR-CNN. Then, for each bounding box, we extract visual feature representations using the CLIP model. To clarify this further, FasterR-CNN is only used to identify and \"crop\" the various objects in a given image. Once we have these cropped bounding boxes for each image in a given sequence, we then compute the similarity scores between each NP in the corresponding story and every bounding box of the image-sequence. Finally, we pick the maximum score for every NP.\n\nD. For this analysis, as mentioned at the beginning of Section 4, we asked the participants to provide ratings for 100 randomly sampled VIST <image sequence, story> pairs. Among these samples, $18$% of the stories exhibit high temporal misalignment (more details in Section 4.1). Similarly, $32.7$% of the subset of samples we consider for the P analysis have a high proportion of NPs ($>$ central tendencies of the P distribution – e.g., mode). We agree that expanding and discussing these distribution details would substantiate our findings further. We will profit from the extra page given to camera-ready papers to do so.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A- The paper is very well motivated.\nB- Authors show the weaknesses of previously proposed (and published) metrics. Specially surprising is the bad behavior of RoVIST-VG. \nC- The methodology to evaluate automatic metrics for Visual Storytelling is sound and interesting.\nD- Their proposed metric is shown to work better than previous metrics, in some important cases.\nE- I personally think that research papers about evaluation are very important in our community and this work goes exactly in that direction.", "reasons_to_reject": "I do not see any (important) reason to reject the paper.", "questions_to_authors": "A- You explain your proposed metric in Section 3. When I first read it, I didn't find any specific way to address temporal alignment between sentences and images. However, in Section 4.1 you comment: \"We hypothesize that GROOViST—since it takes into account the entire story holistically—correlates better with human judgments than CLIPScore\". Don't you think you should mention this explicitly in Section 3? There are some mentions, like in lines 151-152: \"We process the story and extract all the NPs\", but maybe this should be highlighted better.\nB- Regarding Noun Phrase (NP) extraction, starting in line 151, you don't specify how you accomplish the task. What tool(s) are you using for NP extraction?\nC- A similar question for Vision-language alignment (line 158): how do you extract object bounding boxes? Faster R-CNN? Another object detector? And how do you use CLIP to compute the similarity of those object-based visual features and NPs? As far as I know, CLIP has been trained for entire images, using different visual backbones trained from scratch (ViT, Resnet...). If you use Faster R-CNN for object detection, I assume you have to use the same visual backbone as CLIP, at least. Please, explain this further. \nD- Your analysis shows that CLIPScore may be better for some circumstances (Figure 3, samples with low T and samples with low P). Can you provide a distribution of samples for high/low T and P for the datasets you use? This would be interesting, in order to see whether your metric is better \"only for marginal cases\" or instead, those are the most frequent cases. ", "paper_topic_and_main_contributions": "This paper proposes a new automatic metric to assess the quality of Visual Storytelling systems. They show that previous metrics used by the community have several weaknesses. Based on those weaknesses, they propose a new metric, which they show to mitigate those weaknesses. Additionally, they show how their metric correlates better with human judgement.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new evaluation metric for visual storytelling, focusing on how references to objects and events are grounded in the sequence of images on which the story generation was conditioned. The proposed metric is used to evaluate methods applied to two visual storytelling datasets (including a third dataset in the author rebuttal). Evaluation shows that the metric correlates with human judgments of temporal and visual grounding.
Decision: Accept-Main