SlowGuess/NAACL_2022_OCR huggingface dataset:
Eileen Wang Soyeon Caren Han* Josiah Poon
School of Computer Science, The University of Sydney, NSW, Australia
{Eileen.Wang, Caren.Han, Josiah.Poon}@sydney.edu.au
Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on  $n$ -gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) nonredundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics.<sup>1</sup>
Visual storytelling (VST) is a natural language generation (NLG) task that aims to automatically generate a cohesive story given a sequence of images (Huang et al., 2016). The task is fundamental to the development of intelligent agents capable of understanding complex visual scenarios, and can be further applied to assist the visually impaired in understanding images on the web. Recently, progress has been made on designing network architectures to accomplish the VST task but little work has been done to explore new metrics that automatically evaluate and quantify the errors produced by these systems. As to date, a majority of
Input Image Sequence:
Figure 1: Example gold story found in the VIST dataset versus machine output from 2 VST models and their  $n$ -gram based metrics.
Gold Story:
We arrived at the house in the mountains. It was very well kept but the patio and fire pit needed some attention. The guest house was pretty but had a strange style. From the deck, we could see hawks and buzzards soaring over the mountains. We didn't have a bird's eye view, but ours was more than pretty enough for us.
Candidate Story 1 (AREL):
The view from the top of the mountain was beautiful. This is a picture of a tree. The building was very old. We were able to take a picture of a tree in the distance. The view from the top of the mountain was beautiful.
Candidate Story 2 (MCSM+BART):
A family decided to take a trip to the countryside. They took pictures of the different landscape like this picture outside. At the same time, they found a house that was completely in disrepair. At the end of the day, they saw an eagle in the air. We then looked down over the hills.
B-1	B-2	B-3	B-4	C	M	R	S
Story 1	52.0	27.3	11.6	0.0	0.5	12.7	25.2	11.1
Story 2	36.8	16.3	0.0	0.0	1.1	14.9	24.1	3.2
the past works on VST have used existing popular  $n$ -gram based metrics such as BLEU, METEOR, ROUGE, CIDEr, and SPICE to evaluate their models (Wang et al., 2018; Kim et al., 2018; Hsu et al., 2019; Chen et al., 2021). However, it is known that such metrics are unreliable for VST. Figure 1 shows two machine generated stories for a photo sequence and their corresponding  $n$ -gram matching based metrics (BLEU, CIDEr, METEOR, ROUGE-L and SPICE). Evidently, the first candidate story is more repetitive and lacks a narrative style but achieves higher scores across a majority of the  $n$ -gram based metrics in Figure 1. The second story however, has greater word diversity and is more expressive through its use of phrases like 'completely in disrepair'. Relevant words like 'trip', 'country-side' and 'hills' are also used but are not rewarded since they are not mentioned in the gold story.
The low level of agreement between human judgement and current automatic metrics may be because such metrics were originally developed to assess machine translation, summarization and image captioning tasks (Sharif et al., 2018), which
*Corresponding author (caren.han@sydney.edu.au)
<sup>1</sup>The RoViST code: https://github.com/usydnlp/rovist
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2691 - 2702
are significantly different problems to VST. Specifically, VST is a multimodal task that firstly requires: 1) generating text relevant to the image content but unlike image captioning, there is less emphasis on describing relationships between objects and may contain concepts that are inferred from the image. It additionally needs to ensure that: 2) the story must be topically coherent, similar to how a human would tell a story in a social setting. Sentences should not sound disjointed e.g. 'We went to the park. I grew up in Sydney'. And finally 3) avoids repetition which appears to be a common issue in current VST models. For instance, Candidate Story 1 in Figure 1 exhibits inter-sentence repetition between the first sentence and last sentence. We also find that some output stories may contain repetition within sentences (i.e. intra-sentence repetition) e.g. 'we had a good time and had a great time!'.
Moreover, it is noted that open-ended text generation tasks usually suffer from the one-to-many issue, whereby there are multiple plausible outputs for the same input which are not fully reflected in the reference sentences (Guan and Huang, 2020). This issue is even more prominent in the VST task as different individuals may tell significantly different stories and have diverse interpretations given the same image sequence. All these issues suggest that we require evaluation metrics that do not simply rely on comparison with reference sentences. In addition, given that the VST task requires several aspects, one single metric is not sufficient to evaluate a story and there is a need to design multiple interpretable metrics that each target a specific VST criteria. Hence, in this paper, we propose several unreferenced metrics for the VST task based on the three aforementioned criteria: 1) visual grounding, 2) coherence, and 3) non-redundancy.
To address criteria 1), we propose a learned metric to calculate relevance scores between nouns in the VST sentences with the bounding box regions in the images. We decide to focus on nouns as they provide the most visual information. Other words like adjectives and adverbs are difficult to ground and such words may differ significantly depending on the person writing the story. The second criteria which is story coherence requires that consecutive sentences flow and that each sentence is not just an isolated description of the image. Existing methods for measuring coherence have used next sentence prediction (NSP) to find the probability that a sentence comes after a preceding sentence (Hu et al.,
2020). Inspired by this method, we fine-tune the ALBERT (Lan et al., 2019) model on story sentences and build a sentence-order prediction (SOP) model. Finally, to address criteria 3), we propose an additional metric to explicitly measure intersentence and intra-sentence repetition.
The contributions are summarized as follows: 1) We propose an interpretable and reference-free metric that addresses 3 criteria required for VST - visual grounding, coherence and non-redundancy. 2) We conduct human evaluation studies to assess a sample of machine generated stories obtained from 4 state-of-the-arts VST models. 3) We test the effectiveness of our proposed metrics by analyzing its correlation with human scores and show that our metrics outperform other existing metrics that are commonly used for VST and NLG tasks.
Natural Language Generation Metrics The most popular NLG evaluation metrics are BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), CIDEr (Vedantam et al., 2015) and SPICE (Anderson et al., 2016). All these metrics are widely used in evaluating image captioning tasks (Anderson et al., 2018; Zhou et al., 2020) and have also been predominantly used in VST tasks (Wang et al., 2018; Hsu et al., 2019; Chen et al., 2021) due to the lack of metrics designed for VST. While these metrics are computationally efficient, they have limited ability in accounting for synonym matches or phrase reordering. This poses a problem for many open-ended text generation tasks like VST where different annotators may have slightly different (but still plausible) ways of describing the same image. To address this, some metrics focus on comparing distance and similarity between word embeddings such as Word Mover's Distance (Kusner et al., 2015), MoverScore (Zhao et al., 2019) and BERTScore (Zhang et al., 2019). However, these metrics mentioned so far still heavily rely on similarity with references, potentially leading to bias for VST tasks as the references may not fully cover the possible ways to write a story for an image sequence.
Visual Grounding Metrics Past studies have proposed examining the images in addition to human written references. Cui et al. (2018) trained a binary classifier to discriminate between human and machine captions using image and text representations obtained from a CNN and RNN. TIGEr
(Jiang et al., 2019) employs the pretrained SCAN model (Lee et al., 2018) to calculate the text-to-image grounding scores and compares the relevance ranking and grounding weights distribution among image regions between the references and the candidate. Lee et al. (2020) later introduced ViLBERTScore which uses the same approach as BERTScore but utilizes the ViLBERT model (Lu et al., 2019) to retrieve image-conditioned token embeddings. However, we note that these methods are initially designed for evaluating image captioning systems. Hence, while they do consider the text-to-image similarity aspect, they do not explicitly address the extra criteria required for VST such as story coherence. Moreover, such metrics still rely on reference sentences to some extent.
Story Generation Metrics Language models like BERT (Devlin et al., 2018) trained with NSP and masked language modelling tasks can identify appropriate use of words and sentences and hence, may show promising results when applied to evaluating open-ended text generation. Guan and Huang (2020) proposed UNION, an unreferenced metric for scoring machine generated stories. They leverage a BERT model trained with negative samples created by perturbing ground truth stories and predicts a score representing how human-like a story is. They showed the effectiveness of BERT in identifying stories with conflicting logic, repeated plots and incoherence. However, UNION purely evaluates the output text and cannot be applied to analyse the text-to-image relatedness required for the VST multimodal task. Additionally, a single score is outputted which is not informative enough to gauge what specific errors were made by the model. Moving to VST, Hu et al. (2020) designed reward functions to capture story quality for VST models that use a reinforcement learning framework based on 3 criteria: image relevance, coherence and expressiveness. Image relevance is measured by  $n$ -gram precision of entities between candidate and reference sentences, coherence through BERT's NSP task, and word diversity by computing BLEU scores between generated sentences.
Inspired by this, we also analyze story quality from 3 similar perspectives 1) visual grounding, 2) coherence, and 3) non-redundancy. We attempt to extend the methods of Hsu et al. (2019), provide a reference-free approach and conduct a comprehensive analysis with human evaluation.
We describe our proposed metric in detail. Given a machine story, we aim to output 3 scores that explicitly evaluates the story based on 1) visual grounding, 2) coherence, and 3) non-redundancy.
3.1 RoViST-VG: Visual Grounding Scorer
To detect the visual relationship between image and text, we build a model that computes the similarity between the nouns in the story sentences with the bounding box regions in the images. We focus specifically on nouns because despite the diverse range of words one can use when storytelling, we notice that the main commonality among the ground truth sentences is the noun mention. This is most likely because nouns (in particular, tangible nouns) tend to offer the most visual information and is the common element that people would recognize when observing an image. An example of this case is in Figure 2 where we can see that the nouns 'dart' and 'game' tends to appear in multiple gold sentences, even though each sentence is quite different in structure.
Our visual grounding scorer is inspired by the phrase localization task (Plummer et al., 2015) which involves learning to align sentence entities with image regions. We note that we could have just employed typical image-text matching models like SCAN (Lee et al., 2018) to calculate a similarity score between image and text. However, such models are trained on image captioning sentences and do not explicitly focus on the more fine-grained task of word-region alignment. Moreover, retraining these models with VIST images and whole sentence pairs would be challenging as previously mentioned, story sentences tend to differ significantly in semantics and structure due to human imagination. This is in contrast to image captions where ground truth sentences typically tend to be similar to each other even across different human annotators (e.g. see description in isolation sentences in Figure 2).
Inspired by CLIP (Radford et al., 2021), we create a model that learns the image region and text embeddings such that the noun mention corresponding to an image region will have similar vector representations in geometric space. Let  $I_{i}$  be an image of a bounding box region and  $T_{i}$  be the matching noun. For the image encoder, we follow Radford et al. (2021) and leverage the Vision Transformer (ViT) (Dosovitskiy et al., 2020) to first
Description in Isolation:
Figure 2: Example ground truth description in isolation (dii) and story in sequence sentences (sis) sentences corresponding to an image from the VIST dataset.
1. A mother and her daughters look at a carnival game.  
2. A woman and two girls look at a dart-toss carnival game's card.
3. A woman and some children are standing in front of some balloons.
Story in Sequence:
1. Although some of the games seem fixed and a waste of money.
2. We threw a million darts trying to win one!
3. Finally they played the dart game.
4. [female] and [female] notice another family playing
another type of prize game and decides to try it out when they are finished before leaving the carnival.
Algorithm 1 RoViST-VG
Input: 1) A mini-batch of image regions  $I_{n}$  with shape  $(m \times 3 \times 224 \times 224)$  where  $m$  is the batch size, and the last 3 dimensions correspond to the image channels, height and width respectively. 2) A mini-batch of matching noun pairs  $T_{n}$  with shape  $(m \times 300)$  where 300 represents the dimensions of the GLoVe vectors. Output: Symmetric loss for the mini-batch.
Initialization: Pretrained ViT Model with linear head for the image encoder, and a single linear layer for the text encoder.
1:  $h_n = \mathrm{VisionTransformer}(I_n)$
2:  $I_{e} = \tanh (\mathbf{W}_{i}\mathbf{h}_{n} + \mathbf{b}_{i})$  image embeddings; shape  $= [\mathrm{m},1024]$
3:  $T_{e} = \tanh (\mathbf{W}_{t}T_{n} + \mathbf{b}_{t})$  ▷ text embeddings; shape = [m,1024]
4:  $\mathrm{logits} = T_e\times I_e^T$  shape  $= [\mathbf{m},\mathbf{m}]$
5:  $I_{sim} = I_e \times I_e^T$ $\triangleright$  shape = [m, m]
6:  $T_{sim} = T_e\times T_e^T$ $\triangleright$  shape  $= [\mathfrak{m},\mathfrak{m}]$
7: labels  $= \left( {{I}_{sim} + {T}_{sim}}\right) /2$ $\vartriangleright$  shape  $= \left\lbrack  {\mathrm{m},\mathrm{m}}\right\rbrack$
8:  $\mathcal{L}_{image} =$  cross_entropy_loss(labels  $T$ , logits  $T$ )
9:  $\mathcal{L}_{text} =$  cross_entropy_loss(labels, logits)
10:  $\mathcal{L}_{symmetric} = (\mathcal{L}_{image} + \mathcal{L}_{text}) / 2$
extract the image features from  $I_{i}$ . An additional linear head is further added to project the features to a vector embedding of dimension 1024. For the text encoder,  $T_{i}$  is first converted to 300 dimensional GLoVe vectors (Pennington et al., 2014). If  $T_{i}$  is composed of more than one word, the GLoVe vectors of each token are simply averaged. These vector representations are then passed through a single linear layer to project the text features into the 1024-dimensional joint embedding space. We train the model in a contrastive manner to minimize the symmetric loss. The psuedocode for each batch iteration is provided in Algorithm 1.
To compute the visual grounding score, we extract all nouns from the output story sentences and
the top 10 bounding box regions for each image in the story based on the confidence scores generated from Faster R-CNN (Ren et al., 2015). This results in 50 regions for a 5-image story. Each extracted noun and image region is fed through our trained text and image encoder respectively to obtain the image and text embeddings which we denote by  $I_{e}$  and  $T_{e}$ . For each noun, the cosine similarity ( $cos$ ) is calculated between its text embedding with all other region image embeddings. It is noted that a noun mention from a sentence can match with a region from other images and not necessarily just with regions from its corresponding image as we find that words in story sentences may refer to concepts in other images of the sequence. We then use a greedy matching approach to obtain the maximum similarity score for each noun. Following Zhang et al. (2019), we further experiment by multiplying the similarity score by the inverse document frequency (idf) of the noun calculated from the corpus. This is to put less emphasis on abstract nouns that are not visually grounding but frequently occur in stories (such as 'time' and 'today'). Given  $N$  stories, theidf score of a token  $T_{i}$  is:
$$
i d f \left(T _ {i}\right) = \log \left(\frac {N}{1 + d f \left(T _ {i}\right)}\right) \tag {1}
$$
where  $df(T_{i})$  is the number of stories containing token  $T_{i}$ . Finally, inspired by Lee et al. (2018), a recall score is computed by using LogSumExp (LSE) pooling:
$$
S _ {V G} = \log \sum_ {i = 1} ^ {| T _ {e} |} \exp (i d f (T _ {i}) \max  _ {I _ {e, j} \in I _ {e}} \left(\cos \left(T _ {e, i}, I _ {e, j}\right)\right)) \tag {2}
$$
For interpretability, one can optionally scale the score between 0 and 1 using a shifted and scaled version of the sigmoid function:
$$
S _ {V G (s c a l e d)} = \frac {1}{1 + \exp (- 0 . 5 \times S _ {V G})} \times 2 - 1 \tag {3}
$$
3.2 RoViST-C: Coherence Scorer
To measure the story's inter-sentence coherence, we leverage the ALBERT model to perform sentence order prediction (SOP) (Lan et al., 2019). The SOP task is a binary classification task, whereby positive samples are consecutive sentences while negative samples are simply constructed by swapping the two sentences around. This forces the
model to primarily focus on learning coherence properties rather than topic prediction. We fine-tune the ALBERT model with adjacent story sentences extracted from the VIST and ROCStories dataset. In total, 822,920 training samples were created where  $15\%$  was used in the validation split.
Let  $\{s_{i - 1},s_i\}_{i = 1}^N$  denote the training data where  $s_{i - 1}$  and  $s_i$  are adjacent segments. The input sequence fed into ALBERT is in the format  $\mathbf{s}_n =$  [CLS],  $s_{i - 1}$  , [SEP],  $s_i$  , [SEP], where [CLS] and [SEP] are special tokens. Then, the pooled 1024- dimensional vector representation  $\mathbf{h}_n$  of the input sequence is obtained by the output of ALBERT:
$$
\mathbf {h} _ {n} = \operatorname {A L B E R T} (\mathbf {s} _ {n}) \tag {4}
$$
To perform SOP, we add a task-specific linear layer on top of ALBERT to predict the probability that  $s_i$  follows  $s_{i-1}$ :
$$
\hat {p} _ {n} = \operatorname {s o f t m a x} \left(\mathbf {W} _ {c} \mathbf {h} _ {n} + \mathbf {b} _ {c}\right) \tag {5}
$$
where  $\mathbf{W}_c$  and  $\mathbf{b}_c$  are the trainable weights and bias. For the loss function, we optimize the binary cross-entropy loss as follows:
$$
\mathcal {L} = - p _ {n} \log (\hat {p} _ {n}) - (1 - p _ {n}) \log (1 - \hat {p} _ {n}) \tag {6}
$$
To obtain the final coherence score for each story, we compute  $\hat{p}_n$  for each adjacent sentence pair in the story and average the probabilities across all sentence pairs.
3.3 RoViST-NR: Non-redundancy Scorer
A common problem faced by system output stories is redundancy of words in the form of whole sentences or phrases. While existing methods (Hu et al., 2020) for assessing word diversity and repetition do consider inter-sentence repetition, they do not address repetition within sentences. Therefore, to calculate the inter- and intra-sentence nonredundancy score, we propose calculating the Jaccard Similarity (JS) between and within sentences. The JS is defined as the intersection size divided by the union size of two sets (Singh and Singh, 2021). That is, in our problem, the intersection would be the number of co-occurring words between two texts, while the union is the total number of words in both texts. In particular, we compute the Jaccard Similarity with sentence  $\hat{y}_i$  and all its preceding sentences  $\{\hat{y}_1,\dots,\hat{y}_{i - 1}\}$  as in Eq. 7. Here,  $C(\hat{y}_i)$  and  $C(\hat{y}_j)$  are the count of unique words in sentence  $\hat{y}_i$  and  $\hat{y}_j$  respectively. The inter-sentence
repetition score is then just simply the average JS scores across the  $\binom{n}{2}$  sentence pairs where  $n$  is the number of sentences in the story.
$$
J S (\hat {y} _ {i}, \hat {y} _ {j}) = \frac {C (\hat {y} _ {i}) \cap C (\hat {y} _ {j})}{C (\hat {y} _ {i}) \cup C (\hat {y} _ {j})} \tag {7}
$$
We also measure the intra-sentence redundancy by first splitting each sentence into non-overlapping  $n$ -grams and then calculating the JS score between consecutive  $n$ -grams within sentences. The intrasentence repetition score for a story is then the average JS scores across all consecutive  $n$ -gram computations. Lastly, we take the mean of the final inter- and intra-sentence score to obtain the final repetition score for the story and subtract from 1. The result is a score between 0 and 1 where a value closer to 1 means that the story tends to contain less redundancy.
4.1 Supporting Datasets
VIST The Visual Storytelling Dataset (VIST) dataset (Huang et al., 2016) consists of 10,117 Flickr albums and 210,819 unique images. Each sample is one sequence of 5 photos selected from the same album paired with a single human constructed story, where each story is comprised of mostly one sentence per image.
ROCStories Corpora (Mostafazadeh et al., 2016) is used as additional data along with VIST to train the ALBERT model. It contains 98,161 stories where each story consists of 5 sentences written by humans after being given a prompt.
Flickr30K Entities (Plummer et al., 2015) is derived from the Flickr30K dataset (Young et al., 2014), consisting of 31,783 images each matched with 5 captions. The dataset links distinct sentence entities (i.e. a noun/noun phrase) to image bounding boxes, resulting in 70K unique entities and 276K unique bounding boxes. We use the Flickr30K Entities data to train our visual grounding scorer. After filtering out stopwords from the entity mention, we obtained 566K unique entity-region pairs.
4.2 VST Models
We evaluate our proposed metric on the output stories produced by 4 state-of-the-art VST models: 1) AREL (Wang et al., 2018): adopts an inverse reinforcement learning approach trained adversarially. The policy model is a CNN+GRU that generates