Visual Coherence Loss for Coherent and Visually Grounded Story Generation
Year: 2023

Abstract:
Local coherence is essential for text generation models. We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters. In this paper, we propose a loss function inspired by a linguistic theory of coherence for learning image sequence representations. We further propose combining features from an object detector and a face detector to construct stronger character features. To evaluate visual grounding that current reference-based metrics do not measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories. Our code is available at https:// github.com/vwprompt/vcl.

heading: 1 Introduction
text: Storytelling is humans using language to convey stories to audiences, which is one of the oldest human activities of language use (Lucarevschi, 2016). In this paper, we focus on the visual storytelling task of generating stories from visual narratives, i.e., a sequence of images with a plot (Park and Kim, 2015; Huang et al., 2016; Hong et al., 2023). Stories, unlike image captions, contain several characters and events involving recurrent characters and their interactions with each other and the environment. Especially, characters are among the most important aspects of story writing (GoldfarbTarrant et al., 2020).
However, the current state-of-the-art (SOTA) visual storytelling models often fail to generate correct referring expressions for characters (Modi and
Parde, 2019). To confirm this, we generate 50 stories using a SOTA model (TAPM; Yu et al., 2021) and annotate different types of errors.1 The most prevalent error that appears in 60% of the generated stories is lacking local coherence which manifests as the characters appearing in the image not being mentioned correctly in the stories. For instance, in the first two rows of Figure 1, the character Jeremy appears across all images. The SOTA model TAPM first mentions him as Jeremy in the first sentence but misses him in the second sentence and inconsistently misnames him as Adam in the last sentence.
We identify two fundamental causes of this problem: 1) Most previous models do not represent the recurrence of characters in the image sequence explicitly. In particular, it is not clear whether the parameters in the model can capture character recurrence such that the visual coherence, local coherence in image sequences, can be reflected in textual coherence by repeated mentions of the recurrent characters in the story. We argue that a model that captures and represents the visual coherence better would also lead to more coherent stories.
2) Previous models mostly consider visual features without a focus on human characters, such as features extracted with a general vision backbone model. These features lack enough power to represent properties such as age, gender, face, or body shape. As a result, models using these features cannot distinguish between different instances of the same human character, which introduces wrong character mentions in generated stories.
Our contributions for tackling these limitations: (a) To tackle limitation 1), we propose a new ranking loss function, Visual Coherence Loss (VCL), which is inspired by models of cohesion in language (see section 4.1). VCL punishes standalone instances and rewards instances that appear
1TAPM is the only SOTA model that has publically available source code. See Appendix A for the details of annotation.
334
Xudong Hong 1
consecutively in visual narratives. Experiments show that the proposed loss function increases the coherence of the generated stories (see section 5.1).
(b) To obtain better character representations and tackle limitation 2), we experiment with different features that have proven useful for person re-identification tasks (see section 6). We then add the resulting features separately to visual story generation models. Experiments show that the representation from the face detector is most effective for character representation.
(c) When evaluating the stories generated by the models, we find that reference-based metrics cannot capture referring expression errors, so we propose a new evaluation metric character matching (CM) for image-text relevance in visual stories (see section 3). This metric is based on a weighted bipartite matching graph between the instances in the images and the co-referring expressions in the stories. We compute a coefficient from the weighted bipartite matching that measures local coherence. We apply our metric to prove the effectiveness of our loss function.

heading: 2 Related Work
text: Visually-grounded Story Generation. Modelling visual grounding is essential for vision-based language generation tasks like image captioning (PontTuset et al., 2020), paragraph generation (Ilinykh and Dobnik, 2022) and visual dialog (Yu et al., 2022). However, previous work in visual story generation conducted experiments only on the Visual Storytelling dataset (VIST, Huang et al., 2016),
which doesn’t have annotations for visual grounding. Parallel to our research, Liu and Keller (2023) annotate recurring characters in the VIST dataset which enable further exploration of grounding human characters in visual narratives.
Most of the previous models for visual storytelling only use global features, full image features extracted with a general vision backbone model trained on the image classification task (Yu et al., 2017; Wang et al., 2018; Huang et al., 2019). Some recent researches use local features, which are features of a specific part of the image such as objects, to generate visually grounded stories (Wang et al., 2020; Hong et al., 2020; Yu et al., 2021; Qi et al., 2021; Braude et al., 2022). Only a few works make use of human-related features like emotions (Li et al., 2019), sentiments (Chen et al., 2022) or persona (Chandu et al., 2019). One work that focuses on recurrences of characters is Dong et al. (2019), which employs a Bi-LSTM to encode conference chains in texts. However, their model requires textual features from the corresponding conference chains as input in the test time. On the contrary, our model only requires visual features from the images.
Coherent Story Generation. Coherence is one of the major properties that people would like to achieve in story generation (Alabdulkarim et al., 2021). Previous work generates a story conditioned on a prompt such as keywords (Yao et al., 2019), story plot (Rashkin et al., 2020), cue phrases (Xu et al., 2020), or scripts (Pu et al., 2022). Training data for story generation includes the STORIUM
dataset (Akoury et al., 2020). Its collaborativelywritten long stories contain rich annotations such as narrator prompts, character goals, and other attributes to guide story generation. Previous models based on these datasets require access to character labels which are expensive to obtain and not always available in real-world settings. In contrast, our model only needs an additional loss function during model training. Character Representations for NLG. Characters are essential to stories written and read by humans. Unfortunately, only a limited amount of work has been done on constructing explicit character representations. In story generation, Clark et al. (2018) represent characters with separate trainable parameters and integrate them with the latent representations of the language decoder using max pooling. But they only apply their method to referring expression generation instead of end-to-end story generation. In movie description, Rohrbach et al. (2017) extract head and body representations for characters and train the model for character re-identification and movie description jointly. However, all these methods integrate character features into RNN language decoders. Our model is the first attempt to encode character explicitly in Transformer-based models.

heading: 3 Image-Text Relevance Metric
text: In previous visually-grounded story generation work, models are evaluated with reference-based metrics like METEOR. However, the referencebased metrics only measure similarities between reference texts and generated stories such that the actual relevance of input images to output text is not evaluated. Specifically, we design a charactermatching metric (CM) to determine whether expressions referring to human characters in image sequences are generated correctly.
The computation process of CM is in Figure 2. For each pair of image sequence and story, we first construct the appearance matrices V ∈ {1,−1}m×nv and T ∈ {1,−1}m×nt , which indicate for each character whether it is present (1) vs. absent (−1) in an image or a sentence, where nv is the number of characters in the image sequence, nt is the number of referring expressions in generated stories and m is the number of images/texts. We then compute the matrix of matching scores of each pair as: M = VTT. We normalize the matching scores to the domain [0, 1] and obtain
the matrix M′ = (M/m + 1)/2. To obtain the optimal match between the images and sentences by the occurrences of characters, we compute the maximum weighted bipartite matching by applying the Hopcroft-Karp algorithm (Hopcroft and Karp, 1973) implemented in SciPy (Virtanen et al., 2020). The final overall character matching score is obtained by averaging the scores of the maximum matches across all characters (e.g. 0.9 in Figure 2).

heading: 3.1 Analysis of Baselines
text: We apply our CM metric to show the limitation of current models, which requires the labels of characters in image sequences. We use Visual Writing Prompts (VWP; Hong et al., 2023), a vision-based dataset that contains image sequences aligned with human-written stories in English2. In VWP, each image is corresponding to a piece of text. Instances of each main character are annotated with the name of the character (see Figure 1). Our baselines are:
Seq2Seq. (Seq2Seq; Huang et al., 2016) is a
2https://vwprompt.github.io/
seq2seq model with an encoder-decoder architecture. Visual features are first projected with an encoder which is a feed-forward neural network, then fed to the decoder which is a pre-trained GPT2 model.
TAPM. (TAPM; Yu et al., 2021) is a Transformerbased model which adapts the visual features with pre-trained GPT-2. This is the current state-of-theart model for visual story generation.
For generated text, we run a coreference resolution model (Lee et al., 2018) to capture all referring expressions. We compute the CM metric for the generated stories of two baseline models as well as the human-written stories. We compute the CM upper bound (100) by treating gold annotations of characters in image sequences as referring expressions in texts, i.e., using V as the textual appearance matrix. We also estimate the lower bound
(32.04) by randomly shuffling the appearance matrices and then calculating the CM metric using the shuffled matrices.
Results in Table 1 show that humans outperform the baselines significantly, which indicates that human-written stories match better with input image sequences in terms of characters. We believe that the gap between current SOTA models and the human stories stems from the models’ lack of representations for the recurrence of characters in the image sequence.

heading: 4 Visually Grounded Story Generation
text: In this section, we describe the visually grounded story generation model with a new loss function for self-supervised learning of image sequences. In the visual story generation task, given a sequence of images s = (img1, img2, ..., imgm), s ∈ Dm, the model needs to generate a story wo = (w1, w2, ..., wt−1, wt),wo ∈ Dt.
Encoder-decoder architecture Figure 3 represents our whole architecture, which is based on an encoder-decoder architecture that is also used in most previous work; our architectural changes are represented in the dashed blue box.
The first input to the model is the image sequence s. For the image imgi ∈ s, we extract global features f(imgi) ∈ Rd from the output of the last fully-connected layer of a pre-trained vision backbone model f . We then obtain ni bounding boxes of object eij , j ∈ [1, ni] predicted by an
object detector using the image encoder f as the backbone. We crop each detected object eij and feed the cropped image region to f to get the local features f(eij) ∈ Rd.
The global and local features are first fed to the global encoder fθ and the local encoder fζ correspondingly in order to project them to the same hidden dimensions as the transformer-based language decoder Pη, to which the hidden representations are fed. To generate tokens auto-regressively, the context w consisting of previously generated tokens (w1, w2, ..., wt−1) are fed into a pre-trained text embedding layer Q and then to Pη. Given the input sequence (s,w), the model represents the probability distribution of the next token wt as:
P (wt|s,w) = Pη(fθ(f(imgi)), fζ(f(eij)), Q(w))
The major limitation of the encoder-decoder architecture is that there are no representations in both global and local encoders for recurring characters in the input image sequence which we believe are essential to visual coherence in image sequences. The parameters η in the language generator Pη are not sufficient to capture character re-occurrence among the local representations Lij = fζ(f(eij)). So the visual coherence present in the image sequence is not reflected in textual coherence. We hypothesize that a model that captures and represents the visual coherence better would lead to more coherent stories.

heading: 4.1 Visual Coherence Loss
text: We propose Visual Coherence Loss (VCL, in the blue box of Figure 3) as an auxiliary loss to enable the model to capture the visual coherence of image sequences. Our goal is to measure visual coherence in terms of the recurrence of characters.
Centering theory (Grosz et al., 1995) is one way to explain the relationship between character recurrence and narrative coherence. This theory holds that if a pair of adjacent sentences contain the same discourse entities (here: characters), readers are more likely to find the transition between this sentence pair to be coherent. Character recurrence is very common in visual as well as textual narratives. In addition, images in visual narratives can be considered sentences in textual narratives. Inspired by these, we apply centering theory on measuring coherence in visual narratives. Character instance similarities. To apply centering theory, we first need to find the same-character
recurrences, but we do not have labels for characters at test time. Instead, we can take advantage of the similarities between character instances (as bounding boxes of person objects) of the same character across different images. We use the similarities as soft labels to capture different instances of that one character without any manual labels.
To compute similarities between character instances, the first step is to obtain their representations. We identify the objects depicting people automatically and use their local representations L = fζ(f(e)). The representations must be in the latent space of the language decoder Pη. So we pass the local representations through the language decoder to get the contextualized local representations as E = Pη(L). The next step is to compute similarities of character instances across all images. We calculate an instance similarity matrix, consisting of dot products of contextualized local representations, as: S′ = (sijkl) = (ETikEjl), where k ∈ [1, ni], l ∈ [1, nj ]. Character re-occurrence. Another condition of applying centering theory is to be able to identify where the characters actually recur or not in two adjacent images. We first need to identify occurrences of each character instance in each image. We, therefore, compute a matrix of image-instance similarity. We group elements by image along one dimension indexed by k in S′ and compute the average as: S = meank=nik=1 (sijkl) = (sijl) where i ∈ [1,m], j ∈ [1,m], l ∈ [1, nj ]. Each element sijl is the similarity between image i and character instance ejl in image j.
We measure visual coherence in terms of the squared difference between similarities of character instances in the two adjacent images. For an illustration of this idea, consider the following three cases: 1) if the similarity between a character in the present and the previous image is high and the similarity between a character in the present and the next image is low, it implies a sharp transition, the difference is high; 2) if both of them are high, it implies a smooth transition, the difference is low; 3) if both of them are low, it suggests that this character does not appear in either of the two images, the difference is also low. We can thus use the squared difference to measure the level of recurrence of characters across images. Finally, we can compute the average squared difference for all character instances in all pairs of adjacent images as the visual coherence coefficient. We let Sa...b be the
rows from a to b indexed by i of S. We can define the coefficient as LC = ∥S2...m − S1...(m−1)∥2.
Loss function. The lack of local coherence in the visual embeddings is detrimental to the ability of the latent representations in the language to produce coherent stories. We address this by defining a visual coherence loss to encourage the latent representations to be coherent and maintain the coherence structures in the visual narratives.
We design a ranking loss function to punish stand-alone instances and reward instances that appear consecutively in image visual narratives. To do so, we first construct negative samples by randomly shuffling images in the visual narratives which breaks existing coherent groups of recurrent characters. Then we compute the visual coherence coefficient for negative samples as L−C . Lastly, we define the visual coherence loss as a ranking loss: LR = max{0, 1 − L+C + L−C} where L+C is the visual coherence coefficient for positive samples. The final loss function is L = LL + αLR where LL is the cross-entropy loss for language modeling and α is the weight of ranking loss.
Model training. To better adapt the latent space of model parameters to fit the image sequence, we pretrain our model to optimize the visual coherence loss with image sequences only. We can do so because both terms in the visual coherence loss can be computed with images only. After pre-training, we fine-tune our model with paired image sequences and stories to optimize the final loss.

heading: 5 Experiment and Evaluation
text: In this section, we experiment with the Visual Writing Prompt (VWP; Hong et al., 2023) dataset to demonstrate the effectiveness of our loss function (see Section 3.1). We follow their settings to separate the data into train, validation and test split. We extract global features for all images using the Swin Transformer (Liu et al., 2021), a state-of-the-art supervised vision backbone model. We use the base model pre-trained on the ImageNet-21K dataset released on Hugging Face Models. For local features, object features are obtained using a Cascade Mask R-CNN object detector (Cai and Vasconcelos, 2019) with the same Swin Transformer backbone. We crop the bounding boxes of the top 20 objects that the detector predicts for each image. Then we extract the object features (obj) similarly as how we extract the global features. We also construct the oracle (oracle) features by passing the IDs of the characters to the language decoder. We encode the character IDs via the positions of the feature vectors. For example, character #1 takes the 1st position of the character feature matrix. Because the positional encodings in the language decoder are sensitive to the positions, the character IDs are also encoded in the input.
We use GPT-2 (Radford et al., 2019), a Transformer-based language model pre-trained on large-scale text as the language decoder. We use the small version which is widely used in previous works of story generation. We compare baseline models using the visual coherence loss combined with different character features against the two baseline models defined in Section 3.1.

heading: 5.1 Automatic Metrics
text: To check whether referring expressions of human characters in stories are generated correctly, we apply our character-matching (CM) metric defined in Section 3. To compare with previous baselines, we also evaluate our models with reference-based metrics including unigram (B-1), bigram (B-2), trigram (B-3), and 4-gram (B-4) BLEU scores (B; Papineni et al., 2002), METEOR (M; Banerjee and Lavie, 2005), ROUGE-L (R; Lin, 2004), and CIDEr (C; Vedantam et al., 2015), which were used in the visual storytelling shared task (Mitchell et al., 2018).
Results in Table 2 show that our model trained with visual coherence loss (VCL) substantially outperforms previous baselines like Seq2Seq and TAPM on the CM metric. This supports the hypothesis that the visual coherence loss is an effective means to make the texts better reflect the visual coherence of recurrent characters in the image sequence. We also observe that the model outperforms the baseline on most reference-based metrics except ROUGE-L. The Seq2Seq model with oracle features performs best which again shows the importance of character information.

heading: 5.2 Human Evaluation
text: As we mention in Section 1, we find that most errors are related to lack of local coherence (60% of stories). To get a better understanding of how model fails, we subdivide these errors into three types based on character consistency between images and stories: Wrong Referring Expressions (REs): Consistent number of characters in the images and stories, but incorrect reference. Too Few REs: The number of characters in the stories is smaller than the number in the images. Too Many REs: The number of characters in the stories is more than the number in the images.
We ask three graduate students with a sufficient level of English to annotate these three types of
errors. The annotation was preceded by a training phase, during which annotators were instructed to familiarize themselves with the error types and to perform trial annotations. Then, we collected their annotations and gave them feedback to correct for misunderstandings and biases. Formal annotations started after the completion of the training phase, and was blind to condition, i.e., annotators did not know which story was generated by which model. We collected 50 annotated stories for each model.
For each generated story, we counted the occurrence of the three types of character errors and then calculated the mean inter-annotator agreement (IAA) using Pearson’s r and Cohen’s κ. IAA on this task is moderate r(48) = 0.45, p < 0.01 and Cohen’s kappa coefficient κ = 0.42.
Results in Table 3 show that compared with the state-of-the-art models, our model makes much fewer character errors. On the other hand, there is still a large gap between our model and one with oracle features, both in Table 2 and Table 3. The global/obj features are not sufficient to represent characters. We thus explore different character features in the next section.

heading: 6 Character Representation
text: Another possible cause of local incoherence is that previous models mostly consider only the global or local visual features without a specific focus on human characters. The only local features that are related to human characters are object features, i.e., the bounding boxes or masks around the person objects from the object detector. Object features
are effective and widely used in visual story generation. However, these features are either extracted using a general vision backbone model trained on the ImageNet dataset or directly from the output of the last hidden layer of the object detector. These features do not represent human age, gender, face, or body shape. These features can thus not distinguish between different instances of the same human character, which introduces wrong mentions of characters in generated stories.
To improve character representations, we need the following: 1. distinguishing different characters; 2. identifying different instances of the same characters. We claim that with these two properties, the visual encoder can represent the characters effectively, and the language decoder can capture these characteristics and generate more coherent and visually-grounded stories.

heading: 6.1 Types of Features
text: To identify characters, the use of facial features seems most promising, as face detectors have been shown to achieve highly accurate results (Deng et al., 2020). To avoid problems due to covered or invisible faces, we also consider features from other body parts. For visual narratives, it is often safe to assume that clothing over other body parts doesn’t change between the images of a narrative. Joon Oh et al. (2015) employs the full body features and also inferred upper body features and head features to help person re-identification in photo albums. Body features have been proven to be effective on other similar tasks like character re-identification (Yu et al., 2020), social relation recognition (Sun et al., 2017), and movie description (Rohrbach et al., 2017).
Inferred Features. In our case, we cannot guarantee that our gold labels of human characters cover full bodies, because our annotations are based on the movie shots which often only contain half bodies or heads of the characters.
For each person bounding box, we first pass it to the face detector to get the face bounding box. We propose to use the top boundary of the character bounding box and the other three boundaries of the detected face from face detection to infer a head bounding box. Then we crop the images to get the head features (head←face). We follow Joon Oh et al. (2015) to compute body (body←face) and upper body (upper body←face) features using the inferred head features.

heading: 6.2 Analysis of Properties
text: After obtaining the face and body features, the first question that we would like to investigate is whether these features can distinguish different instances of the same character and also can distinguish different characters. Therefore, we compute the average similarities of different instances of the same character (Same) and the average similarities between different characters (Diff). We further compute differences between these two similarities (∆) to show their ability on representing characters.
The results in Table 4 show that general vision backbone models like Swin Transformer are insufficient for differentiating different characters because the Diff scores of Swin-based features are much higher than the insightface features. The most effective method is to use the hidden representations directly extracted from the insightface Face detection system3.

heading: 6.3 Ablation Study
text: To further compare the best-performing features extracted with Swin Transformer (body) and features based on insightface, we conduct experiments on VWP dataset. We apply different character features one-by-one together with object features to VCL model in visual story generation. We evaluate the models with the same metric as Section 5.1 except we report the BLEU score (B) as the average of BLEU-1 to BLEU-4 scores.
The results in Table 5 show that the model with body features performs on par with the model that only uses obj on all metrics. However, insightface features improve the character matching score significantly (t-test, p < 0.05), proving face features to be effective.
3https://insightface.ai/

heading: 6.4 Case Studies
text: To better illustrate the effectiveness of our CM metric, we present a case study in Figure 1. According to the error types defined in Section 5.2, we can see that the Oracle, TAPM and VCL model all make the Too Few REs error and additionally the TAPM model makes an Wrong REs error, while our proposed VCL model with the insightface feature avoids these problems. Furthermore, comparing the results of METEOR and our proposed CM, we find that METEOR does not reflect the errors in referring expressions: although the VCL model makes one reference error, its METEOR score is higher than the VCL model with the insightface feature, even though it does not make any error in referring expressions. In contrast, our proposed CM can faithfully reflect such errors.

heading: 7 Conclusions
text: We identified a major important problem of current vision-based story generation models, lack of local coherence (characters are not mentioned correctly). We dissect this problem into two aspects: 1) whether the latent visual representations fed to the language decoder are sensitive to the coherent structure in the visual narratives; 2) whether the visual features can distinguish different characters or identify instances of the same character. We then propose a visual coherence loss to constrain the latent visual representations together with the parameters in the language decoder such that they can represent recurrences of characters. Our model trained with visual coherence loss generates more coherent stories and obtains superior performance on both automatic metrics and human judgments. We also find that using the visual features of characters allows us to produce stories with better identification of recurring characters, which are as a result more locally coherent.

heading: 8 Limitations
text: One limitation is that the importance and visual salience of character instances are not measured directly. We plan to settle these in future work.

heading: 9 Ethics Statement
text: The VWP dataset we used is publicly accessible. It is described in a paper (Hong et al., 2023) that is published in TACL under the CC-BY license.
The potential risk of this work is that it can be used to generate harmful content. There could be
potentially offensive images in VWP dataset because it is based on movies. Our model might suffer from the risk of learning unwanted correlations between these images and offensive words in the underlying language model GPT-2 because it has been found that there is a significant amount of unreliable or toxic content in the training data of GPT-2 (Gehman et al., 2020). Although we have not seen any generated in our human evaluations, our proposed model and code are for research purposes only.

heading: Acknowledgements
text: This research was funded in part by the German Research Foundation (DFG) as part of SFB 1102 “Information Density and Linguistic Encoding” and a Swedish Research Council (VR) grant (2014-39) for the Centre for Linguistic Theory and Studies in Probability (CLASP). Xudong Hong is supported by International Max Planck Research School for Computer Science (IMPRS-CS) of Max-Planck Institute for Informatics (MPI-INF). We sincerely thank the anonymous reviewers for their insightful comments that helped us to improve this paper. We thank the students who help us to evaluate the generated stories: Mattes Alexander Warning, Sijie Wu and Trevor Atkins.

heading: A Settings of Human Evaluation
text: For the self-assessment in Section 1, we first adopt the list of common errors from Modi and Parde (2019). Then we add additional errors that we identified. We also merge errors that are too similar for simplicity. Finally, we define nine error types which are described in Table 6. After that, one author annotated 50 stories generated by the SOTA model TAPM.
For the human evaluation in Section 5.2, the annotation guideline is in Figure 4. We process all the data we collect to delete the unique identities of annotators. One annotator is paid by the funding source under a student assistant contract. The other two are student volunteers. One annotator is a native speaker of English and the other two are graduate school students of Saarland University who speak proficient English.

heading: B Settings of Experiments
text: For the analysis in Section 3.1, We use this implementation of a coreference resolution system4 from AllenNLP. We use this version5 of Scipy.
4https://demo.allennlp.org/ coreference-resolution
5https://docs.scipy.org/doc/scipy/ reference/generated/scipy.sparse.csgraph. maximum_bipartite_matching.html
For the experiments in Section 5.2, We build our system based on the code6 by Yu et al. (2021). We follow their settings for hyper-parameters as the initial setting. Then we tune our hyperparameters based on evaluation results on the validation split. We use the Huggingface transformer for Transformer-based models: Swin Transformers7 and GPT-28. Due to our limited computing resources, we use the base versions of both Swin Transformers and GPT-2. E.g. our maximum batch size is 6 on a GPU with 32GB memory. The total computation time is about 6 hours for 20 training epochs on one Nvidia V100 32GB GPU. Our code is in the Supplemental, which is unlicensed and for review only. All hyperparameters are in config.py. We will publish our code on GitHub after the review process of this submission.

heading: C Experiments on VIST and VIST-Character
text: We also evaluate our visual coherence loss on the VIST dataset (Huang et al., 2016) and the VISTCharacter dataset (Liu and Keller, 2023). We first train all the models on VIST. Then we evaluate them on the test split of VIST and the VISTCharacter dataset. We use the same set of metrics described in Section 5.1 but we only report BLEU-1 and BLEU-4 for BLEU scores here for a better format. The results in Figure 7 show that model with our VCL only outperforms the Seq2Seq baseline significantly on BLEU-1 scores. The differences between VCL and all other models in all metrics are not significant. The reason is that the VIST dataset is collected in a way to exhibit less visual coherence (lower rates of recurring characters). It contains fewer human characters per story than VWP. Also, some of the features we are using for character re-identification may not be suitable to other datasets to the same extent (for instance, if the clothing of characters changes more often between album photos). We also show that the results on VIST-Character are much lower than the results on the VIST test set, which indicates that modelling character in visual stories is still very challenging.
6https://github.com/JiwanChung/tapm/ 7https://huggingface.co/microsoft/
swin-base-patch4-window12-384-in22k 8https://huggingface.co/gpt2

heading: Future Work
text: We plan to settle these in future work