Visual Storytelling with Question-Answer Plans
Authors: Danyang Liu
Mirella Lapata
Frank Keller
Venue: EMNLP 2023 Findings (2023)
Keywords: visual storytelling
multimodaility
story generation
TL;DR: This paper presents a novel framework for visual storytelling that converts image sequences into visual prefixes and uses a blueprint plan of question-answer pairs to build coherent, interesting and natural stories.

Abstract: Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative.  Automatic and human evaluation on the VIST benchmark demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.

Visual Storytelling with Question-Answer Plans
Danyang Liu, Mirella Lapata, Frank Keller Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB danyang wwwee..ddde......... uk, {mlap, keller]@inf ed. ac uk
Abstract
Visual storytelling aims generate compelling narratives from image sequences. Existing models often focus on enhancing the represen- tation of the image sequence, c.g., with external knowledge sources or advanced graph struc- tures. Despite recent progress, the stories are often repetitive, illogical, and lacking in de- tail. To mitigate these issues, we present novel framework which integrates visual rep- resentations with pretrained language models and planning. Our model translates the image sequence into visual prefix, sequence of continuous embeddings which language mod- els can interpret. It also leverages sequence of question-answer pairs blueprint plan for selecting salient visual concepts and deter- mining how they should be assembled into narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint- based models gen- erate stories that are more coherent, interesting, natural compared to competitive baselines and state-of-the-un systems.
small indie rock group performed for music our city. The girls doing hannonies in the songs were phenomenal. The band had three guitarists, One of the guitarists resembled James Morrison. More independent bands played for the grand opening of the music sture.
Qui What group performed for a music store in our city?
Agi small indic rock group
Q₁: What kind of music did the girls do?
Aji harmonies
Q₃: Who did the harmonies in the songs?
Ag! the girls
Qş: How many guitarists did the group have?
Agi three guitarists
Q4: Who resembled James Morrison?
Agi one guitarists
O5: Who played for grand opening of music store?
Agi more independent bands
Figure Blueprint annotation for visual story. Color- coded answers are extracted from the gold story. Ques- tions are generated by feeding the answers and the gold story context pretrained question generator.
1 Introduction
encoded into a representation which typically in- cludes image embeddings and detected objects. Subsequently, a decoder generates a story token by token based on the encoding of the image sequence. Recent work has mainly focused on enhancing the first stage of the generation process e.g., by leverag- ing external knowledge sources (Hsu al., 2021; Chen et al., 2021; Hsu et al., 2020; Yang et al., 2019). Advanced representations for image quences have also been explored, such as scene graphs (Hong et al., 2020) and story graphs (Hsu et al., 2021). Despite recent progress, these meth- ods struggle to produce meaningful narratives, are prone to ballucination and repetition, often gener- ate vague sentences, and have difficulty identifying salient visual concepts.
Visual storytelling involves narrating an engaging and logically coherent story based on sequence of images (see the example in Figure 1). The task lies at the intersection of natural language process- ing and computer vision and has recently attracted increasing interest from both communities (Wang et al., 2022; Hsu et al., 2021; Xu et al., 2021; Chen et al., 2021; Hsu et al., 2020; Wang et al., 2020; Huang et al., 2016). Visual storytelling differs from image captioning, which typically focuses on gen- erating descriptive text, e.g., by identifying and depicting objects within an image. It requires deeper understanding of how images and the events they illustrate relate to each other in order create a convineing narrative.
Visual storytelling is commonly modeled as a two-stage process. The image sequence is first
We attribute the lack of story quality to at least two reasons. Previous work on text- based genera-

---

tion has demonstrated that planning can improve story coherence, allowing control the trajectory of events, the characters described and their ac- tions (Yao et al., 2019; Xu et al., 2018; Rashkin et al., 2020a; Fan et al., 2019; Yang et al., 2022). However, plan- ning has not been considered in the context of vi- sual storytelling, existing models adopt black-box architectures which not particularly control- lable or interpretable. Another limitation concerns the nature of current models which are essentially trained from scratch, and as a result have limited language modelling and generalization capabilities (they only see multimodal training samples; see top of Figure 1). Although pretrained language mod- els (Raffel et al., 2020; Lewis et al., 2020; Brown et al., 2020) have been widely adopted for general- purpose story generation, their potential for visual storytelling remains unexplored.
ated so far. Experiments on the VIST benchmark (Huang et al., 2016) show that blueprint-based mod- els generate more coherent, interesting. and human- like stories compared to the state of the art and large language models (LLMs) like GPT-3.5,, according to automatic and human evaluation.
et al., 2020; Goldfarb-Tarre
2 Related Work
Visual Storytelling Huang et al. (2016) intro- duced visual storytelling as a vehicle for devel- oping AI tools with human-like understanding of grounded event structure and linguistic abili- ties that go beyond descriptive language. While carlier work (Gonzalez-Ree Rico and Fuentes-Pinedie 2018; Kim et al., 2018) employed simple encoder- decoder architectures (using CNNs to extract visual features and RNNs to generate text), more recent methods (Xu et al., 2021; Chen et al., 2021; Hsu et al., 2020; Yang et al., 2019) leverage external resources (e.g., ConceptNet) as way of instilling commonsense reasoning skills. Sometimes, scene graphs are also used to model relations between objects (Lu et al., 2016; Hong al., 2020; Wang et al., 2020). To our knowledge, none of these ap- proaches make use of plan-based decoding. Hsu et al. (2021) construct a graph representing the im- age sequence (based on training data and external resources) and identify the highest scoring path as the best storyline encapsulated therein. The story- line can be viewed as a form of planning, however, on the encoder side.
In this work we propose an approach to visual storytelling which integrates pretrained language models with visual representations and incorpo- rates an intermediate planning step before gener- ating the full story. Our encoder translates the im- age sequence into a visual preftx, a sequence of continuous embeddings which language models can interpret. Following Narayan et al. (2022), represent plans as a sequence of question-answe annwer pairs, called blueprints, which serve as proxy for content selection (i.e., what to say) and planning (i.e., in what order). Blueprints are loosely related to the Question-unddr-Diseuue
(QUD) theory of discourse (Larsson, 2002; Roberts, 2012; De Kuthy et al., 2020), which posits that text structure can be analyzed by identifying implicit questions raised and answered by subsequent spans of text. We augment visual storytelling training data with story blueprints (see Figure 1), which we obtain automat- ically thanks to state-of-the-art question generation technology.
Most existing approaches (Xu et al., 2021; Hsu et al., 2020; Wang al., 2020; Yang et al., 2019) train Transformer models from scratch, with the ex- ception of Chen et al. (2021), who employ a vanilla BART model as a baseline without task-specific adaptation. In contrast, our work leverages the lan- guage modeling and generalization capabilities of pretrained language models for visual storytelling.
Planning and Generation In the domain of au- tomatic story generation, planning has been effec- tive at capturing the content and structure of sto- ries. The generation process is often decomposed into two stages, namely planning an outline and then elaborating on it, e.g., by filling in specific details of story. Plans have been represented as a sequence of event or phrase keywords (Yao et al., 2019; Xu al., 2018; Rashkin et al., 2020). character actions (Liu et al., 2020), plot structures (Goldfarb-Tamre et al., 2020a), and more clabo- rate descriptions including details about the setting
We fine-tune pretrained language models to gen- erate blueprints from image sequences and the sto- ries based on them. We showcase two types of storytelling models, which vary in how the plan- ning mechanism is implemented. A top-down model generates the blueprint first and then contin- ues to generate the corresponding story in one go, whereas an integrated model interleaves planning with text generation rather than determining a plan in advance; generation is iteratively conditioned on the image input, the blueprint and the story gener-

---

of the story, its characters, and main plot points (Yang et al., 2022).
Visual
Mapping
Network
Prefixx
The idea of having a separate planning stage has also been explored for other text generation tasks including summarization (Narayan et al., 2021; Liu and Chen, 2021) and data-to-text generation (Moryossef et al., 2019; Puduppully et al., 2022). Our work is closest Narayan al. (2022) who propose the use of question-ansee ansoeer pairs as inter- mediate plans for summarization. However, their approach is designed for descriptive text. Our work extends their framework to a multimodal setting, where the input consists of image sequences, and the output are narratives characterized by more ab- stract and figurative language.
FF
Image
Encoder
Concept Embodder
CD
EL
Figure 2: Visual prefix construction. The pretrained image encoder and concept detector are frozen. FF refers to a feed-forward layer, CD and EL denote a concept detector and embedding layer, respectively.
dates with a T5 model (Raffel et al., 2020; Romero, 2021) fine-tuned on the SQuAD reading compre- hension dataset (Rajpurkar et al., 2018). The an- swer and the story provided context to pre- dict the corresponding question. We decontextu- alize stories by replacing pronouns with their cor- responding head mentions, using a state-of-the-art coreference resolution model (Dobrovolskii, 2021). Question-answer pairs are subsequently filtered to eliminate noise which is unavoidable due to the automatic preprocessing steps mentioned above. We thus remove any question-anse answer pairs where the answer is already present in the question. We also employ round-trip consistency check (Alberti et al., 2019) which discards questions if they yield answers different from those used to generate them.
3 Blueprint-based Visual Storytelling
Let represent sequence of images, denoted as {vi, 12.... va). Given this input, our goal is to generate a blueprint plan B (i.e., an ordered set of question-answer pairs) and story S based it. Most generation datasets do not include blueprint annotations, and visual storytelling is no excep- tion. We first describe how we automatically obtain { Br, S W training data samples (Section 3.1), and then introduce our story generation models (Section 3.2).
3.1 Blueprint Annotation
Let S.) denote a dataset consisting of pairs of image sequences and their corresponding sto- ries. We automatically create blueprint B; based story S; using sfate-of-the-art question genera- tors (Romero, 2021; Raffel et al., 2020), coupled with a filtering procedure to remove repetitions and ill-formed questions.
3.2 Blueprint Models
Our approach leverages the generation capabili- ties of pre-trained sequence-to-sequeece models. As our backbone model, we employ BART-base (Lewis et al., 2020) which has been fine-tuned for text generation. We adapt this model to our visual storytelling task two ways. Aside from enabling the generation of blueprints, we convert the image sequence to visual prefix which the pretrained lan- guage model can interpret. The pretrained language model is prompted with this prefix generate the blueprint, and eventually the story.
The generation of question crestion-ansser answer pairs involves two steps, namely answer extraction and question generation. In the context of storytelling, captur- ing key events is crucial for compelling narra- tive. While noun phrases and named entities are commonly recognized as significant content units other tasks such as summarization (Narayan et al., 2022; Deutsch and Roth, 2023), verb phrases also play vital role in conveying story dynamics, actions, and relationships (Trabasso et al., 1989; Eisenberg and Finlayson, 2017; Liu et al., 2020). Therefore, in addition named entities and noun phrases, also extract verb phrases answer candidates using the spaCy library.
Visual Prefix Construction Our model needs to grasp what the image sequence is about, e.g., the depicted objects, actions, and their associations. Drawing inspiration from recent advances in vision and language research (Mokady al., 2021: Tsim- poukelli et al., 2021; Alayrac et al., 2022; Zhai et al., 2022; Liu et al., 2023; Huang et al., 2023), we translate the input sequence of images into a
We then generate questions for answer candi-

---

Blueprint
Story
Embedding Layer
Iter.
Ozi What gome did the drum line prepore for?
The big drumline prepored for the High School footboll gome.
Ayj the high school football game
Og Who walked out onto the field?
The drumline prepared for the High School football game. The boys all walked out onto the field, reody for the game.
Visual Prefix
Context
Ag boys
mony boys walked the fleld?
Pretrained Language Model
Ier.
Figure Iterative blueprint model: the first iteration, the embedding layer uses <START> as context; in the second iteration, the context includes the blueprint and previously generated story (enclosed by blue dashed line). In contrast, the top-down model takes only the visual prefix as input and predicts global blueprint and corresponding story in go. For details on the visual prefix, see Figure 2.
sequence of continuous embeddings, aka a visual prefix (see Figure 2).
log-likelihood of the joint distribution:
N
Following previous work (Wang al., 2018; Xu al., 2021; Chen et al., 2021), we use ResNet-152 (He et al., 2016) extract visual features from im- ages. We next employ a lightweight linear mapping network that consists of a series of feedforward layers, denoted as Fs. to map image features to k visual clues:
max 8,0
log Pa. (B S; 4))
(3)
where (B, refers to the concatenation of blueprint B and story S. 9 represents the param- eters of the pretrained language model, e are the parameters of the mapping visual network Fg, and N denotes the size of the dataset.
We introduce special tokens Story: and Plan: preceding the story and blueprint, respectively. In experiments, our blueprints consist answer- question pairs {ar, QI...., Om, 4m (rather than question-anse answer pairs). We place the answer be- fore its question to encourage the model to zoom in salient visual concepts depicted in the image sequence. This ordering is intuitive for story- telling task: We first decide on what the story is about and then elaborate on key concepts. Inciden- tally, Narayan al. (2022) also find that generating the answer before the question performs better for their summarization tasks. Finally, the model is trained with the standard maximum likelihood ob- jective to generate the joint target.
Pl....:P. Fs (ResNet (V1, U2, (2, ...., 0k)) (1)
where l is the image sequence length, and each visual clue ps has the same dimensionality as token embedding of the pretrained language model.
To further instil world knowledge in our visual prefix, we employ concept detector. The latter identifies specific objects within images, but also actions, scenes, and attributes. For each image vi we retain the K concepts {c], .?....,. c with the highest confidence score:
,....,c Concept (ResNet (u:)
(2)
Concepts for each image are then concatenated with (SEP) token and serve as input the em- bedding layer of the pretrained model. The visual clues and concept embeddings are concatenated to form the visual prefix V. The image encoder and the concept detector remain frozen during the training phase. Only the parameters in the mapping network Fg are updated (see Figure 2).
Iterative Planning This model employs an in- cremental generation strategy create the story, Rather than generating in one step global blueprint and the story, planning and generation are interleaved. At each time step, the iterative model considers the image sequence and context from previous steps, including the blueprint and story generated so far. We gradually construct the blueprint and its corresponding story sentence-by- sentence; our planning is informed by generation and vice versa, which argue should be mutually beneficial (they are conditioned on each other).
Top-down Planning This model takes image se- quence as input and generates blueprint B and story S in one go. More precisely, during decod- ing, the model first generates the blueprint, which then serves as prompt guiding subsequent story generation. Our training objective maximizes the
Let {si, 82....... } denote a target story and = {b.. b2...., by its blueprint where
5k

---

Length of image sequences
5.0
4.2
Implementation Details
Number of sentences in the story
5.0
Our models are built on top of BART-base (Lewis et al., 2020) and finetuned with learning rate of 3e-5, batch size of 64, and warm-up ratio of 0.05. We select the best checkpoint on the validation set using a QA-based metric which quantifies the ex- tent to which the output story follows its blueprint (see Section 4.4). During inference, we employ beam search (size 5). For our visual prefix, we employed the Clarifai Concept Detector which was trained on dataset containing 9,098 concepts and 20 million images (multiple concepts are assigned to each image), and is integrated with the Incep- tion V2 architecture (Szegedy al., 2016).
Number of tokens per story
52.3
Number of QA pairs per story
11.1
Number of tokens per QA pair
10.3
Number of tokens per story plus QA pair
166.2
Table 1: VIST dataset Statistics (average values).
8; represents the i-th sentence in the story, and b; its associated blueprint. Each b; consists of answer- question pairs, denoted as {at, .i..... a Beg J- where (()) is the number of pairs in the i-th blueprint. The training objective for the iterative model is defined as follows:
4.3 Comparison Systems
NN
We compared our models against several base- lines and state-of-the-art visual storytelling mod- els. These included a vanilla BART-base model with the same encoder and visual prefix ours but no planning (VP-BART; it generates the story directly in an autoregressive manner without the blueprint). KG-Story (Hsu et al., 2020) predicts a set of words representative of the image sequence, enriches them using external knowledge graphs, and generates stories based on the enriched word set. PR-VIST (Hsu et al., 2021) is a tart model which constructs a graph representing the relations between elements in the image sequence, identifies the best storyline captured therein, and proceeds to generate a story based on it. The pro- cess of constructing the story graph can be viewed as a form of planning. Along similar lines, Chen et al. (2021) build a common sense knowledge graph capturing concepts in the image sequence, and use MCSM, a Maximal Clique Selection Mod- ule to identify which ones to write story about. They use BART-large to generate the story based selected concepts (and image features).
max 0,6
log po.o (bi+1, 8141 1141/bia; $1:4, 12) (4)
where bis and 814 refer to the blueprint and sen- tences generated so far, from step 1 to step i.
At each time step the encoder takes image sequence I as input, and the decoder takes the context (i.c., blueprint and sentences generated so far (b), b,.... by; 81, 82, $2,...,,;; s;)) as a prompt to predict the next blueprint b;+1 and sentence 81-1-
Therefore, the iterative model is trained on samples {I, (bi.i, $1:4), be+1, 81+1}. We prefix (b1., $1.)). bit1, and 8i+1 with Context: Plan:, and Next Sen- tence, respectively. To handle the first time step. we introduce special token (START) as context to predict bj and $1. We also use (END) to indicate the completion of an iteration (see Figure 3 for an illustration). It is important to note that (b1.s, $1:1) are masked out when computing the loss because they serve as prompts to the decoder. We want to avoid the model repeatedly predicting and overly optimizing the blueprints and sentences that appear at the beginning of the output.
We also compared against an LLM that gener- ates stories via prompting. We provide GPT-3.5 with a visual prefix, namely the concepts identified in the image sequence, and a prompt which ex- plains how to create the blueprint and generate the story together with examples (in-context learning). Details on the prompt can be found in Appendix A.
4 Experimental Setting
4.1 Dataset
We performed experiments on the widely used VIST dataset (Huang et al., 2016), which contains 10, 17 Flickr albums and 210.819 unique photos. Each training sample consists of k = images and a corresponding story of = sentences. As de- scribed in Section 3.1, we augment each story with an automatically generated blueprint.
4.4 Automatic Evaluation
We evaluated our stories using BLEU, ROUGE, METEOR, and CIDER, mainly to compare to pre- vious work. Several studies (Hsu et al., 2022, 2021,

---

2020; Hu al., 2020: Yang al., 2019; Modi and Parde, 2019) have demonstrated the inadequacy of lexical matching metrics: they correlate poorly with human judgments, and not do effectively mea- sure the semantic similarity to human-written sto- ries or the lexical richness of the generated stories.
Pretrained Language Models Produce Better Stories We observe that models based on pre- trained language models (i.e., our models and MCSM, outperform models trained from scratch (i.e., KG-Story and PR-VIST) in terms of trigram- repetition scores and MAUVE. This indicates that we can maintain strong language modeling capabil- ities while enabling pretrained language models to process visual signals effectively.
We further employ story-specific metries to as- sess story quality aspects such as diversity, fluency, naturalness, and grounding. Specifically, we use two types of trigram repetition metrics (Yao et al., et al., 2020b). Intra-story
The Visual Prefix is an Effective Interface be- tween Image and Text MCSM is the only ex- isting model that utilizes pretrained language model for visual storytelling. However, our base- line model (VP-BART) demonstrates superior per- formance in most story-specific metrics. Remark- ably, this achieved using a smaller pretrained model (BART-base, 140M parameters); MCSM is built on top of BART-large (400M parameters). This highlights the effectiveness of our visual pre- fix, indicating it successfully translates the image sequence into space that BART can understand.
2019; Goldfarb-Taree repetition is a fluency metric, it measures the pro- portion of trigrams repeated within a story. Inter- story repetition examines trigram repetition across stories. This metric evaluates diversity, high intra- story repetition suggests that the model tends to generate the same story even when conditioned on different image sequences. We also use MAUVE (Pillutla et al., 2021) to measure the naturalness of the generated stories. MAUVE is a recently intro- duced automatic metric for open- open-ended generation which has high correlation with human judgements. It computes the similarity of the distribution of human-written text and machine-generated text.
Blueprint Models are Most Grounded Our models outperform comparison systems in terms of concept grounding. This confirms that an interme- diate planning step allows the model to effectively select salient concepts based on the visual prefix. The top-down model in particular achieves the high- est concept grounding recall, it stays close to the image sequence, accurately describing the informa- tion conveyed therein. The higher lexical matching scores further support this observation. The iter- ative blueprint model achieves the best concept grounding precision (excluding GPT-3.5 models) which in turn suggests that the stories generated by this model exhibit a stronger grounding the images with fewer hallucinations.
To quantify the extent to which the generated story is grounded, i.e., whether it accurately repre- sents the content of the image sequence, we mea- sure concept precision and recall. Precision mea- sures the number of words in the generated story that align with the detected concept set, while re- call assesses the number of words in the detected concept set that are present in the generated story. Finally, for our own models we also evaluate whether the generated stories are faithful to their blueprint. Drawing inspiration from recent stud- ies summary evaluation (Deutsch et al., 2021; Fabbri al., 2022), we measure how well the gen- erated story answers questions from the predicted blueprint. We utilize a RoBERTa-based based (Liu et al., 2019) QA model finetuned on the SQuAD dataset.
The Iterative Model Generates Most Natural and Faithful Stories Despite not achieving the highest scores in lexical matching metrics, the itera- tive blueprint model stands out in terms of MAUVE evaluation. Compared to other models, it generates more natural stories, closer to those written by hu- mans. This finding suggests that humans might em- ploy similar iterative planning strategy, at least for the short stories considered here; they construct a narrative gradually rather than global plan which they subsequently convert into story.
5
Results
Our results are summarized in Table 2. The first block presents the performance of of the-art storytelling systems. The second block presents variants of our approach: vanilla BART model, enhanced with visual prefix (VP), and two blueprint models which vary in the way plans are generated, i.e., in a top-down fashion or itera- tively. The third block contains GPT-3.5 models with (+BP) and without blueprints.
With regard to faithfulness, we observe that both blueprint models achieve scores higher than 40%, indicating effective translation of blueprints into

---

Repetition (4)
Grounding (
N-gram- based Metrics (1)
Model
MAUVE
Faithful (1)
Intra
Inter
Precis.
Recall
B-4
RLSum
METEOR
CIDER
KG-Story
1.03
88.72
4,55
3.46
3.86
9.8
27.3
32.3
7.9
PR-VIST
1.19
83.80
3,76
3.28
2.31
7.5
26.1
31.4
7.6
MCSM
2.85
77.48
5.12
5.89
11.01
8.1
27.7
31.4
7.6
VP-BART
0.22
83.70
4.31
3.23
11.31
26.6
31.0
BP (top-down)
0.08
81.51
5.17
11.56
8.32
44.73
9.9
28.5
33.6
7.2
BP fiterative)
0.29
72.70
5.22
3.59
28.25
51.66
7.0
26.1
30.3
5.5
RP (gold)
0.12
18.61
6.81
2.97
52.24
29.4
52.0
58.4
36.3
GPT-3.5
0.47
40.61
10.80
7.90
2.30
5.0
24.4
27.3
1.9
GPT-3.5 BP
1.52
31.19
14,70
10.30
2.10
34,56
4.2
23.3
25.1
2.3
Table 2: Automatic evaluation results. We report intra- and inter-story trigram Repetition (lower is better), precision and recall for concept grounding, MAUVE, Faithfulness, and a suite of commonly used metrics which rely on lexical similarity between system stories and references. Best results are highlighted in bold font.
stories. Notably, the iterative model performs best in terms of faithfulness, which suggests that trans- lating the entire global blueprint into story is more challenging, whereas breaking down planning into individual steps more effective. To get idea of the upper bound performance for blueprint mod- els, we ran the top-down model with silver standard blueprints extracted from the human- written stories (see row +BP (gold) in Table 2). As can be seen, the MAUVE score jumps to 52.24, edging closer to human-written written stories (their MAUVE score is 69.6). This further supports our hypothesis that our model successfully leverages the blueprints and retains the information captured in them.
asked to provide pairwise references along the fol- lowing dimensions: Relevance, Fluency, Coher- ence, Interestingness, and Overall. The full instruc- tions are given in Appendix A. We recruited nine native English speakers and elicited preferences for 100 stories (three judgments per story).
Our human evaluation results are summarized in Table 3. The iterative blueprint model outperforms PR VIST across metrics. Our participants perceive VP-BART stories as marginally more fluent and coherent compared to those created by the iterative model (even though they prefer iterative stories overall). This discrepancy is likely due the gen- eration process introduced by the iterative model which requires the decoder produce mix of questions, answers, and corresponding sentences, deviating from the traditional BART pretraining pattern. This added complexity might result in minor grammatical errors and pose challenges for coherence, given that story generation is broken down into separate steps instead of being contin- process. Nonetheless, the coherence scores are fairly close.
GPT-3.5 Struggles with Blueprints We also compared our approach to GPT-3.5, which we adapted to our task with in-context learning. A GPT-3.5 model enhanced with blueprints performs well at concept grounding. i.c., it generates stories which talk about what is depicted in the image. However, these stories neither human-like (see the very low MAUVE score) nor faithful to the intermediate blueprints (in fact they are 10% less faithful compared our iterative model). This sug- gests that GPT-3.5 tends to ignore the plan, despite being explicitly prompted with blueprints.
The blueprint model excels in terms of interest- ingness and grounding, indicating its effectiveness in creating engaging and memorable stories. Our model's superior grounding performance aligns with our hypothesis that blueprints serve not only as a planning strategy but also as a visual concept selector. This is due to the way blueprints are struc- tured (as answer-question pairs), which explicitly forces the model first identify salient visual con- cepts and then generate questions based on them.
6 Human Evaluation
We conducted a judgment elicitation study to fur- ther evaluate the stories generated by our mod- els. Specifically, we compared the best perform- ing blucprint model (iterative) and three other sys- tems: (a) PR-VIST, which represents the current planning-base state of the art; (b) VP-BART, our proposed model without blueprints; and (c) ground truth stories written by humans. Raters were shown an image sequence, alongside two stories and were
Figure shows example stories created by the models used in our human evaluation study and GPT-3.5+BP. The story generated by the iterative model is coherent, rich in detail, and fluent. VP-

---

Iterative
PR-VE VIST
Iterative vs. TP-BART
Iterative Human
Choices(%)
Win
Lose
Tie
Win
Lose
Tie
Win
Lose
Tie
Fluency 47.0
37.6
15.4
40.5
42.9
16.6
10.9
84.3
4.8
Coherence
56.0
24.8
19.2
41.2
41.6
17.2
15.2
75.7
9.1
Interestingness
70.5
19.2
10.3
55.5
38.1
6.4
23.0
70.0
7.0
Grounding 50.9
40.6
8.5
45.1
41.7
13.2
7.8
79.6
12.6
Overall
58.1
25.6
16.2
42.6
41.5
15.9
12.6
80.0
7.4
Table 3: Human evaluation results. Raters provide pairwise story preferences terms of fluency, coherence, interestingness, grounding, and overall. VP-BART is BART model enhanced with visual prefix (VP) but planning; Iterative is our best blueprint model and PR-V FR-VISTI is state-of-the-art visual storytelling model. We report the percentage times the Iterative model Wins, Loses differences between systems are statistically significant (p 0.05; using the Wilcoxon signed-rank test).
a Tie with comparison system. Unless underlined.
bought a cart to the market with some food. It looked great in the market. There were so many of their enthusiasm. They even had sale with their bumper. The market was sold out of fresh vegetables. the [location] car is very nice and beautiful. them. They were
PR-VIST:
VP-BART:
the market yesterday. There were many different kinds fruits there. bought very expensive, Afterward went back home.
Iterative: Today we decided to take small shopping trip the Market. The market had so many wonderful things to choose from that looked and bought many of them. There was also huge urray of various types of pepper. Finally the day had come
array of different fruits that I could hardly resist buying. There end and we piled in the taxi back heud home. GPT-3.5 BP: On Saturay momiing visited the local market. It had lot of fruits and vegetables heing sold. The atmosphere was friendly and buzzing with many vendors ready to help you. The fresh produce was the best thing about the market. You can always find friendly vendors that sell the most delicious produce. The key to finding the best bargains at the market is bargaining. The market is always great place shop for fresh fruits and vegetables.
Human: Shoppers arrived early the market. There was a variety goods sold. Some carried away large bundles, Every item was fresh and colorful. The spot was a favorite among produce shoppers.
Figure 4: Examples of system output and human-written story for an image sequence.
BART generates a grounded and accurate story without hallucination and semantic errors. How- ever, it is relatively plain narrative, offering lim- ited detail about the market the experience of the characters. Compared GPT-3.5+BP, the it- erative model's s story follows the image sequence more closely, mentioning details like an array of different fruits and various types of pepper, which significantly enhances storytelling.
ically, we apply a filtering step which removes non-grounded entities (and corresponding QA pair) from the blueprint before generating the story. We consider as non-grounded any blueprint entity which not included in the output of the concept detector (see Section 3.2).
Figure shows how this refinement approach can be used to adjust the model's output. In the first example, we observe that the story generated with a refined blueprint effectively avoids hallucinations (highlighted in blue) and is overall more faithful. However, it is important to note that imagination plays crucial role in crafting engaging story, pecially when the image sequence provides limited information. Therefore, employing the refinement method may result shorter and less detailed sto- ries, as illustrated in the second example. While the refined blueprint successfully eliminates all hal-
7 Controllable Generation
In this section we showcase how the blueprint plan allows to control the content and length of model output without additional mechanisms or training. For example, in cases where the generated story contains entities which do not appear in the image sequence, possible to refine the story gener- ation process, mitigating hallucinations. Specif-

---

Faithful Story Generation
Long Story Generation
JIFRY
Iterative: We took a trip to Bruzil. We went to the big City where we stay in, and took a lots of pictures. When we arrived, there were compa pompa loompa hompa, and the people were walking uround. We were at the altar. and it was standing. We ended the day getting hungry, decided to
We took cur out the street that day. Here looking the cur that the winner of the was in. The with the winner blue very close throughout the whole race. It neck and neck as the one blue held lead throughout lap and the winner pulled in for first place at the landing line. We were very close to winning the race and finishing in the final place line. It was very very, very close finish!
entire race. Finally, we crossed
little elegant restaurant.
Iterative Grounded-Ent: We were so excited to see where vacation was going. We arrived at the hotel and was quite busy. We couldn't believe how crowded it was getting. Some of the architecture was very cool. This store was our favorite since it was close the city center.
Last night, the band played an awesome concert. The guitarist had an incredible performance. The keyboard player was incredible as well. The guitarist was superb. He played his guitar very well. The next song he sang was very good. They had such a good performance, and I'm glad they went. The audience loved them so much. The night was over when the band wrote and put together their music. It was a fantastic way to finish the night.
Iterative: went my colleagues the other day. My boss was the one that invited me to uttend the meeting. He talked about how to improve the company looked and listened to my After the presentation, coworkers on how the company would be doing. My boss was pleused thut invited them to talk for hours. We talked for hours and we were very pleased when my boss asked. Iterative + Grounded-Ent: There was a question and answer session. Everyone was listening. The speaker was very interesting. Everyone was taking notes. The speaker talked very long.
the future.
Figure Stories generated within 10 iterations.
cration. Blucprint models leverage sequence of question-answer pairs as intermediate plans, en- abling better selection of salient concepts from the image sequence and guiding the construction of the final narrative. Specifically, we have showcased two model variants: top-down model which re- lies on a global plan, and an iterative model, which interleaves planning with sentence generation. Our experiments have shown that blueprint models ex- cel in concept grounding and their ability to create human-likk stories. Additionally, they are control- lable: Blueprints and their details can be refined (e.g., by emphasis- ing specific entities or characters), thus enabling human-in-the-loop and personalized storytelling. We showcase examples of controllability in Sec- tion 7. In the future, we would like to explore visual storytelling with grounded characters and entities, as well as tackle the generation of more complex narratives, such long-form stories.
Figure 5: Comparison of stories generated by original blueprint and refined blueprint models. Hallucinated words are highlighted in blue.
lucinated entities, the resulting story appears plain and lacks depth. Our blueprint method seems to strike the right balance between accurate and capti- vating story generation, prioritizing faithfulness to the image sequence and creativity in storytelling.
be made shorter or longer
Most visual storytelling systems generate 5-sentence stories, following the predefined story structure of the VIST dataset (Huang et al., 2016). Nevertheless, our iterative blueprint model can flex- ibly modulate the length of the story by controlling the number of iterative steps, thereby overcom- ing the conventional sentence limitation. Figure presents stories generated by this model with a max- imum of 10 iterations. Despite the increased length, the stories maintain coherence and are engaging.
Acknowledgments The authors gratefully ac- knowledge the support of the UK Engincering and Physical Sciences Research Council (grant EP/W002876/1). Liu was supported by the UKRI Centre for Doctoral Training in Natural Lan- guage Processing, funded by the UKRI (grant EP/S022481/I 1/11) and the University of Edinburgh.
8 Conclusion
In this work, we have introduced a novel approach to visual storytelling which integrates visual rep- resentations with pretrained language models and a blueprint-basee planning method for story gen-

---

Limitations
Hong Chen, Yifei Huang, Hiroya Takamura, and Hideki Nakayama. 2021. Commonsense knowledge aware concept selection for diverse and informative visual storytelling. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pages 999- 1008.
While our proposed model demonstrates effective story generation, it has certain limitations. Firstly, the grounding relation between the visual concepts and the corresponding text may not always be clear, leading to potential ambiguity in the generated sto- ries. Furthermore, the model can sometimes suffer from hallucinations due to falsely detected visual concepts.
Kordula De Kuthy, Madeeswaran Kannan, Hae- manth Santhi Ponnusamy, and Detmar Meurers. 2020. Towards automatically generating questions under discussion to link information and discourse structure. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5786-5798.
It is worth noting that our model was built on lop BART-base (Lewis et al., 2020). It would be ben- eficial to investigate the performance of larger mod- els, as they could potentially enhance the quality of the planning component and overall storytelling capability.
Daniel Deutsch, Tania Bedrax- Weiss, and Dan Roth. 2021. Towards question wnwwerrng as an automatic metric for evaluating the content quality of a sum- mary. Transactions of the Association for Computa- tional Linguisties, 9:774-789.
Ethics Statement
Daniel Deutsch and Dan Roth. 2023. Incorporating question answering- signals into abstractive summarization via salient span selection. In Pro- ceedings of the 1 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 575 5888.
Large Language Models This paper uses large pretrained language models, which have been shown to be subject a variety of biases, to occa- sionally generate toxic language, and to hallucinate content. Model output used for the human evalua- tion study (Section 6) was screened by the authors for harmful content.
Vladimir Dobrovolskii. 2021. Word-level coreference resolution. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing. pages 7670-7675, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Experimental Participants The departmental ethics panel judged our human evaluation study be exempt from ethical approval, all partici- pants were employces of the University of X, and as such were protected by employment law. Par- ticipants were paid at the standard hourly rate for tutors and demonstrators at the university.
Joshua Eisenberg and Mark Finlayson. 2017. A sim- pler and more generalizable story detector using verb and character features. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2708-2715, Copenhagen, Denmark. Association for Computational Linguis- tics.
Alexander Richard Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. Qafacteval: Im- proved qa-based factual consistency evaluation for summarization. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 2587-2601.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: visual language model for few-shot learning. Advances Neural Information Processing Systems, 15:23716-23736
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2650- 2660.
Chris Alherti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 01686617, Flo- rence, Italy. Association for Computational Linguis- tics.
Seraphina Goldfarb- -arrann,, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020a. Content plan- ning for neural story generation with aristotelian rescoring. arXiv preprint (:Xx::2009009877
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1877-
Seraphina Goldfarb-Tarren Tarrant,, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng 2020b. Content plan- ning for neural story generation with aristotelian rescoring. In Proceedings of the 2020 Conference on

---

Empirical Methods Natural Language Processing (EMNLP), pages 4339, Online. Association for Computational Linguistics.
Staffan Larsson. 2002. Issue-based dialogue manage- ment. Citeseer.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising gquuence-t--squeee
Diana Gonzalez-Rico and Gibran Fuentes-Pinede 2018. Contextualize, show and tell: A neural visual story- teller. arXiv preprint arXiv: 886600738
pre-training for nat- ural language generation, translation, and comprehen- sion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Danyang Liu, Juntao Li, Meng-Hsuan Yu, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. 2020. A character-centric neural model for auto- mated story generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1725-1732.
Xudong Hong, Rakshith Shetty, Asad Sayeed, Khush- boo Mehra, Vera Demberg, and Bernt Schiele. 2020. Diverse and relevant visual storytelling with scene graph embeddings. In Proceedings of the 24th Con- ference Computational Natural Language Learn- ing, pages 420-430.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023, Visual instruction tuning.
Chao-Chun Hsu, Zi- Yuan Chen, Chi- Yang Hsu, Chih- Chia Li, Tzu- Yuan Lin, Ting-Hao Huang, and Lun- Wei Ku. 2020. Knowledge-enriched visual story- telling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7952-7960.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.
Chi-Yang Hsu, Yun-Wei Chu, Vincent Chen, Kuan- Ting-Hao Huang, and Lun- Wei Ku. 2022. Learning to rank visual stories from human ranking data. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6365- 6378.
Chieh Lo, Chacha Chen, Ting Hao
Zhengyuan Liu and Nancy Chen. 2021. Controllable neural dialogue summarization with personal named entity planning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 92-106, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chi-yang Hsu, Yun-We Wei Chu, Ting-Hap Hao Huang, and Lun-Wei Ku. 2021. Plot and rework: Modeling story- lines for visual storytelling. In Findings of the Asso- ciation for Computational Linguistics: ACL-IJCNLP 2021, pages 4443 4453.
Cewu Lu. Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In European conference on com- puter vision, pages 852-869. Springer.
Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig. 2020. What makes a good story? designing composite rewards for visual story- telling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1969-7976..
Yatri Modi and Natalie Parde. 2019. The steep road to happily ever after: an analysis of current visual storytelling models. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 47-57.
Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. 2023. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.1404 14045.
Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0972
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267-2271 Minneapolis, Min- nesota. Association for Computational Linguistics.
Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Ba- tra, et al. 2016. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 1233-1239.
Proceedings of
Tachyeong Kim, Min-Oh Heo, Seonil Son, Kyoung- Wha Park, and Byoung- Tak Zhang. 2018. Glac net: Glocal attention cascading networks for multi- image cued story generation. arXiv preprint arXiv:1805. 10973.
Shashi Narayan, Joshua Maynez, Reinald Kim Am- playo, Kuzman Ganchey, Annie Louis, Fantine Huot, Dipanjan Das, and Mirella Lapata. 2022. Condi- tional generation with a question-answeringg blueprint. arXiv preprint 220:::2207000997

---

Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021. Planning with learned entity prompts for abstractive summarization. Transactions the Association for Computational Linguistics, 9: 144422
Ruize Wang, Zhongyu Wei, Piji Li. Qi Zhang, and Xuan- jing Huang. 2020. Storytelling from an image stream using scene graphs. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 34, pages 9185-9192.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun. Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Mcasuring the gap be- tween neural text and human text using divergence frontiers. In NeurIPS.
Xin Wang. Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018. No metrics are perfect: Adversarial reward learning for visual storytelling. arXiv preprint arXiv:1804.0910)
Chumpu Xu, Min Yang, Chengming Li, Ying Shen, Xi- ang Ao, and Ruifeng Xu. 2021. Imagine, reason and write: Visual storytelling with graph knowl- edge and relational reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 3022-3029,
Ratish Puduppully, Yao Fu, and Mirella Lapata. 2022. Data-to-text generation with variational sequential planning. Transactions of the Association for Com- putational Linguistics, 10:697-715.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555)
Jingjing Xu, Xuancheng Ren, Yi Zhang, Qi Zeng. Xi- aoyan Cai, and Xu Sun. 2018. A Akeleton-basee model for promoting coherence among sentences in narrative story generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4306-4315, Brussels, Belgium. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint arXiv: 03822.
Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.0677..
Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, and Jianfeng Gao. 2020. PlotMachines: Outline- conditioned generation with dynamic plot state track- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages 4274-4295, Online. Association for Computational Linguistics.
Pengeheng Yang, Fuli Luo, Peng Chen, Lei Li, Zhiyi Yin, Xiaodong Hc, and Xu Sun. 2019, Knowledge- able storyteller: A commonsense-driven generative model for visual storytelling. IJCAI, pages 5356- 5362.
Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and pragmatics, 5:6-1.
Lili Yao, Nanyun Peng, Ralph Weischedel. Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan- and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7378-7385.
Manuel Romero. 2021.
T5 (base) fine-tuned on squad for qg via ap.
https: /tttgin///uggnggfacc. wttaaaace.c//////aeee co/mrm8488/ ff-base-finetuned quueetttotteeeeeeetttioooonooo
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022. Lit: Zero-shot transfer with locked-inage text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 18123-18132
-eaeeeeeeee-tieeeeeterirrraaaa generation-ap
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826
Tom Trabasso, Paul Van den Broek, and So Young Suh. 1989. Logical necessity and transitivity causal relations in stories. Discourse processes, 12(1):1
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill, 2021. Mul- timodal few-shot learning with frozen language mod- els. Advances in Neural Information Processing Sys- tems, 34:200-212 212
Bileen Wang, Caren Han, and Josiah Poon. 2022. Ro- vist: Learning robust metrics for visual storytelling. Findings of the Association for Computational Linguistics: NAACL 2022, pages 2691-2702.

---

A GPT-335 Experimental Setting
wise judgments. The evaluation focused on the following dimensions of story quality: Relevance, Fluency, Coherence, Interestingness, and an Over- all judgment. We provide their definitions below.
We designed the following prompts for GPT-3.5:
Prompt (w/o Blueprint): want you to act as visual storyteller by creating story based given sequence of images. For each image, I'll pro- vide the key concepts. The concepts from different images will be sepurated by <SEP>. 'i Concepts: VI Story: ...,.., 'n Concepts: 'n Story:
Relevance Relevance captures whether the sen- tences in the stories relate to the input images. For each sentence, if the sentence accurately describes the content
specific image (i.c., not imagining something that does not exist in the image), it will be marked as relevance. Otherwise, if the sentence does not correspond to an image or is too vague, it will be not marked as relevance.
Prompt (w/ Blueprint): I'd like you to act as visual storyteller by creating story based on given sequence of images. For each image, I'll pro- vide the key concepts separated by <SEP>. Your task is to first generate a series of question-answer pairs for each image as part of the planning pro- cess, and then use these pairs to create the final story. 'I Concepts: ... vn Plan: 'nn Story:
Fluency Fluency evaluates the grammatical cor- rectness of the text. A story is fluent if it has few or no grammatical errors and easy to understand.
Coherence Coherence assesses whether the story makes sense. A coherent story flows well, the sen- tences are related, and logically connected, In con- trast, an incoherent story would be more less incomprehensible, without any logical connection between its sentences.
'v Concepts: ... 'v Plan: .... VI Story:
In-context learning examples in the prompts not shown for brevity. To make the best of in- context learning, we employed max-shot learning, while adhering to the token limit of 4,096.
B Human Evaluation Study
This evaluates whether the story contains unique, possibly unexpected elements. For example, memorable storyline. Below are examples of dull story and interesting story.
Interestingness
As mentioned in Section 6 we conducted a judg- ment elicitation study to further evaluate the stories generated by our models. Specifically, we pared the best performing blueprint model (itera- tive) and three other systems: (a) PR-VIST, which represents the current planning-based state of the art; (b) VP-BART, our proposed model without blueprints; and (c) ground truth stories written by humans. Raters were shown an image sequence, alongside two stories and were asked to provide pairwise preferences along various dimensions of story quality. We describe below our evaluation procedure and reproduce the instructions given to our raters.
Overall Taking all the aforementioned criteria into consideration, the annotators selected their preferred story for the given set of five images.
B.3 Annotation Interface
We designed an annotation interface using Python Flask. Figure 7 shows a screenshot of the interface.
B.1 Evaluation Procedure
We initially conducted a pilot study, based on which we devised our instructions. Subsequently. 100 im- age sequences were randomly selected from the test set, leading to a total of 300 pairwise comparisons. We employed the expertise of native speakers who uripleannooated the 300 pairwise comparisons. resulting in total of 900 judgments.
B.2 Experimental Instructions
Human raters were asked to compare and evaluate stories generated by different systems using pair-

---

Annotation Interface
Progress: 5/100
Story 1:
Story 2:
The students are getting ready for the graduation ceremany.
Here we are at the stadium where the students are sested.
The students are lining up to get their diplomas.
The students are getting their diploma and are starting to go into the flelcl
Mom very to see her son graduate.
finally
diploma.
picture got My
Mom with her after the cerernan..
Here graduate that proud him.
Here my proud accomplishment that her in law over.
Fluency:
Coherence:
Interestingness:
Overall:
Story
Story
Uncertain
Story
Story
Uncertain
Story
Story
Uncertain
Story
Story
Uncertain
Previous
Next
Figure Screenshot of the anmotation interface.


Reviews:
[{"review_id": "YHTPlWhRbD", "reviewer": "Reviewer_9ntA", "summary": "", "questions": "Is there any reason you did not include RoVIST-VG[1] metric in your results section?\nAlso, a short discussion about if and how the proposed architecture can deal with temporal misalignment for this task (e.g. when a concept appears in the visual feed in the third image but in the ground truth story in the first sentence).\n\nIn Table 3 (human evaluation) the model with planning and iterative generation is kind of close in performance to VP-BART (same model without planning). In table 2 the VP-BART + BP seems a lot better (e.g. MAUVE metric). Why do you believe this is the case? On metrics one seems much better, while for human evaluation they are very close.\n\n\n[1] Eileen Wang, Caren Han, and Josiah Poon. 2022. Rovist: Learning robust metrics for visual storytelling. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2691–2702.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: I believe a more thorough analysis of the answer extraction and question generation is needed. As the “ground truth” blueprints are obtained using a learned model (T5) it would be interesting to see how the quality of the question impacts the generated story.\n\nResponse: Our work uses \"silver standard\" blueprints. The latter are not manually created but automatically derived from gold standard stories. Specifically, an answer extractor and question generator. Adopting this approach allows us to scale our experiments more effectively. Employing gold standard blueprints would require manual annotation, which is both time-consuming and costly. A recent annotation study using the Question under Discussion theory (which inspired our blueprints) further highlights QUDs are subjective and  different readers often come up with distinct questions even with the same answer sentence (see https://aclanthology.org/2023.findings-acl.710)\n\n> Q2: there should be clear steps towards preventing data leaking when building blueprints, as they are built from stories themselves.\n\nResponse: Please see response to Reviewer 2.\n\n> Q3: Is there any reason you did not include RoVIST-VG metric in your results section?\n\nResponse: Thank you for bringing this up! We left out the RoVIST metrics because they are very similar to the metrics we used in our study. Specifically, the three components of RoVIST—Coherence, Non-redundancy, and Visual Grounding—match closely with our chosen metrics: the MAUVE score, intra/inter story repetition, and grounding precision/recall.\nTherefore, we focused on these more common metrics in the manuscript.\n\nHowever, it is straightforward to add these RoVIST results. Due to a suspected bug in the official Visual Grounding script, we couldn't generate results for that component. Thus, we present only the Coherence and Non-redundancy scores below:\n\n|model|non-redundancy|coherence|\n|-|-|-|\n|MCSM|0.90|0.67|\n|KE-Story|0.99|0.64|\n|Ours (iterative)|0.93|0.72|\n\n> Q4: Also, a short discussion about if and how the proposed architecture can deal with temporal misalignment for this task (e.g. when a concept appears in the visual feed in the third image but in the ground truth story in the first sentence).\n\nResponse: Thank you for bringing this up! If a visual concept in the third image gets mentioned in the story's first sentence, it suggests that the human annotator felt this was a good way to make the story flow better, even if that concept wasn't present in the first image. They might prioritize narrative flow over strict temporal order, human storytelling often relies on imaginative sequencing.\n\nAs the blueprints offer a way to control the generation process (see Appendix A.1, A,2 for examples of how the output can be controlled), it is possible to eliminate misalignment, e.g., by filtering out QA pairs whose entities are not grounded in the corresponding image. This would be implemented using a pretrained language-vision model like GLIP (​​https://github.com/microsoft/GLIP). \n\n> Q5: In Table 3 (human evaluation) the model with planning and iterative generation is kind of close in performance to VP-BART (same model without planning). In table 2 the VP-BART + BP seems a lot better (e.g. MAUVE metric). Why do you believe this is the case? On metrics one seems much better, while for human evaluation they are very close.\n\nResponse: In the human evaluation (Table 3), the scores for fluency and coherence are indeed close between the two models. However, for attributes like interestingness and grounding, the model with blueprints stands out as noticeably better than VP-BART without planning. These attributes, which play a key role in assessing story quality, demonstrate that the model with blueprints performs significantly better than the VP-BART without planning.\n\nThis aligns with the findings in Table 2. The VP-BART model scores higher on lexical matching metrics, reflecting its strength in fluency and coherence. Yet, its lower MAUVE and grounding scores show that the stories created using the blueprint model feel more real, interesting, and well-anchored in the visual context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work introduces planning (in the form of question-answer pairs) for the task of visual storytelling. The idea is well explained, and it works! The contributions and the setup are clear, the results are good and well discussed.", "reasons_to_reject": "I believe a more thorough analysis of the answer extraction and question generation is needed. As the “ground truth” blueprints are obtained using a learned model (T5) it would be interesting to see how the quality of the question impacts the generated story. \n\nAlso there should be clear steps towards preventing data leaking when building blueprints, as they are built from stories themselves. ", "questions_to_authors": "Is there any reason you did not include RoVIST-VG[1] metric in your results section?\nAlso, a short discussion about if and how the proposed architecture can deal with temporal misalignment for this task (e.g. when a concept appears in the visual feed in the third image but in the ground truth story in the first sentence).\n\nIn Table 3 (human evaluation) the model with planning and iterative generation is kind of close in performance to VP-BART (same model without planning). In table 2 the VP-BART + BP seems a lot better (e.g. MAUVE metric). Why do you believe this is the case? On metrics one seems much better, while for human evaluation they are very close.\n\n\n[1] Eileen Wang, Caren Han, and Josiah Poon. 2022. Rovist: Learning robust metrics for visual storytelling. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2691–2702.\n", "paper_topic_and_main_contributions": "This work introduces a new approach for generating stories from a set of images, task also know as visual storytelling. The authors propose the usage of planning in the form of (question, answer) pairs to guide the story generation and generate more natural and grounded stories. \n\nBriefly, the visual inputs are processed by a frozen image encoder and concept embedder. Output of those is further passed through feed-forward layers and an embedding layer respectively, and finally concatenated to build the input for a sequence-to-sequence BART model. Next, the pre-trained BART model generates the blueprint (question, answer pairs) and the story. This is either done in one go (top-down approach) or iteratively (as questions and story sentences are generated at each step and become input for the next steps). \n\nQuestion-answer pairs are generated automatically using pre-trained SOTA question generation models. Answers, that are given as input together with the ground truth story are automatically extracted from the story as the noun and verb phrases of each sentence.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-\tL939: Table 6 -> Figure 6\n-\tIn my opinion, results of VP-BART with “gt” blueprints should be in a separate table. It makes Table 2 hard to follow. \n-\tPlease provide more details (maybe a footnote) about the concept detector used. Is this model used? https://clarifai.com/clarifai/main/models/general-image-detection?tab=overview . \n"}}, {"review_id": "NTJ61Udfwh", "reviewer": "Reviewer_1NBh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The main concern for me on this paper is the way they augment the VIST dataset with language models to generate QA pairs as the blueprint plan. In section 3.1, the blueprints B_i are generated by the stories S_i. And then the task is to generate stories S_i based on images I_i and blueprints B_i. This makes the proposed task to have potential data leaking. Say, to the extreme, if the blueprint generator is an identical function, the blueprints will just be the stories. We can also see in Figure 3. that the blueprints and stories have a large number of N-gram overlap. This makes me wonder if the method is ill-defined.\n\nResponse: Thank you for your comments! The blueprints are only seen during training, they are **predicted** during inference, please see more explanation below:\n\n**Fine-tuning Phase**: During fine-tuning, our model takes the image sequence as input and is trained to generate the blueprint **and** the story. The loss is computed based on a comparison between the predicted blueprint+story and the corresponding gold standard blueprint+story.  Fine-tuning is performed on **the training split** of the VIST dataset. The model has no access to the test data during this phase. \n\n**Inference Phase**: During inference the model only sees the image sequence. It does not have access to any gold standard information, For a more detailed discussion please see the Introduction and Section 3.2 of our paper. Figure 3, shows how our model words during inference. The blueprints presented here are **model-generated** and constitute a **plan** of how the story should be realized. \n\n**N-gram Overlap**: The N-gram overlap between blueprints and stories is intentional. The blueprint’s QA pairs are designed to reflect salient content (rendered in natural language) and derived from the stories. This alignment simplifies the generation process and embodies the essence of our planning-based approach.\n\nWe hope that this clarification adequately addresses your concerns. \n\n[Note that our gold-standard blueprints are technically silver-standard, see response to reviewer 1. Here we used a simplified terminology to avoid further confusion.]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author brings a new iterative, plan-based method into visual storytelling.\n2. The results on both automatic and human evaluations are very good.", "reasons_to_reject": "The main concern for me on this paper is the way they augment the VIST dataset with language models to generate QA pairs as the blueprint plan. In section 3.1, the blueprints B_i are generated by the stories S_i. And then the task is to generate stories S_i based on images I_i and blueprints B_i. This makes the proposed task to have potential data leaking. Say, to the extreme, if the blueprint generator is an identical function, the blueprints will just be the stories. We can also see in Figure 3. that the blueprints and stories have a large number of N-gram overlap. This makes me wonder if the method is ill-defined.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces planning into visual storytelling. In their case, the plans are a list of QA pairs, which are called blueprints. The method first translates images into visual prefixes based on an image encoder, then iteratively feed the visual prefixes along with the \"blueprint\" to iteratively generate stories. The authors conduct automatic and human evaluation on the VIST benchmark and show their method achieves good results.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c6vJ1AcQUl", "reviewer": "Reviewer_B4N7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The true impact of the generated blueprints seems questionable. In Table 2, the performance gap between VP-BART (w/o blueprints) and the proposed method (w/ generated blueprints) is much smaller than the performance gap between the proposed method and the model with gold blueprints, described as the upper bound performance. It indicates that the generated (i.e., silver) blueprints are noisy. Some techniques are required to reduce the noise in the generated blueprints.\n\nResponse: Thank you for your feedback! \n\nYou are referring to the difference between VP-BART (w/o blueprints) and the proposed method (w/ generated blueprints)  when using n-gram based metrics. Table 2 reports additional metrics which reveal bigger differences between the two models. Using iterative blueprints results in an improvement of 16.94 MAUVE points over VP-BART. Blueprint models are also considerably better at grounding and reducing intra- and inter-sentence repetition. The iterative blueprint model obtains a faithfulness score of 51.66% which suggests that more than half of the QA pairs are effectively incorporated into the final story. All models fall short of the upper bound. We should also note that automatic evaluation results are complemented with human evaluation (see Table 3, third block) which confirm that (iterative) blueprint stories are perceived as overall significantly better than VP-BART.\n\nUsing gold blueprints yields a superior N-gram score. This is largely due to the overlapping N-grams in the gold blueprints. However, as discussed in Section 4.4, lexical matching metrics like these might not always align with human judgments and may not be the most representative metric for story quality.\n\nOur blueprint method allows for tailored adjustments and controllable generation. In Appendix A.1, we show how blueprint quality can be improved removing hallucinated QA pairs, further refining the story generation process.\n\n> Q2: The current manuscript does NOT compare the proposed method with some strong and pioneering studies [1,2] in the experiments. In my current understanding, the authors exclude them because the N-gram-based performance of the proposed method lags behind that of those studies. It would be much more convincing if the authors compare the proposed method with them. The source code of both studies is open-sourced.\n\nResponse: Our decision to exclude these studies [1,2] was primarily due to the page limit, and not an oversight. We prioritized presenting comparisons with the latest state-of-the-art models, such as PR-VIST and KG-Story. It's worth noting that both PR-VIST and KG-Story have been benchmarked against AREL and/or GLAC in the literature, and found to outperform them.\n\nHowever, in light of your comments and the importance of a comprehensive comparison, we will compare our method with these studies [1,2]. Here are the updated results:\n\n|Model           |Rep-Intra|Rep-Inter|Ground-P.|Ground-R.|MAUVE |B4  |RLSum|\n|----------------|---------|---------|---------|---------|------|----|-----|\n|GLAC Net        |2.94     |84.72    |3.5      |2.1      |1.90  |10.7|33.7 |\n|AREL            |11.96    |84.91    |4.2      |3.3      |2.32  |**14.4**|**35.4**|\n|Ours (top-down) |**0.08**     |81.51    |5.17     |**11.56**    |8.32  |9.9 |33.6 |\n|Ours (iterative)|0.29     |**72.70**    |**5.22**     |3.59     |**28.25** |7.0 |30.3 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes the plan-based approach for visual storytelling, which has not been considered in existing studies.\n\n- The authors perform human evaluation to identify the quality of the generated visual stories.\n\n- Overall, the paper is well-written and easy to follow.\n", "reasons_to_reject": "- The true impact of the generated blueprints seems questionable. In Table 2, the performance gap between VP-BART (w/o blueprints) and the proposed method (w/ generated blueprints) is much smaller than the performance gap between the proposed method and the model with gold blueprints, described as the upper bound performance. It indicates that the generated (i.e., silver) blueprints are noisy. Some techniques are required to reduce the noise in the generated blueprints.\n\n- The current manuscript does NOT compare the proposed method with some strong and pioneering studies [1,2] in the experiments. In my current understanding, the authors exclude them because the N-gram-based performance of the proposed method lags behind that of those studies. It would be much more convincing if the authors compare the proposed method with them. The source code of both studies is open-sourced.  \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a method that leverages the blueprint (i.e., a set of question-answer pairs regarding the given image) to generate coherent and interesting narratives for the task of visual storytelling. \n\nWith a carefully designed process, the authors automatically collect the blueprint annotation given images and corresponding textual stories. The pre-trained language model (PLM) then generates the stories given visual prefixes and the blueprints. The parameters of the PLM remain frozen, and some parameters in the visual prefix construction are updated. This paper introduces two variants to generate visual stories: top-down planning and iterative planning.       \n\nExperimental results show that the proposed method outperforms some existing studies along with the ablative model. Furthermore, the authors perform human evaluation to demonstrate the effectiveness of the generated blueprints.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Adversarial Reward Learning for Visual Storytelling. Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. In Proceedings of ACL, 2018.\n\n[2] GLAC Net: GLocal Attention Cascading Networks for multi-image cued story generation. Taehyeong Kim, Min-Oh Heo, Seonil Son, Kyoung- Wha Park, and Byoung-Tak Zhang. In Proceedings of NAACL StoryNLP Workshop, 2018.", "Typos_Grammar_Style_and_Presentation_Improvements": "- L179: the index of the image should start from 1 if $k$ images are given\n\n- L256: the index of the image should start from 1\n\n- L264: the index of the concepts should start from 1 because the authors said that there are $K$ concepts.\n\n- L316-317: the index should also start from 1\n\n- L324-325: the notation that describes a set of elements from index 0 to index i is commonly represented as “0:i” rather than “0,i” \n"}}]

Author rebuttal:

Meta-review: This paper introduces the use of planning in the form of question-answer pairs to guide visual story generation for more natural and grounded stories. On one hand, reviewers found that (1) the idea is interesting and well explained. The setup is clear and results on both automatic and human evaluations are good and well discussed. On the other hand, two reviewers commented that the impact of generating blueprints seems quite marginal as seen in Tables 2 and 3 (especially for fluency and coherence metrics). The authors did not agree on this, and argue that the model achieves better performance in terms of interestingness and grounding than VP-BART.
Decision: Accept-Findings