DiffuVST: Narrating Fictional Scenes with Global-History-Guided Denoising Models
Authors: Shengguang Wu
Mei Yuan
Qi Su
Venue: EMNLP 2023 Findings (2023)
Keywords: visual storytelling
diffusion language models
global history guidance

Abstract: Recent advances in image and video creation, especially AI-based image synthesis, have led to the production of numerous visual scenes that exhibit a high level of abstractness and diversity. Consequently, Visual Storytelling (VST), a task that involves generating meaningful and coherent narratives from a collection of images, has become even more challenging and is increasingly desired beyond real-world imagery. While existing VST techniques, which typically use autoregressive decoders, have made significant progress, they suffer from low inference speed and are not well-suited for synthetic scenes. To this end, we propose a novel diffusion-based system DiffuVST, which models the generation of a series of visual descriptions as a single conditional denoising process. The stochastic and non-autoregressive nature of DiffuVST at inference time allows it to generate highly diverse narratives more efficiently. In addition, DiffuVST features a unique design with bi-directional text history guidance and multimodal adapter modules, which effectively improve inter-sentence coherence and image-to-text fidelity. Extensive experiments on the story generation task covering four fictional visual-story datasets demonstrate the superiority of DiffuVST over traditional autoregressive models in terms of both text quality and inference speed.

DIFFUVST: Narrating Fictional Scenes with Global-History-Guudee Denoising Models
Shengguang Wu, Mei Yuan, Qi Su"
Peking University
edu. cn, sukia@pku. edu.. cn yuanheegguunngg.aumiiiiieee eku.edu.ccm
Abstract
capturing the semantic links across the given quence of images as a whole.
Recent advances in image and video creation, especially AI- bl-da image synthesis, have led to the production of numerous visual scenes that exhibit high level abstractness and diversity. Consequently, Visual Storytelling (VST), a task that involves generating meaning- ful coherent narratives from a collection of challenging and
Hastig good time bunding and [M] gut edurnated by the
Sky illaminated betiliance
images, has become is increasingly desired beyond real-world im- agery. While existing VST techniques, which typically autoregressive decoders, have made significant progress, they suffer from low inference speed and are not well-suited for sym- thetic scenes. To this end, we propose novel diffusion-based system DIFFU VST, which mod- els the generation of series of visual descrip- tions as a single conditional denoising process. The stochastic and von-auvoregressive nature of ST inference time allows it to gener- ate highly diverse narratives more efficiently. In addition, DIFFUVST features unique design with bi-directional text history guidance and multimodal adapter modules, which effectively improve inter-sentence coherence and image- to-text fidelity. Extensive experiments on the story generation task covering four fictional visual-story datasets demonstrate the superior- ity of DIT over traditional autoregres- sive models in terms of both text quality and inference speed.
Jelf wan waiching TV in his coom. haard molse sou Jareil watching
Jelf west in the living reee janed to velumm. joined Javol weldd the gamne.
Mounshille Alle,, Jared's wite dauggter Candl of Cars.
Figure I: Examples from the VIST (Huang et al., 2016) and AESOP (Ravi et al., 2021) dataset.
So far, prior works on visual storytelling (Wang et al., 2018; Kim al., 2018; Malakan et al., 2022) typically utilize autoregressive design that in- volves an extra sequential model built on top of basic image captioners to capture the relationship among images. More claborate frameworks with multi-stage generation pipelines have also been proposed that guide storytelling via egg. storyline- planning (Yao et al., 2018), external knowledge engagement (Yang et al., 2019; Hsu et al., 2019, 2020), and concept selection (Chen al., 2021). Effective these existing methods are for describ- ing real-world photo streams (e.g. the upper row in Figure 1), albeit with their increasingly com- plex structural designs, their ability to generate stories from fictional scenes (e.g. the lower row in Figure 1) remains unverified. In addition, be- cause these models are typically trained to infer in autoregressive way, they produce captions to- ken by token while conditioning on only previous (unidirectional) history, which prohibits these chitectures from refining prefix of sentences based on later generated tokens. As a result, existing au- toregressive modely are restricted to generating less
1 Introduction
Visual Storytelling (VST) is the challenging task of generating series of meaningful and coherent sentences to narrate a set of images. Compared to image and video captioning (Vinyals et al., 2014; Luo et al., 2020; Wang et al., 2022; Lei et al., 2021), VST extends the requirements of image-to-text gen- eration beyond plain descriptions of images in iso- lation. A VST system expected to accurately portray the visual content each image while also
"Corresponding author.

---

informative narratives at a low inference speed.
(2) We propose global-history guidance with adapted multimodal features in DIFFUVST that enhance the coherence and visual-fidelity of the generated stories.
To tackle the above issues of VST especiillyy given the increasing demand for narrating fictional visual scenes - we propose a novel visual sto- rytelling framework DIFFUVST (Diffuusin for Visual StoryTelling) based on diffusion models uti- lizing continuous latent embeddings, which have demonstrated promising capabilities for image syn- thesis (Rombach ct al., 2021; Pan et al., 2022) and language modelling (Hoogeboom al., 2021; Austin et al., 2021; Li al., 2022b). Our model generates a set of visual descriptions simultane- ously in a non-autooeerressvve manner by jointly denoising multiple random vectors into meaningful word embeddings conditioned on the image fea- tures of all panels (i.e., all image-text pairs in a visual-story sequence). Specifically, SIFFUVST leverages transformer encoder learn restore the word embeddings of the ground-truth story texts from sequences of Gaussian vectors. To en- hance the risual-truthfunness of the generated texts, DIFFUVST proposes multimodal feature extractor and adapter modules with pretrained backbone that provide in-domain story-image features as condi- tion and story-text features classifier-free guid- ance (Ho and Salimans, 2022). Since these features represent all panels across the entire picture stream, they serve the "global history" of all the preced- ing and following frames and texts, which connects the captioning of the current image to other context panels more closely, thus improving the coherence of the whole visual narrative.
(3) We demonstrate the effectiveness and prac- tical value of our system in face of the surging demand for narrating synthetic scenes by conduct- ing extensive experiments four fictional visual- story datasets. DIFFU VST outperforms strong au- toregressive models terms of performance while achieving significantly faster inference speeds.
2
Related work
2.1 Visual storytelling
Visual storytelling (VST) aims to produce set of expressive and coherent sentences depict an image stream. Existing work in this area can be broadly divided into two groups of approaches: end-to-end frameworks and multi-stage systems. In the end-to-end pipeline, models are developed to autoregressively generate multi-sentence stories given the image stream in unified structure (Wang et al., 2018; Kim al., 2018). Meanwhile, multi- stage approaches that introduce more planning external knowledge have also shown impressive performance (Yao et al., 2018; Hsu et al., 2020; Chen et al., 2021). Further, some other works are devoted to adopting more elaborate learning paradigms to improve the informativeness and con- trollability of story generation (Yang et al., 2019; Hu et al., 2019; Jung et al., 2020).
We validate the effectiveness of DIFFUVST by performing the visual story generation task on four visual-story datasets with non-real- woold imagery: AESOP (Ravi et al., 2021), FlintstonesSV (Maha- rana and Bansal, 2021), PororoSV (Li al., 2018),
2.2 Diffusion models
Diffusion models are a family of probabilis- tic generative models that first progressively de- struct data by injecting noise, and then learn to restore the original features through incre- mental denoising. Current studies are mostly based on three formulations: Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., 2020; Nichol and Dhariwal, 2021a), Score-based Gen- erative Models (SGMs) (Song and Ermon, 2019, 2020), and Stochastic Differential Equations (Score SDEs) (Song et al., 2021, 2020), which have shown outstanding performance in image synthesis. Re- cently, diffusion models have also been employed to address text generation tasks. Either discrete diffusion method for text data (Austin et al., 2021; He al., 2022; Chen al., 2022) designed, or the word embedding space is leveraged for contin- uous diffusion (Li et al., 2022b; Gong et al., 2022).
and DiDeMoSV (Maharana et al., 2022) with the latter three converted from Story Visualization (SV) to fictional VST datasets. Quantitative results show that DIFFUVST outperforms strong autoregressive baselines across all four datasets. In addition, DIr- FUVST manages to reduce the inference time by large factor thanks to its non-autoregresssve nature.
Our contributions are summarized as follows:
(1) We model the visual narration of set of images as one single denoising process and propose DTFFUSST, a diffusion-bace storytelling. To our best knowledge, this work is the first to leverage diffusion models and adopt a non autorrrrrreee approach in visual storytelling.
method for visual

---

3 Background
trained together with the conditional diffusion model po(z]e) parameterized through cœ(2,, c). The conditioning signal in the unconditional model is discarded by randomly setting the class iden- tifier e to a null token Ø with some probability Puncond which is a major hyperparameter of this guidance mechanism. The sampling could then be formulated using the linear combination of the con- ditional and unconditional score estimates, where w controls the strength or weight of the guidance:
3.1 Diffusion in continuous domain
A diffusion model involves a forward process and a reverse process. The forward process constitutes Markov chain of latent variables 21,...; 2T by incrementally adding noise to sample data:
ax -N / B, Bel)
(I)
where Be E (0, denotes the scaling of variance of time step and Ir tends to approximate a stan- dard Gaussian Noise N (my; 0, I) if the time step T is large enough. In the reverse process. model parameterised by pe (typically a U-Net (Ron- neberger et al., 2015) or Transformer (Vaswani et al., 2017)) is trained to gradually reconstruct the previous state 3y from the noised samples at time step t, where p0( t) is model's prediction on the mean of - conditioned on Zt:
Eo(2x, c) (1 ()) c) wee(2x) (5)
DIFFUVST
Figure 2 presents the overall architecture and train- ing flow of our model DIFFUVST, which consists of two pretrained multimodal encoders (one for im- age and one for text) with their separate adapter layers, a feature-fusion module and transformer- encoder as the denoising learner.
p0( 1 N N 15/0 t), Σ (ate t))
4.1 Multimodal encoding with adapters
(2)
To obtain the cross-modal features of each image- text panel, we employ a multimodal backbone (in this work: BLIP (Li et al., 2022a)), which serves to provide a solid initial representation of the im- age stream and its paired text sequences. However, since BLIPP is pretrained on large-scale internet data which contains primarily real-world photos, the tracted image- and text- features only represent the most general visual and textual knowledge. Hence, we further propose an adapter module for transfer- ring the BLIP-encoded multimodal features to the specific domain of fictional scenes and texts. Moti- vated by the adapter design in parameter-efficient transfer learning (Houlsby et al., 2019), we imple- ment the feature-adapters additional linear layers following the respective last layer of the visual and textual encoders. This mechanism allows the simul- taneous exploitation of both the information stored in the pretrained BLIP checkpoint and the freshly learned knowledge from our training data, which leads to the final feature vectors (F, and F,) that are well adapted for fictional image-text domains.
The original training objective is minimize the negative log- likelihood when generating 20 from the Gaussian noise ør produced by (1):
E,I log(po(zo)))
E / (((
(3)
Furthermore, a simpler loss function can be ob-
tained following DDPM (Ho et al., 2020):
(00 Ma,, zo)l
(4)
where fi(2s, mo) represents the mean of posterior g(12=111(,20).
3.2 Classifier-free guidance
Classifier-free guidance (Ho and Salimans, 2022) serves to trade off mode coverage and sample delity in training conditional diffusion models. As an alternative method to classifier guidance (Dhari- wal and Nichol, 2021) which incorporates gradi- ents of image classifiers into the score estimate of diffusion model, the classifier-free guidance mech- anism jointly trains conditional and uncondi- tional diffusion model, and combines the resulting conditional and unconditional score estimates. Specifically, an unconditional model po pa- rameterized through a score estimator co (2x) is
4.2 Feature-fusion for global history condition and guidance
In line with the formulation in Diffusion-LM (Li et al., 2022b), our model DIFFUVST performs vi- sual storytelling as a conditional denoising task in continuous domain. To condition this denoising process (i.e., the generation of story texts) on the

---

DENOISE/PREDICT
X
X
Poby Loopy Pororo and Crong are clapping
ADD NOISE
Transformer Blocks
Eddy is standing in tront of etght glasane contataing different colors fluid. Pamora appreaches eddy from the right widr.
the glassex Eddy.
Poby talks
eycs.
Loopy standing Poby.
Eddy patting right
EM BED
Textual Takens
Textual Feature
Visual
Visual Tokens
Poby Loopy Ponuro and Crong are clapping.
Haby Losiee clquine
Feature
L night .ite
Eddy slanding freat the eight glasses containing diffecent colons of fiuld, Forero approsches eddy from right side.
Adapter
Adapter
Piroro loobs...ll to. kids
Pday sike... beace siby
fddy hichee .Ns. misu
Text Encoder
Image Encoder
Pororu louks
glasses
Eddy.
epes. Loopy
standing Poby.
by
patting right
Figure 2: Overview of DIFFU VST: First, two pretrained encoders extract the image- and text features of all panels of the story. which are then transferred to the current dataset domain via trainable adapter modules. Next, these feature segments are fused into corrupted word embeddings Ty to provide global history condition (image) and classifier-free guidance (lext) across the entire picture stream. Finally, transformer learns restore the word embeddings of the story (Io) from the fused vector input, which can be grounded back to the actual text.
corresponding visual scenes, we utilize the adapted
accordance with Puncond in the original classifier- free guidance proposal (Ho and Salimans, 2022), Punguide regulates the percentage of training with- out extra textual condition that will be unavailable during inference.
F? visual features of all images F? F?:... FN in each story sample as the condition signal, where N is the number of images in a single story sample. For simplicity, we further denote the features of all image panels [F] F? F.,.....NN. and text panels [F], F.? ...... F.N] in visual story sample as F and F7. Specifically, the corrupted input embeddings 2T to get [æ7; F.). Since these feature segments contain the visual in- formation of all panels in the storyline, they provide global visual history across the entire image stream and thus serve as a more comprehensive condition for coherent storytelling.
Formally, based on the classifier-fiee freef guidance mechanism in (5), our DIFFUVST model samples with the feature guidance of global textual history. as expressed in:
concatenate them with
[[[TT:F:: (1+ w)) ; wee(lors Fel)
(6)
where w controls the guidance strength of the global textual history. We further discuss the effects of the guidance strength w and unguided probabil- ity Punguide in detail in Section 5.4.
Furthermore, we adapt the classifier-free guid- ance mechanism (Section 3.2) to include a certain degree of ground-truth textual information during training. We do this by fusing the textual features of cach panel of the ground-truth story into the above mixed vectors as [2T; F F?), which serves the final input to our denoising model. However, we mask out the textual feature segment Ft with a probability of Punguide (i.c., such textual guid- ance is provided with a chance of 1 Punguide) In
Just like the global visual features, these addi- tional signals from the target texts also represent the global history of the whole set of panels in story sample but from textual angle. Hence, such guidance contributes to the bidirectional at- tention for text generation and improves the overall fluency and coherence of the generated story.

---

4.3 Transformer--ncodeer for denoising Our model adopts a BERT-like transformer- encoder architecture to perform the reconstruction of the ground-truth -tunh word embeddings 20 (of all the consecutive texts) from random noise zy- This noisy input is enriched with the global visual- and textual-history features (Ir;; F Ft) as described in the above fusion module (Section 4.2). We add an additional segment embedding layer to distin- guish the adapted Bl.ttrfetures from the corrupted sentence embedding zT.
relative weight of L' and Lie is equal (i.e., ta 1). This encourages both loss items in (7) decrease at a relatively same rate.
The predicted embedding 20 is then projected through a language model head to produce all pan- els of text sequences in parallel that finally form a narrative together.
5 Experiments
5.1 Experimental setup
Datasets:
We use four visual-story datasets fea- turing synthetic style and content as our testbed: AESOP (Ravi et al., 2021), PororoSV (Li et al., 2018), FLintstonesSV (Maharana and Bansal, 2021) and DiDeMoSV (Maharana et 2022). It is worth noting that while AESOP was originally designed as an image-to-text dataset, the other three datasets were initially created for story visualization (SV) tasks. However, in our work, we employ them in a reverse manner as visual storytelling datasets due to their fictional art style, which aligns well with the objective of narrating non-real-world imagery. For more detailed statistics regarding these datasets, please refer to Appendix A.1.
Our model is trained to directly predict the ground-truth word embeddings from any dif- fusion timestep 1 sampled from large T in the same spirit with Improved DDPM (Nichol and Dhariwal, 2021b) and Diffusion-LM (Li et al., 2022b). Instead of using a large T, we sample a subset of size T' from the total T timesteps to serve as the actual set diffusion time steps: S = Sr < 8 E (0, 7j. This greatly accelerates the reverse process, which then requires significantly fewer denoising steps, as evidenced in Section 5.3. We also add æ1 restoring loss term [ 1) f((11,200 =o)il as absolute error loss (LI-loss) to the L1-version of the basic con- tinuous diffusion loss I. defined in (4) in order to regulate the performance of model on restor- ing do from a slightly noised x1 and to improve the model's stability of prediction. This gives L' as the natural combination of the Ij restor- ing loss and the L1-version of L. Furthermore, we add rounding term LR parameterized by Figl (ogo(1))W))))
Evaluation metries: We evaluate the quality of generated stories using quantitative NLG metrics: BLEU, ROUGE-L, METEOR and CIDEr. We also compare the inference speed of different methods with the average generation time of each test sam- ple using the same hardware and environment. Baselines: Considering the accessibility and repro- ducibility of existing methods on our four fictional visual storytelling datasets, we compare the perfor- mance of our diffusion based model to three widely recognized open- soen-source autoregressive baselines: GLAC-Net (Kim et al., 2018), TAPM (Yu et al., 2021), BLIP-EncDec (Li et al., 2022a), which are retrained on our four fictional visual storytelling datasets respectively:
F log IT'= P(W, [+a]] for rounding the predicted æo embeddings back discrete tokens, where W is the ground truth sentence and the generated sequence length. represents the predicted word embeddings.
Our final loss function is the sum of the restoring no-embedding loss I.' and the rounding loss Li- with rounding term coefficient \ controlling the relative importance of L' and Lx:
GLAC-Net (Kim et al., 2018) employs a LsTM-encodder-eeooderr structure with two- level attention ("global-local" attention) and an extra sequential mechanism cascade the hidden states of the previous sentence the next sentence serially.
Iifinal= = 1' + ALR
Σ Fggzz (lloo ((r,,) (7) ICS
+ +1Iuo(21,11 1) ((21,
TAPM (Yu al., 2021) uses pretrained vi- sual encoder and language generator aligned with Adaptation Loss and finetuned with Se- quential Coherence Loss. For our task, we utilize a pretrained VIT (Dosovitskiy et al.,
-Alog po (w))
In this work, we set the rounding term coefficient (A) as a dynamic À strategy where À is updated after every gradient descent step to ensure that the

---

Dataset
Method
B@1
B@4
Inference Time (sec.)
Denoising Step
F.
GLACNet
44.55
11.98
37.48
29.23
8.93
1,2042
BIee-FncDec
48.75
16.32
42.25
35.3
15.94
2,0486
TAPM
49.05
15.08
42.12
33.03
16.54
1.8923
DIFFUVST ueguided
48.75
16.32
42.25
35.30
15.94
0,0873
DIFFUVST guided
49.76
17.01
42.98
36.22
17.63
0,0981
P.
GLACNet
23.66
2.1
21.19
17.67
2.8
0.7844
DLic-EncDec
31.38
2.96
23.13
21.64
5.63
1,8702
TAPM
28.43
3.78
24.43
21.87
5.55
1.2289
DIFFUVST unguided
30.12
2.94
24.81
21.64
5.32
0.1084
10
DIFFUVST guided
32.09
2.81
25.35
22.03
6,18
0.1196
11
D.
GLACNet
20.28
1.78
16,36
15.15
7.15
0,3030
BLIP-Ene
20.02
2.14
16.01
17.74
14.7
0,7474
TAPM
20.79
2.12
16.35
17.68
15.26
0.4457
DIFFUVST unguided
20.90
1.88
16.41
18.77
12.51
0,0059
ST guided
22.08
2.74
17.33
19.75
15.16
0.0161
GLACNet
20.37
1.02
19.46
2.14
0,9285
BLIP-EncDec
19.16
1.81
21.88
16.71
2,89
1,3141
TAPM
21.95
1.56
21.55
16.3
2.97
1.1629
DIFFUVST unguidddd
9.20
0.66
15.05
13.9
0.83
0,1982
VST guided
23.13
1.41
21.14
19.07
3.4
0.1988
Table I: NLG evaluation results as well as the average inference time of our proposed model DIFFU and the autoregressive baselines (GLACNet (Kim et al., 2018), BLIP-EncDec (Li et al., 2022a). TAPM (Yu al., 2021)) four fictional visual-story datasets in story generation ("Writer" mode) tasks. The suffix unguided refers to guidance strength w = O, where our model receives no global textual history guidance at all during training. On the other hand, the guided versions are our DIFFUVST models trained with the best configuration of and Punguide depending the datasets. F., P., D., and A. respectively stand for the abbreviations of the FlintstonesSV, PororoSV, DiDeMoSV, and AESOP datasets. As shown here, our DIFFUVS ST outperforms various autoregressive baselines with few marginal exceptions, while requiring significantly lower inference time.
2020) as the visual encoder and GPT2 (Rad- ford et al., 2019) for the language generator, ensuring fair comparison, our DIFFUVST also employs a VIT encoder (BLIP-Image- Model) for image features (Section 4.1).
BLIP (Li et al., 2022a) and we keep their param- eters frozen. Our denoising transformer is ini- tialized with the pretrained DISTILBERT-BAL BBAS- UNCASED (Sanh et al., 2019). To keep the ground- truth story-text embeddings (re) stable for training the denoising model, we keep the pretrained word embedding layers and language model head frozen throughout the training process. More detailed hyperparameters for of DIFFU training and inference can be found in Appendix A.2.
BLIP-EneDee (Li et al., 2022a) leverages the pretrained BLIP captioning model (Li al., 2022a), fine-tuned on the image-text pairs from our datasets. During inference, we in- corporate generated captions from previous images the textual history of the story se- quence, introducing a sequential dependency for captioning the current image. Given that BLIP-EncDec builds upon well-established captioning model, it serves strong base- line, with the mentioned modification specifi- cally for visual storytelling.
5.2 Text generation quality and efficiency
The best metric performances of DIFFUST in comparison to the baselines are reported in Ta- ble With regards to the key guidance-related hyperparameters w and Punguide, the best re- sults are achieved with w = 0.3/0.7/1.0m 10/0.3 and Pungnide: - 0.5/0.7/0.7/0.3 for Flint- stones SiinesS//PPoooS/DDDDMWWWWWAS respectively. As evident from Table 1, DIFFU consistently outperforms these baselines across nearly all metrics and datasets. While there are marginal exceptions, such as the slightly lower @4 score on the PororoSV and ASEOP datasets and the CIDEr score on DiDeMoSV compared to
datasets
We follow the hyperparameter settings specified in each of the baselines and utilize their official codes for training
four fictional datascts.
DIFFUVST implementation details: The multi- modal image- and text- feature extractors of DIF- VU are realized as branches the pretrained

---

TAPM, these do not diminish VIFFUVST's overall superior performance. Notably, given TAPM (Yu et al., 2021) has outperformed most wate-of-the-art visual storytellers in their results on VIST, includ- ing AREL (Wang et al., 2018), INet (Jung et al., 2020), HSRL (Huang et al., 2019), etc. (as in- dicated in the TAPM paper), we have good rea- son to believe that our model can also surpass these models, further demonstrating the effective- ness of DIFFUVST. Moreover, the guided versions of DIFFUVST consistently outperform the fully- unguided (guidance strength w = 0) counterparts, which proves our adaptation of classifier-Ff free guid- ance (Ho and Salimans, 2022) for textual history guidance as highly effective.
amount of denoising iterations may lead less satisfactory text generation, especially from the perspective of NLG metrics such as CIDEr. We attribute this phenomenon to the over-filtering of noise. Our denoising transformer is designed to predict ground-truth text embeddings from any noisy inputs Tt. Thus, even single denoising step can effectively filter out a substantial amount of noise. When this filtering process is applied for too many iterations than necessary, however, it continuously tries to reach a "less noisy" text rep- resentation and may inadvertently remove valuable linguistic details and result in more uniform and less varied text. And metrics like CIDEr are known to lay greater weight on less common n-grams, which explains why the degrading of performance is the most noticeable on the CIDEr metric.
In Table I, we also compared the efficiency of story generation by measuring the average in- ference time for a single sample on a NVIDIA- Tesla- 1000-2GB GPU. Since visual storytelling contains multiple image-text panels, our diffusion- based system that produces all tokens of all text panels simultaneously is able generate story at much faster (about 10 times or more) rate than the autoregressive baselines. This acceleration greatly enhances the efficiency of visual storytelling.
BLEU@4 /CIDE/ CIDEr
w
p= 0.3
p = 0.5
p = 0.7
0.0
1.88
/ 12.51
1887/1.55
1.88 12,51
0.3
2.09
/ 10.91
2.45/ 13.02
2.56 / 14.62
0.5
2.23 / 12.03
2.51 / 13.97
2.68 / 14.08
0.7
2.39
/ 10.73
2.54 11.48
2.56 / 12.14
1.0
2.13
/9.43
2.72 13.99
2.74/ 15.16
Table 2: BLEU@4 and CIDEr metric results DIFFU- VsT trained with different configurations of guidance strengths w and unguided probability Panguide (short- ended as p in this table) on DiDeMoSV. w = refers to fully unguided models and corresponds to DifFUVst ungusde in Table 1. Clearly, using both a high Punguside and high w yields the best result with our DIFFUV
In short, VITFUVST not only achieves top-ticr performance (especially with global history guid- ance), but also excels in efficiency, making an overall superior choice for visual storytelling.
5.3 Inference speed and denoising steps:
The inference time of VST is, however, pos- itively correlated with the number of denoising steps, the increment of which is likely result in more refined story output. Figure 4 illustrates the change of evaluation results of DIFFU with re- spect to the number of denoising steps. Apparently, our model does not require many steps to reach a stable performance and surpass the autoregressive (AR)-baseling (GLACNet as demonstrative exam- ple), which is benefit of our model formulation that directly predicts 2o rather than intermediate steps 4T-1 from the random noise æT. This prop- erty further speeds up the story generation, since our model can already produce decent results in only a few denoising steps.
BLEUg4 Performance
CIDEr Porformance
3.0
20
15
unguide=0.3 unguide-0.. -0.5
unguddeu0:3 unguldeno.5
P.nguide-0077
unggiddmm..7
0.2 0.4
6.6
1.0
0.2 0.4
0.6
Guidence Strongth
Guidance Strength
Figure 3: BLEU@4 and CIDEr metric performance of DIFFUVST with varying configurations of guidance strengths w and unguided probability Punguide on DiDe- MoSV. Each curve represents a value of unguided prob- ability Pangaide- This figure accompanies Table 2. Our diffusion-based model benefits the most from a low guidance rate and strong guidance weight.
Also noteworthy from Figure 4 is that more de- noising steps could result in worse performance. Although larger number of denoising steps typ- ically enhances generation quality, as is the usual case in image synthesis, we believe an excessive
5,4 Textual feature guidance
We have also experimentally verified the effec- tiveness of our adaptation of classifier-fres guid- ance (Ho and Salimans, 2022) to including textual

---

features as additional cues for training our DIF- FU VT model, but with specific probabilities and strengths. The key hyperparameters here are the unguided probability Punguide and the guidance strength w. The former regulates the probability of training the model solely on visual representations, without any textual cues (i.e., not guided by global textual history). The latter controls the weighting of the guided model, as shown in (6).
DIFFUVST
B@1
B@4
M
Dataset FlintstonesSV
Adapter(+) Adapter(-)
49.76
17.01
42.98
36.22
17.63
47.6
16.17
41.65
35.41
13.09
Dataset PororoSV
Adapter(+)
32.09
2.81
25.35
22.03
6.18
Adapter(-)
31.01
2.58
25.06
21.75
5.15
Datasct = DiDeMoSV
Adapter(+) Adapter(-)
22.08
2.74
17.33
19.75
15.16
In Table 2 and Figure 3, we take DiDeMoSV as an example dataset and show the varying test per- formances of our DIFFUVST model when trained with different configurations of guidance strength E {0.0, 0.3, 0.5, 0.7, 1.0) and unguided proba- bility Punguide {0.3, 0.5, 0.7}. If w 0.0, the model is essentially trained without textual feature guidance at all, which corresponds the unguided version of DiffUVsT in Table
21.1
1.72
16.93
17.62
9.19
Datasct AESOP
Adapter(+) Adapter(-) 20.52
23.13
1.41
21.14
19.07
3,4
1.28
19.75
16.84
2.42
Table 3: Evaluation results
metrics of our DIFFU- VST with/without (+/-) Adapter layers under the same hyperparameter settings, including À, guidance strength, and guidance probability. The "Adapter(+)" versions are identical to "DifUVST guided reported in Table I. Evidently, the adapters play an important role in boost- ing DIFFU VST's performance.
As Table 2 and Figure 3 show, DIFFUVST achieves the best BLEU@4 and CIDEr results on DiDeMoSV with strong guidance weight 1.0 and low guidance rate (i.c., high un- guided probability Punguide 0.7). Depending on Punguide: the increase of textual guidance strength during training does not necessarily lead to a bet- ter model performance, whereas a higher unguided probability consistently produces superior results regardless of the strength w. With larger Punguide: the model is trained to concentrate more on the visual features to learn to tell a story instead of relying much the ground-truth textual cues, which will be unavailable inference time. How- ever, if DIFFUVST receives no textual guidance at all (w 0.0), it may not satisfactorily produce a fluent narration, because the model has to ig- nore the global textual context altogether, which is also unwanted for storytelling. This is espe- cially demonstrated by the BLEU@A metric where fully-unguided model obtains the lowest score.
5.5 Multimodal feature adapter
In this subsection, we discuss the effects of the adapter modules that transfer the image-text fea- tures extracted from the pretrained encoders. The motivations for including the adapter modules are two-fold: Firstly, the adapters serve to transfer the pretrained image-text features to the domain of fic- tional scenes; Secondly, since these multimodal features utilized to provide guidance for the optimization of the denoising language model, we need the additional trainable parameters introduced in the adapter modules to better align the encoded features with the capabilities of the denoising LM, thereby enhancing the overall framework integrity of DIFFUVST.
To empirically validate the significance of the adapter modules within our DIFFU system, we present here the ablation studies where we compare the performance of our model with and without the adapter layers. The results on four datasets are shown in Table 3. We keep all other hyperparam- eter settings, including X, guidance strength, and guidance probability identical. The " "Adapter(+)" versions are thus identical "DIFFUVST guided" reported in Table
Based on these observations, we conclude that a relatively small portion of ground-trutt textual guidance is ideal for training our DIFFUVST model. Too much textual signals would not fully develop the model's 's cross-modal ability to caption images into texts and too little conditioning supervision from the story context tends to harm the overall text generation quality. Interestingly, our conclusion also resonates with the findings reported by Ho and Salimans (2022), which emphasize that the sample quality would benefit the most from a small amount of unconditional classifier-fret guidance.
As vividly demonstrated in Table 3, the presence of adapter modules significantly impacts model per- formance across all evaluation metrics and datasets, where DSFFUVST built without adapters consis-

---

BOJST FLEUUT
DRAAT SLAUpT
AUTTOE
DELLATT ADUBRA
DEPMATT CIDE
13.0
40
71 24
15 34
27.5
24
Moneeir 80110-1
Soudee COOR
PororoSV
17.3
12
Sareew Dwweung Mags
Stareer of Dwwsung Megs
Sunder Deseuing Maga
Sareeer of Desaung Mago
Staveer of Dessuin Mags
25.0
Dffisss SLEUEL
Difius ELE staaga
DiftUsT, NETGOR
JOUGEL
DiDeMoSV
simgga
Rocolto PETTES
10.0
Blgss
1ln
Blag
Bagn
Mags
ORWEST ALEU01
DRyAST ALE
DRNUSST METROR
DRWIST ROUGEL
BRASTT COC
AESOR
25.01
3.
37.5
35.0
10
Figure 4: Change of NLG evaluation results of DIFFUVST gusded with respect to the number of denoising steps four datasets. The gray line stands for the performance of GLACNet as a representative Autoregressive(AR) Baseline. Our DIFFUVST consistently manages to outperform the baseline and reach a stable performance within very few denoising steps, thus significantly reducing inference time.
tently underperforms the fully-equipped version. Therefore, these adapter modules, designed to transfer encoded features to the current domain while also enabling more learnable parameters for aligned feature representation, indeed crucial component of our DIFFUVST framework.
quality and inference speed compared to the tra- ditional autoregressive visual storytelling models. Furthermore, we present comprehensive ablation analyses of our method, which demonstrate that our elaborate designs, based diffusion-relann related mechanisms, hold great potential for producing expressive and informative descriptions of visual scenes parallelly with a high inference speed.
6 Conclusion
In this paper, we tackle the challenges posed by Vi- sual Storytelling, and present a diffusion-based so- lution, DIFFUVST that stands out novel effort in introducing non-autoregressive approach to narrating fictional image sequences. Underpinned by its key components, including multimodal en- coders with adapter layers, a feature-fusion module and a transformer-based diffusion language model, DIT models the narration as single de- noising process conditioned on and guided by bidi- rectional global image- and text- history represen- tations. This leads to generated stories that are not just coherent but also faithfully represent the visual content. Extensive experiments attest the outstanding performance of our non-autoren non-autoregressive DSF system both in terms of text generation
Limitations
Our diffusion- based system DIFFUVST inherits main limitation that resides in the need to care- fully search for an optimal hyperparameter setting among various choices and to explain the reason be- hind its validity. The diffusion process contains nu- merous key variables that potentially exerts substan- tial influence on the final model performance for instance, the strengths and probability of classifier- free guidance as discussed in Section 5.4. Although are able to deduce reasonable setup of such pa- rameters via extensive experimental runs, the exact mechanism of how these diffusion-hasee parameter choices affect our storytelling model remains to be explored.

---

References
Qiuyuan Huang, Zhe Gan, Asli Celikyilmaz. Dapeng Wu, Jianfeng Wang, and Xiaodong He. 2019. Hier- archically structured reinforcement learning for top- ically coherent visual story generation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8465-8477
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. 2021. Structured denoising diffusion models in discrete state-spaces.
Hong Chen, Yifei Huang, Hiroya Takamura, and Hideki Nakayama. 2.21. Commonsense knowledge aware concept selection for diverse and informative visual storytelling.
Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aish- warya Agrawal, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. 2016. Visual storytelling. In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2016).
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. 2022. Analog bits: Generating discrete data using diffusion models with self-conditioning.
Prafulla Dhariwal and Alexander Nichol. 2021. Diffu- sion models beat gans image synthesis. Advances Neural Information Processing Systems, 34:8780- 8794.
Yunjae Jung, Dahun Kim, Sanghyun Woo, Kyungsu Kim, Sungjin Kim, and In So Kweon. 2020. Hide- and-tell: Learning to bridge photo streams for visual storytelling. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pages 11213- 11220.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image worth 16x words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.11929.
Tachyeong Kim, Min-Oh Heo, Sconil Son, Kyoung- Wha Park, and Byoung-Tak Zhang. 2018. GLAC net: Glocal attention cascading networks for multi-image cued story generation. CoRR, abs/1805 2t//800../ 1 10973.
Shansan Gong, Mukai Li. Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. 2022. Diffuseq: Sequence to sequence text generation with diffusion models.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg. Mohit Bansal, and Jingjing Liu. 2021. Less is more: Clipbert for video-and-language learning via sparse sampling.
Zhengfu He, Tianxiang Sun, Kuanning Wang. Xuan- jing Huang, and Xipeng Qiu. 2022. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.1500 15029.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. Blip: Bootstrapping language-image pre- training for unified vision- vision-language understanding and generation.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. De- noising diffusion probabilistic models.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang. and Tatsunori B. Hashimoto. 2022b. Diffusion-Im improves controllable text generation.
Jonathan Ho and Tim Salimans. 2022. Classifier-free
diffusion guidance.
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carl- son, and Jianfeng Gao. 2018. Storygan: A sequential conditional gan for story visualization.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. 2021. Argmax flows and multinomial diffusion: Learning categorical dis- tributions.
Ilya Loshchilov and Frank Hutter, 2017. Decoupled weight decay regularization.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nip.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: A unified video and language pre-training model for multimodal understanding and generation.
Chao-Chun Hsu, Zi- Yuan Chen, Chi- Yang Hsu, Chih- Chia Li, Yuan Lin, Ting-Hao 'Kenneth' Huang, and Lun- Wei Ku. 2020. Knowledge-enriched visual storytelling. In AAAI.
Adyasha Maharana and Mohit Bansal. 2021. Integrating visuospatial. linguistic, and commonsense structure into story visualization. Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing.
Ting- Yao Hsu, Chieh- Yang Huang, Yen- Cen-Chialn Hsu, and Ting-Hao 'Kennthh' Huang. 2019. Visual story post- editing.
Junjie Hu, Yu Cheng, 7he Gan, Jingjing Liu, Jianfeng Gao, and Graham Neubig, 2019. What makes a good story? designing composite rewards for visual story- telling.
Adyasha Maharana, Darryl Hamnan, and Mohit Bansal. 2022. Storydall-e: Adapting pretrained text-to-image transformers for story continuation.

---

Zainy M Malakan, Ghulam Mubashar Hassan, and Aj- mal Mian. 2022. Vision transformer based model for describing set of images as a story. In Aus- tralasian Joint Conference on Artificial Intelligence. pages 15-28. Springer.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Alexander Quinn Nichol and Prafulla Dhariwal. 202 la. Improved denoising diffusion probabilistic models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8162-8171. PMLR.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2014. Show and tell: A neural image caption generator.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Ofa: Unifying chitectures, tasks, and modalities through a simple sequence-to-sequeenc learning framework.
Alexander Quinn Nichol and Prafulla Dhariwal. 2021b. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171, PMIR.
Xin Wang, Wenhu Chen, Yuan-Fang Wang, and William Yang Wang. 2018. No metrics are perfect: Adversarial reward learning for visual storytelling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 899-909, 909, Melbourne, Australia. Association for Computational Linguistics.
Xichen Pan, Pengda Qin, Yuhong Li, Hui Xue, and Wenhu Chen, 2022. Synthesizing coherent story with auto-regressive latent diffusion models.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models unsupervised multitask learners. OpenAl blog, 188:9
Pengcheng Yang, Fuli Luo, Peng Chen, Lei Li, Zhiyi Yin, Xiaodong He, and Xu Sun. 2019, Knowledge- able storyteller: commonsense-die model for visual storytelling. In Proceedings of the Twenty-Fighth International Joint Conference Artificial Intelligence, IJCAI-19, pages 5356-5362. International Joint Conferences on Artificial Intelli- genee Organization.
generative
Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. 2021. Aesop: Ab- stract encoding of stories, objects, and pictures. In 2021 LEEE/CVF International Conference Com- puter Vision (ICCV), pages 0032-0043.
Lili Yao, Nanyun Peng. Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2018. Plan- and-write: Towards better automatic storytelling.
Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2021. High- resolution image synthesis with latent diffusion mod- els.
Youngjac Yu, Jiwan Chung, Heeseung Yun, Jongseok Kim, and Gunhee Kim. 2021. Transitional adaptation of pretrained models for visual storytelling. In Pro- ceedings of the IEF//VVF Conference on Computer Vision and Pattern Recognition, pages 12658 1166-8.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Cosiuter-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer.
A Appendix A. Further Report on Experimental Details
A.1 Dataset Statisties
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter.
As stated in section 5.1, we four visual-story datasets of synthetic style and content for our exper- iments: AESOP (Ravi et al., 2021), PororoSV (Li et al., 2018), FlintstonesSV (Maharana and Bansal, 2021) and DiDeMoSV (Maharana et al., 2022). Here are the specific statistics of these datasets: AE- SOP contains visual stories made of 3 image-text panels, with the visual parts generated by clipart ob- jects. It contains 6,024/991 samples in train/val sets (we use the validation set for testing on AESOP well). In PororoSV and FlintstonesSV, which con- tain 10/13332.334 and 20, 17/200030 samples in train/val/test sets respectively, cach story sample is a 5 frame image sequence paired
Yang Song, Conor Durkan, Tain Murray, and Stefano Ermon. 2021. Maximum likelihood training of score- based diffusion models.
Yang Song and Stefano Ermon. 2019. Generative mod- eling by estimating gradients of the data distribution.
Yang Song and Stefano Ermon. 2020. Improved tech- niques for training score-based generative models.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through stochastic differential equations.

---

with 5 consecutive texts. DiDeMoSV uses a 3- frame image-text pair sequence as an individual story with a total number of 11.3502..7773..75 samples in train/val/test sets.
A.2 diffuvst implementation parameters
For DIFFUVST's training and inference, we use the following setup: We set the max- imum text sequence length to 32 for AE- SOP/PononSSV/llnnsooonsyy (i.e., maximum story length of 32 N for N images) and 16 for DiDeMoSV which features shorter captions. Dur- ing training, we randomly sample 30 noised xt from 1000 diffusion timesteps to form the noisy input embeddings to DIFFUVST. For the strength of textual feature guidance w and the unguided probability Punguides we have explored multiple configurations and leave a detailed discussion of their effects in Section 5.4. During inference, use random vectors as noisy input Ty' and replace all guidance text features with zeros and mask them out. For testing, we denoise total steps to iteratively refine the predicted word embeddings and report the best metric performances. We also polish the generated text with unique-consecuttiv method, where the consecutively repeated words are reduced to only one. We train the whole de- noising system with the loss function defined in (7) and the rounding term coefficient (A) is set as a dynamic X strategy as stated Section 4.3. For each dataset, we train DST for 30 epochs using A optimizer (Loshchilov and Hutter, 2017) with the initial learning rate of le - 4 which is then updated by Cosine Annealing schedule and weight decay rate of le - 2.


Reviews:
[{"review_id": "ycAHM0zcw2", "reviewer": "Reviewer_AmK5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer AmK5\n\nThank you for such an insightful and constructive review. We truly appreciate your comments and address your concerns and questions as follows: \n\n**[Q1]:** \nMore recent baselines should be compared with the proposed method.\n\n**[A1]:**\nWe appreciate the reviewer's constructive feedback regarding the need for comparing our DiffuVST with more baselines to validate its effectiveness. We acknowledge the importance of such comparisons in the context of more recent advancements in the field. To address this concern, we have incorporated additional state-of-the-art open-source auto-regressive baselines in order to strengthen the validation of our approach:\n\n1. **TAPM [1, CVPR2021]:** \nTAPM aligns the embedding space of a pretrained visual encoder and language generator using and Adaptation Loss. Subsequently, it applies a Sequential Coherence Loss for finetuning, achieving superior VIST results over multiple visual storytelling baselines including AREL[4], INet[5], HSRL[6], etc. For our task, we utilize a pretrained ViT as the visual encoder and GPT2 for the language generator, ensuring a fair comparison, as our DiffuVST also employs a ViT encoder (BLIP-Image-Model) for image features.\n\n2. **BLIP-EncDec [2, ICML2022]:**\nThis baseline leverages both the Encoder and LM-Decoder of the pretrained BLIP captioning model, fine-tuned on the image-text pairs from our datasets. During inference, we incorporate generated captions from previous images in the textual history of the story sequence, introducing a sequential dependency for captioning the current image. Given that BLIP-EncDec builds upon a well-established captioning model, it serves as a strong baseline, with the mentioned modification specifically for visual storytelling.\n\nConsidering the **accessibility** and **reproducibility** of visual storytelling models, we believe these two strong baselines, along with GLACNet[3], which was part of our initial submission, provide a comprehensive quantitative comparison for our model. \n\nThe results of this extended evaluation are presented in the table below, which supplements Table 1 and 2 from our initial submission. We use \"Writer Mode\" as an example task setting and \"DiffuVST\" refers to the optimally guided version of our model, denoted as \"DiffuVST_*guided*\" in Table 1 and 2.\n\n| Dataset | Method | B@1 | B@4 | M | R | C | Inference_Time (sec.) ↓ |\n|   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: | :----: |\n| *F.* | GLACNet [3] | 44.55 | 11.98 | 37.48 | 29.23 | 8.93 | 1.2042 |\n| *F.* | BLIP-EncDec [2] | 48.75 | 16.32 | 42.25 | 35.3 | 15.94 | 2.0486 |\n| *F.* | TAPM [1] | 49.05 | 15.08 | 42.12 | 33.03 | 16.54 | 1.8923 |\n| *F.* | DiffuVST (ours) | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** | **0.0981** |\n|     |   |   |   |   |   |  |  |\n| *P.* | GLACNet [3] | 23.66 | 2.1 | 21.19 | 17.67 | 2.8 | 0.7844 |\n| *P.* | BLIP-EncDec [2] | 31.38 | 2.96 | 25.13 | 21.64 | 5.63 | 1.8702 |\n| *P.* | TAPM [1] | 28.43 | **3.78** | 24.43 | 21.87 | 5.55 | 1.2289 |\n| *P.* | DiffuVST (ours) | **32.09** | 2.81 | **25.35** | **22.03** | **6.18** | **0.1196** |\n|     |   |   |   |   |   |  |  |\n| *D.* | GLACNet [3] | 20.28 | 1.78 | 16.36 | 15.15 | 7.15 | 0.3030 |\n| *D.* | BLIP-EncDec [2] | 20.02 | 2.14 | 16.01 | 17.74 | 14.7 | 0.7474 |\n| *D.* | TAPM [1] | 20.79 | 2.12 | 16.35 | 17.68 | **15.26** | 0.4457 |\n| *D.* | DiffuVST (ours) | **22.08** | **2.74** | **17.33** | **19.75** | 15.16 | **0.0161** |\n|     |   |   |   |   |   |  |  |\n| *A.* | GLACNet [3] | 20.37 | 1.02 | 19.46 | 13.39 | 2.14 | 0.9285 |\n| *A.* | BLIP-EncDec [2] | 19.16 | **1.81** | 21.88 | 16.71 | 2.89 | 1.3141 |\n| *A.* | TAPM [1] | 21.95 | 1.56 | 21.55 | 16.3 | 2.97 | 1.1629 |\n| *A.* | DiffuVST (ours) | **23.13** | 1.41 | **21.14** | **19.07** | **3.4** | **0.1988** |\n\nAs evident from the table, DiffuVST consistently outperforms these baselines **across nearly all metrics and datasets**. While there are marginal exceptions, such as the slightly lower B@4 score on the PororoSV and ASEOP datasets and the CIDEr score on DiDeMoSV compared to TAPM, these do not diminish DiffuVST's overall superior performance. Notably, given TAPM has outperformed most state-of-the-art visual storytellers in their results on VIST (as indicated in the TAPM paper[1]), we have good reason to believe that our model can also surpass these models, further demonstrating the effectiveness of DiffuVST. Moreover, it is essential to emphasize that DiffuVST's non-autoregressive nature significantly accelerates inference, making it **more than 10x times faster** than the auto-regressive baselines. This acceleration greatly enhances the efficiency of visual storytelling. In conclusion, DiffuVST not only achieves top-tier performance but also excels in efficiency, making it an overall successful choice for the task of visual storytelling. \n\nOnce again, we sincerely thank the reviewer for prompting this comprehensive evaluation, and we will incorporate these additional baselines into our revised paper, further enhancing its comprehensiveness and validation of DiffuVST's capabilities.\n\n\n\n**[Q2]:** \nThe texts in Figures 3 and 4 are too small to read.\n\n**[A2]:** \nThank you for bringing this readability issue to our attention. We acknowledge that the text indeed appears too small in Figures 3 and 4. To address this, we will enlarge the text font size in these figures to ensure that all textual content is clearly visible. In addition, we also plan to provide higher-resolution versions of all our figures in the revised paper and further enhance the readability of the figures.\n\n\n\n**[References]:**  \n[1] Transitional Adaptation of Pretrained Models for Visual Storytelling. CVPR 2021.  \n[2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022.  \n[3] GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation. NAACL 2018.  \n[4] No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling. ACL 2018.  \n[5] Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling. AAAI 2020.  \n[6] Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation. AAAI 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed DiffuVST method is new, and the use of diffusion models is well motivated by inference speed. The history guidance and multimodal adapter modules are also well-motivated. \n2. The authors conducted comprehensive experiments on four fictional visual-story datasets, covering story generation and story continuation tasks. The evaluation demonstrates the superior performance of DiffuVST compared to previous autoregressive models. \n3. Comprehensive analyses, e.g., guidance scale, and number of inference steps, have been conducted. ", "reasons_to_reject": "The only baseline being evaluated was published in 2018. More recent methods can be compared with the proposed method.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents DiffuVST, a novel diffusion-based model for visual storytelling that aims to generate meaningful and coherent narratives from sets of pictures. The authors highlight the challenges of efficiency faced by existing autoregressive methods. To address this challenge, DiffuVST uses stochastic and non-autoregressive decoding, enabling faster and more diverse narrative generation. The model also includes history guidance and multimodal adapter modules to improve inter-sentence coherence. Experimental results on multiple fictional visual-story datasets demonstrate the superiority of DiffuVST over previous autoregressive models in terms of text quality and inference speed. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The texts in Figures 3 and 4 are too small to read. "}, "extra_text": {}}, {"review_id": "EpYBze114P", "reviewer": "Reviewer_sjRb", "summary": "", "questions": "Questions about the method:\n1. The eq7 is less clear. N should be T, right? Why does L339 say “we add a x1 restoring loss to eq (4)”?  Is this loss already included in eq4? Why do you use L1 loss instead of L2?\n2. How important is the dynamic \\lambda strategy in L360-364?\n\nQuestions about the results:\n1. Why can the text guidance improve the performance in the writer mode since the texts are masked as zeros?\n2. Why do more diffusion steps result in worse performance as shown in Figure 4?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer sjRb\n\nThank you for such an insightful and constructive review. We truly appreciate your comments and address your concerns and questions as follows: \n\n**[Q1]:** \nMotivations and effectiveness of the adapter modules.\n\n**[A1]:**\nWe appreciate your concern regarding our motivations behind highlighting the adapter modules as well as the experimental verification of their effectiveness. In response to this, we would like to provide the following points to address your concern:\n\n1. **Motivations for Introducing Adapter Modules:**\n   - ***Domain adaptation:*** Firstly, the adapter modules serve a crucial purpose in bridging the gap between general-domain pretrained features and the specific fictional image-text domain, as you precisely pointed out. These modules facilitate domain transfer, enabling our model to adapt and perform effectively in fictional scenes.\n   - ***Enhancing feature alignment:*** Additionally, the image and text features are utilized to provide guidance for the optimization of our DiffusionLM-based decoder. By incorporating adapter modules, we introduce additional trainable parameters that better align the features extracted by the encoders with the decoding capabilities of the DiffusionLM, thereby enhancing the overall framework integrity.\n\n2. **Effectiveness of the adapter modules:**\n    To empirically validate the significance of the adapter modules within our DiffuVST system, we have conducted ablation studies where we compared the performance with and without the adapter layers. We took the \"Writer Mode\" as the representative task setting and evaluated our models on the four datasets. The results are presented in the table below:\n\n    | Dataset | Method | B@1 | B@4 | M | R | C |\n    |   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: |\n    | *F.* | DiffuVST- Adapter(+) | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** |\n    | *F.* | DiffuVST- Adapter(-) | 47.6 | 16.17 | 41.65 | 35.41 | 13.09 |\n    | *P.* | DiffuVST- Adapter(+) | **32.09** | **2.81** | **25.35** | **22.03** | **6.18** |\n    | *P.* | DiffuVST- Adapter(-) | 31.01 | 2.58 | 25.06 | 21.75 | 5.15 |\n    | *D.* | DiffuVST- Adapter(+) | **22.08** | **2.74** | **17.33** | **19.75** | **15.16** |\n    | *D.* | DiffuVST- Adapter(-) | 21.1 | 1.72 | 16.93 | 17.62 | 9.19 |\n    | *A.* | DiffuVST- Adapter(+) | **23.13** | **1.41** | **21.14** | **19.07** | **3.4** |\n    | *A.* | DiffuVST- Adapter(-) | 20.52 | 1.28 | 19.75 | 16.84 | 2.42 |\n  \n    This table presents the evaluation results of our DiffuVST with (+) and without (-) Adapter layers, all under the same hyperparameter settings, including $\\lambda$, guidance strength, and guidance probability. The \"(+)Adapter\" versions are thus identical to \"DiffuVST_guided\" reported in Table 1 & 2 in the paper. The abbreviations *F.*, *P.*, *D.*, and *A.* correspond to the datasets FlintstonesSV, PororoSV, DiDeMoSV, and AESOP, respectively. The metrics include BLEU@1, BLEU@4, METEOR, ROUGE-L and CIDEr. As vividly demonstrated in the above table, the presence of adapter modules significantly impacts model performance **across all evaluation metrics and datasets**, where DiffuVST built without adapters consistently underperforms the fully-equipped version. Therefore, these adapter modules, designed to transfer encoded features to the current domain while also enabling more learnable parameters for aligned feature representation, are indeed a crucial component of our DiffuVST framework.\n\nWe regret that, due to the space constraints in our initial submission, we had to prioritize a detailed discussion on the \"text feature guidance\" mechanism (Section 5.3) over the ablation studies on adapter layers. However, thanks to your insightful suggestion that Table 1 & 2 can be merged to save space (see **[Q10]**), we will incorporate the critical ablation studies on adapter modules into the main text in our revised paper. This will ensure that the importance of adapter modules is appropriately highlighted, considering their substantial impact on our DiffuVST system's performance, as demonstrated in the above table and discussion.\n\n**[Q2]:** \nMore baselines should be compared to validate the effectiveness of our DiffuVST.\n\n**[A2]:**  \nWe appreciate the reviewer's constructive feedback regarding the need for comparing our DiffuVST with more baselines to validate its effectiveness. We acknowledge the importance of such comparisons in the context of more recent advancements in the field. To address this concern, we have incorporated additional state-of-the-art open-source auto-regressive baselines in order to strengthen the validation of our approach:\n\n1. **TAPM [1, CVPR2021]:** \nTAPM aligns the embedding space of a pretrained visual encoder and language generator using and Adaptation Loss. Subsequently, it applies a Sequential Coherence Loss for finetuning, achieving superior VIST results over multiple visual storytelling baselines including AREL[4], INet[5], HSRL[6], etc. For our task, we utilize a pretrained ViT as the visual encoder and GPT2 for the language generator, ensuring a fair comparison, as our DiffuVST also employs a ViT encoder (BLIP-Image-Model) for image features.\n\n1. **BLIP-EncDec [2, ICML2022]:**\nThis baseline leverages both the Encoder and LM-Decoder of the pretrained BLIP captioning model, fine-tuned on the image-text pairs from our datasets. During inference, we incorporate generated captions from previous images in the textual history of the story sequence, introducing a sequential dependency for captioning the current image. Given that BLIP-EncDec builds upon a well-established captioning model, it serves as a strong baseline, with the mentioned modification specifically for visual storytelling.\n\nConsidering the **accessibility** and **reproducibility** of visual storytelling models, we believe these two strong baselines, along with GLACNet[3], which was part of our initial submission, provide a comprehensive quantitative comparison for our model. \n\nThe results of this extended evaluation are presented in the table below, which supplements Table 1 and 2 from our initial submission. We use \"Writer Mode\" as an example task setting and \"DiffuVST\" refers to the optimally guided version of our model, denoted as \"DiffuVST_*guided*\" in Table 1 and 2.\n\n| Dataset | Method | B@1 | B@4 | M | R | C | Inference_Time(sec.) ↓ |\n|   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: | :----: |\n| *F.* | GLACNet [3] | 44.55 | 11.98 | 37.48 | 29.23 | 8.93 | 1.2042 |\n| *F.* | BLIP-EncDec [2] | 48.75 | 16.32 | 42.25 | 35.3 | 15.94 | 2.0486 |\n| *F.* | TAPM [1] | 49.05 | 15.08 | 42.12 | 33.03 | 16.54 | 1.8923 |\n| *F.* | DiffuVST (ours) | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** | **0.0981** |\n|     |   |   |   |   |   |  |  |\n| *P.* | GLACNet [3] | 23.66 | 2.1 | 21.19 | 17.67 | 2.8 | 0.7844 |\n| *P.* | BLIP-EncDec [2] | 31.38 | 2.96 | 25.13 | 21.64 | 5.63 | 1.8702 |\n| *P.* | TAPM [1] | 28.43 | **3.78** | 24.43 | 21.87 | 5.55 | 1.2289 |\n| *P.* | DiffuVST (ours) | **32.09** | 2.81 | **25.35** | **22.03** | **6.18** | **0.1196** |\n|     |   |   |   |   |   |  |  |\n| *D.* | GLACNet [3] | 20.28 | 1.78 | 16.36 | 15.15 | 7.15 | 0.3030 |\n| *D.* | BLIP-EncDec [2] | 20.02 | 2.14 | 16.01 | 17.74 | 14.7 | 0.7474 |\n| *D.* | TAPM [1] | 20.79 | 2.12 | 16.35 | 17.68 | **15.26** | 0.4457 |\n| *D.* | DiffuVST (ours) | **22.08** | **2.74** | **17.33** | **19.75** | 15.16 | **0.0161** |\n|     |   |   |   |   |   |  |  |\n| *A.* | GLACNet [3] | 20.37 | 1.02 | 19.46 | 13.39 | 2.14 | 0.9285 |\n| *A.* | BLIP-EncDec [2] | 19.16 | **1.81** | 21.88 | 16.71 | 2.89 | 1.3141 |\n| *A.* | TAPM [1] | 21.95 | 1.56 | 21.55 | 16.3 | 2.97 | 1.1629 |\n| *A.* | DiffuVST (ours) | **23.13** | 1.41 | **21.14** | **19.07** | **3.4** | **0.1988** |\n\nAs evident from the table, DiffuVST consistently outperforms these baselines **across nearly all metrics and datasets**. While there are marginal exceptions, such as the slightly lower B@4 score on the PororoSV and ASEOP datasets and the CIDEr score on DiDeMoSV compared to TAPM, these do not diminish DiffuVST's overall superior performance. Notably, given TAPM has outperformed most state-of-the-art visual storytellers in their results on VIST (as indicated in the TAPM paper[1]), we have good reason to believe that our model can also surpass these models, further demonstrating the effectiveness of DiffuVST. Moreover, it is essential to emphasize that DiffuVST's non-autoregressive nature significantly accelerates inference, making it **more than 10x times faster** than the auto-regressive baselines. This acceleration greatly enhances the efficiency of visual storytelling. In conclusion, DiffuVST not only achieves top-tier performance but also excels in efficiency, making it an overall successful choice for the task of visual storytelling. \n\nOnce again, we sincerely thank the reviewer for prompting this comprehensive evaluation, and we will incorporate these additional baselines into our revised paper, further enhancing its comprehensiveness and validation of DiffuVST's capabilities.\n\n\n\n**[Q3]:** \nQualitative results, analysis of success/fail cases, and human evaluation.\n\n**[A3]:** \nThank you for raising important points regarding the need for qualitative analysis and possibly human evaluation as well. We appreciate your feedback and would like to address these aspects as follows:\n\n- **Qualitative result comparison**: Expanding our evaluation beyond quantitative metrics surely allows us to delve into nuanced detilas in terms of e.g., text-image alignment and story coherence. To address this, we have selected several visual stories generated by all compared models. In our revised paper, we will present these examples and provide a detailed qualitative analysis, offering insights into the difference in quality of the generated content by compared models.\n- **Human Evaluation**: We fully recognize the importance of human judgment when assessing the quality of narrative generation, considering the subjective nature of storytelling. In response to your valuable suggestion, we are currently conducting a user study to assess the stories generated by our model and the baselines. Our aim is to incorporate a detailed user study in our revised paper or its appendices, which will provide valuable insights from human perspectives, enriching the assessment of our model's capabilities.\n  \nWe sincerely appreciate your advice that help guide us towards a more comprehensive evaluation of our work, and we are committed to addressing these qualitative aspects of evaluation to enhance the integrity of our paper.\n\n\n**[Q4]:** \nMore clarification on Eq7, $N$ or $T$?\n\n**[A4]:** \nThank you for your insightful question concerning Eq7. We appreciate the opportunity to clarify the notations $N$ v.s. $T$ and their underlying concepts. In Eq7, the term $L^\\prime$ indeed employs a different set of diffusion time steps compared to the original $L$ in Eq4. We explain the differences as follows:\n\n- The $L$ term in Eq4 originates from DDPM [7], serving as the foundational loss function for all diffusion-based systems. As initially conceived in DDPM, $T$ should be a large number, such as 1000, for adding noise (refer also to Line 190-191). This choice aligns with the reverse process (denoising), which then necessitates significantly large steps, e.g., 500. Unfortunately, this DDPM approach results in slow inference, which contradicts our intention for a diffusion-based visual storyteller.\n\n- To accelerate the process, we actually employed the strategy inspired by Improved DDPM [8], as implemented in DiffusionLM [9]. Here, we **sample a subset of size $N$ (e.g., 100) from the total $T$ timesteps** to serve as the actual set of diffusion time steps:\n  $$\n  S = \\{s_0, ..., s_N | s_t < s_{t−1} \\} \\in (0,T]\n  $$\n  Our LM is trained to predict $x_0$ directly from $x_t$, where $t\\in(0, N]$, sampled from this subset $S$. Consequently, our model can generate high-quality output within only a few denoising steps, as evidenced in Section 5.2 and Figure 4. This approach significantly reduces inference time.\n\n- In short, the $L^\\prime$ term in Eq7 indeed uses a different set of diffusion time steps ($N$) compared to the original DDPM's $L$ term in Eq4 ($T$). These smaller steps in $L^\\prime$ are designed to accelerate the denoising process to match our goal for DiffuVST.\n\nAgain, we sincerely appreciate your question about this detail and will ensure that future versions of our paper include clearer definitions and necessary explanations of our equations to eliminate any potential confusion.\n\n\n**[Q5]:** \nThe $x_1$ restoring loss (Line 339) in relation to Eq4 & Eq7.\n\n**[A5]:** \nWe highly appreciate your thoughtful question regarding the \"$x_1$ restoring loss\" and its relation to Eq4 and Eq7 in our paper. Allow us to provide here a clarification:\n\n- The \"$x_1$ restoring loss\" is an addition to the usual \"$x_t$ restoring loss\" as implemented in Improved DDPM [8] and DiffusionLM [9]. It serves to regulate the training process of our model, which learns not only to predict $x_0$ from a randomly sampled diffusion timestep $t \\in N$ (see **[A4]** to **[Q4]**), but also consistently from the timestep $t_1$. \n\n- Thus, the \"$x_1$ restoring loss\" is not part of the original DDPM loss function in Eq4. But our proposed final loss function Eq7 does include this \"$x_1$ restoring loss\" in the term $L^\\prime$, as stated in Line 345-346. Here, we revise our final loss function Eq7:\n    $$\n    L_{final}=L^\\prime + \\lambda L_R  \n    =\\sum_{t=1}^{N}E_{q(x_{t}|x_0)}[\\left\\|\\mu_\\theta(x_{t}, t)-\\hat{\\mu}(x_{t},x_0)\\right\\| + \\left\\|\\mu_\\theta(x_1,1)-\\hat{\\mu}(x_1,x_0)\\right\\| -\\lambda\\log{p_\\theta(w|\\hat{x})}] \n    $$\n\n    It comprises of \"$x_t$ restoring loss\" ($x_{0:t}$), \"$x_1$ restoring loss\" ($x_{0:1}$) and a rounding term ($L_R$).\n\nWe regret that we missed the \"$x_1$ restoring loss\" term in Eq7 of our initial submission. And we sincerely thank you for identifying this oversight. We will rectify this in the revised version of our paper and ensure better accuracy and completeness of the content.\n\n\n**[Q6]:** \nThe reasons behind using L1 instead of L2 for $L^\\prime$ ($x_t$ and $x_1$ restoring loss) in Eq7.\n\n**[A6]:** \nThank you for your insightful question regarding our choice of using L1 instead of L2 for the $x_t$ and $x_1$ restoring loss, denoted as $L^\\prime$ in Eq7. Our decision was motivated by both observations and theoretical considerations:\n\n- **Observations:** During our preliminary experiments, we noticed that employing the original L2 formulation for $L^\\prime$ often resulted in rather turbulent training loss curves. In contrast, when we utilized the L1 distance metric, we observed **more consistent and stable loss convergence**. This led us to favor L1 as a means to enhance training stability.\n\n- **Theoretical Rationale:** We hypothesize that the improved stability with L1 can be attributed to its lower sensitivity to outliers, a crucial aspect given the diverse hyperparameter configurations inherent in our diffusion-based system. Consequently, substituting L2 with L1 appears to promote a **more stable training process**. This choice also aligns with findings in related literature, such as [10], which reported similar advantages associated with L1 loss.\n\nOnce again, we greatly appreciate your question about our choice of L1 over L2. We will make sure to state clearly the motivations behind this decision in our revised paper to avoid confusion.\n\n\n**[Q7]:** \nImportance of the dynamic $\\lambda$ strategy (Line 360-364).\n\n**[A7]:** \nWe appreciate your question regarding the significance of the dynamic $\\lambda$ strategy implemented by our work. We would like to highlight its importance from both observational and empirical perspectives:\n\n- **Observations:** $\\lambda$, the rounding term coefficient, serves to regulate the relative weight of $L^\\prime$ and $L_R$ in Eq7, so that both loss terms should decrease at a proper pace. In our preliminary experiments with fixed $\\lambda$ values (ranging from 0.1 to 0.9), we observed that $L^\\prime$ and $L_R$ inevitably diverged in scale during training, regardless of the chosen $\\lambda$. This divergence led to an undesirable situation, where the relative values of $L^\\prime$ or $L_R$ became either too high or too low, undermining training stability. To maintain a more stable training process, we were thus motivated to adopt a dynamic $\\lambda$ setting, enabling both $L^\\prime$ and $L_R$ to contribute equally to gradient updates, according to the respective status of each training step.\n\n- **Empricial evidence:** In the fixed $\\lambda$ setting, we identified that $\\lambda=0.3$ yielded the most stable training loss curve and model performance. To assess the impact of dynamic $\\lambda$, we have conducted a comparison between dynamic $\\lambda$ and the fixed $\\lambda=0.3$ configuration in terms of model performance. Similar to **[A1]**, we use the \"Writer Mode\" as the representative task setting and present the evaluation results on the four datasets in the table below:\n\n    | Dataset | Method | B@1 | B@4 | M | R | C |\n    |   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: |\n    | *F.* | DiffuVST- λ=*dynamic* | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** |\n    | *F.* | DiffuVST- λ=*0.3* | 48.91 | 16.35 | 41.54 | 35.48 | 10.99 |\n    | *P.* | DiffuVST- λ=*dynamic* | **32.09** | 2.81 | **25.35** | **22.03** | **6.18** |\n    | *P.* | DiffuVST- λ=*0.3* | 31.24 | **2.95** | 25.15 | 22 | 5.25 |\n    | *D.* | DiffuVST- λ=*dynamic* | **22.08** | **2.74** | **17.33** | **19.75** | **15.16** |\n    | *D.* | DiffuVST- λ=*0.3* | 21.35 | 2.45 | 16.7 | 19.44 | 13.02 |\n    | *A.* | DiffuVST- λ=*dynamic* | **23.13** | **1.41** | **21.14** | **19.07** | **3.4** |\n    | *A.* | DiffuVST- λ=*0.3* | 21.13 | 1.31 | 20.7 | 16.94 | 2.81 |\n\n    This table shows the evaluation results of our DiffuVST model trained with both dynamic $\\lambda$ and fixed $\\lambda=0.3$ configurations, using identical hyperparameter settings. The \"$\\lambda=dynamic$\" versions correspond to \"DiffuVST_*guided*\" as reported in Table 1 & 2 in the paper. Notably, the dynamic $\\lambda$ strategy consistently enhances the performance of DiffuVST **across all evaluation metrics and datasets**, except for a minor deviation (B@4 on PororoSV dataset). This evidence clearly underscores the significant role of dynamic $\\lambda$ in optimizing our DiffuVST model's loss function.\n\nWe are sincerely grateful for the opportunity to elaborate on how the dynamic $\\lambda$ strategy impacts our framework. We also plan to emphasize this discussion in the revised version of our paper, given its clear influence on model performance, as demonstrated in the table above.\n\n\n\n**[Q8]:** \nWhy does the text guidance improve model performance even if ground-truth texts are masked during inference? \n\n\n**[A8]:** \nWe thank the reviewer for raising this insightful question regarding the mechanism of textual feature guidance, which is unavailable at inference time. We would like to address this issue through the following points:\n\n- Firstly, the reviewer's keen observation is absolutely correct. During inference, we **indeed mask the ground-truth texts**, resulting in the absence of textual feature guidance for our model's generation. In our practical implementation, during training, we provide the model with textual features in fusion with image encoding (as described in our method in Section 4.2). However, during inference, these textual segments are replaced by zeros, rendering them devoid of any meaningful textual \"guidance\" for model generation.\n\n- This naturally leads us to the question of why this textual guidance still significantly benefits our DiffuVST's performance, as empirically demonstrated in our quantitative evaluation results (shown in Section 5.2 and 5.3). We propose a hypothesis that the ground-truth text encodings, serving as guidance during model training, indirectly contribute by embedding certain training signals from the textual modality. This augmentation affects the distribution patterns within the input space of DiffuVST, which is essentially an image-conditioned text generator. Consequently, the model **learns specific attributes and distributions from the ground-truth text data**, and also **associates these features with the visual conditions**. This enhanced understanding of both the textual modality and the interplay between text and image thus empowers our model to perform better in generating natural language descriptions from visual input, as substantiated by our experimental findings.\n\n- However, we acknowledge that this discrepancy between training and inference may initially seem counterintuitive. It is indeed intriguing how the text feature guidance during training continues to exert a substantial impact on our model's generative capacity at inference time, even in the absence of explicit text guidance. We hope that our above hypothesis sheds light on this phenomenon, and our experimental evidence also encourages further exploration and analysis of this matter.\n\nIn conclusion, we extend our sincere gratitude to the reviewer for posing such a valuable question. We highly appreciate its significance and will include necessary discussions on this matter in the revised version of our paper.\n\n\n**[Q9]:** \nWhy do more diffusion steps result in worse performance as shown in Figure 4?\n\n**[A9]:** \nThank you for raising this intriguing question. Although a larger number of denoising steps typically enhances generation quality, as is the usual case in image synthesis, we believe that an excessive amount of denoising iterations may lead to less satisfactory text generation, especially from the perspective of NLG metrics such as CIDEr. We attribute this phenomenon to two primary factors:\n1. **Over-Filtering Noise:** \nOur denoising transformer is designed to predict ground-truth text embeddings $x_0$ from any noisy inputs $x_t$. Thus, even a single denoising step can effectively filter out a substantial amount of noise. When this filtering process is applied for too many iterations than necessary, however, it might inadvertently remove valuable linguistic details and lead to worse text quality.\n1. **Loss of N-Gram Diversity:** \nIncreasing the number of denoising steps may result in more uniform and less varied text, as the process continuously tries to reach a \"less noisy\" text representation. And metrics like CIDEr are known to lay greater weight on less common N-grams, which explains why the degrading of performance is the most noticable on the CIDEr metric.\n\nWhile our DiffuVST excels in generating high-quality text within only a few number of denoising steps, we acknowledge that the degradation in performance with more steps is not ideal. We appreciate the reviewer's insightful question, as it encourages our future work to explore ways for enhancing our system's stability and predictability during inference.\n\n\n**[Q10]:** \nBetter to merge Tables 1 & 2 and add citations for the compared baselines.\n\n**[A10]:** \nThank you very much for your advice. Combining Table 1 and Table 2 is indeed a good way to save space while preserving an organized representation of the results. And including citations for the baselines shown in tables will certainly provide readers with clearer references. In the revised version, we will ensure to merge Tables 1 and 2 and add the proper citations for the compared baselines. Thank you again for these constructive suggestions.\n\n\n**[Q11]:** \nFigure 2 is not very clear in terms of helping understand how the framework works.\n\n**[A11]:** \nThank you for your valuable feedback regarding the clarity of Figure 2. We appreciate your input and would like to address your concerns with the following points:\n\n- **Clarification of Figure 2 for depicting the DiffuVST workflow:**\n  1. *Data Input and Feature Extraction:* Figure 2 begins with an image stream (depicted as five consecutive pictures at the base of Figure 2) and its corresponding story text sequences (represented by five captions within the blue box at the bottom left) as input. Our DiffuVST model then extracts both textual and visual features using BLIP text and image encoders, as well as adapter modules. This process yields in-domain textual and visual features, denoted by the orange and cyan ladder-shaped icons.\n  \n  2. *Forward Diffusion Process:* The forward diffusion process involves iteratively corrupting the ground-truth caption embedding $x_0$ with Gaussian noise over $T$ diffusion stages. This transformation is depicted in the upper left section of Figure 2, where the blue triangles representing $x_0$ tokens progressively take on a noisy texture, ultimately resulting in $x_T$ after $T$ steps.\n  \n  3. *Denoising Phase and Output Generation:* In the denoising process, the corrupted embedding $x_T$, along with the fused textual-visual guidance features (represented by the central light-green rectangle block), is passed to a Transformer. Specifically, we employ a BERT-like transformer-encoder architecture, as detailed in Line 324-325 of our paper. This Transformer learns to restore $x_0$, which is then rounded to produce the output text sequences, displayed in the top-right blue rectangular block.\n\n- **Our plans to improve Figure 2:** We greatly appreciate your feedback, and we are committed to making improvements to enhance the clarity of Figure 2 in the following ways:\n  - We will **add clearer annotations** to the specific elements within Figure 2, such as ladder shapes denoting adapted features, blue triangles representing ground-truth text embeddings, shaded background texture indicating noise corruption, and more.\n  - we will **supplement Figure 2 with additional figures** illustrating the iterative diffusion process and denoising phase, in order to provide a comprehensive understanding of the underlying diffusion-based mechanisms.\n  - We will also **include more detailed caption explanations** to accompany the figures, ensuring that the entire workflow of our framework is clearly conveyed.\n\n\n\n**[References]:**  \n[1] Transitional Adaptation of Pretrained Models for Visual Storytelling. CVPR 2021.  \n[2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022.  \n[3] GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation. NAACL 2018.  \n[4] No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling. ACL 2018.  \n[5] Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling. AAAI 2020.  \n[6] Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation. AAAI 2019.  \n[7] Denoising Diffusion Probabilistic Models.2022.  \n[8] Improved Denoising Diffusion Probabilistic Models. 2021.  \n[9] Diffusion-LM Improves Controllable Text Generation. 2022.     \n[10] WaveGrad: Estimating Gradients for Waveform Generation. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow.\n\n- The DiffuVST model is the first to leverage diffusion models in visual storytelling. It also explores the use of global-history guidance to improve coherence and visual fidelity.\n\n- Experiments on four datasets demonstrate the effectiveness of the approach with better quantitative performance and higher inference speed.", "reasons_to_reject": "- The multimodal encoder contains less novelty. \nThe paper claims the adapter modules in multimodal encoders as one of the major contributions. However, they are just linear layers to transform the pretrained features, which have been used widely in the literature. Besides, the effectiveness of the adapter module has not been verified in the experiments.\n\n- More baselines should be compared to validate the effectiveness of the diffusion model.\nThe paper only utilizes a LSTM based model as a baseline to compare the auto-regressive and non-autoregressive diffusion models. More state-of-the-art auto-regressive models should be compared. It would be easy to adapt existing image or video captioning models for visual storytelling. For example, one can replace the decoder in DiffuVST by a transformer-based auto-regressive decoder and keep the same encoder for comparison.\n\n- No qualitative results are provided to analyze the success and failure cases.\nMore analysis should be provided to show the advantages of the diffusion model besides the inference speed, especially considering that the model is sensitive to hyperparameters.\nHuman evaluations are encouraged since the task is subjective.", "questions_to_authors": "Questions about the method:\n1. The eq7 is less clear. N should be T, right? Why does L339 say “we add a x1 restoring loss to eq (4)”?  Is this loss already included in eq4? Why do you use L1 loss instead of L2?\n2. How important is the dynamic \\lambda strategy in L360-364?\n\nQuestions about the results:\n1. Why can the text guidance improve the performance in the writer mode since the texts are masked as zeros?\n2. Why do more diffusion steps result in worse performance as shown in Figure 4?", "paper_topic_and_main_contributions": "The paper proposes a diffusion model (DiffuVST) to generate sentence captions for describing fictional scenes. The DiffuVST uses a multimodal adapter module for image and text encoding, and uses text history guidance in the diffusion process. Experimental results show that the DiffuVST achieves better captioning performance and also has faster inference speed.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 1 and table 2 can be merged together to save space. It would also be good to add citations for the compared baselines.\n\nFigure 2 is not very clear. It is hard to understand how the framework works such as the inputs, outputs and the diffusion process."}}, {"review_id": "HyT9qacEIc", "reviewer": "Reviewer_Xen1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Responses to Reviewer Xen1\n\nThank you for such an insightful and constructive review. We truly appreciate your comments and address your concerns and questions as follows: \n\n**[Q1]:** \nArchitecture or mathematical formulation of the adapter modules.\n\n**[A1]:**\nThank you for your inquiry regarding the adapter modules implemented in our work. We appreciate the opportunity to provide a more detailed clarification of its architecture and mathematical formulation, as well as to further demonstrate its significance within our DiffuVST framework:\n\n- **Architecture and mathematical formulation**: \n  Our multimodal adapter modules are implemented as additional linear layers, following the final layers of both the visual and textual encoders. These modules are specifically designed to ensure that the encoded visual and textual feature vectors, $F_v$ and $F_t$, are tailored to the fictional image-text domains (see also Section 4.1). To represent this mathematically, let $V$ denote the output of the BLIP-encoded visual encoder, and $T$ denote the output of the textual encoder. Our adapter layers, represented by matrices $A_v$ and $A_t$, transform these outputs as follows:\n  $$\n  F_v = \\sigma(A_v \\cdot V + b_v); \\\\ \n  F_t = \\sigma(A_t \\cdot T + b_t)\n  $$\n  where $\\sigma$ denotes the activation function, while $b_v$ and $b_t$ represent the respective bias terms for the visual and textual transformations.\n\n- **Effectiveness and importance of the adapter modules**:\n  To empirically assess the impact of the adapter modules within our DiffuVST system, we conducted ablation studies comparing model performance with and without these adapter layers. We took the \"Writer Mode\" as the representative task setting and evaluated our models on the four datasets. The results are reported in the table below:\n\n  | Dataset | Method | B@1 | B@4 | M | R | C |\n  |   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: |\n  | *F.* | DiffuVST- Adapter(+) | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** |\n  | *F.* | DiffuVST- Adapter(-) | 47.6 | 16.17 | 41.65 | 35.41 | 13.09 |\n  | *P.* | DiffuVST- Adapter(+) | **32.09** | **2.81** | **25.35** | **22.03** | **6.18** |\n  | *P.* | DiffuVST- Adapter(-) | 31.01 | 2.58 | 25.06 | 21.75 | 5.15 |\n  | *D.* | DiffuVST- Adapter(+) | **22.08** | **2.74** | **17.33** | **19.75** | **15.16** |\n  | *D.* | DiffuVST- Adapter(-) | 21.1 | 1.72 | 16.93 | 17.62 | 9.19 |\n  | *A.* | DiffuVST- Adapter(+) | **23.13** | **1.41** | **21.14** | **19.07** | **3.4** |\n  | *A.* | DiffuVST- Adapter(-) | 20.52 | 1.28 | 19.75 | 16.84 | 2.42 |\n\n  This table presents the evaluation results of our DiffuVST models with (+) and without (-) adapter layers, all using the same hyperparameter settings. The \"(+)Adapter\" versions correspond to the \"DiffuVST_*guided*\" models reported in Tables 1 and 2 of the paper. The abbreviations  *F.*, *P.*, *D.*, and *A.* stand for the datasets FlintstonesSV, PororoSV, DiDeMoSV, and AESOP, respectively. The metrics include BLEU@1, BLEU@4, METEOR, ROUGE-L, and CIDEr. As evident from the table, the inclusion of adapter modules consistently improves model performance **across all evaluation metrics and datasets**. Models built without adapters consistently underperform the fully-equipped version. Therefore, these adapter modules, designed to adapt encoded features to the target domain while enabling more learnable parameters for aligned feature representation, are a vital component of our DiffuVST framework.\n\nWe sincerely appreciate your question, and we are committed to providing a more detailed formal definition of the adapters and incorporating the ablation studies on adapter modules into the main text of our revised paper. This will ensure that the significance of adapter modules, as demonstrated in the table and our discussion, receives appropriate emphasis in the paper.\n\n\n**[Q2]:** \nMore detailed comparison with other state-of-the-art models.\n\n**[A2]:** We appreciate the reviewer's constructive feedback regarding the need for comparing our DiffuVST with more baselines to validate its effectiveness. We acknowledge the importance of such comparisons in the context of more recent advancements in the field. To address this concern, we have incorporated additional state-of-the-art open-source auto-regressive baselines in order to strengthen the validation of our approach:\n\n1. **TAPM [1, CVPR2021]:** \nTAPM aligns the embedding space of a pretrained visual encoder and language generator using and Adaptation Loss. Subsequently, it applies a Sequential Coherence Loss for finetuning, achieving superior VIST results over multiple visual storytelling baselines including AREL[4], INet[5], HSRL[6], etc. For our task, we utilize a pretrained ViT as the visual encoder and GPT2 for the language generator, ensuring a fair comparison, as our DiffuVST also employs a ViT encoder (BLIP-Image-Model) for image features.\n\n1. **BLIP-EncDec [2, ICML2022]:**\nThis baseline leverages both the Encoder and LM-Decoder of the pretrained BLIP captioning model, fine-tuned on the image-text pairs from our datasets. During inference, we incorporate generated captions from previous images in the textual history of the story sequence, introducing a sequential dependency for captioning the current image. Given that BLIP-EncDec builds upon a well-established captioning model, it serves as a strong baseline, with the mentioned modification specifically for visual storytelling.\n\nConsidering the **accessibility** and **reproducibility** of visual storytelling models, we believe these two strong baselines, along with GLACNet[3], which was part of our initial submission, provide a comprehensive quantitative comparison for our model. \n\nThe results of this extended evaluation are presented in the table below, which supplements Table 1 and 2 from our initial submission. We use \"Writer Mode\" as an example task setting and \"DiffuVST\" refers to the optimally guided version of our model, denoted as \"DiffuVST_*guided*\" in Table 1 and 2.\n\n| Dataset | Method | B@1 | B@4 | M | R | C | Inference_Time(sec.) ↓ |\n|   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: | :----: |\n| *F.* | GLACNet [3] | 44.55 | 11.98 | 37.48 | 29.23 | 8.93 | 1.2042 |\n| *F.* | BLIP-EncDec [2] | 48.75 | 16.32 | 42.25 | 35.3 | 15.94 | 2.0486 |\n| *F.* | TAPM [1] | 49.05 | 15.08 | 42.12 | 33.03 | 16.54 | 1.8923 |\n| *F.* | DiffuVST (ours) | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** | **0.0981** |\n|     |   |   |   |   |   |  |  |\n| *P.* | GLACNet [3] | 23.66 | 2.1 | 21.19 | 17.67 | 2.8 | 0.7844 |\n| *P.* | BLIP-EncDec [2] | 31.38 | 2.96 | 25.13 | 21.64 | 5.63 | 1.8702 |\n| *P.* | TAPM [1] | 28.43 | **3.78** | 24.43 | 21.87 | 5.55 | 1.2289 |\n| *P.* | DiffuVST (ours) | **32.09** | 2.81 | **25.35** | **22.03** | **6.18** | **0.1196** |\n|     |   |   |   |   |   |  |  |\n| *D.* | GLACNet [3] | 20.28 | 1.78 | 16.36 | 15.15 | 7.15 | 0.3030 |\n| *D.* | BLIP-EncDec [2] | 20.02 | 2.14 | 16.01 | 17.74 | 14.7 | 0.7474 |\n| *D.* | TAPM [1] | 20.79 | 2.12 | 16.35 | 17.68 | **15.26** | 0.4457 |\n| *D.* | DiffuVST (ours) | **22.08** | **2.74** | **17.33** | **19.75** | 15.16 | **0.0161** |\n|     |   |   |   |   |   |  |  |\n| *A.* | GLACNet [3] | 20.37 | 1.02 | 19.46 | 13.39 | 2.14 | 0.9285 |\n| *A.* | BLIP-EncDec [2] | 19.16 | **1.81** | 21.88 | 16.71 | 2.89 | 1.3141 |\n| *A.* | TAPM [1] | 21.95 | 1.56 | 21.55 | 16.3 | 2.97 | 1.1629 |\n| *A.* | DiffuVST (ours) | **23.13** | 1.41 | **21.14** | **19.07** | **3.4** | **0.1988** |\n\nAs evident from the table, DiffuVST consistently outperforms these baselines **across nearly all metrics and datasets**. While there are marginal exceptions, such as the slightly lower B@4 score on the PororoSV and ASEOP datasets and the CIDEr score on DiDeMoSV compared to TAPM, these do not diminish DiffuVST's overall superior performance. Notably, given TAPM has outperformed most state-of-the-art visual storytellers in their results on VIST (as indicated in the TAPM paper[1]), we have good reason to believe that our model can also surpass these models, further demonstrating the effectiveness of DiffuVST. Moreover, it is essential to emphasize that DiffuVST's non-autoregressive nature significantly accelerates inference, making it **more than 10x times faster** than the auto-regressive baselines. This acceleration greatly enhances the efficiency of visual storytelling. In conclusion, DiffuVST not only achieves top-tier performance but also excels in efficiency, making it an overall successful choice for the task of visual storytelling. \n\nOnce again, we sincerely thank the reviewer for prompting this comprehensive evaluation, and we will incorporate these additional baselines into our revised paper, further enhancing its comprehensiveness and validation of DiffuVST's capabilities.\n\n\n**[Q3]:** \nA more thorough analysis of limitations.\n\n**[A3]:** \nWe sincerely appreciate your feedback and the opportunity to delve deeper into the limitations of our DiffuVST system. As mentioned in the \"Limitations\" section (Lines 562-575) of our paper, we acknowledge a primary limitation revolving around the need for meticulous hyperparameter search. Allow us to provide a more extensive explanation of how these hyperparameter choices affect the performance of our diffusion-based visual storyteller. Specifically, we will focus on two critical hyperparameters: \n\n1. **Textual feature gidance parameters (unguided probability $p_{unguided}$ and strength $w$)**: \n    In our system's textual feature guidance mechanism (see Section 4.2, Lines 291-321, Eq6), $p_{unguided}$ regulates the probability of training the model solely based on visual data, without any textual cues (i.e., not guided by global textual history), and $w$ controls the weighting of the guided model. The various configurations of these two key hyperparameters significantly impact our model's performance.  \n   As detailed in Section 5.3, using the DiDeMoSV dataset as an example, we experimented with different values of guidance strength $w\\in\\{0.0, 0.3, 0.5, 0.7, 1.0\\}$ and unguided probability $p_{unguided}\\in\\{0.3, 0.5, 0.7\\}$. Only after conducting these exhaustive experiments did we determine that the optimal settings for training our DiffuVST model were a strong guidance weight ($w=1.0$) and a relatively low guidance rate (i.e., a high unguided probability of $ p_{unguided}=0.7$) (see also Table 3 and Figure 3).\n\n2. **Rounding term coefficient $\\lambda$**:\n    As defined in Section 4.3 (Lines 352-364) and Eq7, the rounding term coefficient $\\lambda$, which regulates the relative weight of $L^\\prime$ and $L_R$, is another crucial hyperparameter in our model's training. In our preliminary experiments with fixed $\\lambda$ values (ranging from 0.1 to 0.9), we observed that $L^\\prime$ and $L_R$ diverged in scale during training, regardless of the chosen $\\lambda$. This divergence resulted in an undesirable situation where the relative values of $L^\\prime$ or $L_R$ became either too high or too low, destabilizing the training process Consequently, we had to adopt a dynamic $\\lambda$ strategy (see Lines 359-364) that allowed both $L^\\prime$ and $L_R$ to contribute equally to gradient updates, depending on the respective status of each training step. This dynamic $\\lambda$ strategy proved to be more effective in maintaining training stability. \n    To further underscore the superiority of the dynamic $\\lambda$ strategy, we conducted an experimental comparison between dynamic $\\lambda$ and a fixed $\\lambda$ configuration ($\\lambda=0.3$, which yielded the most stable training loss curve and model performance among fixed $\\lambda$ choices), as shown in the table below:\n    \n    | Dataset | Method | B@1 | B@4 | M | R | C |\n    |   :----:  | :----:  | :----:  | :----:  | :----:  | :----:  | :----: |\n    | *F.* | DiffuVST- λ=*dynamic* | **49.76** | **17.01** | **42.98** | **36.22** | **17.63** |\n    | *F.* | DiffuVST- λ=*0.3* | 48.91 | 16.35 | 41.54 | 35.48 | 10.99 |\n    | *P.* | DiffuVST- λ=*dynamic* | **32.09** | 2.81 | **25.35** | **22.03** | **6.18** |\n    | *P.* | DiffuVST- λ=*0.3* | 31.24 | **2.95** | 25.15 | 22 | 5.25 |\n    | *D.* | DiffuVST- λ=*dynamic* | **22.08** | **2.74** | **17.33** | **19.75** | **15.16** |\n    | *D.* | DiffuVST- λ=*0.3* | 21.35 | 2.45 | 16.7 | 19.44 | 13.02 |\n    | *A.* | DiffuVST- λ=*dynamic* | **23.13** | **1.41** | **21.14** | **19.07** | **3.4** |\n    | *A.* | DiffuVST- λ=*0.3* | 21.13 | 1.31 | 20.7 | 16.94 | 2.81 |\n\n    This table presents the evaluation results in \"Writer Mode\" for our DiffuVST model trained with both dynamic $\\lambda$ and fixed $\\lambda=0.3$ configurations, using identical hyperparameter settings. Notably, the dynamic $\\lambda$ strategy consistently enhances the performance of DiffuVST **across all evaluation metrics and datasets**, except for a minor deviation (B@4 on the PororoSV dataset).\n\nOnce again, we sincerely appreciate the opportunity to discuss the limitations of DiffuVST in greater detail. We plan to incorporate this comprehensive limitation analysis into the appendices of the revised version of our paper, as it promises to inspire future research endeavors.\n\n\n**[Q4]:** \nA clearer explanation of the technical details including clarification on the diffusion model and process implemented in our work.\n\n**[A4]:** \nThank you for your valuable feedback regarding the need for a comprehensive explanation of the technical details involved in our proposed method. We fully recognize the importance of clarity, especially for the purpose of reproducibility. To address this concern, we are committed to providing a thorough and accessible account of the technical details behind our implementation: \n\n1. **Diffusion model and process details**:  \n   Our DiffuVST model is constructed upon the foundation of the Diffusion-LM architecture, incorporating both forward process for adding noise and the backward processes for denoising:\n\n   **The forward process** initiates with a ground truth word embedding $x_0$, and progressively introduces noise over $T$ steps (following a noise scheduler, to be discussed in the next paragraph) to produce $x_T$, utilizing a standard cosine noise scheduler. Subsequently, the corrupted embedding, $x_T$, is subjected to **the backward denoising process**, along with the adapter-tuned visual and textual features as guidance, undergoes the backward denoising process, where denoising transformer directly restores the original textual embedding $x_0$.\n   \n   **The noise schedule**, parametrized by $\\bar{a}_{1:T}$, serves as an important hyperparameter in the Diffusion-LM model shared across all diffusion steps. In our work, we employed the standard **cosine noise schedule** in line with the strategy taken in Improved DDPM[7]. The underlying mechanism could be expressed as follows (see also Line 187), where $q$ denotes a forward noise injection process, which introduces Gaussian noise at time t with variance $β_t ∈ (0, 1)$ based on the $\\bar{a}$ cosine schedule:\n   \n    $$\n      q(x_t|x_{t-1}) = N(x_t; \\sqrt{1-\\beta_t}x_{t-1},\\beta_tI), \n    $$\n  \n    $$\n      \\beta_t=1-\\frac{\\bar{a}_t}{\\bar{a}_{t-1}}\n    $$\n\n    $$\n      \\bar{a}_t = \\frac{f(t)}{f(0)}\n    $$\n\n    $$\n      f(t) = \\cos{(\\frac{t/T+s}{1+s} \\cdot \\frac{\\pi}{2})^2}\n    $$\n  \n2. **Other implementation details**:\n \n    **Model backbones**: The textual and visual feature extractors within our DiffuVST are implemented as the text and image encoder of the pretrained BLIP[2] model. The denoising transformer leverages the architecture of DISTILBERT-BASE-UNCASED[8]. \n    \n    **Hyperparameters in training and inference:** All hyperparameters associated with training and inference are available in Appendix A.2 (Line 766-798). Several key parameters warrant emphasis here: We configure the total number of diffusion steps to 1000. The text sequence length is set to 32 for AESOP/PororoSV/FlintstonesSV and 16 for DiDeMoSV. We train DiffuVST for 30 epochs, employing the AdamW optimizer with an initial learning rate of $1e-4$, subsequently adjusted using a Cosine Annealing schedule a weight decay rate of $1e−2$.\n\nFurthermore, we would like to ensure that we are committed to sharing our codebase and dataset publicly to facilitate further research and validation of our method. Once again, thank you for highlighting this clarity issue.\n\n\n\n**[References]:**  \n[1] Integrating Visuospatial, Linguistic, and Commonsense Structure into Story Visualization. EMNLP 2021.  \n[2] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. ICML 2022.  \n[3] GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation. NAACL 2018.  \n[4] No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling. ACL 2018.  \n[5] Hide-and-Tell: Learning to Bridge Photo Streams for Visual Storytelling. AAAI 2020.  \n[6] Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation. AAAI 2019.  \n[7] Improved Denoising Diffusion Probabilistic Models. 2021.  \n[8] DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates the superiority of DiffuVST over traditional autoregressive models in terms of text quality and inference speed on four fictional visual-story datasets covering both story generation and story continuation tasks.", "reasons_to_reject": "In section 4.1 an adaptor module for transferring BLIP-encoded multimodal properties to the specific domain of fictional sceneries and texts has been proposed in the study, but its architecture or mathematical formulation has not been specified.\nThe paper has claimed to a diffusion architecture as the base architecture but the forward training process, backward denoising, and the corresponding scheduler have not been clarified. Is not clear whether a know diffusion model like the Stable DIffuiosn model has been utilized or just a regular LDM module has been used.\n\nThe paper does not provide a detailed comparison with other state-of-the-art models in the field of visual storytelling, which makes it difficult to assess the true performance of DiffuVST\n\nA thorough analysis of the limitations of DiffuVST, which could lead to an overestimation of its capabilities.\nA clear explanation of the technical details of the proposed model, which could make it difficult for readers to understand and replicate the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes DiffuVST, a diffusion-based system for visual storytelling that generates a series of visual descriptions as a single conditional denoising process. DiffuVST features a unique design with bi-directional text history guidance and multimodal adapter modules, which improve inter-sentence coherence and image-to-text fidelity. The model outperforms traditional autoregressive models in terms of both text quality and inference speed on four fictional visual-story datasets. \nThe main contributions of the paper are:\n (1) modeling the visual narration of a set of images as one single denoising process, \n(2) proposing global-history guidance and multimodal feature adapter modules, and \n(3) demonstrating the effectiveness and practical value of the system.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary evaluation**: The authors propose a diffusion based visual story telling method to improve the quality and inference speed of the task. Two active reviewers have a consensus on the novelty of the proposed method and appreciate the comprehensive analyses and evaluations. Both reviewers had similar concerns related to the limited baseline, however the authors provided additional baselines to resolve the concern from the both reviewers. Also, the authors could provide comprehensive answers to the reviewers, so the most concerns were resolved.

**Reviewer Xen1's reviews**: Reviewer Xen1 have been having a different opinion, but the reviewer haven't interacted at all and haven't even acknowledged the rebuttal by the authors. Also, it was pointed out that Xen1's summary of the paper contained exact copy from the paper which was not desirable.

**Reviewers' recommendations**: After the in-depth discussions, two active reviewers made a consensus that the soundness is good or strong. Also, in terms of excitement, both reviewers had similar consensus which is good or strong. The reviewer Xen1's scores have been '2' on both criteria since the initial review, but Xen1 haven't joined the discussion about misunderstanding of the method. Therefore, I am downweighing the opinions from Xen1.
Decision: Accept-Findings