Sound of Story: Multi-modal Storytelling with Audio
Authors: Jaeyeon BAE
SEOKHOON JEONG
Seokun Kang
Namgi Han
Jae-Yon Lee
Hyounghun Kim
Taehwan Kim
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-modal
Story understanding
Audio-video retrieval
Audio generation

Abstract: Storytelling is multi-modal in the real world. When one tells a story, one may use all of the visualizations and sounds along with the story itself. However, prior studies on storytelling datasets and tasks have paid little attention to sound even though sound also conveys meaningful semantics of the story. Therefore, we propose to extend story understanding and telling areas by establishing a new component called background sound which is story context-based audio without any linguistic information. For this purpose, we introduce a new dataset, called Sound of Story (SoS), which has paired image and text sequences with corresponding sound or background music for a story. 
To the best of our knowledge, this is the largest well-curated dataset for storytelling with sound.
Our SoS dataset consists of 27,354 stories with 19.6 images per story and 984 hours of speech-decoupled audio such as background music and other sounds. As benchmark tasks for storytelling with sound and the dataset, we propose retrieval tasks between modalities, and audio generation tasks from image-text sequences, introducing strong baselines for them. We believe the proposed dataset and tasks may shed light on the multi-modal understanding of storytelling in terms of sound.

Sound of Story:
Multi-modal Storytelling with Audio
Jaeyeon Bac
Scokhoon Jeong? Scokun Kang'
Namgi Han
Jae- Yon Lee
Hyounghun Kim 12
Tachwan Kim
Artificial Intelligence Graduate School
DDpaarment of Computer Science and Engineering
Schooo of Liberal Arts
Ulsan National Institute of Science and Technology, Korea {gowodussla, shjd0246, oraclemiso, hng88, jlee2791, h. h.kim, taehwankim> @unist.ac.kr
Abstract
2022; Wang et al., 2023; He et al., 2023), event- centric understanding (Chen et al., 2021) and using multiway-sample (Xu et al., 2023). Also, lots of approaches are proposed that aim understand the stories within the provided text (Mostafazadeh et al., 2016; Fan et al., 2018; Akoury et al., 2020), image (Huang al., 2016; Bensaid et al., 2021;
Storytelling is multi-modal in the real world. When one tells story, one may use all of the visualizations and sounds along with the story itself. However, prior studies on story- telling datasets and tasks have paid little at- tention to sound even though sound also con- veys meaningful semantics of the story. There- fore, we propose extend story understanding and telling areas by establishing new compo- nent called background sound which story context-based audio without any linguistic in- formation. For this purpose, we introduce a new dataset, called Sound of Story (SoS), which has paired image and text sequences with cor- responding sound or background music for story. To the best of our knowledge, this is the largest well- curated dataset for storytelling with sound. Our SoS dataset consists of 27, .743 stories with 19.6 images per story and 984 hours of speech-dccoupled audio such as back- ground music and other sounds. mark tasks for storytelling with sound and the dataset, we propose retrieval tasks between modalities, and audio generation tasks from image-text sequences, introducing strong base- lines for them. We believe the proposed dataset and tasks may shed light the multi-modal understanding of storytelling in terms of sound. Downloading the dataset and baseline codes for each task will be released in the link: https: //gi thub. com/ 'Sosda lesddataeets. tase wassss/Ssosss.. Datasee.
Krojer et al., 2022) and video (Rohrbach et al., 2015; Xu et al., 2016; Tapaswi et al., 2016; Li et al., 2019; Bain et al., 2020; Huang et al., 2020). Despite there being many benchmarks for story understanding and telling, most of them less focus on audio even if it is one of the key elements for hu- mans to understand stories (Lang et al., 1998; Lake et al., 2017). Therefore, we propose to extend story understanding and telling areas by establishing component called background sound which story comtext-basedd audio without any linguistic information. For this purpose, we introduce novel dataset called the Sound of Story (SoS), which aims to leverage audio information for story understand- ing, distinguishing it from other existing story or audio datasets.
bench-
To create the SoS dataset, all the modal- ities are extracted from Condensed Movie Dataset (CMD) (Bain et al., 2020) and Large Scale Movie Description Challenge datasets (LSMDC) (Rohrbach et al., 2015). Image and text data extracted from the same movie clip have their contexts and both of them have the possibility infer the story from the same movie clip. And the case of audio such as background sound and music, it has the ability to enrich storytelling by representing and recognizing the context of the sit- uations (Eronen et al., 2005; Stowell et al., 2015). Due to these characteristics, audio, image, and text extracted from the same movie clip become cor- related and complementary. So, our SoS dataset allows three modalities generated from the same movic clip to be used for story understanding as a pair. In addition, we propose new story under-
1 Introduction
Story understanding and telling is one of the important but challenging problems in natural language processing and machine lcarning, and there have been extensive studies on this prob- lem for decades (Harrison et al., 2017; Kočiský al., 2018). Many prior researches have fo- cused on building models that understand stories such as utilizing pre-trained model (Chen et al., These authors contributed equally to this work.

---

Image Sequence
Video Frames
Clustaring
Srage Soqpences
Merged Refined
Speech
Movie Dataset
Audio
Decoupling
Speech decoupled
<5pech decespled
Audio
He reaches the scrittHayy backs away, holds the serpt chest. Bo Cateeit notiees this So Caneft eys him beat, then pushes out his chair. Bo Cateett nods. Aonnie sits streig striigt.Roriie opens his that tucked belt.l Cedeet gives Ronniz look. follows
Descriptons
Catlett
the
Overall data generating process flow for our Sound of Story (SoS) dataset. All the modalities extracted from the same movie clips. We exclude the speech from audio to make it contain only pure audio data, which does not contain linguistic information.
Figure I:
standing benchmarks and their baseline models by introducing audio retrieval tasks and audio genera- tion tasks using the SoS dataset.
erate them so that the community can gener- ate more data for variety of tasks to utilize sound in storytelling and story understanding research.
These background sound tasks can be utilized in various applications. For example, for movies and cartoons, background sound often produced according to designated scene using various props or objects for sense of reality. In the case of the retrieval task, among the sound sources made, it is possible to recommend the sound source which fits the best to the scene. Furthermore, background sound generation
2 Related Work
Story understanding and telling Storytelling tasks pose significant challenges in the domain of natural language processing (NLP), and researchers have proposed various datasets and benchmarks to tackle these problems. Wikiplots, WritingPrompts (Fan et al., 2018), and ROCStories (Mostafazadeh et al., 2016) introduce datasets that revolve around text-based story plots. Wikiplots comprises col- lection of approximately 113k story plots extracted from English Wikipedia. WritingPromptss (Fan et al., 2018) assembled their dataset by gath- ering pairs of human-writted stories and corre- sponding prompts from online sources. ROCSto- ries (Mostafazadeh et al., 2016) employed the use of Amazon Mechanical Turk (AMT) to generate short stories that incorporate common-sense knowl- edge, with focus everyday topics. These datasets usually used for story understanding or story generation tasks.
produce more realistic audio while reducing human resource costs by directly generating the right audio for given scene. The overall data generating process can be shown in Figure 1.
In summary, our contributions are as follows:
We propose to extend story understanding and telling areas by establishing new compo- nent called background sound which is story context-based audio without any linguistic in- formation, which has not been well explored. For this purpose, we introduce a new dataset named Sound of Story (SoS) consisting of text and image sequences and audio for a story. To the best of our knowledge, this is the largest well-curated dataset for storytelling with sound.
In addition to text-based story plots, there exist numerous story datasets that incorporate images or videos. Visual Storytelling (Huang et al., 2016) introduces the first sequential vision-to-language dataset and explores its upplicability in various tasks. It goes beyond simple object understanding or captioning for concrete scenes by considering input that encompasses temporal events. Similarly, ImageCoDe dataset (Krojer et al., 2022) also con- tains stories but designed to enable models to
As benchmark tasks to show the significance and relevance of sound in a story, we intro- duce retrieval tasks between audio and other modalities and background sound generation tasks, and baseline models for each of them.
We will release the dataset and code to gen-

---

Example Clip1 (CMD)
Appan (APRe Peateeeete (rw (ryeapp PM
partics wit
Example Clip (CMD)
Pols biological fother Li shows up his village.
Example Clip (LSMDC)
Jackie breeks into lears. They share heiplecs helf-laugh - then Frank appeers in the doorway: She starts to hand him large mirror from her purse - then thinks better of But he holds out his hand and she gives him the mirror.Smon holds...
Example Clip 4 (LSMDC)
The footbell coach, the assistants, and Alabema players cheer for Forrest Forreet runs across the field He speeds pest the playersForreed Fornest runs past the opposite players. The football coach runs along the sidalines as he yells. Forrest..
defencing playars
Figure 2: Example of SoS dataset. CMD and LSMDC have different lengths of descriptions.
learn detailed situations from finer visual differ- ences within similar sets of images.
order to comprehend the distinctive characteristics of multiple modalities.
Movie data inherently encompasses storytelling. Various datasets such as MovieQA (Tapaswi et al., 2016) and MovieNet (Huang et al., 2020) have been collected by using movie data and have been used in various multimodal domains. Addition- ally, datasets like CMD (Bain et al., 2020) and LSMDC (Liu et al., 2019) not only provide movie videos but also include descriptions for the corre- sponding clips. This offers researchers an opportu- nity explore models from multimodal perspec- tive. However, there are few datasets for story with sound. Existing benchmarks for audio mostly con- sist of music
The modall retrieval task has been stud- ied using various modalities such as image-text retrieval (Wang et al., 2020; Zhang et al., 2020; Cheng et al., 2022; Luo et al., 2022; Xuan and Chen, 2023), video-text (Ma et al., 2022; Gorti et al., 2022; Zhu et al., 2023), audio-image (Xu, 2020; Yang et al., 2022: Nakatsuka et al., 2023), video-audio (Surís al., 2018; Gu al., 2023;
Cheng et al., 2023) and audio-text (Kim al., 2022: Xin et al., 2023). Particularly, CLIP4CLIP (Luo et al., 2022), which performs well in the video- text retrieval task by calculating the similarities between the features of each modality obtained from the encoder, and X-CLIP (Ma et al., 2022) expands CLIP4CLIP and proposes multi multi-grained regulation function to improve performance. Fur- thermore, Wav2CLIP (Wu al., 2022) conducts retrieval for audio-image modalities based on CL.P architecture (Radford et al., 2021), while CLAP (Elizalde et al., 2023) focuses on audio-text retrieval. In this paper, we propose a story- story-based audio retrieval task for retrieving proper modal fea- tures like image sequences or descriptions.
et al., 201 I; Sturm, 2012; Ferraro et al., 2021) or animal/environmentall sounds (Salamon et al., 2014; Piczak, 2015; Gem- meke et al., 2017; Kim et al., 2019), and are de- signed evaluate simple matching performance between objects (or sentiment) and audio input.
Multi-modal retrieval tasks Multi-modal re- trieval models are trained to find similar features across different modalities by fusing. These mod- els are tasked with understanding both the context given input and the interplay between features provided by other modalities. As a result, retrieval becomes a highly challenging and crucial task in
Multi-modal audio generation There have been prior studies on generating audio by receiving cer-

---

Dataset
Audio Type
# Story
# Images/ Story
Audio Hours
Avg. text length
Text Type
ImageCode
9,402
17
23.3
Captions
VIST
50,136
4.2
10.2
Description
Video Storytelling
open
105
22
162.6
Captions
MovieNet
open 1,100
633
2004
Description
MSR-VTT
open 10,000
41.2
9.6
Description
speech-decouple audio
Ours
27.354
19.6
984
Description
Table I: Statistics compared to different storytelling datasets. In the audio types, "open" represents raw audio audio" represents the speech-excludee audio. In the text types, "Caption"
from the video, and represents the text that explains only the image, and "Description" represents the text that includes the story.
tain modality information as input (Vasquez and Lewis, 2019; Yu et al., 2022; Borsos et al., 2022; Neves et al., 2022; Schneider et al., 2023; Yang et al., 2023; Agostinelli et al., 2023), Among sev- eral audio generation tasks, recently text-audio generation has been actively studied. AUDIO- GEN (Kreuk et al., 2022) uses augmentation technique that mixes different audio samples when generating audio samples from captions. Riffu- sion (Forsgren and Martiros, 2022) fine-tunes the model to generate Spectrogram images using the image-to-text based on Stable Diffusion (Rombach et al., 2022). MUSICGEN (Copet et al., 2023) uses a single-stage language model with efficient token interlcaving patterns and unsupervised melody inter-
to 12 seconds and is accompanied by captions gen- crated from Description Video Services (DVS). our work, we merge these two movie clip datasets to create the audio-image-text paired dataset. To bridge the gap between CMD and LSMDC, we adapt the LSMDC dataset to match the format of CMD. Since the short movie clips in LSMDC are contiguous, we concatenate them in sequential or- der until their combined length similar that of CMD. Specifically, we concatenate around 20 short movie clips together to create the raw movie clip data.
3.2 Audio processing
To facilitate audio processing, we first utilized fim- peg (Tomar, 2006) to extract audio from the video, ensuring that the length of the extracted audio matched that of the raw movie clip video. Sub- sequently, employed the bytesep framework proposed by (Kong al., 2021) decouple speech from mixed audio containing various sounds. The results of decoupled sounds may considered as audio data that share the storyline with the raw movie clip video without any linguistic informa- tion. Speech-decoupligg is proceeded because the speech itself includes the storyline and the model can learn the storyline focused on the speech, not the audio which is not our expectation. Duration distribution of the extracted audios can be seen Figure 3.
conditioning method. In this paper, ested in generating relevant audio given story.
3 Dataset
We propose a novel dataset named the SoS dataset with 27.354 audio-image-text pairs. 22,330 pairs are created from CMD (Bain et al., 2020) and 5,024 from LSMDC dataset (Rohrbach et al., 2015). We extract the image and audio from the same video and pair them with the description accordingly so that the images and text share the same storyline. In the case of audio, it related to the other modalities due to being extracted from the same clip so can be used in complementary. The examples of the SoS dataset are shown in Figure 2.
3.3 Image processing
3.1 Movie Clips
Since videos include many duplicated and simi- lar images, only the key salient images that may represent storyline were extracted through image clustering. We conduct image clustering I using image fingerprints, which are representations for uniquely identifying an image generated based on the visual features. From all image fingerprints
CMD (Bain et al., 2020) provides the key clips and their high-level semantic descriptions from 3,605 movies. Each key clip has duration of around 150 seconds and can be automatically ob- tained from YouTube, making it easily accessi- ble. LSMDC (Rohrbach et al., 2015) consists I 1180881 short clips extracted from 202 movics. Each clip in LSMDC has duration of around
hnttps://githhe https////ttttta..oo/ooororr///se

---

Number
Iseconds)
words
(a) Frames Distriution
(b) Duration Distribution
(0) Number of words from CMD
(9) Number words from LSMDC
Figure 3: Distributions of each modality. (a) the number of extracted images from a movie video; (b) the duration of decoupled and processed audio for each story (c) the number of words each story from CMD (d) the number of words in each story from LSMDC. Note that due video clip concatenation to match video lengths, descriptions have larger lengths for stories from LSMDC, This text variation leads to the model requiring high-level text understanding.
extracted from frames of the movie clips, we grad- ually removed images with fingerprint similarity of greater than 0.5 which was set empirically found. Through this process, fewer frames for static and more frames for dynamic scenes could be retained. The extracted image sequences can represent the entire vidco by capturing salient scenes with signif- icant variations, even though does not include all of the frames. By utilizing this approach, we can create essential image sequences the video by fo- cusing on scenes with meaningful changes. These processed image sequences provide a novel dataset that highlights the storyline with fewer frames and differentiates it from traditional video datasets. Fig- ure 3 shows the distribution of extracted frames for each movie clip.
3.5 Data Statistics and Analysis
Our SoS dataset is derived from 3,807 movies cov- ered in the CMD and LSMDC datasets, resulting in a total of 27,354 clips. After processing, each clip has an average of 20 images, and the total number of images our dataset 535,707. The accumulated length of the extracted audio is around 984 hours, and each audio consists around 150 seconds on average. Each clip's 's text description consists of 88 words average. If divided into two different datasets, CMD and LSMDC consist of average of 83.2 words and 1,378.7 words, respectively, with medians of 80 and 130.8, respec- tively. Quantitative comparison with other story- based datasets is shown in Table 1, and it can be seen that our proposed SoS dataset is the largest well-curated storytelling dataset with extracted au- dios.
3.4 Text processing
We generate paired decoupled audio data that can be used complementarily for story understand- ing and telling with the image and text in the SoS dataset. The extracted decoupled audio, which is in-the-wild audio, contains abundant information. Specifically, we decoupled speech information to leave only pure audio consisting of background sound on the audio dataset. These remaining au- dios contain a mixture of various daily-life sounds and background music. This distinctive characteris- tic sets it apart from other audio datasets, making it an adequate audio dataset for story understanding.
There exist slight differences in clip lengths be- tween CMD and LSMDC even though both of them provide short descriptions for each movie clip. Because we concatenate LSMDC movie clips to match the length of CMD movie clips, descriptions also must be formatted by concatenating the same number of descriptions with the video. As a result, the concatenated descriptions exhibit variations in sentence lengths as in Figure 3.
Furthermore, there are also inherent differences between the two datasets in how the descriptions were generated. CMD utilizes descriptions gener- ated by users from YouTube, while LSMDC em- ploys descriptive video services to generate descrip- tions. Because of this text variation difference be- tween the datasets, makes our task more chal- lenging by requiring high-level text understanding. We can check that the descriptions are well-aligned with other modalities in Figure 2.
Experiment
4.1 Retrieval Tasks
In this section, we introduce four retrieval tasks: audio-to-viln videe,,, audio-to-text, text-to-audio, and Ando--t-aaudi retrieval using our proposed dataset. The retrieval tasks focus on finding proper audio

---

A2T & T2A
A2V & V2A
imilartty
Similanty
keam
koom
scae
Sean
Scare
Text Encoder
Audio Encoder
Temporal Encoder
Strengar
Audio Embedding
Frame Encoder
Description
Audio
Video Frames
Figure 4: Proposed retrieval model architecture. Encoded features obtained from the encoder corresponding each modality. Features are divided into context-level features and sequence-level features. Then we calculate the score for each task and use the score to calculate final similarity Sim.
corresponding to image sequences text infor- mation containing story, or matching image se- quence text given audio, which may be used as recommendation for background sound suitable for image sequence or storyline.
and audio sequence-frame(Sas.)) These similari- ties are calculated following Eqquation to 4:
Sa. (fa)² (
(1)
Sa.1 = (frfa)" ER
(2)
S... SanJo E CREXI
4.1.1 Retrieval Model
S... = (Joo)(S)) E
(4)
In this section, introduce the model used in our proposed retrieval lask. We are inspired when we create a baseline model in that the Attention Over Simplicity Matrix method proposed by X- CLIP (Ma et al., 2022) can be flexibly applied to Audio as well as Video-Text. Our overall architec- ture is illustrated in Figure 4.
where m is the number video frames and n is the number audio sequences. Except for audio-video similarity, an aggregation process is required to make it an instance-level similarity. Therefore, the mudio-franne and audio sequence- video similarity is calculated through the following equations:
For descriptions, we use the CLIP encoder Fr (Radford et al., 2021). The information ob- tained from the E, is defined as sentence feature fa and word feature fw. the case of video, the CLIP image encoder E is used and the temporal encoder E, is added to obtain the temporal sequence feature. The obtained feature is defined as frame feature f and the pooling of the frame feature is defined video feature fr- And in the case of the audio, audio information extracted by performing au- dio embedding on Log Mel-filter bank coefficients and fed into the EA using pretrained Audio Spec- trogram Transformer (Gong et al., 2021). In this case, the first token among the acquired feature is defined as audio fa. and the remaining tokens are defined as audio sequence feature fas-
P
exp(S..
8 = Σ
So.
(5)
L cap(Sa.
exp(Saan((..))
S
SaL
(6)
21=1 earp(S
The audio-level similarity(S* and video-level similarity (ia) are calculated to obtain the audio sequence-frame instance-level similarity:
exp((au..(t.v)) San. E RR 1m
Saud
CEp(San.f(ia))
erp(Sanntt.o)
S
j=1 (xrp(S.ndtii)) Soud(1.0)
(8)
cxp(Sas.f(.))
S
Sa, CRR
2j=1 exp((uuar(s))
In the case of audio-video retrieval, we first cal- culate the similarity between audio-video(S, (S,,), audio-framm (itiooffrae(S...)) audio sequence-video( r),
cap(Seudtnn))
and
Ei eap(Swadj.1)
Svidl1.1)
(10)

---

A2V
V2A
A2T
T2A
Eval
Ours
Way2CLIP
Ours
Wav2CLIP
Ours
CLAP
Ours
CLAP
Random
R@ I(T)
7.615
4.231
6.462
5.154
7.462
6,538
7.077
5.769
0.076
R R@5(())
18.682
12.615
17.923
16.077
17.462
17.231
19.385
16.077
0.378
R @ 10(1)
28.000
19.538
27.000
23.462
24.692
24.615
25.692
23.538
0.750
Median R(I)
40.615
31.231
38.923
33.692
33.000
34.615
32.769
34.385
631,800
Mean R(U)
32.000
52.000
33.000
47.000
60.000
52.000
63.500
53.500
639.132
Table Retrieval performances with the proposed SoS dataset, which are measured on 1300 test samples. Random means the average of the results of five random retrievals.
where * denotes all elements in that dimension. In addition, Sas, are obtained from the means of Sund and Sii obtained by Equation 8 and 10. Therefore, the similarity between video and audio is obtained as the average of the obtained similari- ties, as shown in Equation 11.
utilize Recall@ 1R@@) 1), Recall@5(R@5), Re- call 10(R@ 10), Mean Rank(Mean R), and Me- dian Rank(Median R). Table 2 shows the per- formances for audio-to-video (A2V), video-to- audio (V2A), audio-to-text (A2T), and text-to- audio (T2A).
For A2V and V2A, each story of the SoS dataset consists of fairly long image sequences by using clustering from the video which has an average length of 215 seconds, unlike other datasets. addition, our audio dataset includes various sounds to express story and thus can better represent storylines compared to audio composed of simple sounds.
Simav = (Sa., + Sa., + San, 5a4,
(11)
For the audio-text retrieval, the similarity SimAT can be calculated by replacing video- related features with text-related features.
4.1.2 Implementation Details
Of the 27,354 stories in our SoS dataset, 26,054 are split into train and 1,300 into test sets. In the case of video, total of 40 images are used as input. At this time, if there are fewer than 40 images, 40 images are filled with duplicate sampling in accordance with the time order, and if there more than 40, 40 images are obtained by uniform sampling. In the case of audio, the entire audio used for learning, and random cropping is performed at this time. In the test, the same audio can be used in all tests by cropping and using the information-rich area of the audio. In the case of text, after tokenizing the description with CLIP tokenizer, we crop 77 tokens from the front and use them similar to other CLIP-based models (Luo et al., 2022; Ma al., 2022; Li al., 2023). The batch size is 128 in the audio-video and 256 the audio-text retrieval task, respectively. We conducted the experiment using 2 NVIDIA A100 80GB GPUs using the PyTorch library for 100 epochs, which took about 20 hours.
However, it also presents greater challenges. Therefore, compared to retrieving tasks using short videos and simple audio, it can be considered a challenging task. Nevertheless, with R@ of 7.615 and 6.462 for both A2V and V2A, respectively, our baseline model performed better than its compar- ative model, Wav2CLIP, and showed reasonable evaluation results.
In addition, our text data consists of storyline that implicitly represents the whole sequence, not description of the scene, so it challenging be- cause the model has to understand the context of the scenes rather than a simpler understanding the scene. Therefore, looking at the R @ perfor- mance 7.462 and 7.077 in A2T and T2A, respec- tively. the performance of our proposed baseline model is better than the comparison model CLAP. Also, the results of retrievals can be found in our supplementary material or GitHub link 2
4.1.3 Results and Analysis
4.2 Audio Generation Task
To evaluate the performance of the proposed base- line model with SoS dataset, we compare it with Wav2CLIP (Wu al., 2022) for A2V and V2A. and CLAP (Elizalde al., 2023) for A2T and T2A. Both Wav2CLIP and CLAP are fine-tuned to our SoS dataset from their pre- trained weights. For the evaluation metrics, we
Composing and arranging background music and sound effects that exquisitely match the current scene's 's story is a highly challenging task that even human creators struggle (Thom, 1999). We propose an audio generation model called SoSGen, which
Phttps:/ thub.

---

is designed to generate appropriate audio for the movie scene, given the scene's 's descriptions, frames, or both as the condition.
To this end, we assessed the mean frequency of each segment and excluded any with a frequency below 78 Hz, threshold determined through man- ual inspection.
4.2.1 Audio Generation Model
This process conducted 185,341 audio data of roughly 15 seconds each, which were then con- verted into spectrograms. From this, randomly split the dataset into 184,462 and 879 as training and testing, respectively.
SoSgen takes advantage of the architecture of dif- fusion models (Sohl-Dickstein et al., 2015), which learn a data distribution through gradual denoising. Stable diffusion models (Rombach et al., 2022), variant of diffusion models, enable effective model conditioning by training the model to learn latent features. Following the same approach, we train SoSgen as a -udioo spectrogram diffusion model, using the following objective function:
For our base model, we leveraged Riffu- sion (Forsgren and Martiros, 2022), checkpoint of Stable Diffusion v1-5 (Rombach et al., 2022), fine-tuned on pairs of music descriptions and spec- trograms. Following the same approach, we fine- tuned the diffusion model from the Riffusion check- point with pairs of SoS descriptions and corre- sponding sliced audio spectrograms. In addition, we further trained the model in the setting of im- age and text conditioning, utilizing the Control- Net (Zhang and Agrawala, 2023) architecture. We use sequence of frames extracted through cluster- ing that correspond the sliced audio as the image condition. SoSjexttimage was additionally trained from SoStext checkpoint, while SoSimage resumed from the Riffusion checkpoint and used redundant text condition in both training and testing. The spe- cific model architecture can be found in Figure 5. For training our models, used batch size of 1, 16 gradient accumulation steps, an initial learning rate of 1e-6, and a cosine learning rate scheduler. We fine-tuned the text-to-audio model for 406,000 steps and additional 525,000 steps to further train the sott-image-to-audee model. Sep- arately. the image-to-audio model was trained for 525,000 steps. We ran fine-tuning for 2 days and ControlNet training for another 2 days on single NVIDIA GeForce RTX 3090 GPU.
C= (0,....) [||c -
[llc-co(E,, (,, t, c(d)) (12)
where e is a trainable CLIP text encoder and is the SoS description.
For image+text-to-audee audioo training, we follow ControlNet (Zhang and Agrawala, 2023) architec- ture, a lightweight and powerful method that en- ables additional conditioning of a diffusion model by freezing the original model's parameters. We freeze the texl-to-audio model's parameters and resume training using similar objective function:
--E,,--N(..)).c.cmm [ik--co(a co 1, c(d), Cang [
(13)
where Cang is an additional CLIP image encoder that embeds the SoS frame sequence f.
4.2.2 Implementation Details
Audio in movies rarely remains consistent until the end of single scene. Also, using long audio as an input or output for deep learning model is bur- dening in terms of hardware capacity. Thus, we slice the audio data into a length of approximately 15 seconds. Since extracted frames partially reflect semantic changes within the scene, audio slicing was performed treating audio data between two extracted frames as base unit. The unit audios are accumulated and concatenated until their cu- mulative duration exceeds 15 seconds, ensuring that each sliced audio is never longer than 15 sec- onds. Audios that are originally longer than 15 seconds are not sliced because it implics the ab- sence significant visual changes for the duration, suggesting a high probability of long, consistent audio. Sliced audio files containing only little sound were eliminated using the PyDub² library.
4.2.3 Results and Analysis
Baselines We compare SoSgen two baselines: Riffusion (Forsgren and Martiros, 2022) and MU- SICGEN (Copet et al., 2023), the state-of-the-ari open source text-to-music generation model. The performance of the models is evaluated based on the Fréchet Audio Distance (FAD) (Kilgour et al., 2019). FAD is a reference- -freeneceefeeeeeeeee evaluation met- ric that is computed using means and covariances of VGGish (Hershey et al., 2017) embeddings. We used lightweight pytorch implementation" of FAD for more efficient evaluation.
"https://githen http:/ttuu..mm/jaaaaaaa cow/ji/arr/pyouuu
"https: / thub. com wwtp///ossaaatattt//ooooeeeeeee Dataset

---

CLIP Image Encodor
ControINet
Latent Space
Space
Diffusion
IaggeTTextor (Sequence Frames)
ZT
CLIP Encoder
Word
OR
Description
Denoising
D
"Audio"
Prediction
Image Aus--uiioooo (Redundant Text Condition)
Texc-la-Auuiii 15o5 Descriptton)
Figure 5: SoSgen architecture for training. The components and arrows drawn with dash lines are only used in the ControlNet-involved training, i.e. image-lo-audio and image+texl-t--auuse condition the denoising model with CLIP text embeddings of SoS descriptions. Once the text-to- to-audio model is sufficiently trained, we freeze the model and add image embeddings as a condition to train the Mettimaaee-t-auudee model using ControlNet. Alternatiely, an image-to-audin model can trained from the Riffusion checkpoint, using redundant text condition "Audio" for each training data. & denotes the encoder, D denotes the decoder, Zt denotes latent representation at timestep
train text-to-audio model, we
Table FAD (lower is better) of SoSgen and the base- lines. SoSgenexxrimgge is the only model given with the image and text condition. The rest are only given with SoS descriptions the condition.
5 Conclusion
In this paper, we emphasize that sound also con- veys meaningful semantics of the story. Thus, we propose novel dataset named Sound of Story (SoS) dataset, audio-image-tent pair dataset. One main feature of our dataset is that all the linguis- tic information inside the audio eliminated. We named the audio without any linguistic information as "Background sound".
Model
F FADyGGiah +
Riffusion
20.114
MUSICGEN
11.680
SoSgengex
10.935
SoSgenimage
18.324
SoSgeneetvvmage 9.099
Background sounds can be utilized various applications and we propose two related tasks, re- trieval task and background sound generation task. To give an example of using it from videos such as movies and cartoons, background sound is of- produced according to designated ing various props or objects for sense of reality, In the case the retrieval task, among the differ- ent sound sources made, is possible to recom- mend the sound source which fits the best to the scene. Furthermore, background sound generation can produce more realistic audio while reducing human resource costs by directly generating the right audio for given scene. To the best of our knowledge, it the first attempt do multimodal tasks with the speech-decouphe background sound relevant to the storyline.
Table 3 shows the comparison SoSgen against Riffusion and MUSICGEN. SoSgenixtt Riffu- sion, and MUSICGEN generate an audio spec- trogram given descriptions. For SoSgenimage and SoSgenesttimage image sequence is also given additional condition.
The results indicate that SoSgeniex and SoSgenhex-inage outperform Riffusion and MUSIC- GEN in terms of FAD. Although SoSgemimage only shows slight improvement from Riffusion, com- bining two modalities exceptionally enhances dio generation capability. The generated audio sam- ples can be listened in our GitHub link and are also included in the supplementary material. The samples show SoSgen's superior performance on story audio generation tasks.
Limitations
To use audio data in story understanding, there are some limitations to be solved. First, we create audio data by decoupling the speech to exclude lin-

---

guistic information and preserving only pure audio. However, the generated audio may be yet distant from what people encounter daily life because there is possibility that various noises caused by the generation from the movie clip remain. Second, the audio-image-tex pair is strongly connected, but it is not perfect due to the nature of the audio. Since audio includes lot of subjective elements, various correct answers may exist for one story. To deal with these limitations, more strict and cautious poli- cies process not only audio but also image and text for story understanding should continue to be studied as future work.
generative framework for storytelling, arXiv preprint arXiv:2108.04324.
Thierry Bertin-Mahieun Daniel PW Ellis, Brian Whit- man, and Paul Lamere. 2011. The million song dataset.
Zakán Borsos, Raphaël Marinier, Damien Vincent, Eu- gene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Tehoul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2022. Audiolm: language mod- eling approach to audio generation. arXiv preprint NXKYv2220903144
Muhao Chen, Hongming 7hang, Qiang Ning, Manling Heng Ji, Kathleen McKeown, and Dan Roth. 2021. Event-centric natural language understanding. In Proceedings of the 59th Annual Meeting of the Asso- ciation Computational Linguisties.
Ethics Statement
One potential ethical concern from our proposed dataset the ability of our model to generate au- dio resembling existing work as it may potentially pose copyright issues such as regarding background music. In addition, generated audio may be too un- comfortable or inappropriate to hear.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Pier- giovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et 2022. Pali: A jointly-scaled mul- tilingual language-image model. arXiv preprint arXiv:2209.0679)
Xuxin Cheng, Zhihong Zhu, Hongxiang Li, Yaowei Li, and Yuexian Zou. 2023. Ssvmr: Saliency- based self- training for video-music retrieval. In ICASSP 2023- 2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE
Acknowledgements
This work was partly supported by Institute of In- formation & communications Technology Planning & Evaluation (IIP)) grant funded by the Korea Artificial intelligence research about multi-moda interac- tions for empathetic conversations with humans & No.2021 1-202110-02000 -0-020688 Artificial Intelligence Innova- tion Hub) and the National Research Foundation Korca(NRF) grant funded by the Korea govern- ment(MSITT) (No. RRS20033-00219959..
government (MSIT)
Yuhao Cheng, Xiaoguang Zhu, Jiuchao Qian, Fei Wen, and Peilin Liu. 2022. Cross-modal graph matching network for image-text retrieval. ACM Transactions on Multimedia Computing. Communications, and Applications (TOMM), 18(4):1-33.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve. Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music gen- eration. arXiv preprint arXtv:2306.05284.
References
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Is- mail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEFE International Confer- ence on Acoustics. Speech and Signal Processing (ICASSP), pages 1-5. IEEE
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et 2023. Musiclm: Generating mu- sic from text. arXiv preprint arXiv:2301.11325.
Antti J Eronen, Vesa T Peltomen, Juha T Tuomi, Anssi P Klapuri, Seppo Fagerlund, Timo Sorsa, Gaëtan Lorho, and Jyri Huopaniemi. 2005. Audio-based context recognition. IEEE Transactions on Audio, Speech, and Language Processing, 14(1):321-329.
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit lyyer. 2020. Storium: A dataset and evaluation platform for machine-in cechiine-ii-th--eee -n-the-loop story generation. arXiv preprint arXiv:2010.01717.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. arXiv preprint arXiv: 180s.08833..33
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. 2020. Condensed movies: Story based retrieval with contextual embeddings. In Proceedings of the Asian Conference on Computer Vision.
Andres Ferraro, Yuntae Kim, Soohyeon Lee, Biho Kim, Namjun Jo, Semi Lim, Suyon Lim, Jungtack Jang, Se- hwan Kim, Xavier Serra, et al. 2021. Melon playlist dataset: A public dataset for audio-based playlist
Eden Bensaid, Mauro Martino, Benjamin Hoover, and Hendrik Strobelt. 2021. Fairytailor: A multimodal

---

generation and music tagging. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 536-540. TEEE.
Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi. 2019. Fréchet audio distance: A reference-free metric for evaluating music enhance- ment algorithms. In INTERSPEECH, pages 2350- 2354.
Seth Forsgren and Hayk* Martiros. 2022. Riffusion Stable diffusion for real-time music generation.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. Audiocaps: Generating cap- tions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages 119-132.
Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence. R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEE.
Eungbeom Kim, Jinhee Kim, Yoori Oh, Kyungsu Kim, Minju Park, Jacheon Sim, Jinwoo Lee, and Kyogu Lec. 2022. Improving audio-lan andio-language learn- ing with mixgen and multi-level test-time augmenta- tion. arXiv preprint arXiv:2210.1 17143.
Yuan Gong, Yu-An Chung, and James Glass, 2021. AST: Audio Spectrogram Transformer, In Proc. In- terspeech 2021, pages 571-575.
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Key- van Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. 2022. X-pool: Cross-modal language- video attention for text-video retrieval. In Proceed- ings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 5006-5015.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transuctions of the Asso- ciation Computational Linguistics, 6:317-328.
Qiuqiang Kong, Yin Cao, Haohe Liu, Keunwoo Choi and Yuxuan Wang. 2021. Decoupling magnitude and phase estimation with deep resunet for music source separation. In ISMIR Citeseer.
Xin Gu, Yinghua Shen, and Chaohui Lv. 2023. A dual- path modll network for video-music retrieval. Sensors, 2302)805
Brent Harrison, Christopher Purdy, and Mark RiedI. 2017. Toward automated story generation with markov chain monte carlo methods and deep neu- ral networks. In Proceedings of the AAAI Conference on Artificial Intelligence and Inteructive Digital En- leriuinment, volume 13, pages 191-197.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022. Audiogen: Textually guided audio generation. arXiv preprint wrXiv:200.113552
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy, 2022. Image retrieval from contextual descriptions. arXiv preprint arXiv:2203.1588
Xingjian He, Sihan Chen, Fan Ma. Zhicheng Huang. Xi- aojie Jin, Zikang Liu, Dongmei Fu, Yi Yang, Jing Liu, and Jiashi Feng. 2023. Vlab: Enhaneing video lan- guage pre-training by feature adapting and blending. arXiv preprimi 2305. 13167.
Brenden M Lake, Tomer D Ullman, Joshua B Tenen- baum, and Samuel J Gershman. 2017. Building ma- chines that learn and think like people. Behavioral and bruin sciences, 40:e253.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Sey- bold, et al. 2017. Can architectures for large-scale audio classification. In 2017 ieee international con- ference on acoustics, speech and signal processing (icassp). pages 131-135 IEEE
Peter J Lang, Margaret M Bradley, and Bruce N Cuth- bert. 1998. Emotion, motivation, and anxiety: Brain mechanisms and psychophysiology. Biological psy- chiatry, 44( 12): 1248-1233.3.
Qinggiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. 2020. Movienet: holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings, Part TV 16, pages 709-727. Springer.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. 2019. Video storytelling: Textual sum- maries for events. IEEE Transactions on Multimedia. 22(2):554-565.
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. 2023. Decap: Decoding clip latents for zero-shot captioning via text-only training, arXiv preprint arXiv:2303.03032.
Han Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick. Xiaodong He, Pushmeet Kohli. Dhruv Ba- tra, et al. 2016. Visual storytelling. In Proceedings of the 2016 conference of the North American chap- ter of the association for computational linguistics: Human language lechnologies, pages 2239.
Yang Liu, Samuel Albanie, Arsha Nagrani, and An- drew Zisserman. 2019. Use what you have: Video retrieval using representations from collaborative ex- perts, arXiv preprint 19Xiv:1907. 13487.

---

Huaishao Luo. Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianui Li. 2022. Clip4elip: An empirical study of clip for end to end video clip retrieval and captioning. Nearocomputiig, 508:293- 304.
Jascha Sohl-Dickstein, Eric Weiss, Niru Mah- eswaranathan, and Surya Ganguli. 2013. Deep un- supervised learning using nonequilibrium thermody- namics. In International Conference on Machine Learning, pages 2256-2265. PMLR.
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022. X-clip: End-to- end -autie- contrastive learning for video-text retrieval. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, pages 638-647.
Dan Stowell, Dimitrios Giannoulis, Emmanouil Bene- tos, Mathieu Lagrange, and Mark D Plumbley. 2015. Detection and classification of acoustic scenes and events. IEEE Transactions on Multimedia, 17(10):133--146.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He. Devi Parikh, Dhruv Batra, Lucy Vanderwende. Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Bob L Sturm. 2012. An analysis of the gtzan music genre dataset. In Proceedings of the second interna- tional ACM workshop on Music information retrieval with user-centered and multinodal strategies, pages 7-12.
Didac Surís, Amanda Duarte, Amaia Salvador, Jordi Torres, and Xavier Giró-i Nieto. 2018. mross embeddings for video and audio retrieval. In Pro- ceedings of the european conference computer vision (eccv) workshops, pages 0-0.
Takayuki Nakatsuka, Masahiro Hamasaki, and Masa- taka Goto. 2023. Content-bass based music-image re- trieval using self-and cross cross-modal feature embed- ding memory. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pages 2174-2184.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631-4640.
Pedro Neves, Jose Foraaii, and João Florindo. 2022. Generating music with sentiment using transformer- gans. arXiv preprint 2rXiv:22
Karol Piczak. 2015. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015- 1018.
Randy Thom. 1999. Designing movie for sound, IRIS- PARIS- 27:9-20.
Suramya Tomar. 2006. Converting video formats with ffmpeg. Linux Journal, 2006( 146):10.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages
Sean Vasquez and Mike Lewis. 2019. Melnet: A gener- ative model for audio in the frequency domain. arXiv preprint arXiv: XXX::9906.00083
PMLR.
Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen. 2020. Cross-modal scene graph matching for relationship-awan image-text retrieval. In Proceedings of the IEEF/CVV winter conference on applications of computer vision, pages 1508- 1517.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schicle. 2015. A dataset for movic description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3202-3212.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. 2022. High- resolution image synthesis with latent diffusion mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2023. Image as a foreign language: Beit pretraining for vision and vision-language language tasks. In Proceedings of the IEFF/CVE Conference on Com- puter Vision and Pattern Recognition, pages 19175- 19186.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. 2014. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM interna- tional conference on Multimedia, pages 1041-1044.
Ho- Hoo-Hsiang Wu, Prem Scetharaman, Kundan Kumar, and Juan Pablo Bello. 2022. Wav2clip: Learning robust audio representations from clip. In ICASSP 2022- 2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 4563- 4567. IEEE.
Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. 2023. M usai: Text-to-music generation with long-context latent diffusion. arXiv preprint arXiv:2301.11557

---

Yifei Xin, Dongchao Yang, and Yuexian Zou. 2023. Im- proving text-audio retrieval by text-aware attention pooling and prior matrix revised loss. In TCASSP 2023-2023 IEEE International Conference Acous- lics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Hanxiao Xu. 2020. Cross-modal sound-image retrieval based deep collaborative hashing. In 2020 Sth In- ternational Conference on Information Science, Com- puter Technology and Transportation (ISCTT), pages 188-197. IEEE.
Jiaqi Xu, Bo Liu, Yunkuo Chen, Mengli Cheng, and Xing Shi. 2023. Multi: Efficient video-and-language understanding with mumpeer and multiple choice modeling. arXiv preprint arXiv:2303.0577
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr- vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 5288-5296.
Hong Xuan and Xi Stephen Chen. 2023. Dissecting deep metric learning losses for image-text retrieval. Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 2164- 2173.
Donghhao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. 2023. Diffsound: Discrete diffusion model for text-to- sound generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing.
Rui Yang, Shuang Wang, Yingzhi Sun, Huan Zhang, Yu Liao. Yu Gu, Biao Hou, and Licheng Jiao. 2022. Multimodal fusion remote sensing inage-audio trieval. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:6220- 6235.
Botao Yu, Peiling Lu, Rui Wang. Wei Hu, Xu Tan, Wei Ye, Shikun Zhang, Tao Qin, and Tic- Liu. 2022. Museformer: Transformer with fine-and coarse-grained attention for music generation. arXiv preprint 22Xx:22200.10345
Lvmin Zhang and Maneesh Agrawala. 2023. Adding conditional control to text-lo-image diffusion models. arXiv preprint 302.5533.
Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z Li. 2020. Context-aware attention network for image- text retrieval. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition. pages 3545..
Jinkuan Zhu, Pengpeng Zeng, Lianli Gao, Gongfu Li. Dongliang Liao, and Jingkuan Song. 2023, Complementarity-wwe space learning for video-text retrieval. HEE Transactions on Circuits and Systems for Video Technology.


Reviews:
[{"review_id": "i6HcnOAyCc", "reviewer": "Reviewer_Lmj8", "summary": "", "questions": "* Have the authors tried using a ViT-based encoder for extracting audio representations i.e. **AST [1]**, since it might be better suited for downstream retrieval tasks?\n* In the case of the visual stream, have the authors tried extracting shots followed by clustering on shot-specific frames to determine the sequence of informative frames?\n\n**[1]**  https://arxiv.org/abs/2104.01778", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Lmj8, thank you for your thoughtful feedback. We address your comments below and will incorporate all feedback into our final paper.\n\n> Q.1. In the retrieval part, the contributions are not clear in terms of the dataset construction. The authors primarily augment two datasets by extracting the audio separately from the videos and separating into speech and non-speech components.\n\nIn the prior storytelling datasets, they usually perform various tasks based on image sequences, videos, or summarized story information. In contrast, in our paper, __we would like to expand the storytelling area by establishing a new component called “background sound”__ which is story context-based audio without any linguistic information. To the best of our knowledge, it is the first attempt to do multimodal tasks with the speech-decoupled background sound with storyline.\n\nA typical example of this background sound retrieval task can be used is the production of videos such as movies and cartoons. When producing the above video, in many cases, to provide realistic sound, various objects or props are used to produce sound according to the scene and insert it according to the scene. This background sound retrieval task can help them select the best sound sequence when given a specific scene and a list of audio samples produced. This process may help them make their video more engaging.\n\n> Q.2. Details of score computation in Eq (1) and (2) are not mentioned. Further, for the sequence to single token similarity, it is not clear how the aggregation is performed after computing similarities with multiple token elements.\n\nWe are afraid we could not give you enough explanation due to the page limit and thank you for asking us a good question to make up for the inconvenience. We adopted the Attention Over Similarity Matrix method proposed in X-CLIP [1] for Video-Text Retrievals to Audio-Video and Audio-Text Retrievals, as mentioned in the paper (lines 329 to 332).\n\nThe equation (1) at line 356 used to perform Audio-Video Retrieval is obtained through the following formula. First, in the case of audio, audio feature $f_{a}$ and audio sequence feature $f_{as}$ are obtained through audio encoder $E_{A}$. For video, video feature $f_v$ and frame feature $f_{f}$ are obtained through Clip Image Encoder $E_{I}$ and Temporary Encoder $E_{t}$.\n\nNext, we calculate the similarity between each acquired feature. These are the equations for them:\n\nAudio-Video Similarity: $ S_{a,v}=(f_a)^T(f_v) \\in \\mathbb{R}^{1}$\n\nAudio-Frame Similarity: $S_{a,f}=(f_{f}f_{a})^{T} \\in \\mathbb{R}^{1 \\times m}$\n\nVideo-Audio Sequence Similarity: $S_{as,v}=f_{as}f_{v} \\in \\mathbb{R}^{n \\times 1}$\n\nAudio Sequence-Frame Similarity: $S_{as,f}=(f_{as})(f_{f})^{T} \\in \\mathbb{R}^{n \\times m}$\n\nwhere  $m$ is the number of video frames and $n$ is the number of audio sequences.\n\nExcept for Audio-Video Similarity, an aggregation process is required to make it an instance-level similarity. Therefore, the Audio-Frame and Audio Sequence-Video Similarity is calculated through the following equation.\n\n$S'\\_{a,f} = \\sum\\_{i=1}^{m} \\frac{exp(S\\_{a,f(1,i)})}{\\sum\\_{j=1}^{m}exp(S\\_{a,f(1,j)})} S\\_{a,f}$\n\n$S'\\_{as,v} = \\sum\\_{i=1}^{n} \\frac{exp(S\\_{as,v(i,1)})}{\\sum\\_{j=1}^{n}exp(S\\_{as,v(j,1)})} S\\_{as,v}$\n\nFinally, the audio-level similarity and video-level similarity are calculated to obtain the Audio Sequence-Frame instance-level similarity. In notation, * means all elements in that dimension.\n\naudio-level similarity:\n\n$S\\_{aud}=\\sum\\_{i=1}^{n} \\frac{exp(S\\_{as,f(i,*)})}{\\sum\\_{j=1}^{n}exp(S\\_{as,f(j, *)})} S\\_{as,f(i,\\*)} \\in \\mathbb{R}^{1 \\times m}$\n\n$S'\\_{aud}=\\sum\\_{i=1}^{m}\\frac{exp(S\\_{aud(1,i)})}{\\sum\\_{j=1}^{m}exp(S\\_{aud(1, j)})}S_{aud(1,i)}$\n\nvideo-level similarity:\n\n$S\\_{vid}=\\sum\\_{i=1}^{m} \\frac{exp(S\\_{as,f(\\*,i)})}{\\sum\\_{j=1}^{m}exp(S_{as,f(\\*, j)})}S\\_{as,f(\\*,i)} \\in \\mathbb{R}^{n \\times 1}$\n\n$S'\\_{vid}=\\sum\\_{i=1}^{n}\\frac{exp(S\\_{vid(i,1)})}{\\sum\\_{j=1}^{n}exp(S\\_{vid(j, 1)})}S\\_{vid(1,1)}$\n\nIn addition, $S'\\_{as,f}$ are obtained from the means of $S'\\_{aud}$ and $S'_{vid}$ obtained by the above equations.\nTherefore, finally, equation (1) of paper line 356 is obtained as the average of the similarities obtained above, as shown in the equation below.\n\n$Sim\\_{AV}=(S\\_{a,f}+S'\\_{a,f}+S'\\_{as,v}+S'\\_{as,f})/4$\n\nIn the case of Equation (2) used in Audio-Text Retrieve, it can be calculated from the above calculation formula by replacing video-related features with text-related features.\n\nWe will include these details in supplementary material of our final paper.\n\n\n> Q.3. The details in Figure 4 are confusing. It might be better to define the terms associated with the similarity scores (token to sequence similarity or sequence to sequence similarities)\n\nWe are sorry if our figure was confusing. What we wanted to provide in Figure 4 was to express the equation described in the response to Q.2. We will add the details in the revision. \n\n> Q.4. For the retrieval section, experimental results should be shown with pretrained multimodal audio-visual (wav2CLIP[1]) or audio-text encoders (CLAP[2]).\n\nThank you for your suggestion. We conducted an additional retrieval experiment using the Wav2CLIP and CLAP you suggested. We finetune pretrained Wav2CLIP and CLAP on our dataset. First of all, in the case of our audio encoder, we did not use the pretrained model, but performed audio embedding with Log Mel-filter bank coefficients as mentioned in our paper (lines 342 to 345), and then used the pretrained ViT with conventional images. Therefore, unlike the audio encoders of Wav2CLIP and CLAP, our audio encoders have not been pretrained with audio. The results are shown in Table 1 below.\n\n__Audio-Video Retrieval__\nIn the case of Audio-Video Retrieval, as shown in Table 1, ours showed mostly better performance than Wav2CLIP.\n\n__Audio-Text Retrieval__\nIn the case of Audio-Text Retrieval, CLAP performs slightly better than ours, showing the benefit of using pretrained model of CLAP on audio data. On the other hand, we use AST model not pretrained on audio data but it still shows comparable performance compared to CLAP, which shows that our model may be a reasonable baseline as non-pretrained model. We will include the new results to our final paper. Thank you.\n\n|          |   A2V  | A2V |   |   V2A  | V2A |   |   A2T  |A2T|   |   T2A  |T2A|   |         |\n|:--------:|:------:|:--------:|---|:------:|:--------:|---|:------:|:------:|---|:------:|:------:|---|:-------:|\n| Eval     |  Ours  | Wav2CLIP |   |  Ours  | Wav2CLIP |   |  Ours  |  CLAP  |   |  Ours  |  CLAP  |   |  Random |\n| R@1      |  __7.615__ |   4.231  |   |  __6.462__ |   5.154  |   |  5.923 |  __6.538__ |   |  5.692 |  __5.769__ |   |  0.076  |\n| R@5      | __18.682__ |  12.615  |   | __17.923__ |  16.077  |   | 15.615 | __17.231__ |   | 14.692 | __16.077__ |   |  0.378  |\n| R@10     | __28.000__ |  19.538  |   | __27.000__ |  23.462  |   | 21.308 | __24.615__ |   | 22.308 | __23.538__ |   |  0.750  |\n| Median R | 40.615 |  __31.231__  |   | 38.923 |  __33.692__  |   | __30.231__ | 34.615 |   | __30.769__ | 34.385 |   | 631.800 |\n| Mean R   | __32.000__ |  52.000  |   | __33.000__ |  47.000  |   | 67.000 | __52.000__ |   | 69.000 | __53.500__ |   | 639.132 |\n\nTable 1. Additional retrieval results with Wav2CLIP and CLAP.\n\n\n\n> Q.5. In the audio generation setup, in section 4.2.2, it is mentioned that two separate models are trained (...)\n\nWhile we have referred to our model as \"$SoSgen\\_{image},$\" its structure and training procedure are identical to \"$SoSgen\\_{text+image}$.\" As mentioned in the caption of Figure 5 in the main paper, the primary distinction between the two models lies in the text conditions given on training and inference time. Specifically, for \"$SoSgen\\_{image},$\" we consistently employ a redundant textual condition, namely, the word \"Audio.\" On the other hand, “$SoSgen\\_{text+image}$” uses SoS descriptions as the text condition. This experimental setup was designed in order to validate the impact of each modality in the SoS dataset on the audio generation task.\n\n> Q.6. Have the authors tried using a ViT-based encoder for extracting audio representations i.e. AST [1], since it might be better suited for downstream retrieval tasks?\n\nWe apologize for any confusion. Actually, __we use AST [2] to extract audio representations in our baseline model__. The explanation about the audio encoder mentioned in line 341 to 345 in the main paper is about how the AST works in our model but we missed to add the reference. Thank you and we will add the reference in revision.\n\n> Q.7. In the case of the visual stream, have the authors tried extracting shots followed by clustering on shot-specific frames to determine the sequence of informative frames?\n\n\nWe assume that the question was about whether we have tried to manually divide the movie clip into shots, which are smaller units of scene, and do clustering to determine the sequence of informative frames.\n\nIf our understanding is correct, no, we have not tried it. Actually, we extracted the frame sequence from the scene. As mentioned from line 238 to 245, we extract frames using image fingerprints similarity. And based on the technique and results mentioned from line 245 to 256, __we made this decision based on our own assessment that the frame sequence was adequately coherent and well-aligned with both the accompanying text and audio. The example of extracted frame sequences can be shown in the provided supplementary material.__ But, we agree that your suggestion is also relevant. Thank you for sharing your insight.\n\n__References:__\n> [1] Yiwei Ma, et al. “X-clip: End-to-end multi-grained contrastive learning for video-text retrieval” ACMMM22.\n\n> [2]Yuan Gong, et al.  “AST: Audio Spectrogram Transformer” Interspeech 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Potential of the dataset to be used for visual scene-guided background score generation.\n* Novel multimodal retrieval task for stories in media by considering all possible modalities including audio, visual, and text.", "reasons_to_reject": "* In the retrieval part, the contributions are not clear in terms of the dataset construction. The authors primarily augment two datasets by extracting the audio separately from the videos and separating into speech and non-speech components.\n\n* Details of score computation in Eq (1) and (2) are not mentioned. Further, for the sequence to single token similarity, it is not clear how the aggregation is performed after computing similarities with multiple token elements.\n\n* The details in Figure 4 are confusing. It might be better to define the terms associated with the similarity scores (token to sequence similarity or sequence to sequence similarities)\n\n* For the retrieval section, experimental results should be shown with pretrained multimodal audio-visual (**wav2CLIP[1]**) or audio-text encoders (**CLAP[2]**).\n\n* In the audio generation setup, in section **4.2.2**, it is mentioned that two separate models are trained ($SOS_{image}$ and $SOS_{image+text}$). Whereas in section **4.2.1**, it is mentioned that the text model is frozen, and then the image guidance is performed on top of it ($SOS_{image+text}$ ?). It is not clear how $SOS_{image}$ is used in this setup?\n\n**[1]** https://arxiv.org/pdf/2110.11499.pdf\n\n**[2]** https://arxiv.org/abs/2206.04769", "questions_to_authors": "* Have the authors tried using a ViT-based encoder for extracting audio representations i.e. **AST [1]**, since it might be better suited for downstream retrieval tasks?\n* In the case of the visual stream, have the authors tried extracting shots followed by clustering on shot-specific frames to determine the sequence of informative frames?\n\n**[1]**  https://arxiv.org/abs/2104.01778", "paper_topic_and_main_contributions": "The authors explore the problem of multimodal storytelling in movies with accompanying sound/background music by augmenting two standard datasets. Further they propose baseline models for two related tasks including multimodal retrieval and audio generation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions\n: https://arxiv.org/abs/2112.00431"}, "extra_text": {}}, {"review_id": "ymJvkDUUlc", "reviewer": "Reviewer_sX7X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer sX7X, thank you for your constructive feedback. We address your comments below and will incorporate all feedback into our final paper.\n\n> Q.1. I failed to truly understand the motivation behind the dataset curation. Why is removing speech from the audio meaningful? To me it is a bit too ad hoc. I think, with the speech, the multimodal retrieval problem will be more meaningful because there are semantic multimodal alignment.\n\n__The purpose of this paper is to expand the storytelling area by establishing a new component called “background sound” which is story context-based audio without any linguistic information.__ To the best of our knowledge, it is the __first attempt__ to do multimodal tasks with the speech-decoupled background sound with storyline and these are challenging and not trivial compared to other audio tasks. And because of the goal of our paper, speech audio is not for us to consider even if it could be helpful for the retrieval tasks. The meaning of removing speech from the audio will be answered in the response to Q2, which is closely related.\n\n> Q.2. The proposed two tasks are also not very related to the SoS dataset, any video dataset could be used on the tasks.\n\n\nTo the best of our knowledge, it is the first attempt to use the story context-based background sound to the retrieval task. And __because the background sound is a new component in the tasks, it is different from other existing tasks.__ Also in the __audio generation task, the purpose is to create a story context-based background sound__ so it is not trivial. Existing audio generation models [1, 2, 3, 4] demonstrate their generation proficiency based on a direct description rather than a story-related context. For example, suitable inputs for the aforementioned models are natural language prompt “Jazz” or an image of a singing woman. In contrast, SoSgen is capable of generating appropriate audio for a story context, such as the prompt \"Mrs. Henderson accidentally shuts off Fido's collar which leads to disastrous results\" or the image sequence of the corresponding scene, which is an example that can be seen on our demo website. Such a task can only be accomplished using SoS dataset, since to our best knowledge, no previous dataset has provided pairs of story context and background sound. \n\n__These background sound tasks can be utilized in various applications.__ To give an example of how to use it from videos such as movies and cartoons, background sound is often produced according to a designated scene using various props or objects for a sense of reality. In the case of the retrieval task at this time, among the various sound sources made, it is possible to recommend the sound source which fits the best to the scene. Furthermore, background sound generation can produce more realistic audio while reducing human resource cost by directly generating the right audio for a given scene.\n\n__References__\n> [1] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352.\n\n> [2] Seth Forsgren and Hayk Martiros. 2022. Riffusion - Stable diffusion for real-time music generation. \n\n> [3] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. 2023. Simple and controllable music generation. arXiv preprint arXiv:2306.05284.\n\n> [4] Iashin, Vladimir, and Esa Rahtu. Taming visually guided sound generation. arXiv preprint arXiv:2110.08791 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a new dataset that might be good for the community. And the dataset and code will be made publicly available.\n2. The related work section contains relevant information about multimodal storytelling , which could be beneficial to the community.", "reasons_to_reject": "1. I failed to truly understand the motivation behind the dataset curation. Why is removing speech from the audio meaningful? To me it is a bit too ad hoc. I think, with the speech, the multimodal retrieval problem will be more meaningful because there are semantic multimodal alignment.\n2. The proposed two tasks are also not very related to the SoS dataset, any video dataset could be used on the tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new dataset for multimodal storytelling with audio. Especially, the paper proposes to remove the speech and only retain the speech decoupled audio so that the \"linguistic information\" in the audio is removed. In addition, the paper proposes two tasks on the dataset, one for multimodal retrieval and one for audio generation based on text and image.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EJeH0o2Xva", "reviewer": "Reviewer_s1GB", "summary": "", "questions": "It’s not clear how important is the speech-decoupling process? What portion of the movie clips include speech signals?\n\nWhat’s the rationale of “we crop 77 tokens from the front and use it.” in line 370\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer s1GB, thank you for your positive feedback and sharing your insights. We address your comments below and will incorporate all feedback into our final paper.\n\n> Q.1. The genre of audio clips are not clear. It would be helpful to have a high-level categorization of audio clips. As a follow up concern, having such categorization can provide more information of the capability of current baseline models for error analysis.\n\nThank you for sharing your interesting insight for our dataset. We also agree that the collected audio data will be helpful in many ways if it becomes a high-level categorization. However, unlike text and images, background sounds are ambiguous in performing categorization, so performing high-level categorization itself is another challenging part for background sound audio. For example, when the background sound is heard in the order of a car moving, a car hitting somewhere, a car door closing, and a running footstep, it is relatively difficult to distinguish which category of the background sound is compared to the image and text. However, as you mentioned, if meta-information related to categories can be generated and utilized, it will be an opportunity for the dataset to become more enabled, so we will explore it.\n\n> Q.2. It’s not clear how important is the speech-decoupling process? What portion of the movie clips include speech signals?\n\nThank you very much for pointing out the important content. In the prior storytelling datasets, they usually perform various tasks based on image sequences, videos, or summarized story information. In contrast, in our paper, we would like to expand the storytelling area by establishing a new component called “background sound” which is story context-based audio without any linguistic information. To the best of our knowledge, it is the first attempt to do multimodal tasks with the speech-decoupled background sound with storyline and these are challenging and not trivial compared to other audio tasks.\nIn our SoS dataset, 27,354 movie clips have 984 hours in total, and the audio extracted from them has the same amount of time. And when we checked the time that the audio split with speech and background sound, there were about 408 hours for the speech, and about 480 hours for the background sound. __It means that the background sound is a significant part of the video and is not trivial to use only background sound in storytelling tasks.__\n\n> Q.3. What’s the rationale of “we crop 77 tokens from the front and use it.” in line 370\n\n\n\nThe reason why we crop in front of 77 tokens is to follow the setting of the x-clip[3] paper that we use as baseline. When using the pre-trained model provided by the Open AI CLIP paper[1], the maximum context length of the tokenizer is 77. And in the x-clip, the max-token-length is set to 77 and used accordingly. At this time, if the number of tokens exceeds 77, the x-clip uses the front 77 tokens in the sentence to perform several tasks, all of which show reasonable performance. It is a common technique to use front tokens for clip encoder if the sentence is too long and not only x-clip but also other papers with clip also use. (Clip4clip [2], DeCap[4] etc). Based on these, our experiment also uses a setting that is used by crop in front of the sentence for sentences exceeding 77 tokens.\n\n\n\n__References:__\n\n> [1] Radford, et al. “Learning transferable visual models from natural language supervision” ICML 2021.\n\n> [2] Luo, et al.  “Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning” Neurocomputing 2022\n\n> [3] Yiwei Ma, et al. “X-clip: End-to-end multi-grained contrastive learning for video-text retrieval” ACMMM22.\n\n> [4] Li, et al. “DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training” ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has an unique focus on speech-decoupled audio for the multi-modal story dataset. \nThe demo website shows a promising performance of the baseline models. It’s already an interesting experience to explore benchmark results .\n\n", "reasons_to_reject": "The genre of audio clips are not clear. It would be helpful to have a high-level categorization of audio clips. As a follow up concern, having such categorization can provide more information of the capability of current baseline models for error analysis.\n", "questions_to_authors": "It’s not clear how important is the speech-decoupling process? What portion of the movie clips include speech signals?\n\nWhat’s the rationale of “we crop 77 tokens from the front and use it.” in line 370\n", "paper_topic_and_main_contributions": "The authors curate a large multi-modal story dataset, Sound of Story (SoS), with the focus on the audio part of the story. The dataset of 27,354 stories is built based on two movie-videos datasets, CMD and from LSMDC datasets. The audio parts of SoS are decoupled with the speech to prevent leaking language information for the retrieval task benchmark of this paper.\n\nIn the second part of the paper, the author proposed two tasks: cross-modality (audio <-> video and audio <-> text) retrieval tasks and the audio generation task. The baseline results on retrieval tasks show reasonable performance on Recall@1 with at least 5% compared to random baseline of 0.07%.\n\nAs for audio generation tasks, the demo website also presents examples of good generated audio clips with multiple baseline models, e.g., SoSgen.\n\nThe contributions of this paper are two-fold: (1) curation of the first large audio-focused multi-modal story dataset, and (2) proposing benchmarks of cross-modal retrieval tasks and the audio generation task.\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new dataset of multimodal storytelling (images from the story, the text itself of the story, and sound effects/music that are part of the story. From the demo page, it seems as though the sound is predominantly music). The dataset is quite large at 27k stories with 19 images per story and 984 hours of sound, and is derived from Hollywood movies. The paper also presents some benchmark tasks for the dataset such as retrieval and audio generation from image/text.

Pros:
The dataset is novel in its purpose and scope, and is to be made publicly available

Cons:
-Some of the technical details for how similarity scores are not clear (see comments by reviewer Lmj8)
-The experiments don't include results from pre-trained models like CLAP or wav2Clip
-It is not clear how the proposed dataset offers any advantages over a dataset of videos + captions/narrations
-It is also not clear why speech/dialog was removed from the soundtracks. The author's response to reviewer sX7X on this point doesn't really explain what is more compelling about the music/background sound with the movie dialog removed, compared to the full original audio
Decision: Accept-Findings