TAVGBench: Benchmarking Text to Audible-Video Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-29 04:40:22

Abstract: The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. 
We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. 
We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. 
Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. 
We achieve the alignment of audio and video by employing cross-attention and contrastive learning. 
Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics.

Main paper: 
 introduction: 
 The text to video generation task [2, 10, 13, 39, 47, 51] has been boosted through the integration of computer vision and natural language processing. This task translates textual descriptions into visual representations, enriching multimedia experiences, and improving accessibility for individuals with visual impairments. However, we observe that although existing methods excel in converting textual descriptions into visual content, the endeavor to integrate synchronized audio into these videos remains largely unexplored. This gap underscores a fundamental necessity within the realm of multimodal generation-the imperative to generate video content with auditory components guided solely by textual descriptions.In this paper, considering the clear gap in current research, we introduce a new task: Text to Audible-Video Generation (TAVG). This task marks a significant change, requiring models to move beyond just generating visual content and also creating audio alongside it. Unlike typical text to video tasks that only focus on unimodal video generation, TAVG requires generating both audio and video at the same time, guided by written descriptions. By taking on this task, we aim to push the boundaries of multimodal generation, making it possible to create immersive audio-visual experiences using only text prompts. The task definition is shown in Fig. 1 .To successfully achieve TAVG, a comprehensive dataset with well-aligned audio and video components is essential. However, we find no mature benchmark available for this task to support training and testing, primarily attributable to the absence of such a large-scale dataset. Building upon the foundation of TAVG, we propose establishing a Text to Audible-Video Generation Benchmark (TAVGBench), which allows the model to be trained in a supervised manner. At the core of TAVGBench lies a carefully selected dataset, comprising diverse textual descriptions and their corresponding audio-visual pairs. This dataset facilitates comprehensive evaluation and comparison of various methods. Our dataset consists of over 1.7 million audio-visual pairs sourced from YouTube videos. We design a coarse-to-fine pipeline to automatically achieve text annotation for audio-visual pairs in the dataset. Specifically, we utilize BLIP2 [23] and WavCaps [27] to describe the video and audio components, respectively. Additionally, we employ ChatGPT [29] to rephrase and integrate annotations from both modalities, which enables our annotation pipeline to excel in understanding context and producing human-like text descriptions. To evaluate the alignment degree between the generated audio and video, we introduce a new metric for the TAVG task to measure the harmony of the generated results, called the Audio-Visual Harmoni score (AVHScore). This metric quantifies the alignment between video and audio in a multi-modal, high-dimensional semantic space.Utilizing our proposed TAVGBench, we present a Text to Audible-Video Diffusion (TAVDiffusion) model as the baseline method. This method is based on the latent diffusion model [34] , representing an initial attempt to generate audio and video from text. Given the requirement of multimodal alignment, we propose two strategies to achieve the alignment of multimodal latent variables from the perspective of feature interaction and feature constraints. We extensively evaluate the baseline model against the proposed TAVGBench using both the conventional metrics and our proposed metrics and demonstrate the effectiveness of our method in the task of TAVG.As follows, we summarize our main contributions as:• We introduce the TAVG task, extending multimodal generation by integrating synchronized audio with visual content, addressing a crucial research gap. • We present TAVGBench, a large-scale benchmark dataset with an automatic text description annotation pipeline and a novel Audio-Visual Harmoni score (AVHScore), significantly facilitating the TAVG task. • We propose the Text to Audible-Video Diffusion (TAVDiffusion) model as a baseline method built upon the latent diffusion model.In the background, music is playing.A baby is sitting on top of a table and looking down at something on the floor. The baby appears to be curious and is observing whatever is capturing its attention below.In the background, music is playing as a baby sits on top of a table, peering down at something on the floor with a curious expression. The scene is tranquil as the baby remains focused and engaged in whatever has captivated its interest below.  
related work: 
 Text to video generation. Text to video generation [2, 10, 13, 18, 39, 47, 49, 51] presents a challenging and extensively studied task. Previous studies utilize various generative models, such as GANs [24] , and Autoregressive models [15, 55] . In recent years, the emergence of diffusion models [14] in content generation (i.e. textto-image generation) catalyzes substantial advancements in text-tovideo generation research. Imagen-Video [13] , Make-A-Video [39] , and show-1 [56] propose a deep cascade of temporal and spatial upsamplers for video generation, training their models jointly on both image and video datasets. The majority of subsequent works are grounded in the latent diffusion model [34] , leveraging pre-trained UNet weights on 2D images. VideoLDM [2] adopts a latent diffusion model, where a pre-trained latent image generator and decoder are fine-tuned to ensure temporal coherence in generated videos. LAVIE [47] integrates Rotary Positional Encoding (RoPE) [42] into the network to capture temporal relationships among video frames. AnimateDiff [10] employs a strategy of freezing a pre-trained latent image generator while exclusively training a newly inserted motion modeling module. SimDA [51] proposes an efficient temporal adapter to help a trained 2D diffusion model extract temporal information. These advancements lay the foundation for an efficient multimodal diffusion pipeline.Text to audio generation. Similar to video generation, the task of text to audio generation also evolves from GANs [28, 37] and Autoregressive models [20] to diffusion models. DiffSound [53] proposes a VQVAE model and a mask-based text generation strategy to address scarce audio-text paired data, albeit with potentially limited performance due to the lack of detailed text information. AudioGen [20] employs an autoregressive framework utilizing a Transformer-based decoder to generate tokens directly from the waveform. It applies data augmentation and simplifies language descriptions into labels, sacrificing detailed temporal and spatial information. AudioLDM [25] transfers the latent diffusion model from the domain of visual generation to text-to-audio generation. It encodes text information through CLAP embedding [6] to achieve guidance. Tango [8] follows the LDM pipeline and replaces the CLAP to T5 [33] for more expressive text embedding. Audio video mutual/joint generation. In addition to text-guided content generation, the mutual or joint generation of audio and video gradually become the focus of research in recent years. Typically, audio and video modalities serve as conditional signals for each other to achieve mutual generation, namely generating audio from video or video from audio. For the former, SpecVQGAN [16] , CondFoleyGen [5] , and Diff-Foley [26] implement audio generation from video using VQGAN, autoregressive transformer, and diffusion model, respectively. Regarding the latter, soundini [22] utilizes audio as a control signal to guide the video diffusion model for video editing purposes. Sung et al. [43] constrain the generated video content from audio to align more closely with the original audio using contrastive learning. TempoTokens [54] introduces an AudioMapper that employs a token encoded by a pre-trained audio encoder as a condition for enabling audio-to-video generation within a diffusion framework.Based on the mutual generation of the two modalities, several studies explore the joint generation of audible video content. MMdiffusion [36] employs a diffusion UNet that takes inputs and outputs from both modalities, enabling the joint generation of two modalities for the first time. Zhu et al. [58] employ a video diffusion architecture to generate video and then retrieve audio, presenting an alternative approach to joint generation. Xing et al. [50] propose augmenting the existing diffusion model with optimization operations during the inference process to achieve audio-video generation while maintaining alignment. Uniqueness of our benchmark. Despite the extensive exploration of multimodal generation tasks, there currently lacks a comprehensive benchmark and large-scale dataset specifically for the text to audible-video generation task. Addressing this gap, our solution offers a dataset for both training and evaluation, alongside metrics for assessing multimodal alignment. Additionally, we provide a straightforward baseline method.Table 1 : Statistics of TAVGBench and other existing video generation datasets. The terms "audio" and "video" represent the modality described by the description in the dataset. The terms "Sentence" and "Word" represent the average number of sentences and words per video annotation, respectively.  
the tavgbench 3.1 dataset statistics: 
 The TAVG task entails the generation of audible videos guided by input text prompts. To support this task, we introduce a benchmark named TAVGBench. Our dataset is sourced from AudioSet [7] , comprising 2 million aligned audio-video pairs obtained from YouTube.After excluding invalid videos, we obtained 1.7 million pieces of original data. Each video sample has a duration of 10 seconds, contributing to a total video duration of 11.8K hours in the dataset. To provide a comprehensive understanding of the scale and characteristics of our dataset, we compare it with the datasets from other related tasks. Table 1 presents a comparative analysis of TAVG-Bench against these datasets in terms of size, source, and other relevant attributes. It can be observed from Table 1 that the AudioCaps [19] , MSR-VTT [52] , and WebVid [1] only describe the content of a single modality (only audio or video modality). Although FAVDBench describes two modalities, the scale of the dataset is limited. The TAVGBench that we propose takes into account descriptions of both audio and video modalities while ensuring a sufficiently large dataset scale. In addition, the videos in WebVid have watermarks, which greatly restrict their application in actual scenarios. This comparison highlights the scale and unique characteristics of the TAVG-Bench dataset, emphasizing its potential for advancing research in audible-video generation. Additionally, TAVGBench exhibits a balanced distribution of textual descriptions, with an average of 2.32 sentences and 49.98 words per video annotation, providing substantial contextual information for each clip. These comparative statistics underscore TAVGBench's extensive scale, multimodal nature, and linguistic richness, positioning it as a valuable resource for advancing research in our TAVG task.Given the absence of detailed text annotations in AudioSet [7] for both its video and audio content, we implement a coarse-tofine pipeline to automate the generation of text descriptions. The complete pipeline is shown in Fig. 2 . Initially, we employ two sophisticated methods, namely BLIP2 [23] for video description and WavCaps [27] for audio description, to annotate the video and audio components, respectively.However, despite the effectiveness of these methods in capturing the essence of video and audio content, the generated annotations often lacked coherence and context. To address this limitation and improve the overall quality of the annotations, we introduced a refinement step using ChatGPT [29] , a powerful language model capable of paraphrasing and enriching textual input.During the refinement phase, we utilize ChatGPT to rephrase and enhance the annotations generated by BLIP2 and WavCaps. By feeding the initial annotations into the ChatGPT model, we obtain revised annotations that exhibit enhanced coherence, contextual relevance, and linguistic refinement. Initially, we individually rephrase the video and audio descriptions to rectify grammatical errors and enhance descriptive content. Subsequently, we employ ChatGPT to amalgamate the descriptions from both modalities into a unified, coherent sentence. This iterative process not only enhances the readability of the annotations but also ensures consistency and accuracy throughout the entire annotation corpus.Incorporating ChatGPT into our pipeline has significantly enhanced the ability to detect subtle nuances and semantic complexities within the video and audio content. As a result, our annotation pipeline excels in understanding context and producing human-like text descriptions, thus facilitating the creation of annotations that more precisely capture the essence of the underlying content.Existing metrics for video (FVD, KVD [45] ) and audio (FAD [36] ) generation primarily focus on the quality of each modality separately. However, for The TAVG task, we not only need to generate high-quality audio and video but also need to ensure that these two modalities are accurately synchronized. To address the necessity for evaluating the alignment degree between generated audio and video, we propose a novel metric called the Audio-Visual Harmony Score (AVHScore). This metric quantifies the alignment of audio-video pairs by calculating the product of the extracted audiovisual features. We use a robust feature extractor (ImageBind [9] ) to project the video frames and the audio into a unified feature space. Formally, the we define the AVHScore 𝑆 AVH as follows:EQUATIONwhere cos denotes cosine similarity. 𝐸 𝑣 and 𝐸 𝑎 represent the vision encoder and audio encoder, respectively, in the ImageBind model. 𝑁 signifies the number of video frames, and we compute the similarity between each video frame and the corresponding audio input, averaging the results across all frames. Original video description A man sitting in front of a microphone in front of a computer screenA man is speaking and breathing with background noiseA man is seated in front of a microphone, facing a computer screen.A man is audibly speaking and taking breaths with background noise.The man is audibly speaking and taking breaths, with background noise audible. Seated in front of a microphone and facing a computer screen, he appears to be engaged in recording or broadcasting activities The man in a suit is standing in front of a wall and delivering a speech, using hand movements to emphasize his points. The audience appears to be engaged and listening attentively to the man's words, as seen in the audio followed by applause.Video AudioThe video shows a truck driving on a street, with smoke visibly coming out of its tires as it moves. The sound of a car revving its engine loudly and screeching its tires can be heard as it accelerates, creating a powerful audiovisual experience.T  
a baseline: 
 We propose a new baseline method for the text to audible-video generation (TAVG) task as shown in Fig. 4 , named TAVDiffusion. The entire network structure is based on the latent diffusion model [34] .The latent diffusion model follows the standard formulation outlined in DDPM [14] , which comprises a forward diffusion process and a backward reverse denoising process. Initially, a data sample x ∼ 𝑝 (x) undergoes processing by an autoencoder, consisting of an encoder E and a decoder D. The autoencoder projects x into a latent variable z via z = E (x). Subsequently, the diffusion and denoising process takes place within the latent space. The denoised latent variable is recovered to the input space by x = D (ẑ 0 ).Inspired by non-equilibrium thermodynamics, diffusion models [14, 40] are a class of latent variable (𝑧 1 , ..., 𝑧 𝑇 ) models of the form 𝑝 𝜃 (𝑧 0 ) = ∫ 𝑝 𝜃 (𝑧 0:𝑇 )𝑑𝑧 1:𝑇 , where the latent variables are of the same dimensionality as the input data 𝑧 0 . The joint distribution 𝑝 𝜃 (𝑧 0:𝑇 ) is also called the reverse process:EQUATIONwhereEQUATIONHere, 𝜇 𝜃 and Σ 𝜃 are determined through a denoiser network 𝜖 𝜃 (𝑧 𝑡 , 𝑡), typically structured as a UNet [35] .The approximate posterior 𝑞(𝑧 1:𝑇 |𝑧 0 ) is called the forward process, which is fixed to a Markov chain that gradually adds noise according to a predefined noise scheduler 𝛽 1:𝑇 :EQUATIONwhereEQUATION)The training is performed by minimizing a variational bound on negative log-likelihood:EQUATIONHence, the final training objective of 𝜃 is a noise estimation loss, with a conditional variable c, it can be formulated as: During the training phase, we randomly select a timestep 𝑡 and employ diffusion loss to guide the single-step denoising. In the inference phase, iterative denoising is conducted to finally produce an audible video.EQUATION)With the forward and reverse process in the latent diffusion model defined in Sec. 4.1, we further present the baseline two-stream diffusion pipeline for joint text to audible-video diffusion. Multimodal Latent encoders. We employ two independent latent autoencoders for our multimodal input to conduct latent space encoding and decoding. This process can be formulated as:EQUATIONwhere the subscripts 𝑎 and 𝑣 represent audio and video modality, respectively. Specifically, for 𝐷 𝑣 , we utilize the pre-trained autoencoder in stable diffusion [34] along with its weights. Given an original video with dimensions 𝑇 × 3 × 𝐻 𝑣 × 𝑊 𝑣 , the size of the latent variable becomes 𝑇 × 4 × 𝐻 𝑣 8 × 𝑊 𝑣 8 . As for 𝐷 𝑎 , we employ the pre-trained autoencoder from AudioLDM [25] to encode the audio mel spectrogram into the latent space. For the dimensions 1 × 𝐻 𝑎 × 𝑊 𝑎 of the input mel spectrogram, the size of the latent variable becomes 8 × 𝐻 𝑎 8 × 𝑊 𝑎 8 . Multimodal diffusion process. For the input of audio and video modalities, we use a two-stream structure to perform the forward and reverse diffusion process for the latent variables z 𝑎 , z 𝑣 , as shown in Fig. 4 . Unlike vanilla diffusion where a single modality is generated, we aim to simultaneously recover two consistent modalities (i.e. audio and video) within a single diffusion process.We consider that the reverse and forward processes of each modality are independent because they have distinct distributions. Taking the audio latent variable z 𝑎 as an illustration, its reverse process at timestep 𝑡 is defined as:EQUATIONThe forward process at timestep 𝑡 is defined as follows:EQUATION)For brevity, we omit the reverse and forward process for video z 𝑣 via 𝜃 𝑣 , as it shares a similar formulation. It is important to note that we empirically set a shared schedule for hyper-parameters 𝛽 across audio and video to streamline the process definition.Summarizing the formulations above, the final definition of the multimodal diffusion loss is:EQUATION)In our task, the conditional variable c denotes the text embedding of the input, and we use the CLIP [32] text encoder with its tokenizer to obtain the text embedding. Diffusion UNet architecture. For 𝜃 𝑎 and 𝜃 𝑣 , we employ two parallel UNet branches to perform denoising for audio and video latent variables, respectively. For the audio UNet 𝜃 𝑎 , we directly utilize the UNet architecture from stable diffusion [34] and adjust the number of channels in its input layer from 4 to 8. Regarding the video UNet 𝜃 𝑣 , akin to AnimateDiff [10] , we adapt the 2D convolution in the 2D UNet of stable diffusion1 to a pseudo-3D convolution, which comprises a 1D convolution followed by a 2D convolution. Additionally, we incorporate the Temporal Transformer layer [46] into the network. These changes allow the network to accept video as input and keep its temporal details.In this section, we introduce a methodology for aligning the intermediate features 𝑓 𝑎 , 𝑓 𝑣 , associated with 𝜃 𝑎 , 𝜃 𝑣 respectively, by employing a cross-attention mechanism [46] . Our approach aims to enhance the performance of joint generation tasks involving video and audio by synchronizing the information flow between the two modalities during the diffusion process. Given 𝑓 𝑎 and 𝑓 𝑣 , we represent the process of cross-attention as follows and utilize the resulting f 𝑎 and f 𝑣 as the input of the diffusion UNet decoder.EQUATION)The feature interaction mechanism does not explicitly enforce the alignment of multimodal features. Hence, it is crucial to integrate a loss function that guarantees alignment between the feature representations of audio and visual modalities. To tackle this issue, we propose an Explicit audio-visual Alignment Strategy (EAS) based on contrastive learning [11] .In a multimodal scenario, where we have feature vectors ( f 𝑎 , f 𝑣 ) representing two modalities, contrastive learning is employed to model the alignment between these modalities. The underlying assumption is that features that align well with each other should be brought closer together, whereas those that contradict each other should be pushed apart. We define alignment as a bidirectional process, leading to the formulation of the contrastive loss as follows:EQUATIONwhere 𝜏 = 0.1 is the temperature scalar, f 𝑝 𝑎 ( f 𝑝 𝑣 ) is the positive sample corresponding to f 𝑎 ( f 𝑣 ) from the other modality. The function s(•, •) computes the cosine similarity. We generate positive and negative samples based on the feature representations of two modalities. Specifically, we define embeddings of the two modalities with the same index (paired data) as positive samples, while those with different indexes are considered negative samples (unpaired data).The bottleneck of contrastive learning lies in designing the positive/negative pairs with effective similarity measure, i.e. s(•, •) in our case. We use a linear projection with softmax activation 𝑙 𝛽 (•) to calculate similarity weight based on the input of a specific modality [4] different information contained in different tokens. Given two modalities (𝑎, 𝑣), a weighted similarity function s(•, •) is:EQUATION)The objective functions are divided into task loss (L diffusion ) and feature alignment loss (L EAS ). The total loss is a weighted sum of the above terms:EQUATIONwhere 𝜆 represents the balanced weights during training. Empirically, the loss weight is set as 𝜆 = 0.1. 
implementation details: 
 Datasets. We train our model on the TAVGBench dataset that we introduced. For the evaluation phase, we select 3,000 samples from the TAVGBench evaluation subset. Additionally, we evaluate the performance of our model on the test subset of FAVDBench [38] , which comprises 1,000 samples. The FAVDBench provides more fine-grained audible video descriptions, enabling the generation of more detailed videos. Importantly, because the data of FAVDBench is not utilized in the training phase, we could assess the zero-shot capabilities of our model based on its performance on FAVDBench.Training Details. We implement TAVDiffusion using PyTorch [30] .We adopt a video training resolution of 256 × 320 × 20 to balance training efficiency and motion quality. We convert the audio into a mel spectrogram and perform training and inference on it. We use the MelGAN Vocoder [21] to convert the denoised audio mel spectrogram into the audio waveform. We use a learning rate of 1×10 -4 and train the model with 64 NVIDIA A100 GPUs for 1.0×10 5 steps. At inference time, we apply the DDIM scheduler [41] and only sample 50 timesteps. Evaluation Metrics. We begin by separately measuring the quality of the generated audio and video. To evaluate the video, we adopt the Frechet Video Distance (FVD) [45] , Kernel Video Distance (KVD) [44] , and CLIPSIM [12, 48] metrics. FVD and KVD employ the I3D [3] classifier pre-trained on the Kinetics-400 dataset [17] .For audio evaluation, we employ FAD [36] to gauge the distance between the features of the generated audio and the reference audio. We also use our proposed AVHScore to measure the alignment degree of the generated results. For all evaluations, we generate a random sample for each text without any automatic ranking.Comparison methods setting. To the best of our knowledge, there are no existing usable methods directly relevant to our proposed task for comparison. Therefore, we combine existing related models and design two-stage methods for comparison.(1) AnimateDiff [10] +AudioLDM [25] : Input text and utilize these two models to generate audio and video respectively. (2) AnimateDiff [10] +Diff-Foley [26] : Input text, employ Animate-Diff to generate video, and subsequently utilize Diff-Foley to generate audio based on the video. (3) AudioLDM [25] +TempoToken [54] : Input text, use AudioLDM to generate audio, and then employ TempoToken to generate video based on the audio. Quantitative comparison. We present the quantitative results of our method alongside comparison methods for the TAVGBench and FAVDBench datasets in Table 2 . The results demonstrate that our TAVDiffusion model outperforms all comparisons in terms of video and audio quality metrics. Specifically, the scores for FVD and KVD are 776.25 and 65.53, respectively, while the FAD score is 1.46. This demonstrates significant consistency between the audible video we generated and the original content and is of higher quality. These results underscore a significant consistency between the videos generated by our model and the original content, indicating superior quality. Additionally, our model achieves a notable CLIPSIM score (24.18) , reinforcing the semantic coherence between the generated A pristine sky with a few white clouds, white snow-capped mountains in the distance and the sound of birds chirping in the background.As music plays in the background, a group of people can be seen chatting and applauding in the video, which captures various scenes and attractions from Disneyland. Visitors are shown enjoying rides, meeting characters, and exploring different themed lands throughout the park.A handsome man wearing a blue coat is playing the guitar in the house, showcasing his skill and rhythm control. He seemed to be attracted by the singing and seemed very intoxicated.The white ambulance is traveling quickly. buildings and parked automobiles line the road. alongside the road are lush trees. sound of an ambulance siren. 2). Given the inferior visual quality of method (3) in our task (see Table 2 ), we exclude it from the qualitative comparison. Best viewed on screen. Qualitative comparison. In Fig. 5 , we present qualitative results comparing our method to the compared generators. The figure illustrates that TAVDiffusion outperforms the comparison models in terms of visual fidelity and the alignment of text, video, and audio. In the first example, the "performer" created by TAVDiffusion displays significantly enhanced realism, especially in facial expressions and hand movements. The generated audio also follows the two elements "guitar sound" and "singing voice" in the prompt. In the second example, TAVDiffusion showcases its ability to produce complex real-life scenes, maintaining the precise shapes of essential objects. It skillfully navigates the dynamics between the foreground object (e.g. the car) and the background scene, complemented by realistic audio. We also showcase our model's performance in two distinct scenarios: environments with significant background noise and notably quieter environments. For the former, our model generates various types of audio, such as music and human cheers, as directed by the prompt. For the latter, our model uniquely and accurately produces the sound of "birds chirping". This comparison shows the versatility of the model, demonstrating its effectiveness across a broad spectrum of audio-visual scenes. By evaluating the model in such contrasting settings, we highlight its universal applicability and robustness in processing diverse auditory and visual inputs. We sincerely hope the readers find more examples of audible videos in the supplementary materials.To demonstrate the effectiveness of our proposed modules, we conduct ablation studies from both quantitative metrics (see Table 3 ) and qualitative visualizations (see Fig. 6 ). In Table 3 , it is evident that the two strategies we proposed, multimodal cross-attention and multimodal alignment, enhance both the quality of video and audio generation and the alignment score. In Fig. 6 , we can obverse that the "wolf" produced by our final model appears more realistic than that of the comparisons, with its mouth movements accurately reflecting the "kept barking" prompt and the generated audio. Please refer to the supplementary material for more samples.Our TAVGBench and baseline model TAVDiffusion have a wide range of multimedia application fields. Our dataset, which includes large-scale video, audio, and corresponding text descriptions, is well-suited for a diverse range of multimodal tasks. It allows for the simultaneous use of text and audio as prompts to generate videos. Additionally, TAVGBench can be used to train audible video captioning models [38, 57] can substantially reduce the impact of insufficient audio-video-text data pairs as mentioned in [38] . 
conclusion: 
 We explored the challenge of creating videos with matching audio from text descriptions, a task known as Text to Audible-Video Generation (TAVG). To aid in this research, we introduced a new benchmark called TAVGBench, filled with over 1.7 million video clips. This resource is designed to help improve and evaluate TAVG models. We developed a method to automatically describe each audio-visual element, ensuring detailed and useful annotations for researchers. A new metric called Audio-Visual Harmoni score (AVH-Score) was designed to evaluate the alignment of generated audible video. We introduced TAVDiffusion, a baseline model leveraging latent diffusion. This model incorporates cross-attention and contrastive learning mechanisms to achieve audio-visual alignment within the diffusion UNet framework. Extensive experimental results verified the effectiveness of our proposed framework, thus opening new avenues for multimedia content creation. In the future, we aim to explore a multimodal diffusion transformer [31] to facilitate audible video generation through a unified architecture. 
 
 Appendix: We use stable diffusion version 1.5 and its pre-trained weights as partial initialization. 


Decision: The authors have developed the Text to Audible-Video Generation Benchmark (TAVGBench), a substantial resource featuring over 1.7 million clips and 11,800 total hours of content, which is valuable for future research on Text to Audible-Video Generation. 

However, the proposed annotation pipeline, even with the quality control discussed in the rebuttal, may result in inaccuracies in the annotations, which need to be discussed and/or further addressed. To verify the developed dataset, the authors proposed a model called TACDiffusion. Regarding the comparison with the SOTA, the comparison may be extended to include the latest video captioning models.

Reviews: review: paper_summary: paper_summary: This paper focuses on the Text to Audible-Video Generation (TAVG) task, which involves creating videos with synchronized audio from textual descriptions. This task necessitates precise alignment of audio and video outputs to ensure a cohesive viewing experience. To facilitate research in this area, the authors have developed the Text to Audible-Video Generation Benchmark (TAVGBench), a substantial resource featuring over 1.7 million clips and 11,800 total hours of content. The authors also propose a baseline model —TAVDiffusion to verify the collected dataset.
strength_weakness: strength_weakness: This paper proposes a large-scale TAVG dataset with 1.7M Clips, which is valuable for future research T2AV generation.
The paper is written clearly, and the dataset description contains comprehensive details.
limitations: limitations: Lacking the quality analysis of automatic generated captions.
The discussion of potential applications is somewhat limited in scope, which raises doubts about the broader impact and applicability to downstream tasks.
The cases included in the paper contains a lot of artifacts which compromise the effectiveness of the proposed TAVDiffusion model.
paper_summary: paper_summary: The paper describes a benchmark dataset designed to generate videos and their corresponding audio from text prompts. The creation of the dataset is fully automated, with 1.7 million clips available. The author further trains a baseline diffusion model with audio-video cross-attention and contrastive learning, which can generate both streams. The author proposes a new evaluation metric to measure the alignment of audio and video and conducts initial experiments on this new benchmark.
strength_weakness: strength_weakness: 1. The author introduces a relatively new task that involves generating a video and its corresponding audio using the same multimodal generation model. This is intriguing because future multimodal generation should be capable of producing both streams to ensure coherence and internal alignment.

2. The author then constructs a large-scale, automatically labeled dataset to facilitate the training of the model.

3. The author suggests a baseline multimodal diffusion model that begins with the same text prompt, generates both streams simultaneously, and uses a cross-attention alignment loss of the latent feature to aid in producing more coherent results. This could serve as a solid starting point model.
limitations: limitations: 1. I have concerns about the quality of the data. Given that the process is fully automated, the data quality is determined by the audio caption and video caption models, both of which are separate and can introduce their own inaccuracies and hallucinations. The generated prompts from both models are then fed into ChatGPT to integrate them and correct grammatical errors, among other things. It's crucial to note that ChatGPT lacks multimodal capabilities, and the audio/video features are not connected to the LLM in any way other than the captions produced by the two separate models. At best, the LLM can reduce grammatical errors and enhance text readability, but it cannot address any hallucination issues arising from errors in the audio/video models. Thus the data should be treated as weak label/noisy data.

2. Furthermore, the author uses samples from the dataset as an evaluation benchmark, with no mention of any filtering or quality control. Regardless of the model's score on its test set, a model trained/evaluated on this dataset will likely overfit to the errors introduced by the caption models and cannot serve as a reliable benchmark.

3. Regarding the baseline model analysis, it's not clear of the settings of the baseline model. Are these model trained under the same constructed dataset, or just used in evaluation only? Since the training/eval dataset has the similar distribution, it's not fair to compare those model in the eval set.
paper_summary: paper_summary: The authors have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. They propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. Additionally, they introduce the Audio-Visual Harmoni Score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Furthermore, they present a baseline model for TAVGBench called TAVDiffusion, which uses a two-stream latent diffusion model as a fundamental starting point for further research in this area.
strength_weakness: strength_weakness: TAVGBench introduces a new benchmark for the Text to Audible-Video Generation (TAVG) task. This task is an innovative extension of traditional text-to-video generation by integrating synchronized audio components. This benchmark fills a crucial gap in current research, which predominantly focuses on unimodal video generation.
The dataset includes over 1.7 million clips with a total duration of 11.8 thousand hours, making it one of the largest datasets for this type of multimodal generation task. The size and diversity of the dataset ensure that models trained on it can generalize well to a wide range of real-world scenarios.
The paper is well-structured and clearly presents the problem, methodology, and results. The use of figures, such as the comparison of TAVG with existing tasks and the detailed annotation pipeline, helps in understanding the complexities and innovations introduced.
limitations: limitations: The automatic annotation pipeline, though efficient, may still result in inaccuracies or inconsistencies in the annotations. While using models like BLIP2, WavCaps, and ChatGPT improves quality, human oversight and correction might still be necessary to ensure the highest accuracy and relevance of annotations.
The paper introduces TAVDiffusion as a baseline model, which is a strong starting point. However, having only one baseline model may not provide a comprehensive understanding of the task. Including multiple baseline models or comparing with existing state-of-the-art models in related fields (e.g., text-to-video or text-to-speech) would provide a more robust evaluation framework.
There is a lack of comparison with the results of current advanced methods, and the visualization results lack complex scenes and action changes.
paper_summary: paper_summary: The Text to Audible-Video Generation (TAVG) task entails creating videos with synchronized audio from text descriptions, which demands careful coordination of audio and video components. This paper develops a comprehensive Text to Audible-Video Generation Benchmark to support the Text to Audible-Video Generation task. This paper also proposes an automatic annotation pipeline to ensure each audible video has detailed descriptions
and introduces the Audio-Visual Harmoni score to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, this paper presents a baseline model named TAVDiffusion for TAVG.
strength_weakness: strength_weakness: 1.	This paper introduces a large-scale Text to Audible-Video Generation Benchmark, which contains over 1.7 million clips with a total duration of 11.8 thousand hours. It can advance the research of the Text to Audible-Video Generation task.
2.	This paper designs an automatic annotation pipeline to save costs for TAVG.
3.	This paper proposes a diffusion-based model and a quantitative metric for TAVG.
limitations: limitations: 1.	The proposed automatic annotation pipeline is based on powerful open-source models. It is easy to follow. However, due to the large volume of video data, even automated annotation requires significant computational resources. Please provide clear information on the total annotation time and costs for the dataset in Section 3.2.
2.	Open-sourcing the entire dataset and related code would be more beneficial to the field.
3.	BLIP2 is capable of performing image captioning rather than video captioning tasks, so I'm curious how the paper utilizes BLIP2 to generate captions for a video.
I have two hypotheses. The first is that the features of multiple image frames are simply stacked.
In the original BLIP2,  a 224x224 resolution image correspond to one cls token and 256 patch tokens, which interact with 32 learnable tokens and image tokens. However, if tokens from different frames are merely stacked together, wouldn’t this approach neglect the frame-wise temporal modeling? Also, would using the default 32 tokens for videos result in information loss?
The second hypothesis is that a random frame is selected from a clip. However, for a clipped clip, the frames are still changing, and I wonder if randomly selecting one frame is reasonable?
Please provide additional explanation in section 3.2 Annotation details, and explain why a pretrained video caption model is not used.