TiVA: Time-Aligned Video-to-Audio Generation

Authors: xihua  wang
yuyue  wang
guodong 2024 sui
yihan  wu
ruihua  song
xu  tan
zehua  chen
hongteng  xu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 09:38:47

Abstract: Video-to-audio generation is crucial for autonomous video editing and post-processing, which aims to generate high-quality audio for silent videos with semantic similarity and temporal synchronization. However, most existing methods mainly focus on matching the semantics of the visual and acoustic modalities while merely considering their temporal alignment in a coarse granularity, thus failing to achieve precise synchronization. In this study, we propose a novel time-aligned video-to-audio framework, called TiVA, to achieve semantic matching and temporal synchronization jointly when generating audio. Given a silent video, our method encodes its visual semantics and predicts an audio layout separately. Then, leveraging the semantic latent embeddings and the predicted audio layout as condition, it learns a latent diffusion-based audio generator. Comprehensive objective and subjective experiments demonstrate that our method consistently outperforms state-of-the-art methods on semantic matching and temporal synchronization.

Main paper: 
 introduction: 
 Video-to-audio (V2A) generation aims to generate synchronized and realistic audio solely based on silent videos [1] . This task is significant for many practical applications, including autonomous video editing and post-processing. Especially, achieving high-quality V2A is necessary for the current AI-generated videos because of their silent nature, e.g., Sora generated videos [4] .As a main challenge of V2A generation, given a silent video, we need to not only ensure the semantics of the generated audio to match well with the visual content but also make the audio temporally synchronize with the video. However, most existing V2A generation methods mainly focus on matching the semantics of the visual and acoustic modalities while seldom considering their temporal alignment. In particular, they often follow the learning paradigms of existing text-to-audio methods [14, 15, 21] , applying contrastive learning-based models to encode video/images as conditions and then generating Mel-spectrograms [25] or audio tokens [16, 29] via diffusion or auto-regressive models, respectively. Some recent methods, like DiffFoley [25] , maximize the similarity of audio-visual pairs from the same time segment and minimize the similarity of those across different time segments. However, this temporal contrastive learning only aligns the generated audio with the input video in a coarse granularity, failing to achieve precise temporal synchronization.In this study, we propose a novel time-aligned V2A generation method, called TiVA, to achieve semantic matching and precise temporal synchronization jointly when generating audio. In principle, our method is based on the fact that the low-resolution Melspectrogram of an audio provides an effective audio layout for temporal synchronization. Take Figure 1 as an example. Given a video of cat, in which a cat opens its mouth wide when hissing and more narrowly when meowing, both the ground-truth waveform and its corresponding Mel-spectrogram are highly synchronized with the dynamics of the visual content. By reducing the resolution of the Mel-spectrogram, we obtain an audio layout, as shown in Figure 1 (d) . On one hand the layout provides precise alignment [25] .between the mouth opening and making sounds three times; on the other hand the frequency difference of hissing and meowing can also be easily distinguished by the shapes of bright areas in the layout. Thus the audio layout can be used to supervise the learning of the proposed model. In particular, in the training phase, we learn a conditional diffusion-based audio generator and an associated audio layout predictor jointly. The audio layout predictor predicts the low-resolution Mel-spectrogram based on the input video. Taking the real audio layout and the semantical encoding result of the video as conditions, the audio generator generates a prior of the audio in a latent space. In the testing phase, the predictor estimates the audio layout, which provides useful temporal condition for the learned audio generator. As shown in Figure 1 (e, f), compared to the state-of-the-art V2A method [25] , our TiVA can generate an audio from the input video, with better temporal synchronization.There are three main contributions of our work:• We propose using low-resolution Mel-spectrograms of groundtruth audio to self-supervise the prediction of an audio layout, which bridges temporal encoding and audio generation. Experimental results indicate that it can well capture temporal alignment information and be complementary to semantic information in controlling audio generation. Audio generation exploits various condition information: (1) Sole semantic condition, typically extracted using pre-trained models (e.g., FLAN-T5 [6] , ResNet [11] , VideoMAE [30] , CLIP [27] , CLAP [31] ) for semantic guidance in audio generation [15, 20, 21, 26, 32] . (2) Enriched semantic condition. Make-an-Audio-2 [14] and WavJourney [23] leverage large language models to augment descriptions of audio, while DiffFoley [25] injects temporal dynamics into semantic condition through advanced temporal contrastive learning.(3) Additional information condition. Beyond semantics, SpecVQ-GAN [16] and SyncFusion [7] improve audio quality by employing additional video flow and onset information respectively. Different from them, our work proposes a novel audio layout condition, which captures temporal alignment information and complements semantic information in controlling audio generation. 
generative modeling for audio generation: 
 With encoded conditions from input modalities, previous studies on audio generation generally fall into two streams: (1) Autoregressive methods, like SpecVQGAN [16] , AudioGen [20] , and IM2WAV [29] , employ transformer-based architectures for sequential audio generation, yielding high-quality output but at the expense of computational time.(2) Non-autoregressive methods, like SoundStorm [3] and MagNet [34] , adopt multi-step non-autoregressive designs for more efficient generation, but with lesser audio quality compared to autoregressive methods. Recent advancements incorporate diffusion models, with DiffSound [32] adopting discrete diffusion and works like DiffFoley [25] and the AudioLDM [21, 22] and Makean-audio [14, 15] series employing latent diffusion models (LDM), achieving enhanced audio quality and generation speed through well-designed samplers [24] . Our study further advances LDMs by integrating both semantic embeddings and a novel audio layout as conditions. We also introduce effective strategies for optimizing the two condition encoders/predictors and the audio generator.Previous research [16, 21, 29, 32] assesses the quality of generated audio by comparing it with the ground truth using automatic metrics such as Fréchet Inception Distance (FID) [16] , mean KL divergence (MKL) [16] , and Inception Score (IS) [16] , derived from feature distributions of audio classifiers (Melception [16] , PANNs [19] , VG-Gish [12] ). FID measures dataset-level consistency, MKL quantifies pair-level variation, and IS estimates diversity via audio distribution entropy. Moreover, FoleyGen [26] employs ImageBind score (IB) to measure semantic relevance between generated audio and input video based on features extracted by the pre-trained multi-modal alignment model ImageBind [10] . Recent studies have extended the evaluation to measure the synchronization between generated audio and input video. For instance, DiffFoley [25] applies a binary classifier, while CondFoleyGen [8] employs a multi-class classifier [17] to predict the temporal offset (TO) between an audio-video pair. SyncFusion [7] implements synthesized onset accuracy (Onset Acc) and average precision score (Onset Sync AP) via predicting binary onset signal. However, these onset signal-based metrics may not adequately capture the fine-grained details across the entire temporal distribution. To address this limitation, we propose two fine-grained synchronization evaluation metrics based on the more detailed onset detection function (ODF) curves in this paper. 
method: 
 Video-to-audio (V2A) generation aims to synthesize an audio track 𝑎 from a silent video 𝑣 comprising 𝑁 frames 𝑣 : [𝑓 1 , ..., 𝑓 𝑖 , ..., 𝑓 𝑁 ].Ideally, the generated audio 𝑎 should precisely align with the visual content of 𝑣, both semantically and temporally. Previous works have achieved coarse alignment between the generated audio and input visual information primarily through semantic guidance and control, e.g., accurately generating multiple cat sounds rather than a single dog sound for the video depicted in Figure 1 (a, f ). However, it remains a challenge for these methods to achieve a fine-grained alignment, which includes more intricate semantic sound structures such as the distinction between different sounds produced by the same entity, e.g., a cat's hiss versus its meow, and precise temporal information, e.g., the exact start and end time of a sound. To address the above challenge of precise alignment, we adopt a coarse-to-fine principle. Given the difficulty in directly achieving precise alignment in one step, as evidenced by previous work, our approach initially predicts a simpler intermediate state (lowresolution Mel-spectrograms in our method) before generating the final output (high-resolution raw Mel-spectrograms). We propose defining this low-resolution Mel-spectrogram as a new control signal, termed audio layout, serving as the bridge in video-to-audio generation, as exemplified in Figure 1 (d) and Figure 2 (e) . Then, we propose a new V2A framework, named TiVA, which first produces semantic embeddings and this audio layout, and then utilizes both as condition to control the audio generation, as shown in Figure 3 (a). We will introduce the audio layout, the architecture of TiVA, and the training of TiVA in detail in the following subsections.In the domain of 2D image generation, precise control is typically achieved by 2D layout inputs [33] . As audio is often represented in a 2D Mel-spectrogram format (with time and frequency as its axes), akin to images, we propose the use of a coarse-grained Melspectrogram as an audio layout in the audio domain, denoted as 𝒕. This proposed audio layout is simpler for models to predict compared to raw Mel-spectrograms. Additionally, it can provide more detailed control signals, such as the sound structure and the start and end of a sound, than the Onset signals or ODF curves (extracted from audio waveforms), as illustrated in Figure 2 . Experimental results, detailed in Section 4.3, confirm the superior effectiveness of the proposed audio layout in Mel-spectrograms over raw Melspectrograms target or Onset and ODF forms for V2A tasks.Specifically, a ground truth audio layout 𝒕 𝑎 is generated by downscaling a Mel-spectrogram of dimensions [𝑀 raw , 𝐶 raw ] to a target resolution [𝑀, 𝐶]. Here, 𝑀 raw and 𝐶 raw denote the original number of frames and frequency dimensions, respectively, while 𝑀 and 𝐶 denote the reduced temporal length and frequency dimensions:𝒕 𝑎 = Resize(Norm(mel 𝑎 ); 𝑀, 𝐶).(EQUATIONThis simple but efficient extraction facilitates a self-supervised learning manner, that captures the temporal structure pattern without requiring additional audio annotations. The extracted audio layout in size [𝑀, 𝐶] is further resized to match the latent space dimensions [ℎ, 𝑤] for integration into the latent diffusion model.TiVA integrates a semantic encoder S, a layout predictor T , and a conditional generator G. As illustrated in Figure 3 (a), TiVA extracts semantic embeddings 𝒔 and predicts an audio layout 𝒕 𝑣 from the input video frames [𝑓 1 , ..., 𝑓 𝑖 , ..., 𝑓 𝑁 ] through S and T , respectively. These conditions are then employed by G to guide the generation of an audio within a latent space. The following contents delineate the encoding and generative processes of TiVA.TiVA's Condition Encoders. The semantic encoder and layout predictor transform the video frames into the semantic embeddings 𝒔 and the temporal audio layouts 𝒕 𝑣 . This begins with the extraction of multi-layer features 𝒄 𝑙 𝑖 via CLIP [27] , where 𝒄 𝑙 𝑖 denotes the CLS and patch features from layer 𝑙 of CLIP for frame 𝑓 𝑖 . These features are then utilized to produce the corresponding semantic and temporal conditions 𝒔 and 𝒕 𝑣 , as illustrated in Figure 3 (b) . Semantic Encoder S employs a linear layer to map the CLS features of CLIP's 11th layer into semantic embeddings for each frame:𝒔 𝑖 = Linear(CLIP(𝑓 𝑖 )).(EQUATIONThe 𝒔 𝑖 from each frame collectively form the semantic condition 𝒔. Layout Predictor T employs an encoder-decoder structure. The encoder first concatenates and pools patch features from selected CLIP layers for each frame, serving as encoder inputs, denoted as 𝒄 patch 𝑖. These features are then processed through windowed selfattention layers, facilitating interactions among adjacent frames:𝒄 patch 𝑖 = Pooling(Concatenate([𝒄 4,patch 𝑖 , 𝒄 8,patch 𝑖 , 𝒄 11,patch 𝑖 ])), 𝒄 patch 𝑖 ′ = Window-Encoder(𝒄 patch 𝑖).(3)The decoder starts by combining CLS features and learnable query tokens 𝒒 𝑖 as inputs 𝒄 query 𝑖 , and is followed by multiple decoder EQUATIONwhere ᾱ𝑡 = 𝑡 𝑖=1 𝛼 𝑖 . Consequently, the LDM's diffusion model, denoted as G, is trained to predict the noise at each time step 𝑡: EQUATIONOur generator G utilizes a Transformer-UNet (T-UNet) structure [22] , integrating convolutional, self-attention, and cross-attention layers.The appended latents z𝑡 𝑎 are fed into the T-UNet as input, with the semantic embedding 𝒔 incorporated into the cross-attention layers. It's noted that we only take the first 𝐶 dimensions from the 𝐶 + 1 dimensions of G's direct output as estimated noise ε𝑡 for each step.In the training phase, given a video with sound, comprising silent frames 𝑣 : [𝑓 1 , ..., 𝑓 𝑖 , ..., 𝑓 𝑁 ] and the ground-truth audio 𝑎, we separately optimize the layout predictor T , the semantic encoder S and the generator G, before a joint optimization stage.The layout predictor is trained to predict the audio layout 𝒕 𝑣 from video frames, using features from CLIP as input. This training is guided by the ground-truth audio layout 𝒕 𝑎 , derived from the audio 𝑎's Mel-spectrogram, and and aims to minimize the L2 loss:EQUATIONThe semantic encoder S processes frames to produce the semantic embedding 𝒔, which, along with the ground-truth 𝒕 𝑎 , serves as conditions for the generator G. The generator G predicts the noise sequence to reverse the diffusion process, iteratively denoising 𝒛 𝑡 𝑎 to reconstruct 𝒛 0 𝑎 . The optimization of S and G is driven by an L2 loss on top of G's output, targeting the noise prediction:EQUATIONwhere during training, 10% of the semantic embeddings 𝒔 are randomly zeroed to promote unconditional generation capabilities. Independent optimization of T and G leads to suboptimal performance. This is because G, trained on the precise ground-truth 𝒕 𝑎 is then applied with the predicted 𝒕 𝑣 during inference. Due to G's acute sensitivity to deviations, it tends to magnify even minor discrepancies between 𝒕 𝑎 and 𝒕 𝑣 . Empirically, we find this gap significantly impacts generation quality. To address this, we introduce a joint optimization stage in addition to the initial separate optimization stage. T first predicts 𝒕 𝑣 from the video, and then G takes it as input for the audio layout generation. The learning objective is to optimize the L2 loss on G's prediction to achieve a joint optimization of all components G, T , and S:EQUATIONwhere 10% of 𝒔, i.e., S(𝑣), is set to zero in this joint optimization. Video frames and audio are sampled at 10 FPS and 16 kHz, respectively, followed by preprocessing according to the protocols of CLIP [27] and AudioLDM2 [22] . The size of the preprocessed Mel-spectrogram is [1024, 64], and the size of our audio layout 𝒕 𝑎 ∈ R 𝑀 ×𝐶 is set to [102, 5] . During the training stage, the CLIP, the VAE and vocoder (sourced from AudioLDM2 [22] ) are frozen. The training of latent diffusion model is governed by a variancepreserving DDPM scheduler [13] , with the semantic condition 𝒔 being dropped at a probability of 0.1. For inference, we employ the second-order DPM-Solver sampler [24] and apply a guidance scale of 5.0 to steer the generation. We set the inference step as 25.4.1.2 Baselines. We collect or implement the following baselines that can address the V2A task on VGGSound dataset: For consistency with DiffFoley, we use the first 8s from SpecVQ-GAN, DiffSound-V, and TiVA outputs. For IM2WAV, we concatenate two 4s samples corresponding to the 0-4s and 4-8s clips of the video. 
evaluation metrics.: 
 For objective evaluation, we use the following groups of metrics:• Quality metrics. We employ automatic metrics Fréchet Audio Distance (FAD) [18] , Inception Score (IS), and KL-divergence (KL), each with a classifier name for specificity. The practice is in line with previous work [16, 21, 25, 29] .Onset Sync AP rely on binary onset. However, as depicted in Figure 2 , Onset is a sparse, discrete signal. To achieve a more fine-grained evaluation of synchronization, we adopt Onset Detection Function (ODF) [2] curves, which capture sound energy variations over time. Audio-video synchronization is assessed through a comparison of ODF curves of generated and reference audio. We apply the Dynamic Time Warping Distance2 (DTW-dis) and Wasserstein Distance3 (W-dis) to measure their distance: (a) DTW-dis first employs a dynamic programming algorithm to optimally align two time sequences, allowing the one-to-many cases. Subsequently, it distorts the two sequences according to the alignment and calculates point-by-point Euclidean distance. (b) W-dis regards two distributions 𝑃 and 𝑄 as a series of mounds and evaluates the minimum cost to transform 𝑃 into 𝑄, considering both the distance between mounds and the mound mass. In our evaluation, DTW-dis and W-dis view ODFs as time series and one-dimensional distributions, respectively. Lower distance implies better synchronization. Supplementary analysis reveals that DTW-dis and W-dis correlate well with human judgment, outperforming previous metrics: Onset Acc [7] , Onset Sync AP [7] ,Align Acc [25] , and Temporal Offset (TO) [8] . • Semantic metrics. We use ImageBind score (IB) [10] to assess semantic relevance between input videos and generated audios. • Efficiency metrics. We computed the average inference time for each sample across ten batches, each containing 100 samples, excluding the time attributed to the vocoder module. We benchmark our proposed TiVA against existing V2A methods on VGGSound dataset, following established protocols. As shown in Table 1 , TiVA outperforms existing methods, achieving leading scores in FAD, KL (pas.), IS, W-dis, DTW-dis, and IB metrics, and ranks second in KL (mel.). Notably, TiVA's design enhances not only temporal synchronization but also generation quality and semantic relevance. This is caused by that the 2D audio layout provides a coarse prior of target audio that helps the generation of relevant sounds. Moreover, the results indicate that TiVA has a substantial leap in computational efficiency, accelerating audio generation by approximately 40% compared to baselines. This may be attributed to the well-designed architecture, in particular the sharing of CLIP features between the semantic encoder and audio layout predictor.For subjective assessment, we ramdonly select 50 videos from VGGSound test set. Human evaluators are asked to rate audio samples from different models from four perspectives: Overall quality (Overall), Sound Quality (SoundQua), Semantic Relevance (SemRel), and Synchronization Score (SyncScore) on a 5-level Likert scale. Each model's audio output is rated by ten evaluators, and their scores are averaged and assessed for a 95% confidence interval. According to these subjective ratings presented in Table 2 , TiVA consistently outperforms all baselines by a substantial margin. The IM2WAV model ranked the second, surpassing the other two baselines, SpecVQGAN and DiffFoley, in every evaluative criterion 4 .For qualitative evaluation, Figure 4 presents a series of cases in the in-domain VGGSound test set. Case 1 and 3 show that the audio generated by our TiVA model can better temporally align with the visual content or the ground-truth audio than baselines. In Case 1, DiffFoley exhibits delays for the second cat's hiss and the final meow, whereas IM2WAV delays the last meow. Both of them two miss certain sounds in Case 3. Case 2 and Case 3 demonstrate 4 Visit our demo page for playable examples: https://tiva2024.github.io/TiVA.github.io/ Table 2: Subjective evaluation results for different models. A rating ranges from 0 to 5, the higher the better.Overall ↑ SoundQua ↑ SemRel ↑ SyncScore ↑ SpecVQGAN [16] 1.10 ± 0.09 1.32 ± 0.10 1.14 ± 0.12 1.09 ± 0.12 DiffSound[32]-V 2.01 ± 0.08 2.18 ± 0.09 2.03 ± 0.11 1.88 ± 0.11 DiffFoley [25] 1.98 ± 0.10 2.01 ± 0.10 2.03 ± 0.13 1.97 ± 0.14 IM2WAV [29] 2.17 ± 0.07 2.27 ± 0.09 2.29 ± 0.11 2.01 ± 0.10 TiVA 2.74 ± 0.11 3.12 ± 0.12 2.81 ± 0.12 2.47 ± 0.13that TiVA can generate more semantically similar sound according to the shape and texture of bright areas in the Mel-spectrogram. For example, in Case 3, TiVA generates a 'triangular' bright area in the Mel-spectrogram that closely resemble the ground-truth, in contrast to the 'rectangular' pattern produced by DiffFoley. There are similar vertical bars in TiVA generated and ground-truth audio, whereas the baseline generates horizontal bars. In addition, we observe that although the predicted audio layouts are coarse in resolution, they contains important temporal, shape and texture information, contributing to the enhanced generation quality and temporal synchronization of our proposed method.We investigate three audio layout representations: ( denoted as 𝒕 𝑎 , performs the best across the major metrics FAD, KL, W-dis and DTW-dis. The lower IS score for 𝒕 𝑎 may attribute to this metric is based on entropy and a more precise control may reduce the diversity of generated audios. We further explore the influence of down-sampling ratios (resulting in different resolutions) of our proposed audio layouts. We assess the impact of these ratios by examining the performance of predictors with various audio layout resolutions as learning targets. As shown in Figure 5 , the training loss curves and L2 loss on the validation set reveal that the compressed size [102, 5] ensures effective learning and precise audio layout prediction; whereas, higher and lower down-sampling ratios increase training losses and hinder model convergence and get worse L2 losses on the validation set.Predicting audio layout from visual information serves as the core step in TiVA. We conduct experiments to investigate the optimal design choice for our audio layout predictor: (1) CLIP features: the effectiveness of top-layer versus multi-layer CLS tokens from CLIP in Equation 4; (2) self-attention: the benefits of windowed and global 4 .Based on the results, we have discerned the following insights: (1) Multi-layer CLIP features enhance performance, with three-layer CLS features reducing the L2 distance from 0.208 to 0.187 compared to single-layer CLS features. As it is unclear which visual features are the most important, we retain three-layer CLS features and allow the model to learn on its own; (2) Self-attention contributes Table 4 : Results of different design choices for audio layout predictor T . Each row represents the setting of a variant of the audio layout predictor. CLS 4,8,11 and Patch 4,8,11 refer to the frame features from the CLIP's CLS token and the patch token of the 4-, 8-, 11th layers. 'SA' denotes selfattention layers and 'WA' denotes windowed self-attention layers. L2(𝒕 𝑎 , 𝒕 𝑣 ) denotes the L2 distance of preditced audio layouts 𝒕 𝑣 and ground truth audio layouts 𝒕 𝑎 on validation.CLIP-features self-attention cross-attention L2(𝒕 𝑎 , 𝒕 𝑣 ) ↓ CLS 11 SA -0.208 CLS 4,8,11 SA. -0.187 CLS 4,8,11 WA -0.214 CLS 4,8,11 SA+WA -0.182 CLS 4,8,11 SA+WA Patch more while windowed self-attention is complementary. Although windowed self-attention alone performs much worse than selfattention alone, an extra gain is achieved when using both. This may be because some audio is delayed, such as a thunder lagging behind lightning, and some audio is inherently long, requiring longrange interactions for accurate prediction; (3) Incorporating patch features is helpful, taht further decreases the L2 distance from 0.182 to 0.171. These findings guide our final layout predictor's design.We further conduct ablation studies on TiVA's training strategies:(1) w/o T , in which we remove audio layout predictor and retrain our framework only with semantic encoder; (2) w/o S, in which we set classifier guidance scale with zero to remove semantic information during generation; (3) w/o Joint, which skips the training of joint optimizing parameters of semantic encoder, audio layout predictor, and generator. We compare them with our full version TiVA in automatic metrics and present results in Table 5 . The results indicate that the performance drops the most if without joint training. Compared to w/o T , w/o S drops more in terms of FAD, KL, W-dis, and IB. This indicates the critical role of semantic alignment in V2A generation, establishing it as a key component even when temporal alignment is controlled. In contrast, w/o T results in a significant increase in IS, indicating increased diversity of generated audio. This aligns with the intuition that temporal control may reduce diversity to achieve more tightly synchronized audio. An important application of video-to-audio generation is providing soundtracks for silent videos, including those created by AI. We try to produce audios for videos generated by Sora [4] for testing our TiVA model's generalization capabilities to out-of-domain data. Due to the absence of ground-truth audio, we add our generated audio tracks to Sora-generated videos and show some cases in Figure 6 . These cases cover the videos with creative scenes requiring non-natural sound effects (Case 1), fictional creatures (Case 2), complex transitions (Case 3), and scenes with multiple sound sources (Case 4). Our model TiVA successfully generates temporally aligned and semantically coherent audio for these videos, demonstrating promising generalization capabilities to open domains and the feasibility of dubbing sound effects for videos. 
conclusion and future work: 
 This paper introduces TiVA, a novel framework for video-to-audio generation that excels in creating semantically accurate and temporally synchronized audio tracks, with the innovative concept of audio layout. Results in objective and subjective evaluations consistently indicate TiVA's generation superiority in quality, semantic relevance, and synchronization. TiVA also shows promising results even for out-of-domain videos, e.g., Sora-generated videos. In future work, we aim to explore audio layouts with alternative generative frameworks and various tasks, and investigate TiVA's scalability. 
 
 Appendix: https://openai.com/research/video-generation-models-as-world-simulatorshttps://en.wikipedia.org/wiki/Dynamic_time_warpinghttps://en.wikipedia.org/wiki/Wasserstein_metric 


Decision: The paper proposes a new method for time-aligned video-to-audio generation. The paper received 3 WAs and 1 BA. The reviewers all liked the video-to-text problem and agreed that the proposed method is innovative and the consideration of predicting an audio layout is effective. The AC agrees with the reviewers that the video-to-audio work is timely considering the current trend in visual generative models (many on text to image and text to video, but few on video to audio) and marks a good contribution to the field.

Reviews: review: paper_summary: paper_summary: The paper proposes a new time-aligned video-to-audio generation framework called TiVA, which aims to provide precise temporal synchronization. TiVA uses a unique audio layout to predict the audio based on visual semantics and temporal information, and leverages a ldm for generaton. The results from objective and subjective experiments demonstrate that TiVA outperforms state-of-the-art methods in terms of semantic matching and temporal synchronization. Additionally, TiVA shows promising generalization capabilities when tested on out-of-domain videos.
strength_weakness: strength_weakness: 1. TiVA proposes an interesting solution to capture temporal aspects of video content to generate corresponding audio, resulting in a high degree of synchronization.

2. The use of an audio layout to predict audio based on visual information is new.

3. TiVA shows robust performance even on out-of-domain videos, demonstrating its ability to generalize beyond the training data.
limitations: limitations: 1. The comparison and discussion with existing approaches could be more in-depth. First, there are some related works missing, e.g,

Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (https://arxiv.org/pdf/2402.17723)

which also shows an interesting demo on generating audio for Sora video. I believe a discussion/comparison is needed here. Also, the comparions with existing approaches might not be fully fair. For DiffFoley, it is designed to generation audio for video that have a strong rythm, like drum beating, water flowing, which requires high-resolution audio-video synchronization capability. And in this paper, the examples shown are more related to some long-lasting sounds, like cat meowing and baby laughs. I think that's why a low-resolution audio layout can work well in this case. The authors might need to provide more justisfications for such cases.

2. Presentation could be improved. In fig3(a), t_a is the input to the network. But during inference, it should be t_v, as there is no GT t_a. Similarly, when introduing t_a and t_v in the paper, it could be strengthened more on the relations between them.

3. In Figure 5, what's the reason behind the gap between [102, 5] and other groups of parameters?
paper_summary: paper_summary: This article proposes a TIVA framework, which aims to generate high -quality audio to the silent video by achieving semantic similarity and accurate time synchronization. 
Unlike the existing method that mainly focuses on the method of matching vision and Audio Semantic with the rough time, Tiva encodes Visual Semantic and predicts Audio Layout respectively. 
It then uses these components as Condition Signal to train Latent Diffusion Model. 
Through comprehensive objective and subjective experiments, Tiva shows excellent performance in Semantic Matching and Temporal Synchronization, exceeding the current most advanced method.
strength_weakness: strength_weakness: 1. The starting point of this article is good. To solve the problem of fine-grained alignment in the Audio generation task, it is indeed a interesting and needed research direction in the current field. And from the results provided in this article, the proposed practices are effective in the process of solving this task.
limitations: limitations: 1. I questioned some motivation to use Audio Layout. As the author said, he is just the result of Mel-Spectrogram after sampling, so what exactly is this Audio Representation? Why can I include Temporacture after sampling.
2. Based on the above problems, the author needs to provide an additional experiment: Under the settings of Audio Layout, using Audio Vae and Vocoder to reconstruct the original information, it can maintain the original information. Compared with the reconstruction based on the original Mel-Spectrogram.
3. When processing long videos, how to ensure that all video frames in long videos are encoded, and is there any sampling strategies for some frames? The author should provide more details, instead of simply describing the use of embedding with CLIP.
4. The focus of this article is Time-Aligned Video-to-Audio Generation. So it is very important to evaluate whether the Audio generated by the generated by Video is indeed aligned with Video. Therefore, the author should emphasize the evaluation indicators used in detail, including its principles, and how to measure alignment.
5. Formula (6) is defective. "LDM" contains noise sampling, noise estimation, and loss functions, which are not simply used to estimate noise. It is better to write a specific symbol of Denoising Network.
paper_summary: paper_summary: In this study, the authors propose a new approach to video-to-audio generation called TiVA. The goal is to generate high-quality audio for silent videos with semantic similarity and accurate temporal synchronization. In particular, TiVA encodes the visual semantics and predicts an audio layout separately to achieve joint semantic matching and precise temporal synchronization.
strength_weakness: strength_weakness: 1、The practice of ensuring the consistency of the timing of audio and video generation by first generating the audio layout has certain innovativeness.

2、The authors provide many playable samples in the provided link, which demonstrate the effectiveness of the proposed method.
limitations: limitations: 1、Previous methods have used attention mechanisms in addition to semantic guidance for temporal alignment of audio and video, such as MM_Diffusion [1]. However, such alignment in this method relies solely on the low-resolution audio layout. I suspect that the downsampled audio layout may not provide sufficient temporal guidance for multi-channel audio (such as music for dancing or movie soundtracks) or natural sound (such as the sound of ocean waves) because the downsampling operation loses a lot of the fine details of the mel-spectrogram.

2、In  L402, the author states that they used a Transformer-based UNet. What is the difference between this structure and the UNet used in traditional diffusion models? In addition, in Equ. 2 and Equ. 3, the author used various subscripts and superscripts for "s," but did not explain the meaning of each subscript and superscript.

3、The model proposed in this paper needs to be trained separately and then jointly. Why can't it be trained end-to-end directly?
paper_summary: paper_summary: For the video-to-audio (V2A) generation task, this paper proposes a time-aligned V2A framework TiVA to jointly achieve semantic matching and temporal synchronization. In addition to typical video frames, the method also utilizes low-resolution Mel-spectrograms of audio ground truths (audio layout called in this paper) as an important condition for the diffusion-based model. Experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: I do believe this is a nice work. 
1. The paper is well-written with clear explanations of the proposed method. Figures and tables are effectively used, enhancing understanding of the TiVA framework.
2. The proposed method is simple yet highly effective. 
3. Extensive experimental results, including ablation studies, robustly support the validity of the proposed method.
limitations: limitations: I found no major problems with this paper. I just have a few small suggestions or questions.
1. It would be better to also provide citations in Table 1. The meaning of the `CFG' in Table 1 is not provided.
2.  Utilizing the audio mel-spectrogram in the audio generation task may not be entirely novel.  A more exhaustive review of related literature might be beneficial to better position this work within existing research.
3.  Making the source code available would significantly enhance the paper's impact, allowing for reproducibility and further research based on this work.