Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching
Authors: Yongqi Wang
Wenxiang Guo
Rongjie Huang
Jiawei Huang
Zehan Wang
Fuming You
Ruiqi Li
Zhou Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: video-to-audio generation
rectified flow model
efficient generation
Primary area: speech_and_audio
TL;DR: We design a video-to-audio generation model with higher quality and fewer sampling steps.

Abstract: Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video, and it remains challenging to build V2A models with high generation quality, efficiency, and visual-audio temporal synchrony. 
We propose Frieren, a V2A model based on rectified flow matching. Frieren regresses the conditional transport vector field from noise to spectrogram latent with straight paths and conducts sampling by solving ODE, outperforming autoregressive and score-based models in terms of audio quality. By employing a non-autoregressive vector field estimator based on a feed-forward transformer and channel-level cross-modal feature fusion with strong temporal alignment, our model generates audio that is highly synchronized with the input video. Furthermore, through reflow and one-step distillation with guided vector field, our model can generate decent audio in a few, or even only one sampling step. Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment on VGGSound, with alignment accuracy reaching 97.22\%, and 6.2\% improvement in inception score over the strong diffusion-based baseline. Audio samples and code are available at http://frieren-v2a.github.io.

FRIEREN: Efficient Video-to-Audio Generation Network with Rectified Flow Matching
Yongqi Wangi Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, Zhou Zhao"
Zhejiang University cyanbox8zju. edu. edu. cn
Abstract
Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video, and remains challenging to build V2A models with high generation quality, efficiency, and visual-audio temporal synchrony. We propose FRIEREN, a V2A model based on rectified flow matching. FRIEREN regresses conditional transport vector field from noise to spectrogram latent with straight paths and conducts sampling by solving ODE, outperforming autoregressive and score-based based models in terms of audio quality. By employing autooooorerre vector field estimator based on a feed-forward transformer and channel-level cross- modal feature fusion with strong temporal alignment, model generates audio that is highly synchronized with the input video. Furthermore, through reflow and one- step distillation with guided vector field, model generate decent audio in a few, or even only one sampling step. Experiments indicate that FRIEREN achieves state-of-the-art performance in both generation quality and temporal alignment on VGGSound, with alignment accuracy reaching 97.22%, and 6.2% improvement inception score over the strong diffusion-basee baseline. Audio samples and code are available at http::/ffieeree eeraiiereeeea...... v2a. thub.
Introduction
Recent advancements in deep generative models have significantly enhanced the quality and diversity
of Al-generated content, including text (27), images [29] 30 T. videos 32 23 and audios [19]
20 Among various content-generation tasks, video-to-audio (V2A) generation aims to synthesize semantically relevant and temporally aligned audio from video frames. Due to its immense potential for application film dubbing, game development, YouTube content creation and other areas, the task of V2A has attracted widespread attention.
A widely applicable V2A solution is expected to have outstanding performance in the following aspects: 1) audio quality: the generated audio should have good perceptual quality, which is the fundamental requirement of the audio generation task; 2) temporal alignment: the generated audio should not only match the content but also align temporally with the video frames. This has a significant impact
experience due to keen human perception of audio-visual information: and 3) generation efficiency: the model should be efficient in terms of generation speed and resource utilization, which affects its practicality for large-scale and tigh-throoghhut applications.
Currently, considerable methods have been proposed for this task, including GAN-based models wansformer-basedd autoregressive models [1], and recent Foley 25 However, these methods have not yet achieved a balanced and satisfactory performance
based model, Diff-
Equal contribution. Cooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

across the above aspects. 1) For audio quality, early GAN-based models suffer from poor quality and lack practicality, Autoregressive and diffusion models make improvements in generation quality, but still leave room for further advancement. 2) For temporal alignment, autoregressive models lack the ability to align the generated audio with the video explicitly. And due to the difficulty of learning visual alignment with the cross-attention-bann based conditional mechanism solely, Diff-Foley relies on additional classifier guidance to achieve good synchrony, which not only increases the model complexity but also leads to instability when reducing sampling steps. 3) For generation efficiency, autoregressive models suffer from high inference latency, while Diff-Foley requires considerable sampling steps to achieve good generation quality due to the curved sampling trajectories of diffusion models, increasing the temporal overhead in inference. In a nutshell, existing methods still leave significant room for improvement in performance.
In this paper, We introduce another generative modeling approach, namely rectified flow match- ing [21], into the V2A task. This method regresses the conditional transport vector field between noise and data distributions with as straight trajectories as possible, and conducts sampling by solving the corresponding ordinary differential equation (ODE). With simpler formulations, our rectified- flow-based model achieves higher audio quality and diversity. To improve temporal alignment, we adopt non-autoregressivee vector field estimator network with a feed-forward transformer with no cemporal-dimensee downsampling, thereby preserving temporal resolution. We also employ channel-level cross-modke feature fusion mechanism for conditioning, leveraging the inherent alignment of audio-visual data and achieving strong alignment. These designs lead to high synchrony between generated audio and input video while upholding model simplicity. Moreover, through integrating reflow and one-step distillation techniques, our model can generate decent audio with a few, or even only one sampling step, significantly improving generation efficiency.
We name our model FRIEREN for efficient video-to-audio generation network with rectfied flow matching. Experiments indicate that FRIEREN outperforms strong baselines in terms of audio quality, generation efficiency, and temporal alignment on VGGSound (2], achieving a 6.2% improvement in inception score (IS) and generation speed 7.3x that of Diff-Foley, as well as temporal alignment accuracy of up to 97.22% in 25 steps. Additionally, FRIEREN combining reflow and distillation achieves alignment accuracy of up to 97.85% with just one step, with 9.3x acceleration compared to 25-step sampling, further boosting generation efficiency.
2 Related works
2.1 Video-to-audio generation
Video-to-audio (V2A) generation aims to synthesize audio of which content matches the visual information of a video clip. RegNet 3 designs time-dependent visual encoder to extract appearance and motion features, which are then fed to a GAN for audio generation. FoleyGAN [ also utilizes GAN for audio generation from visual features, together with predicted action category as the conditional input. Spec VQGAN (15) takes RGB and optical flow of videos and uses a transformer to generate indices of spectrogram VQVAE autoregressively. Im2Wav [1] adopts two transformers for different temporal resolutions and takes CLIP (2 features as the condition to generate VQVAE indices. Du et al. [5] mimics the real-worla foley methodology and introduces an additional reference audio as the condition. Diff-Foley (25) designs an audio-visual contrastive feature and adopts a latent diffusion to predict spectrogram latents, achieving decent audio quality and inference speed.
In addition to training a whole model from scratch, some works integrate off-the-shelf audio gen- eration models with modality mappers or multimodal encoders with joint embedding space for conditioning. V2A-Mapper (35) uses a lightweight mapper to transfer CLIP embeddings of videos to CLAP (40) embeddings as the condition for audio generation. Xing et al. 441 utilize an ImageBind[[]] based latent aligner for conditional guidance in audio generation. Despite the existence of plentiful works on V2A, there is still large room left for improvement in quality, synchrony, and efficiency.
2.2 Flow matching generative models
Flow matching 18 models the vector field of transport probability path from noise to data samples. Compared to score-based models like DDPM [2), flow matching achieves more stable and robust training together with superior performance. Specifically, rectified flow matching [21 learns the
2

---

transport ODE to follow the straight paths connecting the noise and data points as much as possible, reducing the transport cost, and achieving fewer sampling steps with the reflow technique. This
modeling paradigm has demonstrated excellent performance in accelerating image generation 22 6
In the area of audio generation, Voicebox [16] builds a large-scale multi-task speech generation model
based on flow matching. Its successor, Audiobox [34], extends the flow-matching mnatching-based model to unified audio generation model with natural language prompt guidance. Matcha-tts (26) trains an encoder-decoder TTS model with optimal-transport conditional flow matching. VoiceFlow [ introduces rectified flow matching into TTS, achieving speech generation with fewer inference steps. However, for the task of V2A, there has been no exploration into utilizing flow matching models to enhance generation quality or inference efficiency.
3 Method
3.1
Preliminary: rectified flow matching
We first introduce the basic principles of rectified flow matching (RFM) (21] that we build our model upon. Conditional generation problems like V2A can be viewed as a conditional mapping from noise distribution Zu ~ (x) to a data distribution æ ~ P1 (z)). This mapping can be further taken as time-dependent changing process of probability density (a.k.a. flow), determined by the ODE:
da u(x lc)ddt, 1. [0, 1],
(1)
where É represents the time position, x is a point in the probability density space at time t, u is the value of the transport vector field (i.e., the gradient of the probability w.r.t t) æ, and c is the condition. In our case, the condition e is the visual features from the video frames, while the data æ1 is the compressed mel-spectrogram latent of the corresponding audio from pre-traines praitreine autoencoder. The fundamental principle of flow matching generative model is to use a neural network to regress the vector field with the flow matching objective:
(0) (x)) [| t|c; (xxxt
where Pe (xx is the distribution of x at timestep t. However, due to lack of prior knowledge of target distribution p1 (x) and the forms of pt and u. it is intractable to directly compute ul (a: tl).. As an alternative, conditional flow matching objec- tive, which is proven in [8] to have identical gradient as eq. 2
Audio
Vocoder
w.r.t 0, is used for regression:
Spectrogram
E
D
Autnencoder
CCFM (0) Et (mm)) [e(x u(x,t/11,10 c) (3)
Mel-Spee
Through designing specific probabilistic paths that enable effi cient sampling from (a(a=ar) and computing of u(x, f| c). we achieve an unbiased estimation of u(x tl, e) with the CFM objective 3 Specifically, rectified flow matching attempts to establish straight paths between noise and data, aiming to fa- cilitate sampling with larger step sizes and fewer steps. Given noise-data pair (æo, x1). is located at (1 t)x, tx at timestep t, with the vector field being u (x, 1/x1, c) = x1 xo. pointing from the noise point to the data point. Hence, for each training step of the vector field estimator, we simply sample the data point 21 and noise point 2n from pr (æ:) and (ax),, respectively, and optimize the network with the rectified flow matching (RFM) loss
ODE Solver
dæz (ar.tle 0)
34 N(O.
Figure Illustration of the sampling process of our rectified-flow based V2A architecture.
lu(2,t|c;0 (x1 xo)]l
Once the vector estimator network finishes training, we can adopt various solvers to approximate the solution of the ODE dæ v(x, t|c; 8) discretized time steps for sampling. A simple and commonly used ODE solver is the Euler method:
EV(x, t|c;
(5)
3

---

0.:0--00-000
Conv ID
Cony ID
Group Norm
+
Vector Field Estimator
Feed-Forwar
XN
Conv DD-FNN
Transformer
Layer Norm
-0000-00000 10000-0000
100000-0000
Conv
Leaened
Self-Aticntion
PosEibb
Length Regulator
Layer Norm
+
Linear
Self Self-Attention
Visual Encoder
S Sinssiddl
Linear
Layer Norm
Encoding
ConviD
Lineur
ConviD
Sepueoo Looel
Framos
Group Norm
Charat-Land Concenaatio
Conarnatio
(a) Model Architecture Overview
(b) Vector Field Estimator Network
(c) Peed-Forward Transformer Block
Figure 2:
Illustration of model architecture of FRIEREN at different levels.
where is the step size. The sampled latent is fed to the decoder of the spectrogram autoencoder for spectrogram reconstruction, and the result is further used reconstruct the audio waveform with vocoder. Figure I provides simple demonstration of the model's sampling process.
3.2 Model architecture
Model overview We illustrate the model architecture of FRIEREN at different levels Figure As shown in Figure first utilize pre-trained visual encoder with frozen parameters extract frame-level feature sequence from the video. Usually, the video frame rate lower than the temporal length per second of the spectrogram latent. To align the visual feature sequence with the mel latent at the temporal dimension for the cross-modal feature fusion mentioned below, we adopt length regulator, which simply duplicates each item in the feature sequence by the ratio of the latent length per second and the video frame rate for regulation. The regulated feature sequence is then fed to the vector field estimator the condition, together with and to get the vector field prediction v.
Visual and audio representations Various audio-aligned visual representations 9 25
38 37
36 39 can potentially be applied video-to-audio generation, and we conduct experiments with two types of visual representations. For a fair comparison with Diff-Foley [25], we mainly utilize the CAVP feature proposed in (25]. which visual-audio contrastive feature considering both content and temporal alignment. Meanwhile, to investigate the impact of visual feature characteristics on model performance, we also attempt the visual feature from MAViL. 14,, which is an advanced self-supervised visual-audio representation learner that employs both maakeed-reoonstruetio and contrastive learning, and exhibits formidable performance in audio-visual understanding (See section 4.3.2 for comparison). For audio representation, follow a previous text-to-audio work [13] to train a mem -specrrogram VAE with ID convolution over the temporal dimension. Details of the VAE are provided in appendix A
Vector field estimator Figure b) demonstrates the structure of the vector field estimator, which is composed of feed-forward transformer and some auxiliary layers. The regularized visual feature and the point æ on the transport path are first processed by stacks of shallow layers separately, with output dimensions being both half of the transformer hidden dimension, and are then concatenated along the channel dimension to realize mross-modal feature fusion. This simple mechanism leverages the inherent alignment within the video and audio, achieving enforced alignment without relying on learning- based mechanisms such as attention. As result, the generated audio and input video sequences exhibit excellent temporal alignment. After appending the time step embedding to beginning, the sequence is added with learnable positional embedding and is then fed into the feed- forward transformer. The structure of the transformer block illustrated in Figure (c), the design of which is derived from the spatial transformer in latent diffusion (29,, with the 2D convolution layers replaced by ID ones. The ferdwforward transformer does not involve temporal downsampling,
Implementation of MAViL is from av-superb 33 https://gitthin htw.shhuu.ccmm//ooere.e leveng/av-supet.

---

thus preserving the resolution of the temporal dimension and further ensuring the preservation of aligmment The output of the stacked transformer blocks then passed through a normalization layer and ID convolution layer to finally obtain the prediction of the vector field.
3.3 Re-weighting RFM objective with logit-normal coefficient
The original RFM objective samples uniformly over time span [0, 1]. However, for modeling the vector field, positions in the middle of the transport path (equivalent to time steps in the middle of [0. IJ) present greater difficulty, these positions are distant from both noise and data distributions. On the other hand, positions near the boundaries of the time span typically lie close to corresponding noise or data points, and their vector field direction tends to align with the lines connecting these points and the centroid of the distribution on the opposite side, and therefore relatively easy to regress. Upon this insight, we introduce time-based re-weighting to the original RFM objective, allocating more weight to intermediate time steps to achieve better modeling effectiveness. This is equivalent to increasing the sampling frequency of intermediate time steps. In practice, logit-normal weighting coefficients have been proven [6] to yield promising results, with the formula being
(In In(1 t))
wit) = 22T ((1 - t)
exp
(6)
We re- re-weight the RFM objective with this weighting function to replace the original objective and observe in our experiment that this re- we-whigtting helps to slightly improve audio quality and temporal alignment at the cost of marginal decrease in audio diversity.
3.4 Classifier-free guidance
Similar to liffusion-based models, we observe that classifier- free guidance (CFG) highly important for generating audio that semantically matches and temporally aligns with the video. During training, we randomly replace the condition sequence e with zero tensor with a probability of 0.2, and during sampling, we modify the vector field using the formula
CCFG (ar, l/c; 0) = 7n( t|c; 0) + (1 - 7), 1):
where 7 is the guidance scale trading off the sample diversity and generation quality, and TCPG degenerates into the original vector field e when 7 = 1. We set y to 4.5 our major experiments.
3.5 Reflow and one-step distillation with guided vector field
In this section, we introduce two techniques we adopt for reducing sampling steps. The first one is reflow, which is crucial component of the rectified flow paradigm 2T (2). Training the estimator network with objective 4 for once is insufficient to construct straight enough transport paths, and an extra reflow procedure is needed to strengthen the transport trajectories without altering the marginal distribution learned by the model, enabling sampling with larger step sizes and fewer steps. Given a model trained with RFM objective, the reflow procedure applies to conduct sampling over the entire training dataset to obtain sampled data é and save the corresponding input noise z/)- finally
obtaining triplets (at) @. c). The noise-data pair (æa, æ1) in the RFM objective 4 is replaced by (#/) #²1) for a secondary training of 0. This process can be repeated multiple times to obtain straighter trajectories with diminishing marginal effects. We conduct reflow for once as it is sufficient for achieving straight enough trajectories.
While many rettiiiddflow-blee flow- models regress the same velocity field during both the initial training and the reflow process, we observe that when incorporating CFG, conducting sampling and reflow with the original vector field v is ineffective in straightening the sampling trajectories with the guided vector field UCFG. Therefore, we use UCFG for generating á and as the target of regression reflow. The reflow objective can written as:
Lreflow (f') = E pe, th (o) [FFCa t/c; - (#) =1)))²
(8)
with same weighting function as eq.
Upon the model 0' obtained from reflow, we further conduct one-step distillation 21 22 to enhance the single-step generation performance of the model. As type of self-distillation, this procedure tries
5

---

Table 1: Results of V2A models on VGGSound dataset. R+F and RN50 denote the RGB+Flow and ResNet50 versions of SpecVQGAN, and CG denotes classifier guidance Dole-Foley.
Model
FD4
1St
KT_
FAD
KID(I0-3)
Acc(5))
MOS-Q7
MOS-A1
SpeeVQGAN (R+F)
31.69
5.23
3.37
8,53
61.83
3.30 0.06
2.35 0.05
SpecVQGAN (RN50)
32.52
3.41
9.00
56.92
3.25 0.07
2.17 0.05
Im2Wav
14.98
7.20
2.57
5.49
3.35
56.70
3.39 0.06
2.29 0.06
Diff-Poley (CG v/)
23.94
11.11
3.38
4.72
9.58
95.03
3.57 0.08
3.74 0.07
Diff-Foley (CG X)
24.97
11.69
3.23
7.10
10.32
92.53
3.64 0.07
3,59 0.06
LDM
11.79
10.09
2.86
1.77
2.26
95.33
3.72 1 0.05
3.79 0.07
FRIEREN
12.26
12.42
2.73
1.32
2.49
97.22
3.78 + 0.06
3,90 = 0.05
FRIEREN (Dopris)
11.64
12.76
2.75
1.37
2.39
96.87
3.81 = 0.06
3.85 = 0.06
to reduce the error between the single-step sampling result æ6 + VCFa l/;; 0) and the multi-step sampling result 1 The objective function can be written as:
Ldista((")) = piae dh pe) || + uCFG (z/) t|c; 8") - 21
Formally, the distillation objective can be viewed as reflow objective with the sampling timestep fixed at L O. We observe in the experiment that due to a limited number of sampling steps reflow data generation, the model may experience a decrease in sampling quality after the reflow process. Therefore, we opt to use the same training data used reflow for distillation, rather than re-sampling the training data with the reflow model, which is based on the theoretical basis that reflow does not alter the marginal distribution modeled by the estimator.
Experiments
4.1 Experiment setup
Dataset and pre-processing Following most previous works, we take VGGSound 2 as the benchmark, which consists of 200k+ 10-second video clips from YouTube spanning 309 categories. Excluding videos already removed from YouTube, we follow the original train and test splits of VGGSound, the sizes of which are about 182.6k and 15.3k. We downsample the audios 16kHz and transform them to mel-spectrogram with 80 bins and hop size of 256. We follow 255 to downsample the videos FPS. Data samples are truncated to 8-second clips for training and inference.
Model configuration The transformer of the vector field estimator mainly used in the experiments has 4 layers and hidden dimension of 576. Each model is trained with 2 NVIDIA RTX-4090 GPUs. We train the estimator for 1.3M steps for the first training, and 600k and 500k steps for reflow and distillation, with the learning rate being 5e-5 for all stages. For waveform generation, we train BigVGAN I7 vocoder on AudioSet (7]. Details of model parameters are provided in appendix A
We combine objective and subjective metrics to evaluate model performance audio quality, diversity, and temporal alignment. For objective evaluation, we calculate Frechet distance (FD), inception score (IS), Kullback-Lebler divergence (KL), Frechet audio distance (FAD), kernel inception distance (KID), and alignment accuracy (Acc). We utilize audio evaluation tools provided by AudioLDM 19 which are widely used in audio generation tasks, as well as the alignment classifier provided in 25 For metrics with reference like FAD, we duplicate the reference audio samples in the test set for 10 times as we generate 10 samples for each data item. For subjective evaluation, we conduct crowd-sourced human evaluations with 1-5 Likert scales and report mean-opine reean-opinionnscovee (MOS) over audio quality (MOS-Q) and content alignment (MOS-A) with 95% confidence intervals (CI). We sample 10 audios for each test video for evaluation. Details of subjective evaluation are provided in appendix
Metrics
We adopt three advanced V2A models as baselines, including: 1) Spec [5]. a transformer-based autoregressive model generating spectrogram VQVAE indices from visual features; 2) Im2Wav (31), hierarchical autoregressive V2A model predicting audio VQVAE indices conditioned on CLIP features; and 3) Diff-Foley 255, strong latent-diffusion--assee based V2A model. For we evaluate two versions using RGB+Flow and ResNet features as input visual
Baseline models

---

Table 2: Results of FRIEREN and Diff-Foley under different sampling steps. CG denotes classifier guidance, R denotes reflow and D denotes one-step distillation.
Model
Steps
FDL
ISt
KT-
FAD1
KID
Acc(Sh)
MO5
MOS-A1
Dir-Foley
Dil-Foley (CG X)
86.97
1.86
4.17
14.66
39.73
37.02
1.17 1: 0.03
1.63 - 0.04
FRIEREN (R D
70.48
2.98
26.99
43.18
FRIEREN (R X)
18.61
6.63
2.60
3.13
3,49
94.96
3.32
1.
3.74
0.06
FRIEREN (R (VD))
17.58
8.66
2.56
1.85
2.91
97.85
3.48
+ 0.06
3.93
0.05
DiT Foley Doley (CG v/)
60.99
3.42
3.62
9.61
3.60
73.30
2.66 1. 0.07
2.98
0.07
Dife-Foley X)
51.52
91.30
5
FRIEREN (R K.
28.78
6.69
3.02
4.34
8.56
87.69
3.30 1.
3.37
0.08
FRIEREN (R (.D
14.65
8.28
2.60
2.11
2.28
96.82
3.43 1: 0.06
3.83
0.06
DiI DullyFuley (CG V)
23.94
11.11
3.28
4.72
9.58
95.03
3.57 1: 0.08
3.74
0.07
Fole. X)
25
24.97
92.53
FRIEREN (R X
12.26
12.42
2.73
1.32
2.49
97.22
3,78 1: 0.06
3.90
0.05
FRIEREN (R V.D
13.39
9.79
2.64
1.66
2.01
97.36
3.61 1. 0.07
3.88
0.05
(a) Input video
(b) Diff-Folcy
(c) LDM
FRIEREN
FRIEREN (rellow)
FRIEREN (reflow distillation)
(no reflow)
Figure 3: One-step generation results of different models. (a): The content of the input video is woman playing the violin. (b): Diff-Foley generates meaningless audio with one step. (c, d): LDM and FRIEREN without reflow generate highly noisy audio. (c. f): reflow enables FRIEREN generate meaningful audio in one step, and distillation further improves the one-step generation quality.
conditions. For Diff-Foley, we evaluate its performance with and without classifier guidance to examine the impact of its complex external alignment mechanism. To better validate the superiority of our rectified flow model, we also train diffusion model sharing the same architecture as FRIEREN but has a different prediction target, labeled as LDM in the following tables. For diffusion models, we use DPM-Solver [24] for sampling. For our rectified flow model, we use the Euler method most cases without further specification. We also explore the more advanced Dormand-Pal --rrnrer method (Dopri5) 4 method for higher generation quality.
4.2 Results and analysis
Video-to-audio generation results The results of different models illustrated in table I We sample with diffusion models and FRIEREN with 25 steps, and report the result of FRIEREN without reflow and distillation, which shows the best overall performance with high number of sampling steps. It can be seen that FRIEREN significantly outperforms other models in IS, FAD, and alignment accuracy, with the values reaching up 12.42, 1.32, and 97. 22%, together with high subjective scores of 3.78 and 3.90 on quality and alignment. For FD, KL, and KID, the scores of FRIEREN are also very close the best values among other models. When we employ the higher-order DopriS ODE solver, FRIEREN achieves further improvements in FD and IS, attaining best values of 11.64 and 12.76, respectively, while maintaining stable performance in other objective metries, at the cost of slower sampling speed. This indicates the effectiveness of our approach. Generally, the performance of FRIEREN surpasses that of the I.DM, demonstrating the superiority of rectified flow matching over the score-based paradigm of diffusion. Additionally, both FRIEREN and LDM outperform Diff-Foley in temporal alignment, proving that our architecture design achieves strong temporal alignment without the need for complex mechanisms, and can produce audio that is highly synchronized with visual input. Additionally, our model also has an advantage in sampling time, with details provided in appendix
Few and single step generation results We further demonstrate the results of Diff-Foley and FRIEREN reduced sampling steps in table 2 to illustrate the impact of reflow and one-step

---

Table 3:
Ablation results on different model size of vector field estimator network.
Model Size
FDL
1S1
KL|
FAD4
KID(10-3) J
Acc(S) 1
MOS-Qt
MOS-A1
Small (70,90 M)
13.02
12.16
2.78
1.50
2.79
96.04
3.71 + 0.07
3,83 0.06
Base (158.88 M)
12.26
12.42
2.73
1.32
2.49
97.22
3.78 + 0.06
3.90 0.05
Large (421.12 M)
12.20
12.29
2.76
1.36
2.97
95.16
3.78 + 0.07
3.80 + 0.06
distillation, together with trend graphs of IS and FAD in figure 4 for intuitive presentation. The data for reflow are generated with the Euler method for 25 steps. We observe an obvious drop in performance of Diff-Foley as well as FRIEREN without reflow when sampling with as few as 5 steps, and their scores become extremely poor when we further reduce the step number to 1. Figure (b) (c) and (d) illustrate that the audio generated by these models as well as LDM degrades into unacceptably noisy or meaningless audio within one step. This is due to the convoluted nature of the sampling trajectories of these models, which disables them from sampling with large step sizes and few steps. We also notice that when sampling with 5 steps, using additional classifier guidance deteriorates the audio quality and synchrony of Diff-Foley, where alignment accuracy and IS drop by 18.0% and 1.72 respectively, while FD, KL., and KID increase by 9.47, 0.17 and 0.94 x x10-3 This indicates the lack of robustness of the complex alignment mechanism that Diff-Fole relies on.
In contrast, FRIEREN with reflow achieves an alignment accuracy of up to 96.82% in just 5 steps, with signif- icant advantages in quality, diversity, and subjective metries. Additionally, it maintains an accuracy of 94.96% in well de-
Inception Score RS}
Frechet Audio Distance (FAD)
Disiokl Co
12
Onttrkky (wo Oc)
Fieenn deei
D Biiieey 2e CD)
single-step generation, cent quality and diversity. This proves that reflow functions significantly in straightening the sampling trajecto- ries, enabling the rectified flow model to generate decent audio with a small number of sampling steps. Furthermore, single-step distillation following reflow further improves the model performance with one step, with alignment accuracy reaching up to 97.85%, and KL. FAD, and KID being close to the 25-step results of FRIEREN trained once, with differences of 0.17, 0.53 and 0.42x10 3.48 and 3.93.
Hroen Traan D4
Infeeence Steps
1rd Inference Steps
Figure 4: IS and FAD of the models with different steps.
10 3 It also achieves high MOS-Q and MOS-A of
Figure B (e) and (f) show that results from FRIEREN with reflow and reflow-distillation have distingutshable spectrograms, with the latter showing higher quality and sharper edges. This fully demonstrates that the combination of reflow and one-step distillation endows our model with strong single-step generation capabilities, significantly enhancing the efficiency on the V2A task. Notice that reflow brings in some quality degradation in sampling with 25 steps. We speculate that this is because the limited number of sampling steps restricts the data quality when generating data for reflow, resulting shift in the marginal distribution learned by the model. This cumulative error might be mitigated by increasing the number of sampling steps during reflow data generation.
4.3 Ablation study
4.3.1 Model size of vector field estimator
We adjust the number of parameters of the vector field estimator and evaluate the model performance at different scales. We label the major model as "base", and obtain "small" and "large" models by decreasing and increasing the hidden dimension and or the number of transformer layers, respectively. The parameter counts of the estimator and results are presented in table
We observe that when the model parameters are reduced to 71M, performance declines across all metrics, where FD, KL, FAD, and KID increase by 0.76, 0.05, 0.18, and 0.3x10-3, and IS, alignment accuracy, MOS-Q and MOS-A drop by 0.26, 181%, 0.07 and 0.07, respectively. However, when the parameter number increases to 421 M, there is a performance degradation across multiple metrics, with KI., FAD, and KID increasing by 0.03, 0.04, and 0.48 - and IS, alignment acc declining by 0.13 and 2.06%. We speculate that this anomalous phenomenon may be due to the convergence difficulty
8

---

Table 4: Results on different types visual features.
Type
Feat. FPS
FD4
1St
KLU
FAD,
KID(10 ")
Acc(5))
MOS-Q1
MOS-A1
CAVP 25
12.26
12.42
2.73
1.32
2.49
97.22
3.78 + 0.06
3.90 = 0.05
MAVIL
14
12.08
12.17
2.49
1.26
2.52
90.17
3.75 0.06
3.46 + 0.07
Table 5: Ablation results RFM objective re-weighting.
Re-weighting
FD4
1S1
KL4
FAD]
KID(10 4
Acc(5)) f
MOS-Q Q1
MOS-A1
11.95
12,20
2.73
1.25
2.12
97.04
3.74 + 0.07
3.82 + 0.06
12.26
12.42
2.73
1.32
2.49
97.22
3.78 + 0.06
3.90 + 0.05
for the larger model under similar training steps, or the redundant model capacity tends to cause overfitting on a relatively small dataset like VGGSound, deteriorating the model's generalization performance. In summary, we achieve relatively balanced model performance with the parameter of the estimator being around 160M. Details of model parameters are provided in appendix
4.3.2 Visual feature characteristics
In table 4 we compare the results of FRIEREN using two different types of visual features from CAVP and MAVIL. Intuitively, the MAViL feature should be more robust and contain richer audio-related semantic information, it utilizes maskeddreconstruuctio together with inter-modal and intra-modal contrastive learning, in contrast to CAVP trained solely with inter-modal contrastive learning. On the other hand, however, due to MAViL's convolutional downsampling in the temporal dimension, its feature sequence has a lower effective FPS of 2 with the same FPS video input as CAVP. The results in the table indicate that the model with MAViL feature excels in audio diversity, with differences of FD, KL, and FAD being 0818. 0.24, and 0.06. Meanwhile, exhibits a 7.05% decrease alignment accuracy and 0.25 decrease in IS. This result yields two insights for V2A tasks: 1) at relatively low frame rates, the frame rate of features, rather than content, is more likely to become the bottleneck for audio quality and visual-audio synchrony; 2) compared to high video frame rates, the semantic information and robustness visual features are more crucial for the diversity generated audio.
4.3.3 Classifier-free guidance scale
In figure 5 we illustrate the im- pact of various CFG scales on the performance of FRIEREN. In terms of audio diversity (FD, KI, KID, FAD), the metrics initially increase with the CFG scale, reaching an optimal value at around 2 and 3. After that, the metrics go down as the in- creasing CFG scale suppresses the diversity. For audio quality (IS) and temporal alignment, as larger scales make the content of the generated audio closer to the visual information, the metrics initially increase with the scale, reaching an optimal value between 4 and 4.5, and decrease after that due to audio distortion. prioritize audio quality and synchrony and adopt a CFG scale 4.5.
15
3.0
CHo Stale
CRG Stale
Cha Scale
05000
0500
00000
Stale
Figure 5: Model performance of FRIEREN under different CFG scales.
4.3.4 Re-weighting RFM objective
We conduct ablation on RFM objective re- re-weighting and report the results in table 5 We can see that compared to the vanilla objective, introducing re-weighting results in improvements 0.22 and 0. 0.18% for IS and alignment accuracy. This validates the positive impact of objective re- weighting on audio quality and temporal alignment. the other hand, objective re-weighting causes a decrease in audio diversity, with differences in FD, FAD, and KID being 0.31, 0.07, and 0.37 1 respectively.

---

5 Conclusion
In this paper, we propose FRIEREN, an efficient video-to-audio generation model based on rectified flow matching. We use a neural network to regress the conditional transport vector field with straight paths from noise to spectrogram latents, and conduct sampling by solving ODE, achieving better performance than cittusiion-bsell and other V2A models. We adopt vector field estimator based on a feed-fore herd-f transformer as well as channel-level cross-modal feature fusion to realize strong audio-video synchrony. Through a combination of reflow and one-step distillation, our model can generate high-quality audio with few or even one sampling step, boosting the generation efficiency significantly. Experiments show that our model achieves state-of-the-ane V2A performance on VGGSound. For future work, we will explore extending the model to larger scales and larger datasets to achieve V2A generation longer video sequences with variable lengths, rather than being limited to ixed-Iength short clips. These efforts aim to build more versatile and widely applicable V2A model.
broader data domain. Besides, we will attempt audio generation from
Acknowledgment
This work is supported by the National Natural Science Foundation of China under Grant No. 62222211 and 60.62272397.
References
[1] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. con/papers/tallle-.. pdf. 2(3):8, 2023.
[2] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
[3] Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 29: 8292-8302, 2020.
[4] John Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1): 1980.
[5] Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Conditional generation of audio from video via foley analogies. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2426-2433 2023.
[6] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- for high-resolution image synthesis. arXiv preprint S.XYv:24033032066 2024.
[7] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and numan-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
[8] Sanchita Ghose and John J Prevost. Foleygan: Visually guided generative adversarial network- based synchronous sound generation in silent videos. IEEE Transactions on Multimedia, 2022.
[9] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15190- 2023.
[10] Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEFE Transactions acoustics, speech, and signal processing, 52(2):236-243 1984.
10

---

[11] Yiwei Guo, Chenpeng Du, Ziyang Ma. Xie Chen, and Kai Yu. Voiceflow: Efficient text-to- speech with rectified flow matching. In ICASSP 2024-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121-11125. IEFF, 2024.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 2020.
[13] Jiawei Huang, Yi Ren, Rongji Huang. Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanned auxioo generation. arXiv preprint 230.:23305.1877 2023.
[14] Po-Yao Yao Huang, Vasu Sharma, Hu Xu, Chaitanya Ryali, Yanghao Li, Shang- Wen Li, Gargi Ghosh, Jitendra Malik, Christoph Feichtenhofer, et al. Mavil: Masked audio-video learners. Advances in Neural Information Processing Systems, 36, 2024.
[15] Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. The 32st British Machine Vision Virtual Conference. BMVA Press, 2021.
[16] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilin- gual universal speech generation scale. Advances in neural information processing systems, 36, 2024.
[17] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2022.
[18] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2022.
[19] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.122500 2023.
[20] Haohe Liu, Qiao Tian, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Yuping Wang, Wenwu Wang, Yuxuan Wang. and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprins arXiv:2308.0m 05734, 2023.
[21] Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. The Eleventh International Conference on Learning Representations, 2022.
[22] Xingchao Liu. Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based rext-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
[23] Yixin Liu, Kai Zhang. Yuan Li. Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17117. 2024.
[24] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dom-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 55:5775-5787 2022.
[25] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models. Advances Neural Information Processing Systems, 36, 2024.
[26] Shivam Mehta, Ruibo Tu, Jonas Beskow, féva Székely, and Gustav Eje Henter. Matcha-tts: fast tts architecture with conditional flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11341-11345. IEEE, 2024.

---

[27] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8788778 -863., PMLR, 2021.
[29] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 10684-10695, 2022.
[30] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- -ottte--mage diffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-364939 2022.
[31] Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
[32] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[33] Yuan Tseng, Layne Berry, Yi-Ting Chen, I-Hsiang Chiu. Hsuan-Hao Lin, Max Liu, Puyuan Peng. Yi-Jen Shih, Hung-Yu Yu Wang, Haibin Wu, et al. Av-superb: A multi-task evaluation benchmark for audio-visual representation models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6890 6894. IEEE, 2024.
[34] Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821 2023.
[35] Heng Wang. Jianbo Ma, Santiago Pascual. Richard Cartwright, and Weidong Cai. V2a-mapperr lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15492-15501 2024.
[36] Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, et al. Freebind: Free lunch in unified multimodal space via knowledge fusion. In Forty-first International Conference on Machine Learning.
[37] Zehan Wang, Ziang Zhang, Luping Liu, Yang Zhao, Haifeng Huang, Tao Jin, and Zhou Zhao. Extending multi-modal contrastive representations. arXiv preprint urXiv:2310.088 2023.
[38] Zehan Wang, Yang Zhao, Haifeng Huang. Jiageng Liu, Aoxiong Yin, Li Tang, Linjun Li, Yongqi Wang, Ziang Zhang, and Zhou Zhao. Connecting Advances in Neural Information Processing Systems, 36:22099 200-222112 14, 2023.
contrastive representations.
[39] Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces. arXiv preprint wwXiv:2407.11899 2024.
[40] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor SirgKKirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-aute audio pretraining with feature fusion and keyword- to-caption augmentation. In /CASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
[41] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Sceing and hear- ing: Open-domain visual-audio generation with diffusion latent aligners. arXiv preprint 11XIv:2402.11772,, 2024.
12

---

A Implementation details
Table 6: Architecture details of ID VAE for spectrogram compression.
Hyperparameter
1D VAE
Input tensor shape for 10-sec audio
(80,624)
Embedding dimension
20
Channels
224
Channel multiplier
1, 2,
Downsample layer position
after block after block
Attention layer position
Output tensor shape for 10-sec audio
(20,312)
Table 7: Hyperparameters of the vector field estimator of FRIEREN with different sizes.
Hyperparameter
Small
Base
Large
Layers
4
6
Hidden dimension
384
576
768
Attention heads
8
8
ConvID-FFN dimension
1,536
2,304
3,072
Number of parameters
70.90M
158.88M
421.12M
In table we provide the architecture details the mel-spectrogram VAE. Different from the commonly used 2D VAE for spectrogram, the ID VAE we adopt does not involve an extra channel dimension, but takes the frequency axis of the spectrogram the channel dimension, and conducts convolution along the temporal axis. This design is derived from the insight that the spectrogram is translation invariant along the frequency axis, and it can better synergize with the feed-forward transformer. In table 7 we present the hyperparameters of the vector field estimator networks with different sizes.
Additionally, we observe that although there is no significant difference in objective metrics, ini- tializing the vector field estimator with the weights of a diffusion model for text-to-audio (T2A) generation [3] helps improve the subjective perceptual quality of the generated audio marginally. This improvement derives from the knowledge of audio generation on broader data domain learned by the T2A model. We adopt this trick in our model training.
B Subjective evaluation
Seleet option
Good good
2.5
unaccapaaa callty...
Figure 6: Screenshot of subjective evaluation on audio quality.
For each evaluated model, we select 150 items for subjective evaluation, accounting for about 1% of the entire test split.
Our subjective evaluation tests are crowd-sourced and conducted via Amazon Mechanical Turk. For audio quality evaluation, ask the testers to examine the audio quality and ignore the content. And
13

---

Select opton
Good syccesiiio
35 Tal- redert
Figure Screenshot of subjective evaluation on temporal alignment.
for temporal alignment, instruct the testers to evaluate the synchrony between the background
audio and the video content, while ignoring the audio quality. The testers rate scores on 1-5 Likert scales. We provide screenshots of the testing interfaces in and 7 Each data item is rated by 6 testers, and the testers are paid $8 hourly.
C Time efficiency
Table 8: Inference time per sample of different models with batch size 1.
Model
Inference Time (sec)
SpecVQGAN
3,936
Im2Wav
333.246
Doff-Foley (step=25)
2.104
FRIEREN (Dopri5, step=25)
1.510
FRIEREN
(Euler, step=25)
0.288
FRIEREN (Euler, step=5)
0.064
FRIEREN (Euler, step=l)
0.031
In table compare the inference time per sample of different models. The inference is conducted on a single RTX-4090 GPU with a batch size of 1. We can see that the inference procedure of wransformer-based autoregressive models, including SpecVQGAN and Im2Wav, is time- consuming, especially for Im2Wav, which takes several minutes to generate a single sample. This because Im2Wav conducts cascaded generation with transformers. Moreover, its use of high- bitrate audio VQVAE results in very long sequences of audio representation, significantly increasing the inference time required for the transformers, which has quadratic time complexity concerning sequence length. In contrast, Diff-Foley and FRIEREN require less inference time, and FRIEREN with Euler solver enjoys a higher speed, achieving 7.3 times faster than Diff-Foley with 25 sampling steps. This is the result of a combination of multiple factors, including model architecture, model parameters, sampling methods, and so on. Furthermore, when using FRIEREN model with reflow and one-step distillation, we can generate 5-step sampled audio in 0.064 seconds and 1 1-step sampled audio in just 0.031 seconds, achieving 4.5 and 9.3x acceleration compared 25-step sampling. This demonstrates the extremely high generation efficiency of our model on the task of V2A.
D Impact of the vocoder on model performance
Different selections of vocoders can significantly impact the performance of various audio generation models. Diff-Foley uses the simple Griffin-iim method [10] to map spectrograms waveforms, while FRIEREN employs the more efficient BigVGAN. To compare the performance of the spec- trogram generation models while minimizing the influence of the vocoder, we apply BigVGAN and Griffin-Lim separately to each model. The output from Diff-Foley is converted into an 80-bin mel-spectrogram and then fed into VGAN. The number Griffin-Lim iterations for FRIEREN the same as Diff-Foley. The results are shown in table 9
14

---

Table 9; Comparison of the performance of Diff-Foley and FRIEREN using the same vocoder.
Model
FDL
1St
KL4
FAD1
KID(I0 4
BigVGAN
Doff-Foley (CG ))
18.02
10.89
2.88
6.32
5.32
FRIEREN
12.26
12.42
2.73
1.32
2.49
Griffin-Lim
Doff-Foley (CG ))
23.94
11.11
3.38
4.72
9.58
FRIEREN
28,29
10.67
3.17
3.70
12.30
be seen that using BiAN for Diff-Foley improves its FD, KL, and KID, indicating its effectiveness. On this basis, FRIEREN outperforms Diff-Foley across all metries, with greater difference than when using Griffin-Lim for both. This further demonstrates that our model is superior to Diff-Foley.
On the other hand, when using Griffin-Lim for both models, despite the performance drop, FRIEREN still surpasses Diff-Foley in KL. and FAD, with FAD showing a significant advantage while maintain- ing competitive FD and IS values. We speculate that the Griffin-Lim algorithm weak that forms a performance bottleneck, narrowing the performance gap between FRIEREN and Diff-Foley. Addi- tionally, differences spectrogram hyperparameters may also lead performance gap. Diff-Foley uses 128 frequency bins, more than the 80 bins used by FRIEREN, allowing it to carry finer-grained information and may give Diff-Foley an advantage when using Griffin-Lim.
E
Limitations and boarder impacts
Limitations Despite that FRIEREN achieves outstanding performance on audio quality, temporal alignment, and generation efficiency, still has two major limitations: 1) Currently, experiments have only been conducted on small-scale dataset, VGGSound, and we have not yet scaled the model to large-scale datasets. Therefore, it is still difficult to apply our model to a wide range real-world scenarios for now; 2) our current model design only targets audio generation for fixed-length short video clips, and it lacks the ability of audio generation long videos with various lengths. We will explore the solutions to these issues in future work.
Potential positive impacts The achievements our model on the V2A task may reduce the cost of sound effect synthesis, and could potentially drive advancements in the film, gaming, and social media industrics.
Potential negative social impacts The automatic sound effect generation technology may lead to job losses for related personnel. Additionally, there is a risk the model being used to generate harmful content fake media. Constraints needed to guarantee that people will not use the model in illegal cases.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to abstract and section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
16

---

Justification: As an application-orienne orieente paper, the theory of the rectified flow model we rely upon has been thoroughly demonstrated and proven the related original papers. We cite these original papers and encourage readers to refer to them for the complete proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer to section 4. and appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due to time limiations, we have not yet organized a publicly shareable version of the code. We will open-source our code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudoueo Coaee//ddeeSSbbmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer section 4.1 appendix A and appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report 95% confidence interval of subjective evaluation results (MOS-Q and MOS-A). No statistical significance applicable for objective metrics
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Model configuration in section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of
Answer: [Yes]
Justification:
We faithfully adhered to the NeurlPS Code of Ethics throughout research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: At this stage, have not made code and models open-source. We will consider relevant protection measures during the open- open-soureing stage.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For code and data our paper, we cite the original papers and comply with their licensing.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Please refer to appendix B
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The crowdsourced evaluations we conduct do not involve potential risks or activities requiring approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
21


Reviews:
[{"review_id": "FyysAfrOLb", "reviewer": "Reviewer_MSJt", "summary": "This paper proposes a diffusion model based on rectified flow matching. Besides, to generate better audio quality, the authors propose re-weighting objective. The method  achieves the state-of-the-art results on V2A benchmark.", "questions": "* Do the authors use any pretrained initialization for transformer?\n\n* The design of Fig2b is very similar to standard ViT block. Any intuitions or differences between these two?\n\n* In Fig2b,  are the c latents, video features, performed any pooling layer?", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed method is the first to leverage rectified flow matching on video-to-audio generation tasks.\n\n* The quantitative and qualitative results demonstrate the superiority comparing with existing baselines.", "weaknesses": "* Although FRIEREN shows impressive results, the competing methods ( i.e., Diff-foley) based on U-Net style diffusion model are relatively weak. The performance gain seems mostly come from transformer architecture.\n\n* Following previous point, DDPM shows pretty similar results when increasing steps. It makes the proposed method less stronger. Thus, it would be great to show the results with more steps. \n\n* The proposed method, reflow, cannot consistently benefit FAD on different number of step. It seems not reasonable.\n\nOverall, the results are good. If the authors can address some questions and more insight (comparing to simple adapting reflowing in V2A like speech model), that would make the paper more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments on our work. We would like to discuss the issues you raised here.\n\n**[Effect of transformer architecture and rectified flow]**\n\nWe believe that both the transformer architecture and the rectified flow (RF) modeling method contribute to the model performance. We will elucidate the role of RF from the following perspectives.\n\n1. According to Table 1 in the paper, our RF model demonstrates an advantage in IS, KL, FAD, and Acc, with differences being 2.33, 0.13, 0.45, and 1.89, as well as higher MOS. These differences are actually quite significant. When the sampling steps increase to 40, these advantages remain consistent (see the table below). We believe these results serve to demonstrate the significant positive effect on model performance that RF has.\n\n    | Model | Step | FD↓ | IS↑ | KL↓ | FAD↓ | KID $\\times10^{-3}$ ↓ | ACC↑ |\n    | --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n    | DDPM | 40 | 11.63 | 10.28 | 2.87 |1.72 | 2.18 | 95.26 |\n    | Frieren | 40 | 11.87 | **12.63** | **2.74** | **1.31** | 2.39 | **97.19** | \n\n2. Differences in the sampler may potentially reduce the performance gap between RF and DDPM. We adopt an advanced solver, DPM-Solver, for DDPM, in contrast to the simplest Euler solver for RF. Due to the differences in the models, it is difficult to eliminate this effect through an entirely consistent sampler. However, we can further unlock RF's potential by employing a more advanced sampler for RF. The following table shows the results of Frieren with the Dormand–Prince method (dopri5). We can see that the RF model holds an advantage in almost all metrics, especially in IS, FAD, and Acc, with only a slight disadvantage in KID. This further indicates the advantage of RF.\n\n    | Model | Sampler | Step | FD↓ | IS↑ | KL↓ | FAD↓ | KID $\\times10^{-3}$ ↓ | ACC↑ |\n    | --- | --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n    | DDPM | DPM-Solver | 25 | 11.79 | 10.09 | 2.86 |1.77 | 2.36 | 95.33 |\n    | Frieren | dopri5 | 25 | **11.63** | **12.76** | **2.75** | **1.37** | 2.39 | **96.87** | \n\n3. Lastly, RF **not only enhances the quality** of generated audio but also reduces sampling steps through reflow and distillation, significantly improving the model's **generation efficiency**, which DDPM cannot achieve.\n\n**[Effect of reflow under different steps]**\n\nWe briefly discussed this issue **at the end of Section 4.2** and we'd like to provide a possible explanation with more details here. Theoretically, reflow should not alter the model’s marginal distribution. Yet in practice, the limited number of steps for reflow data generation (25 steps in our experiments) can affect the data quality, introducing errors into the regression targets during the reflow process. While reflow can straighten trajectories and improve generation quality with a few steps, such errors in the regression targets can degrade the model's generation quality with over 25 steps, leading to reductions in metrics such as FAD and IS. This might be mitigated by increasing the number of sampling steps during reflow data generation. Moreover, \nincreasing the number of iterations of generating reflow data and conducting reflow can lead to the accumulation of more errors, and that's why we only generate data once for both reflow and distillation (as discussed in section 3.5). \n\n**[Design of transformer block]**\n\nOur design of transformer block derives from Make-an-Audio 2[1], which is proven to be efficient in audio generation, despite that it does not necessarily outperform standard ViT block significantly. In other words, it is not necessarily the best choice, but it is sure to be a good one.\n\n**[Model initialization]**\n\nYes. We load the weight of the diffusion denoiser in Make-an-Audio 2[1], which is a text-to-audio ddpm model trained with more data. Despite that we did not observe significant improvement in metrics and the rate of convergence, it seems to slightly help the subjective perception quality. We will supplement these details in the revised version of our paper.\n\n\n**[Pooling on condition latent]**\n\nNo. No pooling is conducted on the condition sequence, as we want to keep the temporal information for generating synchronized audio.\n\n**[Difference with RF-based speech models and other Insights]**\n\nWe agree that our method has some similarities with RF-based speech models, such as VoiceFlow [2]. However, we believe our model delves more deeply into certain aspects.\n\n1. Compared to the strong and highly deterministic content condition (text) in TTS, V2A has a weaker condition and its performance relies more on guidance. Compared to RF TTS models, we found that using the CFG-corrected vector field as the regression target during the reflow stage is crucial for model performance, rather than using the same vector field as in previous RF models (see section 3.5). \n\n2. Upon reflow, we further investigate one-step distillation, which further improves the single-step performance and previous speech models did not address it.\n\n---\n\nWe hope our clarifications address your concerns.\nIf you find our response helpful, we would very appreciate it if you consider increasing your evaluation of our work. And we always welcome further discussion with you. Thank you again for your efforts.\n\n---\n\n[1] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023.\n\n[2] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121–11125. IEEE, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wNzwBrAQlM", "reviewer": "Reviewer_faHj", "summary": "The following work proposes a video to audio generation model. The model architecture closely follows that of prior work diff-foley, which operates on 4 frames per second, fits a temporally aligned latent space between audio and video content, and then a latent diffusion model to map from this latent space to audio. This work proposes to replace the latent diffusion model architecture with a transformer-based rectified flow model, and opts to use the MAViL audio-video joint latent representation instead of the one from diff-foley (CAVP). Results are qualitatively much better than that of diff-foley, and also faster to sample from due to the rectified flow matching formulation.", "questions": "- I'm curious how the authors were able to try MAViL given that the code for this project does not appear to be publicly available?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Qualitative results significantly improve over prior work\n- Decent ablation studies over critical architectural design choices, such as CAVP vs MaVIL, loss-reweighting for training flow matching models.", "weaknesses": "- Despite improvements over prior works such as diff-foley, the contributions of this work remain limited. The time-aligned audio generation  appears to stem from architectural choices made in diff-foley. \n- Furthermore conditional-optimal-transport flow-matching generative models have been applied audio models with similar conclusions. The specific application to the video-to-audio task, in my opinion, is not sufficiently different from prior applications in audio for the findings in this work to be particularly new. Specifically, it should be considered to be very closely related to other temporally-aligned conditional generation tasks such as text-to-speech.\n- It's also worth noting that Diff-Foley uses a very simple griffin-lim to map predicted spectrograms to audio waveforms, whereas this work makes use of the much more effective BigVGAN model. This makes it very difficult to pinpoint the qualitative improvements of the proposed work compared to prior methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We'd like to make some clarification and discussion about the issues you raised.\n\n**[Difference with Diff-Foley in architecture]**\n\nWe'd like to clarify that our model **significantly differs from Diff-Foley** in both **architecture** and **alignment mechanisms**. Diff-Foley adopts a U-Net denoiser with a cross-attention-based conditional mechanism. As stated in sections 1 and 4.2, cross-attention alone struggles to achieve precise audio-visual alignment, and therefore Diff-Foley relies on an additional classifier for guidance, which is complicated and unstable with fewer steps. In contrast, we adopt a transformer vector field estimator with channel-level cross-modal feature fusion, achieving higher synchrony and robustness with simpler architecture. \n\n**[Difference with other flow-matching-based audio models and novelty issues]**\n\nWe agree that our model shares some similarities with previous flow-matching-based audio models. Nevertheless, we have delved deeper into certain aspects compared to previous speech models like VoiceFlow [1].\n\n1. Compared to the strong and highly deterministic content condition (text) in TTS, V2A has a weaker condition and its performance relies more on guidance. We found that using the CFG-corrected vector field as the regression target during the reflow stage is crucial for model performance, rather than using the same vector field in both initial training and reflow as in previous rectified flow models (see section 3.5). \n\n2. Upon reflow, we further investigate one-step distillation, which further improves the single-step performance and previous flow-matching-based audio models did not address it. We also investigate techniques like objective reweighting for further performance improvement.\n\n**[Effect of vocoder on qualitative results]**\n\nWe agree that the vocoder significantly impacts audio fidelity and objective metrics. Our target is to build an integral system for video-to-audio generation with higher quality and generation efficiency, and therefore we replace the slow and low-quality Griffin-Lim with BigVGAN. For reference, we provide the results of Frieren with Griffin-Lim as the vocoder in the following table. The number of Griffin-Lim iterations is the same as Diff-Foley.\n\nIt can be seen that despite the performance drop, Frieren still surpasses Diff-Foley in KL, FAD, and ACC, with FAD showing a significant advantage while maintaining competitive FD and IS values.\n\n| Model | FD↓ | IS↑ | KL↓ | FAD↓ | ACC↑ |\n| --- | :---: | :---: | :---: | :---: | :---: |\n| Diff-Foley (w/ CG) | 23.94 | 11.11 | 3.28 | 4.72 | 95.03 |\n| Diff-Foley (w/o CG) | 24.97 | 11.69 | 3.23 | 7.10 | 92.53 |\n| Frieren (Griffin-Lim) | 28.29 | 10.67 | **3.17** | **3.70** | **95.22** |\n\nMoreover, due to the limitations of objective metrics, we **highly recommend** you refer to our **demo page (https://frieren-v2a.github.io/)**. It can be observed that in addition to audio fidelity, the samples from our model exhibit better semantic content and more precise temporal alignment compared to Diff-Foley, demonstrating the qualitative advantages of our rectified-flow-based model.\n\nIt's also worth mentioning that other than audio quality, Frieren achieves a generation **speed $7.3\\times$ that of Diff-Foley** (see table 8 in the paper, taking only the spectrogram generation for consideration), demonstrating a significant advantage in terms of generation efficiency.\n\n\n**[Source of MAViL]**\n\nWe adopt the MAViL implementation and checkpoints from a public-available audio-visual representation benchmark project (AV-SUPERB, https://github.com/roger-tseng/av-superb). We made a slight modification to the model input so that it takes 4FPS video rather than 2. Due to concerns about potentially violating anonymity policies by using links in our rebuttal, we must state that there is no overlap or connection between the authors of this project and our paper.\n\n---\n\nWe hope our clarifications address your concerns and we are looking forward to your re-assessment of our work. We also welcome further discussion with you. Thank you again for your efforts.\n\n---\n[1] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121–11125. IEEE, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KqxN8YT6Ak", "reviewer": "Reviewer_hYKZ", "summary": "This paper presents a new model for video-to-audio generation. The proposed model is based on rectified flow formulation and adopts Transformer-based architecture. A conditional video is fed into the model via channel-level concatenation to the audio tokens after processed by a length regulator. After training of the model, the model is further fine-tuned with the reflow and distillation. They are conducted with synthetic data generated by the firstly trained model with classifier-free guidance. The experimental results demonstrate that the proposed model outperforms the existing models by a large margin both quantitatively and qualitatively.", "questions": "- Is there any particular challenge (and its solution) when applying rectified flows for audio generation?\n\n- Minor questions:\n  - Is CFG also applied for the reflowed models? I understand that it is applied when generating the training data for the reflow process but cannot find how it is set during the inference phase.\n\n---\n\n<After the rebuttal>\n\nI updated my rating from 5 to 7.", "limitations": "Limitations have been discussed in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The design of the proposed model is simple and reasonable. The proposed model is based on Transformer, and the video condition is fed into the model via channel-level concatenation to the audio tokens after adjusting the number of tokens. This design would be beneficial for boosting the temporal alignment, as it explicitly utilizes the temporal correspondance between the conditional video and the generated audio.\n\n- In the experiments, the proposed method outperforms the other existing methods by a large margin. I have checked the generated examples on the website, and they are really amazing.\n\n- The proposed model is quite light-weight, and it is great to be able to train the model with only two GPUs. In addition, the inference speed is substantially fast thanks to the reflow and distillation as well as the light-weight design.\n\n- The manuscript is well-written and easy to follow.", "weaknesses": "- The experiments have only been conducted with one dataset, which is VGGSound. Training or zero-shot evaluation with other datasets (such as Landscape dataset) would be beneficial to validate the generalization capability of the proposed method.\n\n- The empirical analysis on why the proposed method performs well seems insufficient. According to the results shown in Table 2, DDPM with the proposed model architecture already achieves substantially better performance than the existing methods. Thus, it appears that the model architecture, rather than the usage of the rectified flow, is the key for the impressive performance. As far as I understand, its major difference from the standard Transformer is two-fold: channel-level concatenation for the conditional inputs instead of sequence-level one (or cross-attention mechanism as in [23]) and the usage of 1D-conv instead of 2D-conv. It would be great if this paper could provide an empirical analysis on which component actually boosts the performance for video-to-audio generation. The current manuscript places significant emphasis on the rectified flow aspect, which is not particularly novel as the proposed model largely follows to the settings of previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly grateful for your positive appraisal of our work, and we'd like to discuss the issue you raised here.\n\n**[Generization experiments on landscape dataset]**\n\nFollowing your advice, we conduct experiments on the landscape dataset to investigate the generalization capability of our model. We compare our zero-shot performance with Diff-Foley. We also try to finetune our model on landscape for about 4k steps (about 268 epochs). The results are illustrated in the following table. \n\n| Model | Mode | FD↓ | IS↑ | KL↓ | FAD↓ | KID $\\times10^{-3}$ ↓ | \n| --- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Diff-Foley | zero-shot | 76.98 | 2.96 | 4.16 | 9.70 | 41.50 |\n| Frieren | zero-shot | 34.87 | 4.15 | 4.12 | 2.64 | 12.29 | \n| Frieren | finetuned | 30.38 | 3.74 | 6.12 | 1.94 | 12.28 | \n\nIt can be seen that the zero-shot performance of Frieren significantly outperforms Diff-Foley in multiple metrics. On the other hand, we observe that finetuning improves in FD and FAD, with the differences being 4.49 and 0.70. However, it also leads to degradation of 0.41 and 2.00 in IS and KL, respectively. Due to the limited size of the landscape dataset, there may be a distribution gap between the training and testing splits. Finetuning could lead to a degree of overfitting on the training set, resulting in a decline in certain metrics.\n\n\n**[More empirical analysis on model performance]**\n\nFirst, we'd like to claim that both the transformer architecture and the rectified flow (RF) modeling method contribute to the model performance (please **refer to our response to reviewer MSJt** for details and additional results). Our rectified flow model brings improvement in IS, FAD, and Acc while enabling generation with fewer or even one step. Besides, adopting a better ODE solver can further improve the performance of RF and increase its performance gap with DDPM.\n\nSecond, we have conducted experiments with sequence-level concatenation for the conditional inputs (similar to cross-attention essentially, as the alignment is learned by attention). However, this model fails to generate meaningful audio, and the metrics are unacceptably bad as shown in the table below. \n\n| Cond Mechanism | FD↓ | IS↑ | KL↓ | FAD↓ | KID $\\times10^{-3}$ ↓ | ACC↑ |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Channel-Level Concat | 12.25 | 12.42 | 2.73 | 1.32 | 2.49 | 97.22 | \n| Sequence-Level Concat | 83.92 | 1.62 | 22.16 | 12.31 | 41.63 | 28.91 |\n\nWe also provide two pairs of results of sequence-level and channel-level concatenation in the PDF. It can be seen that the sequence-level model tends to generate flat, monotonous, and meaningless audio. However, the frequency bands where energy is concentrated are similar in the results of the two models, with the bright lines on the spectrograms being of similar height. We speculate that this indicates that the sequence-level model can extract semantic information from conditional inputs, but it fails to learn temporal alignment through attention. Adopting different positional embedding doesn't help. This seems a little weird, as cross-attention shows a fundamental alignment learning ability in the baseline diffusion model, but it just turns out not working in our architecture. Maybe it has something to do with model size and capacity. Anyway, these results illustrate the necessity of channel-level feature fusion.\n\n\nLast, the design of our transformer block with 1D convolution derives from a diffusion-based text-to-audio generation model[1], and it is proven in the paper to perform better than 2D-convolution-based U-Net on audio generation, and has better potential to generalize to audio of longer and variable lengths. Due to the limited response time, we haven't made it to implement and train a 2D version of the model to examine the performance gap yet. We plan to add these results in the final revised version of our paper. \n\n**[Challenges in applying rectified flows for audio generation]**\n\nCompared to other tasks, we think that applying RF in video-to-audio generation faces the following challenges:\n\n1. Our method shares some similarities with RF-based TTS models like VoiceFlow [2]. However, we find that compared to the strong and highly deterministic content condition (text) in TTS, V2A has a weaker condition and its performance relies more on guidance. Compared to RF TTS models, we found that using the CFG-corrected vector field as the regression target during the reflow stage is crucial for model performance, rather than using the same vector field as in previous RF models (see section 3.5). \n\n2. As stated above, compared to text-conditioned generation like T2I and T2A, it turns out attention-based conditional mechanism fails to provide precise information in terms of semantic and temporal alignment for V2A. Hence we propose the channel-level feature fusion used with the feed-forward transformer architecture.\n\n**[CFG in reflow]**\n\nYes. As shown in equations (8) and (9), we use the CFG-corrected vector field as the regression target during the reflow and distillation, where the CFG scale is the same as that used for generating reflow data. And we use the same CFG scale for sampling with the reflowed model.\n\n---\n\nOnce again, thank you for your effort in reviewing our work and your acknowledgment. We hope our clarifications address your concerns, and we always welcome further discussion with you.\n\n---\n\n[1] Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio 2: Temporal-enhanced text-to-audio generation. arXiv preprint arXiv:2305.18474, 2023.\n\n[2] Yiwei Guo, Chenpeng Du, Ziyang Ma, Xie Chen, and Kai Yu. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11121–11125. IEEE, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmjxPWnBwi", "reviewer": "Reviewer_WgLP", "summary": "They propose FRIEREN, an efficient video-to-audio generation model based on rectified flow matching that obtains state-of-the-art results.", "questions": "- Do you plan to release the code?\n- How do you compute alignment accuracy?\n- Why not using CLIP for visual representation?", "limitations": "- 16kHz and short-form videos.\n- No code/weights provided.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- They successfully propose rectified flow matching for video-to-audio, that is a problem that is important in current generative AI setup where most video generative models are generating video without audio.\n- They run a perceptual study.\n- The paper (specially section 3) is very well written and clear.\n- I appreciate the examples in the demo webpage. Also the selection of the examples, that does not feel cherry picked.", "weaknesses": "The introduction lacks scientific rigor.\n- line 36: \"leave room for further advancement\". This is a general statement, can you be more specific?\n- line 36: \"autoregressive models lack the ability to align the generated audio with the video explicitly\". This is not true, because AudioLM and MusicLM are autoregressive models that use explicit semantic tokens to capture structure similar to the conditioning in video-to-audio.\n\nI could not find how you compute alignment accuracy.\n\nMinor comment related to scientific writing:\n- line 18: \"revolutionary enhancements\". It feels like marketing and this is a scientific paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your positive appraisal of our work and would like to discuss the issues you raised here.\n\n**[Computing alignment accuracy]**\n\nAs stated in the **Metrics part, Section 4.1**, we adopt the alignment classifier provided in Diff-Foley[1] for calculating alignment accuracy. Specifically, we convert the generated audio to 128-bin mel-spectrogram and feed it to the classifier along with the CAVP features. We will emphasize these details in the revised version.\n\n**[Alignment learning ability of autoregressive models]**\n\nThanks for pointing out this. We agree that recent large-scale autoregressive models like AudioLM and MusicLM have shown a strong ability to learn alignment between different modalities with self-attention and in-context learning. However, the good performance of these models often relies on large-scale transformer decoders and substantial amounts of training data. Early autoregressive baselines on VGGSound perform poorly in terms of temporal synchrony (see table 1 in our paper), this may be due to the limited model capacity and training data volume. We will modify our statements in the revised version of our paper.\n\n**[Other wording and scientific rigor issues]**\n\nThank you for pointing out the issues in our writing, we will revise our wording in the revised version. For example, we may change line 18 to *...\nsignificantly enhanced the quality and diversity...*, and change line 36 to *...but still has a gap in quality compared to state-of-the-art text-to-audio models and real-world audio.*\n\n**[Selection of visual features]**\n\nIn previous work [1], CLIP has been shown to be not effective enough in generating temporally aligned audio. In this work, we mainly use CAVP for a fair comparison with [1] and attempt to find a better visual representation for video-to-audio generation (taking MAViL as an example).\n\n**[Sampling rate and duration issues]**\n\nFollowing most previous audio generation models, we adopt 16kHz audio. This could be improved by using spectrogram, VAE, and vocoder with a higher sampling rate, which shouldn't be a big problem. On the other hand, most public-available on-the-shelf video-to-audio data, like VGGSound and AudioSet, are short video clips, restricting extending generation lengths. We may delve into these issues in future work.\n\n**[Plan for releasing code and weights]**\n\nThanks for your interest. We plan to release our code and weights on GitHub several weeks later, no matter whether our paper gets accepted or not.\n\n---\n\nOnce again, thank you for your effort in reviewing our work and your acknowledgment. We welcome further discussion with you.\n\n---\n\n[1] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To all reviewers, ACs, and PCs:

We thank all reviewers for their valuable suggestions with their effort and time. Your comments have improved our work. We have individually responded to the comments and concerns of each reviewer. Please refer to each response for details.

In order to better illustrate the effect of the channel-level fusion condition mechanism to respond reviewer hYKZ, we provide a PDF showing the generated spectrogram of our model with channel-level and sequence-level concatenation.

We sincerely hope that our responses have addressed the concerns raised by the reviewers and welcome further discussions. Once again, thank you for your time and efforts.

Best regards, Authors.

Meta-review: Audio generation from videos/images has been studied for sometime now but most models suffer from complexity of proposed architecture (in efficient for real time) or lack of perceptual correctness. The authors propose a simple architecture based on transformer flow with a tweak to diffusion loss, leading to small sample processing based synthesis. The authors show significant improvement over state of the art in both objective and perceptual studies. 

There is general agreement among the reviewers regarding the importance and sufficiently broad impact of the work. The authors are suggested to capture the comments/replies into the final paper.
Decision: Accept (poster)