Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers
Authors: Jibin Song
Mingi Kwon
Jaeseok Jeong
Youngjung Uh
Venue: ICLR 2026 Poster (2026)
Keywords: Audio-to-Video Generation
Multimodal Synthesis
Temporal Synchronization
Diffusion Transformer
Video Generation
Audio-Conditioned Generation
Primary area: generative models
TL;DR: We propose improved audio-aligned video generation by leveraging a pretrained video generation model, while preserving its original performance.

Abstract: Text-to-video and image-to-video generation have made rapid progress in visual quality, but they remain limited in controlling the precise timing of motion. 
In contrast, audio provides temporal cues aligned with video motion, making it a promising condition for temporally controlled video generation. 
However, existing audio-to-video (A2V) models struggle with fine-grained synchronization due to indirect conditioning mechanisms or limited temporal modeling capacity.
We present Syncphony, which generates 380×640 resolution, 24fps videos synchronized with diverse audio inputs. Our approach builds upon a pre-trained video backbone and incorporates two key components to improve synchronization: 
(1) Motion-aware Loss, which emphasizes learning at high-motion regions; 
(2) Audio Sync Guidance, which guides the full model using a visually aligned off-sync model without audio layers to better exploit audio cues at inference while maintaining visual quality.
To evaluate synchronization, we propose CycleSync, a video-to-audio-based metric that measures the amount of motion cues in the generated video to reconstruct the original audio. Experiments on AVSync15 and The Greatest Hits datasets demonstrate that Syncphony outperforms existing methods in both synchronization accuracy and visual quality.

Published as a conference paper at ICLR 2026
SYNCPHONY: SYNCHRONIZED AUDIO-TO-VIDEO GENERATION WITH DIFFUSION TRANSFORMERS
Jibin Song' :2, Mingi Kwon Jaeseok Jeong 1.2 Youngjung Uh'
DDepartment of Artificial Intelligence, Yonsei University Linnelingo"
{s jbpphh, kwonmingi, jete- yj. wh..yooseeii.ccc ac.
ABSTRACT
Text-to-video and image-to-video generation have made rapid progress in visual quality, but they remain limited in controlling the precise timing of motion. In contrast, audio provides temporal cues aligned with video motion, making a promising condition for temporally controlled video generation. However, exist- ing audio-to-vider (A2V) models struggle with fine-grained synchronization due to indirect conditioning mechanisms or limited temporal modeling capacity, We present Syncphony, which generates 380x640 resolution, 24fps videos synchro- nized with diverse audio inputs. Our approach builds upon pre- mre-trained video backbone and incorporates two key components to improve synchronization: (1) Motion-aware Loss, which emphasizes learning high-motion regions; (2) Au- dio Syne Guidance, which guides the full model using a visually aligned off- sync model without audio layers better exploit audio cues inference while maintaining visual quality. To evaluate synchronization, we propose CycleSync, a video-to- daleo-to-audio-- audio-based metric that measures the amount of motion cues in the gen- erated video to reconstruct the original audio. Experiments on AVSync15 and The Greatest Hits datasets demonstrate that Syncphony outperforms existing methods in both synchronization accuracy and visual quality. Project page is available at: httpss/iibnnggguuuubbidooooppoyo
INTRODUCTION
Video generation has achieved remarkable progress especially in text-to-video (T2V) and image-to- video (I2V). They synthesize visually crisp and temporally coherent videos that match the given text prompt and/or starting frame. However, we still need additional ways to control the motions that are difficult control by the texts or the starting frames. example, texts inherently lack explicit timings of when and how motions would occur, although they may describe motions, e.g., "dog barking" and "striking bowling". In what rhythm would the dog bark? When is the ball released, how fast does it roll, and when does it hit the pins? Similarly, image-based conditions also face inherent limitations. An image can convey information about the appearance, pose, background, and layout of the scene, but it represents only static snapshot of single moment.
In contrast, audio signals inherently carry temporal clues because audio and video share the same temporal axis. Returning the earlier examples, the accompanying audio would provide how many times and exactly when the dog barks, when the ball is released, how quickly travels, and when hits the pins. Therefore, tackle generating videos that are synchronized to audios.
Even with audio, text, or image conditions, existing audio-to-vides methods (Lee et al.) 2023 Jeong al. 22023] Yariv al.) 2023 Zhang al.) 20244) struggle with fine-grained synchronization be- tween audio and motion. These approaches rely on indirect mappings, such as magnitude- based adjustments (Lee et al. 2023) or audio-to-text projections (Jeong et al. 2023 Yariv et 2023), which fail to reflect the complex and detailed temporal structures in audio signals. Instead, we di- rectly inject audio features into the visual generation process via cross-attention mechanism, en-
abling audio-motion alignment. In parallel, compared to T2V models (Jin et al. 2024a HaCohen
"[jbii nsong, minnikkwon jasseeochhung hi abs com

---

Published as a conference paper at ICLR 2026
et al. 2024 Blattmann al. 2023: Wan al.
2025 which generate high-resolution, high-frame- rate, temporally coherent videos, Zhang et al. (024) adds temporal layers to an image backbone, training them from scratch with limited data (c.g., 6 fps at 256x256 resolution) leading to broken temporal coherence, such as flickering and saturation artifacts. We address this by leveraging a pre- trained video backhone with strong temporal modeling capabilities, resulting in more stable and consistent motion.
Despite these advancements, audio-to-video generation still faces fundamental challenge: MSE- based objectíves alone are insufficient for modeling accurate motion timing and appropriate motion magnitude. In diffusion or flow models, MSE is effective for reconstructing overall visual appear- ance, but it provides weak supervision for localized, precisely timed motion changes. Therefore, stronger and more targeted supervision needed regions that exhibit significant motion.
To this end, we propose Syncphony, which generates high-quality videos at 380x640 resolution, 24fps, and up to 5 seconds in length, and most of all, synchronized to audio. We design Synephony to have joint self-attention of text-video and audio cross-attention with RoPE on top of DiT archi- tecture.
For training, we introduce motion-focused loss that places greater emphasis on regions exhibiting significant visual movement. This encourages the model to better detect motion that causally linked to the audio signal, even when such motion is highly localized. Furthermore, for sampling, we introduce a novel synchronization guidance strategy that enhances audio-driven motion without compromising visual fidelity. Motivated by the observation that applying traditional classifier-fre guidance to audio conditions makes it difficult to train scenes without audio, we instead propose a guidance method that skips the audio layer itself, rather than dropping the audio condition.
Also, we provide a comprehensive set of experiments that evaluate synchronization, visual quality, and semantic alignment across real-world scenarios. In particular, we propose novel synchroniza- tion metric, CycleSync, designed for high-frame-rate video generation, overcoming the limitations of existing metries that assume unrealistic one-to-one audio-video mappings or operate only low temporal resolution. Using CycleSync, we demonstrate that our approach successfully models varied audio-motion dynamics. Overall, our method, Synephony, outperforms existing approaches across all aspects, and we will release our code, models, and evaluation tools to support future research in this direction.
RELATED WORKS
2.1 TEXT&IMAGE-TO VIDEO GENERATION
Models. Based on the autoregressive models Yann et al. 202 Hong et al. 2022 Jin et al. 2024b) and diffusion models (Ho et al.) 2022 Brooks et al. (024). video generative models have been advanced dramatically. Notably, adapting DiT allows huge improvements in high-quality video gen- eration with scalability (Peebles & Xie 2023 Chen et al. 2023 Wang et al. 2023 HaCohen et al.) 2024]. Chen al. (2024), and Valevski et al. (024) proposed a hybrid approach that combines toregressive and diffusion models. Upon them, Jin et al. (2024a) proposes both a spatial and temporal feature compression enabling the generation of long videos with high fidelity at lower training cost. Notably, they only allow text or an image as conditions. On the other hand, our method takes audio as condition.
Guidance. Guidance mechanisms play a crucial role in improving sample quality across genera- tive models. Classifier-Free Guidance (Ho & Salimans 2022) interpolates between conditional and null-conditional predictions to enhance visual fidelity, but requires models to be explicitly trained with null conditions. Spatiotemporal Skip Guidance (Hyung et al., 2025) constructs a weak model by skipping visually sensitive layers, and interpolates its predictions with those of the full model to improve quality without additional training. However, in T2I and T2V architectures, visual and semantic representations are often deeply entangled, making such selective skipping difficult.
2

---

Published as a conference paper at ICLR 2026
2.2 AUDIO-TO- VIDEO GENERATION
Recent works on Audio-to-Videe (A2V) generation have explored how to synthesize temporally aligned videos conditioned on audio inputs. Lee et al. (203) modulates cross-attention weights based on audio amplitude to control video. Although this approach is simple, amplitude alone does not transfer the semantic and temporal structure of audio, resulting in weak fine-grained synchro- nization. On the other hand, Jeong et al. 2023 ; Yariv et al. (202)) project audio embeddings into a text embedding space and generate frames using pre-trained text-to-vides (T2V) models. This in- direct audio-to- andiotto-motion mapping is a bottleneck in delivering temporal expressiveness and hinders precise alignment between audio cues and motion transitions. AVSyncD (Zhang et al.) 2024) in- jects audio layers into a Stable Diffusion-based text-to-image (T2I) model, but it is limited to the T2I backbone's 's spatial resolution and suffers from relatively shallow temporal modeling capacity. Although Zhang et al. (2024) further introduces synchronization guidance, this requires additional training and often causes flickering, degrading visual smoothness. While talking head models have shown strong lip-syne performance for speech (Wang al.) 2025), they are limited to facial motion and human voice. We instead focus on non-speech sounds and general visual motion, which could complement lip-syne systems in real-world audio scenarios.
In contrast to these prior approaches, our method builds on the strengths of diffusion transformer- based T2V models to directly incorporate fine-grained temporal audio cues. By leveraging a high- capacity backbone capable of high-resolution, high-frame- frame-rate generation and introducing targeted synchronization guidance and loss-level modifications, our model achieves accurate audio-motion synchronization across diverse domains while preserving visual fidelity.
Synchronization metrics. Existing synchronization metrics, such as RelSync (Zhang et al. 2024)
and AlignSync (Zhang et al. 2024), require downsampling to 6 fps, which reduces temporal resolu- tion and undermines the evaluation of fine-grained motion. AV-Align (Yariv et al. 2023) assumes a one-to-one correspondence between motion and audio peaks, which fails to genera lize to woal-world scenarios involving preparatory or residual motion. For example, hammer moves before the impact sound and stops at the sound. To address these limitations, we propose new synchronization metric that supports high frame rates and generalizes to real-world audio-motion scenarios.
3 SYNCPHONY
3.1 OVERVIEW
Our goal is to generate high-quality videos that have motions aligned with audio inputs. We build upon pretrained autoregressive diffusion transformer (Jin al., [2024)), which sequentially syn- thesizes consecutive video chunks by denoising each chunk for given a previous chunk and a text prompt. As shown in Figure [ our model takes an initial frame, a text prompt, and audio wave- form as input. The initial frame is encoded into a latent 2g using a VAF, which serves as the starting point for generating video latents {=)_1. Text features are extracted from pretrained en- coders (Raffel et al. 2020 Radford et al. 2021), and audio features {a:) Hend are obtained from DenseAV (Hamilton al. 2024) encoder. Each transformer block includes joint self-attention layer, which attends over the concatenated sequence of text tokens and video latents. To incorpo- rate audio, we insert cross-attention layer before the joint self-aftention layer in the later blocks, allowing each video latent to attend to its aligned audio segment for fine-grained synchronization.
In the following subsections, propose a motion-aware loss that puts more weight the regions with large motions for training (3.2). introduce a sampling strategy designed to sample the videos toward better audio-conditional outputs (33), and describe additional architectural details (3.4).
3.2 MOTRON-AWARE LOss
Conventional video generation models typically use Mean Squared Error (MSE) loss, which mea- sures the pixel or latent-level diserepancy between predicted and ground-trum frames. While MSE is effective for general reconstruction, it treats all spatial and temporal regions equally, without dis- tinguishing between static and dynamic areas. As a result, even when the model produces inaccurate motion timing or insufficient movement, e.g., a delayed or insufficient gun firing motion, the error
3

---

Published as a conference paper at ICLR 2026
Auto-rogressively generate lntent
Previous Latents (or Latent)
Diffusion Transformer
concat
Noisy Latent
Latce Blvckk
Denoised Latent
Danty
10) (
Ag
dag barking"
Text Eneoder
Joint SelF-attention Layer
Audio
Audio Projector
Audio Cross attention Layer
Encoder
Feed Forward Layer
Figure 1: Overview of our video generation framework. Given an initial frame, a text prompt. and an audio waveform, the model autoregressively predicts each video latent through iterative denois- ing. The Diffusion Transformer is divided into two groups of transformer blocks: the early blocks (frozen, blue) and the later blocks (trainable, orange). Text features are injected into blocks via joint self-attention, whereas audio cross-attention layers are inserted only into the later (trainable) blocks. For brevity, latents are visualized as RGB frames, but they are spatiotemporal features ex- tracted by VAE.
remains low if the overall appearance is visually close to the ground truth. This may lead the model to interpret poorly synchronized predictions as successful outputs, weakening its ability to learn precise audio-visual alignment.
This limitation is particularly critical in real-world scenarios where audio cues correspond to dis- tinct, temporally localized motion, such drum hit bowling pin collision. In such cases, accu- rate timing and appropriate motion magnitude are essential for maintaining natural synchronization. Therefore, it is necessary provide stronger and more focused supervision areas involved in high-motion events.
In Figure 2 observe that latent differences between adjacent frames tend correlate with audio events, even when the corresponding motion is not clearly visible in the video frames, as in (c). Based this observation, cording to the intensity of ground- motion. This amplifies supervision at moments of significant movement, encouraging the model to better capture and align motion with audio cues.
propose a Motion-aware Loss that amplifies the learning signal
The proposed loss function is defined as:
L = Lbase + Lmotion:
(1)
Liase
GT 1le ct 1
(2)
Cmotion
Σ I(E Ee
(T(W)
(
olean
GT())
2
elean
GT 13
11.
(3)
mation
The proposed loss function is defined as:
= |1 C2
12
+
I(e!
GT(,)
GT(L) GT( 1)
(4)
cleam
clean
1=2
motion
where e!" and 6, (T()) denote the predicted and ground-truth noise scores at denoising timestep t GT(()) for the 1-th temporal position in the video latent sequence, respectively, *eksan denotes the ground- truth clean latent at position I, and @ denotes element- multiplication. The second term weights prediction enrors according to the magnitude of ground-truit motion between consecutive frames, with À as hyperparameter (we set 1).
4

---

Published as a conference paper at ICLR 2026
(a) striking bowling
(b) dog barking
(c) machine gun shooting
Figure 2: Visualization of video frames (top), latent difference maps (middle), and audio waveforms (bottom) over time. In (a) and (b), the latent differences correspond to key audio events such as pin collisions and barking. (c), although motion is not clearly visible in raw frames, latent differences still reveal temporal alignment with machine gun audio signals.
This design ensures that prediction errors during dynamic motion are penalized more heavily than those in static periods, encouraging the model to better capture the timing and intensity of important motions.
Importantly, do directly audio signal strength a supervision signal. This is because audio and motion do not always exhibit one-to-one temporal alignment: motion may precede or follow audio events, or span multiple frames. For instance, a lion may move before roaring, or bowling ball may roll before impact. By focusing on ground-truth motion magnitude rather than audio signal strength, our loss design allows the model learn natural synchronization patterns without rigidly assuming direct temporal alignment.
One might worry that the motion-aware loss could be negatively affected by camera motion or background movement that is unrelated to audio events. However, because our formulation weights loss based on ground-truth motion intensity itself, the model naturally learns to differentiate between motion that is causally linked to audio cues and motion that not.
Overall, Motion-awar aware Loss strengthens the model's attention motion reliinnreleevat regions, encourag- ing the model to learn diverse audio-motion relationships and generate natural, well-aligned motion sequences. Additional notes on motion-award loss are provided in Appendix
3.3 AUDIO SYNC GUIDANCE
In audio-conditioned video generation, audio-driven layers responsible for injecting timing into the visual dynamics. However, these cues from audio are not always strong or clear, so it's hard for the model determine whether to reflect them in the generated motion. For example, when drumstick hits a plastic surface, a subtle crinkling sound helps specify the exact target. Relying only on the coarse impact sound misplace the strike.
To address this, we propose Audio Syne Guidance (ASG) that reinforces the influence of audio sig- nals so the model better captures and reflects them in motion. As illustrated in Figure 3b),

---

Published as a conference paper at ICLR 2026
Video Latentt
Audio Sogments
{s (0) tiny 2
(9 yo..
Lineur
Lineur
Linear
6 (x))
RoPE
Audio RoPE
Query
Key
Value
Joint Self-attention Layer
Audio Cross-atiention Layer
Audio cruss attention
Off-sync model
(a) Audio Cross-attention with Audio RoPE
(b) Audio Sync Guidunce
Figure 3: (a) Audio cross-attention with Audio RoPE. Each video latent attends to a local audio segment using cross-attention. RoPE applied to both video queries and audio keys, using shared positional embedder to align modalities in relative position space. (b) Audio Sync Guidance. An off- syne model that skips the audio cross-ard coss-attention layers guides the full model better utilize audio cues during sampling.
two branches that share the same visual backbone: full model with audio cross-attention layers en- abled, and an off-synic model where only those layers are disabled. We found that the off--sync model produces outputs that are visually similar the full model's, s, yet desynchronized (Please see the supporting experiments in Appendix B.2). Thus, the difference between the two predictions isolates the synchronization component and could serve as guidance for synchronization. By adding this difference back into the full model's i output, ASG amplifies the influence of audio and encourages more synchronized motion generation.
Formally, given a latent 2, denoising timestep the guided prediction is
EΦ (z)) = cz(z;) + w (z))
Eg
off-syne (z))
(5)
where (x())) is the denoising output of the full model, ég skipped, and w is the guidance-strength hyperparameter controlling the degree of audio emphasis. For clarity, omit the integration with Classifier-Frs -eeeefie--Free Guidance; please an Appendix B.3 for the connection to CFG.
( (z;) is the output with audio layers
In summary, ASG highlights audio cues by disabling only the audio cross-attention layers in the off- sync model, improving audio-motion alignment while preserving visual fidelity without additional training.
3.4 ARCHITECTURAL DETAILS
Training layer selection. To leverage the pretrained video backbone effectively, we identify which transformer blocks to fine-tune through a layer-wise sensitivity analysis. We find that earlier layers primarily control spatial structure and semantic fidelity, whereas later layers govern temporal dy- namies and motion refinement. Based on this, we insert audio-driven cross-attention only into the later blocks and fine-tune them jointly. This strategy allows the model to focus synchronizing motion with audio signals while maintaining high visual fidelity and leveraging the strong general- ization capability of the pretrained I2V backbone. Details provided in Appendix D.1.2
Audio conditioning. To synchronize video motion precisely with audio cues, we apply Rotary Positional Embedding inject relative temporal information into the audio features during cross- attention (Audio RoPE), as illustrated in Figure Fa). We confirm that Audio RoPE leads to tighter temporal alignment between motion and sound events. Implementation details and an ablation study are provided in Appendix D.2
4 EVALUATING AUDIO-MOTION SYNCHRONIZATION
Although prior synchronization metrics Zhang et al. 2024: Yariv et al. 2023 are useful, they
require low fixed frame-rate introduce wrong assumption that the peak magnitudes of audio

---

Published as a conference paper at ICLR 2026
Pretrained V2A
A2V
Input Audio
Recovered Audio
Wavefoom
Generated Video
Onset Strength Devectad Peaks
Compare Temporal Peaks
Detected Peaks
Detected Peaks
Figure 4: CycleSyne metric pipeline. The generated video is fed into pretrained Video-to-Aur (V2A) model reconstruct audio. We compare temporal peaks between the reconstructed and orig- inal audio signals. High peak correspondence indicates that the generated video accurately preserves the timing structure of the original audio.
and video should match. makes them less reliable for high-frame-rate videos or real-world audio- motion scenarios.
To address these limitations, we propose CycleSync, a synchronization metric based on video- to-audio (V2A) reconstruction process. Instead of directly comparing motion and audio peaks, Cy- cleSync evaluates whether the motion in video provides enough signal to reconstruct the temporal structure of the original audio. As illustrated in Figure 4 we feed the generated video into state-of- the-art V2A model (Viertola al. 2025). and compare the resulting audio to the original input audio by aligning their temporal peaks. By aligning audio peaks between the original and recovered audio, we can assess whether the generated video contains sufficient timing and motion cues to reproduce the original audio structure.
Formally, given an original audio signal a and a generated video ú, we reconstruct the audio using pretrained video-to-aue -ocdiooo model fvza:
à foz
(6)
Let A = P(a) and À = P(ā) be the sets of onset peaks extracted from a and á, respectively. We compute the number matched peaks under temporal tolerance 5 using one-to-one matching (see Appendix Alg. D:
Σ
i
la àl <6]
aE
The CycleSyne score is then computed as an IoU between the two peak sets:
1
CycleSyne
(8)
|
where 8 is a temporal tolerance and 1 is the indicator function.
A higher CycleSync score indicates that the generated video preserves the timing structure of the original audio
5 EXPERIMENT
5.1 EXPERIMENTAL SETUP
Dataset. We evaluate our model using AVSync15 Zhang et al. 2024 and TheGreat- estHits Owens al. 2016), whose samples have synchronized audio and video.
AVSyne15 is curated subset of the VGGSound dataset consisting of 1,500 videos from 15 action-related
classes.
ThhGrreatstHiis is dataset where person strikes various objects with drumsticks, producing distinet impact sounds closely tied to visual motion.
7

---

Published as a conference paper at ICLR 2026
Table
Quantitative results on the AVSyne15 dataset.
User Study
Input
Model
FID
FVD 4
IA
IT
CycleSyne
Syne
T+A
TompoTokens Yariv ct al.
2023
8.9
4187.2
27.24
27.88
13.10
Pyramid Flow Jin et al. 2024a
8.9
$50.7
29.34
14.25
1+T
Pyramid Flow (Ime-tureed)
8.5
294.6
30.02
12.34
CoDi O
14.5
1522.6
28.15
23,42
TP6S Q
11.9
1227.8
38.36
30.73
1+T+
AVSyncD Zhang al. 2024
9.2
491.5
35.23
30.18
16.38
30
18
78
Ours
8,5
293.1
37.02
30.23
16,48=1a
270
282
222
Groundtruch
37.06
30.18
22.15
Table 2: Quantitative results on the TheGreatestHits dataset.
Input
Model
FID
FVD
IA
IT
CycleSync
Pyramid Flow (Jin al 2024a
6.5
350.5
18.42
7.41+0a3
1+T
Pyramid Flow (fie-tuned)
6.9
195.6
20.86
9.23gme
AVSyneD Zhang 2024
6.8
327.8
12.35
21.77
1.891
14TA
Ours
6.7
166.2
13.83
19.64
16.1811.8
Groundtruth
14.68
19.47
15.99
Baselines. We compare our method against the following baseline models: We employ the Pyra- mid Flow Video model (I+T2V) (Jin et al. (2024)), which conditions on text and image inputs, TokeoTokens (T+A2V), which conditions on audio and text inputs, and AVSyneD (I+T+A2V), which conditions audio and image inputs. For closer comparison between I2V and A2V, we also employ fine-tuned version of our model without audio layers, denoted as Pyramid Flow (fine- tuned).
Evaluation metrics. To assess visual quality, we report FID (Heusel et al., 2017) (Fréchet In- ception Distance) and FVD (Unterthiner al. 2019) (Fréchet Video Distance). FID measures the fidelity of individual frames, while FVD evaluater the spatiotemporal coherence of the entire video. To assess semantic alignment with conditioning modalities, we use Image-Text Similarity (TT) and Image-Audio Similarity (IA). IT evaluates how well the generated frames correspond to the in- put text prompt using CLIP (Radford al. 202T), while IA measures semantic alignment between audio signals and visual content using ImageBind (Girdhar et al. 2023). To assess audio-motion synchronization, we report CycleSync, which evaluates whether the generated videos contain suf- ficient motion cues synchronized with audio signals. We also conduct user study 150 videos from the AVSync15 dataset. Participants compare video pairs across three criteria; synchronization (Syne), image quality (IQ), and frame consistency (FC). Implementation details of the user study are provided in Appendix
Implementation details. We use the pretrained Pyramid Flow Video model (Ji et al. 2024a) as the backbone. Generated videos are up to 5 seconds long at 24 fps and 380 x 640 resolution. Audio sampled at 16kHz. During training, we randomly sample training clips from different tem- poral segments of each video to improve generalization various audio-motion alignments. During evaluation, we extract three 2-second clips distinct time points per video. The AVSync dataset provides 450 clips, and TheGreatestHits provides 732 clips for evaluation. We use CLIP (Radford) ctt al.) (02T) and DenseAV (Hamilton et al.) 2024) audio backbone as our text encoder and audio encoder, respectively. We train our model on 4 NVIDIA RTX 3090 GPUs (24GB).
5.2
MAIN RESULTS
5.2.1
MODEL. COMPARISON
Quantitative results. Tables I and show results AVSync15 and TheGreatestHits. Across both datasets, our model consistently outperforms baselines synchronization accuracy while maintain- ing competitive visual and semantic quality. Compared to AVSyneD, model achieves higher
8

---

Published as a conference paper at ICLR 2026
(a) frug croaking
(b) buby babbling crying 100000000000000
(c) machine shooting
Pyramid Flow (fine-tuned)
ASyncD
Figure 5: Qualitative comparison of generated videos. Our method produces more explicit and temporally consistent motion compared both baselines.
Table 3: Ablation results on AVSync15.
Model Variant
FID
FVD
CycleSyne
w/o Motion-awasd Loss
8.4
305.9
18.18
Full model w/o ASG
8.5
299.1
15.31
Full model w/ ASG (w = 1)
8.5
294.2
15.94
Full model w/ ASG (w 4)
8.7
298.3
16,26211
Full model w/ ASG (w = 2)
8.5
293.1
16,48
CycleSync scores and lower FID/FVD, indicating improved temporal coherence. User study further confirms these gains, with clear preference for model in synchronization, image quality, and frame consistency.
On TheGreatestHits, model even surpasses the ground-truth CycleSync score. We attribute this to the generated videos exhibiting strong and clear motion aligned with audio events, whereas ground-truth videos often contain off-event movements sounds, such hovering or background noise. These results suggest that our model demonstrates greater sensitivity to audio cues under syn- chronization metrics. Additional results using existing metries (AV-Align, RelSyne, AlignSync) are reported in Appendix C.4
Qualitative results. Figure 5 presents qualitative comparisons among ours, Pyramid Flow (fine- tuned), and AVSyncD. Our method produces clearer motion dynamics and stable appearances, whereas AVSyncD often suffers from saturation artifacts and weakened motion. We recommend watching the supplemental videos to see additional qualitative results (Appendix F.I.
5.2.2 METRIC COMPARISON
Controlled metric comparison. We analyze synchronization robustness under controlled tempo- ral shifts. Details are provided in Appendix C.2 As shown in Figure CycleSync is markedly more sensitive to temporal misalignment than other metrics, clearly differentiating synchronized from desynchronized cases.
Human alignment validation for CycleSync. We conduct a user study to assess how well Cy- cleSync aligns with human perception. Details provided in Appendix C.3 As shown in Table and Table CycleSync achieves the highest positive correlation with human preference, while prior metrics show weak or negative trends.
5.3 ABLATION STUDY
Effeet of Motion-aware Loss. When trained without Motion-aware Loss, the model tends to produce weaker and less clearly timed motions. As shown in Figure 6 it often fails to initiate or terminate motion in sync with the corresponding audio events. Incorporating Motion-awan Loss improves both the magnitude and temporal precision motion, particularly at the onset and off- set of dynamic actions. This is because Motion-aware Loss selectively amplifies learning signals at

---

Published as a conference paper at ICLR 2026
(a) frog croaking
(b) striking bowling
w/o Motion-aware Loss
Motion-aware Loss
Figure 6: Ablation of Motion-aware Loss. (a) Without Motion-aware Loss, the model fails to ter- minate the motion correctly the end of the audio. (b) It also fails trigger motion the correct audio onset. In contrast, with Motion-aware Loss, the model generates motion that more accurately aligns with the beginning and end of the audio event.
points of high motion intensity, guiding the model to learn more precisely on the timing structure of audio-driven actions.
Effect of Audio Syne Guidance. As shown in Tables 3. applying Audio Sync Guidance (ASG) with scale w = 2 improves synchronization metrics while preserving visual fidelity. Increasing the scale to 4 yields marginal gains in synchronization, but introduces over-exageerated motion (e.g., frog inflation or recoil motion), which slightly degrades visual realism reflected in higher FVD, while FID remains stable.
CONCLUSION
We introduced Syncphony, high-quality audio-synchronized video generation framework. By con- ditioning on text, image, and audio inputs, our model captures both the semantic context and the fine-grained temporal dynamics of motion. To improve audio-motion alignment, we incorporated two key techniques: Motion-aware Loss encourages accurate timing by emphasizing high-motion regions, and Audío Syne Guidance enhances sensitivity to audio signals during inference while maintaining visual quality, To better evaluate synchronization accuracy, we proposed CycleSyne, iido-to--adiio-e
metric that measures whether the generated video retains sufficient motion cues to reconstruct the original audio. This enables a more reliable assessment than the existing metrics in real-world scenarios.
Limitation. Motion-aware Loss reweights training by ground-truth motion magnitude to empha- size dynamic regions. current scope of this design is that the weighting not explicitly selective for audio-relevant motion at the supervision level. This still allows the model to learn audio-motion associations, but a more selective motion proxy (e.g., foreground/action-awn further improve robustness in highly dynamic scenes where audio-drives motion should be empha- sized. We also note that the limitations of CycleSyne as synchronization metric are discussed in Appendix C.5
weighting) could
ETHICS STATEMENT
As generative model, our method could be used to facilitate deceptive interactions that would cause harm, such as fraud. It could be used to impersonate public figures and influence political processes, or tool to promote hate speech or abuse. To address this, we will include explicit license terms and usage guidelines to promote ethical and lawful use, referencing best practices such as the Adobe Generative AI User Guidelines. If the model is released, implement safeguards such as prompt image filtering to restrict high-risk applications, including impersonation or politically manipulative content.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Key components of our implementation are provided in the supplementary materials, and detailed descriptions of our method, training, inference, and evaluation are included in the appendix. We will release our code, trained models, and evaluation tools ensure reproducibility.
ACKNOWLEDGMENTS
This work was supported by LTT grants RRS-2024-00439762, Developing Techniques for Analyz- ing and Assessing Vulnerabilities, and Tools for Confidentiality Evaluation in Generative AI Mod- els] and (RS-2020-I120136 Artificial Intelligence Graduate School Program (Yonsei University)] funded by the Korean government (MSIT).
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominík Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint ar arXiv:2311.15127, 2023.
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitz- mann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neu- ral Information Processing Systems, 37:24081-24125, 2024.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high-quality video generation, 2023.
Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mit- sufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28901-28911, 2025.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725. 2023.
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.000 000103, 2024.
Mark Hamilton, Andrew Zisserman, John R Hershey, and William T Freeman. Separating the" chirp" from the" chat": Self-supervised visual grounding of sound and language. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13117-13127, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems,001
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS Workshop on Deep Generative Models and Downstream Application,,, 2022.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi. and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 8646, 2022.
11

---

Published as a conference paper at ICLR 2026
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: sarge-scale pre- training for text-to-video generation via transformers. arXiv preprint 0rXiv:2205.15868, 2022.
Junha Hyung, Kinam Kim, Susung Hong, Min-Jung Kim, and Jaegul Choo. Spatiotemporal skip guidance for enhanced video diffusion sampling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1100--1001,, 2025.
Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Seo, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7822-7832, 2023.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.0595. 2024a.
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual-motional tokenization. arXiv preprint arXivv2402.03166, 2024b.
Seungwoo Lee, Chaerin Kong, Donghyeon Jeon, and Nojun Kwak. Aadiff: Audio-aligned video
synthesis with text-to-image diffusion. In CVPR Workshop on Content Generation, 2023.
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and
William T. Freeman. Visually indicated sounds. In CVPR, 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo. and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. In arXiv, 2019.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-tine time game engines. arXiv preprint 240i 14837, 2024.
Ipрo Viertola, Vladimir Iashin, and Esa Rahtu. Temporally aligned audio for video with autore- gression. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haim- ing Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.2033 2025.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope to-t-to-vido technical report, 2023.
Mengchao Wang, Qiang Wang, Fan Jiang, Yaqi Fan, Yunpeng Zhang, Yonggang Qi, Kun Zhao, and Mu Xu. Fantasytalking: Realistic talking portrait generation via coherent motion synthesis. arXiv preprint arXiv:2504.00 044422224...4,, 2025.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv: 104.104411555 10157, 2021.
12

---

Published as a conference paper at ICLR 2026
Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. 2023.
Lin Zhang, Shentong Mo, Yijing Zhang, and Pedro Morgado. succo-syychronizeed visual animation. In European Conference Computer Vision, pp. 1-18. Springer, 2024.
13

---

Published as a conference paper at ICLR 2026
Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformers
Supplementary Material
A ADDITIONAL NOTES ON MOTION-AWARE LOSS
Motion-aware Loss is designed based on ground-trute truth motion magnitude, not audio amplitude. This reflects the fact that motion peak and audio peak do not always exhibit one-to-one temporal align- ment. Instead, audio events are often accompanied by diverse motion patterns that vary by context.
For example, some events, such as a gunshot or dog's bark, occur with motion and sound nearly perfectly aligned. However, many others do not. A lion may start moving its mouth and body before emitting a roar. A person winds up before throwing a ball. A hammering motion may start before the sound and end just as the impact occurs. A trombone player moves the instrument before the sound begins. bowling ball rolls with a low rumble before producing a sharp impact sound when it hits the pins.
Therefore, synchronizing motion to audio does not mean matching peak amplitudes. Rather, it in- volves capturing the causal and contextual patterns of motion that correspond to different types of audio events. Our loss focuses on motion regions, encouraging the model to learn this alignment without relying rigid audio-based timing. This design encourages the model to learn various audio-motion relationships, leading to natural audio-visual aligned video generation.
B AUDIO SYNC GUIDANCE
B.1 CLASSIFIER FREE GUIDANCE FOR AUDIO GUIDANCE
Conventional classifier-free guidance (CFG) is trained by randomly dropping conditioning inputs, strategy that has proven highly effective in models such as text-to-image and class-conditional image generation. However, we observe that directly applying this random-drop strategy to audio conditioning leads to degraded performance.
Unlike text, audio carries a meaningful semantic interpretation even when its value is zero. other words, silence is itself an informative condition. When audio inputs are randomly dropped during training, the model becomes unable to properly model silence and fails to understand its distinct role compared the absence of conditioning. Our experiments show that this mismatch significantly harms synchronization quality.
To address this issue, instead of dropping audio conditions, we propose an Audio Sync Guidance mechanism that selectively skips audio-conditionin layers during inference. This approach pre- serves the semantic meaning of silence while preventing the model from being exposed ambigu- ous training signals. In the following section, we describe this strategy in more detail.
B.2 DIFFERENCES BETWEEN FULL AND OFF-SYNC MODEL IN AUDIO SYNC GUIDANCE
To better understand how Audio Sync Guidance contributes to synchronization, we evaluate whether an off-sync model, formed by skipping the audio layers, can still retain appearance and overall motion. As shown in the last row of Figure "Off-sync model" of Table the model remains out of synchronization but still preserves appearance (FID) and motion quality (FVD). Since the visual quality remains similar between the full and off-sync models, their difference primarily captures audio-related motion cues. By adding this difference back into the full model's output, Audio Sync Guidance amplifies the influence of audio and encourages more synchronized motion generation.

---

Published as a conference paper at ICLR 2026
Table Analysis of Audio Syne Guidance. The full model includes audio layers, whereas the off- sync model skips them.
Model Variant
FID |
FVD J
CycleSync 1
Off-sync model
8.5
294.6
12.34
Full model
8.5
299.1
15.31
Full model w/ ASG
8.5
293.1
16.48
B.3 INTEGRATION CFG AND AUDIO SYNC GUIDANCE
Classifier-Fres Guidance (Ho & Salimans 2022) conditional predictions to enhance visual fidelity:
extrapolates between conditional (full) and null-
(24)) = c (2t, ca) + (co e) - co(2e. cœ))
(9)
At inference time, the Audio Syne Guidance and CFG are combined additively:
(0(2)) = c0(2), c) + wa (co(Ze; c) oft-syne (2t, + we (<o(2e, c) c0(2,, co))
(10)
We take the conditional prediction the base and add CFG and ASG residuals. In our implementa- tion, we use Wa = and W/
B.4 AUDIO SYNC GUIDANCE COMPARED PRIOR SKIP-BASED METHOD
Audio Sync Guidance (ASG) inspired by Hyung et (025) but differs in both purpose and design.
Hyung et al. (2025) improves visual fidelity by constructing a weak model that skips visually sensi- tive layers and using it to guide the full model. In T2I/T2V settings. however, semantic and visual features are heavily entangled, making such selective skipping difficult, model-dependent, and prone to unintended degradations.
ASG instead targets synchronization. We skip only the audio-injection layers, creating an off-sync model that preserves appearance but ignores audio cues. The difference between this off-syne and the full model isolates synchronization the guidance signal (see Appendix [..2).
This design is suited to A2V architectures, where audio and visual pathways are explicitly separated. Skipping only the audio pathway perturbs synchronization without affecting visual fidelity, enabling precise and stable guidance for improved audio-motion alignment.
C EVALUATION METRICS FOR SYNCHRONIZATION
C.I IMPLEMENTATION DETAILS OF CYCLESYNC
We use V-AURA Viertola et al. 2025) as the pretrained video-to-aude model fvza. selected for its demonstrated effectiveness generating general-class, temporally aligned audio from videos. For peak extraction, we use bibrosaa.. onset. onset detect, and 8 is fixed at 5 milliseconds,
Alg. summarizes the full CycleSyne evaluation procedure, including peak extraction and one-to- one matching.
C.2 CONTROLLED METRIC COMPARISON
To evaluate the effectiveness CynleSync, we compare with three existing synchronization met- rics: AV-Align (Yariv et al. 2023), AlignSync, and RelSyne (Zhang et al. 2024). We apply six levels of synchronization shift to video clips from the AVSync15 (Zhang et al. 2024 and TheGreat- estHits (wwess et al. 2016) datasets.
2

---

Published as a conference paper at ICLR 2026
Algorithm CycleSyne: audio reconstruction onset-peak IoU with one-to-one matching
Require: GT audio waveform a, generated video é, duration 7 (sec), onset parameter A, tolerance
8, sampling rate 8y
Ensure: CycleSync score [0,
1: ā - fvza(t)
pretrained v2a model
2: function DETECTPEAKSL, 7, 78, A)
3:
I <- RESAMPLELOAD sr)
4: J =00 -s]
5: if' max [x| < 0.01 then
6:
return 9 end if
7:
8:
< ONSETSTRENGTH(22 sr)
9:
F < ONSETDETECT(C: ST; A)
D 1 Eebrosa.onseer onset detect
P c FRAMESTOTIME(RR.M
11: return P
12: end function
13: & DETECTPEAKS(O.7) A)
À DETECTPEAKS(O, A)
15: I <-0
used [Al]] < false
17: for E A. do
D greedy one-to-one matching within
18:
for + 1 to |A| do
19:
if -u-ddd [ and [A[]] al 8 then
20:
I + 1+1
used] [ + true
22:
break
end if
24: end for
end for
I
26:
A + [Å
27: return
implementation detail. We extract three 2-second clips per video with linear intervals. To ensure valid comparison under delay shifts, clips are sampled starting 0.5 seconds into the video, allowing up 0.5 seconds of temporal shift. Videos shorter than 2.5 seconds are excluded. It results in 438 clips from 150 videos in AVSync15, 15, and 732 clips from 244 videos in TheGreatestHits.
AlignSync and RelSync evaluated on videos downsampled fps unless otherwise noted as 24 fps. CycleSync (Ours) is evaluated at 24 fps videos.
fps. AV-Align is measured
Synchronization configurations. A sample type with "Perfect Sync" represents that Ground- truth audio pairs with its original video. The other sample types with "Delay 0.1s-0.5s" represent that the video is temporally shifted by the indicated delay relative to its audio.
C.2.1 RESULTS AND ANALYSIS
Figure shows how each metric responds increasing audio-video misalignment. We observe that existing metrics often struggle to clearly separate perfectly synchronized samples from delayed ones, whereas CycleSync scores drop sharply with the misalignments. For absolute metric values, please refer to Table 5 and T
AV-Align. The performance AV-Align varies significantly depending on the frame rate. At 24 fps, we would expect the highest score for perfectly synchronized samples, but in both AVSyne15 and TheGreatestHits, delayed samples receive higher scores than the ground alignment. At fps, AV-Align becomes more stable, but the separation between perfect and delayed cases remains

---

Published as a conference paper at ICLR 2026
limited. This suggests that the metric may not reliably reflect fine-graines temporal misalignment higher frame rates.
Moreover, shown in Appendix obtain higher AV-Align scores than models explicitly guided by audio. This is because AV-Align assumes strict one-to-one correspondence between peaks in the audio and motion signals, sumption that often does not hold natural scenarios, where motion may precede or follow audio cues.
there are where models without audio conditioning
AlignSyne and RelSyne. AlignSync and RelSync generally show decreasing scores as the degree of delay increases, indicating sensitivity to misalignment. However, they do not show clear dif- ferences between perfectly synchronized samples and delayed ones, especially on TheGreatestHits dataset. In addition, both metrics are designed for evaluation at 6 fps, which makes it difficult to assess the performance of models operating higher frame rates, such
fps.
We also observe cases where models without audio conditioning receive higher scores than those guided by audio (see Appendix C.4). One possible explanation that these metries are more effec- tive when evaluating sequences that are simple temporal shifts of the same ground-truth content, as assumed during training. contrast, when the evaluated sequence differs from the original ground- truth content, the metrics may longer provide reliable scores.
CycleSync. CycleSync clearly distinguishes perfectly synchronized samples from those with tem- poral misalignment. Once the delay exceeds certain threshold, the differences between misaligned cases become less pronounced. In other words, the score is not strictly monotonically decreasing.
This is due to CycleSync's use of a fixed 0.05s tolerance window to determine alignment between onset peaks in the original and reconstructed audio. While this allows for robust separation between synchronized and unsynchronized cases, does not explicitly quantify how far misaligned peaks fall beyond the threshold.
This behavior arises because the tolerance hyperparameter is set to 0.05s, and CycleSync determines alignment between onset peaks in the original and reconstructed audio within this fixed tolerance window. While this design provides robust separation between synchronized and unsynchronized cases, it does not explicitly quantify how far misaligned peaks fall beyond the threshold. It could be addressed by incorporating multi-scale tolerance or continuous scoring mechanisms to capture varying degrees of misalignment. We leave this future work.
AVSync15
TheGreatestiits
1.4
Al-Align 124fps]
Ai-Align (241ps)
AV-Align 18fps!
Algglign (6fps)
Sync
AlignSync
RelSync
CydeSyyc
CycleSync
Relative Scores
1.0
0.4
Delay (s)
Delay (s)
Figure 7: Comparison of relative synchronization scores under increasing audio-video delays on AVSyne15 and TheGreatestHits datasets. The vertical axis denotes each metric's score normalized by its value under perfect synchronization (0.0s).
C.3 HUMAN ALIGNMENT VALIDATION FOR CYCLESYNC
Human evaluation is essential for establishing a reliable synchronization metric. To assess how well CycleSync aligns with human perception, we conduct study with 9 participants, who rate the

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of synchronization metric scores on the AVSync15 dataset. Parentheses show percentage change from perfect synchronization (positive = increase, negative decrease).
Sample Type
AV-Align (24fps) 1
AV-Align f
AlignSync
RelSync
CycleSyne f
Perfect Syne
24.22 (0.0%)
20.30 (-0.0%)
25.04 (-0.0%)
50.00 (0.0%)
20.97 (0.0%)
Delay 0.1s
24.29 (+0.3%)
19.76 (-2.7%)
24.89 (-0.6%)
49.70 (-0.6%)
17.48 (-16.6%)
Delay 0.2s
24.63 (+1.7%)
19.06 (-6.1%)
24.39 (-2.6%)
48.70 (-2.6%)
13.96 (-33,4%)
Delay 0.3s
24.49 (+1.1%)
19.03 (-6.3%)
24.05 (-4.0%)
48.04 (-3.9%)
12.03 (-42.6%)
Delay 0.4s
24.25 (+0.1%)
19.71 (-2.9%)
23.74 (-5.2%)
47.41 (-5.2%)
12.43 (-40.7%)
Delay 0.5s
25.24 (+4.2%)
19.20 (-5.4%)
23.70 (-5.4%)
47.33 (-5.3%)
11.11 (-47.0%)
Table 6: Comparison of synchronization metric scores on TheGreatestHits dataset. Parentheses show percentage change from perfect synchronization (positive increase, negative decrease).
Sample Type
AV-Align (24fps) 1
AV-Align
AlignSyne f
RelSync 1
CycleSyne 1
Perfect Sync
14.84 (0.0%)
27.27 (0.0%)
25,07 (0.0%)
50.00 (0.0%)
16.52 (0.0%)
Delay 0.1s
17.71 (+19.3%)
26.05 (-4.5%)
24.86 (-0.8%)
49.59 (-0.8%)
6.97 (-57.9%)
Delay 0.2s
14.67 (-1.2%)
25.48 (-6.6%)
24.76 (-1.2%)
49.40 (-1.2%)
7.30 (-55.8%)
Delay 0.3s
15.25 (+2.8%)
24.20 (-11.3%)
24.61 (-1.8%)
49.11 (-1.8%)
8.06 (-51.2%)
Delay 0.4s
14.91 (+0.5%)
24.45 (-10.3%)
24.63 (-1.8%)
49.15 (-1.7%)
9.29 (-43.77%)
Delay 0.5s
15.21 (42.5%)
25.15 (-7.8%)
24.59 (-1.9%)
49.06 (-1.9%)
10.30 (-37.65%)
sync quality of 20 videos, sampled from Pyramid Flow and Syncphony, on 1-5 scale. We then com- pute Pearson correlations between the human ratings and the metric scores. We compare CycleSync against existing synchronization metrics: AV-Align (Yariv et al. 2023), AlignSync, RelSync (Zhang ct al. 2024), and DeSync (Cheng et al. 2025).
Correlation with human ratings. As shown in Table CycleSync shows the strongest positive correlation with human ratings (r = 0.486), whereas prior metries exhibit weak or even negative correlations. DeSync also shows positive trend 0.206), but remains notably weaker than CycleSync.
Model ranking agreement. We further compared model-level rankings derived from each metric against human ratings (Table 8). CycleSyne accurately captures the human preferred ordering be- tween the two models. DeSync also reflects the correct direction of preference, though with lower discriminative strength.
human-preferred
These results provide strong empirical evidence that CycleSyne is both quantitatively sensitive to temporal misalignment and best aligned with human perception, making more reliable synchro- nization metric than existing metrics.
C.4 RESULTS OF BASELINES AND SYNCPHONY WITH EXISTING METRICS
We additionally report the performance of baseline models and ours using existing synchronization metries on the AVSync15 and TheGreatestHits datasets Table Table 10 respectively.
On AVSync15, the fine-tuned Pyramid Flow model, which generates audio-independent but plau- sible motion, achieves the highest AV-Align score. similar pattern is observed TheGreat- estHits, where the same model also obtains higher AlignSync and RelSyne scores than other audio- conditioned models.
These results reveal limitation of existing metrics, which tend to favor models that produce highly dynamic motion with plausible timing, even if that motion is not aligned with the audio signal.
In contrast, CycleSyne consistently assigns the lowest scores to the same model across both datasets.
This is because CycleSync penalizes mismatches in temporal structure between the original audio and the reconstructed audio from the generated video. Rather than comparing audio and motion
5

---

Published as a conference paper at ICLR 2026
Table Correlation with human ratings. CycleSync achieves the highest positive correlation with human judgments, while other metrics show weak or negative trends.
Metric
Correlation
95% CI Lower
95% CI Upper
CycleSyne
0.486
0.053
0,919
AV-Align
0.043
-0,451
0.538
RelSyne
-0.623
-1.011
-0.236
AlignSync
-0.625
-1.011
-0.238
DeSync
0.206
-0.279
0.690
Table 8: Model ranking agreement with human ratings. CycleSync correctly reflects human pref- erence, ranking Syncphony above Pyramid Flow.
Model
Human Score 1 CycleSync
AV-Align
RelSync
AlignSyne 1
DeSyne +
Pyramid Flow (12V)
2.68
8.15
24.96
55.36
27.75
0.88
Synephony (Ours)
4.30
22.04
21.88
50.44
25.19
0.86
peaks directly, CycleSync compares the temporal structure of the original and reconstructed audio signals, enabling more precise assessment of synchronization quality.
C.5 LIMITATION OF CYCLESYNC.
As reconstruction-based metric, CycleSyne relies on the quality and behavior of the underlying video-to-audio (V2A) model. The reconstructed audio may sometimes reflect dataset-level biases rather than the visual content of the input video itself.
For example, in frog videas, although only single frog may be visible, many clips in the dataset include ambient sounds from nearby frogs. As a result, the reconstructed audio sometimes contains multiple frog sounds, regardless of the actual motion in the video. Similarly, bowling videos in the dataset often include background music, which can occasionally appear in the reconstructed audio even if it is not visually implied. Such cases may affect CycleSync scores in specific contexts. This issue could potentially be addressed by improving the V2A model or applying post post-processing. which we leave for future work.
D ARCHITECTURAL DETAILS
D.1 TRAINING LAYER SELECTION
D.1.1 VIDEO GENERATION BACKBONE
We adopt Pyramid Flow (Jin et al. 2024a) as the video generation backbone due to its efficiency and scalability in generating long. high-resolution videos. Pyramid Flow is an autoregressive diffu- sion transformer trained with a flow-matching objective, which sequentially synthesizes consecutive video chunks by denoising each chunk for given a previous chunk and a text prompt.
To capture temporal and spatial consistency, it employs 3D Rotary Positional Encoding (RoPE) (Su ct al., 2024) within its self-attention layers, enabling the model to encode relative positions across time, height, and width. In addition, the model dynamically adjusts resolution throughout the de- noising process, using low-resolution frames at early (noisier) timesteps and high-resolution frames at later (cleaner) stages, thereby reducing computational cost while maintaining visual detail.
This design enables resource-efficien training and generation, supporting high-resolution and long- duration video synthesis even under constrained computational resources.

---

Published as a conference paper at ICLR 2026
Table Quantitative results on the AVSync15 dataset.
Input
Model
AV-Align
AlignSync t
RelSyne
CycleSync
T+A
TokpoTokens (Yariv al.
2023)
15.51
22.38
46.91
13.10
Pyramid Flow (Jin al Pyramid Flow (hne-tuned)
2024a
18.85
23.65
47.56
14.25
I4T
20.69
23.97
47.76
12.34
AVSyncD Zhhag et al.
2024
19.31
24.61
48.99
16.38
I+T+A
Ours w/o ASG (Wa 0)
20.01
24.24
48.28
15.31
Ours w/ ASG (1%
19.89
24.45
48.74
16.48
Ours w/ ASG (1% 4)
20.00
24.58
49.04
16.26
Groundtruth
20.84
25.10
50.00
22.15
Table 10: Quantitative results TheGreatestHliss dataset.
Input
Model
AV-Align f
AlignSync
RelSyne f CycleSyne f
IT
Pyramid Flow (Jin et al. Pyramid Flow (fine-tuned)
2024a
25.24
25.12
50.46
7.41
26.76
26.67
53.35
9.23
AVSyncD (Zhag
2024
23.29
26.55
53.07
9.89
I+T+A
Ours w/o ASCr (W0g
27.11
26.08
52.21
11.70
Ours w/ ASG (Wa = 2)
26.92
26.10
52.27
16.18
Ours w/ ASG (Ma = 4)
26.81
27.04
54.14
17.71
Groundtruth
26.00
25.07
50.00
15.99
D.1.2 TRAINING LAYER SELECTION IN VIDEO BACKBONE
Pyramid Flow consists of 24 transformer blocks. To identify which layers to fine-tune, we individ- ually skip each of the 24 transformer blocks during inference and observe the effects on image-to- video (I2V) generation (see Figures and ITI.
Skipping early blocks (0-7) significantly degrades appearance, often causing artifacts in the back- ground and object structure. In contrast, skipping later blocks (8-23) mostly preserves the appear- ance of the input image (first frame) in the generated video, primarily affecting the motion. This suggests that early blocks are critical for preserving the input's appearance, whereas later blocks are responsible for refining motion. This separation aligns with the architecture: early blocks use separate attention weights for text and video, while later blocks share them. Based on this functional and structural separation, we fine-tune only the last blocks (8-23) with minimal impact on the pretrained model's s visual fidelity.
D.2
AUDIO ROPE
D.2.1
IMPLEMENTATION DETAILS
To encode the temporal structure of audio features explicitly, apply Rotary Positional Encod- ing (RoPE) to inject relative temporal information directly into the cross-attention mechanism, as illustrated in Figure a)).
We first obtain video latents {2 1 Listo from VAE encoder, where each (() represents com- Ti=0
pressed spatiotemporal feature at the I-th position in the video sequence. Simultancously. we extract Lunto audio features {a 2 from a DenseAV encoder, capturing the temporal and semantic structure of the audio input.
To align these modalities, we divide the audio sequence into local segments corresponding to each video latent. For each target video latent 2(0) we define the corresponding audio segment (()) as:
A (a))) lie [a(l A), A)]].
(11)

---

Published as a conference paper at ICLR 2026
Table 11: Ablation of Audio RoPE.
Model Variant
CycleSync
w/o Audio RoPE
14,41
w/ Audio RoPE
15.3
where is scaling factor mapping video indices to audio indices (accounting for the different
sequence lengths), and A determines the width of the temporal window(we set A=1).
Then, we apply Audio RoPE to the audio segments. The procedure is as follows:
Step Assign positional indices.
Each video latent z()) is assigned 3D coordinates h1, w) representing its temporal and spatial location within the video sequence.
For each audio segment A the constituent audio features are assigned linearly interpolated temporal indices within the range [t (A + 0.5), + (A + 0.5)], such that the center of the segment aligns exactly with I.
Step Project into query and key spaces.
② = Wo = (Wxa | A
(12)
where WQ and WK are learnable linear projection matrices.
Step 3. Apply RoPE rotations.
i hope ( (1, w)),,
K (1) (RoPE(WRa!)) (r(0,,,,) a AA (13)
where (9) denotes the interpolated temporal index assigned to each (1)
Step Compute cross-attention between video latent
2 and audio segment A
(1) Prope rope T
(ttention=="),AA_ Softmax
20)
(14)
Vd
where x0)
( (a) E A is the set of value projections of the audio features, and is the dimension of the projected space.
By explicitly injecting temporally aligned positional cues into both video and audio features, our model captures the sequential structure of audio signals more effectively, leading improved syn- chronization between generated video motion and corresponding audio events.
D.3 ABLATION STUDY
We conduct ablation experiments to examine the effect of Audio RoPE, The results Table TT indicate that using Audio RoPE leads to higher synchronization quality compared to the model without it. Without applying RoPE to audio features, the model frequently exhibits misalignments, with motions often preceding lagging behind the corresponding audio cues. In contrast, applying RoPE to the audio features results tighter temporal alignment between motion and sound events, enabling the model to better capture the sequential structure of the audio input. Additional ablation examples are included in the supplementary materials (Appendix G
E
USER STUDY
To assess the perceptual quality of our generated videos, we conducted a user study comparing our method with the state-of-the-art Audio-to-Vicee Video model AVSyncD (Zhang et al. 2024). We select
8

---

Published as a conference paper at ICLR 2026
AVSyncD the sole baseline in the study, as other baselines generate noticeably unsynchro- nized motion. The evaluation focused on three aspects: synchronization with audio, image quality, and frame consistency.
The study was conducted using all 150 test videos from the AVSync15 dataset. These were divided into five subsets of 30 videos each, with each subset assigned to participants (10 participants total). For every video, participants were shown two versions, one generated by our model and one by AVSyncD based on the same audio input and initial image. Participants asked answer the three questions for each video pair:
Synchronization: Which video is better synchronized with the audio in terms of motion timing?
Image Quality: Which video has better image quality in terms realism and clarity?
Frame Consistency: Which video is more visually consistent across frames, without flickering or sudden jumps?
As illustrated in Figure participants evaluated video pairs using a web interface showing both videos and three corresponding questions.
The results, summarized in Figure 8, show that our model was consistently preferred: 74% for syn- chronization, 90% for image quality, and 94% for frame consistency.
Ours vs. AVSyncD
Synchronization
74%
Image Quality
90%
Frame Consistency
94%
Figure 8: Human preference rates (%) for our method over ASyneD across three evaluation criteria.
These results demonstrate that our model is consistently favored by human evaluators across all three aspects. This further validates the effectiveness of our synchronization mechanisms and visual fidelity of our methods.
F ADDITIONAL VIDEO SAMPLES
Additional generated videos, ablations, and comparisons are provided on our project page:
https//jibinnbggtttuubioooyyyyyp project page
F.1
SAMPLES FROM SYNCPHONY
Please refer to our project page for generated video samples from Synephony.
G ABLATION SAMPLES
Please refer Section Ablations of the project page which includes ablation results for Motion- aware Loss, Audio Sync Guidance, and Audio RoPE.
H COMPARISON SAMPLES
Please refer to Section Comparison of the project page for video comparisons between model, AVSyncD Zhang et al. (2024). and Pyramid Flow (fine-tuned), a variant of our model without audio cross-attention layers.

---

Published as a conference paper at ICLR 2026
1 IMPLEMENTATION AND EXPERIMENTAL DETAILS
I.1 WHY IMAGE-TO-VIDEE BACKBONE?
We also applied our method Text-to- Vixeeoo-Vidoo (T2V) model, AnimateDiff (Guo al. 2023). and trained it on the AVSync15 dataset, which contains limited 1,350 training samples. We found the model generates motion aligned with audio, but shows overfitting, with limited diversity in appear- ance. This is because T2V models have to generate both appearance and motion without a reference image. With a small dataset, becomes difficult to produce diverse appearances, and even harder to learn various audio-driven motion patterns.
In contrast, Image-to- Mange-to-Video (I2V) models, such as Pyramid Flow (Jin et al., 2024a), are conditioned on an initial image and focus predicting motion rather than full appearance. This simplifies the learning process and reduces the risk of overfitting. For these reasons, we adopt the I2V model as video generation backbone.
1.2 TRAINING AND INFERENCE SETTINGS
We train our model using 4 NVIDIA RTX 3090 GPUs (24GB each) with total batch size of 32. Training takes hours reach 33,000 steps. For all experiments, we use 30 denoising steps. We follow Pyramid Flow in setting the classifier-free guidance (CFG) strength to 7.0 for the first latent and 4.0 for the rest. For Audio Sync Guidance, we use w = 2, where w disables the guidance.
Inference time for 5-second video (with pre-encoded audio and text features) is as follows:
Audio Guidance: 2 min 53 sec
w/o Audio Guidance: min sec
w/o Audio Layers: min 43 sec
At least 16 GB of GPU memory is required generate 5-second videos.
1.3 TRAINING AND EVALUATION DATASETS
We use two datasets for training and evaluation:
AVSync15 (Zhang ef al., 2024): 1,350 videos for training and 150 for testing. For evaluation, we linearly extract 3 clips per video, resulting in 450 evaluation clips.
TieGreatestHliss (Owens et al., 2016): 733 videos for training and 244 for testing, resulting in 732 evaluation clips.
During training, randomly sample clips from different temporal regions of each video improve generalization to various audio-man motionn alignments.
J APPLICABILITY OF SYNCPHONY TECHNIQUES TO OTHER MODALITIES
While Synephony focuses on audio-to-video generation, we believe the proposed techniques are applicable other modalities.
Motion-aware Loss, by amplifying learning signals in high-motion regions, encourages the model focus on dynamic cues that reflect physically grounded movements, This can benefit tasks like mudio--oo3DD animation, text-to-video, and text-to-3D, where generating realistic motion is essential.
In contrast, Audio Syne Guidance are designed improve synchronization between audio and mo- tion. This technique is applicable to tasks such as audio-to-3D animation, provided that the model adopts an attention-baa based architecture with functionally well-separated layers, which enables clean injection of audio signals into the network.
10

---

Published as a conference paper at ICLR 2026
10. frog croaking
0:02
Video 1
Video
Q1. Which video has better image quality terms realism and clarity?
OVideo
CVideo
Q2. Which video more visually consistent across frames, without flickering or sudden jumps?
OVideo
CMidea 2
Q3. Which video better synchronized with the audio in terms of motion timing?
CVideo 1
OVideo 2
Figure 9: Screenshot of the user study interface of each video pair with questions.

---

Published as a conference paper at ICLR 2026
None
1-block
2-block
3-block
4-block
5-hlock
6-hlock
7-block
8-block
9-block
10-block
11-block
12-block
13-block
14-bkock
block
16-block
17-block
18-block
19-block
20-block
21-block
22-block
blokk
block
Figure 10: Frame results of skipping each transformer block individually.
12

---

Published as a conference paper at ICLR 2026
None
1-block
2-block
3-block
4-block
5-hlock
6-block
7-block
8-block
9-block
10-block
11-block
12-block
13-block
blockock
15-block
16-block
17-block
18-block
19-block
20-block
21-block
22-block
block
block
Figure 11: Frame results of skipping each transformer block individually.
13


Reviews:
[{"review_id": "18Ef5UKChK", "reviewer": "Reviewer_nWr3", "summary": "This paper introduces Syncphony, an audio-to-video (A2V) model capable of generating 380×640 resolution, 24fps videos synchronized with diverse audio inputs. Built upon a pre-trained video backbone, it emphasizes on audio-visual synchronization through two main contributions:\n- Motion-aware Loss, a loss weighting mechanism emphasizing on the high-motion regions of the video.\n- Audio Sync Guidance, a classifier-free guidance design focusing solely on audio conditioning to emphasize better synchronization.\n\nMoreover the authors introduce an auxiliary contribution for the model evaluation:\n- CycleSync, an audio similarity metrics between the detected peaks of the ground truth and the A2V -> (pretrained) V2A generated audio.\n\nThe A2V model is finetuned from a pretrained I2V model, by adding cross attention layers in the latter transformer blocks of the pretrained model. The backbone is a pretrained PyramidFlow Video model, trained on videos up to 5 seconds long at 24 fps and 380 × 640 resolution. Audio is sampled at 16kHz and encoded through DenseAV for conditioning. Text is encoded through CLIP. Temporally-aware RoPE frequencies are employed to aligned the modalities in the aforementioned cross attention layers.\n\nEvaluations are conducted on the AVSync15 dataset using both objective (FID, FVD, Image-Audio similarity, Image-Text similarity, CycleSync) and subjective metrics (IQ, FC, Sync). They demonstrate that the proposed method outperforms the AVSyncD on all axes.\n\nOn the Greatest Hits dataset, the proposed method outperforms on the CycleSync and FVD metrics, showing its emphasis on both video generation quality and audio visual synchronization.", "questions": "Formatting:\n- Add spacing before opening parentheses (around the line 347).\n- In the table 3, I suggest better highlighting which model is the final version (maybe in the top row and by adding a row separator).\n- As one of the paper's contributions is a classifier-free-guidance design, I would suggest adding some ablations of the choice of the cfg parameters.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper quality is enforced by thorough ablations and experiments, presented both in the main sections and appendixes, notably:\n- demonstration of the greater sensitivity of the CycleSync metrics compared with previously proposed ones, running correlation with human study and experimenting with controlled settings such as temporal shifts.\n- ablations on the Motion-aware Loss and Audio Sync Guidance contributions\n- pretrained model behavior study to understand where to inject the finetuning layers\n\nAlthough this paper presents an Audio to Video model, its contributions should translate into better the Video to Audio model designs.", "weaknesses": "The Figure 1 is not clear enough to me. It is not clear what the frozen and trainable layers refer to (are those transformer blocks?). I would suggest adding in the captions that the audio features are injected in the latter blocks (the expectation is usually to add cross attention to each block so the presentation is quite counter intuitive without a corresponding explanation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtxrS4k8YS", "reviewer": "Reviewer_4rJg", "summary": "Syncphony demonstrates meaningful progress in the audio-to-video generation domain. Its main contributions include: (1) a motion-aware loss that re-weights the training objective based on optical flow to focus the model’s learning on motion-intensive regions; (2) a training-free guidance technique that enhances the injection of audio information during inference without additional training; and (3) a more intuitive and reasonable evaluation metric for measuring audio-visual synchronization through reconstructed audio from generated videos. Experiments on two public datasets and qualitative case studies further validate the method’s effectiveness in improving both synchronization and visual quality.", "questions": "- Could the authors elaborate on how they plan to address the challenges posed by more realistic (or longer) audio-to-video scenarios, where optical flow estimation can be severely affected by background motion, camera movement, or scene transitions beyond the main subject?\n\n- Could the authors show how the skip-layer behavior manifests in other video generation models and whether similar phenomena can be consistently observed? In addition, if temporal alignment is indeed a crucial aspect of the task, why not consider using energy-based audio features as a more direct form of control or guidance?\n\n- Given that modern multimodal large language models (e.g., Gemini 2.5) already demonstrate strong capabilities in understanding audio-visual information, and that existing video-to-audio alignment metrics (such as DeSync, which can be applied similarity in a2v field) provide reliable proxies for spatiotemporal correspondence, could the authors clarify or compare what specific advantages their proposed metric offers over these established approaches?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed training-free guidance is novel and effectively points out the core challenge of achieving precise spatiotemporal alignment in audio-to-video generation.\n\n- The proposed evaluation metric is intuitively reasonable and addresses the limitation of conventional metrics like FVD, which fail to effectively measure spatiotemporal alignment.\n\n- The use of RoPE for spatiotemporal positional encoding enhances temporal consistency and spatial coherence in video generation, contributing to smoother and more structured motion representation.\n\n- The demo results demonstrate good temporal alignment consistency.", "weaknesses": "Although the work is technically sound, there are several issues that I feel must be discussed.\n\n- Regarding the motion-aware loss, it relies on a strong assumption that the visual content remains static and confined to a single scene. This assumption holds almost perfectly under the authors’ setting where short video clips of around two seconds. However, in more realistic video generation scenarios, background changes, camera motion, and scene transitions can occur without producing strong audio cues, but may catastrophically distort optical flow estimation and thus limit the scalability of the proposed approach.\n\n- The proposed guidance technique randomly drops audio cross-attention layers during inference; however, this inference structure (unlike CFG or Autoguidance) is never encountered during training, making the resulting “unconditional” outputs less predictable. Moreover, although the authors provide some visual demonstrations, the underlying motivation may primarily apply to relatively smaller models where audio conditioning tends to be weaker. In larger-scale video generation frameworks such as Wan or Hunyuan-Video, audio conditions may not be as easily ignored, and models could exhibit different skip-layer behaviors. Rather than validating only on additional datasets, I would prefer the authors to verify this idea across multiple video generation baselines to strengthen the generality of their claims.\n\n- Although the proposed new metric is intuitively reasonable, current video-to-audio (V2A) models also suffer from (or are still addressing) spatiotemporal alignment issues, which means they may not serve as a fully reliable ground-truth proxy. Moreover, the approach fundamentally increases evaluation time, potentially limiting scalability to larger experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fn47zRvXcf", "reviewer": "Reviewer_DdqZ", "summary": "This paper proposes Syncphony, a diffusion-transformer-based framework for audio-to-video (A2V) generation that achieves fine-grained synchronization between sound and motion. The method introduces two key innovations: 1) Motion-aware Loss — reweights reconstruction loss to emphasize high-motion regions, improving temporal alignment. 2) Audio Sync Guidance (ASG) — uses an auxiliary “off-sync” model (without audio layers) to guide the full model during sampling toward stronger audio-motion coupling. To evaluate synchronization, the authors further introduce CycleSync, a new video-to-audio-based metric that measures whether the generated motion contains sufficient cues to reconstruct the original audio. Experiments on AVSync15 and TheGreatestHits datasets show improved synchronization and comparable or better visual quality relative to baselines such as AVSyncD and TempoTokens.", "questions": "1. Fig 5 and 6: it will be helpful to include the ground-truth video as a reference. \n2. How does ASG compare with the vanilla classifier-free guidance? For example, use the features with and without audio input as guidance. \n3. The paper only finetunes the last 16 blocks (8–23) of the Pyramid Flow backbone. Are there any experimental results supporting the benefits of this choice, e.g., compared to finetuning the full model?\nOthers: see the weakness section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes Motion-aware Loss and Audio Sync Guidance to improve the audio-visual synchronization of A2V generation, which are conceptually simple and empirically effective.\n2. The proposed metric CycleSync offers a meaningful step forward over prior metrics (AV-Align, AlignSync, RelSync) by enabling evaluation at 24 fps and better correlating with human perception.\n3. The paper also proposes a principled strategy to adapt a pretrained I2V model for the A2V task by selecting the most relevant layers to inject the audio cross-attention layer.\n4. Syncphony consistently outperforms baselines on synchronization (CycleSync) and achieves competitive or superior FID/FVD, indicating that temporal precision does not come at the expense of visual quality.", "weaknesses": "1. While the paper’s ideas are sound, the architectural novelty is moderate. Syncphony heavily relies on a pretrained Pyramid Flow backbone, and the main innovations are at the loss and sampling levels rather than core model design.\n2. The writing of the paper can be improved. For example, in the introduction, more details/motivations about the proposed methods could be included instead of the background information. \n3. Limited baselines: the proposed method is only compared with AVSyncD and Pyramid Flow. Is it possible to include more baselines, such as those listed in AVSyncD?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udu8DEyPmm", "reviewer": "Reviewer_2TxV", "summary": "The paper proposes Syncphony, an audio-to-video generation model that focuses on improving synchronization between audio and motion.\nTo achieve this, the authors introduce a Motion-Aware Loss for capturing movement intensity and an Audio Sync Guidance (ASG) mechanism for enforcing synchronization between audio and visual dynamics.\nA new evaluation metric, CycleSync, is also proposed to better align with human perceptual judgments of synchronization.\nExperimental results show that Syncphony performs better than most existing baselines.", "questions": "In the demo, only 2-second video samples are provided, even if it can generate 5 seconds video. Why is that?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe introduction of the CycleSync metric sounds reasonable and shows higher alignment with human perceptual evaluations than prior synchronization metrics.\n2.\tThe model achieves consistently better performance than most baselines across several benchmarks.", "weaknesses": "1.\tGenerating a 5-second video takes nearly 3 minutes, which significantly limits practical usability.\n2.\tThe main model builds heavily on Pyramid Flow, with only moderate extensions (audio conditioning and synchronization guidance). As such, the contributions feel incremental.\n3.\tThe CycleSync metric relies on pretrained V2A models that may introduce background or irrelevant audio content, potentially biasing the evaluation.\n4.\tThe proposed Motion-Aware Loss does not explicitly capture semantic motion as authors mentions. It merely measures pixel or latent differences between consecutive frames, which may not correspond to meaningful sound-related motion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)