ViSAGe: Video-to-Spatial Audio Generation
Authors: Jaeyeon Kim
Heeseung Yun
Gunhee Kim
Venue: ICLR 2025 Poster (2025)
Keywords: Audio Generation
Audio-Visual Learning
Spatial Audio
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose novel problem of generating first-order ambisonics from silent field-of-view videos. We introduce novel dataset and method to deal with the problem.

Abstract: Spatial audio is essential for enhancing the immersiveness of audio-visual experiences, yet its production typically demands complex recording systems and specialized expertise. In this work, we address a novel problem of generating first-order ambisonics, a widely used spatial audio format, directly from silent videos. To support this task, we introduce YT-Ambigen, a dataset comprising 102K 5-second YouTube video clips paired with corresponding first-order ambisonics. We also propose new evaluation metrics to assess the spatial aspect of generated audio based on audio energy maps and saliency metrics. Furthermore, we present Video-to-Spatial Audio Generation (ViSAGe), an end-to-end framework that generates first-order ambisonics from silent video frames by leveraging CLIP visual features, autoregressive neural audio codec modeling with both directional and visual guidance. Experimental results demonstrate that ViSAGe produces plausible and coherent first-order ambisonics, outperforming two-stage approaches consisting of video-to-audio generation and audio spatialization. Qualitative examples further illustrate that ViSAGe generates temporally aligned high-quality spatial audio that adapts to viewpoint changes.

Published conference paper at ICLR 2025
VISAGE: VIDEO-TO-SPATIAL AUDIO GENERATION
Jaeyeon Kim, Heeseung Yun & Gunhee Kim
Scoul National University
jaeyeonkim99@nnu.c ac. heeseung. wwnevision..snnnn snu.ac.kr kr, gunhee@snu.
ABSTRACT
Spatial audio is essential for enhancing the immersiveness of audio-visual ex- periences, yet its production typically demands complex recording systems and specialized expertise. In this work, we address a novel problem of generating first- order ambisonics, widely used spatial audio format, directly from silent videos. To support this task, we introduce YT-Ambigen, dataset comprising 102K 5- second YouTube video clips paired with corresponding first-order ambisonics. We also propose new evaluation metrics to assess the spatial aspect of generated audio based on audio energy maps and saliency metrics. Furthermore, we present Video- to-Spatial Audio Generation (ViSAGe), an end-to-end framework that generates first-order ambisonies from silent video frames by leveraging CLIP visual fea- tures, autoregressive neural audio codec modeling with both directional and visual guidance. Experimental results demonstrate that ViSAGe produces plausible and coherent first-order ambisonics, outperforming two-stage approaches consisting of video-to-audie generation and audio spatialization. Qualitative examples further illustrate that ViSAGe generates temporally aligned high-quality spatial audio that adapts to viewpoint changes. Project page: https//jaeyeonkiin9.giit gjthub.io/vvisee..
INTRODUCTION
Humans perceive the world through both auditory and visual cues, each of which conveys significant spatial information. Visual cues enable them to locate objects, while auditory cues help estimate the interaction betweeen objects in the environment based on the origin of sounds. Hence, spatial audio is vital for creating immersive experiences in the visual scenes (Poeschl et al. 2013; Holm jet al. 22000) HHiway et al., 2022] Nguyen & Willson 2023 Hirway al.) 2024). This makes spatial audio production essential for many applications such as film, virtual reality, and augmented reality. However, producing spatial audio typically requires expensive sound-ficld microphones, professional production equipment, and advanced technical expertise (Zotter & Hrank 2019).
Sound effects in videos are often recreated during the post- pestt-roduction stage due to challenges associated with on-location audio capture (Ament (014), adding more complexity to the task of producing spatial audio for visual content. Hence, generating appropriate spatial audio for silent videos has immediate and impactful applications for enhancing the immersive experience in various media. Moreover, recent ad vancements in generative models have enabled the creation of videos from textual descriptions (Ho al.) 2022bb Singer et al. [023 Brooks al. 2024). further increasing the demand for creating audio streams that align with spatial and contextual characteristics of videos.
Previous works have shown remarkable progress in generating audio from silent videos and spatial audio from mono audio. However, generating spatial audio directly from silent videos remains an unsolved challenge. Current video-to-ane audioo generation models (lashin & Rahtu 202T Luo et al.)
2023] Wang et al.) 2024 Pascual et al., 2024) are capable of producing audio based on the content and timing of the video. However, these models generate only mono audio, whereas spatial audio requires the generation of multiple channels with proper arrangement to convey sense of space.
Audio spatialization models can generate binaural audio (Gao & Grauman 2019 Zhou et al. 2020 Li et al. 2024b) or first-order ambisonics (Morgado et al. 2018 Lim & Nam 2024), from mono audio using visual cues. However, they necessitate a reference mono audio, which is not available for silent videos. Combining video-to-audio generation and audio spatialization may introduce additional challenges. The generated audio often deviates from the ground- distribution and may not align with the timing content of the video, potentially leading inaccurate spatialization.

---

Published conference paper at ICLR 2025
Silent video
ViSAGe
Tret-orderAmmisoniic Ambisonics Audio Energy Map
Ground- Truth Audio Energy Map
W
P
Y
Direction
2
YT-Ambigen
102K
Saliency Metrics
Figure 1: Sadeo-to-Spatial Audio Generation. Given a silent video and the camera direction, the model generates corresponding first-order ambisonics. The camera direction gives cue about where the visual event occurs, enabling the model to generate an appropriate three-dimensional sound field.
In this work, we introduce novel task: generating first-order ambisonics, a widely used spatial audio format, from silent videos, as in Figure To address this task, we introduce Y'T-Ambigen, new dataset comprising YouTube videos paired with first-order ambisonics, tailored for the audio generation. To evaluate the spatial quality of the generated ambisonics, we propose novel metrics derived from audio energy maps and visual saliency. Furthermore, we present Siteo-to-Spatill Audio Generation (ViSAGe), end-to-end framework designed to generate and spatialize audio based on visual content and camera direction. ViSAGe leverages CLIP features, patchwise energy maps, and neural audio codecs along with rotation augmentation. Additionally, ViSAGe incorporates code gen- eration scheme and guidance optimized for the simultaneous generation of multiple spatial channels. Extensive experiments on YT-Ambigen show that ViSAGe outperforms two-stage approaches, which separately handle video-to-audio generation and audio spatialization, across all metrics.
2 RELATED WORKS
Video-to-Audio Generation. Earlier works on creating soundtracks for silent videos focused on
limited set of classes (Owens al., 2016 Zhou al. 2018 Chen et al., 2020b). VQGcVQGAN
(lashin & Rahtu (021)) was the first to generate sounds for open-domain videos with autoregressive
transformers trained on VQ-GAN (Esser et al. 202T codebooks of mel-spectrograms. IM2WAV
(Sheffer & Adi 2023) extended this by training on hierarchical VQ-VAE (Razavi et al. 2019)
codebooks. Diff- FoleyFolyy&ee (Luo et al.) 2023) introduced contrastive audio-visual pretraining and latent
diffusion for improved audio-visual synchronization, while V2A-mapper (Wang et al., 2024) used
AudioLDM (Liu et al.
2023) by mapping CLIP (Radford et al.
202T) and CL.AP (Wu et al. 2023).
The works most closely related to ours are FoleyGen (Mei et al.) 2023) and Mask (Pascual et al.
2024), which use CLIP features for visual conditioning and neural audio codecs (Défossez et al. 2023 Kumar et al., 2024) for audio generation. FoleyGen employs autoregressive transformer, while VAT uses masking-basee generation (Chang et al. 2022). Unlike existing approaches that generate mono audio, ViSAGe generates multiple spatial channels simultaneously using spatial cues.
Generating spatial audio typically requires specialized equipment and expertise, motivating research into creating spatial audio from various conditions (Kushwaha et al. 2025 Heydari et al. 2025). In particular, several studies have explored generating binaural audio from mono audio using visual cues. Mono2Binaural (Gao & Grauman| 2019) proposed a UNet-like framework to predict binaural channel masks using visual cues. Sep-Stereo (Zhou et al.
Audio Spatialization with Visual Cues.
2020) extended this by jointly modeling source separation and binauralization, while Binuurlinaural
(Xu et al. (02T)) applied the same approach to pscudo-binaural data generated from mono audio.
More recent works utilize multitask-basee geometry aware features (Garg et al. 2023) and cyclic learning with localization (Li et al., 2024b) for binauralization.
Another line of works generate first-order ambisonics from mono audio based on panoramic videos. SpatialAudioGeen (Morgado et al.] (018)) produced spatial channels by separating sound sources inside the mono omnidirectional channel and localizing them in the correct direction. Similarly, Rana et al. (2019) predicted sound source locations and manually encoded the audio based on those predictions. Lim & Nam (2024) improved SpatialAudioGGen by incorporating a pretrained source separation model and channel panning loss between spatial channels. In this work, focus first-order ambisonics, a versatile spatial audio format that can be decoded into various formats, including binaural audio. Unlike the above methods, ViSAGe generates spatial audio purely from visual content, without the need for a reference mono audio.
2

---

Published conference paper at ICLR 2025
Audio Generation Using Neural Audio Codecs. Neural audio codecs are autoencoders that compress audio signals into sequences of discrete codes (Zeghidour et al. [021] Défossez et al. 2023 Kumar
et al.. 2024). Due to their discrete latent space and superior audio reconstruction, they are widely used for generating audio (Kreuk et al. 2023 Ziv et al., 2024). speech (Borsos et al. [023 Wang et al.
2023), and music (Copet et al., 2023 A Agssinnelli al. 2023 Ziv et al. 2024 Li et al. 2024a). Recent neural audio codecs often utilize residual vector quantization R applying multiple codebooks to quantize the residuals from earlier steps (Zeghidour et al. (021). This creates challenges in managing multiple code sequences, leading to various code generation strategies (Borsos et al. 2023; Wang et al., 2023 Agostinelll et al.] 2023 Coper et al., (023). While Copet al. (2023) and [Li al. (2024a) explored strategies for stereo music, most works focus on mono audio due to the complexity of modeling residual code sequences even for a single channel. In this work, we propose an efficient
method for generating all four channels of first-order ambisonics using neural audio codecs.
3
VIDEO-TO-AMBISONEE GENERATION
3.1 BACKGROUND: FIRST-ORDER AMBISONICS
Ambisonics is a three-dimensional surrounding sound format that captures and recreates sound fields using spherical harmonics. Due to its accurate and scalable representation of sound sources from all directions with desired precision, ambisonics plays a crucial role in immersive audio experiences. Among a variety of formats, First-order ambisonies (FOA) employs four channels (W,X, Y. Z) to encode the sound field with first-order spherical harmonic decomposition. The W-channel corresponds to the sound from an omnidirectional microphone the center, while each directional channel (X, Y. Y) amounts to the sound from figure-of-eight microphone aligned with the corresponding axis. FOA is more widely used than other higher-order representations due to its affordable and efficient nature well as compatibility with popular video streaming services like YouTube. Compared to other surround sound formats like 5.1 surround sound, which favors a fixed frontal field, ambisonics offers unbiased playback of directional information with precision (Courville & Studio 1994). promoting immersiveness in dynamic user-centric scenarios.
One of the primary advantages of ambisonics format is that can explicitly map the energy of auditory information to the spherical coordinate system. This energy map reveals the direction from which the audio energy originates, representing the spatial characteristics of the sound. Using spherical harmonics decomposition, we can derive an audio energy map G(ò, 0) for & E [0, [0, 2T] with respect to real spherical harmonics Y!" (o, B) and audio length L:
G( - 1 Σ (Y) ())()) + Y (,, (0) + ()Z(() + ()())))((t). (1)
-
3.2 TASK DESCRIPTION
Mideo-to-ambisone generation addresses the problem of generating FOA channels (W, W,X.Y. Z) given silent video frames. This brings about major challenges in methodology and evaluation that have not discussed in prior spatialization or video-to-audio generation tasks. Compared to mono audio generation problems, each generated channel should be plausible and visually entailing while maintaining consistency with one another. Moreover, all channels should form a spatially coherent sound field for immersiveness, i.e., the audio perceived by a person viewing from a specific direction should also be plausible. Such spatial coherency should hold no matter what content is conveyed in the generated audio by preserving the directions of dominant sound sources.
For visual conditioning, we use a combination of field-of-vier (FoV) videos and their corresponding camera directions. Although ambisonics are typically provided with panoramic videos due to their three-dimensional nature (Morgado et al. 20188] 2020), we opt for V-b-ed conditioning since FoV has much broader applications compared to panoramic videos and can be easily integrated with traditional video-to-audio generation methods and benchmarks. However, FoV alone lacks information about where visual events occur within the three- direeddimnsiooall environment. To address this, we include the camera direction as input, representing where the visual scene is taking place, allowing the model to create accurate sound field based on the visual content. This approach not only enhances spatial awareness but also offers users greater control when generating the FOA.
3

---

Published conference paper at ICLR 2025
3.3 EVALUATION METRICS
Semantic Metrics. We adopt two widely used metrics, Fréchet Audio Distance (FAD) (Roblek et al. 2019) and Kullback-Leibler Divergence (KLD), evaluate the semantic aspects of the generated audio. FAD is defined as the Fréchet distance between the feature distributions of the generated and ground-truth truntt audio, as extracted by a pretrained audio encoder. FAD reflects the perceptual quality and fidelity of the generated audio, whereas KLD measures the KL divergence between the class distributions of the generated and ground-truth audio, evaluating how well the generated audio captures the intended audio concepts. Since the pretrained classifiers used for metric computation require mono audio input, we decode both the ground-truth and predicted FOA into mono audio based on the ground-truth camera direction (Ø, 0), where é represents the azimuth and 0 the elevation:
$ W X Φ cos + Y sin Φi cos 0 sin 0.
W. X, Y, and represent the respective channels of the FOA. The decoded mono audio equivalent to recording from the virtual 3D cardioid microphone heading (ø) 8), reflecting what the listener would likely hear the scene. We use the decoded mono audio, rather than W, as the representative mono audio to ensure that the semantic coherence of the generated ambisonics channels can be evaluated. We report FAD and KLD evaluated on decoded mono audio, i.e., FAD and KLD
For generated FOA, the fidelity of each channel is also crucial to the listener's 's experience, since these channels can be combined in various ways depending on the listener's location and direction. To assess the fidelity of the individual channels, we report the average of FAD from each channel, i.e., FADagg which can capture the overall plausibility of generated ambisonics.
Spatial Metrics. Previous works audio spatialization (Morgado et al. 2018 Gao & Grauman 2019) have utilized distance-hased metrics such as STFT, Log-spectral, and Envelope distance, which compare spatialized channels from reference
audio with ground-truth spatial channels. However, these metrics are not suitable for evaluating generated audio with varying content and timing, since they cannot be directly compared ground-truth audio. To address this limitation, we propose a new set of metrics to evaluate the spatial aspects of generated ambisonics.
As in Eq. M first-order ambisonics using spherical harmonies decomposition. We adopt visual saliency metrics (Bylinskii et al.) 2018) to evaluate the similarity between the original energy map and the generated energy map, typically in the form of a heatmap over an equirectangular panorama of elevation by azimuth (Cheng et al. 2018). key distinction in this adaptation is that we mitigate oversampling bias evaluating saliency. The energy evaluation between equirectangular panoramas is prone to oversampling around 0 = 0 and = n, making the evaluation less accurate when the auditory source deviates from the center. We prevent this with trivial overhead by reducing the number of sampled points for evaluation by sin 6.
be used to generate an audio energy map over the sphere
We calculate the Correlation coefficient (CC) and the Area Under the Curve (AUC) values between the audio energy maps of the generated ambisonics and the ground-truth audio. To measure the spatial coherence the generated sound field with respect to varying temporal granularity, we report CC and AUC for different temporal windows: energy map over full generated audio (CC.n, AUCan), energy map aggregated every 1000ms (CC AUC))p)) and 200ms (CCsips- sUC)))s).
4 DATASET: YT-AMBIGEN
Motivation. Existing datasets on video-to- generation or spatialization fall short in addressing the Video-to-ambison generation. As outlined in Table []] only restricted number of datasets cover video-to-audin problems at scale (i.e., >100)). Previous datasets with spatial audio either only contain 360 videos visual conditions or have not been demonstrated in audio generation.
In addition, video-to-audio generation itself considerably challenging, even with the largest am- bisonics video dataset available. Experimental results in Table Iableess that competitive generative model on VGGSound (Chen et al. 2020a) struggles to train or finetune with YT360 (Morgado et al., 2020), often producing noise-like like sounds as outputs. We hypothesize this performance gap largely attributes to the quality of existing datasets for generative problems. Therefore, we newly propose large-scale dataset specifically designed to meet the needs of video-t--ambioniie generation, enabling more accurate and contextually relevant spatial audio synthesis. YT-Ambigen dataset com-

---

Published conference paper at ICLR 2025
Table 1:
Comparison of YT-Ambigen with existing datasets. FoV and 360 respectively denote field-of-view videos and panoramic videos. NS, B, and FOA refer to non-spatial audio, binaural audio, and first-order ambisonics, respectively. (" ("Number of audio classes < 15)
# of
Video
Vidco
Audio
All Spatial
Open Domain
Audio Generation
Dataset
Clips
Length
Type
Type
Channels
Greatest Hits (Owens et al.
2016
IK
11h
FoV
NS
X
VEGAS (hou et al. 2018
28K
55h
FoV
NS
VAS (Chen ct al. 2020b
13K
24h
FoV
NS
VGGSound (Chen et al 2020a
200K
560h
FoV
NS
FairPlay (Gao & Grauman 2019
2K
5h
FoV
B
X
OAP (Vasudevan cl al. 2020
64K
26h
360
B
X
X
REC-STEEET (Morgado et al. 2018
123K
3.5h
360
FOA
V
X
AXX
X
YT-ALL (Mogado el al 2018
3976K
113h
360
FOA
X
YT-360 Morgado et al 2020
89K
246h
360
FOA
X
STARSS23 (Shimada (Shmada et al.
2024
0.2K
7.5h
360"
FOA
Yl-Ambigen
102K
142h
FoV
FOA
prises total of 102,364 five-second FoV clips with corresponding FOA and camera direction (ø, 0). which is divided into 81,594 9,604 / 11.166 clips for training, validation, and test, respectively.
Dataset Curation. We address two major issues identified in YT360 for video-to-audio generation: (i) the absence of semantically significant audio events due to amplitude-basee coherence between the audio events and visual in- formation. Using 5.2K panoramic videos with first- order ambisonics collected from YouTube, first filter out videos where the average absolute ampli- tude of any channel per second less than 10-20 to ensure the presence of all four channels.
Table 2:
Video-to-audio generation results of the model in Sec. [ for different datasets.
X + y/ indicates pretraining and finetuning datasets.
filtering and (ii) weak
Dataset
FAD,
KLD.
VGGSound
3.62
2.23
YT360
15.91
3.10
VGGSound + YT360
12.92
3.16
YT-Ambigen
3.95
VGGSound YT-Ambigen
3.42
1.75
We then focus clips with noticeable audio events that are suitable for generation. For each 1s segment in all videos, we determine the validity of the clip by thresholding the root mean square energy of the segment. These segments are merged into 5s clips by retaining only those with >3s of valid audio. We utilize a 5s temporal window to capture coherent events with sufficient length for generation, considering longer clips tend to include fewer valid segments. To further ensure each clip contains semantically recognizable audio events, we use an AudioSet classification model (Koutini jet al. 2022) to recursively select clips with high-probability sound event labels. Selected clips cover over 300 distinct AudioSet classes, demonstrating their suitability for open-domain generation.
Moreover, we try to ensure that the visual cues for audio generation are within the FoV. We calculate the audio energy map for each clip to identify the argmax coordinate, where the sounding events likely happening. We then crop the panoramic video around this point to obtain FoV videos. Finally, we filter out clips based on audio-visual relevance scores (Luo et al. 2023). removing any clips with scores lower than one standard deviation below the mean.
APPROACH: VISAGE
The overall architecture of the proposed model, ViSAGe, is illustrated in Figure (a). Let video frames be V E T3xx where T. H, and W denote the time, height, and width, respectively. The camera direction is given by D (ő, 0) for azimuth o and elevation 0. The goal of ViSAGe is to generate first-order ambisonics A = (W, X, Y, Z) E R44L, where L is the length of the waveform, by modeling the conditional probability p(A[V, D) with transformer encoder-decode architecture.
5.1 CONDITIONAL ENCODING
The video frames V and the camera direction D is conditioned through the transformer encoder. For V. we use CLIP (Radford et al. 202T) features to capture semantic content, while proposing the use of patchwise energy maps capture fine-grained spatial cues within the frames. Meanwhile, D is processed into a direction embedding to provide cues for overall spatiality.
5

---

Published conference paper at ICLR 2025
Inference
Training Cross-Entropy Loss
Training
Inference Autoregressiiee code generation
W
W
Sp
4
Codebook Logits
Linear
Projection
Soae-us PCadePrrdiction Prediction
DAC Decoder
DAC Decoder
Transformer Encoder
Transformer Decoder
Linear Projection
Generated Ambisonics Channels
Trainable Frozen
Transformer Decoder
<BOS>
Concat
Linear Projection
Embedding Layers
Embedding Layers
Layers
MLP Layers
W,
&
W,W. channel codebooks
Patchwise Energy Map
CLIP Image Encoder
DAC
W
12
(S,, S,). Spatial codebooks
Encoder
S
1
(a: Primary /r Reakhwl)
4: DAC Code timestep k
S,
Padding
Reorganize
(#.
Securnes
Camera
Silent Video
Ambisonics Channels
Direction
Frames
DAC Encoder
Directional
Visual
Guidance
Guidance
Ambisonics Channels
(a) Overall Architecture
(b) Ambisonics Generation
Figure 2: (a) Overall architecture of ViSAGe and (b) its ambisonics generation with DAC codes. Each block in (b) represents all residual codes belonging to the corresponding codebook group.
CLIP Embeddings. We choose CLIP over other visual encoders due to its superior performance in previous rideo-to-audio generation works (Sheffer & Adi 2023 Mei et 2023 Pascual et al.. 2024). The video frames V IR x3x are transformed using pretrained CLIP image encoder and joint projection, and then linearly projected to align with the transformer's ' hidden dimension, yielding T E d'd,, where de is the dimension of the transformer encoder's hidden states.
Patchwise Energy Maps. While CLIP effectively captures the overall semantics of visual frames, it lacks spatial information, such as the location and movement of sounding objects. To address this limitation, we introduce using a patchwise energy map as an additional visual input to extract fine-grained, temporally aligned spatial information from video frames. First, obtain patch-level image embeddings before pooling, denoted as ep E 7 bxxba xdp, from the pretrained CLIP image encoder. Here, h H/p and w W/p with the patch size p, and dp is the dimension of the CLIP image embeddings before the joint projection. Similar to the spatial and temporal saliency approach from PAVER (Yun et al. 2022), the spatial and temporal scores of each patch are calculated based on the patch's 's embedding distance from its spatial and temporal neighbors. Let 2ij E IR denote the embedding for the ij-th patch at time-step t. The spatial score S and the temporal score T for embedding æi, given spatial and temporal window N, T and cosine similarity de are defined
S+N Σ
L+T Σ k=1-T
Sts = 22d (2i
Σ N=j-
T 2d, (d 2T
(2N 1)2
High spatial scores indicate that patch contains content distinct from adjacent patches, while high temporal score suggests that patch contains temporally changing information such as moving object. Therefore, these scores are correlated with the location and movement of the sounding object in the scene. Next, the scores are converted into probabilities by applying softmax over the patches and averaged, forming energy map over the patches, i.e., E [o, 1] x This energy map is flattened and passed through MLP layers, resulting in the final energy map embedding I E TT
Direction Embedding. We the direction embedding to control the overall spatial directivity of the sound field. The camera direction D = (ø, 0) is first mapped to a unit vector u E R3 Cartesian coordinates for smooth interpolation across different directions. This unit vector is then projected through MLP layers and duplicated along the axis to form the direction embedding I E R Xxx,

---

Published conference paper at ICLR 2025
Transformer Encoder. To condition the input features, the embeddings 1, Tes and Ta are summed and then concatenated with learnable embeddings that represent the start and end of the sequence along the temporal dimension. Positional embeddings are added to capture the sequential order of the inputs. The resulting features are fed into the transformer encoder layers.
5.2 AMBISONIC' GENERATION
We model first-order ambisonics using a neural audio codec, which encodes waveforms into se- quence discrete codes. This enables the use of discrete modeling techniques such as autoregressive generation, while the predicted codes can be decoded back into waveforms. To facilitate this process, we propose strategies to model neural audio codes for ambisonics generation.
Descript Audio Codec Encoding, The FOA channels are transformed into audio code matrix using the Descript Audio Codec (DAC) encoder (Kumar et al. 2024). a state-of-the art nerual codec for open-domain audios (Wu al. 2024). Based on residual vector quantization (RVQ) that quantizes the residuals of previous codebooks, DAC encoder compresses each ambisonics channel into a discrete code matrix C E vNxLe where N is the number of codebooks used in the RVQ process, Le is the length of the compressed audio, and V is the vocabulary of the codebooks. Each row Cn.: corresponds the code sequence from specific codebook, while each column C. represents the codes at given time step. To handle all channels simultaneously, we concatenate the code matrices from all four ambisonics channels along the codebook dimension, forming C. E 444N1...
The Code Generation Pattern. The proposed code generation pattern is illustrated in Figure 2 (b). Generating first-order ambisonics channels requires handling four times more code sequences compared to mono audio while ensuring both semantic and temporal coherence across all channels. For a given ambisonics code matrix C. E y4NxL= and codebook index I i S 4N,, we divide them into four groups to effectively model the dependencies between code sequences: W,. W,, Sp. and S,- W and S denotes omnidirectional (W) and spatial (X. Z) channels and p and r denotes primary and residual codebooks, respectively. That is, for P { i mod N 1} and W = (i < i < N), W = {(Ca),. S, ((C.).,; i E P, W). and S, = ((C.)..
P,i W). W = ((C.)..: $ P.i E W). WJ.
We hypothesize that two types of dependencies should be modeled among these groups of codebooks. First, must capture the dependency between the primary and residual codebooks (residual dependency), as later codebooks depend on earlier ones in RVQ. Second, we need to model the dependency between the omnidirectional and spatial codebooks (spatial dependency), since the spatial channels should remain semantically coherent with the omnidirectional channel while varying in amplitude according spatial cues. naive approach would model these dependencies sequentially in the order of W, + W, + Sp Sr; yet, it results in sequence length of 4T.--
To address this, we propose a more efficient generation pattern that requires only 2 + steps. We first generate Wp. followed by Sp), then W,,,S, and so forth. For sequential step 1 < < 2Le + 1 and codebook index 1 S < 4N, we modify C. to C² E VVNNx(LL+1);
(Ca).,z'
if 8 mod 2 = 1, + 2L. + (Ca)i, E,, s mod = 1, $ 0, (Ca).: S, if s mod 0, (Ca)..: U. else (padding)
(Ca). (Ca)
(C.)
(4)
Autoregressivley modeling the sequential columns (C)).. enables modeling both residual and spatial dependencies. C% serves as both the input and the prediction target for the transformer decoder.
Transformer Decoder. In order to model discrete code matrix C² with the transformer decoder, each row is embedded using separate embedding layers and then summed with positional embeddings. A learnnable <BOS> embedding is concatenated at the start of the sequence to be used for autoregressive generation. The resulting sequence is passed through the transformer decoder layers. The final hidden states of the decoder are fed into separate linear layers, which predict the logits for each row of C²,-
Training and Inference. During training. use the cross-entropy loss between the ground-trun -trttth code matrix C² and prediction. During inference, the model autorogressively generates codes for each sequential step. These codes reorganized the original DAC code matrix format, and decoded through the DAC decoder to generate respective channels. More details are deferred to the Appendix.
7

---

Published conference paper at ICLR 2025
Rotation Augmentation. To guide the model in capturing spatial aspects and to disentangle visual features from the viewing direction, we utilize a rotation augmentation strategy. For a given rotation matrix R E Rxxx3, the sound field of the first-order ambisonics channels (W, X, Y, Z) can be denotes the rotated
rotated as W" = W and "',,Y'Z'' ambisonics channels. Since elevation is closely tied to visual features- e.g., the sound of river cannot originate from above the river is flowing below we perform azimuth rotation for augmentation. Specifically, during training, with probability of 0.5, rotate the azimuth by 90º along the 2-axis, while simultaneously adjusting the input viewing direction from D = (e) 0) to D' = (Φ+ 0). This rotation is selected for augmentation because it is computationally efficient and can be implemented by (W", X', Y', Z = (W,- -Y, X, 2), while minimizing the loss of information that could arise from altering the relationship between direction and visual features.
= R(X, Y, Z) where ,X,"",, Z')
Directional and Visual Guidance. We employ classifier-free guidance (Ho & Salimans 2022) on the predicted logits for DAC codes enhance the generation of spatial audio. We introduce applying classifier-fi free guidance to the directional condition to improve the spatial accuracy of the audio. During training, the directional unit vectors u are replaced with null embeddings with the probability of 0.1. During inference, unconditional logits for classifier-free guidance are generated by replacing with null embeddings. For the reorganized code matrix C% E y4N x21xx let C²2 = (C").... Given input video frames V and a camera direction D, DAC codes for sequence step are sampled from (C Ch V, D) + wlog Pa (C2 Cir-1 Ø, D) - logp (C C1a-1, R3, Ø)), (5) where w and ps denotes the guidance scale and the probability parameterized by ViSAGe, respectively.
Furthermore, we adopt guiding both the directional and visual conditions simultancously to further improve the semantic quality of the generated audio. CLIP embeddings and the patch-wisc energy map E are additionally replaced with null embeddings with a probability of 0.1 during training. At inference time, we guide both conditions jointly, as we observe that they are closely related. Thus, DAC codes are sampled from the modified log probability that replaces the first Ø to V. Based on a hyperparameter sweep, we use guidance scale 2.5 throughout the experiments.
6 EXPERIMENT
6.1 SETUP
We use YT-Ambigen as main testbed for tideo-to-ambisonics generation with the evaluation protocol explained in Sec. Sec.B.3 [33 We first pretrain ViSAGe on VGGSound for mono audio generation, followed by finetuning on YT-Ambigen. During pretraining, only CLIP features are used as input, and we train the codebook embeddings and projection layers corresponding to the W-channel.
Baselines. Since there is no prior work to directly generate first-order ambisonics from FoV videos like ViSAGe, we compose baselines by merging vidco-to-audio generation with audio spatialization. For video-to-audio generation, we adopt SQGCN (lashin & Rahtu 202T) and Diff-Foley (Luo) al. 2023) as state-of-the-art open-domain generation models with publicly available implementation. We finetune the models pretrained on VGGSound to generate W-channel audio for YT-Ambigen.
We employ two methods to spatialize the W-channel audio generated by the iideo-t--audio models. First, we encode first-order ambisonics based on the ground-trun direction (Ambi Enc.). We encode mono sound source s(t) at direction (c) 0), into first-order ambisonics as follows (ootter & Frank 2019): W = -2
1 (t), X = cos & cos (t), Y = sin cos (s()), and Z = sin As((). Spatial audio
workstations typically encode mono audio into ambisonics by applying the eqution to each sound source composing the mono signal. As straightforward spatialization approach, we manually encode W-channel to FOA by treating (t) = V2W. This is conceptually equivalent to encoding the generated sound from a speaker located at (Ø. 0) into first-order ambisonics.
Additionally, we train an audio spatialization model based on visual cues (Audio Spatial.). Since none of the previous methods are fully compatible with our current setup, we train a spatialization model from scratch as a bascline. By closely following the architectures in Garg et al. (2023) and Liu et al.) (2024). the spatializer consists of U-Net architecture where the model learns to predict directional audio (X, Y. Z) from the complex spectrogram of the Wchannel. We tile and concatenate the visual features to the output of the audio encoder and decode train with 1.2 loss. We use CLIP as the visual feature backbone and encode camera direction for a fair comparison.

---

Published conference paper at ICLR 2025
Table 3: Results on YT-Ambigen. PT, DIR, PE, and RA stand for pretraining, directional embedding. patchwise energy map, and rotation augmentation, respectively.
Model
Semantic Metries
Spatial Metrics
CC₁
AUC; 1fps
V2A
Spatialization
BADee
Kl.D
FAD
All
1ps
5fps
All
5ips
Comparison to baseline models
SpecVQGAN
Ambi Enc.
5.94
2.56
5.62
0.349
0.337
0.322
0.687
0.680
0.670
Audio Spatial,
6.40
2.43
7.90
0.619
0.587
0.547
0.848
0.828
0.802
Ambi Enc.
5.68
2.60
5.53
0.349
0.337
0.322
0.687
0.680
0.670
Diff-foley
Audio Spatial,
7.24
2.51
8.76
0.577
0.537
0.494
0.826
0.803
0.777
ViSAGe (Directional)
5.56
0.721
0.671
0.624
0.864
0.839
ViSAGe (Directional& & Visual)
3.86
1.71
4,20
0.635
0,584
0.531
0.846
0.819
0.790
Ablation on Model Components
PT
DIR
PE
RA
3.73
1.76
4.19
0.430
0.398
0.362
0.708
0.689
3.74
1.77
4.04
0.524
0.482
01.439
0.778
0.757
0.734
4.44
1.89
4.49
0.544
0.498
0.449
0.794
0.770
0.745
1.78
4,30
0,531
0,486
0.441
0.782
0.759
0.735
3.86
1.71
4.20
0,635
0.584
0.531
0.846
0.819
0.790
Ablation Code Generation Pattern
Sequential Delay
10.75
2.66
8.66
0.386
0.341
0,311
0.746
0.716
0.695
Residual Only
4.68
1.97
4.37
0.480
0.448
0.413
0.777
0.759
0.738
Sputial Only
5.89
2.01
5.69
0,511
0,468
0.425
0.779
0.756
0.732
Ours
4.44
1.89
4,49
0,544
0.498
0.449
0.794
0.770
0.745
6.2 RESULTS
Table presents presents the overall results. ViSAGe with directional guidance outperforms the two-stage baselines in both semantic and spatial metrics, demonstrating its capability to generate semantically rich and spatially coherent first-order ambisonics. Importantly, manually encoded FOA exhibits better semantic quality but fail to capture spatial aspects adequately. On the other hand, ambisonics produced via the spatialization model effectively capture spatial information but suffer from reduced audio fidelity. In contrast, ViSAGe successfully balances semantic and spatial aspects, generating semantically and spatially accurate audio. Additionally, ViSAGe with both visual and directional guidance further improves semantic quality, albeit with some degradation in spatial accuracy. Never- theless, it performs comparably to the best-performing two-stage approach in spatial metrics, while significantly outperforming it in semantic metries. In subsequent experiments, we use both guidance.
Ablation on Model Components. We conduct ablation studies on several key components, including pretraining VGGSound, direction embedding, patchwise energy maps, and rotation augmentation. The results in Table]]show that while the overall semantic quality remains consistent when pretrained on VGGSound, both direction embedding and rotation augmentation significantly enhance spatial accuracy. Patchwise energy maps also contribute to increased spatial metrics by capturing fine- ineiiiai spatial details in the FoV scenes. Moreover, pretraining diverse video clips from VGGSound helps the model to generate semantically plausible audio.
We compare the proposed code generation pattern with alternative patterns that generates ambisonics channels in a similar or shorter number steps. Let C, V LL be the ambisonics code matrix. First, we adopt the sequential delay pattern (Li et al. 2024a) from stereo music generation, which delays the generation of later codebooks to condition them on earlier ones, requiring Le + 4N steps. We also compare our method with patterns that model only residual dependency, following (Wp. p. Sp) - (W, S,). and only spatial dependency, using (W,, W.) + (Sp: both requiring 2L, steps. Patterns are illustrated in Figure
Ablation on Code Generation Pattern.
The results in Table show that the sequential delay pattern fails to model both semantic and spatial aspects, indicating that generating first-order ambisonics cannot be achieved by simply adopting patterns that are successful for mono stereo audio generation. Additionally, modeling only residual dependency improves semantic quality compared to modeling only spatial dependency, but performs worse in capturing spatial accuracy. This suggests that residual dependency is closely linked to semantic quality, while spatial dependency critical for spatial accuracy. In contrast, our proposed code generation pattern outperforms all other patterns in both semantic and spatial metrics, effectively modeling both dependencies while maintaining a similar number of steps.
Qualitative Examples. As illustrated in Figure 3] the linear spectrograms demonstrate that ViSAGe generates semantically and spatially coherent ambisonics channels. We also compare the audio

---

Published conference paper at ICLR 2025
(a) Qualitative Examples
6fo
ViSAGe
W
Z
Audio Energy Visualization
Baseline
viSAGe
Figure 3: (a) Qualitative examples of generated audios and (b) audio energy visualization. Blue boxes highlight ViSAGe's ability to capture differences between spatial channels, while green boxes demonstrate that ViSAGe generates semantically plausible events.
energy maps from ViSAGe with those from the ablated model, which conditioned only on CLIP features. While the ablated model captures some degree of spatiality due to the inherent relationship between visual features and spatial aspects, ViSAGe captures significantly more fine-graid fineggrained details and temporal dynamics, highlighting the effectiveness of the proposed components. Additional qualitative examples are provided in Appendix
7 CONCLUSION
We addressed the challenging task of generating spatial audio, specifically first-order ambisonics, directly from silent videos-a task that has significant implications for enhancing the realism and immersiveness of audio-visual media. We introduced YT-Ambigen as large-scale dataset that pairs YouTube video clips with corresponding first-order ambisonics, providing valuable resource for future research in this area. To rigorously assess the spatial fidelity of generated audio, we proposed novel metrics that incorporate audio energy maps and visual saliency. Our proposed framework, ViSAGe, uniquely integrates neural audio codecs with CLIP-derived visual features, enabling the generation of semantically rich and spatially coherent ambisonics from video frames only. Experimental results confirmed that ViSAGe outperforms two-stage methods both semantic and spatial evaluations. The promising performance of ViSAGe, along with its ability to adapt to dynamic visual contexts, underscores its potential for broad application in immersive media production. Future work will explore further enhancements in spatial audio realism and extend the framework's 's applicability to other forms like higher-order ambisonics.
10

---

Published conference paper at ICLR 2025
Acknovledgement. This work was supported by Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant (No. RS-2019-11191082, SW StarLab; No. RS- 2021-I121 1343, Artificial Intelligence Graduate School Program (Seoul National University; No. RS- 2022-11220156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation) the National Research Foundation of Korea (NRF) grant (No. corresponding author.
funded by the Korea government (MSIT). Gunhee Kim is the
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv:2301.
2023.
Vanessa Theme Ament. The Foley grail: The art of performing sound for film, games, and animation. Routledge, 2014.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. Audiolm: A language modeling approach to audio generation. JEEE/ACM TASLP, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL httpt::////p/aaa www.opppennn..crreeeeeeeeeeeeeee com/research/
video-generation wate--mooel-s-se worwar-wwwooooooooo 1d- simulators
Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba. and Frédo Durand. What do different evaluation metries tell us about saliency models? IEEE TPAMI, 2018.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP. 2020a.
Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. JEEE TIP. 2020b.
Hsien-Tzu Cheng, Chun Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, and Min Sun. Cube padding for weakly-superrised saliency prediction in 360 videos. In CVPR, 2018.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In NeurIPS, 2023.
Daniel Courville and Ambisonic Studio. Procédés et systèmes d' d'enregistrement et de reproduction sonores en trois dimensions. Université du Québec à Montréal, 1994.
Tri Dao. Flashattention-2:: Faster attention with better parallelism and work partitioning. In ICLR, 2024.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. TMLR, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Ruohan Gao and Kristen Grauman. 2.5d visual sound. In CVPR, 2019.
11

---

Published conference paper at ICLR 2025
Rishabh Garg, Ruohan Gao, and Kristen Grauman. Visually-guided audio spatialization in video with geometry-aware multi-task learning. IJCV, 2023.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron Weiss, and Kevin Wilson. Can architectures for large-scale audio classification. In ICASSP. 2017.
Mojtaba Heydari, Mehrez Souden, Brumo Conejo, and Joshua Atkins. Immersediffusion: generative spatial audio latent diffusion model. In ICASSP, 2025.
Amit Hirway, Yuansong Qiao, and Niall Murray. Spatial audio 360 videos: does influence visual attention? In ACM MMSys, 2022.
Amit Hirway, Yuansong Qiao, and Niall Murray, Evaluating visual attention and qoe for 360 videos with non-spatial and spatial audio. In MMSys, 2024.
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv:2207.12590 2022.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv:2210.02300. 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In NeurIPS, 2022b.
Jukka Holm, Kaisa Väänänen, and Anas Battah. User experience of stereo and spatial audio in 360 live music videos. In AcademicMindtrek, 2020.
Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. In BMVC, 2021.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, 2025.
Khaled Koutini, Jan Schlüter, Hamid Lighal-zadeh, and Gerhard Widmer, Efficient training of audio transformers with patchout. In Interspeech, 2022.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In ICLR, 2023.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. In NeurIPS, 2024.
Saksham Singh Kushwaha, Jianbo Ma, Mark RP Thomas, Yapeng Tian, and Avery Bruni. Diff-sage: End-to-end spatial audio generation using diffusion models. In ICASSP, 2025.
Yeonghyeon Lee, Inmo Yeon, Juhan Nam, and Joon Son Chung. Voiceldm: Text-to-speech with environmental context. In ICASSP, 2024.
Xingda Li, Fan Zhuo, Dan Luo, Jun Chen, Shiyin Kang, Zhiyong Wu, Tao Jiang, Yang Li, Han Fang. and Yahui Zhou. Generating stereophonic music with single-stage language models. In ICASSP, 2024a.
Zhaojian Li, Bin Zhao, and Yuan Yuan. Cyclic learning for binaural audio generation and localization. In CVPR, 2024b.
12

---

Published conference paper at ICLR 2025
Wootaek Lim and Juhan Nam. Enhancing spatial audio generation with source separation and channel panning loss. ICASSP, 2024.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In ICML, 2023.
Miao Liu, Jing Wang, Xinyuan Qian, and Xiang Xie. Visually guided binaural audio generation with cross-modal consistency. In ICASSP, 2024.
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audie synthesis with latent diffusion models. In NeurlPS, 2023.
Xinhao Mei, Varun Nagaraja, Gael Le Lan, Zhaoheng Ni, Emni Chang, Yangyang Shi, and Vikas Chandra. Foleygen: Visually-guided audio generation. arXiv:2309. 10537, 2023.
Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), NeurIPS, 2018.
Pedro Morgado, Yi Li, and Nuno Nvasconcelos. Learning representations from audio-visual spatial alignment. In NeurIPS, 2020.
Huyen Nguyen and Madeline Willson. Spatial audio in youtube vr videos and its impacts on audience engagement. In 13DA, 2023.
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William Freeman. Visually indicated sounds. CVPR, 2016.
Santiago Pascual, Chunghsin Yeh, Ioannis Tsiamas. and Joan Serrà. Masked generative video-to-audio transformers with enhanced synchronicity. In ECCV, 2024.
Sandra Poeschl, Konstantin Wall, and Nicola Doering. Integration of spatial sound in immersive virtual environments an experimental study on effects of spatial sound on presence. In IEEE VR, 2013.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aakanksha Rana, Cagri Ozcinar, and Aljosa Smolic. Towards generating ambisonics using audio- visual cue for virtual reality. In ICASSP. 2019.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq- In NeurlPS, 2019.
Dominik Roblek, Kevin Kilgour, Matt Sharifi, and Mauricio Zuluaga. Fr) 'echet audio distance: reference-free metric for evaluating music enhancement algorithms. In Interspeech, 2019.
Roy Sheffer and Yossi Adi. hear your true colors: Image guided audio generation. In ICASSP. 2023.
Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annota- tions of sound events. In NeurlPS, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
13

---

Published conference paper at ICLR 2025
Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Semantic object prediction and spatial sound super-resolution with binaural sounds. In ECCV, 2020.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, al. Neural codec language models are zero-shot text speech synthesizers. arXiv:2301.02/11, 2023.
Heng Wang. Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In AAAI, 2024.
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan- Wu, Xuanjun Chen, Yu-Chi Pai, -suan Wang, Kai-We Chang, Alexander H Liu, and Hung-yi Lee. Codec-superb: An in-depth analysis of sound codec models. arXiv: 13071, 2024.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kong-Kiipattrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP, 2023.
Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In CVPR, 2021.
Jinhyeok Yang, Junhyeok Lee, Hyeong-Seok Choi, Seunghoon Ji, Hyeongju Kim, and Juheon Lee. Dualspeech: Enhancing speaker-fidelity and text-intelligibility through dual classifier-free guidance. In Interspeech, 2024.
Heeseung Yun, Sehun Lee, and Gunhee Kim. Panoramic vision transformer for saliency detection in 360o videos. In ECCV, 2022.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE TASLP, 2021.
Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In ECCV, 2020.
Yipin Zhou, Zhaowen Wang, Chen Fang. Trung Bui, and Tamara Berg. Visual to sound: Generating natural sound for videos in the wild. In CVPR, 2018.
Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non- con-autoregressive transformer. ICLR, 2024.
Franz Zotter and Matthias Frank. Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforcement, and virtual reality. Springer Nature, 2019.
14

---

Published conference paper at ICLR 2025
Table 4: Ablation on Classifier-Free Guidance. For Dual wi & w2, wi denotes the guidance scale for the directional guidance, while w2 denotes the guidance scale for the visual guidance.
Semantic Metrics
Spatial Metrics
Guidance Scheme
CC+
AUC+
FADaa
KLD
FAD
All
pps
Sips
All
1sss
5íps
Directional Visual
3.86
1.71
4.20
0.635
0.584
0.531
0.846
0.819
0.790
Directional only
5.56
2.01
4.76
0.721
0.671
0.624
0.890
0.864
0.839
Visual Only
3,85
1.71
4.14
0.488
0,446
0.406
0.757
0.736
0.716
Dual 2.5 2.5
3,93
1.72
4.22
0.549
0.505
0.459
0.795
0.772
0.748
Dual 7.5 & 2.5
4.32
1.76
4.59
0,596
0.540
0.487
0.829
0.798
0.769
A IMPLEMENTATION DETAILS OF VISAGE
We utilize pretrained CLIP model based on ViT-B/32 (Dosovitskiy et al. 2021 where output dimension is 512 and dp 768. We obtain CLIP embeddings at 4 frames per second (FPS). For the patchwise energy map, the window sizes N and T are both set to 1. For computing the compute energy map E from the scores, temperature of 0.1 is used for the softmax, and top-p filtering is applied after averaging the probabilities, with a top-p threshold of 0.7. For the DAC, we adopted the 44100 Hz variant, which employs N 9 codebooks per audio channel, each with size of 1024, producing 86 codes per second audio.
The transformer architecture have hidden dimension of d, 1024 with 16 attention heads. It consistes of 6 layers in the encoder and 12 layers in the decoder. Since the sequence length of visual features is much shorter than that of audio features and adjacent CLIP embeddings are often highly similar, we halve the number of layers in the encoder compared to that of the decoder. Both the energy map E and unit vector are processed through MLP layers composed of two linear layers with GELU activation in between, using a hidden dimension of 1024. Overall, ViSAGe have 360M trainable parameters.
B ABLATION ON CLASSIFIER-FREB GUIDANCE
We conduct an ablation study on different classifier-free guidance schemes. We compare our approach alternative methods that modify the second term in Eq. 5 including guiding only the visual condition, as in previous video-to-audio generation works (Shetfer & Adi 2023 Mei et al.
2023), and guiding both conditions separately using dual classifier-free guidance (Lee al. 2024]]
Yang et al. 2024). which has been used to improve different aspects of text-to- -sxeech-speech generation. Dual classifier-fres guidance assumes that the two input conditions are independent, allowing each condition to be guided separately. However, in our case, visual and directional conditions are closely related, and guiding them separately may not capture their combined effect on spatial audio generation effectively.
For reorganized DAC code matrix C₁ E +21x let C = (C.)., e and V and D respectively denotes the video frames and the camera direction. Each guidance can be formulated as follows. First, for visual only guidance, we sample DAC codes from
log Po Ch.-1: D) + wlog (C) V, - log (C' 1 Char1, 8, Ø)), (6)
where w denotes the visual guidance scale and denotes the probability parametrized by the ViSAGe model.
Lastly, dual guidance can be formulated as sampling DAC codes from
log ( C, D) + wi (log Ci 0, D) - PC &,,11,, Ø)) +wo Ps (C Ci.t-1, - log po C/.r-1,,, 2, 2)),
(7)
where W and W2 respectively denote the guidance scale for directional guidance and visual guidance.
The results in Table 4 show that omitting directional guidance degrades spatial accuracy, while the absence of visual guidance reduces the semantic quality of the generated audio. Our approach, which
15

---

Published conference paper at ICLR 2025
Table 5: Results of the subjective test. "Win" represents the percentage of participants who preferred ViSAGe, "Lose" represents those who preferred the baseline, and "Tie" indicates the percentage of participants with no preference.
Criteria
Win
Lose
Tie
Win
Lose
Tie
(a)
SpecVOGAN
(b)
Diff-Foley
Natural
43.33
25.56
31.11
44.44
14.44
41.11
Relevent
50.00
27.78
22.22
40.00
23.33
36.67
Spatial
52.22
31.11
16.67
42.22
30.00
27.78
Overall
50.00
23.33
26.67
44.44
16.67
38.89
jointly guides both conditions, outperforms guiding them separately, indicating that the two conditions are closely related. Furthermore, increasing the guidance scale for the directional condition improves the spatial aspect but worsens the semantic quality, and still underperforms compared guiding both conditions jointly. This provides additional evidence that the two conditions are interdependent and better guided together rather than guided separately.
C TRAINING AND EVALUATION DETAILS
Training Loss. For the reorganized DAC code matrix C² (1NN(22c+11 given input video frames V and a camera direction D, the training loss is formulated as
2h,+1 AN
(2L + +1 x 4N Σ Σ log Pe ((C)),,
( (V,
(8)
where denotes the probability parametrized by the ViSAGe model.
Training Hyperparameters. For pretraining on VGGSound, we use constant learning rate of le-4 with 4000 warmup steps. For finetuning YT-Ambigen, apply a constant learning rate of le-4 without warmup. When training from scratch on YT-Ambigen, we use a constant learning rate of 2e-4 with 4000 warmup steps. The AdamW optimizer is adopted with weight decay of 1e-2 and a gradient clipping norm of 1.0. Training is conducted on NVIDIA A6000 or A40 GPUs with batch size of 64. We also utilize bfloat16 precision and FlashAttention-2 the training process.
Dao 2024 to accelerate
pretrained model based ResNet-50 (He et al. 2016) features at 21.5 fps, with a total of 310M trainable parameters. VQeAN generates audio 22050 Hz. For Diff-Foley, adopt the large variant of the pretrained model, which has 860M trainable parameters and generates audio at 16000 Hz.
Evaluation Details. For SpecVQGAN,
For all models, evaluate the checkpoint with the lowest validation loss. To compute the FAD, use features from the VGGish network (Hershey et al. (2017) pretrained on AudioSet (Gemmeke et al.) 2017) classification. For KLD, we follow previous works (Sheffer & Adi 2023 Mei al.) 2023) and use the PaSST model (Koutini et al. 2022) pretrained on AudioSet classification, to calculate class distributions. We use the audioldm eval library (Liu et al.) (2023) to compute all metrics. Since these pretrained audio classifiers accept 16000 Hz audio as input, we resampled all generated ambisonics 16000 Hz before evaluation.
D ILLUSTRAION OF CODE GENERATION PATTERNS
The code generation patterns compared in the ablation study in Section 6.2 are illustrated in Figure
E SUBJECTIVE TEST RESULTS
We conducted human preference analysis with two-sample hypothesis testing of generated audio with respect to four subjective criteria:
16

---

Published conference paper at ICLR 2025
(Wp.WAr J. W channel codebooks
W
(Sp Sj): Spatial codebooks
W
b
(p: Primary/ r. Residual)
Sp
f
C n-th codebook
S,
b
t. DAC Code of timestep
Padding
(a) Proposed
Cn.:
E
f
W
2
fs
W
b
[s
C,+,
12
1g
W
4
12
W
f
Cnt 1g
Sp
12
Sp
Crr, h
12
S,
f2
S
4
(c) Residual
(d) Spatial
(b) Sequential Delay
Dependency Only
Dependency Only
Illustration different code generation patterns from Section 6.2 For (a), (c), and (d), each block represents all residual codes belonging to the corresponding codebook group. In (b), each block represents a single code from the codebook C,..-
Figure 4:
Naturalness: Which audio sounds more natural?
Relevance: Which audio is more closely related to objects and surroundings in the video?
Spatiality: After observing different viewpoints of a 360 video by rotating, which audio better captures the spatial effects perceived both ears?
Overall preference: Which audio do you prefer overall?
Due to the characteristics of 360 videos and spatial audio, we recruited 12 participants person instead of crowdsourcing (e.g., MTurk). Each annotator evaluated an average of 15 videos out of 30 randomly selected samples from the test split. The results are summarized Table showing that samples generally preferred over the prior arts across all four criteria. It is worth noting that the gap is particularly large for the spatiality criterion.
F ADDITIONAL QUALITATIVE EXAMPLES
Comparsion to two-stage baselines. Linear spectrograms generated by ViSAGe and two-stage approaches based on Diff-Foley (Luo et al., 2023) are shown Figure 5 ViSAGe consistently produces semantically and spatially coherent ambisonies channels. While Diff-Foley generates semantically plausible spectrograms, the spatial channels produced through audio spatialization exhibit limited fidelity and contain several artifacts. We attribute this two factors: (1) the commonly approach based on complex masking tends to introduce artifacts, and (2) the generated audio has limited fidelity compared to ground-truth audio. This worsens the semantic quality of the generated spatial channels by introducing gap between the training and inference data in spatialization models, highlighting the advantage of our end-to-end approach.
used mix-and-separee
Ambisonics Generation for Dynamic Visual Scenes. Figure demonstrates the patchwise energy map of the video frames, along with the audio energy map of the first-order ambisonics generated by ViSAGe. The proposed patchwise energy map effectively highlights dynamic changes in visual scenes. When objects move dynamically within a scene or when specific regions undergo temporal changes, these areas represented by high energy values due to significant differences with their spatially and temporally neighboring patches.
Role of Camera Direction. Figure 7 illustrates the effect of the camera direction parameter the ambisonics generation. The orientation from which the visual information captured significantly
17

---

Published conference paper at ICLR 2025
Diff- Diff-foley
W
Ambi Enc.
ViSAGe
Difi-fold Diff-foley
Audio Spatialization
Figure 5: Qualitative examples of generated audios from ViSAGe and two-stage approaches. Blue boxes shows that while ViSAGe captures the acoustic characteristics of surroundings (indicated by less black area in spectrogram) in Y-channel, while Ambi Enc, generates almost identical spectrogram for spatial channels. Green box show that spectrogram of spatial channels generated by Audio Spatialization may introduce artifacts.
18

---

as Pubbssiad
condcrrnvee
IC
2003
paper di
E1
12
1-3
E22
AALASI
SLASS
ALLEC
P44AS
NE
1F2
L-3
E-4
1E3
Ee4
E-2
E-3
scene esppuccei at famne 4 Figure 6: Daalilatii
example of Badewwiii
changing
euergy
eppcrrde and map
audio
rapddty in
per secont
(b)
Figuue 7: Qualitaioo ssampee of the camera directio aataneerr (a) Camera
Camera direction is sei to front:
(0. 0). (b) Cameer firecttonn is set to monttaeii ( 0).
19

---

Published conference paper at ICLR 2025
impacts the output ambisonics. As described in Sec 3.2 ambisonics capture the full three-dimensional sound field and are commonly used with panoramic videos. However, when paired with field-of-view (FoV) video, ambiguity arises regarding the visual scene's placement within the three-dimensional space. While treating the FoV scene as a frontal view simplifies processing, it compromises the immersiveness and controllability of ambisonics generation since all sounds appear to originate from directly front the listener. To address this, we introduce a camera direction parameter as an additional condition that specifies the visual scene's 's position within the three- diceeniimenio sound field, enabling proper audio-visual spatial alignment. In practice, the camera direction parameter guides the directivity of spatial audio generation. For instance, in orchestra recording. if the camera faces front, the audio originates primarily from the front. If the camera turns left, the audio follows, enhancing spatial realism.
More qualitative examples and demo videos are available at http:///yycoonmmm...9.iib.b...o.
G DETAILS OF YT-AMBIGEN
We analyze the content and distribution of the proposed YT-Ambigen dataset. To examine the audio distribution, we classify each audio clip using the PaSST (Koutini et al. 2022) model trained on AudioSet Gemmeke BudioSet/Gemmmee et al. (2017). For video content distribution, we employ an FPN (Lin et al. (0177) to identify the most salient object in each video. Additionally, we utilize CoTracker (Karaev et 2025) to capture object motion and trajectories over time.
In Figure 8 (a), our audio distribution similar that of AudioSet, where YT-Ambigen covers 314 out of 527 classes in AudioSet, accounting for 97.91% of the entire AudioSet videos. Figure (b-d) summarizes the semantic, spatial, and temporal distributions of the most salient object per video. These objects cover 79 out of 80 classes in COCO and are located in diverse positions within the field of view. These objects would often move around significantly during five-second segments, creating more challenging scenarios for vidco-to to--ttosommiiiiccooo generation.
20

---

Published conference paper at ICLR 2025
10
10
10
10
Spezch
Ocean
Waterfall
Stream
Anima
Insoch
Bost, Watte dole
imicrophone
Camalic music
Beep, bleep
Chole
Silence
Wind
Trein
Applluoe
Environmental noiee
capells
Rain Freg
Cello
Rowbool, canoe, Outsiifo nual or natura
Wak foorsteps
Vialin, Indil
Cake
Raindrog
Drum kit
Hellcopper
Rumble
Crumpling cinkking
White noite noise
Trumpes
Pinoo
rnstumment
Saaophhne
Thurder
Harmonice
Waver,
(a) Audio class distribution
10
(c) Object detection
10
Nytop
bosr
25e
col
(b) Object class distribution
(d) Object tracking
Figure Distribution statistics YT-Ambigen. (a) The top-50 AudioSet labels distribution predicted with PaSST. (b) The top-50 COCO object class distribution of the most salient object per video with FPN. (c) The center coordinates of each salient object's 's bounding box. (d) The tracking of center pixels per video predicted with CoTracker (randomly selected 1K samples for visibility).
21


Reviews:
[{"review_id": "XR8qlZIVxk", "reviewer": "Reviewer_Xpfx", "summary": "The paper titled \"ViSAGe: Video-to-Spatial Audio Generation\" introduces a novel framework for generating first-order ambisonics, a spatial audio format, directly from silent video clips. The authors address the challenge of enhancing the immersiveness of audio-visual experiences without the need for complex recording systems or specialized expertise. They present YT-Ambigen, a dataset of 102K YouTube video clips paired with first-order ambisonics, and propose new evaluation metrics based on audio energy maps and saliency metrics. The core of their work is the Video-to-Spatial Audio Generation (ViSAGe) framework, which leverages CLIP visual features and autoregressive neural audio codec modeling to generate spatial audio that is both semantically rich and spatially coherent. The framework outperforms two-stage approaches and demonstrates the ability to adapt to dynamic visual contexts, showing potential for applications in immersive media production.", "questions": "The audio energy map shown in Figure 3 seems to be highly correlated with the location of the sound source, which can be related to the sound source localization task. In other words, can the relevant design ideas under the sound source localization task provide some guidance here? More specifically, can the cross modal contrastive learning strategy [1] commonly used in sound source localization tasks be applied here to impose some additional constraints?\n\n[1] Chen H, Xie W, Afouras T, et al. Localizing visual sounds the hard way[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 16867-16876.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a groundbreaking approach to directly generate spatial audio from video, addressing a previously unsolved problem and offering a significant advancement in the field of immersive media.\n2. The ViSAGe framework is an end-to-end solution that integrates neural audio codecs with visual features, which is a novel combination in the context of audio generation from video.\n3. The paper is well-structured, with a clear problem statement, including the introduction of a new dataset and evaluation metrics, which are crucial for the field.\n4. The creation of YT-Ambigen dataset and new evaluation metrics shows a comprehensive approach to both generating and validating the spatial audio.", "weaknesses": "As shown in Figure 2, the framework designed in this paper uses many different modules. Therefore, the computational complexity (model parameters) and running time (inference time) of the overall framework need to be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AC8NYfTQeR", "reviewer": "Reviewer_T5M6", "summary": "This paper presents a novel task: generating spatial audio from silent video. Specifically, given a silent video and the camera direction, the proposed model, ViSAGe, leverages CLIP features, patchwise energy maps, and neural audio codes, incorporating a code generation scheme to simultaneously produce multiple spatial channels of audio in the first-order ambisonic format. For this task, the authors introduce a new dataset, YT-Ambigen, which consists of YouTube videos paired with first-order ambisonics. Compared to two-stage models, which first generate mono-channel audio and then perform audio spatialization, the proposed ViSAGe outperforms in overall quantitative metrics.", "questions": "- Two numbers are highlighted in bold in Table 3: Ablation on Model Components.\n- Please check that all figure and table references are correctly cited, e.g., L476 references Figure D.\n- Could the authors clearly describe how the 9 codes per timestep in each audio channel are generated? The figures seem to illustrate the process for generating a single code, while the neural codec contains 9 RVQ codes per timestep.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written and well-presented.\n- The proposed task of generating spatial audio from silent video is interesting and takes one step further of existing works that tackle this task separately.\n- The dataset curation pipeline effectively addresses the limitations of existing datasets, and the dataset would make contribution to the community.", "weaknesses": "- Lack of moving object samples or objects that are not centered.\n    - When listening to the synthesized audio while watching the video, most content appears centered, which makes the synthesis task relatively simple and appears to be similar to the mono audio generation task.\n    - To fully demonstrate the effectiveness of the proposed method and task, more qualitative examples are needed that show results in challenging scenarios (e.g., objects moving from left to right, objects not centered, or visual events requiring time-synced audio synthesis).\n- This lack raises questions about dataset quality.\n    - Since the dataset seems to be collected automatically, is the test set clean or challenging enough to serve as a reliable benchmark? Wouldn’t human annotation or verification be necessary to validate its quality?\n    - Furthermore, what distribution does this dataset cover? What types of events or objects appear in it? If the task involves not only audio spatialization but also requires semantic information, then details on the dataset (e.g., statistics, categories) should be provided.\n- Clearer visualization and explanation in qualitative results would enhance the work.\n    - The generated spectrograms in Fig. 3(a) appear different from the ground truth. If the authors do not highlight the specific areas readers should focus on, important details may be overlooked.\n    - Similarly, in Fig. 3(b), overlapping the heatmap with the original video would improve clarity. While the predicted heatmap seems better than the baseline model, it still does not fully align with the real visual events.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzUIGA14Wc", "reviewer": "Reviewer_NiUt", "summary": "- The paper introduces a new problem of generating spatial audio for silent videos.\n- The problem is formalized as:  Given silent video and direction of camera -> generate First Order Ambisonics (FOA)\n- Due to lack of suitable datasets, the authors collect YT-AmbiGen from Youtube for this task\n- They use discrete audio representations for encoding FOA and autoregressive model for generation\n- New baselines and evaluation metrics are proposed to compare and evaluate their approach\n- A comprehensive comparison with baselines, along with ablation study, show the effectiveness of their proposed approach", "questions": "- Could the authors include subjective test results evaluating metrics such as audio quality and relevance to the specified direction? \n\n- Would it be possible to create a subset of clean, single-source static sounds as a benchmark and demo set? This would enable evaluation using metrics like Direction of Arrival (DoA) and provide clearer, more focused demo examples.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is new, interesting, and important for the research community\n- The authors introduce a new dataset for this new task, setting a benchmark for future spatial audio generation research.\n- Suitable baselines and metrics are proposed to compare on this new task\n- Carefully designed components are incorporated, for eg. FOA encoding, sequence of its generation, rotation augmentation, patchwise energy maps", "weaknesses": "Major:\n\nMissing Subjective tests:\n- The paper lacks subjective evaluations; studies assessing quality, and directionality ( or localization accuracy) should be included. Authors should compare their approach with baselines on metrics like mean opinion score (or other subjective metrics).\n \nDemo examples\n- While the demo examples appear semantically good, the sounds are often too diffuse, making it challenging to precisely localize the direction of the audio.\n- Including some static sources with smaller, more focused sound-generation areas can help in better experiencing and analyzing the sound source direction (subjectively). \n\nMinor:\n- Missing relevant references:\n\nSome recent work on spatial audio generation should be referenced. These methods also generate spatial audio (FOA) given some conditions (eg. direction of arrival, and sound source category).\n\n[1] Heydari et. al, \"Immersediffusion: A generative spatial audio latent diffusion model\"\n\n[2] Kushwaha et. al, \"Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0dc1MB4f2", "reviewer": "Reviewer_XV3v", "summary": "ViSAGe is a framework designed to generate spatial audio, specifically first-order ambisonics, directly from silent videos, enabling immersive audio experiences without complex equipment. Utilizing the YT-Ambigen dataset of over 102,000 video clips with ambisonic audio, ViSAGe integrates CLIP visual features and a neural audio codec for synchronized, viewpoint-adaptive sound. It outperforms traditional two-stage methods by producing temporally aligned, spatially coherent audio, evaluated through innovative metrics based on audio energy and saliency.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written\n2. The approach of generating spatial audio from FoV holds considerable value for practical applications. Open-sourcing the proposed dataset would provide a valuable resource to the community.\n3. The design methodology is reasonable and effective.", "weaknesses": "1. There appears to be no explanation of the camera orientation parameter, which leaves me unclear on how it enhances spatial perception.\n2. The method assumes that the CLIP visual representation lacks spatial information. Could replacing it with a visual representation that has stronger local perception, such as DINOv2, improve the quality of spatial audio generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SZN0m2a3O9", "reviewer": "Reviewer_vsEj", "summary": "The paper introduces ViSAGe, a novel framework for generating first-order ambisonics (a spatial audio format) directly from silent videos. This is significant for enhancing the immersiveness of audio-visual experiences without the need for complex recording systems or specialized expertise.", "questions": "Could the authors elaborate on the diversity of the YT-Ambigen dataset in terms of different acoustic environments and video content types? How does this diversity compare to real-world scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to generating spatial audio directly from silent videos, aiming to enhance immersion without complex recording setups. A key contribution is the creation of the YT-Ambigen dataset, which includes 102K video clips paired with spatial audio, providing valuable resources for training and evaluation. The authors propose innovative evaluation metrics that utilize audio energy maps and visual saliency to assess the spatial quality of the generated audio, offering a deeper understanding of audio-visual coherence. Their end-to-end framework, ViSAGe, integrates CLIP visual features with neural audio codecs, avoiding issues associated with traditional two-stage approaches. Lastly, this research has significant implications for media production in film, virtual reality, and augmented reality, potentially revolutionizing audio generation for visual content.", "weaknesses": "The current approach may not fully capture the dynamic changes in audio that correspond to rapidly changing visual scenes. The generation of spatial audio is based on neural networks, which may not always adhere to the physical principles governing sound propagation and perception.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)