OmniAudio: Generating Spatial Audio from 360-Degree Video
Authors: Huadai Liu
Tianyi Luo
Kaicheng Luo
Qikai Jiang
Peiwen Sun
Jialei Wang
Rongjie Huang
Qian Chen
Wen Wang
Xiangtai Li
ShiLiang Zhang
Zhijie Yan
Zhou Zhao
Wei Xue
Venue: ICML 2025 poster (2025)
Keywords: video-to-audio generation
360-degree video-to-spatial audio generation
spatial audio generation
Primary area: applications->language_speech_and_dialog
TL;DR: We introduce OmniAudio, the first 360-degree video-to-spatial audio generation model.

Abstract: Traditional video-to-audio generation techniques primarily focus on perspective video and non-spatial audio, often missing the spatial cues necessary for accurately representing sound sources in 3D environments. To address this limitation, we introduce a novel task, \textbf{360V2SA}, to generate spatial audio from 360-degree videos, specifically producing First-order Ambisonics (FOA) audio - a standard format for representing 3D spatial audio that captures sound directionality and enables realistic 3D audio reproduction. We first create \textbf{Sphere360}, a novel dataset tailored for this task that is curated from real-world data. We also design an efficient semi-automated pipeline for collecting and cleaning paired video-audio data. To generate spatial audio from 360-degree video, we propose a novel framework \textbf{OmniAudio}, which leverages self-supervised pre-training using both spatial audio data (in FOA format) and large-scale non-spatial data. Furthermore, OmniAudio features a dual-branch framework that utilizes both panoramic and perspective video inputs to capture comprehensive local and global information from 360-degree videos. Experimental results demonstrate that OmniAudio achieves state-of-the-art performance across both objective and subjective metrics on Sphere360. Code and datasets are available at~\href{https://github.com/liuhuadai/OmniAudio}{\texttt{github.com/liuhuadai/OmniAudio}}. The project website is available at \href{https://OmniAudio-360V2SA.github.io}{\texttt{OmniAudio-360V2SA.github.io}}.

OmniAudio: Generating Spatial Audio from 360-Degree Video
Huadai Liu
Tianyi Luo
Kaicheng Luo Qikai Jiang
Peiwen Sun' Jialei Wang Rongjee Huang
Qian Chen Wen Wang?
Xiangtai Li
Shiliang Zhang'
Zhijie Yan Zhou Zhao Wei Xue
Abstract
1. Introduction
The rapid advancement of virtual reality and immersive technologies has created urgent need for realistic audio- visual experiences. Although many video-to-audio gen- eration methods (Iashin & Rahtu, 2021; Luo et al., 2024; Zhang et al., 2024; Mei et al., 2024) are proposed, they face two critical limitations. First, they typically generate non-spatial audio (mono/stereo), which lacks the directional information essential for immersive experiences. Second, they operate on perspective videos with limited field-of- view, missing crucial visual context for sound generation, as illustrated in Figure 1.
Traditional video-to-audio generation techniques primarily focus on perspective video and non- spatial audio, often missing the spatial cues necessary for accurately representing sound sources in 3D environments. To address this limitation, we introduce a novel task, 360V2SA, to generate spatial audio from 360-degree videos, specifically producing First-order Ambisonics (FOA) audio standard format for representing 3D spatial audio that captures sound directionality and enables realistic 3D audio reproduction. We first create Sphere360, novel dataset tailored for this task that is curated from real-world data. We also design an efficient semi-automated pipeline for collecting and cleaning paired video-audio data. To generate spatial audio from 360-degree video. we propose novel framework OmniAudio, which leverages self-supervised pre-training using both spatial audio data (in FOA format) and large-scale non-spatial data. Furthermore, OmniAudio features a dual-branch framework that utilizes both panoramic and per- spective video inputs to capture comprehensive local and global information from 360-degree videos. Experimental results demonstrate that OmniAudio achieves state-of-the-art performance across both objective and subjective metrics on Sphere360. Code and datasets are available at github. com/ 1
Spatial audio, particularly in First-order Ambisonics (FOA) format (Zotter & Frank, 2019), offers a solution to the first limitation by preserving cent works have begun exploring spatial audio generation: Anonymous (2024) generate FOA audio from video per- spectives using autoregressive models, Heydari al. (2024) synthesize FOA audio from text and room parameters, and Kushwaha et al. (2024) condition on sound categories and source locations. However, these approaches still rely on fixed camera perspectives, inheriting the second limitation of restricted visual context.
sound positioning. Re-
To overcome both limitations, we introduce 360-degree video to spatial-audio generation (360V2SA), novel task that generates FOA audio directly from 360-degree (panoramic) videos. This task is particularly timely given the increasing accessibility of 360-degree cameras. Panoramic videos offer significant advantages over tradi- tional perspective videos they capture a complete 360- degree field of view, allowing simultaneous observation of all sound-emitting objects and their spatial relationships, regardless of direction. By utilizing this comprehensive spherical visual coverage, 360V2SA enables the the genera- tion of spatially-aware audio that naturally aligns with the visual content, without requiring additional control parame- ters like camera angles.
The project website available http:/..githu......... github.
'Hong Kong University Science and Technology, Hong Kong, China Tongyi Lab, Alibaba Group, Hangzhou, China 'Zhejiang University. Hangzhou. China 'FAIR, Meta, USA 'Nayyang Technological University. Singapore. Cor- respondence Wei Xue <weixue@ust.hk>. Zhou Zhao Giaaozhou@zju.cdu..o
The 360V2SA task presents three major challenges: (1) the scarcity of paired 360-degree video and spatial audio data, (2) the need for precise audio-visual synchronization across the entire sphere, and (3) the complexity of generating high- fidelity spatial audio.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
The trinn enlers.
The trint pusses.
The train deparis.
Audio Rotation
Frontal Perspective
After Rotation
Audio)
After Rotation (FOA Audio)
Even wils noustonn the relaiive posiizo of the cound icerce changrs, yet the priseived effict remaiin unclunged.
After rotatioe, relative prsition shend oanc chiffs. leading to correpanding change In the preceled offect.
Panoramic View
(a) Comparison of panoramic video and perspective video.
(b) Comparison of stereo audio and FOA audio under head rotation.
Figure 1. (a) shows the scene of a moving train that appears and gradually disappears in panoramic view without being visible in the frontal perspective. (b) compares the audio localization before and after head rotation, illustrating how stereo audio fails to maintain sound localization while spatial audio (in FOA format) retains accurate positioning.
We propose an end-to-end framework OmniAudio for 360V2SA. To address the data scarcity challenge, we con- struct Sphere360, the first large-scale dataset for 360V2SA, containing 103,000 real-world video clips, each with 10- second duration, spanning 288 audio events. We develop semi-automated pipeline for dataset construction. Moreover, we also propose a novel training strategy in OmniAudio that leverages existing non-spatial audio datasets through a spa- tial autoencoder and masked token prediction pre- gre-training, in self-supervised coarse-to-fine manner. By masking and reconstructing portions of audio tokens, the model learns general audio patterns that transfer to spatial audio gener- ation via the autoencoder, effectively bridging the domain gap. For precise visialviuual synchronization across the entire sphere and generating high-fidelity spatial audio, we design dual-branch architecture in OmniAudio that com- bines latent flow matching with both local and global video processing. This architecture captures fine-grat fine-grained object details while maintaining awareness of the complete spheri- cal context, enabling precise audio-visual synchronization through a two-stage optimization process.
2. Related Work
Video-to-Audio Generation. This direction (Zhang et al., 2024; Wang et al., 2024a; Xu et al., 2024c; Tian et al., 2024) focuses on synthesizing audio that aligns seamlessly with the visual content of a video clip. Some approaches, such as SpecVQGAN (lashin & Rahtu, 2021), FoleyGen (Mei et al., 2024), and V-AURA (Viertola et al., 2024), leverage au- toregressive techniques to produce audio from silent video. Meanwhile, other methods (Luo et al., 2024; Cheng et al., 2024b; Xu et al., 2024b) adopt diffusion models (Song et al., 2020; Xing et al., 2024: Liu et al., 2024a) or flow match- ing (Lipman et al., 2022; Vyas et al., 2023; Wang al., 2024b) generative models. Diff-Foley (Luo et al., 2024) employs an audio-visual contrastive feature and latent dif- fusion to predict spectrogram latent. Similarly, MovieGen Audio (Polyak et al., 2024) uses flow matching conditioned on multi-mode inputs, including videos and texts. Despite these advancements, a significant gap remains in the re- search focused on generating spatial audio from 360-degree video, essential component for crafting genuinely im- mersive experiences. We introduce OmniAudio, which, to the best of our knowledge, represents the first approach in spatial audio generation from 360-degree video.
Our contributions can be summarized as follows:
We introduce 360V2SA, novel task addressing funda- mental limitations in traditional video-to-audio generation, and propose OmniAudio, an end-to-end solution using latent flow matching.
Spatial Audio Generation. Existing spatial audio gen- eration methods (Xu et al., 2021; Leng et al., 2022; Liu et al., 2023c; Gao & Grauman, 2019b; Kushwaha al., 2024; Anonymous, 2024) predominantly focus on produc- ing binaural (Sun et al., 2024; Yoshida et al., 2023; Garg et al., 2021) or First-order Ambisonics (FOA) audio (Kush- waha et al., 2024; Dagli et al., 2024) from fixed perspec- tive inputs, such as mono audio, visual features, text, and
We develop an effective training strategy combining coarse-to-fine pre-training and dual-branch video encod- ing for spatial-aware generation.
We create and release Sphere360, comprehensive dataset of 103,000 video clips with spatial audio, along with its semi-automate construction pipeline.
Binaural audio captures sound using two microphones to simu- late human car positioning for 3D effect when using headphones, while spatial audio involves a broader range techniques that create 3D sound in dynamic environments, including with head tracking or multi-speaker setups.
We establish Sphere360-Beae -BnhhhhBBchh as standardized evalua- tion framework where our AmniAudio achieves of the-art performance across both objective and subjective metrics.
2

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
source locations. For example, 2.5D Visual Sound (Gao & Grauman, 2019b) transforms monoaural audio into binau- ral sound using convolutional neural networks, while Diff- SAGe (Kushwaha et al., 2024) and ViSAGe (Anonymous, 2024) generate FOA audio by incorporating sound cate- gories, source locations, and camera parameters. However, these approaches typically assume a static camera perspec- tive, limiting their applicability in dynamic environments where the field of view can change in real-time. To over- come these limitations, we propose framework for gener- ating FOA audio from 360-degree video, which inherently supports dynamic and panoramic visual inputs. Our method leverages self-supervised flow matching pre-training and dual-branch video design to effectively synthesize spatial audio that adapts to the comprehensive and evolving visual context captured by 360-degree videos. This advancement enables more immersive and flexible audio-visual experi- ences in applications requiring real-time spatial audio ad- justments.
to Figure 8 for the distribution of different audio events.
Data Collection. Based on YT-360 (Morgado et al., 2020), we construct the dataset by crawling YouTube, leveraging its extensive collection of 360-degree videos and spatial audio content. The data collection process follows a systematic approach with several key steps. The initial search strategy involves using carefully formulated search keywords (e.g.. "sking spatial audio 360") to ensure class diversity and retrieve more 360-degree and FOA content. Technical filter- ing is applied to exclude videos that do not support either 360-degree visual content or FOA audio. Guided by target lists and technical filters, the crawling process proceeds in two stages: first, a channel-based approach where we iden- tify frequently appearing channels and collect videos on a per-channel basis, covering the majority of search results; second, video-based approach, which involves manually reviewing the remaining videos and applying final filters for dataset completion. Throughout the process, we maintain rigorous quality assurance through periodic re-evaluations of search results to refine keywords, as well as by sampling clips within target channels and videos to manually filter out content with unrealistic scenes or excessive post-production modifications. Detailed copyright and collection protocol in- formation is provided in Appendix A and B.1, respectively.
Flow Matching, Flow matching is used as the backbone for audio generation due to its superior generation perfor- mance. This framework (Lipman et al., 2022) trains a model to learn a vector field, enabling the generation of a desired probability path from noise to data. Unlike score-based models such as Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020), FM offers more stable and ro- bust traiming and enhanced performance. This generative model has proven effective in audio generation, as seen in applications like Audiobox (Vyas et al., 2023) and FlashAu- dio (Liu et al., 2024b). Moreover, SpeechFlow (Liu al.,, 2023a) has integrated self-supervised pre-training with FM to improve speech processing tasks, demonstrating its adapt- ability. In our work, we adapt FM for the novel task of spatial audio generation from 360-degree videos. By adopt- ing a coarse-to-fine self-supervised pre-training strategy, our approach first learns general audio patterns from large- scale non-spatin con-spatial audio data and then fine-tunes the model to capture spatial-specific characteristics. This enables the generation of high-quality First-order Ambisonics (FOA) audio that complements the immersive visual context of 360-degree videos.
Data Cleaning. While the initial collection focuses on gathering the 360-degree video and FOA audio, we fur- ther clean the dataset to remove stationary videos, silent audio, excessive speech, and videos with mismatched audio- processes
visual content from the dataset. are adopted. Stationary videos are identified by measuring the similarity between frames using the mean squared er- ror (MSE), and video segments containing more than 85% stationary frames are removed. Silent audio segments are identified using a 20ms sliding window to calculate the maximum decibels relative to full scale (dBFS), with clips containing over 90% silence being filtered out. To remove excessive speech content, we employ the Sense Large model? for speech detection. Videos containing more than 5 detected words are filtered out, though we retain those with minimal vocal content to preserve naturally occurring sounds. Finally, we ensure strong audio-visual correspon- dence using Imagebind (Girdhar et al., 2023) for alignment assessment, removing videos with similarity scores below 1. Detailed cleaning procedures are documented in Ap- pendix B.2.
3. The Sphere360 Dataset
Overview. The Sphere360 dataset comprises over 103,000 paired audio and 360-degree video clips sourced from YouTube, amounting to 288 hours. The dataset covers a wide range of real-world acoustic environments and noise conditions. We partition the dataset into training and test sets based on video IDs, to maintain the integrity of audio event distributions for evaluation and prevent data leakage. We implement comprehensive data filtering pipeline to ensure high-quality video and audio samples. Please refer
Comparison with Existing Datasets. As shown in Ta- ble 1, Sphere360 stands out as the largest 360-degree video dataset with FOA audio format, specifically tailored for 360- degree video-to-spatial audio generation. In comparison, YT-360, previously the largest dataset providing 360-degree
Phttps: / /t/gttttttttt com/ FunAudi SenseVoice
3

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
The model is trained by minimizing an objective function aimed at optimizing 0. Specifically, linear interpolation is defined as:
Table Comparison between Sphere360 and existing datasets. 360" represents panoramic videos. FOA refers to first-order am-
bisonies.
tz + (1 t)
Total Duration
V/A Type
Audio Generation
Dataset
#Clips
360
FOA
where 1i data points from the training distribution con- ditioned on C, and Zo is sampled from the standard normal distribution. The velocity field u(ze 21) at any inter- mediate point zy is given by:
VGGSound
200K
550b
x
X
(Chen al., 2020)
FairPlay
1.8K
5.2b
X
X
(Gao & Grauman, 2019a)
OAP
(Vaudevan al., 2020)
64K
15h
X
X
u( No, x1) = 21 200-
REC-STREET
123K
3.5h
(Morgado et al., 2018)
YT-ALL
The objective for conditional flow matching is:
3976K
11.3h
df
(Morgado et sl., 2018) 2018)
YT-Ambigen
102K
142h
X
[t,g((o), [llva C, u( Io, a 113)
(3)
(Anonymous, 2024)
STARRS23
0.2K
7.5h
where the expectation is over ((2g) (noise), g(21, C) (train- ing data), and t uniformly sampled from [0. 1].
(Shimada al., 2023)
YT-360
89K
246h
x
(Morgado et al., 2020)
Sphere360
103K
288h
4.2. Spatial Audio Representation
Variational Autoencoder (VAE) plays an essential role in audio processing by compressing audio signals into latent representations (Hsu et al., 2017: Caillon & Esling, 2021; Evans et al., 2024). Traditionally, audio VAEs have involved converting waveforms into mel-spectrograms before com- pressing them into latent spaces. However, recent advance- ments demonstrate that the incorporation of Snake activa- tions in the Descript Audio Codec architecture significantly enhances audio reconstruction quality at high compression ratios (Kumar et al., 2023), outperforming alternatives like EnCodec (Défossez et al., 2022). Building on these devel- opments, the Stable Audio framework (Evans et al., 2024) succeeded in directly compressing stereo waveforms into latent representations, marking an advance in audio VAEs. Despite advances, there is still a lack of VAES for four- channel FOA audio. The channels W. X, Y. and serve different roles: W captures overall sound pressure, X dif- ferentiates front and back sounds, Y differentiates left and right, and Z encodes vertical audio.
videos with FOA, offers a considerable volume of video con- tent (approximately 246 hours). However, as evidenced by ViSAGe (Anonymous, 2024), YT-360 is not optimized for video-to-audio generation (a competitive generative model on VGGSound struggles to train or finetune with YT360, often producing noise-like sounds as oufputs), probably due to the less stringent quality control. Here, Sphere360 has been developed using meticulously designed data collection and cleaning protocols, ensuring that the dataset adheres to high-quality standards, making it reliable resource for research in 360-degree video-to-spatial audio generation. Additional related works and detailed comparative analyses are provided in Appendix C.
4. OmniAudio
Overview. As illustrated in Figure 2, OmniAAdio consists of two main stages: (1) we employ a coarse-to-fine self- supervised flow matching pre-training (Figure 2a) to alle- viate the issue of data scarcity using both unlabeled spatial and non-spatial audio. (2) In the fine-tuning stage (Figure 2b), we fine-tune the diffusion transformer by efficiently in- tegrating panoramic video representation. In the following, we describe these stages in detail.
To address this gap and effectively encode FOA audio, we propose several modifications based on the Stable Audio framework: (1) We initialize our four-channel VAE with weights from pre-trained stereo VAE leverage existing non-spatial audio knowledge. (2) We eliminate the Mid-
Side Short Time Fourier Transform (MS-STFT) which is specifically designed for stereo reconstruction, and adapt the system to transform the left/right components into FOA components W, X, Y, and Z, each with an loss weight of 1/4. For detailed information on VAE, please refer to Ap- pendix D.
4.1. Preliminaries
Conditional Flow Matching. Conditional flow match- ing an effective generative technique applied across im- age (Esser et al., 2024; Lipman et al., 2022), audio (lashin & Rahtu, 2021; Liu et al., 2024b), and video domains (Liu et al., 2024c). It employs a time-dependent velocity vec- tor field parameterized by a neural network, denoted as (t, C, z). where represents time, is the conditioning input (e.g., video or text), and is a data point in dR.d.
4.3. Dual-Branch Video Representation
We propose a architecture to effectively model spatial-temporal dynamics in 360-degree videos. Given an

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
Noised Latent x
Channel Concal
Lincar Project
Noised Latent x
Lincar Project
Temporal Resample
Gated Linear Project
Non-Spatial: HES
Video Encoder
Masked Token
Vt
1V Global Condition
Flow
Flow
DiT Blocks
Linear Project
DiT Blocks
Linear Project
Spatial:
Perspective Video
VAE
Embedding
+V360
fg
Encoder
Embedding
Time Step
Audio Context
Time Step t
360-dcgree Vidco
Spatial
Video Encoder
Resize
(a) Self-Supervised Coarse-to-Fine Pre-training
(b) 360-degree Video-Guided Fine-tuning
Figure 2. A high-level overview of OmniAudio. The model leverages stereo and FOA audios for self-supervised pre- pre-trainne trainiiinng using token masking. OmniAudio efficiently trains for conditional generation during fine-tuning, supported by robust panoramic video representation. DiT denotes Diffusion Transformer.
input video sequence V360 E RTxH*WxC which is in equirectangular projection format with height-to-width ratio of 1:2, we first pad it to square format (1:1) and then resize it to match the input requirements of the image encoder. Here, T is the number of frames, H is the frame height, W is the frame width, and C' is the number of color channels. We extract the perspective video 4, denoted as Vio E R Tx capture local information. Our framework employs two complementary branches leveraging frozen pre-trained MetaCLIP-Huge image encoder (Xu al., 2024a):
4.4. Self-supervised Flow Matching Pre-training
The scarcity of pair 360-degree video-spatial audio dataset presents a significant challenge when compared to the abun- dance of unlabeled non- non-spatial audio resources. To address this issue, adopt novel self-supervised coarse-to-fine pre-training approach that leverages both the extensive un- labeled non-spatial audio and our curated FOA audio data. Inspired by the successful application of masked audio con- texts in frameworks like Audiobox (Vyas et al., 2023) and Voicebox (Le et al., 2023), we propose a two-phase pre- training paradigm with masking strategy.
H' xW'XC'" xC, from the 360-degree video V360 to
In the initial phase, we utilize the vast non-spatial audio dataset. We first convert the non-spatial audio into FOA audio and then compress these inputs into latent represen- tations using our spatial VAE, and apply token masking to these latent representations. Specifically, we condition the velocity vector field v, on partially masked audio latents Tmask with a probability Peond during training. This approach means the model has a 1Podd Poond chance of receiving a fully masked Tmask The masked condition Tmask is obtained by randomly selecting Prmnsk frames to be masked, with mini- mum masking span length of bmask This phase allows the model to learn general audio characteristics and temporal structures a resource-cfficient manner.
fg Emetactip (1s6a)) fi = Emetaatip (P(Vyov))
where Emeraclip processes both global panoramic features through equirectangular projection and local perspective features via linear projection P(-). The dual representations fused through diffusion transformers. This parameter- efficient design enables simultaneous modeling of both global scene context and fine-grained field-of-vie details, crucial for high-fidelig FOA audio synthesis.
'The Equirectangular Projection (ERP) is a popular method for projecting 360-degree videos, commonly used by platforms like YouTube. It maps a spherical view onto 2D image using latitude and longitude coordinates, with 2:1 aspect ratio (360 horizontal, 180 vertical).
In the subsequent phase, we focus exclusively on FOA audio as input. This refined stage serves pre-train the model to the specific characteristics and spatial dynamics of FOA data. By narrowing the focus to only FOA audio in the latter phase, the model effectively aligns its learned representation
*Prsseective video denotes standard rectilinear projections with less than 120-degree horizontal field-of-view (FoV).
5

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
to these more complex auditory patterns, thus enhancing its capability for high-fidelity FOA audio generation from 360- degree video content. We use Equation 3 for our training objective for only masked pieces.
audio (Copet et al., 2024), with the metric computed using the PaSST model trained on AudioSet (Koutini et al., 2021). For Spatial Audio Metrics, following Heydari et al. (2024), we employ directions of arrival (DoA) and the mean error to evaluate spatial audio quality. Specifically, we use three spatial angle-related metrics: Aabs0, Aabe and Angular- (2) Subjective Evaluation: We conduct human evaluation using the Mean Opinion Score (MOS) evaluate both spatial audio quality (MOS-SQ) and video-audio alignment faithfulness (MOS-AF). Details of the objective metrics and subjective evaluation can be found in Appendix E.1 and Appendix E.2.
4.5. Spatial-Aware Supervised Fine-tuning
Building upon our pre-trained flow matching model and dual-branch video representation, we implement an effi- cient fine-tuning strategy for video-guided spatial audio generation. Given a panoramic video sequence V360 and its corresponding perspective video Vrov. we integrate video features with dual-branch design while the FOV features are upsampled to match the audio latent sequence length and combined through element-wise addition, while 360 fea- tures pass by max- max-pooling layer and then serve as global condition in the diffusion transformer. The conditional flow matching during fine-tuning is formulated as:
Baselines. Since our work pioneers in generating spatial audio from 360-degree videos, we construct the following baselines for comparison: (1) A cascaded system that inte- grates the -fate-of-the-art (video (ext)-to-audin generation model MMAudio (Cheng et al., 2024a) with an audio spa- tialization component. Appendix E.4 includes details of the spatialization component. (2) Another cascaded system that integrates the classic video-to-audio generation model Diff- Foley (Luo et al., 2024) with the audio spatialization com- ponent. (3) ViSAGe (Anonymous, 2024), a recent model designed for generating spatial audio from perspective video inputs. We implement both ViSAGe with Fo video in- puts and ViSAGe with panoramic videos for comparisons. ViSAGe is considered a competitive model as it specifically targets spatial audio generation. For all baselines except ViS- AGe (360), we use the field-of-view of panoramic videos as video inputs, while for ViSAGe (360), we use the 360- degree video input. Both baselines are trained on the Sphere360 dataset. We assess the performance of Omni- Audio and baseline models on both the Sphere360-Bench and the YT360 test set. The YT360 test set is particularly valuable for evaluating the generalizability of our model due to its -ut-of-distriuttion characteristics. Note that audio spatialization is essentially the inverse process of calculat- ing spatial angles: the spatialization is performed using the ground truth spatial angles. Therefore, "+AS" models are not evaluated using the three metrics related to evaluating spatial angles.
(e. Vaea) (1, J360: S #t) u(xe To, F1 )l?]
(4)
where the time steps / are sampled from a logit-normal distribution. During inference, we sample trajectories using the learned velocity field with the dual-branch conditioning strategy, followed by the VAE decoder to generate the final FOA audio output.
5. Experiments
5.1. Experimental Setup
Datasets. The non-spatial audio datasets we utilize include FreeSound (Fonseca et al., 2017), AudioSet (Gemmeke et al., 2017), and VGGSound (Chen et al., 2020), comprising total of approximately 2M samples. We first segment each video into 10-second clips, then sample the video at 8 frames per second (fps), with audio sampled at 44.1 kHz. To incorporate non-spatial audio into the pre- pre-training phase, we convert them to the FOA format. Specifically, the Y and Z channels are initialized as zero, the W channel is set as the sum of the two original audio channels, and the X channel is set as the difference between the two original audio channels. Details of constructing our Sphere360-Bend -Bnhh benchmark are in Appendix C.
5.2. Main Results
We conduct comprehensive evalua- tions using both objective and subjective metrics to measure spatial audio quality and video-audio alignment. (1) Ob- jective Evaluation: For Non-Spatial Metrics, we adopt the widely used Fréchet Distance (FD) (Kilgour et al., 2018: Copet et al., 2024) measure the similarity between the fea- ture distributions of generated and reference audio, leverag- ing the 3penL3 feature space for audio projection (Cramer et al., 2019; Evans et al., 2024). Additionally, we use the Kullback-Leibin (KL) Divergence to measure the differ- ence between label distributions of generated and reference
Evaluation Metrics.
Objective Evaluation Results. As shown in Table 2, the objective evaluation results demonstrate the superior per- formance of Audioudio across multiple metrics on both YT360 and Sphere360 datasets: 1. Non-Spatial Metries:
On the YT360 dataset, OmniAudio achieves FD of 92.57 and KL divergence of 1.64, outperforming the baselines
'For all comparisons, we use the frontal-view large FoV (120") as the default perspective video. This choice aligns with woal-world conditions where front-facing viewpoints capture main sound sources.
6

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
Table Performance comparison hetween OmniAudio and the baselines the Sphere:360-Bench (in-distribution) and YT360 (out-of- distribution) test sets. We use objective metrics computing FD, KL divergence, Antz0, A.t.O, and Angute between estimated DoA and ground truth, well subjective metrics including MOS for spatial audio quality (MOS-SQ) and video-audio alignment faithfulness (MOS-AF). We report the mean and standard deviation for MOS-SQ and MOS-AF. +AS denotes adding an audio spatialization component. metrics with a downward arrow (4), lower values represent better performance, while for metrics with an upward arrow (t), higher values indicate better quality.
Model
Params
FDI
KL|
Aaba +
A Aabs |
A Angular
MOS-SQ
MOS-AFI
Inf. Time
In-distribution
(Sphere360- Bench)
GT
88,4110.79
90.1211.000
Diff-Foley + AS
0.94B
331.05
3.56
69.87:1.84
11.12
2.40s
MMAudio + AS
1.03B
271.15
2.39
75.34
77.56+1.22
3.01s
ViSAGe (FoV)
0.36B
210.87
2.90
1.51
0.71
1.49
1..45+1.42
74.89+1.71
22.37s
ViSAGe (360)
0.36B
219.66
2.96
1.52
0.74
1.51
1..12
75.34+1.03
22.37s
OmniAudio
1.22B
88.30
1.58
1.36
0.52
1.28
84.67
87.23
0.92s
Out-of-distribution (YT360-Test)
GT
85.38
87.8511.21
Diff-Foley AS
0.94B
361.65
2.22
0..2
1..3
2.40s
MMAudio + AS
1.03B
190.40
1.71
1.02
76.77+11.2
3.01s
ViSAGe (FoV)
0.36B
199.09
1.86
2,21
0.88
1.99
7..82
72.17
22.37s
ViSAGe (360)
0.36B
225.52
1.95
2.18
0.86
1.98
1.645=1.64
72.96+1.39
22.37s
OmniAudio
1.22B
92.57
1.64
1.27
0.53
1.27
80.37
83,49
0.92s
Diff-Foley AS (FD: 361.66 66,,6,,6666 KI. 2.22) and MMAudio + AS (FD: 190.40, KL: 1.51). This indicates substantial improvement in perceptual audio quality. Similarly, on the Sphere360 dataset, OmniAudio achieves FD of 88.30 and KL divergence of 1.58, surpassing Diff-Foley + AS (FD: 331.05, KL: 3.56) and MMAudio AS (FD: 271.15, KL: 2.39). 2. Spatial Audio Metries: OmniAudio consistently outperforms all baselines in spatial audio metrics. On the YT360 dataset, OmniAudio achieves A Angular of 1.27, com- pared to 1.99 for ViSAGe (FoV), 1.98 for ViSAGe (360), and higher values for the other baselines. On the Sphere360 dataset, Audioo yields Angular of 1.28, outperforming ViSAGe (FoV) 1.49 and ViSAGe (360) at 1.51. These results underscore the advantages of directly generating FOA audio from 360-degree video in accurately capturing spatial information, as opposed to generating FOA from perspective video inputs.
5.3. Case Study
In our qualitative analysis, we compare spectrograms of audio generated by OmniAudio and those produced by base- lines, as illustrated Figure 3. We make the following observations: (1) As demonstrated in case 1, MMAudio struggles to generate spatial audio when the source object is out of view. In contrast, OmniAudio effectively gener- ates FOA audio even when the object moves behind the camera, highlighting the importance of 360-degree video in accurately recreating the 3D sound environment required for FOA audio. (2) The spectrograms produced by OmniAu dio consistently exhibit higher fidelity and aligmment with video compared to those generated by baselines. In case 2, OmniAudio maintains a more consistent musical rhythm, evidenced by the performance of the percussionist in the video. These enhancements are indicative of OmniAudio 's ability to better replicate the true acoustic scene, leading to more authentic and spatially precise audio outputs.
Subjective Evaluation Results. Evaluating 360V2SA models is inherently challenging due to the subjective na- ture of perceptual quality assessment. We conducted human evaluations and report the findings in Table 2. OmniAudio demonstrated the highest perceptual quality, attaining MOS- SQ and MOS-AF scores of 84.67 and 87. 37.23, respectively. These scores highlight a clear preference by evaluators for the synthesized outputs of our model compared baseline models, in terms of spatial audio perception and video-to- audio alignment. However, it is essential to acknowledge the inherent subjectivity in evaluating spatial audio quality, which contributes to the relatively large standard deviations observed in the MOS scores.
5.4. Ablation Study
We conduct an ablation study on the Sphere360 dataset to ensure consistency and reliability. Additional analyses are presented in Appendix G.
Effect of Self-supervised Pre-training. We evaluate ef- ficacy of the self-supervised coarse-to-fine pre-training by comparing four configurations: (1) full model with coarse- to-fine pre-training (coarse-to-fine) (2) pre-training with spatial audio only (w/ fine), (3) pre-training with non-spatial audio only (w/ coarse), and (4) without any self-supervised

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
W
Z
AucioSpaceeours)
W
ViSAGe
MMAudioAAudi Spatialization
W
Figure 3. Qualitative Comparison. The first case on the left shows an agricultural machine moving behind, with the rectangular annotation indicating decreasing trend in sound intensity in the GT audio. The second the right features a person playing musical instrument. Since ViSAGe only generates 5-second audio, we concatenate the segments.
pre-training (w/o PT). As presented in Table can draw the following observation: (1) the coarse-to-fine approach consistently outperforms the other configurations across all metrics, highlighting the benefits of integrating both spa- tial and non-spatial audio datasets for pre-training. (2) the absence of pre-training and coarse stage results in notable degradation performance, emphasizing the importance of self-supervised pre-training and non-spatial audio datasets in enhancing the generalization of the model.
and non-spatial metries, demonstrating the advantages of full-view 360-degree video for generating immersive spatial audio. By combining perspective video with equirectangular representations in the dual-branch design, the model
Table 4. Effect of the bual-branch Design.
Model
FDJ
KLJ
A
A
A Angutar
ERP+Per
88.30
1.58
1.36
0.52
1.28
EAC+Per
89.89
1.66
1.33
0.55
1.29
w/ Per only
88.80
1.87
1.41
0.59
1.33
Table 3. Effect of Self-supervised Pre-training.
w/ FAC only
93.37
1.84
1.37
0.57
1.30
ERP only
97.83
1.87
1.35
0.59
1.28
Model
FDJ
KT,
A
Auts
Augular
coarse-to-fine
88,30
1.58
1,36
0.52
1.28
Impact of Model Size. To evaluate the impact of the model size on OmniAudio's performance, we compare three model configurations: Large (1.2B), Medium (472M), and Small (291M), using the Sphere360 dataset. We detail configu- rations of the different-sized models in Appendix D and use the Large model by default. We make the following observations from Table 5: (1) The Large model achieves the best performance across all metrics, including achieving the lowest FD and KL divergence. The results show that the capacity of a larger model enhances the generative quality and improves alignment with the ground truth distribution. (2) As the model size decreases from Large to Small, the performance degrades substantially. The Medium model yields moderately higher FD and KL. results, while the Small model produces the highest divergence and spatial errors. These results highlight the difficulty maintaining audio fidelity and spatial precision with smaller models, empha- sizing the necessity of adequate model capacity for effective
w/ fine
97.57
1.82
1.36
0.57
1.28
w/ course
97.26
1.78
1,36
0.66
1.30
104.57
1.83
0.58
1.32
Effect of Dual-branch Design. We investigate the effective- ness of the dual-branch design by comparing it with models that utilize only the input perspective video (w/ Per), only the Equi-Angular Cubemap (a format of 360-degree video) (w/ EAC), and only equirectangular representations (w/ ERP). The results are presented in Table 4. We find that: (1) The Atallbranch architecture outperforms single-branch models across all metrics, highlighting the benefit of inte- grating multiple representations for spatial audio generation. (2) While EAC outperforms ERP in single-branch settings, combining ERP and Per captures complementary spatial information more effectively, likely because EAC's division into six faces loses global context. (3) Panoramic videos significantly outperform perspective inputs in both spatial
8

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
audio-spatial generation.
generated from visual input can significantly enhance im- mersion in virtual and augmented reality experiences.
Table 5. Impact of Model Size.
Model Size
FD
KL,
Aab,0l
Angatar
Historical Preservation: Silent historical footage can be brought to life by adding generated audio based on visual cues, preserving valuable historical moments.
Large
88.30
1.58
1.36
0.52
1.26
Medium
104.19
1.82
1.36
0.60
1.28
Small
108.50
1.91
1.37
0.67
1.29
6.2. Negative Impacts
6. Conclusion
Misinformation and Deepfakes: The ability to generate realistic audio from video could be exploited to create fake news or manipulate audio content to misrepresent somcone's words or actions, raising ethical concerns.
In this work, we introduced the novel task of generating spatial audio from 360-degree video, termed 360V2SA. We demonstrated that traditional video-to-audio generation tech- niques often fell short in delivering immersive auditory ex- periences due to their reliance on non-spatial audio formats, which fail to capture the crucial spatial cues inherent in 360-degree video. By leveraging FOA audio, we aimed to capture the spatial cues necessary for accurately represent- ing sound sources in dimen-iimesiional environments. To facilitate research in this area, we created Sphere360, the first large-scale dataset specifically curated for 360V2SA. Our semi-automated pipeline for data collection and clean- ing ensured high-quality paired video-audio data, which was essential for training robust models. We proposed the OmniAudio framework, which utilized self-supervise pre- training and a dual-branch architecture to effectively capture both local and global information from panoramic video inputs. Our experimental results demonstrated that Om- niAudio achieved sfae-off-hhearr performance across both objective and subjective metrics the Sphere360 dataset. We envisage that our work could serve as a basis for future 360-degree video-to-spand to--ttttolllll audio generation studies.
Privacy Concerns:
There risks of privacy violations if video-to-audio technology is used to generate audio from personal videos without consent.
Job Displacement: comes more sophisticated, professionals in audio editing and sound design may face job losses.
As Al-powered audio generation be-
Generating audio that closely resembles existing copyrighted music or sound effects could lead to legal complications.
Copyright Issues:
6.3. Key Considerations
Developing guidelines and regulations to address the potential misuse of video-to-audio generation technology is essential.
Regulation and Transparency:
Ethical Development: Researchers and developers should prioritize responsible development practices to mitigate po- tential harms and ensure
Impact Statement
A Acknowledgements
This paper advances the field of Machine Learning by pre- senting novel work in video-to-audio generation. This tech- nology holds significant societal implications, both benefi- cial and detrimental, which are outlined below.
The research was supported by NSFC (No.62206234) from Mainland China, Early Career Scheme (ECS- HKUST22201322) from Hong Kong RGC, and Na- tional Natural Science Foundation of China under Grant No.62222211 and No.U24A20326.
6.1. Positive Impacts
Accessibility: By generating audio from silent videos, this technology can make visual content more accessible to peo- ple with hearing impairments, thereby improving inclusivity in media consumption.
References
An, K., Chen, Q. Deng, C., Du, Z., Gao, C., Gao, Z., Gu, Y.. He, T., Hu, H., Hu, K., Ji, S., Li, Y., Li, Z., Lu, H., Luo, H., Lv, X., Ma. B., Ma, Ni, C., Song, C., Shi, J., Shi, X., Wang, H., Wang, W., Wang, Y., Xiao, Z., Yan, Z., Yang, Y., Zhang, B., Zhang, Q. Zhang, S., Zhao, N., and Zheng, S. Funaudiollm: Voice understanding and generation foundation models for natural interaction between hu- mans and 1lms. arXiv preprint arXiv:2407.04051, 2024.
Content Creation: Creators can effortlessly add sound effects, background music, or narration videos without requiring extensive audio editing skills, opening up new creative possibilities.
Education and Training: Educational videos can be en- riched with automatically generated audio explanations, making learning experiences more engaging and accessible.
Anonymous. ViSAGe: -ideo-t--spatial audio generation. Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL, https://
Virtual Reality and Augmented Reality: Realistic audio

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
un- Caillon, A. and Esling. P. Rave: A variational autoencoder for fast and high-quality neural audio synthesis, 2021. URL https::///// / /aaxivivvv../////a//aa////aaavv. org/ abs/ 2111. 05011.
openreview. net wttp://wwwneeffouuuuimiieeeee der review.
audio datasets. In Cunningham, S. J., Duan, Z., Hu, X., and Turnbull, D. (eds.), Proceedings of the 18th Interna- tional Society for Music Information Retrieval Confer- ence, ISMIR 2017. Suzhou, China, October 23-27, 2017, pp. 486-493, 2017.
Gao, R. and Grauman, 2.5d visual sound. arXiv preprint arXiv: (t/i::///2242000 2019a.
Chen, H., Xie, w., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. arXiv preprint arXiv:2004. 14368, 2020.
Gao, R. and Grauman, K. 2.5 d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 324-333, 2019b.
Cheng, H. K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A. G., and Mitsufuji, Y. Taming multimodal synthesis.
joint training for high-quality video-to- CoRR, abs/2412. 15322, 2024a. doi: 10.48550/ARXIV. 2412.15322. URL https:t//////t httpp::///....oee httpe://ddoo.oooo///// wrw/orggo//... 48550/ arXiv. 2412. 15322.
Garg, R., Gao, R., and Grauman, K. Geometry-aware multi- task learning for binaural audio generation from video. CoRR, 11. 11.02111110882, 2021. URL https: //rxivv. org/ abs/ 21t.21111.1111111 10882.
Cheng, X., Wang, X., Wu, Y. Wang, Y., and Song, R. Lova: Long-form video-to-audio generation. CoRR, abs/2409.15111 2024b. doi: 10.48550/ARXIV. 2409.15157. URL https:://///. : /ttp://dooo..eee org/00..111.... 48550/ arXiv. D.Xiv.2409.152 15157.
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017, pp. 776-780. IEEE, 2017. doi: 10. N0.1109/CCASSP2207.779222
Copet, Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances Neural Information Pro- cessing Systems, 36, 2024.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Mísra, I. Imagebind: One embedding space to bind them all. arXiv preprint arXiv:2305.0560 05665, 2023.
Cramer, L.,,
Wu, H.-H., Salamon, J., and Bello, J. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3852 3856. IEEE, 2019.
Heydari, M., Souden, M., Conejo, B., and Atkins, J. Im- mersediffusion: A generative spatial audio latent diffu- sion model. arXiv preprint arXiv:2410. 14945, 2024.
Dagli, R., Prakash, S., Wu, R., and Khosravani, H. SEE-2- SOUND: zero-sioe shot spatial environment-tee to-stta sound. CoRR, abs/2406. 06666,06662... 2024. doi: 10.48550/ARXIV.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
2406.06612. URL https::t////.... //t:::/doooooo org/10 48550/ arXiv. 2406. 06612.
Hsu, W.-N., Zhang, Y., and Glass, Learning latent repre- sentations for speech generation and transformation, 2017. URL httptttp///// //axiiv..iiivvi///a/ orgg /a////b///// 04222.
Défossez, A., Copet, J., Synnaeve, G., and Adi, High fidelity neural audio compression, 2022. URL https: // marxiv.ee orgg/aggge abs 221 13438.
lashin, V. and Rahtu, E. Taming visually guided sound generation. arXiv preprint arXiv:21 10.08791, 2021.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., al. Scaling rectified flow transformers for nigh-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Kilgour, K., Zuluaga, M.. Roblek, D., and Sharifi, M. Fr\'echet audio distance: A metric for evaluat- ing music enhancement algorithms. arXiv preprint arXiv:1812.08466 2018
Evans, 7.,, Carr, C., Taylor, J., Hawley, S. H., and Pons, latent audio diffusion. arXiv preprint 4rXiv:2402.044 2024.
Koutini, K., Schlüter, J., Fghbal-77deh, H., and Widmer, G. Efficient training of audio transformers with patchout, arXiv preprint 1 2021.
Fast
Fonseca, E., Pons, J., Favory, X., Font, F, Bogdanov,
Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Ku- mar, K. High-fidelin audio compression with improved rvggan. In Oh, A., Naumann, T., Globerson, A., Saenko,
D., Ferraro, A., Oramas, S., Porter, A., and Serra, X. Freesound datasets: platform for the creation of open
10

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 27980- 27993, Curran Associates, Inc., 2023.
large vision models. arXiv preprint arXiv:2402.17177 2024c.
Loshehilov, and Hutter, Decoupled weight decay regu- larization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9. 2019. OpenReview.net, 2019. URL https: / oooprreeeeeee www.reviee..... net forum?
Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Ku- mar, K. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Sys- tems, 36, 2024.
Kushwaha, S., Ma, J., Thomas, M. R., Tian, Y., and Bruni, A. Diff-sage: End-to-end spatial audio generation using diffusion models. arXiv preprint arXiv:2410.1122 2024.
Luo, S., Yan, C., Hu, C., and Zhao, H. Diff-foley: Syn- chronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Sys- tems. 36, 2024.
Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L.., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., and Hsu, W. Voicebox: Text-guided multilingual univer- sal speech generation at scale. CoRR, abs/2306. 15687, 2023. doi: 10.485500AARXXI223666..55 //doi. org.//rg//0 10. R0..0.885555 48550//55iii 2306 15687.
Mei, X., Nagaraja, V., Le Lan, G., Ni, Z., Chang, E., Shi, Y., and Chandra, V. Foleygen: Visually-guided audio generation. In 2024 IEEE 34th International Workshop on Machine Learning for Signal Processing (MLSP), pp.
URL https:
1-6. IEEE, 2024.
Morgado, P., Vasconcelos, N., Langlois, T. R., and Wang, O. Self-supervised generation of spatial audio for 360 video. arXiv preprint D0XXvv:8099.025500
Leng, Y., Chen, Z., Guo, J., Liu, H., Chen, J., Tan, X., Mandic, D., He, L., Li, X., Qin, T., et al. Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances in Neural Information Processing Systems, 35:23689-23700 2022.
htss/889.02587, 2018.
Morgado, P., Li, Y., and Vasconcelos, N. Learning repre- sentations from audio-visual spatial alignment. arXiv preprint arXiv:201 1.01819, 2020.
Lipman, Y., Chen, R. T., Ben Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint (xXiv:22/0002744 2022.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y.. Chuang, C.-Y., et al. Movie gen: cast of media foundation models. arXiv preprint arXiv:2410. 13720, 2024.
Liu, A. H., Le, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N. Generative pre-training for speech with flow matching. arXiv preprint 1XXx:23100.63388 2023a.
Shimada, K., Politis, A., Sudarsanam, P., Krause, D.,
Uchida, K., Adavanne, S., Hakala, A., Koyama, Y. Taka- hashi, N., Takahashi, S., Virtanen, T., and Mitsufuji, Y. Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events, arXiv preprint urXiv:2306.09122 2023.
Liu, H., Huang, R., He, J.. Sun, G., Shen, R., Cheng, X., and Zhao, Z. Wav2sql: Direct generalizable speech-to-sql parsing. arXiv preprint arXiv: /2552, 2023b.
Liu, H., Huang, R., Lin, X., Xu, W., Zheng, M., Chen, H., He, J., and Zhao, Z. Vit-tts: visual text-to-speech with scalable diffusion transformer. arXiv preprint arXiv:2305.I 12708. 2023c.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. CoRR, abs/201 1.13456, 2020. URL abs 00t./2111.13456
Liu, H., Huang, R., Liu, Y., Cao, H., Wang, J., Cheng, X., Zheng, S., and Zhao, Z. Audiolem: Efficient and high- quality text-to-audio generation with minimal inference steps. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7008-7017, 2024a.
Sun, P., Cheng, S., Li, X., Ye, Z., Liu, H., Zhang, H., Xue, W., and Guo, Y. Both wide open: Towards language- driven spatial audio generation. CoRR, abs/2410.1066 2024. doi: 10. 1A..5////AAAX12211110000777. URL https: / //doi. wtt.:////......../......o..o. 2tt.550/rXxv...444 10676.
Liu, H., Wang, J., Huang, R., Liu, Y., Lu, H., Xue, W., and Zhao, Flashaudio: Rectified flows for fast and high-fidelin text-to-audio generation. arXiv preprint arXiv:2410. 1202440..222666 12266, 2024b.
Tian, Z., Liu, Z., Yuan, R., Pan, J., Huang, X., Liu, Q., Tan, X., Chen, Q. Xue, W., and Guo, Y. Vidmuse: A simple video-to-music generation framework with long- short-term modeling. CORR, abs/2406.0432) 2024. doi: 10. 1044550/ARXXI22200000222 URL https: /doi. 0055/10.4855//40
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. Sora: A review on background, technology, limitations, and opportunities of
04321.

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
Vasudevan, A. B., Dai, D., and Gool, L. V. Semantic ob- ject prediction and spatial sound super-resolution with binaural sounds, arXiv preprint ar arXiv:2003.04210, 2020.
Yoshida, M., Togo, R., Ogawa, T., and Haseyama, M. Binaural audio generation with data augmenta- tion from 360 videos. In International Conference on Consumer Electronics Taiwan, ICCE-Taiwan 2023, PingTTnn, Taiwan, July 17-19, 2023, pp. 795-796. IEEE, 2023, doi: htttps://doi..rr///// 10227056. URL
Viertola, I., lashin, V., and Rahtu, E. Temporally aligned audio for video with autoregression. arXiv preprint arXiv:2409.1368% 2024.
2023.
1109/
ICCE Tww.aaaawaann59.. 2023. 2ww.aan587999..2..122222
Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.-C., Guo, B., Zhang, J.. Zhang, X., Adkins, R., Ngan, W., et al. Au- diobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312. 15821, 2023.
Zhang, Y., Gu, Y., Zeng, Y., Xing, Z., Wang, Y., Wu, Z., and Chen, K. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint urXhv:2407.01444 2024.
Wang, X., Wang, Y., Wu, Y., Song, R., Tan, X., Chen, Z., Xu, H., and Sui, G. Tiva: Time-aligned video-to-audio gen- eration. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 573-582, 2024a.
Zotter, F. and Frank, M. Ambisonics: A practical 3D audio theory for recording, studio production, sound reinforce- ment, and virtual reality. Springer Nature, 2019.
Wang, Y., Guo, W., Huang, R., Huang, J., Wang, Z., You, E, Li, R., and Zhao, 7. Frieren: Efficient video-to- audio generation network with rectified flow matching. In The eight Anmual Conference on Neural In- formation Processing Systems, 2024b. URL https: //openreview.oo net forum? dd
Xing, Y., He, Y., Tian, Z.., Wang, X., and Chen, Q. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In IEEF/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 7151-7161. IEEE, 2024. doi: 10.1 10.1109/CVPPR22732224400062 URL httt::://doi.ogg////// 2024. 00683.
Xu, H., Xie, S., Tan, X. E., Huang, P., Howes, R., Sharma, V., Li, S., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C. Demystifying CLIP data. The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024a. URL https::p///. httpa////pppeeeeeeeeeeeeomm id- 5BCFFIIIff...
Xu, M., Li, C., Ren, Y., Chen, R., Gu, Y., Liang. W., and Yu, D. Video-to-audio generation with hidden alignment. CoRR, 025/2407.07464 2024b. doi: 10.48550/ARXIV. 2407.07464. URL https:ttt//... /tttts://ee http:///o..oo..oo ooro/..org/10.. / arXiv.2407.07455
Xu, M., Li, C., Tu, X., Ren, Y., Chen, R., Gu, Y., Liang, W., and Yu, D. Video-to-audio generation with hidden alignment. arXiv preprint arXiv:2407.0r7 07464444,,4 2024c.
Xu, X., Zhou, H., Liu, Z., Dai, B., Wang, X., and Lin, D. Visually informed binaural audio generation without binaural audios. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 15485-15494, 2021.
12

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
Table of Contents
1. Disclaimer on Copyright and Data Usage
2. Dataset Construction Pipeline
(a) Data Crawling
Label Searching
Type-Scale Filter
Channel-Based Crawling
Video-Basse Crawling
(b) Data Cleaning
Stationary Videos
Silent Audio
Excessive Speech
Audio-visual Mismatch
Manual Inspection
(c) Pipeline Statistics
3. Dataset Comparisons and Benchmark Construction
(a) Dataset Related Works and Comparisons
(b) Benchmark
4. Model Configurations and Architecture
(a) Model Configurations
(b) Variational Autoencoder
(c) Diffusion Transformer
5. Evaluation
(a) Objective Metrics
(b) Subjective Metries
(c) Spatial Audio Format
(d) Audio Spatialization Component
6. Limitations and Future Work
(a) Limitations
(b) Future Work
7. Additional Quantitative Results
(a) Impact different Classifier-Free Guidance (CFG) Scale
(b) Effect of Spatial VAE
8. Additional Qualitative Results
A. Disclaimer on Copyright and Data Usage
The video data utilized in this study were sourced from the YouTube platform. All content copyrighted by their respective creators and owners. The videos included in this research adhere YouTube's 's terms of service and, where applicable, to Creative Commons licenses. Specifically, videos under the Creative Commons license have been appropriately attributed to the original authors in accordance with the license terms (CC BY 4.0).
For videos not governed by a Creative Commons license, we acknowledge that they are protected by copyright and are used exclusively for academic research purposes. No commercial use of these videos or content is intended. The use these videos falls under the fair use doctrine for educational and research purposes, as permitted by copyright law.
13

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
B. Dataset Construction Pipeline
B.1. Data Crawling
Label Search
+
spatial audio 360
Basic Search Items
Dog
Qualifying Term
Event Labe]
Type-Seale Filter
dlp
Youlube Data API
yt-ilp
Non-360 videos
360 videos
FOA
X No FOA
Channel-Based
Video-Based
FOA
Type Items
Channel List
Channel Analasis
Channel Filter son vallow
Uc/wAS4EISSNNNAAAKATTT.g (9 Videos)
Manual Selection
Manual Selection
Channel--asse based Crawled Items
Video-based Crawled Items
Figure 4. The process of dataset crawling
Label Searching To identify relevant videos, we employ keyword- search strategy that targets entities commonly associated with 360 and FOA content. Our search keywords combine specific event labels to ensure class diversity with qualifying terms aimed at retrieving more 360 and FOA content. We derive our event labels from the ontology used in AudioSet (Gemmeke al., 2017). To ensure the labels are simple, comprehensive, and likely to appear in video titles, we limit selection the first three levels of the hierarchical ontology. Through manual curation, we have compiled list of 316 event labels. Throughout the process, periodically re-evaluate the search results to refine the keywords, removing event labels that yield few or low-quality results to improve class diversity, and testing multiple qualifying terms to increase the proportion of 360 content. As a result, we selected "spatial audio 360" as the primary qualifier. This process led to the compilation of 316 search keywords in the format "[event label] spatial audio 360".
Type-Scale Filter API (by checking the "projection" field within "contentDetails" to ensure is set to "360") and yt-dlp (using the format filter miudio_channels' to exclude videas that do not support 4-channel audio for any player client). This approach enables us identify videos that contain both 360 video and FOA audio.
We filter out videos that lack either 360-degree visual content or spatial audio using the YouTube Data
14

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
Channel-Based Crawling In the initial stage of crawling, we use large-scale approach to filter relevant channels. We Start by searching up to 5 pages (a maximum of 250 results) for each keyword and then filter out 360 and FOA entities to create smaller search set. Next, we identify channels that appear more than three times within the S-page search set. For each prominent channel, we randomly select 10 videos and manually filter for relevance. This process results in 124 relevant channels out of an initial 246. By collecting all videos from these channels, we compile a list of 1488 highly relevant videos. Additionally, we create blacklists (for totally irrelevant channels) and whitelists (for highly relevant channels) based on manual filtering in the first stage. These lists will be used in the next crawling stage to streamline the filtering process.
Video-Based Crawling In the second stage, we expand the search scale to create a larger search set. Since entities beyond the first 5 search pages tend to be less relevant and contain fewer 360 and FOA items, we manually filter out irrelevant videos. By searching up 12 pages for each keyword and excluding videos from the blacklists or whitelists established in the previous stage, we obtain 652 360 and FOA items, of which 438 are manually identified as relevant. Additionally, we experiment with other qualifying terms for searching, and the top two are detailed in Table 6. The overall details of the two-stage crawling are shown in Table 7, and the whole procedures are illustrated in Figure 4.
Finally, by adding 83K clips (total duration of 230.6 hours) successfully downloaded from 5128 videos in the YT-360 dataset, we obtained a total of 166.5K clips (approximately 462.5 hours) from 7179 videos.
Table 6. Details of videos included in the dataset.
Crawling. Different qualifying terms are applied independently. Selected Videos refers to the number of
After
360 & FOA
Qualifying Term
360
Videos
Videos Selected
Channel Filter
Videos
spatial audio 360
5996
2036
652
438
3D audio 360
4353
2899
384
245
Total
8746
4356
848
563
Table 7. Outcome of the Two Stages. Videos Covered denotes the number of 360 videos in the search set (for Channel-Based Crawling.
this refers to the total number of 360" videos from all checked channels). Videos Selected, Clips, and Total Duration indicate information of relevant videos manually identified as relevant.
Stage
Videos Covered
Videos Selected
Clips
Total Duration
Channel- Cruwling
13733
1488
58.9K
163.6
Video-Based Crawling
8746
563
24.6K
68.3h
Total
22479
2051
83.5K
231.9h
B.2. Data Cleaning
Stationary Videos To clean stationary videos, we measure the similarity between two frames at fixed intervals using the mean squared error (MSE). A certain threshold of similarity is set to classify frames as stationary. If a video segment contains more than 85% of stationary frames, it is classified as a stationary video and subsequently removed from the dataset.
Silent Audio across all channels for each segment. If this value falls below -35, we consider the segment to be silent. the number of silent segments in the entire audio exceeds 90%, the audio is classified as silent and removed from the dataset.
For cleaning silent audio, we divide the audio into several segments and calculate the maximum dBFS value
Fie ID. [3
Fe byoe. Inngerzgi
-| Dipio Me)
File D 2000Usa sE
Fe bype (ede (961
Dep Be
le typec (Woes Lwebnn Dipaay
File preview:
File preview:
File preview:
000 /010
100/18
Figure 5, Manual inspection webpage.
15

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
Diny
Data eleaning
Demee:
Chred2
Dhame
Other data cleaning methods
Stationary video
Dynamic video
Silent audio
Informative audio
Manual inspection
Clean Dataset
Figure 6. The process of dataset cleaning.
Excessive Speech For videos with excessive speech, and remove videos where the number of detected words exceeds 5. We do remove all videos containing speech, this approach allows to retain videos that feature moderate levels of speech, which may still be valuable for the dataset. This ensures that videos with important context or minimal speech are not erroneously discarded, preserving a more balanced range of content.
Sense Voice (An al., 2024) detect the presence of speech
Audio-visual Mismatch For cleaning based audio-visual alignment, the degree of alignment between video and audio. We stipulate that videos with degree higher than are considered qualified. Videos that do not meet the threshold are removed from the dataset, ensuring that only videos with strong and coherent connection between audio and visual content are retained. This approach helps to eliminate videos where the audio-visual mismatch may hinder the quality of data analysis or model training.
Imagebind (Girdhar al., 2023) assess
Manual Inspection We develope clear and user-friendly webpage (Figure 5) to manually inspect the effectiveness of the data cleaning process. We sample and verify the data that has been removed to ensure it truly does not meet the criteria, preventing valid data from being mistakenly deleted. Additionally, we sample the remaining data after cleaning to confirm meets the required conditions. any discrepancies are found during this process, we review the appropriateness of the cleaning thresholds and the accuracy the cleaning process, restart the cleaning, and make manual adjustments when needed, to ensure the effectiveness of the cleaning process. The overall data cleaning process diagram shown in Figure 6.
16

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
B.3. Pipeline Statisties
During the cleaning process, following the specific methods mentioned above, we remove approximately 7,500 silent clips, 12,000 static clips, 34,000 clips with excessive human voices, and 23,500 clips with audio-visual mismatches (some of the clips removed during the cleaning process may overlap). The number of clips in each category of the cleaned dataset is shown in Figure 7.
Figure 7. The har chart shows the occurrence count different classes the cleaned dataset (displayed on a log scale).
C. Dataset Comparisons and Benchmark Construction
Dataset Related Works and Comparisons Several datasets (Morgado et al., 2020; Liu et al., 2023b) have been developed to support research in 360-degree video and spatial audio processing. REC-STREET (Morgado et al., 2018). YT-ALL (Mor- gado et al., 2018), 8), and YT-360 (Morgado et al., 2020) provide large-scale 360-degree video collections with First-order Ambisonics (FOA) audio (with total durations 3.5, 113, and 246 hours, respectively), although their primary focus on audio-visual correspondence learning. STARRS23 (Shimada et al., 2023), while supporting spatial audio generation from 360-degree videos, contains only 200 clips (7.5 hours), limiting its applicability for deep learning approaches. More recently, YT-Ambigen (Anonymous, 2024) introduced dataset of 102K clips (totaling 142 hours), specifically designed for spatial audio generation; however, it is limited to fixed field-of-view (FoV) videos rather than full 360-degree content. Our
Table Comparison of Sphere360 with existing datasets. FoV 360 respectively represent field-of-view videos and panoramic videos. NS, Bin, and FOA refer non-spatial audio, binaural audio, and first-order ambisonies, respectively. "y indicates that is not explicitly mentioned the paper.
Clip duration
Video & Audio Type
Audio
Contains Benchmark
Dataset
#Clips
Video length
#Class
Generation
VGGSound (Chen et al., 2020)
200K
550h
FoV NS
Yes
Yes
300
FairPlay (Gao & Grauman, 2019a)
1.8K
10s
5.2h
FoV & Bin
No
/
/
OAP (Vasudevan al., 2020)
64K
2s
15h
360 & Bin
No
/
23
REC-STREET (Morgado et al., 2018)
123K
0.1s
3.5h
360 & FOA
No
Yes
YT-ALL (Morgado et al., 2018)
3976K
0.1s
113h
360 & FOA
No
Yes
28
YT-Ambigen (Anonymous, 2024)
102K
5s
142h
FoV & FOA
Yes
/
>00
STARRS23 (Shimada et al., 2023)
0.2K
10min
7.5h
360 & FOA
Yes
Yes
13
YT-360 (Morgado et al., 2020)
89K
10s
246h
360" & FOA
No
/
32
Sphere360
103K
10s
288h
360 & FOA
Yes
Yes
288
17

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
Erquuescy Clos occurr sce 2.lg scale)
Figure 8. The bar chart shows the occurrence count of different classes in the benchmark dataset (displayed on log scale).
Sphere360 dataset offers over 103K high-quality 360-degree video clips with FOA audio, totaling 288 hours in duration, specifically curated for spatial audio generation through meticulous data collection and cleaning pipelines. A comparison of Sphere360 with existing datasets is shown in Table 8.
Benchmark Currently, there is a lack of an accessible benchmark for 360-degree video-to-spatial audio generation. To address this gap, we construct benchmark encompassing about 180 distinct audio events using our semi-automated data-cleaning process. During the construction of the benchmark, we first label and categorize the dataset based on the audio. Then, we apply the semi-automated cleaning process for initial screening with lower standards remove low-value video segments. Next, remove the video classes that are not of interest from the remaining dataset and select several video segments with the highest confidence for each class. During this selection, we try to retain complete video clips to better assess the generalization capabilities of the models. Following this, we utilize our designed web platform for manual verification, where samples failing to meet the standards are rejected and reprocessed. We iterate this process, adjusting the parameters used for filtering at each step, ultimately selecting around 3,000 high-quality clips and covering about 180 events. The class occurrence frequency of the benchmark dataset is shown in Figure
D. Model Configurations and Architecture
Model Configurations For VAE training, we initialize our Spatial VAE using the VAE model weights trained on stereo data provided by Stability AI We employ mixed precision training with a batch size of 144 across 24 A800 GPUs for 500,000 steps. Subsequently, following Evans et al. (2024), we freeze the VAE encoder and train the VAE decoder with a latent mask ratio of 0.1 for an additional 300,000 steps. We use Adam W (Loshchilov & Hutter, 2019) as the optimizer, setting the generator learning rate to 3e-5 and the discriminator learning rate to 6c-5.
In the self-supervised pre-training phase, we apply mask with a conditioning probability Peond of 0.1. We utilize exponential moving average and automatic mixed precision for 100,000 steps on 8 A100 GPUs, with an effective batch size of 256. For the Video-Guided fine-tuning stage, we similarly apply exponential moving average and automatic mixed precision for 50,000 steps on 8 A100 GPUs, maintaining an effective batch size of 256. AdamW remains our optimizer of choice, with a learning rate set at 5e-5.
Variational Autoencoder The spatial VAE is initialized with pre-trained weights from a stereo VAE, To reconstruct FOA audio, we train the VAE using a weighted four-channel multi-resolution STFT loss. Additionally, we apply KL divergence
18

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
Discriminator
X
Weighted STFT Loss
Z
WHN0
Encoder
Latent
Decoder
Encoder:
Conv
D(s)
Snake
Conv
D:
R(s)
Snake
Strided Conv
mm
R:
M
M
Snake
Dilated Conv
Snake
Ixl Conv
Figure 9. The architecture of autoencoder. D and R are designations for specific structures.
Flow Time Step
Video Features
Lin
Loue
Vr
Audio Context
Transforer
(a layer)
Input
Inputz
Layer Norm
Layer Norm
Layer Norn
Self Attention
Cross Attention
Gated MLP
Figure 10. The architecture of diffusion transformer.
19

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
loss the VAE bottleneck and discrimination loss enhance high-fidelity audio reproduction. The architecture of the VAE is illustrated in Figure 9.
Diffusion Transformer The flow component employs Diffusion Transformer (DiT) with an embedding dimension of 1536. It comprises 24 layers and 24 attention heads, with local and global conditioning dimensions of 768 and 1536, respectively. The transformer operates by projecting condition tokens and adheres to continuous transformer architecture. The overall architecture is illustrated in Figure 10, while detailed configurations for different model scales are provided in Table 9. Given the scale of the existing dataset, we believe that model with 1.2 billion parameters is already sufficiently large, and therefore, we did not further increase the capacity of our model.
Table Diffusion Transformer Configurations at Different Model Size.
Embedding Dimension
Attention Heads
Condition Token Dimension
Global Condition Dimension
Total Parameters
Model Scale
Depth
Large
1536
24
768
1536
1.2B
Medium
1024
16
512
1024
472M
Small
768
12
12
384
768
291M
E. Evaluation
E.1. Objective Metrics
The Fréchet Distance (FD) used measure the similarity between the feature distributions of generated and reference audio. A lower FD indicates that the generated audio is closer to the reference in terms of feature distribution ((Kilgour et al., 2018), (Copet et al., 2024)). We use the OpenL3 feature space for audio projection ((Cramer et al., 2019), (Evans et al., 2024)).
The Kullback-Leible (KL) Divergence measures the difference between the label distributions of generated and reference audio. A lower KL indicates better semantic alignment between the generated and reference audio. The KL metric utilizes the PaSST model, an audio tagger trained on the AudioSet dataset ((Koutini al., 2021)), compute the KL divergence (CCopet et al., 2024)).
To assess the spatial accuracy of FOA (First-Order Ambisonic) audio, we caleulate the intensities 4. mcan W X), ly mean W Y), and I mean(W Z) for the directional channels and report three key metrics ((Heydari et al., 2024)):
Theta Error () measures the difference between the ground truth azimuth (0) and the estimated azimuth (0). The azimuth is the angle in the horizontal plane, defined
Ty 1
0= = tan
The azimuth error is computed using the circular difference method ((Heydari et al., 2024)):
Aahs0 / = min (10 0, 2n - 10- -00 01)
Phi Error (A quantifies the difference between the ground truth elevation (Φ) and the estimated elevation (δ). The elevation angle is calculated as:
1
Lan
12 - 13
The elevation error is computed as the absolute difference between the ground truth and the estimated value:
20

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
Aabs
10
Spatial-Angle Error Spatialllll))) quantifies the difference between the ground truth and estimated directions of arrival (DoA). The spatial angle ASpatial-Angle is calculated using the following equations:
/AA
a = sin?
+ cos( cos(Ø) sin?
2
2
A Abbs Spaaiaaaial = 2 arctan (Va, v1
Where AO and Ao represent the linear and circular differences for azimuth and elevation, and and & represent the ground truth and estimated elevations.
The lower the values of these three metrics, the better the generation quality.
E.2. Subjective Metrics
To probe spatial audio quality, we conduct the MOS (mean opinion score) tests and explicitly instruct the raters to "focus on examining the audio quality, naturalness, spatiality, and overall preference.". The testers present and rate the samples, and each tester is asked to evaluate the subjective naturalness on a 20-100 100 Likert scale.
To probe video-audio alignment, human raters are shown a spatial audio and 360-degree video and asked "Does the spatial audio align with 360-degree video faithfully??". They must respond with "completely", "mostly"" or "somewhat" on a 20-100 Likert scale.
Because of the characteristics of 360-degree videos and spatial audio, recruit 15 participants in person rather than crowdsoureing like Amazon Mechanical Turk. We randomly select 50 samples from our Sphere360-Bor Bench for each annotator.
E.3. Spatial Audio Format
Spatial audio is technology that captures and reproduces sound in such way that it mimics the natural experience of hearing, providing listeners with a full 360-degree auditory environment. This involves accurately encoding sound direction, distance, and amplitude to create immersive and realistic audio experiences. this context, FOA a popular format due to its ability to effectively balance spatial resolution with computational simplicity. FOA achieves this using four specific channels labeled W. X, Y, and Z. Each channel serves unique function: the W channel captures overall sound pressure from all directions, the channel differentiates sounds from the front and back, the Y channel distinguishes left from right, and the Z channel encodes vertical audio information, such sounds coming from above or below.
E.4. Audio Spatialization Component
Given the ground-truth direction (ø, 0) calculated from the previous section E.1, we can spatialize mono audio signal into First-Order Ambisonies (FOA) format. Following (Zotter & Frank, 2019), the encoding process converts mono sound source x()) into FOA channels as: W (1), X cos 0 cos dss(),, Y = sin 0 cos @s((),, and Z sin @s((). To spatialize our generated mono audio, we directly apply this encoding with st) being our input signal, thereby positioning the sound in the corresponding direction the ambisonic domain.
F. Limitations and Future Work
While our dataset demonstrates strong performance for the majority of test cases, it is worth noting that some samples, derived from real-world scenarios, contain an unusually high number of sound-emitting objects, as illustrated in Figure 11. In such cases, OmniAudio struggles to accurately identify event types, leading to errors such as misclassifying musical instrument sounds as applause.
Limitations
21

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
Future Work We plan explore techniques for better understanding 360-degree videos with multiple targets. Additionally, current cleaned dataset comprises 100,000 samples, which remains insufficient for robust real-world 360V2SA. To address this, will leverage semi-automated pipeline to continuously collect and expand the dataset, thereby advancing progress in this field.
front view
back view
AudioSpaceeours)
X fails to generate
W
Figure 11. An failure case of OmniAudio. In this case, the front view shows instrumental ensemble and the back view shows the audience. The sound source the instrument sound from front, but model mistakenly identifies the sound source as applause from the audience behind.
G. Additional Quantitative Results
Impact different Classifier-Free Guidance Scale (CFG-Scale) Testing with different CFG-Scale settings, the results shown in Figure 12, illustrating how FD and Aabs Spatial-Angle change.
126
132-
Figure 12. Variation of FD and Aata Spatial-Angle under Different CFG-Scale Settings
Balancing the performance FD and spatial results, we choose CFG-Scale 5.
Effect of Spatial VAE The results of the Variational Autoencoder (VAE) model in terms of STFT, MEL distances, FD, and KI. divergence presented in Table 10. A comparison is made between the proposed model (ours), -es-ript-audio-cddee (Kumar et al., 2024), and non-spatial audio VAE model. Better performance is indicated by lower values of STFT, MEL distances, FD, and KL. Considering all these metrics, our model demonstrates superior overall performance, outpacing the others.
Table 10. Performance of VAE model in terms of STFT and MEL distances. For metrics with downward arrow (L), lower values represent better performance.
Model
STFT Distance).
MEL Distance
FD)
K1.-
Ours
0.82
1.48
94.15
Descript-Audio Codec
0.71
4.74
123.41
Non-Spatial Audio VAE
4.12
6.65
272.89
2.12
22

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
More Analysis of Dual-Branch Strategies To further investigate the effectiveness of dual-branch approach, conducted additional experiments comparing various FOV-cut strategies. Our original design employs 120" frontal view focus on prominent sound sources while incorporating panoramic features contextual conditioning. We evaluated three alternative FOV-cut strategies to replace the local FOV video:
Hexadirectional 360 cuts (ERP+6cuts): Capturing views from front, back, left, right, up. and down directions.
Quadrant cuts (ERP+4cuts): Capturing views from front, left, right, and back directions.
Bipolar cuts (ERP+2cuts): Capturing views from front and back directions.
Table 11 presents the results of this comparison.
Table II. Comparison of Dual- Branchh Methods
Dual- mual-Branch Method
FD +
KL +
aba +
Aaha 4
A Angular +
ERP+Front
88.30
1.58
1.36
0.52
1.28
ERP+2FOV Cuts
95.84
1.77
1.37
0.55
1.30
ERP+4FOV Cuts
92.89
1.65
1.36
0.51
1.27
ERP+6FOV Cuts
90.16
1.59
1.37
0.52
1.26
The results demonstrate that our original dual-branch strategy (ERP+Front) still achieves the best audio quality, as reflected in the FD and KL metrics. Moreover, it achieves similar spatial metrics compared to the multiple FOV view approaches. This analysis supports our choice of using a single frontal view combined with equirectangular projection in our final model design.
H. Additional Qualitative Results
We include more qualitative results in this section, as shown in Figure 13, 14, and 15.
23

---

OmniAudio: Generating Spatial Audio from 360-Degree Vidco
GT
Z
ourr))
ViSAGe
Aududio+Audio Spatialization
Figure 13.. Additional Quantitative Results. This shows train passing by. The rectangular annotation indicates that the audio generated by our model continues to capture the sound the train leaving the frontal perspective, even after it has passed, while the audio generated by other models almost entirely fades the train moves out of the frontal view.
24

---

OmniAudio: Generating Spatial Audio from 360-Degree Video
GT
(pcioSpaace(urs)
ViSAGe
Audio Spatialization
Figure 14. Additional Quantitative Results. The case on the left shows a continuous display of fireworks rising into the sky and exploding. The case the right depicts several motorcycles chasing each other dirt road, with intense wind and engine sounds.
GT
W
OodssS
ViSAGe
MMAudio AarAAudio-Audii Audio Spatialization
Figure Additional Quantitative Results, case shows camera mounted on navigating through waves, with the bow plunging into the water and splashing onto the screen. The case on the right shows the viewpoint moving through a noisy crowd in an indoor environment.
25


Reviews:
[{"review_id": "GXacsPKnah", "reviewer": "Reviewer_Xvna", "summary": "This paper proposes the task of generating spatial audio from 360-degree videos. To support this task, the authors construct a dataset named Sphere360, comprising curated real-world 360-degree videos collected from YouTube. Leveraging this dataset, the authors introduce the AudioSpace model, which employs self-supervised pretraining with both spatial and non-spatial datasets, along with a dual-branch architecture utilizing panoramic and field-of-view (FOV) video inputs. Given the preliminary nature of this task, several baseline models are established for comparison. Experimental results indicate that AudioSpace achieves promising outcomes in both objective and subjective evaluations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback and detailed suggestions. We hope our response below fully resolves your concerns and questions.\n\n## Claims that AudioSpace achieves SOTA performance\nWe sincerely apologize for the two typos in Table 2 main results and typos and boldfacing errors in Table 3 and Table 4. These were completely unintentional oversights on our part. **As shown in the updated Table 2 below, AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test.**\n### 1. Open-Source Commitment\nCodebase and model weights will be open-sourced by April 2025. The Sphere360 dataset is already released at https://anonymous.4open.science/r/Sphere360-CF51/.\n### 2.Updated Table 2 (Bold=Best, Highlight fixed numbers):\nFixed typos in YT360-Test results. AudioSpace achieves **best objective and subjective results and fastest inference speed**:\n\n|Model|FD|KL|$\\Delta_{abs}\\theta$|$\\Delta_{abs}\\phi$|$\\Delta_{Angular}$|Params|Infer Time(s)|TFlops|\n|---|---|---|---|---|---|---|---|---|\n|Diff-Foley+AS|361.65|2.22|/|/|/|0.94B|2.40|9.11|\n|MMAudio+AS|190.40|`1.71`|/|/|/|1.03B|3.01|7.18|\n|ViSAGe(FOV)|199.09|1.86|2.21|0.88|1.99|0.36B|22.37|0.68|\n|ViSAGe(360)|225.52|1.95|2.18|0.86|1.98|0.36B|22.37|0.68|\n|AudioSpace|**92.57**|**1.64**|**1.27**|**`0.53`**|**1.27**|1.22B|**0.92**|21.15|\n\n### 3. Extended V2A Validation:\nAudioSpace also excels in video-to-audio generation. Please see **Response to Reviewer cQFE** for results.\n\n### 4. Fixed Table 3 and Table 4:\n  AudioSpace results (row1 in both tables) are correct and best in FD, KL, and most spatial metrics. Fixed Tables 3 and 4 are on the demo page. **Coarse-to-fine pre-training and dual-branch design are superior to alternatives**.\n\n## Fairness of Experimental Comparisons\n\n### 1.Fair Comparison Protocol\nAll baselines in Table 2 follow their original pre-training and are fine-tuned on Sphere360 with the same setup.\n\n|Model|Training Data (Type/Scale)|Pretraining Strategy|\n|---|---|---|\n|ViSAGe|VGGSound/YT-360 (~300k pairs)|Video-guided init.|\n|MMAudio|VGGSound/WavCaps/AudioCaps (~1.1M pairs)|Text+video init.|\n|AudioSpace|FreeSound/AudioSet/VGGSound/Sphere360 (~2M audio samples)|**Audio-centric pretraining**|\n\n**Uniform pretraining is unsuitable as baselines need multimodal pairs (text/video+audio)**, while AudioSpace focuses on **audio diversity** through domain-specific pretraining.\n\n### 2. Architectural Superiority: Table 3 **w/o PT** (trained from scratch on Sphere360) and Table 2 show structural advantages: **Without pre-training, AudioSpace surpasses all pretrained-finetuned baselines on Sphere360 in all objective metrics.**\n\n## Threshold for audio-visual similarity score for data cleaning\nThe threshold of 1 corresponds to the original cosine similarity score scaled by 10 for easier processing, which does not impact comparative results or conclusions. The threshold was chosen based on video-audio alignment control and data analysis.\n\n## 360˚ Video Representation\nWe appreciate the question about motion representation in ERP-formatted videos. Our dual-branch design addresses ERP limitations as follows:\n\n### 1. Geometric Distortion in ERP\nERP causes planar distortions and boundary discontinuities, conflicting with CLIP encoders’ planar continuity assumptions, leading to feature misalignment. Table 4 shows single-branch ERP-only model suffers from severe feature distortion (FD: 97.83 vs. our 88.30) and semantic inconsistency (KL: 1.87 vs. our 1.58), indicating naive planar encoding fails in spherical contexts.\n\n### 2. Dual-Branch Design with Frontal Perspective\nERP distorts global spatial relationships, but motion trajectories remain coherent within FOV. Our temporal modeling FoV branch isolates motion modeling to distortion-free frontal regions, ensuring stable motion encoding, as shown by Y-channel intensity variations in Fig.13.\n\n### 3. Limitations and Future Directions\nObjects entirely outside the frontal viewport may challenge our current approach, though they represent <5% of training data in typical 360° videos. Future work will explore geometric-aware feature rectification to address ERP distortions at polar regions.\n\n## Clarification of Inference\nDuring inference, the frontal-view segment of the panoramic video is used for local conditioning, and the full panoramic video for global conditioning. Both features guide the diffusion transformer's ODE solver (Euler method, CFG scale=5). The audio patents are decoded by the pre-trained spatial VAE decoder to produce the final spatial audio outputs.\n\n## FOV selection criteria\nDuring training and testing, we use the frontal-view large FOV (120°) as the default FOV video. This choice aligns with real-world conditions where front-facing viewpoints capture main sound sources.\n\n## Typos\nThank you for your reminder. Line 158 should refer to Figure 8, not Table 8, for the audio event distribution. We will make all the fixes in the revised paper.", "claims_and_evidence": "- The claim that AudioSpace achieves state-of-the-art (SOTA) performance is questionable. The authors frequently emphasize their final model's results in bold throughout Tables 2, 3, and 4, even in cases of tied scores or superior performance by other models. This could mislead readers into perceiving AudioSpace as consistently superior across all metrics.\n\n- The introduced 360V2SA dataset represents a valuable contribution to the community, facilitating further research into 360-degree video-guided spatial audio generation.", "methods_and_evaluation_criteria": "- In Section 3 (Data Cleaning), the authors state that videos with an audio-visual similarity score below 1 are discarded. Clarification is needed regarding this threshold. Specifically, what is the range of audio-visual similarity scores? If this score refers to cosine similarity, the threshold value provided may be incorrect or misleading.\n\n- During 360-degree video-guided fine-tuning, authors mention max-pooling 360 features to serve as a global condition. However, extracting only a single vector from the image encoder might not sufficiently represent motion information (e.g., a car moving from right to left). Additional explanation or justification of this method is necessary.\n\n- Are the outputs depicted in Figure 2(b) fed directly into the decoder of the spatial VAE? The inference of each component requires clearer explanation.\n\n- The selection method for the field of view (FOV) input in the visual encoder is not explained. Additionally, how should the FOV be determined during test time? Clarification on FOV selection criteria is necessary.", "theoretical_claims": "No theoretical claims are presented.", "experimental_designs_or_analyses": "- The comparison setup may not be entirely fair, as existing methods are trained exclusively on Sphere360, whereas AudioSpace undergoes initial training on FreeSound/AudioSet/VGGSound before Sphere360. Conducting comparisons with identical training datasets across all methods would better highlight the effectiveness of the proposed model. Additionally, the authors note in Table 3 that coarse training significantly contributes to performance; thus, applying similar training strategies to existing methods would clarify the specific impact of the proposed model design.\n\n- In Table 2, specifically for $\\Delta_{abs}\\phi$ in YT360-Test, AudioSpace is incorrectly marked in bold despite performing worse than ViSAGe. Similar inconsistencies appear in Table 3, where numbers ($\\Delta_{angular}$) are incorrectly highlighted or tied results ($\\Delta_{abs}\\theta$) are unnecessarily emphasized for AudioSpace. Such inconsistencies may create misleading interpretations, suggesting the authors intentionally emphasize favorable results. Similar issues appear in Table 4.", "relation_to_broader_scientific_literature": "No explicit relation to broader scientific literature is identified.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See other sections.", "other_comments_or_suggestions": "- Line 158 references Table 8, which does not contain information about audio event distribution as suggested.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wkk2Wy64xm", "reviewer": "Reviewer_MEu5", "summary": "This paper addresses an interesting problem of generating spatial audio from panoramic videos. They first propose a real-world dataset, Sphere360, for 360 videos and their spatial audios.  They also propose an effective training strategy combining coarse-to-fine pre-training and dual-branch video encoding for spatial-aware generation. The proposed method, AudioSpace, achieves state-of-the-art performance on the Sphere360-Bench", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of the strong motivation and exceptional quality of our paper. We hope our response thoroughly addresses your concerns and questions.\n\n## Updated Table 2 and Inference Latency\n\nWe sincerely apologize for the typos in Table 2 in the submission. Please check our **Response to Reviewer Xvna** under **Claims that AudioSpace achieves SOTA performance.** As shown in the updated Table 2, **AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test.** We also added the number of parameters and inference time into comparison. Although AudioSpace has slightly more parameters than MMAudio+AS, **it achieves notably faster inference than all baselines.**\n\n## Roles of Training Stages\n\nAudioSpace includes a coarse-to-fine self-supervised pre-training stage (**Stage 1**) and a spatial-aware supervised fine-tuning stage (**Stage 2**).\n\n### 1. Roles of Training Stages\n\n**Stage 1**: Utilizes large-scale unlabeled audio datasets (~2M samples) to build foundational audio understanding capabilities, crucial for 360-degree video-to-spatial audio generation. It focuses on **General Audio Distribution Learning** through masked audio context modeling, enhancing audio feature representation and temporal coherence.\n\n**Stage 2**: Tailors the model for spatial audio generation from 360-degree video. Key aspects of Stage 2 are:\n\n* **Modality Alignment**: Integrates video data for precise spatial audio generation.\n* **Task-Specific Optimization**: Refines the model for the 360V2SA goal, balancing audio quality and spatial accuracy.\n\n### 2. Validation on 360V2SA Task\n\n| Configuration | FD | KL |\n| --- | --- | --- |\n| Stage 2 only (full data) | 104.57 | 1.83 |\n| Stage 1+2 (full data) | **88.30** | **1.58** |\n| Stage 1+2 (80% data) | 89.86 | 1.60 |\n| Stage 1+2 (60% data) | 93.11 | 1.80 |\n| Stage 1+2 (40% data) | 105.26 | 1.88 |\n\nThe table shows that after Stage 1, using 40% of Stage 2 data matches Stage 2 full-data baseline, 60% outperforms it, and 100% substantially exceeds it, in FD and KL: Stage 1 leads to reduced reliance on labeled data and increased robustness.\n\n### 3. Further Validation on V2A Task\n\nWe validate the effects of training stages on traditional V2A tasks on Sphere360. Stage 1 reduces FD by 16.3% relative and KL by 19.2% relative, confirming its effectiveness in improving audio quality.\n\n| Stage | FD | KL |\n| --- | --- | --- |\n| Stage 2 | 41.30 | 2.03 |\n| Stage 1+2 | **34.56** | **1.64** |\n\nThe two stages complement each other. Stage 1 builds a strong audio base, while Stage 2 adds modality-specific constraints, preventing overfitting to limited video-audio pairs. The tables highlight the necessity of both stages. Without pre-training, the model struggles with coherent audio, even with video guidance.\n\n## Video Length Constraints\n\nOur training uses fixed 10-second clips and rotary positional encoding rather than absolute positional embedding for flexible inference lengths. However, the 10-second limit of the video datasets restricts generated audio to under 10 seconds.\n\nThe self-supervised pre-training helps with audio coherence, but reliability beyond 10 seconds is uncertain. This is due to potential degradation in modality alignment and feature consistency. Future research is needed to develop extended temporal modeling for long-form audio synthesis.\n\n## Scene Transition Cases\n\nThank you for raising this interesting and insightful question.\nOur training and demo data mainly use fixed-camera 360° videos for immersive experiences. To test AudioSpace’s robustness to scene transitions, we used 50 single-scene videos, paired them into 25 groups, and spliced 5-second segments into 10-second clips with abrupt transitions. **We added interactive spatial audio for these clips on the demo page - [Scene Transition Cases]**. Results show that in some examples, our model can generate transitions that are natural and seamless. It even integrates the content from both segments into a coherent whole—for instance, when the two segments showcase performances by different musical instruments, the synthesis can merge them into a balanced duet at times. However, because our training data predominantly consists of single-scene videos, the model occasionally emphasizes one segment over the other, sometimes overlooking the acoustic nuances of the less dominant scene.\n\n## More Visual Comparisons for Ablation Studies\nDue to the time limit of the first rebuttal phase, we will add more visual comparisons for the ablation study in the final response.", "claims_and_evidence": "* The high-quality demo videos show great performance in generating spatial audio from the 360 video.\n\n* The quantity results, e.g., Tab.2, demonstrate significant performance improvement of the proposed method.", "methods_and_evaluation_criteria": "The evaluation is reasonable.\n\nFor the proposed method:\nThere are two main stages:  (1) A coarse-to-fine self-supervised flow matching pre-training to alleviate the issue of data scarcity using both unlabeled spatial and non-spatial audio. (2) Fine-tuning the diffusion transformer by efficiently integrating panoramic video representation.", "theoretical_claims": "There is no proof for theoretical claims", "experimental_designs_or_analyses": "The experiments and ablation study are well-designed.\n\nI want to see more visual comparisons for the ablation study.", "relation_to_broader_scientific_literature": "Relate to some research area of CG.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Weaknesses:\n* As the methodology consists of two phases of training, are there relevant experiments analyzing the respective roles of these two phases of training and the relationship between the two phases of training?\n* I noticed that the demo videos are all single scenes, what would happen if the given 360 video had scene transitions? And the authors should discuss what the length limit of a 360 video should be.", "other_comments_or_suggestions": "The motivation and quality of this paper are very good and I would have increased my score if the author had answered my questions adequately at the rebuttal.", "questions_for_authors": "Refer to the \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fdkPZY5Hvi", "reviewer": "Reviewer_nf2N", "summary": "This paper addresses a novel task called 360V2SA, which involves generating First-order Ambisonics (FOA) spatial audio from 360-degree videos. To tackle this challenge, the authors introduce a new dataset called Sphere360, containing more than 100k clips of real-world 360-degree videos paired with their FOA audio. They also propose AudioSpace, a dual-branch flow-matching framework that combines both panoramic and field-of-view (FOV) video representations. The model is first pre-trained in a self-supervised manner using both spatial (FOA) and non-spatial audio, and is then fine-tuned to generate high-quality spatial audio. Experimental results demonstrate that AudioSpace outperforms several baseline models.\n\n## Update after rebuttal\nAfter the discussion, I raised my rating since the authors addressed my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our motivation and demo, as well as acknowledging our experimental validations. We hope our response addresses all your concerns and questions.\n\n## Updated Table 2 and Inference Latency\n\nWe sincerely apologize for the typos in Table 2 in the submission. Please check our **Response to Reviewer Xvna** under **Claims that AudioSpace achieves SOTA performance**  As shown in the updated Table 2, **AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test.** We also added the number of parameters and inference time into comparison. Although AudioSpace has slightly more parameters than SOTA MMAudio+AS, **it achieves notably faster inference than all baselines.**\n\n## Combining Equirectangular Projection with Multiple FOV Cuts\n\nWe are grateful for your suggestion regarding the dual-branch approach and its comparison with dual-branch variants. Our dual-branch design employs a 120° frontal view to concentrate on prominent sound sources while integrating panoramic features as contextual conditioning to overcome the limitations of visual information beyond the central view. To elucidate the impact of combining equirectangular projection with multiple FOV cuts, we have assessed three innovative FOV-cut strategies to replace the local FOV video: hexadirectional 360° cuts (ERP+6cuts) which capture views from front, back, left, right, up, and down directions, quadrant cuts (ERP+4cuts) which capture views from front, left, right, and back directions, and bipolar cuts (ERP+2cuts) which capture views from front and back directions.\n\n| Dual-Branch Method| FD   | KL   | $\\Delta_{abs}\\theta$ | $\\Delta_{abs}\\phi$ | $\\Delta_{Angular}$ |\n|--------------------|------|------|----------------------|--------------------|--------------------|\n| ERP+Front          | **88.30** | **1.58** | **1.36** |0.52 | 1.28 |\n| ERP+2FOV Cuts      | 95.84 | 1.77 |  1.37  |  0.55 | 1.30 |\n| ERP+4FOV Cuts      | 92.89 | 1.65 |1.36| **0.51** | 1.27 |\n| ERP+6FOV Cuts      | 90.16 | 1.59 | 1.37 | 0.52 | **1.26**   |\n\nThe results show that **our dual-branch strategy still achieves the best audio quality**, reflected in FD and KL, and **achieves similar spatial metrics compared to multiple FOV views**.\n\n## Ratio between non-spatial data and FOA data for pre-training\n\nThe pre-training corpus consists of approximately 2M samples from general non-spatial audio datasets (detailed in Section 5.1) and 100K samples from the specialized Sphere360 FOA dataset, yielding a ratio of 20:1 between non-spatial and spatial audio data. Table 3 shows that coarse-to-fine pre-training substantially improves FD and KL than pre-training with non-spatial data only or FOA data only, confirming that general audio tasks effectively complement the specialized FOA domain.\n\n## Open-Source Sphere360 Dataset\n\nWe fully agree that transparency and reproducibility are critical for the research community.\n\n1. Open-Source Release: Sphere360 dataset with metadata (including all `youtube_id` identifiers and timestamps) and semi-automated construction pipeline are released at https://anonymous.4open.science/r/Sphere360-CF51/.\n\n2. Legal-Compliant Data Access\n   Due to YouTube's Terms of Service restrictions, we cannot directly redistribute original videos. Instead, we provide a version-controlled crawling script that reconstructs the raw dataset using the provided `youtube_id` list.\n\nThis framework ensures that researchers can fully reconstruct our dataset while complying with content distribution policies. We will refine the documentation based on community feedback to ease the procedure.\n\n## More Different Audio Events in Appendix\n\nThank you for the valuable suggestion. We will expand Appendix by incorporating additional examples of diverse audio events to better demonstrate the coverage and diversity of our dataset. \n\n## More Detailed Figure for Dual-Branch Architecture\n\nThank you for highlighting this critical architectural detail. **We will revise Figure 10 to explicitly illustrate the dual-branch fusion process** with the following additions:\n\n1. Local Feature Pathway (FOV-Audio Alignment)\n   * Visual FOV patches are processed through a linear layer to align their dimensionality with audio latents.\n   * The adapted FOV features are directly combined with audio latents via element-wise addition before entering the Diffusion Transformer, ensuring pixel-level spatial correspondence.\n\n2. Global Context Pathway (360° Scene Guidance)\n   * The full 360° video features are condensed into a global descriptor using max-pooling, capturing holistic scene semantics.\n   * This global descriptor is fused with the diffusion timestep embedding through element-wise addition, providing consistent scene-level conditioning across all transformer layers.", "claims_and_evidence": "The claims made by the authors are largely supported by the experiments on Sphere360 and YT360. They demonstrate meaningful improvements in various spatial audio metrics (like DoA errors) as well as in perceptual studies (MOS scores). The large scale of their new dataset and the multiple comparisons to baselines also strengthen their evidence.", "methods_and_evaluation_criteria": "The approach focuses on a flow-matching generative model combined with a two-stage training pipeline. They evaluate the generated audio via both objective metrics (e.g., FD, KL divergence, DoA angle errors) and subjective ratings (MOS-SQ for spatial audio quality and MOS-AF for video alignment). These metrics are highly relevant for audio generation tasks. The evaluation on both Sphere360 and an out-of-distribution set (YT360) is also a good indicator of generalization.", "theoretical_claims": "I did not find major issues with the theoretical discussion -- it is mostly referencing known generative modeling frameworks.", "experimental_designs_or_analyses": "The experiments are well-structured, with a thorough comparison to baselines. The results consistently favor the proposed approach. However, I think more analysis would help clarify a few points:\n1. The dual-branch approach is compared against single-branch variants (Section 5 and Table 4), but I would like more clarity on whether combining equirectangular with multiple FOV “cuts” (instead of just one) provides additional improvement.\n2. The ratio between non-spatial data and FOA data for pre-training could be further detailed, helping readers understand how general audio tasks complement the specialized FOA domain.", "relation_to_broader_scientific_literature": "This paper advances video-to-audio generation by focusing on FOA audio from 360-degree sources, which has a meaningful place in multi-modal generative modeling.", "essential_references_not_discussed": "I think the references cited are comprehensive.", "other_strengths_and_weaknesses": "I like this paper's motivation and demo, here're some additional weaknesses:\n1. I think adding more detail about the variety of ambient scenes or specific events in Sphere360 in the Appendix would help demonstrate coverage.\n2. The authors briefly mention the dual-branch system (Section 4.3), but I found the exact architectural interplay between the local and global features a bit missing. A more detailed figure of how these features combine could help.\n3. The authors should consider discussing the inference latency of this model, as it is an important factor for AR/VR applications.", "other_comments_or_suggestions": "None.", "questions_for_authors": "See the content in the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uj8jJY0CgT", "reviewer": "Reviewer_cQFE", "summary": "This work created Sphere360, a real-world dataset for realistic 3D audio reproduction. An efficient sem-automated pipeline for collecting and cleaning paired video-audio data is established. The challenges of the created task are clearly described. The demos are interesting. Code and datasets will be made publicly available.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our demo page and our dataset and all your valuable feedback. We plan to open-source the codebase in April 2025 to facilitate community-driven improvements in this direction and welcome the reviewer's specific recommendations on cutting-edge techniques worthy of prioritized exploration. We hope our response fully addresses your concerns and questions.\n\n## Open-Source Sphere360 Dataset\n\nPlease check our **Response to Reviewer nf2N** under **Open-Source Sphere360 Dataset**.\n\n## Updated Table 2 with Model Size, Inference Time, and TFLOPS\n\nWe sincerely apologize for the typos in Table 2 in the submission. Please check our **Response to Reviewer Xvna** under **Claims that AudioSpace achieves SOTA performance** As shown in the updated Table 2, **AudioSpace indeed achieves state-of-the-art performance on both in-domain Sphere360 and out-domain YT360-Test.** We also added the number of parameters, inference time, and TFLOPS into comparison. Although AudioSpace has slightly more parameters than MMAudio+AS, **it achieves notably faster inference than all baselines.**\n\n## Extended V2A Validation\nWe further compared AudioSpace with baselines on the perspective video-to-stereo audio (V2A) generation task on VGGSound. Under the same multimodal setting as MMAudio (i.e., + Text modality), **AudioSpace+Text notably outperforms SOTA MMAudio and sets new SOTA performance on the traditional V2A task**, in addition to the new 360V2SA task.\n\n| Model|Params|FD|KL|\n|-------------------------|--------|--------|--------|\n| Diff-Foley              | 859M   | 324.88 | 3.03   |\n| Seeing-and-Hearing      | 415M   | 261.60 | 2.30   |\n| Frieren                | 159M   | 80.69  | 2.83   |\n| MMAudio                 | 1.0B   | 43.26  | 1.56   |\n| AudioSpace (ours)       | 1.2B   | 34.56  | 1.64   |\n| **+ Text modality** | 1.2B   | **33.24** | **1.40** |\n\n## More Ablation Study on Dual-Branch Design\nThank you for the insightful suggestion for more FOV settings. We attach more dual-branch experimental results in Response to Reviewer nf2N under [Combining Equirectangular Projection with Multiple FOV Cuts]. **They further demonstrate that our \"Frontal + ERP\" dual-branch design outperforms other dual-branch variants.** We will add these experimental results and analyses to the revised paper.\n\n## Limitation and Future Work\n\nThank you for the kind reminder. Please refer to Appendix F for discussions of limitations and future work.\n\n## Related Works on Panoramic Video Processing\n\nWe greatly appreciate your guidance in strengthening the contextualization of our work. In the revised paper, we will add discussions on advancements in panoramic video processing into Section 2 Related Work.\n\n## User Study\n\nIn this work, we employ human evaluation based on the Mean Opinion Score (MOS) to quantitatively assess both spatial audio quality (MOS-SQ) and video-audio alignment faithfulness (MOS-AF), as detailed in Section 5.1. Additionally, we present a comprehensive case study in Section 5.3 to comparatively analyze the generated outputs of AudioSpace against baseline methods. Supplementary qualitative results are included in Appendix H. We will further expand it with additional case studies in the revised paper. To enhance reproducibility and community accessibility, we are planning to deploy an online interactive demo platform via Gradio and Hugging Face Spaces for real-time generation and evaluation.\n\n## Usage of Terminology\n\nThank you for this suggestion. We agree that \"perspective video\" more accurately conveys the conventional narrow-FOV nature of the inputs compared to panoramic formats. In the revised paper, we will replace all instances of \"FOV video\" with \"perspective video\", and add a footnote in Section 3.1 clarifying that \"Perspective video\" denotes standard rectilinear projections with ≤120° horizontal FOV, distinct from 360° equirectangular formats.\n\n## The Integration of more recent Flow-matching strategies\nWe sincerely appreciate the reviewer's constructive suggestion regarding flow-matching strategies. The proposed coarse-to-fine self-supervised pre-training and dual-branch design are agnostic to flow matching strategies.\nOur current framework achieves state-of-the-art performance while maintaining the fastest inference speed in both 360V2A and V2A tasks. While our experiments demonstrate that the existing configuration optimally balances accuracy and efficiency for target scenarios, we fully agree that deeper integration of advanced flow matching strategies could unlock further theoretical insights.  \nWe plan to open-source the codebase in April 2025 to facilitate community-driven improvements in this direction, and welcome the reviewer's specific recommendations on cutting-edge techniques worthy of prioritized exploration.", "claims_and_evidence": "Yes. The dual-branch framework which utilizes panoramic and FoV video inputs is verified on the created Sphere360 dataset. The claimed contributions are well supported.", "methods_and_evaluation_criteria": "Yes. The created Sphere-Bench provides a credible benchmark.", "theoretical_claims": "Yes. The usage of the equirectangular representation and the extraction of the FoV video follow standard projections.", "experimental_designs_or_analyses": "Yes. The experimental results and comprehensive analyses provide evidences of the effectiveness of the proposed solution.", "relation_to_broader_scientific_literature": "Yes. The method is relevant to autonous driving and augmented reality.", "essential_references_not_discussed": "No. The related work analysis is comprehensive.", "other_strengths_and_weaknesses": "1. If it is possible, please consider conducting some experiments with different FoV setups for comparison to help understand the most suitable configuration of the framework depicted in Fig. 2.\n2. The computational complexity results like FLOPs/MACs, the number of parameters, and the training/inference time of the proposed method could be presented to help understand the efficiency of the proposed solution.\n3. It would be nice to discuss some limitations of the presented work and point out some future work directions.\n4. In the related work section, it would be nice to also discuss some research works on panoramic video processing such as panoramic semantic segmentation and panoramic generation, etc.\n\nMost of the concerns have been addressed in the rebuttal. The reviewer would like to maintain the positive rating.", "other_comments_or_suggestions": "If it is possible, please consider conducting a user study to compare the quality of the generated spatial audio data.\n\nThe term of \"FOV video inputs\" could be revised to \"perspective video\". FOV means the field of view, which could also be associated to panoramic data with a large FOV.", "questions_for_authors": "Would you consider your proposed method with more recent flow-matching strategies to help better understand the effectiveness of your solution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received consistently positive feedback following the authors’ rebuttal. Reviewers highlighted the paper's experimental validation, and the quality of the results with the high-quality demo (cQFE, nf2N, MEu5). While there were initial concerns about the comparison setup and presentation errors in the table, these were effectively addressed in the rebuttal, leading to stronger overall reviews.

After carefully reviewing the paper, the reviews, and the rebuttal, The AC concurs with the reviewers' positive consensus and recommends acceptance of this paper.

For the camera-ready version, the authors should integrate all discussions from the rebuttal into the main paper and supplementary materials. Specifically, the authors should implement the following changes:
1. Manuscript Revisions: typos and boldfacing errors in the tables (nf2N, Xvna). The usage of terminology and more ablations on the dual-branch design and validations (cQFE). More different audio events in appendix (nf2N).
2. Inference Code Release
3. Enhanced Dataset Documentation
Decision: Accept (poster)