TAS: Personalized Text-guided Audio Spatialization

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-16 10:17:18

Abstract: Synthesizing binaural audio according to personalized requirements is crucial for building immersive artificial spaces. Previous methods employ visual modalities to guide the spatialization of audio because it can provide spatial information about objects. However, the paradigm is dependent on object visibility and strict audiovisual correspondence, which makes it tough to satisfy personalized requirements. In addition, the visual counterpart to the audio may be crippled or even non-existent, which greatly limits the development of the field. To this end, we advocate exploring a novel task known as Text-guided Audio Spatialization (TAS), in which the goal is to convert mono audio into spatial audio based on text prompts. This approach circumvents harsh audiovisual conditions and allows for more flexible individualization. To facilitate this research, we construct the first TASBench dataset. The dataset provides a dense frame-level description of the spatial location of sounding objects in audio, enabling fine-grained spatial control. Since text prompts contain multiple sounding objects and spatial locations, the core issue of TAS is to establish the mapping relationship between text semantic information and audio objects. To tackle this issue, we design a Semantic-Aware Fusion (SAF) module to capture text-aware audio features and propose a text-guided diffusion model to learn the spatialization of audio, which can generate spatial audio consistent with text prompts. Extensive experiments on TASBench compare the proposed method with several methods from related tasks, demonstrating that our method is promising to achieve the personalized generation of spatial sense of audio under text prompts.

Main paper: 
 introduction: 
 Generating music, speech, and sound effects based on personalized requirements [3, 22, 39] is critical for applications such as augmented reality, virtual reality, game development, and video editing [6, 36] . In recent years, audio generation research has made surprising progress [13, 28, 30, 34] . However, existing studies mostly focus on the semantic generation of audio and ignore the significance of audio spatiality. Due to the binaural structure, human hearing conveys critical spatial information [4] . We can easily identify the approximate location of a sound source and even track it without relying on vision.An artificial head prosthesis records realistic binaural audio to provide users with a rich listening experience. However, binaural audio recording is expensive and relies on professional equipment [32] . Even though the device has been around for many years, binaural audio is still scarce. It is wise to eschew hardware and design algorithms to obtain binaural audio. Therefore, converting mono audio to binaural audio has attracted increasing attention [18, 20, 21] . Binaural audio with real human hearing experience can provide users with an immersive artificial 3D experience and has broad application prospects [1] .Recent work attempts to exploit visual modalities to synthesize binaural audio from mono audio [8, 20, 23, 38, 40] . Ideally, it works because the visual modality contains the spatial information of the objects required for audio spatial recovery. However, relying on visual modality requires two prerequisites: (i) The visual and audio modalities must be strictly time-synchronized; (ii) The sounding object must be visible. Object visibility guarantees may be contradictory. To include the object, the field of view needs to be as wide as possible. However, a larger field of view introduces more irrelevant objects and background while narrowing the object. In the case of audiovisual synchronization, we summarized 5 common situations of visual guidance, including invisibility, occlusion, interference, silence and their combinations, as shown in Figure 1 (a). Therefore, it is very challenging to exploit visual modalities to generate customized spatial audio.The natural defects of visual guidance motivate us to search for a new way to synthesize spatial audio. Recently, text promptbased generative methods have achieved impressive performance, such as audio style conversion [14, 17, 26 ], text-to-speech synthesis [2, 9, 31] , etc. Generative models demonstrate powerful language understanding and generation performance. Inspired by this, we made the first attempt in this field to reconstruct the spatial sense of audio using text prompts, as shown in Figure 1 (b). Our approach circumvents the pitfalls of visual guidance and allows for greater personalization. In addition, it is capable of serving as a promising remedy when visual guidance methods fail.In this paper, we raise the issue of text-guided audio spatialization, which requires models to generate spatially consistent spatial audio from mono audio based on text prompts. To facilitate this research, we present the TASBench dataset, the first dense frame-level annotated spatial audio dataset that provides textual annotations for the spatial locations of sounding objects. The text prompt contains multiple sounding objects and different spatial locations. To connect the semantic information in the text prompts with the sounding objects in the audio, we design a Semantic-Aware Fusion (SAF) module to capture text-aware audio features. Then, we propose a new text-guided audio spatialization method as a first baseline for the TAS task. The method follows the probabilistic diffusion model structure, using mono waveforms and text prompts as guiding conditions to generate differential waveforms. We conduct extensive and rich experiments to verify that the proposed method and module are practicable and effective. Experimental results show that it is prospective to convert mono audio into spatial audio using text prompts. In general, the contributions of the paper can be summarized as follows:1) We explore a novel text-guided audio spatialization task, and introduce TASBench, a new dataset that provides dense frame-level spatial location annotations of sounding objects for TAS.2) We design a semantic-aware fusion module to establish the perceptual relationship between text and audio modalities and propose a text-guided diffusion model for spatial audio generation.3) Extensive experimental results demonstrate the feasibility and potential of our text-guided audio spatialization method. Comparisons with several methods from related tasks demonstrate the superiority of the proposed method. 
related work 2.1 text-guided audio generation: 
 By integrating text and audio, we can mine diverse audio information and expand the use scenario of audio [16] . Recently, with the vigorous growth of deep generative models, text-guided generation has been intensively investigated in the audio field [29] .A prevalent text-guided generation task is audio style conversion, which seeks to convert only specific attributes while leaving other properties unaltered, such as speech conversion [14, 16] , emotion conversion [10, 41] , etc. Le et al. [16] proposed a large-scale speechtext guided generation model. It is based on non-autoregression and can achieve zero-shot style conversion within a single or crosslanguage. Guo et al. [10] proposed an emotion-controllable speech generation model in which the intensity of emotion is guided by a soft label. Another common task is text-to-speech synthesis, which attempts to synthesize a target audio-style speech within a given text. Kreuk et al. [15] proposed an autoregressive audio generation model to generate high-quality audio under conditional and unconditional settings. However, it is difficult for autoregressive methods to achieve fine-grained control of audio properties. Huang et al. [12] proposed a diffusion model with prompt enhancement, which utilizes latent diffusion mechanisms and autoencoders to generate audio. Yang et al. [35] explored text-to-audio synthesis based on a discrete diffusion process that handles audio codes in VQ-VAE [33] , employing masked text generation with CLIP [24] representation. To solve the quality problem caused by quantization, Shen et al. [29] proposed an audio codec based on a residual vector quantizer, and then utilized a diffusion model to synthesize latent vectors based on text prompts. Huang et al. [11] proposed a diffusion probabilistic method for generating music from text prompts that outperformed methods conditioned on simple music attributes. Currently, most text-guided audio generation mostly focuses on the mono audio level and ignores the spatial properties of audio.Reconstructing the spatial properties of sound from mono audio has received long-standing attention. Traditional methods utilize linear mapping relationships to convert mono audio to spatial audio, making it difficult to simulate realistic non-linear scenes [7, 37] . Therefore, synthesizing spatial audio using the nonlinear modeling capabilities of deep learning has received widespread attention [38, 40] . Due to the strong correlation between vision and hearing, it is natural to associate spatial audio generation with visual modality. Visually guided audio spatialization has been extensively studied in … … … … … … … … … … … … … … … … … …N (0,1) recent years [8, 18] . In the pioneering approach, Gao et al. [8] proposed a visually guided UNet-structured codec to generate binaural audio via spectrograms. Next, Zhao et al. [40] introduced sound source separation into binaural audio generation. The method improves spatialization performance by incorporating both tasks into a unified training framework through a shared visual network. However, this paradigm makes binaural audio generation subject to additional source-separated datasets and models. Li et al. [18] proposed a generative adversarial approach to synthesize binaural audio. The visual modality provides guidance information to the generator while providing visual reference to the discriminator. Although these existing audio spatialization methods can generate spatial audio that corresponds to visual content, they rely on strict audiovisual correspondence and have limited flexibility. Different from the above methods, we explore a new task of textguided audio spatialization and propose a text-guided diffusion model to reconstruct the spatial sense of audio. Experimental results demonstrate that our approach can generate spatial audio that is consistent with text prompts and outperforms some visually guided methods. 
method: 
 To address the TAS issue, we present a text-guided audio spatialization method based on the diffusion model to realize audio spatialization reasoning guided by text prompts. Our method recovers differential audio from random noise conditioned on mono audio and text prompts. Mono audio provides the original signal basis, while text prompts provide spatial information during generation. Figure 2 shows the pipeline illustration of the proposed method.Given a mono audio 𝐴 𝑚 and a text prompt 𝑇 , our model aims to synthesize the binaural audio 𝐴 𝑏 = {𝐴 𝑙 , 𝐴 𝑟 } corresponding to the text prompt. 𝐴 𝑚 can be obtained by mixing channels of binaural audio 𝐴 𝑏 :EQUATIONFor simplicity, instead of directly synthesizing the binaural audio itself, we predict the differential audio 𝐴 𝑑 of the binaural audio:EQUATIONTo convert the differential audio waveform distribution 𝑞(𝐴 𝑑 ) into a Gaussian noise distribution N (0, I), we diffuse it via 𝑁 steps in the diffusion process. 𝑎 is used to represent 𝐴 𝑑 for convenience. Then, the diffusion process can be described as:EQUATIONIn the reverse process, our diffusion model with parameter 𝜃 generates clean differential audio from Gaussian noise 𝑝 (𝑥 𝑇 ) ∼ N (0, I) guided by both mono audio 𝐴 𝑚 and text prompts 𝑇 . The reverse process can be expressed as:EQUATIONFinally, the generated differential audio 𝐴 ′ 𝑑 is computed with the mixed audio 𝐴 𝑚 to obtain the final binaural audio 𝐴 ′ 𝑏 = {𝐴 ′ 𝑙 , 𝐴 ′ 𝑟 }: Text Encoder: For input prompts, the text encoder of pre-trained CLIP [24] is employed to extract text features 𝑓 𝑡 ∈ R 𝑑×1 , where 𝑑 = 512 is the embedding dimension. We also utilize diverse text encoders to extract text features to verify the impact of text representation on audio spatialization, including BERT [5] and T5 [25] (introduced in Sec. 4.6). We perform average pooling on the embedding representations of all tokens to obtain the information of the entire text prompt in BERT and T5. A learnable fully connected layer is used to reduce the feature dimension of BERT from 768 to 512. Following the common practice, the weights of all text encoders are frozen during training. Semantic-Aware Fusion Module: To highlight key audio semantic information closely related to text prompts, we propose a semanticaware fusion module. Specifically, given text features 𝑓 𝑡 and audio features 𝑓 𝑎 , the semantic-aware fusion module learns to aggregate text-aware audio features 𝑓 ′ 𝑎 . The text-aware audio features can be computed as:EQUATION)EQUATIONThen, the text features are fused with text-aware audio features:EQUATIONwhere ⊙ represents the matrix dot product, and FC represents the fully connected layer. Finally, 𝑓 ′ 𝑎𝑡 is repeated 𝐿 times to splice with audio features and fused through a convolutional layer to obtain the final aggregated text-aware audio features 𝑓 𝑎𝑡 :EQUATIONwhere  
experiments: 
 We first introduce the construction of TASBench, implementation details, and evaluation metrics in this section. Then, we conduct comparative experiments between the proposed baseline and methods from related tasks. We also provide experimental results of several task-related methods guided by text prompts. In addition, we compare the impact of different text representations on audio spatialization and verify the personalization generation capability of the proposed method. Finally, we analyze the proposed method in ablation experiments and user study.Dataset Statistics: We build the first text-guided audio spatialization dataset to facilitate related research. A straightforward idea is to provide only one text annotation for an audio clip. In contrast, we provide dense spatial location annotations for sounding objects in spatial audio, which allows us to engage in fine-grained control of spatial sense generation. All spatial audio is collected from the FAIR-Play dataset [8] , which was recorded in a music room with professional equipment. The TASBench dataset contains 1871 10-second audio clips and 190625 text annotations over 12 categories, covering sounds from musical instruments, humans, and their various combinations. Figure 3 shows the specific categories Annotation: To better annotate the spatial location of sounding objects, we combine spatial audio and video provided by the FAIR-Play dataset. Specifically, we first extract visual images from each video at 10 frames per second. We then provide a spatial location annotation for each sounding object in each image frame based on its visual and auditory spatial location. There are some special cases in the annotation process: (i) Audio spatial perception may not exactly match the visual situation when the sounding object is visible. In this case, we utilize visual information as the main annotation to faithfully reflect the spatial position of the sounding objects. (ii) We utilize spatial audio as the main annotation object when the sounding object is invisible. Since the FAIR-Play dataset lacks specific spatial location records, the spatial location of the sounding object can be roughly divided into three directions: Left, Center, and Right. We use text to annotate the spatial location and status of the sounding objects in order from left to right. We only annotate sounding objects. When the object is completely silent, it will not appear in our annotations. Figure 4 shows some samples of the constructed dataset.Dataset Settings: We train and test the proposed method and other methods from related tasks on the constructed TASBench dataset. We retain the original partition of the dataset, which is 1497/187/187for training, validation and testing. The average of 10-fold crossvalidation of the model is used as the final result. We randomly intercept a 1-second clip from a 10-second audio sample with a sampling rate of 16kHz. The text closest to the timestamp in the center of the clip is chosen as the input prompt.Training and Inference: The batchsize of the model is 12 and the maximum number of training steps is 3000. The proposed model is optimized using Adam with a learning rate of 2e-4 during training.We utilize a window with a hop size of 0.1 seconds to generate audio during inference.Metrics: Six evaluation metrics are employed to comprehensively evaluate generated binaural audio, including STFT Distance [8] , Envelope (ENV) Distance [19] , Wave L2 (WAV×10 -3 ) [27] , Amplitude L2 (AMP), Phase L2 (PHA), and Signal-to-Noise Ratio (SNR) [21] . See supp. for details. Methods from Related Task: We compare the proposed method with several visually guided methods from related tasks, including weakly semi-supervised method: L2BNet [23] ; autoencoder-based methods: MONO2BINAURAL [8] , AVSN [20] ; multi-tasking-based methods: APNet [40] , Sep-stereo [40] ; attention-based methods: Main network [38] , Complete network [38] ; GAN-based method: Table 3 : The ablation results of our approach on the split1 segmentation of TASBench dataset.SAGM [18] . Mono-Mono represents fake binaural audio created by channel duplication of mono audio.To investigate the impact of different guidance modalities on audio spatialization and prove the superiority of our method, we organize extensive quantitative experiments. Table 1 demonstrates the quantitative results under different guidance. Mono-Mono is audio without modal guidance, which has no spatial awareness. The improved evaluation metrics show that both our method and other methods can generate audio with spatial awareness. As can be seen, our method even outperforms some visually guided methods. This is challenging because most of the visual images in the FAIR-Play dataset have clear visual expressions.For intuitive comparison, we replaced the visual modality in MONO2BINAURAL and APNet with text prompts and retrained it. Some metrics decreased in text guidance compared to visual guidance in the MONO2BINAURAL method, which further supports our observations. In the text-guided method, Ours represents the baseline model without the SAF module, and Ours+SAF represents the baseline model with the SAF module. After eliminating the influence of modality, our method outperforms other methods in all metrics, indicating the superiority of the proposed model. The proposed SAF module further significantly improves the performance of our method by incorporating text-aware audio features. This demonstrates the effectiveness of the proposed module.We visualize the differential spectrogram generated by different methods in Figure 5 . The first row is the visual image corresponding to the text prompt. The second and third rows are the results generated by the visually guided method, while the fourth row is the result generated by our text-guided method. It can be seen that the time-frequency structure of the spectrogram generated by our method is comparable to the ground truth.To visually observe the similarity of the generated waveforms, we visualize the envelope curves of the left and right waveforms, as shown in Figure 6 . The warping of the waveform envelope curve generated by the proposed method follows the ground truth situation well. Compared with MONO2BINAURAL and APNet, the proposed method has better warpage peak, and the waveform shape is closer to the ground truth. Overall, our method demonstrates competitive performance with visually guided generation methods in both spectrogram and waveform domains. This demonstrates the promise of using text prompts to synthesize mono audio into binaural audio.We use existing text encoders for feature extraction on text prompts, including CLIP, BERT, and T5. CLIP is a pre-trained model obtained through contrastive learning of multimodal data and is widely used in text feature extraction. BERT is a large language model pre-trained on a large amount of plain text corpus. Compared with BERT, T5 is a more versatile large language model that can adapt to a variety of natural language processing scenarios. Table 2 shows the quantitative results of binaural audio generated based on different text representations. It can be seen that the text representation of CLIP is superior to other methods in most metrics, demonstrating its potential in audio spatialization tasks.To evaluate the personalization capability of the proposed method, we conduct personalization experiments. Specifically, given a mono audio, we then feed it to our model with text prompts of different spatial senses. Finally, the difference curve of the sound pressure level is employed to visualize the spatial variations of the audio.The SPL represents the difference in sound pressure level between the left channel and the right channel of audio. The spatial sense is biased to the left when SPL > 0, to the right when SPL < 0, and toward the center when SPL is near 0. The personalization results of our method are demonstrated in Figure 7 . The purple line is the visualization result of the ground truth, and the sense of space is biased to the left. It can be seen that the spatial sense of the audio generated by our method is comparable to the ground truth (blue line). When the model is provided with different text prompts (orange and green lines), it can generate audio consistent with the text prompts, demonstrating the personalization capabilities of the proposed method.We implement ablation experiments to demonstrate the impact of text modality and fusion methods on audio spatialization. Table 3 shows the results of the ablation experiments. The w/o Modality represents Mono-Mono, which is obtained by channel duplication of mono audio, and it has no spatial information. Audio-Only means using only mono audio as guidance without text prompts. Interestingly, most metrics of Audio-Only still improve without spatial information guidance. The reason may be that the model can learn some audio with insignificant spatiality, which does not need to rely on explicit spatial information, such as audio with the object in the center. The bottom of Table 3 shows the results of audio-text joint guidance. It can be observed that the introduction of text modality further boosts the performance of the model because it allows the model to generate audio corresponding to the spatial information in the text prompt. In addition, we also compare different fusion methods of text and audio modalities, including element-wise addition, 1D convolution and the proposed SAF module. Compared to these methods, our method achieves leading performance through perceptual interaction between text and audio modalities.We conduct a user study to artificially estimate binaural audio generated by different methods. The user study recruited 8 subjects. We provide each subject with 30 sets of samples. Each set of samples contains audio generated by Mono-Mono, MONO2BINAURAL, APNet, and Ours, as well as the ground truth. The samples in each group are shuffled and named with random numbers. To ensure generalizability, all subjects were novices and were asked to give corresponding answers based on the audio and questions provided. Specifically, we use text prompts with location information removed as questions for each group of samples. Then, each subject was asked to listen to each audio one by one and given the spatial location of the sounding object, including left, center, and right. Finally, we compute the average accuracy of all participants under diverse methods as the final experimental results.In Figure 8 , we provide two levels of accuracy for user study, including rank-object and rank-audio. Rank-object evaluates the position accuracy of the sounding object from the object level. Rankaudio evaluates the accuracy of spatial perception from the entire audio level, 𝑖.𝑒., the positions of all objects need to be consistent with the ground truth. It can be seen that our approach can generate credible spatial audio, demonstrating the effectiveness of using text prompts to guide audio spatialization. Compared with other methods, the spatial audio generated by the proposed method is more realistic. 
conclusion: 
 In this paper, we explore a novel text-guided audio spatialization task, which aims to convert mono audio into spatial audio based on text prompts. To promote research in this area, we construct the first dense text-annotated spatial audio dataset, named TASBench. We design a semantic-aware fusion module to capture text-aware audio features to enhance the correlation between text and audio features and propose a new text-guided audio spatialization model as a first baseline for this task. We compare the proposed method with several visually guided methods from related tasks and demonstrate that the proposed method can establish a connection between text prompts and the spatial perception of audio. Personalization experiments demonstrate that the proposed method can generate audio with a specified spatial sense based on text prompts. In the future work, we believe this research will facilitate multimodal audio spatialization, 𝑖.𝑒., generation under the joint guidance of text, audio, and visual modalities.  
: 
 the core issue of TAS is to establish the mapping relationship between text semantic information and audio objects. To tackle this issue, we design a Semantic-Aware Fusion (SAF) module to capture text-aware audio features and propose a text-guided diffusion model to learn the spatialization of audio, which can generate spatial audio consistent with text prompts. Extensive experiments on TASBench compare the proposed method with several methods from related tasks, demonstrating that our method is promising to achieve the personalized generation of spatial sense of audio under text prompts. 
 
 Appendix:  


Decision: The paper proposed a novel framework for text-guided audio spatialization, and contributed a TASBench dataset with dense annotation. Although there are certain limitations such as generalization issue to real-world scenarios, most reviewers agreed that the strength of the paper outweighs the weakness.

Reviews: review: paper_summary: paper_summary: The paper introduces Text-guided Audio Spatialization, a novel task for converting mono audio into spatial audio using text prompts. The authors highlight the limitations of visually guided audio spatialization, which requires strict audiovisual synchronization and object visibility. To address this, they propose a method that leverages text prompts for greater personalization and flexibility. The paper also presents TASBench, the first dataset with dense frame-level annotations for audio spatialization tasks. Experimental results demonstrate the effectiveness of the proposed method, showing it outperforms some visually guided approaches and can generate personalized spatial audio based on text inputs.
strength_weakness: strength_weakness: 1. The paper introduces a new task text-guided audio spatialization, which is interensting as it provides an alternative to visually guided methods which have limitations related to object visibility and synchronization.

2. The authors collect a TASBench dataset, which is the first dataset with dense frame-level annotations for audio spatialization, and could potentially serve as a valuable resource for the research community and facilitates further research in this area.
limitations: limitations: 1. While the motivation to use text for audio spatialization is good, the current method only adopts very coarse spatial resolutions, annotating sound source locations as left, center, and right. This approach overlooks intermediate positions that could be more precisely captured with visual guidance. The text-based approach needs a better definition of spatial resolutions to avoid limiting the task's contribution and the model's performance.

2. The proposed text-guided spatialization method performs worse than many visually guided counterparts, raising questions about its effectiveness. This disparity may be due to the coarse spatial resolution in the text annotations or other factors. The authors should provide a more detailed justification and analysis to address this concern.

3. The paper would benefit from a case study demonstrating scenarios where visually guided methods fail due to occlusion, interference, or other issues, and where the text-guided method succeeds. This would highlight the advantages of the text-guided approach in specific challenging situations.
paper_summary: paper_summary: The paper explores the task of converting mono audio into spatial audio by providing the audio and text prompts as input, defined as Text-guided Audio Spatialization (TAS). This task aims to provide a more flexible and personalized approach compared to previous visually guided methods that rely on strict audiovisual data. The audiovisual methods are limited when not strictly time-synchronized or if audio sources are not visible. The paper addresses the task extensively by presenting methods based on machine learning to accomplish the tasks, as well as a dataset for training and evaluation.
strength_weakness: strength_weakness: * The paper presents a novel approach to generating spatial audio.
* The paper contributes a dataset, TASBench Dataset, a fusion module Semantic-Aware Fusion (SAF) module and a text-guided diffusion model. The dataset is used to train and evaluate the model.
* The author(s) present a theoretical concept together with a technical implementation and extensive evaluation, including ablation study and user study.
* Overall the paper is well written and fits the conference topics.
limitations: limitations: * The dataset is based on the FAIR-Play dataset, which includes music instruments only, is recorded in a controlled environment, and has an uneven distribution. Future work could evaluate the presented models on more complex sounds.
* The paper does not provide information about the performance in terms of the speed of generating the spatial audio.
paper_summary: paper_summary: The paper presents TAS, a novel framework for text-guided audio spatialization, aiming to convert mono audio into spatial audio based on text prompts. Unlike traditional methods that rely on visual modalities for spatial information, TAS leverages textual descriptions to achieve greater flexibility and personalization in audio spatialization. The authors introduce TASBench, a new dataset with dense frame-level annotations of spatial audio, which enables fine-grained spatial control. The core contributions include a Semantic-Aware Fusion (SAF) module to integrate text and audio features and a text-guided diffusion model to generate spatial audio consistent with the text prompts. The method is evaluated extensively on the TASBench dataset, showing promising results compared to existing approaches.
strength_weakness: strength_weakness: 1. Novel Approach: The shift from visually guided to text-guided audio spatialization is innovative. This approach circumvents the limitations of visual guidance, such as dependency on object visibility and strict audiovisual correspondence.
2. Dataset Contribution: The introduction of TASBench, a densely annotated spatial audio dataset, is a significant contribution. It provides a valuable resource for future research in text-guided audio spatialization.
3. Semantic-Aware Fusion Module: The SAF module is a key innovation that effectively integrates text and audio features, enhancing the model's ability to generate spatial audio that aligns with textual descriptions.
4. Diffusion Model: The use of a text-guided diffusion model for generating differential waveforms adds robustness and precision to the spatial audio generation process.
5. Comprehensive Evaluation: The paper includes extensive experiments, including quantitative and qualitative analyses, ablation studies, and user studies. These evaluations demonstrate the method's effectiveness and potential for personalized audio spatialization.
limitations: limitations: 1. Generalization to Other Datasets: The evaluation is primarily conducted on the TASBench dataset. Additional experiments on other datasets would help validate the generalization capabilities of TAS.
2. Complexity: The model introduces several novel components, which adds to the implementation complexity. This might pose challenges for practical deployment and requires substantial computational resources.
3. Parameter Sensitivity: The performance of the model is sensitive to the choice of hyperparameters, such as the weights in the loss function. This sensitivity requires careful tuning and may affect the model's robustness.
4. Limited Real-World Scenarios: While the TASBench dataset provides a controlled environment for evaluation, real-world scenarios with more complex and diverse audio-visual contexts could provide a more rigorous test of the method's capabilities.
paper_summary: paper_summary: Text-guided audio spatialization (TAS) is indeed a novel idea, however, the manuscript still lacks much.
strength_weakness: strength_weakness: As mentioned in the manuscript, this work advocates exploring a novel task called text-guided audio spatialization (TAS), which aims to convert mono audio into spatial audio based on text prompts. This is a change from the previous fixed idea of using visual modalities to guide the audio spatialization. In order to build the mapping relationship between text semantic information and audio objects, a semantic-aware fusion (SAF) module is designed in the manuscript to capture text-aware audio features and learn the audio spatialization using the diffusion model, which is also different from the previous practice of using convolutional networks for audio spatialization. In addition, this work creates a TASBench dataset that provides a dense frame-level description of the spatial location of sounding
objects in audio, providing data support for text-guided audio spatialization.
limitations: limitations: Firstly, in the description of the method:

Q1: The emphasis of the method presentation is not satisfactory. Some common knowledge aspects of audio spatialization, such as the processing of the input audio and the principles of the diffusion model, are over-introduced, while the proposed text-guided audio spatialization method is only briefly introduced. In particular, the description of the Semantic-Aware Fusion (SAF) module, which is summarized as one of the innovations in the contribution.
(1)	Equations (6)-(8): Missing description of the parameters involved in the equations.
(2)	The parameters labeled in Figure 2(b) cannot correspond to Equations (6)-(8) in the manuscript. Is it true that Figure 2(b) has two parameters, the letter l and the number 1, or has it been plotted incorrectly?

Q2: Spatialization Network: The diffusion process as well as the output of the SAF are redundantly described without presenting a clear presentation of how this spatialized network handles these two inputs? What is the relationship between the two inputs? Sample concat or other processing?

Secondly, for the new dataset TASBench:

Q3: Except for the expanded categories of “Woman” and “Man”, the remaining category names are not fully aligned to the FAIR-Play dataset. Also, the FAIR-Play dataset involves some non-common instruments or non-instruments, such as wooden boxes that can be struck and bell-like objects that can be shaken to make them sound; how are these types of videos labeled?

Q4: What is the role of the statistics on the number of sound sources mentioned in Figure 3 in the process of training or testing? Which aspect does the phrase “Overall, the distribution of the TASBench dataset is reasonable.” (Section 4.1 Dataset Statistics) mean?

Q5: Section 1 mentions: “In the case of audiovisual synchronization, we summarized 5 common situations of visual guidance, including invisibility, occlusion, interference, silence and their combinations, as shown in Figure 1 (a)” However, the Section 4.1 does not explicitly describe how the dataset solves these problems. For example, how does it avoid interference from identical instruments? (e.g., the presence of guitars around guitars as well.)

Q6: Is the annotation manual or model-based? If it is model labeling, are there any metrics to judge whether the labeling is correct or not?

Thirdly, for the experiment:

Q7: The four metrics commonly used for binaural audio are STFT,ENV, Mag, and SNR; the manuscript and supplemental materials do not explain the reasons for not using the metric Mag, and the reasons for choosing a few other new metrics (which don't seem to have much to do with the use of text or visuals)

Q8: Comparisons of binaural audio performance using text-guided instead of vision-guided could not demonstrate superiority. As shown in Table 1, some methods, e.g.[8][40], the performance instead decreases after using the text instead of the visual. The manuscript explains that it is to highlight that text guidance has some effect, but obviously does not show the level of performance of the model proposed in this work.

Some additional details:

Q9: The layout of figures and tables. A figure or table is far away from the corresponding text description. They should be repositioned to avoid page breaks.

Q10: “Dataset Settings: … We retain the original partition of the dataset,” It has been pointed out in Paper 《Visually Informed Binaural Audio Generation without Binaural Audios》 that there are some problems with the division provided by the FAIR-Play dataset. It is recommended that the new division provided in Paper 《Visually Informed Binaural Audio Generation without Binaural Audios》 be used.

Q11: Does the differential spectrogram provided in Figure 5 take into account that the generated left and right channels are reversed? In addition, the spectrogram should be plotted with the appropriate coordinate system.