Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from fMRI

## Abstract

Abstract Drawing inspiration from the hierarchical processing of the human auditory system, which transforms sound from low-level acoustic features to high-level semantic understanding, we introduce a novel coarse-to-fine audio reconstruction method. Leveraging non-invasive functional Magnetic Resonance Imaging (fMRI) data, our approach mimics the inverse pathway of auditory processing. Initially, we utilize CLAP to decode fMRI data coarsely into a low-dimensional semantic space, followed by a fine-grained decoding into the high-dimensional AudioMAE latent space guided by semantic features.
These fine-grained neural features serve as conditions for audio reconstruction through a Latent Diffusion Model (LDM). Validation on three public fMRI datasets—Brain2Sound, Brain2Music, and Brain2Speech—underscores the superiority of our coarse-to-fine decoding method over stand-alone fine-grained approaches, showcasing state-of-the-art performance in metrics like FD, FAD, and KL. Moreover, by employing semantic prompts during decoding, we enhance the quality of reconstructed audio when semantic features are suboptimal.
The demonstrated versatility of our model across diverse stimuli highlights its potential as a universal brain-to-audio framework. This research contributes to the comprehension of the human auditory system, pushing boundaries in neural decoding and audio reconstruction methodologies.

## 1 Introduction

Hearing is one of the most important senses for humans, responsible for receiving external auditory stimuli and transmitting the information to the brain for processing and understanding. Researchers aim to explore the auditory perception mechanisms of the human brain from the fields of both neuroscience and computer science [, , , , ]. A key goal is to decode neural information from the human brain and reconstruct the original stimuli.
The common brain-to-audio reconstruction tasks can be categorized into brain-to-sound task [, ] for reconstructing all natural sounds in the environment, brain-to-music task [, , ] for the music, and brain-to-speech task [, , , , , ] for the human voice, based on the different stimulus audios.

Some researchers first attempted to map brain signals to the spectrograms or mel-spectrograms of the stimulus audios using linear regression [, , , ].
Others introduce non-linear units and use simple networks such as MLP [, , ], BiLSTM [, ], Transformer [], etc. This approach can restore the detailed temporal and frequency information of the spectrogram, but the reconstructed audio lacks semantic information, especially for non-invasive brain signals.

As research progresses, researchers have found that Deep Neural Network (DNN) features are closer to neural responses in the human brain compared to artificial acoustic representations like spectrograms [, , , , ].
Therefore, researchers [, , ] first decode neural signals into DNN features as an intermediate representation and then use generative models to reconstruct the spectrogram.
The intermediate representation is typically chosen from the intermediate layers of DNN, serving as high-dimensional fine-grained features that contain both semantic and acoustic information of sound. However, directly decoding these fine-grained features is often challenging and yields limited results, especially for non-invasive neural signals.

There are also works that decode neural signals coarsely into the low-dimensional semantic space. For example, Denk et al. [] decodes neural signals into 128-dimensional MuLan embeddings and then predicts high-dimensional acoustic embeddings and generates music using MusicLM.
However, this reconstruction model relies on the generative ability of MusicLM and cannot be transferred to audio forms other than music.

Let’s turn our focus back to neuroscience. As shown in Figure 1(a), research has indicated that in the cochlea and subcortical structures of the human ear, sound is decomposed into frequency-specific temporal patterns similar to spectrograms [, , ].
Further into the cerebral cortex, the human auditory system has two information processing pathways from low-level to high-level [, , , , ].
In recent years, an increasing amount of research has found that this cortical processing hierarchy aligns with the functional hierarchy of auditory DNN [, , , , , , ].
The primary auditory cortex is more sensitive to shallow or intermediate DNN features, which represent low-level acoustic features, while the nonprimary auditory cortex is more sensitive to deep DNN features, which represent high-level semantic features.

Figure: Figure 1: (a) The hierarchical auditory processing pathway of humans. The stimulus audio is gradually decomposed into time-frequency representation, low-level acoustic features, and high-level semantic characteristics. (b) The pipeline for our coarse-to-fine reconstruction from fMRI. Brain activity is decoded progressively into semantic, acoustic, and spectrogram levels, ultimately resulting in reconstructed audio.
Refer to caption: x1.png

Inspired by the acoustic-to-semantic stream, we model each physiological structure of the auditory processing pathway and propose an opposite coarse-to-fine audio reconstruction method, as shown in Figure 1(b). We use non-invasive fMRI as the neural signal.
First, a coarse-to-fine brain decoding process is conducted. We decode fMRI data into the low-dimensional CLAP [] space to obtain coarse-grained semantic features, and then under the guidance of these semantic features, we decode the fMRI data into the high-dimensional AudioMAE [] latent space to obtain fine-grained acoustic features.
Next, we use the decoded fine-grained neural features as conditions to reconstruct the mel-spectrogram using a Latent Diffusion Model (LDM) [], and then restore the stimulus waveform using a Vocoder [].

We validate our approach on three publicly available fMRI datasets: Brain2Sound, Brain2Music, and Brain2Speech. The results demonstrate that coarse-to-fine decoding outperforms directly fine-grained decoding in both details and semantics. Our method achieves state-of-the-art levels in metrics such as FD, FAD, and KL for the reconstructed audio across all three datasets.

Due to the complexity and diversity of sound signals, as well as the low resolution of neural signals [], brain-to-audio decoding is generally considered challenging, especially in semantic decoding.
Therefore, we attempt to provide the model with simple semantic prompts during decoding, such as the music genres or the genders of speakers.
The experimental results demonstrate that providing prompts can improve the semantic quality of the reconstructed audio when the decoded semantic features are poor.

Our contributions are as follows: (1) We propose a coarse-to-fine neural decoding model and reconstruct high-quality waveforms with both semantic and detailed information. We also confirm that coarse-to-fine decoding is superior to solely fine-grained decoding. (2) Our model achieves good results on datasets with three different kinds of stimuli, demonstrating its strong transferability. It can serve as a universal brain-to-audio framework. (3) We attempt to provide semantic prompts and prove that they can enhance the reconstruction quality when semantic decoding is challenging.

## 2 Method

Let $y\in\mathbb{R}^{L}$ represent an audio stimulus and $x\in\mathbb{R}^{V}$ represent the corresponding fMRI signal, where $L$ is the length of the audio samples and $V$ is the number of voxels in $x$. The brain-to-audio reconstruction process can be formulated as $\mathcal{R}:x\mapsto y$.
Our approach is to first decode an intermediate representation $c$ from $x$, and then generate $y$ using a generative model $\mathcal{G}$ conditioned on $c$.
To obtain the condition $c$, we follow a coarse-to-fine process. First, we perform a coarse-grained decoding by a Semantic Decoder $\mathcal{D}^{Sem}:x\mapsto s$ to extract the semantic embedding $s$ from fMRI.
Then, we use a semantically-guided Acoustic Decoder $\mathcal{D}^{Aco}:(s,x)\mapsto c$ to jointly decode the condition c with both semantics and acoustic details.
After decoding, we use an LDM as the generative model $\mathcal{G}:c\mapsto y$ to reconstruct the stimulus audio conditioned on $c$.
We will introduce the coarse-grained decoding process of $\mathcal{D}^{Sem}$ in Section 2.1.1, discuss the design of $\mathcal{D}^{Aco}$ and the fine-grained decoding process in Section 2.1.2, and describe the training of $\mathcal{G}$ in Section 2.2.

### 2.1 Coarse-to-fine brain decoding

#### 2.1.1 Coarse-grained semantic decoding

Figure: Figure 2: (a) Coarse-to-fine brain decoding. In the coarse-grained decoding, fMRI is decoded into the semantic space of CLAP. In the fine-grained decoding, fMRI is decoded into the acoustic space of AudioMAE. (b) Detailed structure of Acoustic Decoder.
Refer to caption: x2.png

We use the CLAP feature as the coarse-grained semantic embedding of audio.
CLAP, or contrastive language-audio pretraining [], is a pretrained multi-modal model that aligns representations of audio with natural language descriptions.
Pretrained on LAION-Audio-630K dataset [] containing audios of human speech and song, natural sounds, and audio effects music, CLAP features are semantically aligned with various categories of audios, providing rich semantic information.

We model the Semantic Decoder $\mathcal{D}^{Sem}:x\mapsto s$ as a ridge regression model.
As shown in Figure 2, we firstly use CLAP’s Audio Encoder to extract the ground truth semantic feature of the stimulus audio $y$, denoted as $s_{gt}\in\mathbb{R}^{512}$.
Then, we perform the L2-regularized linear regression from $x$ to $s_{gt}$ using PyFastL2LiR(^1^11https://github.com/KamitaniLab/PyFastL2LiR) toolkit, which provides fast ridge regression and voxel selection functionalities.
For each dimension of $s_{gt}$, we only select 500 voxels for regression based on the correlation coefficient. Thus, we obtain a sparse mapping matrix $W\in\mathbb{R}^{V\times 512}$ and a bias $b\in\mathbb{R}^{512}$.
The semantic embedding $s$ of fMRI can be inferred by $s=xW+b$ and $s\in\mathbb{R}^{512}$.

#### 2.1.2 Fine-grained acoustic decoding

We use the AudioMAE latent feature as the fine-grained acoustic embedding of audio.
AudioMAE, or audio mask autoencoder [], is a self-supervised pretrained model, which consists of an encoder $\mathcal{E}^{A}$ and a decoder $\mathcal{D}^{A}$ and focuses on the reconstruction of the masked patches.

The reason we choose the AudioMAE latent embedding as the acoustic feature instead of other DNN features is threefold: (1) AudioMAE is trained on a generative task, which retains more low-level acoustic details compared to the discriminative models like VGGish-ish [] used in Park et al. [] and Wav2Vec 2.0 [] used in Kim et al . [].
(2) Compared to the normal autoencoder used in Chen et al. [], AudioMAE performs a masked patch prediction task, which models the whole patches of the spectrogram. The empirical evidence [] shows that this makes the AudioMAE feature space more inclined to cluster audio of the same category together compared to VAE, indicating that AudioMAE better preserves high-level semantic information.
(3) Pretrained on AudioSet [] which consists of natural sounds, human and animal sounds, and music, AudioMAE can work well in the general audio domain. In comparison, the MusicLM [] used in Denk et al. [] can only model the music domain.
Taking all the points above into consideration, AudioMAE features are highly suitable for fine-grained features in our method, containing rich semantic and acoustic details.

As shown in Figure 2, we first transform the stimulus waveform $y$ into 128 Kaldi []-compatible Mel-frequency bands with a 25ms Hanning window that shifts every 10 ms following AudioMAE [], obtaining the mel-spectrogram $m$.
Then we divide $m$ into 16$\times$16 patches $m^{p}\in\mathbb{R}^{N_{patch}\times 256}$ and encode the patches into
$c_{gt}=\mathcal{E}^{A}(m^{p})\in\mathbb{R}^{N_{patch}\times 768}$ with no mask, where $N_{patch}$ represents the number of patches.
Then we decode $c_{gt}$ into the reconstructed patches $m^{p}_{upp}=\mathcal{D}^{A}(c_{gt})$ and unpatchify them into the mel-spectrogram $m_{upp}$.
We consider $c_{gt}$ as the ground truth acoustic feature of the stimulus audio $y$ and $m_{upp}$ as an upper bound for the reconstructed mel-spectrogram.

We model the Acoustic Decoder $\mathcal{D}^{Aco}:(s,x)\mapsto c$ as a Transformer-based model, which captures the dependencies between $s$ and $x$, and decodes fMRI into the latent space of AudioMAE through a Seq2Seq generation.
First, we project $s$ and $x$ into the 768-dimensional representation space of the Transformer. For $s$, we use a linear layer to project it to a semantic token $s^{\prime}$. For $x$, we select 768 voxels with the highest responses based on the mapping matrix $W$, forming the fMRI token $x^{\prime}$.
Then we concat the tokens and encode them with a Transformer Encoder $\mathcal{E}^{T}$, obtaining the neural embedding $n=\mathcal{E}^{T}([s^{\prime},x^{\prime}])$.
We create a learnable embedding $q$ as the query to a Transformer Decoder $\mathcal{D}^{T}$ along with $n$ as key and value, obtaining the decoded acoustic feature $c=\mathcal{D}^{T}(q,n)$.

We train $\mathcal{D}^{Aco}$ from scratch with three different loss functions that measure the distance between $c$ and $c_{gt}$.
The first one is $\mathcal{L}_{cond}$, which directly calculates the L2 distance in the latent space.
Then, we use the AudioMAE Decoder $\mathcal{D}^{A}$ to decode $c$ into the reconstructed patches $m^{p}_{recon}=\mathcal{D}^{A}(c)$ and unpatchify them into the mel-spectrogram $m_{recon}$.
During the decoding process, we calculate the L2 distance between each intermediate layer representation of $\mathcal{D}^{A}$ and the ground truth, which is the perceptual loss $\mathcal{L}_{perceptual}$.
Finally, we calculate the L2 distance between $m_{upp}$ and $m_{recon}$ with the original mel-spectrogram $m$ as the reconstruction loss $\mathcal{L}_{mel}$.
The overall loss is given by:

$$ $\mathcal{L}=\overbrace{\|c-c_{gt}\|_{2}^{2}}^{\mathcal{L}_{cond}}+\overbrace{ \sum_{i\in layer}\|\mathcal{D}^{A}_{i}(c)-\mathcal{D}^{A}_{i}(c_{gt})\|_{2}^{2 }}^{\mathcal{L}_{perceptual}}+\overbrace{\|m_{upp}-m\|_{2}^{2}+\|m_{recon}-m\| _{2}^{2}}^{\mathcal{L}_{mel}}.$ (1) $$

Since the pretrained AudioMAE model is accustomed to working with masked patches, and our method utilizes all patches, we freeze $\mathcal{E}^{A}$ and fine-tune the parameters of $\mathcal{D}^{A}$ in order to achieve the best performance of reconstruction.

Furthermore, we follow Liu et al. [] by setting a $P_{gt}=0.25$ during training, which means that $\mathcal{D}^{Aco}$ has a 0.25 probability of receiving the ground truth semantic feature $s_{gt}$ as input and a 0.75 probability of receiving the decoded semantic feature $s$ from $\mathcal{D}^{Sem}$. This trick enhances the decoding performance of $\mathcal{D}^{Aco}$ and facilitates the utilization of semantic prompts. We will discuss it in Section 3.5.

### 2.2 Brain-to-audio reconstruction

Figure: Figure 3: Brain-to-audio reconstruction. The LDM generates mel-spectrograms under the condition of fine-grained acoustic features, followed by the Vocoder to generate reconstructed audios.
Refer to caption: x3.png

In this section, we use a generative model $\mathcal{G}:c\mapsto y$ to reconstruct the stimulus audio conditioned on $c$.
When performing fine-grained decoding, although we use the AudioMAE Decoder to reconstruct the mel-spectrogram, it is not suitable to serve as the generative model for our method. We will discuss this in detail in Section A.4.
Instead, we model the process with a Latent Diffusion Model (LDM) [].
LDM is a powerful generative model that can model complex data distributions in the latent space. It has been extensively used in the audio generation task, such as AudioLDM [], AudioLDM2 [] and DiffVoice [].

We follow the formulation in AudioLDM2 [] to implement the LDM.
As shown in Figure 3, we first use a Hanning window with 64 frequency bins, a window size of 1024, and a hop size of 160 to convert the stimulus audio into the mel-spectrogram. Then compress it to a latent representation $z$ using a VAE.
The forward diffusion process is a $T$ steps Markov chain that gradually adds Gaussian noise as

$$ $q(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}\textbf{ I})$ (2) $$

where $\beta_{t}$ is a variance schedule. Then the distribution of $z_{t}$ given $z_{0}$ can be formulated as

$$ $q(z_{t}|z_{0})=\prod_{s=1}^{t}q(z_{s}|z_{s-1})=\mathcal{N}(z_{t};\sqrt{\alpha_ {t}}z_{0},(1-\alpha_{t})\textbf{I})$ (3) $$

where $\alpha_{t}=\prod_{s=1}^{t}(1-\beta_{s})$. The distribution of $z_{T}$ at the final step will be a standard Gaussian distribution [].
The LDM learns a reverse denoising process from the prior distribution $\mathcal{N}(\textbf{0},\textbf{I})$ to the data distribution $z$ conditioned on $c$. The loss function [, ] in our method can be given as

$$ $\mathcal{L}=\mathbb{E}_{z_{t},\epsilon\sim\mathcal{N}(\textbf{0},\textbf{I}),t \sim\{1,...,T\}}[\|\epsilon_{\theta}(z_{t},t,c)-\epsilon\|_{2}^{2}+\|\epsilon_ {\theta}(z_{t},t,c_{gt})-\epsilon\|_{2}^{2}]$ (4) $$

where $\epsilon_{\theta}$ is the denoising network, for which we utilize a Transformer-UNet (T-UNet) following AudioLDM2 [].
After the LDM reconstructs the mel-spectrogram, it will be converted to the waveform using a pretrained HiFiGAN [] vocoder.
We initialize the LDM with pretrained weights from AudioLDM2 and fine-tune $\mathcal{D}^{Aco}$ and the T-UNet during training, while keeping other weights frozen.

### 2.3 Conditional reconstruction

In practical applications, brain-to-audio reconstruction is not always unconditional. Firstly, compared to images, audios exhibit strong temporal correlations.
If a subject listens to a long segment of stimulus audio but only a portion needs to be reconstructed using brain signals, considering that other audio segments and the target segments may be semantically similar, they can serve as conditions to provide additional semantic information.
Secondly, we may know in advance the coarse-grained category (e.g., human speech or animal sound) of the audio to be reconstructed. Given the challenge of semantic decoding from fMRI (see Section  3.4 for details), we can use the coarse-grained category as the semantic prior to guide the reconstruction process. Thus, we attempt to provide semantic prompts in the form of audio or text to our model for conditional reconstruction, to assess whether it can enhance the quality of the reconstructed audio.

The conditional reconstruction process is straightforward. We utilize CLAP’s Text Encoder and Audio Encoder to extract the semantic embedding $s_{prompt}$ of the text prompt and audio prompt. Then we replace $s$ with $s_{prompt}$ as input to $\mathcal{D}^{Aco}:(s_{prompt},x)\mapsto c$, to obtain the fine-grained acoustic embedding $c$. Finally, we use $\mathcal{G}:c\mapsto y$ to reconstruct the stimulus audio conditioned on $c$.

Figure: Figure 4: Reconstruction results of S1, sub-001 and UTS01 on the three datasets.
Refer to caption: x4.png

## 3 Experiments

### 3.1 Datasets

We use three publicly available fMRI datasets to validate our method’s performance across different kinds of stimuli: Brain2Sound [], Brain2Music [], and Brain2Speech [] datasets.
Brain2Sound Dataset [] comprises fMRI signals from five subjects listening to 4-second natural sounds, including human speech, animal, musical instrument, and environmental sounds. The dataset consists of 14,400 training samples and 150 test samples.
Brain2Music Dataset [] comprises fMRI signals from five subjects listening to 1.5-second music clips, consisting of 4,800 training samples and 600 test samples.
Brain2Speech Dataset [] comprises fMRI signals from seven subjects listening to 2-second voice segments, consisting of 9,137 training samples and 595 test samples.
For detailed information about the datasets and the preprocessing methods, please refer to section A.1.

### 3.2 Metrics

We use FD, FAD, KL, and KL-S, commonly utilized in audio generation tasks, to evaluate the quality of brain-to-audio reconstruction.
FD (Fréchet Distance) calculates the distance in features between generated samples and target samples, extracted from an audio classifier PANNs [].
KL and KL-S (Kullback–Leibler divergence) calculate the KL divergence of classification logits, also based on PANNs, using Softmax and Sigmoid activation functions respectively.
FAD (Fréchet Audio Distance) is similar to FD, but it uses VGGish [].
All metrics are implemented through the Audio Generation Evaluation toolbox.(^2^22https://github.com/haoheliu/audioldm_eval)
The above metrics evaluate high-level representations of audio. In addition to these, we use PCC and SSIM to measure the similarity between the mel-spectrograms of reconstructed audio and stimulus audio, evaluating low-level spectrotemporal representations.
Each subject conducts independent experiments, and the metrics are averaged.

### 3.3 Reconstruction results

We compare the reconstruction results of three methods:
(1) The direct decoding methods, which map fMRI signals to mel-spectrograms, including a linear regression model [, , , ] implemented through Ridge in sklearn, a three-layer MLP [, , ] implemented through MLPRegressor in sklearn following Bellier et al. [], a Bidirectional LSTM [, ] and a Transformer Encoder [], both with the same configuration as our Transformer.
(2) The fine-grained decoding methods, which map fMRI signals to high-dimensional intermediate features directly [, , ]. We remove the coarse-grained decoding process of our method and decode fMRI into the latent space of AudioMAE using the Acoustic Decoder $\mathcal{D}^{Aco}:x\mapsto c$.
Then we used the LDM $\mathcal{G}:c\mapsto y$ to reconstruct the audio. This method is called Fine-LDM.
In addition, for the Brain2Sound Dataset, we use the code and checkpoints open-sourced by Park et al. [] to reproduce their experimental results.
(3) The coarse-to-fine decoding methods proposed by us, including C2F-Decoder, which utilizes the AudioMAE Decoder as the generative model (see details in Section A.4) and C2F-LDM using the LDM (ours).
Please refer to section A.2 for specific details on the experimental setup.

Figure: Figure 5: PCC between the ground truth and decoded acoustic features for 17 subjects in the Brain2Sound, Brain2Music and Brain2Speech datasets. Our coarse-to-fine method consistently outperforms the directly fine-grained method.
Refer to caption: extracted/5628114/img/decoding_PCC.png

**Table 1: Reconstruction results on the three datasets.**
| Model | PCC$\uparrow$ | SSIM$\uparrow$ | FD$\downarrow$ | FAD$\downarrow$ | KL$\downarrow$ | KL-S$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Brain2Sound Dataset  [] |  |  |  |  |  |  |
| LiR [, , , ] | 0.607 | 0.184 | 105.113 | 40.877 | 4.027 | 9.650 |
| MLP [, , ] | 0.566 | 0.181 | 98.358 | 38.045 | 4.020 | 9.811 |
| BiLSTM [, ] | 0.526 | 0.134 | 92.172 | 33.442 | 4.187 | 9.668 |
| Transformer [] | 0.526 | 0.132 | 74.048 | 27.526 | 3.817 | 8.974 |
| Park et al. [] | 0.394 | 0.133 | 88.456 | 12.694 | 2.251 | 6.560 |
| Fine-LDM [, , ] | 0.376 | 0.107 | 49.827 | 10.803 | 2.895 | 7.500 |
| C2F-Decoder | 0.595 | 0.177 | 95.565 | 35.775 | 3.748 | 10.097 |
| C2F-LDM (ours) | 0.418 | 0.118 | 44.003 | 9.324 | 2.697 | 7.046 |
| Brain2Music Dataset [] |  |  |  |  |  |  |
| LiR [, , , ] | 0.637 | 0.197 | 47.710 | 18.247 | 0.997 | 4.105 |
| MLP [, , ] | 0.591 | 0.191 | 48.980 | 19.895 | 0.732 | 3.460 |
| BiLSTM [, ] | 0.628 | 0.200 | 57.030 | 22.673 | 1.008 | 4.117 |
| Transformer [] | 0.646 | 0.195 | 60.969 | 22.195 | 1.079 | 4.692 |
| Fine-LDM [, , ] | 0.426 | 0.112 | 6.544 | 1.439 | 0.526 | 2.132 |
| C2F-Decoder | 0.643 | 0.199 | 63.039 | 26.053 | 1.191 | 4.334 |
| C2F-LDM (ours) | 0.454 | 0.124 | 6.102 | 1.504 | 0.520 | 2.010 |
| Brain2Speech Dataset [] |  |  |  |  |  |  |
| LiR [, , , ] | 0.511 | 0.126 | 68.146 | 24.988 | 3.483 | 8.132 |
| MLP [, , ] | 0.409 | 0.116 | 75.174 | 27.983 | 4.153 | 9.057 |
| BiLSTM [, ] | 0.580 | 0.179 | 112.031 | 39.895 | 3.948 | 10.312 |
| Transformer [] | 0.581 | 0.173 | 104.118 | 39.484 | 3.764 | 10.228 |
| Fine-LDM [, , ] | 0.357 | 0.077 | 12.706 | 4.820 | 0.885 | 2.573 |
| C2F-Decoder | 0.518 | 0.128 | 96.032 | 26.917 | 4.278 | 9.831 |
| C2F-LDM (ours) | 0.393 | 0.083 | 9.726 | 4.623 | 0.616 | 1.989 |

We select one representative from each of the three methods, Linear Regression (LiR), Fine-LDM and C2F-LDM, and display the reconstructed mel-spectrograms in Figure 4.
All quantitative results are displayed in Table 1. We find that direct decoding methods are able to capture the overall spectrotemporal distributions of mel-spectrograms but struggle to reconstruct details and semantics.
They show high performance in low-level metrics but low performance in high-level metrics.
In contrast, the fine-grained decoding methods exhibit a significant improvement in high-level metrics but a decrease in low-level metrics, which increase details in the reconstructed spectrograms.
It suggests that these methods can provide a preliminary reconstruction of semantics but at the cost of losing spectrogram similarity.

Our coarse-to-fine method reconstructs the spectrogram details that are closer to the original stimulus and shows improvements in both low-level and high-level metrics compared to the fine-grained decoding methods, as shown in Figure 4 and Table 1.
Although it has not yet reached the level of direct decoding methods in PCC and SSIM, it has achieved state-of-the-art performance in FD, FAD, KL, and KL-S.
Compared to Park et al. [], our method significantly improves on FD, surpasses FAD and PCC, and closely approaches the remaining metrics.
The comparison of reconstructed samples can be found in Section A.3. The comparison with C2F-Decoder can be found in Section A.4.

We further analyze the impact of our coarse-to-fine method on decoding the fine-grained acoustic features. We compute the PCC between the ground truth and decoded acoustic features for 17 subjects across the three datasets. Then we compare the experimental results between the coarse-to-fine decoding and the directly fine-grained decoding, with a baseline established by directly mapping fMRI to the acoustic features using L2-regularized Linear Regression.
As shown in Figure 5, the consistent decoding performance of the PCC across different methods reflects the varying signal quality of the subjects.
Across almost all participants, our coarse-to-fine method consistently outperforms the fine-grained method. It suggests that coarse-to-fine decoding can effectively enhance the fine-grained acoustic features widely across the participants.

Figure: Figure 6: Semantic decoding accuracy in the (a) Brain2Music and (b) Brain2Speech Dataset. The gray dashed lines represent the upper bound of accuracy and the chance level. Significance test is performed (paired t-test, $p<0.001$(***), $p<0.01$(**), $p<0.05$(*)).
Refer to caption: x5.png

### 3.4 Semantic analysis of acoustic features

An intuitive idea is that the introduction of coarse-grained semantic decoding enhances the semantic information in the acoustic features, thereby improving the fine-grained decoding. Is this really the case? We will discuss this issue in this section.

It is generally believed that a representation space with strong semantic information exhibits the following characteristics: samples of the same category cluster in this space, while samples from different categories are far apart.
Therefore, we conduct a classification experiment using two datasets with clear category labels, the Brain2Music Dataset [] with the labels of music genres (10 classes) and the Brain2Speech Dataset [] with the labels of speakers’ genders (2 classes).
We use SVM to classify the decoded acoustic features after the coarse-to-fine decoding and the directly fine-grained decoding.
The experiment is conducted on the test set and the average classification accuracy using 5-fold cross-validation is used to assess the semantic information in the acoustic features. The chance levels are $0.1$ and $0.5$ for the two datasets, while the upper bounds are the classification accuracy on the ground truth acoustic features, which are $0.33$ and $0.95$.

As shown in Figure 6, after introducing the coarse-grained decoding, the classification accuracy for each subject in the Brain2Music Dataset either increases or remains essentially unchanged. However, in the Brain2Speech Dataset, the classification accuracy for some subjects decreases. It suggests that the semantic information in the acoustic features is diminishing, while the coarse-grained decoding primarily enhances the low-level acoustic information.

To explain this phenomenon, we further analyze the coarse-grained semantic features of guidance.
The same SVM classification task is performed. As shown in Figure 6, the classification accuracy of the semantic features in the Brain2Music Dataset is relatively high, whereas the classification accuracy in the Brain2Speech Dataset is poor, approaching the chance level.
It indicates that there is little semantic content in the semantic features, making it nearly impossible to differentiate speakers’ genders.
We visualize the distribution of the semantic features using t-SNE for the two datasets. As shown in Section A.5, some categories in the Brain2Music Dataset cluster well, while samples from the two categories in the Brain2Speech Dataset are mixed and almost indistinguishable, which is consistent with our experimental results.
It might be because participants were more focused on listening to the content of the stories during fMRI signal collection and disregarded the speaking style of the speaker.
The lack of semantic richness in the guided semantic features leads to a decrease in the semantic content of the acoustic features.

In summary, although introducing the coarse-grained semantic decoding can enhance the decoding of the fine-grained acoustic features, the semantic content of the acoustic features may not be enhanced if the semantic features of guidance are poor. In such cases, the coarse-grained decoding mainly enhances the low-level acoustic information.

### 3.5 Conditional reconstruction results

From the previous section, we can see that due to the issues with the dataset, when the semantic decoding does not perform well, guidance on the acoustic decoding may worsen the semantic content of the acoustic features.
Therefore, when we conduct the conditional reconstruction task mentioned in Section 2.3, it’s better to use the semantic features of prompts as the coarse-grained features instead of the decoded features.
In this experiment, we define two kinds of prompts. The first one is a text prompt that includes the category of the stimulus audio.
The prompts in the Brain2Music Dataset [] consist of 10 music genres such as pop music and rock music, while the prompts in the Brain2Speech Dataset [] include two options for the speaker’s gender: man speaking and woman speaking.
The second one is an audio prompt. We utilize the last 10 stimulus audio clips from two stories in the test set as prompts for the Brain2Speech Dataset and then average the results obtained.

Since the hyperparameter $P_{gt}$ mentioned in Section 2.1.2 is related to the input ground truth semantic feature, we attempt to evaluate the conditional brain-to-audio reconstruction with prompts under three conditions: $P_{gt}=0.0$, $0.25$ and $0.5$.
The results are shown in Table 2.
We observe that all metrics for $P_{gt}=0.25$ surpass those for $P_{gt}=0.0$ without a prompt.
For the Brain2Music Dataset, the high-level metrics are higher for $P_{gt}=0.25$, while for the Brain2Speech Dataset, the high-level metrics are higher for $P_{gt}=0.5$. However, on both datasets, the PCC is highest when $P_{gt}=0.25$.
It suggests that appropriately incorporating ground truth semantic features during training can help enhance the quality of brain-to-audio reconstruction.

For the Brain2Music Dataset, the high-level metrics decrease after incorporating the text prompts. It suggests that the text prompts of music genres we provide may not effectively represent the semantic content of the stimulus audio.
After adding the text prompts, samples of the same music genre have identical coarse-grained features.
Considering the relatively strong semantic decoding in the Brain2Music Dataset as mentioned in Section 3.4, the loss of the individual specificity across samples leads to a decrease in the reconstruction quality, particularly in semantics.
In contrast, the semantic decoding performs poorly in the Brain2Speech Dataset, so the introduction of both text prompts and audio prompts can significantly enhance the semantics of the reconstructed audio.
In summary, when coarse-grained semantic features are suboptimal, conditional reconstruction with prompts can effectively enhance the quality of the reconstructed audio.

**Table 2: Conditional reconstruction results with text prompts and audio prompts.**
|  |  | Brain2Music Dataset [] | Brain2Speech Dataset [] |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $P_{gt}$ | Prompt | PCC$\uparrow$ | SSIM$\uparrow$ | FD$\downarrow$ | FAD$\downarrow$ | KL$\downarrow$ | KL-S$\downarrow$ | PCC$\uparrow$ | SSIM$\uparrow$ | FD$\downarrow$ | FAD$\downarrow$ | KL$\downarrow$ | KL-S$\downarrow$ |
| $0.0$ | no prompt | 0.442 | 0.113 | 6.121 | 1.606 | 0.520 | 2.070 | 0.379 | 0.081 | 11.636 | 4.866 | 0.758 | 2.303 |
| text prompt | 0.421 | 0.135 | 8.283 | 2.169 | 0.641 | 2.606 | 0.331 | 0.079 | 10.772 | 5.265 | 0.513 | 1.650 |  |
| audio prompt | - | - | - | - | - | - | 0.315 | 0.077 | 7.160 | 4.148 | 0.430 | 1.427 |  |
| $0.25$ | no prompt | 0.454 | 0.124 | 6.102 | 1.504 | 0.520 | 2.010 | 0.393 | 0.083 | 9.726 | 4.623 | 0.616 | 1.989 |
| text prompt | 0.405 | 0.136 | 7.358 | 2.219 | 0.584 | 2.430 | 0.340 | 0.082 | 9.265 | 4.712 | 0.449 | 1.419 |  |
| audio prompt | - | - | - | - | - | - | 0.300 | 0.075 | 7.722 | 4.383 | 0.416 | 1.394 |  |
| $0.5$ | no prompt | 0.422 | 0.110 | 6.587 | 1.341 | 0.536 | 2.156 | 0.374 | 0.084 | 7.957 | 4.013 | 0.493 | 1.665 |
| text prompt | 0.393 | 0.138 | 10.304 | 2.624 | 0.559 | 2.479 | 0.350 | 0.089 | 6.698 | 5.102 | 0.348 | 1.142 |  |
| audio prompt | - | - | - | - | - | - | 0.303 | 0.076 | 7.129 | 3.930 | 0.398 | 1.338 |  |

## 4 Conclusion

In this paper, we propose a novel coarse-to-fine audio reconstruction method inspired by the hierarchical processing of the human auditory system.
Our method begins by decoding fMRI data into the CLAP space to extract coarse-grained semantic features. Subsequently, leveraging these semantic features, we decode the fMRI data into the AudioMAE latent space to capture fine-grained acoustic features.
Next, we use the acoustic features as conditions to reconstruct the stimulus audio using a Latent Diffusion Model.
By validating our method on three diverse fMRI datasets—Brain2Sound, Brain2Music, and Brain2Speech—our method has shown superior performance in brain-to-audio reconstruction compared to previous fine-grained methods.
We have illustrated its state-of-the-art capabilities, achieving remarkable results in FD, FAD, KL, and KL-S. The integration of semantic prompts during decoding further enhances the semantics of reconstructed audio, particularly when dealing with suboptimal semantic decoding.
We will discuss the limitations of our paper in Section A.6 and the broader impacts in Section A.7.

## Appendix A Appendix

### A.1 Datasets

**Table 3: ROIs and voxels of the datasets.**
| Dataset | ROIs | Subjects | Voxels | Training samples | Test samples |
| --- | --- | --- | --- | --- | --- |
| Brain2Sound [] | AC (A1, LBelt,<br>A4, A5, etc.) | S1 | 6,662 | 13,872 | 150 |
| S2 | 6,624 | 13,944 |  |  |  |
| S3 | 6,713 | 13,944 |  |  |  |
| S4 | 6,157 | 13,944 |  |  |  |
| S5 | 7,143 | 13,944 |  |  |  |
| Brain2Music [] | N/A | sub-001 | 60,784 | 4,800 | 600 |
| sub-002 | 53,927 |  |  |  |  |
| sub-003 | 64,700 |  |  |  |  |
| sub-004 | 61,899 |  |  |  |  |
| sub-005 | 53,421 |  |  |  |  |
| Brain2Speech [] | AC, FFA, OFA,<br>PPA, etc. | UTS01 | 836 | 9,137 | 595 |
| UTS02 | 2,093 |  |  |  |  |
| UTS03 | 1,303 |  |  |  |  |
| UTS05 | 920 |  |  |  |  |
| UTS06 | 980 |  |  |  |  |
| UTS07 | 1,584 |  |  |  |  |
| UTS08 | 1,109 |  |  |  |  |

##### Brain2Sound Dataset

As proposed by Park et al. [], this dataset(^3^33https://github.com/KamitaniLab/SoundReconstruction) records the fMRI signals of five subjects (one female) while they are listening to natural sounds, including human speech, animal, musical instrument, and environmental sounds.
fMRI data are acquired using a 3.0-Tesla Siemens MAGNETOM Verio scanner at the Kyoto University Institute for the Future of Human Society. Functional images that cover the entire brain are obtained with TR = 2,000 ms, TE = 44.8 ms, flip angle = 70 deg, FOV = 192 × 192 mm, voxel size=2 × 2 × 2 mm, number of slices = 76 and multi-band factor = 4.

The stimuli consist of 1,250 8-s natural sound segments, with 1,200 for the training set and 50 for the test set, selected from the VGGSound dataset []. All the sounds are extracted from the videos uploaded to YouTube.
To increase the sample number, we preprocess the audio segments in the same way as Park et al. []: 4-s sliding windows are utilized with a 2-s stride to extract 3 4-s segments. All audio clips are resampled to 16kHz.
During the collection of fMRI signals, each stimulus is repeated four times, resulting in 14,400 samples(^4^44When downloading, we discovered that some audios in the training set were no longer available on YouTube, hence, the amount of training samples is slightly less than 14,400. See Table 3 for details.) for the training set (1,200 stimuli × 4 repetitions × 3 samples = 14,400 samples). For the test set, we average the multiple fMRI samples, resulting in 150 samples (50 stimuli × 3 samples = 150 samples).

##### Brain2Music Dataset

Following Denk et al. [], we use the music genre neuroimaging dataset(^5^55https://openneuro.org/datasets/ds003720) from Nakai et al. [], which records the fMRI signals of five subjects (two female) while they are listening to music clips.
fMRI data are acquired using a 3.0T MRI scanner (TIM Trio; Siemens, Erlangen, Germany) at the Center for Information and Neural Networks (CiNet), National Institute of Information and Communications Technology (NICT), Osaka, Japan. Functional scanning is performed with TR = 1,500 ms, TE = 30 ms, flip angle = 62 deg, FOV = 192 × 192 mm, voxel size = 2 × 2 × 2 mm and multi-band factor = 4.

The dataset contains music stimuli from 10 genres (blues, classical, country, disco, hip-hop, jazz, metal, pop, reggae, and rock) which are sampled from the GTZAN dataset []. A total of 54 15-s music pieces are selected from each genre, with 48 for the training set and 6 for the test set.
All music pieces are resampled to 16kHz and segmented into 10 clips of 1.5 seconds each to match the TR of functional scanning. As a result, the dataset consists of 4,800 samples (48 stimuli × 10 genres × 10 samples = 4,800 samples) for training and 600 (6 stimuli × 10 genres × 10 samples = 600 samples) for testing.

##### Brain2Speech Dataset

We use the dataset(^6^66https://openneuro.org/datasets/ds003020/versions/1.1.1) proposed by LeBel et al. []. The dataset contains fMRI responses recorded while 7 participants(^7^77Subject UTS04 lacks a story, hence it will not be utilized.) (three female) are listening to 27 complete, natural, narrative stories.
fMRI data are collected on a 3T Siemens Skyra scanner at the UT Austin Biomedical Imaging Center. Functional scans are collected with TR = 2.00 s, TE = 30.8 ms, flip angle = 71 deg, multi-band factor = 2, voxel size = 2.6 × 2.6 × 2.6 mm and FOV = 220 mm.

The stimulus set consists of 27 10–15 minute stories from The Moth podcast. We select two stories (Hang time by a male speaker and Where there’s Smoke by a female speaker) as the test set, and the remaining 25 stories are used as the training set.
All stories are resampled to 16kHz and segmented into 2-s clips to match the TR of functional scanning.
To account for the hemodynamic response, we form a sample pair by combining the fMRI signal of each TR with the stimulus audio clip from 4 seconds ago.
We use the last 10 stimulus audio clips from two stories in the test set as audio prompts. These prompts are not used for testing, ensuring that the participants could not have possibly heard the audio prompts in the preceding trials.
As a result, the dataset consists of 9,137 samples for training and 595 samples for testing per subject.

### A.2 Experimental setup

In the stage of coarse-grained decoding, for the Brain2Sound and Brain2Speech datasets, we only utilize voxels from the auditory cortex (AC) area, whereas for the Brain2Music Dataset, we use voxels from the entire brain. The specific brain regions and voxels can be found in Table  3.

In the stage of fine-grained decoding, we utilize a 4-layer Transformer Encoder and Decoder in $\mathcal{D}^{Aco}$ and initialize AudioMAE with the pretrained weights.(^8^88https://github.com/facebookresearch/AudioMAE)
Since the pretrained AudioMAE requires 10-second audios (128$\times$1024 mel-spectrograms) as inputs, we duplicate the stimulus waveforms to 10 seconds. After encoding with the AudioMAE Encoder $\mathcal{E}^{A}$, we select the embeddings of the first $N_{patch}$ patches as $c_{gt}$, corresponding to the length of the stimulus audio.
We set $N_{patch}=208$ for the Brain2Sound Dataset, $N_{patch}=80$ for the Brain2Music Dataset, and $N_{patch}=112$ for the Brain2Speech Dataset.

In the stage of brain-to-audio reconstruction, we utilize two checkpoints of AudioLDM2(^9^99https://github.com/haoheliu/AudioLDM2) as the initialization weights for our LDM $\mathcal{G}$: audioldm2-full for the Brain2Sound and Brain2Music datasets, and audioldm2-speech-gigaspeech for the Brain2Speech dataset.

We use the AdamW [] optimizer to train $\mathcal{D}^{Aco}$ and $\mathcal{D}^{A}$ with a learning rate of 1e-6, and train $\mathcal{G}$ with a learning rate of 1e-4.
All training is completed on a single NVIDIA A100 80GB GPU.

### A.3 Comparison with Park et al.

We reproduce the experimental results of Park et al. [] using the features from the conv5_3 layer of VGGish-ish [] and voxels from the entire AC region, and compare them with our method.

Figure: Figure 7: Comparison with the reconstruction results of Park et al. [].
Refer to caption: x6.png

### A.4 C2F-Decoder vs. C2F-LDM

When performing fine-grained decoding, although we use the AudioMAE Decoder to reconstruct the mel-spectrogram, it is not suitable to serve as the generative model for our method. There are two main reasons for this:
(1) The mel bins and window parameters of the mel-spectrograms in AudioMAE do not align with those of commonly used pretrained Vocoders. This mismatch prevents the generated mel-spectrograms from being directly converted into audio. Moreover, the cost of training a compatible Vocoder from scratch is prohibitively high. (2) The primary task of the AudioMAE Decoder is to predict masked patches, with a focus on low-level details of the spectrogram. This limitation leads to insufficient reconstruction quality in terms of semantic content.
In contrast, the mel-spectrograms generated by LDM can be directly restored to audio using the pretrained HiFiGAN [] vocoder, and the generated audio has richer semantic and acoustic details.

To investigate the reconstruction performance of C2F-Decoder, we need to transform the mel-spectrograms generated by the AudioMAE Decoder, denoted as $m^{A}$, into mel-spectrograms that the Vocoder can accept, denoted as $m^{V}$. We assume that $m^{A}$ and $m^{V}$ have a linear relationship, so we use an L2-regularized linear regression model trained on $m^{A}$ and $m^{V}$ of the stimulus audio in the training set. The results in the test set are as follows:
PCC $=0.938$ in the Brain2Sound Dataset, PCC $=0.967$ in the Brain2Music and Brain2Speech datasets.
Based on the results, we believe that this transformation is almost lossless.
As shown in Figure 8 and Table 1, C2F-Decoder is similar to the direct decoding methods in that they both focus on modeling the overall spectrogram but lack detail and semantic information compared to C2F-LDM.

Figure: Figure 8: Comparison of the reconstruction results between C2F-Decoder and C2F-LDM.
Refer to caption: x7.png

### A.5 Coarse-grained semantic decoding

We perform t-SNE visualization on the space of ground truth and decoded semantic features on the Brain2Music and Brain2Speech datasets.
We choose the decoding spaces of sub-003 and UTS05, which show significant effects after incorporating the semantic features based on Figure 6.
For the Brain2Music Dataset, some genres like classical and jazz exhibit good decoding performance, while others like rock show poor decoding performance, as shown in Figure 9.
For the Brain2Speech Dataset, the semantic decoding performance is poor, and it cannot differentiate between male and female speakers, as shown in Figure 10.

Figure: Figure 9: Visualization on the space of ground truth (left) and decoded (right) semantic features on the Brain2Music Dataset.
Refer to caption: extracted/5628114/img/B2Music-gt.png

Figure: Figure 10: Visualization on the space of ground truth (left) and decoded (right) semantic features on the Brain2Speech Dataset. The cross markers represent samples used as audio prompts, providing coarse-grained semantic features in the conditional reconstruction.
Refer to caption: extracted/5628114/img/B2Speech-gt.png

### A.6 Limitations and future work

The main purpose of this article is to illustrate the superiority of hierarchical decoding over direct decoding.
Therefore, we build a generic brain-to-audio framework, selecting the most suitable models, CLAP and AudioMAE, without comparing them to other representation models.
In the future, we will switch to different models within our framework to attempt further improvement of reconstruction results.
Furthermore, we utilize all the voxels of the auditory cortex (AC) in our work. However, there are gradients in the voxels of different brain regions within the AC. In the future, we plan to consider the gradients of voxels to further enhance the hierarchy of information processing.

### A.7 Broader Impacts

This research contributes to enhancing our understanding of brain function and cognitive processes, playing a crucial role in further exploring the mechanisms of the human auditory system and promoting the development of related technologies.
The experiments are conducted in a controlled laboratory setting with the participants’ consent and cooperation. It is not applicable in real-world scenarios, thereby posing a minimal risk of privacy leakage.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: **Improving clarity and motivations**

- The distinction between low-level (coarse) and high-level (fine-grained) features is confusing, lacks clarity and contributes negatively to the overall appreciation of the paper. Could the authors precise what they refer to as high-level or fine-grained features and as coarse or low-level features? It is confusing to refer to "semantics" as coarse, as they are supposed to provide very precise and high-level descriptive information. Similarly, the spectrogram is often referred to as high-level, although it describes low-level features similar to the features extracted by the cochlea (line 86).

- The description of the CLAP training (with aligned textual descriptions and audio) (line 147) and the use of prompts for incorporating music genres and phenotype information  (line 117), are considered as coarse features, why? 

- The paper refers to as "inverse pathway of auditory processing" (line 17). If it is inverse; shouldn't it go from high-level to low-level? seems that the method goes from coarse-to-fine ... "AudioMAE latent feature as the fine-grained acoustic embedding of audio" (line 182) isn't it coarse information?

- Typo line 466 ->"it's better"

- In line 52, what are the DNN features referred to? could the authors provide more explanations?


**Methodology/Results:**

-  It is unclear how the Semantic Decoder extracts semantically rich information (line 137); is it validated somehow? 

- One of the baseline modes is a Transformer that goes from voxel space to mel-spectrogram space. Could the authors provide more information about this implementation, as it does not seem straightforward?

- Could the authors provide some motivations for using ridge regression to model brain signals (referred to as $x$ in the paper)? does this consider the autocorrelation of brain cortical activation and using the structure and spatial organisation of the auditory cortex?

- Regarding the generation process and the use of the latent diffusion model, how much do the authors consider providing the ground truth as input to help the model (figure 3)? Isn't the conditioning should be enough?

- In tables 1, 2, and 3, are the results averaged across subjects? if yes could you provide the std? 

- In Table 1, why is C2F-LDM low for PCC and PSNR?

- The objective of section 3.3. and 3.4 are not very clear, for instance, lines 457 to 460. More importantly, the results do not seem to go in the direction of the conclusion of the paper, e.g. in Figure 5. b. why does the fine-grained decoding perform better for almost all experiments? The hypothesis made in section 3.3, e.g. "This could be attributed to participants being more focused on the content of the stories during fMRI signal collection, potentially disregarding the speaking style of the speaker." (line 453) is used to motivate the next section 3.4 but is relatively weak and unclear. In section 3.4, The Brain2Music dataset performs better without prompts, although the prompts are supposed to incorporate important information such as the music genre. Why use prompts, then? seems somehow superficial. 

**Data**:

- what about variation in performance across subjects? are the models trained per sujects? or models are used across subjects?

- how are the voxels in Table 4 extracted? Is it from a template? subject-based parcellation?

- Are the datasets aligned functionally (on top of anatomical registration) so that similar voxels and auditory regions across subjects can be compared?

- For all datasets, the audio clips seem to be between 1.5 and 4 seconds. Is it enough to aim at reconstructing audio from a very short fMRI temporal window? What about using larger windows?
soundness: 3
strengths: - The technical contributions of the paper are strong. The methodology, with the threefold approach, is well thought out and carefully implemented. Many implementation details are provided, and the notations are clear and easy to follow. This is a great point.

- Figure 2 is clear and provides sufficient details to understand the intricate methodology

- The new method is compared against various baselines and three openly available datasets. Also, the code is made public, and additional results are provided. This is a great point for the openness of science. 

- Various ablation studies are done to confirm the contributions of specific modules (for instance the diffusion reconstruction vs using the MAE decoder).
summary: This paper aims to achieve audio reconstruction from an fMRI brain signal via a coarse-to-fine approach. The idea is to replicate the audio processing stream in the human auditory cortex. The method is threefold: first, it uses a CLIP-based approach to extract an audio representation (low-level description) from the fMRI data signal. From these initial features, a high-dimensional description of the auditory feature is obtained via a guided AudioMAE. Finally, these high-level features are used as a condition of a Latent Diffusion model for mel-spectrogram reconstruction and, thus, audio reconstruction. The method achieves high results on three publicly available datasets, demonstrating strong performance for both low-level and high-level audio metric reconstructions, showing improvement compared to the direct reconstruction of mel-spectrogram and other high-level approaches which omit low-level features.
weaknesses: - One weakness of the paper is the limited neuroscientific motivations for audio reconstruction as a way to understand auditory mechanisms (first paragraph of the introduction). It is not clear from the introduction or the conclusion how solving this task could help to understand better the auditory processes (line 28  "This research contributes to the comprehension of the human auditory
system" ). Are these models generalisable between subjects? can a model trained on a single subject be used for another subject? can they be used to identify specific features related to language disorders? 

- Another major weakness in this study is the lack of clarity in the writing, which prevents us from understanding some of the motivations and results/discussion. The characterisation of "high-level/low-level" features and "coarse/fine-grained" features throughout the paper is very unclear. Some of these terms seem interchangeable in how they are employed; the paper would greatly benefit from a clear definition (see more related questions in the next section).

- Some points of the methodology lack details and/or seem to diminish the results (see questions in the next section). For instance, the modelling of the brain signal with ridge regression, which overlooks the spatial autocorrelation of the brain signal and structure of the auditory cortices, is not motivated. Sections 3.3 and 3.4 lack clarity, it is not clear what they aim to achieve with respect to the general motivation of the paper, and the conclusion are not clear and even seem to diminish previous results (for instance Figure 5 and  line 457 to 460) - see more questions below. 

- There are some important missing descriptions regarding the dataset and the training procedure. How is the data split between training and testing? is it subject-wise or dataset-wise? Are the same subjects used both for training and testing? functional alignment between subjects if direct comparison between them?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: 1)	For each of the metrics, it would be helpful to understand what a good value is, and if these are getting at all close to that value. Currently, it is difficult to interpret whether the observed differences are at all significant. For instance, for the semantic representations one could use a baseline score that is two different samples in time from the same audio clip.  And maybe a PCC and PSNR score could be referenced to samples with additive Gaussian noise of various SNR levels at the waveform?  

2)	Listening to the supplement audio files, I am not so sure that the proposed method is actually doing something reasonable. The samples “sound” more natural (which is expected from including a diffusion model in the pipeline), but they are often completely different sounds than the initialized audio, essentially hallucinations. PSNR levels of ~15-18 seem like they might be noise, relative to the original sound.  

3)	In the decoding section, one of the experiments is on male/female decoding which is called a “semantic” task. However, I believe a decent amount of male/female decoding can be achieved from simply looking at the overall power spectrum of the sound (male voices are generally lower than female voices). Thus, this doesn’t seem particularly “semantic”.  

4)	The metrics are not defined enough in the main text (Lines 311-319). At a minimum, the acronyms need to be defined. For instance, what is “PCC” and “PSNR”? Although some audio researchers may be familiar with these measurements, a more general ICLR attendee may not. 

5)	How many repetitions of each sound are present in each dataset and are these averaged for the analysis? More generally, how is fMRI measurement noise taken into account for the analysis? I.e., is there a sort of “noise ceiling” defined on the reconstruction? 

Somewhat minor suggestions for specific lines: 
* Line 075-077: This sentence doesn’t make sense to me. How does the “high-dimensionality” play a role here? The main challenge of fine-grained decoding is the lack of resolution in time and the inherently noisy signal of the fMRI response.  
* Lines 086-088: These references seem out of place, as the decomposition of sound into difference frequencies in the cochlea dates much further back than these papers. Some good references might be work by Shihab Shamma (maybe https://ieeexplore.ieee.org/abstract/document/119739, or https://pubmed.ncbi.nlm.nih.gov/16158645/) , but I would encourage the authors to perhaps cite classic textbooks or review articles on auditory processing 
* Lines 147-152: Which features are used in CLAP? The final output features or an intermediate stage? This should be mentioned in this section.  
* Line 231: “unpatchify” is not defined.
soundness: 2
strengths: The paper tackles an interesting and challenging problem of trying to decode auditory fMRI activity. I like how it ties back into classic ideas of course-to-fine reconstruction. It also takes advantage of many recently proposed auditory models, similarity metrics, and auditory datasets.
summary: This work designs a system to reconstruct audio signals from human fMRI data collected while listening to natural sounds. The system separately handles fine-grained information (i.e., the precise timing and frequencies of the sounds) and course-grained information (semantic information such as the class of sound) by using different neural network features for each branch of the architecture. The authors evaluate their method on three different fMRI datasets and multiple different evaluation methods capturing the fine-grained and semantic nature of the sounds.
weaknesses: * The paper combines many different pre-trained systems. This makes the methods used difficult to understand, but also makes it hard to evaluate where things might be going wrong, and which pieces are critical. Due to the unknown inherit biases of each system, it is difficult to see how the work could actually be useful for understanding the human auditory system.
 
* As currently written, the presentation of results is confusing. The text should point to each set of results when they are discussed, rather than stating (line 357-359) that qualitative results are displayed in Tables 1 and 2. Additionally, it would be helpful if the columns were in some way separated or labeled for the “fine grained” vs. “semantic” measures.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Figure 5 is confusing. What are the ‘features’ that are input into the SVM? And given the results and discussion that there is ‘little semantic content in the semantic features’ (line 449) the initial claim that semantic prompts during decoding ‘[enhances] the quality of reconstructed audio’ seems perhaps exaggerated (lines 25-26). 

A lot of the text is pushed to the appendix, making the official 10 pages lacking in sufficient detail and discussion. 

A lot of the writeup feels pretty inside baseball to this reviewer (or perhaps this reviewer is just too far outside this particular game). e.g. CLAP is not explained (though there is a reference) until page 3.
soundness: 3
strengths: Technically, the method introduced here is interesting and seems sophisticated. 

The paper is well-written and makes use of highly relevant contemporary work—including studies done in machine learning and neuroscience—and uses this grasp of the literature to provide interpretations on different steps in their method. 

The paper does a nice job of comparing many different methods in addition to their own.
summary: This paper presents a new method for reconstructing audio signals from fMRI responses to sounds. Specifically, it proposes a course-to-fine decoding strategy in which  fMRI responses from auditory cortex are decoded in a course-grained fashion into the semantic space of CLAP (which is not defined until page 3), and decoded in a fine-grained fashion into acoustic space.

I would not recommend this paper for acceptance. While the high-level approach seems interesting, the purpose of integrating neural data seems lacking in motivation, and the use of pre-trained representations in both training and evaluation seem to present a major confound. 



Figure 5 is confusing. What are the ‘features’ that are input into the SVM? And given the results and discussion that there is ‘little semantic content in the semantic features’ (line 449) the initial claim that semantic prompts during decoding ‘[enhances] the quality of reconstructed audio’ seems perhaps exaggerated (lines 25-26). 

A lot of the text is pushed to the appendix, making the official 10 pages lacking in sufficient detail and discussion. 

A lot of the writeup feels pretty inside baseball to this reviewer (or perhaps this reviewer is just too far outside this particular game). e.g. CLAP is not explained (though there is a reference) until page 3.
weaknesses: 1. Lack of Clear Motivation: This reviewer got stuck at square one: what is the goal of this work? The authors don't tell us. If the goal were to understand brain function this work would proceed differently, for example separately analyzing primary versus non primary auditory cortex, asking which brain regions are best modeled by each component of the model, comparing decoded signals to human behavioral data, etc. Alternatively, perhaps it could be intended for BCI applications. But what would those applications be? If you have access to a person's auditory cortex then you would already have access to the sound they were hearing, so there would be no point trying to decode that information from the brain. A paragraph in the appendix (A.7) attempts to provide a motivation, but it doesn't make sense to me. For example it mentions that this work could  "aid individuals with voice disorders". But if an individual had a voice disorder, one would want to generate the intended spoken information, not the heard information from auditory cortex.

It’s unclear how the findings here contribute to the ‘comprehension of the human auditory system’ as stated in the abstract (lines 27-29). While the coarse-to-fine method is inspired by the human auditory system, it is not quite a model of ‘each physiological structure of the auditory processing pathway’ (lines 96-97)

2. Possible confound.
The results on many of the metrics suggest that the direct decoding methods better reconstruct audio than the C2F-LDM proposed here (Table 1, 2). The novel C2F-LDM method does improve on measures of FD, FAD, KL, and CLAP, but these results seem to present a confound. The reconstruction makes use of many model features from pre-trained models and are subsequently evaluated on their similarity to pre-trained model features. Thus would higher scores not be unsurprising?

3. Is the fMRI data even relevant?
The paper should address the role of fMRI data and how it improves the methods described here. The majority of steps requires pre-trained models and predicting their ‘ground truth’ representations, but the optimal value of P=0.25 seems to suggest that using these ground truth representations without the neural data may improve reconstruction.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 4
questions: Questions:
- Why was sex selected as the semantic class for the brain-to-speech case?
- Just confirming that: semantic features of guidance = course-grained features in the CLAP space? (might be helpful to state this more explicitly)
soundness: 3
strengths: Overall, this paper represents a novel improvement for brain-to-audio decoding and its acknowledgement would benefit the advancement of the field. The approach introduced is novel, performance state-of-the-art, and the presentation clean.
summary: This paper proposes a coarse-to-fine framework for audio reconstruction from fMRI brain recordings that outperforms the leading solely fine-grained approaches. This approach draws inspiration from the hierarchical processing found in the human auditory system. It first projects the audio and fMRI into a coarse-grained semantic embedding in the CLAP (Wu et al, 2023) space before paring that embedding again with the fMRI signal for fine-grained decoding, generating features in the space of AudioMAE (Huang et al, 2022). Lastly, an LDM is used to reconstruct the mel-spectrogram of the stimulus audio with the fine-grained embedding, before it is converted to the waveform using a pretrained HiFiGAN (Kong et al, 2020) vocoder.

The authors evaluate the performance of this framework over three different datasets encompassing three different classes of audio / reconstruction task (sound, music, and speech). The quality of the reconstructed audio is assessed by FD, FAD, KL, and CLAP score and the mel-spectrograms are evaluated using PCC and PSNR. Performance is benchmarked against direct decoding approaches and fine-grained only approaches. The authors also offer an investigation into the quality of the semantic information captured through these approaches.
weaknesses: There are a few places where choices are made without explanation and parts of the discussion on the semantic analysis are unclear. For example, in the discussion of the semantic analysis it is stated that there is less semantic richness in the brain-to-speech case because listeners might be more focused on content. However, this effect does not necessarily seem to hold for the fine-grained only decoding (which is not addressed). It seems more likely that coarse-grained decoding may just be ill-suited to capturing semantic quality from speech as the signal clearly exists given the performance of the solely fine-grained approach. This represents a potential limitation of this framework for brain-to-speech.

Meta-review: This submission provides a method for audio reconstruction from fMRI, presumably to be used for the study of those representations/encodings. The authors provide validation on three open datasets of reconstruction, and discussion of their work in the context of modelling in the auditory pathway.

There is a very healthy discussion of multiple aspects, but there are specific objections raised about motivation, contribution, and usefulness of the proposed work that remain unaddressed. In particular, reviewers `DqUq`, `JVh6`, and `Fjq3` all found that the modelling claims are not well supported, even after discussion. Quoting the response by `Fjq3` as summary: 
> The motivation and utility of pre-trained model features in your method still also confuses me. The different components of your framework are inspired by the brain’s hierarchy, but you also claim that they ‘provide valuable insights into the human auditory system’ and reveal something of its ‘processing mechanisms.’ This reasoning seems a bit circular to me. 

The other two reviewers raising this point responded similarly, highlighting a tension between engineering efforts and contribution to understanding (of the brain). This, in my opinion, speaks to a conceptual flaw in the work, or at the very least an unsuccessful attempt to provide two different things in one computational model, with an imperfect attempt to anneal their differences.

I find that there is merit in the present work, as evidenced by the relatively high scores from one reviewer, and based on comments from the lower scoring reviewers. However, I also find that there is still unrefined questions and conceptual issues that remain unanswered and unaddressed. Based on scores alone this work would be difficult to include; given it's particular flaws and the comments provided by reviewers, I think this work needs restructuring and possibly further meditation on its high level direction, choosing between reconstruction for reconstruction's sake (an engineering feat, though with dubious utility) or modelling interpretability, even noting the loaded nature of "interpretation".

Decision: Reject