DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection

Authors: sheng  yan
cunhang  fan
hongyu  zhang
xiaoke  yang
jianhua  tao
zhao  lv
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:37:22

Abstract: At a cocktail party, humans exhibit an impressive ability to direct their attention. The auditory attention detection (AAD) approach seeks to identify the attended speaker by analyzing brain signals, such as EEG signals. 
However, current AAD algorithms overlook the spatial distribution information within EEG signals and lack the ability to capture long-range latent dependencies, limiting the model's ability to decode brain activity.
To address these issues, this paper proposes a dual attention refinement network with spatiotemporal construction for AAD, named DARNet, which consists of the spatiotemporal construction module, dual attention refinement module, and feature fusion \& classifier module. Specifically, the spatiotemporal construction module aims to construct more expressive spatiotemporal feature representations, by capturing the spatial distribution characteristics of EEG signals. The dual attention refinement module aims to extract different levels of temporal patterns in EEG signals and enhance the model's ability to capture long-range latent dependencies. The feature fusion \& classifier module aims to aggregate temporal patterns and dependencies from different levels and obtain the final classification results.
The experimental results indicate that DARNet achieved excellent classification performance, particularly under short decision windows. While maintaining excellent classification performance, DARNet significantly reduces the number of required parameters. Compared to the state-of-the-art models, DARNet reduces the parameter count by 91\%. Code is available at: https://github.com/fchest/DARNet.git.

Main paper: 
 introduction: 
 The auditory attention detection (AAD) aims to study human auditory attention tendencies by analyzing brain signals [1, 2, 3] . The auditory attention refers to the ability of individuals to isolate or concentrate on specific sounds, which aids them in focusing on a single speaker amidst a multispeaker environment, a scenario commonly referred to as the "cocktail party scenario" [4] . However, this ability may diminish or even completely disappear for individuals with impairment. Therefore, finding solutions to assist these individuals in overcoming this challenge has become an urgent matter. [5] have demonstrated a close connection between auditory attention and brain activity, which indicates that researchers can study auditory attention by analyzing brain activity.Following this concept, many methods such as electrocorticography (ECoG) [5] , magnetoencephalography [6, 7] , and electroencephalography (EEG) [8, 9] have been used to implement auditory attention detection. Among these methods, EEG-based approaches are widely applied in AAD due to their high temporal resolution, non-invasive mode, and excellent maneuverability [9, 10, 11] .According to the conclusions of Mesgarani and Chang [5] , previous studies have utilized stimulusreconstruction or speech envelope reconstruction methods, which necessitate clean auditory stimuli as input [12, 13] . However, in most real-world scenarios, environments consist of multiple sounds simultaneously. Listeners are exposed to a mixture of these sounds, posing a challenge in obtaining clean auditory stimuli. Therefore, in recent years, the academic community has increasingly focused solely on utilizing EEG signals as input for AAD research [14, 15, 16] . The research method proposed in this paper also exclusively utilizes EEG signals.Traditional AAD tasks relied on linear models to process EEG signals [17, 18] . However, brain activity is inherently nonlinear, posing challenges for linear models in capturing this complexity. Consequently, they necessitate longer decision windows to extract brain activity features [19] . Some previous studies have indicated that decent decoding performance can be achieved by analyzing different spatial distribution features within each frequency band. These studies project the extracted differential entropy (DE) values onto 2D topological maps and decode them with convolutional neural networks [20, 15] . However, EEG signals are fundamentally time-series data, these methods overlook the dynamic temporal patterns of EEG signals. Other studies analyze EEG signals only in the time domain. For instance, they use long short-term memory (LSTM) networks to capture dependencies within EEG signals and achieve decent decoding performance [2] . However, these studies only focus on the temporal information within EEG signals, neglecting the spatial distribution features, which reflect the dynamic patterns of different brain regions when receiving, processing, and responding to auditory stimuli. Meanwhile, numerous noise points and outliers make it difficult to capture long-range latent dependencies.To address these issues, this paper proposes a dual attention refinement network with spatiotemporal construction for AAD, named DARNet, which effectively captures the spatiotemporal features and long-range latent dependencies of EEG signals. Specifically, our model consists of three modules: (1) Spatiotemporal Construction Module. The spatiotemporal construction module employs a temporal convolutional layer and a spatial convolutional layer. The temporal convolutional layer effectively captures the temporal dynamic features of EEG signals, and the spatial convolutional layer captures the spatial distribution features among different channels, thereby constructing a robust embedding for the next layer. (2) Dual Attention Refinement Module. The dual-layer self-attention refinement module consists of two layers, each comprising a multi-head self-attention and a refinement layer. This design is intended to capture long-range latent dependencies and deeper sequence patterns in EEG signals. (3) Feature Fusion & Classifier Module. The attention features generated by the dual-layer self-attention refinement module, comprising both shallow and deep levels, are fed into the feature fusion module to obtain richer representations, enhancing the model's robustness and generalization. The fused features are input into a classifier to predict the auditory attention tendencies of the subjects.To this end, We evaluated the decoding performance of DARNet on three datasets: DTU, KUL, and MM-AAD. The results demonstrate that DARNet outperforms the current state-of-the-art model on all three datasets. The main contributions of this paper are summarized as follows:• We propose a novel auditory attention decoding architecture, which consists of a spatiotemporal construction module, a dual attention refinement module, and a feature fusion module. This architecture could fully leverage the spatiotemporal features and capture long-range latent dependencies of EEG signals.  
methodology: 
 The previous AAD methods overlooked the influence of spatial distribution characteristics on decoding performance and struggled to capture the long-range dependencies in EEG signals [14, 20] .To address these issues, we proposed DARNet, which consists of a spatiotemporal construction module, a dual attention refinement module, and a feature fusion & classifier module, see Figure 1 .Our proposed DARNet effectively captures the spatiotemporal features of EEG signals and has the capability to capture long-range latent dependencies in EEG signals.By employing a moving window on the EEG data, we obtain a series of decision windows, each containing a small duration of EEG signals. Let R = [r 1 , ..., r i , ..., r N ] ∈ R T ×N represents the EEG signals of a decision window, where r i ∈ R N ×1 represents the EEG data at the i-th time point within a decision window, contains N channels. Here N represents the number of EEG channels and T denotes the length of the decision window. Before inputting EEG data into the DARNet, we employ a common spatial patterns (CSP) algorithm to extract raw features from the EEG data under different brain states [21, 22] .EQUATIONwhere CSP (•) represents the CSP algorithm, E ∈ R N ×T represents the processed EEG signal. c_in is the components of the CSP algorithm and T denotes the length of the decision window.EEG signals record the brain's neuronal electrical activity, varying over time and reflecting activity patterns and connectivity across brain regions [23] . By constructing spatiotemporal features from EEG signals, it's possible to analyze the brain's response patterns to auditory stimuli. However, previous studies only focused on local temporal patterns in EEG data, overlooking the spatial distribution features. Therefore, in addition to the conventional use of temporal filters, we introduced a spatial filter [24] to construct the spatiotemporal features of EEG signals.Firstly, we use temporal convolution layers to capture the instantaneous changes in EEG signals, thereby constructing the temporal patterns E t of the EEG signals. It can be formulated as follows:EQUATIONwhere T emporalConv2d( (3) where SpatialConv2d(•) performs an 2-D convolutional filters with a c_in × 1 kernel size on spatial dimension. By doing so, we not only capture the temporal patterns in EEG signals but also integrate the spatial distribution characteristics of EEG signals, thereby constructing input embedding S containing comprehensive spatiotemporal information for the next layer. This integrated input better reflects the complex features within EEG signals, providing richer information for subsequent analysis and processing.Previous psycho-acoustic research has demonstrated that human attention is a dynamic and timerelated activity [25, 26] . The brain activity from the preceding moment can profoundly influence subsequent brain activity [27] . However, previous AAD algorithms were hindered by model depth and the noise and outliers in EEG data, making them ineffective at capturing the long-range latent dependencies in EEG signals.To address this issue, we proposed a dual self-attention mechanism, which has greater potential for capturing long-range latent dependencies and deeper sequence patterns in EEG signals. Inspired by Zhou et al. [28] , Yu et al. [29] , We introduced a self-attention refinement operation, which refines the dominant temporal features through convolution and pooling operations, compressing the original EEG series of length T to half its length. This self-attention refinement operation reduces the impact of noise and outliers, while also decreasing the model's parameter count. This enhances the model's generalization and robustness. The single-layer attention refinement module can be formulated as follows:F = M axP ool(ELU (Conv1d(M ultiHeadAttention(x)))) (4) where M ultiHeadAttention(•) denotes multi-head self-attention algorithm [30] , Conv1d(•) represents an 1-D convolutional filters (kernel width=3) on time dimension. The ELU (•) is the activation function proposed by Clevert et al. [31] , M axP ool(•) denotes a max-pooling layer with stride 2.Before applying the temporal attention feature extraction module, we add the absolute positional embedding [30] to the input embedding S as follows:EQUATIONwhere s i represents the embedding vector of the i th time step, p i ∈ R d model represents i th time step position.To obtain different levels of temporal features from EEG signals and to capture the long-range latent dependencies, we stacked two of the above attention refinement extraction modules.EQUATIONF 2 = M axP ool(ELU (Conv1d(M ultiHeadAttention(F 1 )))) ∈ R d model × T 4 (7)where F 1 and F 2 contain different levels of dependencies and temporal patterns in the EEG signals, respectively.Features at different levels can reflect various characteristics of the pattern. By optimizing and combining these different features, it preserves effective discriminative information from features at different levels while also to some extent eliminating redundant information [32] . Therefore, we designed a feature fusion module as follows:First, we project the features F 1 and F 2 into the same dimension.EQUATIONEQUATIONwhere AdaptiveAvgP ool(•) denotes an adaptive average pooling layer, Linear denotes a linear layer.Second, We concatenate feature F ′ 1 and F ′ 2 to obtain the fused feature vector F .EQUATIONFinally, we employ a fully connected layer to obtain the final auditory attention prediction.EQUATIONwhere w and b are the weight and the bias of the fully connected layer, p denotes the predicted direction label. In the training stage, we employ the cross entropy loss function to supervise the network training.3 ExperimentsThe dual attention refinement module and feature fusion module of DARNet comprehensively capture temporal patterns and dependencies at different levels, enabling the model to better understand the temporal dynamics within EEG signals.This holistic consideration of features at different time scales is crucial for the analysis of EEG data.3. Robust feature representation: Despite observing that removing the feature fusion module did not lead to a significant decrease in accuracy across the three datasets, the performance variability of DARNet increases substantially. We believe that the feature fusion module integrates temporal patterns and dependencies at different levels, enabling the model to better understand and utilize the complex relationships within the data, thus enhancing the robustness and generalization of the model. 4.21M MBSSFCC [15] 83.91M DBPNet [20] 0.91M DARNet (ours) 0.08M 
dataset: 
 In this section, we conduct experiments on three publicly available datasets, namely KUL [33, 34] , DTU [35, 36] and MM-AAD [20] , which are commonly used in auditory attention detection to evaluate the effectiveness of our DARNet. KUL and DTU only contain EEG data of the auditory stimulus scenes. MM-AAD contains EEG data of the audio-only scene and the audio-visual scene.We summarize the details of the above datasets in Table 1 .1) KUL Dataset: In this dataset, 64-channel EEG data were collected from 16 normal-hearing subjects using a BioSemi ActiveTwo device at a sampling rate of 8,192 Hz in a soundproof room. Each subject was instructed to focus on one of two simultaneous speakers. The auditory stimuli were filtered at 4kHz and set at 60dB through in-ear headphones, which contain four Dutch short stories, narrated by three male Flemish speakers. Two listening conditions were employed: dichotic (dry) presentation with one speaker per ear, and headrelated transfer function (HRTF) filtered presentation, simulating speech from 90°left or right. Each subject listened to 8 trials, which lasted 6 minutes.2) DTU Dataset: In this dataset, 64-channel EEG data were collected from 18 normal-hearing subjects using a BioSemi ActiveTwo device at a sampling rate of 512 Hz. Each subject was instructed to focus on one of two simultaneous speakers, who presented at 60°relative to the subject. The auditory stimuli were set at 60dB through ER-2 earphones, which contain Danish audiobooks, narrated by three male speakers and three female speakers. Each subject listened to 60 trials, which lasted 50 seconds.3) MM-AAD Dataset: In this dataset, 32-channel EEG data were collected from 50 normalhearing subjects (34 males and 16 females) at a sampling rate of 4kHz, following the 10/20 international system. Each subject was exposed to both audio-only and audio-visual stimuli.They were instructed to focus on one of two simultaneous speakers, who presented at left or right spatial direction relative to the subject. The auditory stimuli comprised 40 classic Chinese stories narrated by both male and female speakers. Each subject listened to 20 trials, which lasted 165 seconds.To fairly compare the performance of the proposed DARNet model, specific preprocessing steps are applied to each dataset (KUL, DTU, and MM-AAD). For the KUL dataset, the EEG data were firstly re-referenced to the average response of mastoid electrodes, then bandpass filtered between 0.1 Hz and 50 Hz, and finally down-sampled to 128 Hz. For the DTU dataset, the EEG data were filtered to remove 50 Hz linear noise and harmonics. Eye artifacts were eliminated through joint decorrelation and the EEG data were re-referenced to the average response of mastoid electrodes. Finally, the EEG data were down-sampled to 64 Hz. For the MM-AAD dataset, the EEG data were firstly bandpass filtered between 0.1 Hz and 50 Hz, then removed 50 Hz noise through a notch filter. Additionally, eye artifacts were eliminated, and further noise removal was achieved, using independent component analysis (ICA). Finally, the EEG data were down-sampled to 128 Hz.We evaluated our proposed DARNet model and compared it with other state-of-the-art models under three decision window lengths: 0.1s, 1s, and 2s. Specifically, we selected three publicly available models as our baseline for comparison: SSF-CNN [37] , MBSSFCC [15] , and DBPNet [20] .In previous AAD research, the accuracy of auditory attention prediction classification has been used as a benchmark for model performance. We followed this convention and evaluated our proposed DARNet on the KUL, DTU, and MM-AAD datasets. As follows, we take the KUL dataset with a 1-second decision window as an example to illustrate implementation details, including training settings and network configuration.Firstly, we set the proportions of the training, validation, and test sets to 8:1:1. For each subject of the KUL dataset, we get 4,600 decision windows for training, 576 decision windows for validation, and 576 decision windows for testing. Meanwhile, we set the batch size to 32, the maximum number of epochs to 100, and employ an early stopping strategy. Training will stop if the loss function value on the validation set does not decrease for 10 consecutive epochs. Additionally, we utilize the Adam optimizer with a learning rate of 5e-4 and weight decay of 3e-4 to train the model. The DARNet is performed using PyTorch.Before inputting EEG data into the DARNet, we employ the CSP algorithm to extract raw features E ∈ R 128×64 from the EEG data. The data is transposed and expanded, represented as E ′ ∈ R 1×64×128 . Then, through the spatiotemporal construction module (c in is set to 16), we can get embedding data S ∈ R 16×1×128 . After dimensionality reduction, transposition, and the addition of absolute positional embedding, the data is fed into the dual attention refinement module, resulting in two distinct level features, F 1 ∈ R 16×64 and F 2 ∈ R 16×32 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: All the theorems, formulas, and proofs in the paper have been properly numbered and cross-referenced, fulfilling the guidelines provided.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
performance of darnet: 
 To evaluate the performance of DARNet, we conducted comprehensive experiments under decision windows of 0.1-second, 1-second, and 2-second, respectively, as shown in Figure 2 . Additionally, We compared our DARNet with other advanced models, as shown in Table 2 . The results are replicated from the corresponding papers.DARNet has outperformed the current state-of-the-art models on the KUL, DTU, and MM-AAD datasets, achieving further enhancements in performance. On the KUL dataset, the DARNet achieves average accuracies of 91.6% (SD: 4.83%), 96.2% (SD: 3.04%), 97.2% (SD: 2.50%) under 0.1-second, 1-second and 2-second decision window, respectively. On the DTU dataset, the DARNet achieves average accuracies of 79.5% (SD: 5.84%) for 0.1-second decision window, 87.8% (SD: 6.02%) for 1-second decision window, 89.9% (SD: 5.03%) for 2-second decision window, respectively. On the MM-AAD dataset, the DARNet also demonstrates outstanding decoding accuracies of 94.9% (SD: 4.79%) for 0.1-second, 96.0% (SD: 4.00%) for 1-second, 96.5% (SD: 3.59%) for 2-second in the audio-only scene, and 95.8% (SD: 4.04%) for 0.1-second, 96.4% (SD: 3.72%) for 1-second, 96.8% (SD: 3.44%) for 2-second in the audio-visual scene.Overall, DARNet's decoding accuracy increases with larger decision windows, consistent with prior research [15, 14] . This is because longer decision windows provide more information for the model to make judgments while also mitigating the impact of individual outliers on the predictions. However, DARNet still maintains excellent performance under the 0.1-second decision window. Additionally, we observe that in the MM-AAD dataset, performance is better in the audio-visual condition compared to the audio-only condition in two different scenarios. We attribute this improvement to the visual cues aiding humans in localizing sound sources.We conducted comprehensive ablation experiments by removing the spatial feature extraction module, the temporal feature extraction module, and the feature fusion module. Additionally, we supplemented our study with ablation experiments using a single-layer attention refinement module on the KUL and DTU dataset, referred to as single-DARNet. All experimental conditions remained the same as in previous settings. Additionally, we ensured that all model network parameters were fully optimized to guarantee that the model's performance reached its best under each condition, whether a module was removed or added. The results of the ablation experiments are shown in Table 3 .Experimental results show that on the DTU dataset, after removing the spatial feature extraction module from DARNet, the average accuracy decreased by 10.1% under a 0.1s decision window, 13.1% under a 1s decision window, and 12.0% under a 2s decision window. After removing the temporal feature extraction module, the average accuracy for the 0.1s, 1s, and 2s decision windows decreased by 10.9%, 14.2%, and 11.9%, respectively. After removing the feature fusion module, the average accuracy decreased by 2.5% under a 0.1s decision window, 1.6% under a 1s decision window, and 1.4% under a 2s decision window. On the KUL dataset and the MM-AAD dataset, removing the aforementioned modules also resulted in similar trends of decreased average accuracy. To ensure fairness in comparisons, we aligned our previous experimental setup and data processing methods with those of DBPNet. However, during the code review for the final version of this paper, we found that DBPNet applies CSP to the data prior to dataset splitting, which can lead to data leakage. Consequently, we corrected the processing steps and conducted additional experiments, as shown in Table 4 . The data preprocessing steps for the supplementary experiments are as follows:1. Split each trial of the dataset into the first 90% for training and the last 10% for testing.2. Fit the CSP transformation matrix using the training data, project the data to extract features, and apply the matrix to the testing data. 3. Apply a sliding window with 50% overlap to the processed training data, then randomly split into a training (90%) and a validation (10%) set. Similarly, apply a sliding window to the processed testing data as the final test set.Following these steps, we rigorously avoided risk of data leakage. Experimental results show that both corrected DBPNet 2 and DARNet 2 exhibit a performance decline compared to the original DBPNet 1 and DARNet 1 . However, DARNet 2 still achieves state-of-the-art performance under most test conditions, as shown in Table 4 . For instance, on the KUL dataset with a 0.1-second decision window, DARNet 2 maintains a decoding accuracy of 89.2%. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We have detailed our model and experimental setup thoroughly in the Methodology and Experiments sections, providing all necessary information to reproduce the main experimental results. 
comparative analysis: 
 To further evaluate the performance of our proposed DARNet, we compared it with other advanced AAD models, as shown in Table 2 . The results indicate that our DARNet has achieved a significant improvement over the current state-of-the-art results.For example, on the DTU dataset, our DARNet has shown relative improvements of 27.2%, 18.8%, 9.7%, and 5.9% for 0.1-second decision window, compared to the SSF-CNN, MBSSFCC, EEG-Graph Net and DBPNet models, respectively. Compared to the SSF-CNN, MBSSFCC, BSAnet, EEG-Graph Net, and DBPNet models, the relative improvements achieve 25.8%, 16.1%, 5.7%, 11.6%, 4.6% for 1-second decision window, and 22.6%, 14.2%, 5.0%, 13.2%, 3.9% for 2-second. On both the KUL and MM-AAD datasets, DARNet has achieved similar improvements compared to the state-of-the-art models. The particularly outstanding results achieved across all three datasets under the 0.1-second decision window indicate the potential of DARNet for real-time decoding of auditory attention.Overall, the excellent performance of DARNet across different datasets and decision windows demonstrates its robustness and versatility in various contexts. This further validates the potential of DARNet as an effective EEG analysis model and provides strong support for its widespread application in real-world scenarios.As shown in Table 3 and Figure 3 , compared with removing the spatial feature extraction step, removing the temporal feature extraction step, removing the feature fusion module, and using a single-layer attention refinement module, we believe DARNet performs excellently for the following reasons:1. Integrating multiple sources of information: DARNet integrates temporal and spatial distribution features from EEG signals, constructing richer and more robust spatiotemporal features. This enables the model to comprehensively understand the spatiotemporal information within EEG signals, thereby enhancing the understanding of brain activity. In contrast, removing any single feature may lead to information loss or the inability to capture the transient changes in EEG signals, thereby impacting the model's performance.We compare the training parameter counts of our DARNet, SSF-CNN [37] , MBSSFCC [15] , and DBPNet [20] , with the results shown in Table 5 . The parameter count of DARNet is 51.6 times lower than that of SSF-CNN, 1331.5 times lower than that of MBSSFCC, and 10.4 times lower than that of DBPNet. Compared to other models, DARNet demonstrates superior parameter efficiency. Despite having fewer parameters, DARNet maintains good performance, indicating its ability to be applied in resource-constrained environments for AAD analysis, thus demonstrating practical utility. 
conclusion: 
 In this paper, we propose the DARNet, a novel dual attention refinement network with spatiotemporal construction for auditory attention detection. By employing spatial convolution operations across all channels, DARNet effectively leverages the spatial information embedded in EEG signals, thereby constructing a more robust spatiotemporal feature. Additionally, DARNet integrates dual attention refinement and feature fusion techniques to comprehensively capture temporal patterns and dependencies at various levels, enhancing the model's ability to capture the temporal dynamics within EEG signals. We evaluate the performance of DARNet on three datasets: KUL, DTU, and MM-AAD. DARNet achieves a decoding accuracy of 96.2% on the 1-second decision window of the KUL dataset and 87.8% on the 1-second decision window of the DTU dataset, demonstrating significant improvements compared to current state-of-the-art models. The experimental results validate the effectiveness and efficiency of the DARNet architecture, indicating its potential for practical applications. In future research, we plan to further explore DARNet's performance on cross-subject tasks to verify its generalization and robustness.NeurIPS Paper Checklist Justification: We outlined the limitations in the conclusion. Current AAD research primarily employs two experimental strategies: subject-dependent and subject-independent.Subjectdependent refers to the training and evaluating procedures containing only samples from a single subject, while subject-independent contains samples from all subjects in the dataset.Our proposed model has been validated under the subject-dependent condition and has demonstrated exceptional results. However, further exploration and resolution of the issue of inter-subject variability are necessary to enable our model to be more widely applicable to real-world brain-computer interface applications. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: Our research does not directly produce societal impacts as it focuses on technical advancements in a specific field without direct societal applications. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: Auditory and audio-visual attention tasks are quite difficult to model, characterize and enable prediction. Specifically, these are not well studied from a learnability point of view. The authors propose a model that combines deep signal processing / classical insights with established spatio-temporal modeling (convolutions in time with attention in space) to build a better than state of the art EEG-driven multi-modal model for attention. The resulting work also improves performance quite significantly compared to baselines / sota. 

There is general agreements among reviewers regarding the impact, and I would suggest the authors to address / capture content from comments on rationale for proposed models' novelty, single vs. dual attention and a summary from additional experimental insights in the final version for publication.

Reviews: review: paper_summary: paper_summary: The manuscript is aim to capture the and spatial distribution information and long-range dependencies in EEG signals. Two modules are designed to solve the upper two challenges, spatiotemporal construction module and dual attention refinement module. The experiment  have shown the superiority of the proposed method. And the analysis have shown the advantage of each module.
strength_weakness: strength_weakness: 1)The manuscript is lack of novelty.
2)It is less clarity in session Abstract on which dataset the accuracy is improved using different sliding window size. 
3)There is no analysis that the level of computation cost and time consuming of the mentioned methods in Table 2.
questions: questions: 1)Many researchers have done this work, to capture the long-range dependencies and spatial distribution information within EEG signals. Is there any more innovation of your proposed method? (From framework design and experimental performance.)
2)What sliding window size is the best performance with? Have you ever try other size longer than 2 s?
3)Please prove that the long-range dependencies are captured within EEG signals.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a new architecture for auditory attention detection (AAD) that consists of three key components: 1) Convolutional layers applied to the temporal and spatial dimensions of EEG signals in a sequential manner to extract features. 2) Two attention layers to process these features. 3) A feature fusion module to combine the outputs of the two attention layers. The method was tested on three different datasets and achieved state-of-the-art (SOTA) classification results.
strength_weakness: strength_weakness: * Using convolutional layers on the temporal dimension followed by the spatial dimensions of EEG signals is not a novel idea. Classic EEG+DL+BCI papers, such as ConvNet (Schirrmeister, 2017) and EEGNet (Lawhern, 2018), already employ similar concepts. Many EEG+DL+AAD papers also adopt spatial processing modules (e.g., references 3 and 14 in the paper). The author should provide a more in-depth discussion on why their spatial module is more effective than existing methods. Specifically, they should highlight any unique innovations or improvements their approach offers over previous works.
* It would be interesting to see the comparison between dual-attention layer and one attention layer. 
* It is crucial to explicitly mention in the methods section that this is a subject-dependent study. 
* There is a typo in the notations on row 91.
questions: questions: How did 48 minutes of recording in the KUL dataset yield over 5000 1-second decision windows? If a sliding window with overlap was used, please describe the procedure in detail to ensure that there are no repeated signal segments in the training and test sets.
limitations: limitations: * Reproducibility of the paper would be significantly enhanced if the author provided the full code.
paper_summary: paper_summary: The paper proposes DARNet, a dual attention refinement network with spatiotemporal construction for auditory attention detection (AAD). The network captures spatiotemporal features and long-range latent dependencies from EEG signals, leading to improved classification accuracy and reduced parameter count compared to state-of-the-art models.
strength_weakness: strength_weakness: 1. The experiments are all subject-dependent, which may limit the generalizability of the results. Cross-subject conditions could be more relevant for practical applications, where the model needs to generalize across different individuals.
2. While the model achieves high performance, its complexity might make it difficult to interpret and understand the underlying mechanisms contributing to its success. More insights into the workings of the dual attention mechanism and its impact on EEG signal processing would be beneficial.
questions: questions: NA
limitations: limitations: Limitations are discussed in the checklist.
paper_summary: paper_summary: This paper makes a substantial contribution to the field of auditory attention detection (AAD) by presenting a more accurate and efficient model, the dual attention refinement network with spatiotemporal construction (DARNet). The authors effectively address critical limitations in current AAD algorithms, specifically the lack of spatial distribution information and the inability to capture long-range latent dependencies within EEG signals. The methodological innovations and empirical results suggest that DARNet represents a significant advancement in decoding brain activity related to auditory attention tasks. The experiments show significant improvements over state-of-the-art methods.
strength_weakness: strength_weakness: Overall, the writing of this paper is relatively clear and the structure is complete. However, there are still some issues, such as grammatical errors and some sentences that are not clearly articulated. 3）There are many writing problems. The authors should further polish this manuscript:
1）Line 100-102: The expression is not very clear.
2）Line 103: "However previous" -> "However, previous".
3）Line 188: "or" -> "and".
4）Line 292: "increased" -> "increases".
questions: questions: 1. Why did you use a dual attention refinement module instead of three layers or more? The authors should provide more analysis
2. Did you lose the FC layer for the final classification part in Figure 1?
limitations: limitations: The DARNet has only been validated under the subject-dependent condition, and it may not perform as well under subject-independent conditions.

Comments: title: Thank you for your prompt and comprehensive response to my review.
comment: We sincerely thank you for your positive feedback and for raising the rating of our paper. We greatly appreciate your recognition of our additional experiments and the overall contribution of our work. We are delighted that our paper is considered a significant contribution to the field and will be of interest to the community.
We greatly appreciate your constructive comments and are pleased that our cross-subject study addressed your concerns.
We sincerely appreciate your recognition of our supplementary experiments and your decision to increase the score.
Thanks for your supplementary experiments and I have increased the score from 5 to 6.
Your detailed answers have effectively addressed my concerns. The additional cross-subject experiments are a valuable addition, significantly strengthening the overall contribution of the paper. Additionally, the authors have provided a thorough rationale, supported by ablation studies, demonstrating the effectiveness of their approach within the constraints of low-latency scenarios.
I am confident that this paper makes a significant contribution to the field of auditory attention detection and will be of great interest to the community. Therefore, I am raising my rating to "Strong Accept."
I thank the author for the detailed response. The cross-subject study improved the quality of the work. I changed my score to  6: Weak Accept