DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection
Authors: Sheng Yan
Cunhang Fan
Hongyu Zhang
Xiaoke Yang
Jianhua Tao
Zhao Lv
Venue: NeurIPS 2024 poster (2024)
Keywords: auditory attention decoding (AAD)
electroencephalography (EEG)
brain-computer interface (BCI)
Primary area: neuroscience_and_cognitive_science
TL;DR: A dual attention refinement network with spatiotemporal construction for audutory attention detection.

Abstract: At a cocktail party, humans exhibit an impressive ability to direct their attention. The auditory attention detection (AAD) approach seeks to identify the attended speaker by analyzing brain signals, such as EEG signals. 
However, current AAD algorithms overlook the spatial distribution information within EEG signals and lack the ability to capture long-range latent dependencies, limiting the model's ability to decode brain activity.
To address these issues, this paper proposes a dual attention refinement network with spatiotemporal construction for AAD, named DARNet, which consists of the spatiotemporal construction module, dual attention refinement module, and feature fusion \& classifier module. Specifically, the spatiotemporal construction module aims to construct more expressive spatiotemporal feature representations, by capturing the spatial distribution characteristics of EEG signals. The dual attention refinement module aims to extract different levels of temporal patterns in EEG signals and enhance the model's ability to capture long-range latent dependencies. The feature fusion \& classifier module aims to aggregate temporal patterns and dependencies from different levels and obtain the final classification results.
The experimental results indicate that DARNet achieved excellent classification performance, particularly under short decision windows. While maintaining excellent classification performance, DARNet significantly reduces the number of required parameters. Compared to the state-of-the-art models, DARNet reduces the parameter count by 91\%. Code is available at: https://github.com/fchest/DARNet.git.

DARNet: Dual Attention Refinement Network with Spatiotemporal Construction for Auditory Attention Detection
Sheng Yan-* 1*
Cunhang Fan 1xt Hongyu Zhang
Xiaoke Yang!
Jianhua Tao?
Zhao Lv
Anhui Province Key Laboratory of Multimodal Cognitive Computations, School of Computer Science and Technology, Anhui University Department of Automation, Tsinghua University
Abstract
At cocktail party, humans exhibit impressive ability direct their attention. The auditory attention detection (AAD) approach seeks to identify the attended speaker by analyzing brain signals, such EEG signals. However, current AAD algorithms overlook the spatial distribution information within EEG signals and lack the ability to capture long-range latent dependencies, limiting the model's s ability to decode brain activity. To address these issues, this paper proposes dual attention refinement network with spatiotemporal construction for AAD, named DARNet, which consists of the spatiotemporal construction module, dual attention refinement module, and feature fusion & classifier module. Specifically, the spa- tiotemporal construction module aims to construct more expressive spatiotemporal feature representations, by capturing the spatial distribution characteristics of EEG signals. The dual attention refinement module aims extract different levels temporal patterns in EEG signals and enhance the model's ability to capture long-range latent dependencies. The feature fusion & classifier module aims aggregate temporal patterns and dependencies from different levels and obtain the final classification results. The experimental results indicate that DARNet achieved excellent classification performance, particularly under short decision windows. While maintaining excellent classification performance, DARNet signifi- cantly reduces the number of required parameters. Compared to the state-of-the-arr models, DARNet reduces the parameter count by 91 %. Code is available https://gttubbcoo///cs/s//DRRte
Introduction
The auditory attention detection (AAD) aims to study human auditory attention tendencies by analyzing brain signals 2 3. The auditory attention refers to the ability of individuals to isolate or concentrate on specific sounds. which aids them in focusing on a single speaker amidst multi- speaker environment, a scenario commonly referred to as the "cocktail party scenario" [4. However, this ability may diminish or even completely disappear for individuals with impairment. Therefore, finding solutions assist these individuals in overcoming this challenge has become an urgent matter.
Mesgarani and Chang [5] have demonstrated close connection between auditory attention and brain activity, which indicates that researchers can study auditory attention by analyzing brain activity.
"Equal contribution.
Corresponning author. Correspondence to cunhang, fan @ ahu.edu.cn.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Following this concept, many methods such as electrocorticograpty (BCoG) [5], magnetoencephalog- raphy 7 and electreeneehhaloraphh (EEG) 8 have been used to implement auditory attention detection. Among these methods, EEGi-based approaches are widely applied in AAD due to their
high temporal resolution, non-invasive mode, and excellent maneuverability
[ 10
According to the conclusions of Mesgarani and Chang [5]. previous studies have utilized stimulus- reconstruction or speech envelope reconstruction methods, which necessitate clean auditory stimuli as input 12 [3] However, in most real-world scenarios, environments consist of multiple sounds mixture of these sounds, posing challenge in obtaining clean auditory stimuli. Therefore, in recent years, the academic community has increasingly focused solely on utilizing EEG signals as input for AAD research [5] The research method proposed in this paper also exclusively utilizes EEG signals.
simultaneously. Listeners are exposed
Traditional AAD tasks relied linear models process EEG signals 117 18 However, brain activity is inherently nonlinear, posing challenges for linear models in capturing this complexity. Consequently, they necessitate longer decision windows to extract brain activity features [19]. Some previous studies have indicated that decent decoding performance can be achieved by analyzing different spatial distribution features within each frequency band. These studies project the extracted differential entropy (DE) values onto 2D topological maps and decode them with convolutional neural networks IS). However, EEG signals are fundamentally time-series data, these methods overlook the dynamic temporal patterns of EEG signals. Other studies analyze EEG signals only in the time domain. For instance, they use long short-term memory (LSTM) networks capture dependencies within EEG signals and achieve decent decoding performance (2). However, these studies only focus on the temporal information within EEG signals, neglecting the spatial distribution features, which reflect the dynamic patterns of different brain regions when receiving, processing, and responding to auditory stimuli. Meanwhile, numerous noise points and outliers make it difficult to capture long-range latent dependencies.
To address these issues, this paper proposes dual attention refinement network with spatiotemporal construction for AAD, named DARNet, which effectively captures the spatiotemporal features and long-range latent dependencies of EEG signals. Specifically, our model consists of three modules: (1) Spatiotemporal Construction Module. The spatiotemporal construction module employs temporal convolutional layer and a spatial convolutional layer. The temporal convolutional layer effectively captures the temporal dynamic features of EEG signals, and the spatial convolutional layer captures the spatial distribution features among different channels, thereby constructing a robust embedding for the next layer. (2) Dual Attention Refinement Module. The dual-layer self-attention refinement module consists of two layers, each comprising a multi-head self-attention and a refinement layer. This design is intended to capture long-range latent dependencies and deeper sequence patterns in EEG signals. (3) Feature Fusion & Classifier Module. The attention features generated by the dual-layer self-attention refinement module, comprising both shallow and deep levels, fed into the feature fusion module to obtain richer representations, enhancing the model's 's robustness and generalization. The fused features are input into classifier to predict the auditory attention tendencies of the subjects.
To this end, We evaluated the decoding performance of DARNet on three datasets: DTU, KUL, and MM-AAD, The results demonstrate that DARNet outperforms the current state-of-the-an model on all three datasets. The main contributions of this paper are summarized as follows:
We propose novel auditory attention decoding architecture, which consists of a spatiotem- poral construction module, dual attention refinement module, and feature fusion module. This architecture could fully leverage the spatiotemporal features and capture long-range latent dependencies of EEG signals.
The DARNet achieves remarkable decoding accuracy within very short decision windows, surpassing the current otate-of-the-art (SOTA) model by 5.9% on the DTU dataset and 4.9% on the KUL. dataset, all under a O. 0.1-second decision window. Furthermore, compared to the current state-of- th-tth-rrtt model with 0.91 million training parameters, DARNet achieves further parameter reduction, requiring only 0.08 million parameters.
2

---

Conv2d
Conv2d
N
CSP
N
mode
T
T
(a) Spatiotemporal Construction
GAP
FC
Refine
Refine
FC
C
Iilention
attention
labels
GAP
FC
F
FEll IR
T:4
T12
(C) Feature Fusion & Classifier
(b) Dual Attention Resinement
T: Time series
N Channels
Refine MaxPool(EL Id))
Concatenation
+ Element-wise Addition
Positional Emhedding
Figure 1: The framework of the DARNet model for AAD, which mainly consists three modules: (a) spatiotemporal construction module, (b) dual attention refinement module, and (c) feature fusion & classifier module. The model inputs are common spatial patterns (CSP) extracted from EEG signals, and the outputs are two predicted labels related to auditory attention.
2 Methodology
The previous AAD methods overlooked the influence of spatial distribution characteristics on de- coding performance and struggled to capture the long-range dependencies in EEG signals [4] 20). To address these issues, we proposed DARNet, which consists a spatiotemporal construction module, dual attention refinement module, and a feature fusion & classifier module, Figure Our proposed DARNet effectively captures the spatiotemporal features of EEG signals and has the capability capture long-range latent dependencies in EEG signals.
By employing a moving window on the EEG data, we obtain a series of decision windows, each containing small duration of EEG signals. Let R [r1...., Tt...., ... ] E IR. X represents the EEG signals a decision window, where r; E N'1 represents the EEG data at the i-th time point within decision window, contains N channels. Here N represents the number of EEG channels and T denotes the length of the decision window. Before inputting EEG data into the DARNet, we employ common spatial patterns (CSP) algorithm to extract raw features from the EEG data under different brain states 22
CSP(() R
where CSP(-) represents the CSP algorithm, E NxT represents the processed EEG signal. c_in is the components of the CSP algorithm and T denotes the length of the decision window.
2.1 Spatiotemporal Construction Module
EEG signals record the brain's neuronal electrical activity, varying over time and reflecting activity patterns and connectivity across brain regions (23] By constructing spatiotemporal features from EEG signals, it's possible analyze the brain's response patterns to auditory stimuli. However, previous studies only focused on local temporal patterns in EEG data, overlooking the spatial distribution features. Therefore, addition the conventional use temporal filters, we introduced spatial filter 24] to construct the spatiotemporal features of EEG signals.

---

Firstly, we use temporal convolution layers to capture the instantaneous changes in EEG signals, thereby constructing the temporal patterns E, of the EEG signals. It can be formulated as follows:
= GELU E.=GEIUU (Temporl(oon2d(())) E R 3gddmoaxxe_ixxt
where TonporalConv2d(() performs 2-D convolutional filters (kernel size=1 x 8) time dimension with (.) activation function. dmoddt represents the embedding dimension.
Subsequently, we employ spatial convolutional layer with receptive field spanning all channels to capture the spatial distribution features S of EEG signals across different channels, thereby aiding the model in comprehensively understanding the brain's activity patterns in response to various auditory stimuli.
S (SLUU((paaaaCooo2220 E,)) modertX
where SpatialCono2d()) performs an 2-D convolutional filters with c_in x kernel size on spatial dimension. By doing so, we not only capture the temporal patterns in EEG signals but also integrate the spatial distribution characteristies of EEG signals, thereby constructing input embedding containing comprehensive spatiotemporal information for the next layer. This integrated input better reflects the complex features within EEG signals, providing richer information for subsequent analysis and processing.
2.2 Dual Attention Refinement Module
Previous -scousccc acoustic research has demonstrated that human attention is dynamic and time- related activity 25 26). The brain activity from the preceding moment can profoundly influence subsequent brain activity [27]]. However, previous AAD algorithms were hindered by model depth and the noise and outliers in EEG data, making them ineffective at capturing the long-range latent dependencies in EEG signals.
To address this issue, proposed dual self-attention mechanism, which has greater potential for capturing long-range latent dependencies and deeper sequence patterns in EEG signals. Inspired by Zhou et al. (28). Yu al. [29), We introduced a self-attention refinement operation, which refines the dominant temporal features through convolution and pooling operations, compressing the original EEG series of length T to half its length. This self-attention refinement operation reduces the impact of noise and outliers, while also decreasing the model's parameter count. This enhances the model's s generalization and robustness. The single-layer attention refinement module can be formulated as follows:
F Max Pool (Conel di Multi Head Attention((z)))
where Multi H ead Attendion(()) denotes multi-head self-attention algorithm 300, Conveld()) repre- sents an 1-D) convolutional filters (kernel width=3) on time dimension. The (.TU(.)) is the activation function proposed by Clevert et al. (31, Max: (ool(o)..) denotes a max-pooling layer with stride 2.
Before applying the temporal attention feature extraction module, we add the absolute positional embedding (30] to the input embedding S as follows:
8; + pi
(5)
where s; represents the embedding vector of the inh time step, Ps R.dmadel represents inh time step position.
To obtain different levels of temporal features from EEG signals and to capture the long-range latent dependencies, we stacked two of the above attention refinement extraction modules.
Fi MaxPool LE (Cono ( M ulti eaditntttttnnttioon (S))) R ddmonii??
(6)
F2 MaxPool ELU (Conold(AM) (M
E R x
(7)
where F1 and F2 contain different levels of dependencies and temporal patterns in the EEG signals, respectively.
2.3 Feature Fusion & Classifier Module
Features at different levels can reflect various characteristics of the pattern. By optimizing and combining these different features, it preserves effecctve discriminative information from features

---

at different levels while also to some extent eliminating redundant information 32 designed feature fusion module as follows:
Therefore, we
First, we project the features Fi and F2 into the same dimension.
F Linear Liinar (Aduptive L
IR'
(8)
F2 Linear Ara E IR.
(9)
where Adaptive denotes an adaptive average pooling layer, Linear denotes a linear layer.
Second, We concatenate feature Fí and F22 obtain the fused feature vector F.
F
F2
(10)
Finally, we employ fully connected layer to obtain the final auditory attention prediction.
p p= w +
(11)
where and are the weight and the bias of the fully connected layer, p denotes the predicted direction label. In the training stage, we employ the cross entropy loss function to supervise the network training.
Experiments
3.1 Dataset
In this section, we conduct experiments on three publicly available datasets, namely KUL 33 34],
DTU (35 (36] and MM-AAD (20], which are commonly used in auditory attention detection to evaluate the effectiveness of our DARNet. KUL and DTU only contain EEG data the auditory stimulus scenes. MM-AAD contains EEG data of the audio-only scene and the audio-visual scene. We summarize the details the above datasets in Table
1) KUL Dataset: In this dataset, 64-channel EEG data were collected from 16 normal-hearing subjects using BioSemi ActiveTwoo device at a sampling rate of 8, 191 Hz soundproof room. Each subject was instructed to focus auditory stimuli were filtered at 4kHz and set at 60dB through in-ear headphones, which contain four Dutch short stories, narrated by three male Flemish speakers. Two listening conditions were employed: dichotic (dry) presentation with one speaker per ear, and head- related transfer function (HRTF) filtered presentation, simulating speech from 90" left or right. Each subject listened to 8 trials, which lasted 6 minutes.
one of simultaneous speakers. The
2) DTU Dataset: In this dataset, 64-channel EEG data were collected from 18 normal-hearing subjects using BioSemi ActiveTwo device sampling rate of 512 Hz. Each subject was instructed to focus on one of two simultancous speakers, who presented at 60% relative to the subject. The auditory stimuli were 60dB through ER-2 earphones, which contain Danish audiobooks, narrated by three male speakers and three female speakers. Each subject listened to 60 trials, which lasted 50 seconds.
3) MM-AAD Dataset: In this dataset, 32- 22-channel EEG data were collected from 50 normal- hearing subjects (34 males and 16 females) at a sampling rate of 4kH7, following the 10/20 international system. Each subject was exposed both audio-only and audio-visual stimuli. They were instructed to focus on one of two simultaneous speakers, who presented at left right spatial direction relative to the subject. The auditory stimuli comprised 40 classic Chinese stories narrated by both male and female speakers. Each subject listened to 20 trials, which lasted 165 seconds.
3.2 Data Processing
To fairly compare the performance of the proposed DARNet model, specific preprocessing steps applied to each dataset (KUL., DTU, and MM-AAD). For the KUL dataset, the EEG data were firstly re-referenced to the average response of mastoid electrodes, then bandpass filtered between 0.1 Hz
5

---

Table Details of three datasets used in the experiments.
Duration per subject (minutes)
Total duration
Dataset
Subjects
Scene
Language
(hours)
KUL
16
audio-only
Dutch
48
12.8
DTU
18
audio-only
Danish
50
15.0
50
audio-only
Chinese
55
45.8
MM-AAD
50
audio-visual
Chinese
55
45.8
and 50 Hz, and finally down-sampled to 128 Hz. For the DTU dataset, the EEG data were filtered to remove 50 Hz linear noise and harmonics. Eye artifacts were eliminated through joint decorrelation and the EEG data were re-referenced to the average response of mastoid electrodes. Finally, the EEG data were down-sampled to 64 Hz. For the MM-AAD dataset, the EEG data were firstly bandpass filtered between 0.1 Hz and 50 Hz, then removed 50 Hz noise through a notch filter. Additionally, eye artifacts eliminated, and further noise removal achieved, using independent component analysis (ICA). Finally, the EEG data were down-sampled to 128 Hz.
We evaluated our proposed DARNet model and compared it with other state-of-the-ar models under three decision window lengths: 0. Is, 1s, and 2s. Specifically, we selected three publicly available models as our baseline for comparison: SSF-CNN (37], MBSSFCC [15], and DBPNet (20
3.3 Implement Details
In previous AAD research, the accuracy of auditory attention prediction classification has been used as a benchmark for model performance. We followed this convention and evaluated our proposed DARNet the KUL, DTU, and MM-AAD datasets. As follows, take the KUL dataset with a I-second decision window as an example to illustrate implementation details, including training settings and network configuration.
Firstly, we set the proportions of the training, validation, and test sets to 8:1:1. For each subject of the KUL dataset, we get 4,600 decision windows for training, 576 decision windows for validation, and 576 decision windows for testing. Meanwhile, we set the batch size to 32, the maximum number of epochs to 100, and employ early stopping strategy. Training will stop if the loss function value on the validation set does not decrease for 10 consecutive epochs. Additionally, we utilize the Adam optimizer with a learning rate of 5c-4 and weight decay of 3e-4 to train the model. The DARNet is performed using PyTorch.
Before inputting EEG data into the DARNet, we employ the CSP algorithm to extract raw features E E TR. 128x61 from the EEG data. The data is transposed and expanded, represented as E' R1x64x 128 Then, through the spatiotemporal construction module (Cin is set to 16), we can get embedding data S E 16611x12 After dimensionality reduction, transposition, and the addition of absolute positional embedding. the data is fed into the dual attention refinement module, resulting in two distinct level features, Fi E 116664 and P22 16x332 The F, and F2 are sent to the feature fusion module, where they undergo global average pooling and dimensionality reduction via fully connected (FC) layer (input: 16, output: 4) before being concatenated to obtain the fused feature, F E R Finally, F is passed through another FC layer (input: 8, output:2) to obtain the final auditory attention prediction p E R?.
4 Result
4.1
Performance of DARNet
To evaluate the performance of DARNet, we conducted comprehensive experiments under decision windows of 0.1 --scccnd, 1 1-second, and 2-second, respectively, shown in Figure Additionally, We compared our DARNet with other advanced models, as shown in Table The results are replicated from the corresponding papers.
DARNet has outperformed the current state-of-the-arr models on the KUI,, DTU, and MM-AAD datascts, achieving further enhancements in performance. On the KUL dataset, the DARNet achieves

---

Table Auditory attention detection accuracy comparison on DTU, KUL and MM-AAD dataset. The results annotated by are taken from [20] Our experimental setup is consistent with theirs to ensure fairness comparison. Hence, we directly cited their results.
Decision Window
Dataset
Scene
Model
0.1-second
1-second
2-second
SSF-CNN'
37
76.3 + 8.47
84.4 8.67
87.8 + 7.87
MBSSFCC'
13
79.0 + 7.34
86.5 7.16
89.5 + 6.74
BSAnet
[8
93.7 + 4.02
95.2 + 3.08
KUL
audio-only
DenseNet-3D
94.3
95.9 + 4.3
DBPNet" 20
87.1 + 6.55
95.0 4.16
96.5 3.50
DARNet(ours)
91.6 + 4.83
96.2 3.04
97.2 2.50
SSF-CNN' 37
62.5 + 3.40
69.8 5.12
73.3 + 6.21
MBSSFCC 15
66.9 + 5.00
35.6::6.55
78.7 + 6.75
BSAnet 38
8.71
85.6 6.47
DTU
audio-only
EEG-Graph Net 40
72.5 7.41
78.7
+ 1
DBPNet
20
75.1 +.1174877 4.87
5.99
86.5 + 5.34
DARNet(ours)
79.5 5.84
87.8 6.02
89.9 + 5.03
SSF-CNN' 37
56.5 + 5.71
57.0
57.9 + 7.47
MBSSFCC
15
7
76.5
77.0 9.92
audio-only
DBPNet
20
91.4 4.63
92.5 + 4.59
DARNet(ours)
94.9 4.79
96.0 4.00
96.5 + 3.59
MM-AAD
SSF-CNN'
37
+ 56.6
57.2 5.59
58.2 + 6.39
audio-visual
MBSSFCC'
15
77.2
78.1 + 10.1
78.4 9.57
DBPNet 20
92.1 1 4.47
92.8 + 5.94
93.4 + 4.86
DARNet(ours)
95.8 + 4.04
96.4 3.72
96.8 + 3.44
average accuracies of 91.6% (SD: 4.83%), 96.2% (SD: 3.04%), 97.2% (SD: 2.50%) under 0. O.1-second, 1-second and 2-second decision window, respectively. On the DTU dataset, the DARNet achieves average accuracies of 79.5% (SD: 5. 5.84%) for 0.1-sccond decision window, 87.8% (SD: 6.02%) for I-second decision window, 89, 9% (SD: 5.03%) for 2-second decision window, respectively. On the MM-AAD dataset, the DARNet also demonstrates outstanding decoding accuracies of 94.9% (SD: 4.79% for 0. -se-ooc,,,, 96.0% (SD: 4.00%) for 1-second, 96.5% (SD: 3.59%) for 2-second in the audio-only scene, and 95.8% (SD: 4.04%) for 0. I-second, 96.4% (SD: 3.72%) for 1-second, 96.8% (SD: 3.44%) ) for 2-second in the audio-visual scene.
Overall, DARNet's decoding accuracy increases with larger decision windows, consistent with prior research 15 14. This is because longer decision windows provide more information for the model to make judgments while also mitigating the impact of individual outliers the predictions. However, DARNet still maintains excellent performance under the 0. 0.1-second decision window. Additionally, observe that in the MM-AAD dataset, performance is better in the audio-visual condition compared to the audio-only condition in two different scenarios. We attribute this improvement to the visual cues aiding humans in localizing sound sources.
4.2 Ablation Study
We conducted comprehensive ablation experiments by removing the spatial feature extraction module, the temporal feature extraction module, and the feature fusion module. Additionally, we supplemented study with ablation experiments using single-layer attention refinement module the KUL and DTU dataset, referred to as previous settings. Additionally, ensured that all model network parameters were fully optimized to guarantee that the model's performance reached its best under each condition, whether a module was removed added. The results of the ablation experiments are shown in Table
All experimental conditions remained the same as in
Experimental results show that on the DTU dataset, after removing the spatial feature extraction module from DARNet, the average accuracy decreased by 10.1% % under 0. decision window,
7

---

Table Ablation Study on KUL, DTU, and MM-AAD dataset.
Decision Window
Dataset
Scene
Model
0.1
I-second
2-second
w/o spatial feature
81.1 + 6.31
87.5 + 7.24
89.0 + 4.93
w/o temporal feature w/o feature fusion
81.2 + 6.94
86.7 + 7.30
89.1 5.37
KUL.
audio-only
90.5 + 5.09
95.2 3.58
96.1 + 3.46
single-DARNet DARNet(ours)
91.1 5.18
95.5 3.28
96.2 = 2.98
91.6 4.83
96.2 3.04
97.2 + 2.50
w/o spatial feature
76.3
79.1 + 6.79
w/o temporal feature
70.9
79.2 + 6.84
DTU
audio-only
w/o feature fusion
77.5
86.3 96.3 6.16
88.6 + 5.71
single-DARNet DARNet(ours)
7
86.3 5.83
88.0 4.03
79.5 5.84
87.8 6.02
89.9 5.03
w/o spatial feature
90.0 5.76
91.0
92.5 4.76
w/o temporal feature
87.8 = 5.66
8
91.1 + 5,31
audio-only
w/o feature fusion
9
94.8 4.05
95.7 = 4.21
DA DARNet(ours)
94.9 4.79
96.0 4.00
96.5 - 3.59
MM-AAD
w/o spatial feature
91.2 5.56
93.1 + 5.36
w/o temporal feature
89.4 6.96
90.5 6.04
92.1 5.72
audio-visual
w/o feature fusion
95.3 + 4.57
95.7 + 3.88
96.1 + 3.96
DARNet(ours)
95.8 4.04
96.4 = 3.72
96.8 + 3.44
13.1% under Is decision window, and 12.0% under 2s decision window. After removing the temporal feature extraction module, the average accuracy for the 0.1s, 1s, and 2s decision windows decreased by 10.9%, 14.2%, and 11.9%, respectively. After removing the feature fusion module, the average accuracy decreased by 2.5% under a 0. decision window, 1.6% under a 1s decision window, and 1.4% under a 2s decision window. On the KUL dataset and the MM-AAD dataset, removing the aforementioned modules also resulted in similar trends of decreased average accuracy.
1,00
6.90
Acunasy
0.50
8.75
DIU
GIL
0.-
MMN AAD audko
temporal Feature
wrl feature fusion Dentter:
ovdloviual
temponal feature
0.1
0.15
15
Timefs)
Figure 2: AAD accuracy(%) of DARNet across all subjects on three datasets.
Figure 3: AAD accuracy(%% of the ablation study across all subjects on the DTU dataset.
4.3 Experimental Correction
To ensure faimess in comparisons, aligned our previous experimental setup and data processing methods with those of DBPNet. However, during the code review for the final version of this paper, found that DBPNet applies CSP the data prior to dataset splitting, which lead data leakage. Consequently, we corrected the processing steps and conducted additional experiments, as shown in Table The data preprocessing steps for the supplementary experiments are follows:
1. Split each trial of the dataset into the first 90% for training and the last 10% for testing.

---

2. Fit the CSP transformation matrix using the training data, project the data to extract features, and apply the matrix the testing data.
3. Apply sliding window with 50% overlap to the processed training data, then randomly split into a training (90%) and a validation (10%) set. Similarly, apply a sliding window to the processed testing data as the final test set.
Following these steps, we rigorously avoided risk of data leakage. Experimental results show that both corrected DBPNet and DARNet? exhibit performance decline compared to the original DBPNet' and DARNet'. However, DARNet still achieves state-of-ther performance under most test conditions, shown in Table 4 For instance, the KUL dataset with 0.1-scd decision window, DARNet maintains a decoding accuracy of 89.2%.
Table 4: Experimental Correction on KUL, DTU, and MM-AAD dataset. The experimental setup for the results marked with is consistent with DBPNet 20) where CSP applied prior dataset splitting. In contrast, for the results marked with CSP is applied after dataset splitting.
Decision Window
Dataset
Scene
Model
0.1-second
1-second
2-second
DBPNet
87.1 + 6.55
95.0 + 4.16
96.5 + 3.50
DBPNet
85.3 + 6.22
94.4 + 4.62
95.3 + 4.63
KUL
audio-only
DARNet
91.6 + 4.83
96.2 3.04
97.2 2.50
DARNet
89.2 + 5.50
94.8 + 4.53
95.5 + 4.89
DBPNet!
75.1 4.87
8 5.95
86.5
DBPNet?
74.0
79.8 = 6.91
80.2
DTU
audio-only
DARNet!
79.5 + 5.84
87.8 + 6.02
DARNet2
74.6 6.09
80.1 6.85
81.2
DBPNer
91.4 + 4.63
92.0 5.42
92.5 4.59
DBPNet2
90.0 5.51
4.66
91.6 4 4.82
audio-only
DARNet
4.79
96.0
DARNe12
91.5 5.27
92.2 4.54
4:22
MM-AAD
DBPNet
92.1 4.47
92.8 2 5.94
93.4 4.86
DBPNe12
92.0
93.9
92.7 + 6.04
audio-visual
DARNeL
95.8 4.04
96.4 3.72
96.8 + 3.44
DARNet
92.7 5.34
93.4 5.23
94.2 + 4.84
5
Discussion
5.1 Comparative Analysis
To further evaluate the performance of our proposed DARNet, compared it with other advanced AAD models, as shown in Table The results indicate that our DARNet has achieved a significant improvement over the current -fate-of-hhe-at results.
For example, on the DTU dataset, our DARNet has shown relative improvements 27.2%, 18.8%, 9.7%, and 5.9% for 1-eeood decision window, compared to the SSF-CNN, MBSSFCC, BEG-Graph Net and DBPNet models, respectively. Compared to the SSF-CNN, MBSSFCC, BSAnet, BEG-Graph Net, and DBPNet models, the relative improvements achieve 25.8%. 16.1%. 5.7%. 1 11.6%, 4.6% for I-second decision window, and 22.6%. 14.2%. 5.0%, 13.2%, 3.9% for 2-second. On both the KUL and MM-AAD datasets, DARNet has achieved similar improvements compared to the otate-of-the-art models. The particularly outstanding results achieved across all three datasets under the 0.1 -ssonnd decision window indicate the potential of DARNet for real-time decoding of auditory attention.
Overall, the excellent performance of DARNet across different datasets and decision windows demonstrates its robustness and versatility in various contexts. This further validates the potential of DARNet as an effective EEG analysis model and provides strong support for its widespread application in real-world scenarios.

---

5.2
Ablation Analysis
As shown in Table and Figure E] compared with removing the spatial feature extraction step. removing the temporal feature extraction step, removing the feature fusion module, and using single-layer attention refinement module, we believe DARNet performs excellently for the following reasons:
1. Integrating multiple sources of information: DARNet integrates temporal and spatial distribution features from EEG signals, constructing richer and more robust spatiotemporal features. This enables the model to comprehensively understand the spatiotemporal information within EEG signals, thereby enhancing the understanding of brain activity. In contrast, removing any single feature may lead to information loss or the inability to capture the transient changes in EEG signals, thereby impacting the model's performance.
2. Comprehensive capture of temporal dependencies: The dual attention refinement module and feature fusion module of DARNet comprehensively capture temporal patterns and dependencies different levels, enabling the model to better understand the temporal dynamics within EEG signals. This holistic consideration of features at different time scales crucial for the analysis of EEG data.
Robust feature representation: Despite observing that removing the feature fusion module did not lead to significant decrease in accuracy across the three datasets, the performance variability of DARNet increases substantially. We believe that the feature fusion module integrates temporal patterns and dependencies at different levels, enabling the model to better understand and utilize the complex relationships within the data, thus enhancing the robustness and generalization the model.
Table 5: The training parameter counts comparison. "M" denotes million.
Model
Trainable Parameters Counts
SSF-CNN 37
4.21M
MBSSFCC 15
83.91M
DBPNet (20)
0.91M
DARNet (ours)
0.08M
5.3 Computational Cost
We compare the training parameter counts of our DARNet. SSF-CNN (37). MBSSFCC 15]. and DBPNet (20). with the results shown in Table The parameter count of DARNet is 51.6 times lower than that of SSF-CNN, 1331.5 times lower than that of MBSSFCC, and 10.4 times lower than that of DBPNet. Compared to other models, DARNet demonstrates superior parameter efficiency. Despite having fewer parameters, DARNet maintains good performance, indicating its ability to be applied in resource-constrainge environments for AAD analysis, thus demonstrating practical utility.
Conclusion
In this paper, we propose the DARNet, a novel dual attention refinement network with spatiotem- poral construction for auditory attention detection. By employing spatial convolution operations across all channels, DARNet effectively leverages the spatial information embedded in EEG signals, thereby constructing a more robust spatiotemporal feature. Additionally, DARNet integrates dual attention refinement and feature fusion techniques to comprehensively capture temporal patterns and dependencies at various levels, enhancing the model's ability capture the temporal dynamics within EEG signals. We evaluate the performance of DARNet on three datasets: KUL, DTU, and MM-AAD. DARNet achieves decoding accuracy of 96.2% on the 1-seoond decision window of the KUL dataset and 87.8% on the I-second decision window of the DTU dataset, demonstrating significant improvements compared to current ofate-of-the-art models. The experimental results validate the effectiveness and efficiency of the DARNet architecture, indicating its potential for practical applications. future research, we plan to further explore DARNet's performance on cross-subject tasks verify its generalization and robustness.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the STI 2030-Majoo Projects (No. 20212D00015000,, the National Natural Science Foundation of China (NSFC) (No. (No.62201002, 6247077204), Excellent Youth Foundation of Anhui Scientific Committee (No. 2408085Y034), Distinguished Youth Foundation of Anhui Scientific Committee (No. 2208085105), Special Fund for Key Program of Science and Technology of Anhui Province (No. 202203a07020008) Open Fund of Key Laboratory of Flight Techniques and Flight Safety, CACC (No, FZ2022KF155, Cloud Ginger XR-1.
References
[1] Cong Han, James O' Sullivan. Yi Luo, Jose Herrero, Ashesh D Mchta, and Nima Mesgarani. Speaker-indepeene auditory attention decoding without access to clean speech sources. Sci- ence advances, 5(5) 2019.
[2] Mohammad Jalilpour Monesi, Bernd Accou, Jair Montoya- Martinez, Tom Francart, and Hugo Van Hamme. An Istm based architecture to relate speech stimulus to eeg. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 941-945, IEEE, 2020.
[3] Siqi Cai, Enze Su, Longhan Xie, and Haizhou Li. Eeg-based auditory attention detection via frequency and channel neural attention. IEEE Transactions on Human-Machine Systems, 52(2): 256-266, 2021.
[4] Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the acoustical society of America, 25(5):975-979) 1953.
[5] Nima Mesgarani and Edward F Chang. Selective cortical representation of attended speaker in multi-talker speech perception. Nature, 485(7397):233 236, 2012.
[6] Nai Ding and Jonathan Simon. Neural coding of continuous speech in auditory cortex during monaural and dichotic listening. Journal of neurophysiology, 107(1):78-89, 2012.
[7] Sahar Akram, Jonathan Z Simon, and Behtash Babadi. Dynamic estimation of the auditory temporal response function from meg in competing-speaker environments. IEEE Transactions on Biomedical Engineering, 6498):18961190 2016.
[8] Inyong Choi, Siddharth Rajaram, Lenny A Varghese, and Barbara G Shinn- Cunnnigunin Quantifying attentional modulation of auditory-evoked cortical responses from single-trial electroencephaloggaphy Frontiers in human neuroscience, 7:115, 2013.
[9] James A O'sullivan, Alan Power, Nima Mesgarani. Siddharth Rajaram, John J Foxe, Barbara G ChininCCunn,,hmm, Malcolm Slaney, Shihab A Shamma, and Edmund C Lalor. Attentional selection in cocktail party environment can be decoded from single-trial eeg. Cerebral cortex, 25(7):1697-1706, 2015.
[10] Faramarz Faghihi, Siqi Cai, and Ahmed A Moustafa. A neuroscience-inspeee spiking neural network for eeg- eeg-based auditory spatial attention detection. Neural Networks, 152:5555565. 2022.
[11] Simon Geirnaert, Tom Francart, and Alexander Bertrand. Riemannian geometry-basel decoding of the directional focus of auditory attention using eeg. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1115-1119. IEEE, 2021.
[12] Ali Aroudi, Daniel Marquardt, and Simon Daclo. Eeg-based auditory attention decoding using steerable binaural superdirective beamformer. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 851-855. IEEE, 2018
[13] Neetha Das, Simon Van Eyndhoven, Tom Francart, and Alexander Bertrand. Beg-based attention- driven speech enhancement for noisy speech mixtures using n-fold multi-channel wiener filters. In 2017 25th European Signal Processing Conference (EUSIPCO), pages 1660-1664. IEEE, 2017.
[14] Enze Su, Siqi Cai, Longhan Xie, Haizhou Li, and Tanja Schultz. Stanet: A spatiotemporal attention network for decoding auditory spatial attention from eeg. IEEE Transactions on Biomedical Engineering, 69(7):2233223) 2022.

---

[15] Yifan Jiang, Ning Chen, and Jing Jin. Detecting the locus of auditory attention based on the seat--ppaatttpponoe analysis of eeg. Journal of neural engineering, 19(5):55033,, 2022.
[16] Cunhang Fan, Hongyu Zhang, Wei Huang. Jun Xue, Jianhua Tao, Jiangyan Yi, Zhao Lv, and Xiaopei Wu. Dgsd: Dynamical graph self-distillation for eeg-based auditory spatial attention detection. Neural Networks, 179: 179:106580, 2024.
[17] J Crosse, Giovanni M Di Liberto. Adam Bednar, and Edmund Lalor. The multivariate temporal response function (mtrf) toolbox: a matlab toolbox for relating neural signals to continuous stimuli. Frontiers in human neuroscience, 10:604, 2016.
[18] Daniel DE Wong, Søren A Fuglsang, Jens Hjortkjær. Enea Ceolini, Malcolm Slaney, and Alain De Cheveigne. A comparison of regularization methods in forward and backward models for auditory attention decoding. Frontiers in neuroscience, 12:352049, 2018.
[19) Sina Miran, Sahar Akram, Alireza Sheikhattar, Jonathan Z Simon, Tao Zhang, and Behtash Babadi. Real-time tracking of selective auditory attention from m/eeg: A bayesian filtering approach. Frontiers in neuroscience, 12:262, 2018.
[20] Qinke Ni, Hongyu Zhang, Cunhang Fan. Shengbing Pei, Chang Zhou, and Zhao Lv. DBPNet: Dual-branch parallel network with Samporal-frequeene fusion for auditory attention detection. In IJCAI, 2024.
[21] Herbert Ramoser, Johannes Muller-Gerking and Gert Pfurtscheller. Optimal spatial filtering of single trial eeg during imagined hand movement. IEEE transactions rehabilitation engineering, 8(4):441-44) 2000.
[22] Benjamin Blankertz, Ryota Tomioka, Steven Lemm, Motoaki Kawanabe, and Klaus-Robert Muller. Optimizing spatial filters for robust eeg single-trial analysis. IEEE Signal processing magazine, 25(1):41-56, 2007.
[23] Mahnaz Arvaneh, Cuntai Guan, Kai Keng Ang, and Chai Quek. Optimizing the channel selection and classification accuracy in eeg-based bei. IEEE Transactions on Biomedical Engineering.
2011.
[24] Navid Mohammadi Foumani, Chang Wei Tan, Geoffrey I Webb, and Mahsa Salehi. Improving position encoding of transformers for multivariate time series classification. Data Mining and Knowledge Discovery, 38(1):22-48, 2024.
[25] Mari Riess Jones and Marilyn Boltz. Dynamic attending and responses to time. Psychological review, 96(3):459, 1989.
[26] Mari Riess Jones, Heather Moynihan, Noah MacKenzie, and Jennifer Puente. Temporal aspects of stimulus-driven attending in dynamic arrays. Psychological science, 13(4):313 1191133-1,,22 2002.
[27] Klaus Linkenkaer- Hannen,, Vadim V Nikouline, Matias Palva, and Risto J Ilmoniemi. Long- range temporal correlations and scaling behavior in human brain oscillations. Journal of Neuroscience, 1777 2001.
[28] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115. 2021.
[29] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 472-480, 2017.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neurul information processing systems, 30, 2017.
[31] Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter, Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
[32] Quan-Sen Sun, Sheng- Gen Zeng, Yan Liu, Pheng-Ann Heng. and De-Shen Xia. A new method of feature fusion and its application in image recognition. Pattern Recognition, 38(12): 2437-2448, 2005.
[33] Neetha Das, Wouter Biesmans, Alexander Bertrand, and Tom Francart. The effect of head- related filtering and ear-specific decoding bias on auditory attention detection. Journal of neural engineering, 13(5):056014, 2016.
12

---

[34] Neetha Das. Tom Francart, and Alexander Bertrand. Auditory attention detection dataset kuleuven. Zenodo, 2019.
[35] Søren Asp Fuglsang, Torsten Dau, and Jens Hjortkjær. Noise-robust cortical tracking of attended speech in real-world acoustic scenes. Neurolmage, 156:435-444, 2017.
[36] Søren A Fuglsang. DD Wong, and Jens Hjortkjær. Eeg and audio dataset for auditory attention decoding. Zenodo, 2018.
[37] Siqi Cai, Pengcheng Sun, Tanja Schultz, and Haizhou Li. Low-latency auditory spatial attention detection based on spectro-spatial features from eeg. 2021 43rd Annual International Conference of the IEFE Engineering in Medicine & Biology Society (EMBC), pages 5812-5815. IEFE, 2021.
[38] Siqi Cai, Peiwen Li, and Haizhou Li. bio-inspired spiking attentional neural network for attentional selection in the listening brain. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[39] Xiran Xu, Bo Wang, Yujie Yan, Xihong Wu, and Jing Chen. A densenet-basee method for decoding auditory spatial attention with eeg. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1946-1950. IEEE, 2024.
[40] Siqi Cai, Tanja Schultz, and Haizhou Li. Brain topology modeling with ceg-graphs for auditory spatial attention detection. IEEE transactions bio-medical engineering, 71(1):171-12,, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction of the paper clearly state the main contributions and scope the research. The claims made are aligned with the theoretical and experimental results presented in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We outlined the limitations in the conclusion. Current AAD research primarily employs two experimental strategies: subject-dependent and bubject-independen..S. Subject- dependent refers to the training and evaluating procedures containing only samples from single subject, while subject-independent contains samples from subjects the dataset. Our proposed model has been validated under the subject-dependent condition and has demonstrated exceptional results. However, further exploration and resolution of the issue of inter-subject variability necessary to enable our model to be more widely applicable to real-world brain-computer interface applications.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
14

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All the theorems, formulas, and proofs in the paper have been properly numbered and cross-referenced, fulfilling the guidelines provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We have detailed our model and experimental setup thoroughly in the Method- ology and Experiments sections, providing all necessary information to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
15

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: The paper includes code as an attachment, facilitating the reproduction of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ctipss.ccpuubic/// gwwc////u//////eeeoeoooc. CodeeSumiia ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed specifications of experimental settings in the "Methodology" and "Experiments" sections of the paper. This includes descriptions of the data splits. hyperparameters, selection criteria, and the type of optimizer used. The comprehensive documentation these parameters ensures that our results can be understood and replicated by other researchers.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
16

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experimental results do not include confidence intervals or statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our model contains only 0.78 million training parameters, making lightweight and capable of running on most machines. Therefore, we did not provide detailed specifications of the compute resources required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Yes, our research adheres to all ethical guidelines required by NeurlPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
17

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research does not directly produce societal impacts as it focuses on technical advancements specific field without direct societal applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeen of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

Justification: All datasets used in our paper are publicly available datasets, and we have cited the respective literature for each dataset. Any researcher can download these datasets from the provided sources.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwpesssipppeseeeeeee thcode. coodsattett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper includes the submission of the model's source code.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
20


Reviews:
[{"review_id": "er5H3JtqzK", "reviewer": "Reviewer_eWL2", "summary": "The manuscript is aim to capture the and spatial distribution information and long-range dependencies in EEG signals. Two modules are designed to solve the upper two challenges, spatiotemporal construction module and dual attention refinement module. The experiment  have shown the superiority of the proposed method. And the analysis have shown the advantage of each module.", "questions": "1)Many researchers have done this work, to capture the long-range dependencies and spatial distribution information within EEG signals. Is there any more innovation of your proposed method? (From framework design and experimental performance.)\n2)What sliding window size is the best performance with? Have you ever try other size longer than 2 s?\n3)Please prove that the long-range dependencies are captured within EEG signals.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)Good writing and organization.\n2)Method is good understanding and can be replicated.\n3)The experiment and analysis is sufficient.", "weaknesses": "1)The manuscript is lack of novelty.\n2)It is less clarity in session Abstract on which dataset the accuracy is improved using different sliding window size. \n3)There is no analysis that the level of computation cost and time consuming of the mentioned methods in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal:\n\nThank you so much for your thoughtful comments and the time to provide constructive feedback!\n\n### Weakness:\n\n1. **Innovation summary:**\n\n   We apologize for any confusion that may have led to the perception of a lack of novelty in our manuscript. We have clarified and summarized our key innovations as follows:\n\n   Firstly,our method highlights EEG channel interactions. Unlike some spatial feature explorations through the frequency domain that exist in the AAD field, our work effectively extracts spatial information directly from the temporal domain and uses GELU to capture nonlinear relationships. We compared temporal-only, spatial-only, and spatiotemporal feature extraction, offering significant insights for summarizing, inspiring, and guiding future works in AAD.\n\n   Secondly, unlike methods that stack multiple self-attention layers to capture long-range dependencies, we employ a refinement operation that progressively compresses the EEG signal sequence. This approach models global dependencies more effectively and reduces the model's parameter count. By halving the sequence length at each step, our method reduces the complexity of self-attention to one-fourth of the original.\n\n   Thirdly, our feature fusion module effectively balances local EEG features and long-range dependencies, enhancing decoding accuracy.\n\n   Finally, our model outperforms current SOTA models across multiple datasets while maintaining a lower parameter count.\n\n2. **Dataset clarity in Abstract.**\n\n   Thank you for your thorough review. We realize we overlooked specifying the dataset annotations and will update this in the final version of the paper.\n\n3. **The computation cost and time consuming of the mentioned methods .**\n\n   Table 1 includes data on computation cost and time consumption, conducted on the KUL dataset for 1s. The results demonstrate that our model has fewer parameters, as well as lower MACs and test time compared to current SOTA models.\n\n   Table1. DBPNet is currently the SOTA model.\n\n   | Model      | Param(M) | MACs(M)   | Test time(ms) |\n   | :--------- | :------- | :-------- | :------------ |\n   | MBSSFCC    | 83.91    | 89.15     | 11.20         |\n   | DBPNet     | 0.91     | 96.55     | 11.98         |\n   | **DARNet** | **0.78** | **16.36** | **10.03**     |\n\n###  Questions:\n\n1. **Is there any more innovation of your proposed method?**\n\n   See response to weakness1.\n\n2. **What sliding window size is the best performance with? Have you ever try other size longer than 2s?**\n\n   Previous research shows that longer sliding windows capture more information and typically improve performance. However, in AAD tasks, longer windows increase latency, which is less suitable for human attention switching. Therefore, current research focuses on improving decoding accuracy with short decision windows, specifically 0.1s, 1s, and 2s. Our results for a 5-s window, presented in Table 2 & 3, confirm that our model continues to outperform current SOTA models.\n\n   Table2: AAD accuracy(%) comparison on DTU dataset.\n\n   | Model      | 0.1s             | 1s               | 2s              | 5s              |\n   | :--------- | :--------------- | :--------------- | :-------------- | :-------------- |\n   | MBSSFCC    | $66.9 \\pm 5.00$  | $75.6 \\pm 6.55$  | $78.7 \\pm 6.75$ | $80.2\\pm 8.64$  |\n   | DBPNet     | $75.1  \\pm 4.87$ | $83.9  \\pm 5.95$ | $86.5 \\pm 5.34$ | $90.1 \\pm 5.82$ |\n   | **DARNet** | $79.5\\pm 5.84$   | $87.8\\pm 6.02$   | $89.9 \\pm 5.03$ | $92.4\\pm 4.71$  |\n\n   Table3: AAD accuracy(%) comparison on KUL dataset.\n\n   | Model      | 0.1s            | 1s              | 2s              | 5s              |\n   | :--------- | :-------------- | :-------------- | :-------------- | :-------------- |\n   | MBSSF      | $79.0 \\pm 7.34$ | $86.5 \\pm 7.16$ | $89.5 \\pm 6.74$ | $92.8 \\pm 5.32$ |\n   | DBPNet     | $87.1 \\pm 6.55$ | $95.0 \\pm 4.16$ | $96.5 \\pm 3.50$ | $97.0 \\pm 4.05$ |\n   | **DARNet** | $91.6\\pm 4.83$  | $96.2\\pm 3.04$  | $97.2 \\pm 2.50$ | $98.0\\pm 3.17$  |\n\n3. **Please prove that the long-range dependencies are captured within EEG signals.**\n\n   We supplemented our study with ablation experiments using a single-layer attention refinement module, referred to as single-DARNet, while the original model is denoted as DARNet. The results, shown in Tables 4 & 5, reveal the performance gap between single-DARNet and DARNet widens with increasing decision window lengths.\n\n   Our analysis indicates that with a short decision window of 0.1s, the EEG sequence contains few data points, making each point significantly impactful on decoding accuracy. At this short window length, local EEG features are crucial, so the difference in performance between single-DARNet and DARNet is minimal. However, as the decision window lengthens to 1s and 2s, the gap between single-DARNet and DARNet increases. This suggests that longer decision windows better capture the long-range dependencies within EEG signals, enhancing decoding accuracy. Our experiments with even longer sliding window sizes, such as 5s, confirm that our model effectively captures these long-range dependencies.\n\n   Table4: Ablation Study on DTU dataset.\n\n   | Model         | 0.1s            | 1s              | 2s              | 5s              |\n   | :------------ | :-------------- | :-------------- | :-------------- | :-------------- |\n   | single-DARNet | $79.1 \\pm 5.66$ | $86.3 \\pm 5.83$ | $88.0 \\pm 4.03$ | $90.2 \\pm 5.62$ |\n   | **DARNet**    | $79.5\\pm 5.84$  | $87.8 \\pm 6.02$ | $89.9\\pm 5.03$  | $93.1\\pm 4.37$  |\n\n   Table5: Ablation Study on KUL dataset.\n\n   | Model         | 0.1s            | 1s              | 2s              | 5s              |\n   | :------------ | :-------------- | :-------------- | :-------------- | :-------------- |\n   | single-DARNet | $91.1 \\pm 5.18$ | $95.5 \\pm 3.28$ | $96.2 \\pm 2.98$ | $96.7 \\pm 4.03$ |\n   | **DARNet**    | $91.6\\pm 4.83$  | $96.2 \\pm 3.04$ | $97.2\\pm 2.50$  | $98.0\\pm 3.17$  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0A27gTqMH0", "reviewer": "Reviewer_FMXQ", "summary": "This paper proposes a new architecture for auditory attention detection (AAD) that consists of three key components: 1) Convolutional layers applied to the temporal and spatial dimensions of EEG signals in a sequential manner to extract features. 2) Two attention layers to process these features. 3) A feature fusion module to combine the outputs of the two attention layers. The method was tested on three different datasets and achieved state-of-the-art (SOTA) classification results.", "questions": "How did 48 minutes of recording in the KUL dataset yield over 5000 1-second decision windows? If a sliding window with overlap was used, please describe the procedure in detail to ensure that there are no repeated signal segments in the training and test sets.", "limitations": "* Reproducibility of the paper would be significantly enhanced if the author provided the full code.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an effective and innovative adoption of convolution and attention layers for auditory attention detection. The model has been rigorously validated on three different datasets. The inclusion of ablation studies further strengthens the validity of the findings by systematically analyzing the contribution of each component of the model.", "weaknesses": "* Using convolutional layers on the temporal dimension followed by the spatial dimensions of EEG signals is not a novel idea. Classic EEG+DL+BCI papers, such as ConvNet (Schirrmeister, 2017) and EEGNet (Lawhern, 2018), already employ similar concepts. Many EEG+DL+AAD papers also adopt spatial processing modules (e.g., references 3 and 14 in the paper). The author should provide a more in-depth discussion on why their spatial module is more effective than existing methods. Specifically, they should highlight any unique innovations or improvements their approach offers over previous works.\n* It would be interesting to see the comparison between dual-attention layer and one attention layer. \n* It is crucial to explicitly mention in the methods section that this is a subject-dependent study. \n* There is a typo in the notations on row 91.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal:\n\nThank you so much for your thoughtful comments and the time to provide constructive feedback!\n\n### Weakness:\n\n1. **A more in-depth discussion on why their spatial module is more effective than existing methods.**\n\n   Auditory attention decoding requires processing EEG signals under complex acoustic stimuli. Temporal regularity is crucial in selective hearing, making the extraction of spatiotemporal relationships key to decoding. Traditional ConvNet models for motor imagery first perform temporal convolution and then spatial convolution, potentially overlooking complex interactions between spatial and temporal dimensions. Most current EEG+DL+AAD methods use DE features, such as transforming the temporal domain into the frequency domain and projecting it onto a 2D topological map, which can lose important temporal variations. Recent methods like STANet (Su et al.) explore spatial features in the temporal domain but are limited to assigning weights to EEG channels without considering nonlinear relationships between them.\n\n   + **Rich Spatiotemporal Feature Extraction:** We capture spatial dependencies between EEG channels and temporal patterns within each channel. By emphasizing channel interactions and using GELU to capture nonlinear relationships, we achieve comprehensive spatiotemporal representations.\n   + **Multi-level temporal pattern extraction:** We integrate a dual attention mechanism on the basis of effective convolution design, enabling the model to focus on distant time points and capture long-term attention pattern changes.\n\n   Additionally, we compared the performance of temporal-only, spatial-only, and spatiotemporal feature extraction. This comparison offers valuable insights for future research.\n\n2. **The comparison between dual-attention layer and one attention layer.**\n\n   We have supplemented our study with ablation experiments using a single-layer attention refinement module, referred to as single-DARNet, while the original model is denoted as DARNet. The results, shown in Tables 1 and 2, reveal that the performance gap between single-DARNet and DARNet widens with increasing decision window lengths.\n\n   Our analysis indicates that with a short decision window of 0.1 seconds, the EEG sequence contains few data points, making each point significantly impactful on decoding accuracy. At this short window length, local EEG features are crucial, so the difference in performance between single-DARNet and DARNet is minimal. However, as the decision window lengthens to 1 and 2 seconds, the gap between single-DARNet and DARNet increases. This suggests that longer decision windows better capture the long-range dependencies within EEG signals, enhancing decoding accuracy. \n\n   Table1: Ablation Study on DTU dataset\n\n   | Model         | 0.1s            | 1s              | 2s              | 5s              |\n   | :------------ | :-------------- | :-------------- | :-------------- | :-------------- |\n   | single-DARNet | $79.1\\pm 5.66$ | $86.3\\pm 5.83$ | $88.0 \\pm 4.03$ | $90.2 \\pm 5.62$ |\n   | **DARNet**    | $79.5\\pm 5.84$  | $87.8\\pm 6.02$ | $89.9\\pm 5.03$  | $93.1\\pm 4.37$  |\n\n   Table2: Ablation Study on KUL dataset\n\n   | Model         | 0.1s            | 1s              | 2s              | 5s              |\n   | :------------ | :-------------- | :-------------- | :-------------- | :-------------- |\n   | single-DARNet | $91.1\\pm 5.18$ | $95.5\\pm 3.28$ | $96.2\\pm 2.98$ | $96.7\\pm 4.03$ |\n   | **DARNet**    | $91.6\\pm 4.83$  | $96.2 \\pm 3.04$ | $97.2\\pm 2.50$  | $98.0\\pm 3.17$  |\n\n3. **Supplementary cross-subject experiment results:**\n\n   Recently, we conducted additional leave-one-subject-out cross-validation experiments on the publicly available DTU and KUL datasets, using a 1-second decision window that closely matches human attention shifts. The results showed that our model performed exceptionally well under subject-independent conditions, surpassing the performance of current SOTA model. The specific results are shown in Table 3.\n\n   Table3: Cross-subject experiment comparison for 1s. The GCN is currently the SOTA model for cross-subject tasks.\n\n   | Model      | KUL                               | DTU                               |\n   | :--------- | :-------------------------------- | :-------------------------------- |\n   | SSF-CNN    | $54.1 \\pm 6.60$                   | $48.7 \\pm 3.96$                   |\n   | MBSSFCC    | $59.0 \\pm 8.72$                   | $49.3 \\pm 4.86$                   |\n   | DBPNet     | $59.7 \\pm 8.12$                   | $53.7 \\pm 5.98$                   |\n   | GCN        | $64.4 \\pm 6.36$                   | $53.5 \\pm 7.53$                   |\n   | **DARNet** | $\\textbf{74.0} \\pm \\textbf{11.4}$ | $\\textbf{56.0} \\pm \\textbf{5.71}$ |\n\n\n4. **There is a typo on row 91.**\n\n   We are very grateful to reviewer for reviewing the paper so carefully. \n\n   Modified version: By employing a moving window on the EEG data, we obtain a series of decision windows, each containing a small duration of EEG signals. Let $R=[r_1,...,r_i,...,r_N] \\in \\mathbb{R}^{T×N}$ represent the EEG signals of a decision window, where $r_i \\in \\mathbb{R}^{N \\times 1}$ represents the EEG data at the $i$-th time point within a decision window, containing $N$ channels.\n\n### Questions:\n\n1. **48 minutes of recording yield over 5000 1s decision windows.**\n\n   The 48 minutes of recording contains 2880 seconds of EEG data. Consistent with previous studies, we set the repetition rate to 0.5. Thus, we obtain $2880 \\times 2 = 5760$ 1s decision windows.\n\n2. **Prevent data leakage.**\n\n   To prevent data leakage, we designated the first 90% of the EEG data from each trial as the training set and the remaining 10% as the test set. We then applied the sliding window technique separately to both sets.\n\n### Limitations:\n\n1. Reproducibility of the paper would be significantly enhanced if the author provided the full code.\n\n   We will upload the complete code once the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yyzn931JiJ", "reviewer": "Reviewer_vVL1", "summary": "The paper proposes DARNet, a dual attention refinement network with spatiotemporal construction for auditory attention detection (AAD). The network captures spatiotemporal features and long-range latent dependencies from EEG signals, leading to improved classification accuracy and reduced parameter count compared to state-of-the-art models.", "questions": "NA", "limitations": "Limitations are discussed in the checklist.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. DARNet shows significant improvements in classification accuracy across multiple datasets, demonstrating its effectiveness in AAD tasks.\n2. The model reduces the number of parameters by 14% compared to the state-of-the-art, which is beneficial for practical applications requiring efficient computation.\n3. The integration of spatial and temporal convolutions to capture dynamic patterns in EEG signals enhances the model's ability to decode brain activity accurately.\n4. The dual attention refinement module effectively captures long-range dependencies in EEG signals, addressing a common limitation in previous models.", "weaknesses": "1. The experiments are all subject-dependent, which may limit the generalizability of the results. Cross-subject conditions could be more relevant for practical applications, where the model needs to generalize across different individuals.\n2. While the model achieves high performance, its complexity might make it difficult to interpret and understand the underlying mechanisms contributing to its success. More insights into the workings of the dual attention mechanism and its impact on EEG signal processing would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal:\n\nThank you so much for your thoughtful comments and the time to provide constructive feedback!\n\n## Weakness:\n\n1. **Supplementary cross-subject experiment results:**\n\n   Recently, we conducted additional leave-one-subject-out cross-validation experiments on the publicly available DTU and KUL datasets, using a 1-second decision window that closely matches human attention shifts. The results showed that our model performed exceptionally well under subject-independent conditions, surpassing the performance of current SOTA models. The specific results are shown in Table 1:\n\n   Table1: Cross-subject experiment comparison on the KUL and DTU dataset for 1s. The GCN is currently the state-of-the-art (SOTA) model for cross-subject tasks.\n\n   | Model   | KUL                               | DTU                               |\n   | ------- | --------------------------------- | --------------------------------- |\n   | SSF-CNN | $54.1 \\pm 6.60$                   | $48.7 \\pm 3.96$                   |\n   | MBSSFCC | $59.0 \\pm 8.72$                   | $49.3 \\pm 4.86$                   |\n   | DBPNet  | $59.7 \\pm 8.12$                   | $53.7 \\pm 5.98$                   |\n   | GCN[1]  | $64.4 \\pm 6.36$                   | $53.5 \\pm 7.53$                   |\n   | DARNet  | $\\textbf{74.0} \\pm \\textbf{11.4}$ | $\\textbf{56.0} \\pm \\textbf{5.71}$ |\n\n   [1] S. Cai, R. Zhang and H. Li, \"Robust Decoding of the Auditory Attention from EEG Recordings Through Graph Convolutional Networks,\" *ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, Seoul, Korea, Republic of, 2024, pp. 2320-2324, doi: 10.1109/ICASSP48485.2024.10447633.\n\n2. **More insights into the workings of the dual attention mechanism and its impact on EEG signal processing would be beneficial：**\n\n   We supplemented our study with ablation experiments using a single-layer attention refinement module, referred to as single-DARNet, while the original model is denoted as DARNet. The results, shown in Tables 1 and 2, reveal that the performance gap between single-DARNet and DARNet widens with increasing decision window lengths.\n\n   Our analysis suggests that with a short decision window length of 0.1 seconds, the EEG sequence contains few data points, making each point significantly impactful on the decoding accuracy. At this stage, the local features of the EEG signal play a more decisive role in the decoding process. Therefore, the performance difference between single-DARNet and DARNet is not substantial. However, under experimental conditions with medium to long decision window lengths, such as 1 second and 2 seconds, the performance gap between single-DARNet and DARNet gradually increases. This indicates that as the decision window lengthens, the long-range dependencies within the EEG signals increasingly influence the decoding accuracy. This experiment demonstrates that the long-range dependencies are captured within EEG signals. Additionally, under experimental conditions with even longer sliding window sizes, such as 5 seconds, our model continues to perform exceptionally well, further proving that the long-range dependencies are captured within EEG signals.\n\n   Table2: Ablation Study on DTU dataset.\n\n   | Model         | 0.1s                             | 1s                                | 2s                               | 5s                               |\n   | :------------ | :------------------------------- | :-------------------------------- | :------------------------------- | :------------------------------- |\n   | single-DARNet | $79.1 \\pm 5.66$                  | $86.3 \\pm 5.83$                   | $88.0 \\pm 4.03$                  | $90.2 \\pm 5.62$                  |\n   | **DARNet**    | $\\textbf{79.5}\\pm \\textbf{5.84}$ | $\\textbf{87.8} \\pm \\textbf{6.02}$ | $\\textbf{89.9}\\pm \\textbf{5.03}$ | $\\textbf{93.1}\\pm \\textbf{4.37}$ |\n\n   Table3: Ablation Study on KUL dataset\n\n   | Model         | 0.1s            | 1s              | 2s              | 5s              |\n   | :------------ | :-------------- | :-------------- | :-------------- | :-------------- |\n   | single-DARNet | $91.1 \\pm 5.18$ | $95.5 \\pm 3.28$ | $96.2 \\pm 2.98$ | $96.7 \\pm 4.03$ |\n   | **DARNet**    | $91.6\\pm 4.83$  | $96.2 \\pm 3.04$ | $97.2\\pm 2.50$  | $98.0\\pm 3.17$  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCG585Giq8", "reviewer": "Reviewer_XzJY", "summary": "This paper makes a substantial contribution to the field of auditory attention detection (AAD) by presenting a more accurate and efficient model, the dual attention refinement network with spatiotemporal construction (DARNet). The authors effectively address critical limitations in current AAD algorithms, specifically the lack of spatial distribution information and the inability to capture long-range latent dependencies within EEG signals. The methodological innovations and empirical results suggest that DARNet represents a significant advancement in decoding brain activity related to auditory attention tasks. The experiments show significant improvements over state-of-the-art methods.", "questions": "1. Why did you use a dual attention refinement module instead of three layers or more? The authors should provide more analysis\n2. Did you lose the FC layer for the final classification part in Figure 1?", "limitations": "The DARNet has only been validated under the subject-dependent condition, and it may not perform as well under subject-independent conditions.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper uses a simple yet effective method to address the previous auditory attention detection algorithms' insufficient utilization of the spatiotemporal features of EEG signals.\n2. The proposed method is technically sound and the idea seems to be interesting.\n3. The experimental setting is valid and extensive,  and the experimental results indicate significant improvements over some comparison methods on several datasets.", "weaknesses": "Overall, the writing of this paper is relatively clear and the structure is complete. However, there are still some issues, such as grammatical errors and some sentences that are not clearly articulated. 3）There are many writing problems. The authors should further polish this manuscript:\n1）Line 100-102: The expression is not very clear.\n2）Line 103: \"However previous\" -> \"However, previous\".\n3）Line 188: \"or\" -> \"and\".\n4）Line 292: \"increased\" -> \"increases\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal:\n\nThank you so much for your thoughtful comments and the time to provide constructive feedback!\n\n### Weaknesses:\n\n1. **Regarding grammatical errors and unclear sentence expressions in the paper:**\n\n   We are very grateful to the Reviewer for carefully reviewing the paper. We have thoroughly examined and corrected the grammatical errors and unclear sentences to ensure the paper is clearer and more accurate.\n\n2. **For the specific issues you pointed out, we have made the following revisions:**\n\n   - **Lines 100-102:** We have reorganized the sentence to make it clearer. The revised sentence is: \"EEG signals record the brain's neuronal electrical activity, varying over time and reflecting activity patterns and connectivity across brain regions.\"\n   - **Line 103:** We have changed \"However previous\" to \"However, previous.\"\n   - **Line 188:** We have changed \"or\" to \"and.\"\n   - **Line 292:** We have changed \"increased\" to \"increases.\"\n\n### Questions:\n\n1. **Why did you use a dual attention refinement module instead of three layers or more?**\n\n   In the field of auditory attention detection, researchers are more focused on addressing the issue of low decoding accuracy in low-latency scenarios. Specifically, they aim to reflect the model's effectiveness in low-latency conditions by examining decoding accuracy within a 0.1-second decision window. However, EEG sequences within this 0.1-second window typically contain 12 or 6 data points, which is insufficient for our model to use three or more layers of refinement. Additionally, after two refinement operations, the EEG sequence is already compressed to 3 or 1 data points, making further compression meaningless. Therefore, we chose to use only a two-layer attention refinement module.\n\n   Furthermore, we conducted ablation experiments, as shown in Table 1 and Table 2. The results indicate that using a single-layer attention refinement compared to a two-layer attention refinement resulted in only a 0.5% decrease in performance, which aligns with our assumptions.\n\n   Table1: Ablation Study on DTU dataset.\n\n   | Model         | 0.1s                             | 1s                                | 2s                               | 5s                               |\n   | :------------ | :------------------------------- | :-------------------------------- | :------------------------------- | :------------------------------- |\n   | single-DARNet | $79.1 \\pm 5.66$                  | $86.3 \\pm 5.83$                   | $88.0 \\pm 4.03$                  | $90.2 \\pm 5.62$                  |\n   | **DARNet**    | $\\textbf{79.5}\\pm \\textbf{5.84}$ | $\\textbf{87.8} \\pm \\textbf{6.02}$ | $\\textbf{89.9}\\pm \\textbf{5.03}$ | $\\textbf{93.1}\\pm \\textbf{4.37}$ |\n\n   Table2: Ablation Study on KUL dataset\n\n   | Model         | 0.1s            | 1s              | 2s              | 5s              |\n   | :------------ | :-------------- | :-------------- | :-------------- | :-------------- |\n   | single-DARNet | $91.1 \\pm 5.18$ | $95.5 \\pm 3.28$ | $96.2 \\pm 2.98$ | $96.7 \\pm 4.03$ |\n   | **DARNet**    | $91.6\\pm 4.83$  | $96.2 \\pm 3.04$ | $97.2\\pm 2.50$  | $98.0\\pm 3.17$  |\n\n2. Did you lose the FC layer for the final classification part in Figure 1?\n\n   Thank you for your thorough review and feedback. We acknowledge that we missed labeling the fully connected layer (FC layer) in the final classification part of Figure 1. We will correct this in the final version of the paper. We appreciate your attention to detail.\n\n### Limitations:\n\n1. **The DARNet has only been validated under the subject-dependent condition, and it may not perform as well under subject-independent conditions.**\n\n   Recently, we conducted additional leave-one-subject-out cross-validation experiments on the publicly available DTU and KUL datasets, using a 1-second decision window that closely matches human attention shifts. The results showed that our model performed exceptionally well under subject-independent conditions, surpassing the performance of current SOTA models. The specific results are shown in Table 3:\n\n   Table3: Cross-subject experiment comparison on the KUL and DTU dataset for 1s. The GCN is currently the state-of-the-art (SOTA) model for cross-subject tasks.\n\n   | model            | KUL                               | DTU                               |\n   | ---------------- | --------------------------------- | --------------------------------- |\n   | SSF-CNN          | $54.1 \\pm 6.60$                   | $48.7 \\pm 3.96$                   |\n   | MBSSFCC          | $59.0 \\pm 8.72$                   | $49.3 \\pm 4.86$                   |\n   | DBPNet           | $59.7 \\pm 8.12$                   | $53.7 \\pm 5.98$                   |\n   | GCN[1]           | $64.4 \\pm 6.36$                   | $53.5 \\pm 7.53$                   |\n   | **DARNet(ours)** | $\\textbf{74.0} \\pm \\textbf{11.4}$ | $\\textbf{56.0} \\pm \\textbf{5.71}$ |\n\n   [1] S. Cai, R. Zhang and H. Li, \"Robust Decoding of the Auditory Attention from EEG Recordings Through Graph Convolutional Networks,\" *ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, Seoul, Korea, Republic of, 2024, pp. 2320-2324, doi: 10.1109/ICASSP48485.2024.10447633.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Auditory and audio-visual attention tasks are quite difficult to model, characterize and enable prediction. Specifically, these are not well studied from a learnability point of view. The authors propose a model that combines deep signal processing / classical insights with established spatio-temporal modeling (convolutions in time with attention in space) to build a better than state of the art EEG-driven multi-modal model for attention. The resulting work also improves performance quite significantly compared to baselines / sota. 

There is general agreements among reviewers regarding the impact, and I would suggest the authors to address / capture content from comments on rationale for proposed models' novelty, single vs. dual attention and a summary from additional experimental insights in the final version for publication.
Decision: Accept (poster)