# A Simplified Query-Only Attention for Encoder-Based Transformer Models
## 4 Results  
The results indicate that our method achieves a performance that is comparable to the original, while also simplifying the model's architecture. The accuracy transitions on the dataset I were similar between the previous attention and the simple attention, as shown in Figure 3. The prediction accuracies rapidly increased, according to the training in both methods. Then, the accuracy was saturated. The blue and red lines show the average accuracy among the subjects. The sky blue and light red bounded lines illustrate the standard deviation among the subjects. The \(X\)-axis shows the number of training epochs. The \(Y\)-axis represents the prediction accuracy. The accuracy of the previous method is depicted in Figure 3a. The results of the proposed methods are shown in Figure 3b.  
As shown in Tables 1 and 2, the accuracy across the subjects was similar between the previous and simple attention methods. Tables 1 and 2 depict the average accuracy of all the epochs for each subject on datasets 1 and 2, respectively. PrevConformer means the EEG conformer with previous attention. SimpleConformer implies the EEG conformer with simple attention. The SimpleConformer shows similar accuracy to the PrevConformer, despite its simple structure. Tables 1 and 2 also represent the accuracy of state-of-the-art models [26], such as FBCSP [35], ConvNet [36], EEGNet [37], and DRDA [38]. The results show that the transformer model that is based on attention mechanisms outperforms the other machine learning algorithms.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Model** & **S01** & **S02** & **S03** & **S04** & **S05** & **S06** & **S07** & **S08** & **S09** & **Average** & **Kappa** \\ \hline ConvConformer & 82.29 & 60.69 & 90.31 & 71.02 & 72.57 & 60.68 & 81.94 & 83.55 & 79.30 & 75.82 & 0.6776 \\ SimpleConformer & 81.11 & 60.96 & 90.12 & 73.87 & 73.06 & 61.05 & 82.58 & 85.03 & 77.41 & 76.13 & 0.6817 \\ FBCSP & 76.00 & 56.50 & 81.25 & 61.00 & 55.00 & 45.25 & 82.75 & 81.25 & 70.75 & 67.75 & 0.5700 \\ ConvNet & 76.39 & 55.21 & 89.24 & 74.65 & 56.94 & 54.17 & 92.71 & 77.08 & 76.39 & 72.53 & 0.6337 \\ EEGNet & 85.76 & 61.46 & 88.54 & 67.01 & 55.90 & 52.08 & 89.58 & 83.33 & 86.81 & 74.50 & 0.6600 \\ DRDA & 83.19 & 55.14 & 87.43 & 75.28 & 62.29 & 57.15 & 86.18 & 83.61 & 82.00 & 74.75 & 0.6633 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Prediction accuracy of dataset I.  
Figure 3: Accuracy transition on dataset I. The \(X\)-axis indicates the number of training epochs, and the \(Y\)-axis represents prediction accuracy. (**a**) The results from the EEG conformer with the conventional attention mechanism. The blue line represents the average accuracy across subjects, with the sky-blue shaded area showing the standard deviation. (**b**) The results from the EEG conformer with the simplified attention mechanism. The red line represents the average accuracy among subjects, with the light red shaded area indicating the standard deviation among subjects.  
The prediction accuracies of the conventional and proposed methods were not statistically different in either dataset I or II. For dataset I, the ConvConformer and the SimpleConformer accuracies were 75.82 \(\pm\) 10.33 and 76.13 \(\pm\) 10.09, respectively. For dataset II, the ConvConformer and the SimpleConformer accuracies were 84.72 \(\pm\) 9.59 and 84.62 \(\pm\) 10.45, respectively. The \(p\)-values of the paired-sample \(t\)-tests for datasets I and II were 0.513 and 0.770, respectively.