# A Simplified Query-Only Attention for Encoder-Based Transformer Models
## 5 Discussion
### Model Simplification and Its Implications  
The simplification of model architectures has often driven advancements in technology. Using only the attention mechanism, the original transformer achieved a superior performance by eliminating the recurrence and convolution [1]. Similarly, the performance of AlphaGo, the first algorithm to win over humans in the game of Go, was improved by simplifying the architecture. AlphaGo employed separate policy and value networks [39]. After AlphaGo's improvement, AlphaGo Zero, AlphaZero, and MuZero utilized a unified network for policy and value [40; 41; 42].  
The proposed query-only attention mechanism could simplify the architecture by eliminating the redundant \(K\) and \(V\) vectors while achieving a performance that is comparable to the traditional model. As a result, the number of linear models to generate \(Q\), \(K\), and \(V\) vectors could be reduced by one-third. It implies that the linear model's training time and memory space also be decreased by one-third.  
The relatively complex structure of conventional attention mechanisms can make it challenging to understand the underlying principles of the algorithm. The meanings and roles of the \(Q\), \(K\), and \(V\) vectors differ between the encoder and decoder, which can further complicate intuitive understanding. Particularly in the case of encoder-based transformers, the \(Q\), \(K\), and \(V\) vectors are generated through the same process from the same input, making it unclear what each of these vectors signifies and why different values are necessary. In contrast, the proposed method simplifies this process by replacing the ambiguous \(K\) and \(V\) vectors with \(Q\), as illustrated in Figure 2, potentially making the algorithm more straightforward to comprehend. The \(Q\) vector can be considered as a transformed input. Therefore, the proposed method clarifies that attention values are calculated based on the relationships within the input itself. In conclusion, the proposed attention mechanism enhances interpretability by replacing the unnecessary and ambiguous \(K\) and \(V\) values with the interpretable \(Q\) value. This enhanced intuitiveness could facilitate the development of new algorithms.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Model** & **S01** & **S02** & **S03** & **S04** & **S05** & **S06** & **S07** & **S08** & **S09** & **Average** & **Kappa** \\ \hline ConvConformer & 72.80 & 67.81 & 80.07 & 94.76 & 93.98 & 82.80 & 89.56 & 92.53 & 88.17 & 84.72 & 0.6944 \\ SimpleConformer & 71.79 & 66.18 & 79.49 & 96.63 & 93.91 & 82.40 & 89.52 & 92.66 & 88.98 & 84.62 & 0.6924 \\ FBCSP & 70.00 & 60.36 & 60.94 & 97.50 & 93.12 & 80.63 & 78.13 & 92.50 & 86.88 & 80.00 & 0.6000 \\ ConvNet & 76.56 & 50.00 & 51.56 & 96.88 & 93.13 & 85.31 & 83.75 & 91.56 & 85.62 & 79.37 & 0.5874 \\ EEGNet & 75.94 & 57.64 & 58.43 & 98.13 & 81.25 & 88.75 & 84.06 & 93.44 & 89.69 & 80.48 & 0.6096 \\ DRDA & 81.37 & 62.86 & 63.63 & 95.94 & 93.56 & 88.19 & 85.00 & 95.25 & 90.00 & 83.98 & 0.6796 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Prediction accuracy of dataset II.