# A Simplified Query-Only Attention for Encoder-Based Transformer Models
## 3 Materials and Methods
### Transformer Architecture  
We applied the proposed simple attention and the previous attention to the structure of the EEG conformer and compared the accuracies. The EEG conformer is a state-of-the-art deep learning model for EEG signal classification [26]. It extracts local and global features using convolution and attention modules. The EEG conformer consists of three main modules: a convolutional module, a self-attention module, and a classifier module, as illustrated in Figure 2.  
(1) Convolutional module: This module consists of two convolutional layers and one pooling layer. The first convolutional layer extracts temporal features using 40 kernels (1 \(\times\) 25) with a stride of (1, 1). The second convolutional layer plays a role as a spatial filter, using 40 kernels (22 \(\times\) 1) with a stride of (1, 1). The number of electrode channels was 22. The average pooling layer reduces the amount of feature dimension using a kernel (1 * 75) with a stride of (1, 15). Consequently, the convolutional module effectively captures the local spatio-temporal features of EEG signals. (2) Self-attention module: This module utilizes a self-attention mechanism to capture global interactions. Self-attention learns relationships between distant EEG signals, enabling more accurate classification. Figure 1: Experimental paradigm. (**a**) Paradigm of dataset I. (**b**) Paradigm without feedback of dataset II. (**c**) Paradigm with feedback of dataset II. The previous self-attention used query (\(Q\)), key (\(K\)), and value (\(V\)) to calculate the attention, as in Figure 2a. The previous attention is computed using Equation (1). \[Attention(Q,\ K,\ V)=softmax\Big{(}\frac{QK^{T}}{\sqrt{d_{k}}}\Big{)}V \tag{1}\]  
where \(d_{k}\) denotes the length of \(Q\), \(K\), and \(V\). A single-layer, feed-forward network generates the \(Q\), \(K\), and \(V\) from processed EEG signals. These values can be considered latent vectors of the input (EEG). Therefore, the \(K\) and \(V\) are redundant with the \(Q\), although the \(Q\), \(K\), and \(V\) might differ slightly. The \(Q\) can substitute the \(K\) and \(V\). The proposed query-only attention mechanism calculates attention scores using only the \(Q\) vector, as in Figure 2b. It can be formulated using Equation (2). \[Attention(Q)=softmax\Big{(}\frac{QQ^{T}}{\sqrt{d_{k}}}\Big{)}Q \tag{2}\]  
In the encoder, the \(Q\), \(K\), and \(V\) are linear combinations of the same input X. Therefore, the \(Q\), \(K\), and \(V\) can be represented as \(AX\), \(BX\), and \(CX\), respectively. The expression \(QK^{T}V\) in Equation (1) can be rewritten as \(AXX^{T}B^{T}CX\). If we represent \(Q\) in Equation (2) as \(DX\), where \(D\) is a matrix trained from a different linear network, then \(QQ^{T}Q\) becomes \(DXX^{T}D^{T}DX\). Thus, when \(A\) equals \(D\) and \(B^{T}C\) equals \(D^{T}D\), Equations (1) and (2) become equivalent. Therefore, in this case, Equation (1) can be entirely replaced by Equation (2). However, a matrix \(D\) that mathematically satisfies the conditions \(A=D\) and \(B^{T}C=D^{T}D\) may not always exist, and \(A\), \(B\), and \(C\) are not fixed matrices with specific val  
Figure 2: EEG conformer structure. The convolutional module is illustrated in sky blue, the self-attention module in light green, and the fully connected layer module in orange. (**a**) EEG conformer with the conventional attention mechanism using query (\(Q\)), key (\(K\)), and value (\(V\)) vectors. (**b**) EEG conformer with the proposed simplified attention mechanism using only a query (Q). ues. The matrices can be changed based on the initial conditions and the training iterations. Therefore, a \(D\) matrix may be found that produces outputs similar to those generated by the flexible combination \(AXX^{T}B^{T}CX\). The replacement makes the model efficient and allows for a more intuitive understanding. Figure 2 illustrates the procedures of the previous and the proposed attention. Multi-head attention performs a self-attention \(h\) times to capture the diverse interactions.