# A Simplified Query-Only Attention for Encoder-Based Transformer Models
###### Abstract  
Transformer models have revolutionized fields like Natural Language Processing (NLP) by enabling machines to accurately understand and generate human language. However, these models' inherent complexity and limited interpretability pose barriers to their broader adoption. To address these challenges, we propose a simplified query-only attention mechanism specifically for encoder-based transformer models to reduce complexity and improve interpretability. Unlike conventional attention mechanisms, which rely on query (Q), key (\(K\)), and value (\(V\)) vectors, our method uses only the \(Q\) vector for attention calculation. This approach reduces computational complexity while maintaining the model's ability to capture essential relationships, enhancing interpretability. We evaluated the proposed query-only attention on an EEG conformer model, a state-of-the-art architecture for EEG signal classification. We demonstrated that it performs comparably to the original \(Q\)\(K\)V attention mechanism, while simplifying the model's architecture. Our findings suggest that query-only attention offers a promising direction for the development of more efficient and interpretable transformer-based models, with potential applications across various domains beyond NLP.  
t +
Footnote †: journal: Applied  
1  
class=AMS] [Primary = ]  
global dependencies by combining CNNs and transformers [18]. Similarly, the EEG conformer performed excellently by combining CNNs and transformers for neural signal processing [26]. The D-FaST applied attention to frequency, space, and time domains [29].  
The transformer is composed of encoders and decoders [1]. The encoder consists of multi-head attention and a feed-forward network, and the decoder comprises masked multi-head attention, multi-head attention, and a feed-forward network. The feed-forward network is a conventional neural network, and multi-head attention is the core of the transformer. Attention is calculated from query, key, and value inputs by computing the dot product, scaling, softmax, and another dot product. Multi-head attention performs the process several times to take different attention views. Masked multi-head attention computes attention by masking a part of the input data to prevent the use of future data as input.  
Numerous modified versions of the transformer algorithms have been proposed, using only the encoder part for classification [20; 26; 27; 28; 29]. Most of them use query (\(Q\)), key (\(K\)), and value (\(V\)) to calculate the attention as initially proposed [20; 26; 27; 28; 29]. The \(K\) and \(V\) can be redundant in the encoder. However, many modified algorithms for the encoder-based transformer still use all the \(Q\), \(K\), and \(V\) components.  
In this paper, we propose a simplified query-only attention mechanism to enhance the efficiency and interpretability of transformer models by removing redundant elements. The transformer, using only a \(Q\), performed similarly to the transformer using \(Q\), \(K\), and \(V\), even in a simple structure. To evaluate the proposed method, we applied the simple attention and conventional attention mechanisms to the equal transformer model and compared their accuracy. To show that the proposed method plays the same role as the previous method, even in simple structures, we applied the simple attention method and the previous attention method to the equal structure of the transformer. We compared the performance of the two cases. An EEG conformer was used as the transformer structure because the state-of-the-art model performs well by extracting local features with a convolutional layer and recognizing global relations with attention. The open datasets (BCI competition IV dataset 2a and b) were used, which are commonly used for evaluation [30].