# A Simplified Query-Only Attention for Encoder-Based Transformer Models
## 2 Related Works  
Transformers were proposed for sequence-to-sequence tasks, such as machine translation, in which a sentence in one language is converted into another [1]. The transformer architecture consists of an encoder and a decoder. The encoder is responsible for encoding the input sequence into a compact representation, while the decoder generates the output sequence (e.g., a translated sentence) from this representation. The attention mechanism plays a crucial role by allowing the model to focus on different parts of the input sequence at each step of the output generation, modeling the relationships within and between the input and output sequences.  
Owing to their exceptional performance in NLP, transformers have been widely adopted across domains beyond text processing [20; 26; 27; 28; 29; 31; 32; 33; 34]. In computer vision, where convolutional neural networks (CNNs) were traditionally dominant, the vision transformer (ViT) introduced a novel approach by dividing an image into patches and processing them using a transformer without any convolutional layers [20]. The ViT could achieve superior performance compared to state-of-the-art (SOTA) CNNs. This breakthrough led to the widespread adoption of transformers in image recognition tasks [31; 32; 33; 34].  
Transformers have also been increasingly applied in neural signal processing [26; 27; 28; 29]. For instance, the EEG conformer model combined convolutional layers and a transformer [26]. The EEG conformer extracts features from EEG signals through convolutional layers and processes them within a transformer to predict the user's intent. The EEG conformer has shown superior performance compared to previous state-of-the-art models in this domain. Many BCI studies have employed transformers for various classification tasks [27; 28; 29].  
In many classification tasks, such as image recognition and neural signal processing, the encoder is often used independently, without the decoder. This is because the encoder compresses the high-dimension input sequences into the low-dimension outputs, whereas the decoder expands the low-dimension inputs into the high-dimension output sequences [20; 26; 27; 28; 29]. As a result, the encoder-based transformer does not require attention between input and output sequences, as used in sequence-to-sequence tasks like translation. In the encoder, the \(Q\), \(K\), and \(V\) vectors are all derived from the same input sequence through linear transformations, while in the decoder, \(Q\), \(K\), and \(V\) are generated from the input and output sequences [1]. Consequently, the \(Q\), \(K\), and \(V\) vectors in the encoder can be similar to each other, leading to redundancy. Despite this, many applications use distinct \(Q\), \(K\), and \(V\) components, even when only the encoder is utilized.  
In this paper, we propose a simplified query-only attention mechanism for encoder-only transformer models. Our experiments demonstrated that the query-only attention mechanism achieves good performance comparable to traditional attention, which uses \(Q\), \(K\), and \(V\), even in a simple structure. The proposed mechanism saves computational time and memory by eliminating the need to generate and learn the \(K\) and \(V\) components. This approach can be applied to most transformer models that rely solely on the encoder.