# Time-Aware and Feature Similarity Self-Attention in Vessel Fuel Consumption Prediction
## 2 Related Work
### Attention Mechanism  
The attention mechanism is developed, especially in the machine translation domain. It was introduced in [19] to preserve previous information and emphasize significant data in the sequence. This improved model performance and alleviated the gradient vanishing problem of RNNs since it used weighted previous information again. The achievement of attention accelerated the development of other attention. [PERSON]. [20] compared various alignment functions and verified the global and local approach for attention. [PERSON]. [21] used attention based _bi-directional_ LSTM (BiLSTM) for sentiment classification.  
Recently, Transformer [22] achieved state-of-the-art performance in machine translation. Transformer's self-attention consists of a scaled dot product between \(Q,K\), and \(V\), which refer to query, key, and value, respectively. Each \(Q,K\), and \(V\) is projected by input \(X\) and its corresponding weights \(W_{q},W_{k}\), and \(W_{v}\). The self-attention calculates the dot products between the query and key, divides them by \(\sqrt{d_{k}}\) as a scaling factor, where \(d_{k}\) is the dimension of features. It additionally applies a softmax function to obtain the attention weight, \(\alpha\), and is used to emphasize data on value. The process of self-attention is defined as follows:  
\[Attention(Q,K,V)=softmax(\frac{QK^{\top}}{\sqrt{d_{K}}})V \tag{2}\]  
Using the outputs of the previous layer as the inputs of self-attention, it emphasizes significant information in a sequence. Due to good compatibility, other studies employed a part of Transformer's self-attention and applied it on RNNs [23; 24]. Besides, the attention mechanism's effectiveness was not confined to machine translation, it was proved in diverse domains (e.g., NLP, recommendation, and time series) [25; 26; 27].  
In the irregular time interval domain, several studies have been conducted to apply self-attention to address different time intervals in a sequence [28; 29]. The authors of [29] used a Transformer encoder, a symmetric time interval matrix, and a position matrix as inputs to consider irregular time in the recommendation system. It also applied embedding to represent hidden information from the time interval and position information. The attention process is similar to the Transformer's self-attention. It calculates the scaled dot product between \(Q,K^{\prime}\), and \(V^{\prime}\), where \(K^{\prime}=K+E_{K}^{R}+E_{K}^{P}\) and \(V^{\prime}=V+E_{V}^{P}+E_{V}^{P}\). Each \(E^{R}\) and \(E^{P}\) is the result of embedding the time interval and position information. The attention process is as follows:  
\[Attention(Q,K^{\prime},V^{\prime})=softmax(\frac{QK^{\prime\top}}{\sqrt{d_{K }}})V^{\prime} \tag{3}\]  
The embedding of the irregular time intervals was useful in obtaining the hidden information and time meaning of users when constructing the recommendation system. Inspired by the work in [29], the difference of the domain and time meaning motivated us to propose other approaches to represent the time interval. The irregular time interval occurs in the vessel domain mainly because of the problem of data acquisition and preprocessing for noise. The irregular time in the previous study reflects the behavior information of the specific user, whereas the irregular time in the vessel does not represent hidden information of the specific vessel. This indicates it is necessary to make representations considering discontinuous time steps above all things.