# A CNN-LSTM-Attention Model for Near-Crash Event Identification on Mountainous Roads
## 4 Near-Crash Event Identification Model
### Attention Layer  
During the detection of near-crash events from naturalistic driving data, the dynamic features of various variables play a critical role. For example, rapid changes in vehicle acceleration and heading angle often signal the occurrence of near-crash events. However, as the LSTM compresses the entire sequence into a singular, fixed-length hidden vector and assigns equal weights to information at different times, it might overlook key information at certain moments, such as drastic changes in acceleration, thereby reduce the predictive accuracy of the model. To address this limitation, this study introduces an attention mechanism [34]. Its inspiration comes from the human biological system, where humans tend to focus on distinct parts when processing large amounts of information. With the development of deep neural networks, the introduction of attention mechanisms helps networks capture more important information, thereby optimizing the computational performance of the entire network. The attention mechanism enhances prediction accuracy and improves computational efficiency by selectively focusing on important input elements.  
The attention mechanism is divided into hard attention and soft attention. The hard attention applies attention directly to a specific position in the input sequence. It is akin to a selection operation, and only inputs the features of that position into the model to obtain precise attention information. The soft attention distributes attention in the form of weighted vectors to the features at each position in the input sequence. By considering the entire sequence, it makes the model more fault-tolerant and suitable for longer sequence data.  
The attention mechanism is widely used in fields such as drone controlling and naturalistic driving. [PERSON] et al. [35] added a self-attention mechanism to the Transformer  
Figure 6: LSTM neural network basic unit.  
network to minimize drift issues caused by drone movement. [PERSON] et al. [36] proposed a multi-instance learning approach based on the attention mechanism to accurately locate anomalous events in drone flight data. [PERSON] et al. [37] proposed a Transformer Encoder model with an attention mechanism to predict vehicle lane change intentions.  
This paper adopts the soft attention mechanism, which allocates variable levels of attention to different segments of the data based on a probability-weighted distribution. This approach ensures that significant moments, especially those indicative of near-crash events, are accentuated, thereby enhancing the model's capacity to capture essential temporal information more accurately.  
The implementation of this mechanism within this study employs the scaled dot-product attention technique. This method linearly maps the input sequence to produce three different interpretation vectors: the query vector \(Q\), the key vector \(K\), and the value vector \(V\). Following this, an attention score for each sample is derived through a similarity function \(F(Q,K)\). These scores are then normalized through the softmax function, establishing the weight coefficients for each temporal segment. These coefficients finally multiply with the value vector \(V\) to produce the output of the attention mechanism \(Attention\) (\(Q,K,V\)).  
\[Attention(Q,K,V)=softmax\left(\frac{F(Q,K)}{\sqrt{d_{k}}}\right)V \tag{13}\]  
where \(d_{k}\) represents the dimension of \(Q\), \(K\), and \(V\). It refers to the dimension of the input vector, which is also the dimension of \(Q\), \(K\), and \(V\) vectors. \(F(Q,K)\) is the similarity function:  
\[F(Q,K)=QK^{T} \tag{14}\]