# A Deep Reinforcement Learning-Based Decision-Making Approach for Routing Problems
## 3 Methodology
### Encoder  
The encoder in our framework comprises \(L\) stacked layers, each consisting of a multi-head attention (MHA) sublayer followed by a feedforward network (FFN) sublayer. While conventional DRL-based methods adopt residual connections and post-layer batch normalization within the transformer architecture to produce high-dimensional node embeddings, this design often falls short of capturing effective state representations. To address this limitation, we draw upon advances in natural language processing [26] and the gated transformer-XL (GTrXL) architecture [27] by incorporating a batch normalization fronting (BNF) mechanism and a gate-like threshold (GT) block. These components jointly enhance the encoder's capacity for feature extraction and representation learning, leading to improved modeling of the routing environment. The encoding process begins by projecting the raw input features into a high-dimensional space to obtain the initial node embeddings \(G^{(0)}\in\mathbb{R}^{N\times d_{g}}\) through a linear projection:  
\[G^{(0)}=XW_{\text{linear}}+b_{\text{linear}}, \tag{3}\]  
where \(W_{\text{linear}}\) and \(b_{\text{linear}}\) represent denote trainable weight and bias parameters, respectively. The resulting matrix \(G^{(0)}\) is then passed through a sequence \(L\) attention layers. For the \(l\)-th layer, where \(l\in\{1,2,\cdots,L\}\), the input \(G^{(l-1)}\) is firstly normalized in a batch-wise manner and subsequently processed by the MHA block to capture contextual dependencies among nodes. \[G^{(l)}_{\text{BN}}=\text{BN}\Big{(}G^{(l-1)}\Big{)}, \tag{4}\]  
\[\hat{G}^{(l)}=\text{MHA}\Big{(}G^{(l)}_{\text{BN}}W^{Q},G^{(l)}_{\text{BN}}W^ {K},G^{(l)}_{\text{BN}}W^{V}\Big{)} \tag{5}\]  
\[Z^{(l)}_{y}=\text{Softmax}\Bigg{(}\frac{Q_{l,y}K^{T}_{l,y}}{\sqrt{d_{g}}} \Bigg{)}V_{l,y}, \tag{6}\]  
\[Q_{l,y}=G^{(l)}_{\text{BN}}W^{Q},\ K_{l,y}=G^{(l)}_{\text{BN}}W^{K},\ V_{l,y}=G^ {(l)}_{\text{BN}}W^{V}, \tag{7}\]  
where \(\text{BN}(\cdot)\) denotes the batch normalization operator, \(\text{Concat}(\cdot)\) represents matrix concatenation, and \(\text{Softmax}(\cdot)\) is the softmax activation function. The trainable projection matrices for the query, key, value, and output transformations are defined as \(W^{Q},W^{K}\in\mathbb{R}^{d_{s}\times d_{q}}\), \(W^{V}\in\mathbb{R}^{d_{g}\times d_{q}}\), and \(W^{O}\in\mathbb{R}^{d_{g}\times d_{g}}\), respectively. Here, the dimensionalities satisfy \(d_{g}=d_{q}=d_{k}=Yd_{v}\), where \(Y\) denotes the number of attention heads. An overview of the MHA mechanism is provided in Figure 2a. Building upon this design, we replace the conventional residual connection with a GT block, as illustrated in Figure 2b, to further enhance state representation learning. In particular, the GT block computes a weight matrix from the input vector \(g_{\text{in}}\) and the output vector \(g_{\text{out}}\). This mechanism enables adaptive modulation of information flow, thereby improving the quality of the learned representations. \[\text{GT}(g_{\text{in}},g_{\text{out}})=g_{\text{in}}+\text{Sigmoid}\big{(}g_ {\text{in}}W_{\text{gate}}+b_{\text{gate}}\big{)}\odot g_{\text{out}}, \tag{8}\]  
where \(\text{Sigmoid}(\cdot)\) represents the sigmoid function, \(\odot\) denotes the element-wise product, \(W_{\text{gate}}\) and \(b_{\text{gate}}\) are trainable parameter matrices. Therefore, the output of the MHA sublayer can be expressed as:  
\[\widetilde{G}^{(l)}=\text{GT}\Big{(}G^{(l-1)},\text{MHA}\Big{(}\text{BN}\Big{(} G^{(l-1)}\Big{)}\Big{)}\Big{)}. \tag{9}\]  
In the subsequent FFN sublayer, the input \(\widetilde{G}^{(l)}\) is first processed by the BNF block, followed by transformation through the FFN block.