# Learning-Based Optimisation for Integrated Problems in
## 3 Methodological Preliminaries
### DNN Model  
During the past decade, DL scholars have extensively experimented with various DNNs, including Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM) networks, and Convolutional Neural Networks (CNN) to predict target actions in dynamic systems [50; 51; 52; 53; 54]. The breakthrough in this direction was achieved by [PERSON] et al. [10], demonstrating that sophisticated encoder-decoder networks can essentially predict the next value based on all previous context information. Today, many state-of-the-art solutions in the field of combinatorial optimisation employ encoder-decoder architecture, which involves training an encoder to learn a meaningful representation of the input problem and a decoder to use this representation to infer a probability over the next actions. The encoder's role in the context of combinatorial optimisation is to embed the problem state, including its features (e.g., distance, customer demand, and time windows in context of the VRP), into a fixed vector representation. The encoder thus encodes the input vector \(s_{t}\in\mathbb{R}^{x}\) into a new extended vector \(emb_{s_{t}}\in\mathbb{R}^{n}\), where \(x\) is the number of problem state features and \(n\) is the embeddings size (usually \(n=128\)). Typically, the encoder is represented by multiple feedforward networks containing several Multi-Head Attention (MHA) layers, whose role is to learn a new, extended representation of the input data. The encoder's output can subsequently be used by the decoder to create a probability distribution for all possible next nodes. Most state-of-the-art encoder-decoder models today rely on the attention or MHA mechanisms at their core [55; 56; 57]. In general, MHA layers quantify the relationships between two nodes in an embedded space by calculating a dot product. The elements (e.g., node embeddings \(emb_{s_{t}}\)) are each multiplied by three matrices: (i) Query \(Q^{i}=W_{q}emb_{s_{t}}\), (ii) _Key_\(K^{i}=W_{k}emb_{s_{t}}\), and (ii) Value \(V^{i}=W_{v}emb_{s_{t}}\), where \(W_{q},\ W_{k},W_{v}\) are trainable parameters, and \(emb_{s_{t}}\) represents the embeddings of the problem state \(s\) at timestep \(t\). By adjusting the weights \(W_{q},\ W_{k}\), and \(W_{v}\) during training, the importance of each element (e.g., embedded node) is calculated in relation to itself and to all other problem elements. Concretely, this means that the probability of selecting the next action \(a_{t+1}\) is derived from the attention score produced by the decoder: \(a_{t+1}=P(u_{dec})\), where \(u_{dec}\) is the attention score calculated according to [10] as  
\[u_{dec}=softmax\bigg{(}\frac{QK^{T}}{\sqrt{\text{dim}K}}\bigg{)}V \tag{1}\]  
In Equation (1), \(softmax\) represents the softmax function that output the probabilities over the dot product between queries, keys and values, divided by the scaling factor \(\sqrt{\text{dim}K}\), where \(dim\) indicates the size of the key vector \(K\). To learn additional information from the problem context, such as dynamic problem states and implicit information, the context vector can be additionally concatenated and projected in the decoder. In this context, the additional context information from the environment can be added to the vector outputted by the encoder as follows: \(c_{t}=FF(emb_{s_{t}}||c_{0})\), where \(FF\) represents a feedforward network (usually with a single hidden layer), \(c_{0}\) defines the problem context added to the learned representation, and (\(||\)) denotes concatenation. In fact, by including the context vector \(c_{t}\) we inject additional state information, enabling the encoder-decoder models to capture problem dynamics during the learning episode, such as the remaining vehicle capacity or the remaining number of empty vehicles in VRP. To prevent the selection of inappropriate actions, practitioners apply masking, a technique which eliminates the probabilities for undesirable actions in the decoder [55; 58; 59].