# Learning-Based Optimisation for Integrated Problems in
## 3 Methodological Preliminaries
### RL Training Algorithm  
DRL is a machine learning (ML) paradigm for learning actions based on observations and feedback information in a dynamic environment. In the context of NCO, DRL is primarily applied to train encoder-decoder Neural Networks (NNs), including MHA layers--usually referred to as a policy network for action selection. The RL algorithms rely on a Markov Decision Process (MDP), representing the problem environment as a tuple of state \(s_{t}\), action \(a_{t}\), reward \(r_{t}\), and transition \(T_{t}\): \(MDP=(s_{t},a_{t},r_{t},T_{t})\). The state \(s_{t}\) at each timestep \(t\) forms the input vector for the policy network. The role of the RL training algorithm is to calculate the loss \(L\)--the reward difference between the previous state \(s_{t-1}\) and the current state \(s_{t}\) after taking action \(a_{t}\). The backpropagation of the loss \(L\) enables the encoder-decoder networks to learn a behaviour that maximises the reward.  
For example, in the classical VRP using RL, the reward is defined as a negative tour length and the parameters \(\theta\) are adjusted during the training, where \(\theta\) is a vector of all trainable parameters in embedding, encoder-decoder and attention weighting matrices (e.g., \(W_{q}\), \(W_{k}\), \(W_{v}\)).  
\[J(\theta|s)= \mathbb{E}_{\pi_{q}}[r(a|s)] \tag{2}\] \[\theta=arg\max_{\theta}J(\theta|s) \tag{3}\]  
Figure 2: A prototypical illustration of the NCO framework, including the models typically applied in each NCO design block (Source: Own diagram).  
Equation (2) represents the objective function, where \(\theta\) is the policy parameters and \(s\) defines the current state of the environment. The objective is to maximise the expected reward obtained by following policy \(\pi_{\theta}\). The expected reward is denoted by \(E\), and \(r(a|s)\) is the reward obtained by executing policy \(\pi_{\theta}\) and taking the action \(a\) in state \(s\). Equation (3) denotes the policy parameters optimised by RL that maximise objective function \(J(\theta|s)\).  
In fact, the design choice of DRL algorithm is very problem-specific. Generally, the first choice lies on whether we want to train the policy network directly (policy-based learning) or can evaluate and learn from every state-action tuple (value-based learning or Q-Learning). Policy-based RL is more suitable for problems with a large state and action spaces due to the fact, that it only updates parameters of the policy NN. In contrast, value-based RL learns values for each state or state-action pair, which can lead to insufficient convergence when dealing with vast state-action spaces. Although value-based algorithms traditionally found their application in control problems, such as inventory and supply chain management (see [67]), transport researchers have also applied Q-Learning in train scheduling and dispatching, as demonstrated by [68; 69; 70]. We note that the majority of the studies in the current State of the Art use hybrid approaches called Actor-Critic (e.g., Proximal Policy Optimisation (PPO), REINFORCE) where policy update (actor) and value-based learning (critic) are performed in a feedback loop. Thus, the goal of (i) an actor network is to estimate the probability of the next action in each state, and (ii) a critic is to predict the reward (value) from a given state. To reduce the variance during training, the researchers subtract a so-called baseline from the loss (i.e., the best value so far, produced by the previous policy [55; 56]. For instance, [PERSON] et al. [71] subtract \(\hat{Q}\), an additional policy network with less frequent updates, from the current value function Q to stabilise training. For a comprehensive overview of DRL application in the field of transport, we refer to [72; 73; 74; 75].