# Learning-Based Optimisation for Integrated Problems in
## 4 Overview of the State of the Art in the Field of NCO
[105] develop advanced models for solving VRPs incorporating encoder-decoder and RL. [PERSON] and [PERSON] [106] propose an advanced encoder-decoder model for Capacitated Vehicle Routing Problem with Time Windows (CVRP-TW), extending the encoder with rich context embedding. [PERSON] et al. [107] argue that a Convolutional Network can be suitable in cases where the adjacency matrix of a graph is structured similarly to the pixel image. [PERSON] et al. [108] solve the Dynamic Pickup and Delivery Problem (DPDP) with online generated orders. The environment of the dynamic dispatching problem is abstracted as a graph, where each vehicle is a node. Then, the attention network is used to calculate the relationships between the vehicle and all the neighbouring vehicles considering spatial and vehicle features. In this study, the authors tackle the stochastic environment by designing a spatial-temporal network, which can predict the demand for each factory at timestep \(t+1\). [PERSON] et al. [109] tackle the stochasticity in the online Large-scale DPDP by utilising the order buffer, where the n-orders are first cashed and then provided to the policy network designed as HRL. Based on the upper-level policy, the agent decides on the orders to be served from the cash buffer. The lower-level agents aim to serve these orders by assigning them to the most appropriate vehicles and constructing routes similar to classical heuristics. These agents apply four swap actions (inner-exchange, inner-relocate, inter-exchange, and inter-relocate) on the graph nodes of the constructed routes by utilising REINFORCE and GNNs. Thus, we refer to this model as learning-to-configure. We note that this study is among the first to introduce the HRL solution for the real-world large-scale DPDP. Alternatively, [PERSON] et al. [110] adopt a similar architecture to [55], but instead of selecting the next node, two nodes are permuted. To preserve positional information within the tour, authors propose sinusoidal positional encoding. [PERSON] et al. [111] extend this approach with dual encoding and pointing attention, allowing faster convergence to a near-optimal solution in TSP and CVRP. [PERSON] and [PERSON] [112] combine Ruin-Recreate and neural optimisation for CVRP, improving solution search by leveraging local information. We refer to these studies as learn-to-improve. [PERSON] et al. [113] introduce a learn-to-delegate technique for large-scale VRP, integrating attention neural networks and a Multi-Head Attention and VRP solver. This technique demonstrates promising potential for nested problems such as the Location Routing Problem (LRP) or vehicle fleet optimisation with integrated routing problems. There is a notable trend in the prediction and forecasting domain towards the utilisation of the GNN alongside attention mechanisms. Earlier studies attempted to combine LSTM and Convolutional Neural Network (CNN) techniques to capture temporal and spatial patterns [114]. [PERSON] et al. [115] use RL to adjust weights in different time-series models to forecast freight demand movements over short- to long-term horizons. More recently, researchers have successfully applied GCNs and attention mechanisms to predict the dynamics of spatial-temporal sequences [116, 117]. For instance, [PERSON] et al. [23] develop a sophisticated Graph Attention Network (GAN) and supervised learning approach to predict train delays. Interestingly, the authors first formulate the dynamic railway system as a multigraph, where each timestep is a graph containing nodes representing running trains and terminals. Subsequently, they leverage a GNN to dynamically predict delay propagation in the railway network. From the presented State of the Art, it is evident that classical combinatorial optimisation problems, such as VRP and scheduling problems, are primarily treated using end-to-end attention-based encoder-decoder architectures, which are then trained with Actor-Critic DRL.