# Learning-Based Optimisation for Integrated Problems in
## 4 Overview of the State of the Art in the Field of NCO  
In the following, we provide an overview of the studies published between 2018 and 2023 related to the six problem classes introduced in Section 2.2 and NCO. We design the main search query for Google Scholar using the following syntax: PUBLICATION NAME (\"Arxiv OR Transport OR Logistics\") AND TITLE_WORDS (\"(reinforcement OR learn OR neural) AND (freight OR transport OR logistics)\") AND KEYWORDS (problem class name from Table A1). In the first step, we include publishers from the fields of logistics, transport or computer science. In the second step, we narrow the list by including publications that relate to both NCO and transport logistics. Afterwards, we use the problem field keyword from Section 2.2 to further narrow the search space and review selected studies associated with the methods outlined in Section 3. In addition, we also analyse studies from adjacent reviews such as [73] and include articles from the manual search. The reason for the manual search was the broad range of possible topics related to the NCO. For example, [PERSON] et al. [88] propose RL optimisation of the sewer network. Although this study is not directly related to the transport field, we chose to include it due to its methodological transferability. We explicitly exclude studies having main focus on topics such as signal controlling, supply chain, robotics, navigation, public transport, maritime logistics, and Multi-Agent bidding. This literature review also does not cover the classical RL methods  
Figure 4: Illustration of the three NCO implementation strategies (Source: Own diagram). such as simple Q-Learning from the control theory (see, e.g., [70, 89]) as our research aims to explore more advanced NCO methods, which usually rely on Transformer architecture. Below, the surveyed studies are organised according to the problem classes outlined in Section 2.2. Strategic network optimisation in the fields of OR and computer science is traditionally approached through discrete graph optimisation methods. In the reviewed State of the Art, only two studies address network optimisation using NCO. [PERSON] et al. [90] apply hybrid end-to-end DRL, where an Actor-Critic RL with Graph Convolutional Networks (GCNs) policy first finds the maximum capacity for the web links, and then an Integer Linear Programming (ILP) is applied to improve the final network solution. The authors emphasise that RL-based optimisation offers a more natural solution approach than traditional local search heuristics, as it can generate decisions based on global information and consider delayed outcomes. [PERSON] et al. [88] propose a DRL framework based on GCNs for dynamic pipe network optimisation, where a centralised agent learns a maintenance plan based on network features for a 100-year period. Resource assignment, scheduling, and dispatching class covers a broad range of problems related to the railway network and terminal processes. Since this class of problems can be formulated as a sequential decision problem (e.g., Flexible JSSP), the encoder-decoder and end-to-end methods find a natural application here [76, 91]. [PERSON] et al. [92] and [PERSON] and [PERSON] [93] employ an encoder with LSTM/RNN and a decoder with attention to produce sequence-to-sequence solutions. [PERSON] et al. [94] develop an end-to-end approach that can operate on problems with varying state and action sizes. They first encode the trajectories as a graph (with states \(s\) as nodes and actions \(a\) as edges) and process it using a GNN. Subsequently, Q-Learning is employed to select actions (edges). The learning-based approach outperforms benchmark heuristics when applied to various combinatorial optimisation problems, such as Capacitated Vehicle Routing Problem (CVRP) and Job Scheduling. [PERSON] et al. [87] encode a scheduling plan (Gantt chart) as a multi-graph and apply a learn-to-improve strategy, where PPO with a GNN learns search operators to update the MIP solution.