# Learning-Based Optimisation for Integrated Problems in
## 3 Methodological Preliminaries
### Simulator Setup Strategies  
Given the complexity of transport systems, simulators are widely applied. Based on [85], the following three strategies for combining NCO with simulators can be highlighted: (i) end-to-end, where the entire problem is built and solved by NCO independently; (ii) learning-to-configure, where the DRL algorithm parametrises the external model, i.e., computes parameter for an optimisation algorithm and ML alongside optimisation algorithms; or (iii) learn-to-improve, where an algorithm or heuristics iteratively calls the ML model to improve the outcomes.  
End-to-end: The first strategy requires the model and the agent implemented in the same environment. Here, the solution search is completely controlled by the RL agent. The simulator, if needed, is used solely to execute actions and provide the reward. This setup strategy is well-suited for one-dimensional combinatorial problems or problems with a comparatively low degree of interdependencies between decision-makers (VRP, JSSP, Facility Location Problem (FLP), network optimisation, prediction), where the model and the RL algorithm can be formulated as a system in one software environment. The end-to-end approach benefits from improved efficiency by treating the entire system as a single entity, allowing for direct learning from input to output without intermediate steps. However, it requires more expert knowledge and a deep problem understanding due to the need to model the entire transport problem as a dynamic MDP, usually formulated in line with the principles outlined in [49].  
Learn-to-configure: The second strategy focuses on leveraging DRL as an oracle in scenarios where the environment is already implemented and building a new environment is costly. Using DRL to parametrise the simulator or even to select heuristics can help to encapsulate the entire problem logic and yield an optimal partial solution for each timestep. In this case, the algorithm being optimised is treated as a black box. Here, the DRL model can provide the state (e.g., demand and customer locations at timestep \(t\)) and the action  
Figure 3: Three agent setups for solving integrated sequential decision problems with Multi-Agent Reinforcement Learning (MARL). Single agent with centralised learning (**left**), Multi-Agent with decentralised learning (**centre**, (**A**)) and centralised (**centre**, (**B**)), and Hierarchical Multi-Agent (HRL) (**right**). (Source: Own diagram).  
(e.g., the number of vehicles or the set of parameters for the simulator). [PERSON] et al. [86] define this technique as a Dynamic Algorithm Configuration (DAC). When the simulation environment for a certain transport problem is used, the DAC can dynamically determine the set of input parameters (e.g., optimal vehicles, facilities, demand points) for each instance.  
Learn-to-improve: The third strategy is well suited for large-scale, one-dimensional combinatorial problems where traditional heuristics are computationally expensive and solution reliability is crucial. By applying DRL, the solution search process conducted by an external algorithm can be enhanced. In this case, the simulator provides an initial solution to the DRL model, which then executes the search and applies search operators based on the rewards received. The learn-to-improve paradigm consumes more computational time but can obtain higher-quality solutions [87]. Figure 4 illustrates these three scenarios in which the simulator is deployed alongside the NCO.