# Learning-Based Optimisation for Integrated Problems in
## Appendix A
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Studies** & **Problem** & **Learning** & **Models** & **ma*** & **Solution Strategy** \\ \hline \multirow{4}{*}{[PERSON] et al. [77]} & \multicolumn{4}{c}{Resource Allocation/Balancing/Fleet Composition} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & End-to-end. Cooperative \\  & Resource balancing & DQL & MLP for each agent & X & Multi-Agent RL (MARL) with shared state and rewards \\ \cline{1-1}  & \multirow{2}{*}{Assignment and rebalancing} & Factorised Actor–Critic & GCN & X & End-to-end. Decentralised policy for each zone with parameters sharing \\ \cline{1-1}  & \multirow{2}{*}{Wehicle repositioning} & Actor–Critic & Q-Network for each agent & X & Hierarchical RL (HRL) where each sub-agent adopts LSTM \\ \cline{1-1}  & \multirow{2}{*}{Vehicle repositioning} & PPO & DQN and LSTM & X & 3-Level HRL. DQN for workers and LSTM for prediction. \\ \cline{1-1}  & \multirow{2}{*}{Vehicle fleet} & \multirow{2}{*}{Q-Learning} & NN and tabular Q & X & End-to-end \\ \cline{1-1}  & & & & & \\ \hline \multirow{4}{*}{[PERSON] et al. [83]} & Order batching and sequence problem & \multirow{2}{*}{PPO} & \multirow{2}{*}{DNN} & End-to-end with reward shaping for two-objective problems \\ \cline{1-1}  & \multirow{2}{*}{Batching and sequencing orders} & \multirow{2}{*}{PPO} & \multirow{2}{*}{DNN} & Hybrid-method. Heuristics to take sequencing decisions and DRL to take batching decisions. \\ \cline{1-1}  & \multirow{2}{*}{Reordering of containers} & \multirow{2}{*}{Supervised} & DNN & Learn-to-improve. Heuristics utilise RL for improvement \\ \hline \multirow{4}{*}{[PERSON] et al. [55]} & \multicolumn{4}{c}{Vehicle Routing Problem} & \multicolumn{1}{c}{} & End-to-end. Prediction of next nodes using GAN Attention Network, \\ \cline{1-1}  & \multirow{2}{*}{TSP} & REINFORCE & with GAN and attention & masking and RL-baseline. \\ \cline{1-1}  & & & Encoder-decoder & & \\ \cline{1-1}  & \multirow{2}{*}{TSP} & REINFORCE & with self-attention & & End-to-end \\ \cline{1-1}  & \multirow{2}{*}{TSP} & REINFORCE & and Pointer & & & End-to-end \\ \cline{1-1}  & \multirow{2}{*}{CVRP with heterogeneous vehicle fleet} & Network & & & & \\ \cline{1-1}  & \multirow{2}{*}{VRP with TW} & \multirow{2}{*}{policy DRL with baseline.} & \multirow{2}{*}{Multi-Head Attention} & \multirow{2}{*}{End-to-end} \\ \cline{1-1}  & & & & & & \\ \cline{1-1}  & \multirow{2}{*}{VRP with TW} & Policy gradient & \multirow{2}{*}{Encoder-decoder} & X & Centralised training with shared parameters and observation \\ \cline{1-1}  & \multirow{2}{*}{CMVRP} & Actor–Critic & Encoder-decoder & X & End-to-end MARL with centralised training and decentralised execution \\ \cline{1-1} \cline{3-4}  & \multirow{2}{*}{VRP} & Policy-based & \multirow{2}{*}{Encoder-decoder with RNN and attention} & \multirow{2}{*}{End-to-end} \\ \cline{1-1}  & & & RL & attention & & \\ \cline{1-1} \cline{3-4}  & \multirow{2}{*}{TSP} & Supervised & GNN & & End-to-end \\ \cline{1-1}  & \multirow{2}{*}{Pick-up and delivery VRP} & \multirow{2}{*}{policy DRL with baseline.} & \multirow{2}{*}{Encoder-decoder with self-attention} & \multirow{2}{*}{End-to-end} \\ \cline{1-1}  & & & & & & End-to-end \\ \cline{1-1} \cline{3-4}  & \multirow{2}{*}{CVRP-TW} & \multirow{2}{*}{REINFORCE} & Encoder-decoder & & End-to-end. Extended \\ \cline{1-1}  & \multirow{2}{*}{CVRP–TW} & \multirow{2}{*}{REINFORCE} & with attention & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & \multirow{2}{*}{Assignment and routing} & \multirow{2}{*}{modified PPO} & \multirow{2}{*}{CNN} & X & \multirow{2}{*}{actor-networks for node selection and assignment} \\ \cline{1-1}  & & & & & End-to-end. Predict demand and \\ \cline{1-1}  & \multirow{2}{*}{Dynamic pick-up and delivery} & Double & Graph & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & \multirow{2}{*}{Q-Learning} & spatial–temporal attention network & & & End-to-end \\ \hline \hline \end{tabular}
\end{table}