# Transfer Reinforcement Learning for Combinatorial Optimization Problems
## 5 Comparison with Other Studies  
This section presents a comparison of this proposal with other literature studies. For this, three papers that applied AI algorithms to solve the SOP were selected [37; 39; 80]. Table 11 shows a summary of this analysis. The first point to emphasize is the relevance of the dataset used in this paper. In this sense, TSPLIB is frequently adopted in the literature when applying methods to combinatorial optimization problems, especially those related to the TSP and SOP. The second criterion to be observed in Table 11 is the algorithm adopted to resolve SOP instances. Several traditional optimization techniques have already been used, such as the ant colony system [39] and particle swarm optimization [80]. It is noteworthy that the authors of [39; 80] achieved important results in solving several TSPLIB instances, finding the best known solution in several simulated situations, thus showing great potential in these methods for solving combinatorial optimization problems based on the ATSP and SOP. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**ATSP**} & \multirow{2}{*}{**SOP**} & **Best Known Solution** & **Without Auto\_TL\_RL** & **With Auto\_TL\_RL** \\ \hline \multirow{2}{*}{br17} & br17.10 & 55 & 57 & **55** \\  & br17.12 & 55 & **57** & **57** \\ \hline \multirow{4}{*}{p43} & p43.1 & 28,140 & 28,765 & **28,715** \\  & p43.2 & 28,480 & 29,265 & **29,170** \\  & p43.3 & 28,835 & 29,545 & **29,535** \\  & p43.4 & 83,005 & 84,110 & **83,985** \\ \hline \multirow{4}{*}{ry48p} & ry48p.1 & 14422 & 18,154 & **17,922** \\  & ry48p.2 & 16,074 & 18,549 & **18,459** \\  & ry48p.3 & 19,490 & **22,789** & 22,853 \\  & ry48p.4 & 31,446 & 38,235 & **37,679** \\ \hline \multirow{4}{*}{ft53} & ft53.1 & 7531 & **8852** & 9056 \\  & ft53.2 & 8026 & 9839 & **9588** \\  & ft53.3 & 10,262 & 12,598 & **12,594** \\  & ft53.4 & 14,425 & 17,650 & **16,935** \\ \hline \multirow{4}{*}{ft70} & ft70.1 & 39,313 & 43,460 & **42,707** \\  & ft70.2 & 40,101 & 44,841 & **44,499** \\ \cline{1-1}  & ft70.3 & 42,535 & **48,015** & 48,311 \\ \cline{1-1}  & ft70.4 & 53,530 & 60,049 & **59,275** \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Results of running SOP instances without the Auto\_TL\_RL algorithm and with the Auto\_TL\_RL algorithm, with the asymmetric traveling salesman problem (**ATSP**); sequential ordering problem (**SOP**); **best known solution** value presented by TSPLIB; and the proposed algorithm (**Auto\_TL\_RL**). \begin{table}
\begin{tabular}{l l c c c} \hline \hline  & & **Proposed** & [37] & [39] & [80] \\ \hline \multirow{2}{*}{Dataset} & TSPLIB & ✓ & ✓ & ✓ & ✓ \\  & SOPLIB & – & – & – & ✓ \\ \hline \multirow{4}{*}{Algorithm} & Ant Colony System & – & – & ✓ & – \\  & Particle Swarm Optimization & – & – & ✓ \\  & Reinforcement Learning & ✓ & ✓ & – & – \\ \hline \multirow{4}{*}{Meta-learning} & Hyperparameter Tuning & – & ✓ & – & ✓ \\  & Transfer Learning & ✓ & – & – & – \\ \cline{1-1}  & AutoML & ✓ & – & – & – \\ \hline \hline \end{tabular}
\end{table} 
Table 11: Comparison of this proposal with different studies that applied algorithms for the sequential ordering problem’s solution. However, this paper did not aim to make a direct comparison with these conventional methods from the literature. The objective was to investigate and propose advances in reinforcement learning algorithms for solving combinatorial optimization problems. In this sense, it is noteworthy that, due to the best knowledge of these authors, this work is only the second paper that addresses reinforcement learning methods for solving the SOP. For this aspect, the work in [37] made important advances in the application of RL and hyperparameter tuning for the SOP. For example, the work in [37] assessed that, in general, the SARSA algorithm outperforms the Q-learning method in the search for better SOP solutions. Following this line, this paper continued to investigate the possibilities for improving RL techniques for combinatorial optimization problems.