# Transfer Reinforcement Learning for Combinatorial Optimization Problems
## 4 Results
### Results of Preliminary Analysis
#### 4.1.3 Results Visualization and Interpretation  
In this stage, the learning evolution graphs (distance calculated over the episodes) are analyzed for the following SOP problems: kro124p.1, kro124p.2, kro124p.3 and kro124p.4. These four problems are based on kro124p (ATSP), and the data between these instances vary depending on the number and arrangement of precedence constraints between the nodes (see Tables 1 and 5). \begin{table}
\begin{tabular}{l r r r r r} \hline \hline
**Problem** & **Q0** & **QATSP** & **D(\%)** & \(t\)** & \(p\)** \\ \hline br17.10 & 99.6 & 117.2 & 17.67 & \(-\)3.20 & 0.01 \\ br17.12 & 92.7 & 100.8 & 8.74 & \(-\)2.14 & 0.06 \\ ft53.1 & 19,054.3 & 10,003.9 & \(-\)47.50 & 126.10 & 0.00 \\ ft53.2 & 19,735.9 & 12,057.4 & \(-\)38.91 & 138.95 & 0.00 \\ ft53.3 & 19,583.6 & 16,173.2 & \(-\)17.41 & 25.40 & 0.00 \\ ft53.4 & 19,360.0 & 18,245.4 & \(-\)5.76 & 26.56 & 0.00 \\ kro124p.1 & 179,266.4 & 56,146.9 & \(-\)68.68 & 789.13 & 0.00 \\ kro124p.2 & 179,859.2 & 59,280.3 & \(-\)67.04 & 432.24 & 0.00 \\ kro124p.3 & 168,223.3 & 71,605.7 & \(-\)57.43 & 221.92 & 0.00 \\ kro124p.4 & 124,900.0 & 99,131.0 & \(-\)20.63 & 67.09 & 0.00 \\ p43.1 & 72,411.4 & 30,453.1 & \(-\)57.94 & 117.94 & 0.00 \\ p43.2 & 71,953.6 & 32,726.0 & \(-\)54.52 & 101.90 & 0.00 \\ p43.3 & 67,146.8 & 33,151.5 & \(-\)50.63 & 84.45 & 0.00 \\ p43.4 & 93,488.1 & 86,304.4 & \(-\)7.68 & 25.96 & 0.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Average solution (distance) over the course of learning to solve the SOP, percentage difference (D) between the results of Q0 and the QATSP and results of the \(t\) test. There was a significant difference between the mean solutions of Q0 and the QATSP if \(p\leq 0.05\). \begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Problem** & **Q0** & **QATSP** & **D(\%)** & \(t\)** & \(p\)** \\ \hline br17.10 & 0.33 & 0.22 & \(-\)33.33 & 3.24 & 0.01 \\ br17.12 & 0.30 & 0.22 & \(-\)26.67 & 9.62 & 0.00 \\ ft53.1 & 0.65 & 0.59 & \(-\)9.23 & 2.87 & 0.01 \\ ft53.2 & 0.84 & 0.66 & \(-\)21.43 & 9.95 & 0.00 \\ ft53.3 & 1.79 & 0.89 & \(-\)50.28 & 60.65 & 0.00 \\ ft53.4 & 2.62 & 1.27 & \(-\)51.53 & 73.37 & 0.00 \\ kro124p.1 & 1.63 & 1.15 & \(-\)29.45 & 10.42 & 0.00 \\ kro124p.2 & 2.12 & 1.12 & \(-\)47.17 & 73.63 & 0.00 \\ kro124p.3 & 3.36 & 1.57 & \(-\)53.27 & 57.10 & 0.00 \\ kro124p.4 & 7.61 & 2.88 & \(-\)62.16 & 158.77 & 0.00 \\ p43.1 & 0.55 & 0.59 & \(-\)7.27 & \(-\)1.49 & 0.15 \\ p43.2 & 0.79 & 0.72 & \(-\)8.86 & 4.53 & 0.00 \\ p43.3 & 1.13 & 1.08 & \(-\)4.42 & 2.73 & 0.01 \\ p43.4 & 1.84 & 1.51 & \(-\)17.93 & 27.04 & 0.00 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Average computational times (in seconds) for solving the SOP, percentage difference (D) between the results of Q0 and the QATSP and the results of the \(t\) test. There was a significant difference between the mean computational times for Q0 and the QATSP if \(p\leq 0.05\). Figures 4-7 show that the QATSP learning curves started the episodes closer to the optimal value of the instance than the simulations with Q0. This is most evident when comparing the \(d_{0}\) values. In all the graphs shown, the difference between the solution in the initial episode of the QATSP and Q0 was greater than \(0.8\times 10^{5}\) (80,000) distance units. This highlights the importance of transferring knowledge between the ATSP and SOP domains in the initial episodes, favoring the acceleration of learning. In addition, the graphs in Figures 4-6 also show the differences between the \(d_{f}\) values for Q0 and the QATSP. For instances kro124p.1, kro124p.2 and kro124p.3, adopting transfer learning resulted in better results in the last episode of the simulation. In the worst case (experiments on problem kro124p.4), the Q0 learning curve still needed around 50 episodes to achieve results close to those presented by the simulations using the ATSP knowledge base. Finally, it is worth highlighting that for the graphs presented in Figures 4-7, employing transfer learning between the ATSP and SOP domains resulted in calculating better solutions during learning.