# Transfer Reinforcement Learning for Combinatorial Optimization Problems  
[PERSON]  
1 Technologic and Exact Center, Federal University of Reconcavo da Bahia, R. Rui Barbosa, Cruz das Almas 44380-000, Bahia, Brazil; [EMAIL_ADDRESS] (G.K.B.S. ); [EMAIL_ADDRESS] (A.L.C.O.) [PERSON]  
2 Hamilton Institute, Maynooth University, W23 VP22 Maynooth, Co. Kildare, Ireland; [EMAIL_ADDRESS]  
[PERSON]  
1 Technologic and Exact Center, Federal University of Reconcavo da Bahia, R. Rui Barbosa, Cruz das Almas 44380-000, Bahia, Brazil; [EMAIL_ADDRESS] (G.K.B.S. ); [EMAIL_ADDRESS] (A.L.C.O.) [PERSON]  
3 Department of Mathematics and Statistics, Federal University of Sao Joole del-Rei, Praca Frei Orlando, Sao Joole del Rei 36309-034, Minas Gerais, Brazil; [EMAIL_ADDRESS] (M.S.O. ); [EMAIL_ADDRESS] (D.C.R.O.) [PERSON]  
to transfer knowledge between tasks in different domains and provide various benefits, such as improving the performance of the agent in the target task, improving the agent's total reward and reducing the time needed to carry out the learning [17; 18]. Along these lines, transfer reinforcement learning techniques have been applied in important domains, especially in the areas of robotics [19; 20; 21; 22; 23; 24] and multiagent systems [25; 26; 27]. However, the literature has paid little attention to the transfer reinforcement learning for combinatorial optimization problems. Indeed, the field of combinatorial optimization has several studies using RL [6; 28; 29; 30; 31; 32; 33; 34]. Some domains with applications of RL techniques are the traveling salesman problem (TSP) [29; 31], k-server problem [32; 34], multidimensional knapsack problem [35], vehicle routing [36] and sequential ordering problem (SOP) [37]. The TSP [29; 31; 33] and the SOP [38; 39; 40; 41] are problems that have great relevance in the literature, which has led to several studies on them. The TSP is a problem with numerous relevant applications, and as a result, several studies have been carried out to solve it [42; 43; 44; 45]. The SOP, like the TSP, also has several important studies, including energy optimization in compilers [46], search optimization [39; 47] and parallel machine scaling [48]. Moreover, the TSP and SOP are NP-hard combinatorial optimization problems; that is, in practice, it is necessary to adopt approximate algorithms in the search for better solutions [39]. However, the literature still lacks studies that address tranfer RL in these two problems. Therefore, the goal of this work is to propose and analyze transfer reinforcement learning between these two relevant combinatorial optimization domains: the ATSP and SOP. For this purpose, instances of the TSPLIB library [49; 50; 51] and the state-action-reward-state-action (SARSA) algorithm [4] were used. The approach proposed in this work enables generation of a knowledge base in the source domain (ATSP) to transfer learning to the target domain (SOP). In addition to the extensive variety of studies on the ATSP and SOP, these domains were selected due to the similarities in the characteristics of these two combinatorial optimization problems. In this sense, in this paper, three criteria are proposed to be evaluated when carrying out transfer reinforcement learning across combinatorial optimization problems: (1) problems with the same objective function; (2) similar datasets; and (3) transfer from the simpler domain to the more complex domain. The main contributions include the following:  
* A transfer reinforcement learning approach between two classical combinatorial optimization problems. The asymmetric traveling salesman problem is the source domain, and the sequential ordering problem is the objective domain. * Apply transfer learning to these problems and statistically analyze the results obtained with the transfer. * Develop a new AutoRL algorithm, apply it to the problems studied and analyze its results. The present paper is organized into six sections.