# Transfer Reinforcement Learning for Combinatorial Optimization Problems
## 3 Methodology
### Automated Transfer Reinforcement Learning Method
#### 3.4.1 Automated Transfer Reinforcement Learning Algorithm (Auto_TL_RL)  
The flowchart depicted in Figure 3 illustrates the proposed automated transfer reinforcement learning algorithm flow, which has been named Auto_TL_RL. Initially, an SOP instance is selected, and if a knowledge base already exists for the selected SOP instance, then the problem is executed directly. Otherwise, the knowledge base is generated from an ATSP instance, this database is then stored, and finally the SOP is executed. Algorithm 3 represents the proposed Auto_TL_RL algorithm procedurally, which was developed using the R language. It is worth noting that the proposed algorithm uses the SARSA algorithm to perform the ATSP and SOP. ```
1 Specify the SOP instance to be executed
2 Extract the size of the SOP instance
3 Set the parameter: \(\epsilon\)
4 Set the number of episodes: 1000
5 Checking the existence of the knowledge base
6 Stage 2
7 Specify the ATSP instance that will be executed
8 Extract the size of the ATSP instance
9 Set the parameters: \(\alpha\) and \(\gamma\)
10 Set the number of epochs: 5
11 foreach\(\alpha_{t}\in\alpha\)do
12 foreach\(\gamma_{t}\in\gamma\)do
13 foreoch to numberEpochsdo
14 SARSA(\(\alpha\), \(\gamma\), sizeOfInstance)
15 end if
16
17 end for
18
19 end for
20
21 Stores the database generated
22
23 Stage 3
24 Set the parameters: \(\alpha\) and \(\gamma\)
25 Set the number of epochs: 10
26 for epoch to numberEpochsdo
27 SARSA(\(\alpha\), \(\gamma\), sizeOfInstance)
28 end for
```  
**Algorithm 3**Auto_TL_RL Algorithm  
Initially, the user must select the instance of the SOP to execute (line 1), and from this, the size of the instance is extracted (line 2). Next, the value of the \(\epsilon\)-greedy parameter and the number of episodes that will be executed during learning must be defined (lines 3 and 4). Next, the execution of the algorithm is divided into three stages. The first stage checks for the existence of the database related to the selected problem (line 5). To check for the existence of the knowledge base, a document is used that records the name of the instance, the dimension, the status of the knowledge base (whether it exists or not) and the name of the file that stores the knowledge base corresponding to the instance analyzed. The second path is taken if there is no pre-existing knowledge base for the selected SOP instance. It defines an ATSP instance and extracts its dimension (lines 6 and 7). Furthermore, the values for the learning rate, the discount factor and the number of epochs are defined (lines 8 and 9). In this sense, with the learning rate and discount factor values defined, different parameter combinations are performed. For each simulated parameter combination, the selected ATSP instance is executed for the defined number of epochs (lines 10-16). At the end of the execution, the knowledge base that resulted in the shortest route distance in the ATSP instance used is stored among all generated databases (line 17), allowing it to be reused in future applications. Finally, in the third step, the SOP is executed (lines 18-22) using the knowledge base generated in the previous step. The second path is adopted when there is no pre-existing knowledge base for the selected SOP instance. Thus, an ATSP instance is defined, and its dimension is extracted (lines 6 and 7). Furthermore, the values for the learning rate, discount factor and number of epochs are defined (lines 8 and 9). In this sense, with the learning rate and discount factor values established, various combinations of parameters are performed. For each simulated parameter combination, the selected ATSP instance is executed for the defined number of epochs (lines 10-16). At the end of the execution, the knowledge base that resulted in the shortest route distance in the ATSP instance used is stored among all generated databases (line 17), allowing it to be reused in future applications. Finally, the third stage is where the SOP is executed (lines 18-22) using the knowledge base generated in the previous step.