# Transfer Learning for Operator Selection: A Reinforcement Learning Approach
## 3 Proposed Approach for Transfer Learning with RL
```
1:Initial Phase
2:if learning is not activated or first run then
3: Initial credit and C cluster values
4:endif
5:Operator Selection
6: Assign probabilities
7: Choose operator using Roulette-Wheel selection
8:Operator Evaluation
9: Execute operator and get reward
10:if positive reward and learning is activated then
11: Update cluster of operator
12: Update operator total reward
13:endif
14:At the end of iteration
15:if learning is activated then
16: Update credit values using Equation (5)
17:endif
```  
**Algorithm 1** General overview of RL-AOS  
We present below the above discussed concepts more formally. Let \(\mathcal{X}\) be a population of solutions that makes up a bee colony handled by the ABC implemented in this study, where \(\mathcal{X}=\{x_{i}|i=1\ldots|\mathcal{X}|\}\). Each solution \(x_{i}\) is defined as a \(D\) dimensional binary set, \(x_{i}=\{x_{i,j}\in[0,1]|j=1\ldots D\}\), and its quality of solution is measured with \(F(x_{i})\). The ABC is implemented uses a pool of actions \(\mathcal{A}=\{a_{k}|k=1\ldots|\mathcal{A}|\}\), where each \(a_{k}\) is a function defined as \(a_{k}(x_{i})\xrightarrow[]{x_{i}}y_{i}\), where \(y_{i}\in\mathcal{X}\). Meanwhile, a set of clusters, \(\mathcal{C}=\{c_{a}|a=1\ldots|\mathcal{A}|\}\), is defined to represent the set of actions, \(\mathcal{A}\), where each cluster center is the centroid measure of the \(D\) dimensional dataset, \(c_{a}=\{c_{a,j}|j=1\ldots|D|\}\), and calculated using the following equation:  
\[c_{a,j}=\frac{\sum_{i=0}^{t}x_{i,j}b_{a,i}}{\sum_{i=0}^{t}b_{a,i}}\quad\forall j \in D \tag{4}\]  
where \(t\) is the number of iterations done so far, \(b_{a,i}\in[0,1]\) is a binary value indicating if the action is successful, (i.e., if the operator \(a\) helped produce a better fitness), where it take value 1 if successful and 0 otherwise. The centroids are optimised online with Q learning algorithm collecting the rewards, \(r_{a,i}\), based on the fitness values, \(F(x_{i})\) as detailed in [4]. All \(c_{a}\) values are initialised with 0, while random values are allocated to \(Q(\mathbf{x},a)\). Earlier iterations impose a random selection of operators, \(a\in\mathcal{A}\), whereas subsequent stages enforce the selections through fine-tuned \(Q\) values throughout the experience-gaining process. The \(Q(x_{i},a)\) values are updated immediately after an action is taken, (i.e., operator \(a\) is chosen and applied to \(x_{i}\)) using the following rule:  
\[Q(x_{i},a)=Q(x_{i},a)+\beta(r_{a,i}+\gamma(E(y_{i})-Q(x_{i},a)) \tag{5}\]  
where \(\beta\) is the learning coefficient, \(\gamma\) is the discounting factor, and \(E(y_{i})\) is the expected \(Q\) value for the new problem's state, i.e., a solution. The expected value is calculated with \(d=|x_{i}-c_{a}|\) as the Euclidean distance between \(x_{i}\) and \(c_{a}\) as the current solution and the centroid for operator \(a\), respectively. The algorithm runs repeating this procedure until a stopping criterion is satisfied. More opportunities to experience would be required to build a wide range of experience across the search space, necessitating the adoption of an exploration policy alongside the exploitation of learned cases. This study employs a \(\epsilon\)-greedy policy to accomplish this goal. It requires randomising a value and performing a random selection if the random number is less than a \(\epsilon\) threshold and \(Q\)-values-based selection otherwise. More details can be found in [4; 24]. The transfer learning can be adopted into a data model \(\mathcal{M}(D)=\{m_{i}(d)|i=0\ldots|\mathcal{M}|\}\), where each component of the model comprises \(m_{i}(d)=\alpha_{i}(d)\bigoplus\beta_{i}(d)\). The model combines the terms \(\alpha\) and \(\beta\) with the implementation of a union operator represented by \(\bigoplus\), where \(\alpha\) and \(\beta\) represent the experience gained previously and the experience to be gained over upcoming attempts, respectively.