## Appendix A  
### HQL  
Below is an example for of the HQL method on four DTs. Each learning episode starts with an empty random forest \(RF=[]\). We assume that \(\alpha=0.9\), \(\gamma=0.1\). The initial Q-Matrix is a \(4\times 4\) matrix of 0's. The rows represent states, the columns--actions. The initial state for the episode is _state_1_ which means DT 1 is added to the random forest. **Step 0**: Initial State: state_1, Hidden State RF = [1], previous validation accuracy is \(valid\_acc_prev=68\). \begin{tabular}{|l|c|c|c|} \hline add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 0 \\ \hline state\_3 & 0 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 1**: Action Taken: add_2; Hidden State RF = [1, 2]; current validation accuracy \(valid\_acc_curr=70\); \(Reward=valid\_curr-valid_prev=70-68=2\); New State: state_2;\[Q(1,2)=(1-0.9)\times 0+0.9\times(2+0.1\times 0)=1.8.\]  
The updated Q-value matrix is as follows. \begin{tabular}{|c|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 1.8 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 0 & 0 \\ \hline state\_3 & 0 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 2**: Action Taken: add\_3; Hidden State RF = [1, 2, 3]; \(valid\_acc_curr=72\); \(Reward=72-70=2\); New State: state\_3;  
\[Q(2,3)=(1-0.9)\times 0+0.9\times(2+0.1\times 0)=1.8.\]  
The updated Q-value matrix is as follows. \begin{tabular}{|c|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 1.8 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 1.8 & 0 \\ \hline state\_3 & 0 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 3**: Action Taken: add\_1; Hidden State RF = [1, 2, 3, 1], \(valid\_acc_curr=71\), \(Reward=71-72=-1\), New State: state\_1;  
\[Q(3,1)=(1-0.9)\times 0+0.9\times(-1+0.1\times 0)=-0.9.\]  
The updated Q-value matrix is as follows. \begin{tabular}{|c|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 1.8 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 1.8 & 0 \\ \hline state\_3 & -0.9 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 4**: Action Taken: add\_2; Hidden State RF = [1, 2, 3, 1, 2]; \(valid\_acc_curr=72\); \(Reward=72-71=1\); New State: state\_2;  
\[Q(1,2)=(1-0.9)\times 1.8+0.9\times(1+0.1\times 1.8)=1.242.\]  
The updated Q-value matrix is as follows. \begin{tabular}{|c|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 1.242 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 1.8 & 0 \\ \hline state\_3 & 0.9 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 4**: Action Taken: add\_2; Hidden State RF = [1, 2, 3, 1, 2]; \(valid\_acc_curr=72\); \(Reward=72-71=1\); New State: state\_2;  
\[Q(1,2)=(1-0.9)\times 1.8+0.9\times(1+0.1\times 1.8)=1.242.\]  
The updated Q-value matrix is as follows. \begin{tabular}{|c|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 1.242 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 1.8 & 0 \\ \hline state\_3 & 0.9 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 2**: Action Taken: add\_3; Hidden State RF = [1, 2, 3]; \(valid\_acc_curr=72\); \(Reward=72-70=2\); New State: state\_3;  
\[Q(2,3)=(1-0.9)\times 0+0.9\times(2+0.1\times 0)=1.8.\]  
The updated Q-value matrix is as follows. \begin{tabular}{|c|c|c|c|c|} \cline{2-5} \multicolumn{1}{c|}{} & add\_1 & add\_2 & add\_3 & add\_4 \\ \hline state\_1 & 0 & 1.8 & 0 & 0 \\ \hline state\_2 & 0 & 0 & 0 & 0 \\ \hline state\_3 & 0.9 & 0 & 0 & 0 \\ \hline state\_4 & 0 & 0 & 0 & 0 \\ \hline \end{tabular}   
**Step 3**: Action Taken: add\_1; Hidden State RF = [1, 2, 3, 1], \(valid\_acc_curr=71\), \(Reward=71-72=-1\), New State: state\_1;  
\[Q(3,1)=(1-0.9)\times 0+0.9\times(-1+0.1\times 0)=-0.9.\]  
The updated Q-value matrix is as follows.