# Iterative Oblique Decision Trees Deliver Explainable RL Models
## 3 Methods
### DT Training Methods
#### 3.4.3 Iterative Training of Explainable RL Models (ITER)  
The DTs in [1] were trained solely based on samples from the oracle episodes. In the iterative algorithm, ITER, the tree with the best performance (from all iterations so far) is used as a generator for new observations.  
The procedure is shown in Algorithm 1 and can be described as follows: The initial tree is trained on samples of oracle evaluation episodes. We use these initial samples to fit the tree to the oracle's predictions and use the tree's current, somewhat imperfect decision-making to generate observations in areas the oracle maybe does not access anymore. Often, the points in the observation space that are essential for the formation of a successful tree lie in regions adjacent to the oracle trajectories, as will be shown in Section 4.1. The imperfect decision-making is demonstrated in the figures of Section 4.1 as well with the low return \(\overline{R}\) in early iterations. Generating samples next to the oracle trajectories with a not yet fully trained tree allows the next iteration's tree to fit its decisions to these previously unknown areas.  
```
0: Oracle policy \(\pi_{O}\), maximum iteration \(I_{max}\), number \(n_{b}\) of base samples, number of trees \(T_{max}\), evaluation episodes \(n_{eps}\), DT depth \(d\), number of samples added in each iteration \(n_{samp}\). \(\mathcal{O}\): observations, \(\mathcal{A}\): actions.
1:\(S_{c}\leftarrow(\mathcal{O}_{b},\mathcal{A}_{b})\)\(\triangleright\) collect \(n_{b}\) samples by running \(\pi_{O}\) in environment
2:\(\mathcal{M}\leftarrow\emptyset\)\(\triangleright\) the set \(\mathcal{M}\) collects triples for each tree
3:for\(i=0,\ldots,I_{max}\)do
4:for\(j=1,\ldots,T_{max}\)do
5:\(T_{i,j}\leftarrow\texttt{train\_tree}(S_{c},d)\)\(\triangleright\) tree \(T_{i,j}\)
6:\(\mathcal{M}_{j}\leftarrow\left(T_{i,j},\texttt{eval\_tree}(T_{i,j},n_{eps})\right)\)\(\triangleright\) triple \(\mathcal{M}_{j}\left(\texttt{eval\_tree}\) returns \(\mathcal{O}_{T_{i,j}},\mathcal{R}_{T_{i,j}}\right)\)
7:\(\mathcal{M}.\texttt{append}(\mathcal{M}_{j})\)
8:endfor
9:\(\left(T_{i,*},\mathcal{O}_{T_{i,*}},\mathcal{R}_{T_{i,*}}\right)\leftarrow \texttt{pick\_best}(\mathcal{M})\)\(\triangleright\) pick triple \(\mathcal{M}_{j}\) with highest reward \(\mathcal{R}_{T_{i,j}}\)
10:\(\widehat{\mathcal{O}}_{T_{i,*}}\leftarrow\texttt{choice}\Big{(}\mathcal{O}_{T_{i,* }},n_{samp}\Big{)}\)\(\triangleright\) pick \(n_{samp}\) random observations
11:\(S_{new}\leftarrow\Big{(}\widehat{\mathcal{O}}_{T_{i,*}},\pi_{O}(\widehat{ \mathcal{O}}_{T_{i,*}})\Big{)}\)
12:\(S_{c}\gets S_{c}\cup S_{new}\)
13:endfor
14:return\(T_{max,*}\)\(\triangleright\) return the best tree from all iterations
```  
**Algorithm 1** Iterative Training of Explainable RL Models (ITER)  
After each iteration, the generated observations from the so-far best tree are labeled with the oracle's predictions. The tree will likely revisit the region of these observations in the future, and therefore has to know the correct predictions at these points in the observation space. Finally, the selected samples are joined with the previous ones, and the next iteration begins. After a predetermined number of iterations, the overall best tree is returned. A flowchart of the iterative generation of samples for a given tree depth is shown in Figure 1.  
Figure 1: Flowchart of ITER (Algorithm 1).  
A possible variant is to include in line 11 of Algorithm 1 only samples for which the action of the tree \(\mathcal{A}_{T}\) and of the oracle differ; while this potentially reduces the training set for the DTs, both the oracle's and the DT's predictions are still required for comparison. Since our tests showed no noticeable difference in performance, we do not delve into this variant further.