# Multiple Instance Learning with Trainable Soft Decision Tree Ensembles
## 5 Soft Tree Ensemble for MIL
### Trees to Neural Networks  
Suppose that the RF is trained on the artificial dataset (12). Then, it can be converted to a neural network with a specific structure. A tree with \(M\) internal decision nodes and \(L\) leaves is represented as a neural network with the following three layers:  
1. The first layer aims to approximate the node predicates. It is a fully connected layer with \(m\) inputs (dimensionality of \(\mathbf{x}\)) and \(M\) outputs, i.e., it is held that: \[f^{(1)}(\mathbf{x})=\sigma(\mathbf{W}\mathbf{x}+\mathbf{b}\mid\omega),\] (15) where \(\mathbf{W}\in\mathbb{R}^{r\times m}\) is a matrix of non-trainable parameters consisting of \(r\) vectors \(\mathbf{w}_{i}\in\mathbb{R}^{m}\); \(r\) is the total number of the tree nodes; \(\mathbf{b}\in\mathbb{R}^{r}\) is the trainable bias vector; \(\omega\) is the trainable temperature parameter of the sigmoid \(\sigma\). As a result, the first layer has only trainable parameters \(\mathbf{b}\) and \(\omega\). The matrix \(\mathbf{W}\) consists of one-hot vectors having \(1\) at positions corresponding to the node features. 2. The second layer aims to estimate the leaf indices. It is fully connected layer with \(M\) inputs and \(L\) outputs having one trainable parameter \(\tau\): \[f^{(2)}(\xi)=\text{softmax}(\mathbf{R}\xi+\mathbf{s}\mid\tau),\] (16)where \(\mathbf{R}\in\mathbb{R}^{L\times M}\) is a non-trainable routing matrix that encodes decision paths, such that one path forms one row of \(\mathbf{R}\); \(\mathbf{R}_{\xi}^{\mathsf{T}}\in\mathbb{R}^{M}\) is the input vector; \(\mathbf{s}\in\mathbb{R}^{L}\) is the non-trainable bias vector; \(\tau\) is the trainable temperature parameter of the softmax operation. Matrix \(\mathbf{R}\) consists of values from the set: \(\{-1,0,1\}\). If the path to \(i\)-th leaf does not contain \(j\)-th node, then \(R_{i,j}=0\). Otherwise, if the path goes to the left branch, then \(R_{i,j}=-1\), and \(R_{i,j}=1\) if the path goes to the right branch. The vector \(\mathbf{s}=(s_{1},\ldots,s_{\texttt{2h}})\) is needed to balance the decision paths. The sum of the sigmoid functions from the path to the \(k\)-th leaf in (11) can be represented as:  
\[\sum_{d=1}^{h}\sigma\Big{(}[-\mathbf{w}_{I(d,l)}^{\mathsf{T}} \mathbf{x}+b_{I(d,k)}]\cdot s(d,k)\Big{)}\] \[=\sum_{i=1}^{M}\Bigl{(}R_{k,i}\sigma\Bigl{(}-\mathbf{w}_{I(d,l)}^ {\mathsf{T}}\mathbf{x}+b_{I(d,k)}\Bigr{)}+\mathbb{I}[R_{k,i}=-1]\Bigr{)}\] \[=\sum_{i=1}^{M}\Bigl{(}R_{k,i}\sigma\Bigl{(}-\mathbf{w}_{I(d,l)}^ {\mathsf{T}}\mathbf{x}+b_{I(d,k)}\Bigr{)}\Bigr{)}+s_{k}, \tag{17}\]  
because it holds that \(\sigma(-\omega)=1-\sigma(\omega)\). 3. The third layer aims to calculate the output values (embeddings). It is trainable and fully connected. Each leaf generates the class probability vector of the size \(C\). We take the probability \(v_{1}(\mathbf{x})\) of class \(1\) and repeat it \(E-1\) times, such that the whole embedding \(\mathbf{v}(\mathbf{x})=(v_{1}^{(1)}(\mathbf{x}),\ldots,v_{1}^{(E)}(\mathbf{x}))\) has the length \(E\). The final output of the network (or the third layer) is of the form \[f(\mathbf{x})=\mathbf{V}f^{(2)}\Bigl{(}f^{(1)}(\mathbf{x})\Bigr{)},\] (18) where \(\mathbf{V}\in\mathbb{R}^{E\times L}\) is a trainable leaf value matrix consisting of \(L\) vectors \(\mathbf{v}(\mathbf{x})\). An example of the transformation of a tree to a neural network is illustrated in Figure 1. A full decision tree with three decision nodes and four leaves was considered and is depicted in Figure 1. The first layer of the neural network computes all decisions at internal nodes of the tree. Matrix \(\mathbf{R}\) is constructed, such that each row represents a path to the corresponding leaf of the tree. For example, values of the first row are \((-1,-1,0)\) because the path to the leaf \(l_{1}\) passes through the nodes \(d_{1}\) and \(d_{2}\) to the left.