# Nonlinear Random Forest Classification, a Copula-Based Approach
## 3 Copula-Based Random Forest  
The connection between mutual information and copula function has been investigated in the literature. We can also represent the conditional mutual information via the copula function through the following proposition:  
If the random vector \((\mathbf{X},\mathbf{Y},\mathbf{Z})\) is associated with copula \(C_{X,Y,Z}(\mathbf{u},\mathbf{v},\mathbf{w})\), then Equation (7) is  
\[I(\mathbf{X},\mathbf{Y}|\mathbf{Z})=h_{c}(\mathbf{X},\mathbf{Z})+h_{c}(\mathbf{Y},\mathbf{Z})-h_{c}(\mathbf{X},\mathbf{Y},\mathbf{Z})-h_{c}(\mathbf{Z}) \tag{8}\]  
**Proof of Proposition 1. ** By an appropriate equivalent modification of the argument of the log function in the integrand of (7), we readily obtain:  
\[I(\mathbf{X},\mathbf{Y}|\mathbf{Z})=\oint_{\mathcal{Z}}\oint_{\mathcal{Y}}\oint_{\mathcal{ X}}p_{X,Y,Z}(\mathbf{x},\mathbf{y},\mathbf{z})\log\frac{p_{Z}(\mathbf{z})p_{X,Y,Z}(\mathbf{x},\mathbf{y}, \mathbf{z})}{p_{X,Z}(\mathbf{x},\mathbf{z})p_{Y,Z}(\mathbf{y},\mathbf{z})}dxdyd\mathbf{z}\]  
\[=-h(\mathbf{X},\mathbf{Z})-h(\mathbf{Y},\mathbf{Z})+h(\mathbf{X},\mathbf{Y},\mathbf{Z})+h(\mathbf{Z})\]  
\[=h_{c}(\mathbf{X},\mathbf{Z})+h_{c}(\mathbf{Y},\mathbf{Z})-h_{c}(\mathbf{X},\mathbf{Y},\mathbf{Z})-h_{c}( \mathbf{Z}).\]  
The last equality comes from one of the results of [PERSON] and [PERSON] (2011), which proves that for \(\mathbf{X}=(X_{1},X_{2},\ldots,X_{n})\),  
\[h(\mathbf{X})=\sum_{i=1}^{n}\,h(\mathbf{X}_{i})+h_{c}(\mathbf{X}).\]  
In order to use the mutual information in the decision trees, assume we have a dataset \(\mathbf{D}=\big{\{}(\mathbf{x}_{1},y_{1}),(\mathbf{x}_{2},y_{2}),\ldots,(\mathbf{x}_{n},y_{n} )\big{\}}\), where \(\mathbf{x}_{i}\in\mathcal{X}\) is the \(i\)-th input or observation and \(y_{i}\in\mathcal{Y}\) is the corresponding outcome variable. In a machine learning approach, the major goal is constructing (or finding) a classification map \(f:\mathcal{X}\rightarrow\mathcal{Y}\) which takes the features \(\mathbf{x}\in\mathcal{X}\) of a data point as its input and outputs a predicted label. The special case of the outcome variable is a class label \(y_{i}\in\{-1,1\}\); i.e., it has two possible values, such as: negative/positive, pathogenic/benign, patient/normal, etc. The general objective function that must be maximized is:  
\[J_{CMI}(\mathbf{X}_{k})=I(\mathbf{X}_{k},Y)-\beta\sum_{i=1}^{n}I(\mathbf{X}_{i},\mathbf{X}_{k })-\gamma\sum_{i=1}^{n}I\big{(}\mathbf{X}_{i},\mathbf{X}_{j}\big{|}\mathbf{Y}\big{)} \tag{9}\]  
where \(I(\mathbf{X}_{k},Y)\) measures the relation between term \(\mathbf{X}_{k}\) and target variable \(Y\), \(I(\mathbf{X}_{i},\mathbf{X}_{k})\) quantifies the redundancy between \(\mathbf{X}_{i}\) and \(\mathbf{X}_{k}\); while \(I(\mathbf{X}_{i},\mathbf{X}_{j}\big{|}Y)\) measures the complementarity between terms \(\mathbf{X}_{i}\) and \(\mathbf{X}\).\(\square\)  
Similar to Proposition 1, one may state the Equation (9) based on copula as:  
\[J_{CMI}(\mathbf{X}_{k})=h_{c}(\mathbf{X}_{k},Y)+\beta\sum_{i=1}^{n}\,h_{c}(\mathbf{X}_{i},\mathbf{X}_{k})\]  
\[-\gamma\sum_{i=1}^{n}\,[h_{c}(\mathbf{X}_{i},Y)+h_{c}(\mathbf{X}_{k},Y)-h_{c}(\mathbf{X}_ {i},\mathbf{X}_{k},Y)-h_{c}(Y)]\]  
\[=(1-\gamma(n+1))h_{c}(\mathbf{X}_{k},Y)+n\gamma h(Y)\]  
\[+\beta\sum_{i=1}^{n}\,h_{c}(\mathbf{X}_{i},\mathbf{X}_{k})-\gamma\sum_{k\
eq i=1}^{n} \,h_{c}(\mathbf{X}_{i},Y)+\gamma\sum_{i=1}^{n}\,h_{c}(\mathbf{X}_{i},\mathbf{X}_{k},Y),\]  
where the first term of the last part of equality refers to the relevancy of the new feature \(\mathbf{X}_{k}\). [PERSON] et al. (2019) [17] introduced the \"Minimum Redundancy Maximum Relevance (mRMR)\" criterion to set the value of \(\beta\) to be the reverse of the number of selected featuresand \(\gamma=0\). We generalize their results by simplifying \(J_{CMI}(\mathbf{X}_{k})\).