## 3 Proposed Method
### Simulation with Artificial Examples
#### 3.5.2 Artificial Example II: Four Modules with Discrete Variables
This is discussed in Section 4.2 and please review the architecture in Figure 5 for a detailed description. Though not as high as \(\theta_{c}(S_{1})=75\%\), it is better than random guessing. Similarly, we can compute the theoretical prediction rate for the remaining modules as \(\theta_{c}(S_{3})=\theta_{c}(S_{4})=12.5\%+(1-12.5\%)*50\%=56.25\%\), which is only slightly higher than random guessing. All together, we should expect \(\theta_{c}(\{S_{1},S_{2},S_{3},S_{4}\})=\max(0.75,0.625,0.5625)=75\%\). In other words, the derivation for the theoretical prediction of each variable module in the underlying Model (12) helps us to understand that with correct model specification, we should expect simulation results to be approximately 75% on average. **Scenario II. ** In practice, we assume we do not have the knowledge of the underlying model. To illustrate the performance of the I-score as a feature selection methodology, we conducted the following simulation: We created data with 49 variables drawn from Bernoulli(\(1/2\)) random variables independently. We allowed the number of in-sample training size to be \(\{50,100,1000\}\). For each value of the training sample size, we conducted experiments using machine learning algorithms such as bagging (We used the statistical package called ipred: Improved Predictors. Source: [[https://cran.r-project.org/web/packages/ipred/index.html](https://cran.r-project.org/web/packages/ipred/index.html)]([https://cran.r-project.org/web/packages/ipred/index.html](https://cran.r-project.org/web/packages/ipred/index.html)) (2 November 2021)), logistic, random forest (RF) (We used the statistical package called randomForest. Source: [[https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest](https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest)]([https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest](https://www.rdocumentation.org/packages/randomForest/versions/4.6-14/topics/randomForest)) (2 November 2021)), iterative random forest (iRF) (We use the statistical package irf. Source: [[https://www.rdocumentation.org/packages/vars/versions/1.5-3/topics/irf](https://www.rdocumentation.org/packages/vars/versions/1.5-3/topics/irf)]([https://www.rdocumentation.org/packages/vars/versions/1.5-3/topics/irf](https://www.rdocumentation.org/packages/vars/versions/1.5-3/topics/irf)) (2 November 2021)), and neural network (NN) (We used the Keras package and produced a neural network with no hidden layer and one output neuron. Source: [[https://github.com/yiqiao-yin/YinsLibrary_/blob/9822f36](https://github.com/yiqiao-yin/YinsLibrary_/blob/9822f36) ca097b1e19f7b669e4f42 ca39 ea9 aa608/r/KerasNN.R#L50-L57]([https://github.com/yiqiao-yin/YinsLibrary_/blob/9822f36](https://github.com/yiqiao-yin/YinsLibrary_/blob/9822f36) ca097b1e19f7b669e4f42 ca39 ea9 aa608/r/KerasNN.R#L50-L57) (2 November 2021)). We used default values from the packages for the first four algorithms. For the NN, we used the variables modules (i.e., \(X^{\dagger}\)) as the input layer and the architecture takes the form shown in Figure 5, which states the classification rules using a forward propagation structure (as shown in Section 4.2, Figure 5, there is a linear transformation using weights \(\vec{w}\) and a nonlinear transformation using an activation function \(a(\cdot)\)). The performance was measured using out-of-sample test set with 1000 data points. The metric for performance is area under the curve (AUC) from the receiver operating characteristic (ROC) curve (see Section 4.4 for a detailed discussion of AUC values). An AUC value of 50% indicates that the area under the curve mapped from a list of precision and recall rates has no prediction power of the ground truth. An AUC value of 100% indicates that the predictor can perfectly guess the ground truth. For each machine learning algorithm with each value in the training size, we conducted experiments using all variables as the benchmark.