# Feature Interaction in Terms of Prediction Performance
## 3 Results  
The feature interaction function described in the previous section was implemented in the R programming environment ([[https://www.r-project.org](https://www.r-project.org)]([https://www.r-project.org](https://www.r-project.org))). The CARET package [20] was used to build the prediction model. The feature permutation was repeated 10 times, and the average of the reduced performance of the \(Err()\) function was computed to accurately measure the feature permutation. Only the interaction between two features was measured for simplicity. Feature interaction was demonstrated in two sample datasets. The PimaIndiansDiabetes [21] dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases and is used to predict whether or not a patient has diabetes. It has nine features and 768 observations and is used for the classification task. The features in the PimaIndiansDiabetes dataset are described in Table 1.  
The BostonHousing [22] dataset has housing data for 506 census tracts of Boston from the 1970 census. It is used for predicting the price of houses (mdev in the dataset) and is essentially a regression task. The dataset has 14 features and 506 observations. Features in the BostonHousing dataset are illustrated in Table 2.  
The C5.0 prediction model was constructed using the PimaIndiansDiabetes dataset to test the classification, and the entire dataset was used for model fitting. The training accuracy was 0.8112. Figure 4 shows the feature interaction for various combinations of any two features in the dataset. In the interaction table, the (\(i\)-th, \(j\)-th) cell shows the feature interaction between features \(F_{i}\) and \(F_{j}\). The reduced prediction performance due to feature \(F_{k}\) as calculated by the \(Err(F_{k})\) is shown in the cells along the (gray) diagonal (\(k\)-th, \(k\)-th). Figure 4 shows that \"glucose\" is the most effective feature for prediction as the \(Err(\text{glucose})=0.1503\), which is the largest value of the \(Err()\) function. It means that if \"glucose\" is removed from the prediction model, the model performance (prediction accuracy) decreases by 0.1503. The feature \"mass\" offers the best interaction with \"glucose\" (_Internact_(glucose, mass) = 0.0301) as the prediction error decreases by 0.0301. The features \"age\", \"insulin\", and \"pressure\" have an \(Err()=0\) and \(Interact()=0\), which indicates that these features are not effective predictors of diabetes. The prediction model when rebuilt excluding the above three features resulted in an accuracy of approximately 0.80, which is approximately equal to that obtained with the original full dataset. Figure 5 shows the feature interaction between \"glucose\" and other features in the dataset. The feature \"mass\" is most interactive and \"pregnant\" is second. \"Pressure\", \"insulin\", and \"age\" have no interaction with \"glucose\". The three features do not co-work with \"glucose\" to increase or decrease the prediction performance. A linear regression model using the BostonHousing dataset was built to evaluate regression. The complete BostonHousing dataset has an RMSE of 4.6792. Figure 6 shows the interaction table where it can be observed that \"lstat\" (percentage of lower status of the population) is the most useful feature for predicting house price (\(Err(\text{lstat})=2.423\)). The features \"b\", \"crim\", \"dis\", \"rad\", and \"rm\" are important for prediction, whereas \"age\" and \"indus\" are less influential. The regression model also displays negative interaction. Figure 7 shows that the \"dis\" (weighted distances to five Boston employment centers) and \"rad\" (index of accessibility to radial highways) features interact negatively with the \"lstat\". This suggests that \"dis\" and \"rad\" decrease the prediction performance of \"lstat\".