## 2 Materials and Methods
### Interpretability and Feature Importance
#### 2.7.1 Model Interpretability
### Prediction Results with Recursive Feature Elimination (RFE)

The utility of each feature can be determined by an automatic feature selection method like RFE. By using this method, the most beneficial features are chosen from the original features. RFE is able to build several models for various data subsets and, most importantly, to exclude useless features. This experiment was deployed in the R language environment. RFE was combined with rfFuncs, i.e., random forest, to select necessary features. A popular data modeling technique for calculating prediction error while effectively avoiding the overfitting condition is cross-validation. The most popular is K-fold cross-validation (K-fold CV), which divides the entire sample set into K divisions, selects one of them as the validation set each time, and the remaining K-1 division as the training set. The next step is to calculate the error of the model on validation sets and average K errors after K cycles, as the estimation of the prediction error. In this experiment, 10-fold cross-validation was employed. Table 10 below shows the selected features by RFE. ### Prediction Results with Explainable AI (XAI) and AutoML

The XAI method we used is SHAP. In this experiment, the SHAP analysis was divided into three main steps: model training, model interpretation, and analysis of the entire dataset. To begin model training, we read the dataset and used the train/test split method to divide it into 80% training data and 20% test data. Next, we trained an XGBoost classifier using the dataset. In the second step, we interpreted the model by creating an interpreter on top of the XGBoost model to calculate SHAP values. Once the first two steps were completed, we used the summary plot method to interpret the entire dataset. This method identifies the top 20 most important features based on their significance. Finally, we implemented AutoML using the twenty most important features. Figure 7 below shows the workflow of how we used SHAP to extract important features. ### AutoML

We implemented the AutoML method using H\({}_{2}\)O. This framework automatically selected 17 AutoML models to train our datasets, including GBM-grid-1-AutoML, GBM-5-AutoML, StackedEnsemble-BestOfFamily-AutoML, XGBoost-3-AutoML, GLM-1-AutoML, DRF-1-AutoML, etc. In the first step of our experiment, we used the SHAP method to identify the 20 most significant features, as mentioned in the above section. We then used an iterative elimination method to select the top N most important features and trained them with AutoML models. We repeated this process with N-1 most important features until we selected the most vital features that have relatively high accuracy. The results are shown in Table 11 below. Our experimental results show that the accuracy achieved for project A was 84.16%. This was reached by picking the top three most important features, i.e., Id, Grado, \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Project A** & **Project B** & **Project C** & **Project D** & **Project E** & **Project F** \\ \hline Number of & & & & & & \\ variables chosen & 5 & 4 & 2 & 5 & 3 & 2 \\ \hline \hline \end{tabular} Project A selects Id, Modularity Class, CL English, CL German and EG French. Project B selects Id, Modularity Class, intensity.1.PR, and intensity.2.PR. Project C selects intensity.5.PR and intensity.2.PR. Project D selects intensity.5.PR, Location, intensity.1.PR, Gender female and intensity.3.PR. Project E selects intensity.5.PR, intensity.6.PR, and Tenure. Project F selects Tenure and intensity.3.PR.

\end{table} 
Table 10: Important features selected by the method Recursive Feature Elimination (RFE) in real-world projects A and B at an IT company, projects C and D at a global consulting company, and projects E and F at an international manufacturing company. and Communication-Language-English. The corresponding AutoML model used was GBM-grid-1-AutoML. The highest accuracy achieved for project F was 91.41%.