# A Study on Dropout Prediction for University Students Using Machine Learning
## 3 Proposed Method
### Model Implementation  
As an algorithm to build a model for the dropout prediction, various machine learning algorithms were used and compared, including Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), Deep Neural Network (DNN), and Light Gradient Boosting Machine (LightGBM). For the LR, DT, RF, SVM, and LightGBM models, Scikit-learn [36] was used, which is a machine learning library in Python. For the DNN models, Keras [37] was used, which is an open source software library for artificial neural networks provided by Google. The summary table shown in Table 6 is used as input data for training the model. The table is input in the form of a CSV file and converted to a data frame in Python. From the data frame, the X and T lists are extracted as follows, which denote the list of input records and the list of values indicating whether each input record results in dropout or not, respectively. Since there is a derivation between the values of X, its values are normalized to values between 0 and 1 using Scikit-learn's StandardScaler. Then, X and T are split into the training and test data sets using the train_test_split() function. In the code below, X_train and T_train denote the training data sets, while X_test and T_test denote the test data sets. \begin{tabular}{l} \hline \hline X = df[[\"AdmType\",\"NumSem\",\"Grade\",...]].values \\ T = df[\"Dropout\"].values \\ X\_scaled = StandardScaler().fit_transform(X) \\ X\_train, X\_test, T\_train, T\_test = train\_test\_split(X\_scaled, T, test\_size = 0.2) \\ \hline \hline \end{tabular}   
Table 7 shows the hyperparameters and their settings for each machine learning algorithm used to implement the model. First, the LR model was implemented using the LogisticRegression class, which is included in the linear_model package of Scikit-learn. The value of C, the regularization parameter used to control overfitting, was set to 100. \begin{tabular}{l} \hline \hline from sklearn.linear_model import LogisticRegression \\ model = LogisticRegression(C = 100) \\ model.fit(X\_train, T\_train) \\ \hline \hline \end{tabular}   
The DT model was implemented using the DecisionTreeClassifier class included in the tree package of Scikit-learn. The parameter max_depth was set to 10, which denotes the depth of a Decision Tree to control overfitting. The parameter random_state was set to 0 to get the same Decision Tree when conducting experiments. \begin{tabular}{l} \hline \hline from sklearn.tree import DecisionTreeClassifier \\ model = DecisionTreeClassifier(max_depth = 10, random_state = 0) \\ model.fit(X\_train, T\_train) \\ \hline \hline \end{tabular}   
The RF model was implemented using the RandomForestClassifier class included in the ensemble package of Scikit-learn. The parameter n_estimators was set to 100, which represents the number of trees that make up a forest, and the parameter random_state was set to 0 to get the same forest when conducting experiments. \begin{tabular}{l} \hline \hline from sklearn.ensemble import RandomForestClassifier \\ model = RandomForestClassifier(n_estimators = 100, random_state = 0) \\ model.fit(X\_train, T\_train) \\ \hline \hline \end{tabular}   
The SVM model was implemented using the SVC class included in the svm package of Scikit-learn. The parameter C was set to 100, the same as for the LR model. The DNN model was implemented using Keras 2.0 and consisted of three Dense layers. The Dense layer has two parameters to specify the number of hidden nodes and the activation function. For the first two layers, relu was used as an activation function to calculate the output value from each node while avoiding the vanishing gradient problem [38]. For the last layer, sigmoid was used as an activation function to calculate the probability of student dropout. The parameter input_shape in the first layer was used to specify the size of input data. As an optimizer for the DNN model, adam was used, and binary-crossentropy was adopted as a loss function to correct the weights and biases based on the T values.