# University Student Dropout Prediction Using Pretrained Language Models
## 4 Experiments
### Comparisons with Baseline Models  
We evaluated the performance of seven methods. We first adopted multimodal spatiotemporal neural fusion (MSNF) [11] as a state-of-the-art method for the dropout prediction task. Other baselines included machine learning methods, such as logistic regression, support vector machine (SVM), decision tree, random forest, as well as deep learning methods, such as MLP. In our experiments, we compared the following methods:  
* Logistic regression (baseline): This model is used to estimate the probability that dependent variables belong to a particular class. If the estimated probability exceeded a specific threshold (0.5), the model predicted the variables belonged to the dropout class. To counterbalance the lower proportion of student dropouts, we set the influence of the dropout class on the cost function to eight times that of the other class. The regularization parameter was set to 0.01 [5; 6; 22].
* SVM (baseline): This model is a binary linear classification model that determines which category the data belong to. In the feature space mapped to the dataset, the model finds the appropriate decision boundaries that divide the data into two classes. We set \"rbf\" as the kernel type and \"scale\" as a kernel coefficient. To address data imbalance, we set the influence of the dropout class on the cost function to six times that of the other class. The regularization parameter was set to one [5; 8; 9; 22].
* Decision tree (baseline): This model shows the data patterns as predictable combinations of features. The model then proceeds with a binary classification depending on whether the features match. We used \"gini\" as a function to measure the quality of he division [42] and adjusted the weight of the dropout class in the gini calculation to be six times higher than that of the other class to address data imbalance. We set the maximum depth to 5 [4; 5; 22].
* Random Forest (baseline): This model is an ensemble of decision trees. Instead of creating a decision tree for the entire feature set, a random forest randomly selects some features to generate several decision trees and collects the classification results from them. We use \"gini\" as a function to measure the quality of the division and set the weight of the dropout class in the gini calculation to be eight times higher than that of the other class. We set the maximum depth to 10 [4; 5; 8; 9; 22].
* MLP (baseline): This model is a multilayer structure that combines multiple perceptrons. We employed an MLP with a supervised learning technique using a gradient descent algorithm for training [5; 6; 22].
* MSNF (baseline): This model is the state-of-the-art dropout prediction method. It is a multitask model that uses long short-term memory, 1D convolutional neural networks (CNN), and BERT models to predict future dropout, next semester dropout, type of dropout, duration of dropout, and cause of dropout. Because we only used data from the first semester of the first year, we did not use temporal information. Therefore, we reimplemented the model using a 1D CNN and a BERT structure to predict future dropout [11].  
STUD: This is our proposed dropout prediction framework. We first formulated student demographic and academic information in a natural language format. We then fine-tuned PLMs (e.g., BERT [27], SimCSE-BERT [31]) by recasting the problem as an NLI task, prepending a continuous hypothesis at each layer of the model.