# University Student Dropout Prediction Using Pretrained Language Models
## 1 Introduction  
According to a report from the U.S. National Center for Education Statistics, the rate of enrollment in higher-education institutions (e.g., universities) decreased to 66% in 2019 and 63% in 2020 [1]. By contrast, South Korea's enrollment rate increased to 69.4% in 2019 and 71.0% in 2020 ([[https://www.index.go.kr/](https://www.index.go.kr/)]([https://www.index.go.kr/](https://www.index.go.kr/)), accessed on 15 April 2023), the highest worldwide. However, the average dropout rate among freshmen at universities in South Korea is large at 8.0% ([[https://www.academyinfo.go.kr/index.do](https://www.academyinfo.go.kr/index.do)]([https://www.academyinfo.go.kr/index.do](https://www.academyinfo.go.kr/index.do)), accessed on 15 April 2023). Students discontinue their studies for various reasons, including a lack of alignment with academic programs, financial constraints, and social relationships. Student dropout negatively affects not only students but alsolectrpers and universities, which further impacts social and economic costs [2, 3]. Therefore, improving student retention is crucial for universities, necessitating the development of a system that predicts student dropout. Several studies have been conducted to predict university and college student dropout [4, 5, 6, 7, 8, 9]. Recent studies [5, 6, 7] have utilized machine learning techniques to analyze the causes of university dropout. These studies use only the demographic information of students, such as their gender, region of origin, and family income. These approaches have the advantage of predicting the dropout before the beginning of the first semester. However, students who are not predicted to drop out by models based on demographicfactors may fail to complete the course, and vice versa. For instance, some students may experience issues related to their friendships or encounter challenges when adapting to a university environment. In this case, records of academic performance and activities may play an important role in the dropout prediction task. According to previous studies [8; 9], after enrolling in a university, the behaviors of students who drop out and those who successfully complete their degree programs differ. These studies empirically discovered that the predictive performance of their models increased when term exam scores and the number of course credits were included. Following these previous studies, we also utilize various pieces of academic information from students (e.g., GPA, club activities, and course evaluations), as well as demographic information. Students' demographic and academic information throughout the semester is recorded in various formats. For instance, the GPA and student club participation status are recorded as numerical and categorical data, respectively, whereas course evaluation comments are recorded in textual format. Recently, several studies [10; 11] have analyzed unstructured textual data (e.g., advisor notes) to predict student dropout. The study [10] used sentiment analysis to extract positive or negative words from the advisor's notes. The authors then utilized the extracted words as features and trained machine learning models. However, that approach considered only unstructured textual data to predict student dropout. Another study [11] developed a multimodal neural network model that combined structured data (e.g., GPA and SAT scores) and unstructured data (e.g., advising notes and forum posts). The authors showed that fusing structured and unstructured data was effective for predicting student dropout. However, this approach employed two different neural networks (i.e., long short-term memory and transformer) to analyze the structured and unstructured data. It may not capture the implicit relationships between structured and unstructured data.