# Customer Churn Prediction Approach Based on LLM Embeddings and Logistic Regression
## 2 Background and Related Works
### Related Works  
Customer churn prediction has attracted the attention of numerous researchers and organizations, particularly in the telecommunications sector, leading to the proposal of various models. In [15], authors have trained and tested the performance of four machine learning models. Namely, logistic regression, K-nearest neighbors, random forest, and stochastic gradient booster to predict churn. They used a dataset that consists of information on a made-up telecom organization that provides home telephone and internet services. This dataset compasses general information and characteristics about customers such as their gender, age, inscription date, whether they have a telephone/mobile connection, and the number of months customers, have retained with the organization. The authors of this study achieved the best result by using the stochastic gradient booster with an accuracy of 83.9%. Similarly, in [1], researchers tested the performance of two ML models (i.e., logistic regression and logit boost) using data from an American telecommunications company called Orange that contains customer behavior data in a particular period. The logistic regression model outperformed the logit boost, achieving an accuracy of 85.23%. In [16], the authors focused on predicting customer churn in the telecom industry, which includes detailed customer information such as demographics, account details, service subscriptions, monthly charges, and the total number of services used. They evaluated various machine learning models, including support vector machines (SVM), logistic regression (LR), random forest (RF), and neural networks (NN), ultimately identifying the gradient boosting machine (GBM) as the most effective model for this dataset. GBM achieved a high accuracy of 81%, outperforming the other models. Similarly, in [17], the authors tested a range of machine learning methods on a tabular dataset to predict churn, including logistic regression (LR), random forest (RF), support vector machine (SVM), and XGBoost. Among these models, XGBoost achieved the highest performance, with an accuracy of 83%. Within the respected efforts dedicated by researchers to develop models predicting churners, the majority have focused on numerical data, as it aligns well with traditional machine learning and deep learning models, which are currently the preferred methods in prediction churn. However, as datasets increasingly on more complex forms, greater attention should be directed towards leveraging diverse data types to build more robust churn prediction models. The study [18] is the only research that has utilized an embedding approach to vectorizing categorical data, achieving an F1-score of 81.16%. Our study distinguishes itself by applying embedding methods to textual data, achieving a high accuracy of 89%, setting it apart in the context of churn prediction. Given the nature of available datasets in any domain, consisting of numerical, categorical, and textual data, structuring the dataset becomes a primordial preprocessing step, and the importance of embedding becomes more pronounced as it plays a key role in maximizing the utility of the data and ensuring a meaningful representation. Therefore, researchers are proposing employing embedding techniques to vectorize the textual data using techniques such as Sent2 vec [19], Doc2 vec [20], and Google BERT [21; 22; 23]. Accordingly, in [24], the authors propose using the BERT embedding technique for sentiment analysis in Twitter classification tasks. They demonstrate that this approach, when combined with various deep learning classifiers, such as convolutional neural networks (CNN), recurrent neural networks (RNN), and bidirectional long short-term memory (BiLSTM), achieves higher performance than Word2 Vec-based approaches. The contextualized embeddings generated by BERT enable it to capture the sentiment, opinions, and writer's attitudes more effectively, as it considers word context within sentences.