# Customer Churn Prediction Approach Based on LLM Embeddings and Logistic Regression
## 4 Materials and Methods
### Dataset
#### 4.1.2 Data Preprocessing  
Data preprocessing is an important step in the machine learning process. It includes structuring, cleaning, and transforming raw data to prepare it for further analysis or feeding it to machine learning and deep learning models. For our dataset, we have employed various preprocessing techniques to make it more relevant and structured. The preprocessing we employed is as follows:  
Figure 3: Churn Predictive Model Building Methodology.  
1. Deleting unnecessary columns. For instance, the \"Security_no\" contained one unique value and thus did not contribute to the analysis.
2. Handling missing values: Columns with a high percentage of null values (exceeding 40%) were eliminated to maintain data integrity; for columns with a smaller percentage of missing values, we adopted the following approach: For categorical features, missing values were imputed using the mode; for numerical features, we applied the KNN imputer.
3. Outlier Treatment: Outlier removal was performed to improve the quality of the dataset; to detect and remove outliers, we used the quantile method.
4. Data structuring: To build structured datasets, we converted numerical attributes into categorical attributes. For instance, the \"age\" attribute, originally represented as numeric values, was converted into age groups. Date and time data were excluded, as they could not be effectively transformed into meaningful text, and for categorical data, we retained them in text to enable concatenation with other columns. This transformation allows for more effective embedding and analysis of the data.  
As represented in Table 1, these preprocessing steps resulted in including 14 attributes out of the 23 in the dataset. These attributes are all categorical and provide valuable information that will be beneficial in the customer classification phase. The description of these 14 attributes is provided in Table 1.  
After preprocessing the dataset, we conducted several experiments using embedding techniques. However, embedding each attribute individually from a dataset of 14 attributes would lead to a considerable increase in dataset size (each attribute embedding will create a vector of at least 40 dimensions), which will explicitly increase the computation complexity. Therefore, to employ the embedding technique, we opted to merge the values of the 14 features into a unified single column structured as a coherent, semantically meaningful sentence.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Attribute Name** & **Description** \\ \hline age & Represents the age of a customer \\ \hline Gender & Represents the gender of a customer \\ \hline Region\_category & Represents the region to which a customer belongs \\ \hline Membership\_category & Represents the type of membership category that a customer is utilizing \\ \hline Joined\_through\_referral & Indicates whether a customer joined by using a referral code or ID \\ \hline Preferred\_offer\_types & Represents the type of offer that a customer prefers \\ \hline Medium\_of\_operation & Represents the medium of operation that a customer uses for transactions \\ \hline Internet\_option & Represents the type of internet service a customer uses \\ \hline Used\_special\_discount & Represents whether a customer uses special discounts offered \\ \hline Offer\_application\_preference & Represents whether a customer prefers offers \\ \hline Past\_complaint & Indicates whether a customer has submitted any complaints \\ \hline Complaint\_status & Indicates whether the complaints submitted by a customer have been resolved \\ \hline Feedback & Represents the feedback provided by a customer \\ \hline Churn\_risk\_score & Indicates whether the customer is a churner (1) or not (0) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Description of the 14 attributes included in our study.