# Customer Churn Prediction Approach Based on LLM Embeddings and Logistic Regression
## 6 Result and Discussion  
In this section, we will evaluate the performance of various combinations of embedding techniques and classifiers by measuring accuracy, precision, recall, and F1-score. A summary of the comparison results is presented in Table 3.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Embedding Model**} & \multirow{2}{*}{**Classifier**} & \multicolumn{4}{c}{**Measure**} \\ \cline{3-6}  & & **Accuracy** & **Precision** & **F1 Score** & **Recall** \\ \hline \multirow{6}{*}{OpenAI Text-embedding-ada-002} & LR & **89\%** & 89\% & 88\% & 89\% \\ \cline{2-6}  & SVM & 88.5\% & 89\% & 88\% & 89\% \\ \cline{2-6}  & RF & 83\% & 84\% & 83\% & 83\% \\ \cline{2-6}  & MLP & 83\% & 83\% & 83\% & 84\% \\ \cline{2-6}  & NB & 81\% & 83\% & 83\% & 81\% \\ \cline{2-6}  & KNN & 76\% & 76\% & 75\% & 75\% \\ \cline{2-6}  & DT & 72\% & 72\% & 72\% & 72\% \\ \cline{2-6}  & Zero-shot classification & 66\% & 68\% & 65\% & 89\% \\ \hline \multirow{6}{*}{Google Gemini text-embedding-001} & LR & **85\%** & 85\% & 85\% & 85\% \\ \cline{2-6}  & SVM & **85\%** & 87\% & 85\% & 86\% \\ \cline{2-6}  & RF & **85\%** & 85\% & 85\% & 85\% \\ \cline{2-6}  & RF & **85\%** & 85\% & 85\% & 85\% \\ \cline{2-6}  & MLP & 84\% & 85\% & 83\% & 83\% \\ \cline{2-6}  & NB & 78\% & 78\% & 78\% & 78\% \\ \cline{2-6}  & KNN & 76\% & 76\% & 76\% & 76\% \\ \cline{2-6}  & DT & 77\% & 77\% & 77\% \\ \cline{2-6}  & DT & 85\% & 86\% & 85\% & 86\% \\ \cline{2-6}  & NB & 77\% & 76\% & 76\% & 76\% \\ \cline{2-6}  & NB & 77\% & 76\% & 76\% & 76\% \\ \cline{2-6}  & DT & 72\% & 72\% & 71\% & 71\% \\ \cline{2-6}  & Zero-shot classification & 54\% & 27\% & 35\% & 50\% \\ \hline \multirow{6}{*}{Sentence-Transformers paraphrase-MiniLM-L6-v2} & LR & 85\% & 85\% & 85\% & 85\% \\ \cline{2-6}  & SVM & **86\%** & 87\% & 86\% & 87\% \\ \cline{2-6}  & RF & 84\% & 84\% & 84\% & 84\% \\ \cline{2-6}  & MLP & 84\% & 84\% & 84\% & 84\% \\ \cline{2-6}  & NB & 76\% & 76\% & 76\% & 76\% \\ \cline{2-6}  & KNN & 78\% & 78\% & 78\% & 77\% \\ \cline{2-6}  & DT & 76\% &As shown in Table 3, the OpenAI embedding model demonstrated superior performance in terms of accuracy across all assessed machine-learning models. It outperformed other embedding techniques by reducing the error ratio by nearly 3% for the sent2 vec model and Sentence-Transformers, 4% for Google Gemini and Google Bert, and 30% for the Doc2 vec embedding model. Sent2 Vec also demonstrates its efficacy as an embedding model, achieving an accuracy of 86% with LR and 85% with MLP. Sentence-Transformers exhibit an accuracy of 86% with SVM and 85% with LR. Meanwhile, Google Gemini achieves an accuracy of 85% with LR, SVM, and RF, whereas Google Bert reaches an accuracy of 85% with LR, SVM, and MLP. However, Doc2 Vec's results are suboptimal, with the highest accuracy reaching only 57%, observed with SVM as the classifier. This suggests that Doc2 Vec may not be a robust embedding model due to its limited performance on short texts. Furthermore, the results indicate that zero-shot classification, while trendy, proves to be an ineffective classification technique based on the obtained accuracies. Conclusively, the study reveals that the combination of OpenAI's embedding model with a logistic regression classifier stands out for customer churn prediction, achieving the highest accuracy of 89%. Figure 4 represents the final settings and approach that outperformed the other model.