# ChurnKB: A Generative AI-Enriched Knowledge Base for Customer Churn Feature Engineering
## 4 Evaluation
### Evaluating the ChurnKB Development Approach  
We formulated four hypotheses to test and validate the information in the knowledge base and the proposed method. For this purpose, we used an iterative approach that closely mirrors reinforcement learning with human feedback. To test the hypotheses, we designed and distributed anonymous questionnaires consisting of multiple-choice items. Participants were asked to indicate their level of agreement with each statement using a Likert scale ranging from 1 to 5 (1: irrelevant; 2: weakly relevant; 3: neutral; 4: relevant; 5: strongly relevant). No identifying information was collected, and the identities of participants remained undisclosed throughout the study. This approach focuses on refining the knowledge base through multiple feedback loops, gathering domain experts' opinions on the relevance of the taxonomy's elements. Initially, the included questions were reviewed by domain experts to ensure their relevance to the churn problem. Once the questionnaires were finalized, it was recommended by the experts that a 20% threshold be applied in the analysis of the collected responses. All responses were provided anonymously, and no identifying information about the participants was collected or disclosed. On the questionnaire, an explanation of its purpose and goals is first provided. This is followed by a target hypothesis. Next, a snapshot of the ChurnKB components related to the hypothesis questions is presented. Finally, the corresponding questions are provided. Four hypotheses, designed to validate our approach on developing ChurnKB, are outlined as below:  
* H1: The structure of the initial churn taxonomy, including concepts and instances, etc., is relevant to customer churn. * H2: The use of the pronoun \(I\), _absolute words_, and _certainty words_ by customers in their communications or feedback is relevant to identifying customers' churn-related cognitive and behavioural patterns. * H3: Churn-related features (i.e., sub-instances) and the corresponding phrasal/lexical list are relevant to identifying customers' chum-related cognitive and behavioural patterns. * H4: The developed classifier leads to reliable results derived from the application of ChurnKB to enhance the feature engineering phase. The initial version of the churn taxonomy was developed based on customer journey studies and GenAI capabilities. To test H1, we presented various parts of the initial churn taxonomy, including different concepts and instances, to the experts. We asked them to evaluate the relevance of each part to customer churn. An example question designed for this purpose is shown in Figure 4A. To test H2, various questions were posed to validate an assumption, leading to modifications in the initial churn taxonomy. During the development of the initial churn taxonomy, we noted that several sub-instances, such as sadness and negative emotions, were present in a KB developed in previous work [16]. This KB includes different feelings and mental statuses that individuals suffering from mental health issues, such as depression, are likely to experience. Based on this, the use of terms like \(I\), _absolute words_, and _certainty words_ in communication could reflect specific psychological and emotional statuses. Motivated by that understanding, we hypothesised that dissatisfied customers may experience a negative emotional status (e.g., similar to depressed individuals) that confirms their decision to leave a company. Therefore, those individuals may use the pronoun \(I\), _absolute words_, and _certainty words_ in their feedback, reviews, and/or interactions with a company to emphasise a specific decision of theirs. Hence, we came up with the idea of adding those three as sub-instances of _negative word of mouth_, assumed to be probable behavioural signs of unhappy customers who wish to discourage others from using the product/service. To validate the H2, we asked experts about their opinions of adding those sub-instances to the latest version of the churn taxonomy.