## 3 Mental Health Root Cause Categories
### Whole Post Analysis
#### 3.2.5 Validation and Quality Assurance  
After the completion of the annotation process, our study progressed into the crucial phase of validation and quality assurance. The credibility and analytical utility of our annotated dataset depended significantly on a thorough validation process. The foundation of our methodology sits on the adoption of a consensus-based approach, complemented by inter-annotator agreement (IAA) measures [83]. These measures were critical in ensuring both the reliability and objectivity of our annotations. They were instrumental not only in verifying the accuracy of our initial annotations but also in continuously improving our annotation guidelines. This further contributed to the overall consistency and dependability of our dataset. The subsequent sections will outline in detail how we seamlessly integrated IAA measures into our annotation workflow, thereby reinforcing the methodological integrity of our research. Consensus-Based Approach  
Our two annotators, equipped with comprehensive guidelines, independently classified each post into one of four mental health root causes: personality, trauma, drug and alcohol, and early life. However, independent classification was only the first step. The key aspect of our methodology was the requirement for a unanimous consensus between annotators on each post, which was particularly crucial in cases of initial disagreement. Such dialogue was instrumental in capturing the intricate details of both the subjective experiences conveyed by the authors and the objective circumstances of their mental health situations. During the dialogue, both annotators were tasked to identify whether the author of the post was expressing one or more of the root causes, and whether the overall situation described in the post fell under these categories while following annotation strategy. This dual focus allowed us to capture both the subjective experiences of the authors and the objective circumstances of their situations, providing a more comprehensive view of the root causes of mental health issues. Inter-Annotator Agreement (IAA) Measures  
The cornerstone of our quality assurance process involved the implementation of inter-annotator agreement (IAA) measures. Following the first phase of independent annotation, we had a inter-annotator agreement (IAA) contingency Table 6, which we used to implement IAA calculations, to quantitatively assess the agreement level between the annotators. By employing Cohen's Kappa, we moved beyond mere percentage agreement to a statistical measure that accounts for the probability of chance agreement. This step was pivotal, especially when addressing the initial annotator disagreement [84]. Our annotated dataset consisted of 800 posts, out of which there were disagreements on 78 posts, while a high level of agreement was observed on the remaining 722 posts. The Cohen's Kappa calculation was conducted as follows:  
* Total number of posts: 800 (722 agreements + 78 disagreements);\(P_{o}\):
* Observed Agreement \(P_{o}\): The proportion of times both annotators agreed (including the 722 agreed posts);* Expected Agreement by Chance \(P_{e}\) : Remained the same, calculated based on the proportion of each category chosen by each annotator. \[\kappa=\frac{P_{o}-P_{e}}{1-P_{e}}\] (1) Our analysis revealed a Cohen's Kappa value of approximately 0.869, indicating a very high level of agreement beyond chance. This value reflects the fact that we could confidently assert the high degree of reliability in our annotations--a testament to the robustness of our annotation process. Domain Expert Consultation Recognizing the complexity of mental health narratives, we further strengthened our annotation process with domain expert consultation. In instances where the content of the posts presented exceptional complexity, or when the annotators' discussions reached a deadlock, we sought the insights of a domain expert.