## 2 Materials and Methods
### Deep Learning Model for Classification of Mental Health Disorders
#### 2.2.3 Model Training  
BERT (Bidirectional Encoder Representations from Transformers) is an encoder-centric model based on the Transformer architecture [57]. It employs an unsupervised training approach with emphasis on two principal loss functions during its pre-training phase: Masked Language Model loss (MLM) and Next Sentence Prediction loss (NSP). For the MLM loss computation, a subset of tokens from an initial input sequence, denoted as \(x=(x_{1},x_{2},\cdots,x_{T})\), are substituted with the special token [MASK], which produces a modified sequence \(m\). The MLM loss is then determined by \(L_{t}=-\log P(x_{t}|m)\). Here, \(P(x_{t}|m)\) represents the predicted probability of the model for the genuine token \(x_{t}\) at the masked position, conditioned on the modified sequence \(m\). Regarding NSP loss, individual examples are separated into sentences, establishing a dataset composed of examples with sentences A and B. Typically, for half of the instances, sentence B sequentially follows A in the corpus. For the other instances, B is a sentence arbitrarily selected from the corpus. The goal of the model is to predict a label \(1\) when B is the sequential sentence in the corpus and \(0\) when randomly chosen. The NSP loss is computed using binary cross-entropy: \(L_{NSP}=-y\log(y^{*})-(1-y)\log(1-y^{*})\). Following pre-training, the model is fine-tuned via a supervised approach for specific tasks. This synergy between unsupervised and supervised training has shown superior results across diverse domains and is the foundation of many modern language models. BERT authors trained the model unsupervised on a combined set of BookCorpus and Wikipedia. The models obtained from this pretraining were made available to the  
Figure 2: (a) The distribution of number of examples per each subreddit in the Reddit dataset. (b) The distribution of number of examples per each diagnosis category in the extended dataset. community [58] for various versions of the base transformer model of different sizes in terms of the number of weights. Several subsequent studies [52; 53; 54] have demonstrated that incorporating an additional phase of unsupervised learning or pre-training can improve classification performance in downstream tasks across multiple domains. This added phase allows the model to adapt to distributional shifts in the corpus, which are anticipated due to the general nature of the corpus used during model training. For example, technical corpora, or those abundant in domain-specific terminology, are likely to exhibit a distribution with significantly higher conditional probability for certain terms that would be rare in a more general corpus. Notably, it has been recognized that discussions related to mental health on Reddit display a particular distribution in linguistic features [45]. Furthermore, terms associated with treatments and diagnoses are widely used, which are atypical in broader contexts. With this in mind, the first step involved pretraining the BERT model using the entire collection of Reddit posts from the training dataset. For this endeavor, a training set was assembled by masking 15% of the tokens, acquired via WordPiece tokenization, and constructing pairs of sentences for the next sentence prediction task. This was balanced at a 50% ratio between genuinely consecutive sentences and randomly selected ones. After the initial pretraining step, we conducted a subsequent pretraining phase using data from the extended training set. We separated this from the Reddit set due to the considerable size difference, leading to an imbalance particularly for the less common conditions in the extended set. Adapting the model to these less common conditions was the primary reason for extending the initial dataset. Hence, we opted for a distinct, shorter pretraining process, separate from the first pretraining phase. The processing for this phase was performed in the same manner as for the Reddit data. For the fine-tuning step, we explored two approaches.