## 3 Methodology
### Stage 4: Investigating Semantic Differences by Cross-Domain Means  
[PERSON] et al. [28] found that single-domain models perform better on their corresponding domain, but they perform poorly on multi-domains. On the other hand, when it comes to utilizing multi-domain datasets, multi-domain models show superior performance as compared to single-domain models. We decided to compare the performances of multiple models on single and multiple domains.  
To extract data from the datasets, we utilized Python's Pandas library for handling CSV format. Single-domain data are defined as J (Jira), T (Twitter), and R (Reddit). To make use of the multi-domain datasets, we combine single-domain data subsets and generate the following sets: J + T (Jira + Twitter), T + R (Twitter + Reddit), J + R (Jira + Reddit), and J + T + R (Jira + Twitter + Reddit).  
Once we completed the extraction process, we had to refine the raw data. Both Jira and Twitter datasets have seven basic labels of emotions, which are joy, love, surprise, neutral, sadness, fear, and anger. On the other hand, the Reddit dataset has 28 labeled emotions. To simplify it, we grouped the 28 different labels into an Ekman-style group [29], which consists of six categories as mentioned earlier. This allowed us to better capture the full spectrum of emotional expressions in the comments, including those that did not fit into the initial categories. As fear and surprise data emotions are scarce, we have removed them entirely. Having a severely imbalanced data collection, with some datasets being more dominant than others, biased model predictions and inaccurate results may arise as a consequence. To address the issue, we decided to sample all datasets uniformly. The number of entries for each label was selected based on the Jira dataset's maximum label size. The collected data are described in Figures 2-4 as follows:  
* Figure 2 represents the single-domain data distribution for cases J, T, and R; consisting of 5 labels, each with their corresponding size.
* Figure 3 shows the two-domain data distribution for J + T, T + R, and J + R, with 5 labels equally divided between the two domains.
* Figure 4 illustrates a three-domain data distribution for J + T + R, including 5 labels, where every third of the labels corresponds to each domain.  
Figure 1: Semantic distribution of short messages.  
In the following step, the raw text is transformed into individual tokens using a pre-existing tokenization method, namely RobertaTokenizer. Tokenization is a process of  
Figure 4: Three-domain data distribution.  
Figure 3: Two-domain data distribution.  
Figure 2: Single-domain data distribution.  
converting text into individual words (tokens), which can then be analyzed and processed further. To specify the maximum number of tokens for encoding, we have first checked the number of tokens per comment. Given that most comments contain fewer than 200 tokens, we set the limit at 128 tokens.  
For the training process, we initialized the \"roberta-base\" model and specified its layers using PyTorch Lightning. We accomplished this by creating the LightningModule class. To enable data forwarding, we redefined the class methods training_step, validation_step, and test_step for our case. The final steps of the training process were concluded by configuring the Adam optimizer and setting up training configurations such as batch size, number of epochs, and learning rate.  
Figure 5 illustrates that the transformer models were trained on 70% of the dataset, validated on 20%, and tested on 10% of the data for both self-domain and cross-domain scenarios.