## 2 Materials and Methods
### Supervised Classification  
In this work, we train two sets of supervised text classifiers. The first set consists of the binary emotion-specific classifiers that perform the pseudo-labeling process described in Section 2.5. The second set consists of the classifiers used in the experiments described in Section 4. Both sets follow the same architecture as shown in Figure 3. This is a pre-trained language model based on the transformer architecture [27] with a classification head for task-specific fine-tuning, which has come to dominate the state of the art in NLP. We use RoBERTa [55] as the encoder, which replicates BERT [28] with better training. It also replaces BERT's character-level Byte-Pair Encoding (BPE) [56] with a byte-level implementation \begin{table}
\begin{tabular}{l c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Batch Size & 32 \\ Learning Rate & \(3\times 10^{-5}\) \\ Max Epochs & 2 \\ Smoothing & 0.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Hyperparameter values used for training the binary classifiers. Figure 2: Pseudo labeling using weakly labeled data to train a model. Positives in green, negatives in red. similar to GPT [57]. This step is commonly called tokenization, as it segments text into a list of integer identifiers mapped to embeddings, in the model's embedding layer. We do not use additional preprocessing or text encoding steps. The classification head consists of dropout [58], followed by a fully connected dense layer, a hyperbolic tangent activation, dropout, and a fully connected dense projection layer. This classification head is used by the Transformers library in the _\"RobertaForSequenceClassification\"_ model. We adopt it as it allows for easier replication of our work and for our model weights to be shared without the need to share additional code (See [[https://huggingface.co/lrei/emolit-roberta-large-soft](https://huggingface.co/lrei/emolit-roberta-large-soft)]([https://huggingface.co/lrei/emolit-roberta-large-soft](https://huggingface.co/lrei/emolit-roberta-large-soft)) for an example (accessed on 23 June 2023). The pseudo-labeling models use the smaller \(BASE\) transformer with 12 transformer layers, a hidden dimension of 768, with 12 attention heads (L = 12, H = 768, A = 12), totaling 125 M parameters. The models in the experiments section primarily use the \(LARGE\) transformer (L = 24, H = 1024-hidden, A = 16, 355 M parameters). Additionally, in Section 4.2 we also provide results for a smaller and faster model based on the _DIS-TIL_ ([[https://huggingface.co/distiroberta-base](https://huggingface.co/distiroberta-base)]([https://huggingface.co/distiroberta-base](https://huggingface.co/distiroberta-base)) (accessed on 23 June 2023)) transformer (L = 6, H = 768, A = 12, 82 M parameters) [59]. Pseudo-labeling models use the Softmax as the final activation function, whereas the models in the experiments use either the Softmax or Sigmoid based on whether the target dataset is multiclass or multi-label, respectively. The final FC layer has a size of \(H\times N\) where N is the number of classes, for pseudo-labeling N = 2, for models trained on our data N = 38, while models trained on other datasets have dataset-specific values of N. The dropout probability used was always 0.1. All models are trained using mixed precision (FP16) and the Adam [60] optimizer with the linear scheduler without warmup. Hyperparameters are taken directly from Ref. [55] without any hyperparameter tuning specific to the dataset. For pseudo-labeling, we used a batch size of 32 with a learning rate of \(3\times 10^{-5}\). Experiment-specific hyperparameters are presented in their respective sections. All models trained on our dataset were limited during training to a maximum sequence length of 48. The computational complexity of the transformer architecture, resulting from the use of self-attention, is \(\mathcal{O}(L\cdot(n^{2}\cdot H))\), where \(L\) is the number of self-attention layers, \(n\) is the sequence length, and \(H\) is the hidden dimension size.