SlowGuess/ACL_2020_OCR huggingface dataset:
In Section 3, we describe how we obtain subreddits that are balanced in terms of sentiment. Here, we note the distribution of sentiments across subreddits before we apply the filtering: neutral (M=28%, STD=11%), positive (M=41%, STD=11%), negative (M=19%, STD=7%), ambiguous (M=35%, STD=8%). After filtering, the distribution of sentiments across our remaining subreddits became: neutral (M=24%, STD=5%), positive (M=35%, STD=6%), negative (M=27%, STD=4%), ambiguous (M=33%, STD=4%).
E BERT's Most Activated Layers
To better understand whether there are any layers in BERT that are particularly important for our task, we freeze BERT and calculate the center of
gravity (Tenney et al., 2019) based on scalar mixing weights (Peters et al., 2018). We find that all layers are similarly important for our task, with center of gravity  $= 6.19$  (see Figure 4). This is consistent with Tenney et al. (2019), who have also found that tasks involving high-level semantics tend to make use of all BERT layers.
Figure 4: Softmax weights of each BERT layer when trained on our dataset.
F Number of Emotion Labels Per Example
Figure 5 shows the number of emotion labels per example before and after we filter for those labels that have agreement. We use the filtered set of labels for training and testing our models.
Figure 5: Number of emotion labels per example before and after filtering the labels chosen by only a single annotator.
G Confusion Matrix
Figure 6 shows the normalized confusion matrix for our model predictions. Since GoEmotions is a multilabel dataset, we calculate the confusion matrix
similarly as we would calculate a co-occurrence matrix: for each true label, we increase the count for each predicted label. Specifically, we define a matrix  $M$  where  $M_{i,j}$  denotes the raw confusion count between the true label  $i$  and the predicted label  $j$ . For example, if the true labels are joy and admiration, and the predicted labels are joy and pride, then we increase the count for  $M_{joy,joy}$ ,  $M_{joy,pride}$ ,  $M_{admiration,joy}$  and  $M_{admiration,pride}$ . In practice, since most of our examples only has a single label (see Figure 5), our confusion matrix is very similar to one calculated for a single-label classification task.
Given the disparate frequencies among the labels, we normalize  $M$  by dividing the counts in each row (representing counts for each true emotion label) by the sum of that row. The heatmap in Figure 6 shows these normalized counts. We find that the model tends to confuse emotions that are related in sentiment and intensity (e.g., grief and sadness, pride and admiration, nervousness and fear).
We also perform hierarchical clustering over the normalized confusion matrix using correlation as a distance metric and ward as a linkage method. We find that the model learns relatively similar clusters as the ones in Figure 2, even though the training data only includes a subset of the labels that have agreement (see Figure 5).
H Transfer Learning Results
Figure 7 shows the results for all 9 datasets that are downloadable and have categorical emotions in the Unified Dataset (Bostan and Klinger, 2018). These datasets are DailyDialog (Li et al., 2017), Emotion-Stimulus (Ghazi et al., 2015), Affective Text (Strapparava and Mihalcea, 2007), CrowdFlower (CrowdFlower, 2016), Electoral Tweets (Mohammad et al., 2015), ISEAR (Scherer and Wallbott, 1994), the Twitter Emotion Corpus (TEC) (Mohammad, 2012), EmoInt (Mohammad et al., 2018) and the Stance Sentiment Emotion Corpus (SSEC) (Schuff et al., 2017).
We describe the experimental setup in Section 6.2, which we use across all datasets. We find that transfer learning helps in the case of all datasets, especially when there is limited training data. Interestingly, in the case of CrowdFlower, which is known to be noisy (Bostan and Klinger, 2018) and Electoral Tweets, which is a small dataset of  $\sim 4\mathrm{k}$  labeled examples and a large
taxonomy of 36 emotions, FREEZE gives a significant boost of performance over the BASELINE and NOFREEZE for all training set sizes besides "max".
For the other datasets, we find that FREEZE tends to give a performance boost compared to the other setups only up to a couple of hundred training examples. For 500-1000 training examples, NOFREEZE tends to outperform the BASELINE, but we can see that these two setups come closer when there is more training data available. These results suggest that our dataset helps if there is limited data from the target domain.
Figure 6: A normalized confusion matrix for our model predictions. The plot shows that the model confuses emotions with other emotions that are related in intensity and sentiment.
Figure 7: Transfer learning results on 9 emotion benchmarks from the Unified Dataset (Bostan and Klinger, 2018).