Contrastive Learning of Sociopragmatic Meaning in Social Media
Year: 2023

Abstract:
Recent progress in representation and contrastive learning in NLP has not widely considered the class of sociopragmatic meaning (i.e., meaning in interaction within different language communities). To bridge this gap, we propose a novel framework for learning taskagnostic representations transferable to a wide range of sociopragmatic tasks (e.g., emotion, hate speech, humor, sarcasm). Our framework outperforms other contrastive learning frameworks for both in-domain and out-of-domain data, across both the general and few-shot settings. For example, compared to two popular pre-trained language models, our model obtains an improvement of 11.66 average F1 on 16 datasets when fine-tuned on only 20 training samples per dataset. We also show that our framework improves uniformity and preserves the semantic structure of representations. Our code is available at: https: //github.com/UBC-NLP/infodcl

heading: 1 Introduction
text: Meaning emerging through human interaction such as on social media is deeply contextualized. It extends beyond referential meaning of utterances to involve both information about language users and their identity (the domain of sociolinguistics (Tagliamonte, 2015)) and the communication goals of these users (the domain of pragmatics (Thomas, 2014)). From a sociolinguistics perspective, a message can be expressed in various linguistic forms, depending on user background. For example, someone might say ‘let’s watch the soccer game’, but they can also call the game ‘football’. In real world, the game is the same thing. While the two expressions are different ways of saying the same thing (Labov, 1972), they do carry information about the user such as their region (i.e., where they could be coming from). From a pragmatics perspective, the meaning of an utterance depends on its interactive context. For example,
while the utterance ‘it’s really hot here’ (said in a physical meeting) could be a polite way of asking someone to open the window, it could mean ‘it’s not a good idea for you to visit at this time’ (said in a phone conversation discussing travel plans). We refer to the meaning communicated through this type of socially embedded interaction as sociopragmatic meaning (SM).
While SM is an established concept in linguistics (Leech, 1983), NLP work still lags behind. This issue is starting to be acknowledged in the NLP community (Nguyen et al., 2021), and there has been calls to include social aspects in representation learning of language (Bisk et al., 2020). Arguably, pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) learn representations relevant to SM tasks. While this is true to some extent, PLMs are usually pre-trained on standard forms of language (e.g. BookCorpus) and hence miss (i) variation in language use among different language communities (social aspects of meaning) in (ii) interactive settings (pragmatic aspects). In spite of recent efforts to rectify some of these limitations by PLMs such as BERTweet on casual language (Nguyen et al., 2020), it is not clear whether the masked language modeling (MLM) objective employed in PLMs is sufficient for capturing the
2405
rich representations needed for sociopragmatics. Another common issue with PLMs is that their sequence-level embeddings suffer from the anisotropy problem (Ethayarajh, 2019; Li et al., 2020). That is, these representations tend to occupy a narrow cone on the multidimensional space. This makes it hard for effectively teasing apart sequences belonging to different classes without use of large amounts of labeled data. Work on contrastive learning (CL) has targeted this issue of anisotropy by attempting to bring semantic representations of instances of a given class (e.g., positive pairs of the same objects in images or same topics in text) closer and representations of negative class(es) instances farther away (Liu et al., 2021a; Gao et al., 2021). A particularly effective type of CL is supervised CL (Khosla et al., 2020; Khondaker et al., 2022), but it (i) requires labeled data (ii) for each downstream task. Again, acquiring labeled data is expensive and resulting models are task-specific (i.e., cannot be generalized to all SM tasks).
In this work, our goal is to learn effective taskagnostic representations for SM from social data without a need for labels. To achieve this goal, we introduce a novel framework situated in CL that we call InfoDCL. InfoDCL leverages sociopragmatic signals such as emojis or hashtags naturally occurring in social media, treating these as distant/surrogate labels.1 Since surrogate labels are abundant (e.g., hashtags on images or videos), our framework can be extended beyond language. To illustrate the superiority of our proposed framework, we evaluate representations by our InfoDCL on 24 SM datasets (such as emotion recognition (Mohammad et al., 2018) and irony detection (Ptácek et al., 2014)) and compare against 11 competitive baselines. Our proposed framework outperforms all baselines on 14 (out of 16) in-domain datasets and seven (out of eight) out-of-domain datasets (Sec. 4). Furthermore, our framework is strikingly successful in few-shot learning: it consistently outperforms baselines by a large margin for different sizes of training data (Sec. 4). Our framework is also language-independent, as demonstrated on several tasks from three languages other than English (Sec. E.3).
Our major contributions are as follows: (1) We introduce InfoDCL, a novel CL framework for learning sociopragmatics exploiting surrogate la-
1We use distant label and surrogate label interchangeably.
bels. To the best of our knowledge, this is the first work to utilize surrogate labels in language CL to improve PLMs. (2) We propose a new CL loss, Corpus-Aware Contrastive Loss (CCL), to preserve the semantic structure of representations exploiting corpus-level information (Sec. 3.3). (3) Our framework outperforms several competitive methods on a wide range of SM tasks (both in-domain and out-of-domain, across general and few-shot settings). (4) Our framework is language-independent, as demonstrated by its utility on various SM tasks in four languages. (5) We offer an extensive number of ablation studies that show the contribution of each component in our framework and qualitative analyses that demonstrate superiority of representation from our models (Sec. 5).

heading: 2 Related Work
text: Our work combines advances in representation learning and contrastive learning.
Representation Learning. PLMs encode discrete language symbols into a continuous representation space that can capture the syntactic and the semantic information underlying the text. Since BERT is pre-trained on standard text that is not ideal for social media, Nguyen et al. (2020) propose BERTweet, a model pre-trained on tweets with MLM objective and without intentionally learning SM from social media data. Previous studies (Felbo et al., 2017; Corazza et al., 2020) have also utilized distant supervision (e.g., use of emoji) to obtain better representations for a limited number of tasks. Our work differs in that we make use of distant supervision in the context of CL to acquire rich representations suited to the whole class of SM tasks. In addition, our methods excel not only in the full data setting but also for few-shot learning and diverse domains.
Contrastive Learning. There has been a flurry of recent CL frameworks introducing selfsupervised (Liu et al., 2021a; Gao et al., 2021; Cao et al., 2022), semi-supervised (Yu et al., 2021), weakly-supervised (Zheng et al., 2021), and strongly supervised (Gunel et al., 2021; Suresh and Ong, 2021; Zhou et al., 2022) learning objectives.2 Although effective, existing supervised CL (SCL) frameworks (Gunel et al., 2021; Suresh and Ong, 2021; Pan et al., 2022) suffer from two
2These frameworks differ across a number of dimensions that we summarize in Table 6 in Sec. A in Appendix.
major drawbacks. The first drawback is SCL’s dependence on task-specific labeled data (which is required to identify positive samples in a batch). Recently, Zheng et al. (2021) introduced a weaklysupervised CL (WCL) objective for computer vision, which generates a similarity-based 1-nearest neighbor graph in each batch and assigns weak labels for samples of the batch (thus clustering vertices in the graph). It is not clear, however, how much an WCL method with augmentations akin to language would fare for NLP. We propose a framework that does not require model-derived weak labels, which outperforms a clustering-based WCL approach. The second drawback with SCL is related to how negative samples are treated. Khosla et al. (2020); Gunel et al. (2021) treat all the negatives equally, which is sub-optimal since hard negatives should be more informative (Robinson et al., 2021). Suresh and Ong (2021) attempt to rectify this by introducing a label-aware contrastive loss (LCL) where they feed the anchor sample to a taskspecific model and assign higher weights to confusable negatives based on this model’s confidence on the class corresponding to the negative sample. LCL, however, is both narrow and costly. It is narrow since it exploits task-specific labels. We fix this by employing surrogate labels generalizable to all SM tasks. In addition, LCL is costly since it requires an auxiliary task-specific model to be trained with the main model. Again, we fix this issue by introducing a light LCL framework (LCL-LiT) where we use our main model, rather than an auxiliary model, to derive the weight vector wi from our main model through an additional loss (i.e., weighting is performed end-to-end in our main model). Also, LCL only considers instancelevel information to capture relationships between individual sample and classes. In comparison, we introduce a novel corpus-aware contrastive loss (CCL) that overcomes this limitation (Sec. 3.3).

heading: 3 Proposed Framework
text: Our goal is to learn rich and diverse representations suited for a wide host of SM tasks. To this end, we introduce our novel InfoDCL framework. InfoDCL is a distantly supervised CL (DCL) framework that exploits distant/surrogate label (e.g., emoji) as a proxy for supervision and incorporates corpus-level information to capture inter-class relationships.

heading: 3.1 Contrastive Losses
text: CL aims to learn efficient representations by pulling samples from the same class together and pushing samples from other classes apart (Hadsell et al., 2006). We formalize the framework now. Let C denote the set of class labels. Let D = {(xi, yi)}mi=1 denote a randomly sampled batch of size m, where xi and yi ∈ C denote a sample and its label respectively. Many CL frameworks construct the similar (a.k.a., positive) sample (xm+i) for an anchor sample (xi) by applying a data augmentation technique (T ) such as back-translation (Fang and Xie, 2020), token masking (Liu et al., 2021a), and dropout masking (Gao et al., 2021) on the anchor sample (xi). Let B = {(xi, yi)}2mi=1 denote an augmented batch, where xm+i = T (xi) and ym+i = yi (i = {1, . . . ,m}). Self-supervised Contrastive Loss. We consider |C| = N , where N is the total number of training samples. Hence, the representation of the anchor sample xi is pulled closer to that of its augmented (positive) sample xm+i and pushed away from the representations of other 2m− 2 (negative) samples in the batch. The semantic representation hi ∈ Rd for each sample xi is computed by an encoder, Φ, where hi = Φ(xi). Chen et al. (2017) calculate the contrastive loss in a batch as follows:
LSSCL = 2m∑
i=1
− log e sim(hi,hp(i))/τ
∑2m a=1 1[a ̸=i]e sim(hi,ha)/τ , (1)
where p(i) is the index of positive sample of xi,3 τ ∈ R+ is a scalar temperature parameter, and sim(hi, hj) is the cosine similarity h⊤i hj ∥hi∥·∥hj∥ .
Supervised Contrastive Loss. The CL loss in Eq. 1 is unable to handle the case of multiple samples belonging to the same class when utilizing a supervised dataset (|C| < N ). Positive samples in SCL (Khosla et al., 2020) is a set composed of not only the augmented sample but also the samples belonging to the same class as xi. The positive samples of xi are denoted by Pi = {ρ ∈ B : yρ = yi ∧ ρ ̸= i}, and |Pi| is its cardinality. The SCL is formulated as:
LSCL = 2m∑
i=1
−1 |Pi|
∑
p∈Pi
log esim(hi,hp)/τ
∑2m a=1 1[a ̸=i]e sim(hi,ha)/τ . (2)
In our novel framework, we make use of SCL but employ surrogate labels instead of gold labels to construct the positive set.
3If i ≤ m, p(i) = i+m, otherwise p(i) = i−m.

heading: 3.2 Label-Aware Contrastive Loss
text: Suresh and Ong (2021) extend the SCL to capture relations between negative samples. They hypothesize that not all negatives are equally difficult for an anchor and that the more confusable negatives should be emphasized in the loss. They propose LCL, which introduces a weight wi,ya to indicate the confusability of class label ya w.r.t. anchor xi:
LLCL = 2m∑
i=1
−1 |Pi|
∑
p∈Pi
log wi,yi · e sim(hi,hp)/τ
∑2m a=1 1[a ̸=i]wi,ya · esim(hi,ha)/τ .
(3)
The weight vector wi ∈ R|C| comes from the class-specific probabilities (or confidence score) outputted by an auxiliary task-specific supervised model after consuming the anchor xi. LCL assumes that the highly confusable classes w.r.t anchor receive higher confidence scores, while the lesser confusable classes w.r.t anchor receive lower confidence scores. As stated earlier, limitations of LCL include (i) its dependence on gold annotations, (ii) its inability to generalize to all SM tasks due to its use of task-specific labels, and (iii) its ignoring of corpus-level and inter-class information. As explained in Sec. 2, we fix all these issues.

heading: 3.3 Corpus-Aware Contrastive Loss
text: In spite of the utility of existing CL methods for text representation, a uniformity-tolerance dilemma has been identified in vision representation model by Wang and Liu (2021): pursuing excessive uniformity makes a model intolerant to semantically similar samples, thereby breaking its underlying semantic structure (and thus causing harm to downstream performance).4 Our learning objective is to obtain representations suited to all SM tasks, thus we hypothesize that preserving the semantic relationships between surrogate labels during pre-training can benefit many of downstream SM tasks. Since we have a large number of fine-grained classes (i.e., surrogate labels), each class will not be equally distant from all other classes. For example, the class ‘ ’ shares similar semantics with the class ‘ ’, but is largely distant to the class ‘
’. The texts with ‘ ’ and ‘ ’ belong to same class of ‘joy’ in downstream emotion detection task. We thus propose a new CL method that relies on distant supervision to learn general knowledge of all SM tasks and incorporates corpus-level information to capture inter-class relationships, while improving uniformity of PLM and preserving the
4For details see Sec. G in Appendix.
underlying semantic structure. Concretely, our proposed corpus-aware contrastive loss (CCL) exploits a simple yet effective corpus-level measure based on pointwise mutual information (PMI) (Bouma, 2009) to extract relations between surrogate labels (e.g., emojis) from a large amount of unlabeled tweets.5 The PMI method is cheap to compute as it requires neither labeled data nor model training: PMI is based only on the co-occurrence of emoji pairs. We hypothesize that PMI scores of emoji pairs could provide globally useful semantic relations between emojis. Our CCL based on PMI can be formulated as:
LCCL = 2m∑
i=1
−1 |Pi|
∑
p∈Pi
log esim(hi,hp)/τ
∑2m a=1 1[a ̸=i]wyi,ya · esim(hi,ha)/τ ,
(4)
where the weight wyi,ya = 1 − max(0, npmi(yi, ya)), and npmi(yi, ya) ∈ [−1, 1] is normalized point-wise mutual information (Bouma, 2009) between ya and yi.6

heading: 3.4 Overall Objective
text: To steer the encoder to learn representations that recognize corpus-level inter-class relations while distinguishing between classes, we combine our LCCL and LLCL.7 The resulting loss, which we collectively refer to as distantly-supervised contrastive loss LDCL is given by:
LDCL = γLLCL + (1− γ)LCCL, (5) where γ ∈ [0, 1] is a hyper-parameter that controls the relative importance of each of the contrastive losses. Our results show that a model trained with LDCL can achieve sizeable improvements over baselines (Table 1). For a more enhanced representation, our proposed framework also exploits a surrogate label prediction (SLP) objective LSLP where the encoder Φ is jointly optimized for the emoji prediction task using cross entropy loss. Our employment of an SLP objective now allows us to weight the negatives in LLCL using classification probabilities from our main model rather than training an additional weighting model, another divergence from Suresh and Ong (2021). This new LCL framework is our LCL-LiT (for light LCL),8
5We experiment with a relatively sophisticated approach that learns class embeddings to capture the inter-class relations in Sec. 5, but find it to be sub-optimal.
6Equation for NPMI is in Appendix B.1. 7Note that LLCL operates over surrogate labels rather than task-specific downstream labels as in (Suresh and Ong, 2021), thereby allowing us to learn broad SM representations.
8The formula of LCL-LiT is the same as Eq. 3 (i.e., Loss of LCL).
giving us a lighter DCL loss that we call DCL-LiT:
LDCL−LiT = γLLCL−LiT + (1− γ)LCCL. (6)
Our sharing strategy where a single model is trained end-to-end on an overall objective incorporating negative class weighting should also improve our model efficiency (e.g., training speed, energy efficiency). Our ablation study in Sec. 5 confirms that using the main model as the weighing network is effective for overall performance. To mitigate effect of any catastrophic forgetting of token-level knowledge, the proposed framework includes an MLM objective defined by LMLM .9 The overall objective function of the proposed InfoDCL framework can be given by:
LInfoDCL = λ1LMLM + λ2LSLP + (1− λ1 − λ2)LDCL−LiT , (7)
where λ1 and λ2 are the loss scaling factors. We also employ a mechanism for randomly re-pairing an anchor with a new positive sample at the beginning of each epoch. We describe this epoch-wise repairing in Appendix B.4.

heading: 3.5 Data for Representation Learning
text: We exploit emojis as surrogate labels using an English language dataset with 31M tweets and a total of 1, 067 unique emojis (TweetEmoji-EN). In addition, we acquire representation learning data for (1) our experiments on three additional languages (i.e., Arabic, Italian, and Spanish) and to (2) investigate of the utility of hashtags as surrogate labels. More about how we develop TweetEmoji-EN and all our other representation learning data is in Appendix C.1.

heading: 3.6 Evaluation Data and Splits
text: In-Domain Data. We collect 16 English language Twitter datasets representing eight different SM tasks. These are (1) crisis awareness, (2) emotion recognition, (3) hateful and offensive language detection, (4) humor identification, (5) irony and sarcasm detection, (6) irony type identification, (7) sentiment analysis, and (8) stance detection. We also evaluate our framework on nine Twitter datasets, three from each of Arabic, Italian, and Spanish. More information about our English and multilingual datasets is in Appendix C.2.
9The Equations of LSLP and LMLM are listed in Appendix B.2 and B.3, respectively.
Out-of-Domain Data. We also identify eight datasets of SM involving emotion, sarcasm, and sentiment derived from outside the Twitter domain (e.g., data created by psychologists, debate fora, YouTube comments, movie reviews). We provide more information about these datasets in Appendix C.2.
Data Splits. For datasets without Dev split, we use 10% of the respective training samples as Dev. For datasets originally used in cross-validation, we randomly split into 80% Train, 10% Dev, and 10% Test. Table 7 in Appendix C describes statistics of our evaluation datasets

heading: 3.7 Implementation and Baselines
text: For experiments on English, we initialize our model with the pre-trained English RoBERTaBase.10 For multi-lingual experiments (reported in Appendix E.3), we use the pre-trained XLM-RoBERTaBase model (Conneau et al., 2020) as our initial checkpoint. More details about these two models are in Appendix D.1. We tune hyper-parameters of our InfoDCL framework based on performance on development sets of downstream tasks, finding our model to be resilient to changes in these as detailed in Appendix D.3. To evaluate on downstream tasks, we fine-tune trained models on each task for five times with different random seeds and report the averaged model performance. Our main metric is macro-averaged F1 score. To evaluate the overall ability of a model, we also report an aggregated metric that averages over the 16 in-domain datasets, eight out-of-domain tasks, and the nine multi-lingual Twitter datasets, respectively.
NPMI Weighting Matrix. We randomly sample 150M tweets from our original 350M Twitter dataset, each with at least two emojis. We extract all emojis in each tweet and count the frequencies of emojis as well as co-occurrences between emojis. To avoid noisy relatedness from low frequency pairs, we filter out emoji pairs (yi, ya) whose cooccurrences are less than 20 times. We employ Eq. 8 (Appendix B.1) to calculate NPMI for each emoji pair.
Baselines. We compare our methods to 11 baselines, as described in Appendix D.2.
10For short, we refer to the official released English RoBERTaBase as RoBERTa in the rest of the paper.

heading: 4 Main Results
text: Table 1 shows our main results. We refer to our models trained with LDCL (Eq. 5) and LInfoDCL (Eq. 7) in Table 1 as DCL and InfoDCL, respectively. We compare our models to 11 baselines on the 16 Twitter (in-domain) datasets and eight out-of-domain datasets.
In-Domain Results. InfoDCL outperforms Baseline (1), i.e., fine-tuning original RoBERTa, on each of the 16 in-domain datasets, with 1.93 average F1 improvement. InfoDCL also outperforms both the MLM and surrogate label prediction (SLP) methods with 1.19 and 0.46 average F1 scores, respectively. Our proposed framework is thus able to learn more effective representations for SM. We observe that both Mirror-BERT and SimCSE-Self negatively impact downstream task performance, suggesting that while the excessive uniformity they result in is useful for semantic similarity tasks (Gao et al., 2021; Liu et al., 2021a), it hurts downstream SM tasks.11 We observe that our proposed variant of SimCSE, SimCSE-Distant, achieves sizable improvements over both Mirror-BERT and SimCSE-
11The analyses in Sections 5 and E.6 illustrate this behavior.
Self (3.20 and 4.49 average F1, respectively). This further demonstrates effectiveness of our distantly supervised objectives. SimCSE-Distant, however, cannot surpass our proposed InfoDCL framework on average F1 over all the tasks. We also note that InfoDCL outperforms SCL, LCL, and WCL with 2.05, 1.87, and 1.90 average F1, respectively. Although our simplified model, i.e., DCL, underperforms InfoDCL with 0.20 average F1, it outperforms all the baselines. Overall, our proposed models (DCL and InfoDCL) obtain best performance in 14 out of 16 tasks, and InfoDCL acquires the best average F1. We further investigate the relation between model performance and emoji presence, finding that our proposed approach not only improves tasks involving high amounts of emoji content (e.g., the test set of EmoMoham has 23.43% tweets containing emojis) but also those without any emoji content (e.g., HateDav). 12 Compared to the original BERTweet, our InfoDCL-RoBERTa is still better (0.36 higher F1). This demonstrates not only effectiveness of our approach as compared to domain-specific models pre-trained simply with
12Statistics of emoji presence of each downstream task is shown in Table 7 in Appendix.
MLM, but also its data efficiency: BERTweet is pretrained with ∼ 27× more data (850M tweets vs. only 31M for our model). Moreover, the BERTweet we continue training with our framework obtains an average improvement of 0.77 F1 (outperforms it on 14 individual tasks). The results demonstrate that our framework can enhance the domain-specific PLM as well.
Out-of-Domain Results. InfoDCL achieves an average improvement of 1.01 F1 (F1 = 75.54) over the eight out-of-domain datasets compared to Baseline (1) as Table 1 shows. Our DCL and InfoDCL models also surpass all baselines on average, achieving highest on seven out of eight datasets. We notice the degradation of BERTweet when we evaluate on the out-of-domain data. Again, this shows generalizability of our proposed framework for leaning SM.
Significance Tests. We conduct two types of significance test on our results, i.e., the classical paired student’s t-test (Fisher, 1936) and Almost Stochastic Order (ASO) (Dror et al., 2019). The t-test shows that our InfoDCL-RoBERTa significantly (p < .05) outperforms 9 out of 11 baselines (exceptions are SimCSE-Distant and BERTweet) on the average scores over 16 in-domain datasets and 10 baselines (exception is SLP) on the average scores over eight out-of-domain datasets. ASO concludes that InfoDCL-RoBERTa significantly (p < .01) outperforms all 11 baselines on both average scores of in-domain and out-of-domain datasets. InfoDCL-BERTweet also significantly (p < .05 by t-test, p < .01 by ASO) outperforms BERTweet on the average scores. We report standard deviations of our results and significance tests in Appendix E.1.
Additional Results. Comparisons to Individual SoTAs. We compare our models on each dataset to the task-specific SoTA model on that dataset, acquiring strong performance on the majority of these as we show in Table 12, Sec. E.2 in Appendix. Beyond English. We also demonstrate effectiveness and generalizability of our proposed framework on nine SM tasks in three additional languages in Sec. E.3. Beyond Emojis. To show the generalizability of our framework to surrogate labels other than emojis, we train DCL and InfoDCL with hashtags and observe comparable gains (Sec. E.4). Beyond Sociopragmatics. Although the main objective of our proposed framework is to improve
model representation for SM, we also evaluate our models on two topic classification datasets and a sentence evaluation benchmark, SentEval (Conneau and Kiela, 2018). This allows us to show both strengths of our framework (i.e., improvements beyond SM) and its limitations (i.e., on textual semantic similarity). More about SentEval is in Appendix C.2, and results are in Sections E.5 and E.6.
Few-Shot Learning Results. Since DCL and InfoDCL exploit an extensive set of cues, allowing them to capture a broad range of nuanced concepts of SM, we hypothesize they will be particularly effective in few-shot learning. We hence fine-tune our DCL, InfoDCL, strongest two baselines, and the original RoBERTa with varying amounts of downstream data.13 As Table 2 shows, for in-domain tasks, with only 20 and 100 training samples per task, our InfoDCLRoBERTa strikingly improves 11.66 and 17.52 points over the RoBERTa baseline, respectively. Similarly, InfoDCL-RoBERTa is 13.88 and 17.39 over RoBERTa with 20 and 100 training samples for out-of-domain tasks. These gains also persist when we compare our framework to all other strong baselines, including as we increase data sample size. Clearly, our proposed framework remarkably alleviates the challenge of labelled data scarcity even under severely few-shot settings.14

heading: 5 Ablation Studies and Analyses
text: Ablation Studies. We investigate effectiveness of each of the ingredients in our proposed framework through ablation studies exploiting TweetEmoji-EN for pre-training. We evaluate on the 16 in-domain SM datasets with the same hyper-parameters identified in Sec. D.3 and report
13Data splits for few-shot experiments are in Appendix C.2. 14We offer additional few-shot results in Appendix E.7.
results over five runs. As Table 3 shows, InfoDCL outperforms all other settings, demonstrating the utility of the various components in our model. Results show the SLP objective is the most important ingredient in InfoDCL (with a drop of 0.80 average F1 when removed). However, when we drop both SLP and MLM objectives, DCL (our second best proposed model) only loses 0.20 F1 as compared to InfoDCL. Results also show that our proposed CCL is more effective than LCL: CCL is second most important component and results in 0.42 F1 drop vs. only 0.08 F1 drop when ablating LCL. Interestingly, when we remove both CCL and LCL, the model is relatively less affected (i.e., 0.19 F1 drop) than when we remove CCL alone. We hypothesize this is the case since CCL and LCL are two somewhat opposing objectives: LCL tries to make individual samples distinguishable across confusable classes, while CCL tries to keep the semantic relations between confusable classes. Overall, our results show the utility of distantly supervised contrastive loss. Although distant labels are intrinsically noisy, our InfoDCL is able to mitigate this noise by using CCL and LCL losses. Our epoch-wise re-pairing (EpW-RP) strategy is also valuable, as removing it results in a drop of 0.18 average F1. We believe EpW-RP helps regularize our model as we dynamically re-pair an anchor with a new positive pair for each training epoch. We also train an additional network to produce the weight vector, wi, in LCL loss as Suresh and Ong (2021) proposed instead of using our own main model to assign this weight vector end-to-end. We observe a slight drop of 0.02 average F1 with the additional model, showing the superiority of our end-to-end approach (which is less computational costly). We also adapt a simple self-augmentation method introduced by Liu et al. (2021a) to our dis-
tant supervision setting: given an anchor xi, we acquire a positive set {xi, xm+i, x2m+i, x3m+i} where xm+i is a sample with the same emoji as the anchor, x2m+i is an augmented version (applying dropout and masking) of xi, and x3m+i is an augmented version of xm+i. As Table 3 shows, InfoDCL+Self-Aug underperforms InfoDCL (0.38 F1 drop). We investigate further issues as to how to handle inter-class relations in our models and answer the following questions: Should we cluster or push apart the large number of fine-grained (correlated) classes? In previous works, contrastive learning is used to push apart samples from different classes. Suresh and Ong (2021) propose the LCL to penalize samples that is more confusable. In this paper, we hypothesize that we should also incorporate inter-class relations into learning objectives (our CCL). Hence, we introduce the PMI score into SCL to scale down the loss of a pair belonging to semantically related classes (emojis) as defined in Section 3.3 (which should help cluster our fine-grained classes). Here, we investigate an alternative strategy where we explore using the PMI scores as weights to scale up the loss of a pair with related labels (which should keep the fine-grained emoji classes separate). Hence, we set wyi,ya = 1 + Sim(yi, ya) where Sim(yi, ya) = max(0, npmi(yi, ya)). We train RoBERTa on 5M random samples from the training set of TweetEmoji-EN with the overall loss function in Eq. 7, one time using this new weighting method and another time using the weighting method used in all our reported models so far: wyi,ya = 1− Sim(yi, ya). Given these two ways to acquire wyi,ya in Eq. 4, we fine-tune the trained model on the 16 Twitter tasks. Our results in Table 4 show the penalizing strategy to perform lower than our original clustering strategy reported in all experiments in this paper. We also present their performance on each dataset in Table 5.
Can we use the emoji class embedding (ECEmb) for corpus-level weighting? We experiment with using the embedding of the emoji class
(EC-Emb) as an alternative weighting method in place of PMI. Namely, we train RoBERTa on SLP (using the training set of TweetEmoji-EN) for three epochs with a standard cross-entropy loss. We then extract weights of the last classification layer and use these weights as class embeddings, E = {e1, e2, . . . , eC}, where ei = Rd, d is hidden dimension (i.e., 768), and |C| is the size of classes (i.e., 1, 067). The correlation of each pair of emojis is computed using cosine similarity, i.e., Sim(yi, ya) = e⊤i ea ∥ei∥·∥ea∥ .
15 As Table 4 and 5 shows, using PMI scores performs slightly better than using class embeddings in both the clustering and penalizing strategies mentioned previously in the current section. For more intuition, we handpick three query emojis and manually compare the quality of similarity measures produced by both PMI and class embeddings for these. As Table 17 in Appendix shows, both PMI and EC-Emb are capable of capturing sensible correlations between emojis (although the embedding approach includes a few semantically distant emojis, such as the emoji ‘ ’ being highly related to ‘ ’).
Qualitative Analysis. To further illustrate the effectiveness of the representation learned by InfoDCL, we compare a t-SNE (Van der Maaten and Hinton, 2008) visualization of it to that of two strong baselines on two SM datasets.16 Fig. 2 shows that our model has clearly learned to cluster the samples with similar semantics and separate semantically different clusters before fine-tuning on the gold downstream samples, for both in-domain
15Self-similarity is set to 0. 16Note that we use our model representations without down-
stream fine-tuning.
and out-of-domain tasks. We provide more details about how we obtain the t-SNE vitalization and provide another visualization study in Appendix F.2.
Uniformity-Tolerance Dilemma. Following Wang and Liu (2021), we investigate uniformity and tolerance of our models using Dev data of downstream tasks.17 As Fig. 3 shows, unlike other models, our proposed DCL and InfoDCL models make a balance between uniformity and tolerance (which works best for SM).

heading: 6 Conclusion
text: We proposed InfoDCL, a novel framework for adapting PLMs to SM exploiting surrogate labels in contrastive learning. We demonstrated effectiveness of our framework on 16 in-domain and eight out-of-domain datasets as well as nine non-English datasets. Our model outperforms 11 strong baselines and exhibits strikingly powerful performance in few-shot learning.
17For details see Sec. G in Appendix.

heading: 7 Limitations
text: We identify the potential limitations of our work as follow: (1) Distant labels may not be available in every application domain (e.g., patient notes in clinical application), although domain adaptation can be applied in these scenarios. We also believe that distantly supervised contrastive learning can be exploited in tasks involving image and video where surrogate labels are abundant. (2) We also acknowledge that the offline NPMI matrix of our proposed CCL method depends on a dataset (distantly) labeled with multiple classes in each sample. To alleviate this limitation, we explore an alternative method that uses learned class embeddings to calculate the inter-class relations in Section 5. This weighting approach achieves sizable improvement over RoBERTa on 16 in-domain datasets, though it underperforms our NPMI-based approach. (3) Our framework does not always work on tasks outside SM. For example, our model underperforms self-supervised CL models, i.e., SimCSESelf and Mirror-BERT, on semantic textual similarity task in Appendix E.6. As we showed, however, our framework exhibits promising performance on some other tasks. For example, our hashtag-based model acquires best performance on the topic classification task, as shown in Appendix E.5.

heading: Ethical Considerations
text: All our evaluation datasets are collected from publicly available sources. Following privacy protection policy, all the data we used for model pretraining and fine-tuning are anonymized. Some annotations in the downstream data (e.g., for hate speech tasks) can carry annotator bias. We will accompany our data and model release with model cards. We will also provide more detailed ethical considerations on a dedicated GitHub repository. All our models will be distributed for research with a clear purpose justification.

heading: Acknowledgements
text: We gratefully acknowledge support from the Natural Sciences and Engineering Research Council of Canada (NSERC; RGPIN-2018-04267), the Social Sciences and Humanities Research Council of Canada (SSHRC; 435-2018-0576; 895-2020- 1004; 895-2021-1008), Canadian Foundation for Innovation (CFI; 37771), Digital Research Al-
liance of Canada (the Alliance),18 and UBC ARCSockeye.19 Any opinions, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSERC, SSHRC, CFI, the Alliance, or UBC ARC-Sockeye.

heading: Appendices

heading: A Survey of Contrastive Learning Frameworks.
text: There has been a flurry of recent contrastive learning frameworks introducing self-supervised, semisupervised, weakly-supervised, and strongly supervised learning objectives. These frameworks differ across a number of key dimensions: (i) type of the object (e.g., image, sentence, document), (ii) positive example creation method (e.g., same class as anchor, anchor with few words replaced with synonyms), (iii) negative example creation method (e.g., random sample, anchor with few words replaced with antonyms), (iv) supervision level (e.g., self, semi, weakly, hybrid, strong), and (v) weighing of negative samples (e.g., equal, confidencebased). Table 6 provides a summary of previous frameworks, comparing them with our proposed framework.

heading: B Method

heading: B.1 Normalized Point-Wise Mutual Information
text: The normalized point-wise mutual information (NPMI) (Bouma, 2009) between ya and yi. npmi(yi, ya) ∈ [−1, 1] is formulated as:
npmi(yi, ya) = ( log p(yi, ya)
p(yi)p(ya)
) /− log p(yi, ya). (8)
When npmi(yi, ya) = 1, ya and yi only occur together and are expected to express highly similar semantic meanings. When npmi(yi, ya) = 0, ya and yi never occur together and are expected to express highly dissimilar (i.e., different) semantic meanings. We only utilize NPMI scores of related class pairs, i.e., npmi(yi, ya) > 0. As the NPMI score of ya and yi is higher, the weight wyi,ya is lower. As a result of incorporating NPMI scores into the negative comparison in the SCL, we anticipate that the representation model would learn better inter-class correlations and cluster the related fine-grained classes.

heading: B.2 Surrogate Label Predication
text: Our proposed framework also exploits a surrogate label prediction (SLP) objective, where the encoder Φ is optimized for the surrogate label prediction task using cross entropy. Specifically, we pass the hidden representation hi through two feed-forward
layers with Tanh non-linearity in between and obtain the prediction ŷi. Then, the surrogate classification loss based on cross entropy can be formalized as:
LSLP = − 1
2m
2m∑
i=1
C∑
c=1
yi,c · log ŷi,c, (9)
where ŷi,c is the predicted probability of sample xi w.r.t class c.

heading: B.3 Masked Language Modeling Objective
text: Our proposed framework also exploits a MLM objective to mitigate the effect of catastrophic forgetting of the token-level knowledge. Following Liu et al. (2019), we randomly corrupt an input sentence by replacing 15% of its tokens with ‘[MASK]’ tokens. Given the corrupted input sequence, we then train our model to predict original tokens at masked positions. Formally, given an input sequence, xi = {t1, . . . , tn}, the loss function of MLM is formulated as:
LMLM = − 1
2m
2m∑
i=1
∑
tj∈mk(xi) log(p(tj |tcor(xi))),
(10) where mk(xi) indicates the set of masked tokens of the input sequence xi and cor(xi) denotes the corrupted input sequence xi.

heading: B.4 Epoch-Wise Re-Pairing
text: Rather than augmenting a batch D with using some data augmentation technique, in our framework, the positive sample xm+i of the anchor xi is a sample that uses the same emoji. To alleviate any potential
noise in our distant labels, we introduce an epochwise re-pairing (EpW-RP) mechanism where the pairing of a positive sample with a given anchor is not fixed for epochs: at the beginning of each epoch, we flexibly re-pair the anchor with a new positive pair xm+i randomly re-sampled from the whole training dataset using the same emoji as xi. This ensures that each anchor in a given batch will have at least one positive sample.20

heading: C Data

heading: C.1 Representation Learning Data and Pre-Processing.
text: Emoji Pre-Training Dataset. We normalize tweets by converting user mentions and hyperlinks to ‘USER’ and ‘URL’, respectively. We keep all the tweets, retweets, and replies but remove the ‘RT USER:’ string in front of retweets. We filter out short tweets (< 5 actual English word without counting the special tokens such as hashtag, emoji, USER, URL, and RT) to ensure each tweet contains sufficient context. Following previous works (Felbo et al., 2017; Barbieri et al., 2018; Bamman and Smith, 2015), we only keep the tweet that contains only a unique type of emoji (regardless of the number of emojis) and that uses a emoji at the end of the tweet. We then extract the emoji as a label of the tweet and remove the emoji from the tweet. We exclude emojis occurring less than 200 times, which gives us a set of 1, 067 emojis in 32M tweets. Moreover, we remove few tweets overlapped with Dev and Test sets of our evalua-
20Note that each sample in the training dataset is used only once at each epoch, either as the anchor or as a positive sample of the anchor.
tion tasks by Twitter ID and string matching. We refer to this dataset as TweetEmoji-EN and split it into a training (31M) and validation (1M) set.
Hashtag Pre-Training Dataset. We also explore using hashtags as surrogate labels for InfoDCL training. Following our data pre-processing procedure on TweetEmoji-EN, we randomly extract 300M English tweets each with at least one hashtags from a larger in-house dataset collected between 2014 and 2020. We only keep tweets that contain a single hashtag used at the end. We then extract the hashtag as a distant label and remove it from the tweet. We exclude hashtags occurring less than 200 times, which gives us a set of 12, 602 hashtags in 13M tweets. We refer to this dataset as TweetHashtag-EN and split the tweets into a training set (12M) and a validation (1M) set.
Multilingual Emoji Pre-Training Dataset. We collect a multilingual dataset to train multilingual models with our proposed framework. We apply the same data pre-processing and filtering conditions used on English data, and only include tweets that use the 1, 067 emojis in TweetEmojiEN. We obtain 1M tweets from our in-house dataset for three languages, i.e., Arabic, Italian, and Spanish.21 We refer to these datasets as TweetEmoji-AR, TweetEmoji-IT, and TweetEmoji-ES. We also randomly extract 1M English tweets from our TweetEmoji-EN and refer to is as TweetEmoji-EN-1M. We then combine these four datasets and call the combined dataset TweetEmoji-Multi.

heading: C.2 Evaluation Data
text: In-Domain Datasets. We collect 16 English witter datasets representing eight different SM tasks to evaluate our models, including (1) crisis awareness task (Olteanu et al., 2014), (2) emotion recognition (Mohammad et al., 2018), (3) hateful and offensive language detection (Waseem and Hovy, 2016; Davidson et al., 2017; Basile et al., 2019; Zampieri et al., 2019a), (4) humor identification (Meaney et al., 2021), (5) irony and sarcasm detection (Hee et al., 2018; Riloff et al., 2013; Ptácek et al., 2014; Rajadesingan et al., 2015; Bamman and Smith, 2015), (6) irony type identification (Hee et al., 2018) (7) sentiment analysis (Thelwall et al., 2012; Rosenthal et al., 2017), and (8) stance detection (Mohammad et al., 2016). We present the
21However, we were only able to obtain 500K Italian tweets satisfying our conditions.
distribution, the number of labels, and the short name of each dataset in Table 7.
Out-of-Domain Datasets. We evaluate our model on downstream SM tasks from diverse social media platforms and domains. For emotion recognition task, we utilize (1) PsychExp (Wallbott and Scherer, 1986), a seven-way classification dataset of self-described emotional experiences created by psychologists, and (2) GoEmotion (Demszky et al., 2020), a dataset of Reddit posts annotated with 27 emotions (we exclude neutral samples). For sarcasm detection task, we use two datasets from the Internet Argument Corpora (Walker et al., 2012; Oraby et al., 2016) that posts from debate forums. For sentiment analysis, we utilize (1) five-class and binary classification versions of the Stanford Sentiment Treebank (Socher et al., 2013) (SST-5 and SST-2) that include annotated movie reviews with sentiment tags, (2) movie review (MR) for binary sentiment classification (Pang and Lee, 2005), and (3) SentiStrength for YouTube comments (SSYouTube) (Thelwall et al., 2012).
Multilingual Datasets. As explained, to evaluate the effectiveness of our framework on different languages, we collect nine Twitter tasks in three languages: Arabic, Italian, and Spanish. For each language, we include three emotion-related tasks, (1) emotion recognition(Abdul-Mageed et al., 2020; Bianchi et al., 2021; Mohammad et al., 2018), (2) irony identification (Ghanem et al., 2019; Cignarella et al., 2018; Ortega-Bueno et al., 2019), and (3) offensive language/hate speech detection (Mubarak et al., 2020; Bosco et al., 2018; Basile et al., 2019).
Few-Shot Data. We conduct our few-shot experiments only on our English language downstream data. We use different sizes from the set {20, 100, 500, 1, 000} sampled randomly from the respective Train splits of our data. For each of these sizes, we randomly sample five times with replacement (as we report the average of five runs in our experiments). We also run few-shot experiments with varying percentages of the Train set of each task (i.e., 1%, 5%, 10%, 20% . . . 90%). We randomly sample five different training sets for each percentage, evaluate each model on the original Dev and Test sets, and average the performance over five runs.
Topic Classification Datasets. To investigate the generalizability of our models, we evaluate our models on two topic classifcation datasets: AGNews (Corso et al., 2005) and TopicDao (Daouadi et al., 2021). Given a news title and a short description, AGNews classifies the input text into four categories, including world, sports, business, and Sci/Tech. TopicDao identifies if a given tweet is related to politics or not. The data distribution is presented in Table 7.
SentEval. We utilize SentEval benchmark (Conneau and Kiela, 2018)22, a toolkit for evaluating the quality of sentence representations, to evaluate on seven semantic textual similarity (STS) datasets and eight transfer learning datasets. Seven STS datasets include STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), SICKRelatedness (Marelli et al., 2014), and STS Benchmark (Cer et al., 2017). Eight transferring classifi-
22https://github.com/facebookresearch/ SentEval
cation datasets consist of four sentiment analysis (i.e., movie review (MR) (Pang and Lee, 2005), product review (CR) (Hu and Liu, 2004), SST2, and SST5 (Socher et al., 2013)), subjectivity detection (SUBJ) (Pang and Lee, 2004), opinion polarity (MPQA) (Wiebe et al., 2005), question-type classification (TREC) (Voorhees and Tice, 2000), and paraphrase detection (MRPC) (Dolan and Brockett, 2005). The data distribution and evaluation metrics are presented in Table 8. The STS datasets only have test set since they do not train any model. Tasks of MR, CR, SUBJ and MPQA are evaluated by nested 10-fold cross-validation, TREC and MRPC use cross-validation, and two SST datasets have standard development and test sets.

heading: D Experiment
text: D.1 Implementation
For experiments on English language datasets, we initialize our model with a pre-trained English RoBERTaBase (Liu et al., 2019) model from Hug-
gingface’s Transformers (Wolf et al., 2020) library. RoBERTaBase consists of 12 Transformer Encoder layers, 768 hidden units each, 12 attention heads, and contains 110M parameters in entire model. RoBERTa uses a byte-pair-encoding vocabulary with a size of 50, 265 tokens. RoBERTa was pretrained on large English corpora (e.g., Bookcorpus) with the MLM objective. In accordance with convention (Liu et al., 2019; Gao et al., 2021), we pass the hidden state corresponding to the ‘[CLS]’ token from the last layer through a feedforward layer with hidden size of 768 and a hyperbolic tangent function and, then, use the output as the sentence-level embedding, hi. For the classification objective, we feed hi into a feedforward layer with hidden size of 1, 067 23, a softmax function and a dropout of 0.1. For multilingual experiments, we utilize the pre-trained XLM-RoBERTaBase model24 (Conneau et al., 2020) as our initial checkpoint. XLM-RBase has the same architecture as RoBERTa. XLM-R includes a vocabulary of 250, 002 BPE tokens for 100 languages and is pre-trained on 2.5TB of filtered CommonCrawl.
We fine-tune pre-trained models on each downstream task for five times with different random seeds and report the averaged model performance. Our main metric is macro-averaged F1 score. To evaluate the overall ability of a model, we also report an aggregated metric that averages over the 16 Twitter datasets, eight out-of-domain tasks, and the
23The number of Emoji classes is 1, 067. 24For short, we refer to the official released XLM-
RoBERTaBase as XLM-R in the rest of the paper.
nine multi-lingual Twitter datasets, respectively.
NPMI weighting matrix. We randomly sample 150M tweets from the 350M tweets with at least one emoji each. We extract all emojis in each tweet and count the frequencies of emojis as well as cooccurrences between emojis. To avoid noisy relatedness from low frequency pairs, we filter out emoji pairs, (yi, ya), whose co-occurrences are less than 20 times or 0.02× frequency of yi. We employ Eq. 8 to calculate NPMI for each emoji pair. Similarly, we calculate the NPMI weighting matrix using 150M with at least one hashtag each and filtering out low frequency pairs.

heading: D.2 Baselines
text: We compare our proposed framework against 11 strong baselines, which we describe here. (1) RoBERTa: The original pre-trained RoBERTa, fine-tuned on downstream tasks with standard cross-entropy loss. (2) MLM: We continue pretraining RoBERTa on our pre-training dataset (TweetEmoji-EN for emoji-based experiment and TweetHashtag-EN for hashtag-based experiment) with solely the MLM objective in Eq. 10 (Appendix B.3), then fine-tune on downstream tasks. (3) Emoji-Based MLM (E-MLM): Following Corazza et al. (2020), we mask emojis in tweets and task the model to predict them, then finetune on downstream tasks.25 (4) SLP. A RoBERTa model fine-tuned on the surrogate label prediction task (e.g., emoji prediction) (Zhang and AbdulMageed, 2022) with cross-entropy loss, then finetuned on downstream tasks. Supervised Contrastive Learning: We also compare to state-ofthe-art supervised contrastive fine-tuning frameworks. We take the original pre-trained RoBERTa and fine-tune it on each task with (5) SCL (Gunel et al., 2021) and (6) LCL (Suresh and Ong, 2021), respectively. Both works combine supervised contrastive loss with standard cross-entropy as well as augmentation of the training data to construct positive pairs. We follow the augmentation technique used in Suresh and Ong (2021), which replaces 30% of words in the input sample with their synonyms in WordNet dictionary (Miller, 1995). SelfSupervised Contrastive Learning. We further train RoBERTa on different recently proposed selfsupervised contrastive learning frameworks. (7)
25For hashtag-based experiment, we adapt this method to masking hashtags in tweets and refer to it as Hashtag-based MLM (H-MLM).
SimCSE-Self. Gao et al. (2021) introduce SimCSE where they produce a positive pair by applying different dropout masks on input text twice. We similarly acquire a positive pair using the same droput method. (8) SimCSE-Distant. Gao et al. (2021) also propose a supervised SimCSE that utilizes gold NLI data to create positive pairs where an anchor is a premise and a positive sample is an entailment. Hence, we adapt the supervised SimCSE framework to our distantly supervised data and construct positive pairs applying our epochwise re-pairing strategy. Specifically, each anchor has one positive sample that employs the same emoji as the anchor in a batch. (9) MirrorBERT. (Liu et al., 2021a) construct positive samples in Mirror-BERT by random span masking as well as different dropout masks. After contrastive learning, sentence-encoder models are finetuned on downstream tasks with the cross-entropy loss. (10) Weakly-supervised Contrastive Learning. We simplify and adapt the WCL framework of Zheng et al. (2021) to language: We first encode unlabelled tweets to sequence-level representation vectors using the hidden state of the ‘[CLS]’ token from the last layer of RoBERTa. All unlabelled tweets are clustered by applying k-means to their representation vectors. We then use the cluster IDs as weak labels to perform an SCL to pull the tweets assigned to the same cluster closer. Following Zheng et al. (2021), we also include an SSCL loss by augmenting the positive sample of an anchor using random span as well as dropout masking. We jointly optimize the SCL and SSCL losses in our implementation. (11) DomainSpecific PLM (BTw): We compare to the SoTA domain-specific PLM, BERTweet (Nguyen et al., 2020). BERTweet was pre-trained on 850M tweets with RoBERTaBase architecture. We download the pre-trained BERTweet checkpoint from Huggingface’s Transformers (Wolf et al., 2020) library and fine-tune it on each downstream task with crossentropy loss. More details about hyper-parameters of these baselines are in Appendix D.3.

heading: D.3 Hyper-Parameters
text: InfoDCL Training Hyper-Parameters. For hyper-parameter tuning of our proposed InfoDCL framework, we randomly sample 5M tweets from the training set of our TweetEmoji-EN. We continue training the pre-trained RoBERTa for three epochs with Adam optimizer with a weight de-
cay of 0.01 and a peak learning rate of 2e − 5. The batch size is 128, and the total number of input samples is 256 after constructing positive pairs. As Gao et al. (2021) find contrastive learning is not sensitive to the learning rate nor batch size when further training a PLM, we do not finetune these (i.e., the learning rate and batch size) in this paper. Following (Liu et al., 2019), we mask 15% of tokens for our MLM objective. We fine-tune the loss scaling weights λ1 in a set of {0.1, 0.3, 0.4}, λ2 in a set of {0.1, 0.3, 0.5}, and γ in a set of {0.1, 0.3, 0.5, 0.7, 0.9}. To reduce search space, we use the same temperature value for the τ in Eq. 3 and Eq. 4 and fine-tune in a set of {0.1, 0.3, 0.5, 0.7, 0.9}. We use grid search to find the best hyper-parameter set and evaluate performance on the Dev set of the 15 English language Twitter datasets (excluding SentiThel). 26 We select the best hyper-parameter set that achieves the best macro-F1 averaged over the 15 downstream tasks. Our best hyper-parameter set is λ1 = 0.3, λ2 = 0.1, γ = 0.5, and τ = 0.3. As Figure 4 shows, our model is not sensitive to changes of these hyper-parameters, and we observe that all the differences are less than 0.45 comparing to the best hyper-parameter set. Finally, we continue training RoBERTa/BERTweet on the full training set of TweetEmoji-EN with InfoDCL framework and best hyper-parameters. We train InfoDCL model for three epochs and utilize 4 Nvidia A100 GPU (40GB each) and 24 CPU cores. Each epoch takes around 7 hours.
Downstream Task Fine-Tuning HyperParameters. Furthermore, we take the model trained with the best hyper-parameters and search the best hyper-parameter set of downstream task fine-tuning. We search the batch size in a set of {8, 16, 32, 64} and the peak learning rate in a set of {2e − 5, 1e − 5, 5e − 6}. We identify the best fine-tuning hyper-parameters based on the macro-F1 27 on Dev sets averaged over the 16 English language Twitter datasets. Our best hyper-parameters for fine-tuning is a learning rate of 1e − 5 and a batch size of 32. For all the downstream task fine-tuning experiments in this paper, we train a model on the task for 20 epochs with early stop (patience = 5 epochs). We use the
26We fine-tune the learned model on each downstream task with an arbitrary learning rate of 5e − 6, a batch size of 16, and a training epoch of 20. The performance is macro-F1 over three runs with random seeds.
27We run three times and use the mean of them.
same hyper-parameters identified in this full data setting for our few-shot learning. For each dataset, we fine-tune for five times with a different random seed every time, and report the mean macro-F1 of the five runs. Each downstream fine-tuning experiment use a single Nvidia A100 GPU (40GB) and 4 CPU cores.
Baseline Hyper-Parameters. Our Baseline (1) is directly fine-tuning RoBERTa on downstream tasks. We fine-tune Baseline (1) hyper-parameters as follows: The batch size is chosen from a set of {8, 16, 32, 64} and the peak learning rate in a set of {2e−5, 1e−5, 5e−6}. The best hyper-parameters for RoBERTa fine-tuning is a learning rate of 2e−5 and a batch size of 64.
For Baseline (2-3), we further pre-train the RoBERTa model for three epochs (same as our InfoDCL) with the MLM objective with an arbitrary learning rate of 5e − 5 and a batch size of 4, 096. We mask 15% of tokens in each input tweet. For Baseline (3), we give priority to masking emojis in a tweet: if the emoji tokens are less than 15%, we then randomly select regular tokens to complete the percentage of masking to the 15%. Baseline (4) is about surrogate label prediction (with emojis). We also train Baseline (4) for three epochs with a learning rate of 2e − 5 and a batch size of 4, 096. After training, models are fine-tuned on downstream tasks using the same hyper-parameters as our proposed model.
Baselines (5-7). SimCSE (Gao et al., 2021) was trained in two setups, i.e., self-supervised and supervised by label data. We also train RoBERTa on both settings. For self-supervised SimCSE, we train RoBERTa on our pre-training dataset for three epochs with a learning rate of 2e− 5, a batch size of 256, and τ of 0.05. For the distantly-supervised SimCSE, we construct positive pairs as described in Section B.4. Similar to self-supervised SimCSE,
we train RoBERTa for three epochs with a learning rate of 2e − 5 but with a batch size of 128. 28 The pre-training of Mirror-BERT is similar to the pre-training of self-supervised SimCSE. We set the span masking rate of k = 3, a temperature of 0.04, a learning rate of 2e − 5, and a batch size of 256. Trained models, then, are fine-tuned on downstream tasks. For downstream task finetuning with baselines 2-7, we use the same hyperparameters identified with InfoDCL downstream task fine-tuning.
Baselines (8-9). SCL (Gunel et al., 2021) and LCL (Suresh and Ong, 2021) directly fine-tune on downstream tasks with cross-entropy loss. We reproduce these two methods on our evaluation tasks. For SCL, we follow Gunel et al. (2021) and fine-tune each task with a temperature of τ = 0.3, a SCL scaling weighting of 0.9, and a learning rate of 2e− 5. For LCL, we fine-tune each task with a temperature τ of 0.3, a LCL scaling weighting of 0.5, and a learning rate of 2e− 5.
Baselines (10). We implement WCL (Zheng et al., 2021) to continue train RoBERTa with our emoji dataset. We remove all emojis in the 31M tweets and encode tweets using the hidden state of ‘[CLS]’ token from the last layer of RoBERTa. The tweets are then clustered by k-means clustering algorithm.29 For hyper-parmeter tuning of WCL, we randomly sample 5M tweets from the training set of TweetEmoji-EN and train a model for three epochs with different hyperparmeter sets. We search the number of clusters in a set of {200, 500, 1067, 2000} and temperature τ in a set of {0.1, 0.3}. To reduce the search space, we use the same temperature value for SSCL and SCL losses. We evaluate performance on the Dev set of the 16 English language Twitter
28After pairing, each batch include 256 unique tweets. 29We use mini-batch k-means clustering from scikit-
learn (Pedregosa et al., 2011).
datasets 30 and find the best hyper-parameter set is k = 1067 and τ = 0.1. We then train WCL on the TweetEmoji-EN dataset for three epochs with our best hyper-parameters and fine tune the model on 24 downstream tasks with the same hyperparameters identified for InfoDCL downstream fine-tuning.31
Baseline (11). We fine-tune BERTweet with hyperparameters utilized in (Nguyen et al., 2020) that are a fixed learning of 1e− 5 and a batch size of 32.

heading: Multi-Lingual Experiment Hyper-Parameters.
text: For multi-lingual experiments, we utilize the pretrained XLM-RoBERTaBase model (Conneau et al., 2020) as our initial checkpoint. We continue training XLM-R on multi-lingual tweets with our framework and the best hyperparameters identified for English. For the downstream fine-tuning, we use as same as the best hyperparameters identified for English tasks.
Hahstag Experiment Hyper-Parameters. For the hashtag-based experiments presented in Section E.4, we use the same hyper-parameter optimization set up to find the best hyper-parameter set for hashtag-based models. The best hyperparameter set for hashtag-based models is λ1 = 0.4, λ2 = 0.1, γ = 0.1, and τ = 0.1. We then use the same downstream fine-tuning hyper-parameters identified with emoji-based InfoDCL for downstream task.

heading: E Results

heading: E.1 Standard Deviation and Significance Tests
text: Table 10 shows the standard deviations of our emoji-based InfoDCl models and all baselines over five runs. We conduct two significance tests on our results, i.e., the classical paired student’s ttest (Fisher, 1936) and Almost Stochastic Order (ASO) (Dror et al., 2019) (better adapts to results of neural networks). As we pointed out earlier, we run each experiment five times with different random seeds. Hence, we conduct these two significance tests by inputting the obtained five evaluation scores on the Test set. Table 11 presents p-values for t-test and minimal distance ϵ at significance level of 0.01 for ASO test. We also conduct significance tests on the results of individual tasks, finding that our InfoDCL-RoBERTa significantly (p < 0.05) improves the original RoBERTa on 13 (out of 24) and 24 (out of 24) datasets based on t-test and ASO, respectively. InfoDCLRoBERTa also significantly (p < 0.05) outperforms BERTweet (the strongest baseline) on 10 (out of 24) and 15 (out of 24) tasks based on t-test and ASO, respectively.

heading: E.2 Comparisons to Individual SoTAs.
text: Although the focus of our work is on producing effective representations suited to the whole class of SM tasks, rather than to one or another of these tasks, we also compare our models on each dataset to other reported task-specific SoTA models on that particular dataset in Table 12. We compare our methods on each dataset to other reported task-specific SoTA models on that particular dataset as shown. Due to diverse metrics used in previous studies, we compare models of each task reporting the corresponding metric of the SoTA method. Some SoTA models are trained on different data splits or use different evaluation approaches (e.g., Olteanu et al. (2014) is evaluated by cross-validation). To provide meaningful comparisons, we thus fine-tune BERTweet on our splits and report against our models. Our InfoDCL-RoBERTa outperform SoTA on 11 out of 16 in-domain datasets and four out of eight out-ofdomain datasets. We achieve the best average score over 16 in-domain datasets applying our model
30We fine-tune the trained WCL model with a learning rate of 1e− 5 and a batch size of 32.
31For hashtag-based experiment, we use the same hyperparameters.
on BERTweet. Further training RoBERTa with our framework obtains the best average score across the eight out-of-domain datasets. We note that some SoTA models adopt task-specific approaches and/or require task-specific resources. For example, Ke et al. (2020) utilize SentiWordNet to identify the sentiment polarity of each word. In this work, our focus on producing effective representations suited for the whole class of SM tasks, rather than one or another of these tasks. Otherwise, we hypothesize that task-specific approaches can be combined with our InfoDCL framework to yield even better performance on individual tasks.

heading: E.3 Multilingual Tasks
text: We also investigate the effectiveness of our proposed model on multilingual tasks. Table 13 shows the performance on nine downstream tasks in three different languages. Here, we continue training XLM-R with our proposed objectives. We experiment with three settings: (1) English only: training on the TweetEmoji-1M and evaluating on the nine multilingual datasets, (2) Target mono-lingual: training on each 1M mono-lingual tweets in the target language independently (i.e., TweetEmoji-AR for Arabic, TweetEmoji-IT for Italian, and TweetEmoji-ES for Spanish) and evaluating on the respective dataset corresponding to the same language as training data, and (3) Multilingual: training on the TweetEmoji-Multi dataset and evaluating on the nine multilingual datasets. We still use the NPMI weighting matrix generated from English tweets in these experiments. 32 Table 13 shows that our models outperform the original XLM-R on all the datasets and obtains improvements of 1.44 and 0.85 average F1 across the nine datasets under the multilingual and target mono-lingual settings, respectively. Training on English mono-lingual data helps four datasets, but cannot benefit all the nine non-English datasets on average. Compared to previous SoTA models, our proposed methods outperform these on six out of
32We plan to explore generating the NPMI weighting matrix from mutlilingual data in future work.
nine datasets. 33 These results demonstrate that our methods are not only task-agnostic within the realm of SM tasks, but also language-independent.

heading: E.4 Using Hashtag as Distant Supervision
text: As Table 14 presents, our proposed framework also can enhance the representation quality using hashtags as distantly supervised labels. InfoDCLRoBERTa, the model further training RoBERTa on the training set of TweetHashtag-EN with our framework, obtains average F1 of 77.36 and
33For Emo-esMoham, we use fine-tuning XLM-R as SoTA model because we convert the intensity regression task to a emotion classification and there is no SoTA model.
75.43 across the 16 in-domain and eight out-ofdomain datasets, respectively. Comapred to baselines, our DCL obtains the best performance average F1 score across 16 in-domain datasets ( F1 = 77.64). InfoDCL-BERTweet, the further pre-trained BERTweet on the training set of TweetHashtag-EN with our framework, obtains average F1 of 78.29 and 74.44 across the 16 in-domain and eight out-of-domain datasets, respectively.

heading: E.5 Topic Classification
text: We fine-tune baselines and our models on two topic classification datasets and report macro F1 scores in Table 15. We find that our hashtagbased InfoDCL model acquires best performance on both datasets, for AGNews F1 = 97.42, and for TopicDao F1 = 94.80. These results indicate that our framework can also effectively improve topic classification when we use hashtags as distant labels.

heading: E.6 SentEval
text: Each STS dataset includes pairs of sentences each with a gold semantic similarity score ranging from 0 to 5. We encode each sentence by the hidden state of ‘[CLS]’ token from the last Transformer encoder layer. We then calculate the Spearman’s correlation between cosine similarity of sentence embeddings

heading: Emoji-based Hashtag-based Model AGN Topic Ave Model AGN Topic Ave
text: and the gold similarity score of each pair. Same as Mirror-BERT (Liu et al., 2021a) and SimCSE (Gao et al., 2021), we report the overall Spearman’s correlation. For transfer learning tasks, we follow the evaluation protocal of SentEval, where a trainable logistic regression classifier is added on top of a frozen encoder that is an PLM. We report classification accuracy of eight transfer learning datasets in Tables 16. Although our InfoDCL underperforms Mirror-BERT on all STS datasets, but it still outperforms than Baseline 1, 2, and 3. Our InfoDCL is not designed to improve STS task but it does not
hurt performance compared to Baseline 2. Moreover, our InfoDCL achieves the best average performance on eight transferring datasets. We note that four datasets are SM tasks. Only regarding the other four non-SM tasks, our InfoDCL model still outperforms most baselines and achieves the second best performance on average, which is only 0.40 F1 points lower than Mirror-BERT.

heading: E.7 Few Shot Learning
text: Since InfoDCL exploits an extensive set of cues in the data that capture a broad range of finegrained SM concepts, we hypothesize it will be also effective in few-shot learning. Hence, we test this hypothesis for both in-domain and outof-domain tasks. Figure 5 and Table 19 compare our models to three strong baselines when they are trained with different percentages of training samples. Results show that our proposed InfoDCL model always outperforms all baselines on average F1 scores across both in-domain and out-of-domain tasks. For 16 in-domain tasks, our InfoDCL-RoBERTa remarkably surpasses the RoBERTa baseline with a sizable 12.82 average F1 scores when we only provide 1% training data from downstream tasks. Compared to other strong baselines, fine-tuning BERTweet and SimCSEDistant (also our method), InfoDCL-RoBERTa outperforms these with 12.91 and 3.55 average F1 scores, respectively, when we use 1% training data for downstream fine-tuning. With only 5% of gold data, InfoDCL-RoBERTa improves 5.76 points over the RoBERTa baseline. For eight outof-domain tasks, InfoDCL-RoBERTa outperforms the RoBERTa, BERTweet, and SimCSE-Distant baselines with 16.23, 15.52, and 2.89 average F1 scores, respectively, when the models are only finetuned on 1% training data of downstream tasks. As Figure 5b and Table 19 show, InfoDCL-RoBERTa consistently outperforms all the baselines given any percentage of training data. Tables 20, 21, 22, 23, 24, and 25, respectively, present the performance of RoBERTa, BERTweet, SimCSE-Distant, DCL, InfoDCL-RoBERTa, InfoDCL-BERTweet on all
our 24 English downstream datasets and various few-shot settings.

heading: F Analyses

heading: F.1 Model Analysis
text: Table 17 shows that both PMI and EC-Emb are capable of capturing sensible correlations between emojis (although the embedding approach includes a few semantically distant emojis, such as the emoji ‘ ’ being highly related to ‘ ’).

heading: F.2 Qualitative Analysis
text: We provide a qualitative visualization analysis of our model representation. For this purpose, we use our InfoDCL-RoBERTa to obtain representations of samples in the TweetEmoji-EN’s validation set (‘[CLS]’ token from the last encoder layer) then average the representations of all tweets with the same surrogate label (emoji). We then project these emoji embeddings into a two-dimensional space using t-SNE. As Fig. 6 shows, we can observe a number of distinguishable clusters. For instance, a cluster of love and marriage is grouped in the left region, unhappy and angry faces are in the right side, and food at the bottom. We can also observe sensible relations between clusters. For instance, the cluster of love and marriage is close to the cluster of smiling faces but is far away from the cluster of unhappy faces. In addition, the cluster of aquatic animals (middle bottom) is close to terrestrial animals while each of these is still visually distinguishable. We also note that emojis which contain the same emoji character but differ in skin tone are clustered together. An example of these is emojis of Santa Claus (left bottom). This indicates
that our InfoDCL model has meticulously captured the relations between the emoji surrogate labels.

heading: G Uniformity and Tolerance
text: Wang and Liu (2021) investigate representation quality measuring the uniformity of an embedding distribution and the tolerance to semantically similar samples. Given a dataset D and an encoder Φ, the uniformity is based on a gaussian potential kernel introduced by Wang and Isola (2020) and is formulated as:
Luniformity = log E xi,xj∈D
[e−t||Φ(xi)−Φ(xj)|| 2 2 ],
(11) where t = 2. Wang and Liu (2021) use −Luniformity as the uniformity metric, thus a higher uniformity score indicates that the embedding distribution is closer to a uniform distribution.
The tolerance metric measures the mean of similarities of samples belonging to the same class, which defined as:
Tolerance = E xi,xj∈D
[(Φ(xi) TΦ(xj)) ·Il(xi)=l(xj)], (12)
where l(xi) is the supervised label of sample xi. Il(xi)=l(xj) is an indicator function, giving the value of 1 for l(xi) = l(xj) and the value of 0 for l(xi) ̸= l(xj). In our experiments, we use gold development samples from our downstream SM datasets.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Sec. 7
3 A2. Did you discuss any potential risks of your work? Section of Ethical Considerations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Sec. 1
3 A4. Have you used AI writing assistants when working on this paper? We use Grammarly to assist with grammar correction.
B 3 Did you use or create scientific artifacts? Section C.2
3 B1. Did you cite the creators of artifacts you used? Section C.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section of Ethical Considerations
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section C.2
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 3.6 and C.2
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3.5, 3.6, C1, and C.2
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.5, 3.6, C1, and C.2
C 3 Did you run computational experiments? Section 3.7, D.2, and D.3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 3.7, D.2, and D.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 3.7, D.2, and D.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4, 5, and E
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section D and E
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Compared to previous SoTA models, our proposed methods outperform these on six out of
32We plan to explore generating the NPMI weighting matrix from mutlilingual data in future work. nine datasets. 33 These results demonstrate that our methods are not only task-agnostic within the realm of SM tasks, but also language-independent