mCPT at SemEval-2023 Task 3: Multilingual Label-Aware Contrastive Pre-Training of Transformers for Few- and Zero-shot Framing Detection
Year: 2023

Abstract:
This paper presents the winning system for the zero-shot Spanish framing detection task, which also achieves competitive places in eight additional languages. The challenge of the framing detection task lies in identifying a set of 14 frames when only a few or zero samples are available, i.e., a multilingual multi-label fewor zero-shot setting. Our developed solution employs a pre-training procedure based on multilingual Transformers using a label-aware contrastive loss function. In addition to describing the system, we perform an embedding space analysis and ablation study to demonstrate how our pre-training procedure supports framing detection to advance computational framing analysis.

heading: 1 Introduction
text: Approaches for computational framing detection are diverse (Ali and Hassan, 2022), as the framing concept itself is often just casually defined (Entman, 1993). Consequently, framing detection is challenging on its own, but also suffers from a lack of sufficient data (Kwak et al., 2020), especially in multilingual settings. The SemEval 2023 Task 3 Subtask 2 (Piskorski et al., 2023) aims at predicting 14 distinct media frames (Boydstun et al., 2013) present within news articles in 9 languages. Due to label imbalances, as a result of the high dimension of the label space compared to the number of samples, traditional paradigms, e.g., per-label binary classification, do not apply well to the given setting without adaptions (Tarekegn et al., 2021).
We introduce mCPT, the label-aware Contrastive Pre-training of Transformers based on a multilingual encoder model (original team name on the leaderboard1: PolarIce). We exploit two features
∗equal contribution †corresponding author
1https://propaganda.math.unipd.it/ semeval2023task3/SemEval2023testleaderboard.html
of the task: (i) multi-label information and (ii) multilingual data for pre-training.
First, we leverage the label information by adopting a contrastive loss function, i.e., HeroConLoss (Zheng et al., 2022), for natural language processing that optimizes the embedding space with respect to the similarities of the label space. Therefore, samples with more similar labels occupy similar regions in the embedding space, whereas mostly dissimilar samples regarding their shared labels are pushed apart (refer to Figures 1 top and 2a).
Second, we design a custom two-phase procedure with multiple stages for multilingual training to maximize the available data (see Figure 1). In phase one, we train on all languages, while in phase two, we further fine-tune the model on the target language if such data exist i.e. few-shot setting, or continue training on all languages if not i.e. zeroshot setting.
Our system performs competitively (top 10) on
941
all six few-shot (i.e., English, German, French, Italian, Polish, and Russian) and three zero-shot (i.e., Spanish, Greek, and Georgian) settings, beating the baselines on all languages. On Spanish, which is the only zero-shot language with a common language family and alphabet as the training languages, our system is the winning contribution with a Micro-F1 of 0.571 (compared to 0.120 of the baseline). Therefore, we argue that our system generalizes well to unseen data, even when no training data is available in similar target languages.
In sum, our contribution is three-fold2:
C1 We adopt a multi-label contrastive loss function for natural language processing to optimize the embeddings of textual data. C2 We describe a two-phase multi-stage training procedure for multilingual scenarios with limited data, i.e., few- and zero-shot predictions. C3 We demonstrate the effectiveness of our winning system for framing detection supported by embedding and ablation studies.

heading: 2 Related Work
text: Framing Detection. According to Entman (1993), to frame is to select and emphasize some aspects of reality to encourage particular interpretations. That is, messages centered around a common topic may draw the receiver’s attention to distinct features, thus suggesting different courses of action, causal interpretations, etc. As such, computational framing detection requires natural language processing (NLP) methods that capture nuances of how content is presented rather than just what topic is present. Therein, studies focus on detecting vastly different conceptualizations of framing, such as blame frames (Shurafa et al., 2020), war frames (Wicke and Bolognesi, 2020), moral frames (Reiter-Haas et al., 2021), or media frames (Boydstun et al., 2014; Kwak et al., 2020).
Regarding media frames, Boydstun et al. (2013) identified a set of relevant frames that formed the basis for the media frame corpus (Card et al., 2015). Within this supervised frame detection scenario, Liu et al. (2019) indicate that Transformer-based approaches vastly outperform approaches using less powerful architectures such as LSTMs. As such, we also employ Transformer models with label-aware contrastive pre-training.
2Our code and model are publicly available at: https://github.com/socialcomplab/semeval23-mcpt
Supervised Contrastive Learning. Contrastive learning, originally mainly used in computer vision settings (e.g., Chopra et al., 2005), has recently found increased attention in the NLP research community due to its efficacy on tasks with limited amounts of data and its applicability to Transformer embeddings (e.g., Tunstall et al., 2022). The general concept of supervised contrastive learning (Khosla et al., 2020a) is that latent representations (or embeddings in NLP) of samples with the same labels should be close in embedding space, while samples with different labels should be further apart.
Su et al. (2022) and Zheng et al. (2022) have independently proposed contrastive learning methods for multi-label settings that weight similarities of samples by the similarity of their label vectors, i.e. hidden representations of samples with similar label vectors should be more similar than hidden representations of samples with less similar label vectors. Su et al. (2022) weight a Euclidean distance-based measure of embeddings by a normalized dot product of the label vectors. HeroCon (Zheng et al., 2022) generalizes supervised contrastive loss (Khosla et al., 2020b) and beats previous state-of-the-art contrastive learning paradigms in multi-label settings on multiple image data sets. Tunstall et al. (2022) introduce SETFIT, an algorithm for the data-efficient fine-tuning of sentence embeddings, primarily on binary labels. SETFIT first fine-tunes sentence embeddings in a contrastive manner before training a classification head.
We combine the idea of the contrastive pretraining stage from SETFIT and adopt HeroCon for NLP loss to improve performance on multi-label datasets.

heading: 3 Methods
text: At the core of our system lies a multilingual Transformer model with dense neural layers comprising the head. Contrastive fine-tuning is performed as part of a multi-stage training procedure.

heading: 3.1 Contrastive Fine-Tuning
text: Our contrastive fine-tuning objective (C1, Figure 1 top) is centered around the idea that embeddings of samples with similar labels should be close while embeddings of samples with very distinct label vectors should be distant. Following Zheng et al. (2022) for every batch and every class, we com-
pute the similarity between positive samples, i.e., samples that are of that class, and all others. As such, samples may both repel and attract each other within different classes yet do neither if they are both negative.
Our loss function is a linear combination of two terms: A binary cross entropy term LBCE that jointly optimizes the head and body in the contrastive fine-tuning stage and a contrastive term LCON : L = LBCE + αLCON (1) where α is a weighting hyperparameter. The contrastive loss is given by:
LCON = 1|C| ∑ c∈C −EXi,Xj∈P(c) [ log σijf(Xi,Xj) δij ] (2)
where C is the set of all classes (e.g. Economic), P(c) is the set of all positive samples i.e., all embeddings Xi that are of class c, and f(·, ·) is the cosine similarity measure between embeddings. The loss is normalized by:
δij = σijf(Xi,Xj)+
∑ Xk∈N (c) γikf(Xi,Xk) |N (c)|+1 (3)
where N (c) is the set of all negative samples and σij and γik are given by:
σij = 1− d(Yi, Yj)/|C|, γik = d(Yi, Yk)
respectively where d describes the Hamming distance between label vectors Yi.

heading: 3.2 Training Procedure
text: We follow a two-phase training procedure illustrated in Figure 1 (bottom), focusing on optimally utilizing the available data, which in our case are the six languages of subtask 2 (C2). Inspired by Tunstall et al. (2022), we first optimize the embedding space of a (multilingual) Transformer model. Herein, our approach makes the assumption that the embeddings, regardless of language, possess mutual information given similar labels (Zheng et al., 2022). While the embedding space may be improved in this manner, we fine-tune on the target language to further improve the performance.
Phase one consists of two separate stages: head pre-training and contrastive fine-tuning. For phase two, the head is re-used for zero-shot settings, while discarded and randomly re-initialized for fewshot settings. In the former case, we conduct the post-training stage on all languages, while in the latter it is essential to pre-train the head on the target language before proceeding to the contrastive fine-tuning and head post-training stages. Both the head pre-training and head post-training stages only compute the binary cross entropy term LBCE ,
while simultaneously leaving the body unchanged, i.e., frozen. While the stages are identical, the rationale for each of them is very distinct: As the head is randomly initialized, we first pre-train it to avoid high gradients in the subsequent contrastive fine-tuning stage. In contrast, post-training allows the head to better fit the fine-tuned embeddings.

heading: 4 Experiments
text: We now present the results of our mCPT system, supported by embedding space and ablation studies (C3). We base mCPT on the multilingual3 sentence Transformer model paraphrase-multilingualMiniLM-L12-v2 (Reimers and Gurevych, 2019) and demonstrate that competitive results can be achieved with a relatively small amount of parameters, i.e., 117M parameters (Wang et al., 2020), given a training method tailored to the task. The model was chosen for its sentence embedding performance on multiple languages and its small size relative to similar state-of-the-art multilingual Transformer models.
Our model architecture comprises mean-pooling, no normalization of embeddings, a dense head with one hidden layer of size 256, and a dropout of 0.5. We train the model with separate learning rates for the classification head (1e−3) and the body (2e−5), a weighting parameter α of 0.01, a batch size of
3The base model was trained on 50+ languages including all nine of the shared task, thus being suitable for the problem.
26 for 10 and 50 epochs for head pre-training and contrastive fine-tuning respectively in phase one (more details in Appendix A).
Baseline Models. We compare the performance of our system against two baselines. First, we consider the results of the official baseline Base (i.e., n-grams and support vector classification; Piskorski et al., 2023). Second, we compare against SETFIT (Tunstall et al., 2022) with the same base encoder as ours, on the post-challenge test set (details in Appendix B).

heading: 4.1 Main Results
text: mCPT performs better on Latin alphabets (marked by L) in both few- and zero-shot settings, and improves upon the two baselines (as presented in Table 1). In Slavic languages (S), Polish is second-best in terms of Micro-F1 (0.597), but only slightly outperforms the baseline, whereas the improvement on Russian is very significant but only achieves the second-lowest score of 0.409. In comparison, both Germanic languages (G), German and English perform well, where we also have our highest overall Micro-F1 of 0.622 for German, but also a high baseline of 0.487. Although we find significant improvement on Greek and Georgian (which are zero-shot languages that do not share a major branch with any other language) over the baseline, both perform poorly in terms of MicroF1 (i.e., Georgian having the lowest Micro-F1 of
0.400). Hence, we suspect that not enough information from the other languages could be transferred. Finally, in the Romance languages (R), Italian and Spanish perform well, while French with a MicroF1 of 0.516 performs lower than Greek.
The performance of Spanish is especially noteworthy, as it is the only zero-shot language that shares an alphabet as well as a language family with the training data languages. Therefore, we argue that our winning performance on Spanish (Micro-F1 of 0.571 compared to 0.12 of the baseline) stems from the fact that the knowledge was successfully transferred from the other languages to the zero-shot setting. This is further supported by SETFIT, which improves upon the baseline but shows lower performance on Latin-based languages.

heading: 4.2 Embedding Space Analysis
text: Figure 2 demonstrates how our contrastive training procedure optimizes the embeddings of the Transformer body. Figure 2a exemplarily shows the repositioning of samples due to the loss function in twodimensional space. Observe how the trained labels (opaque) align, with the blue-yellow label between the two blue-only labels and two yellow-only labels, while simultaneously pushing the red labels to the side (more detailed analysis in Appendix C). For the analysis of the high-dimensional embeddings (384) and label spaces (14) on the dev set, we use boxen plots concerning embedding cosine similarity for all pairwise samples within a given Hamming distance.
Regarding the pre-trained base model on English (Figure 2b), we find a suboptimal correlation with R2 = 0.005 and β = −0.009. In comparison, after contrastively training the model, the correlation becomes much more pronounced, i.e., R2 = 0.241 and β = −0.079 for English as shown in Figure 2c. Furthermore, the spread of pairwise embedding similarity distribution increases, as a greater amount of samples become dissimilar to each other, especially for higher label distance. Thus, we conclude that our system leads to higher utilization of the available embedding space, which in turn boosts performance. Appendix D contains the remaining languages.
Finally, we want to emphasize that our data set has no perfectly dissimilar label vector pairs, which would make hard negative mining approaches (Gao et al., 2021) infeasible, e.g., for using plain con-
trastive loss (Chopra et al., 2005).

heading: 4.3 Ablation Study and Extension
text: Table 2 indicates the effectiveness of our combined training approach mCPT. From mCPT, we remove components iteratively, first removing the multilingual pre-training phase, then the contrastive term (see Equation 2), and finally, end-to-end training leaving only a trained classification head with no embedding fine-tuning. Comparing the results of the ablation study with those of Table 1 it is interesting to note that our approach works best on languages with lower scores. A hypothesis is that the out-of-the-box Transformer embeddings already fit the data well and that mCPT is not able to improve upon the already strong baseline. Finally, we find that adding a contrast sampling extension could further improve the results (see Appendix E).

heading: 5 Conclusion
text: In this paper, we describe our system (mCPT) for the framing detection shared task (Piskorski et al., 2023). We introduce an approach based on a labelaware contrastive loss and training procedure for Transformers to deal with the challenges of multilingual multi-label prediction with few or even zero samples. The generalization ability of our system is demonstrated by providing the winning contribution for the Spanish framing detection subtask where no training samples were available4. Hence, we believe that our system is a notable advancement for computational framing research.
4We refer to Appendices F and G for discussions on limitations and ethical considerations, respectively.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for taking the necessary time and effort to review the manuscript and appreciate their universal positive feedback.

heading: A Training Environment
text: We performed the main experiments on the Kaggle platform (www.kaggle.com) with the P100 graphics card. We chose a free platform for the computation to demonstrate that our system is tailored towards the task at hand and is accessible for everybody, rather than relying on large amounts of computational resources. We empirically selected the hyperparameters to fit the platform. Herein, we chose a batch size of 26 which optimally utilizes the available GPU memory. The multilingual pretraining takes approximately 1.5 hours, while the language-specific fine-tuning takes 1 hour each.

heading: B SETFIT Parameters
text: We choose SETFIT as it is similar in concept to our system, i.e., contrastive learning for Transformers, but not aligned with the shared task, i.e., does not explicitly consider multi-label problems. Hence, the comparison demonstrates how our system is an improvement over established approaches in this setting and emphasizes that the adaptions of the contrastive loss and training procedure are indeed beneficial. We report the results of SETFIT on the post-challenge leaderboard without further adaption after the initial submissions for fair comparisons on the test set.
We mimic the parameters setting where applicable while preserving the standard training procedure to maximize comparability. We first contrastively train the body for 10 epochs before training the full model end-to-end for 50 epochs with a batch size of 26 with learning rates of 1e− 3 and 2e− 5 for the head and the body respectively. The body-then-end-to-end procedure was suggested by the usage guide. The training runtime is approximately 10 hours on the Kaggle platform (which again was chosen for a fair comparison). Initially, we experimented with SETFIT in the challenge period but decided to submit our presented system instead.

heading: C Repositioning of Samples
text: In Figure 2a, we show the effect of the loss function and how the label and embedding space are intertwined. Specifically, we demonstrate how randomly generated embeddings in two-dimensional space with three-dimensional label vectors shift towards more optimal positions after applying the contrastive loss function. Accordingly, the initially random positions of embeddings with equivalent labels end up on straight lines drawn from the origin. The observed effect is a direct consequence of similar label vectors attracting and opposite labels repelling each other. Moreover, a partial label similarity with two distinct groups ends between those groups, as a result of both forces being active. For instance, consider the line from top left to bottom right: blue-only labels become attracted, and repel yellow-only labels, while the sample with blue and yellow lies in between. Due to the resulting positioning, the embedding space becomes disentangled leading to an increase in linear separability, which benefits classifiers such as our differentiable head.

heading: D Embeddings and Labels Correlation
text: Here, we present the remaining languages for the embedding space analysis in Figure 3. When considering the plots before contrastive training is performed, the correlations between embedding spaces and label spaces are very weak, as well as most embeddings being very similar regardless of their label distance. Hence, virtually no pair of samples are opposite to each other, i.e., has a noteworthy negative cosine similarity. Conversely, all pairs are similar to a certain extent. We argue that in these given embedding spaces, it is challenging
for a classifier to learn a discriminative function. Moreover, due to the suboptimal positioning of embeddings, a substantial amount of the embedding space does not contribute to the prediction performance, thus wasting the model’s potential expressiveness. Hence, the interpretation from Section 4.2 can be directly applied to the five other languages, as the effect is the same (although differently pronounced). For instance, models of German and Russian even have a slight upward slope when applied without contrastive training. Hence, the negative slope and regression fit increases with contrastive training, as intended and expected.

heading: E Contrast Sampling Extension.
text: Adding a custom contrast sampler which ensures that at least one sample from every class is present per batch further improves consistency as well as performance. Due to the nature of the contrastive objective, it is imperative that every batch contain negative as well as positive pairs of samples for every class. This is not guaranteed by sampling randomly, especially if the label distribution is imbalanced. As illustrated by Table 2 it outperforms pure mCPT in five out of six languages while coming in second by a small margin in Russian. We attribute this largely to the variance introduced by using a relatively small batch size of 26 compared to the number of labels (14). This variance may lead to undesirable gradient updates in some iterations when batches contain label distributions that are not representative.

heading: F Limitations
text: We recognize three main limitations of our work, which are distinct in their aspect.
First, the performance limitation; while our system has competitive results across the board, it only performs best in one of the nine languages on the leaderboard. In comparison, team MarsEclipse (Liao et al., 2023), which also focused on the framing detection subtask, wins all six few-shot languages and performs second on two of three (i.e., Greek and Georgian) zero-shot languages. They only perform worse at Spanish (6th), which is opposite to our placement. Team SheffieldVeraAI (Wu et al., 2023), who also participated in the other two subtasks regarding news genre and persuasion technique detection, perform well across the board and wins the Greek and Georgian framing detection tasks. Hence, our system occupies the niche
of zero-shot prediction when trained with similar languages (i.e, in our case Latin-based).
Second the technical limitation, our system was trained using a small multilingual model as we aimed towards adapting Transformer pre-training for the multi-label challenge in particular rather than achieving the highest performance with computationally expensive training. However, as a consequence, we do not know how well our system scales to bigger models, such as MPNet (Song et al., 2020), and plan to address this limitation in future work.
Third the task setting limitation, we want to emphasize a potential limitation resulting from the shared task setting. Ali and Hassan (2022) argue that the specified labels in the media frame corpus (Card et al., 2015) revolve around topics (i.e., the what) rather frames (i.e., the how). As the same labels were adopted for the shared task, the conceptualizations of frames are expected to be similar to a certain extent. They thus would also affect the resulting models and predictions.

heading: G Ethics Statement
text: We want to discuss three ethical considerations of our system. First, our system is based on pretrained Transformers, which inherit biases from their training data. For the shared task, these biases are negligible, but are a concern for real-world applications. The second consideration relates to fairness concerns. The performance varies strongly between languages, with more researched languages typically resulting in better performance. We, thus, embrace the multilingual setting of the shared task with one-third zero-shot languages, but similarly achieved better performance in Latin-based languages. Third, our system leads to better detection of media frames, which is an important research direction. However, the system could in theory also be used in a disputed or even malicious manner, e.g., for reframing political statements. Hence, we do not advise specific applications of our system besides better framing detection.

heading: Future Work
text: , 2020), and plan to address this limitation in future work. Third the task setting limitation, we want to emphasize a potential limitation resulting from the shared task setting. Ali and Hassan (2022) argue that the specified labels in the media frame corpus (Card et al. , 2015) revolve around topics (i. e. , the what) rather frames (i. e. , the how). As the same labels were adopted for the shared task, the conceptualizations of frames are expected to be similar to a certain extent. They thus would also affect the resulting models and predictions