PAI at SemEval-2023 Task 4: A general multi-label classification system with class-balanced loss function and ensemble module
Year: 2023

Abstract:
The Human Value Detection shared task (Kiesel et al., 2023) aims to classify whether or not the argument draws on a set of 20 value categories, given a textual argument. This is a difficult task as the discrimination of human values behind arguments is often implicit. Moreover, the number of label categories can be up to 20 and the distribution of data is highly imbalanced. To address these issues, we employ a multi-label classification model and utilize a class-balanced loss function. Our system wins 5 first places, 2 second places, and 6 third places out of 20 categories of the Human Value Detection shared task, and our overall average score of 0.54 also places third. The code is publicly available at https://www.github.com/ diqiuzhuanzhuan/semeval2023.

heading: 1 Introduction
text: The Human Value Detection shared task (Kiesel et al., 2023) aims at building a system to identify human values behind a textual argument. Though human values have been considered in formal argumentation for about 20 years (Bench-Capon, 2003), it is the first task to draw values from arguments computationally. The organizers provide the first dataset Touché23-ValueEval Dataset (Mirzakhmedova et al., 2023) for studying human values behind arguments. It contains 5270 arguments extracted from four different cultural backgrounds: Africa, China, India, and USA. All arguments are written in English. Every argument consists of three parts: a premise, a conclusion, and a stance indicating whether the premise is in favor of or against the conclusion. Table 1 demonstrates examples from the dataset.
In this paper, we propose a general multi-label classification system for the Human Value Detection shared task. To handle imbalanced data, we apply a class-balanced loss function and perform data augmentation for classes with fewer data.
Transformer-based models have demonstrated remarkable performance across a variety of NLP tasks in recent years. As a result, we adopt these models as encoder in our work. To select the best encoder model, we evaluate 4 candidates, including BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), DeBERTa (He et al., 2020) and etc. The sentence vector computed by the encoder is used as input for the multi-label classifier. We select Sigmoid function as the activation function for the classifier. Given the highly imbalanced nature of the data, we experiment with 5 loss functions that are specifically designed for such case, including focal loss, distribution-balanced loss and etc., and select CB-NTR loss (Huang et al., 2021) as our final loss function. In terms of data preprocessing, we conduct rewriting and data augmentation to solve the weaknesses in the data and the model. In the end, we conduct 8-fold cross-validation and use a weighted voting ensemble based on evaluation scores of each class to get the final prediction.
Besides the system description, we also have some interesting findings: (1) setting individual threshold for each classifier in a multi-label classification model may not be an effective approach, (2) concatenating data that does not conform to human language can have a negative impact on model performance.

heading: 2 Background
text: Despite many years of research on the relationship between human values and arguments, identifying human values from argumentative texts computationally is a novel approach. In this task, the heart of value-based argumentation framework has been determined as a value taxonomy(or a set of values) that is both accepted and relevant by the organizers. The organizers extend the refined theory of (Schwartz et al., 2012) by adding and integrating nine values and building multi-level taxonomy in
256
line with psychological research and publish the Touché23-ValueEval Dataset, in which there are a total of 54 values and 20 value categories. However, in this task, we only care about the classification of 20 value categories. It’s worth mentioning that the organizers have already conducted experiments on Touché23-ValueEval Dataset and obtained results with an average F1 score of 0.25.

heading: 3 Our System
text: In this section, we will illustrate how our system works. We use Transformer-based model as our encoder model. We build four kinds of data modules: BaselineArgumentDataset, PremiseArgumentDataset, RewriteArgumentDataset and AugmentArgumentDataset, apply the best loss function, and finally integrate the results from multiple data modules. We also provide an overview of the system in Figure 1.

heading: 3.1 Data Module
text: Based on the characteristics of these data, we construct four different datasets.
BaselineArgumentDataset As mentioned in previous section, each argument contains a premise, a conclusion, and a stance. We simply concatenate the three part with a special separator token [SEP], make it work as our baseline data set.
PremiseArgumentDataset As shown in Table 1, the premise field obviously holds the richest information. Only using the premise part is also a good option, so we build a new data module (named after PremiseArgumentDataset) that only uses the premise field.
RewriteArgumentDataset The simple way of concatenating data in BaselineArgumentDataset is not in line with natural language and therefore is also inconsistent with the input data used by
pretrained models. We rewrite all the arguments by modifying the positive or negative attitude in the conclusion and use the premise and the modified conclusion to build new input data. We also provide an illustration in Figure 2.
AugmentArgumentDataset The label distribution is highly imbalanced, to mitigate data imbalance issue, we concatenate ordinary samples with sparse-label samples to augment the sparse-label samples. Concretely, for each sparse sample, we randomly sample from the remaining samples, perform data concatenation, and merge the label of these two samples. The iteration continues until the number of samples in each category can reach 15% of the overall proportion.

heading: 3.2 Multi-label Classification Module
text: We approach Human Value Detection task as a multi-label classification problem. We employ a transformer-based model as our encoder and derive a representation vector V ∈ RH from the [CLS] token’s embedding in the output of the encoder. Then we learn |L|(the number of labels) binary classifiers Hl : X → {l,¬l}, one for each different label l in L. We use the Sigmoid function as the activation function and set the activation threshold to 0.5.
ȳi = sigmoid(W Ti ∗ V + bi) (1)
yi ∈ li if {ȳi < 0.5} yi ∈ ¬li if {ȳi ≥ 0.5}
where W T ∈ RH∗|L| denotes a logic matrix, b denotes a bias, and i is the index of the matrix and the bias.
We also attempt to set different classification thresholds for each label based on its frequency but find that it does not improve the performance.

heading: 3.3 Loss function
text: Binary Cross Entropy(BCE) (Bengio et al., 2013), widely used for multi-label classification, is our baseline loss function. The main challenge lies in the imbalance of label distribution and the complex dependencies between labels. A lot of work has been published on the design of loss function. Focal loss(FL) (Lin et al., 2017) places a higher weight on samples with low probability to emerge. Class-balanced loss(CB) (Cui et al., 2019) design a re-weighting scheme that uses the effective number of samples for each class. Distributed-balanced loss(DB) (Wu et al., 2020) takes into account the impact caused by label co-occurrence and adds a negative tolerant regularization(NTR) to mitigate the over-suppression of negative labels. CB-NTR loss (Huang et al., 2021) integrates NTR with CB.
We experiment with all the mentioned loss functions and find that the CB-NTR loss function performs the best in this task.

heading: 3.4 Ensemble
text: When submitting the results of the final test set, we apply the model ensemble. Given the m predictions {yθ1, ..., yθ3} generated by m models, following (Ma et al., 2022), we form the final results by voting based on the F1 score of each value category on validation set.

heading: 4 Experimental Setup
text: The implementation is based on PyTorch Lightning1, Transformers2, and AllenNLP3. During training, we set the batch size as 16, random seed as 42, learning rate as 1e-5, and warmup steps as 1000. We use the AdamW optimizer (Loshchilov and Hutter, 2017) and the polynomial decay scheduler with the power of 0.01. We set a maximum epoch of 25 and set patience of 3 for early stop. All experiments are run on one single Tesla V100 GPU.
The macro F1 is our metric. As for data splits, we directly use the official train and development set. But for the test set, we conduct 8-fold crossvalidation and ensemble 8 results to one.

heading: 5 Results and Analysis
text: In this section, we display our results and analyze the impact of each component through ablation studies.

heading: 5.1 Results
text: In this competition with a total of 47 teams, we achieve the third place with a macro F1 score of 0.54. We outperform the official baseline by 0.28 and the official BERT by 0.12. The result is shown in Table 2.
1https://www.pytorchlightning.ai 2https://huggingface.co/docs/transformers/index 3https://allenai.org/allennlp

heading: 5.2 Ablation Study
text: We also conduct ablation experiments to validate our designs, including encoder model, data modules, loss function, and ensemble. Due to limited space, we only compare the macro F1 of all categories.
Encoder Model We build our baseline model with BaselineArgumentDataset and BCE loss and run experiments to find out the best encoder model among RoBERTa (Liu et al., 2019), BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), DeBERTa (He et al., 2020), etc. As shown in Table 3, the large version of RoBERTa achieves the best score.
Data Module We use the best encoder model to conduct ablation experiments on the data module. The results are shown in Table 4
Loss Function We further evaluate different loss functions by average F1 scores, including BCE (Bench-Capon, 2003), FL (Lin et al., 2017), CB (Cui et al., 2019), DB (Wu et al., 2020), and CBNTR (Huang et al., 2021). As shown in Table 5, CB-NTR loss achieves the best score.
Ensemble Our ensemble approach can significantly improve performance. We integrate the results of different data modules based on their performance on the development set. The result is shown in Table 6, where we can see our ensemble approach outperforms the best single-model by 0.04 F1 score over test set.

heading: 6 Conclusion
text: In this paper, we describe our system on how to use multi-label classification model to handle Human Value Detection task (Kiesel et al., 2023). We win 5 first places, 2 second places, and 6 third places on the leaderboard. We analyze both the data and model characteristics in-depth and implement modifications on data processing and loss function design. These adjustments are verified through extensive evaluations. Nevertheless, we don’t leverage information contained within the labels. For future research, how to incorporate the information of individual labels into a multi-label classification model is also a promising direction for us.

heading: A Appendix
text: The team name of us in TIRA system: theodorzwinger

heading: Future Work
text: For future research, how to incorporate the information of individual labels into a multi-label classification model is also a promising direction for us