Prodicus at SemEval-2023 Task 4: Enhancing Human Value Detection with Data Augmentation and Fine-Tuned Language Models
Year: 2023

Abstract:
This paper introduces a data augmentation technique for the task of detecting human values. Our approach involves generating additional examples using metadata that describes the labels in the datasets. We evaluated the effectiveness of our method by fine-tuning BERT and RoBERTa models on our augmented dataset and comparing their F1-scores to those of the non-augmented dataset. We obtained competitive results on both the Main test set and the Nahj al-Balagha test set, ranking 14th and 7th respectively among the participants. We also demonstrate that by incorporating our augmentation technique, the classification performance of BERT and RoBERTa is improved, resulting in an increase of up to 10.1% in their F1-score.

heading: 1 Introduction
text: When it comes to arguments, different people sharing the same values may have different opinions about whether one argument is persuasive or not. One cause of this is the difference between people ordering of values used to assess arguments. Within computational linguistics, human values can provide context to categorize, compare, and evaluate argumentative statements, allowing for several applications: to inform social science research on values through large-scale data sets; to assess argumentation; generate or select arguments for a target audience; and to identify opposing and shared values on both sides of a controversial topic (Mirzakhmedova et al., 2023). Human Value Detection task (Kiesel et al., 2023) is concerned with automatic classification of textual arguments to determine whether an argument draws on a specific human value. Provided dataset of arguments (Mirzakhmedova et al., 2023) is collected from different resources within different countries and beliefs. All arguments are presented either in English or have been translated into it.
To address the problem of detecting values behind arguments, we can design a multi-label clas-
sifier to determine whether an argument draws on a specific value or not. To tackle the classification problem, we propose two multi-label classifiers based on two widely used language models: 1) RoBERTa (Liu et al., 2019) and 2) BERT (Devlin et al., 2018); We evaluated these models performances on multiple test sets in two scenarios: 1) fine-tuned on the Main training set (Mirzakhmedova et al., 2023), 2) fine-tuned on our Augmented dataset. We provide all the code necessary to replicate our work in the paper’s GitHub repository1.

heading: 2 Background
text: The main challenge of identifying human values behind an argument is that in most cases the argument does not explicitly refer to the desired value. The first attempt to computationally extract human values behind an argument was done by (Kiesel et al., 2022). They first proposed a 4-level taxonomy of human values. After that they provided three baseline models to classify values in each level. For sake of this paper and this task (Kiesel et al., 2023)
1https://github.com/ErfanMoosaviMonazzah/SemEval2023Task4-Human-Value-Detection
2033
we only classified values from level 2 which consists of 20 value categories (Classes). Later on we compare our results to the result (Kiesel et al., 2022) obtained for level 2 value categories using these three baselines: 1-baseline, BERT-kiesel, as well as one of our own baselines, which is zero-shot classification of arguments.

heading: 3 Dataset
text: Task’s Main dataset contains 9324 arguments on a variety of statements written in different styles, including religious texts (Nahj al-Balagha), political discussions (Group Discussion Ideas), free-text arguments (IBM-ArgQ-Rank-30kArgs), newspaper articles (The New York Times), community discus-
sions (Zhihu), and democratic discourse (Conference on the Future of Europe) (Mirzakhmedova et al., 2023). All the arguments are either in English or translated into English. Each argument has a Premise, a Stance and a Conclusion and a list of binary values corresponding to value categories selected from all 20 value categories in level 2. (See Table 1) Arguments are then divided into 5 splits: Main Train, Main validation, Zhihu validation, Main test, Nahj al-Balagha test and New York Times test. They also accompany description data to describe the meaning of each value category by providing examples for them. The original training set consists of 5393 examples.
A challenging aspect of this task is that the models tend to struggle with predicting labels that have only a small number of examples in the training set (See Figure 2). To tackle this problem, we propose a data augmentation method which will create additional examples from meta data provided by task organizer. Including more examples can enhance the prediction of values that constitute a smaller fraction of the dataset. This, in turn, leads to an overall improvement in the macro-average F1 score of models across all labels, by increasing the F1 score of each label.
The main datasets consist of arguments, each of which comprises three parts: a premise, a stance, and a conclusion (Table 1). The conclusion takes either an in favor or against stance on its corresponding premise. To input these arguments into our models, we concatenate the three parts to form a single sequence (Table 2).
The value definition data is structured as a dictionary within a dictionary. The outer dictionary contains twenty key-value pairs, where each key represents a value category to be predicted. For each key (value category), the corresponding value
is an inner dictionary. These inner dictionaries consist of various key-value pairs, where each key corresponds to a level-1 label, and each value is the definition of that level-1 label ((Kiesel et al., 2022)).
We propose a template for generating new examples from the definition data. The template involves placing the definition before its corresponding level-1 value and inserting the phrase "is an example of" in between (See Table 3). Although the newly generated sentences have a different structure compared to the arguments in the dataset, we hypothesize that using these sentences can aid in predicting categories by expanding the vocabulary size of categories.
Through the application of this method on description data, we generated a new dataset that consists of 218 examples, each containing a single label corresponding to the value category which it defines. We use an up-sampling factor in order to control the number of augmented examples in dataset. For up-sampling factor k, the augmented sentences repeated k times in the main training set. Our method was evaluated for k ∈ [0, 10].

heading: 4 System Overview
text: Our proposed classifiers are heavily relied on transformer based pre-trained language models. Language modeling is the task of predicting the next word given a sequence of input words.
P(wi|w1, w2, . . . , wi−1) = P (w1,w2,...,wi−1,wi)P (w1,w2,...,wi−1) Currently, state-of-the-art results in language modeling are achieved by training transformer models that use an attention mechanism to extract a representation for an input sequence (encoding) and generate the next word from the extracted representation in a separate decoding module (Vaswani et al., 2017). Although language modeling differs from our classification task, we can transfer the learned knowledge about language structures from these models to our task by changing the model’s head, which is the output layer of the decoder part of the transformer. Instead of predicting the next word using the output embeddings of the decoders, one can classify these embeddings using a fully connected layer. It has the benefit of only training the classifier head instead of the entire model. To accomplish this task, we fine-tuned two commonly used language models, BERT and RoBERTa. We provided arguments as input and generated a binary
vector as output. Each digit in the binary vector corresponds to whether the input argument draws on that particular value category or not (refer to Table 2). To obtain zero-shot classification results, we employed bart-large-mnli ((Lewis et al., 2019)) which is a NLI model. (Yin et al., 2019) presented a technique for utilizing pre-trained NLI models as off-the-shelf zero-shot sequence classifiers. This method involves formulating the sequence to be classified as the NLI premise, and constructing a hypothesis for each candidate label. For instance, to determine whether a sequence belongs to the "politics" class, a hypothesis could be constructed as "This text is about politics." The probabilities for entailment and contradiction are subsequently converted into label probabilities.

heading: 5 Experimental Setup
text: The task organizer divided the data into three main sets (train, validation, test) and three supplementary sets (Zhihu validation, Nahj al-Balagha test, The New York Times test) (Mirzakhmedova et al., 2023). We generated eleven versions of the main training dataset, each one uses a different up-sampling factor k. We trained each model for 10 epochs and recorded its performance after each epoch. Both the BERT and RoBERTa models performed reasonably well after 10 epochs. We compared the effectiveness of augmentation with dif-
ferent up-sampling factors across different datasets and models (Figure 1). In terms of the main validation set, increasing the up-sampling factor usually resulted in an increase in the F1-score. Both the BERT and RoBERTa models were trained on an NVIDIA GeForce GTX 1080 Ti for 10 epochs. For the task leaderboard, we submitted the results of RoBERTa fine-tuned on the augmented dataset with k = 6. We carefully tuned the batch size, learning rate and number of traning epochs of the model with respect to both Main and Zhihu validation sets. We repeated the training with different initial weights five times and submitted the best results out of the five attempts for the task. Our RoBERTa results placed 14th out of 41 on the leaderboard for the main test datasets and 7th among 20 submissions for Nahj al-Balagha test set (Table 5).

heading: 6 Results
text: We evaluated the effectiveness of our approach on the original test datasets using two models, BERT and RoBERTa, and the results are presented in Table 4. Our findings indicate that the proposed augmented dataset leads to an increase in F1-score for the Nahj al-Balagha test set by 3% and the New York Times test set by 10.1% for BERT. For RoBERTa, the F1-score increases are 1.9%, 6.5%, and 2.2% for the Nahj al-Balagha test set, the New York Times test set, and the main test set, respectively. Our carefully fine-tuned RoBERTa model achieves an F1-score of 48% on the main test set, which is 6% higher than the results reported by (Kiesel et al., 2022) and 35% higher than the zeroshot baseline.
Regarding the Nahj al-Balagha test set, the RoBERTa model’s F1-score is 30%, which is 2% higher than (Kiesel et al., 2022) and 22% higher than the zero-shot baseline. A detailed summary of our results can be found in Table 5. We also observed that data augmentation had a positive impact on values with a smaller amount of data, as depicted in Figure 2. Augmenting the data for these values resulted in an increase in their F1-score, as well as a slight increase in the F1-scores of other values.

heading: 7 Conclusion
text: In this paper, we proposed two multi-label classification systems for the Human Value Detection task, using a fine-tuned version of BERT and RoBERTa language models. We also introduced a data aug-
mentation method to improve the performance of our models. Our experimental results showed that our augmentation method can increase the F1-score of the models. We also achieved competitive results on the Main test set and Nahj al-Balaghafor set, ranking 14th and 7th respectively among the participants. Our work demonstrates the effectiveness of using pre-trained language models for the Human Value Detection task and the potential benefits of data augmentation methods for improving model performance. Future work can explore additional augmentation methods and further improve the performance of the models by incorporating external knowledge or domain-specific information. Additionally, expanding the dataset to include arguments in other languages and from diverse cultures and perspectives can enhance the robustness of the models and better reflect the real-world scenarios.

heading: 8 Acknowledgments
text: I would like to express my gratitude to my parents and friends for their unwavering support throughout this project.

heading: Future Work
text: Future work can explore additional augmentation methods and further improve the performance of the models by incorporating external knowledge or domain-specific information. Additionally, expanding the dataset to include arguments in other languages and from diverse cultures and perspectives can enhance the robustness of the models and better reflect the real-world scenarios