I2C Huelva at SemEval-2023 Task 4: A Resampling and Transformers Approach to Identify Human Values behind Arguments
Year: 2023

Abstract:
This paper presents the approaches proposed for I2C Group to address the SemEval-2023 Task 4: Identification of Human Values behind Arguments (ValueEval)" (Kiesel et al., 2023), whose goal is to classify 20 different categories of human values given a textual argument. The dataset of this task consists of one argument per line, including its unique argument ID, conclusion, stance of the premise towards the conclusion and the premise text. To indicate whether the argument draws or not on that category a binary indication (1 or 0) is included. Participants can submit approaches that detect one, multiple, or all of these values in arguments. The task provides an opportunity for researchers to explore the use of automated techniques to identify human values in text and has potential applications in various domains such as social science, politics, and marketing. To deal with the imbalanced class distribution given, our approach undersamples the data. Additionally, the three components of the argument (conclusion, stance and premise) are used for training. The system outperformed the BERT baseline according to official evaluation metrics, achieving a f1 score of 0.46.

heading: 1 Introduction
text: Human values refers to the beliefs, principles and standards that individuals or groups hold to be important and worthwhile. These values guide people´s attitudes and behaviors; they can vary across cultures (Civitillo et al., 2019), communities, and individuals. According to a study of Global Values Survey (White et al., 2020), the most widely held values across the world are a sense of community, a sense of national pride, and a desire for social order. Other commonly held values include equality, respect for others, and a desire for a peaceful world.
In this context, the classification of human values in textual arguments is an important task in the field
of Natural Language Processing. Understanding the values that underlie an argument could provide valuable insights into people´s beliefs, attitudes, and motivations. It could also be useful in various applications such as opinion analysis (Hemmatian and Sohrabi, 2019), argumentation mining, emotion recognition, and persuasive technology, among others.
Despite its importance, the automatic classification of human values in arguments remains a challenging problem. The task requires the ability to identify values in a text, understand the argument structure, and make a binary judgment about the presence of a value in the argument. It aims to advance the state of the art in human value classification in textual arguments , these textual arguments are compiled from the social science literature and described in detail in the accompanying ACL paper (Kiesel et al., 2022).
ValueEval had the advantage of bringing together 40 teams, taking in 112 different runs (including the competition organizers). They provided a set of labelled data to solve the task. This dataset was highly imbalanced over the 20 categories, with a large number of instances belonging to the negative class. Undersampling (Arefeen et al., 2020) is a common technique used in machine learning to balance the class distribution in imbalanced datasets. To tackle this problem, we employed an undersampling strategy to decrease the number of instances within the negative class. On the other hand, we conducted experiments using various combinations of the information provided for the arguments.
Finally, our approach uses the premise, conclusion and stance of the argument as input features. These three components provide important information about the argument to help identify the values. In this work, we utilized transfer learning techniques by fine-tuning state-of-the-art pre-trained language models, using the transformers library. This approach allowed us to leverage the knowl-
1382
edge learned from large-scale datasets and apply it to our specific task of argumentative text classification.
The results of our experiments show that our approach is effective in classifying arguments based on human values. The combination of undersampling and the use of all the data given leads to improved performance compared to other methods. Our findings contribute to a better understanding of how human values can be identified in arguments and have implications for a range of applications, including opinion analysis and argumentation (Lawrence and Reed, 2020). However, as is common in such tasks, some categories have a low number of positive instances, i.e. instances where the argument draws on that category. This low number of positive instances can pose a challenge for machine learning algorithms, as they may not have enough examples to learn from. This can result in overfitting, where the algorithm memorizes the training data instead of generalizing to new data, therefore the model may not be able to accurately predict the outcome for new instances.

heading: 2 Background
text: The input data for the study consists of two tabseparated value files, "arguments-training.tsv" and "labels-training.tsv". Both contains 5,394 rows, on arguments each row represents a unique argument with its ID, conclusion, stance and premise, Figure 1 shows an example of an argument from the training dataset. . For the label each row corresponds to the unique argument ID, and one column for each of the 20 value categories, indicating whether the argument aligns with the particular value category (1) or not (0). The dataset used in this paper was extracted from the descriptions of articles in arXiv, as described in the source (Mirzakhmedova et al., 2023).
[Argument ID] A01010 [Conclusion] We should prohibit school prayer [Stance] against [Premise] it should be allowed if the student wants to pray as long as it is not interfering with his classes.
Figure 1: Example of argument from training dataset
The identification of human values behind arguments is an important aspect of argument mining. Some work have been researched in this area,
which aims to extract natural language arguments and their relations from text (Cabrio and Villata, 2018), there are a lot of use cases like (Passon et al., 2018) predicting the usefulness of online reviews based solely on the amount of argumentative text that they contain, or finding relevant evidence (on argument premises) in the study of adjudication decisions about veteran´s claims for disability (Walker et al., 2018)

heading: 3 System Overview
text: The task of classifying textual arguments based on human values categories is challenging due to the subjective nature of human values. However, the system was able to address this challenge by using advanced deep learning algorithms such as BERT and RoBERTa.

heading: 3.1 Implemented Models
text: For our argument classification task, we employed the BERT and RoBERTa models. Both of these models are based on the transformer architecture and have been pre-trained on massive amounts of text data. BERT, short for Bidirectional Encoder Representations from Transformers, was introduced by (Devlin et al., 2019) and has achieved state-of-the-art performance on various natural language processing tasks. RoBERTa, a variant of BERT, was introduced by (Liu et al., 2019) and further improved the pre-training process by optimizing hyperparameters and using larger batch sizes. We used the Hugging Face1 library to finetune the pre-trained BERT and RoBERTa models for our task. We employed the common hyperparameters, including batch size, learning rate, and weight decay, and used early stopping with a maximum of five epochs. We did not tune any specific hyperparameters.Our choice of these models was based on their proven success in various natural language processing tasks and their pre-training on large amounts of text data. We fine-tuned the models on our argument classification task to leverage their ability to understand and extract meaningful information from natural language text. Recent studies have demonstrated the effectiveness of these models in various tasks, such as sentiment analysis (Khan and Fu, 2021), question answering (Ju et al., 2019), and document classification (Liu et al., 2021).
1https://huggingface.co/

heading: 3.2 Selected Inputs
text: We experimented with different input formats for our models, including using just the conclusion or just the premise as input. However, we found that using the conclusion stance and premise together as a single input yielded the best results in our experiments. This is because the conclusion stance provides important context for the premise, allowing the model to better understand the argument being made. To illustrate the impact of using different input formats, we show the F1 scores for each format in Table 2. As can be seen, the model using the combination of conclusion stance and premise as input achieved the highest F1 score.2

heading: 3.3 Preprocessing Text
text: We have applied text preprocessing to clean and simplify the text:
• Conversion of all characters to lowercase: All the characters in the text were converted to lowercase for consistency and ease of processing.
• Expansion of all possible contractions in English: Contractions such as "don’t" were expanded to "do not" and "can’t" was expanded to "cannot" to ensure that the model could understand the text properly.
2Note C+S+P stands for Conclusion+Stance+Premise.
• Removal of special characters: Special characters such as punctuation marks and symbols were removed from the text. This helped to simplify the text and remove any unnecessary noise.
• Removal of multiple spaces between characters: Multiple spaces between characters were reduced to a single space. This was done to ensure that there was consistency in the text and that all the spaces were uniform.
By applying these text processing techniques, we were able to simplify and clean the text, making it easier to process and analyze.

heading: 3.4 Undersampling Techniques
text: In order to address the issue of class imbalance in the provided training dataset, we implemented an undersampling technique that reduces the size of the majority class to increase the representation of the minority class. Specifically, we used a multiplier on the size of the majority class to determine the number of samples to keep for each label, with the multiplier being determined by the size of the minority class. To ensure the models had sufficient data to train on, we trained at least with 1000 arguments. Table 1 shows the distribution of argument categories after performing undersampling. Overall, these preprocessing steps helped improve (see Table 3) the models’ performance on the data.

heading: 4 Experimental Setup
text: For our experimental setup, we used a trainvalidation-test split to evaluate the performance of our models. We split the dataset into 80% for training, 10% for validation, and 10% for testing. This allowed us to train our models on a sufficiently large amount of data while still having a separate set of data to test the models’ generalization ability. We chose not to use the validation dataset during the training process because its arguments were substantially different from those in the test dataset, which could have affected the generalization ability of our models. Therefore, we solely relied on the training and test datasets for all the experiments. We used the PyTorch library and the transformers package to implement our models. We used the Hugging Face Transformers library to fine-tune pre-trained transformer models for the argument classification task. TIRA (Fröbe et al., 2023) is the platform used for the shared task that we submitted our system to.
For the experiment, the models were trained with 5 epochs, 32 batch size, 64 token length. Early stopping was used to avoid overfitting while training. Labels like "Stimulation" only left us with 198 positive examples and 4116 negatives for our training dataset. This made the identification of values a more difficult task.
By balancing it to an equal number of 0s and 1s, we have observed worse results compared to the original one. Then, to improve our approach which is identifying the human value, the dataset has been adjusted to bring at least 1000 arguments for training, which has shown better results as we can see in Table 4, our model gave us a 56% improvement over the original one.
In addition to that, we have more balanced labels like "Self-direction: action" with 1116 positive examples and 3198 negatives. With our undersampling approach (see Table 5) we improved the result by 18%.

heading: 5 Results
text: Our system achieved performance above the BERT baseline (shown in Figure 2), as measured by official evaluation metrics, we got better results overall. But the biggest gap is in more imbalanced classes like "Stimulation", "Humility" and "Face" where we nearly doubled the performance. Table 6 shows the overall results. In the competition, there were 39 participating teams. Our best-performing system was a BERT pretrained model with resampling. This approach achieved a f1 score of 0.46, obtaining the 24th position in the final leaderboard, demonstrating its effectiveness.

heading: 6 Conclusion
text: To conclude, our system for detection of human values behind arguments achieved competitive results, as it presents better results than the baseline BERT model. Our approach of utilizing undersampling to balance the dataset and focusing on the positive class proved to be effective, especially given
the nature of the competition where identifying it is more crucial than not identifying it. However, there is still room for improvement. One potential direction for future work is to explore data augmentation techniques to increase the size of our training arguments and potentially improve model performance. There is also the possibility of extending our system to other languages, which would require additional preprocessing and potentially the development of language-specific models. Finally, investigating the performance of other transformerbased models and their variants on this task could also lead to better results in future researches.

heading: 7 Acknowledgments
text: This paper is part of the I+D+i Project titled “Conspiracy Theories and hate speech online: Comparison of patterns in narratives and social networks about COVID-19, immigrants, refugees and LGBTI people [NONCONSPIRA-HATE!]”, PID2021-123983OB-I00, funded by MCIN/AEI/10.13039/501100011033/ and by “ERDF A way of making Europe”.

heading: Future Work
text: One potential direction for future work is to explore data augmentation techniques to increase the size of our training arguments and potentially improve model performance. There is also the possibility of extending our system to other languages, which would require additional preprocessing and potentially the development of language-specific models. Finally, investigating the performance of other transformerbased models and their variants on this task could also lead to better results in future researches