Tübingen at SemEval-2023 Task 4: What can Stance Tell? A Computational Study on Detecting Human Values behind Arguments
Year: 2023

Abstract:
This paper describes the performance of a system which uses stance as an output instead of taking it as an input to identify 20 human values behind given arguments, based on two datasets for SemEval-2023 Task 4. The rationale was to draw a conclusion on whether predicting stance would help predict the given human values better. For this setup—predicting 21 labels—a pre-trained language model, RoBERTa-Large was used. The system had an F1-score of 0.50 for predicting these human values for the main test set while this score was 0.35 on the secondary test set, and through further analysis, this paper aims to give insight into the problem of human value identification.

heading: 1 Introduction
text: Arguing can take place in our everyday life to express and rationalize ideas, and the way arguments are delivered can vary in their style, language and goal (Boltužić and Šnajder, 2014). In parallel with this variation, it is stated that it is a challenge to detect values expressed in the arguments as they are not always opinionated explicitly (Kiesel et al., 2022). Since argumentation is a part of our everyday life—for instance, while making decisions—it is likely that our values play a role, although this may not always be overt. Therefore, a large amount of literature on human value studies is an understandable effort. With this study, I hope to contribute to computational approaches to identifying human values behind arguments.
Kiesel et al. (2023) describe this current human value identification task as classifying whether a given textual argument falls into a given human value or not.
Out of all the experiments, the versions with stance used as a separate label in addition to the existing 20 labels (resulting in 21 labels) outperformed the rest. The underlying motivation was to mark whether the model would withdraw any information from the stance of the premise towards the
conclusion. After the data pre-processing, the pretrained transformer model, RoBERTa-Large (Liu et al., 2019), was implemented.
This approach did not turn out to perform as well on the secondary test set, Nahj al-Balagha, as it did on the main test set when the F1-scores are taken into account—corresponding scores and ranks will be discussed later in the paper in addition to other approaches. This can very well ring a bell to how daunting it can be to identify human values due to their covert usage in arguments (Kiesel et al., 2022), especially considering that the arguments in the Nahj al-Balagha test set are from Islamic religious texts with grandiloquent content that includes sermons (Mirzakhmedova et al., 2023).
The aim of this paper is to present the role of stance and a pre-trained language model to identify human values behind given arguments for SemEval2023 Task 4 by Kiesel et al. (2023) by drawing attention to where the system shows no immaculateness, along with two other approaches. Stanley Grenz was the code name I used for the system submission. For those interested, the source code is publicly available on GitHub.1

heading: 2 Background
text: An argument is comprised of a premise (or premises) and a conclusion (Boltužić and Šnajder, 2014), and for this task, we have the additional information of stance: whether the premise is in favor of the conclusion or against it. While the object of the task still remained as human value identification, I included the stance information as a separate label and used the RoBERTa-Large model to predict the stance as well as the 20 human values, which in principle resulted in this becoming a stance detection task, too.
According to Al-Khatib et al. (2020), an argument generally includes a core claim with a few
1https://github.com/fidan-c/human-value-detection.git
1763
supporting evidence for that. As stance expresses information in terms of premise and conclusion relationship, predicting the stance could mean for the model to learn how an argument can be constructed with parallel or contrasting ideas.
In addition, stance classification or detection tries to find out the position of a text towards a certain topic that is generally more abstract and may not be explicated in the text (Kobbe et al., 2020), and considering that the secondary test set, Nahj al-Balagha, differs in the way arguments are expressed with implicit messages, as a result of which predicting human values can become extra challenging, stance prediction could be a possible help to alleviate this challenge.
I chose a pre-trained language model for this task. As Wang et al. (2022) states, pre-trained language models (PTMs) have made great achievements in the field of NLP, which shifted the way from supervised learning approaches to "pre-training and fine-tuning".
Since the task in question is in line with the Natural Language Understanding (NLU) problem (i.e. comprehending text sequences) transformerencoder-only (e.g. BERT and RoBERTa) architecture was chosen over transformer-decoder-only and transformer-encoder-decoder architectures.
More specifically, the system used RoBERTaLarge. For one of the two alternative approaches described later, another transformer model, BERTLarge (uncased), was used so as to compare performances. While RoBERTa-Large model is a model pre-trained on English language with the Masked Language Modelling (MLM) objective (Liu et al., 2019), BERT is a pre-trained language model on English—with MLM and Next Sentence Prediction goals (NSP) (Devlin et al., 2018).
Liu et al. (2019) points out that BERT was significantly undertrained, and with certain modifications, such as training the model with bigger batches on more data and on longer sequences and removing the NSP objective, an optimized model named RoBERTa was built.

heading: 3 System Overview
text: As it was briefly mentioned earlier, the main system is based on using the stance information as an additional label rather than using it as a part of the input. For this reason, the goal turned into predicting 21 labels in total, 20 of which are the given human values. For the input, the conclusion and the
premise columns in the dataset were concatenated together. While concatenating these two, I used the 
 token in-between. The input formation can be seen in Figure 1.
The input then was fed into RoBERTa (more specifically RoBERTa-Large), and I fine-tuned this pre-trained language model, which included setting the number of iterations to 5 and the the batch size for training to 16 to predict 21 labels: predicting the 20 human values and the stance. The following sections will elaborate on the system setup and performance further.

heading: 4 Experimental Setup

heading: 4.1 Data
text: For this task, two different argument datasets were used, one being the main dataset and the other being a secondary test set, Nahj al-Balagha, for the purpose of checking the robustness of the model.
The main dataset was already split into training (61% with 5393 rows/arguments), validation (21% with 1896 arguments) and test (18% with 1576) sets (arguments-training/validation/test.tsv) each of which includes 4 columns: "Argument ID", "Conclusion", "Stance" and "Premise", respectively. The premise’s stance towards the conclusion is described as in favor of and against. The labels for 20 human values were given as separate datasets with the corresponding splits (labelstraining/validation.tsv).
This main argumentation dataset is compiled from IBM-ArgQ-Rank-30kArgs (2019-2020), Conf. on the Future of Europe (2021–22), where English, French, and German are the main languages among the other 23 languages— they were automatically translated into any EU24 languages— and from Group Discussion Ideas (2021–22). Therefore, it is safe to state that this dataset in-
cludes English as the original language as well as automatic translations into English.
As for the secondary test set, Nahj al-Balagha, it contains 279 arguments extracted from Islamic religious texts written originally in Arabic and includes advice and sayings where ideas are delivered implicitly due to its eloquent content. This dataset was manually translated to English. Further elaboration on these two datasets can be found in Mirzakhmedova et al. (2023).

heading: 4.2 Data Pre-processing
text: Having used the aforementioned datasets, building the system required data pre-processing and model fine-tuning. Following the random seed being set to 64, stance was used as an output, and conclusion and premise were concatenated together with the 
 token in-between and included as an input. After lower-casing this new input, RoBERTa-Large model was used through the Simple Transformers library (Rajapakse, 2019).

heading: 4.3 Hyperparameter Tuning
text: Hyperparameter tuning was solely comprised of specifying the number of epochs as 5 and the batch size for training as 16.
The other hyperparameters had the default values of the Simple Transformers library for the Multi-Label Classification model architecture: 4e5 for the learnig rate, AdamW for the optimizer and Binary Cross-Entropy for the validation loss (Rajapakse, 2019).

heading: 4.4 External Libraries and Tools
text: The versions of the external tools and libraries used in the GoogleColab environment for the system are as follows:
• Python - 3.8.10.
• numpy - version: 1.21.6. 2
• pandas - version: 1.3.5. 3
• scikit-learn (Buitinck et al., 2013) - version: 1.0.2. 4
• simpletransformers (Rajapakse, 2019) - version: 0.63.9. 5
2https://www.numpy.org 3https://pandas.pydata.org 4http://scikit-learn.org 5https://github.com/ThilinaRajapakse/
simpletransformers/
• transformers (Wolf et al., 2020) - version: 4.26.1. 6

heading: 4.5 Experimental Setup - Other Approaches
text: Two more experiments were done to analyze input and the model difference; however, these were not submitted. I believe they still carry meaning that can be used to discuss the main approach.
First, for the sake of comparison on whether using stance as an additional label contributed to model performance or not, stance was included as a part of the input. In this setup, conclusion, premise and stance were concatenated together with the same data pre-processing that the main approach had (i.e. lower-casing and 
 token in-between), and everything else was kept the same (i.e. same seed number, using RoBERTa-Large with the same hyperparameters, etc.). This first alternative approach resulted in F1 = 0.45 on the argumentsvalidation dataset.
The last approach concerned changing the model. Stance was again used as a label, but with a different pre-trained language model. Instead of RoBERTa-Large, BERT-Large (uncased) was used, and everything else was kept the same except for lower-casing and replacing the 
 token: 
 with [SEP]. This setup, when evaluated on the main validation set, resulted in F1 = 0.39.
These two approaches were to determine which input design and which pre-trained model could contribute to the overall performance, and neither could outperform the main approach. The score details for the submitted/main approach will be given in the following section.

heading: 5 Results and Discussion
text: A glance at the published F1-scores for the main argument test set and the Nahj al-Balagha test set can give a broad idea for the performance of the main system this paper is focused on. Table 1 in the Appendix is provided for the readers in case of a need for looking into further details.
It is noticeable that the model performed better on the main argument test set than the secondary test set: F1 = 0.50—ranked ninth, with 0.56 being the highest score—compared to F1 = 0.35—ranked second, with 0.40 being the highest.
There are 10 labels on the main test set where F1score is lower than 0.50, with Stimulation having the lowest score of 0.10, followed by Humility with
6https://github.com/huggingface/transformers
a score of 0.12. The top score is seen in Security: personal with 0.77.
For the secondary test set, the label that got the highest F1-score is Achievement with 0.66 while Security: personal was ranked third with 0.51. Similar to the main test set, Stimulation and Humility are among the lowest scoring labels, with 0.00 and 0.08 respectively.
In fact, 4 labels that each have F1 = 0.00 are in this secondary test set: Stimulation, Power:dominance, Power:resources and Universalism:tolerance. Another 13 labels are below 0.35.
Across these two datasets, one can deduce that Stimulation and Humility were poorly predicted. There can be several reasons for that, but one that can be already seen in Mirzakhmedova et al. (2023) is that the data frequency for these two human values is relatively low for the main dataset.
However, though one can consider the influence of data frequency as a possible reason—which might be a factor for Power:resources having F1score of 0.00 for the secondary test—Table 2 in the Appendix shows another participant’s system (code name r-m-haare) giving F1 = 0.33 for this label on the secondary test set. It is intriguing to hypothesize the potential reasons behind this result.
Another question arises pertaining to the way of expressing ideas based on where the arguments were taken from. As mentioned earlier, the secondary test set has a nature of including covert messages embedded in its eloquent content, and that might be a challenge for the pre-trained models. For this, training on bigger data with more variety may help to better predict the hidden human values behind covert arguments.

heading: 6 Conclusion
text: This paper presented a system to predict 20 human values behind arguments by using a pre-trained language model, RoBERTa-Large, and with specific hyperparameter tuning; the main strategy employed was to use stance as a label, which meant that the system was set to predict the human values and the stance for each argument. For comparison, two other approaches were also described in the paper— one for comparing the effect of another pre-trained model and the other for the impact of including stance in the input.
Overall, it was shown that the main system outperformed the other two. Despite the reasonable rank that the system got in SemEval-2023 Task 4,
a close look into where the system did not perform well was worth delving into.
Based on the secondary test set results, it seems that language use for expressing ideas may vary depending on the content. This can be a phenomenon to investigate in a future comparative study with a bigger and a more balanced dataset, which could potentially give an insight into the performance of pre-trained models for pragmatics and see whether or not human values in arguments can be predicted better despite implicity and covertness. Another suggestion is linked to transfer-learning: a model trained for ideology detection can be used on this task to see if it would contribute to the system performance on identifying human values behind arguments.

heading: Future Work
text: This can be a phenomenon to investigate in a future comparative study with a bigger and a more balanced dataset, which could potentially give an insight into the performance of pre-trained models for pragmatics and see whether or not human values in arguments can be predicted better despite implicity and covertness. Another suggestion is linked to transfer-learning: a model trained for ideology detection can be used on this task to see if it would contribute to the system performance on identifying human values behind arguments