LRL_NC at SemEval-2023 Task 4: The Touche23-george-boole Approach for Multi-Label Classification of Human-Values Behind Arguments
Year: 2023

Abstract:
The task ValueEval aims at assigning a subset of possible human value categories underlying a given argument. Values behind arguments are often determinants to evaluate the relevance and importance of decisions in ethical sense, thereby making them essential for argument mining. The work presented here proposes two systems for the same. Both systems use RoBERTa to encode sentences in each document. System1 makes use of features obtained from training models for two auxiliary tasks, whereas System2 combines RoBERTa with topic modeling to get sentence representation. These features are used by a classification head to generate predictions. System1 secured the rank 22 in the official task ranking, achieving the macro F1-score 0.46 on the main dataset. System2 was not a part of official evaluation. Subsequent experiments achieved highest (among the proposed systems) macro F1-scores of 0.48 (System2), 0.31 (ablation on System1) and 0.33 (ablation on System1) on the main dataset, the Nahj al-Balagha dataset, and the New York Times dataset.

heading: 1 Introduction
text: Judgement and opinion of each person are typically governed by a set of principles which is based on a certain value system. Gaining an understanding of human values behind a natural language argument can serve an important role in the field of social sciences, and policy making.
The aim of the current task (Kiesel et al., 2023) is to assign to each argument a set of underlying values, where each argument is given in the form of a Conclusion, Stance and Premise of the argument. With each argument having multiple underlying values, the task effectively boils down to a multi-label classification task, since each argument can have multiple underlying values. English language Touché23-ValueEval Dataset (Mirzakhmedova et al., 2023) has been used for the present challenge.
The labels for the challenge are given as a multilevel taxonomy of 54 human values (Kiesel et al., 2022), called Level 1 labels which are further grouped into 20 human values categories called Level 2 values. The aim of the task is to predict these Level 2 values.
The present work describes two systems: System1 and System2. Both the systems use RoBERTa (Liu et al., 2019) to extract sentence encodings. System1 incorporates Level 1 label information. This system also incorporates linguistic information, learned by predicting the Stance of an argument presented by a Conclusion based on a certain Premise. System2 uses topic modeling to incorporate corpus-level semantic information. System1 was submitted for official evaluation though it was later observed that System2 achieves a higher macro F1-score as compared to System1.
The participating teams were required to make their submissions on the platform TIRA (Fröbe et al., 2023). The official task evaluation platform ranks System1 on the main dataset at the position 22. The proposed systems either outperform or perform at par with the organizer’s baseline. But, the systems struggle at capturing low frequency labels.
The paper is organized as follows. Section 2 discusses the task background and Section 3 discusses related past works. Section 4 provides a detailed system overview. The experimental setup and results are given in Section 5 and Section 6, respectively. The paper is concluded with Section 7. The code for the proposed systems have been made available on GitHub.1

heading: 2 Background
text: The data made available by the organisers comprises of a main dataset and three supplementary datasets, namely, Zhihu, Nahj al-Balagha and The
1https://github.com/KushagriT/SemEval23_ValueEval_ TeamLRL_NC
136
New York Times dataset. The main dataset consists of Train/ Validation/Test counts as 5393/ 1896/ 1576. The task dataset is a collection of 9324 arguments on statements which include religious texts (Nahj al-Balagha), political discussions (Group Discussion Ideas), free-text arguments (IBM-ArgQRank-30kArgs), newspaper articles (The New York Times), community discussions (Zhihu), and democratic discourse (Conference on the Future of Europe). The supplementary Zhihu dataset consists of 100 validation instances. Nahj al-Balagha and The New York Times datasets have 279 and 80 test examples, respectively.
A general instance of the dataset can be interpreted as, Arguing in favor of/against a <Conclusion> by saying <Premise>. An example of such an instance of the dataset, which has been assigned labels or value categories as Security: societal and Universalism: concern, is given in Table 1.

heading: 3 Related Past Works
text: Recent works have explored role of values and frames in argument mining. A frame is a set of arguments that focus on the same aspect. Ajjour et al. (2019) introduces frame identification towards splitting a set of arguments into a set of nonoverlapping frames. They present a fully unsupervised approach to the task which identifies frames using clustering. Kobbe et al. (2020) present models for automatically predicting moral sentiment in debates and explore how moral values in arguments relate to argument quality, stance, and audience reactions. Trautmann (2020) discusses the task of Aspect-Based Argument Mining (ABAM), while Schiller et al. (2021) present a language model for argument generation to generate sentence-level arguments for a given topic, stance, and aspect. Maheshwari et al. (2017) attempt to understand whether individuals in a community possess simi-
lar personalities, values and ethical background. Several recent works also make use of human values. Qiu et al. (2022) present ValueNet, which is a large-scale text dataset for human value modeling. The dataset contains human attitudes on a number of text scenarios. Hendrycks et al. (2023) introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. They also analyse the ability of language models to predict basic human ethical judgements. Ammanabrolu et al. (2022) create interactive chat agents which act in alignment with socially beneficial norms and values. Liu et al. (2023) introduce a new learning paradigm Second Thoughts that can make current language models aware of the human value alignment.

heading: 4 System Overview
text: This section discusses System1 and System2 as mentioned in Section 1 . The first step in the systems is to generate an encoding for each instance using RoBERTa (Liu et al., 2019).
Figure 1 and Figure 2 give the architecture of the System1 and System2, respectively.

heading: 4.1 System1
text: The core idea for this system is to combine the information of factors that classify each argument into Level 1 labels with the information which can predict the stance indicated by the conclusion drawn from a particular premise. To generate sentence encodings, the RoBERTa model has been fine-tuned for Masked Language Modeling (MLM) task on the main task dataset. Each instance consists of three entries, namely, Conclusion, Stance and Premise, and input text is represented as, <Conclusion> + ‘ 
 ’ + <Stance> + ‘ 
 ’ + <Premise>, where 
 is the separator token. The RoBERTa layer of this fine-tuned model is called MyRoBERTa in the rest of the paper.
This system consists of two sub-models trained for related auxiliary tasks.
• The first sub-model, called Model_Level1, is essentially the MyRoBERTa model fine-tuned for multi-label classification on the set of 54 Level 1 labels from the human values taxonomy. It takes as input <Conclusion> + ‘ 
 ’ + <Premise>. This model is trained using Binary Cross Entropy loss which is preceeded by
application of sigmoid layer on the output.2.
• The second sub-model, called Model_Stance, is essentially a MyRoBERTa model fine-tuned for binary classification on stance of the argument given the input text, <Conclusion> + ‘ 
 ’ + <Premise>. This model is trained using Cross Entropy loss.
The model for multi-label classification on the set of Level 2 labels of size 20, is called
2BCEWithLogitsLoss in huggingface transformers
Model_Level2. The first layer of which is MyRoBERTa3. The CLS embedding from this layer is input to a dense neural network layer with input and output dimensions as embedding size of the RoBERTa model. The output from this layer is concatenated with the output from the first dense layer of the classification head of each of the two auxiliary models trained above. Tanh activation is applied to this concatenated embedding and input to another dense neural network layer with output dimension as the target space size corresponding to the number of labels. This model is trained using Binary Cross Entropy loss which is preceded by application of sigmoid layer on the output.
While training the three models Model_Level1, Model_Stance, and Model_Level2, all the parameters of the MyRoBERTa layer are frozen, except the parameters corresponding to the last and second-tolast encoder layer.

heading: 4.2 System2
text: This approach combines topic modeling with RoBERTa. Topic modeling takes into account the general semantic properties of the entire corpus. Analysis of the corpus reveals that often the arguments are centered around certain human values and tend to involve some specific words. Moreover, they exhibit a significant correlation between the semantically coherent units or topics that are represented by the data. Hence we train a Correlated
3The pooling layer is not added to this RoBERTa model.
Topic Model (CTM) (Lafferty and Blei, 2005) on the corpus, where each instance is transformed as ‘Arguing ’+ <Stance> + <Conclusion> + ‘ by saying ’ + <Premise>. CTM produces a K-dimensional vector which represents the topical composition of that argument.
The first layer of this system is RoBERTa4 from pretrained roberta-base model. The instance ‘Arguing ’+ <Stance> + <Conclusion> + ‘ by saying ’ + <Premise> is input to this model, and the corresponding CLS embeddings are concatenated with the K-dimensional text topic proportions. This serves as an encoding for each instance. This encoding is passed as input to a classification head, of architecture as described in case of Model_Level1 of System1. This model is trained for multi-label classification on the set of Level 2 labels of size 20, using Binary Cross Entropy loss on the output, where a sigmoid layer has been applied to the output to convert the values to probabilities.

heading: 5 Experimental Setup
text: The experiments were carried on Google Colaboratory in Python 3.8.10 with Nvidia Tesla P100 GPU. PyTorch (Paszke et al., 2019), and Huggingface Transformers (Wolf et al., 2020) were used as the key frameworks for the experiments.
Training is carried on the Train + Validation subsets of the main dataset. For validation, the Validation subset of the supplementary Zhihu dataset, has been used in order to make the system more robust.

heading: 5.1 Description of MyRoBERTa
text: For finetuning RoBERTa for MLM task the following settings were followed.
• The roberta-base is fine-tuned on the combined Train + Validation subsets of the main dataset.
• The text was tokenized using RobertaTokenizerFast from Huggingface Transformers and RoBERTa special tokens were added. The data was prepared by masking tokens in the input with probability 0.15. The instances were padded to the maximum sequence length in the batch. Truncation was done at a sequence length of 256. The model was implemented using RobertaForMaskedLM from Huggingface Transformers.
4The pooling layer is not added to this RoBERTa model.
• The model was fine-tuned with batch size 4, for 6 training epochs and learning rate 5e-5.

heading: 5.2 Description of System1
text: To train Model_Level1 and Model_Stance the following steps were used.
• The data was prepared and batched by sorting the input text, <Conclusion> + ‘ 
 ’ + <Premise> according to text length, padding to maximum document length in the batch, and truncating at a maximum sequence length of 256. The text was tokenized using RobertaTokenizerFast, and RoBERTa special tokens were added.
• The optimizer used is Adam with weight decay (Kingma and Ba, 2014; Loshchilov and Hutter, 2017).
• The task of Model_Level1 is multi-label classification with 54 labels, and the task of Model_Stance is binary classification with 2 labels.
• The hyperparameters for the model are given in Appendix.
The two sub-models are not further trained as a part of the architecture of Model_Level2. The model settings for both the sub-models and Model_Level2 are given in the Appendix. The final model state for Model_Level2 was selected according to the highest macro F1-score on the development dataset.

heading: 5.3 Description of System2
text: The Correlated Topic Model (CTM) was trained using using tomotopy package5 in Python. The text was preprocessed using the simple_preprocess in Gensim (Řehůřek and Sojka, 2010). The standard English stopwords were removed using the NLTK toolkit6. The words having POS (part-of-speech) tags as NOUN, ADJ (adjective), VERB and ADV (adverb) were lemmatized. But proper nouns were retained without change. The POS tagging and lemmatization were done using Spacy7 framework in Python.
The settings for training the CTM model are given in Appendix. For training the main model,
5https://bab2min.github.io/tomotopy/v/en/ 6https://www.nltk.org/ 7https://spacy.io/
the data was prepared by considering the input text, ‘Arguing ’+ <Stance> + <Conclusion> + ‘ by saying ’ + <Premise>, and sorted according to text length. The instances in a batch were padded to maximum document length in the batch, and truncation was done at a maximum sequence length of 256. The final model state was selected according to the highest macro F1-score on the development dataset. The model settings used for training the main multi-label classification model are given in Appendix.

heading: 6 Results
text: Ablation studies on System1 were conducted as follows. The results are given in Table 2.
• Ablation 1.1: System1 without Model_Level1
• Ablation 1.2: System1 without Model_Stance
• Ablation 1.3: System1 without Model_Level1 and Model_Stance
In each of three test sets the submitted systems either outperform or perform at par with the organizer’s BERT and 1-Baseline. Among the proposed systems, System2 performs the best on the Main
test set. Whereas, Ablation 1.2 performs best in case of Nahj al-Balagha test set. In case of New York Times test set, the Ablation 1.1 performs almost at par with the best participant model on the official task leaderboard based on the the macro-F1 score. This can be attributed to the use of the supplementary Zhihu subset for validation, improving the robustness of the system. It is observed that use of topic modeling improves the performance of the system, in general.
In our experiments, on the main dataset, the F1score is highest for eleven labels, for System2. Similarly, on the Nahj al-Balagha dataset, for five labels, the the F1-score is highest for Ablation 1.2. For this dataset, System2 performs at par with the BERT baseline. On the New York Times dataset, for five labels, the F1-score is highest for Ablation 1.1.

heading: 7 Conclusion
text: The current work proposes two systems for the task of value identification in arguments. This is broadly a multi-label classification task. The proposed systems use RoBERTa to get sentence encodings for each document. System2, combines RoBERTa with topic modeling to get sentence representation. This is followed by a classification head to generate the label subsets. The other system, namely, System1, also makes use of features obtained from training models for auxiliary tasks. The auxiliary tasks, are that of prediction of Level 1 values from the human-value taxonomy given an argument, and stance prediction given conclusion and premise of an argument.

heading: A Appendix
text: All transformer-based models were trained using Trainer from Huggingface Transformers.
For finetuning RoBERTa for MLM task, the dataset was batched using, DataCollatorForLanguageModeling.
The model settings for both the sub-models and Model_Level2 are given in Table 3. These models
are implemented using RobertaForSequenceClassification from Huggingface Transformers, with pretrained model as MyRoBERTa.
The settings for training CTM model are given in Table 4. The settings for train the main multilabel classification model in System2, are given in Table 5.

heading: Future Work