Lauri Ingman at SemEval-2023 Task 4: A Chain Classifier for Identifying Human Values behind Arguments
Year: 2023

Abstract:
Identifying expressions of human values in textual data is a crucial albeit complicated challenge, not least because ethics are highly variable, often implicit, and transcend circumstance. Opinions, arguments, and the like are generally founded upon more than one guiding principle, which are not necessarily independent. As such, little is known about how to classify and predict moral undertones in natural language sequences. Here, we describe and present a solution to ValueEval, our shared contribution to SemEval 2023 Task 4. Our research design focuses on investigating chain classifier architectures with pretrained contextualized embeddings to detect 20 different human values in written arguments. We show that our best model substantially surpasses the classification performance of the baseline method established in prior work. We discuss limitations to our approach and outline promising directions for future work.

heading: 1 Introduction

heading: 1.1 Motivation
text: Human values are inextricably linked to the formation of thoughts, opinions, and actions. They underlie an individual’s moral, ethical, and philosophical convictions which are manifest in their worldview. In essence, public and private discourse are expressions of human values. As online communication platforms increasingly provide social alternatives to the "town square," understanding written content, as it relates to the human values contained within it, is an interesting and important NLP challenge. ValueEval presents a unique opportunity to investigate the human values behind the formation and presentation of opinions on controversial topics, particularly those with social, economic, and political implications (Kiesel et al., 2023).
The main objective of ValueEval1 is to detect 1https://touche.webis.de/semeval23/
touche23-web/index.html#evaluation
human values in arguments. Given an argument with three attributes and a finite set of human value categories, the system must classify whether or not the argument is associated with a given value for all values in the set. ValueEval is composed of a single task rather than sub-tasks; however, the committee specifies the option to focus on a subset of the 20 human value categories for classification. Here, we employ a classification model using all 20 human value categories.

heading: 1.2 Related Work
text: Since its debut, BERT (Devlin et al., 2018) has become a bedrock of many NLP applications (Tenney et al., 2019). BERT enables researchers to expedite the task of creating contextual word embeddings. More recently, newer alternatives to BERT have become increasingly popular, including the GPT models created by OpenAI. Yet, BERT continues to transform the traditional architecture for NLP problems: provided a sample as input to a pre-trained transformer language model, one can generate contextualized embeddings to then be passed to downstream tasks. Following in the steps of many others, we too utilize BERT in this way in our proposed solution to the present multi-label classification problem.
Multi-label classification is not new, and there are numerous methods proposed in the literature, which typically span one of two categories: problem transformation and algorithm adaption (Zhang and Zhou, 2014). Problem transformation methods decompose the complexities of multi-class classification into more familiar approaches. The simplest method of doing so is to treat each possible class as a binary classification problem (Zhang and Zhou, 2014), but this tends to be rather crude and ignores potential inter-class dependencies. On the other hand, algorithm adaption methods attempt to reshape existing methods to directly accommodate the multi-label data. This, however, requires exten-
193
sive knowledge of the method to be adapted itself as well as the changes to the loss metric required to handle the new labels.
All things considered, we contend that chain classifiers are a particularly good algorithm to approach the present problem because they grant the simplicity of binary classifiers in tandem with the capability to incorporate inter-label dependencies (Read et al., 2011). Each classifier in the chain is, in addition to the original input, provided with all predictions from preceding classifiers. In a 10-class setting, for instance, the first classifier in the chain receives the base input, while the tenth (and final) classifier receives the base input and the predictions for the nine previous classes. This provides the model more information to discover inter-class dependencies and make informed choices. Notably, this approach significantly relies on the order of classifiers, the importance of which can be circumvented by building an ensemble of chains with different orderings (Zhang and Zhou, 2014).

heading: 2 Methods

heading: 2.1 Data
text: The data for this task2 are adapted from the task committee’s previous work (Kiesel et al., 2022). We used the provided Argument dataset for training. This dataset contains 5220 uniquely identified natural language arguments (presented in English) based on African, Chinese, Indian, and American cultures (Mirzakhmedova et al., 2023). Each argument is comprised of a conclusion, stance, and premise string. The conclusion of an argument concisely describes the ultimate judgement or decision, the stance indicates whether the conclusion is in the affirmative or the opposition, and the premise statement justifies the conclusion. Each argument is also annotated with a binary vector of 20 human value categories, such that a given argument can be
2https://zenodo.org/record/6818093# .Y4pqGuzMLIA
founded upon more than one human value. Table 1 provides a sample of five of the human value categories relative to an example argument from the dataset, which contains sentiments that evoke those particular values.
In preparation for our model, we sample the data uniformly at random to create a training set (80%) and development set (20%). Figure 1 displays the distribution of all possible human value labels across the training and development sets. While the representation of each human value is not balanced across the entire dataset, our training/development split is proportionally balanced within each human value category, relative to the composition of the original training dataset. We preprocess the data using subword-based tokenization with Byte-Pair Encoding from a small pretrained BERT model variant intended for use with limited computational resources (Bhargava et al., 2021; Turc et al., 2019). We adapt the traditional pairwise input method which is often used for question-answering models, to create an input representation suitable for both the BERT model and the input triples in our data (i.e., conclusion, stance, premise).

heading: 2.2 Model
text: Although we tested multiple architectures, our final model is divided into two main components, as illustrated in Figure 2. First, we use a pretrained BERT model to generate contextualized embeddings for each of the conclusion, stance, and premise of a given sample. We then extract the pooling layer output of BERT, which corresponds to BERT’s <CLS> token and serves as a contextualized embedding of the entire input. We pass that output to a hidden layer, comprised of one or more linear layers, to project to a specific dimension.
The output of the hidden layer is then connected to the head of the classifier chain. Each classifier in the chain is relatively simple; it includes a dropout layer for regularization, a linear layer for the pre-
diction, and a sigmoid activation function. Each classifier makes a prediction on its assigned class, which is then appended to the input and passed to the next classifier in the chain. This process repeats for all 20 classifiers in the chain, at which point all generated predictions on each class are returned.

heading: 2.3 Experimental Design
text: Each model is trained over 100 epochs with the AdamW optimizer. Preliminary results show more
epochs yield no improvements. We train our model with Binary Cross-Entropy loss in order to make 20 binary classifications. For each training session, we change up to two of a number of model-defining parameters–either architecture choices or hyperparameters. For our architecture, we experiment with unfreezing BERT weights, adjusting the number of layers in each classifier in the chain, and varying the number of hidden layers between BERT and the classifier chain. In terms of hyperparameters, we experiment with varying the dropout probability, the hidden size to project to, batch size, learning rate, and the classification threshold.
The number of hidden layers and prediction threshold appear to be the parameters with the largest impact on performance. We observe that due to the complexity of the task, adding an additional layer between BERT and the classifier chain improves performance; yet, a third layer slightly decreases performance, as shown in Table 3. We use the prediction threshold to determine the binary prediction status of an output. We initially define a threshold of 0.5, but discover that associating the threshold with the data label density significantly improves performance. That is, we incorporate the label density into the predictions in order to leverage the general class distribution. Our best results are associated with a threshold of 0.3426, twice the label density of the entire dataset. The results from our various training sessions are listed in Table 3 in Appendix A.

heading: 2.4 Evaluation Metrics
text: We compare models using two different F1-scores. We calculate a micro F1-score on the development
set after every epoch to gauge our model’s performance during training. This micro F1-score does not calculate an F1-score per class, but instead assumes an entry-by-entry approach. After training, we use the provided evaluation method, which calculates a per-class F1-score before averaging. In most cases, our score is reasonably close to the official score, so we continue to use our method for simplicity.

heading: 3 Results
text: We reference the original task paper to establish a baseline (Kiesel et al., 2022). The authors separate tasks into different levels of granularity. We focus on Level 1, the highest granularity. Kiesel et al. (2022) test three Level 1 baseline models: BERT, an SVM, and a simple model that always outputs 1. The highest F1-score in this context is 0.25, with BERT. Our model achieves an F1-score of 0.67, a significant improvement from the baseline. We compare our results with the F1 evaluation metric in Table 2. Our competitive model performance in the official testing session, relative to other teams, is publicly available3.

heading: 4 Discussion

heading: 4.1 Limitations and Future Work
text: Our most notable restriction on performance optimization is model size. We do not observe evidence of overfitting during training, which suggests that a larger model could potentially learn more. We further see that additional layers and parameters in our model are associated with top results. Even still, as model size increases, so do the demands of computing resources and training time, both of which posed strict limitations on our ability to expand. Unfreezing the BERT parameters, for instance, resulted in reduced model performance; if
3https://touche.webis.de/semeval23/ touche23-web/index.html#results
computationally feasible, implementing a different learning rate for BERT than that of the newly initialized classifier may be a strategy worth pursuing. Accordingly, we did not explore other models such as GPT or ALBERT to generate contextualized embeddings, which may improve performance, at the cost of training speed and computation complexity.
There are many multi-label classification methods to explore in this context. Classifier chains, while superior to independent classifiers, are somewhat limited in their exploitation of inter-class dependencies. Other approaches, such as ensemble methods, may more intelligently incorporate interclass dependencies to identify samples correctly. Our simple chain classifier could be augmented with additional chains using different permutations of class ordering.
Data augmentation is another direction worthy of future investigation. We note that the the present implementation of our model was developed with the provided training dataset exclusively, prior to the release of the official validation and test datasets. The training set used here is not substantial, despite the relatively small size of our model. Aside from using the full official datasets, a model like GPT could generate additional samples from those given, as slightly noisy data could improve training performance. A more expansive and complex model might be required to leverage the benefits of more data, imposing considerable tradeoffs.

heading: 4.2 Conclusions
text: Overall, our model offers a vastly improved alternative to the relatively simplistic baseline models, and attains results competitive with the approaches of other teams. The class labels associated with human values in this context are, evidently, not independent. We observe values to often co-occur in groupings, and thus emphasize that additional functionality to account for inter-class dependencies within the model is a key component to optimize classification performance. We believe this to be our main model attribute that contributes to improvement over the baselines, none of which contain special features to leverage inter-class dependencies. Moreover, we note that our model trains quickly, only requiring 20 to 30 minutes to complete each training session, which we deem to be another major strength relative to the complex nature of the task. We acknowledge that a larger model may outperform ours, but we underscore that
the reduction in training time and feasible computational load of small models such as ours are highly beneficial for development. All code and data for our implementation are available on GitHub4.

heading: 5 Acknowledgements
text: We thank Adam Wiemerslage, Rehan Ahmed, and James Martin for their helpful guidance.

heading: A Appendix

heading: Future Work
text: Data augmentation is another direction worthy of future investigation. We note that the the present implementation of our model was developed with the provided training dataset exclusively, prior to the release of the official validation and test datasets. The training set used here is not substantial, despite the relatively small size of our model. Aside from using the full official datasets, a model like GPT could generate additional samples from those given, as slightly noisy data could improve training performance. A more expansive and complex model might be required to leverage the benefits of more data, imposing considerable tradeoffs