Sina at SemEval-2023 Task 4: A Class-Token Attention-based Model for Human Value Detection
Year: 2023

Abstract:
The human values expressed in argumentative texts can provide valuable insights into the culture of a society. They can be helpful in various applications such as value-based profiling and ethical analysis. However, one of the first steps in achieving this goal is to detect the category of human value from an argument accurately. This task is challenging due to the lack of data and the need for philosophical inference. It also can be challenging for humans to classify arguments according to their underlying human values. This paper elaborates on our model for the SemEval 2023 Task 4 on human value detection. We propose a class-token attentionbased model and evaluate it against baseline models, including finetuned BERT language model and a keyword-based approach.

heading: 1 Introduction
text: The social sciences and humanities provide insight into understanding the world and its people, with a primary responsibility of solving human-based issues and providing recommendations. The study of human argumentation and causality is an approach that aids in understanding human relationships and culture, with applications in areas such as faceted search (Amsterdamer and Gáspár, 2022), valuebased argument generation (Bostrom et al., 2022), and value-based personality profiling (Liu et al., 2019). The Semantic Evaluation 2023 includes the human value detection task (Kiesel et al., 2023), which aims to classify the human value category based on textual argument.
Our study investigates the effectiveness of keyword extraction and attention-based neural models. Our findings indicate that while context keywords contain the primary argument value, incorporating the class embedding of arguments as queries that
*Equal contribution
focus on the most important concepts of each argument can improve classification results. We also observed that simple models like SVM (Cortes and Vapnik, 1995) perform well compared to neural networks due to the dataset’s small size, multi-class prediction, and numerous labels.
We participated in the human value detection task at TIRA (Fröbe et al., 2023) and achieved an average score of 0.47 for all labels, which was 0.09 less than the first team’s score. However, we attained the best F1 score of 0.54 for the Power: Resources label, which was 0.01 better than the top-performing approach. These findings suggest that there is room for improvement in the task.
To facilitate easier evaluation and reproducibility of our results, we have made our baselines and proposed models1 available open-source as several Jupyter-notebooks and docker images.

heading: 2 Background
text: Human values are ubiquitous in social sciences, and identifying them in argumentative texts can help understand cultures, conflicting beliefs, and opinions. In the human value detection classification task, it is required to determine human values, given human arguments containing premises and conclusions (Kiesel et al., 2022).

heading: 2.1 Dataset
text: The task dataset comprises 9324 arguments with corresponding classes from various sources, such as political and religious texts, newspapers, and free-text arguments (Mirzakhmedova et al., 2023). The train, validation, and test sets contain 5393, 1896, and 1576 data items. There is an additional validation and test dataset from community discussion and religious texts. For more information
1https://github.com/language-ml/ human-value-detection
2164
about the dataset, refer to Table 1. The dataset contains three components: premise, stance, and conclusion representing a moral inference. The objective is to determine the value type employed to make this inference. This task is multi-label and multi-class classification.

heading: 3 System Overview
text: This section provides a review of the baseline methods and the proposed methods for addressing the task at hand, including keyword extraction and attention-based models.

heading: 3.1 Baselines
text: We utilized two baseline models for our experiments: a Support Vector Machine (SVM) and a fully connected neural network. To obtain the sentence embedding, we combined each input sentence’s premise, stance, and conclusion parts and fed them to the LABSE model (Feng et al., 2022), which we found to be more appropriate than traditional models like Word2Vec (Mikolov et al., 2013) or BERT (Devlin et al., 2019) for our task.
The fully connected neural network consisted of four layers, and to ensure stable training, we incorporated batch normalization (Ioffe and Szegedy, 2015). We also used dropout (Srivastava et al., 2014) to counter overfitting and improve the model’s accuracy. We utilized the sigmoid activation function in the last layer with a threshold of 0.5 to predict the output class. On the validation dataset, this model achieved macro-F1 and microF1 scores of 0.32 and 0.47, respectively. The SVM model with a linear kernel and LABSE embedding obtained macro-F1 and micro-F1 scores of 0.31 and 0.46 on the validation dataset, respectively.
Our experiments demonstrated that even simple models such as SVM can perform comparably to neural networks for our task. To further enhance the performance of the SVM, we employed an ensemble of seven SVMs with different kernel functions, including polynomials (with 2, 3, and 4 degrees), RBF, and sigmoid. We randomly se-
lected the kernel function for each SVM to reduce their total variance. This ensemble model achieved macro-F1 and micro-F1 scores of 0.41 and 0.51, respectively, on the validation dataset.

heading: 3.2 Keyword Extraction
text: Since keywords within text data can encompass the fundamental concepts of the text, and these concepts are essential in deriving conclusions from premises, the personal values of the individual making inferences can influence these keywords. Consequently, we developed an approach based on keywords to predict human values. This involved extracting keywords from each human value’s class descriptions and training data using Yake (Campos et al., 2020). We then assigned positive labels to data classes with scores above a pre-defined threshold based on the number of intersections between arguments in the test data and each class’s keywords. We also repeated this approach using the embeddings of keywords instead of the surface forms. Despite these efforts, both experiments yielded poor results, with an average F1 score overall categories of only 0.28, which was only slightly better than the 1-Baseline provided by organizers.
Therefore, we can conclude that although the main concepts of argumentative texts may contain informative data about human values, there are strong hidden connections between keywords and human values that result in the human values not automatically discernibly classifiable.

heading: 3.3 Attention-based Model
text: In this model, we utilized BERT to compute embeddings for every token in the concatenated input. Subsequently, we implemented an attention layer over these embeddings to generate a singular embedding. This attention layer utilized the token embeddings as both the value and key, while a classspecific embedding served as the query. These class-specific embeddings were randomly initialized and then learned during the training phase. The final step involved using a binary classifier to predict if the input belonged to the selected class. The entire network was trained end-to-end without freezing the BERT model. We provided a visual representation of this architecture in Figure 1.
To address the issue of imbalanced training data,
where most of the labels were negative, we oversampled the positive data to achieve balance in each epoch. Our experiments showed that without this technique, the models did not converge.

heading: 4 Results
text: Table 2 presents the results obtained by applying the method described in section 3.3. We compare it with the best approach, baselines provided by the organizers, and the best model per category. The overall average F1 score for 20 labels was 0.47, which outperformed the organizers’ 1-Baseline and BERT models. Our difference with the best approach was 0.09, and we even surpassed it by 0.01 in the F1 score of the power: Resources label. These results show that using class-specific embedding can be effective in any attention-based approach.
For Nahj al-Balagha data, the result of our model was better than the 1-Baseline; it could not improve the BERT baseline, and to improve the result, we need more related human-value data from a religious source.

heading: 5 Conclusion
text: Detecting human values is a practical task that challenges natural language processing methods because of the necessity to comprehend moral and philosophical concepts. This paper proposes a solution to this problem by learning class-specific embedding and utilizing an attention mechanism to find the best features for a binary classifier. Although we have introduced novel models to address this issue, we have not achieved the desired level of accuracy because we used BERT-base instead of models with more parameters and a single transformer-based model due to our computational resource constraints. Given the paucity of data for numerous classes, unsupervised techniques such as training a dedicated language model for philosophical and moral text could be utilized in future research.

heading: Future Work
text: Given the paucity of data for numerous classes, unsupervised techniques such as training a dedicated language model for philosophical and moral text could be utilized in future research