ValueEval: Identification of Human Values Behind Arguments
Year: 2023

Abstract:
Argumentation is ubiquitous in natural language communication, from politics and media to everyday work and private life. Many arguments derive their persuasive power from human values, such as self-directed thought or tolerance, albeit often implicitly. These values are key to understanding the semantics of arguments, as they are generally accepted as justifications for why a particular option is ethically desirable. Can automated systems uncover the values on which an argument draws? To answer this question, 39 teams submitted runs to ValueEval’23. Using a multi-sourced dataset of over 9K arguments, the systems achieved F1-scores up to 0.87 (nature) and over 0.70 for three more of 20 universal value categories. However, many challenges remain, as evidenced by the low peak F1-score of 0.39 for stimulation, hedonism, face, and humility.

heading: 1 Introduction
text: How come people disagree on the best course forward in controversial issues, even if they use the same information to form their opinion? We observe that people have different beliefs and priorities of what is generally worth striving for (e.g., personal achievements vs. humility) and how to do so (e.g., being self-directed vs. respecting traditions), often referred to as (human) values (Searle, 2003). Some values tend to conflict, others tend to align (see Figure 1). This can cause disagreement on the best course forward, but also the support, if not formation, of political parties that promote respective highly revered values.
Due to their outlined importance, human values have been studied both in the social sciences (Schwartz, 1994) and in formal argumentation (Bench-Capon, 2003) for decades. According to the former, a “value is a (1) belief (2) pertaining to desirable end states or modes of conduct, that
∗ Contact: johannes.kiesel@uni-weimar.de
(3) transcends specific situations, (4) guides selection or evaluation of behavior, people, and events, and (5) is ordered by importance relative to other values to form a system of value priorities.” Consider the following example argument:
“Social media is good for us. Though it might make people less polite, it fosters free speech.”
To understand this argument, a reader has to acknowledge the belief (Point 1 in the definition) that the “end state” (2) of allowing for self-directed action, as in free speech, is desirable in general (3). To concur with the statement (4), the reader further has to prefer this state, in doubt, over interpersonal conformity (politeness; 5)—ignoring other arguments on the topic for the sake of the example.
The identification of human values behind arguments by means of natural language processing
2287
can assist in argument categorization, evaluation, and generation. Also, it may support the research of social scientists. The identified values can enable models of audience-specific argument strength (Bench-Capon, 2021), and it can support studies of morals (Feldman, 2021; Alshomary et al., 2022) and frames (Entman, 1993; Ajjour et al., 2019).
To advance computational methods of human value identification, we organized the ValueEval’23 shared task at SemEval as part of the Touché series.1 Simply put, the task is to classify whether a given argument resorts to a given value (Section 3). As part of the shared task, we extended our previous dataset (Kiesel et al., 2022) to more than 9K arguments (Section 4). The task attracted submissions from 39 teams (Section 5), which we benchmarked and analyzed on our dataset (Section 6). Overall, the teams achieved respectable results, beating our BERT baseline by a large margin on all dataset parts for all 20 value categories.

heading: 2 Related Work
text: A dedicated line of research studied human values extensively. Rokeach (1973) first defined values as certain end states or modes of conduct that humans desire. Accordingly, they introduced the value system as a prioritization over these values based on cultural, social, and personal factors. The authors developed a practical survey of 36 values distinguishing between end states and behavior. For cross-cultural analysis, Schwartz et al. (2012) derived 48 value questions from universal individual and societal needs, including concepts such as obeying all the laws and being humble. Cheng and Fleischmann (2010) consolidated 12 taxonomies into a “meta-inventory” with 16 values, revealing significant overlap. Another effort in unifying value taxonomies is the linked open data scheme ValueNet by Giorgis et al. (2022), though the authors do not compare the taxonomies as such. Based on these taxonomies are several studies in the social sciences: see Scharfbillig et al. (2021) for a recent overview and practical insights (directed at policy makers). An automated detection of human values, as is the goal of this shared task, could greatly assist such analyses (Scharfbillig et al., 2022).
Some computational frameworks of argumentation consider the strength of an argument subject to the audience’s preferences defined via their values. Example frameworks include value-based ar-
1 https://touche.webis.de/semeval23/touche23-web
gumentation schemes (van der Weide et al., 2009), defeasible logic programming (Teze et al., 2019), and the value-based argumentation framework of Bench-Capon (2003). Automatically identifying values in natural language arguments is an important step in operationalizing these frameworks.
Outside of argumentation, several works in natural language processing utilize values. For example, in the context of interactive systems, Ammanabrolu et al. (2022) aim to tune interactive chatbased agents towards morally acceptable behavior. However, since their operationalization of values is limited to valence (good or bad) and target (self or other), the model can not explain in abstract terms why something would be acceptable or not. Liu et al. (2023) follow a similar approach based on human edits that change text to morally acceptable (“value-aligned”) behavior. A related dataset to the ours is ValueNet by Qiu et al. (2022),2 which contains 21K one-sentence descriptions of social scenarios (taken from SOCIAL-CHEM-101 of Forbes et al. (2020)) annotated for the 10 value categories of an earlier version of Schwartz’ value taxonomy. A major difference to our dataset are the more ordinary situations in ValueNet (e.g., whether to say “I miss mom”). A conceptual difference is that while ValueNet’s scenario descriptions could be seen as conclusion and its “utility” annotation (-1 to +1) as stance, the link between value category and description—the premise in our data—remains implicit in ValueNet. The implicit premise is a key difference: our annotations specifically target the premise, as it is the locus of ethical reasoning.

heading: 3 Task Description
text: We define the task of identifying the human values behind arguments as follows:
Given a textual argument and a human value, classify whether the argument resorts to that value or not.
Specifically, we employ a set of 20 value categories from our previous work (Kiesel et al., 2022) in a multi-label classification setup and evaluate approaches using macro F1,3 but also provided the teams with per value category F1-scores, precision, and recall for deeper analyses. We use macro instead of micro F1 to weigh each category the same,
2Which is not related to ValueNet by Giorgis et al. (2022). 3To be precise, we employ the harmonic mean of macro-
averaged precision and recall.
no matter their frequency. Figure 1 shows the employed value taxonomy, which is largely based on that of Schwartz (1994) and described in full detail by Kiesel et al. (2022). For ValueEval’23, we use the 20 value categories (level 2) in a multi-label classification setup: one yes-no decision for each pair of argument and value category. We selected level 2 for the task as it is the one usually used in social science analyses (e.g., by Scharfbillig et al. (2021)), whereas level 1 is mostly used for data collection (i.e., when surveying people).

heading: 4 Data
text: ValueEval’23 employs the Touché23-ValueEval dataset, which consists of 9324 arguments collected from 6 diverse sources, namely religious texts, political discussions, free-text arguments, newspaper editorials, and online democracy platforms. Each argument consists of three parts: two short texts— premise and conclusion—, and a stance attribute that indicates whether the premise supports the conlusion (“in favor of”) or its anti-thesis (“against”). Each argument was annotated by 3 crowdworkers for 54 values, which were then mapped to 20 value categories (cf. Table 4). Mirzakhmedova et al. (2023) describe the collection and annotation in detail. The dataset is publicly available online4 and has over 1900 downloads as of April 2023.
The dataset consists of two parts (cf. Table 2): the main dataset (8865 arguments; 95%) and the supplementary dataset (459 arguments; 5%). For the main leaderboard, we provide the main dataset as three separate sets as it is customary in machinelearning tasks: one set each for training, validation, and testing. To avoid train-test leakage from argument similarity, we ensured that all arguments with the same conclusions (but different premises) were in the same set. The ground truth for the test dataset has been kept secret until the camera ready deadline for ValueEval’23 participant papers.
While the main dataset reflects a classical indomain machine learning setup, the supplementary dataset simulates an out-of-domain application. This domain difference is reflected in several aspects of the datasets. As Figure 2 shows, the majority of arguments in the main dataset have more than one (= at least two) value categories assigned to them (between 88% and 92%). As for the supplementary dataset, the Zhihu split shows a similar pattern (with 81% of arguments having more than
4 https://doi.org/10.5281/zenodo.6814563
:
one value category), while much fewer labels are assigned to the arguments of the New York Times (41%) and the Nahj al-Balagha split (33%). Moreover, for the latter two splits a few arguments have no value assigned (i.e., resort to no ethical judgement): 2% for Nahj al-Balagha and 15% for the New York Times. As Figure 3 shows, also the distribution of value categories is very similar within the main dataset, but quite different to the distributions for the supplementary dataset, reflecting the difference in genres and topics. Table 1 shows one example argument for each dataset source.

heading: 4.1 Main dataset
text: The main dataset is compiled of 8865 arguments from the following three sources, with approximately the same ratio of arguments pre source in the train, validation, and test splits (cf. Table 2):
Argument Value categories Source
◦ Con “We should end the use of economic sanctions”: Economic sanctions provide security and ensure that citizens are treated fairly.
Security: societal, Universalism: concern IBM-ArgQRank-30kArgs
◦ Pro “We need a better migration policy.”: Discussing what happened in the past between Africa and Europe is useless. All slaves and their owners died a long time ago. You cannot blame the grandchildren. Universalism: concern Conf. on the Future of Europe
◦ Con “Rapists should be tortured”: Throughout India, many false rape cases are being registered these days. Torturing all of the accused persons causes torture to innocent persons too. Security: societal, Universalism: concern Group Discussion Ideas
◦ Con “We should secretly give our help to the poor”: By showing others how to help the poor, we spread this work in the society.
Benevolence: caring, Universalism: concern Nahj al-Balagha
◦ Con “We should crack down on unreasonably high incomes.”: If the key to an individual’s standard of living does not lie in income, then it is useless to simply regulate income. Security: personal, Universalism: concern Zhihu
◦ Pro “All of this is a sharp departure from a long history of judicial solicitude toward state powers during epidemics.”: In the past, when epidemics have threatened white Americans and those with political clout, courts found ways to uphold broad state powers. Power: dominance, Universalism: concern The New York Times
IBM-ArgQ-Rank-30kArgs We collected 7368 arguments from this dataset by Gretz et al. (2020), a collection of free-text arguments on 71 controversial topics that are common in the US. Arguments are collected using the US-based crowdsourcing platform Figure Eight and labeled for argumentation quality. We sampled arguments by topic, starting with that of highest quality. We used the topics as conclusions and the “arguments” as premises.
Conference on the Future of Europe We collected 1098 arguments for 431 unique conclusions from the Conference on the Future of Europe portal,5 an online participatory democracy platform
5Portal: https://futureu.europa.eu/; dataset: https://data.europa.eu/data/datasets/conference-onthe-future-of-europe
that involves citizens, experts, and EU institutions in a dialogue focused on the future direction and legitimacy of Europe. We sampled from an existing dataset (Barriere et al., 2022) proposals and corresponding comments which were originally written in English and for which the users marked their comments as supporting or contesting. We manually extracted the conclusions from the proposals and one or more premises from their comments.
Group Discussion Ideas We collected 399 arguments from the Group Discussion Ideas web page.6
The web page aggregates pros and cons on various topics covered in Indian news to help users participate in a group discussion or debate in English.
6 https://www.groupdiscussionideas.com/
We crawled the web-page and manually extracted one or more conclusion-premise pairs from each topic discussion, and manually labeled the stance for each pair where it was not stated explicitly.

heading: 4.2 Supplementary dataset
text: In addition to the main dataset, we collected a supplementary dataset of 459 arguments of different written forms and ethical reasoning. This dataset is intended as an out-of-domain challenge for submitted approaches. The arguments from the latter two sources were provided by other researchers in response to our call for data on NLP mailing lists:
Zhihu We collected 100 arguments from the recommendation and hotlist section of Zhihu, 7 a Chinese question-answering website. We translated the answers to English using automated translation and manually extracted and rephrased key points (premises and conclusions) from these.
Nahj al-Balagha We collected 279 arguments from the Nahj al-Balagha, an Islamic religious text. Conclusions and premises were deduced manually from a Farsi translation of the Arabic text, with similar conclusions being unified, and conclusions and premises translated to English. These arguments were provided by the language.ml lab.
The New York Times We collected 80 arguments from 12 news articles of The New York Times8 that were published between July 2020 and May 2021 and contain at least one of these keywords: coronavirus (2019-ncov), vaccination and immunization, and epidemics. The premises, conclusions, and stances were manually extracted by three annotators per text and curated by two linguist experts. These arguments were provided by Lea Kawaletz and Zeljko Bekcic of the HeinrichHeine-Universität Düsseldorf.

heading: 5 Submissions
text: The task received submissions from 39 teams, of which 37 provided information about their approaches—including the 29 teams who submitted a paper to SemEval. The task used TIRA9
(Fröbe et al., 2023) for evaluation. The following is a cross-sectional overview of the approaches by the 37 teams, after an overview of the top-ranked
7 https://www.zhihu.com/explore 8 https://www.nytimes.com/ 9 https://www.tira.io/task/valueeval-at-semeval-
2023-human-value-detection
approach. For details on single approaches, refer to the papers cited in Table 3. Each team was, per test set, allowed one submission before December 16th (“early bird”) and four additional submissions on January 27th. Teams were allowed more submissions after the deadline for analyses, but ground truth labels are released only after the camera-ready paper submission deadline to prevent the report of over-engineered results in the papers.
We employed anonymous submissions: teams had to choose a code name from Wikipedia’s list of ethicists10 on registration. Five teams preferred to stay anonymous and 26 other teams kept their code name as their team name.
Top-ranked Approaches Team Adam Smith uses an ensemble of 12 transformer-based models: DeBERTa and RoBERTa, both trained for either loss minimization or F1-score maximization on three different folds each.11 The RoBERTa models were pretrained on the full IBM-ArgQ-Rank30KArgs dataset (Gretz et al., 2020), which is the source for most arguments in the main set (cf. Table 2). For ensembling, they averaged the predictions of the single models and used a single decision threshold that they optimized on a hold-out set. They also tried a stacked meta-classifier based on logistic regression, which performs better on the Nahj al-Balagha but not on the main test set.
Team John Arthur fine-tuned a DeBERTa model (microsoft/deberta-v2-xxlarge) on the task’s data. They represented each input as a concatenation of stance, premise, and conclusion, separated by special tokens. They also found that using separate token symbols for stance (“Favour” and “Against”) slightly boosted classification results. They trained their model to minimize binary cross-entropy loss. The outputs were passed through a sigmoid function to make a binary prediction for each of the value categories. A single decision threshold (0.2) was used for all categories. The team observed that having more data in the training set benefits the model’s performance. Just adding as many as 100 arguments (they used the Zhihu validation set as an addition to the training set) lifted the F1 results from 0.53 to 0.55.
Team PAI (Theodor Zwinger) used a combination of transformer models. The training data, con-
10 https://en.wikipedia.org/wiki/List_of_ethicists
11They report on an ablation study in their paper, which reveals that using only the three F1-score-optimized DeBERTa models raises the F1-score slightly.
approach. If available, the table also contains hyperlinks to the team’s source code ( ), Docker image ( ), and web demo ( ). Also shows the approaches’ scores on the test datasets: Main, Nahj al-Balagha (Nahj.), and New York Times (NYT).
sisting of premise, conclusion, and stance, were combined in several ways in order to create four different input datasets. For example, in one dataset, sample labels were merged in order to equalize data imbalance. Each trained transformer has classifiers for all labels. The final results were ensembled by weight voting based on the F1 score in the validation set of each value category. Several loss functions were considered, where a class-balanced loss combined with a negative tolerant regularization proved to be the best approach. Also, different classification thresholds were tried but without performance improvement.
Employed Models Transformer-based models were the most dominant techniques (35/37; cf. Table 3). All submissions that used transformers finetuned them on data except team StFX-NLP (Jesus of Nazareth), who integrated BERT’s embeddings as features into their approach. Only two teams relied on classical machine learning models such as decision trees and SVMs (team Friedrich Nietzsche) or logistic regression (team Joseph Fletcher). Among the 35 transformer-based submissions, four approaches considered reformulating the task as a non-classification task, while 31 approaches performed direct classification on the provided labels.
Non-classification Approaches Four teams formulated the task as natural language inference: Søren Kierkegaard, Rudolf Christoph Eucken, Jesus of Nazareth, and Quintilian). For example, Team Søren Kierkegaard constructed samples from the premise and the corresponding value category with entailment or contradictory labels inferred from whether the premise has the value category. Then, they fine-tuned a RoBERTa model that was already pre-trained on the natural language inference task (Bowman et al., 2015). One team formulated the task as question-answering: Team Hitachi (R. M. Hare) used BART and T5 as backbone models. The predictions were made by feeding the model a yes/no question for each value category given a (premise, stance, conclusion) triple. They also experimented with formulating the task as zero-shot question-answering with chain-of-thought prompting (Kojima et al., 2022) using GPT-3 (text-davinci-003).
Employed Data Out of the 37 submissions, all used the premise as an input, 25 used the conclusion, and 24 used the stance. For example, team Tübingen (Stanley Grenz) found that utiliz-
ing the stance as part of the output to predict instead of being in the input led to better effectiveness. Four teams augmented the data with our definitions of the value categories (4 approaches): team Prodicus created extra instances by concatenating human values and their descriptions as input and the corresponding value category as a label; team Noam Chomsky used the similarity between premises and the corresponding human values as extra features. Team Hitachi also employed value category descriptions and formulated the task as a question-answering problem, as described above. Team Epicurus augments their data by adding definitional statements formulated based on several human value surveys, such as the world value survey (Haerpfer et al., 2020), and Rokeach value survey (Rokeach, 1973).
Special Training Besides team Adam Smith, as described above, three more teams pre-trained on other corpora: team Niccolo Machiavelli finetuned DeBERTaV3, and team Hitachi fine-tuned RoBERTa on ValueNet (Qiu et al., 2022), while team MaChAmp (Robert S. Hartman) fine-tuned on data from the other SemEval’23 tasks (“intermediate training”). However, for most SemEval tasks—including this one—the intermediate training did not result in higher effectiveness. Two teams employed contrastive learning: Mao Zedong and T. M. Scanlon). Team SUTNLP (David Gauthier) employed adversarial training to improve the generalization of their model.
Label Semantics Five submissions (Mao Zedong, SUTNLP, Hitachi, Prodicus, Epicurus) employed label semantics by learning it from data or using the definitions of human values. For example, team Mao-Zedong utilized a label-guided attention mechanism to learn label-specific representations from the input. They also utilize a contrastive loss function to pull instances with similar labels together. Team SUTNLP learns a semantic representation of labels by encoding them jointly with their corresponding premises and then employs adversarial training to enhance the generalization of their model. They further capture label correlations by constructing a graph over labels using a Graph Convolutional Network (Zhang et al., 2019), where nodes (hidden label representations) are connected if there is at least one argument in the training data that contains these labels.

heading: 6 Results
text: We evaluated the submitted approaches on the Touchè23-ValueEval dataset, both the main and supplementary parts (Section 6.1). Though we allowed teams to submit predictions only for a subset of the 20 value categories, only one team did so, and not on their best run. We use two baselines from our previous work (Kiesel et al., 2022): BERT Fine-tuned multi-label bert-base-uncased with batch size 8 and learning rate 2−5 (20 epochs). 1-Baseline Classifies each argument as resorting to all categories: always achieves a recall of 1.
Additionally, we report on a meta-classification study for an analysis of straightforward approachcombinations (Section 6.2) and describe our efforts to disseminate the results (Section 6.3).

heading: 6.1 Results of Participants
text: Table 3 shows the best macro F1-score each team achieved for each of the three test datasets. Team Adam Smith reached the highest F1-score for the main and Nahj al-Balagha test sets, whereas team Hitachi (R. M. Hare) was best on the New York Times dataset. The table shows a considerable increase over the BERT baseline for all datasets: 0.56 vs. 0.42 for main, 0.40 vs. 0.28 for Nahj al-Balagha, and 0.34 vs. 0.24 for New York Times. The naive 1-baseline performs even worse (0.26, 0.13, and 0.15), though the F1-score favors it in comparison with accuracy or other measures. Though the scores for the supplementary datasets are lower, one has to take into consideration that the teams had no training data from these datasets. In light of the very different nature of these datasets compared to the main dataset, we find these results to be very encouraging for building robust systems.
However, not every approach performed equally well for all value categories. Table 3 shows the “best per value category” system at the top, which is a hypothetical system that uses for each value category the run that performed best for that category— among all submitted runs, not just the ones that achieved a best macro F1-score. The improvement of this hypothetical system over the best single runs is small but noticeable for the main test set (0.59 vs. 0.56), but quite large for Nahj al-Balagha (0.48 vs. 0.40) and even larger for New York Times (0.47 vs. 0.34). For more details, Tables 5 and 6 in the appendix show the F1-score of the best approaches per team for each value category. For example, the best approach for stimulation and hedonism
outperforms the approach by Adam Smith considerably (0.39 for both vs. 0.22 and 0.29), but does not perform well overall (it is not in the list of best approaches per team).
For a visual impression of the submitted runs’ performance, the radar plot in Figure 4 shows the F1-score of every submitted run for each value category, where each run corresponds to one line. The plot shows that most runs did improve over the baselines for most value categories: there is a “dark band” that is consistently outside of the red “BERT” line. Moreover, the plot reveals some peculiarities, like that only few runs are at the top for stimulation, hedonism, face, conformity: interpersonal and humility. Since stimulation and hedonism are next to each other, one can also see that the top run in each one of these performed worse on the respective other category. The plot also reveals the anomaly of the universalism: objectivity category, for which the 1-baseline performed considerably better that the BERT baseline and many submitted runs. This is the only category that was added—based on a comparison with other value taxonomies (Kiesel et al., 2022)—to the 19 categories of the Schwartz taxonomy. Schwartz et al. (2012) refrained from including such a category as they found truth-seeking to be correlated with several other categories on all sides of the wheel. This observation might also be connected to the anomaly we observe in the figure.

heading: 6.2 Results from Meta-classification
text: As a form of meta-analysis we tested the use of basic ensemble methods on top of the submitted run files (excluding baselines). Figure 5 shows the effectiveness of a simple voting scheme: the ensemble classifies an argument as resorting to a value category if the same was done in at least X of the submitted runs (“run threshold”). Thus, if the threshold is 0, the voting scheme corresponds to the 1-Baseline. We employ either the best run for each team (per dataset, as determined by macro F1score), or only the top-5 runs of these. We refrained from using more involved ensemble techniques as this analysis is of theoretical nature either way: the computational costs of such an ensemble makes its use unfeasible in practice.
Unlike in a previous shared task of ours (Kiesel et al., 2019), the voting scheme does not (main test dataset, Figure 5 a,b) or only marginally (supplementary test datasets, Figure 5 c–f) improve over the best single run (Table 3, solid line in Figure 5).
The achieved F1-score is similar to the best single run for relatively large ranges of the run threshold, for which every increase in precision is traded with a similar decrease in recall. For top-5 runs on the main test set (Figure 5 b) this equilibrium even holds for all run thresholds except 0, which corresponds to a even field among the top 5 teams. The decrease in recall from 74% to 44% shows that, though many argument-value category pairs seem to be clear in that they were found unanimous among the top teams (44%), the approaches are diverse enough to detect many more pairs. This situation is more extreme for the Nahj al-Balagha set (73% to 29%; Figure 5 d) but similar for the New York Times set (75% to 37%; Figure 5 f).
The extreme values for the run threshold show both the very difficult cases—ratio of argumentvalue category pairs which were found in none of the runs (1 - recall for run threshold of 1)—and very easy cases—ratio of pairs found in each run (recall at highest run threshold). The ratio of difficult (easy) cases is 6% (<1%), 12% (9%), and 2% (5%) for main, Nahj al-Balagha, and New York Times, respectively (Figure 5 a,c,e). Though these numbers are not fully comparable given the varying number of submitting teams, they highlight that most pairs have indeed been found by at least one of the submitted (best) runs.

heading: 6.3 Results Dissemination
text: Table 3 cites all submitted papers and links to the source code and Docker images of the teams. Moreover, we worked together with the top-ranked team, Adam Smith, to provide an online demo and executable of their best-performing approach.12

heading: 7 Conclusion
text: This paper reports on the setup, participation, results, and insights gained from the Touché task on identifying human values behind arguments (ValueEval’23), hosted as Task 4 at SemEval-2023. We detail the construction both of the main dataset of 8865 arguments, as well as the supplementary dataset of 459 arguments used to test model generalizability. Moreover, we provide a systematic overview of the 29 papers submitted by the participants, compare their approaches, and perform an ad-hoc meta classification.
Through the use of TIRA (Fröbe et al., 2023) we can directly evaluate the approaches of the 8 teams who used Docker submission on new datasets for human values detection, provided they are formatted like the datasets presented here. Moreover 23 teams set up an open source repository for making their code available to other researchers.
Given the implicit nature of human values, very promising results were achieved during the task, with F1-scores above 0.56 on the main test set and up to 0.87 for individual categories. However, many challenges remain, as evidenced by the low peak F1-score of 0.39 for stimulation, hedonism, face, and humility. The results show that, like in many other NLP tasks, transformer models could be used to great effect, and in particular ensembling approaches showed to be quite effective. The results between the main and supplementary datasets were quite different as anticipated due to their dissimilarity, yet teams managed to beat the baselines on the supplementary datasets by a wide margin despite the total lack of in-domain training data. Moreover, several teams reported that using our supplementary validation set as an addition to the training set boosted results, suggesting that a larger dataset of this kind will probably assist in improving the performance of future models even further. Thus, there is still room for improvement on this task, and whether progress will come from novel models or creative applications of existing techniques remains to be seen.
12 https://values.args.me/

heading: Acknowledgements
text: Our thanks go out to all participating teams; your contributions made this task a success. Our special thanks go out to the SemEval organizers and to the TIRA team for their constant support.

heading: A Appendix
text: Value category descriptions (Table 4) and results per value (Tables 5 and 6).
Value category (level 2) Contained values (level 1)
◦ Self-direction: thought It is good to have own ideas and interests. Be creative: arguments towards more creativity or imagination Be curious: arguments towards more curiosity, discoveries, or general interestingness Have freedom of thought: arguments toward people figuring things out on their own, towards
less censorship, or towards less influence on thoughts
◦ Self-direction: action It is good to determine one’s own actions. Be choosing own goals: arguments towards allowing people to choose what is best for them, to decide on their life, and to follow their dreams
Be independent: arguments towards allowing people to plan on their own and not ask for consent Have freedom of action: arguments towards allowing people to be self-determined and able to do what they want Have privacy: arguments towards allowing for private spaces, time alone, and less surveillance, or towards more control on what to disclose and to whom
◦ Stimulation It is good to experience excitement, novelty, and change.
Have an exciting life: arguments towards allowing people to experience foreign places and special activities or having perspective-changing experiences Have a varied life: arguments towards allowing people to engage in many activities and change parts of their life or towards promoting local clubs (sports, ...) Be daring: arguments towards more risk-taking
◦ Hedonism It is good to experience pleasure and sensual gratification.
Have pleasure: arguments towards making life enjoyable or providing leisure, opportunities to have fun, and sensual gratification
◦ Achievement It is good to be successful in accordance with social norms. Be ambitious: arguments towards allowing for ambitions and climbing up the social ladder Have success: arguments towards allowing for success and recognizing achievements Be capable: arguments towards acquiring competence in certain tasks, being more effective,
and showing competence in solving tasks Be intellectual: arguments towards acquiring high cognitive skills, being more reflective, and
showing intelligence Be courageous: arguments towards being more courageous and having people stand up for
their beliefs
◦ Power: dominance It is good to be in positions of control over others. Have influence: arguments towards having more people to ask for a favor, more influence, and more ways to control events
Have the right to command: arguments towards allowing the right people to take command, putting experts in charge, and clearer hierarchies of command, or towards fostering leadership
◦ Power: resources It is good to have material possessions and social resources.
Have wealth: arguments towards allowing people to gain wealth and material possession, show their wealth, and exercise control through wealth, or towards financial prosperity
◦ Face It is good to maintain one’s public image.
Have social recognition: arguments towards allowing people to gain respect and social recognition or avoid humiliation Have a good reputation: arguments towards allowing people to build up their reputation, protect their public image, and spread reputation
◦ Security: personal It is good to have a secure immediate environment.
Have a sense of belonging: arguments towards allowing people to establish, join, and stay in groups, show their group membership, and show that they care for each other, or towards fostering a sense of belonging Have good health: arguments towards avoiding diseases, preserving health, or having physiological and mental well-being Have no debts: arguments towards avoiding indebtedness and having people return favors Be neat and tidy: arguments towards being more clean, neat, or orderly Have a comfortable life: arguments towards providing subsistence income, having no financial worries, and having a prosperous life, or towards resulting in a higher general happiness
◦ Security: societal It is good to have a secure and stable wider society. Have a safe country: arguments towards a state that can better act on crimes, and defend or care for its citizens, or towards a stronger state in general
Have a stable society: arguments towards accepting or maintaining the existing social structure or towards preventing chaos and disorder at a societal level
◦ Tradition It is good to maintain cultural, family, or religious traditions. Be respecting traditions: arguments towards allowing to follow one’s family’s customs, honoring traditional practices, maintaining traditional values and ways of thinking, or promoting the preservation of customs
Be holding religious faith: arguments towards allowing the customs of a religion and to devote one’s life to their faith, or towards promoting piety and the spreading of one’s religion
Table 4 (continued on next page).
Table 4 (continued).
Value category (level 2) Contained values (level 1)
◦ Conformity: rules It is good to comply with rules, laws, and formal obligations.
Be compliant: arguments towards abiding to laws or rules and promoting to meet one’s obligations or recognizing people who do Be self-disciplined: arguments towards fostering to exercise restraint, follow rules even when no-one is watching, and to set rules for oneself Be behaving properly: arguments towards avoiding to violate informal rules or social conventions or towards fostering good manners
◦ Conformity: interpersonal It is good to avoid upsetting or harming others. Be polite: arguments towards avoiding to upset other people, taking others into account, and being less annoying for others
Be honoring elders: arguments towards following one’s parents or showing faith and respect towards one’s elders
◦ Humility It is good to recognize one’s own insignificance in the larger scheme of things. Be humble: arguments towards demoting arrogance, bragging, and thinking one deserves more than other people, or towards emphasizing the successful group over single persons and giving back to society
Have life accepted as is: arguments towards accepting one’s fate, submitting to life’s circumstances, and being satisfied with what one has
◦ Benevolence: caring It is good to work for the welfare of one’s group’s members.
Be helpful: arguments towards helping the people in one’s group and promoting to work for the welfare of others in one group Be honest: arguments towards being more honest and recognizing people for their honesty Be forgiving: arguments towards allowing people to forgive each other, giving people a
second chance, and being merciful, or towards providing paths to redemption Have the own family secured: arguments towards allowing people to have, protect, and care
for their family Be loving: arguments towards fostering close relationships and placing the well-being of
others above the own, or towards allowing to show affection, compassion, and sympathy
◦ Benevolence: dependability It is good to be a reliable and trustworthy member of one’s group. Be responsible: arguments towards clear responsibilities, fostering confidence, and promoting reliability Have loyalty towards friends: arguments towards being a dependable, trustworthy, and loyal friend, or towards allowing to give friends a full backing
◦ Universalism: concern It is good to strive for equality, justice, and protection for all people.
Have equality: arguments towards fostering people of a lower social status, helping poorer regions of the world, providing all people with equal opportunities in life, and resulting in a world were success is less determined by birth Be just: arguments towards allowing justice to be ’blind’ to irrelevant aspects of a case, promoting fairness in competitions, protecting the weak and vulnerable in society, and resulting a world were people are less discriminated based on race, gender, and so on, or towards fostering a general sense for justice Have a world at peace: arguments towards nations ceasing fire, avoiding conflicts, and ending wars, or promoting to see peace as fragile and precious or to care for all of humanity
◦ Universalism: nature It is good to preserve the natural environment.
Be protecting the environment: arguments towards avoiding pollution, fostering to care for nature, or promoting programs to restore nature Have harmony with nature: arguments towards avoiding chemicals and genetically modified organisms (especially in nutrition), or towards treating animals and plants like having souls, promoting a life in harmony with nature, and resulting in more people reflecting the consequences of their actions towards the environment Have a world of beauty: arguments towards allowing people to experience art and stand in awe of nature, or towards promoting the beauty of nature and the fine arts
◦ Universalism: tolerance It is good to accept and try to understand those who are different from oneself. Be broadminded: arguments towards allowing for discussion between groups, clearing up with prejudices, listening to people who are different from oneself, and promoting to life within a different group for some time, or towards promoting tolerance between all kinds of people and groups in general
Have the wisdom to accept others: arguments towards allowing people to accept disagreements and people even when one disagrees with them, to promote a mature understanding of different opinions, or to decrease partisanship or fanaticism
◦ Universalism: objectivity It is good to search for the truth and think in a rational and unbiased way.
Be logical: arguments towards going for the numbers instead of gut feeling, towards a rational, focused, and consistent way of thinking, towards a rational analysis of circumstances, or towards promoting the scientific method Have an objective view: arguments towards fostering to seek the truth, to take on a neutral perspective, to form an unbiased opinion, and to weigh all pros and cons, or towards providing people with the means to make informed decisions
Table 4: Descriptions for each of the 20 employed value categories (level 2) and their respective contained values (level 1; 54 total). These value descriptions were also used (in slightly adapted form as bullet points) in the dataset annotation and are distributed along with the dataset.

heading: Future Work
text: Moreover, several teams reported that using our supplementary validation set as an addition to the training set boosted results, suggesting that a larger dataset of this kind will probably assist in improving the performance of future models even further. Thus, there is still room for improvement on this task, and whether progress will come from novel models or creative applications of existing techniques remains to be seen. 12 https://values. args. me/