Hitachi at SemEval-2023 Task 4: Exploring Various Task Formulations Reveals the Importance of Description Texts on Human Values
Year: 2023

Abstract:
This paper describes our participation in SemEval-2023 Task 4, ValueEval: Identification of Human Values behind Arguments. The aim of this task is to identify whether or not an input text supports each of the 20 pre-defined human values. Previous work on human value detection has shown the effectiveness of a sequence classification approach using BERT. However, little is known about what type of task formulation is suitable for the task. To this end, this paper explores various task formulations, including sequence classification, question answering, and question answering with chain-of-thought prompting and evaluates their performances on the shared task dataset. Experiments show that a zero-shot approach is not as effective as other methods, and there is no one approach that is optimal in every scenario. Our analysis also reveals that utilizing the descriptions of human values can help to improve performance.

heading: 1 Introduction
text: SemEval-2023 Task 4 involves detecting human values behind argumentative sentences in English. Given a conclusion, stance, and premise, the task is to classify whether they entail each of the 20 pre-defined values (Kiesel et al., 2023). The performance improvement in this task can contribute to more precise analysis of argumentative texts using argument and opinion mining techniques.
In this paper, we explore various task formulations for detecting human values in texts, including sequence classification, question answering, and question answering with chain-of-thought prompting. Previous work on human value detection (Kiesel et al., 2022) thus far has only investigated the effectiveness of sequence classification using BERT (Devlin et al., 2019), and it is still unknown what type of task formulation is effective for human value detection.
To this end, we test a variety of pretrained language models. For sequence classification, we employ encoder-based models such as RoBERTa (Liu et al., 2019) and DeBERTaV3 (He et al., 2021). For question answering, we test both encoder-decoder and decoder models, including T5 (Raffel et al., 2020), BART (Lewis et al., 2020), and GPT-3 (textdavinci-003) (Brown et al., 2020). For question answering with chain-of-thought prompting, we use GPT-3 (text-davinci-003) in a zero-shot manner. We measure the detection performance of all approaches on three shared task datasets (Mirzakhmedova et al.), including the two optional test datasets: Nahj al-Balagha and New York Times. We also verify the effectiveness of three additional training strategies for sequence classification: loss weighting, pretraining on a similar corpus: ValueNet (Qiu et al., 2022), and adding a value description to an input.
In the competition1, our approach using RoBERTaLARGE trained with sequence classification and weighted loss achieved macro F1 scores of 0.51 and 0.34 on the Main and New York Times test datasets and was ranked 7th and 1st, respectively. For the Nahj al-Balagha dataset, the submitted approach using a mixture of the highest performing models with respect to each value on the validation set demonstrated a macro F1 score of 0.34 and was ranked 4th.
Our contributions are three-folds.
• This study is the first to explore various task formulations for detecting human values in argumentative texts (Section 2).
• We conduct five analyses to examine the behavior of our approaches and verify the effec-
1This paper reports the results not only with our submitted systems but also other approaches for a comprehensive comparison of various task formulations. Table 4 includes the detailed results of our submitted system on the Main test dataset, and Appendix A details our submitted systems.
1723
tiveness of three additional training strategies for sequence classification (Section 5).
• We demonstrate that utilizing the descriptions of human values can improve detection performance (Section 5.2 and 5.3).

heading: 2 Methodology
text: This section explains our approaches in this work. Table 1 provides an overview of each approach with the input and output formats and corresponding example model names.

heading: 2.1 Sequence Classification
text: Our first approach is sequence classification (SC), which has been tested in previous work (Kiesel et al., 2022). Let s = [w0, . . . , wn] be an input sequence, where w denotes a token. In SC, a model takes s as input and outputs a hidden vector hs = [e0, . . . , en] corresponding to s. We put a linear layer on top of the model for final classification. Because the task is formulated as multilabel classification, the model computes the score of each value category using a sigmoid function and is trained with the binary cross-entropy loss averaged over all 20 classes (values).

heading: 2.2 Question Answering
text: Because the shared task is fine-grained and has a variety of value categories, it may be difficult for a model to accurately detect human values in texts only using a (conclusion, stance, premise) triple as in SC. If the model can capture subtle differences in value definitions, the correct values in texts can be identified more easily. To this end, we investigate adding descriptions of values to the model’s input.
Here, we formulate the task as question answering (QA) to accommodate a value description in input. We obtain all descriptions of the 20 values from the shared task website2 and generate a yes/no question for each value description given a (premise, stance, conclusion) triple (see Table 1 for an example). This means we create 20 samples per triple and need to feed them into a model one by one to obtain predictions for a particular triple. We fine-tune both encoder-decoder and decoder models, including T5, BART, and GPT-3 (text-davinci-003).
2https://touche.webis.de/semeval23/ touche23-web/index.html#task

heading: 2.3 Question Answering with Chain-of-Thought Prompting
text: Previous studies (Kojima et al., 2022; Wei et al., 2022) have reported that large language models (LLMs), e.g., GPT-3, are effective even in complicated tasks that need reasoning. We assume that the shared task falls in this category; thus, utilizing an LLM can be an effective approach to detecting human values behind texts. One practical problem arises in using an LLM: financial costs. To reduce the cost as much as possible, we use a zero-shot approach with chain-of-thought prompting (Kojima et al., 2022), which shows significant performance improvement over zero-shot LLMs on various reasoning benchmark datasets. This approach is a two-step procedure exemplified in Table 1 (QA w/ CoT). Specifically, we employ a similar prompt used for the question answering approach as a first query with the addition of “Let’s think step by step.” to its end. After obtaining the corresponding output from GPT-3 (text-davinci-003), we concatenate the first prompt, output, and “Therefore, the answer (YES or NO) is” and feed it to GPT-3 again to obtain the final decision.

heading: 3 Experimental Setup
text: Data We used the official training dataset for training and the official validation dataset for monitoring generalization performance. For evaluation, we used the Main, Nahj al-Balagha, and New York Times test datasets. Details of each dataset can be found in the task paper (Kiesel et al., 2023) and on the website3.
Evaluation Metrics The evaluation metrics are a macro F1 score averaged over all value categories and F1 score for each value.
Models Table 2 presents the list of models tested for each approach. For simplicity, we only compare the results of bold models that showed the highest macro F1 score on the validation set in the remainder of this paper.
Implementation Details We implemented each approach with PyTorch (Paszke et al., 2019) and Hugging Face transformers (Wolf et al., 2020) except for GPT-3. For GPT-3, we used the official API provided by OpenAI to use both the base and
3https://touche.webis.de/semeval23/ touche23-web/index.html
Approach Models
SC roberta-large, microsoft/deberta-v3-large, google/rembert, nghuyong/ernie-2.0-large-en, microsoft/infoxlm-large, google/electra-largediscriminator, google/canine-s QA facebook/bart-large, t5-base, t5-large, google/flan-t5-base, google/flan-t5-large, GPT-3 (text-davinci-003) (on some values) QA w/ CoT GPT-3 (text-davinci-003) (zero-shot)
Table 2: List of models tested in this paper. In the experiments, we used bold models that showed the highest F1 macro score on the validation set for each approach.
fine-tuned text-davinci-003 models. The hyperparameter settings are detailed in Appendix A.

heading: 4 Results
text: Table 3 presents the macro F1 and F1 score of each value category with respect to the three approaches introduced in Section 2. We also include the results of the baseline (“1-Baseline”) that always assigns 1 to all instances for reference. Overall, QA exhibited the highest macro F1 scores of 0.54 and 0.42 on the Main and Nahj al-Balagha datasets, respectively. SC showed the highest macro F1 score of 0.28 on the New York Times dataset. QA w/ CoT resulted in the lowest macro F1 score among the three approaches though it outperformed the baseline. This indicates that it is essential to train a model on annotated data for human value detec-
tion in order to effectively identify values behind argumentative texts.
As for the value-wise F1 results, the most effective approach depends on the value categories and datasets, indicating that there exists no onesize-fits-all approach for human value detection. However, we see a general trend that QA slightly outperforms SC the majority of the times. The main difference between the two is whether or not an input has a value description. Thus, we hypothesize that utilizing the descriptions of human values can boost the detection performance. We verify our hypothesis in the Sections 5.2 and 5.3.

heading: 5 Analysis

heading: 5.1 Mitigating Data Imbalances
text: As shown in Table 4, the official dataset is imbalanced with respect to all value categories, which makes it challenging for a model to acquire the useful representations for the task due to the limited number of positive samples. We found in the preliminary experiments that value-wise F1 scores on the validation data were often low for value categories with low frequency. To address with this problem, we utilized an adjusted loss function with respect to label frequency in SC4 and verified its effectiveness on the Main test dataset. For im-
4We only targeted SC because it achieved the highest macro F1 score on the validation data.
plementation, we adjusted the weight of positive samples in each value category according to the frequency in the training data when computing the binary cross-entropy loss.
As Table 4 shows, SC with adjusted loss slightly outperformed SC on the validation dataset, but not on the Main test dataset in most cases. The Pearson and Spearman correlations between the ratio of positive samples in the training data for each value category and its performance improvement over SC were extremely small, -0.09 and -0.07, respectively. Although the loss is adjusted to improve performance on low-frequency labels, we did not see a correlation between the ratio of positive samples and performance improvement over SC. These results suggest that the adjusted loss function does not have much impact on the performance improvement.

heading: 5.2 Pre-training on ValueNet
text: It is very unlikely that argumentative texts in the official datasets explicitly mention underlying human values. Therefore, it may be difficult for models to acquire the semantic representations for effectively identifying human values. To address this challenge, we pre-trained an SC model on a similar corpus, ValueNet (Qiu et al., 2022), motivated by the assumption that it could provide the model with
prior knowledge of human values and thus improve performance. ValueNet contains short texts about social scenarios (e.g., “applying to a far-away university against my dad’s wishes”) and the relatedness (-1/0/1) between the texts and specific human values. The value categories in ValueNet are similar to those in the shared task as both are based on Schwartz’s theory (Schwartz, 2012).5
During pre-training, a model predicted the relatedness (-1/0/1) between a given scenario text and value and was trained with the mean squared error loss, following Qiu et al. (2022). We generated input texts with the following template.
The premise {scenario text} is supporting the value towards {value}.
As a {value}, we used “Definition goal” defined by Schwartz (2012). The model was then finetuned on the shared task dataset as described in 2.1. We used RoBERTa LARGE, which achieved the highest macro F1 score on the validation set for the experiments. During fine-tuning, we applied the adjusted loss as this approach exhibited the best F1 score on the validation set (see Section 5.1).
As Table 4 shows, SC with the adjusted loss and ValueNet outperformed or was on par with SC
5Some values in this task are integrated in ValueNet, and some do not exist in ValueNet. Appendix E describes the correspondence between the two datasets.
with the adjusted loss on the Main test dataset in many values. The performances of value categories whose positive ratio in the training data was lower than 0.1 improved compared to those with SC w/ adjusted loss except for “Face.” The Pearson and Spearman correlations between the ratio of positive samples in the training data for each value category and its performance improvement over SC w/ adjusted loss were -0.56 and -0.57, respectively. These results suggest that ValueNet improves the performance of low-frequency value categories.

heading: 5.3 Sequence Classification with Human Value Descriptions
text: As observed in Section 5.2, the prior knowledge of human values seems to improve the detection performance of low-frequency human values. Here, we try to incorporate Human Value Descriptions into SC (SC w/ HVD), which should help a model capture the semantic representations to identify human values. We followed the same setting as that of SC except for the input and output formats shown in Table 1, and employed deberta-v3-large, which achieved the highest macro F1 score on the validation dataset. This time, we need to feed an individual query for each value into the model to obtain a prediction.
As seen in Table 4, SC w/ HVD exhibited the highest macro F1-score and outperformed SC on almost all value categories. In addition, the Pearson and Spearman correlations between the ratio of positive samples in the training data for each value category and its performance improvement over SC were -0.55 and -0.61, respectively. These results indicate that incorporating human value descriptions into an input is effective for identifying human
values behind argumentative texts, especially for low-frequency labels.

heading: 5.4 Fine-tuning with GPT-3
text: As explained in Table 2, we only fine-tuned GPT-3 (text-davinci-003) on a limited number of value categories due to budget constraints, including “Selfdirection: thought,” “Stimulation,” “Face,” “Tradition,” “Humility,” “Benevolence: dependability,” and “Universalism: tolerance,” which exhibited low macro F1 score in the validation data regardless of the approaches used.
The results in Table 4 show that some F1 scores improved, but their differences were small except “Humility.” This suggests that although fine-tuned GPT-3 does not always show the highest result, there might be human values that fine-tuned GPT3 is good at identifying. We leave the thorough investigation of such values as future work.

heading: 5.5 Prediction Tendency of Question Answering with Chain-of-Thought Prompting
text: As shown in Table 3, zero-shot question answering with chain-of-thought prompting (QA w/ CoT) by GPT-3 exhibited the lowest macro F1 scores on all datasets. To determine the causes of the low performance of QA w/ CoT, we examined its output trends and error cases. To simplify the discussion, we focused on the validation and Main test datasets.
Table 5 presents the types of answers generated by QA w/ CoT, the number of each type, and the example answers. While the ratio of “Yes” in gold answers is 0.168 on the validation dataset, the ratio of answers starting with “yes” by QA w/ CoT is 0.291. Thus, QA w/ CoT tends to answer “Yes,”
Prompt (# Input 1 in Table 1): Question: I agree with the conclusion that the use of public defenders should be mandatory. I have the premise: "the use of public defenders should be mandatory because some people don’t have money for a lawyer and this would help those that don’t." Do you think I EXPLICITLY support the value that it is good to be successful in accordance with social norms? Answer: Let’s think step by step.
Reasoning Text (# Output 1 in Table 1): The premise states that the use of public defenders should be mandatory because it would help those who don’t have money for a lawyer. This premise does not explicitly support the value that it is good to be successful in accordance with social norms. However, the conclusion that the use of public defenders should be mandatory implies that it is good to be successful in accordance with social norms, as it implies that those who don’t have money for a lawyer should be given the same opportunity to be successful as those who do. Therefore, while the premise does not explicitly support the value that it is good to be successful in accordance with social norms, the conclusion does
excessively.
In addition, QA w/ CoT sometimes generates answers that do not follow the instructions. As for an example of an answer starting with “yes,” the model answered whether to implicitly support the values, although the first prompt (# Input 1) in Table 1 asked whether to explicitly support the values. For all examples of the “Other” type in Table 5, the answers were ambiguous, although the second prompt (# Input 2) in Table 1 instructed the model to answer with “Yes” or “No.”
To understand how the model wrongly outputs answers starting with “yes” on the validation dataset (i.e., false positive cases), we analyzed the intermediate reasoning texts generated by the model based on the first prompt (# Input 1) in Table 1. Some of the reasoning texts stated that a given premise did not explicitly support the given value but the given premise or conclusion implicitly supported the value. Table 6 presents one such example. In these error cases, the model generally paraphrased the given premise first, stated that the premise did not explicitly support the given value, and then explained the interpretation that associates the premise or conclusion with the value. The ratio of reasoning text having the strings of “does not explicit” and “however” among the false positives case on the validation dataset was 0.662
(5,277/7,966). To remedy these cases, we need to further investigate how to make the model not allow implicit support for the values, such as through prompt engineering. Post-hoc modification of the answer based on the content of its reasoning texts is also promising.

heading: 6 Discussion and Conclusion
text: This paper has explored three task formulations for human value detection: SC, QA, and QA w/ CoT, and investigated the effectiveness of each approach on the shared task dataset. Experimental results demonstrated that an one-size-fits-all approach does not exist for this task, but supervised learning with the dedicated task data is necessary to obtain decent detection performance.
As mentioned in Section 4, we hypothesized that utilizing the descriptions of human values can boost detection performance, which was supported by the results in Sections 5.2 and 5.3. Although there is no one task formulation that is always effective for all value categories, different value categories can favor different task formulations. For instance, QA largely outperformed SC w/ HVD in “Selfdirection: thought” and “Universalism: tolerance,” though the only difference between the two is in their architecture. Investigating this trend in detail can be a future research direction.

heading: Acknowledgments
text: We would like to thank anonymous reviewers for their valuable feedback. We would also like to thank Gaku Morio, Yuichi Sasazawa, and Ken-ichi Yokote for their initial feedback on the experimental design and Dr. Masaaki Shimizu for the maintenance and management of large computational resources.

heading: A Submitted Systems
text: Here, we detail our submitted systems that achieved the highest macro F1 score among our runs in each dataset.
A.1 Main
Our most accurate system for the Main test dataset utilized RoBERTaLARGE trained with sequence classification and loss weighting6. We used one NVIDIA Tesla V100 (32GB) and optimized the model with AdamW (Loshchilov and Hutter, 2019). Table 7 shows the hyperparameter settings.
A.2 Nahj al-Balagha
Our most accurate system for the Nahj al-Balagha test dataset used a mixture of models and task formulations. We selected a highest-performing model for each value category on the basis of a F1 score on the validation set. Table 8 shows the detailed configuration for each value category.
A.3 New York Times
Our most accurate system for the New York Times test dataset is the same as that one for the Main test dataset.

heading: B Cost of GPT-3
text: The total cost of QA w/ CoT was $2324.63 to obtain predictions for the validation set and the three test datasets in a zero-shot manner, while that for QA was $964.04 to fine-tune GPT-3 and make predictions on the test datasets over the seven value categories listed in Table 4.
6See details in Section 5.1

heading: C Fine-tuning with Instructions
text: Some pre-trained models (e.g., Flan-T5) are finetuned with texts beginning with instructions, and they have shown promising results on various NLP benchmarks (Chung et al., 2022). Here, we also verified the effectiveness of the QA approach with instructions (QA w/ Instruct) in the task of human value detection. The only difference between QA w/ Instruct and QA is whether or not an input has an instruction sentence as demonstrated in Table 18.
As shown in Table 19, QA w/ Instruct outperformed QA on the validation dataset, but it did not always improve F1 scores on the test dataset. While the F1 score of “Conformity: interpersonal” largely improved compared to QA, that of “Hedonism” largely degraded. Therefore, no positive impact on detection performance was observed with respect to the usage of QA w/ Instruct.

heading: D Prediction Tendency
text: Here, we analyze the tendency of prediction for the sequence classification (SC), question answering (QA), and question answering with chain-ofthought prompting (QA w/ CoT) based on precision and recall to better understand the characteristics of each approach in human value detection. Table 20 presents the precision and recall of the three approaches on the Main test dataset. QA consistently achieved the highest precision for all value categories, while its recall values were consistently lower than those of SC except “Hedonism” and “Power: resources.” QA w/ CoT achieved the highest recall values in several value categories but exhibited significantly low precision, indicating that GPT-3 with zero-shot chain-of-thought prompting tends to say “YES” to a given question regardless of correctness in this task. This is consistent with the tendency described in Section 5.5.
One possible reason for the low recall values in QA is that the ratio of “yes” in the training data was quite low (0.17), forcing a model to generate “no” most of the time. We assume that negative sampling can be effective for improving recall values for QA.
E ValueNet
Table 21 presents the correspondence between human values in Task 4 (Mirzakhmedova et al.) and ValueNet (Qiu et al., 2022). Table 21 also presents
the frequency of each human value category on both training dataset.

heading: Future Work
text: Investigating this trend in detail can be a future research direction