What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations
Authors: Kavel Rao
Liwei Jiang
Valentina Pyatkin
Yuling Gu
Niket Tandon
Nouha Dziri
Faeze Brahman
Yejin Choi
Venue: EMNLP 2023 Findings (2023)
Keywords: moral reasoning
defeasible reasoning
commonsense reasoning
language groundings
knowledge distillation
TL;DR: We present defeasible moral reasoning (a new task for contextualization of moral situations), produce a model to perform the task by iterative self-distillation, and create a high-quality dataset of moral contextualizations and rationales.

Abstract: Moral or ethical judgments rely heavily on the specific contexts in which they occur. Understanding varying shades of defeasible contextualizations (i.e., additional information that strengthens or attenuates the moral acceptability of an action) is critical to accurately represent the subtlety and intricacy of grounded human moral judgment in real-life scenarios.

We introduce defeasible moral reasoning: a task to provide grounded contexts that make an action more or less morally acceptable, along with commonsense rationales that justify the reasoning. To elicit high-quality task data, we take an iterative self-distillation approach that starts from a small amount of unstructured seed knowledge from GPT-3 and then alternates between (1) self-distillation from student models; (2) targeted filtering with a critic model trained by human judgment (to boost validity) and NLI (to boost diversity); (3) self-imitation learning (to amplify the desired data quality). This process yields a student model that produces defeasible contexts with improved validity, diversity, and defeasibility. From this model we distill a high-quality dataset, $\delta$-Rules-of-Thumb, of 1.2M entries of contextualizations and rationales for 115K defeasible moral actions rated highly by human annotators 85.9% to 99.8% of the time. Using $\delta$-RoT we obtain a final student model that wins over all intermediate student models by a notable margin.

What Makes it Ok to Set Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations
Kavel Rao V* Liwei Jiang
Valentina Pyatkin
Yuling Gu
Niket Tandon
Nouha Dziri
Faeze Brahman
Yejin Choi
Paul G. Allen School of Computer Science & Engineering, University of Washington
Allen Institute for Artificial Intelligence
(kavelrao, lwjiang]@cs washington. edu
Abstract
(a) Seed Knowledge Generation
Everyday
Moral or ethical judgments rely heavily on the specific contexts in which they occur. Under- standing varying shades of defeasible conter- tualizations (i.e., additional information that strengthens or attenuates the moral acceptabil- ity of action) is critical to accurately rep- resent the subtlety and intricacy of grounded human moral judgment real-life scenarios.
Action
Context
?
Rationale
Context
Rationale
www.wa..........................
(b) Targeted Filtering
Setting fire
Teacher Model GPT-3 (175B)
We introduce defeasible moral reasoning: a task to provide grounded contexts that make an action more or less morally acceptable, along with commonsense rationales that justify the reasoning. To elicit high-quality task data, take an iterative self distililllation approach that starts from a small amount unstructured seed knowledge from GPT-3 and then alternates be- tween (1) self-distillation from student models; (2) targeted filtering with critic model trained by human judgment (to boost validity) and NLI (to boost diversity); (3) self self-imitation learning (to amplify the desired data quality). This pro- cess yields student model that produces defea- sible contexts with improved validity, diversity, and defeasibilliy. From this model distill a high-quality dataset, 5- 5-RULES BELLEES-OOTTUMMM (8-ROT), 1.2M entries of contextualizations and rationales for 115K defeasible moral ac- tions rated highly by human annotators 85.9% 99.8% 9% of the time.' Using 8-RoT we obtain a final student model that wins over all interme- diate student models by notable margin.
Btar
(d) Self-imitation Learning
Knowledge Self-distillatin
Iter 2
At BBQ
Student Model
Flan- (3B)
On a plane
When cemping
Because It's way to stay warm and cook
In a fleld of dry gress
Because It's likelly burn out of control
When you are camping dry grass
An illustration of the iterative delf-distillation pipeline on eliciting defeasible moral reasoning contex- ualizations and rationales. For the event "setting fire." different contextualizations of the action can bend its moral acceptability up (e.g.. "at a BBQ") or down (e.g., "to get revenge"). Capturing the nuances of how addi- tional contexts interplay with base actions is critical for grasping the flexible defeasibility of moral judgments.
Figure 1:
moral acceptability. For instance, the act of "know- ing where somcone lives" may carry no inherent moral weight. But when supplemented by the con- text that "the purpose is to provide assistance to a person in need," the action becomes more morally justified. Conversely, if the context shifts to "for the purpose of surveillance or spying," the same tion loses its moral grounding. This phenomenon of flexibly bending moral rules in instantiations of scenarios is widely recognized assorted cog- nitive science studies (Kwon et al., 2022; Levine et al., 2020; Awad et al., 2022; Levine et al., 2018).
1 Introduction
Moral or social judgments play a vital role in decision-making, influencing how we perceive ac- tions and behaviors daily. However, these judg- ments are far from fixed; instead, they are highly context-dependeen Contexts surrounding core action significantly strengthen weaken its
"quual contribution.
DDtasett is publically available at http::://huugingfaeee co/ addaaattss/aaaa /aaeetsskakkaaaaee rao/ Rules- of Thumb
The inherent context dependence of moral judg-

---

ments underscores the importance of understand- ing the complex interplay between actions and their grounded contexts in real-life scenarios. Delving into how different contexts bend the moral accept- ability of an action, along with the reasons behind these shifts, enables us to make informed moral judgments geared toward situational nuances.
Generate Contexts & Rationales
Self-imitation
Learning
Action: "Trespassing" Variance: Strengthener
Iterate
LM
Gen
Gen 10
New LM
Targeted Filtering
Previous works about contextualized moral judg- ment pose several challenges. First, they focus primarily on atomic contexts with limited situa- tional complexity. For instance, Ziems et al. (2023) rigidly prescribe grounded contexts to fall under concepts such as settings, roles, and behaviors, nar- rowing and fragmenting the scope of contextual- ization. Pyatkin et al. (2023) propose stepwise clarification question generation system to elicit elementary contexts of moral actions. Another limitation lies the emphasis on the defeasibil- ity of assumed moral and social judgments (e.g., "it's wrong to yell to your friend"), rather than the natural defeasibility of moral scenarios tee (Rudinger et al., 2020). Finally, existing works lack the rationales to explain why a particular context renders situation more or less morally acceptable. We address all the above limitations in this work.
NLI to diversity
Critic to boost validity
"To save someone's i life"
To save someone's life"
"You didn't know it's private property"
In order to do graffiti
"You'e rescuing person from danger"
"You didn't know it's private property"
Itero Itert
Iter
IterO
Iter1
Iter
Figure 2: Iterative self-distillation that repeats genera- tion, filtering, and self-imitation learning.
and rationales have been rated highly by human annotators on both validity (85.9%) and language quality (99.8%), ensuring their reliability in cap- turing the complexities of moral judgment within various contexts. Using 8-RoT, we further train a final downstream student model that prevailed over all intermediate student models.
In sum, in this work, we introduce the defeasi- ble moral reasoning task that involves contexts and rationales for making defeasible moral judgments. We present the iterative self-distillation methodol- ogy to gather high-quality and diverse training data for student models ($3) along with careful abla- tions of each component involved in the pipeline ($5). We distill sizeable, human-verified dataset under the defeasible moral reasoning task formula- tion ($2) and a high-performing downstream task model. We will release all student models and 8- RULES-OF-THUMB, along with subset of human- annotated gold data for training supervised critic model that mimics human ratings of the validity of contextualizations.
We introduce defeasible moral reasoning, a task to provide grounded contextualizations (or con- texts) that alter the moral acceptability of action, accompanied by commonsense rationales that jus- tify the reasoning. We aim to explicitly state underspecified contexts of actions, providing ance and interpretability moral judgments. To substantiate this task, we introduce 6- RULLES-OF- THUMB (6-RoT), a high-quality dataset of 1.2M entries of combined contextualizations and ratio- nales for 115K defeasible moral actions. 5-RoT is created through an iterative self-distillation ap- proach. Starting with a small amount of unstruc- tured seed knowledge from GPT-3, we alternate between (I) self-distillation from student models move away from the reliance on expensive API calls for GPT-3; (2) targeted filtering using critic model trained with human judgment to enhance generation validity and natural language inference (NLI) to enhance diversity; and (3) self-imitation learning to magnify the desired model properties.
2 8- 6-RULES-OF-THUME Dataset Design
We introduce 8- G-RULES-OF-THUMB (6-RoT), a dataset for the defeasible moral reasoning task. Given an everyday action with a default common- sense moral judgment, 5-ROT captures nuances of the defeasible moral action through contextu- alizations that either strengthen or attenuate the acceptability of action. It also contains ratio- nales that explain why the contextualizations affect the judgment. See example data in Figure 1. As shown in Table 1, 8-ROT contains 115K ac-
The iterative self-distillation process yields student model that generates defeasible contexts with enhanced validity, diversity, and defeasibility. From the pertooeefornnng student model, we distill the final dataset, 8-ROT. These contextualizations

---

tions and 578K entries each of contextualizations and rationales. Extensive human evaluation con- firms that 6-RULES-OF-THUME
Statisties
Human Val.
Type
Pol.
#Entry
#3-Grams
% Vld. Lan.
of high qual- ity, demonstrating high validity (85.9% for con- texts; 98.5% for rationales) and language qual- ity reflected by fluency and grammar correctness (99.8% for contexts; 99.7% for rationales), on par with human written datasets (West et al., 2022).
Action
115K
110K
All
578K
182K
85.9
99.8
Context
Stren.
266K
108K
84.2
100
Weak.
312K
130K
87.6
99.7
All
578K
275K
98.5
99.7
Rationale
Stren.
266K
170K
98.6
99.6
Weak. 312K
182K
98.4
99.8
Action and Commonsense Moral Judgment. We source our scenarios from SOCIAL-CHEM-ION 101 (Forbes et al., 2020), a bank of rules-of- of-thumb (RoTs) that describe various social, cultural, and moral norms. Each RoT consists of an action and a sociomoral judgment annotated by workers based on natural language snippets of real-life situations.
Table 1: Statistics and human validation results of 8- RULES-OF- -EEEMB..-T #Entry is the total number of data entries, and #3-Grams is the number of unique 3- 3-Grams of each data type. % Vld. is the percentage of valid data rated by humans, and % Lan. the percentage with proper language form (i.e., fluency and grammar).
Human Critic Gold Data. In addition 8-RoT, we also release a dataset of human- aanaan-n qual- ity assessments of machine -eteennnnggeeeedd contextual- izations and rationales used to train the critic model for distillation filtering ($3.1). Actions are sampled from SOCIAL-CHEM- 101, and we use GPT-3 to generate contextualiiaatios and rationales for both moral variances, each annotated by three crowd- workers. Labels are obtained by majority votes across annotators, but we keep only the subset with full agreement for validation and test sets to ensure high confidence in their labels. The critic gold data contains 1 1K actions and 20K contextualizations with quality labels across all splits.
Example RoT (judgment, action):
"It's dangerous to set fire"
Because RoT's combine everyday atomic actions (e.g., "set fire") with commonsense sociomoral judgments (e.g., "it's dangerous"), they serve as ideal seeds to be expanded with contextual nuances.
Morally Variant Contextualization. Moral vari- ance is a binary label such that strengthening contextualizations further ground the original ac- tion to be more morally acceptable, while weak- ening contextualizations have the opposite effect. Note that meaningful morally variant contextual- izations range from simple properties such as lo- cations (e.g., "in a field of dry grass") auxil- iary actions (e.g., "when you're 'r're camping") to com- plex compositional contexts with an intricate in- terplay between multiple atomic variations such as "when you're 're camping in a field of dry_grass. Thus, we focus on eliciting flexible contexts that exercise concrete and natural effects tailored to given actions, instead of pre-defining the categories of the contextualizations regardless of situational nuances (Ziems et al., 2023).
Dataset Creation via Iterative Self-distillation
Competent large language models has opened up new opportunities for automatic dataset creation through symbolic knowledge distillation (West et al., 2022). In this framework, knowledge is gen- erated from a large teacher model, filtered to im- prove data quality, and then instilled into smaller student model. Previous works have found that machine-generated datasets can surpass human- authored datasets in their quality and diversity while also achieving greater seale (West al., 2022; Bhagavatula et al., 2023; Sclar et al., 2022; Jung et al., 2023; Wang et al., 2023).
Commonsense Rationales. A critical missing piece from previous works on grounded moral judg- ments is rationale that ties together the actions and contextualizations by explaining the reasoning be- hind the defeasible effect (e.g., the context "in a field of dry grass" might make the action "setting fire" less morally acceptable "because it's likely burn out of control"). provides complete picture of how each context achieves a moral variance, paving the way toward dissecting and understanding the varying shades of moral judgments.
In this work, we create --RULESS-F- THUMM with an iterative self-distillation approach which minimizes the resource bottleneck from expen- sive GPT-3 API calls. Our approach follows three stages after producing an initial student model us- ing relatively small-scale seed knowledge from GPT-3: (1) self-distillation from student mod-

---

Top Greedy
Top Sampling
Auto (Critic)
Human
Auto (Critic)
Human
Model
#Trm.
Vld.
Avg.
Vid.
Defease.
Lan. Rationale.
#VId. #Unq. Vld.
#VId. #Unq. Vid.
GPT-3
0.53
0.69
0.56
0.37
0.98
0.93
Distilluse
85K
0.68
0.75
0.54
0.42
0.98
0.91
6.40
5.63
5,36
4.78
No Critic
143K
0.60
0,68
0.51
0.39
0.98
0.94
5.63
5.00
4.66
4.23
SeltDistill
434K
0.75
0.80
0.60
0.48
0.97
0.93
7.08
5,83
6.04
5.05
'Top 1 Only
53K
0.71
0.77
0.59
0.48
0.98
0.93
7.06
3.16
5.74
2.54
No NLJ
492K
0.75
0.80
0.64
0.50
0.98
0.92
7.12
5.89
5.93
4.97
SeliDistlla
466K
0.79
0.83
0.62
0.50
0.98
0.93
7.60
6.15
6.26
5.21
'Top 1 Only
57K
0.73
0.78
0.62
0.49
0.97
0.93
7.28
2.60
6.13
2.16
-No NIJI
567K
0.80
0.83
0.63
0.51
0.98
0.91
7.65
5.92
5.93
4.73
-oo Self-distill
869K
0.75
0.80
0.62
0,50
0.98
0.94
7.19
5.95
6.01
5,08
Distillana
578K
0.86
0.88
0.71
0.56
0.99
0.92
8.40
6,45
7.26
5.69
Table 2: Automatic and human evaluation of distilled models across three iterations. We evaluate both the top model generation by greedy decoding and the top candidates by nucleus sampling. Best results are bolded and second best results are underlined (to declutter the table, we remove the styles for Lan. and Rationale. as their results are approximately the same across all models).
els to move away from reliance the expensive GPT-3 model: (2) targeted filtering to critically select high-quality and diverse data; and (3) self- imitation learning to amplify learning signals.
quality-assessmenn gold labels introduced in $2. We fine-tune DeBERTa-V3 (He al., 2021) on these annotations, resulting in a critic model that achieves high accuracy on a held-out validation set. Using the trained critic model, we filter the teacher model generations to remove errors from the distillation data and obtain an initial medium- quality training corpus, Do. of 85K examples.
3.1 Gathering Medium-quality Seed Data to Train an Initial Student Model
Eliciting Raw Seed Data from GPT-3. To set the stage for the later knowledge amplification pro- cess via iterative self dif-distillation, we gather seed knowledge from GPT-3 3 (175B)², the teacher model, instill into Flan-Tr T5 (3B), smaller base student model. To do so, we jointly generate defeasible contextualizations for moral actions and associated rationales with carefully engineered task-directed prompts. 3 To encourage diverse alternatives this initial seed, we generate two contexts/rationales each for both strengthening and weakening moral variances; in total, we obtain 212K contextualiza- tions and rationales for 60K base actions.
Training Initial Student Model Our goal is to train an initial student model capable of generating contextualizations and rationales for given action and moral variance. We fine-tune Flan-T5 (3B) (Chung et al., 2022) on Do for epochs produce Distill our base student model.
3.2 Refining Intermediate Student Models via Iterative Self-distillation
We refine the quality of the base student model by further amplifying desired generation properties through an iterative self-distillation process, uti- lizing no additional generations from the teacher model. Our iterative process has some key differ- ences from Bhagavatula et al. (2023), in that we focus improving diversity in addition quality.
Filtering Raw Seed Data with Critic Model. Despite careful prompting following the task for- mulation, raw generations from GPT-3 remain noisy. Thus, we train a binary classifier to sim- ulate human quality assessments on GPT-3 gener- ated contextualizations inspired by West et al. (2022).4 To train the critic model, we the hu-
Self-distillade from Student Models. First, we generate corpus of contextualizations and ratio- nales using Distill on set of newly sampled actions from the training split of SOCIAI-CHRM- 101. Given an action, we use nucleus sampling (Holtzman et al., 2020) (p = 0.9) to produce 10 contextualizations and rationales for each moral variance.
-vii--dvvnee--00 is used wherever GPT-3 is mentioned *See Appendix A for GPT-3 prompt details.
*Preliminary human evaluation results show that whenever contextualization is deemed high quality, the rationale is most likely to be high quality too (over 90% of the time). There- fore. although some improvements could be gained on the rationales, in this work, we focus on improving the quality of contexts which starts at ~50% valid, as there's much more room to improve their quality.
Seec critic model training details in Appendix D.

---

Targeted Filtering. Next, we again perform geted filtering on the newly self-distileed data to (1) ensure the validity of the data via the super- vised critic model, similarly to the treatment of Do described in $3.1; (2) encourage diverse model outputs by reducing repetition among valid contex- tualizations using a Natural Language Inference filter (NLI) (Liu et al., 2022).
Model
Vid.
Avg.
GPT-3 (Teacher)
0.53
0.69
Falcon-7B-Insten
0.39
0.54
GPT-3.5 (ChatGP))
0.71
0.77
GPT-4
0.77
0.82
Distill
0.86
0.88
Table 3: Our Distill model outperforms all other
baseline models on the top 1 generation via the auto- matic evaluation.
NLI is an NLP task that determines whether premise statement entails or implies the truth of a hypothesis statement (Bowman et al., 2015; Liu et al., 2022). For a given pair contextualizations A and B. we say the pair is mutually entailed both A + B and B + A are entailments, indicat- ing high confidence of not only lexically but also semantically repetitive content. We filter out these mutual entailments such that most one exam- ple from each pair remains in the dataset, thereby removing redundant signals in the training data. We use RoBERTa- Large pretrained on the WANLI dataset (Liu et al.. 2022) to compute entailment scores between each of the generated contextual- izations for given input. Formally, the filtering process is defined as:
and 98.5% of rationales are deemed high quality (see details of dataset stats and human validation in Table 1). Using this large-scale, high- quality dataset, we train a final student model, Distill outperforming all previous intermediate student models and the teacher model (i.e., GPT-3).
Experimentation Setups
4.1 Evaluation Data
We hold out test set of 6K actions from the same distribution in SOCIAL-CHEM-10 For each model we generate contextualizations and ratio- nales over the test set using both greedy decoding (top 1 candidate) and nucleus sampling (top 10 candidates) with p = 0.9.
(C) =Vj E -acceptwrrcc (PNLI(C, < 0.5 V (NLI(C,, < 0.5)
where accept determines a context is accepted into the filtered set of unique candidates, Cj is the k candidate, and PNLz(C1, 62) is the predicted score that context C1 entails C2-
4.2 Evaluation Metries
We aim evaluate the overall quality of model- generated contextualizations and rationales (valid- ity), a model's capability to produce diverse contex- tualizations for given action and moral variance (diversity), and also the degree of moral variance that the model's 's contextualizations provide (defea- sibility). Finally, guage quality of model generations reflected by fluency and grammatical correctness.
This process results in a filtered self-generated corpus, D1, of 434K examples. We then train SelfDistill, using Dj starting from Distillase-
To improve the student model further, we repeat the self-distillation process using SelfDistill, as the base. Using SelfDistill,, we generate a high- quality corpus, D2, automatically filtered by the supervised critic model and NLI to train a second- iteration self- distilisd student model, SelfDistillg.
also evaluate the general lan-
Validity. For contextualizations, we the critic model for automatic evaluation. For greedy gen- erations, we compute the ratio of generations in the test set that pass the critic filter threshold as de- fined in $3 (Auto/VId.) and the average predicted critic score across the test set (Auto/Avg.). For sam- pled generations, we compute the average number of contextualizations out of the top 10 candidates that pass the critic filter threshold (Auto/#Vld.). We also conduct human evaluation vet validity of contextualizations to complement conclusions drawn from automatic evaluation (Human/Vld., #VId.). We evaluate the validity of rationales with human evaluation (Rationale.).
3.3 Training Final Student Model with Large-scale, High-quality Data from Refined Self-distilled Model
Using SelfDistill,, we produce 5-RULES-OFF THUMB by generating contextualizations and ra- tionales for 94K actions and combining them with previous training sets. We apply the NLI filter as above and filter by a restrictive critic model thresh- old of 0.96 to ensure high confidence of quality in the dataset. Human evaluation subset of 1000 samples shows that 85.9% of contextualizations

---

200
lidity (26.8%) and defeasibility (51.4%), demon- strating the effectivenesss of distilling small special- ized knowledge models from large general-purpose close-sourced models like GPT-3.
300
150
250
200
100
150
100
50
Filtering by the critic model improves the qual- ity of contextualizations. Our results show that filtering training examples by the critic model im- proves the quality of generated contextualizations, in line with previous findings (West et al., 2022; Bhagavatula et al., 2023). In partieular, duct an ablation study without using critic filter- ing (Distill critic), resulting in lower perfor- mance on almost all contextualization metrics and similar performance on others, despite its training set being 70%% larger than Distillaase
Pet
Project
Vulnerable
Public
Supervisor
Safety
Gift
Elderll
Commitment
Doctor
Friend
Students
Children
Distress
Compassion
Advantage
Risk
Productivity
Power
Abuse
(a) Contextualization
(b) Rationale
con-
Figure 3: Top 10 topics and their counts among 10K sampled contextualizations and rationales from 6-RoT.
Diversity. We use automatic evaluation assess the diversity of generated contextualizations as is a generally well-defined dimension. Similarly to the mutual entailment NLI filter in $3, we compute the bidirectional entailment probabilities between each pair of contextualizations across valid candi- dates and report the average number of semanti- cally unique generations (Auto/#Unq. Vld.). This metric deseribes the model's capability to produce multiple varied contextualizations for a given input, directly indicating the diversity of model outputs.
Training student models on diverse self- generated data improves validity and diversity over greedy decoding. We find that omitting di- verse candidates during the self-distillation process results in drastic decrease in the diversity of the subsequent models. In particular, ablations using only the top candidate in distillation (SelfDistilll- Top I Only and SelfDistillz-Top I Only) produce significantly less valid and unique generations com- pared to SelfDistill, (5.05 >2.54) and SelfDistill (5.21 This insight is critical as previous symbolic knowledge distillation works (West al., 2022; Bhagavatula et al., 2023) focused primarily on improving the validity of downstream student models without screening the diversity.
Defeasibility. We break down human evaluation of contextualization validity into more granular an- or "slightly" shifting the moral implication of the base action. Defeasibil- ity of the contextualizaaioon (Defease.) is computed (#contextSaggiticam=> 1 # Hall, indicating the degree to which the contextualiza- tions affect the morality of the original actions. See Appendix B for annotation details.
swer
Filtering repetitions with NLI improves the di- versity of candidates from student models. Is training on more candidates itself, without filtering out repetitions, sufficient for improving the diver- sity of downstream models? To answer this ques- tion, we conduct ablation studies without using the NLI mutual entailment filter, i.e., SelfDistill -No NLJ and SelfDistillz-No NLI, Our results show that despite being trained with more data, these two models generate less valid and unique contextual- izations compared to SelfDistill, (5.05-4..97) and SelfDistill, -4.733,, shedding light on the im- portance of having truly diverse training data by removing redundancy.
Language Quality. We evaluate the language quality (i.e., fluency and grammar correctness) of generated contexulizations and rationales with hu- man evaluations (Lan.).
5 Results and Discussions
In this section, we present results and insights of the iterative self-distillation process and an analysis of the resulting dataset, 8- R-RO..
5.1 Insights of Iterative Self-distillation
As shown in Table 2, student models improve across iterations during the iterative selfldistillation process on all of validity (0.54 +0.71), diversity +5.69,, and defeasibility (0.42-40.56). In particular, the final student model, Distill wins over GPT-3 (the teacher model orders of magni- tude larger) by substantial relative gain on va-
Successive iterative training leads to higher quality student model than single iteration. We train an ablation model (SelfDistillz-No Self- distill) combining the actions in the training sets of both the first and second rounds of distillation,

---

ture of captured contextualizations and rationales.
30%
Contextualization. To understand what topics contextualizations in 6-RoT represent, we con- duct topic analysis with BERTopic (Grootendorst, 2022), an easily interpretable off-the-shelf topic modeling technique. Frequent topics of contextu- alizations are shown in Figure 3(a), which involve daily objects entities (e.g., pet, project, super- visor, gift, doctor) and characters or properties that carry moral weights (e.g., vulnerable, public, safety, elderly, commitment). In particular, vul- nerability serves as a critical weakening context if characters in the action are among vulnerable popu- lations (e.g., "taking advantage of people" becomes less acceptable if "they are in vulnerable state"). See topics analysis details in Appendix $E.1.
16% role (own)
(othar)
Interaction
role (other)
5%
fooling
action
13% timing
6%
4%
(own))
0%
6%
location
object
9%
Intent
intent
object
(a) strengthening
(b) weakening
Figure 4: Qualitative analysis of contextualization cate- gorics per moral variance. (a) and (b) are for strength- ening and weakening contexulizations, respectively.
Warting spend time family long and difioutt work day
Timing
Teling another person's 's secret Tellng secret about close friand to stranger
Interaction
location
Farting lot
crowded elevator
Role (other)
Trying repeateclly contact someone
We also manually analyze the categories of 200 contextualizations sampled from 6-ROT. As shown in Figure 4, frequent types of contextualizations in- clude role specifications of the action taker (own), other characters involved in the scene (other), and setting specifications such object, timing, and lo- cation. In addition to individual role specifications, interactions, relationships, and dynamics between multiple roles add rich groundings that carry moral significance. Figure 5 shows example contextual- izations under each category. Contextualizations in 8- RoT have an average of 1 11.7 words, providing concrete, specific contexts tailored to each action. We further conduct qualitative error analysis over generations from the Distilliinal to gauge what makes a context implausible or incorrect.
dingeeros situation and needs help
Causing between friends
order to manipultte tham for personal gain
Feeling
Leaving a party early You are fnaiing and need to rest
Role (own)
Getting involved in school activities
You are under the infleenc drugs alcohol
Action
Risking
person is climbing mountain other's life
Object
wanting reletionahip certain people The relationshpp domaging
Figure 5: Example rich contextualizations per category.
but with only one iteration of self-lcarning from Distill SelfDistill,, which has been trained us- ing the same actions over two rounds of successive training for the same number of total training steps, outperforms this ablation on almost all metrics, showing the effectiveness of amplifying learning signals via successive iterations of self-learning,
Trivial Context: Context that adds details that may often be relevant to morality for other actions, but is rendered trivial in this particular case, e.g., "Staying up all night" vs. "Staying up all night while in a relationship."
The final student model outperforms orders of magnitude larger off-the-shelf models. We evaluate the zero-shot performance of some of the most powerful general-purpon LLMs of varied sizes with 1000 sampled examples from our test set. We use the same instructions as we used to dis- till the seed knowledge from GPT-3 to prompt these baselines in a zero-shot manner (see Appendix $A). Results in Table 3 show that despite some of the zero-shot models being orders of magnitude larger than our final student model (e.g., 175B vs. 3B), our model outperforms them all, proving the effec- tiveness of our proposed approach.
Infeasible/U Unlikely/Unnatuueeeu Context: Con- text that is infeasible, highly unlikely in real world setting, or unnatural by itself and/or in relation with the action, e.g., "Participating in chess team" vs. "The team is made up of people who have been convicted of serious crime."
Opposite Context: Context that adds details op- posite to the desired moral variance, e.g., "Offering people money" vs. "Offering money to someone who is in vulnerable financial situation" when prompted for weakening context.
5.2 Delving into 5- ---RLES-OP-THMME
We analyze 8- RF THUMB to gauge the dataset composition and gain a comprehensive pic-
We conduct the same topic analy- sis on rationales. Results in Figure 3(b) highlight
Rationales.

---

that common topics in justifying a moral decision involve important roles (e.g., friend, students, chil- dren) and common human values (e.g, distress, compassion, risk, productivity, power, abuse). See topics analysis details in Appendix $E.1.
Cultural Biases It's also important note the sensitivity of moral reasoning to cultural differ- ences. In fact, previous studies have pointed out that cultural bias is a pervasive phenomenon across many NLP models (e.g., GPT-3/3.5/4) and tasks (c.g., hate speech detection with Perspective API, RewireAPl, HateRoberta) (Santy et al., 2023). To better represent diverse perspectives to our contex- tualizations, we (1) abstain from producing an ab- solute moral judgment given an action and (2) pro- mote diverse distillation as discussed previously.
Diving into specific examples of why contexts shift the moral implications of actions, we find common values that uplift the acceptability of an action include empathy, kindness, support, and re- spect (e.g., "...someone who is in need of emo- tional support, which shows empathy and kind- ness"). Additionally, (in)equality or (un)faimess is another dimension of value that carries signifi- cant weight on moral implications (e.g.. "...taking advantage of someone else's generosity when you have the resources to provide for yourself") Fi- nally, contexts that are explained to promote or im- pede physical/nental wellbeing. financial health, or learning/vorking productivity (e.g., "...helping them learn, which is beneficial for their future") are also common. These qualitative results show consistency with the automatic topic analysis.
However, these measures cannot eliminate all traces of bias in the final model, so we also qual- itatively probe 8-RoT to examine cultural biases. Admittedly, our dataset and student models are distilled from GPT-3, which shown to present Western-centric perspectives, it likely that our dataset and models inherit this cultural bias as well (Santurkar et al., 2023; Abdulhai et al., 2023). For example, when prompted for a weakening contex- tualization for the action "Not having freedom of speech in (country): )." For some countries such Japan, the United Kingdom, and the United States, the top generated context is "in workplace set- ting." Yet for other countries such as China, India, Thailand, Korea, and Russia, Distill produces different results which might imply these countries have varying levels of human rights concerns (see details in Appendix $E.3). This example aligns with our intuition that the student model might dis- play Westem-centree biases, and it fits with previ- ous finding by Fraser et al. (2022) that such models are likely to encode the cultural biases aligned with those involved in training data annotation.
Toxicity Analysis. Since the seed actions of 8- ROT sourced from SOCIAL-CHEM-10 (Forbes et al., 2020) mainly concern everyday situations, they are at a low risk of containing toxic content. In addition, due to the careful filtering with the critic model and the iteratively refined self-distillation process, we expect most of the low-quality (includ- ing potentially biased data points) to be already filtered out from the final dataset. However, be- cause any toxicity in moral reasoning data is es- pecially detrimental and could easily propagate through downstream tasks, we run toxicity anal- ysis on subset of 40K data points from 8-ROT using the Perspective API (Lees et al., 2022). Our results show that the average toxicity score is 0.09, indicating very low toxicity overall. In a qualita- tive analysis of the data rows with higher toxicity scores (with a max of 0.83), we observe a strong pattern where the base action itself is problematic, and the distilled contexts produce the target moral variance without contributing significantly to the toxicity of the complete statement (see examples in Table 9 of Appendix $1.22.. While no existing toxi- city detection method can accurately measure all potential biases, this analysis provides reasonable confidence in the lack of toxicity in our generated contextualizations and rationales.
Thus, despite our careful filtering process, it is clear that culturally biased generations can still be produced by our model and may be present in 8- ROT. As such, users of this dataset must exercise discretion and care when applying it to novel use cases, and it should never be used as prescriptive ethical advice. This points to key direction for future work to further enrich multicultural repre- sentations in computational moral reasoning and other commonsense understanding tasks.
6 Related Work
Computational Morality. Jiang al. (2022) present Delphi, a commonsense moral model trained to present descriptive view of ethical judg- ments. Ammanabrolu et al. (2022); Hendrycks et al. (2021); Pan et al. (2023) incorporate moral values

---

in an interactive game environment align agent actions with social norms. Kim al. (2022) uses social norms to guide conversational agents' proso- cial responses. Jin et al. (2022) introduce MoralEx- ceptQA, a task of identifying the acceptability of breaking well-known moral rule in different sit- uations. Fung et al. (2023) introduce NormSAGE, a framework to discover multi-Lingual and multi- cultural norms on-the-fly. There is also a promi- nent line of work in quantifying social, political, and moral values and views presented in language models by using well-established public opinion surveys or social science instruments (Santurkar et al., 2023; Hartmann et al.. 2023; Fraser et al.. 2022). Recently, Sorensen et al. (2023) builds the Kaleido model to capture the importance of plural- istic human values in moral decisionnmakingg
et al., 2020; Brahman et al., 2021). The advent of in-context learning (Brown et al., 2020; inter alia) led to growing interest in using LLMs to generate rationales in few-shot prompting mode (Wiegreffe et al., 2022; Marasovic et al., 2022; Wei et al., 2022). While so-called explanation-basse prompting shows encouraging results, it is hindered by costly API calls. We instead endow accessible models with joint generation of contextualizations and rationales, reducing the computation required.
Automatic Data Generation. Previous works on automatic data generation have worked on cre- ating datasets for commonsense reasoning (West et al., 2022; Bhagavatula et al., 2023; Liu et al., 2022; Wang et al., 2023; Kim et al., 2023), dia- logues (Kim et al., 2023; Xu et al., 2023; Geng et al., 2023; Chiang et al., 2023), and summariza- tion (Sclar et al., 2022; Jung et al., 2023). West et al. (2022) propose the symbolic knowledge dis- tillation framework, with several follow-up works to extend it with iterative distillation (Sclar et al., 2022; Jung et al., 2023; Bhagavatula et al., 2023), We further build on this paradigm encourage diversity in distillation and apply our method to moral contextualization and rationales.
Defeasible Reasoning. Defeasibility describes the idea that new information might strengthen or weaken a given interpretation (Rudinger al., 2020). This concept has been used in multi- ple works for different applications: Rudinger et al. (2020) introduced two task formulations, one which concerns generating strengthening or weak- ening updates to a premise and hypothesis, and the other one which concerns classifying whether premise and an update strengthen weaken the hy- pothesis. Madaan al. (2021) improved upon the latter task by modeling inference graphs. Our work relates to recent efforts towards contextualizing moral reasoning. Pyatkin et al. (2023) developed ClarifyDelphi, a system capable of asking clarifi- cation questions to elicit the context surrounding a judgment. With NormBank, Ziems et al. (2023) in- troduce framework for grounded reasoning about norms, adding environmental conditions and agent characteristics. Rather than QA setup or provid- ing an atomic groundings, in 8-RoT, we instead provide free-text contextualizations along with sup- porting rationales which justify how each piece of context alters the morality of the an action.
Conclusion
In this work, we highlight the importance of dy- namic contexts in shaping moral reasoning. We introduce defeasible moral reasoning, task of providing grounded contexts elucidate varying degrees of moral acceptability, accompanied by commonsense rationales to justify the reasoning. We employ an iterative self-distillation method- ology to create a high-quality and diverse dataset, SUULES-OF-THUME comprising over 1.2M com- bined entries of contextualizations and rationales for 16K defeasible moral actions. Through this iterative approach, we also obtain small student model capable of generating defeasible contexts with improved validity, diversity, and defeasibility.
Our work aims to promote a deeper understand- ing of the intricate interplay between defeasible moral actions and grounded contexts that shape moral acceptability in nuanced and complicated way, building a strong foundation for future works on enriching cultural representations in computa- tional moral reasoning research. We hope 8-RoT serves as a rich resource for the community study how moral judgments are made to unveil this unique perspective of human intelligence.
Explanations and Rationales. Free-form ratio-
nales have emerged as promising direction to promote models' reasoning capabilities and aid in- terpretability by filling in the knowledge gap. Prior works on rationale generations take either su- pervised approach by training on human-written explanations (Camburu et al., 2018; Rajani al., 2019; Narang et al., 2020; Kumar and Talukdar, 2020) or a weakly supervised approach (Glockner

---

Limitations & Ethical Considerations
could not be justified even with further contextual- izations.
Large-language models can generate text that might be biased and insensitive to a user's socio-cultural context (Bordia and Bowman, 2019; Sharma et al.. 2021; Hovy and Prabhumoye, 2021). By intro- ducing the defeasible moral reasoning task, we consider different contexts and rationales, making a step towards being more diverse and inclusive in accounting for different perspectives.
A Acknowledgement
The authors thank the anonymous reviewers. This research was supported in part by DARPA under the ITM program (FA8650-23-C-7316) and the Allen Institute for AI.
with filtering by the critic model, it is possible for biased or incorrect out- puts to be produced by distilled models. The critic model is trained to be a strong approximation of human judgment, but it is not perfect, and due to the scale we cannot collect human annotations to verify all examples in model training data or 8- RULES-OF-THUMB.
However,
References
Marwa Abdulhai, CI' ement Crepy, Daria Valter, John Canny, and Natasha Jaques, 2023, Moral foundations of large language models.
Prithviraj Ammanabrolu, Liwei Jiang. Maarten Sap, Hannaneh Hajishirzi, and Yejin Choi. 2022. Aligning to social norms and values in interactive narratives.
In addition, determining moral variance is form of moral judgment and so may not have a clear answer in some cases, egg., trolley problems. There are certainly contextualizations which differ- ent groups of people disagree over whether they make the base action more or less acceptable, as could be seen in our critic gold data, where we em- ployed inter-annotator voting to reconcile differing opinions.
Edmond Awad, Sydney Levine, Andrea Loreggia, Nicholas Mattci, lyad Rahwan, Francesca Rossi, Kartik Talamadupula, Joshua Tenenbaum, and Max Kleiman- Weiner.. 2022. When is it acceptable break the rules? knowledge representation of moral judgement based on empirical data, arXiv preprint arXiv:2201.077633
Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Lianhui Qin, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, and Yejin Choi. 2023. 12d2: Inductive knowledge distil- lation with neurologic and self-imitation.
With these points mind, our dataset and mod- cls should never be used as direct moral advice to humans; they are solely intended to be resources for more nuanced reasoning and interpretation in computational morality research.
Shikha Bordia and Samuel R. Bowman. 2019. Identify- ing and reducing gender bias word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Student Research Workshop. pages 7-15, Minncapolis, Minnesota. Association for Computational Linguistics.
We will publicly release our dataset and final student model promote open-source research but for gated research purposes only. To mitigate risks misuse, we will inform all users about potential ethical implications and risks of the resource and require them to complete a data/model user agree- ment form to acknowledge their consent to proper usage of the resource. This will also help track how our resource repurposed for downstream applications.
Samuel R. Bowman, Gabor Angeli. Christopher Potts. and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Finally, for extremely morally negative situa- tions, there might not be any reasonable contex- tuaiizations to make the action justifiable (e.g., "genocide"). However, the situations we source from SOCIAL-CHEM-10 101 focus everyday ac- tions that do not carry extreme moral implica- tions. Therefore, we consider identifying impossi- ble cases to further contextualize out of the scope of our current study. Future work could investi- gate more inherently morally charged cases that
Faeze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2021. Learning to rationalize for non- monotonic reasoning with distant supervision. Pro- ceedings of the AAA/ Conference on Artificial Intelli- gence, (5(14)::11))))) 12502-2222222000
Tom Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 197-1

---

Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natu- ral language inference with natural language explana- tions. In Advances Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging non- linearities and stochastic regularizers with gaussian error linear units. CoRR, abs/ 600.084155
Wei-L.in Chiang, Zhuohan Li, Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Pa- tel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. 2021. What would jiminy cricket do? towards agents that behave morally. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Dulasels and Benchmarks Track (Round 2).
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun. Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling fineetued language mod- els.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration.
Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Lan- guage and Linguistics Compass, (5(8)e11433)
Maxwell Forbes, Jena D. Hwang. Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chem- istry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 653-670, Online. Association for Computational Linguistics.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ro- nan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap. Regina Rini, and Yejin Choi. 2022. Can machines learn morality? the delphi experiment.
Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mi- halcea, Josh Tenenbaum, and Bernhard Schölkopf. 2022. When to make exceptions: Exploring language models as accounts of human moral judgment.
Kathleen C. Fraser, Svetlana Kiritchenko, and Esma Balkir. 2022. Does moral code have a moral code? probing delphi's 's moral philosophy. In Proceedings of the 2nd Workshop on Trustworthy Natural Lan- guage Processing (TrusiNLP 2022), pages 26 26-42, Seattle, U.S.A. Association for Computational Lin- guistics.
Jachun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2023. Impossible distillation: from low-quality model to high-quality dataset & model for summa- rization and paraphrasing.
Yi R. Fung, Tuhin Chakraborty, Hao Guo, Owen Ram- bow, Smaranda Muresan, and Heng Ji. 2023. Norm- sage: Multi-lingual multi-cultural norm discovery from conversations on-the-fly.
Hyunwoo Kim. Jack Hessel, Liwei Jiang, Peter West, Ximing La. Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhami, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization.
Xinyang Geng, Arnv Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: dialogue model for academic re- search. Blog post.
Hyunwoo Kim, Youngjae Yu, Liwei Jiang, Ximing Lu, Daniel Khashabi, Gunhee Kim. Yejin Choi, and Maarten Sap. 2022. ProsocialDialog: A prosocial backbone for conversational agents. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4005-4029 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Max Glockner, Ivan Habernal, and Iryna Gurevych. 2020. Why do you think that? exploring faithful sentence-level rationales without supervision. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1080-1095, Online. Association for Computational Linguistics.
Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with blass-based tf-idf procedure. arXiv preprint arXiv:2203.0577
Sawan Kumar and Partha Talukdar, 2020. NILE: : Natu- language inference with faithful natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 8730-8742, Online. Association for Computational Linguisties.
Jochen Hartmann, Jasper Schwenzow, and Maximil- ian Witte. 2023. The political ideology of conver- sational ai: Converging evidence on chatgpt's pro- environmental, left-iibertarin orientation.

---

Joseph Kwon, Josh Tenenbaum, and Sydney Levine. 2022. Flexibility in moral cognition: When is it okay to break the rules? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44,
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain your- self! leveraging language models for commonsense reasoning. In Proceedings of the 57th Anmual Meet- ing of the Association for Compulational Linguistics, pages
Alyssa Lees, Vinh Q. Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective api: Efficient multi- lingual character-level transformers.
Florence, Italy. Association for Computational Linguistics.
Rachel Rudinger, Vered Shwartz, Jena D. Hwang. Chan- dra Bhagavatula, Maxwell Forbes, Ronan Le Bras, Noah A. Smith, and Yejin Choi. 2020. Thinking like skeptic: Defeasible inference in natural language. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4661-4677 Online. Association for Computational Linguisties.
Sydney Levine, Max Seriman-Weineee Nick Chater, Fiery Andrews Cushman, and Joshua B. Tenenbaum. 2018. The cognitive mechanisms of contractualist moral precision-making Cognitive Science.
Sydney Levine, Max Wleiman-Weinedt Laura Schulz, Joshua Tenenbaum, and Fiery Cushman. 2020. The logic of universalization guides moral judgment. Proceedings of the National Academy of Sciences, (1744))26158-26166
Shibani Santurkar, Esii Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang. and Tatsunori Hashimoto, 2023. Whose opinions do language models reflect?
Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. NLPositionalit:: Characterizing design biases of datasets and models. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 9080-9102, Toronto, Canada. Association for Computational Linguisties.
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. WANLI: Worker and Al collabora- tion for natural language inference dataset creation. Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826-686 (666-6647440 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Melanie Sclar, Peter West, Sachin Kumar, Yulia Tsvetkov, and Yejin Choi. 2022. Referee: Reference- free sentence summarization with sharper controlla- bility through symbolic knowledge distillation. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9649 96699966.. Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! improving defeasible reasoning by first mod- eling the question scenario. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6291-6311
Ana Marasovic, Iz Beltagy, Doug Downey, and Matthew Peters. 2022. Few-shot ratioaattinilinatoionn with nat- ural language prompts. In Findings of the Associa- tion for Computational Linguistics: NAACL 2022, pages 410-424, Seattle, United States. Association for Computational Linguistics.
Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating gender bias in natural language inference. arXiv preprint arXiv: 20X.:22005.555 005.05541.
Taylor Sorensen, Liwei Jiang, Jena Hwang, Syd- ney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Choi. 2023. Value kaleidoscope: Engaging ai with pluralistic hu- man values, rights, and duties.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. 2020. Wt5?! training text-to-text models to explain their predictions. ArXiv, abs/2004. 14546.
Peifeng Wang. Zhengyang Wang. Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. 2023. Scott: Self- consistent -hain-of-thought distillation.
Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li. Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendryeks. 2023. Do the rewards justify the means? Measuring trade-off's be- tween rewards and ethical behavior in the machiavelli benchmark. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 26837-26867. PMIRR
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Xia, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.1
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang. Ronan Le Bras, Ximing Lu. Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of
Valentina Pyatkin, Jena Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chandra Bhagavatula. 2023. Clarifydelphi: Reinforced clarifi- cation questions with defeasibility rewards for social and moral situations.

---

the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-466 1602-4625, Seat- tle. United States. Association for Computational Linguistics.
Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta. Mark O. Riedl, and Yejin Choi. 2022. Reframing human-ai collaboration for generating free-text ex- planations. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 632-658.. Association for Computational Linguistics.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: open-source chat model with parameter-efficient tuning on self-chat data.
Caleb Ziems, Jane Dwivedi- Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2023, Normbank: A knowl- edge bank of situational social norms.

---

A
GPT-3 Prompt for Seed Data Generation
Human Val.
Type
Pol.
Vid.
Signifieant. Slight
We prompt GPT-3 with the action and moral vari- ance to jointly generate the contextualization and rationale in zero-shot setting. We use nucleus sampling with p 0.9 and presence and frequency penalties of 0.5.
All
85.9
79.5
20.5
Context
Stren.
84.2
79.0
21.0
Weak.
87.6
80.0
20.0
All
98.5
93.4
6.6
Rationale
Stren.
98.6
92.9
7.1
Weak.
98.4
93.9
6.1
We explore multiple prompts, including few-shot versus zero-shot prompting and multiple variations wording. From qualitative analysis, we find that this zero-shot prompt performs well, while also reducing the number of tokens required for each request.
Table 4: Expanded human validation results of 6- breaking down the degree of moral variance and logical completeness. Vld. the same as Table 1. % Signiificant and Slllight are (for Context) the percent Vld. which significantly or slightly im- pact the moral variance relative the base action; and (for Rationale) the percent of Vld. which are annotated as fully or somewhat valid respectively.
Given an action, write down a situation in which the action is more (ethical/unethical] and give a reason for why it makes the action more
B Human Annotation Details
Use the following format.
We use Amazon Mechanical Turk as the interface for all human annotations and evaluations. For each task, we estimate the completion time by doing selection of jobs ourselves in order target compensation rate of $15 per hour.
Action:
Situation:
Explanation:
Action: (action)
Situation:
B.1 Critic Model Data Collection
For the critic training data, collect human notations the quality of GPT-3 generated textualizations, which we then portion into an 80%/10%/10% vrain/validationntest split. We com- bine the "Neutral" and "Opposite" answer choices into single "Invalid" label. As described in $2, to reduce noise in the dataset, we collect 3 annota- tions per generation and vote to produce the gold label; for validation and test sets include only cases where all three annotators agree.
B.2 Human Evaluation of Model Generations and the Final Distilled Dataset
We design the dataset human evaluation to gather fine-grained assessments contextualizations and rationales. As such, we include options for "slightly" valid contextallizations and "somewhat" valid rationales along with "invalid". which allows us to gain a more in-depth understanding of the quality of the data. In Table 1 we collapse these labels into simply "valid" and "invalid". consider- ing the first two options "valid" and only the last option "invalid".
We find high inter-annotator agreement on these evaluations, with the questions on language qual- ity and rationale validity at over 90% full three-

---

Sentence:
Sisource-action-O) $(modiffer-O)
when Sigeneraalon-uudatee))) because Sigemeration-xxplaaatoonn
Update Effectiveness Does the update make the action S(modifier-0)
makes 5(modifler-o).
Neatral does not make the action 5(modifler-0) or the opposite.
Oppenisy has the agposite effect of makung the action 0/modifier-O).
Explanation Effectiveness Does the explanation provide facts reasoning that update makes the action
5/modifier-0p2
applicable: Update Effectiveness
Yes
Sentence:
$isource-action-O) S(modifier-O)
5igeneration-updaae-)) because Sigemeraaioonexplaaatioooo
Q1. Update Effectiveness Does the update make the action 5(modifier-0) O)?
makes action #(modifler-0)
#0modifier-0p opposite.
Oppaett opposite making the action S/modiffer--)
Explanation Effectiveness Does explanation provide facts reasoning
update makes the action
$/modifier-O)
Yol provides facts or reasoning that dearly explain how the update makes the action $(modifler-0)
Samewhus provides facts reasoning that somewhat explam how the update makes the action $pmodifler-).
does not effectively reason about how the update makes the action S(modifier-o)
Figure 6: The human data collection template for the critic model gold training data collection.

---

annotator agreement. On the context validity ques- tion, we find 57% full agreement, and in 82% of cases, two out of three annotators agree on a label. We expect a lower agreement on this question com- pared to others, since the moral judgment of the action and context is an inherently subjective task, and the answer may not be clearly defined in all cases. With the high two-way agreement, have confidence in the accuracy of labels after applying voting across annotators.
Since the moral variance of given contextual- ization may be ambiguous or contested between different annotators, we filter the validation and test data to only the subset on which all annotators agree on label.
We conduct small search of hyperparameters using the dev set around the values suggested by West et al. (2022), and we find training with batch size of 4, learning rate of 5e-06, and dropout of 0. to be effective to produce a discerning critic model. Because the dataset is heavily class-imbalanced with about 3:1 high-quality to low low-quality contextu- alizations, weight the loss of low-quality exam- ples in training with the reciprocal of the imbalance. We employ early stopping to save the checkpoint with the lowest validation loss after 15000 training steps. We use the special tokens [ACTION] to de- note the start of the action, and [POS] and [NEG] respectively to denote contextualization with pos- itive and negative moral variance. The critic model training takes approximately 3 hours to train single NVIDIA Titan XP GPU,
C Full Iterative Self-distillation Algorithm
Algorithm 1 Iterative Self-distillation of 8-RoT
Require: teacher model T, critic model AsocialChmm
I: Ag f- sample AgocislChem
2: Asocialthem f- Asocuaktien Au
3: Do GENERATESDIVERSEE Ao)
4: Do FILTER(P, Do. Thresholdgau)
5: Distillaaae Fine-tune base model on Da
6: for
7: A: 4- sample AsociaChhem
8: AsucialChom AsueisiChem \
SelfDistill, 1), SELFDISTILLO, A., Asocatctem)
10: Drom + A
Accuracy
F1 Score
AUC PR Curve
11: return FILTER(p. Da U D2 U D2 Drem. Thresholdamse)
Val
0.88
0.93
0.98
12: procedure SELFDISTILL(auu. p)
13:
D
DIVERATTEIVERSECC A)
Test
0.86
0.92
0.98
14:
D < FILTER(p, D, Thresholdamm)
15:
+- Fine-tune return dace, D,
Dy
Table 5: Critic model metrics evaluation sets from gold auman-annotated data ($2)
16:
17:
procedure GENERATEDIVERSS(JAAA
18:
D 3--
19:
for in A do
20:
for p +. do
21:
b - (Top 10 beams from Hol for o.p)
Critic Precision Recall Curve
22:
Ds + 0
1.00
23:
for (a,p, ) in do
24:
if vé' E Do: MUTUALNLI(C,e)) then
0.98
25:
D3 + D. U ((a,p.c,r))
26:
1 DUD
0.96
27: return D
Precision
0.94
28:
procedure FILTER((. 1), Dy
29: for (a.p, C, r)) in 17 do
0.92
30:
if V/la, e) then D - Dy ((ap,,,r))
31:
0.90
32:
return D3
0.88
33:
procedure MUTUAL.NI.KCI. (2) return Entail(cl,, c2) A Entail(c2,
Classifier (AP 0.98)
34:
1.0
Recall
D
Model Training Details
Figure 8: PR Curve on Validation Set. Red lines display recall of 0.8 with high precision used to select threshold
D.1 Critic Model
We fine-tune the critic model from DeBERTa-Va V3- Large (He al., 2021) on the human critic gold data $2, attaching a 2-layer classificaton head with a hidden size 512 and GELU (Hendrycks and Gimpel, 2016) as the activation function.
Using the precision-recal. curve, we determine critic threshold of 0.8 for distillation, which achieves high precision and recall on the valida- tion set.

---

Original Situation: S(situation;
Update Scenario: S(update)
Explonation: $(explanationy
Update Scenario Questions
Q1. Does the update scenario convey meaning fluently and with good grammar, independent from the original situation?
fluent and well-formed.
Somewhat somewnat
well-formed,
errors.
is not Tluent and well-formed, with many errors.
Q2. Does the update scenario make the original situation more morally acceptable?
Yos sipniicatlyy It makes the original situation significantly more moraly acceptable.
slghdy makes the original situation slightly marally acceptable.
not the original situation more morally acosptable.
Explanation Questions
Q3. Does the explanation convey meaning fluently and with good grammac, Independent from the original situation update scenario?
is fluent and well-formed.
Semowhat somewnat
wel-formed, errors.
No is noc fluent and well formed, with many errors.
Q4. the explanation provide rationale how the update scenario makes the original situation more morally
acceptable?
Yau provides valid rationale with commonsense reasoning.
valid rationale, potentiall incomplete reasoning does not provide a valid retionale.
provides
No
Figure 7: The human evaluation template for evaluating or validating distilled generations from teacher and student models.

---

D.2 Student Model
The base model for distillation is Flan-T5-XL
(Chung et al., 2022), with 3 billion parameters. Student model prompts follow the format:
Attribute
Content
Action: (action).
Action
Not wanting to be friends with my ex
Modifier: more (ethical/unethical))
Judgment
ot'
And target generations follow this format:
Strengthen
My ex and I had really bad breakup and they now dating somcone new
Update: (context). Explanation: (rationale).
Context
It would be really awkward and uncomfortable to be friends with my ex, especially since
We fine-tune each student model for a fixed epochs using a maximum target length of 512, per- device batch size of 8, and learning rate of 5e 5 We use the AdamW optimizer, keeping other hy- perparameters at default. Each student model takes about 28 hours to fine-tune using 8 NVIDIA RTX A6000 GPUs.
Strengthen Rationale
they are dating someone new
Weaken Context
My ex and have kids together
Weaken
we' re not friends, it'll be harder co-parent our kids and it' be confusing for them
Rationale
Action
Letting your mom borrow your car
E
Data Analysis
Judgment
It's nice
E.1 Dataset Examples and Topics Analysis
Strengthen
Your mom is unable to afford a car and needs transportation for her job interview
Context
We show examples from the --RULES-OF-THUME in Table 6, and common topics of contexualizations in Table 7 and of rationales in Table 8.
Strengthen
demonstrates kindness and generosity towards your mother
Rationale
Weaken Context
Your mom driving under the influence of drugs alcohol
E.2 Toxicity Analysis
It increases the risk of her driving while impaired, which could put other people danger
Table 9 shows examples of 6-RUI.ES-FF-THUNE with high toxicity rated by the Perspective API. Es- sentially, all data with higher toxicity scores are cases where the base action itself is inappropriate. and the distilled contexts with the desired moral variance do not contribute significantly to the toxi- city of the total statement.
Weaken Rationale
Action
Flaking out on someone
Judgment
It's rude
Strengthen Context
The person has been acting in way that damaging to themselves or someone else
Strengthen Rationale
is act of self-preservation and ensuring that the person's safety is prioritized
E.3 Cultural Bias Analysis
Flaking out close friend who is going through difficult time and you know that they need your support
Weaken Context
As it's extremely challenging to systematically quantify cultural biases with state-of-the-ar tools such as hate speech detectors, we thus have probed the model qualitatively to gauge evidence of cul- tural biases inherent in the dataset/model. Table 10 Shows examples generated by the final distilled stu- dent model that potentially implies cultural biases. We can see that the model indeed comes back with different updates for the prompt "Not having free- dom of speech in [country]" )" for different countries. For some countries such as Japan, the United King- dom, and the United States, the generated weakener context is "in a workplace setting." Yet, for other countries such as China, India, Thailand, and Ko- rea, or Russia, the model comes back with different results, which might imply these countries have varying levels of human rights concerns. This ex- ample confirms our intuition that the student model might encode Western- centric biases.
Weaken Rationale
It can be seen as taking advantage of their vulnerability and disregarding their feelings
Action
Buying lottery tickets at the store
Judgment
It's common
Strengthen Context
The lottery tickets are bought in order support a charitable cause
It supports good cause and helps to raise money for a cause that can benefit those in need
Strengthen Rationale
Weaken Context
Buying lottery tickets at the store to cover up a theft
Weaken
is enabling illegal activity
Rationale
Table 6: Example data from R-ROT.

---

Count|
Topic
Count|
Topic
191
Pet
328
friends
162
project
258
students
121
gift
213
children
124
supervisor
199
taking advantage
143
public
121
family
111
doctor
144
commitment
90
siblings
124
partner
93
race
118
vulnerability
94
food
88
marriage
162
vulnerable
121
care
123
safety
88
interview
109
decision
134
protect
84
teacher
72
pet
95
minor
87
virus
90
ex
88
parent
116
committed
69
animals
86
friends
76
discrimination
98
years
71
elderly
86
stealing
91
respect
117
elderly
89
helping
93
abusive
157
mental health
99
relationship
88
informed
80
pandemic
96
decision
85
overwhelmed
77
minors
59
roommate
64
endangering
98
mental health
101
financial hardship
88
power
62
customer
98
child
75
violates trust
53
interview
60
motivated
78
neglecting
68
hard work
93
harm
60
safety
59
expensive
68
self-preservattion
50
gatherings
80
autonomy
55
married
59
relax
72
stranger
55
growth
65
workplace
110
resources
79
protect
49
pedestrians
71
advice
57
overspending
64
parents
105
support
51
formal
60
colleagues
60
consistently
61
exploitation
Table 7: Topics and their Appearance Counts for 10K Sampled Contextualizations from ò-ROT, supplement- ing Figure 3
Table 8: Common Topics and their Appearance Counts for 10K Sampled Rationales from 8- 8-ROT, supplement- ing Figure 3

---

Action
Moral Variance
Content
Telling someone to f*ck off
Weakener
in professional setting
Your friend has been being verbally abusive
Telling someone to f*ck off Strengthener
manipulative to you for long time
Engaging in bullshit
Strengthener
in order protect vulnerable person from harm
Touching your friend'ss butt
Weakener
while they are in the middle of private conversation
Table 9: Example data with high toxicity rated by the Perspective API.
Country
Context
China
In situation where the government using its power to oppress citizens
India
In country where people expressing their opinions controversial topics
Thailand
A country with a history of human rights abuses
Korea
A country with a history of human rights abuses
Russia
In country with a history of human rights abuses
Japan
In workplace setting
United Kingdom
In workplace setting
United States
In workplace setting
Table 10: Examples that potentially imply cultural biases generated by the final distilled student model, for the action "Not having freedom of speech in (country)" )" and the moral variance "weakener."


Reviews:
[{"review_id": "gtX6nRnLbI", "reviewer": "Reviewer_Uk2j", "summary": "", "questions": "In the Human Critic Gold Data section you used the Majority Rule to aggregate labels, Is there any reason? See:\nKruger, J., Endriss, U., Fernández, R., & Qing, C. (2014). Axiomatic analysis of aggregation methods for collective annotation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Uk2j for their endorsement of acceptance and constructive feedback! We are thrilled to see that they recognize our proposed task as “novel and challenging”, our contributed dataset as a “useful resource for future work”, our distillation process as “valuable”, and rate our paper as highly exciting! Below, we address specific comments they made:\n\n### **1. How hallucination from the teacher model will impact the student model?**\n\nThank you for calling our attention to the important point of hallucination of large language models. You’re correct that as our distillation framework ports knowledge from the large teacher model to smaller student models, it’s likely that the smaller models may also inherit undesirable behaviors. However, as we employ the critic model that’s trained with human annotations during the iterative self-distillation process, low-quality teacher model generations, including potential hallucinations such as logical inconsistencies and factual errors, are filtered out by the critic model, and thus might not present in the successive rounds of training of student models. As a result, we are able to increase the quality of intermediate student models via a richer set of training data from iterative self-distillation, and eventually get a final student model that produces much higher quality outputs based on human evaluation (**Validity**: 0.56 -> 0.71, **Defeasibility**: 0.37 -> 0.56, **Language Quality**: 0.98 -> 0.99). To explicitly show the effectiveness of the critic model in promoting student model quality, we included ablation experiments to show the contrast between training student models *with* vs. *without* filtering in Table 1 in the paper (**Validity**: 0.54 vs. 0.51, **Defeasibility**: 0.42 vs. 0.39). In the final version, we will clarify our careful data filtration process designed to counter the error propagation due to hallucination. Finally, we acknowledge that even with our filtering by the critic model, hallucinated outputs can still be produced by distilled models due to the limitations of critic models (see discussion in Section 8). We will stress in the final version that users must exercise discretion and care when applying our dataset to novel use cases.\n\n### **2. Clarification on the Human Critic Data Aggregation Strategy**\n\nThank you for providing this reference and calling out the possibility of different aggregation strategies! The work on bias-correcting rules is quite interesting. We used a majority rule for simplicity because there are only two classes (valid/invalid) and three annotators, and since previous work that similarly used crowdsourcing for commonsense-domain critic data also used a majority rule [1].\n\nHowever, when releasing the final dataset along with the camera-ready paper, we plan to include the individual gold annotations along with the aggregated results so that downstream users of the dataset can choose any preferred aggregation strategy. We agree that exploring additional aggregation strategies could be a promising direction to seek improvements to the distillation process in future work!\n\t\n- [1] Symbolic Knowledge Distillation: from General Language Models to Commonsense Models: https://arxiv.org/abs/2110.07178", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The defeasible moral reasoning task is novel and challenging.\n- Introduction of the δ-ROT dataset. The dataset size and quality metrics suggest δ-ROT will be a useful resource for future work.\n- The iterative distillation process is effective at improving model quality over raw GPT-3 generations. Using this approach to move from expensive API calls to smaller models is valuable.\n- This paper is detailed and easy to follow with clear logic. \n", "reasons_to_reject": "- We know that a frequent problem with using large language models is hallucinations, even if we carefully control the model. In the teacher -student model presented in this thesis, I think there would also be teacher-misleading student behavior, and I hope that a corresponding discussion will appear in the final version.", "questions_to_authors": "In the Human Critic Gold Data section you used the Majority Rule to aggregate labels, Is there any reason? See:\nKruger, J., Endriss, U., Fernández, R., & Qing, C. (2014). Axiomatic analysis of aggregation methods for collective annotation.", "paper_topic_and_main_contributions": "This paper introduces the task of defeasible moral reasoning, which involves providing contexts that alter the moral acceptability of an action, along with rationales that explain the reasoning. The main contributions are:\n\n- Formulation of the defeasible moral reasoning task.\n- Introduction of the δ-RULES-OF-THUMB (δ-ROT) dataset containing 1.2M contexts and rationales for 115K actions.\n- An iterative self-distillation methodology to create high-quality training data and models for the task.\n- Analysis of the dataset contents and model outputs, showing improved diversity and validity.", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dwy9plTY5T", "reviewer": "Reviewer_7Yd4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 7Yd4 for their endorsement of acceptance and valuable feedback! We are excited that they recognize our proposed task as “interesting” and “useful in daily applications” and recognize our proposed dataset and curation pipeline as having extensive details that could be useful for future related research! We address their specific comments on the following points:\n\n### **1. Writing Improvement**\n\nThank you for pointing this out; we will do another pass over the paper to revise the writing quality and flow for the camera-ready version and move the related work section to follow the introduction per your suggestion.\n\n### **2. Clarification of Paper Contribution and Data Analysis**\n\nWe agree that a more in-depth study of the dataset would be a fitting direction to continue this line of work. Recognizing the importance of it, we have done a preliminary dataset analysis, presented in Section 5.2 (see Fig. 4 & 5). We present insights over topics represented in δ-ROT surfaced through both modeling and qualitative analysis, categories of contextualizations and rationales broken into each moral variance, and the most common failure modes in the second-iteration distilled model. Our preliminary analysis reveals interesting insights on contextualizations that vary the degree of moral acceptability and underlying rationales that justify moral decisions. The results from this analysis include that vulnerability of the subject of an action is a critical weakening context and that (in)equity or (un)fairness in practice carry significant weight on moral implications; these observations also align with theories in cognitive science literature [1].\n\nIn addition, we would like to clarify that our contributions go beyond the dataset. Our work also includes the introduction of the defeasible moral reasoning task and a novel self-distillation pipeline, which we empirically show to improve generation quality and diversity through automated metrics and human evaluations of generations from our trained model.\n\n- [1] The Theory of Dyadic Morality: Reinventing Moral Judgment by Redefining Harm: https://journals.sagepub.com/doi/full/10.1177/1088868317698288\n\n### **3. Distinction from ProsocialDialog**\n\nThank you for pointing out this missing reference! We want to highlight some key differences between our work and ProsocialDialog. Our work introduces the task of **defeasible moral reasoning**, which reasons about grounded contexts that make a given action more or less morally acceptable, along with rationales based on implicit social and cultural norms. Yet, ProsocialDialog introduces a **multi-turn dialogue dataset** that teaches dialogue agents to respond to problematic content following social norms. Although the motivation of both datasets is guided by an understanding of social and cultural norms, they are significantly different in terms of both the goal (*ours*: defeasible moral reasoning vs. *ProsocialDialog*: dialogue safety) and the form (*ours*: open-text generation of defeasible contexts and rationales vs. *ProsocialDialog*: dialogues). We will add ProsocialDialog as a reference and clarify this distinction in the final draft!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting task definition and clear reason for it to be useful in daily applications\n- Large dataset introduced that can be used by other researchers working in a similar field.\n- Extensive detail about the proposed dataset and the curation pipeline.\n- Good diagrams with detailed captions.", "reasons_to_reject": "- A bit hard to follow. Writing can be improved. Also, can bring the related work section after introduction so that readers can get accustomed to the content of the paper.\n- The paper seems like a complete data paper. A preliminary study on the proposed dataset would have been highly beneficial.\n- The dataset introduced is extremely similar to the ProsocialDialogue dataset [1]. However, it is not cited anywhere.\n\n[1] Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., ... & Sap, M. (2022). Prosocialdialog: A prosocial backbone for conversational agents. arXiv preprint arXiv:2205.12688.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a new task of defeasible moral reasoning where an action is associated with its moral connotations based on the context. They curate a dataset to support this task with 1.2M instances. They use distinct models as teacher and student models to obtain the required annotations and perform human evaluation to validate them.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Kim, H., Yu, Y., Jiang, L., Lu, X., Khashabi, D., Kim, G., ... & Sap, M. (2022). Prosocialdialog: A prosocial backbone for conversational agents. arXiv preprint arXiv:2205.12688."}, "extra_text": {}}, {"review_id": "Sr9Kfbz7e9", "reviewer": "Reviewer_RfsH", "summary": "", "questions": "**A.** How did you ensure that the initial seed knowledge from GPT-3 was free from biases, and how did you mitigate the risk of these biases propagating through the iterative self-distillation process?\n\n**B.** Can you elaborate on the generalizability of your methods? Specifically, how do you envision the iterative self-distillation approach and the δ-RULES-OF-THUMB (δ-ROT) dataset being adapted for other NLP tasks or diverse cultural contexts?\n\n**C.** Given the sensitivity of moral reasoning and its cultural nuances, how did you account for potential cultural biases in the δ-ROT dataset? Were there any measures taken to ensure a diverse representation of moral perspectives?\n\n**D.** What are the potential real-world applications you envision for the models trained on the δ-ROT dataset, and how do you plan to address the ethical implications associated with deploying such models?\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer RfsH for their constructive suggestions! We are excited to see that they recognize our proposed task as “novel” and “shedding light on how context influences moral judgments in NLP” and recognize the value of our dataset and distillation pipeline! We address all their suggestions on the following points below:\n\n### **1. Clarification of Over-reliance on Iterative Self-distillation and Method Generalizability**\n\nThank you for raising the important question of whether our proposed iterative self-distillation method can be generalizable beyond the defeasible moral reasoning task and whether errors/biases from the teacher model will compound into student models.\n\nIn a broader picture, our novel approach of iterative self-distillation is rooted in the rich line of research of symbolic knowledge distillation (i.e., distilling knowledge from teacher models to benefit the downstream student models), where **different variations of the distillation pipeline have been proven to be more effective than human annotations in a wide net of applications.** For example, commonsense knowledge [1], generic reasoning [2], chain-of-thought reasoning steps [3], summarization [4,5], paraphrasing [5], and dialogues [6]. Thus, we are confident that our iterative self-distillation approach has the potential to be highly flexible and adaptable to tasks beyond defeasible moral reasoning itself.\n\n> **The Iterative Self-distillation Method is Generalizable**\n\nZooming into our iterative self-distillation approach, as it requires minimal task-specific constraints, it’s expected to be highly generalizable by nature. The approach's success comes from **three** main ingredients detailed below, and  **none of them are specifically bound to defeasible moral reasoning and thus could be generalizable to other reasonable, well-defined, open-text generative tasks.**\n\n(1) **An initial set of seed knowledge** that could be sourced using either automatic data generation or from existing datasets. In our case, we carefully prompted GPT-3 to generate the initial knowledge set. Still, we envision other variations of LLMs that can produce reasonable quality knowledge (e.g., Llama2) could also serve the purpose here.\n\n(2) **A critic model** that is trained on human annotations to mimic human ratings of the dataset's quality. Technically, for all tasks humans could evaluate, we can source human annotation data to train the critic model. Similar critic models have been proven to be effective in different applications in symbolic knowledge distillation research, such as commonsense knowledge [1] and generic reasoning knowledge [2].\n\n(3) **Iterative self-distillation and filtering** that amplifies desirable characteristics of the trained model. Our approach shows key advantages compared to previous symbolic knowledge distillation work. First, our approach minimizes the reliance on expensive API calls from closed-sourced models like GPT-3 as opposed to sourcing all data from GPT-3 [1,3]. Second, we use a simple over-generating and filtering pipeline instead of evoking complicated/slow constrained decodings during self-distillation [2]. We believe our simple but effective techniques could further lift the burden of operationalizing different tasks.\n\n> **The Critic Model Promotes Validity/Correctness of Knowledge Ported from the Teacher to the Student Models**\n\nAdmittedly, the data quality from the teacher model is mediocre as-is without any treatment and thus will not serve as a good source for student models to train on (see Table 2 in the paper). However, with the careful screening of the critic filter and the NLI filter, we can increase the quality of intermediate student models via a richer set of training data from iterative self-distillation, and eventually get a final student model that produces much higher quality outputs rated by human evaluation (**Validity**: 0.56 -> 0.71, **Defeasibility**: 0.37 -> 0.56, **Language Quality**: 0.98 -> 0.99). To explicitly show the effectiveness of the critic model in promoting student model quality, we included ablation experiments to show the stark contrast between training student models for one iteration *with* vs. *without* filtering in Table 1 in the paper (**Validity**: 0.54 vs. 0.51, **Defeasibility**: 0.42 vs. 0.39). In the final version, we will clarify our careful data filtration process designed to counter the error propagation. Finally, we acknowledge that even with our filtering by the critic model, hallucinated outputs can still be produced by distilled models due to the limitations of critic models (see discussion in Section 8). We will stress in the final version that users must exercise discretion and care when applying our dataset to novel use cases.\n\n- [1] Symbolic Knowledge Distillation: from General Language Models to Commonsense Models: https://arxiv.org/abs/2110.07178 \n- [2] I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation: https://arxiv.org/abs/2212.09246 \n- [3] Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step: https://arxiv.org/abs/2306.14050\n- [4] Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation: https://arxiv.org/abs/2210.13800\n- [5] Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing: https://arxiv.org/abs/2305.16635 \n- [6] SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization: https://arxiv.org/abs/2212.10465\n\n### **2. Dataset Size and Quality**\n\nWe appreciate the concern about dataset quality—high-quality data is indeed important, and increasing dataset size at the expense of quality would arguably decrease the value of the dataset. To counter this trend, to produce δ-ROT, we set a high critic threshold of 0.96 to maintain high quality in the data (Section 3.3). This threshold removes 73% of the candidate generations to produce the restricted set of generations in δ-ROT. Human annotators deem 85.9% of the contextualizations and 99.8% of the rationales as high-quality (Section 2). This is on par with similar datasets in the field [1], and using the critic model to enforce quality in the data is much more efficient than human annotations, which could be very prohibitive when scaling up. Thus, it’s fair to say that our dataset is both large-scale and high-quality.\n\nWe also note that scale is an important factor in the value of the dataset to train models. In our own data, we can compare SelfDistill1 to the ablation with Top 1 Only, which has high data quality yet smaller size due to the greedy sampling. In this ablation, we observe that SelfDistill1, with a larger scale training set, performs better on the number of human-judged valid generations, even when ignoring the uniqueness constraint (Table 2). In other literature, it is also observed that language model performance scales with dataset size according to the LM scaling law, demonstrating that both dataset scale and quality are critical to producing high-quality models [2].\n\nIt’s also a valid concern to ensure that the dataset reflects real-world moral dilemmas. Hence, we source the actions in δ-ROT from Social Chemistry 101, which is filled with a “rich spectrum of real-life situations” that carry sociomoral norms and judgments [3]. By building our dataset on top of these real-world actions, δ-ROT is representative of moral contextualizations that people are likely to encounter in the real world.\n\n- [1] Symbolic Knowledge Distillation: from General Language Models to Commonsense Models: https://arxiv.org/abs/2110.07178\n- [2] Scaling Laws for Neural Language Models: https://arxiv.org/abs/2001.08361\n- [3] Social Chemistry 101: Learning to Reason about Social and Moral Norms: https://arxiv.org/abs/2011.00620\n\n### **3. Discussion on Potential Biases**\n\nThank you for calling our attention to discuss the important point of potential biases of our dataset. We sourced the seed actions from Social Chemistry 101 [1] that concern mostly everyday situations, such as pet, project, gift, supervisor, public, and siblings (see details of the topic analysis of our dataset in Table 6/7 in the appendix). Thus, we find them less prone to biases/toxicity to begin with. In addition, due to the careful filtering with the critic model, and the iteratively refined self-distillation process, we expect most of the low-quality (including potentially biased data points) are already filtered out from the final dataset. This is also why human annotations deem our final distilled dataset high quality overall.\n\nHowever, we acknowledge the importance of examining the potential biases, especially considering the sensitive nature of morality and the potentially limited scope of the critic model. Therefore, we run additional experiments of toxicity analysis with the Perspective API on a subset of 40K data points of our final dataset. Our results show that the average toxicity score is **0.09**, indicating very low toxicity overall. In a qualitative analysis of the data rows with higher toxicity scores (with a max of *0.83*), essentially all of them are cases where the base action itself is inappropriate/explicit, and the distilled contexts that have the desired moral variance do not contribute significantly to the toxicity of the total statement. *We provide some examples with high toxicity scores in the Table below.* While *no* existing toxicity detection can accurately measure all potential biases, we will incorporate Perspective API analysis and thoroughly discuss potential limitations of this toxicity analysis.\n\n| **Action**                      | **Moral Variance** | **Context**                                                                         |\n|---------------------------------|--------------------|-------------------------------------------------------------------------------------|\n| Telling someone to f*ck off     | Weakener           | in a professional setting                                                           |\n| Engaging in bullshit            | Strengthener       | in order to protect a vulnerable person from harm                                   |\n| Touching your friend's butt     | Weakener           | while they are in the middle of a private conversation                              |\n| Telling your friend to f*ck off | Strengthener       | Your friend has been being verbally abusive and manipulative to you for a long time |\n\nFinally, we want to highlight that our dataset should **never** be used as prescriptive advice to humans. Our work aims to promote the consideration of defeasible contexts to state underspecified contexts in moral reasoning explicitly. This opens future avenues for more nuanced interpretation/consideration in computational morality research. We will add this important point to the final paper as well.\n\n- [1] Social Chemistry 101: Learning to Reason about Social and Moral Norms: https://arxiv.org/abs/2011.00620\n\n### **4. Discussion about Accommodating Diverse Perspectives and Cultural Nuances**\n\nWe appreciate the reviewer for raising the important questions of how we account for diverse representations of moral perspectives and potential cultural biases, which we underspecified in our original draft. We will discuss this point in the following response and will add them to the camera ready.\n\n> **Diversity**\n\nWe readily agree with the reviewer that “moral reasoning is a complex and culturally sensitive area” and thus, it is important to accommodate various different but equally valid opinions for making judgments. Our work is precisely motivated by this observation: *oftentimes, the failure to accommodate variations of moral judgments is due to the underspecified and unspoken assumptions or contexts of the given default action.* For example, the default/average judgment of the action “talking loudly” will be shifted in various directions with more detailed context specifications (e.g., if you try to warn people during a fire, in a public library). **To this goal, we introduced the defeasible moral reasoning task to explicitly gauge various rich defeasible contextualizations that underpin alternative perspectives and shifts of moral judgments in nuanced but essential ways.**\n\nWith this goal in mind, we took precautions to accommodate/promote diversity of representations during data distillation. **First**, we don’t presume any default moral judgment given an action (i.e., if the action is *absolutely* good or bad) but rather to understand various contexts that shift the relative moral implications of a situation differently (i.e., whether a context makes the given action *MORE* or *LESS* morally acceptable). This assumption is helpful to not force default/average interpretation or frame onto an action, but to attempt to justify the rich variability of its judgments based on alternative contexts and interpretations. **Second**, it is important to note that an action could be contextualized in many ways. Thus, we deliberately promote the distillation of multiple contextualizations to account for diverse ways of defeating moral judgments. As a result, we improved the diversity scores significantly throughout the self-distillation process (4.78 -> 5.69, see Table 2 in the paper). Therefore, although we do not directly engage different moral perspectives during distillation, such as philosophical moral theories, we contribute to understanding various perspectives that moral judgments could be shifted due to variable contexts, complementing the existing landscape of computational morality research.\n\n> **Culture Sensitivity and Nuances**\n\nAmong all perspectives that shape people’s morality differently, one crucial dimension of variation is people’s cultural background. As we responded to ethical implications and general biases, our final dataset exhibited *very low* indications of overall toxicity. However, it is hard to conclude that our dataset/model is immune from cultural biases, as even the best hate speech detector to date might fail to capture this more nuanced type of bias. In fact, previous studies have pointed out that cultural bias is a pervasive phenomenon across many NLP models (e.g., GPT-3/3.5/4) and tasks (e.g., hate speech detection with Perspective API, RewireAPI, HateRoberta) [7]. Admittedly, as our dataset and student models are distilled from GPT-3, which is shown to present *Western-centric* perspectives, it is highly likely that our dataset and models inherit this cultural bias as well. \n\nAs it’s extremely challenging to systematically quantify cultural biases with state-of-the-art tools such as hate speech detectors, we thus have probed the model qualitatively to gauge evidence of cultural biases inherent in the dataset/model. We provide some interesting examples in the Table below. We can see that the model indeed comes back with different updates for the prompt “Not having freedom of speech in {country}” for different countries. For some countries such as Japan, the United Kingdom, and the United States, the generated weakener context is “in a workplace setting.” Yet, for other countries such as China, India, Thailand, and Korea, or Russia, the model comes back with different results, which might imply these countries have varying levels of human rights concerns. This example confirms our intuition that the student model might encode Western-centric biases.\n\n**Action Template**: “Not having freedom of speech in {country}”\n\n**Moral Variance**: “weakener”\n\n| **Country**    | **Context**                                                                     |\n|----------------|---------------------------------------------------------------------------------|\n| China          | In a situation where the government is using its power to oppress citizens      |\n| India          | In a country where people are expressing their opinions on controversial topics |\n| Thailand       | A country with a history of human rights abuses                                 |\n| Korea          | A country with a history of human rights abuses                                 |\n| Russia         | In a country with a history of human rights abuses                              |\n| Japan          | In a workplace setting                                                          |\n| United Kingdom | In a workplace setting                                                          |\n| United States  | In a workplace setting                                                          |\n\nWe will add the full probing results to the paper and an extensive discussion of the potential Western-centric cultural biases due to distillation to the final paper. Further, we will deepen the discussion of future directions to lay out concrete road maps of future research on enriching cultural representations in computational moral reasoning research. Finally, we will make our data and code completely open source to allow for scrutiny and improvement by the research community.\n\nFinally, it's worth noting that discussions on the feasibility of instilling morally-informed AI are often marked by deep-seated disagreements concerning the influence of culture, personal experiences, and specific circumstances on people's moral perspectives. These disparities have led some to assert that endeavoring to systematically imbue machines with human moral values is futile. However, evidence spanning various disciplines, including comparative law [1], anthropology [2], psychology [3,4,5], and philosophy [6], challenges this notion by suggesting that fundamental similarities underlie human moral systems, potentially outweighing their divergences. **In light of this, it’s equally, if not more, important to start investigating the possibility of teaching models to capture matters of widespread moral consensus under defeasible variations.** In particular, such differences do not necessarily preclude the need for investigating cases where people tend to agree. To this end, we will add the discussion of the importance of moral consensus in addition to moral disagreement into the paper.\n\n- [1] John Mikhail. “Is the Prohibition of Homicide Universal-Evidence from Comparative Criminal Law”. In: Brook. L. Rev. 75 (2009), p. 497.\n- [2] Oliver Scott Curry, Daniel Austin Mullins, and Harvey Whitehouse. “Is it good to cooperate? Testing the theory of morality-as-cooperation in 60 societies”. In: Current Anthropology 60.1 (2019), pp. 47–69.\n- [3] PR Blake et al. “The ontogeny of fairness in seven societies”. In: Nature 528.7581 (2015), pp. 258–261.\n- [4] Edmond Awad et al. “Universals and variations in moral decisions made in 42 countries by 70,000 participants”. In: Proceedings of the National Academy of Sciences 117.5 (2020), pp. 2332–2337.\n- [5] H Clark Barrett and Rebecca R Saxe. “Are some cultures more mind-minded in their moral judgements than others?” In: Philosophical Transactions of the Royal Society B 376.1838 (2021), p. 20200288.\n- [6] Bernard Gert. “Common Morality and Computing”. In: Ethics and Information Technology 1.1 (1999), pp. 53–60. doi: 10.1023/a:1010026827934.\n- [7] Sebastin Santy et al. “NLPositionality: Characterizing Design Biases of Datasets and Models”: https://arxiv.org/abs/2306.01943\n\n### **5. Comparative Analysis**\n\nOur current work tackles the novel and unique task of defeasible moral reasoning. To the best of our knowledge, no existing work tackles this exact task, and thus, it's challenging to find other comparative task-specific baselines. The closest related works that we had discussed in the paper introduction are [1] (on clarification question generation for moral situations) and [2] (simple situational norms), but none of these provide outcome models that directly produce defeasible moral contexts and rationales compatible with our task. \n\nInstead, we conducted new experiments to investigate the zero-shot performance of some of the most powerful general-purpose LLMs of varied sizes with 1000 sampled examples from our test set. We use the same instructions as we used to distill the seed knowledge from GPT-3 to prompt these baselines in a zero-shot manner (see Section A in the paper's appendix). We evaluate them using the comparable automatic evaluation metric via the critic model (i.e., the same metrics used in Table 1 in the paper). Such automatic evaluation scores are shown to be highly correlated with human evaluation results in our paper. **Our results show that despite some of the zero-shot models being orders of magnitude larger than our final student model (e.g., 175B vs. 3B), our model outperforms them all, proving the effectiveness of our proposed approach.** It's also worth noting that our models will also be open-sourced, providing further accessibility advantages compared to some of these massive and opaque closed-sourced LLMs. We will add a detailed discussion of these new baselines to the final paper!\n\n| **Model**          | **Percentage of Valid Contexts**     | **Avg. Critic Score**     |\n|--------------------|--------------|--------------|\n| *(NEW)* Falcon-7B-Instruct | 0.39\t\t | 0.54 \t\t |\n| GPT-3              | 0.53         | 0.69         |\n| *(NEW)* GPT-3.5 (ChatGPT)  | 0.71         | 0.77         |\n| *(NEW)* GPT-4              | 0.77         | 0.82         |\n| **Ours**           | **0.86**     | **0.88**     |\n\n- [1] ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations: https://arxiv.org/abs/2212.10409 \n- [2] NormBank: A Knowledge Bank of Situational Social Norms: https://arxiv.org/abs/2305.17008\n\n### **6. Potential Real-world Application and Strategies for Addressing Ethical Implications**\n> **Potential Real-world Applications**\n\nHere’s a list of potential use cases of our task/dataset. This is by no means a complete list, but we hope it could provide a fuller picture of the downstream potential of our work!\n- Incorporating the nuanced understanding of defeasible context into reasoning about social norms and values in interactive narrative games, e.g., [1,2,3]\n- Using the richer set of defeasible contexts to reward more interesting clarification question generation for disambiguating social and moral situations, e.g., [4]\n- Using the contextualized understanding of ethical situations to enhance prosocial conversation in dialogue applications, e.g., [5]\n\nIn the broader scope, it’s critical to understand that the defeasible, contextualized nuances of moral reasoning itself is largely understudied in the current landscape of computational morality research, and thus our task and dataset inspires further research in understanding the interleaving complexity of moral reasoning. Finally, as we mentioned in the method generalization response, we believe our proposed iterative self-distillation method serves as a simple but effective method for other tasks.\n\n> **Strategies for Addressing Ethical Implications**\n\nWe will publicly release our dataset and final student model to promote open-source research but for ​​gated research purposes only. To mitigate risks of misuse, we will inform all users about potential ethical implications and risks of the resource and require them to complete a data/model user agreement form to acknowledge their consent to proper usage of the resource. This will also help us track how our resource is repurposed for downstream applications. We will also add explicit discussions to encourage future research to tackle limitations of our current work, such as potential cultural biases. We'll add the above discussion to the final paper.\n\n- [1] What Would Jiminy Cricket Do? Towards Agents That Behave Morally: https://arxiv.org/abs/2110.13136\n- [2] Aligning to Social Norms and Values in Interactive Narratives: https://arxiv.org/abs/2205.01975\n- [3] Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark: https://arxiv.org/abs/2304.03279\n- [4] ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations: https://arxiv.org/abs/2212.10409 \n- [5] ProsocialDialog: A Prosocial Backbone for Conversational Agents: https://arxiv.org/abs/2205.12688\n\n### **7. More Extensive Discussion of the Literature on Morality and LLMs**\n\nThank you for pointing out that we should expand our discussion on computational morality and LLMs! We originally discussed computational morality in the related work section, and we appreciate the additional literature you suggested. We’ll include them all in the camera-ready to address related work more comprehensively!\n\n### **8. Typos and Presentation Improvements**\nThank you for pointing out the typos; we’ll fix them all in the camera-ready draft!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a novel \"defeasible moral reasoning\" task, shedding light on how context influences moral judgments in NLP. Its main strength lies in the creation of the δ-RULES-OF-THUMB (δ-ROT) dataset, a comprehensive resource with over a million entries. The iterative self-distillation method used to generate this dataset is both interesting and scalable, with potential applications beyond this study. ", "reasons_to_reject": "\n1. **Over-reliance on iterative self-distillation**: The paper heavily depends on the iterative self-distillation method. If there are inherent biases or errors in the initial seed knowledge from GPT-3, they might propagate through the iterations, affecting the final dataset's quality. \n\n2. **Dataset size over quality**: While the δ-RULES-OF-THUMB (δ-ROT) dataset is extensive, but it is unclear why is this better than a highly curated small dataset where examples are more representative of real-world moral dilemmas.\n\n2. **Generalizability concerns**: The paper's focus on the specific task of \"defeasible moral reasoning\" might limit its generalizability. It's unclear how the proposed methods would perform on other NLP tasks or in different cultural and moral contexts.\n\n3. **Potential ethical implications**: Moral reasoning is a complex and culturally sensitive area. There's a risk that the models and datasets might inadvertently perpetuate biases or oversimplify intricate moral dilemmas, leading to potential misuse in real-world applications.\n\n4. **Lack of comparative analysis**: The paper might benefit from a more in-depth comparison with existing works or methods. Without this, it's challenging to gauge the true novelty or superiority of the proposed approach.\n\n5. **Disregarding existing literature on morality and LLMs**: The paper fails to address the important and growing literature on the moral and social choices of LLMs. Please see missing references.\n\n", "questions_to_authors": "**A.** How did you ensure that the initial seed knowledge from GPT-3 was free from biases, and how did you mitigate the risk of these biases propagating through the iterative self-distillation process?\n\n**B.** Can you elaborate on the generalizability of your methods? Specifically, how do you envision the iterative self-distillation approach and the δ-RULES-OF-THUMB (δ-ROT) dataset being adapted for other NLP tasks or diverse cultural contexts?\n\n**C.** Given the sensitivity of moral reasoning and its cultural nuances, how did you account for potential cultural biases in the δ-ROT dataset? Were there any measures taken to ensure a diverse representation of moral perspectives?\n\n**D.** What are the potential real-world applications you envision for the models trained on the δ-ROT dataset, and how do you plan to address the ethical implications associated with deploying such models?\n\n", "paper_topic_and_main_contributions": "The paper addresses the challenge of understanding how specific contexts influence moral judgments. Recognizing that moral decisions are often shaped by the nuances of their surrounding contexts, the authors introduce a task called \"defeasible moral reasoning\" to provide grounded contexts and rationales that determine the moral acceptability of actions. To gather data for this task, they employ an iterative self-distillation method, resulting in a high-quality dataset named δ-RULES-OF-THUMB (δ-ROT). This dataset, containing over a million entries, aids in training a model that captures the intricacies of human moral judgment in diverse real-life scenarios.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 5 - “Example *of* rich contextualizations per category.”\n\nLine 619 - comprising *of* over 1.2M combined \n"}, "extra_text": {"Missing_References": "**1.** Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. “Does Moral Code Have a Moral Code? Probing Delphi’s Moral Philosophy\n\n**2.** M. Abdulhai, S. Levine, and N. Jaques. “Moral Foundations of Large Language Models”. In: AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI (2022).\n\n**3.** Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. “The Political Ideology of Conversational AI: Converging Evidence on ChatGPT’s Pro-Environmental, Left-Libertarian Orientation”\n\n**4.** Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. “Whose Opinions Do Language Models Reflect?”\n\n**5.** Julian Coda-Forno, Kristin Witte, Akshay K Jagadish, Marcel Binz, Zeynep Akata, and Eric Schulz. “Inducing Anxiety in Large Language Models Increases Exploration and Bias”\n\n"}}]

Author rebuttal:

Meta-review: The paper presents a new task, defeasible moral reasoning, to provide grounded contexts that make an action more or less morally acceptable, along with commonsense rationales that justify the reasoning. 

The authors introduced the δ-RULES-OF-THUMB (δ-ROT) dataset of 1.2M entries of combined context and rationales for 115K defeasible moral actions. δ-ROT is created through an iterative approach, which includes self-distillation, targeted filtering using a critic model trained with human judgment and (3) self-imitation learning.


Pros:

- The δ-ROT dataset is an important contribution with over 1.2M entries.  It also contains rationales that *explain the moral judgment of an action* that is useful for the community. 
- The method to generate the data is not novel but effective and can be used for other tasks. 
- The qualitative analysis of contextualised categories per moral variance is interesting.

Cons: 
- Strong baselines such as finetuned open-sourced models are missing. 

In the result section, there are too many **bold** words. Try to use italics instead. 

Finally, The AreaChair appreciate the authors' response to clarify the doubts of the reviewers.
Decision: Accept-Findings