ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Year: 2023

Abstract:
Context is everything, even in commonsense moral reasoning. Changing contexts can flip the moral judgment of an action; Lying to a friend is wrong in general, but may be morally acceptable if it is intended to protect their life. We present CLARIFYDELPHI, an interactive system that learns to ask clarification questions (e.g., “why did you lie to your friend?”) in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, informative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and computational investigations of moral judgments.

heading: 1 Introduction
text: Commonsense moral reasoning of social situations and actions depends squarely on their context. Offering someone a cup of coffee is generally considered appropriate. If offered to a work colleague, it may even be viewed as a courteous gesture. However, offering coffee to a toddler would be deemed morally irresponsible.
Delphi (Jiang et al., 2022), a recently proposed commonsense moral reasoning model, generates moral judgments for simple actions described in text. However, Delphi’s judgments are made in isolation, without any knowledge of surrounding context. Grounding moral reasoning in context is crucial (Talat et al., 2022). How can moral reason-
ers elicit missing salient context? A natural way to do so is by asking clarification questions.
We present CLARIFYDELPHI, an interactive system that learns to ask questions to elicit salient context. Prior research in cognitive science shows that human reasoning exhibits the flexibility not only to articulate where a certain moral rule should hold, but also to imagine valid exceptions where the rule can be bent or defeated based on the demands of the context (Kwon et al., 2022; Levine et al., 2020; Awad et al., 2022).
We present a first step toward computationally exploring and discovering these defeasible contexts which can potentially flip the moral judgement of a situation. Given a situation and its default judgment (e.g., it is nice to offer a cup of coffee to someone), defeasible contexts can strengthen (e.g., offering it to a colleague) or weaken (e.g., giving it
11253
to a toddler) the judgment (Rudinger et al., 2020; Madaan et al., 2021; Allaway et al., 2022). We aim to generate questions whose answers might uncover missing context for making better-informed moral judgments, and we propose to do so in a conversational setting between a user and CLARIFYDELPHI.
Our method for clarification question generation is based on reinforcement learning. Using Proximal Policy Optimization (PPO; Schulman et al. 2017; Ouyang et al. 2022) we optimize for generating questions that invoke responses that provide morally salient contexts. CLARIFYDELPHI “imagines" answers to a generated question, using a trained answer generation model. A reward is calculated by comparing the probability distributions Delphi assigns to the imagined answers. Fig. 1 provides an overview of CLARIFYDELPHI.
The intuition behind our approach is that questions that lead to maximally divergent answers (e.g., “Who did you offer it to?”) are also those that elicit most morally salient contexts and therefore are more consequential to the situation. These morally consequential questions surface latent ambiguities that may directly affect the moral decision process. Questions with little divergence in its imagined answers (e.g., “When did you offer it?”) have little to offer in terms of resolving contextual moral ambiguities.
Our results show that our approach outperforms other strong clarification question generation baselines; its generated questions lead to consequential answers. We additionally quantify how much supervised clarification question training data is needed for a good initial policy. Lastly we show that questions help with generating defeasible updates.
Our contributions are as follows. We introduce the task of clarification question generation for social and moral situations. For this task we propose an RL based approach, defining defeasibility as a new type of relevance for clarification questions. We publicly release δ-CLARIFY, a dataset of 33k crowdsourced clarification questions, and δ-CLARIFY silver containing generated questions conditioned on a defeasible inference dataset. We also release trained models with their code.1

heading: 2 Problem Setup
text: Given a situation, such as lie to my friend, we aim to generate question(s) that are the most relevant for uncovering the most consequential context with respect to making a social or moral judgement. While situations could evoke a multitude of potential questions, the following work is concerned with predicting questions whose answers are likely to be consequential, i.e. answers that could function as either weakeners or strengtheners of the default judgement. The terms weakener and strengthener come from the concept of defeasible inference (Rudinger et al., 2020), which defines a way of reasoning that takes into consideration (new) evidence which could either support (e.g. strengthen) or cancel/weaken an initial inference.
Formally, the task is to predict a question q given a base situation s. The base situation has a default moral judgement j ∈ {bad, ok, good}. For every input tuple of (si, qi, ji) there is a hypothetical set of strengthening answers AS and weakening answers AW . Adding the additional information obtained from any qi and corresponding answer ai to
1Data and code are available at: https://github. com/allenai/clarifydelphi.
Algorithm 1 Training CLARIFYDELPHI Input initial policy model θ0, initial value model ϕ0, Delphi ψDelphi Dδ-CLARIFY ← Get dataset of clarification questions. θQ ← Fine-tune θ0 with Eqn 1 from Dδ-CLARIFY. ▷ Sec. 3.1 Dδ-CLARIFYsilver ← Get silver dataset of defeasible answers to questions. θA ← Fine-tune lm with Eqn 2 from Dδ-CLARIFY. ▷ Sec. 3.2 θCLARIFYDELPHI ← REINFORCEDLEARNING(SSocialChem, θQ, θA, ϕ0, ψDelphi) ▷ Sec. 3.4
procedure REINFORCEDLEARNING(SSocialChem, θQ, θA, ϕ, ψDelphi)
θQold ← θQ, ϕold ← ϕ for iterations = 1, 2, . . . do
Sample a minibatch of situations s from SSocialChem.
for step = 1, 2, . . . , s do Calculate r using θA and ψDelphi with Eqn 3. Compute lossPPO on the minibatch with Eqn 6. Optimize θ and ϕ with LPPO for one step.
θQold ← θQ, ϕold ← ϕ return θQ
Output θCLARIFYDELPHI
the base situation si results in an updated situation sui, with an updated judgement jui.

heading: 3 CLARIFYDELPHI: A Reinforced Clarification Question Generator
text: The CLARIFYDELPHI approach is based on reinforcement learning. Algorithm 1 gives an overview of the training process. As a first step, before performing reinforcement learning, we obtain a question generation model θQ and an answer generation model θA, which we both train on data that we curated, described in the later Sec. 4. The question generation model predicts the clarification questions and the answer generation model provides (defeasible) answers to the generated questions. By using these two models in addition to Delphi (ψDelphi) for calculating the rewards, we do not require any supervised data during RL training.
We consider question generation conditioned on a given situation a sequential decision making process over the natural language vocabulary space, where the generated question q with T tokens has an episode of length T . At step t ∈ [1, T ], the state st = (s, q<t) is the combination of the given situation and the question decoded up to the (t− 1)-th token; the action ct = qt would be the t-th token to decode. The question generation model, θQ(qt|q, q<t; θ), is the policy model that we optimize. We define a reward function r(s, q, aw, as)
that characterizes the divergence of answers from θA conditioned on generated question q and discuss the definition of this reward function in §3.3.

heading: 3.1 Supervised Question Generation
text: The first subcomponent is a basic question generation system θQ that outputs a question q conditioned on a situation s. It is used as the initial policy model during RL training.
q̂ = argmax q
P (q|s) (1)

heading: 3.2 Defeasible Answer Simulation
text: For each generated question q, we need to generate a weakening answer aw and a strengthening answer as in order to calculate the reward r (Formula 3). For the defeasible answer generation system θA, we take as input a situation si, the generated question qi (§3.1), and an update type u ∈ {weakener, strengthener} to predict a weakener-strengthener answer pair aw and as:
a = argmax a
P (a|s, q, u) (2)
An example of an instantiated input/output:
Input It’s bad to be a snitch, TYPE: Weakener, Q.: Why would being a snitch be beneficial? Output doing so would save someones life.
The crucial element in the input is the update type, as it allows to generate two types of answers for the same s and q. When computing the reward during training, for each question, we filter out all its generated answers which either contradict or are entailed (i.e. no new information) by the given situation, using an off-the-shelf NLI model.

heading: 3.3 Reward
text: As a reward for generating a question, we aim to quantify how well the generated questions are able to elicit consequential answers. For this purpose we query Delphi (Jiang et al., 2022) for feedback, using situations updated with answers.
We optimize for questions that lead to maximally divergent answers by defining a reward function which uses the JS-Divergence, between the Delphi probability distribution of the weakener updated situation and the strengthener updated situation:
r(s, q, aw, as) = JSD(Pjw||Pjs) (3)
Sentence Fusion To create an updated situation that sounds natural and can be used to query Delphi, the situation s, question qi and answer (both aw and as separately) have to be fused together into sui. For example: Situation refraining from doing something bad Question When do you do something bad? Answer when I’m angry Fusion: refraining from doing something bad when you’re angry. We train a model to distill fusion in-context examples obtained from GPT-3 (text-curie-001).
Delphi for Feedback Delphi is then queried with the updated situation sui for a judgement, leveraging the probability distribution that Delphi provides over three classes: j ∈ {bad, ok, good}. The probability scores are the probabilities of the special T5 tokens representing each of the three classes, normalized by their sum.
j = argmaxj q
P (j|s) (4)
JS-Divergence We calculate the Jensen-Shannon divergence between the Delphi probability distributions jw and js obtained from two updated situations originating from defeasible answers to q.
Reward normalization We normalize the reward during training as follows:
r(x, k)← r(x, k)− µ0 σ0 . (5)
The µ0 and σ0 of rewards are obtained before training begins, by generating a question and calculating its rewards for all s in the training data.

heading: 3.4 Proximal Policy Optimization (PPO)
text: We maximize the reward using Proximal Policy Optimization (PPO) (Schulman et al., 2017) as our RL algorithm, which previous works have shown to be suitable for NLG tasks (Liu et al., 2022b; Ramamurthy et al., 2022). Our implementation of PPO is an adaptions of Ouyang et al. (2022)’s, which includes a KL penalty between the initial policy model θQold and the updated policy θQ. In addition to the policy model, PPO employs a value model (parameterized by ϕ) to estimate the value function for states with incomplete decoded text, i.e. V (st;ϕ) for any t. PPO’s loss consists of a value model (ϕ) loss and the policy loss, which is
jointly minimized:
lossPPO(θ, ϕ) = α · lossValue(ϕ) + lossPolicy(θ) (6)

heading: 4 δ-CLARIFY: a Dataset of Clarification Question
text: We require data for various components of our CLARIFYDELPHI model: The policy needs bootstrapping from a clarification question dataset and the answer generation model needs data to learn to generate defeasible answers to questions. To the best of our knowledge no such datasets exist. We therefore collect a crowdsourced dataset of clarification question for social and moral situation and a silver dataset of defeasible QAs to train θQ .
The situations are sampled from SOCIAL-CHEM101 (Forbes et al., 2020) and the COMMONSENSE NORM BANK (Jiang et al., 2022). We call our dataset δ-CLARIFY and it consists of crowdsourced questions, enriched with questions generated by GPT-3 (Brown et al., 2020).
δ-CLARIFYgold: We crowdsource clarification questions by showing annotators a situation and asking them to write a clarification question they would ask an imagined colleague requesting advice on the situation. Each of the 6425 situations is presented to 5 annotators, resulting in 5 questions per situation (500 situations are used for dev and test respectively). Details of the annotation are found in Appendix A.1.
δ-CLARIFYsilver: The δ-SOCIAL part of the defeasible inference dataset (Rudinger et al., 2020) consists of statements that express default judgments over situations (It is good to protect your kids) and updates that weaken (Your protection is overbearing) or strengthen (Your child is in danger) the default. These updates could be viewed as potential answers to an implicit question about a base situation: “What are you protecting your child from?” for Your child is in danger. We 5-shot prompt GPT-3 to generate questions, conditioned on situation and answer, resulting in ≈ 80K (situation, update type, question, answer) tuples.
Dataset Analysis Fig. 3 shows that the crowdsourced δ-CLARIFYgold has more variety in its most common question starts, which reflects the general diversity of the dataset: For only 10% of the situations, more than 1 Turker asked exactly the same question, and for only 8% of the situations all 5 Turkers used the same Wh-word to start the question. This highlights that there is more than one possible salient clarification question to be asked for any given situation. For the situation tipping people decently in Tab. 1, all 5 Turkers chose to start their questions differently, even though three out of these five questions ask in one way or the other about the service quality. For the other situation 4/5 Turkers asked for a specification “What was the comment?” and 1 Turker asked about the missing agent role. We also see that polar (yes/no) questions appear less frequently, as Turkers were explicitly asked to avoid them unless no other suitable question comes to mind.2
The δ-CLARIFYsilver questions are generated by conditioning on weakener or strengthener updates. Since we aim to predict defeasible questions, the most desirable questions are those whose answers can be both weakeners and strengtheners.
2This is to prevent leading questions such as “Do you intend to give it to a kid?" for "offering a cup of coffee".
In the silver data, 53% of situations have at least one question that has been generated by GPT-3 for both update types. The situation Your kids should be your number one priority, for example, has the same question “What are your kids’ ages?” for the weakener update They are adult children. and the strengthener update Your children are toddlers. Interestingly, among the subset of defeasible questions in δ-CLARIFYsilver, we find that the most frequent question start is ‘why’. This suggests that it is easiest to come up with both weakener and strengthener answers to why-questions.

heading: 5 Baselines
text: We consider four baselines in our experiments.
Question Generation Without RL To assess what additional improvements training an RL model with a defeasibility rewards provides, we report performance of the supervised question generation model θQ on its own (§3.1). We refer to this baseline as t5 fine-tuned. We decode using nucleus sampling with top-p = 0.6.
Pipelines with Question Selection Next, we implement two pipelines where, as the first step, a diverse set of questions is generated for a given situation and then, as the second step, the best question is selected according to a score.
In order to generate a diverse set of questions we fine-tune T5 on δ-CLARIFY, conditioned on a modified input compared to the model from §3.1: Input <Situation>. Q.: <wh-word> - Output <Question>
By also conditioning on the first wh-word of the question it is possible to generate different questions. During inference we generate questions for 14 different question starts.3 We propose two approaches to scoring these questions: using a discriminator model and using divergence ranking, which we describe as follows.
Discriminator We train a discriminator classifier which labels these questions as either relevant or irrelevant to a given situation. We then choose the question that has been assigned the relevant label with the highest probability.
The discriminator is a binary classifier based on DeBERTa (He et al., 2020). The positive examples are situations and their respective 5 questions written by annotators. The negative question examples
3what, how who, do, are, did, is where, have, was when, would
are sampled from other situations, in a way that ensures that the relevant and irrelevant questions are similar enough to make training more challenging.
Divergence Ranking We run the defeasible answer simulation with feedback from Delphi for each question in the set. This process is the same as the reward function of the RL approach, except that the JS-divergence score is used to rank the questions instead of being used as a reward for question generation. We compare two variations of this baseline: one with answer filtering using an NLI model as described in Sec. 3.2 (pipeline-nli) and one without filtering (pipeline).
Why-Baseline We saw in §4 that questions conditioned on weakener/strengthener updates are usually causal questions. Using the same input/output configuration as in the pipeline baseline, we generate a why-question for each situation (called why).

heading: 6 Evaluation and Analysis

heading: 6.1 Human Evaluation
text: Automatic evaluation of questions and their usefulness for clarifying moral situations is tricky. While we do have gold reference questions, we have shown that humans will produce diverse questions for the same situation (§4) and just because a question does not appear in the reference set does not necessarily indicate that it is not a consequential question. We therefore perform human evaluation of the models’ outputs on Amazon Mechanical Turk on the 500 test set instances from δ-CLARIFY. Given a situation and a question, Turkers are asked to rate the question along three different attributes: Grammaticality (Is the question grammatical?), Relevance (Is the question relevant and plausible to the situation?), and Informativeness (Does the question access new information or regurgitate the situation?). The attributes are further detailed in Appendix A.1.
Additionally, and most importantly, we aim to evaluate the defeasibility of the questions, e.g. how well the generated questions can elicit weakener or strengthener answers. For this purpose, Turkers are given a situation with a question and are first asked to judge this situation (generally ok vs. generally not ok). They then need to say whether and specify if they can think of an answer to the question which might support their judgement and also of an answer which would flip their judgement.

heading: 6.2 Results of Human Evaluation
text: We first run the grammaticality, relevance and informativeness evaluation. All questions which are given the lowest rating (e.g. irrelevant and/or uninformative) by at least two annotators are excluded from the second evaluation. It does not make sense to ask about defeasibility for questions which already are irrelevant, and additional weakening or strengthening context is not feasible for uninformative questions.
We find, as displayed in Fig. 4, that CLARIFYDELPHI has the biggest percentage of relevant and informative questions in the test set, compared to the baselines. We also see that a big majority of the generated questions, from all models, are relevant and informative, with the lowest performing model (discriminator) still producing 448/500 questions that are passed on to the next evaluation round.
We also find that grammaticality across all systems is high with the lowest average score being 0.98 and the highest 0.99 (on a scale from 0 to 1, with 1 being grammatical). The minimal variation in grammaticality score is expected since all models are based upon the same transformer model.
The CLARIFYDELPHI questions also outperform the baselines in terms of defeasibility, as seen in Table 2: annotators can more often think of a strengthener answer and/or a weakener answer to our questions. The evaluation also shows that adding the answer-filtering with NLI step to the pipeline improves the question selection on all 4 evaluation criteria. The why-baseline is shown to be a strong baseline, indicating that motives and reasons are important for moral reasoning.

heading: 6.3 How much supervision does the policy require?
text: Our approach uses RL in conjunction with a supervised policy that has been fine-tuned on question generation. This has been shown to outperform approaches which use RL on top of a “vanilla" lmpolicy (Ramamurthy et al., 2022). To assess the effect of supervision on question generation performance, we trained multiple initial policies on varying percentages of δ-CLARIFY training data: 25%, 50%, 75% and 100%. To compare against more traditional supervised question generation approaches we additionally trained a policy on SQuAD v1.1 data (Rajpurkar et al., 2016).
We report two automatic evaluation metrics. To measure informativeness we use an off-the-shelf QA model trained on SQuAD 2.0 from AllenNLP (Gardner et al., 2018). This model either answers a question by pointing to a span in the input or outputs that the question in unanswerable with respect to a given context. For a clarification question to be informative it would not ask about anything already mentioned in the situation. For the QA-metric we thus report the percentage of non-answerable questions.4 We also report the average maximum BERTScore (Zhang et al., 2019) between a generated question and one of the 5 gold questions in δ-CLARIFY.
Fig. 5 shows the following trends with regards to training a supervised policy. More training data leads to more informative questions. The policy trained on SQuAD produces the most uninformative questions which can be explained by the fact that SQuAD questions are conditioned on existing answers in a text. While performance consistently increases from 25% to 75% of the training data,
4The Pearson correlation coefficient shows that this metric (moderately) correlates with the human informativeness evaluation (r = 0.31).
improvements after 75% are minimal. We conclude that for our use case training on about 5000 (75%) situations with 5 questions each leads to a sufficiently good policy. These results are also supported by the BERTScore.

heading: 6.4 Analysis
text: Answer Simulation The answer generation model generally succeeds at generating diverse weakener and strengthener answers to the same question: for only about 0.05% of questions per 1000 PPO epochs the model generates the same answer for both weakener and strengthener.
Our answer generation could be looked at as question-guided defeasible update generation. Rudinger et al. (2020)’s task of Generative Defeasible Inference generates an update given a situation, a moral judgement and the update type (e.g. weakener/strengthener). In our answer generation approach we condition on the same input together with a generated question. This intermediate question generation step functions as a type of macro planning which has been shown to be effective for NLG (Puduppully et al., 2022; Narayan et al., 2022). We evaluate our approach on the same test set using the same evaluation metrics as Rudinger et al. (2020). Table 3 shows that by first predicting the question and then the updates, we improve upon generating defeasible updates for δ-SOCIAL.
Questions We qualitatively inspect the types of generated questions: There are many specification questions asking about a hyponym of an argument
in the base situation, for example, exterminating pests on your property - “What kind of pests?”. The situations extracted from SocialChem often include underspecified pronouns, such as ‘something’ or ‘somewhere’. 60% of the situations containing ‘something’, for example, elicit what-questions from our model. Note that while such specification questions are valid clarification questions, the SQUAD 2.0 QA model would mark them as answerable given the situation. It is also interesting to see that often when a situation has a missing or implicit semantic argument, such as being anxious sometimes, CLARIFYDELPHI inquires about it: “What are you anxious about?” The generated why-questions most often ask about the motives of the agent in the situation, such as Ben tells Maggie that he’s traveling alone - “Why is Ben traveling alone?”. More rarely the model generates questions asking about the viewpoint of the patient: asking a friend [...] whether your attire is appropriate for an event - “What is the friend’s opinion of your attire?”
Analysis of Delphi’s Probabilities In Tab. 4 we quantify the JSD of Delphi’s judgments. Even though the human evaluation showed that CLARIFYDELPHI produced the most questions leading to defeasible answers, the JSD and the precentage of judgment flips is higher for the pipeline_nli approach, where we explicitly filter questions to maximize the JSD. Nevertheless, CLARIFYDELPHI leads to more Delphi judgment flips and higher JSD between answers than the fine-tuned t5 model without RL (and also all other baselines besides the pipeline). This automatic evaluation and the disagreement with the human annotators also reveals that Delphi’s probabilities are not always perfectly calibrated and relying too much on a model’s output might potentially lead to some error propagation.

heading: 7 Interactive Judgements
text: While we use answer simulation during PPO training, inference only requires a situation as input. The clarification questions can then be used to elicit additional context, in the form of answers, through interaction. Fig. 2 illustrates examples of such an interaction between a user, Delphi as the moral reasoning system and CLARIFYDELPHI. After each turn, the situation is updated with the user provided context, for which Delphi produces a new decision. We limit the interaction to three turns. This is based on the observation that after the third turn the sentence fusion starts to deteriorate, resulting in less relevant and more repetitive questions. Additionally, we find that the first two questions generally can capture missing contexts that are most central to making moral decisions. We provide more examples of generated questions in the Appendix.

heading: 8 Related Work
text: Question Generation Clarification question generation has been studied for various domains from image recognition questions to product description questions (Rao and Daumé III, 2018; Majumder et al., 2021; White et al., 2021), defining the goodness of clarification questions along the lines of information theoretic measures such as relevance, informativeness or utility (Rao and Daumé III, 2018; White et al., 2021; Warstadt and Agha, to appear; Rao and Daumé III, 2018, 2019). Most of existing works focus on questions that lead to single true answer, whereas we focus on generating clarification questions based on social situations, defining the relevance and utility of a question in terms of defeasibility. Additionally, we offer a high-quality clarification question dataset for social and moral situation—comprising of more than 30K questions—that breaks the mold from the domainspecificity of previous clarification datasets (Kumar and Black, 2020; Aliannejadi et al., 2019).
Some general question generation approaches
have incorporated an RL-based approach. Buck et al. (2018) learn to paraphrase questions with a reward that maximizes the QA answer F1 score. And Rao and Daumé III (2019) optimize a binary utility reward, using Reinforce in an adversarial setup for generating clarification questions. In our setup, we use Proximal Policy Optimization (Schulman et al., 2017; Ouyang et al., 2022) with a trained model for feedback as part of the reward.
Commonsense Moral Reasoning Delphi (Jiang et al., 2022) is a commonsense moral reasoning model trained on COMMONSENSE NORM BANK, a dataset with 1.7M instances of descriptive knowledge of people’s general sense of what’s ethically acceptable or not in everyday situations. COMMONSENSE NORM BANK is compiled from five existing large-scale datasets, including SOCIAL CHEMISTRY (Forbes et al., 2020), ETHICS Commonsense Morality (Hendrycks et al., 2021), MORAL STORIES (Emelin et al., 2021), SOCIAL BIAS INFERENCE CORPUS (Sap et al., 2020), and SCRUPLES (Lourie et al., 2021).
Delphi is based on pre-trained UNICORN, a universal commonsense reasoning model, trained on a number of commonsense reasoning tasks. Delphi can predict the ethical judgment given a description of a situation.

heading: 9 Conclusion
text: In this work we introduce CLARIFYDELPHI, which generates clarification questions for social and moral situations. We show how a RL approach that optimizes for maximally divergent answers in terms of defeasibility outperforms other clarification question baselines. While we start with a supervised policy, the reward function makes use of already trained models and does not rely on any additional training data. We believe that our questions can be useful for providing more disambiguating context through interaction.
Limitations
On Western-centricity The majority of the crowdworkers producing the source data (δ-Social and Delphi) and δ-CLARIFY were located in the United States. Due to this, the predictions generated by CLARIFYDELPHI are currently limited to representing only the perspectives of western culture (particularly the United States). Overcoming the western-centric bias is a compelling direction for future research.
On Defeasibility We rely upon Delphi to produce acceptable judgments given a situation and the modifying context as a measure of defeasibility. We recognize that, however, Delphi is not perfect and is characterized by a variety of limitations such as limited cultural awareness and inconsistent predictions (Jiang et al., 2022). Investigating improved methods for identifying answer divergences that will better capture defeasibility is a topic for future investigation.
Ethics Statement
Annotations are conducted on Amazon Mechanical Turk (MTurk). We maintain an average pay of $15 per hour for all our crowdsourcing data collection and evaluation tasks. Our crowdsourcing tasks do not collect personal information and are strictly limited to gathering workers’ general knowledge. We do not keep any deanonymizing information such as MTurk IDs so that the identity of the workers cannot be directly or indirectly ascertained. Finally, our crowdsourcing task meets the standards for exemptions as human research and has obtained the necessary documentation that deems it exempt through our internal IRB procedures.
Our model is intended to be used for research purposes only and it is not supposed to provide any sort of advice applicable outside of that domain.

heading: Acknowledgements
text: We thank our colleagues on the Beaker Team at the Allen Institute for AI for helping with the compute infrastructure. This work was supported in-part by DARPA MCS program through NIWC Pacific (N66001-19-2-4031).

heading: A Appendix
text: A.1 Crowdsourcing and Annotation
Annotations are collected on Amazon Mechanical Turk (MTurk). We run two varieties of HITs: (1) data collection HIT in which we collect questions given statements, and (2) evaluation HIT in which the workers are asked to judge validity of the generated questions. The group of 145 Turkers working on the HIts were manually vetted and selected through an open paid qualification round. We maintain an average pay rate of $15/hour for all HITs.
Question Collection: We crowdsource clarification question by prompting annotators with a situation. The crowdworkers are asked to imagine a hypothetical situation where a colleague came to them requesting advice or judgment on the shown situation. The workers are then instructed to write a clarification question they would want to ask that would help them make a better judgment or give a better advice that they would without it. Each of the 6425 situation is presented to 5 distinct annotators; we collect 5 questions per situation. A screenshot of the HIT is shown in Figure 6.
Human Evaluation: We ask crowdworkers to evaluate model outputs. Given a situation and a question Turkers are asked to rate the question along three different attributes:
Grammaticality Is the question grammatical? - yes/no Relevance Does the question fit the situation and is it plausible that someone might ask this question? - very relevant/somewhat relevant/entirely irrelevant Informativeness Can the question lead to new information or does it ask about things
already mentioned in the situation? - very/somewhat/uninformative
A screenshot for the evaluation HIT w.r.t grammaticality, relevance, and informativeness is found in Figures 8. Additionally, we evaluate the defeasibility of a question. A screenshot of the defeasibility evaluation is shown in Figure 10.
IRB approval: We sought and received exemption from our internal IRB. In accordance to the regulations, we do not collect sensitive information. If we do publish WorkerIDs, we will do so by fully anonymizing the information. The exemption received does not require a consent form.
Language and Demographics: We have not collected any demographic information from the workers. However, all crowdsourcing was conducted in English and the region (current location of the crowdworker) was set to US. Consequently, what counts as a context of consequence is centered around western views, or views of the English speaking cultures within the United States.
A.2 Prompting for Answer Generation
One way to elicit a set of opposing answers is through prompting. We instruct GPT-3 to provide a so-called “bad" and a so-called “good" answer to a question about a situation. For the situation learning how to take a joke and the question “What was the joke?”, the two answers could be: “it was a lighthearted joke among friends” and “it was an offensive joke”. In order to determine which of the answers is a weakener and which a strengthener, we compare the difference in Delphi’s judgement for s and s+ agood or s+ abad.
A.3 Details of PPO
Policy loss. To compute the policy loss, we first define the truncated estimated advantage function,
Ât = T−1∑
t′=t
(γλ)t ′−tδt′ ,
where δt′ = rt′ + γVϕ(st′+1)− Vϕ(st′),
where the value function of a certain state st is estimated by the value model Vϕ(·), rt denotes intermediate reward obtained at time step t, and γ and λ denote reward decay factor. PPO then maximizes the empirical expectation of clipped
surrogate objective term, defined as
cso(Ât, rt(θ), ϵ) =
min ( rt(θ)Ât, clip(rt(θ), 1− ϵ, 1 + ϵ)Ât ) ,
Value loss. The value model is trained with MSE loss with respect to the target value, V targt :
LValue(ϕ) = Ê [( V (st;ϕ)− V targt )2] ,
A.4 Implementation Details The question generation model θQ is trained with T5-large (Raffel et al., 2020) on δ-CLARIFY.
We train the answer generation model θA (T5large) on 77k instances from δ-CLARIFYsilver (§4) and about 4k instances obtained through prompting GPT-3. For the answer-filtering using NLI we use WaNLI (Liu et al., 2022a) as an off-the-shelf NLI model.
We run the PPO algorithm for 6000 steps, with a batch size of 64, top_p= 0.6 and a temperature of 0.7 for the sampling policy. Training for 6000 step took approximately 2 days on 4 A6000, 48G, GPUs.
Our models are based on T5-large which has 770 million parameters.
For the DELPHI model we use the t5 − large variant.
A.5 Licensing CLARIFYDELPHI and the code will be released under the Apache License 2.0. We will release the dataset under CC-BY.
A.6 Sample of Generated Questions We show a sample of 15 generated questions in Tab. 5.
A.7 Sample of Generated Questions in a Compositional Setting
We show a sample of generated questions in a twoturn interactive setting in Tab. 6.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations and Ethics Statement

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations and Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 0 and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendix

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix and Ethics Statement

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 3 and 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Appendix

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: We only report basic geographic characteristics in the appendix.

heading: Future Work
text: Overcoming the western-centric bias is a compelling direction for future research. On Defeasibility We rely upon Delphi to produce acceptable judgments given a situation and the modifying context as a measure of defeasibility. We recognize that, however, Delphi is not perfect and is characterized by a variety of limitations such as limited cultural awareness and inconsistent predictions (Jiang et al. , 2022). Investigating improved methods for identifying answer divergences that will better capture defeasibility is a topic for future investigation