Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: To achieve capturing the flexibility of the human moral mind, authors created a challenge set consisting of rule-breaking question answering.
They use the state of the art large language model (InstructGPT) and developed MoralCOT that is based on theories of moral reasoning, which outperformed other existing large language models, they say.
 
strengths_and_weaknesses: [Strength]
In my opinion, authors' ideas (e.g., breaking down into a few decision processes and estimating costs) are quite novel from the psycholinguistic view point. They also provide detailed examples of input and output, which may be beneficial for the community.

[Weaknesses]
Dealing with only 3 types of norms are not very sufficient. 
The proposed method fully relies on open AI's API, which leads less reproducibility. 
Details of human's responses are not shown, which results in less clarity.
questions: - I think relying on InstructGPT is problematic for reproducibility, I'm wondering if InstructBERT can be compared? 
   I would at least like to see the results of applying the same mechanism as MoralCOT to a pretrain model such as BERT.

- For binary predictions of human moral judgement, how the inter-rater correlation would be? I'm wondering if each model could be comparable to individual's assessment.  
limitations: Authors employed only 3 types of norms, which could be regarded as preliminary study Therefore they need to prove the types are sufficient to cover the human morality. At least they should show how these 3 types of norms are important for dealing with human morality.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses the problem of improving language models by taking into account human moral judgment: basically, the idea is to build AI systems that have to establish whether it is permissible to break a moral rule in several situations. This is a crucial task in AI safety, as it deals with the capability of understanding ethics in relation to heterogeneous contexts. The proposed approach consists in a prompting strategy that generates a sequence of questions and answers that can be exploited to predict human moral judgment.

strengths_and_weaknesses: + Building AI systems that account for human moral judgment is a crucial need
+ The motivation related to AI safety is particularly strong
+ Rule-Breaking Question Answering (RBQA) is a new challenging task/dataset for NLP systems that aim to incorporate human moral judgment

- The description of the model and of the proposed approach is too brief, and several details are lacking or just mention without a proper analysis.


questions: * How is the sequence of questions q_1, ..., q_N chosen? Is it fixed (i.e., pre-determined) or adapted to each case?

* Did you try other aggregation functions rather than concatenation to obtain the chained prompt c_i?
limitations: The main limitation of the paper is that the decription of the model is quite short. I would have liked to see more details on the cognivitely-inspired model: for example, whether the posed questions are fixed or not, or whether the authors tested a different aggregation function rather than concatenation (e.g., similarly to what happens with memory networks).

Other comments and typos:
-- In Table 3, the last three columns are misleading: they represent an F1-score, but this becomes clear only after reading the caption. I would suggest either to split the table in two, or to move the three columns next to the initial F1 column, and include a supercolumn named F1 covering all the four cases (the average, plus the three single scenarios).
-- Line 255, "to a multi-step prompt in Figure 1" -> "to a nulti-step prompt, as shown in Figure 1"
-- Line 344, "that contradicts" -> "that contradict"
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the important question whether LLMs understand human flexible moral judgements. Human moral judgements are based on rules (like "don't cut the line"), but most important are flexible. Depending on the situation, it might be morally OK to break the rule. Prior work (like MoralQA) doesn't address this important aspect of moral judgements.
This work proposes a challenge set that addresses precisely this. The set contains three different rules, each with a set of situations that either allow breaking them or not. The "ground-truth" is a large set of judgements made by humans. The situations are made such that novel scenarios occur that are unlikely to have occurred in training. This is important, because central to human morality is the ability to respond in a novel situation that is hasn't been exposed to in training. 
The authors evaluate a set of language models on this challenge set, most notably of which Delphi (domain-specific, trained on moral judgements) and instructGPT (one of the SOTA LLMs). They show that most models don't achieve much better than random on this task, but that their proposed method called moral chain-of-thought prompting improves the SOTA by ~11% F1. The authors do an extensive error analysis and include high-stake wrong predictions in the appendix (e.g. when the model doesn't realise a life being at stake is important).
strengths_and_weaknesses: **Strengths**
- This paper addresses a very significant subject (to what extent do the LLM align with human moral judgements)
- It does so by realising that this is not an objective task. The moral judgements are made by a large group of human annotators and not only binary accuracy by the models is reported (OK to break rule vs not OK), but also the % of people saying it's OK.
- If it's true that a challenge set like this (flexible moral judgements) doesn't exist yet (reviewer can't judge because doesn't know the literature), then it's a clearly important and original addition.
- The clarity of writing is good, the paper is easily understood in a single pass.
- The dataset construction is sound and contains a lot of human judgements.

**Weaknesses**
- Information about the type of human annotators is missing (demographic, how were they recruited, etc.)
- It's quite difficult to judge the results without looking at the appendix. E.g. in "checking subquestion answers" I have no idea what this accuracy means. Mostly I would at least add a conclusions derived from the results in the main paper.
- It's a well-known issue that LLMs are sensitive to prompt wording and even ordering ([1], [2]). Although the authors do mention they did an analysis on the sensitivity on the model to this and add it in the appendix, to make the claims that this paper makes I think this should be part of the main results (i.e. in the form of variance over prompt wording in Table 3).

[1] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=ShnM-rRh4T
[2] Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In ACL, 2022.
questions: Questions that could address limitations:
- How sensitive is the SOTA model (MoralCoT) to prompt wording and ordering?
- Would it be possible to add some conclusions from analyses that are part of the appendix in the main text?
- Same but for human annotator demographic (seems like things could be cultural dependent, i.e. in some cultures it might be absolutely not OK to step on money when it is about to be blown away by the wind, whereas in others that's considered helpful)
- Do you have any data on how much humans agree usually in the judgements, i.e. what is the distribution of percentages of permissibility?

General suggestions:
- Consider also citing the following work for Chain-of-Thought prompting (scratchpad): https://arxiv.org/abs/2112.00114
- s/that/those line 74
- 131-133: I would rephrase this a bit less strong, it seems to me like the previous setup (moralQA + finetuning) can test flexibility (simply by having questions that reflect flexible judgement), but they don't currently, and your data is in a sense complementary; moralQA can test whether the model even has knowledge of the vast amount of norms out there, you test wether they understand the flexibility but with less data/norms.
- typo line 206
- I don't fully understand what's going on in the line 259 within the `concat()` operation
- Reference to table 8 in line 312 should be table 4
limitations: Yes, the authors talk about the dataset size, the fact that there are different theories of moral judgement, the fact that models shouldn't be making moral decisions automatically.
review_title: null

Meta-review: metareview: This paper addresses an important question of whether LLMs understand human flexible moral judgments. This is a crucial task in AI safety, as it deals with the capability of understanding ethics in relation to heterogeneous contexts. The proposed approach consists in a prompting strategy that generates a sequence of questions and answers that can be exploited to predict human moral judgment.

The reviewers agree that the problem is important and timely, the constructed data resource is sound and of great interest to the community, and the evaluation is done thoroughly. Reviewers' raised concerns and questions are properly addressed by the author's response. 
confidence: Certain
Decision: Accept