Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates whether large language models can make causal judgments and reason about moral permissibility in text scenarios. Two new datasets are collected by aggregating stories from 24 cognitive science papers and standardizing the human annotations from the datasets accompanying these papers into ML-ready datasets (additional expert annotation is performed as well). The authors find that LLMs perform poorly on both datasets. To improve performance, they propose three methods: cognitive scaffolding (predicting useful intermediate attributes), thought-as-text translation (abstracting away irrelevant details), and expert reasoning engine (predicting label directly from the intermediate attributes with a shallow network). In experiments, these additions greatly improve performance.
strength_and_weaknesses: Strengths:
- The problem area is important.
- The proposed methods are intuitively reasonable, and they seem to work well.

Weaknesses:
- This paper has a lot to take in. I found myself needing to jump back and forth between R1, R3, etc.

Questions:
- Could the proposed methods for improving performance on these tasks also be used to improve performance on existing datasets, e.g., ETHICS? This would greatly strengthen the paper.


Minor points:
- Typo: "With the crowd sourced votes, we can make our Since each of the story is designed to verify a scientific"
- Typo: "The inter-rator agreement"
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper could be improved. This is the main weakness of the paper right now. It's just a lot to take in, and I suspect readers will have trouble skimming it. There is plenty of novelty, and there are enough details for reproducibility.
summary_of_the_review: I think this paper could generate good discussion, and the new datasets seem like useful tests for LLMs. I vote to accept.

-----------------------
Update after reading rebuttal and other reviews:

The other reviewers raise some interesting points. In particular, Reviewer XfnA's point about simplifying stories and lexical cues may need to be further addressed. (Note: I'm not convinced that the paper has serious ethical concerns.) Overall, I'm still convinced that this paper is valuable. This is mainly because the only real avenue people have considered until now for increasing performance on ethics / morality benchmarks is using bigger LLMs. While this does work, it's very valuable to explore methods that differentially improve performance on these tasks using domain knowledge. This work accomplishes precisely that, so I think it provides value to the community. Consequently, I'm keeping the same recommendation of acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies how well-correlated human judgments and LLM predictions are for causal and moral reasoning tasks. They collect a dataset of human judgments for a number of text problems used in cognitive science literature for probing human judgments, including expert judgments for relevant factors in the judgment process (e.g., norms, inevitability). They perform a few experiments that find that (1) correlation between human and LLM predictions is low out-of-the-box, but that (2) explicitly providing factors only, or combining the two methods, can yield better performance.
strength_and_weaknesses: Strengths:
- The description of the task and crowdsourcing (especially expert judgment) annotation is quite thorough.
- Experiments are easy to understand.
- Evaluation of the LLM by disentangling text complexity with reasoning is interesting (Section 4.2).

Weaknesses:
- There are many missing details about the crowdworkers involved in the judgment tasks. Who was recruited to participate? Where were they located? 
- There is also missing detail about the dataset itself. Perhaps the most interesting part of this dataset might be the actual distributions (between yes/no) of user annotations on the stories, but this is not provided in the paper or materials as far as I can tell. Instead, it appears that stories are bucketed into "yes", "no", and "ambiguous" buckets (when agreement is too low). 
- There is missing discussion on the motivations and framing of these tasks at a meta-level. "when something bad happened, we naturally ask: who did what, and why?", "people's moral knowledge helps them tell apart good from bad" -- these claims are from a particular philosophical perspective, e.g. where "good" and "bad" are standalone values that can be placed on an action or event in absence of enough context.
- Similarly, framing the task of causality as placing blame on an agent rather than on the structural context is a specific choice. I understand this study is coming from the perspective that cogsci has created on causality, but this framing shapes event understanding in a particular way (i.e., that structural context, e.g., that Billy/Suzy's computer has this particular flaw, is not the focus of blame). I also understand that the goal is to study how predictions correlate with human judgments, but it also seems that the questions asked of annotators ("Did Billy cause the emails to be deleted?"), rather than asking open-ended questions, may result in a bias towards agents becoming the sole focus of blame in LLMs (if we want to assume that LLMs will or should adopt the moral judgments of humans as studied in tasks like these).
- The metrics themselves are not well-motivated. Accuracy with discretized categories of judgments (yes, no, ambiguous) seems like an arbitrary choice. Did the authors consider directly evaluating the distribution similarity between the LLM's prediction and the distributions among 25 human raters?
- It appears there isn't human evaluation for the generated T3 stories, which would make the comparison more accurate in Table 3.
- I don't think the claim "we ... only test if LLMs can have any implicit or innate ability to combine these factors and produce the same judgment a human would" is accurate. It is still possible that the authors' choice in generating the T3 versions of stories influences the model's predictions (e.g., with a spurious bias towards "yes" across the stories) simply in how they were worded, so although the details of stories are removed, the surface form itself is still not being controlled for.
- I'd be interested in seeing "error analysis" of the predictions in Table 4, as well as precision/recall of factor prediction rather than plain accuracy.
- There is an implication in Section 5 that "aligning" our LLMs to perform well in these causal/moral judgment tasks will result in better text generation in general (please correct if I am wrong about this implication), yet there isn't really substantial evidence of these LLMs (1) being "misaligned" with human judgments in general, free-form text generation, or that (2) "aligning" them with the proposed method will improve "alignment" in free-form text generation.
clarity,_quality,_novelty_and_reproducibility: Clarity:
- There are a number of typos in the dataset, e.g., "colleccted", "theoriezed", "severval", "inter-rator"; also in the "Dataset" paragraph of 3.1, there seems to be a partial sentence cut off.
- Citations for COPA and SuperGLUE appear to be transposed

Quality:
- The experiments are thorough.
- The dataset itself could be valuable, especially for cogsci researchers, but there should be more details provided about its collection and the distribution of annotations in each story. The framing of questions and stories could also affect the moral judgments in a way that results in biases in human decision-making.

Novelty:
- The dataset itself appears to be new, and relatively thorough (reaching back to decades of cogsci research to acquire more high-quality scenarios).
summary_of_the_review: This paper is thorough, and the paper is clearly written. However, I am a bit wary of underlying assumptions made in the design of the paper, which makes me lean towards rejection:

- That moral and causal judgments can be bucketed into "yes", "no", and "ambiguous".
- That reasoning about causality is equivalent to answering the question of which agent is to blame under a particular structural context, rather than placing blame on the structural context of the event itself.
- That we should prioritize "aligning" LLMs with some measure (in this paper, majority vote) of human judgments in terms of morality and causality as it's studied from the perspective of cogsci researchers, even though human judgments, reasoning, and perspectives about morality and causality vary significantly across personal experience, cultures, etc. In particular, this paper is focusing on a set of crowdworkers to provide judgments (while providing no details about how those workers were selected).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper examines behaviors of large language models in causal and moral judgment tasks drawn from cognitive science literature studying human judgments in these domains. They first test on simple cloze tasks using stories and prompts that human participants responded to, and identify model preferences for continuations of "Yes" or "No". They find low (near-chance) levels of alignment with human judgments. However, when they simplify to giving the LMs abstract descriptions capturing high-level properties of the stories, alignment improves a bit. They then do a factor inference task, where the models are supposed to complete a prompt with a factor label from the cognitive science literature. Lastly they combine the latter two experiments, getting tags from GPT-3 and using these to along with an abstract prompt. This last experiment shows a small amount of improvement in the largest GPT-3 model. 
strength_and_weaknesses: Strengths: I'm happy to see engagement with cognitive science literature, and this paper deals with the important topic of models' ability to do humanlike causal and moral reasoning. The experiments are interesting creative. 

Weaknesses: 

There are issues with methodological clarity, and I'm skeptical about interpretations of a number of the results. 

For R2, it seems likely that simplifying to the abstract stories creates simpler lexical cues that the models are able to latch onto to show the bump in performance that is observed. There may also be biases in terms of frequency of mentions, e.g. "A" is mentioned more times in Fig 2a than "B" is. Additionally, it seems worth considering the possibility that humans actually wouldn't show the same patterns with these abstract versions of the stories as with the original stories, since these seem like potentially less intuitive inputs to reason about.

For R3, the authors initially describe the tests as zero-shot, but based on the subsequent description and Appendix examples, it seems what they are doing is few-shot. So this is an issue with methodological clarity, and it makes me concerned about my interpretation of other experiments as well -- as far as I know, the other experiments are truly zero-shot, but I'm not sure if this is actually correct. If that interpretation is correct, however, then the next concern: why is R3 the only experiment where the model is given examples in the prompt? My immediate concern is that the models could identify simple cues in the prompt examples that predict the factor tag (e.g. "both" --> conjunctive). The authors acknowledge something like this concern, and argue that because their prompt examples are higher-level than the final item the model needs to respond to, this concern can be dismissed. However, this really does not exclude the possibility that there are overlapping cues between the prompt examples and the real examples. Additionally, given that R3 is (I think) the only few-shot experiment, and is also the only experiment in which the models show notably higher performance than any of the other experiments, this raises further suspicion that the strong performance is attributable to simpler solutions based on the few-shot examples. 

As for R4, I'm having trouble understanding what we stand to gain/learn from this experiment, and how we should interpret improvements relative to R1/R2, since it seems that the R4 is simply adding less accurate factor labels to what is otherwise the R2 setup. I'm (again) not completely clear on the method here, but as far as I can tell R4 is just changing the list of labels prepended to the abstract story from R2, and making them the GPT-3 predicted labels, which are decent but worse. What do we stand to learn from this? Why is it useful to check how the models perform if the labels become the GPT-3 labels? The results table only compares to R1, so I'm having to scroll back and forth to try to check the (seemingly more direct) comparison with R2 -- it looks like it's often worse, but sometimes on par/better.  So is the conclusion that if we do a noisier version of R2, performance is often worse but sometimes better? Why would that be? Something else that is unclear: the description of the "Expert Reasoning Engine" comes out of the blue and is not enough for me to be clear what was done (and is also confusing because it uses the phrase "one can imagine" multiple times, suggesting this is merely a hypothetical, but then it turns out to be in the reported results). Did the authors train a separate classifier mapping between GPT-3's noisy labels and the "Yes/No" labels? Is this no longer a cloze task but a classification? There really needs to be a lot more methodological clarity here. 

Finally, zooming out a bit more: the authors are claiming that cognitive science-based scaffolding is helpful for the models' performance, but I'm not sure in what way this is shown. It seems like it's mostly the labeling of factors that comes from the cognitive science literature, but it's not clear that listing those factors along with the story (which I think, but am not sure, is what's being done) is actually making much of a contribution to models being able to make humanlike judgments on the stories. The other candidate for "cognitive scaffolding" is the translation of the stories into abstract versions -- this does seem to help the models, but it's not clear that this is a contribution from cognitive science, and I also had concerns about lexical shortcuts as outlined above.


clarity,_quality,_novelty_and_reproducibility: I've outlined issues with clarity above, which I think are non-trivial, and also affect the reproducibility. I think the quality of the paper can be improved by addressing the various concerns I've raised about interpretation of the results as well as validity of the claims about contribution of insights from cognitive science. 
summary_of_the_review: I am happy to see this engagement with cognitive science, and I think this is an interesting and useful topic -- but I have numerous non-trivial concerns about methodological clarity and validity of conclusions, and I'm skeptical that the paper has given a strong demonstration of usefulness of insights from cognitive science, as it claims. I think these concerns should be addressed before publication, but I am optimistic that the paper can be much improved and make an interesting contribution.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper investigates whether the language models can make causal judgements and reason about moral permissibility. To this end, the authors collected two datasets by aggregating stories from 24 cognitive papers. By conducting experiments on these two datasets, the authors show that the existing language models do not perform well. Three methods were proposed to improve the performance.

This paper studies an interesting and important problem. The research questions are clear and the answers to these questions provide new knowledge. However, there are a series of concerns raised by the reviewers. For instance,  simplifying to the abstract stories creates simpler lexical cues that the models are able to latch onto.  Framing the task of causality as placing blame on an agent rather than on the structural context is a specific choice that lacks detailed justifications. The method part is clearly described and it is not clear how cognitive science-based scaffolding is helpful to improve performance. Given the concerns, the paper needs a major revision before publication. Although we think the paper is not ready for ICLR in this round, we believe that the paper would be a good one if the concerns can be well addressed.

justification_for_why_not_higher_score: how the proposed method is inspired by cognitive science needs further justifications.
some technical steps in the method need further clarifications.
justification_for_why_not_lower_score: n/a
Decision: Reject