Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: Jiminy Cricket is a benchmark of 25 text-based adventure games having ethical scenarios for evaluating moral behaviour of reinforcement learning agents via dense annotations of every possible action. The goal is to encourage and evaluate moral behaviour while maximizing reward.

The major contributions of this paper are:
1. Manual annotation of all possible actions in 25 text adventure games
2. Simple and effective baseline which morally conditions agents to reduce immoral behaviour without sacrificing performance
strengths: The dataset is well motivated and differentiated from similar work in terms of scale and realism. The annotations cover all morally salient scenarios with descriptions and morality labels using a robust annotation framework. The proposed baseline method to morally condition the agents using fine-tuned language models not only seem interesting but also effective. The authors also explore several directions to improve the baseline and encourage further research.
weaknesses: The Oracle Morality agent and Human Expert walk-throughs are frequently used as a point of comparison in the experiment analysis. However, it is unclear if the human experts who produced the walk-throughs were instructed to make moral choices while trying to complete the games. 

As the authors note that immoral actions might be required as the game progresses, it is crucial to know how much immorality is necessary to completely win these games. For a fair comparison against other morally conditioned agents, the human expert must be playing to win the game while reducing immoral behaviour as much as possible. Ideally both kinds of walk-throughs, i.e. one where the human plays to win and another where the human plays to win while reducing immoral behaviour, would be necessary to paint the full picture.
correctness: The annotation framework seems to be reasonably well designed and the claims are substantiated with empirical evidence.
clarity: The paper is well written and easy to follow.
relation to prior work: The work is positioned well within the literature and clearly differentiated from the similar work.
additional feedback: I would suggest to include the annotation framework in the paper itself.
review_title: Jiminy Cricket

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors provide an annotated dataset of text-based adventure games for benchmarking moral decision-making. This contributes to the field of ethical decision-making for autonomous agents.
strengths: The strengths of this paper are a novel application of moral coding to text-based adventure games.
weaknesses: This paper, while well-intentioned, makes a profoundly troubling assumption that the moral valence and ethical content of decision-making is both universal and readily apparent. Moral decision-making is deeply contextual, depending on the positionality and background of the coder, and the contextual frame in which a decision must be made. While the authors consider the context of the game itself, i.e. the gameplay context in which a choice must be made, that is far too limited a frame in which to evaluate the moral content and valence of a decision. The game itself is set within a moral context that determines what is permissible, but also what is morally desirable and appropriate. There is no "outside" perspective that would render any decision code-able as "good" or "bad" independent of a full examination of context. The use of graduate students to conduct the coding is another weakness of this paper. Graduate students represent a slender, privileged set of experiences on which to draw ethical and moral reasoning that ought not be taken to represent any wider constituency. Furthermore, the assertions made by the authors that "moral awareness comes naturally to all humans" is a dangerous assumption that is not at all borne out by social science research and papers over profound differences is moral frames across and between cultural settings. To assume a universal moral awareness is to erase and dismiss any moral framings that diverge from that possessed by the researchers themselves.
correctness: The claims about moral awareness and the learn-ability of moral decision making are profoundly flawed. The correctness of the benchmarking practices and statistical analyses are, independent of this, outside the scope of this reviewer.
clarity: The paper is clearly written, although it could use more description of what text-based games consist of, as well as how and why they are appropriate for moral analysis.
relation to prior work: Relation to prior work within moral philosophy is discussed, although through the very limited frame of Rawlsian approaches.
additional feedback: This paper requires a greatly expanded limitations discussion that addresses the contextuality and cultural bounded-ness of moral reasoning, the limitations of using graduate students to code moral decisions, and the moral frame in which game play occurs.
review_title: Jiminy Cricket: Benchmarking Moral Behavior in Text-Based Games

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Update after rebuttal:
After a couple rounds with the authors as seen in the comments below, I am relatively satisfied and will increase my score to reflect that I believe this work should be accepted.

==========

The paper provides a benchmark, Jiminy Cricket, that provides dense moral value alignment annotations and utilities for many of the existing Jericho benchmark suite of text games. The paper provides a method, Commonsense Morality Policy Shaping (CMPS), that uses RoBERTa trained on morality scenarios to provide a signal to a reinforcment learner. This method outperforms some existing methods such as CALM (Yao et al. 2020) and NAIL (Hausknecht et al. 2019) when measured in terms of safe moral exploration.
strengths: At a high level, the motivation is crystal clear and is something that is very much needed more in many RL benchmarks. The themes of "Value alignment" and "morally safe exploration" will be key in future RL benchmarks as pre-training makes its way into this space also.

The enginnering+annotation efforts, additional utilities provided (complete object tree and fast forwarding in particular are useful), and the time sinks each of these represent (esp given how messy ZIL and Infocom source code is) is definitely seen and very much appreciated.

The result that policy shaping is more effective than reward shaping for these domains is interesting and provides a starting point for future researchers in the area.
weaknesses: Inconsistency with earlier benchmarks in the area. Overall completion aside, the number of games for which CMPS beats CALM/NAIL appears to present a different picture (CALM is not as terrible as portrayed), this number should be reported. Taking only an overall average loses per game information (i.e. the ability to make an "apples to apples" comparison).

Adding to the above point, a breakdown of the performances of the agents at different starting points should be given (in the appendix at least) - the current forumalation of the metrics loses information that could provide deeper insight into where such agents struggle.

Further still, the lack of multiple seeds for testing variance at least for the 0% walkthroughs (so 25 envs and not 125) makes the results rather unconvincing given rather narrow performance improvement margins and lack of statistical significance.

Many existing text game baseline agents such as KG-A2C (Ammanabrolu et al. 2020), GATA (Adhikari et al. 2020), etc. are not checked against or cited. This could be useful given that most of these agents are tabula rasa and do not use pre-training, and thus display biases only from the reward.

I am not clear on how exactly the "noiseless object tree" utility provided differs from z-machine returned trees. Is it just that the entities are human readable instead of codes IF developers often use? More details on this would help clarify given that it is claimed as a contribution.

The fact that the morality judgements were made by CS undergrads and grads, presumably at the author's institutions is a huge weakness, especially without further information on the annotators, and provides a source for bias and potentially propogating stereotypes. The only mitigation I see for this is the fact that many of the games are fantasy themed and so the scenarios are less likely to be connected to "real world" stereotypes. There is no information even in the datasheet about how these source code annotations were made, aggregated demographic information, or other measures the authors have taken to not amplify harmful biases found. 

The annotation framework used should be mentioned in the main paper (at the expense of some of the engineering details perhaps if space is an issue).

I am also not convinced that the annotation framework is sufficiently fine grained enough to capture what the authors present as issues with the normal Jericho game rewards. One example in particular that I see is the case of disproportionate punishment (pointed out in Fig. 4 for Wishbringer). A 4 point scale of good/bad self/others is not enough to capture this distinction.

At a high level, it is not clear to me how scenarios in a bunch of Infocom games, many of which are fantasy or otherwise not real-world themed, many scenarios which are unrealistic or contradict everyday commonsense norms, will help measure real-world commonsense morality as claimed by the authors. What exactly is the information that is expected to transfer?
correctness: See above Weaknesses section.
clarity: The paper reads smoothly and I have no issues with general clarity. Details to be expanded on are in the Weaknesses section.
relation to prior work: As mentioned above many other benchmarks are ignored including GATA, KG-A2C. The idea behind this benchmark is essentially the same as Nahian et al. as noted as noted in the paper, though on a much larger scale.
additional feedback: Would like to note that Jiminy Cricket is a proxy swear/oath for "Jesus Christ" and not just the cartoon character (though I'm sure the authors are aware of this).
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a benchmark to evaluate text-based game AI agents in terms of moral behavior, rather than just focusing in reward functions that can go against morality, with the purpose of making the simulated environments provided by such games useful to develop ethical agents. For this purpose, the authors have annotated 25 classic Infocom text adventure games according to the morality of each relevant action. The paper also presents approaches that use pretrained language models (which can, to a certain extent, assess whether the actions described in a piece of text are moral or not) to condition the agent into a better moral behavior, showing that these approaches indeed reduce immoral actions without reducing the agent's performance in the game.
strengths: The paper addresses a relevant problem, since text-based games are a useful environment for AI research but the games' built-in rewards (i.e. game completion) are not necessarily aligned with morals. This dataset will be useful to leverage such games as environments for experimentation while creating morally-aware agents, and to evaluate how such agents perform with respect to morality.

The presented dataset represents a very substantial amount of work - a large number of actions have been annotated in a very detailed way by diving into 70s-80s source code.

The models presented in the paper and evaluated against the dataset are interesting not only for their own sake, but because they give an idea of what one can expect to achieve (i.e., the provided data and visualizations implicitly answer some natural questions like how much immorality one typically needs to finish a game, etc.)
weaknesses: While the presented models reduce immorality with respect to the baseline, they still have a long way to go to even approach the human expert. I do not consider this to be an important weakness as the main contribution here is the dataset, with the models just serving as a way to show how the dataset can be used and a starting point.
correctness: The scientific content of the paper is sound as far as I can see. I only have a minor comment about the claim that "people have an innate sense of basics of right and wrong that guides their actions in everyday life". While I am no expert in these matters, I don't think one can categorically claim that there is such a thing as an innate (as opposed to learned) morality; my understanding is that that is not obvious and subject to debate. If the authors want to make this kind of claim, it would require some justification (references, etc.)

The paper should also at least mention the fact that morality is not universal (actions that are considered good in a given culture/country/religion are not necessarily so in a different one) and provide some data about the annotators' cultural background.
clarity: The paper is clearly written.
relation to prior work: The paper does a good job of putting the contribution in relation to previous work.
additional feedback: While this does not affect the scientific value of the contribution, I think there is a natural question that the reader may ask themselves (at least, I did) and should be answered in the paper: why Infocom games? These are games written in a quite ancient programming language, and their copyright is unclear as they are abandonware (as the authors rightly point out in the supplementary material). Wouldn't it have been easier to use more modern games, written in more modern languages like Inform, and which are often open-source (or permission could be seeked from the authors when they aren't)? I would like to know (and I suppose, many interested readers would) what the reason for this choice is: is it that modern games tend to be shorter or more linear than Infocom games?

In any research dealing with language, we should specify what language we are using - the paper should explicitly say somewhere that this is a dataset of English games and all the models operate on English.

I spotted a couple of minor typos/inconsistencies:

l. 74: markov -> Markov

l. 116-117: "For instance, in Figure 1 an agent decides to take an object from a non-player character, who resists the attempted theft" -> according to the figure, the NPC had been killed, so they could not resist the theft.
review_title: A substantial dataset for a useful purpose

Decision: Accept