Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the question: how can we train agents to behave 'morally' in text-based games, while achieving a high task reward?  The paper proposes a two stage learning method, which is iterated over time. In my understanding, this involves training both a task-specific Q function ('task policy'), which only cares about task reward, and a 'moral policy' which is fine-tuned from GPT-2. During inference, candidate actions are sampled by the moral policy, and the final action is chosen via a weighted combination of the logprobs of the moral policy and the task policy (derived via softmax on the Q function). The task policy is updated via normal TD learning. The moral policy is refined using self-imitation learning -- the 'best' trajectories (based on task reward, length, and novelty) are selected into a buffer, they get assigned a 'morality score' according to a 'commonsense prior' (a BERT model fine-tuned on a part of the ETHICS dataset), and a weighted behavior cloning objective (weighted by the commonsense prior) is used to update the morality policy. 

The paper shows that this method improves upon baselines on the Jiminy Cricket benchmark (15 text-based games w/ morality scores), both on immorality score and completion percentage.
strength_and_weaknesses: Strengths:
+ The problem being tackled is interesting, and fairly important. Studying this problem in text-based games is an interesting angle.
+ The paper includes a nice ablation section.
+ The proposed algorithm does seem to improve over the baselines.

Weaknesses:
- I found the method wasn't very clearly explained, and took a while to understand with multiple re-reads. Specifically, I think it'd be useful to explain DRRN and CALM more in detail when first presented, since these are the core pieces on which MorAL is based.
- The improvements over baselines are quite small, and given there are no error bars it's hard to tell if the results are significant. 
- The closest competing algorithms (CMPS and CMRS from Hendrycks et al) are only described briefly in the introduction, but they seem like very similar to the proposed method. The overall strategy is the same (train a Q function that optimizes task reward, and modify it using a commonsense prior to make it more moral). From what I can tell, only difference seems to be whether the correction term is used to modify the game reward or Q-value, rather than separately choosing actions using a mixture of the Q value and moral policy. I'm not convinced by the drawbacks mentioned in the paper for the Hendrycks et al. method:

"First, adding a correction term to the game reward or Q-value will generate extra noise, particularly for
game rewards that are extremely sparse. In addition, some immoral actions are necessary for progressing through the game. For instance, in the game “Zork1”, the agent must steal the lantern to
reach the following location on the map, as shown in Figure 1. The trade-off between task progress
and morality is a dilemma that agents may encounter while making decisions."

Specifically, it's not clear to me how the proposed MorAL algorithm materially changes the problem of "some immoral actions are necessary for progress in the game" -- it seems like both methods have some way of trading off morality and task performance, and it's unclear to me which is better. I think this paper would be much stronger if it had a better argument / analysis for why MorAL performs better than CMPS/ CMRS, and why we'd expect this to be a general finding, rather than a specific quirk of this environment. 


----------------------------------------
Small points:
"RL agents may select immoral actions unconsciously"
> unclear if 'unconsciously' is an appropriate word here (what would it mean for an RL agent to select actions consciously?)

"The LM is then equipped with a conscience" 
> I don't really like using these kinds of human analogies, since I don't think the 'conscience' described here is anything like a human conscience. 

"To sustain the agent’s exploration, we define that
within an episode, if the current steps t exceed the max length of trajectories lmax in buffer B, πT
should be used instead of the mixture policy for selecting actions."
> So if the episodes are long enough, the agent behaves non-morally? Or is this only during training?

"The walkthrough is constructed by human experts to quickly obtain the maximum possible score while taking less immoral behaviours"
> Seems like the method needs human expert data to warm-start learning. Is this also a limitation of other methods?

"The framework eliminates the assumption that dense human feedback is required
during training, as we only perform morality learning using a limited number of trajectories at
specific stages."
> It seems like other methods also don't need dense human feedback, unless I'm mistaken? 

- I'm not clear why you call MorAL a 'framework', rather than simply an 'algorithm'. To me it seems more like an algorithm.

- Eq1 -- what is A? Is it the set of candidate actions?

clarity,_quality,_novelty_and_reproducibility: 
Clarity: okay overall, though I found the clarity of the methods could be improved.

Quality: also okay, see above

Novelty: I think the novelty is fairly limited, see above

Reproducibility: seems quite good, lots of details given on hyperparameters, etc.


summary_of_the_review: 
Overall, this paper proposes a new algorithm for moral learning in text games and shows slight improvements over baselines. My biggest concern is that there is little space devoted to analyzing how / why the method is different from previous work (specifically Hendrycks et al.), and it's not intuitively clear to me from the paper why the proposed method should be better. Thus, the paper relies on its empirical results, which, while not bad, seem to me to be only small improvements over the baseline (though I'm also not an expert in this field). Thus, I lean reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Moral Awareness Adaptive Learning (MorAL) to enhance the morality capacity of an agent using a plugin moral-aware learning model. The paper develops a mixture policy to interleave  task learning and morality learning. 

 

For each episode, the moral policy decodes a set of valid action candidates. The task policy pairs it with current observation and computes a Q-value. To pick an action, a combination of Q-value and a score returned by moral policy is used. These states are stored in a memory. Later, morality scores are computed for high-quality trajectory to update the morality awareness control module. The morality scores are computed by a commonsense prior module. 

 

The paper compares MorAL technique with four other prior techniques (NAIL, CALM, CMRS, CMPS) on the Jiminy Cricket benchmarks. The metrics of evaluation are Immorality Score, Completion Percentage and Relative Immorality (which are used in prior work). The paper presents a comprehensive ablation of the different components of MorAL and their contributions. 
strength_and_weaknesses: Strengths: 

1. The MorAL technique achieves overall a significant boost of 19% in game completion rate while decreasing the immorality score by 5% compared to CMPS. Hence, overall 15 games the MorAL technique seems successful. 

2. The paper presents comprehensive ablation experiments studying the effect of each component of MorAL on immorality score and completion percentage. The ablations show that using the moral-enhanced objective has the highest impact on immorality score and completion percentage. Removing this significantly increases the immorality score and completion percentage. 

3. The paper also presents an example of actions generated by morality policy and the action picked by the MorAL. 

Weakness: 

1. Even if on average MorAL achieves the least immorality score and the highest completion percentage, for 7/15 tasks NAIL has lower immorality score and for 4/15 tasks CMPS has lower immorality score. In total for 11/15 tasks, MorAL does not achieve the least immorality score. These results are not discussed and no reasoning is provided why this is the case. The reason MorAL achieves an overall lower immorality score is because for certain games like Ballyhoo, Moonmist and Deadline NAIL has a high immorality score. There should be a discussion on what is the reason for this. 

2. Similarly, for a total of 8/15 tasks, other prior techniques achieve a higher completion percentage. Similar to point (1), this should be discussed. Are there certain properties of certain games because of which MorAL performs better or worse on them. 
clarity,_quality,_novelty_and_reproducibility: Overall the paper is clearly written. It needs more explanation about the main results. The MorAL technique is novel and it doesn't need dense human feedback during training. The code will be released and that would make MorAL reproducible. 

1. The commonsense prior model which is used to obtain morality scores is a RoBERTa-large model finetuned on commonsense morality portion of the ETHICS benchmark. It is unclear what is the performance of this finetuned model on the test set of the benchmark I.e how good is the quality of this model in predicting morality scores especially on out-of-domain data (because the Jiminy Cricket dataset would be out of domain for this model). 

2. Similarly, the use of commonsense prior model could also be a limitation because the errors of this model would be propagated to the morality control module of MorAL. This should be made clear and what definition of morality is used should also be clarified in the paper. 
summary_of_the_review: Overall, the paper proposes a novel MorAL framework to reduce immoral actions in text based games without impacting the game completion percentage. The MorAL technique is overall successful in doing so but not when we look at majority of the games individually.

> Making changes to the score to reflect the points made in the discussion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes MorAL, a new method for training text-based agents to avoid immoral actions. MorAL modifies the action generator in CALM with an imitation learning procedure that is weighted for immoral actions. On the Jiminy Cricket benchmark of text-based games with morality annotations, this notably improves game completion and reduces immoral actions, improving over existing baselines. Ablations show that various components of the method are working as expected.

Compared to previous work, the main contribution seems to be the idea to modify the action generator to reduce the sampling of immoral actions via imitation learning on a replay buffer. This is in contrast to CMPS, which directly modifies Q-values, and GALAD, which pretrains the action generator to reduce immoral action sampling without updating it during RL training. The proposed idea works well (compared to CMPS; GALAD is not compared to).
strength_and_weaknesses: Strengths:
- Good improvements over CMPS (the previous SOTA on these environments)
- Artificial consciences are an interesting area of research
- The literature review throughout the paper is good. It is comprehensive and accurately represents the prior work. (although )
- Interesting idea to train an action generator with "moral-enhanced cross-entropy"
- Helpful implementation details in the appendix
- The ablations seem to be informative. Removing the moral cross-entropy loss (MorAL w/o Mixture w/o MeO) leads to the highest completion percentage but also many more immoral actions. This seems to validate that the mixture approach and moral cross-entropy loss are working as expected.

Weaknesses:
- Lack of clarity in Section 4.3 about what is old vs new. For instance, the data buffer is already a component of CALM, yet this is not made clear in the writing and it sounds like the authors are presenting something new. If there are differences compared to the CALM buffer, this should be made clear.
- Why only 15 games from Jiminy Cricket? How were these selected? Out of the 15 games selected, there are a few where immorality increases slightly, so I don't think the authors engaged in cherry-picking. However, it would be useful to know why the remaining 10 games were not included and whether the authors plan to include them (this would make it easier for future work to compare to the MorAL method).
- The paper should include more discussion of why comparison with GALAD is infeasible. This could be a subsection of the appendix. In particular, this paper proposes modifying the action generator language model, and GALAD does something similar. This should be discussed.
- It's not clear what the moral policy language model is. Is this a GRU on top of a pre-trained GPT-2 model? Or are you fine-tuning the GPT-2 model?
clarity,_quality,_novelty_and_reproducibility: Clarity:
Major points:
- Equation 4 and its description are confusing. The description mentions that c is the scale factor, and \alpha = c*(1 - 0.05*i), where I presume that (1 - 0.05*i) tends to zero as the training progresses. Then it says that the scaling factor decays to zero as the probability of immorality increases (it isn't specified whether this is within one learning cycle or across the entire training run). I understand that m(c_i, a_i) goes to zero as the probability of immorality increases, but m(c_i, a_i) is different from the scaling factor, right? This whole paragraph is confusing. I get the overall idea that a policy network is being trained with imitation learning on successful buffer examples weighted by the conscience, but the details need to be made more clear.
- Figure 3 is somewhat poorly described in the text. How is the figure generated?

Minor points:
- The first paragraph of the introduction is more like related work
- In Figure 1's caption, it may be helpful to the reader to mention that the house that the agent is in does not belong to them.
- Typo: "use the CALM" should be "use CALM"
- Typo: "repeated learning circles" should probably be "repeated learning cycles"


Novelty:
There is some novelty with the proposed approach. It's a meaningfully different approach than CMPS or GALAD. Imo the main contribution is the improved results, though (backed up by ablations, although see my point about only 15 environments).


Reproducibility:
The authors say they will release code. The method does not seem reproducible from the description in the paper alone, which could be improved.
summary_of_the_review: I think this is a good paper, but it feels like it needs more polishing. If the authors address most of my concerns listed above, I would be happy to recommend acceptance.

---------------------
Update after rebuttal:

The authors have addressed all of my concerns about how polished the original submission was. After checking the updated paper and the other reviews, I will now recommend acceptance. This is primarily because of the novelty of moving the morality prior into the GPT-2 language model (including fine-tuning it during RL training), the meaningful improvements to the RI metric, and the ablations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors focus on studying the morality of RL agents on text-based adventure games. In such games, due to many reasons, agents often resort to behaviours that are identified as immoral, e.g., stealing things, attacking NPCs or monsters even unnecessary. 

To mitigate this issue, the authors propose a novel iterative training framework, namely MorAL. Every iteration consists of a task learning phase and a morality learning phase. During task learning, the task policy conditions on state information selects its action from a list of action candidate that is generated by the (fixed) morality policy. Meanwhile, high-quality trajectories are pushed into a replay buffer. During the morality learning phase, a commonsense prior is utilized to score the transitions in the buffer, the morality policy is subsequently updated to generate action candidates that have higher morality scores. 

Comparing against a set of top performing text-based game systems, on a set of 15 text-based games, the author show that the proposed MorAL framework indeed decreases the average immorality scores, and at the same time slightly increased the game completion percentage. 
strength_and_weaknesses: ### Strength:

* **Multi-disciplinary & social impact**: This work brings some important ideas from another community (FAccT) to text-based games, which makes this study multi-disciplinary. A successful agent needs skills from RL, NLP, social science and more. I'd love to see work on this direction, which can have broader social impact than the specific task used in the paper. 
* **Two-step learning**: The proposed MorAL pipeline makes sense to me, experiment results suggest that the two step training procedure indeed helps the agent performing well on both game playing and morality sides. Note the framework is not necessarily limited to only the morality learning, in principle, the second step can leverage any metric of interest, depending on the task. 
* **Game completion improvement**: Without considering the morality dimension, the ablation study (Table 2) suggests that the self-imitation learning can quite significantly improve the average game completion percentage. 


### Weaknesses:

Please see my questions and concerns below. 
clarity,_quality,_novelty_and_reproducibility: ### Questions and concerns
1. As my main question, I want to hear more from the authors regarding the motivation of this research direction. Object collection and combat are probably at the core of many text adventure games, or any games in general. Many games are designed in a way that the players/agents have to follow some pre-defined storyline, or to reach some key points, in order to proceed the story forward. I fully understand and agree the necessity of having morality as an important evaluation dimension in sequential decision making problems, but are existing games the best place to start with?
2. Related to the previous question, I also want to grab the authors' thoughts on, if designing new games (let's say, text-based adventure games), what are some ways to put this morality dimension systematically in the designing process? For instance, how to explicitly model/measure the trade-off between morality and task progress, and how to evaluate morality. One possibility is, as the authors briefly discussed in the paper, to take social aspects into consideration in building tasks/games. On a related note, in [1], they proposed a set of minimal tasks that require agents to perform certain social interactions as part of the skills solving some tasks. This can potentially be used in text-based game design, in a way that the agent needs to borrow a lantern from an NPC (and later return it!).
3. Regarding human expert immorality scores in Table 1. Are they the accumulated scores until the end of the game, or until the same step budget as given to the agent? Would it make more sense if computing human immorality scores until the step where MorAL ends each game?
4. To me it was a bit confusing at the beginning of reading, I thought the morality training might have led to higher task completion scores --- which is actually not the case. The task completion boost might rather come from a better action candidate generator. Because the majority of the paper discuss the effects of morality learning, so the other part of the contribution (the boost on task completion scores) might not be as clear. To me they are both important and worth some emphasizing.


### Minor questions
1. In Section 5.4, the authors provide the value of $\lambda = 0.14$, this seems require some hyper parameter tuning. What was the hyper-param search space?
2. How much training speed does the extra modality learning phase sacrifice?


### References:
[1] [SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents. Kovač et al., 2021](https://arxiv.org/abs/2107.00956)
summary_of_the_review: In general I like this work, the idea of introducing FAccT considerations into machine learning tasks is great; the two-step framework is neat, results suggest that it's working as expected. However, I am a bit hesitant because I am not fully convinced if existing games are the best platform for such work. For now, I would rate it above borderline, but I definitely look forward to reading the authors' responses and other reviewers' comments so I can better understand the work and thus have a more precise evaluation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new method (MorAL) for training text-based agents to minimize immoral actions. MorAL is an extension of CALM, adding an imitation learning procedure that is weighted for immoral actions. An advantage of this new method is that it does not need dense human feedback during training. Its results are good with a boost of 19% in game completion rate while decreasing the immorality score by 5% compared to CMPS (the previous SOTA on of environments of the paper).

The discussion addressed most of the reviewers concerns. During the reviewer-AC meeting, some of the reviewers still found the work to be relatively "incremental" (extension of CALM) and its technical applicability to other tasks or domain is relatively unclear (as the work is applied to one benchmark). However, the reviewers also noted other benefits of this work, e.g., that the problem of training agents to reduce immoral behaviors is an important one. Results are strong and the paper presents a comprehensive ablation of the different components of MorAL and their contributions.

As this is a solid paper with applicability to responsible AI, I think the pros outweigh the cons and recommend accept. 

Note: If the paper is accepted, I would recommend that the authors add plots akin to Figure 3, but applied to the main results of the paper (Table 1 and 2) as suggested by one of the reviewers. This would help better understand the tradeoff between task completion and immorality.
justification_for_why_not_higher_score: The reviewers all agreed that the work is relatively incremental, and the scope of the paper is relatively limited (one benchmark).
justification_for_why_not_lower_score: Solid paper with meaningful technical contributions, and it has applications to responsible AI.
Decision: Accept: poster