SlowGuess/NAACL_2022_OCR huggingface dataset:
Prithviraj Ammanabrolu† Liwei Jiang‡† Maarten Sap†
Hannaneh Hajishirzi†
Yejin Choi†
†Allen Institute for Artificial Intelligence ‡University of Washington
{raja,maartens}@allenai.org
{lwjiang,hannaneh,yejin}@cs.washington.edu
We focus on creating agents that act in alignment with socially beneficial norms and values in interactive narratives or text-based games—environments wherein an agent perceives and interacts with a world through natural language. Such interactive agents are often trained via reinforcement learning to optimize task performance, even when such rewards may lead to agent behaviors that violate societal norms—causing harm either to the agent itself or other entities in the environment. Social value alignment refers to creating agents whose behaviors conform to expected moral and social norms for a given context and group of people—in our case, it means agents that behave in a manner that is less harmful and more beneficial for themselves and others.
We build on the Jiminy Cricket benchmark (Hendrycks et al., 2021b), a set of 25 annotated interactive narratives containing thousands of morally salient scenarios covering everything from theft and bodily harm to altruism. We introduce the GALAD (Game-value Alignment through Action Distillation) agent that uses the social commonsense knowledge present in specially trained language models to contextually restrict its action space to only those actions that are aligned with socially beneficial values. An experimental study shows that the GALAD agent makes decisions efficiently enough to improve state-of-the-art task performance by  $4\%$  while reducing the frequency of socially harmful behaviors by  $25\%$  compared to strong contemporary value alignment approaches.
An inherent difficulty in designing and training AI agents lies in simultaneously ensuring that agents are competent at a given task while following socially beneficial behaviors (Nahian et al., 2020; Hendrycks et al., 2021b). Such agents—particularly those trained via reinforce
Figure 1: Excerpt from the game Zork1 in the Jiminy Cricket benchmark where the agent breaks into someone's house and proceeds to steal items and the corresponding value annotations and game rewards. Note the conflicting valence of the two rewards when the agent is in the Kitchen.
ment learning (RL) in sequential decision making environments—are prone to learning behaviors harmful to themselves and their surroundings due to optimal task performance being often misaligned with socially beneficial human values (Moor, 2006; Soares and Fallenstein, 2017; Russell, 2019). Further, despite work showing the need for dataset bias analysis in supervised settings (Gebru et al., 2018), most reinforcement learning benchmarks do not provide equivalent analysis regarding their reward functions—making harmful agent behaviors difficult to diagnose (Gilbert et al., 2022). Fig. 1 shows an example of such misalignment in the context of text based games—long puzzles or quests where an agent perceives and interacts with the world
Human Language Technologies, pages 5994 - 6017
through incomplete English language descriptions.
We focus on the task of socially beneficial value alignment, a subset of value alignment concerned with creating agents that better conform to the expected social norms and values of a given group of people in a specific context. In particular, this refers to agents that act in a manner that reduces harm to both themselves and surrounding entities. We propose to do this by using the social commonsense knowledge present in specially trained large language models (Jiang et al., 2021) to contextually constrain an agent's actions to those that align with these expectations. Evaluating the relative social harmfulness of such agents requires us to focus not only agent design but also on the contexts, or environments, in which they operate.
As such, we build on the Jiminy Cricket benchmark (Hendrycks et al., 2021b), a set of 25 text-based adventure games containing annotations regarding what constitutes socially beneficial behavior in thousands of grounded and morally salient scenarios. They are structured as long puzzles and quests that require agents to reason about over 2000 locations, hundreds of characters, and nearly 5,000 objects over hundreds of steps, creating chains of dependencies that an agent must fulfill to complete the overall task. Contained within these quests are a diverse range of morally salient scenarios covering everything from bodily harm and theft, to altruism and other positive human experiences.
Given the complexity of these scenarios, variations in what is perceived as socially beneficial behavior in a particular context may vary greatly depending on the group of people judging the situation. We conduct a relatively large scale human participant study to better understand these variations. Participants are presented with transcripts of these scenarios—similar to what is seen in Fig. 1—and asked to determine how they perceive scenario's relative moral valence and salience. As noted by Hendrycks et al. (2021b), requiring such dense human feedback for training purposes is unrealistic in most sequential decision making environments and is thus used only for evaluation.
Keeping this in mind, we introduce the GALAD (Game-value Alignment through
Action Distillation $^{2}$  agent that learns to inherently constrain its action space to only actions that align with socially beneficial human values even before it ever begins training a policy for a task. We use the social commonsense norms encoded in specially trained transformer-based language models (Jiang et al., 2021) to endow our agent with the ability to contextually distinguish between socially beneficial and harmful behaviors. An experimental study shows that translating these sources of commonsense knowledge through distillation into soft constraints on the action space results in an agent that aligns more closely to social values when compared to popular contemporary policy and reward shaping based value alignment approaches while improving state-of-the-art task performance.
In short, our contributions are threefold, we provide: (1) a broad human participant study to verify the moral valence and salience of scenarios in the Jiminy Cricket environment; (2) the GALAD agent which constrains its action space using social commonsense knowledge encoded in large scale language models; and (3) an experimental study that showcases the relative importance of action distillation when compared to reward and policy shaping methods in value aligning RL agents.
Text game playing. Recent works in this area have focused on tackling three primary challenges: (1) how to represent agent knowledge to effectively operate in partially observable environments (Adhikari et al., 2020; Sautier et al., 2020); (2) scaling RL algorithms to handle combinatorial natural language state-action spaces (Zahavy et al., 2018; Yao et al., 2020; Ammanabrolu et al., 2021; Jang et al., 2021); and (3) giving agents commonsense priors to better reason about the world (Murugesan et al., 2020, 2021; Dambekodi et al., 2020; Ammanabrolu and Riedl, 2021a). All of these works focus exclusively on improving task performance, often in the form of increasing overall game completion rates, and do not analyze agent behaviors.
Value alignment and safe RL. Value alignment is often defined as a property of intelligent agents that biases them towards acting in a manner that is similar to a human in a given situation (Bostrom,
<sup>1</sup>We note that prior works in AI value alignment refer to this as normative (Nahian et al., 2021) or moral (Hendrycks et al., 2021b) value alignment. As our work focuses on imbuing agents with social commonsense knowledge, it is more accurate to refer to it as social value alignment.
Inspired by both the noble Arthurian Knight of the Round Table Sir Galahad (originally spelled Galaad) and the honorable elven king Gil-Galad from Lord of the Rings—Galad itself meaning "radiant" in the fictional Sindarin language.
2014). Value alignment and safe exploration for RL agents are usually cast as constrained optimization problems wherein an agent attempts to learn a policy for a task while adhering to a given set of constraints (García et al., 2015; Leike et al., 2017; Achiam et al., 2017; Tessler et al., 2019; Ray et al., 2019). Contemporary approaches often rely on imitating expert demonstrations to learn safe trajectories (Gao et al., 2018) or by modeling rewards that best fit human values (Hadfield-Menell et al., 2016; Reddy et al., 2020).
Closest in spirit to our work are those of Nahian et al. (2021) and Hendrycks et al. (2021b), the latter being the work which we build on. Both of these works introduce text game based environments with inherent morally salient scenarios for an agent to reason about—Nahian et al. (2021) building on the procedurally generated TextWorld framework (Côté et al., 2018) and Hendrycks et al. (2021b) borrowing from human-made games in the Jericho benchmark (Hausknecht et al., 2020). Both further design value-aligned agents that use priors regarding socially beneficial behavior learned by training language models on domain specific datasets such as stories (Nahian et al., 2020) or ethical understanding benchmarks (Hendrycks et al., 2021a) to perform reward and policy shaping (Sutton and Barto, 1998; Griffith et al., 2013). In contrast, GALAD distills social commonsense information regarding socially beneficial behaviors into contextual knowledge about what actions to not take in a given state before policy learning begins.
3 Environment Alignment Verification
The Jiminy Cricket benchmark (Hendrycks et al., 2021b) is a set of 25 text adventure games—drawn from the Jericho suite (Hausknecht et al., 2020)—that contains morality annotations for possible action-outcome in a particular world state. The original annotation framework used in Hendrycks et al. (2021b) contains two primary dimensions with two options apiece: (1) valence - does the action constitute socially beneficial behavior as defined by the annotation rubric; and (2) target - does this action affect the agent itself or others. For each of these categories, a further severity level is assigned between 1-3 to better account for variations in the relative seriousness of a situation. This results in 12 possible annotation labels. The original annotations are made pro tanto at a source code level, i.e. annotations do not consider how
the world has changed until that particular state (e.g. breaking into someone else's house is always bad regardless of context). Further details on the original framework drawn from Hendrycks et al. (2021b) are provided in Appendix A.1.
Text games from the Jericho benchmark (and subsequently Jiminy Cricket) provide semantically rich, complex environments to study value alignment. This, along with the inherent quest-like structure of such games, provides thousands of potentially morally ambiguous scenarios for an agent to reason through—many of which contain game rewards that can easily encourage socially unacceptable behaviors as seen in Fig. 1. The relative complexity and common fantastical elements present in such environments, however, does significantly complicate the process of annotating the relative harmfulness of an action.
To complement the source code level annotations in Jiminy Cricket, we perform a human evaluation of actions in context through crowdsourcing. We randomly select  $n = 210$  world states with moral source code annotations, along with their 15 preceding turns in a game played by an oracle agent. Using a pool of trained Amazon MTurk workers, we collect 5 annotations per game snippet of moral saliency of the agent's actions, as well as valence, target, and severity using the same definitions as Jiminy Cricket. We refer the reader to Appendix A.1.1 for further details on the annotation setup.
According to our workers,  $81\%$  of the situations with source code level morality annotations were deemed morally salient. We find that workers agreed with the valence of the source code annotations  $67\%$  of the time, and with both the valence and target  $50\%$  of the time. However, our workers matched the exact valence, target, and severity annotations only  $24\%$  of the time. Finally, as discussed in greater detail Appendix A.1.2, our own workers exhibited variation in their annotations, showing moderate agreement on annotations of valence (83% pairwise agreement, Krippendorf's  $\alpha = 0.30$ ) and valence and polarity (70% pairwise agreement,  $\alpha = 0.30$ ). While these results highlight that most source-level scenarios marked as morally salient pro tanto are also morally salient in context, they suggest that as source level annotations become more fine grained, they become noisier proxies for determining in context social acceptability. In our work, we therefore do not consider the target and severity dimensions of the
Figure 2: Overview of the action distillation process in GALAD.
annotations and use only the high level annotations regarding the moral valence—having relatively high annotator agreement—of a situation to judge whether an agent has behaved acceptably.
4 Value Alignment via Action Distillation
This section covers different parts of GALAD, first describing the pre-training required to create social commonsense models and then detailing how an RL agent uses them while exploring the game world. GALAD has three primary components: (1) Section 4.1 explains the value prior, a large language model specially trained with knowledge of commonsense morality; (2) Section 4.2 showcases a relatively small action candidate generator which learns to contextually generate less socially harmful actions by distilling the knowledge present in the value prior as seen in Fig. 2; and (3) Section 4.3 combines all the parts, describing the overall RL policy network as seen in Fig. 3 which uses the action candidate generators to generate (socially acceptable) candidate actions that are sampled from during exploration.
We first formally define interactive narratives, building on the definition seen in Côté et al. (2018) and Hausknecht et al. (2020), accounting for the objective of socially beneficial value alignment. In our setting, text games are Partially-Observable Markov Decision Processes (POMDP) defined as an 8-tuple of  $\langle S, P, A, O, \Omega, R, \chi, \gamma \rangle$ , representing: the set of environment states  $(S)$ , conditional transition probabilities between states  $(P)$ , the vocabulary or words used to compose text commands  $(A)$ , observations  $(O)$ , the conditional probability of receiving an observation  $(\Omega)$ , reward function  $(R)$ , value alignment scoring function  $(\chi)$ , and discount factor  $(\gamma)$ , respectively.
4.1 Value Priors from Social Commonsense
Our value prior is based on Delphi (Jiang et al., 2021), a moral reasoning model taught with 1.7M instances of publicly polled declarative knowledge of what's ethically acceptable or not in everyday situations. It is fine-tuned from UNICORN, a universal commonsense reasoning model derived from T5-11B, the largest T5 model with 11 billion parameters (Raffel et al., 2020).
Datasets and Training. Delphi is trained on COMMONSENSE NORM BANK, a knowledge repository of everyday ethics, sourced from five existing large-scale datasets, including SOCIAL CHEMISTRY (Forbes et al., 2020), ETHICS Commonsense Morality (Hendrycks et al., 2021a), MORAL STORIES (Emelin et al., 2021), SOCIAL BIAS INFERENCE CORPUS (Sap et al., 2020), and SCRUPLES (Lourie et al., 2021). COMMONSENSE NORM BANK contains diverse set of descriptive social, cultural and norms grounded in complex real world situations. The version of Delphi used for our work is trained via a question answering task that infers the ethical judgment regarding a text description of an everyday situation (e.g., "killing a bear" is "wrong"). We use this model to generate social acceptability scores (i.e., probability scores among "positive," "neutral" and "negative" categories) for given game actions and context.
4.2 Action Distillation
Requiring the use of multi-billion parameter language models for value feedback at every single step is computationally infeasible, especially given the large number of environment interactions that are often required to make progress in sequential decision making environments. Further, exhaustive search by exploring every single possible action is similarly infeasible given the combinatorially sized
action spaces—a text game of average difficulty such as Zorkl has  $\mathcal{O}(10^{14})$  possible actions per step. GALAD thus uses an action candidate generator that takes the current state of the world into account and produces a limited set of contextually relevant action candidates. The GALAD action candidate generator used in this work is a  $117\mathrm{m}$  parameter autoregressive language model (100x smaller than the value prior) with the architecture of GPT-2 (Radford et al., 2019). Fig. 2 provides an overview of this entire action distillation process.
Datasets. We use two offline datasets designed to help agents produce contextually relevant actions. The first dataset is the ClubFloyd dataset introduced by Yao et al. (2020) and Ammanabrolu and Hausknecht (2020). It contains transcripts of human playthroughs of over 500 games in the form of alternating observations and actions. The second dataset is the JerichoWorld dataset introduced by Ammanabrolu and Riedl (2021b). It consists of similar samples from 31 games but collected through the use of oracle agents that can play a game perfectly. Both datasets are processed into (context, action) pairs with context being the observations from two subsequent steps  $c_{i} = (o_{i-1}, o_{i}) \forall i \in \{1 \dots n\}$  and corresponding action being  $a_{i}$ . We note that we have taken additional precautions via data curation to ensure that there is no overlap between the games in the Jiminy Cricket benchmark and the games used to collect data in either of these datasets.
Training. The autoregressive action candidate generator is trained on this combined dataset via a modified sequence-to-sequence strategy, with the language model being trained to produce action  $a_{i}$  given context  $c_{i}$ . Standard autoregressive techniques factor the distribution over the tokens  $k$  in the target sequence of length  $M$  into a chain of conditional probabilities with a left to right structure.
$$
P \left(a _ {i} \mid c _ {i}; \theta\right) = \prod_ {k = 1} ^ {M + 1} P \left(a _ {i, k} \mid a _ {i, 0: k - 1}, c _ {i}; \theta\right) \tag {1}
$$
Where  $\theta$  represents the overall action candidate generator network parameters and each action  $a_{i}$  consists of a sequence of tokens  $a_{i,k}$ . This can then be used to formulate a maximum likelihood training objective with cross-entropy for each individual
Figure 3: An overview of RL training for the agent. Policy network components are learned during RL training and the action candidate generator is pre-trained.
$$
\begin{array}{l} \mathcal {L} _ {s e q} = - \log P \left(a _ {i} \mid c _ {i}; \theta\right) (2) \\ = - \sum_ {k = 1} ^ {M + 1} \log P \left(a _ {i, k} \mid a _ {i, 0: k - 1}, c _ {i}; \theta\right) (3) \\ \end{array}
$$
As seen in Fig. 2, each of the data samples is fed into the value prior language model to measure the relative contextual morality of performing that particular action given the context. The model outputs a probability distribution over three labels corresponding to whether the action corresponds to good, neutral, or bad behavior given the specific context of that scenario. Each of the samples used to train the action candidate generator is scaled by these values to bias it towards taking actions that represent less harmful behaviors by learning which action to not perform. This gives an action distillation loss:
$$
\mathcal {L} _ {A D} = \lambda (1 - P (b a d | c _ {i}, a _ {i}; \theta^ {\prime})) \mathcal {L} _ {s e q} \tag {4}
$$
Where  $\theta^{\prime}$  represents the parameters of the value prior and  $\lambda$  is a scaling weight hyperparameter.
4.3 RL Policy Training
The overall GALAD agent, as seen in Fig. 3, is trained inspired by the Deep Reinforcement Relevance Network (DRRN) (He et al., 2016; Yao et al., 2020) used for natural language state-action spaces. It uses  $Q$ -Learning (Watkins and Dayan, 1992) to learn a control policy  $\pi(a_{t}|c_{t})$ ,  $a_{t} \in A$  that directly maximizes long horizon expectation over  $R$  and implicitly minimizes expectation over  $\chi$ .
At each game step  $t$ , the environment provides context consisting of the last two observations
$c_{t} = (o_{t - 1},o_{t})$  to both the action candidate generator and the policy network. The action candidate generator autoregressively decodes a set of  $j$  action candidates  $\mathcal{A}_t = \{a_t^{(0)}\dots a_t^{(j)}\}$  through nucleus sampling (Holtzman et al., 2020) given the context. These actions and also the context are passed into two separate encoders in the policy network. These encoders are recurrent GRU networks whose hidden state is initialized from the final hidden state in step  $t - 1$ . Actions and context are encoded into vectorized representations  $\mathbf{c}_{\mathbf{t}}$ ,  $\mathbf{a}_{\mathbf{t}}^{(\mathbf{i})}\forall i\in 0\dots j$ . An additional joint encoder takes  $\mathbf{c}_{\mathbf{t}}$ ,  $\mathbf{a}_{\mathbf{t}}^{(\mathbf{i})}$  and learns a  $Q$ -value function  $Q(\mathbf{c}_{\mathbf{t}},\mathbf{a}_{\mathbf{t}}^{(\mathbf{i})})$  representing the relative utility of performing action  $a_{t}^{(i)}$  given  $c_{t}$  to maximize long term expected reward. Softmax sampling is applied to these  $Q$ -values to pick the action  $a_{t}$  that is then executed in the game. The network is trained using experience replay (Lin, 1993) with prioritized sampling (Moore and Atkeson, 1993) for experiences with positive reward. Experiences in the form of tuples of  $\langle c_t,a_t,r_t,c_{t + 1}\rangle$  collected during training are stored in a replay buffer and then minibatches are priority sampled to calculate temporal difference (TD) loss:
$$
\begin{array}{l} \mathcal {L} _ {T D} (\phi) = r _ {t + 1} + \\ \gamma \max  _ {\mathbf {a} \in \mathcal {A} _ {t + 1}} Q (\mathbf {c} _ {\mathbf {t}}, \mathbf {a}; \phi) - Q (\mathbf {c} _ {\mathbf {t}}, \mathbf {a} _ {\mathbf {t}}; \phi) \tag {5} \\ \end{array}
$$
Where  $\phi$  represents the parameters of the policy network. Further training details and hyperparameters are found in Appendix A.3.
We conduct two evaluations to compare GALAD to other contemporary agents and a secondary ablation study to identify the relative importance of the components of GALAD.
Metrics. We base performance on three primary metrics: (1) Harmfulness score which is returned by the annotated Jiminy Cricket environment and verified by the human participant study in Section 3 as a measure of how harmful an agent's actions were for both itself and others; (2) Completion percentage which measures the overall task performance of an agent by way of looking at collected game score as a proportion of the maximum score; and (3) Relative harmfulness which is defined as Harmfulness score/Completion percentage to account for the fact that agents that complete more
of the games will encounter more scenarios where they might potentially act unacceptably.
As the human participant study in Sec. 3 suggests that the more fine-grained annotations in the benchmark relating to severity might be prone to being more noisy, we modify our metrics from Hendrycks et al. (2021b) and do not consider labels associated with the relative severity of harm and only consider the more high-level annotations pertaining to valence (good or bad). Unadjusted results calculated taking severity also into account are found in Appendix A.4. No trends discussed change due to this adjustment.