SlowGuess/NAACL_2022_OCR huggingface dataset:
We follow the overall experimental setup of Hendrycks et al. (2021b) for fair comparison, testing each agent at 5 evenly spaced starting locations in each of the 25 games in the Jiminy Cricket benchmark—resulting in 125 total environments. Additionally, we run each individual experiment with 5 random seeds and report averaged results and standard deviations. Number of training steps, number of parallel environments for experience collection, and parameter size of both the action candidate generator and policy network are held constant across all agents we test. Further details are found in Appendix A.3.
5.1 Comparison to Baselines
We compare to recent, state-of-the-art, text game works that create agents belonging to the same class of agents as GALAD as defined by Hausknecht et al. (2020)—i.e. none of these agents have the commonly used valid action handicap that gets the set of ground truth contextually relevant action candidates at each step during training.
NAIL is a heuristic rules-based agent created by Hausknecht et al. (2019) to function as a general text game playing agent.
CALM is developed by Yao et al. (2020), this agent only uses the ClubFloyd dataset to train its action candidate generator without any value prior and uses the DRRN (He et al., 2016) architecture otherwise.
CMPS is the best performing baseline agent provided in the Jiminy Cricket benchmark by Hendrycks et al. (2021b). It is identical to the CALM agent but uses an additional RoBERTa (Liu et al., 2019) morality model trained on the commonsense portions of the ETHICS dataset (Hendrycks et al., 2021a) to perform policy shaping.
$\mathbf{CMPS}+$  is an enhanced version of CMPS. For a
Game/Agent	NAIL	CALM	CMPS	CMPS+	GALAD	NAL	CALM	CMPS	CMPS+	GALAD
Metric	Harmfulness (↓)	Completion (↑)
ballyhoo	2.97	2.58	2.21	2.9	1.46	0.33	2.47	1.18	2.19	1.61
borderzone	1.89	1.91	1.4	2.2	1.7	1.38	3.55	3.33	3.7	3.47
cutthroats	0.74	1.4	1.21	1.18	1.12	4.21	3.94	3.8	3.59	3.79
deadline	4.51	3.44	3.49	4.37	1.8	0.76	1.55	1.56	1.68	1.82
enchanter	0.34	0.58	0.34	0.51	0.37	0.03	1.79	1.72	3.57	3.22
hitchhiker	2.5	2.7	1.63	2.79	1.12	0	7.94	7.21	10.53	9.98
hollywoodhijinx	2.13	0.88	0.54	1.06	0.69	0.33	1.66	1.76	1.56	1.84
infidel	0.32	0.16	0.08	0.14	0.08	0.12	0.38	0.38	0.38	0.38
lurkinghorror	3.58	2.61	0.88	2.42	1.54	0	0.39	0.81	0.32	0.3
moonmist	4.05	6.63	1.77	5.95	1.68	7.09	9.26	9.28	8.21	10.85
planetfall	0.87	2.95	2.4	3.37	1.94	0.51	1.58	1.3	1.59	2.16
plunderedhearts	2.87	3.23	1.94	3.63	1.64	0.95	2.67	2.81	2.8	3.21
seastalker	5.72	1.83	1.94	3.81	2.01	0.96	3.37	4.41	3.88	4.41
sorcerer	1.29	0.58	0.57	0.4	0.15	0.54	2.6	2.6	2.62	1.8
spellbreaker	1.11	0.92	0.58	0.93	0.87	0.64	3.39	3.4	3.39	3.29
starcross	1.49	8.16	1.07	1.23	1.28	-1.67	-0.09	-0.06	-0.07	1.3
stationfall	2.89	0.64	0.42	0.59	0.44	0.7	0.31	0.22	0.27	0.43
suspect	3.58	4.13	1.73	4.43	2.09	3.51	5.06	4.27	4.75	4.36
suspended	9.77	2.5	3.28	2.44	0.93	-1.66	-0.67	-0.76	-0.36	-0.74
trinity	4.89	1.82	2.12	2.09	1.92	0.06	1.58	1.59	1.54	1.61
wishbringer	3.69	1.88	1.53	0.96	1.35	0.29	5.04	5.09	5.04	5.21
witness	2.14	1.34	1.36	1.24	1.49	2.83	9.22	8.58	9.22	9.91
zork1	1.48	3.48	2.45	2.8	1.02	-2.4	5.32	5.07	5.32	5.22
zork2	2.13	1.31	1.24	1.4	1.6	-2.49	2.54	4.01	2.54	2.38
zork3	1.57	1.11	0.84	1.14	1.19	5.22	12.19	11.08	12.19	11.96
Average	2.74	2.35	1.48	2.16	1.26	0.89	3.48	3.39	3.62	3.76
Relative	3.08	0.68	0.44	0.6	0.33	-	-	-	-	-
Table 1: Normalized harmfulness and completion percentages across the baselines. Each individual experiment is averaged across 5 independent runs and the last 50 episodes. Bolded results represent a statistically significant  $(p < 0.05)$  difference in means with a one-way ANOVA test followed by a post-hoc Tukey test. Standard deviations do not exceed 0.1 for any single agent's overall performance. GALAD represents a relative  $25\%$  reduction in harmful behavior and  $4\%$  task performance increase over separate next best baselines.
Figure 4: Performance trade-off curves averaged across 5 runs showing the relative accumulation of harmfulness score as an agent encounters and completes scenarios across all the games for select baselines. Shaded regions indicate one standard deviation.
more fair comparison given our use of additional datasets, we provide results for CMPS+ that uses an action candidate generator trained on both of the datasets shown in Sec. 4.2, without distillation.
Table 1 and Fig. 4 outline the results for this evaluation. The main trend to note here is that for most of the baselines, completion rates and socially harmful behavior appear to be directly proportional to each other—i.e. the more an agent explores,
the more chances it has to accumulate harmfulness score. In particular, we see this when comparing the results of CMPS and  $\mathrm{CMPS + }$ ,  $\mathrm{CMPS + }$  uses data better suited to predicting all possible valid actions for a given state to train its action candidate generator and so achieves  $6.3\%$  higher relative completion rate than CMPS. This comes at the expense of effectively aligning its behavior and it performs actions that are deemed harmful  $31.4\%$  more than CMPS—implying that the data used to train action candidate generators contain a bias that skews agents towards harmful behaviors.
In contrast, GALAD shows a  $4\%$  greater completion rate than the next best agent (CMPS+) while simultaneously reducing harmful behavior by  $25\%$  when compared to the next best agent for the metric (CMPS)—with over  $50\%$  more socially beneficial agent behaviors being shown in some of the more realistic games such ballyhoo, suspect and up to  $10x$  in more fantasy themed games like starcross, suspended. This indicates that distillation from large language model based value priors is an effective way to translate social commonsense knowledge into value aligned agent actions without sacrificing competency. Qualitative examples of GALAD's behavior are found in Appendix A.5.
Expt.	GALAD- +RS	GALAD +PS	GALAD +Oracle	GALAD +Oracle	GALAD- +RS	GALAD +PS	GALAD +Oracle
Metric	Harmfulness (↓)	Completion (↑)
ballyhoo	2.73	0.75	2.04	1.46	0.08	2.45	1.6
borderzone	2.38	1.17	1.98	1.7	0.9	3.74	2.6
cutthroats	0.88	1.14	0.96	1.12	0.1	3.8	2.13
deadline	2.27	0.83	1.38	1.8	0.15	0.73	1.55
enchanter	0.43	0.57	0.29	0.37	0.31	3.22	3.24
hitchhiker	2.95	2.19	0.75	1.12	0.15	6.94	2.27
hollywoodhijinx	1.33	1.48	0.56	0.69	0.07	1.32	1.9
infidel	0.17	0.13	0.09	0.08	0.11	0.38	0.38
lurkinghorror	3.65	3.16	1.65	1.54	2.22	0.43	0.27
moonmist	4.41	1.34	2.53	1.68	0.94	7.48	5.46
planetfall	4.41	1.02	1.91	1.94	0.65	1.83	2.69
plunderedhearts	2.75	1.26	2.5	1.64	0.01	2.58	3.33
seastalker	3.19	2.24	2.19	2.01	0.3	2.61	0.62
sorcerer	0.62	1.31	0.67	0.15	0.01	2.52	0.38
spellbreaker	0.98	1.19	0.64	0.87	0.02	3.33	0.8
starcross	1.69	1.41	1.49	1.28	0.24	-0.03	1.11
stationfall	0.82	0.68	0.42	0.44	0.15	0.09	0.15
suspect	3.61	2.08	1.72	2.09	0.37	6.28	1.7
suspended	0.86	2.21	0.82	0.93	0.96	-0.62	1.52
trinity	2.04	1.28	1.22	1.92	0.15	1.58	5.4
wishbringer	2.03	1.31	1.58	1.35	0.44	4.97	3.73
witness	1.5	2.86	1.17	1.49	0.17	10.16	3.03
zork1	4.19	3.76	1.77	1.02	1.72	5.12	1.89
zork2	3.43	1.49	1.45	1.6	0.3	2.15	0.81
zork3	1.59	1.42	1.44	1.19	0.79	14.21	12.65
Average	2.2	1.53	1.33	1.26	0.45	3.49	2.45
Relative	0.63	0.63	0.38	0.33	0.13	-	-
Table 2: Normalized harmfulness and completion across the ablations. Methodology is similar to Table 1.
We test four variations of GALAD to analyze the behavior and performance of our agent.
GALAD-. To better understand the relative trade-offs between optimizing for task performance and socially beneficial behaviors in this environment, we negate the valence seen in Eq. 4 to encourage the action candidate generator to take actions perceived by the value prior as being socially unacceptable.
GALAD+PS. We use a smaller version of the value prior used by GALAD—trained similarly but with less than 1 billion parameters—to perform policy shaping in a manner similar to CMPS.
GALAD+RS. We use the same value prior as given above to perform reward shaping by subtracting from the reward given to the agent at each step by a factor proportional to how socially unacceptable an agent's behavior is perceived as.
GALAD+Oracle. We train GALAD by using the dense, ground-truth harmfulness score feedback returned by the Jiminy Cricket benchmark to perform policy shaping for the agent similarly to CMPS—solely for analysis purposes as an upper bound on harmfulness.
Table 2 shows the results of this evaluation. There are two major trends to note in this table. First of all, GALAD performs best when no additional constraints are placed during policy network training through reward or policy shaping.
Reward and policy shaping techniques are popular approaches to value alignment but our results indicate that they pose difficult dual optimization problems—prone to incurring more noise during policy training—and that an easier way to align agents is to pre-train them to contextually learn what actions to not take in a scenario. Adding in the Oracle as an upper bound to provide dense feedback significantly drops harmfulness, suggesting that there is ample room for improvement.
Further, we note that GALAD-, an agent trained to behave unacceptably, performs worse than GALAD, GALAD+PS, CMPS, and CMPS+ across all metrics. From this we conclude that there is not always a direct trade-off between acting harmfully and task performance, sometimes acting altruistically in these environments is necessary to improve task performance—i.e. when in doubt, defaulting to socially beneficial behavior is more effective than defaulting to socially unacceptable behavior.
Modern testbeds for developing intelligent interactive agents often contain incentive structures that can bias agents towards acting in ways that are harmful both for themselves and towards the environment and entities within. Value alignment is often seen as being directly at odds with task performance, i.e. assuring altruistic behavior requires a proportional sacrifice in general task competency.
This is a consequence of the fact that many value alignment approaches are based on reward or policy shaping—trying to learn socially beneficial behavior from expert feedback—which directly places task performance and socially acceptable norms against each other in a dual optimization problem. In an attempt to encourage a greater volume of work that treats value alignment as an principal property of AI systems, we show that with careful design—in our case the GALAD system made by distilling social commonsense knowledge present in large language models to contextually learn soft constraints on what actions not to take—it is possible to create agents that act less socially harmfully with respect to themselves and other agents without loss in competency.
As mentioned, this work is an attempt to tackle the issue of creating agents that consider the relative harmfulness caused by their behaviors as a first class citizen in their design in addition to task-based rewards. Agents that simply focus on task rewards are at significantly greater risk of acting in a manner harmful to themselves and others. Text games, in particular the games in the Jiminy Cricket benchmark, provide semantically rich, grounded, and morally salient scenarios for agents to navigate through. To better understand and mitigate the inherent biases found within the games of the benchmark, we conduct a large scale human participant study to judge the relative moral valence and salience of the scenarios present in these games—attempting to verify how accurately our evaluation metrics map to values considered to be socially beneficial by this particular set of humans.
We further note that agents trained in text environments are more suited for domains in which change in the world is affected via language, which mitigates physical risks—downstream lines of work are not directly relevant to robotics—but not cognitive and emotional risks (Hausknecht et al., 2020). As noted earlier, any system capable of generating natural language, even within the limits of fantasy domains as seen in certain games, is capable of accidental or intentional harmful and biased language use—a property which we mitigate but do not entirely eliminate through our value prior (Sheng et al., 2019; Dinan et al., 2020). We note that we do not use this value prior to reason about these moral scenarios by itself in vacuum—
it is instead used to implicitly bias the actions of an RL agent towards actions that are deemed to be more aligned with socially beneficial behaviors grounded in a particular context within the dynamics of the environment.
We acknowledge that the data used to train the value prior and the games themselves in addition to the added annotations heavily skew towards Western cultural and social norms. Further, despite attempts at data curation and downstream task verification, it is possible that the values encoded within such language model priors represent socially unacceptable views (Bender et al., 2021). We thus emphasize that these are not universally applicable values and that agents trained in these environments cannot be used to directly assess the social acceptability of human made actions in more real world scenarios. This work presents a first step in attempting to reduce the implicit harms that arise from training interactive, intelligent agents to focus only on task performance in popular, contemporary sequential decision-making environments.
This work was supported in part by DARPA MCS program through NIWC Pacific (N66001-19-2-4031), Google Cloud Compute, and the Allen Institute for AI.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 22-31. PMLR.
Ashutosh Adhikari, Xingdi Yuan, Marc-Alexandre Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. 2020. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems, 33.
Prithviraj Ammanabrolu and Matthew Hausknecht. 2020. Graph Constrained Reinforcement Learning for Natural Language Action Spaces. In International Conference on Learning Representations.
Prithviraj Ammanabrolu and Mark Riedl. 2021a. Learning knowledge graph-based world models of textual environments. In Thirty-fifth Conference on Neural Information Processing Systems.
Prithviraj Ammanabrolu and Mark Riedl. 2021b. Modeling worlds in text. In Thirty-fifth Conference on
Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur Szlam, Tim Rocktäschel, and Jason Weston. 2021. How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds. In Proceedings of 2021 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Nick Bostrom. 2014. Superintelligence: Paths, dangers, strategies. Superintelligence: Paths, dangers, strategies. Oxford University Press, New York, NY, US.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. 2018. Textworld: A learning environment for text-based games. In Workshop on Computer Games, pages 41-75. Springer.
Sahith Dambekodi, Spencer Frazier, Prithviraj Ammanabrolu, and Mark O Riedl. 2020. Playing text-based games with common sense. In 2nd Wordplay: When Language Meets Games Workshop at NeurIPS.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020. Queens are powerful too: Mitigating gender bias in dialogue generation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP-20).
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In EMNLP.
Yang Gao, Huazhe Xu, Ji Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. 2018. Reinforcement learning from imperfect demonstrations. *ArXiv*, abs/1802.05313.
Javier García, Fern, and o Fernández. 2015. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford. 2018. Datasheets for datasets. arXiv preprint arXiv:1803.09010.
Thomas Krendl Gilbert, Sarah Dean, Nathan Lambert, Tom Zick, and Aaron Snoswell. 2022. Reward reports for reinforcement learning. arXiv preprint arXiv:2204.10817.
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2020. Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7903-7910.
Matthew J. Hausknecht, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D. Williams. 2019. NAIL: A general interactive fiction agent. CoRR, abs/1902.04259.
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Li hong Li, Li Deng, and Mari Ostendorf. 2016. Deep reinforcement learning with a natural language action space. In ACL.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a. Aligning {ai} with shared human values. In International Conference on Learning Representations.
Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. 2021b. What would jiminy cricket do? towards agents that behave morally. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
Youngsoo Jang, Seokin Seo, Jongmin Lee, and Kee-Eung Kim. 2021. Monte-carlo planning and learning with language action value estimates. In International Conference on Learning Representations.
Liwei Jiang, Jena D. Hwang, Chandrasekhar Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge,
Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2021. Delphi: Towards machine ethics and norms. CoRR, abs/2012.15738.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. 2017. Ai safety gridworlds. ArXiv, abs/1711.09883.
Long-Ji Lin. 1993. Reinforcement learning for robots using neural networks. Ph.D. thesis, Carnegie Mellon University.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. Scruples: A corpus of community ethical judgments on 32,000 real-life anecdotes. In AAAI.
J.H. Moor. 2006. The nature, importance, and difficulty of machine ethics. IEEE Intelligent Systems, 21(4):18-21.
Andrew W. Moore and Christopher G. Atkeson. 1993. Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time. Machine Learning.
Keerthiram Murugesan, Mattia Atzeni, Pavan Kapani-pathi, Pushkar Shukla, Sadhana Kumaravel, Gerald Tesauro, Kartik Talamadupula, Mrinmaya Sachan, and Murray Campbell. 2021. Text-based RL Agents with Commonsense Knowledge: New Challenges, Environments and Baselines. In Thirty Fifth AAAI Conference on Artificial Intelligence.
Keerthiram Murugesan, Mattia Atzeni, Pushkar Shukla, Mrinmaya Sachan, Pavan Kapanipathi, and Kartik Talamadupula. 2020. Enhancing text-based reinforcement learning agents with commonsense knowledge. arXiv preprint arXiv:2005.00811.
Md Sultan Al Nahian, Spencer Frazier, Brent Harrison, and Mark Riedl. 2021. Training value-aligned reinforcement learning agents using a normative prior. CoRR, abs/2104.09469.
Md Sultan Al Nahian, Spencer Frazier, Mark Riedl, and Brent Harrison. 2020. Learning norms from stories: A prior for value aligned agents. In AAAI/ACM Conference on AI, Ethics, and Society, page 124-130, New York, NY, USA. Association for Computing Machinery.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the
limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Alex Ray, Joshua Achiam, and Dario Amodei. 2019. Benchmarking safe exploration in deep reinforcement learning.
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. 2020. Learning human objectives by evaluating hypothetical behavior. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8020-8029. PMLR.
Stuart Russell. 2019. Artificial intelligence and the problem of control. *Perspectives on Digital Humanism*, page 19.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In ACL.
Corentin Sautier, Don Joven Agravante, and Michiaki Tatsubori. 2020. State Prediction in TextWorld with a Predicate-Logic Pointer Network Architecture. In In Workshop on Knowledge-based Reinforcement Learning at IJCAI-20.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412. Association for Computational Linguistics.
Nate Soares and Benya Fallenstein. 2017. Agent Foundations for Aligning Machine Intelligence with Human Interests: A Technical Research Agenda, pages 103-125. Machine Intelligence Research Institute (MIRI) technical report, Springer Berlin Heidelberg.
Richard S Sutton and Andrew G Barto. 1998. Introduction to Reinforcement Learning, 1st edition. MIT Press, Cambridge, MA, USA.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. 2019. Reward constrained policy optimization. In International Conference on Learning Representations.
Christopher J C H Watkins and Peter Dayan. 1992. Q-learning. Machine Learning, 8(3):279-292.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and explore: Language models for action generation in text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8736-8754, Online. Association for Computational Linguistics.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018. Learn what not to learn: Action elimination with deep reinforcement