Safe Deep RL in 3D Environments using Human Feedback
Authors: Matthew Rahtz
Vikrant Varma
Ramana Kumar
Zachary Kenton
Shane Legg
Jan Leike
Venue: ICLR 2022 Submitted (2022)

Abstract: Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, a simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims to solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn a reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans - whether sufficient pixel-based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train an agent to perform a 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits an order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.

Under review as a conference paper at ICLR 2022
SAFE DEEP RL IN 3D ENVIRONMENTS USING HUMAN FEEDBACK
Anonymous authors Paper under double-blind review
ABSTRACT
Agents should avoid unsafe behaviour during both training and deployment. This typically requires a simulator and a procedural specification of unsafe behaviour. Unfortunately, simulator is not always available, and procedurally specifying constraints can be difficult or impossible for many real-world tasks. A recently introduced technique, ReQueST, aims solve this problem by learning a neural simulator of the environment from safe human trajectories, then using the learned simulator to efficiently learn reward model from human feedback. However, it is yet unknown whether this approach is feasible in complex 3D environments with feedback obtained from real humans whether sufficient pixel- based neural simulator quality can be achieved, and whether the human data requirements are viable in terms of both quantity and quality. In this paper we answer this question in the affirmative, using ReQueST to train agent perform 3D first-person object collection task using data entirely from human contractors. We show that the resulting agent exhibits order of magnitude reduction in unsafe behaviour compared to standard reinforcement learning.
INTRODUCTION
Many of deep reinforcement learning's recent successes have relied on the availability of procedu- ral reward function and simulated environment for the task in question, As a result, research has been largely insulated from many of the difficulties of learning in the real world.
One of these issues is safe exploration (Garcıa & Fernández) 2015). Online reinforcement learning dependent first-han hand experience in order learn the constraints of safe behaviour: the agent must drive the car off a cliff to learn not to drive the car off a cliff, While such actions may be fine in simulation, in the real world these actions may have unacceptable consequences, such injury to humans. Further, such constraints are not always easy to describe using procedural functions.
Real environment
Learned simulation
Reward model
Agent
Safe human trajectories
Human feedback on simulated trajectories
Figure/ Our approach. Using number of safe trajectories demonstrated by humans in the real environment, we train a dynamics model that functions as learned simulator. Using this simulator, we train a reward model by asking humans for feedback on hypothetical trajectories, Once the reward model is robust, deploy agent in the real environment using model predictive control.

---

Under review as a conference paper at ICLR 2022
A recently proposed approach to safe exploration is reward query synthesis via trajectory optimiza- T
tion. ReQueST (Reddy et al., 2019). In this approach, the agent is trained in a learned dynamics model (a neural environment simulator) with rewards from learned reward model. Given models of sufficient fidelity, this should allow us to train an agent with close to zero instances of unsafe behaviour in the real environment. However, of Reddy al. (20T9), ReQueST has only been simple navigation task, and a 2D car racing
demonstrated to work in simple 2D environments game with a dynamics model learned from (potentially unsafe) random exploration, and a reward model learned from binary feedback generated by a procedural reward function.
In this work, we aim answer the question: ReQueST feasible complex 3D environments, with data used to train both dynamics and reward models sourced from real humans? In particular, can we learn a pixel-based dynamics model of sufficient quality to enable informative human feedback, and are the data requirements viable, especially in terms of quantity? Our key contributions in this work are as follows.
We demonstrate that ReQueST feasible complex 3D environment, training pixel- based dynamics model and reward model from 160 person-hours of safe human exploratory trajectories and 10 person-hours
reward sketches. We also show that performance de- grades smoothly when models are trained on smaller amounts of data.
On 3D first-person object collection task, we show that ReQueST enables training of competent agent with 20 times fewer instances of unsafe behaviour during training (close to zero instances if not counting mistakes by human contractors) than traditional RI. algorithm.
Safe trajectories
Learned dynamics model
Random
Reward
Initial
Optimised
Reward sketches
Improved
Agent
imagined
sketches
reward
imagined trajectories
reward
trajectories
model
model
Figure 2: Our training procedure for apple collection task. Light blue: steps performed using learned dynamics model. First, a human demonstrates number of trajectories, exploring thoroughly while avoiding the unsafe (red) parts of the state space. From these trajectories, train dynamics model, and use it to generate a number of random trajectory videos. We ask humans to provide feedback on these videos in the form of reward sketches, then use these sketches to train an initial reward model. Since this reward model may not be sufficiently robust, we generate another set of trajectory videos by optimising for maximum and minimum reward as predicted by the current reward model, exposing for example instances where an agent would otherwise exploit the reward model. Using sketches on these trajectories, we train an improved reward model. This cycle can be repeated a number of times until a human deems the reward model to be good enough, which point we deploy the agent using model predictive control.
2 RELATED WORK
Safe exploration Safe exploration has been studied extensively (Garcia Fernández 2015). Most existing work achieves safety by making strong assumptions about the state space, such as all unsafe
2

---

Under review as a conference paper at ICLR 2022
states being known in advance (Geibel & Wysotzki 2005 Luo & Ma
the state space Other approaches require
2021
being reasonably smooth (Berkenkamp et al., 2017; Dalal et al. 2018).
additional inputs, such procedural constraint function (Altman 1999 Achiam et al.) 2017 Ray let al. 2019, Dalal et al. 2018). a safe baseline policy (Garcia & Fernández 2012), or a separate system that determine whether action is safe (Alshiekh al.)
2018). In contrast, the only
assumption we make is that a human can recognise when a trajectory contains or is heading towards an unsafe state.
Acceptability of unsafe behaviour Another important dimension is whether safety is treated as a soft or hard constraint. Most work assumes the former, seeking to minimise time spent in unsafe states (e.g. /eibel & Wysotzki (2005)). Two notable examples of the latter include [Saunders et al.) (2177), which avoids unsafe behaviour by having a human intervene during training to block unsafe actions, Luo & Ma (2021), which starts with a trivial-but-safe policy and slowly broadens the policy while guaranteeing the policy will avoid set of unsafe states specified by the user in advance.
Learned dynamics models in prior work The broad structure of our approach learning dy- namics model (Chiappa et al. 2017) from trajectories, then learning a policy or planning using that model has been successfully used in simple control tasks (Hafner et al. 2019), autonomous heli- copter flight (Abbeel et al. 2010), fabric manipulation (Hoque et al., 202 Atari games (Buesing et al. 2018 and in simple 3D environments (Ha & Schmidhuber 2018). Our work shows that this approach is viable even with complex 3I) scenes, with a dynamics model learned from human- demonstrated trajectories, and with reward model learned from human feedback rather than rely- ing on environment rewards (Hafner et al. [2119] Buesing et al. 20, trajectory following (Abbeel al. 2010), or maximisation of episode length (Ha & Schmidhuber 2018). Prior work on ReQueST Our work expands on the original ReQueST (Reddy et al. 2019) in two main ways. First, we source all data from humans, showing that ReQueST is still applicable with imperfect, real-world data. Second, rather than simple 2D environments, we use visually 3D environments, requiring a much more sophisticated dynamics model.
Reward modeling As with the original ReQueST, we rely on a learned model of the reward func- tion (Knox) 2012, Leike et al.] 2018). In contrast to the classification-bett based reward model used in the original, our reward model regresses to continuous-valued reward, trained using reward sketches (Cabi et al. 2019) on imagined trajectories. Other forms of feedback on which such re- ward models can be trained include real-time scalar rewards (Knox & Stone 2009 Warnell et al.) 2018), goal states (Bahdanau et al. 2018), trajectory demonstrations (Finn et al.) 2016), trajectory comparisons (Christiano et al. 2017), ) and combinations thereof (Ibarz et al. 2018: Stiennon et al.
2020 Jeon et al. 2020).
3 METHODS
3.1 REQUEST
Approach overview For online reinforcement learning algorithms, the only way an agent can learn about unsafe states is by visiting them. Intuitively, ReQueST avoids this problem by allowing agents to explore these states in simulated model without having to visit them in the real environment. First, the agent learns a model of the environment by watching human, who already knows how to navigate the environment safely. The agent then uses this model 'imagine" various scenarios in the environment both safe and unsafe, asking the human for feedback on those scenarios. This process continues until the human is satisfied with the agent's understanding of the task and its safety constraints, at which point the agent can be deployed in the real environment. If all goes to plan, this agent should avoid unsafe states in the real environment without having needed to visit those states in the first place. For further details, see Reddy et al. (2019).
RuQueST in this work ReQueST is flexible in i) the type of feedback used to train the reward model, ii) the proxies for value of information used to elicit informative feedback, and iii) the algo- rithm used to train the agent in the learned simulation. In this work, for i) we use reward sketch- ing (Cabi al. 2019), one of the highest- bandesttt-bwwddthh feedback mechanisms currently available. For ii), we use maximisation and minimisation of reward as predicted by the current reward model. Finally, for iii) we use model predictive control appealing for its simplicity, requiring no addi-
3

---

Under review as a conference paper at ICLR 2022
tional training the dynamics model and reward model are complete. See Fig. for details, and Appendix A for pseudocode.
3.2 PROBLEM SETTING
Our 3D environment consists of an arena with two apples out the open, and third apple behind gate that must be opened by stepping on a button (shown in Fig. 3). Agents receive 96 72 x 3 RGB pixel observations from a first-person perspective, and take actions in a two-dimensional continu- action space allowing movement forward and backward and turns left and right. Agent spawn position, positions of the two apples out in the open, and wall and floor colour are all randomised.
Task The task is eat the apples by moving close to them, ideally eating all 3 apples. The episode ends when either the agent eats the apple behind the gate or 900 steps have elapsed. To test Re- two variations on this setup.
QueST's ability to avoid unsafe states,
Cliff edge environment In the first environment variant, we remove the walls from the arena, so that it is possible for the agent fall off the side. Such fall is considered unsafe, and immediately ends the episode.
Dangerous blocks environment The second environment variant models
subtle case:
where some part of the agent's internal mechanisms is exposed in enviromment, and we would like to safe exploration discourage the agent from tampering (Everitt et al.) (22TT with those mech- anisms. We use a scenario based on the REALab framework (Kumar et al. 2020), where instead of training rewards being communicated the agent directly, rewards are communicated through pair of blocks, where the reward given to the agent is based on the distance between these blocks. If the agent happens bump into one of these blocks, changing the distance between them, this will affect the reward the agent receives, likely interfering with the agent's 's ability to learn the task. Unsafe behaviour in this environment corresponds to making contact with the blocks.
Sized subvariants We further use three subvariants, small, medium and large, of each of these two main variants. each subvariant, the apples are the same distance from the agent spawn position, but the arena size is different, varying the difficulty. The larger the arena, the easier it is to avoid unsafe states: the edges are further away in the cliff edge variant, and the blocks are further away in the dangerous blocks variant. See Appendix B for details.
3.3 DYNAMICS MODEL
Architecture Our dynamics model predicts future pixel observations conditioned on action se- quences and past pixel observations using latent states as predicted by an LSTM. See Fig. for an example rollout. Our model is similar that used in Reddy et al. (2019), except that use larger encoder network, a deconvolutional decoder network, and train using simple mean-squared- error loss between ground-truit -rruh pixel observations and predicted pixel observations. Crucially, base this loss on predictions multiple steps into the future for each ground-truth step (Gregor et al.
Figure 3: 3D environments used for our experiments. Environments consist of green apples, red blocks, and a gate opened by switch. Wall and floor colours are randomised. Also shown are agent's body in the top corner of the environment, and two timer columns, indicating how much time left in the episode by how much of the column green. Left: cliff edge environment. The agent must avoid falling off the edge of the environment. Right: dangerous blocks environment. The agent should avoid interfering with the red blocks, the distance between which encodes the reward sent to the agent.
environment

---

Under review as a conference paper at ICLR 2022
2019), enabling the dynamics model to remain coherent over long rollouts. See Appendix D for additional details.
Data collection We collect safe exploratory trajectories used train the dynamics model using a crowd computing platform. All contractors provided informed consent prior to starting the task, and were paid fixed hourly rate for their time. Note that instruct contractors explore the environment rather than simply eat apples, in order that the dynamics model fully covers the state space. See Appendix H for contractor instructions, and Table for data requirements.
Step 0
Step 30
Step 70
Step 90
Step 100
Step 130
Step 180
Figure 4: Example rollout from dynamics model using test set initial state and action sequence. We require high-quality rollouts for humans to be able to give meaningful feedback. Top row: ground- truth frames. Bottom row: frames predicted by dynamics model, conditioned ground-truth truth frame at step and (for second column onwards) action sequence. Note that the rollout remains coherent over many timesteps, modeling the opening of the gate when the agent steps the button.
3.4 REWARD MODEL.
Reward sketches Our reward models are trained using reward sketches (Cabi et al. 2019): a curve for each episode drawn by a human using the mouse cursor. representing what they think the dense reward between- 11 and should be at each moment in time. We instruct contractors to sketch rewards based on the distance to the nearest visible apple positive increasing rewards when moving closer, positive decreasing rewards when moving away, and negative rewards when close to an unsafe state (the edge the environment or red block). See Appendix and our website for examples.
Data collection We collect reward sketches using a crowd compute platform, asking contractors to sketch 50-frame segments of each episode at time. contractors provided informed consent prior to starting the task, and were paid fixed hourly rate for their time. See Appendix H for contractor instructions, and Table T for data requirements.
Architecture and training Our reward model is a feed-forward network that takes input indi- vidual 96 x 72 3 RGB pixel observations predicted by the dynamics model. The network trained to regress to the sketch values, and the output of the network is then used directly by the agent. See Appendix B for further details on architecture and training.
Programmatic reward bonus Because sketches based the distance the nearest apple, the optimal policy would simply walk up to an apple and stay there. To overcome this, we augment predicted rewards with programmatically-speeee
bonus that encourages the agent actually eat each apple. This bonus gives a reward of + +100 when the predicted reward becomes 2 0.7 and then S 15 steps later drops S 0.1. This corresponds to being close an apple then the apple suddenly disappearing because it has been eaten. Note that the reward model is nonetheless strongly bearing, predicting distance the nearest apple a function that would be difficult compute procedurally from pixels. (This bonus is not applied for the baselines, which instead use ground- truth sparse rewards, as discussed below.)
Trajectory optimisation We use two iterations of human feedback. First, we train an initial reward model based on sketches from 400 trajectories generated by the dynamics model conditioned on random action sequences. Second, we use the reward model from the first iteration to synthesise a set of 100 more informative trajectories that the reward model believes are high or low reward (50 sketches each). We do this by computing the gradient of the predicted reward through both the reward model and the dynamics model to the actions used condition dynamics model predictions, then optimise for maximum minimum reward using gradient descent.
5

---

Under review as a conference paper at ICLR 2022
Model
Data
People
Per person
Total time
Dynamics model
15k trajectories
20 contractors
hours
160 person-hours
2 10M frames
Reward model
400 reward sketches
contractor
hours
person-hours
(First iteration)
2 20k reward values
Reward model
00 reward sketches
contractor
hours
person-hours
(Second iteration)
5k reward values
Table I: Data requirements. A large amount of data required to train the dynamics model, but much less for the reward model. Each trajectory contains up to 900 steps about 15 seconds. Reward sketches are done on video clips 50 steps long, each sketch taking about 90 seconds.
3.5 AGENT
We deploy the agent using the learned dynamics and reward model in the real environment using a simple model-based control algorithm, model predictive control (Garcia et al., 1989). On each planning iteration, sample 128 random trajectories each 100 steps long from the dynamics model. We pick the trajectory with the highest predicted return using a discount of 0.99, take the first 50 actions from that trajectory in the real environment, then replan. See Appendix [for details.
We compare to three baseline agents. The first is alternate ReQueST agent trained on sparse feedback, matching Reddy et al. (2019): +1 when the agent eats an apple, when near an unsafe state, and () otherwise. The second is model-free agent, R2D2 (Kapturowski et al.) 2018), which we train using the ground-trun rewards +1 for each apple eaten (and in the safe exploration environment, penalty of 11 when falling off the edge). The third is an agent that takes correlated random actions. See Appendix [ for details.
Baselines
4 RESULTS
We evaluate ReQueST by running MPC for 100 episodes in each environment variant. For the model-free baseline, we train just up to convergence, and then examine metrics from the most recent 100 evaluation episodes. For the random-action baseline, simply run for 100 episodes. In all cases, we run 10 different seeds and report the median, with error bars showing the 25th and 75th percentiles. Videos are available at http:://sttesooool...oooovwvv//s wtt.e//wwwwweeeeeee...ooooooeeee flrm---ee-ooo---eeeaeeee-
4.1 CLIFF EDGE ENVIRONMENT
In the cliff edge environment,
able ReQueST to train competent agent with between 3 and 20 times fewer instances of unsafe behaviour than by training with a traditional model-free RL algorithm (see Fig. 5). Moreover, the bottleneck is actually the safety of the numan-demonstrated trajectories used to train the dynamics model. For these experiments, we didn' try very hard to min- imize human safety violations. Since our training procedure makes very few errors at deployment time, reducing or eliminating human eтror through e.g. incentives for contractors to be more careful would allow us to train an agent with very close to zero instances of unsafe behaviour.
In terms of performance, the ReQueST agent eats 2 out of the 3 apples average. This is signif- icantly better than the random baseline, which eats on average only out of the apples, and falls off the edge in about half of the 100 evaluation episodes. The model-free baseline (trained using ground-truth sparse rewards) eats 3 apples, but at the cost of safety: even the best case, must fall off the edge over 900 times before it learns not to.
4.2 DANGEROUS BLOCKS ENVIRONMENT
Results for dangerous blocks environments shown in Figure Here the model-free baseline barely able to perform the task at all, instead simply tampering with the blocks (as shown by the large number of safety violations). This is because the agent moves blocks so that the distance

---

Under review as a conference paper at ICLR 2022
between them is as large possible, causing the rewards the agent receives be set permanently high until the end of the episode (when the blocks are reset). Furthermore, since the reward channel saturated, rewards from eating apples capable of performing the task in these environments with a greater number of safety violations than the safe exploration environments only because of the high difficulty of avoiding the blocks in the smaller environment sizes.
longer be observed. ReQueST is the only agent
6k
Safety violations
Train
1225
3
3,0
Apples eaten
300
2
50
48
12
1.3
0.9
Test
1:0
Easy
Medium
Hard
Easy
Medium
Hard
ReQueST (ours)
ReQueST
Model-free
Random actions
(sparse)
baseline
baseline
Figure 5: Cliff edge environment results. Left: safety, measured by the number times the agent fell off the edge of the environment. Note that ReQueST train-time safety violations consists entirely of mistakes by human contractors while providing demonstration trajectories. Right: performance, measured by the average number of apples eaten per episode. Each difficulty represents different arena size, with larger arenas being easier. Runs are repeated over 10 seeds, with bar heights/labels showing medians and error bars showing 25th/75th percentiles.
50k
Safety violations
Train
Apples eaten
2
2.0
2.0
1.9
La
100
11p8
L.0
Test
0.5
0
Easy
Medium
Hard
Easy
Medium
Hard
ST (ours)
ReQueST
Model-free baseline
Random actions
(sparse)
baseline
safety, measured by the total number of times the agent touched the blocks. Note that ReQueST train-time safety violations consist entirely of mistakes by human contractors while providing demonstration trajectories. Right: performance, measured by the average number of apples eaten per episode.
Figure 6: Dangerous blocks environment results. Left:
4.3 SCALING
How much data is required to achieve good performance, for both the dynamics model and the re- ward model? We answer this question by training number of models with varying sizes dataset. See Fig. 7 for these results. The main takeaway is that with less data, performance decreases grad- ually, rather than catastrophically. Our dynamics model appears to follow trends (Henighan al. 2020 in which performance varies smoothly with the dataset size according to power law. For the reward model, however, agent performance appears to plateau after roughly 250 sketches.

---

Under review as a conference paper at ICLR 2022
2.1
Mean return
1.8
Test loss
1,2
0.9
10a
10
103
125
250
375
500
Number of trajectories
Number of sketches
Figure 7: Scaling results for dynamics model and reward model. Top: representative predictions from dynamics models trained on Ik, 3k, 9k, 30k, and 100k trajectories. Left: final smoothed test losses from training dynamics models on 19 dataset sizes. See Appendix D for details. Right: performance of ReQueST agent using reward models trained on different fractions of full sketches dataset (comprised sketches on both random and optimised trajectories). Bar height shows median apples eaten in 100 evaluation episodes over 10 seeds; error bars show 25th and 75th percentiles.
5 DISCUSSION
From these results, should we consider ReQueST realistic, scalable approach to safe exploration?
5.1 ASSUMPTIONS
ReQueST relies on the availability of sufficient quantity of safe exploratory trajectories. Though we have shown the required quantity to be practical (see Table [T for a 3D environment of mod- erate complexity, it difficult to say how well our numbers would generalise to a real-world task (e.g. dishwasher loading). Also, not every task will be amenable to the parallelisation among many humans that made the data requirements viable in our case. For example, the task may be difficult enough that only small number of expert humans are capable of safely navigating the state space, such as helicopter flight. This especially true when dynamics are non-uniform, such as helicopter aerobatics. Here, merely exploratory trajectories would be insufficient; demonstrations of the aero- batic manoeuvres themselves would be needed to learn the unusual parts the state space. For some tasks, however, this issue may be mitigated by unsupervised pretraining (Stiennon et al.! 2020).
ReQueST also relies to some extent on the existence of near-unsafe states. Not all tasks contain such states; for example, if robot is gripping a porcelain cup high above stone floor, the switch between safe and unsafe is near-instantancout In some cases it may be possible to train the dynamics model on similar but low-stakes trajectories (e.g. dropping plastic cups) hoping that the dynamics model will generalise to high-stakes scenarios. In other cases, it is possible that a ReQueST agent would stick close enough to the positive paths (carrying the cup while keeping the grippers closed) that being pushed away from the negative paths (dropping the cup) is less important. However, this will not always work in our dangerous blocks environments, for example, unsafe blocks are on the way to the apples, repulsion from unsafe blocks is necessary.
Finally, ReQueST assumes that unsafe states (or superset of states surrounding unsafe states) can be recognised by humans. This assumption is likely reasonable for tasks of current practical interest. However, may be too strong an assumption looking to the future, particular for tampering: future AI systems may find ways tamper that pernicious exactly because they not be recognised by humans. To guard against these kinds of issues, we will need further progress on scalable oversight techniques, e.g. Christiano al. (2088) Irving al. (018)); Leike et al. (2018).
8

---

Under review as a conference paper at ICLR 2022
5.2 ALTERNATIVES
Imitation learning First, note that in imitation learning, performance is limited by the quality of the demonstrations (Mandlekar et al.) 2021), while the RL-based nature of our approach should enable superhuman performance to be reached. Second, note that demonstration quality is less of an issue for our approach in the first place. We do not require optimal demonstrations, or even necessarily demonstrations of the task at all: to train the dynamics model we require only set of trajectories that provide good coverage of the state space. (However, this does require more data than needed for imitation learning, which does not not require knowledge of the whole state space.)
Offline reinforcement learning Since we assume the environment does not itself provide rewards, for offline reinforcement learning we would, as this work, need to use rewards from a reward model trained using e.g. reward sketches on subset of the human demonstration trajectories (Cabi) et al., 2019) Konyushkova et al. 2020. Zolna et al. 2020). However, note that in this case, assuming no simulator is available, the agent could not be evaluated without running it in the real environment and thus there might be no way to estimate the safety of such an agent without risking an accident in the real world.
Constrained reinforcement learning As noted in Section 2 a common approach to safe exploration is to provide a procedural constraint function which the agent should violate as little as possible dur- ing training. Replacing this procedural function with a learned constraint function would make for an interesting comparison to the present work. However, note that similarly to offline reinforcement learning, such an agent could not be safely evaluated without access to a simulator.
CONCLUSION
In this work we have shown that ReQueST can be used train an agent in a 3D environment with an order-of-magnitude reduction in instances of unsafe behaviour than typically required with reinforcement learning. If unsafe behaviour can be avoided by humans (or trajectories are collected by watching humans carry out tasks they are already doing, such that no additional safety violations are incurred) our method can come close to achieving zero instances of unsafe behaviour. Moreover, this is possible without a procedural specification of safe behaviour, and with minimal assumptions other than unsafe or ncar-unsafe states being recognisable by humans.
Our results are exciting for two reasons. First, they demonstrate that ReQueST is plausibly general- purpose solution to (one version of) the safe exploration problem. Arguably, this is the version we're most interested in long-teu dasy-to-misspest
where safe behaviour is learned from humans, rather than given by an procedural function. Some tasks will require higher-fidethe dynamics models than those used in this work, but given recent advancements in generative image modelling (Karras) et al.] 2018 2019 Mildenhall et al. 2020), we are optimistic for future progress in this area. Second, our results hint at a future where our ability to verify agent behaviour before deployment is not bottlenecked by the availability of a simulator - the simulator can be learned from data.
In terms of future work, one particularly important question is: what challenges arise when trying to use ReQueST in a real-world environment, such as a robotics task? We look forward to future work addressing this line of research.
7 ETHICS STATEMENT
Contractor welfare Our method relies on a large quantity of data from human contractors, requir- ing them to perform a task that is extremely monotonous for several hours per person. Additionally, during one round of feedback our contractors reported mild motion sickness due to latency our interface. Those wishing to use our method in practice should ensure that contractors are well- compensated; that they feel free to take many breaks as they need; and ideally that no individual contractor is required to perform the same task for too long (relying on a large number of contractors rather than long sessions in order to gather sufficient data).

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pieter Abbeel, Adam Coates, and Andrew Y Ng. Autonomous helicopter aerobatics through ap- prenticeship learning. The International Journal of Robotics Research, 29(13):1608-1639, 2010.
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. arXiv 2017.
preprint
Mohammed Alshickh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Eitan Altman. Constrained Markov Decision Processes, volume 7. CRC Press, 1999.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefen- stette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946, 2018.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Advances in Neural Information Processing Systems, pp. 908-918, 2017.
Lars Buesing, Theophane Weber, Sébastien Racaniere, SM Eslami, Danilo Rezende, David P Re- ichert, Fabio Viola, Frederic Besse, Karol Gregor, Demis Hassabis, al. Learning and querying fast generative models for reinforcement learning. arXiv preprint arXiv: 1802.03006 2018.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning, arXiv, pp. arXiv-1909, 2019.
Silvia Chiappa, Sébastien Racaniere, Daan Wierstra, and Shakir Mohamed. Recurrent environment simulators. arXiv preprint arXiv: 02254, 2017.
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08:. 005500000055,..... 2018
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4299-4307, 2017.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik. Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint 1ttt::1801008777 2018.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018. doi: 10.2391 15/distil.00011 1. https://distill.pu/200 S/feeture-wiie--rrasfoonn..
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning. arXiv preprint arXiv:1908.044 047344. 2021.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference Machine Learning, pp. 49-58, 2016.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and prac-
tice-a survey. Automatica, 25(3):335-33) 348, 1989.
Javier Garcia and Fernando Fernández. Safe exploration of state and action spaces in reinforcement learning. Journal of Artificial Intelligence Research, 45:515-564, 2012.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Peter Geibel and Fritz Wysotzki. Risk-sensitive reinforcement learning applied to control under constraints. Journal of Artificial Intelligence Research, 24:81-108, 2005.
10

---

Under review as a conference paper at ICLR 2022
Karol Gregor, Danilo Jimenez Rezende, Frederic Besse, Yan Wu, Hamza Merzic, and Aäron van den Oord. Shaping belief states with generative environment models for RL. Advances in Neural Information Processing Systems, 015-13475-13455 2019.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint
2018.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp.
PMLR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780) 1997.
Ryan Hoque, Daniel Seita, Ashwin Balakrishna, Aditya Ganapathi, Ajay Kumar Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. VisuoSpatial foresight for physical sequen- tial fabric manipulation. arXiv preprint arXiv:2102.09754 2021.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in Atari, arXiv preprint 1.06551,, 2018.
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint 1805 2018.
Hong Jun Jeon, Smitha Milli, and Anea D Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. arXiv preprint arXiv:2002.0483: 2020.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabuey. Recurrent ex- perience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. URL fhttps::: openreveivvv ew net. forum? 9ezkkk22eeekKK
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6986 2014.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
William Bradley Knox. Learning from human-generated reward. 2012.
Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. semi-supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06889 2020.
Ramana Kumar, Jonathan Uesato, Richard Ngo, Tom Everitt, Victoria Krakovna, and Shane Legg- REALab: An embedded perspective on tampering, 2020.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction, 2018.

---

Under review as a conference paper at ICLR 2022
Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.01101, 2017.
Yuping Luo and Tengyu Ma. Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations. arXiv preprint vrXiv:2108.01840 2021.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint wxXiv:210.033200 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. In European Conference on Computer Vision, pp.
Springer, 2020.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 2019.
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning human objectives by evaluating hypothetical behavior, 2019.
William Saunders, Girish Sastry, Andreas Stuhlmueller, and Owain Evans. Trial without enror: Towards safe reinforcement learning via human intervention. arXiv preprint arXiv:170..05133, 2017.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01 0235, 2020.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep TAMER: Interac- tive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Ay- tar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011.13880 2020.
12

---

Under review as a conference paper at ICLR 2022
Appendices
A
PSEUDOCODE
Denoting:
Observation o from observation space f2 Observation predicted by dynamics model 6.
Action from action space A. Action sequence A. Optimised action sequence A".
Trajectory Sequence of observations predicted by dynamics model 7. Sequence of predicted observations sent to user for reward sketching &query-
Reward sketch value Distribution reward sketch values assigned by human Puser.
Algorithm 1: Dynamics and reward model training
Data: Safe trajectories M train. M testi Reward sketches D 0 Parameters: Dynamics model parameters o, Reward model parameters 0 Models: Dynamics model D& 52 A Ω Reward model Rş Ω2
Hyperparameters: Number of reward sketches to obtain on each iteration Nskatches // Train dynamics model
+ arg ming MSE(T Mtrain, Dol-ooo, 00,01 ai ... ~ r)]
// Train reward model (convergence judged by human)
2 while not converged
repeat Nevetchee times
Og ~ M
if A not initialised
f query t 00, o1. Do(+ (0,100,,100 RANDOM)
else
A + arg max L(inDa((oord)) Ro(
A Querry Ôo, 01, Døl: (0,0000,-7'
for ô gquery
r ~ Puser (rlo, fguerz)
11
D - DU r),
0 4 arg ming Σ (B,)ED MSE/r, Ro(
Initial real environment state so sampled from start-state distribution Pu- Subsequent states computed by state transition function (s)
Observation conditioned on real environment state ((s).
Random action sequence for 7ith rollout A", Action at index within each sequence Ag.
Algorithm 2:
Model predictive control
Models: Dynamics model D& f2 x A + Ω. Reward model Rg 92 - Hyperparameters:
Number of sample rollouts per replan N saamples. Steps per sample rollout Nptan. Discount Number of steps after which to replan N Nreplan & N plan
Initialise t < 0, so
2 while S; not terminal
((s))
A RANDOM E [1, Nsampies]
Npten 6-1 - Ro(o Dollon, AR))
A arg mag Σk=1
for a E at,..., Ot AA"
St41 f(s, // Take action in environment
1 1+1+1
See Appendix for more details model predictive control implementation.
13

---

Under review as a conference paper at ICLR 2022
B ENVIRONMENT DETAILS
Our six total environment variants are illustrated in Fig. Our environments are implemented using Unity, with randomisation of wall and floor colours improve generalisation.
In the rest this appendix, we discuss spawn positions.
B.I APPLES
The two apples that are the open always spawn in the same area, near the gate, regardless of the size of the arena. This ensures that the margin for error increases as the size of the arena increases: the larger the arena, the further away the unsafe parts of the arena (the edges of the arena/the red blocks) are from the important parts of the task, the apples. Spawn positions are selected randomly from a discrete set eight possible locations.
B.2 BLOCKS
In the dangerous blocks environment, the blocks spawn in random position around semicircle centred on the button by the gate. The radius of the semicircle increases with the size of the arena, such that the larger the arena, the further away that blocks spawn.
B.3 AGENT
In the safe cliff edge environments (top row of Fig. 8). the agent can spawn in one of the four cor- ners of smallest arena size: by either of the two timer columns, or in either of the two corners surrounded by open space. These spawn locations remain unchanged in the larger sizes of envi- ronment, so that when the arena is larger, the agent starts further from the edge. The agent always spawns looking towards the centre of the arena, such that the first frame of the episode is more likely to show both apples. This is important for helping the dynamics model to establish the positions of the apples.
In the dangerous blocks environments, the agent can only spawn by either of the two timer columns again, looking out towards the centre of the arena. Otherwise, the red blocks may not be visible in the first frame of the episode, which causes the dynamics model to 'hallucinate" blocks in arbitrary locations.
Figure 8: All six of our environments variants, showing the three possible arena sizes. First column: small. Second column; medium. Third column: large.
14

---

Under review as a conference paper at ICLR 2022
C MODEL PREDICTIVE CONTROL
We implement model predictive control with sampling- sepping-basedd optimisation. This is because we aug- ment predicted rewards using non-differentiable procedural reward bonus (see Section 3.4). This functions as follows.
First, generate Naapples 128 random action sequences A, each Npian 100 steps long. Each action sequence starts with random number (between 0 and plon)) of 'turn left' or 'turn right' actions, then all remaining actions are 'move forward'. (Note that try match this for the random-actions baseline; see Appendix(])
Y (1, N samples
01, O.N pinn RANDOM
A" f (a), O.
For each action sequence, we generate an imagined rollout starting from the current state 8, in the real environment. This involves first initialising the latent state of the dynamics model lo by feeding the history of observations from the real environment in the current episode 01, through our encoder network E.
to t E(o
We then use the dynamics model D to predict future states 4,, 1,, conditioned on each action from the sequence. We use the decoder network F transform each latent state to a predicted pixel observation é, then use the reward model R to predict reward ý for each predicted observation. This yields predicted return 1tth (discount 0.99) for each action sequence A".
11 M(olaa)
rz+1 F(li)
Prt1 R(θ++1)
1000 M(199la100)
0e4100 F(1100)
i1+100 R(004100)
itm E100 n-1 fetk
We then select the action sequence with the best predicted return, and take the first N reptan steps in the environment, and repeat.
N + arg max Aim) A' t A for a in A [:Nelln env. step (a)
We selected the hyperparameters Nommples Nptan and N repian using grid search. Rollouts are run in parallel using 50 CPU workers, with each worker running separate cluster node. Using this setup, each episode 900 steps takes about 15 minutes.
15

---

Under review as a conference paper at ICLR 2022
D DYNAMICS MODEL
Our dynamics model consists three components: i) recurrent encoder network that transforms a history of pixel observations into initial latent state, ii) recurrent next-state network that com- putes future latent states conditioned on actions, and iii) a feed-forward decoder network that trans- forms latent state back into predicted pixel observation. The model consists of 50M parameters total.
D.1 ENCODER NETWORK
The encoder network computes a latent state corresponding to the current state of the real environ- ment, taking as input the history of observations from the current episode in the real environment. To do this, we first process each observation using residual network (He et al. 2016 as constructed by the following pseudocode:
observation
for num num_channels in [16, 32, 32]
x conv_24 (x, kernel_shape- 3, stride-1, Bhannels-num_chaneee max_pool (x, window_shapeste stride=2)
for - in ra)ge
block_input
X = relu (x)
conv_2d (x, kernel_shape=3, stride=1, channeels-num_chammeee
X = relu (x)
X conv conv_2d (x, kernel phanel_shape-,, stride=1, Channels-nun_channe
x += block input
relu (x)
For N observations, this yields N outputs. However, each of these outputs is a function of only single observation. To produce latent state which is function of the history of observations, additionally apply single-layer LSTM (Hochreiter & Schmidhuber, 1997) with 1024 hidden units over the sequence of these outputs, and take the final output from the LSTM to produce the latent state. (Note that this is separate LSTM than the one used for the next-state model.)
The encoder residual network consists of 100k parameters, and the encoder LSTM consists of 18M parameters.
D.2 NEXT-STATE MODEL
The next-state model computes future latent states using the initial latent state and a sequence of actions. This is also implemented using single-layer LSTM with 1024 units. The LSTM consists of 4M parameters.
D.3 DECODER NETWORK
The decoder takes single latent state and transforms it back into pixel observation. is feed- forward deconvolutional network using FiLM conditioning (Dumoulin et al. 2018) as constructed by the following pseudocode:
first_output_shapee = [3, 4, 1024] putput_shapa
[9,, 12) (18, 24), (36, 48), (72, 96), (72, 96) ] output output_channels
channels = [512, 256, 128, 64, 3]
strides = [3, 2, 2, 2, 1] kernel_shaper [4, 4, 4, 4, 3]
latent
x linear (x,
reshape (x, first_output_ate
for layer_num in range(
16

---

Under review as a conference paper at ICLR 2022
prev_layen num channels x.shape [-1]
offset = linear(
latent,
Scale should have a mean of 1.
scale 1 linear(
latent,
x x * scale offse
= leaky_relu(x)
x conv_2d_transposet
x, termee_ssappeeeeeess_hhhppp (Layer_numm
str:deesstrldeesllyyeennnli
[layer_num)
sigmeid (x)
In total, the decoder network consists of 28M parameters.
D.4 Loss
We compute the loss based the MSE between the predicted and actual pixel observations. Ad- ditionally, we use latent overshooting (Gregor et al.] 2019) to improve long-term coherence, and subsampling to GPU memory. The overall procedure is follows.
crajectory_num__
= len (observations)
steps_to__s ser_____sarrt____e
random. vandom.choice(
trajectory_nun_steps altzemmmmpprrdddctttooooeeeeepee
loss = 0
for starting_ster in steps_ latent = encoder [eess_tt_start_pprcciiinnng____
predicted_latente = []
for - in range
latent aatt__sttte__mdellaaatte predicced_lateetts.apuue
steps_to_decon random.choice ieaiicttions_eengtte
for step_ta_decode in steps_to_decode:
latent presicted_observatiion decoder (latent)
[ecooee
actual_observatio: observations [starting_step step_to_decomt
loss +=
- actual_observation ** 2
We use num nun_prediatioonsequennees nun.prreciccioo_ttppeeeeemeeepee
100 and
17

---

Under review as a conference paper at ICLR 2022
D.5 TRAINING
We train using the Adam optimizer IKingma & Ba 2014) with a learning rate of x 10-4 and batch size of 48 trajectories (about 43k individual steps).
We train using 8 GPUs (with batch parallelism) for 1M steps about 2 weeks. We train a total of dynamics models for each the environment variants (see Section 3.2). For all models, the final train loss is about 15, and the final test loss is about 30.
D.6 SCALING
Recent work has found performance of large models to vary surprisingly predictably as a function of, for example, dataset size (Henighan et al. 2020). We wanted to check whether this true for our dynamics model too, so we trained a number of dynamics models on varying dataset sizes, as shown in Fig.
This figure was produced as follows. First, we generated synthetic dataset of 100,000 trajectories in the real environment comprised of random action sequences. We then trained 20 different dynamics models on sgarithmicalll-spacee fractions of this dataset, all models using the same hyperparam- eters. We trained until all models' test losses had definitely plateaued about 1.2M steps, 17 days. The model trained on 1,600 trajectories did not train properly so we ignored it. We then smoothed the test loss curve by taking an exponential moving average (smoothing factor a 0.001), and took the minimum of this smoothed curve to be the loss value reported for each model. See Fig. for training curves.
Number of trajectories
1000
1300
6 x 10a
2100
2600
4 x 102
3400
Test loss
x 10a
4300
5500
10a
7000
8900
11300
10a
14400
18300
23400
0.0
0.2
0.4
0.6
0.8
1.0
1.2
29800
37900
Training steps
x 106
48300
61600
78500
100000
Figure 9: Dynamics model training curves.
18

---

Under review as a conference paper at ICLR 2022
E REWARD MODEL
Our reward model is a 2.2M parameter 11-layer convolutional network with residual connec- tions (He et al. ( that takes RGB 96 72 input and produces scalar reward prediction, constructed as follows:
observation
for num num_ num_channels in [16, 32, 32]
x conv conv_2d(x: (x, kernel kernel_shape-3; stride=1, channels-num channelsh)n)) x = dropout (x, rate=0.1)
x max_pool (x, window_shape stride=2)
for in range
b block_input
X = relu(x)
X = conv_2d (d (x, kernel_shapers stride=1, chamels=nuu_chameells
X dropout (x, rate=0.1)
X = relu(x)
conv_2d (x, kernel_shapese stride=l, channe saw.mllcmmnmhhnelll...
X +- block_input
X dropout (x, rate-0.1)
relu (x)
flatten (x)
= linear (x, num num_ cum_units=128
X dropout (x, rate=0.1) 1)
= relu(x)
x linear (x, num (u__unitt=1)))
dropout (x, rate=0.1)
x linear (x, num_units-1) predicted_reward =
We train using mean-squared error loss, using the AdamW optimiser with hyperparameters shown in Table 2 We train using batch size of 64 (where each data point consists of one observation and the corresponding sketched reward value), for a total of 20,000 batches in all cases (which was consistently sufficient for the validation loss to plateau). Training using single GPU, this takes about hour.
Batches must be sampled carefully due to dataset imbalances. For example, when collecting reward sketches from random trajectories, only small number of those trajectories will actually show the apple being eaten. To account for the imbalance, all sketches into individual timesteps, and collect the timesteps in pool. Second, we split the pool into groups of timesteps, where each group represents one type of situation far away from both apples and hazards, ii) being close to hazard, and iii) being close an apple as determined by the sketched reward value for that timestep. Finally, when sampling a batch, we sample roughly equal number of timesteps from each group (placing slightly more weight on the "close to an apple' group, which the most important), ensuring that each batch contains data for all situations of interest. See Table 3 for numbers used for this procedure.
adopt the following procedure. First, we split
our case, i) being
Hyperparameter
Value
Situation
Value range
Batch proportion
Weight decay
2 x 10-4
Close to apple
0.7 to 1.0
0.4
Learning rate
8 x 10
Close hazard
-1.0 to -0.5
0.3
81
0.7
Close to neither
-0.5 to 0.7
0.3
82
0.9
Table 2: AdamW hyperparameters.
Table 3: Batch sampling values.
19

---

Under review as a conference paper at ICLR 2022
F EVALUATING DATA AND MODEL QUALITY
By learning a neural simulation, ReQueST aims to produce an agent that is safe to deploy directly in the real world. But how can we be sure of this how do we know when the agent has been trained enough that it really is both competent and safe?
In principle, the model-based nature of ReQueST allows this determination to be made by simply running agent rollouts the neural simulation. In practice, this requires that the dynamics model both a) remains coherent over the entire length of simulate episode, and b) matches the real en- vironment closely; criteria that the dynamics models we trained in this work do not meet. Further progress in generative video modelling will be needed for this ambition be realised.
In the meantime, we can get feel for how the agent is likely to behave by assessing the performance of the individual models that the agent relies on. In this appendix, we detail the methodology used to do to this, for both the models themselves and the data used to train them.
F.1 EXPLORATORY TRAJECTORIES
A model is only as good as the data it's trained on. To ensure the data used to train the dynamics model was of sufficient quality, we found the following three checks helpful.
Qualitative inspection Simply viewing videos of the trajectories, making it easy to get through many videos quickly by arranging videos in a grid and watching high speed, was an important first step in getting sense for what's going on in the trajectories and what problems might exist in them.
Action counts Given how dull the task is, it's not surprising that the quality of trajectories produced by contractors was somewhat mixed. particular, some contractors would spend some trajectories mostly idle, or just moving consistently in straight line. To ensure such trajectories did not make up a significant fraction of the dataset, we found it helpful to plot histograms of a) the number of no-op actions per trajectory; b) the number of times the action changed per trajectory; and c) the number of distinct actions per trajectory.
State space coverage It's important that the trajectories cover all relevant parts of the state space for example, that there are a good number of trajectories which the apple is eaten. In our case were able do this by examining the distribution of ground-truth rewards provided by the Unity environment. For real-world environments, a good approach here would be to train classifiers for various situations from the trajectories, and examine the distribution of those classifiers' outputs over the dataset.
F.2 DYNAMICS MODELS
There are two main techniques we used to evaluate dynamics models.
Test set trajectories We do not train the dynamics model on all human exploratory trajectories; we reserve some trajectories in test set to use for evaluation. This test set can be used to evaluate the dynamics model quantitatively, comparing the test loss to the train loss, and qualitatively, manually inspecting the predicted rollouts produced by the dynamics model given test set initial state and action sequence.
Interactive exploration It is also possible to explore the simulated environment produced by the dynamics model interactively, starting from an initial state sampled from the test set and then gen- erating actions in real time using keyboard input. This allows for fine-grain investigation of specific details that may not be covered by action sequences the test set for example, what happens one walks off the edge of the cliff edge environment backwards?
The combination of these two techniques worked great, and allowed us to be confident in our as- sessment of dynamics model quality.
20

---

Under review as a conference paper at ICLR 2022
F.3 REWARD SKETCHES
As with the dynamics models, the quality of the reward models depends on the quality of the reward sketches the models are trained on. To ascertain sketch quality, we used the following techniques.
Qualitative inspection As with trajectories, taking a quick look over the sketches themselves, plotting them in a grid to make it easy look through them quickly, was an important first step in our process.
Sketch statistics Next, we examined the number of sketches where i) all sketch values are close to zero, ii) any sketch value is greater or iii) less than zero, and iv) any sketch value is greater than 0.7 or v) less than -0.7. We also plotted the total number of individual timesteps (rather than the number of trajectories) matching the previous criteria. Finally, we plotted histograms of i) all sketch values pooled, ii) the return for each sketch, iii) the maximum and iv) minimum value each sketch, v) the difference between the highest and lowest value in each sketch, and vi) the number of distinct binned values in each sketch.
F.4 REWARD MODELS
We evaluated reward models using similar techniques the dynamies model.
Interactive probing As with the dynamics model, we also found it helpful to play around with reward models interactively. We did this by rumning the dynamics model interactively and overlaying a graph of the predicted rewards. This enabled us to quickly check e.g. how consistently the reward model responded to apples, whether the reward model was responsive the apple behind the gate when the gate is closed, and so on.
Test set feedback: Again, we do not train on all sketches. instead reserving some in order compute a test loss. Due to dataset imbalance described in Appendix [ the test loss must be computed carefully, as different situations will not be represented evenly the dataset. For example, when collecting sketches of random trajectories, only a small fraction of trajectories will show an apple being eaten; splitting the dataset naively could lead to all these instances being placed in the train set. We therefore split the dataset by first pooling and grouping the data as described in Appendix D and then constructing split each individual group into a train set and a test set. This ensures that the test loss is representative of all situations of interest.
In practice, we found reward model test loss to correlate relatively poorly with agent performance. We believe the main reasons for this are:
Coverage Because of the way we generate trajectories to be sketched (in the first stage using random action sequences, and in the second stage by optimising for maximum/minimum predicted reward), the set of states covered by the trajectories is not exhaustive.
Feedback quality Because of the way our reward bonus works (triggering when the predicted reward exceeds 0.7 and then quickly drops), our agent is unfortunately quite sensitive to exact reward values, which can vary from sketch to sketch. This is partly function of contractors' diligence, but mostly a function of a) the difficulty of judging distances to apples consistently, exacerbated by b) the somewhat low fidelity of the trajectories generated by the dynamics model.
To mitigate both of these problems, we briefly experimented with:
High-quality test set This involved manually demonstrating a set of easy-to-sketee trajectories in the neural simulation (starting far away from an apple and walking straight towards it), then carefully sketching the resulting rewards ourselves. However, this turned out be sufficiently time- consuming (needing to be done for all six environment variants) that we judged it not to be worth the effort.
We ultimately didn't feel very satisfied with these techniques. Interactive probing provided the strongest signal, but due to its interactive nature was unsuitable for evaluating large numbers of models when doing e.g. hyperparameter sweeps. In the end, we chose reward model hyperparame- ters on the basis of resulting agent performance in the 'real' environment - a technique not applicable to real-world scenario. If we'd had more time, taking the time to construct high-quality test set would probably have been the most principled solution.
21

---

Under review as a conference paper at ICLR 2022
G UNDERSTANDING AGENT FAILURES
Even after having carefully evaluated the data and models that the agent relies on (Appendix F). the agent still may behave poorly at deployment. This appendix details the methodology we developed to understand agent failures, and the failure modes we observed for our task.
G. DEBUGGING METHODOLOGY
We used two main techniques to understand failures.
Interactive probing from failed state As described in Appendix F key tool is the ability to run the dynamics model interactively, providing actions using keyboard input and watching how graph of the rewards predicted by the reward model changes depending the situation. When evaluating the models, we initialise the dynamics model from an arbitrary state sampled from the dataset of exploration trajectories. However, it is also possible initialise the model from state from a real-world trajectory. For example, the deployed agent moves towards an apple but stops before getting close enough eat we initialise an interactive simulation from that state, and explore to try and understand the problem is that the reward model does not give additional rewards for moving closer? Is that the dynamics model loses coherence when moving towards the apple? And so on.
Examining counterfactual rollouts To recap, model predictive control functions by sampling number of rollouts from the dynamics model, then picking the rollout with the highest predicted reward, and taking the actions from that rollout in the real environment. actions, a second possible debugging strategy is therefore to examine the other rollouts which the agent didn't pick. For example, one case an apple would be in view but the agent would simply turn around rather than going to the apple. By examining the counterfactual rollouts, we were able to see that the agent did consider going to the apple, but chose not to because the reward model did not respond to the apple.
the agent takes poor
G.2 FAILURE MODES
Using the tools above, we identified the following failure modes.
Dynamics model fidelity One class of failure was caused by the dynamics model failing to gen- erate crisp predicted observations. This affected not only agent performance but also the quality of human feedback. The main two failures modes where apples would shimmer in and out of existence as the agent moved around the environment, and b) loss of coherence, where long sequences of predictions would sometimes result in the scene dissolving into a jumble of colours.
were a) blurriness and mirrage- like effects,
Dynamics model quality could likely be improved to some extent by more (or better-quality) data, as shown by the results in Fig. 7 However, the same figure suggests that the improvements from more data plateau around 100k trajectories, and even the model trained that number of trajectories still exhibited these issues to some extent - leading us to believe that architectural limitations of our model are probably the bottleneck here.
Reward model mistakes The second class of failure was the reward model not being robust enough. This took several forms. In the most egregious cases, the reward model would simply not respond to an apple being clearly in view - though this was rare. The more common failures were a) the predicted rewards fluctuating as the observations predicted by the dynamics model them- selves fluctuated - for example, as apples shimmered partly into and out of existence; b) the shaping of rewards being incorrect for example, the predicted reward not increasing monotonically when moving towards apple; and c) the precise value of the predicted rewards being incorrect (at least according to the specification we laid out in our instructions to contractors see Appendix H). in a way that caused our reward bonus not to trigger.
Overall, we suspect the difficulties here were mainly due to poor fidelity of the dynamics model. There could be some lessons here related the difficulty of accurately and consistently sketching precise reward values, but problems here could equally be explained by irregularities in the output of the dynamics model, so we hold off making any firm recommendations here.
22

---

Under review as a conference paper at ICLR 2022
Reward bonus hacking The reward bonus itself caused number of problems. In particular, the bonus was sensitive to reward hacking the agent managing to trigger the bonus in a manner other than the intended one. For example, the agent would raise the predicted reward above the required threshold by moving close to an apple, but then cause the predicted reward to suddenly drop not by moving through the apple but by turning sharply away. The agent could then look back towards the apple and repeat to continually trigger the reward bonus. This is an unfortunate example of the difficulty getting programmatic reward functions (even programmatic augmentations learned reward functions) correct.
We learned two things from this experience. The first is that it difficult use learned dense reward models for multi-stage tasks. If one does use dense rewards, one must either a) use separate reward curves for each stage (as we did this work), necessitating additional mechanisms to cope with the discontinuity between each stage, or b) have reward curve which keeps track of progress stages at once, necessitating the human giving feedback remember everything that's happened in the episode far. The second is the importance of thinking hard at the start of project whether cerd-forward reward model will be sufficient. our case, the key behaviour we needed the reward model to recognise the eating of an apple was fundamentally incompatible with a feed-forward reward model, because required awareness of history: seeing the apple there one moment, and not there the next.
all
23

---

Under review as a conference paper at ICLR 2022
H CONTRACTOR INSTRUCTIONS
Note: we give contractors different instructions depending on whether they are working with the cliff edge or the dangerous blocks variant. For conciseness, below we have combined both of instructions.
H.1 DEMONSTRATIONS
You will be providing demonstrations of how to move around simple environment safely, without [falling off the edge touching certain items in the environment].
Apples
Blocks
Gate
Apple
Switch
Gate
Switch
Note: contrary to what these images suggest, contractors both the cliff edge and dangerous blocks environments at the same resolution. The safe exploration environment image on the left is lower-resolution because those instructions made at earlier point in time.
H.2 TASK
In this task, should move around the environment [without falling off the edge of the envi- ronment without touching the red blocks]. The other things in this environment are two apples out in the open, and another apple behind gate that opens when you step switch. When you get close enough to an apple, you 'eat' it, and the apple disappears. Your goal is to generate diverse demonstrations of (safe) behaviour in this environment.
Try to explore the environment as thoroughly as possible. Specifically:
Try and go many different parts of the environment possible. Go to the gate, the black bars, the middle, the edges, and so on.
Try to view as many parts of the environment as possible (the walls, the apples, the sky, the switch, the gate, different parts of the floor, and so on) from many angles and distances as possible.
Try to move as many differrntt ways as possible. For example, sometimes you can move straight forward or backward for several seconds without turning, sometimes you can spend most of the demonstration just turning around, and so on.
Try to do many things as possible. For example, sometimes you can eat the apples, and sometimes you can leave them alone. Sometimes you can press the switch, and sometimes you can avoid and so on.
Note: you may have played this task before, but the objective here is different to normal. Your main objective is not to eat apples. Your main objective is to explore the environment as thoroughly as possible. That includes eating some apples, but your main objective explore. You shouldn's be aiming to eat apples in every single demonstration.
Remember, it's very important that you don't [fall the edge touch the red blocks]! Please try and be careful about this as possible.
24

---

Under review as a conference paper at ICLR 2022
H.2.1 INTERFACE AND CONTROLS
Info window
Info
Reward Episodes Episode
Episode counter
Demonstration window
To start recording demonstration, click in the demonstration window. You should then be able move about using the following controls:
Move forward: press key W
Move backwards: press key
Turn left: press key A
Turn right: press key D
You can press multiple keys at the same time. Note that you will not be able to look around using the mouse. This is normal.
Note that the controls may be a bit "jerky". For example, when holding down the W key, instead of moving forward smoothly, you will move forward changing speed. This is normal don't worry about it.
Each demonstration will end automatically after 16 seconds. At the end, demonstration will automatically start, with a different version of the environment with different colours and different apple positions.
The episode counter in the info window helps you keep track of how many demonstrations you' 've done. 'Episode number' tells you how many demonstrations you've 'vee done in the current session. 'Episodes recorded' tells you how many demonstrations have been saved for all your sessions for this task. Please make sure that both 'pissdde number' and 'Episodes recorded" increase by each time you do demonstration.
25

---

Under review as a conference paper at ICLR 2022
H.3 REWARD SKETCHING
You will be giving feedback to an agent in order to train it to move towards apples and "eat" them, while avoiding [the edge of the platform they're located on / touching red blocks]. You will be shown a video showing the agent's view from first-person perspective. Your job to give feedback in the form of frame-by-frame framee 'sketch' that tells the agent how well it's doing.
H.3.1 INTERFACE
1. Video
1. Video
10
10
7.5
3. Submit with this button
3. Submit with this button
2. Sketch here
2. Sketch here
-10
SUBMIT
The important parts of the interface are:
1. The video of what the agent sees, The video plays forwards and backwards as you move your mouse cursor over the sketching area.
2. The sketching area. You will draw here by clicking and holding with the mouse, creating blue drawing. The height of your sketch at each moment in time tells the agent how well it's doing at that moment. Note the scale shown on the right which we will refer to later: the top, with guide lines O, 2.5, 5, 7.5 and 10. you
-10 is the bottom, and 10 want to tell the agent it's doing well, sketch a high value, and if you want to tell the agent it's doing badly, sketch low value.
H.3.2 TASK
In this task, you want train an agent move towards apples while avoiding [the edge of the platform / red blocks]. There are two small apples, and a large apple behind a gate that opens when the agent steps on a switch. Moving towards or turning towards apples is good. Getting too close to [the edge/ red block] is bad.
Gate
Apoles
Apple
Raid Mock
Swiltsh
Switch
Once the agent gets to an apple, the apple should disappear (though sometimes might not).
Note that the video is low-resolution, and both the apples and other parts of the videos may become blurry or distorted, making it difficult to see what's going on. This normal don't t worry about this, and just do the best you can.
26

---

Under review as a conference paper at ICLR 2022
H.3.3 SKETCHING GUIDELINES
Please use the following rules of thumb when deciding what height to sketch.
If there are no apples within view (e.g. the agent is looking at wall), sketch 0:
If there are one or more apples in view, but far away, sketch 2.5:
If the agent is close to an apple, sketch 5:
27

---

Under review as a conference paper at ICLR 2022
As the agent moves closer to apple, sketch 7.5:
Only sketch 10 if very close to the apple:
If the agent gets [too close to the edge, or falls off the edge / close to a red block],
immediately sketch -10:
28

---

Under review as a conference paper at ICLR 2022
If the video gets very blurry distorted blurry that you can't 't tell what's happening- sketch 0.
It's more important to [stay away from the edge avoid red blocks] than eat apples so if you're re near [the edge / a red block] and near an apple, it's better to sketch -10.
As soon as the apple is no longer visible sketch O.
Note that the apple may disappear while you are moving towards it. Don't worry about this. When it happens, sketch once the apple has disappeared.
Do not sketch extra height for more than one apple. For example, if there are two apples very close, the sketch height should be the same as if only one apple is very close.
In other situations, sketch heights somewhere in between the heights above. In particular, as the agent moves closer to apple, the sketch height should gradually increase from to 10. Similarly, as the agent moves away from an apple, the sketch should gradually decrease in height. However, make sure that the sketch height is 0 the apple is not on the screen.
However, you should sketch -10 as soon as the agent gets close to [the edge red block]. Treat [the edge red blocks] "all nothing": either the agent is far enough away that everything's fine, or the agent is too close and you should sketch -10. That is, you should not sketch any heights between and -10. Either sketch -10. or sketch height greater than equal O.
H.3.4 NOTES
Sometimes the agent might move forwards towards an apple and apple disappears, but then the agent moves backwards and the apple reappears again. In this case, treat the apple as a 'fresh' apple. Your sketch height should decrease as the agent moving away from the apple, and your sketch height should increase if the agent then moves back towards it.
In some cases the apple may only be partially visible. For example, parts of it may be missing, see-through or distorted, or the video might be too dark. you're not sure, treat the apple as if it wasn't 't there. For example, treat the apple as if it wasn't 't there in cases like these:
H.3.5 GENERAL GUIDELINES
It's important be consistent in the heights you sketch across different videos. For example, an annotation of height 5 should mean the same thing in every video the agent should be the same distance away from an apple in every place you sketch height of 5. Please try to be as consistent as you can.
Drawing the correct sketch in one go is very hard and there is no need to do this. It's usually easier go back over it few times until looks right. Only submit after you't satisfied with the sketch.
You should not take more than 2 minutes to annotate a video.
29

---

Under review as a conference paper at ICLR 2022
1 BASELINES
L.1 MODEL-FREE BASELINE: R2D2
For our model-free bascline, use R2D2 (Kapurowskii 2018) with procedural reward of +1 for each apple eaten (and in the safe exploration environment, a penalty of -1 when falling off the edge). Hyperparameters were chosen by sweeping over learning rate, target network update period, AdamW W epsilon and batch size. We evaluate using the three sizes of safe exploration environment, choosing the hyperparameters which lead to convergence (all three apples being eaten) as few ac- tor steps possible. (We omit the dangerous blocks environments from hyperparameter evaluation because training never converges in these environments.)
Hy Hyperparameter
Value
Number of actors
32
Actor parameter update interval
100
Target network update interval
400
Replay buffer size
105
Batch size
64
Optimizer
A Loshchilov & Hutter 2017)
Learning rate
3 x 10
AdamW epsilon
10 10-4
Weight decay
10
Network hy beeparraaeeeer
Value
Number of groups of blocks
4
Number of residual blocks per group
2, 2, 2,
Number of convolutional layers per residual block
2, 2, 2,
Number of convolutional channels per group
16, 32, 32, 32
LSTM size
512
Linear layer size
512
Other hyperparameters in Kapturowski et al. (2018).
1.2 RANDOM-ACTIONS BASELINE
For the random- -actionss baseline, in order to give it the best chance of good performance possible, we hand-craft a random action generator specifically for our task order to give the agent a high chance of properly exploring the environment.
Each sequence of actions generated consists of random number of turn actions (committing ran- domly to either turning left or turning right, then following that turn direction for all actions), fol- lowed by a random number of 'move forward' actions, followed by another random number of turn actions, and so on.
The random-action baseline uses two hyperparameters: the maximum number of sequential 'turn' actions, and the maximum number of sequential 'move forward' actions. We chose values for these hyperparameters through a grid search over all six environment variants, and choosing the combina- tion of values which lead to the highest average number of apples eaten.
Hyperparameter
Value
Maximum number of sequential 'turn' actions
5
Maximum number of sequential 'move forward' actions
20
30

---

Under review as a conference paper at ICLR 2022
J EXAMPLE REWARD SKETCHES
In each of the following images, the top half shows a reward sketch given by contractor (with horizontal lines reward values +1, 0 and -1), and the bottom half shows predicted observations corresponding to marked points on the sketch.
J.1 SMALL SAFE EXPLORATION ENVIRONMENT, RANDOM TRAJECTORIES
31

---

Under review as a conference paper at ICLR 2022
J.2 SMALL SAFE EXPLORATION ENVIRONMENT, OPTIMISED FOR MINIMUM PREDICTED REWARD
32

---

Under review as a conference paper at ICLR 2022
J.3 SMALL DANGEROUS BLOCKS ENVIRONMENT, OPTIMISED FOR MINIMUM PREDICTED REWARD
33

---

Under review as a conference paper at ICLR 2022
J.4 SMALL SAFE EXPLORATION ENVIRONMENT, OPTIMISED FOR MAXIMUM PREDICTED REWARD
34

---

Under review as a conference paper at ICLR 2022
K EXAMPLE DYNAMICS MODEL ROLLOUTS
Each of the following rollouts generated by generating reset state from the real environment, using the corresponding observation to initialise the dynamics model, and then stepping both the real environment and the dynamics model using the same sequence of 300 actions, sampled from the test set of demonstrations given by contractors. The top row shows ground-truth observations from the real environment, while the bottom row shows predicted observations, both sampled at regular intervals from the full 300-step trajectory.
These rollouts are not cherry-picked or otherwise curated.
K.1
SMALL CLIFF EDGE ENVIRONMENT
35

---

Under review as a conference paper at ICLR 2022
36

---

Under review as a conference paper at ICLR 2022
K.2 SMALL DANGEROUS BLOCKS ENVIRONMENT
37

---

Under review as a conference paper at ICLR 2022
38


Reviews:
[{"review_id": "srpfTdns_2", "reviewer": "Reviewer_yf75", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I generally like the idea of the paper to bring ReQueST together with human feedback through reward sketches. The paper is well-written and easy to follow. The experimental setup is also well-defined and described. However, the second environment should be better justified as the setting looks very unusual.\nThe authors clearly state their contributions and delineate it from previous work. The use of real human data and reward sketches is laudable, and an important contribution. Experiments are replicated 10 times, which is a sensible amount.\n\nHowever, this only seems to be a very incremental work over the existing algorithm. Additionally benchmarking ReQueST on 3D environments is valuable but not a contribution that is sufficient for publication in ICLR. The paper is better suited as a workshop contribution. \n\nAlso, I'm not convinced about the numerical results. Their main safety result shows that ReQuest is unable to outperform a simple baseline (more on this baseline in a bit) during the evaluation, performing less safe and gathering fewer apples in the \"Cliff Edge\" environment. The authors should identify the problems encountered by ReQuest and their modifications here and propose a solution that improves the performance to be better than the baseline. I would also actually argue that having constraint violations in the gathered dataset should improve the performance for an algorithm (not necessarily for ReQuest), since the agent should exploit the knowledge on constraints this dataset contains.\n\nSome further points:\n-\tIn Fig. 2 you could more explicitly show the difference to original ReQueST\n-\tWhile the authors use a large amount of replications, they only include a classical RL baseline. This baseline uses a simplistic, sparse reward, rather then the richer reward sketches used by ReQuest. Also, it penalizes constraint violations by the same amount that it rewards successes. This kind of baseline method will never perform safely during training.\n-\tThe paper should also compare itself against well-known SafeRL algos and/or against classifier-based approaches (such as the original version, why don’t you compare to vanilla-ReQueST in order to show the benefits?). Also, while the authors argue against imitation and offline learning, such approaches could also serve as baselines and provide experimental evidence.\n-\tSince they claim to provide superior safety, the authors should compare their method to recent Safe RL methods, like 2020 Stooke: Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, or 2020 Srinivasan: Learning to be Safe: Deep RL with a Safety Critic. The latter method uses a pretraining step to bootstrap the safety critic before training, which could use a similar dataset than the one used in this study. Rather than only evaluating ReQuest, it could be a more interesting paper if multiple Safe RL approaches based on the dataset are evaluated.\n", "summary_of_the_paper": "The paper proposes an extension to ReQueST, which learn learns a neural simulator of the environment from safe human trajectories and then learns a reward model from human feedback. This work extends ReQueST by dense reward sketches on imagines trajectories and evaluates the idea on a visually-complex 3D environment. The paper also discusses the amount of training data that is needed for ReQueST to work in such a setting. The authors collect 160 person-hours of \"safe\" human exploratory trajectories and 10 person-hours of reward sketches. Their results show that the application of ReQuest results in \"3 to 20\" times less constraint violations compared to a non-safe baseline.", "summary_of_the_review": "I generally like the idea of the original paper (and hence also this paper) but in my opinion this paper is not enough for publication in ICLR. The contribution is very limited over the original ReQueST.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6QEaweQOXT", "reviewer": "Reviewer_gqNf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows that ReQueST is plausibly a general purpose solution to  the safe exploration problem: where safe behaviour is learned from humans, rather than given by a procedural function, and the simulator can be learned from data.\n\nThere are several weakness, for example, the task of apple picking, is this sufficient to show the challenge of RL or more difficult tasks should be used?\n\nThe paper also does not do a detailed comparison with existing approaches. If no existing approaches can achieve what the paper proposed to do, then please clearly say so. Otherwise please do a comparison to strengthen the  claims of novelty of the paper.", "summary_of_the_paper": "This paper shows that  ReQueST can be used to train an agent in a 3D environment with an order-of-magnitude reduction in instances of unsafe behaviour than typically required with reinforcement learning. No procedural specification of safe behaviour is required with minimal assumptions other than unsafe or near-unsafe states being recognisable by humans.\n\n", "summary_of_the_review": "Overall it is an interesting paper with some good results shown. I am not sure the results are sufficient enough. I would also like to see more detailed comparisons with the state-of-the-art.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0QO1ilEDJY", "reviewer": "Reviewer_rmJm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Concerns\n1. This method ensures safety by learning the dynamics of safe human demonstrations. However, if some of the human demonstrations are unsafe, would there be a way to know? Or would this method be sensitive to those cases?\n2. This method requires a lot of human data, which is ok considering the challenge of the task. One concern is that it seems unclear how to determine when the training for dynamics (and reward) are \"completed.\" Especially for safety-critical tasks like the ones suggested by the authors, what would be the indicators in the proposed method that can suggest: \"ok, now the learned model is safe enough, please feel free to deploy.\"\n3. A programmatic reward bonus is applied to the proposed method, not baselines. Would this mean that the reward learning component in this method is insufficient to recover the reward function?\n\n### Minor points\n1. Env2 has blocks in the world, where if the agent moves the block, the reward it will receive will change. This is the main reason why in the results, model-free RL performs much worse than the proposed method.\n    * I think this is a good example of reward hacking, where the defined reward - apple picking and avoiding touching blocks, does not match the actual goal of the task - apple picking. Hence, model-free RL performed poorly. The proposed method leverage human demonstration to avoid suffering from the misspecified reward.\n    * This indicates the power of the proposed method to handle misaligned rewards. However meanwhile, this might also indicate that the proposed method cannot completely outperform model-free RL when the reward is specified well, such as in the cliff env.\n", "summary_of_the_paper": "\n### Contributions\n* This paper proposes a safe model-based deep RL approach where\n    * No simulator is needed. The dynamic model is learned from data.\n    * No constraint is specified.\n* This work is an extension of reward query synthesis via trajectory optimization (ReQueST).\n\n* ReQueSt (previous work)\n    * Algorithm\n        * Dynamics learned from (potentially unsafe) random exploration.\n        * Reward learned from binary human feedback that is generated by a procedural reward function.\n        * Demonstrated in State-based 2D navigation (non-pixel-based), Image-based Car Racing (pixel-based, 64x64×3).\n    * Regarding safety\n        * ReQueST avoids the problem of safe exploration by allowing agents to explore these states in a simulated model without having to visit them in the real environment.\n* This work\n    * Algorithm\n        * Dynamics learned from safe demonstration provided by humans (160 person-hours).\n          * Similar to ReQueSt, `except that we use a larger encoder network, a deconvolutional decoder network, and train using a simple mean-squared error loss between ground-truth pixel observations and predicted pixel observations.`\n        * Reward learned from\n            * Type of feedback from humans: reward sketch (previous work) (10 person-hours).\n            * The proxies for value of information: maximization and minimization of reward as predicted by the current reward model.\n            * The algorithm for reward learning: MPC.\n        * Demonstrated in 3D env\n          * Task goal = eat all the 3 apples\n          * Env 1 = Cliff edge environment\n            * Safety = not fall off the end of the world\n          * Env 2 = Dangerous blocks environment\n            * Safety = bumping into blocks\n          * Each env has 3 Sized subvariants.\n          * Each env is pixel-based, 96×72×3.\n    * Regarding safety\n        * Safety is achieved by focusing the learned dynamic model in the safe scenarios, with only the safe data. `Given models of sufficient fidelity, this should allow us to train an agent with close to zero instances of unsafe behaviour in the real environment.`\n\n### Results\n* Env 1 = Cliff edge environment (Fig5)\n    * Safety violations during training: proposed method << model-free RL\n    * Safety violations during testing: proposed method = model-free RL << random policy\n    * Apples eaten: random policy <= proposed method < model-free RL\n    * => Proposed method vs model-free RL => tradeoff between safety violations during training and apples eaten\n* Env 2 = Dangerous blocks environment (Fig6)\n    * Safety violations during training: Proposed method << model-free RL\n    * Safety violations during testing: Proposed method << model-free RL = random policy\n    * Apples eaten: random policy = model-free RL < proposed method\n    * => Proposed method is better than model-free RL and random\n", "summary_of_the_review": "This paper is an extension of the previous work ReQueSt, with the focus on learning safe policy from human demonstrations only, without simulators or specifications. There are some concerns about the approach. It has a very nice demonstration in challenging pixel-based 3D tasks.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work takes ReQuest, an approach for safe deep reinforcement learning utilizing human feedback, and studies it's feasibility in pixel-based 3D environments (previously it was only shown to work in simple 2D environments). In order to apply ReQuest in this much more challenging settings, this novel instantiation of ReQuest learns a pixel-based dynamics model from a lot of human demonstration data, and a different (as compared to the "base" ReQuest) reward sketching approach to infer the reward function from human feedback. 

**Strengths**
globally a well motivated and well written/structured manuscript
Adresses an important problem, and shows promising results

**Weaknesses**
on the more detailed level, there are some clarity concerns (even with the lengthy appendix)
evaluation was missing some more relevant comparison (partially fixed after rebuttal/revision)
lack of technical novelty, and lack of in depth analysis of results
motivation of algorithmic choices : why did you choose the reward sketching approach that you chose? How is it different, and does it improve performance?

**Rebuttal**
The authors addressed most questions/things that were unclear and updated the paper to include an additional baseline.  

Additional baseline: First, it's great that you added this additional baseline! Yet, to me it's unclear what that additional baseline really represents (Request + sparse rewards). The original ReQuest paper also learns reward from human feedback, is that what you did for this paper? If yes then what does the sparse reward mean? Why does this version of request perform worse than your proposed version?

**Summary**
I agree with the reviewers and authors that this is a promising direction. However, in it's current form this manuscript is not ready yet for publication. My  concern are centered around motivation of algorithmic choices: The reward sketching part (while not novel in itself) is a novel component of the ReQuest pipeline, but you do not evaluate what it adds, and neither do you motivate that choice. Furthermore, the additional baseline is not clearly described, it's unclear how it's different for your proposed approach and why we see the performance improvement of your ReQuest version vs the baseline ReQuest version.
Decision: Reject