Deep Reinforcement Learning with Implicit Human Feedback
Authors: Duo Xu
Mohit Agarwal
Raghupathy Sivakumar
Faramarz Fekri
Venue:  (2020)
Keywords: Error-Potentials
Implicit Human Feedback
Deep Reinforcement Learning
Human-assistance

Abstract: We consider the following central question in the field of Deep Reinforcement Learning (DRL): How can we use implicit human feedback to accelerate and optimize the training of a DRL algorithm? State-of-the-art methods rely on any human feedback to be provided explicitly, requiring the active participation of humans (e.g., expert labeling, demonstrations, etc.). In this work, we investigate an alternative paradigm, where non-expert humans are silently observing (and assessing) the agent interacting with the environment. The human's intrinsic reactions to the agent's behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials. The implicit feedback is then used to augment the agent's learning in the RL tasks. We develop a system to obtain and accurately decode the implicit human feedback (specifically error-related event potentials) for state-action pairs in an Atari-type environment. As a baseline contribution, we demonstrate the feasibility of capturing error-potentials  of a human observer watching an agent learning to play several different Atari-games using an electroencephalogram (EEG) cap, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm with the intent of accelerating its learning of the game. Building atop the baseline, we then make the following novel contributions in our work:
(i) We argue that the definition of error-potentials is generalizable across different environments; specifically we show that error-potentials of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the error-potentials.  
(ii) We propose two different frameworks to combine recent advances in DRL into the error-potential based feedback system in a sample-efficient manner, allowing humans to provide implicit feedback while training in the loop, or prior to the training of the RL agent.
(iii) Finally, we scale the implicit human feedback (via ErrP) based RL to reasonably complex environments (games) and demonstrate the significance of our approach through synthetic and real user experiments.


Under review as a conference paper at ICLR 2020
DEEP REINFORCEMENT LEARNING WITH IMPLICIT HUMAN FEEDBACK
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the following central question in the field of Deep Reinforcement Learning (DRL): How can we use implicit human feedback f to accelerate and op- timize the training of a DRL algorithm? State-of-the-a human feedback to be provided explicitly, requiring the active participation of hu- mans (e.g., expert labeling, demonstrations, etc.). this work, we investigate an alternative paradigm, where non-expert humans are silently observing (and assess- ing) the agent interacting with the environment. The human's ' intrinsic reactions to the agent's behavior is sensed implicit feedback by placing electrodes human scalp and monitoring what are known as event-related electric potentials. The implicit feedback is then used augment the agent's learning in the RL tasks. We develop system to obtain and accurately decode the implicit human feedback (specifically error-related event potentials) for state-action pairs in an Atari-type environment. As a baseline contribution, we demonstrate the feasibility of cap- turing еrrоr arror-ooeeniias of human observer watching an agent learning to play several different Atari-games using an electroenceepalloran (EEG) cap, and then decoding the signals appropriately and using them as an auxiliary reward func- tion to DRL algorithm with the intent of accelerating its learning of the game. Building atop the baseline, we then make the following novel contributions our work: (i) We argue that the definition of error-potentials is generalizable across different environments; specifically we show that error-potentials an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the error- prroo-potentials. (ii) We propose two differ- ent frameworks to combine recent advances in DRL. into the error-potential based feedback system in a sample-efficient manner, allowing humans to provide im- plicit feedback while training in the loop, prior to the training of the RL agent. (iii) Finally, we scale the implicit human feedback (via ErrP) based RL to rea- sonably complex environments (games) and demonstrate the significance of our approach through synthetic and real user experiments.
methods rely on any
the
INTRODUCTION
Deep Reinforcement Learning (DRL) algorithms have now beaten human experts in Go (Silver et al., 2017). taught robots become parkour masters (Heess al. 2017). and enabled truly autonomous vehicles (Wang et al. 2018). However, current state-of-the-art RL agents equipped with deep neural networks are inherently complex, difficult and time-intensive to train. Particularly in complex envi- ronments with sparse reward functions (e.g., maze navigation), the DRL agents need an inordinate amount of interaction with the environment to learn the optimal policy. Human participation can po- tentially help DRL algorithms by accelerating their training and reducing the learning costs without compromising final performance. This potential has inspired several research efforts where either an alternative (or supplementary) feedback is obtained from the human participant (Knox) 2012). Such approaches despite being highly effective, severely burden the human-in intim-iittth---t -oo-hhelooppp demanding either expert demonstrations (Ross et al. 2011) explicit feedback (Christiano et al. (017).
In this paper, we investigate an alternative paradigm that substantially increases the richness of the reward functions, while not severely burdening the human-in-the
We study the use of clec- troencephalogram (EEG) based brain waves of the numan-in-the--ooee to generate the reward func- tions that can be used by the DRL algorithmss. Such a model will benefit from the natural rich activity

---

Under review as a conference paper at ICLR 2020
of powerful sensor (the human brain), but at the same time not burden the human if the activity being relied upon is intrinsic. This paradigm is inspired by a high-level error- prooo-prcesssing system in humans that generates error-related potential/negatiiity (ErrP or ERN) (Scheffers et al. [996). When a human recognizes an error made by an agent, the elicited ErrP can be captured through EEG to inform agent about the sub-optimality of the taken action in the particular state.
As a baseline contribution, we demonstrate the feasibility of capturing pror-potentials of a human observer watching an agent learning play several different Atari-games, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm. We show that a full access approach to obtain feedback on every action pair while RL agent is learning, can significantly speedup the training convergence of RL agent. We contend that while obtaining such implicit human feedback through EEG is less burdensome, it is still time-intensive task for the subject and the experimenter alike. This, combined with the noisy EEG signals and stochasticity in inferring prror-potentias, raises significant challenges in terms of the practicality of the solution.
In this context, we first argue that the definition of ErrPs is generalizable across different environ- ments. We show that ErrPs of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the EmP. This is notably different from previ- ous approaches (Chavarriaga & Millán 2010 Salazar- Salazar-Gomez et al. 2017), where the labeled EmPs obtained in the same environment (where the RL task is performed). For any new and unseen environment, it does not require the human to go through the training phase again, and assumes no prior knowledge about the optimal state-action pairs of the environment.
We present two different frameworks to combine recent advances in DRL into the implicit human feedback mechanism (via EmPP in practical, sample-efficient manner. This reduces the cost of human supervision sufficiently allowing the DRL systems to train. Relying on Active Learning (AL) methods, our first framework allows humans to provide implicit feedback in the loop, while an RL agent is being trained. An uncertainty based acquisition function is modeled to select the samples state-action pairs for querying the implicit human feedback. However, as human is always required to be in the loop, our second framework allows humans to provide their feedback implicitly before the agent starts training. Based on the human feedback obtained during pre-training, quality (Q) function is learned over these imperfect demonstrations to provide the supplementary reward to the RL agent. We present results from real EmP experiments to evaluate the acceleration in learning, and sample efficiency, in both frameworks. In summary, the novel contributions of our work are,
1. We demonstrate the generalizability of error- prtor-potentials over various Atari-like environments (dis- crete grid-based navigation games, studied in this work), enabling the estimation of implicit hu- man feedback in new and unseen environments.
2. We propose two different frameworks combine recent advances in DRL into ErPP based feed- back system in a practical, sample-efficient manner. The first framework allows humans to pro- vide implicit feedback while training in the loop. Taking advantage of recent approaches in learning from imperfect demonstrations, in the second framework, the implicit human feedback obtained prior to the training of the RL agent.
3.
We scale the implicit human feedback (via ErP) based RL to reasonably complex environments and demonstrate the significance of our approach through synthetic and real user experiments.
1.1 RELATED WORK
Daniel et al. (2055) El Asri et al. ( Wang et al. (0016) studied RL from human rankings or rat-
ings, however rely on explicit human feedback, and assume that the feedback is noiseless. Demon- strations have been commonly used to improve the efficiency of RL (Kim et al. 2013] Chemali & Lazaric 2013) Piot al. [0114) and a common paradigm is to initialize RL algorithms with good
policy or Q function (Nair et al. 2018 Hester et al. 2018 Gao et al. 2018). In this work, we use rely on implicit feedback from non-expert humans (via ErrPs) which is inherently noisy.
Chavarriaga & Millán 2010; Iturrate et al.) 2010, Salazar- et al.) 2017) demonstrate the
benefit of ErrP in a very simple setting (i.e., very small state-space), and use ErrP-based feedback as the only reward. Moreover, in all of these works, the EmP decoder is trained on a similar game (or robotic task), essentially using the knowledge that is supposed to be unknown in the RL task. In work, we use labeled ErrPs examples of very simple and known environments to train the EmP decoder, and combine with the recent advances in DRL in a sample-efficiee manner for reasonably complex environments.
2

---

Under review as a conference paper at ICLR 2020
Game:
Catch
Game:
503
10
10
10
504
0
502
0.0
0.2
0.4 0.6
0.6
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
505
time (in 5)
time (in 5)
time (In 5)
Avg
Figure 1: Manifestation of error- proor-potentils in time- time-domain: Grand average potentials (error-minus: correct conditions) are shown for Maze, Catch and Wobble game environments. Thick black line denotes the average all the subjects. The game environments explained in section
DEFINITIONS AND PRELIMINARIES
Consider a Markov Decision Process (MDP) problem M, as tuple < P,, >, with state-space X, action-space A, transition kernel P. initial state distribution Po. accompanied with reward function R, and discounting factor 0 < 7 S 1. Here the random variable ((8, a) denotes the accumulated discounted future rewards starting from state and action a.
In this work, we only consider MDP with discrete actions and states. In model-free RL method, the central idea of most prominent approaches is to learn the Q-function by minimizing the Bellman residual, i.e., ({)) = E.r [(Q(xr,) a) - QQ(a", , a))² " and temporal difference (TD) (Tesauro) 1995) update where the transition tuple (z, a, r. x²) consists a consecutive experience under be- havior policy Modern techniques in DRL such DQN (Mnih et al., 2015) and the target network
Van Hasselt et al. 2016) are also adpoted throughout the paper.
INTEGRATING DRL WITH IMPLICIT HUMAN FEEDBACK: AN IDEAL APPROACH
The humans intrinsic reactions to the agents behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials. error-related potentials
We rely on the Riemannian Geometry framework for the classification (Barachant & Congedo 2014, Congedo et al. 2013) presented in Appendix 7. We consider the classification of reror-related potentials as a binary variable indicating the presence (i.e., action taken by the agent is incorrect) and absence of error (i.e., action taken by the agent is correct).
With the availability of implicit human feedback, we explore how the training of state-of-the-art art DRL algorithms can be accelerated. A trivial approach is to obtain feedback on every state-action pair while RL agent is learning (also known as full access). It is to add a negative penalty to the reward when ErrP is detected, and keep using the original reward from the environment without EmP detected. The evaluation result this method based on real ErPP data is shown in section [. The results validate that this method can speed up the training convergence of RL agent significantly.
We contend that while obtaining such implicit human feedback through EEG is less burdensome, it is still time-intensive task for the subject and the experimenter alike. This, combined with the noisy EEG signals and stochasticity in inferring ErrPs, raises significant challenges in terms of the practicality of the solution.
TOWARDS PRACTICAL INTEGRATION OF DRL WITH IMPLICIT HUMAN FEEDBACK
In this section, we discuss three approaches towards integrating the ErrP with recent advances in DRL in a practical manner. Firstly, we show that ErPs of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the EmP. Fur- ther, we discuss two frameworks to combine the recent advances in DRL into the implicit human feedback mechanism (via ErrP) to accelerate the RL agent learning in sample-efficient manner. The first framework allows humans to provide implicit feedback while training in the loop, without any prior knowledge on the game. In the second framework, the implicit human feedback is obtained prior to the training of the RL agent. It exploits the initially given trajectories with ErrP labels to leam reward function for augmenting the RL agent, where human with some prior knowledge is needed to specify some non-expert trajectories. Recently, Q function can be shown to have better generalization in state-space if trained with non-expert demonstrations (Luo et al. 2019).
3

---

Under review as a conference paper at ICLR 2020
4.1 ERRP GENERALIZATION ACROSS ENVIRONMENTS
Error-potentials in the EEG signals is studied under two major paradigms in human machine inter- action tasks, (i) feedback and response ErrPs: error made by human (Carter et al. 1998| Falkenstein
er et 2000 Blankertz et al., 2003, Parra et al., 2003) Holroyd & Coles 22022) (ii) interaction ErrPs:
error made by machine in interpreting human intent (Ferrez & Millán 2005). Another interesting paradigm is when human is watching (and silently assessing) the machine performing specific task Chavarriaga & Millán 2010). The manifestation of these potentials across these paradigms were found quite similar in terms of their general shape, timings of negative and positive peaks, frequency characteristics etc., (Ferrez& & Millán 2005] Chavarriaga & Millán 2010). This prompts us to ex- plore the consistency of the error-potentials across different environments (i.e.. games, in our case). We restrict the score of our work to the paradigm of human acting as a silent observer of the machine actions. In Fig we plot the grand average waveforms across three environments (Maze, Catch and Wobble), to visually validate the consistency of potentials. We can see that the shape of negativity, and the timings of the peaks is quite consistent across the three game environments studied in this work. Further, we perform experimental evaluation in section 5.2.1 to show that error-potente potenpotessals are indeed generalizable across environments, and can further be used to inform deep reinforcement learning algorithm in new and unseen environments.
4.2 FIRST FRAMEWORK: TRAINING WITH IMPLICIT HUMAN FEEDBACK IN THE LOOP
Active Learning (AL) frameworks have been proved quite successful in optimizing the learning task while minimizing the required number of labeled examples (Cohn et al. 1996; Gal et al.] 2017). In AL, an acquisition function is used efficiently select the data points requested for labeling from an external oracle. We introduce framework of training RI. agents with implicit non-expert human feedback in the loop, leveraging recent advances in active learning methods.
We present our active learning based framework in Fig. 2(a) We use an uncertainty- bassd acquisi- tion function to select the state-action pairs required for non-expert human labeling (via ErrP). Since it is critical to keep the coherence between consecutive state-action pairs shown to the human sub- ject, a full trajectory from start to end of the game can be shown. The calculation of the acquisition function is based on the state-action pair uncertainty along the trajectory, as explained in Appendix 7.3 Specifically, we model the Deep-Q-Network (DQN) by Bayesian learning methods, which have strong capacity of uncertainty estimation (Gal et al. (017). The DQN is trained with experience collected in the reply buffer, a structure commonly used in deep RI. algorithms.
In contrast the full access method, the presented framework queries for ErrP based state-action pair labeling only at the end of every NE episodes. We further store the decoded ErrP labels into buckets, to be used for future training augmentation. In every step, the RL agent inquire the negativ- ity of the current state- acttin pair from buckets, instead of EmP labeling, which reduces the number of ErrP inquiries significantly. This negativity can add negative penalty to the environmental re- ward as auxiliary.
Trajectory Generation and Selection: ErrP labeling informs the RL agent about negativity of selected actions, ideally preventing the agent from deviating from the optimal paths in the game. However, these optimal paths are unknown a priori. For generating trajectories for ErrP labeling. we empirically found that following greedily the action with largest Q value in every state based on the most updated DQN performs very well. Then the trajectory with the largest acquisition function output is selected for querying EmP labels. Three acquisition functions evaluated in experiments are all formulated based on the uncertainty estimation of Q values, and their formulations and approxi- mations are introduced in Appendix 7.3 The framework are presented in Algorithm T
4.3 SECOND FRAMEWORK: LEARNING FROM IMPERFECT DEMONSTRATIONS WITH IMPLICIT HUMAN FEEDBACK
RL algorithms deployed in the environment with sparse rewards demand heavy explorations (require a large number of trial-and-errors) during the initial stages of training. Imitation learning from a small number of demonstrations followed by RL fine-tuning is a promising paradigm to improve the sample efficiency in such cases (Večerik et al., 2017; Hester et al.) 2018 Gao et al., 2018). Inspired by the paradigm of imitation learning, we develop a novel framework that can robustly
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Integrating Implicit Human Feedback while Training
Input: Parameters Ng. Nr, acquisition function a(-)
Initialize DQN 2):,)) for 1, do
Starting from random initial state, the RL agent plays the game until the end of the episode; Update the DQN Q by experiences randomly selected from the replay buffer; In every NE episode:
Generate Nr trajectories { k=1
Nr by following the actions induced from Q:
Select the trajectory maximizing the acquisition function a(-) for obtaining EmP data;
Return the decoded ErrP labels to the RL agent for future training augmentation;
end
ROM
Tagertory
experiment
eopertment
ExP cocoder
Roward Tunition
At Nywe
Ozservaice
Roward
Action
Asvend
Fruirosmnn:
(a) Training with implicit human feedback in the loop. (b) Learning from imperfect demonstration with im- Trajectory block: trajectory generation and selection plicit human feedback. The dashed line shows trajec- tories D Dr and are used reward learning
Figure 2: Integrating DRL with Implicit Human Feedback
learn reward function to augment the DRL algorithms and accelerate the training of RL agent. This reward function is derived from reward function with imperfect demonstrations, achieved by obtaining the implicit human feedback in the form of ErrP over set of trajectories.
The flowchart of the second framework in Fig. 2(b) this framework, the trajectories in the demonstration are first criticized by ErrP labeling experiments, and quality (Q) function is learned from the labeled trajectories in the reward learning step. An alternative reward is derived from the learned quality function, augmenting the following RL algorithm. This approach is siderably different from our first framework (section[42) as we only make queries for ErrP labeling on trajectories initially given in the demonstration (rather than making queries continuously during every training step). These queries are made before the RL agent starts training, improving the ef- ficiency of the total number of labeling (implicit, ErrP based) queries made to the external oracle (human). Similar to the first framework, the demonstrations for ErP labeling can only consist of complete trajectories. We assumed that the trajectories in the demonstration are initially specified by human or other external sources, without any reward information. This is reasonable assumption since the rewards may be unknown to humans in general cases. The human subject the experiment provides feedback in an implicit manner (via EmP) state-action pairs along the trajectories, label- ing every state-action pair as a positive or negative sample. Based on the decoded ErrP labels and initially given trajectories, the proposed framework learns the reward function based maximum entropy RL methods (iebbat/2010) as explained in details in Appendix 77.4
Different from conventional imitation learning, these trajectories not given by expert policies, allowing the non-experts to demonstrate. Moreover, the Q function learned from imperfect demon- strations have better estimations states unseen in the demonstration, and provide better gen- eralization in the state-space (Luo et al. 2019).
5
EVALUATION
We have developed three discrete-grid based navigation games in OpenAl Gym emulating Atari framework (Brockman et al., 2016),, namely (i) Wobble, (ii) Catch, and (iii) Maze, shown in Fig.
We use the default Atari dimensions (i.e., 210x 160 pixels). The source codes of the games can be found in the public repository and can be used with the OpenAl Gym module.
3(a)
source code is attached with the submission for anonymity purposes
5

---

Under review as a conference paper at ICLR 2020
1207
33
(a) Game Environments
(b) Experiment Bench
(a) Learning Curve
(b) Complete Episode
Figure 3: Experimental framework
Figure 4: RL with fill access ErrP feedback.
Wobble: Wobble is simple 1-D cursor-target game, where the middle horizontal plane is divided into 20 discrete blocks. At the beginning of the game, the cursor appears at the center of the screen, and the target appears more than three blocks away from the cursor position. The action space for the agent is moving one step either left or right. The game is finished when the cursor reaches the target. Once the game is finished, new game is started with the cursor in place.
Catch: Catch is a simplistic version of Eggomanid] (Atari 2600 benchmark), where we display a divided into 10x 10 grid space, where the egg and the cart, both occupies one block. The action space of the agent consists of "NOOP" (no operation), "moving left" and "moving right". At the start of the game, the horizontal position of the egg is chosen randomly. At each time step, the egg falls one block in the vertical direction. fixed target. The Atari
single egg on the screen
time. The screen dimensions
Maze: Maze a 2-D navigational game, where the agent has to reach screen is centered and divided into 10x10 equal-sized blocks. The agent and target occupy one block. The action space consists of four directional movements. The maze architecture is kept fixed for the purpose of this work. If an agent moves, but hits wall, quick blinking of the agent is displayed, to show the action taken by the agent.
We designed and developed an experimental protocol, where ma- chine agent plays a computer game, while a human silently observes (and assesses) the actions taken by the machine agent. These implicit human reactions captured by placing raw electrodes on the scalp of the human brain in the form of EEG, The electrode cap was attached with the OpenBC platform, which was further connected to a desktop machine over the wireless channel. In the game design (developed on OpenAl Gym), we open a TCP port, and continuously transmit the current state-action pair using the TCP/IP protocol. We used OpenViBE software (Renard al.) 2010) to record the human EEG data. OpenViBB. continuously listens to the TCP port (for state-action pairs), and timestamps the EEG data in a synchronized manner. A total of five human subjects were recruited using standard procedures. We recruited five human subjects (mean age 26.8 + 1.92, female) for collecting the EEG data. For each subject, we conducted three separate sessions over multiple days. For each subject- game pair, the experimental duration was less than 15 minutes. The agent took action every 1.5 seconds. All the research protocols for the user data collection were reviewed and approved by the Institutional Review Board" ff
EEG experimental protocol:
5.1
FULL ACCESS
The full access method discussed in section B is the most preliminary approach to make EmP labels augment the RL algorithm. It has the fastest training convergence rate (provides upper bound) but makes the maximum possible queries the external oracle (human) for the implicit feedback. We use this method as a benchmark for comparing the data-efficiency of other RL augmentation methods. The results with real ErrP data of subjects are shown in Figure 4 Here the training data of ErTP decoder is from Catch game while the testing data is from Maze. We can see there significant improvement in the training convergence. It further validates the generalization capability of ErrP decoding from 1-D to 2-D navigation games. In this paper, "No ErPP" method refers to regular RL algorithms without the help of any human feedback. The success rate is deñned as the ratio of success plays in the previous 32 episodes. The training completes when the success rate reaches to 1. In all plots of this paper, solid lines are average values over 10 random seeds, and shaded regions correspond to one standard deviation. In the evaluations of this paper, the Q network is modeled by Bayesian deep learning methods, such as Bayesian DQN or bootstrapped DQN, introduced in Appendix 7.2
http:://wwnniieeeiii.o//////iige 'htt:///peeheii.cm
"The Institution name not disclosed ensure the anonymity of the author affiliations.

---

Under review as a conference paper at ICLR 2020
Maze
Catch
Webble
Generalization
10-fold CV
Game 1
Game 1 4
Game
10-fold CV
0.9
AUC Score
0.9
0.9
0.8
0.8
0.8
0.7
0.7
AUD
0.6
0.6
0.5
0.5
0.5
AUC
Sensitivity Specificity
S01
502
S06
807
S16
Maze
Wobble
(a) 10-fold CV
(b) Generalizability over subjects
(c) Generalizability over games
Figure 5: Detection performance and generalizability of EmP: (a) 10-fold CV performance of each
game i.e. no generalization, (b) generalizability from Catch to Maze over subjects compared with
10-fold CV, (c) generalizability over all combinations of three games compared with 10-fold CV.
5.2 PRACTICAL SOLUTION
In this subsection, we evaluate the performance of three approaches to practially integrate the DRL with implicit human feedback (via ErrPs).
5.2.1 GENERALIZABILITY
We first validate the feasibility of decoding ErrP signals using 10-fold cross-validation scheme for each game. In this scheme, we train and test on the ErrP samples of the same game environment. In Fig. 5(a) we show the performance of three games in terms of AUC score, sensitivity and specificity, averaged over subjects. The Maze game has the highest AUC score (0.89 + 0.05) followed by Catch (0.83 + 0.08) and Wobble (0.77 + 0.09). To evaluate the generalization capability of error- potential signals and the decoding algorithm, we train on the samples collected from Catch and test on Maze game. In Fig- [(b) we provide the AUC score performance compared with the 10-fold CV AUC score of Maze. We can see that the Catch game is able to capture more than 80% of the variability the ErrPs for Maze game. To provide deeper insights into the generalizability extent, we present the AUC score of generalizability performance over all combinations in fig. ((c) In the later subsections, we experimentally show that these performance numbers are sufficient achieve 2.25x improvement in training time (in terms of the number of episodes required).
We performed preliminary experiments to gain fundamental insights into the extent of generalizabil- ity. All the three games considered in this work, differ in terms of their action space. Wobble can move either left or right (two actions), Catch has an additional "NOOP" (3 actions), and the agent in the Maze can move in either direction (4 actions). To understand the generalizability of EmP in terms of the actions taken by the agent, we train on the Wobble, and test on the Catch game for two groups (i) when the agent moves in either direction, and (ii) when the agent stays in the place. We obtain an average AUC score of 0.7359 (+ 0. 1292) and 0.6423 (# 0. 0.1451) for both groups, respec- tively. Through paired I-test, we found the difference in mean statistically significant. Similarly, for the Catch game, we test two groups (i) when egg is close to the paddle, and (ii) when egg is far from the paddle. We found the mean AUC scores 0.71 (# 0.1) and 0.84 (= 0.12) for each group, respectively. The difference of the mean of both groups was found statistically significant.
5.2.2 EVALUATION OF FIRST FRAMEWORK
In evaluating active RL framework, we explore three forms of acquisition functions, i.c., entropy, mutual information, and confidence interval. Their expressions and approximation techniques are illustrated with details in Appendix 7.3 The benchmark performance of full access method is shown section 5.1 We first evaluate the performance of first framework with synthesized human feed- back, which is presented in Appendix 7.5. on box world (irironnent(ambaan et al. 2018).
In this section, we evaluate the first framework on Maze game with real ErrP experimental data. We use Bayesian DQN for the Q network. Three acquisition functions compared in Figure 6 with detailed statistics on Table 7.5.1 7.5.I which has similar conclusions as the synthetic case. Based on real EmP data, show that compared with full access method, the first framework can reach similar performance with much less feedback inquiries.
5.2.3 EVALUATION OF SECOND FRAMEWORK
In the evaluation of this framework, the trajectories given initially are generated based on optimal paths randomly corrupted by wrong actions, which appear with the probability of 0.2. We evaluate
7

---

Under review as a conference paper at ICLR 2020
200
Arees
Aeaditice
Tundian
Optidd
Acadelice Fundter
(a) Subject 04
(b) Subject 04
(c) Subject 05
(d) Subject OS
Figure 6: Evaluation of First Framework: Training with Implicit Human Feedback in the loop.
Comparison of Three Acquisition Functions in Figure (b) and (d).
the performance with 10 and 20 trajectories given initially. Prior training the RL agent, each subject is asked to provide feedback via ErrP on the state-action pairs along these trajectories. We conducted experiments on subjects, based on Maze game. Here the Q network modeled by Bayesian DQN. The performance of augmented RI. algorithms is shown in Figure
10 Dnf
10231
14233
10234
302
206
453
03
24
60
202
200
330
2.2
02
(a) Learning Curve
(b) Complete Episode
(c) Learning Curve
(d) Complete Episode
Figure 7: Evaluation of Second Framework: Learning from Imperfect Demonstrations Labeled by
ErrP. Figures (a) and (b) are for the demonstration with 10 trajectories, and figures (c) and (d) are
for 20 trajectories.
The reward function is shown to speed up the training convergence of the RL agent significantly. Since trajectories are randomly generated initially, the number of EmP inquiries of the second frame- work is equal to 372. (1:28.))) based on the statistics our simulations. The second framework even outperforms the full access method, with EmP inquiries on only 20 trajectories, proving its data efficiency. However, this framework needs a human or external source, who has some prior knowledge of the game, to specify the initial trajectories.
CONCLUSIONS AND FUTURE WORK
We first demonstrate the feasibility of capturing error-potentials of human observer watching agent learning to play several different Atari-games, and then decoding the signals appropriately and using them as an auxiliary reward function DRL algorithm. Then we argue that the definition of ErrPs is generalizable across different environment. In the ideal approach, we validate the aug- mentation effect of ErrP labels on RL algorithms by the full access method. Then, in the practical approach, we propose two augmentation frameworks for RL agent, applicable to different situations. The first is to integrate human into the training loop of RL agent based on active learning, while second is to learn reward function from imperfect demonstrations labeled by ErP.
The demonstration of the generalizability of error crror-potentials limited across the environments pre- sented in the paper. We have considered discrete grid-based reasonably complex navigation games. The validation of the generalization to a variety of Atari and Robotic environments is the subject of the future work. We also plan test framework of integrating implicit human feedback (via ErrPs) over robotic environments, and text the generalization capability of error- eror-potentials between virtual and physical worlds.
As future work, we plan to investigate as to how machines can be assisted in RL by using intrinsic EEG-based cooperations among humans and machines.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexandre Barachant and Stephane Bonnet. Channel selection procedure using riemannian distance for bei applications. In 201/ 5th International IEEE/EMBS Conference Neural Engineering, pp. 348-351. IEEE, 2011.
Alexandre Barachant and Marco Congedo. A plug&play p300 bei using information geometry. arXiv preprini arXiv: 1409.0107, 2014.
Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. Multiclass brain- computer interface classification by riemannian geometry. IEEE Transactions on Biomedical Engineering, 59(4):920-922) 2011.
Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. Classification of covariance matrices using a riemannian-barn based kernel for bei applications. Neuracomputing, 112: 172-178, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 459-458. JMLR. org, 2017.
Benjamin Blankertz, Guido Dornhege, Christin Schafer, Roman Krepki, Jens Kohlmorgen, K-R Muller, Volker Kunzmann, Florian Losch, and Gabriel Curio. Boosting bit rates and error detec- tion for the classification of fast- fast-paced motor commands based single-trie trial eeg analysis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 17 2003.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.011 015400. 2016.
Cameron S Carter, Todd S Braver, Deanna M Barch, Matthew M Botvinick, Douglas Noll, and Jonathan D Cohen. Anterior cingulate cortex, error detection, and the online monitoring of per- formance. Science, 280(5364)744-7493 1998.
Ricardo Chavarriaga and José del R Millán. Learning from eeg error-related potentials in noninva- sive brain-computer interfaces. IEEE transactions on neural systems and rehabilitation engineer- ing, 18(4):381-388 2010.
Jessica Chemali and Alessandro Lazaric. Direct policy iteration with demonstrations. In Twenty- Fourth International Joint Conference on Artificial Intelligence, 2015.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pp. 4299 4307, 2017.
David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145, 1996.
Marco Congedo, Alexandre Barachant, and Anton Andreev. A new generation of brain-computer interface based on riemannian geometry. arXiv preprint 17Xiv:1310.8115, 2013.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Autonomous Robots, 39(3):389 405, 2015.
Layla EI Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin. Score-based inverse reinforcement learning. In Proceedings of the 2016 International Conference Autonomous Agents & Mulriagent Systems, pp. 457-465. International Foundation for Autonomous Agents and Multiagent Systems, 2016
Michael Falkenstein, Jörg Hoormann, Stefan Christ, and Joachim Hohnsbein. Erp components reaction errors and their functional significance: a tutorial. Biological psychology, 51(2-3):87- 107, 2000.

---

Under review as a conference paper at ICLR 2020
Pierre W Ferrez and José del R Millán. You wrong! automatic detection of interaction rors from brain waves. In Proceedings of the 19th international joint conference on Artificial intelligence, number CONF, 2005.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1183- 1192. JMLR. org, 2017.
Yang Gao, Lin, Fisher Yu, Sergey Levine, Trevor Darrell, et al. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv:/802.05311 2018.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707 02286,, 2017.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Clay B Holroyd and Michael GH Coles. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological review, 109(4):679, 2002.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for 2011.
classification and preference learning. arXiv preprint
Iñaki Iturrate, Luis Montesano, and Javier Minguez. Robot reinforcement learning using eeg-based reward signals. In 2010 IEEE International Conference on Robotics and Automation, pp. 4822- 4829. IEEE, 2010.
Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems, pp. 2859-2867, 2013.
William Bradley Knox. Learning from human-generated reward. 2012.
Yuping Luo. Huazhe Xu, and Tengyu Ma. Learning self-correctable policies and value functions from demonstrations with negative sampling. arXiv preprint 11X7. 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 2292-6299. IEEE, 2018.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. arXiv preprint 1812::181, 2018.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via
posterior sampling. In Advances in Neural Information Processing Systems, pp. 3003-3011, 2013.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Lucas C Parra, Clay D Spence, Adam D Gerson, and Paul Sajda. Response error correction-a demonstration of improved human-machine performance using real-time eeg monitoring. IEEE
fransactions on neural systems and rehabilitation engineering, 1732):133-177, 2003.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Dis- covery in Databases, pp. 549-564. Springer, 2014.
10

---

Under review as a conference paper at ICLR 2020
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School Machine Learning. pp. 63-71. Springer, 2003.
Yann Renard, Fabien Lotte, Guillaume Gibert, Marco Congedo, Emmanuel Maby, Vincent Delan- noy, Olivier Bertrand, and Anatole Lécuyer. Openvibe: An open-source software platform to design, test, and use brain-computer interfaces real and virtual environments. Presence: tele- operators and virtual environments, 19(1):35-533 2010.
Bertrand Rivet, Antoine Souloumiac, Virginie Attina, and Guillaume Gibert. xdawn algorithm to en- hance evoked potentials: application to compute interface. IEEE Transactions on Biomed- ical Engineering. 56(8):2035-22)) 2043, 2009.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statisties, pp. 627-635, 2011.
Andres F Salazar-Gomez. Joseph DelPreto, Stephanie Gil, Frank H Guenther, and Daniela Rus. Correcting robot mistakes in real time using eeg signals. In 2017 IEEE International Conference Robotics and Automation (ICRA), pp. 6570-6577. IEEE, 2017.
Marten K Scheffers, Michael GH Coles, Peter Bernstein, William Gehring, and Emanuel Donchin. Event-related brain potentials and error-related processing: An analysis of incorrect responses to go and no-go stimuli. Psychophysiology, 33(1):42-555) 1996.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.018.33 2017.
Gerald Tesauro. Temporal difference learning and l-gammon. Communications of the ACM, 38(3):
58 58-68.17 1995.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforeement learning with double q- learning. Thirtieth AAAI conference on artificial intelligence, 2016.
Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang. Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint 1.XY:17077.0817. 2017.
Sen Wang. Daoyuan Jia, and Xinshuo Weng. Deep reinforcement learning for autonomous driving. arXiv preprint arXiv: 1111:1811.113299 2018.
Sida Wang, Percy Liang, and Christopher D Manning. Learning language games through interac- tion. arXiv preprint arXiv:1606. 120,044700.24447,,, 2016.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et Relational deep reinforcement learning. arXiv preprint
2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. PhD thesis, figshare, 2010.
7 APPENDIX
7.1
OBTAINING THE IMPLICIT HUMAN FEEDBACK: DECODING ERRPS
The Riemannian Geometry based framework was first proposed in (Barachant & Congedo 2014 Congedo et al. 2013 to translate teh raw EEG signals into meaningful label The raw EEG data
'The authors successfully applied the framework and won multiple Kaggle challenges. E.g., wwtpgg/l...o///eaaabiiiiiccc
Later, this framework was successfully adapted in many other crror-potential decoding works (Salazar-Gomez al. 2017)

---

Under review as a conference paper at ICLR 2020
bandpass filtered in [0.5, 40] Hz. Epochs of 800ms extracted relative to pre- pre-stimulus 200ms baseline, and were subjected to spatial filtering. In spatial filtering. prototype responses of each class, Le., "correct" and "erroneous", are computed by averaging all training trials in the corresponding daasess""DAAVA Spatial Filter" (Rivet et al. 2009 Barachant & Congedo 2014; Congedo et al., 2013). "xDAWN filtering" projects the EEG signals from sensor space (i.e., electrode space) to the source space (i.e., low-dimensional space constituted by the actual neuronal ensembles in brain firing coherently). The covariance matrix of each epoch is computed, and concatenated with the prototype responses of the class. Further, dimensionality reduction is achieved by selecting relevant channels through backward elimination (Barachant & Bonnet 2011). The filtered signals are projected to the tangent space (Barachant et a0., 3 2011) for feature extraction. The obtained feature vector is first normalized (using LI norm) and fed to regularized regression model. A threshold value is selected for the final decision by maximizing accuracy offline on the training set. We present the algorithm to decode the ErrP signals in Algorithm
Algorithm 2:
Riemannian Geometry based EmP classification algorithm (Barachant et al. (013))
Input : raw EEG signals EEG
Pre-process raw EEG signals:
Spatial Filtering: xDAWN Spatial Filter (nfilter);
Electrode Selection: ElectrodeSelect (nelec, metric= "atric"'riemann))
Tangent Space Projection TangentSpace(metrrcc "logeuclid") Normalize using L1 norm
Regression: ElasticNet:
Select decision threshold by maximizing accuracy
7.2 DEEP Q NETWORK MODELS
Here we introduce two DQN models adopted in this paper.
Bayesian DQN The first model we use is DQN architecture where the Q-function is approximated as linear function, with weights wig of the feature representation states (x(x) E R parameter- ized by neural network with weights 8 (Osband et al. 2013). Here by utilizing the DQN architecture and imposing Gaussian distributions on wa, based on Bayesian linear regression (BLR) [Rasmussen] 2003), the posterior of wa can be calculated by
D D.yo Wa NN Cov.), Wa Cov,
Cov.
(
1
22
DeT
1 +
oZ
where we construct disjoint replay buffer Da corresponding to experience with action a, and matrix 4% Rdx[[aal, vector ya i.e., the concatenation of state features and target values in set D... Therefore the posterior of Q value can be the following the Gaussian distribution,
Q(x,a) ~ N (a (o(x), (x(x)* Cova)o(x))
(2)
Bootstrapped DQN Another Bayesian DQN model we use is bootstrapped DQN (Osband et al., 2016). It explores in a similar manner as the Bayesian DQN introduced above, but uses a boot- strapped neural network to approximate posterior sample for the value. Bootstrapped DQN is also provably efficient, but adopts neural network instead of linear value function and bootstraps instead of Gaussian sampling. It implemented by K N bootstrapped estimates of the Q value in parallel, i... Qu (s, a; 0), I....,K. K.
7.3 ACQUISITION FUNCTION
In this work, three acquisition functions are explored in selecting trajectory for ErrP labeling. The trajectory is defined as a sequence of state-action pairs 7 := { au)...., (sr. ax) }. We denote the trajectory set D, the learned Q network as Q, and acquisition function as a(T, Q). The selected trajectory maximizes the acquisition function given Q. In this work, we explore three acquisition functions:
12

---

Under review as a conference paper at ICLR 2020
Entropy: Select the trajectory with the maximum entropy, which measures the uncertainty state-action pairs along the trajectory.
T(T, = = Σ H|s a
Specifically, in Bayesian and bootstrapped DQN, since Q value is approximately Gaus- sian distributed, we can directly use the differential entropy of Gaussian random variable here, i.e., [r, ag 2 = 1 log 2TreCoa.. where the variance term Cov, can be replaced by equation the variance of K heads of bootstrapped DQN.
Mutual Information: choose the trajectory with maximum mutual information between priors and posterior of state-action distributions (Houlsby et al. (011). The mutual infor- mation represents the information gain of action selections. We have
T
a(T, Q) = 1[
Σ [ss a
t=0
Following (Nikolov et al.
2018 we approximate the mutual information I[s,, a+| by where (st a,) characterizes the parametric uncertainty of Q value, and al
log(1+ a(ar.a]""
p( az) represents the intrinsic uncertainty. In this work, a(-,,) .)2 is obtained by Cov, in equation for Bayesian DQN, and by the variance of K heads in bootstrapped DQN. We approximate p( a;) by the variance of accumulated rewards Z( az). obtained by histogram statistics, C51 (Bellemare et al.. 2017) or QR-DQN (Dabney al. 2018)).
Confidence Interval: Choose the trajectory with the maximum uncertainty in ErrP label- ing. Some state-action pairs may have ambiguous results of ErrP decoding, with similar numbers of true and false results. And taking these pairs for ErrP labeling can decrease the uncertainty, making the RL agent more decisive on action selection. Confidence level of Binomial random variable is used here as the metric of uncertainty, Then acquísition function based on confidence interval can be expressed as
T
1.96
N NÝ + N!
No
N
where N₂ (Ng) are number of True (False) ErP decoding results for state-action pair at step on the trajectory.
7.4 REWARD LEARNING
Since implicit human feedback via ErrP is noisy (hence imperfect demonstrations), we model the reward learning as probabilistic maximum entropy RL problem. Following the principle of max- imum entropy, given Q function Q(:,),, the policy distribution and value function terms of Q function can be expressed as follows,
V²(x) = a log Σ expp(((s,
TQ(a s) = exp(((((x Vo(*))/o)
where is free parameter, tuned empirically. The likelihood of positive and negative state-action pair are denoted as (als) and 1 (a)ss)) When demonstrations and corresponding implicit human feedback are ready, train the function by maximizing the likelihood of both positive and negative state-action pairs in the demonstrations.
In order refine the reward shape and attenuate the variance of learning updates, we introduce another baseline function t(s) in the Q function. Hence, the Q function becomes (s(s, a) Q(8.. a) ((x). It can be proved that (:(:.)) and Q(++)) induce the same optimal policy (Ng et al. 1999). The baseline function t" (.) can be learned by optimizing 1" = arg min; J(t,, and the
objective is defined as
J(() == arg min
Σ
((Q(s,a) - 1(s)) max (ax'sQ(s', ((s')))
a'EA
(EDDD
13

---

Under review as a conference paper at ICLR 2020
where the loss function t(-) is chosen to be I 1o-rrrm through empirical evaluations. In addition to the demonstration D, we incorporate another set demonstrations DR. containing transitions ran- domly sampled from environment without reward information. The set Dn is to help the function t(-) to efficiently learn the state dynamics, and does not require human labeling, essentially keeping the number of queries same. After reward learning, consisting of learning function and base- line function, for any transition tuple (s, a, s'), the learned reward function can be represented as QB(s, a) - maXa'eA (s a'). We then use this reward function augment the following RL agent.
7.5 Box WORLD GAME ENVIRONMENT
This environment consists of an 8 x 8 pixels room with keys and boxes randomly scattered. The room also contains an agent, represented by a single black pixel, which can move four directions: up, down, left, and right. Keys are represented by single colored pixel, and boxes are represented by two adjacent colored pixels, where the pixel on the right represents the box's lock. A key can open lock if its color matches the lock. Its screen shot is shown in Figure g (a).
7.5.1 EVALUATION WITH SYNTHETIC HUMAN FEEDBACK
Here, evaluate the first framework the Box World game (Zambaldi al. 2018) with synthetic human feedback. This environment is introduced with details in Appendix 77.5 The synthetic feed- back gives noisy label on each state-action pair, where the correct (optimal) one is labeled as wrong (sub-optimal) with the probability of é1, and the wrong (sub-optimal) one is labeled as correct with the probability 62. Here, the Q network is modeled by bootstrapped DQN.
This game has combinatorially complex environment which cannot be quickly solved by a regular RL algorithm. The simulation results are shown in Figure where "No Feedback" refers to the RL algorithms without the help of human feedback. The detailed statistics of evaluations are illus- trated in Table 7.5.1 Three acquisition functions are evaluated for comparison. The mean complete episode for mutual information, entropy and confidence level are 167.0, 177.5 and 231.2 2 for Human 1, 166.3, 184.2 and 195.2 for Human We can see the first framework can achieve similar perfor- mance as full access method in terms of convergence speed, with much smaller number of inquiries. The acquisition function of confidence interval performs worst, because it does not consider the properties of the trained model. The mutual information performs better than entropy, but needs larger number of human feedback inquiries.
10
2.0
2.8
2.8
Sucoem
0.4
tu Acces
Ful Access
Ni. Info. (31
No. Info (20
Darrsy
Entray 1:01
0.
3.0
200
200
Spsede
(a) Game Screen
(b) Human
(c) Human 2
Figure 8: First Framework Evaluation on Box World Game. The number in brackets denotes the inquiry interval Ng.
Table 1: First Framework Evaluation on Box World Game with Synthesized Human (Ng: Inquiry Interval Ne: Complete Episode Ny: Number of Inquiries). Standard deviation is presented in the brackets.
Humon
Nw
Ne
Full Access
N₂
No
Mutual Information
N;
Ne:
Entropy
N;
No:
Confidence Interval Ny
0.3,
= 0.25
5
166.9/36.22 2)
721996(1236.0)
167.0(36. 166. 333.0)
1010.3(208.3 2) 713.5(177.5
177 53333)
758 91298.4
231 231.0/26.3) 195.2
1003 254 62) 809.91154
0.2, 0.15
10
155..(2007)
4562. 2132533
184.2/6199
510.6(165.3)
14

---

Under review as a conference paper at ICLR 2020
Table 2: First Framework Evaluation Maze Game with Real ErrP
Subject
N
No
Pull Access
Ny
NC
Mutual Information
Ny
NC
Entropy
N1
No
Confidence Level
N7
Subject 04
153. (5344(1:27.8) 150.0(2:0.4)
1973-1
161.2
636 17033
47200247.8
189. 189.30114999
26
145.202-135.2
Subject 05
21501((157.8
(5311226688
505 26222
2648)
194.7((109.1)
867.1((2792.0 19.
15


Reviews:
[{"review_id": "Syg9sWlRFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose using implicit human feedback, by means of error potential (ErrP) measured using an EEG cap, as an additional input in deep reinforcement learning. The authors first argue that even though feedback is implicit, it is still costly. To overcome this, they propose: (i) using active learning, i.e., an acquisition function, during training in order to reduce the need of ErrP's and (ii) building a model of ErrP that they fit during training and then use to simulate ErrP during test. The authors evaluate their ideas in a real experiment which consists of game playing with three different games and argue that ErrP generalize across games.\n\nThe use of ErrP in RL appears novel, however, there are several concerns that prevent me from recommending acceptance:\n\n1. The motivation for the use of ErrP is very weak. The authors claim that it is better to use ErrP than explicit feedback by users. However, to use ErrP, the humans have to still pay attention and follow the game. In the BCI literature, ErrP have been used mainly in the context of people with disabilities who are unable to provide explicit feedback. \n\n2. The authors use known methods to detect ErrP. As a result, basically, their system just receives a binary signal which indicates the presence of ErrP and the methodological contribution in this paper is minimal.\n\n3. Given the minimal methodological contribution, the experiments should be much more thorough. \n\n4. The authors only provide a hand wavy visual validation of the consistency of the potentials.\n\n5. In section 5.2.1, the authors talk about AUC in the first paragraph, however, they do not clearly specify what was the prediction/classification task. It is also unclear overall what is the \"success rate\". Mor generally, the authors do not provide sufficient details of their experimental setup and evaluation metrics.\n\nMinor comment: Last paragraph of section 2, adpoted -> adopted. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyx8T-E-qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of obtaining feedback from humans for training RL agents, in a way that does not require too much time or mental burden. The proposed approach does this by using implicit human feedback, obtained by measuring EEG signals from the brains of human observers. When humans spot an error (e.g., in the behaviour of the RL agent), an error potential (ErrP) is generated, which differs from human to human. The proposed approach first uses supervised learning to learn what the ErrP looks like for a particular person, and then uses this to provide an additional reward signal to the agent -- the agent receives a negative reward for state-action pairs that result in an ErrP.\n\nThe goal of this paper is to show that using implicit human feedback via ErrPs is a feasible way to speed up the training of RL agents. Towards this, the paper makes two main contributions: (1) experimental evidence that the ErrP for a human can be learned for one game, and transferred zero-shot to other games, and (2) two approaches of collecting this implicit feedback from humans for a smaller number of trajectories, instead of collecting it for all trajectories encountered by the agent during learning. The experimental evaluations, on three simple Atari-like domains, show that agents learn more quickly with implicit feedback, even when gathered for a small number of trajectories, compared to not having any feedback.\n\nThis paper is well-motivated, and tackles an important problem. However, I have several concerns related to feasibility and reproducibility, as described below. The writing also needs quite a bit of editing; there are typos, grammatical errors, and incorrect / missing figure and section references.\n\nRegarding feasibility, my first concern is that the domains considered here are relatively simple. Is there evidence that learned ErrPs would also generalize between tasks with more complex visuals? Also, in order to obtain these error signals, agent trajectories need to be slowed down significantly (from multiple actions a second to one action every 1.5 seconds). I imagine this would quickly become very tedious for people. This seems to be an inherent limitation of this approach, because relevant EEG signals happen up to 1+ seconds after an event occurs. I have doubts that this would generalize to more complex, long-horizon tasks, in which trajectories are at least hundreds of timesteps long. Finally, the study was conducted on a small number of participants (five), with a narrow range of ages and limited gender diversity. I sympathize with the difficulty in obtaining subjects for such studies, but the small sample size makes it questionable whether these results on transferability of learned ErrPs would apply to the wider population.\n\nReproducibility questions:\n- How are the labels (of error vs. not error) obtained for learning per-person error potentials? For more complex domains, it seems that it would be harder to obtain these ground-truth labels, because if there are substantially different strategies for playing the game, humans could disagree on which state-action pairs they consider to be errors.\n- In Section 4.2, how exactly does the RL agent use the saved state-action error labels (from queries to the human) to determine if the current state-action pair is an error? Is it using some form of nearest-neighbor? The notion of buckets isn't clearly described.\n\nAdditional questions and comments:\n- Why does Wobble have the worst AUC for learning ErrPs (Figure 5a)? It seems to be the game that has the most obvious errors (i.e., agent moves away from the target), so I would expect it to have the best AUC.\n- In Section 5.2.1 (Generalizability), what are hypotheses for *why* the ErrP trained on Wobble generalizes better to certain situations in Catch and worse for others? It would be useful to be able to characterise / predict how good generalization will be from one type of game / situation to another.\n- I would like to see reports of wall-clock time for collection of implicit human feedback, for full access and the two other approaches. In other words, how long do humans have to spend connected to the electrodes?\n- I'm curious how imperfect the demonstrations need to be, in order for the ErrPs for those demonstrations to provide a useful reward signal to the agent. In the paper, incorrect actions appear with a probability of 0.2, but there's no explanation for how this number was selected, and whether others were tested.\n\nMinor comments:\n- The diagram in Figure 2a is confusing because DQN is there twice —- explicitly as a component, and implicitly as part of the RL agent.\n- The x-axis labels in Figure 5b don't agree with the subject labels in Figures 4b and 7b.\n- In Figure 6, data for only two of the five participants is reported; it would be useful to include the results for the other participants in the Appendix.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl3cLLIFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces several methods for training reinforcement learning agents from implicit human feedback gained through the use of EEG sensors affixed to human subjects. The EEG data is interpreted into error-related event potential which are then incorporated as a form of noisy feedback for training three different reinforcement learning agents. The first agent (full-access) requires feedback for each state-action selected. The second agent (active-learning) requires feedback on trajectories generated every N_E episodes. The third (learning from imperfect demonstrations) requires the human to provide EEG feedback over an initial set of demonstration trajectories, and subsequently requires no further human access. These methods are evaluated across several handmade gridworld-like environments including a 10x10 maze, a game of catch, and 1-D cursor game called Wobble. Using the EEG training procedures is shown to improve the speed of reaching a good RL policy for all of the three different training algorithms. Additional experiments are conducted to test the generalizability of the error-related event potentials across games, with results indicating a reasonable degree of generalization.\n\nI like the idea of using EEG a way to reduce the burden of collecting human feedback for training reinforcement learning agents. This paper does a good job of investigating several different methodologies for combining ErrP feedback into the main loop of DQN-based RL agents. Additionally, the fact that ErrP feedback seems to generalize between domains is a promising indicator that a single person may be able to provide feedback across many different domains without re-training the ErrP decoder. While I like the paper as an interesting idea and proof of concept, there are some flaws that make me doubt it would be realizable for more complex tasks.\n\nThe drawback of this paper are the many open questions relating to the experiments:\n\n1) In Figures 4b, 6b, and 6d, what is the meaning of 'Complete Episode'?\n\n2) In order to assess how efficient each of these methods was in terms of the number of human labels required, how many human responses were needed for the \"full-access\" and \"First Framework\" experiments?\n\n3) In Figure 6 - what happened to the \"No Errp\" baseline?\n\n4) In Figure 5c - what are Game 1 and Game 2?\n\n5) Why are all the results shown on the Maze domain? Why are no results shown for Catch or Wobble?\n\n6) At an action speed of 1.5 seconds per action, I imagine that EEG is not much faster than having a human subject press a button to indicate their label. What prevents the use of faster speeds?\n\nMore broadly, I think it would be interesting to compare how effective is EEG at collecting human preferences versus pressing buttons (such as in Knox et al) or selecting preferences between trajectories (as in Christiano et al)? \n\nIt's my feeling that the experiments are more of a proof of concept and many open questions exist about whether this method would scale beyond these simple domains that DQN masters in ~300 episodes. In particular, scaling up to actual Atari games as a would go a long way towards showing scalability to a well-studied RL domain.\n\nI thought the overall clarity of the writing was somewhat lacking with many grammatical mistakes throughout, and the necessity to refer repeatedly to the Appendices in order to understand the basic functioning of the RL algorithms and reward learning (7.4). It took several passes to understand the full approach.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper explores the idea of using implicit human feedback, gathered via EEG, to assist deep reinforcement learning. This is an interesting and at least somewhat novel idea. However, it is not clear that there is a good argument why it should work, or at least work well. The experiments carried are more exploratory than anything else, and it is not clear that much can be learned from the results. It's a proof of concept more than anything else, of the type that would work well for a workshop paper. More systematic empirical work would be needed for a good conference paper.

The authors did not provide a rebuttal to reviewers, but rather agreed with their comments and that the paper needs more work. In light of this, the paper should be rejected and we wish the authors best of luck with a new version of the paper.

Decision: Reject