ReNeg and Backseat Driver: Learning from demonstration with continuous human feedback

1 PROBLEM FORMULATION . We seek a way learn autonomous vehicle control using only RGB input from a front-facing camera . ( Example input image in the appendix . ) Specifically , we want to produce a policy network to map states to actions in order to follow a lane as close to center as possible . We would like the agent to behave optimally in dangerous situations , and recover to safe situations . For this to happen , the data the agent trains on must include these sub-optimal states . However , we do not want to leave the safety determination up to any machine . Thus , we require that a human explorer take all the actions . This is referred to as “ Learning from Demonstration ” or LfD . An expert human does the exploration and thus can limit the danger by controlling how dangerous the negative examples are ( i.e . the human can swerve to show suboptimal driving but never drive off the road , as an RL agent likely would ) . Still , in order to get into sub-optimal states , the explorer will need to take sub-optimal actions . Ideally we could replace these sub-optimal actions with labels corresponding to the correct and optimal actions to take so we could perform supervised learning , which provides the strongest signal . However , it is notoriously difficult to assign supervised labels to driving data because it is hard to know the correct steering angle to control a vehicle when your steering has no effect on what happens to the vehicle ( Ross et al. , 2012 ) . Instead of trying to enable humans to assign supervised labels by somehow showing the consequences of their actions , we focus on letting humans assign feedback that evaluates the sub-optimal actions , without actually representing the optimal action . Thus our goal is to learn a deterministic continuous control policy from demonstration , including both good and bad actions and continuous scores representing this valence . Our problem statement is somewhere in between supervised learning and RL : we focus on a general approach that is capable of mapping continuous sensor input to an arbitrary ( differentiable ) continuous policy output and capable of using feedback for singular predetermined actions . Our problem falls within the supervised setting since our agent can not chose want actions to take , in order to avoid the agent exploring dangerous states , and all data collection is done prior to training . However , we also wish to incorporate evaluative scalar feedback given to each data point , which traditionally falls under the RL framework . We refer to this problem setting as the ReNeg framework , since we are essentially performing a regression with scalar weights attached to each data point , that can be positive and negative . For the demonstration , a human demonstrator drives , yielding state-action pairs the agent can learn from . In order to teach the agent which actions are good or bad , an expert critic , or “ backseat driver , ” labels the actions with a continuous scalar . For a viable solution to this problem , we need to define a loss function that induces an effective and robust policy . We also need to choose what driving data to collect to expose the agent to a variety of good and bad states , and in particular show the agent how to get out of bad states where an accident is imminent . What data to collect is non-obvious : we want to explore a range of good and bad states so the agent learns a reasonable policy for how to act in all kinds of states . However , to make this feasible in the real world , we want to avoid exploring dangerous states . Finally , we need to carefully choose a way to collect human feedback that contains signal from which the agent can learn , but is not too hard to collect . We will discuss the choices we made for these three parts of the problem in the Loss Function , Driving Data , and Feedback sections respectively . Although we validate in simulation , specifically Unity , for the sake of ease , the algorithms we test could be easily trained in the real world . 2 RELATED WORK . Learning from demonstration has mostly been studied in the supervised learning framework and the Markov Decision Process ( MDP ) framework . In the former , it is generally known as imitation learning ( or behavioral cloning in the AV setting ) , and in the latter , it is generally known as apprenticeship learning . There are other relevant RL algorithms , but none for LfD in our problem setting , as we will show . The supervised learning in this area has focused on a least squares regression that maps from an input state to action . The easiest approach to take here is to have an expert perform an optimal demonstration , and then simply use that as training data . The main issue here is that the runtime and training distributions can be vastly different . That is , once the trained agent is acting on its own after training and encounters states it has not seen , it does not know how to act , and strays further from the intended behavior . This problem is known as the Covariate Shift and the widely accepted solution generally follow the approach laid out in DAgger Ross et al . ( 2010 ) . DAgger allows the agent to explore and then uses the expert to label the new dataset , then training on all of the combined data . Such an approach has even been improved upon both to address scalar feeddback by incorporating experts that can label Q values with the AggreVaTeD algorithm ( Ross & Bagnell , 2014 ) , and to and to address deep neural networks by finding a policy gradient with the Deeply AggreVaTeD algorithm ( Sun et al. , 2017 ) . However , these DAgger based policies require the agent to ex-plore and make mistakes . The MDP research on apprenticeship learning has largely been on inverse reinforcement learning , or IRL ( Abbeel & Ng , 2004 ) , in which a reward function is estimated given the demonstration of an expert . However , often in this framework , the reward function is restricted to the class of linear combinations of the discrete features of the state , and the framework only allows for positive examples . In addition , there has been work on inverse reinforcement learning from failure ( Shiarlis et al. , 2016 ) , which allows for a sequence of positive or a sequence of negative examples . There is also distance minimization for reward learning from scored trajectories ( Burchfiel et al. , 2016 ) , which allows for gradation in the scores , but does not allow for an arbitrary reward function on continuous inputs or labels for atomic actions as opposed to a trajectory or sequence of actions . Moreover , these IRL methods are not a candidate for our problem , since they require an exact MDP solution with a tractable transition function and exploration to find the optimal policy . The issue we have is not that we don ’ t have the reward function , but that even with the more informative feedback , we can not use exploration to learn the optimal policy . It is interesting to note that there are off-policy RL algorithms as well , but we would like to highlight that this is not the same thing as LfD . LfD , as we use it , means that we have collected all of our data before training . This could be thought of as on-policy only if our policy and start state never changes . Whereas , off-policy RL ( e.g . Off-Policy Actor-Critic ( Degris et al. , 2012 ) , Q-Learning ( Watkins & Dayan , 1992 ) , Retrace ( Munos et al. , 2016 ) ) generally requires agents to have a non-zero probability of choosing any action in any state , in order for the algorithm to converge . ( Moreover , it is the somewhat parenthetical opinion of at least one of authors of this paper that in the off-policy policy gradient RL framework , using importance sampling to calculate an expectation over a different action distribution is fine , but changing the objective function from an expectation over the learned policy state-visitation distribution to an expectation over the behavior ( exploratory ) state-visitation distribution , as in ( Degris et al. , 2012 ) , is an unsatisfactory answer that does not give the optimal policy for the agent when running on the learned policy , as we will want to do . ) Normalized Actor Critic ( NAC ) does attempt to bridge the gap between off-policy and LfD , and works with bad as well as good demonstration , however , NAC does not allow for restricted exploration either , since it adds entropy to the objective function to encourage exploration ( Gao et al. , 2018 ) . NAC has also only been done for discrete action control , not continuous as we want to do . Finally , there are many RL algorithms that use human evaluative feedback , but none for LfD . One RL algorithm with human feedback of note is COACH ( MacGlashan et al. , 2017 ) , which is based on the stochastic policy gradient . COACH is an on-policy RL algorithm that uses human-feedback to label the agent ’ s actions while exploring . COACH ’ s view on human feedback helps us to draw connections to RL , as we will discuss later . However , COACH was designed for on-policy exploration and uses discrete feedback values of 1 and -1 , whereas we generalize to continuous values in [ −1 , 1 ] . We can not use a stochastic policy gradient in a justified manner , since we do not explore with a stochastic policy . Out of all the LfD work in the AV context , the most notable has either been on behavioral cloning ( Bojarski et al. , 2016 ) ( Pan et al. , 2017 ) or using IRL to solve sub-tasks such as driving preferences that act on top of a safely functioning trajectory planner ( Kuderer et al. , 2015 ) . To the best of our knowledge , no research so far has focused on using any kind of evaluative scalar feedback provided by a human in the context of AV control with LfD . That is , no one has solved how to take states , actions , and continuous feedback with respect to those actions , and convert them into a control policy for an AV , without having the AV explore . We believe that this is a major oversight : many AV research groups are investing huge amounts of time into collecting driving data ; if they used our model , they could improve performance simply by having an expert labeler sit in the car with the driver for no additional real time . 3 OUR APPROACH . 3.1 LOSS FUNCTION . Let θ be the steering angle in the demonstrated example ; θ̂ the steering angle predicted by the policy network ( or PNet ) ; D the absolute difference between θ and θ̂ , |θ - θ̂| ; and fθ , or f , the feedback for the demonstrated angle . We collect f ∈ [ −1 , 1 ] . The loss function we choose should have the following 3 properties : 1 . Minimizing the loss should minimize the D for positive examples . That is , when f > 0 , ∂Loss ∂D ≥ 0 . 2 . Minimizing the loss should maximize the distance between θ and θ̂ for negative examples . That is , when f < 0 , ∂Loss∂D ≤ 0 . 3 . The rate at which the loss is minimized should be determined by the magnitude of the feedback . That is , when f > 0 , ∂|Loss|∂f > 0 , and when f < 0 , ∂|Loss| ∂f < 0 . These three properties together ensure that the network avoids the worst negative examples as much as possible , while seeking the best examples . Given an input state s , the first loss function that comes to mind is what we term “ scalar loss ” : Lossscalar = fs ∗ ( θs − θ̂ ( s ) ) 2 This loss function is notable for several reasons . First , it is a generalization of mean squared error , the standard behavioral cloning loss function : LossMSE = ( θs − θ̂ ( s ) ) 2 Mean squared error is a well-principled loss function if you assume Gaussian noise in your training data . That is , you assume the probability of your data can be given by Gaussian noise around some mean and you learn to predict θ̂ as that mean . Given this assumption , you can derive MSE as the loss that produces a maximum likelihood estimate for your parameters . Let the parameters of the model be represented by p and probability be Pr . θ̂ is parameterized by p and will be used interchangeably with θ̂p , when clarity is needed . Please note that θ refers to the angle label , and not the model parameters : argmaxpPr ( data ) argmaxp ∏ θ∈data Pr ( θ|θ̂p ) argmaxp ∑ θ∈data log ( Pr ( θ|θ̂ ) ) argmaxp ∑ θ∈data log ( Pr ( θ|θ̂ ) ) argminp ∑ θ∈data −log ( Pr ( θ|θ̂ ) ) Loss = −log ( Pr ( θ|θ̂ ) ) Loss = −log ( e − ( θ−θ̂ ) 2 2σ2 √ 2πσ2 ) Loss = ( θ − θ̂ ) 2 2σ2 − log ( √ 2πσ2 ) Generally , log ( √ 2πσ2 ) is left out since it is a constant w.r.t . our parameters and so will go away when the gradient is taken , leaving : Loss = ( θ − θ̂ ) 2 2σ2 Generally , 2σ2 is also left out , since it only acts to scale the gradient , and can be accounted for by adjusting the learning rate of gradient descent , leaving : Loss = ( θ − θ̂p ) 2 However , we note that , if we interpret |f | , the magnitude of our feedback , as 12σ2 , we can view |f | as a measure of certainty . This certainly applies to a Gaussian distribution with a variance of at least 12 , since f ∈ ( 0 , 1 ] for positive examples . For negative examples , we generalize further by removing the magnitude calculation , and allowing our feedback to be negative . This enforces that we minimize the probability of negative data : Loss = f ∗ ( θ − θ̂p ) 2 To be able to easily recover behavioral cloning , we introduce two hyperparameters . The first such parameter is the ability to threshold feedback values . If we threshold , we simply replace every f with sign ( f ) . Thresholding eliminates gradations in positive and negative data . Additionally , we introduced the parameter α , which scales down all our negative examples ’ feedback : f : = max ( f , αf ) . This trades off between behavioral cloning and avoiding negative examples . We apply max ( f , αf ) after we threshold , so if we threshold with and set α to 0.0 , we recover behavioral cloning . Our scalar loss is also notable since it closely resembles a loss that induces a stochastic policy gradient . In a standard RL policy network such as REINFORCE , the gradient would be ∇p = ∇p ( Qπ ( θ ) ∗ −log ( Pr ( θ ) ) ) ( Williams , 1992 ) . The loss then , that would induce this gradient is Loss = Qπ ( θ ) ∗ −log ( Pr ( θ ) ) . R , the return , or a sample contributing to Qπ ( θ ) , is generally used . In continuous control , one could instead predict a mean θ̂ for a normal distribution and then sample your action θ from that normal distribution . As demonstrated above , if you replace log ( Pr ( θ ) ) with the probability density function for a normal distribution , the loss you wind up with is precisely MSE scaled by R. Substituting this scalar into the derivation above at every step , you get : Loss = R ∗ ( θ − θ̂ ) 2 A full derivation of this loss given the stochastic policy gradient and Gaussian policy can be found in the appendix . Clearly if we view fθ , our feedback , as Rθ̂ and assume a Gaussian policy , we get our scalar loss function . COACH in fact points out that you can view online feedback given for the current policy , fθ̂ as a sample fromQ π ( θ̂ ) , orAπ ( θ̂ ) , the advantage function , and empirically verifies that this works for on-line RL with discrete feedback ( MacGlashan et al. , 2017 ) . This similarity was useful for inspiration and ideation , but actually falls short of rigorous justification for two reasons . Since we are training off-policy , and , more specifically , on a pre-determined policy that does not explore stochastically and does not vary depending on the current predicted policy , we run into major issues justifying our loss this way . First , the main “ off-policy ” issue here is that for the stochastic policy gradient to hold , the exploration must be stochastic . However , in our case , the data is drawn from a pre-determined , deterministic policy . We can illuminate the intuition for why the stochastic policy gradient no longer works by considering a simple example . Consider the network attempting to learn the correct predicted θ̂ for a given state . Consider that there is only one demonstrated θ , -1 , with a feedback of -1 . Now , no matter what θ̂ the network predicts , the θ action that is taken during training will always be -1 , and the feedback will always be -1 . Moreover , consider that the actual optimum is θ̂ = 1 , and the network is currently predicting θ̂ = −2 . Using our scalar loss , the network will increase distance between θ and θ̂ by decreasing θ̂ , making the policy worse and worse . This would not happen when using a stochastic RL policy , since the network can explore states around the current predicted θ̂ by choosing appropriate actions . Using a stochastic RL policy , given enough samples on either side of θ̂ , the network will have larger and larger gradients , the more negative θ̂ is . But these gradients will not keeping “ pushing ” the prediction to the left of -1 , but rather will randomly cause θ̂ to move around , gradually moving to the right as it finds better feedback , and eventually converging at the optimum of θ̂ = 1 . The network will move less “ violently ” and more “ stably ” the closer θ̂ gets to 0 . And when the network eventually reaches the positive numbers , it might get “ pulled ” to the left a bit when it happens to sample a worse action , but it will not get pulled as strongly as when it samples a better action to the right . We can now intuitively see the issue : the neural network can not not influence the probability of seeing an example again , which can lead to problems with learning the policy . In RL , a policy network can try a bad action , and then move the policy away from that action and not revisit it . On the other hand , if we have a bad example in our training set for a given state , on every epoch of training , our neural net will encounter this example and take a step away from it , thus pushing our network as far away from it as possible . Taking these steps is not necessarily helpful since the network may not have favored taking the bad action before . We could use some sort of importance sampling ( Silver et al. , 2014 ) ( Degris et al. , 2012 ) , as is done with stochastic off-policy exploration , to scale down the loss for examples we are far away from . However , this would make our update have almost no effect when we are far away from positive examples , and with the deterministic exploration of ReNeg , the distinction between positive and negative examples now matters . We can ’ t have it both ways just by multiplying by the probability of θ̂ given our model . ( This happens since the Gaussian PDF decreases exponentially with difference |θ − θ̂| , but the loss only increases quadratically with the difference , due to the logarithm . Thus the gradient tends toward 0 due to the differing rates of growth , as the difference gets large . ) Moreover , importance sampling consistently reduced performance for learning from demonstration in the NAC paper ( Gao et al. , 2018 ) . Another , perhaps less significant issue , is that our feedback represents Q∗ , and not Qπ . Even if the human critic could re-assigned labels as the agent trains , he/she would have no way to sample the return from Qπ , without letting the agent explore freely . This is significant because an action that is optimal for the expert may be a dangerous and poor decision for a non-optimal policy . This is perhaps less significant than the other issue , since there are no actions in our data that could be considered both good and risky . What is good for one policy ( i.e . steering back to the middle of the road ) is generally good for all policies , and so can act rather greedily with respect to Q∗ , even though it will not always be following the optimal policy . For these reasons , we find the comparison to stochastic policy gradients useful , but ultimately uncompelling . Applying RL losses to supervised learning does not provide the mathematical justification we need . The stochastic policy gradient is no longer computing the gradient of the current policy at all . Thus , we focus primarily on extending and generalizing MSE . Yet , we can still learn from the policy gradient comparison . In particular , we acknowledge that the sign of our feedback is far more significant than it was in the RL context ( which is one of the reasons we introduce the α parameter , which scales down the importance of negative examples , in the event that we collect too many negative examples ) . In RL , using the stochastic policy gradient , it did not matter if negative examples “ pulled ” θ̂ closer to them , so long as the positive examples pulled θ̂ more , since you would eventually try one of those actions , if it is a nearby optimum . However , since actions are no longer sampled dependent on the current policy , suddenly the sign matters very much . In fact , even if we have a positive example for a state , if we have more negative examples than positive examples , we may wind up ignoring our positive examples entirely in an effort to get away from our negative examples . This case highlights the trouble inherent in using negative examples : It is hard to know how and when to take into account the negative examples . Now , let us consider negative examples in the ReNeg framework . For example , as shown in Figure 1 , if we perform a regression on positive and negative examples with more negative examples than positive in one state , we may wind up in a case where our loss is minimized by a prediction of positive or negative infinity , and thus our regression is “ overwhelmed ” by the negative examples . This led us to our second “ exponential ” loss : Lossexp = |θ ( s ) − θ̂ ( s ) |2f Using this loss , negative examples will have infinite loss at distance 0 , and then drop off exponentially with distance . We hope that this will create regressions more akin to the second image in Figure 1 . In this image , adding more negative points will still nudge the regression away more and more , but one positive point not too close by should be enough to prevent it from diverging to positive or negative infinity . It should be noted that the loss in a particular state still could only have negative examples , especially in a continuous state-space like ours where states are unlikely to be revisited . However , the reduction in loss caused by diverging to infinity would be so small that it should not happen simply due to continuity with nearby states enforced by the structure of the network . In addition , one concern with this loss could be that for positive fractional differences , and negative non-fractional differences , the desired property 3 ) of loss functions no longer holds . That is , our positive loss will not grow with f if the difference being exponentiated is a fraction . And for negative exponents , the loss will only grow if the difference is a fraction that shrinks as it is raised to increasing powers of f . However , we hope that for negative examples , distances that are more than 1 unit away will not occur often ( since 1 unit is half the distance range ) . We discuss a potential future solution in the appendix to patch this loss function . Our final loss function should produce regressions more like the final image in Figure 1 : We propose directly modelling the feedback with another neural network ( which we call the FNet ) for use as a loss function for our PNet . If this FNet is correctly able to learn to copy how we label data with feedback , it could be used as a loss function for regression with our PNet . Thus , in order to maximize feedback , our loss function would be as follows : LossFNet = −FNet ( s , θ̂ ) After learning this FNet , we can either use it as a loss function to train a policy network or , every time we want to run inference , we can run a computationally expensive gradient descent optimization to pick the best action . Because the latter does not depend on the training distribution ( so we do not have the issue of the runtime and training distributions being different ) , and it is more efficient , we choose an even easier version of the latter : we pick the best action out of a list of discrete options according to the FNet ’ s predictions . One feature of the FNet is that adding more negative points will not “ push ” our regression further away from this point , but rather just make our FNet more confident of the negative feedback there . This may not be the desired effect for all applications . Moreover , the FNet can not operate on purely positive points with no gradation . That is , behavioral cloning can not be recovered from it . 3.2 DRIVING DATA We recorded 20 minutes of optimal driving and labeled all of this data with a feedback of 1.0 . Choosing the suboptimal data and how to label it was a bit more tricky . One reason that we are not using reinforcement learning is that letting the car explore actions is dangerous . In this vein , we wanted to collect data only on “ safe ” driving . However , the neural network needs data that will teach it about bad actions well as good actions that recover the car from bad states . In order to explore these types of states and actions , we collected two types of bad driving : “ swerving ” and “ lane changing ” . The first image in Figure 2 is swerving . In swerving , the car was driving in a “ sine wave ” pattern on either side of the road . We collected 10 minutes of this data on the right side of the road and 10 minutes on the left . The second image is lane changing . For this , we drove to the right side of the road , straightened out , stayed there , and then returned to the middle . We repeated this for 10 minutes , and then collected 10 minutes on the left-hand side as well .

Summary: Despite many high profile successes in research, DeepRL is still not widely used in applications. One reason for this is that current methods typically assume the agent can learn by exploring many states and actions, however, in many real world tasks, such as driving used here, poor actions can be dangerous. Therefore, methods that can provides the flexible benefits of RL while avoiding this are of significant interest, one promising general ideas pursued for this has been to use human demonstrations.