Title: COMBINATION OF SUPERVISED AND REINFORCEMENT LEARNING FOR VISION-BASED AUTONOMOUS CON-TROL

Abstract: Reinforcement learning methods have recently achieved impressive results on a wide range of control problems. However, especially with complex inputs, they still require an extensive amount of training data in order to converge to a meaningful solution. This limitation largely prohibits their usage for complex input spaces such as video signals, and it is still impossible to use it for a number of complex problems in a real world environments, including many of those for video based control. Supervised learning, on the contrary, is capable of learning on a relatively small number of samples, however it does not take into account rewardbased control policies and is not capable to provide independent control policies.
In this article we propose a model-free control method, which uses a combination of reinforcement and supervised learning for autonomous control and paves the way towards policy based control in real world environments. We use Speed-Dreams/TORCS video game to demonstrate that our approach requires much less samples (hundreds of thousands against millions or tens of millions) comparing to the state-of-the-art reinforcement learning techniques on similar data, and at the same time overcomes both supervised and reinforcement learning approaches in terms of quality. Additionally, we demonstrate the applicability of the method to MuJoCo control problems.

1. INTRODUCTION:

Recently proposed reinforcement learning methods (Mnih et al. (2013) , Kober & Peters (2012) , Lillicrap et al. (2015) ) allowed to achieve significant improvements in solving complex control problems such as vision-based control. They can define control policies from scratch, without mimicking any other control signals. Instead, reward functions are used to formulate criteria for control behaviour optimisation. However, the problem with these methods is that in order to explore the control space and find the appropriate actions, even the most recent state-of-the-art methods require several tens of millions of steps before their convergence (Mnih et al. (2013) , Mnih et al. (2016) ).

One of the first attempts to combine reinforcement and supervised learning can be attributed to Rosenstein et al. (2004) . Recently proposed supervised methods for continious control problems, like Welschehold et al. (2016) , are known to rapidly converge, however they do not provide independent policies. Contrary to many classification or regression problems, where there exists a definitive labelling, as it is presented in Szegedy et al. (2017) , Krizhevsky et al. (2012) , for many control problems an infinitely large number of control policies may be appropriate. In autonomous driving problems, for example, there is no single correct steering angle for every moment of time as it is appropriate to follow different trajectories; however such control sequences can be assessed by their average speeds, or time of the lap, or other criteria appropriate for some reason. The situation is the same for other control problems connected with robotics, including walking (Li & Wen (2017) ) and balancing (Esmaeili et al. (2017) ) robots, as well as in many others (Heess et al. (2017) ). In these problems, also usually exist some criteria for assessment (for example, time spent to pass the challenge), which would help to assess how desirable these control actions are.

The problem becomes even more challenging if the results are dependent on the sequence of previous observations (Heess et al. (2015) ), e.g. because of dynamic nature of the problem involving speed or acceleration, or the difference between the current and the previous control signal.

In many real-world problems, it is possible to combine the reinforcement and the supervised learning. For the problem of autonomous driving, it is often possible to provide parallel signals of the autopilot in order to use this information to restrict the reinforcement learning solutions towards the sensible subsets of control actions. Similar things can also be done for robotic control. Such real world models can be analytical, or trained by machine learning techniques, and may use some other sensors, which are capable to provide alternative information (e.g., the model trained on LiDAR data can be used to train the vision based model). However, although there were some works using partially labelled datasets within the reinforcement learning framework (Večerík et al. (2017) ), as far as we believe, the proposed problem statement, injecting supervised data into reinforcement learning using regularisation of Q-functions, is different from the ones published before. In Večerík et al. (2017) , the authors consider the problem of robotic control which does not involve video data, and their approach considers sharing the replay buffer between the reinforcement learning and demonstrator data.

The novelty of the approach, presented in this paper, is given as follows:

1. the regularised optimisation problem statement, combining reinforcement and supervised learning, is proposed;

2. the training algorithm for the control method is proposed, based on this problem statement, and assessed on the control problems;

3. the novel greedy actor-critic reinforcement learning algorithm is proposed as a part of the training algorithm, containing interlaced data collection, critic and actor update stages.

The proposed method reduces the number of samples from millions or tens of millions, required to train the reinforcement learning model on visual data, to just hundreds of thousands, and also improves the quality against the supervised and reinforcement learning.

2. PROPOSED METHOD:

The overall idea of the method is shown in figure 1 and can be described as follows: to perform an initial approximation by supervised learning and then, using both explicit labels and rewards, to fine-tune it.

For supervised pre-training, the annotated dataset should contain recorded examples of control by some existing model. The aim of this stage is to mimic the existing control methods in order to avoid nonsense control behaviour of the trained model.

For supervised and reinforcement learning based fine-tuning, a pretrained model is used as an initial approximation. Contrary to the standard reinforcement learning approaches, the pre-trained model helps it to avoid nonsense control values right from the beginning. Also, for the control it is assumed that there is access to labels, which helps to divert the reinforcement learning model from spending its time on exploring those combinations of inputs and control signals, which are most likely not to provide meaningful solutions.

2.1. SUPERVISED PRETRAINING:

Hereinafter we consider the following problem statement for supervised pretraining. Let Z be the space of all possible input signals. To simplify the formalisation, we restrict ourselves to the image signals z ∈ Z = R m×n , where m and n are the height and the width of the image, respectively. Such signals can form sequences of finite length l > 0 : z 1 , z 2 , . . . , z l ∈ Z l . We define an operator from the subsequences of real valued quantities to d-dimensional control signals π(z i , z i-1 , . . . , z i-p+1 |Θ π ) : Z p → R d , where i ∈ [p, l], p ∈ N is the number of frames used to produce the control signal, and Θ π are the parameters of the operator π(•). We denote The problem is stated as follows: given the set of N sequences ẑj ∈ Z lj N j=1 and the set of corresponding signal sequences

c i = π(z i , . . . , z i-p+1 |Θ π ), x i = (z i , z i-1 , . . . , z i-p+1 ).

ĉj i ∈ R d lj i=1 N j=1

, produced by some external control method called a 'reference actor', find the parameters Θ π of the actor π(•|Θ π ), which minimise a loss function (the used loss function is defined in formula ( 7)).

2.2. LABEL-ASSISTED REINFORCEMENT LEARNING:

The reinforcement learning method is inspired by the DDPG algorithm (Lillicrap et al. (2015) ), however, it is substantially reworked in order to meet the needs of the proposed combination of supervised and reinforcement learning working in real time. First, the problem statement and basic definitions, necessary for formalisation of the method, need to be given.

In line with the standard terminology for reinforcement learning, we refer to a model, generating control signals, as an agent, and to control signals as actions. Also, as it is usually done for reinforcement learning problem statements, we assume the states to be equivalent to the observations. Initially, the model receives an initial state of the environment x 1 . Then it applies the action c 1 , which results in transition into the state x 2 . The procedure repeats recurrently, resulting in sequences of states X n = x 1 , x 2 , . . . , x n and actions C n = c 1 , c 2 , . . . , c n . Every time the agents performs an action it receives a reward r(x i , c i ) ∈ R (Lillicrap et al. (2015) ).

The emitted actions are defined by the policy π, mapping states into actions. In general (Silver et al. (2014) ), it can be stochastic, defining a probability distribution over the action space, however here the deterministic policy is assumed. For such a deterministic policy, one can express the discounted future reward using the following form of the Bellman equation for the expectation of discounted future reward Q(•, •) (Lillicrap et al. (2015) ):

EQUATION

where π(•) is an operator from states to actions, γ ∈ [0, 1] is a discount factor.

Q(•,:

•) is approximated by the critic neural network in line with the actor-critic approach. Similarly to Lillicrap et al. (2015) , the proposed method uses a replay buffer to provide a training set.

In many of the state-of-the-art works on reinforcement learning, the actor's parameters are trained by maximisation of the critic's approximation of Q(•, •) , using gradient descent with gradients, obtained using the chain rule (Lillicrap et al. (2015) , Silver et al. (2014) ):

EQUATION

where Θ Q and Θ π are the trainable parameters of the actor and the critic, respectively. In Lillicrap et al. (2015) , Silver et al. (2014) , the greedy optimisation algorithm is not used as it might be impossible to collect diverse training sample for the critic as the actor would not be able to advance through the task. Instead, small steps in the gradient direction are applied to slightly modify the policy.

In the proposed method, contrary to many state-of-the-art methods, the optimisation is carried out in a greedy way, so that the steps of testing the current policy, updating the critic and the actor are interlaced. This is done in order to meet the requirements of real world scenarios, namely minimising the difference between the measurements per second rate in testing and training scenarios and providing the reasonable performance of the actor after the smallest possible number of epochs.

In order to avoid deterioration of performance in the pretrained model (which would essentially lead to the number of steps comparable with the state-of-the-art reinforcement learning models), the regularisation is used to bring the parameters closer towards some pre-defined (reference) policy.

In the following derivations we use a restriction that Q(•, •) ≥ 0, which, as one can easily see from Equation (1), will be true if the rewards are non-negative. We also assume (due to the practical reasons, it doesn't affect further derivations) that the control signal is bounded between values (t 1 , t 2 ), and t 1 and t 2 are not included into the appropriate operational values. Based on these practical assumptions, the training sample for the critic is augmented with the values Q(x, t 1 ) = Q(x, t 2 ) = 0 for every value x.

The optimisation problem is based on the regularised Q-function f (x, π(x|Θ π )) :

EQUATION

EQUATION

) where α ≥ 0 is some coefficient, π(•) is the reference actor, and ρ(•, •) is a differentiable distancelike function. One can easily see that in the case α = 0 it completely coincides with the reinforcement problem, and with α → ∞ the problem becomes a standard supervised training one. The weight Q(x, π(x)) is given to encourage the actor to follow the reference policy if the expected reward is high, and not to encourage much otherwise.

After differentiating this expression with respect to Θ π one can see that

EQUATION

As in Lillicrap et al. (2015) , the update of the critic is carried out by solving the following optimisation problem:

EQUATION

)

where X and Ĉ are taken from the replay buffer, Qπ is the discounted reward function approximation. One can see that this equation is recurrent, and therefore the current target for training is dependent on the values of the previous training stage.

The training procedure is carried out in the way described in Algorithm 1. In this algorithm, NUM EPOCHS denotes the maximum number of epochs during the training procedure.

In a contrast to Lillicrap et al. (2015) , the proposed model detaches the procedures of filling the replay buffer, training critic and actor into three subsequent steps, making the algorithm greedy rather than making steps every time control happens. As one can see from the algorithm, the 0-th epoch's testing episodes reflect the performance of the model with supervised pretraining. This was done to assess the performance of the same model parameterisation during the epoch, as well as to exclude the problem when the control signal frequency (and hence performance) is affected by the optimisation time.

3. RESULTS:

To demonstrate the ability of the method to learn control signals, TORCS (Espi et al.) / SpeedDreams (SpeedDreams) driving simulator was used. As a reference actor and for the baseline, the Simplix bot, which is a part of the standard package, was used. The outputs of the bot steering angles were restricted between -0.25 and 0.25. The car name was Lynx 220. In the case if the car has stuck (average speed is less than one unit as measured by the simulator), the recovery procedure is provided by the bot. The time and rewards for the recovery are excluded from consideration. The reward is defined as the current car speed measured by the simulator. The assessment has been carried out on a single computer, using NVIDIA GeForce GTX 980 Ti graphical processor on Ubuntu Linux operating system. The model parameters are described in the Appendix A.

Algorithm 1 The description of the training algorithm:

Initialise and train the parameters Θ π for the actor π(x|Θ π ) on a supervised dataset with observations and labels {X S , C S } Initialise the parameters of the critic Θ Q , k = 1 Initialise the empty replay buffer B while k ≤ NUM EPOCHS do Perform N TESTING EPISODES testing episodes of the length L TESTING EPISODES with the current parameters Θ π , where the states x i , actions c i , reference actor actions c GT i , rewards r i = r(x i , c i ) and subsequent states x i+1 are collected and put into the replay buffer B Perform N GD UPDATE CRITIC iterations of gradient descent, according to the optimisation problem (6), in order to update the parameters of the critic Θ Q ; values x i , x i+1 , c i , r i are taken from the replay buffer B Perform N GD UPDATE ACTOR iterations of gradient descent for the optimisation problem The network architecture, used for the proposed actor-critic approach, is shown in Figure 2 . For supervised pretraining, the dataset has been collected from the SpeedDreams simulator, combining the sequences of images and the corresponding control values produced by the bot. Using this dataset, the InceptionV1 network (Szegedy et al. (2015) ) is first fine-tuned from the Tensorflow Slim implementation (Silberman & Guadarrama (2016) ) on the collected dataset, mapping each single image directly into the control signals. The last (classification) layer of the network is replaced by a fully connected layer of the same size (1001) as the preceding logits layer; this layer is referred to as a feature extraction layer as it serves as an input space for the actor-critic model. Such an approach is used in order to avoid expensive storage of images in the replay buffer, as well as circumvent the challenge of training deep architectures within the reinforcement learning setting. Each of the actor and critic layers except the last contains ReLU nonlinearity (Nair & Hinton (2010) ). The actor's last layer with dimensionality d of the control signal is followed by the tanh nonlinearity, which restricts the output between the boundary values (t 1 , t 2 ) = (-1, 1); the last layer of the critic has no nonlinearities.

For supervised finetuning and pretraining stages, we use the following loss function:

EQUATION

where is a reasonably small constant (with respect to c, we use = 10 -2 ), π(x) is an operator, transforming input vectors x to the control signals, c is the reference actor control signal. This was done because some of the control signals, supplied by the bot, were large enough to devaluate the impact of the smaller values within the loss. Similarly, the distance-like function ρ in formula ( 4) is defined as:

EQUATION

where is a reasonably small constant (we use = 10 -4 ).

To implement the formalisation, described in section 2.1, we use a syamese network architecture (see Bertinetto et al. (2016, October) ), given in Figure 2 . The features, calculated by the fine-tuned network, are submitted for the current and the previous frame (p = 2 in terminology of section 2.1).

It is done in order to take into account the dynamic state of the environment, including speed. Also we need to mention that the previous control signal is submitted as an input for the critic (for the initial video frame in each sequence, we assume that the frames and control signals are the same for the current and the previous frames).

3.2. EXPERIMENTS:

Table 2 and Figure 3 contain the comparison of the results given different values of the parameter α. When α = 0, we rely solely on reinforcement learning; the higher is the parameter α, the more restriction is put to stick to the supervised labels rather than improving the policy by reinforcement learning.

In Figure 3 , the scatter points depict total rewards during each of the testing episodes, and the curves are depicting the mean total rewards for each epoch over all testing episode rewards. The total reward is calculated as an arithmetic sum of the rewards during one training episode. The left figure shows the results for all tested parameters, while the right one shows the comparison between the best performing one and the one with the largest value of α (i.e. the closest to supervised active learning). The performance of the pretrained model corresponds to the first epoch in the graph. The shaded area in the right figure shows the standard deviation of the testing episodes performance during the epoch. One can see from the figure that the smaller values of the coefficient α tend to yield worse performance; however, it may be attributed to longer convergence time as it implies more reliance on reinforcement learning exploration. At the same time, the unlimited increase of the coefficient α does not help gaining further performance improvements. Even more, one can see from the right figure that after certain point the curve for α = 100 slowly declines; we suggest that it could be caused by overfitting to the bot values. We also see that for α = 0, when the supervised data is used only during the pre-training stage, the performance is much lower than for the rest of the graphs.

Table 2 shows the total rewards for different values of parameter α. The value max R shows the maximum total reward over one testing episode during all the training time (corresponds to the highest scatter point in Figure 3 for a given parameter α), the value max R shows the maximum mean total reward, where the mean is calculated for each epoch over all testing episode total rewards (corresponds to the highest point of the curve in Figure 3 for a given parameter α).

In order to compare the rewards shown in these graphs and tables, we have also measured them the Simplix bot in the same conditions (frame per second rate) as the proposed algorithm. The total rewards for the Simplix bot, available in the SpeedDreams/TORCS simulator, are given in Table 1 .

The mean value of these rewards is R bot = 81214.07. The maximum value achieved by the bot is max R bot = 81931.15. The percentage of the proposed algorithm's rewards with respect to the bot one, in average and for the best performance, is also given in Table 2 . 2016)), while we achieve up to 92.63% of the bot's performance as reported in Table 2 . The graphs in Figures 4 and 5 illustrate the difference between the unregularised Q-function Q(x, π(x)) and its regularised counterpart f (x, π(x)) (see Equation 4). The images on the top of the figures, recorded during the testing episode of the first epoch (directly after the supervised pretraining), have been used for computation of the Q-values. The graphs below these images show the evolution of the Q-function (and its regularised version f ) across the epochs. One can see that, as expected, the regularised Q-function is steeper than the unregularised one as it penalises the values which are too far away from the reference actor control signal. The difference between Figures 4 and 5 shows the importance of the factor Q(x, π(x)) in Equation 4: the penalty for not following the control signal is less, if the expected reward is smaller. It makes the algorithm explore values further from the bot's values, if it does not provide high expected reward.

Another interesting aspect to be considered is how the proposed method would behave for other well-known control scenarios. Appendix B discusses the performance of the method for MuJoCo environments (Todorov et al. (2012) ) using OpenAI Baselines package (Dhariwal et al. (2017) ).

4. CONCLUSION:

The proposed method shows dramatic improvement in the number of samples for video data (down to just several hundred thousand) comparing to the reinforcement learning methods, as well as improves the performance comparing to both supervised and reinforcement learning. We believe that such approach, combining reinforcement and supervised learning, could help to succeed in the areas of complex spaces where the state-of-the-art reinforcement learning methods are not working yet, as well as towards practical usage for real world models such as autonomous cars or robots.

However, there are still a few limitations of the proposed method. First, it still requires label data through all the course of training. We believe that in the future work it should be possible to reduce usage of training data to a limited number of labelled episodes. Such decrease of the training data could benefit to the range of practical tasks solvable by the proposed approach.

These results could suggest that in certain cases the regularisation can prevent the model of further exploration beyond the reference actor performance.

For the Hopper scenario, the peak in the performance of original DDPG algorithm beyond the reference actor baseline near step 270000 suggests that the original DDPG algorithm may be unstable for this task, which also holds for DDPG with exponential decay as the regularisation coefficient becomes negligibly small. At the same time one can see that the model with the fixed regularisation coefficient can reach performance beyond the reference actor.

It could be concluded from the graphs for the InvertedDoublePendulum task that convergence depends in this case on the initial value of parameter α. The larger initial value for the DDPG with exponential decay appears to give better results due to heavier reliance on the supervised part. What is important is that all versions of the method are able to maintain stable performance after the initial training episode.

For the Swimmer scenario, the exponential α setting allowed to go for some period of time beyond the reference actor baseline; at the same time, the version with α = 0.1, while not beating the reference actor, shows smaller variance than the original algorithm, and most of the time exposes better average performance than the original DDPG method. 

:



Figures and Tables:

Figure 1: The overall scheme of the proposed method

Figure 2: The scheme of the actor (left) and critic (right) networks

Figure 3: Sum-of-rewards for different parameters α

Figure 6: Results in MuJoCo environments, top to bottom: HalfCheetah, Hopper, InvertedDou-blePendulum, Swimmer. The average reference actor performance is shown by black horizontal bar, α = 0 corresponds to the original DDPG algorithm

Simplix bot rewards

Table content: <table><tr><td colspan="8">Test episode 1 Test episode 2 Test episode 3 Test episode 4 Test episode 5 Average</td></tr><tr><td>81315.25</td><td colspan="2">81779.02</td><td>79481.26</td><td/><td>81563.64</td><td colspan="2">81931.15</td><td>81214.07</td></tr><tr><td colspan="8">Table 2: Maximum sum-of-rewards values for different parameters α (for α = 0 these values are</td></tr><tr><td colspan="4">obtained during supervised pretraining)</td><td/><td/><td/></tr><tr><td>α</td><td>max R</td><td colspan="2">100% • max R max R bot</td><td colspan="2">Epoch max R</td><td>100% •</td><td>max R R bot</td><td>Epoch</td></tr><tr><td colspan="3">100 75511.69 92.16</td><td/><td>9</td><td colspan="2">72180.06 88.88</td><td>10</td></tr><tr><td colspan="3">0.25 75896.42 92.63</td><td/><td>17</td><td colspan="2">73339.50 90.30</td><td>14</td></tr><tr><td>0.1</td><td colspan="2">75866.78 92.60</td><td/><td>32</td><td colspan="2">73380.70 90.35</td><td>15</td></tr><tr><td colspan="3">0.05 75379.34 91.97</td><td/><td>27</td><td colspan="2">71670.40 88.25</td><td>23</td></tr><tr><td colspan="3">0.01 67766.42 82.71</td><td/><td>20</td><td colspan="2">65806.41 81.03</td><td>20</td></tr><tr><td>0</td><td colspan="2">62724.68 76.56</td><td/><td>1</td><td colspan="2">58469.25 71.99</td><td>1</td></tr></table>

Peer review (iclr2018):
 This paper proposes leveraging labelled controlled data to accelerate reinforcement-based learning of a control policy. It provides two main contributions: pre-training the policy network of a DDPG agent in a supervised manner so that it begins in reasonable state-action distribution and regalurizing the Q-updates of the q-network to be biased towards existing actions. The authors use the TORCS enviroment to demonstrate the performance of their method both in final cumulative return of the policy and speed of learning. This paper is easy to understand but has a couple shortcomings and some fatal (but reparable) flaws:. 1) When using RL please try to standardize your notation to that used by the community, it makes things much easier to read. I would strongly suggest avoiding your notation a(x|.Theta) and using .pi(x) (subscripting theta or making conditional is somewhat less important). Your a(.) function seems to be the policy here, which is invariable denoted .pi in the RL literature. There has been recent effort to clean up RL notation which is presented here: https://sites.ualberta.ca/~szepesva/papers/RLAlgsInMDPs.pdf. You have no obligation to use this notation but it does make reading of your paper much easier on others in the community. This is more of a shortcoming than a fundamental issue. 2) More fatally, you have failed to compare your algorithm*s performance against benchline implementations of similar algorithms. It is almost trivial to run DDPG on Torcs using the openAI baselines package [https://github.com/openai/baselines]. I would have loved, for example, to see the effects of simply pre-training the DDPG actor on supervised data, vs. adding your mixture loss on the critic. Using the baselines would have (maybe) made a very compelling graph showing DDPG, DDPG + actor pre-training, and then your complete method. 3) And finally, perhaps complementary to point 2), you really need to provide examples on more than one environment. Each of these simulated environments has its own pathologies linked to determenism, reward structure, and other environment particularities. Almost every algorithm I*ve seen published will often beat baselines on one environment and then fail to improve or even be wors on others, so it is important to at least run on a series of these. Mujoco + AI Gym should make this really easy to do (for reference, I have no relatinship with OpenAI). Running at least cartpole (which is a very well understood control task), and then perhaps reacher, swimmer, half-cheetah etc. using a known contoller as your behavior policy (behavior policy is a good term for your data-generating policy.) 4) In terms of state of the art you are very close to Todd Hester et. al*s paper on imitation learning, and although you cite it, you should contrast your approach more clearly with the one in that paper. Please also have a look at some more recent work my Matej Vecerik, Todd Hester & Jon Scholz: *Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards* for an approach that is pretty similar to yours. Overall I think your intuitions and ideas are good, but the paper does not do a good enough job justifying empirically that your approach provides any advantages over existing methods. The idea of pre-training the policy net has been tried before (although I can*t find a published reference) and in my experience will help on certain problems, and hinder on others, primarily because the policy network is already *overfit* somewhat to the expert, and may have a hard time moving to a more optimal space. Because of this experience I would need more supporting evidence that your method actually generalizes to more than one RL environment.