# Design of a Reinforcement Learning-Based Lane Keeping Planning Agent for Automated Vehicles
## 1 Introduction  
Deep Learning (DL) has gained tremendous interest in the field of vehicle control and motion planning in general thanks to the success of DL in other fields where the advantages of Convolution Neural Networks (CNN) are heavily utilized such as semantic segmentation, scene understanding, object detection, and recognition [1, 2]. The early concepts for designing autonomous functions used rule-based systems where the parameters of the conditions are tuned over tests. This type of system has limitations because it is hard to cover all the rare events in tests that can happen in the real world. At the same time, Deep Learning techniques feature adaptation and self-tuning. Consequently, these techniques are able to generalize behaviors for unseen cases. These characteristics make DL an obvious choice for solving vehicle control problems. The vehicle control problem can be categorized into three main groups. The first is lateral motion control of the vehicle, which is dedicated to managing the in-lane location of the vehicle and also handling maneuvers that require lateral control such as lane-changing. The second is a longitudinal motion control, which is dedicated to managing the brake and throttle pedals to hold speed, thus maintainingthe appropriate following distance. The third group is when the lateral and longitudinal motion have to be controlled simultaneously. For a more comprehensive study on the field of DL based vehicle control, see in [3,4]. The authors in the Autonomous Land Vehicle in a Neural Network (ALVINN) [5,6] project endeavor to utilize supervised learning (SL), where the input is raw camera data (\(30\times 32\)), and the neural network only has one hidden layer with four neurons, while the output is one discrete steering angle out of thirty. The training dataset is enlarged with data augmentation, which also decreased the occurrence of overfitting. After the training, the real-word test shows that the trained neural network is able to hold the vehicle on the centerline of the lane more precisely than the human driver could. The following significant result of SL in the vehicle control field is the DARPA Autonomous Vehicle (DAVE) project [7,8]. It has to be mentioned that the innovation in computational hardware has a significant role in the success of this project because it uses a CNN that has 250,000 tunable parameters. The training data is collected from a human driver with cameras implemented to the vehicle. For the steering of the vehicle, an end-to-end solution is carried out with the help of a CNN. The trained CNN is able to detect the main signs that are painted on the road. Moreover, it can perform lane-keeping in a way when it does not need any human intervention. Another remarkable result is in [9], where the authors utilize CarSim [10] for data generation, which is trained with various optimizers after filtering out the degenerated samples from the dataset. The trained neural networks are only evaluated in the same simulator. The results showed that the steering signal created by the neural network is noisy, which suggests the use of Recurrent Neural Networks that has a memory of earlier inputs. By utilizing RNNs, the neural network can understand being in a maneuver, which can result in a smooth steering signal. In [11], the authors train a Convolution Long Short-Term Memory Neural Network (C-LSTM). In training, camera data are used with time dependencies. The performance of the trained network compared to another network trained with simple CNN [12] shows that the C-LSTM reaches a higher performance level in steering the vehicle. The first reinforcement learning (RL) based control is inspired by the concept of ALVINN [13]. The output of the neural network is discrete steering angles as in ALVINN. It has an essential advantage over the SL-based concepts, which is the ability to learn in an online manner from experiences gathered through interactions with the environment. The architecture is a simple feedforward network in which raw camera data are used as input.