# Deep Learning Applied to Scenario Classification for Lane-Keep-Assist Systems
## 4 Driving Data  
The full dataset used to train the neural networks consists of real-world recordings obtained in seven highway and four motorway runs in the area around Graz, Austria. The dataset contains 183 channels and 304 thousand samples per channel, with a sampling rate of ten samples per second. The channels are raw sensor measurements coming from various sensors such as the lane detection camera, inertial measurement unit, encoders, etc., recorded directly from a CAN-Bus interface. As the focus of this research is classifying scenarios relevant for LKA systems, 10 relevant input channels were chosen to learn the distinct features. The used channels are: _Lane Assist Lateral Deviation_, _Lane Assist Lateral Velocity_, _Lane Assist Lateral Velocity SMO20_--smoothed with a 20-step window, _Lane Curvature_, _Lane Width_, _Steering Wheel Angle_, _Longitudinal Acceleration_, _Lateral Acceleration_, _Velocity_ and _Yaw Rate_. The model should learn to segment the data into 13 classes, as shown in Figure 5, which were hand labeled using a custom labeling tool. The labels are: _TR: Turn Right_, _TL: Turn Left_--these two classes are defined with slow velocity sharp turns, _LCLR: Lane Change Left to Right_, _LCRL: Lane Change Right to Left_, _S: Straight_--straight driving, _CL: Curve Left_, _CR: Curve Right_--curves with fixed radius, _TOL: Turn Out Left_, _TOR: Turn Out Right_--curves with increasing radius, _TIL: Turn In Left_, _TIR: Turn In Right_--curves with decreasing radius, _R: Roundabout_ and _SS: Stand Still_. The labeled data contains class probabilities for each sample from the training data. When a sample could not be assigned to one of the thirteen classes, a uniform distribution between the classes was introduced to represent the unknown state. In other words, the model should not be able to tell which class the sample belongs to by generating low probabilities for all classes. Figure 4: Window partitioning for class predictions. In the online model the input is only the current stride and an aggregated past. No information of what happened after the prediction is available. On the offline model the information what the vehicle did after the detection is available, enabling the model to make a prediction based on past and future behavior of the vehicle. To process the whole data, the windows are moved with a fix stride length generating corresponding class probabilities. Figure 5: Example scenario classes. From left to right, top to bottom the labels are: **TR**: Turn Right, **TL**: Turn Left, **LCLR**: Lane Change Left to Right, **LCRL**: Lane Change Right to Left, **S**: Straight, **CL**: Curve Left, **CR**: Curve Right, **TOL**: Turn Out Left, **TOR**: Turn Out Right, **TIL**: Turn In Left, **TIR**: Turn In Right, **R**: Roundabout and _SS:_ Stand Still. The distribution of samples per class is shown in Figure 6. A strong imbalance between the classes exists which limits the capacity to train the models. Classes TR, TL and SS appear very rarely in the dataset, whereas S, CL, CR appear with a much higher occurrence. There was no possibility to balance the classes more evenly during the recording. In addition, various time series augmentation techniques were applied but they did not show any improvements on the accuracy of the models. Furthermore, time series data, such as the present one, contain classes which repeat for a long period of time, i.e., during parts of the training the network gets as an input only one class which hinders generalization. A remedy for this problem is to shuffle the data. However, shuffling of the data is not possible if RNNs are used. The requirement of these networks is that all data is introduced sequentially. CNNs on the other hand are applied on segments of the data, i.e., windows. Inside the windows the data is sequential but individual windows can be shuffled which increases the accuracy. The average duration per class is shown in Figure 6. This information is used to determine the size of the windows used for the CNNs.