Semantic Segmentation & Deep Learning for Autonomous Driving Simulation
By: Jose Rojas
Tags: Machine Learning, Self Driving Cars, TensorFlow, Deep Learning, Udacity

Building and training the FCN

To begin, I set out to find an appropriate FCN architecture by doing some initial research by reading papers and pouring through various GitHub repositories. After having built and trained a few networks from scratch, I knew this approach would be time-consuming. A less time-consuming approach to build a prototype would be to use transfer learning, i.e use a pre-trained network and fine-tune it with new data samples instead of building and training one. Another objective during my research was to find a FCN architecture that would run in real-time using a lower-end GPU. Not only would this be essential for steering a car in simulator, eventually, I’d aspire to run the architecture in a car, making real-time performance paramount.

After some research with various papers, I discovered the ENet architecture, which is one of the fastest FCN that I could find with a working GitHub repository from the original authors along with several others. The network was pretrained on CityScapes, a dataset of 5000 semantically segmented dash cam images and labels of city driving. The network offers 15+ FPS on a NVIDIA TX1 platform, along with good accuracy (81 mIOU) on CityScapes. This network essentially checks all of the requirement boxes and would be very useful in a first pass implementation.

Evaluation of various semantic segmentation networks, accuracy vs frames per second.

After evaluating it on some camera simulator images, although not terrible, the network’s accuracy was not high enough to use without fine-tuning. The most important object class, road, was working to a degree, but many detections were incorrect enough to affect the determination of the road shape. The goal would be to produce semantically labeled output that would generally follow the road contour. The reasoning behind this accuracy failure is due to two factors. ENet is pre-trained on real dash camera images from the CityScapes dataset, not simulated images from a virtual racetrack. Thus, the features it is trained to detect do not include some of the features in the simulated images, such as red-white striped shoulder markings and cobblestone bridges. Also, CityScapes image resolution (1024x512) is also much higher than the simulator camera input (320x160), which would result in large differences in feature detection, due to the ConvNet layers not being scale invariant.

After the mediocre initial evaluation results, fine-tuning the network was the next step. Fine-tuning involved further training the network with simulator images and labels, along with reducing the number of detected object classes. The CityScapes dataset includes 19 object classes and 1 “other” class, however only 2 classes were necessary for this prototype: road and other. The fine-tuning thus involved reducing the decoder size and also generating semantically segmented labels for the new training data. The network was fine-tuned for 100 epochs using TensorFlow with 800 simulator images as a training set and 100 images as a validation set. No significant data augmentation was utilized (only image mirroring). The training time took roughly 20 minutes on a NVidia 1050ti.

Top: Original image; Middle: Original ENet model trained on CityScapes; Bottom: Fine-tuned ENet model

Displayed above are some qualitative image results before and after fine-tuning. The top image is the original sample, the middle image shows the results of the original CityScapes trained ENet model, the bottom image shows the results after fine-tuning. The middle image detects the most of the road pixels correctly, highlighted in magenta, however incorrectly interprets the red-white shoulder as part of the road. The fine-tuned model produces a more accurate result, covering almost all drivable areas, while filtering out the road shoulder. The non-magenta labels in the original ENet model are predictions of non-road classes types. The fine-tuned model only focuses on 2 classes, road (magenta) and non-road.

Automated Label Generation

An essential part of the training processes for a semantic segmentation network is generating labels for the training image set. Manually creating image labels is a labor intensive process requiring some image processing tools and patience. The label image can be generated by opening the training image in a tool like Photoshop, creating a transparent layer over the image, then creating a colored silhouette around the various objects that need classification, then exporting this layer into a tool that could perform one-hot encoding. The silhouette would need to fit tightly around the contours of these objects and would need to be consistent across images for the best possible convergence and inference accuracy. Having to do this several hundred times is an epic chore and would evoke accuracy errors through repetition fatigue.

Luckily, for this particular problem scope, the classification required only detecting road pixels in an image and not other complex shapes. Roads are fairly well-defined in shape. From the point of view of a camera traveling along the surface of a road, they are generally a contiguous area between two converging lines heading towards a vanishing point off into the horizon or towards the edge of the image if the road is curving. Roads that cars travel on are usually made of concrete and have white or yellow road markings. With these characteristics, it would be possible to build a computer vision approach to labeling the images with an automated script to save on the human effort along with improving the consistency.

While the approach was not trivial, I was able to generate an algorithm that would select pixels based on an adaptive color space thresholds. The algorithm essentially works by transforming the image into a few different color spaces, then dynamically adjusting a set of lower and upper bound thresholds for pixel values that fall within these boundaries. Afterward this initial filtering, I applied topology filters to locate contiguous pixel regions and then plotted regression lines at the region boundaries to determine the ‘edges’ of the road within the image. For this, I made an assumption that all relevant camera images would require the ego vehicle to be traveling on the surface of the road, not outside of the road boundary. I felt this was a safe assumption for training purposes, as the goal of this agent is to drive the car while it’s on a known road, not while it was off-road.

Various cases of accurately auto-labeled images using computer vision techniques. Note the variety of ground textures, shadows and edge conditions

You might now be wondering if I’m able to auto-generate image labels by detecting road pixels, then why bother generating a neural network to infer the road pixels? There are a few reasons. Firstly, the label generator was not always 100% correct — there are many edge cases that would need to be hand labeled, although the automated solution greatly reduced the human effort. Secondly, a neural network would be far more optimized and better at discovering nuanced characteristics across many road types than relying on a computer vision approach. Because of the accuracy was important in the training, the label generator’s filtering performance was much slower than the required real-time performance. Thirdly, the neural network could be re-trained to detect more than just roads if more classes were eventually added to the labels, whereas an ensemble of computer vision filters would need to be implemented to detect multiple object classes. This undoubtedly would operate more slowly as additional class types are supported.

Training the Steering Model

With the FCN part of the network working, the next step was to train an extension to the model for inferring the steering angle from the output of the FCN. Building on the principles of deep learning, my goal here was to create a deeper neural network, with one portion of the network detecting road pixels and another portion taking those outputs and using them as inputs for generating a steering angle.

The architecture for this part of the network was very simple; it would be a feed-forward network consisting of two dense layers, the first using the softmax pixel probabilities from the FCN output as new inputs and the second ending with a single output neuron, representing the steering angle. Between the two layers, a PreLU activation function was introduced as a non-linearity into the model. Additionally, L2 regularization was added to the layers to prevent the model weights from excessive over-fitting.

The training for this portion of the network would only involve these two final layers, so the FCN layers weights were “frozen”, i.e. made read-only, so back-propagation of the loss function only traversed these two layers instead of the entire model. I’ll explain more about the advantages of this isolation in the evaluation below. The label for this portion of the network is the target steering angles, so the network would learn the best steering angle for the given road pixel inputs.

To determine the steering angle labels, I chose to use a simple heuristic instead of providing the collected human operated steering angle data samples. This simple heuristic is to take the road labels used for training the FCN, determine the left and right edge of the road region, then calculate the set of middle points between the two lines across all horizontal rows in the camera image. With some assumptions about camera orientation, this would give an approximated center line for the road. A far enough point on this line from the camera would represent the point at which the car should be heading towards to continue traveling down the road. Using this sampled point and some simple trigonometry, the angle steering angle from the perspective of the driver can be approximated.

If you think about how a person would drive a car intuitively, this is roughly a similar model for steering. A driver’s eyes would focus on a point relatively far enough ahead of the vehicle along the path of the road. Steering towards that point would keep the car within the lane with enough time to react and apply minimal error correction. However if a driver continually chose a point not far ahead of the vehicle, they would afford themselves less reaction time to compensate for an over or undershoot in their intended trajectory. Estimation using a far enough point is essential in this case to minimize error by calculating the delta steering angle across a longer distance. For the heuristic used in the algorithm, I chose an appropriate ‘heading’ point that was a fixed number of pixels before the detected horizon line in the road region. For sharp turns where only one road edge is visible, the heading point would become the point where the visible road edge intersects the camera image edge.

With a means to calculate a steering angle from the road’s shape, I was able to train the model using TensorFlow. The model was trained for 10–15 epochs before converging to a minimal RMSE for the steering angles of a validation set. However an important concept to discuss is dealing with label imbalances. To correctly build a model to steer at any appropriate angle, the model must equitably train on the entire range of valid steering angles. For this data set, there are many more likely steering angles than others, causing an imbalance in the steering angle distribution. For instance, the training track had a few curves and several straight sections, causing there to be more cases for steering at lower angles than at higher angles. Another problem is given the inference is a regression, there is a continuous range of values to infer, but only a finite number of samples with which to train against.

Angle distribution: -25 to 25 degrees

In order to deal with the imbalances in the distribution and continuous range, two concepts are introduced. To handle the continuity problem, the steering angles are grouped into discrete ranges of 1 degree, a small enough value to minimize the margin of error in the angles. To handle the distribution imbalance, a method to influence of the loss during training is introduced. The idea here is to scale the error with a set of weights during back-propagation so that less likely ranges of angles have greater influence while more likely values have less influence. To generate the weights, the concept of median frequency balancing is used to scale the training loss during back-propagation by the label sample distribution. To implement, a histogram of the sample distribution frequency with a 1 degree bin size is created and is used calculate a set of weights for each range set using the following equation:

w(x) = median_frequency(X) / frequency(x),

where median_frequency(X) is the median frequency of the entire sample distribution, frequency(x) is the frequency of a particular sample bin x, w(x) is the weight for sample bin x

Evaluation

To evaluate the network’s accuracy quantitatively, there would need to be a ground truth test set of steering angles to calculate a loss based on the values generated by inference. As an equitable comparison, the test set was generated using the same auto-labeling technique used to generate the training and validation data. In cases where the auto-labeling failed, a human labeled approximation was used instead. The graph below breaks down the accuracy by steering angle to account for the distribution inequality. The RMSE for test output is fairly high (all over 1 degree of error; over 3 degrees of error for higher steering angles), while the validation was far lower. I’ve only trained with limited data and evaluated on one track at this time, so this model is over-fitting the data and have some room for improvements.

Model Evaluation: RMSE for various steering angles

I’ve also recorded a video of the model in action on the track as a qualitative evaluation. The right side of the video shows the camera sensor view, the FCN output and the steering direction based on the steering angle output from the final layer of the network (white line). You’ll note that the semantic segmentation output is imprecise; there are many gaps in road and often times there are incorrectly predicted road segments towards the horizon. However a rough contour of the road can be inferred from this noisy data (see green overlay). This is a representation of what the 2nd half of the network is tuned to ‘perceive’ from this FCN output. From this, an estimate of the final steering angle will be generated.

You’ll notice the car does oscillate from side to side, and very noticeably near the end of the video. This is due to errors in both the road pixel predictions and in the steering angle estimates. For the road predictions, the end of the video shows evidence that the model generates noisy predictions while driving on the cobble-stone bridge. The transition from the bridge to road asphalt appears to have prediction errors which most likely cause the car to improperly detect the road direction. To solve this, more data of this transition or further data augmentation is needed to improve the training data set. As far as minor oscillations, the root cause is the simulator introduces latency into the steering angle update of about 0.25 seconds, as a mild effort to model the time needed to turn the steering wheel. The model is not trained to estimate and compensate for this, resulting in delays to the steering update after prediction and eventual overshooting of the intended trajectory. A PID controller is used to reduce this effect, however further tuning is necessary.

Looking beyond the performance, this network was far more tractable and pliable in it’s training and my understanding of what to tune. There was a case for instance where there was a significant amount of noise in the predictions, resulting in large variances in predicted angles. By analyzing the output of the FCN, I was able to inspect there was considerably poor and unstable predictions in the pixels, with the root cause being a hyper-parameter setting error in batch normalization. After fixing the FCN predictions, the next challenge was to tune the steering angles so there would be less oscillations. Adjusting the heuristic for steering angle training and re-training the 2nd half of the network helped improve the results.

In summary , this approach is advantageous for a few reasons. Firstly, the isolation between the semantic segmentation training phase from the steering angle training allows for easier tuning and debugging of failure cases. By tuning the layers of the network independently, the trainer has a greater level of control over modeling the correct inference output. Secondly, the heuristic-based approach of steering angle modeling avoids errors in human-based training input collection and offers yet another hyper-parameter to tune for greater control. Thirdly, this network architecture is far more forward thinking with respect to augmenting it with capabilities to detect new types of objects and their positions, such as other vehicles. With respect to disadvantages, this network structure is not as optimized as the original network and needs GPU acceleration for real-time operation. Also, there’s additional pre-processing necessary to label the training data whereas in the original solution, the training data was unprocessed.

Next Steps

After having achieved my goal of creating a prototype of this new semantic segmentation network, my next set of goals are as follows:

Investigate if an algorithmic approach to determining the road curvature would be more accurate than a neural network. This would involve replacing the 2nd half of the network after the FCN with a computer vision algorithm to estimate the road curvature over time. This could lead to creating more hyper-parameters for further customizable control of the driving agent. Investigate the affects of data augmentation on the FCN. Adding more complexity to pre-processing the training images with geometric affine transformations, clipping, color and contrast distortions could help generalize the semantic segmentation model further and provide smoother and more accurate predictions. Test this method on new data sets and tracks. I’d like to test the steering model potentially on other simulator tracks and also on real-world driving data sets!