A Complete Guide of Udacity Self-Driving Nanodegree Program — Part Two
By: Charing Chen
Tags: Udacity Nanodegree, Self Driving Cars

In this part, I will go over what I learned in every project and provide some insights into how I approach those problems.

Here is part one of this article:

Project 1: Find Lane Lines

Code: Python3 using Jupyter Notebook

Packages used: OpenCV, Numpy, matplotlib

This project introduces how to use the Jupyter Notebook and the OpenCV library. In the later lessons, you will write deep learning code using Jupyter. Jupyter Notebook is widely used by data scientists to visualize intermediate results so that they don’t have to run the whole script over and over again.

In this project, you will practise using basic image processing methods like Canny Edge Detection, Hough Line Transformation and Grayscale transformation.

Hints

Continuity: When processing a video, which essentially is a stream of images, the objects shown within adjacent images would not change dramatically. Therefore, the lane lines would not change too much in terms of slopes and positions. If slopes and positions change more than 10% in two adjacent images, something is wrong. Averaging: When the algorithm detects a really off lane line, the previous results need to be used to fill in the void. There could be multiple detect fails, so you cannot just use the previous result. The average previous result is a better idea. The size of the average result: You not only need to consider the lane lines would not change drastically in a short amount of time but also need to take account of those lines that do change over time. If the averaging starting from time zero to the end of time, the result lane line would be off if the car goes through any curve, i.e slope gets averaged instead of reflecting the real slope.

Project 2: Advanced Lane Finding

Code: Python3 using Jupyter Notebook

Packages used: OpenCV, Numpy, matplotlib

This project takes a step further in image processing, including image distortion, HSV/HLS colour spaces transformation, perspective transformation and gradient filter. And then, you need to use the Sliding Window algorithm to find lane lines based on the processed image.

Hints

Runtime: The runtime of the image processing should not be too long since we are processing video in real-time. Therefore, the runtime saving and code complexity need to be considered for this project. Thresholds: There are many threshold parameters you could tune during the image processing step. Be sure to cover some extreme cases, including but not limited to the shaded area, lane colour difference and more.

Project 3: Traffic Sign Classifier

Code: Python

Package: Pickle, Sklearn, Numpy, matplotlib, random, pandas, Tensorflow

This project practises using TensorFlow and builds Convolutional Neural Networks. The German traffic signs are used to train your neural network model.

Hints

Training Data Set: The data Udacity provides is not balanced. This means when you train using these data, the trained model would bias towards a more weighted data set. Most likely it would predict everything to that sample. Therefore, the training data need to be balanced by creating more samples on those has a lower sample size. Training time: Don’t worry about your accuracy after reaching the minimum goal. There are tons of research papers that did hundreds of hours of tuning and training to achieve single-digit accuracy improvement. No one would expect you to design and trained a model in a couple of days to reach the 95% percentile. So, my advice, don’t get suck in too much.

Project 4: Behavior Cloning

Code: Python

Packages: OpenCV, Numpy, Keras

In this project, you will build a neural network to drive a car in the simulator. You will be driving around the track to create your data sets to train your model. Instead of Tensorflow, this project practices using Keras.

Hints:

Training Data Generator: Since you are creating your datasets by taking snapshots in front of the simulation vehicle, you will have a lot of data to process. You could use a partial shuffled training set to feed into the training every epoch by using a Generator. This would save you memory usage when dealing with a large amount of training data. However, due to the TensorFlow 2 compatibility issue, this did not work for me. You can try to solve the problem by look through the new Tensorflow documentation and this great tutorial. Simplify Existing Model: Instead of designing a completely new model, you can use some established model that has been trained and tested by other researchers. I referenced Nvidia PilotNet to get started. However, in theory, you don’t need all the layers in the PilotNet since that model is for driving a real car in the real environment. Our feature maps could be less since we are driving inside a simulator. I took off the first 1152 dense layers to reduce training time. Training Time and learning rate: This model would be harder to train compared to the previous project since we have more variables as input and more features to classify. Therefore, the Adaptive learning rate method could be used to approach high training accuracy faster.

Project 5: Extended Kalman Filter

Code: C++

Packages: Eugen, iostream,uWebSockets

This project starts to use C++ since it is a run-time efficiency language compare to Python. You are getting all the sensor readings via uWebsockets protocol from the simulator. This project lets you use the Radar and Lidar measurements to determine the location of a moving vehicle.

Hint: This project is straight forward. Just follow the logic and matrix equations in the class.

Project 6: Kidnapped Vehicle

Code: C++

Libraries: normal_distribution, default_random_engine

In this project, instead of using the Kalman Filter, the Particle Filter is used. Compare to Kalman Filter, the Partial Filter is easy to program and would deal well with Non-Gaussian distribution. Here is a really good and graphic video by Matlab, showing how the Particle Filter works:

Hints:

Yaw Rate of the Motion Model: During the motion prediction of our vehicle as a bicycle model, you need to pay close attention to the Yaw rate being zeros. Rise a special alert whenever you see some variable at the denominator position of a division! Check zeros!!! When the yaw rate is zeros, the prediction uses different equations. For a detailed explanation, please check out my Github Repo. Landmark Association: When associate the landmarks with the sensor data, be sure to define a reasonable sensor range to reduce processing time. If the landmarks are outside of this range, discard those landmarks in the list to reduce processing time. Debugging: One advantage of using a simulation is you can debug all variables during “driving”. This is especially true for this project, you can set breakpoints in your code and the simulation vehicle would stop until receiving new data.

Project 7: Highway Driving

Code: C++

Libraries: spline

This the most fun project I had for the entire course. Very fun to see how many things you need to consider when transferring your own driving skill to code. The simulator will take the output waypoints of your code to drive the vehicle. After setting up waypoints that meet the jerk and acceleration requirement, your self-driving code needs to process the sensor data and drive safely on this highway without crashing.

Hints:

Location of the ego car: The location of the ego car (the car you are controlling) in S coordinate is really important. If you follow the tutorial from the class, you will find this variable is overwritten to the previous last waypoint location. If you need the car_s for the lane gap calculation, be sure the save the car_s right after receiving from the simulator. Sensor Reading: The sensor reading that comes from this simulator is really realistic. They can disappear randomly and if the sensor car is close to the driven car, the reading disappears too. Your code has to handle those situations Cost function: Please do spend time and energy to come up with a cost function. This is really important because your lane changing behaviour is dependent on this.

Project 8: PID Controller

Code: C++

In this project, you get to tune PID parameters to have a stable non-crazy vehicle. A tuning logic call Twiddle is used. However, this logic is not a magic cure for PID tuning. In theory, if this logic has a variable stepping rate (gets bigger when the error is big and smaller when the error is small) and infinite run time, it would reach an ideal value with a small error.

PID tuning in the real world is a complex problem. First of all, the three variables affected each other and there will be more than one set of parameters would yield reasonable result. Second, manual tuning or giving the algorithm a reasonable starting parameter would save so much runtime. Therefore, knowing the basic principle of tuning the PID parameter is crucial when going into this project.

Hints:

Accumulated Error or Average Error: To use Twiddle, you need to know how good your PID is controlling the vehicle. There are two ways to calculate the error in my opinion: Accumulate all errors in a run or take the average error of a run. Both methods would work for the twiddle but the average error would represent a more accurate behaviour improvement after you have a stable system. Rise time vs oscillation: A faster rise time means the controller would react faster when there is an increase of error, but this would mean it would create overshoots and it would take time again for the controller to settle these overshoots. Plus, this might even introduce oscillation. However, the slower rise time would mean the vehicle would react slower and therefore more error accumulated too. Try it yourself to see how to coordinate those two to get a better controller.

Project 9: Capstone Project Program an Autonomous Vehicle

Code: ROS

This is the final project of this program and this has been marketed as the highlight of this course. It is an open-end project. You can program a real car that runs in a closed lot with a traffic light to detect. It is a good marketing strategy to let more people get interested in the course.

However, I took this during COVID, and the office was closed. So, I did not get the program a real car. This project can also be a team project. You can talk to fellow students and form a group to work on the project. I did this project myself since I am running late and want to save one-month of tuition. The final graduation only requires the submission of this project and it is not evaluated.

After all, this project is a great practice to use ROS. The CNN of traffic light recognization can be used in the ROS, but that is also optional. you could use the ground truth that being broadcast by the simulator to control the vehicle to stop at the traffic light.

Hints: