How I Created a Course on Lane Detection and Lane Keeping
By: Mario Theers
Tags: Python, Self Driving Cars, Education, Jupyter Notebook, Deep Learning

A while ago I was searching the web because I wanted to learn how lane-keeping systems work. I knew that these systems use a camera to detect the lane boundaries and then some control algorithms to keep the vehicle centered within the lane. But I wanted to understand this in more detail, and ideally implement a simple version of lane detection and lane-keeping myself.

I love Massive Open Online Courses on platforms like Coursera and Udacity, so naturally, I started looking there first. Udacity offers the famous “Self-Driving Cars Nanodegree”, but I didn’t want to spend thousands of euros. On Coursera, you can find the “Self-Driving Cars Specialization” by the University of Toronto, and since you can audit it for free I tried it out. I learned how a vehicle can follow a path using a method called Pure Pursuit. And even better, I saw that the course offered an exercise where you would implement Pure Pursuit and try it out in the Carla Simulator. Neat! However, the course did not cover lane detection.

So I continued searching for lane detection tutorials. I found a lot of stuff online that explained how to detect which pixels in the image are the lane markers. But I also wanted to know how to go from detecting lane marker pixels to generating a path on the road that a Pure Pursuit controller can follow. How do you go from pixels to (x,y) coordinates in meters? I did not find any easy to understand tutorials for that. Eventually, I found some older papers mentioning “Inverse Perspective Mapping”, which was kind of what I was looking for. But the mathematical formulas seemed complicated and I found no derivations. I understood that the idea is to use the assumption that the road is flat to invert the projection equation of the pinhole camera model. In the end, I derived the equations which I needed myself. Naturally, I wanted to implement and test them. What I wanted to do consisted of three steps

South Park reference. Image from Wikipedia.

Implement a method to detect lane boundary pixels Convert lane boundary pixels to a list of road coordinates (x,y), measured in meters. Fit a polynomial y(x) for the left and the right lane boundary Feed polynomials into Pure Pursuit to control a vehicle

Since I had learned about the Carla Simulator on Coursera I decided to apply this pipeline there. I had found the equations for step 2 and had watched the Coursera videos on Pure Pursuit, so I knew how to implement step 3. I only needed to pick a method for step 1.

Lane detection is a computer vision/perception problem and you probably heard that deep learning methods are dominating that field. You might also know that it is common for deep learning researchers and practitioners to publish data sets as “challenges” and to compare the performance of their neural nets on leaderboards. For lane detection, one important data set is the TuSimple Lane Detection Challange and I scanned some research papers that focussed on this. One paper stood out to me, because of its elegant approach and excellent exposition: End-to-end Lane Detection through Differentiable Least-Squares Fitting. In their paper, they also described a baseline model, which is extremely simple but still performs quite well. Good enough for me!

So finally I had found resources to implement my own lane-detection and lane-keeping system. Since it had taken me so long to gather all this information, I decided to create my own online course: “Algorithms for Automated Driving”. This course should guide the reader in implementing lane-detection and lane-keeping for themselves. I published the course is in the form of an online book today and here you can see a screenshot of the landing page (you can also just visit the course by clicking here)