Implement Your First Ever Lane Detection In Any Car Racing Game | Towards Data Science

Lane detection and control has become a common feature in many vehicles today. Additionally it is a basic starting point for anyone going in the direction of Autonomous Driving. But for most people who are not working on autonomous driving or computer vision, starting with it might appear much more of a daunting task than it actually is.
There is a lot of technical research involved in doing actual lane detection. In this case, we see a practical perspective so that anyone interested can try it out without much hassle. The detailed links to the theory are still there in the relevant sections for the curious ones: those of you who do not get satiated with just practical applications and like to get their teeth to sink deeper into the subject matter.
Here is everything you need to start:
Asphalt 8 Airborne: If you are a Windows 10 user, you can download it free from the Windows Store: https://www.microsoft.com/en-us/p/asphalt-8-airborne/9wzdncrfj12h?activetab=pivot:overviewtabPython 3.7: You could install the latest version of Anaconda, most of the required packages will be already bundled and ready to use. Here is the link to download the latest version: https://www.anaconda.com/products/individualOpenCV: It is a library mainly aimed at real time computer vision. You can find the documentation about how to install and use it here: https://www.learnopencv.com/install-opencv3-on-windows/
Asphalt 8 Airborne: If you are a Windows 10 user, you can download it free from the Windows Store: https://www.microsoft.com/en-us/p/asphalt-8-airborne/9wzdncrfj12h?activetab=pivot:overviewtab
Python 3.7: You could install the latest version of Anaconda, most of the required packages will be already bundled and ready to use. Here is the link to download the latest version: https://www.anaconda.com/products/individual
OpenCV: It is a library mainly aimed at real time computer vision. You can find the documentation about how to install and use it here: https://www.learnopencv.com/install-opencv3-on-windows/
Now that we have all we need, let’s cut to the chase!
Step 1: Find a way to access the game screen
This one is easy enough. I did a quick google search to check out python codes that can be used to access the screen. Here is a really great tutorial, from which I used the basic code and adapted it for this case:
holypython.com
If we run the code directly, you would see a similar result as seen below in a random game level. You will notice that the colors are a bit different, and the screen rate results in some lag (which is OK for our purpose).
Let’s correct the color part with some help from the OpenCV documentation. There is a argument in cv2 that makes the screen recording look like the actual color (or at least as far as my vision allows) which is COLOR_BGR2RGB, that is what I used for the correction here. Also, the time function is used to get the screen rate, which comes to around 10 fps, not bad at all! The modified code looks as follows:
# import librariesfrom PIL import ImageGrabimport cv2import numpy as npimport time# for timinglast_time = time.time()# screen capture loopwhile True:        screen = np.array(ImageGrab.grab(bbox=(0,40,800,700)))        print(f'the screen rate is {(1/(time.time()-last_time))}')    last_time = time.time()        cv2.imshow('Python Window', cv2.cvtColor(screen, \               cv2.COLOR_BGR2RGB))    if cv2.waitKey(25) & 0xFF == ord('q'):        cv2.destroyAllWindows()        break
For the above code to correctly capture screen, I have minimized the game window to the top left corner of my primary monitor, with the dimensions as 800 x 700. You could adjust it appropriately in your code (by changing bbox in the ImageGrab above), especially if you are using multiple monitors.
Once you run this, you get the screen capture as follows:
This method of screen capture is extremely handy, particularly when we need to generate images for machine learning use cases.
Now that we can capture the game window, we can move on to the next step.
Step 2: Image processing for edge detection
Since lane detection is basically an edge identification from image, you would have to process the image to actually get the main outlines. One of the most popular method used in computational vision processing is the Canny algorithm for edge detection. It basically extracts and retains the useful structural information from images, all the while drastically reducing its size.
Open CV has a neat little implementation of the Canny edge detection. This reduces the complexity of the image much further and we get the edges of the image in the screen with pretty much a three line code.
Let’s create a small function for processing the image using the Canny implementation in cv2. You can experiment with various values of thresholds by looking into the documentation, the code looks like this:
# canny image processing for detecting edgesdef edgeprocessed(image):        gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)    edgeprocessed_img = cv2.Canny(gray_image,               \                        threshold1 = 200, threshold2 = 300)    return edgeprocessed_img
We have to run the processed image again in the image capture loop, so it makes more sense to define this loop as a function to take into consideration the processing as the input. I will get a bit ahead of myself here and modify it to general function taking the inputs as list, which will make more sense further down in the analysis.
def screen_capture(processing = None):     # takes in a list for type of processing    # [processing_function, masking_function]        last_time = time.time()        while True:                screen = np.array(ImageGrab.grab(bbox=(0,40,800,700)))        count = 0                    if processing is not None:                        for i in processing:                count += 1                        processed_screen = processing[0](screen)                if count > 1:                                    masked_screen = processing[1] \                                    (processed_screen,[vertices])                    screen_1 = masked_screen                                else:                                    screen_1 = processed_screen            else:                        screen_1 = screen                                cv2.imshow('Python Window 2', screen_1)        cv2.imshow('Python Window', cv2.cvtColor(screen, \                    cv2.COLOR_BGR2RGB))                                        print(f'the screen rate is {(1/(time.time()-last_time))}')        last_time = time.time()                 if cv2.waitKey(25) & 0xFF == ord('q'):            cv2.destroyAllWindows()            break
Makes it much better to use the function now. You will see why I structured the function this way in the next section. Till then, let us check out the result with the edge processing.
screen_capture([edgeprocessed])
This is what you will see:
We pretty much get the edges of the objects from the game screen. This is how it looks like when I tried it in a test track in the game:
As you notice, there is still too much unwanted information in the image. The trees, rocks, electricity lines, basically everything above the horizon of the road. Not to mention the overlay controls display from the game itself. If we want to focus on the lanes, we have to find a way to somehow filter out or mask the rest of the image lines.
Step 3: Masking additional information from the image
For this, let’s define a masking function. What we need is to define a polygon shape that masks all the information from the image, except the region that lets us focus on the lanes. This is a very basic method used, in this case I defined the vertices of the exact region I need with a bit of trial and error, as seen in the image below:
The required region function uses a masked frame and a polygon with vertices. The intersection of the masked and the polygon gives us only the required section of the image. The function and the vertices (defined above) are as you see here:
def required_region(image, vertices):        mask = np.zeros_like(image) # black image            cv2.fillPoly(mask, vertices, 255) # fill within vertices    masked = cv2.bitwise_and(image, mask) # intersection        return maskedvertices = np.array([[120,500],[120,400],[150,330],[650,330], \                     [680,400],[680,500],[260,500],[260,450], \                      [325,370],[475,370],[540,450][540,500]], \                      np.int32)
Now, we can run the screen capture function with these inputs as well.
screen_capture([edgeprocessed, required_region])
And, the result is as follows:
As you notice now, I have opaqued out all the unnecessary information except for the lanes. For a rudimentary level code, I’d say that this is pretty neat!
But we are not quite there yet.
Step 4: Finding lines with Hough Lines
Finding the actual geometric lines, we need to do some kind of transformation on the edge processed image. In this case, I used the Hough Transform implementation in Open CV. For this to work properly, the image has to be blurred a bit before giving as the input for Hough transform. I used the Gaussian Blur from Open CV for this case.
Additionally, there is a need to draw the detected lines on the image frames. So I defined one more function overlay the detected lines on the frames.
The modified overlay line and the edge processing codes are:
def overlay_lines(image, lines):        for line in lines:        coordinates = line[0]        cv2.line(image,(coordinates[0],coordinates[1]), \                (coordinates[2],coordinates[3]),[255,255,255],3)def edgeprocessed(image):        gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)            edgeprocessed_img = cv2.Canny(gray_image, threshold1 = 200,\                                  threshold2 = 300)            edgeprocessed_img = cv2.GaussianBlur(edgeprocessed_img,(5,5),0)        lines = cv2.HoughLinesP(edgeprocessed_img, 1, np.pi/180, \                            180, np.array([]), 100, 5)        overlay_lines(edgeprocessed_img, lines)        return edgeprocessed_img
As you can see, that is not bad at all, and with just a few lines of code! We are almost there.
Step 5: Get Lanes
There are a variety of lines detected in the previous step. We have to find a way to merge them in some way so that we can get the actual lanes. This is a bit more complex and I checked out several resources online to see if there is a simpler way that it could be done. I found a great draw lanes function in the Github project here which I used here directly (code not added below as it is too long, but you can refer my project notebook in Github, linked at the end). All that was needed to be done then was to modify the edge processing function as follows:
def edgeprocessed(image):        original_image = image        gray_image = cv2.cvtColor(original_image, cv2.COLOR_BGR2GRAY)    edgeprocessed_img = cv2.Canny(gray_image, threshold1 = 200, \                                  threshold2 = 300)        edgeprocessed_img = cv2.GaussianBlur(edgeprocessed_img,(5,5),0)    edgeprocessed_img = required_region(edgeprocessed_img, \                                        [vertices])        lines = cv2.HoughLinesP(edgeprocessed_img, 1, np.pi/180, 180, \                            np.array([]), 100, 5)    #overlay_lines(edgeprocessed_img, lines)    m1 = 0    m2 = 0        try:        l1, l2, m1, m2 = draw_lines(original_image,lines)                cv2.line(original_image, (l1[0], l1[1]), \                 (l1[2], l1[3]), [0,255,0], 30)                cv2.line(original_image, (l2[0], l2[1]), \                 (l2[2], l2[3]), [0,255,0], 30)        except Exception as e:        pass    try:        for coords in lines:            coords = coords[0]            try:                cv2.line(edgeprocessed_img,                          (coords[0], coords[1]),                          (coords[2], coords[3]), [255,0,0], 2)                                            except Exception as e:                print(str(e))    except Exception as e:        passreturn edgeprocessed_img,original_image, m1, m2
And accordingly a few tweaks in the capture function:
def screen_capture(processing = None):     # takes in a list for type of processing    # [processing_function, masking_function]        last_time = time.time()        while True:                screen = np.array(ImageGrab.grab(bbox=(0,40,800,700)))        count = 0                    if processing is not None:                        for i in processing:                count += 1                        processed_screen ,original_image, m1, m2 = \                                           processing[0](screen)                if count > 1:                                    masked_screen = processing[1] \                                    (processed_screen,[vertices])                    screen_1 = masked_screen                                else:                                    screen_1 = processed_screen            else:                        screen_1 = screen                                cv2.imshow('Python Window 2', screen_1)        cv2.imshow('Python Window', cv2.cvtColor(screen, \                    cv2.COLOR_BGR2RGB))                                        print(f'the screen rate is {(1/(time.time()-last_time))}')        last_time = time.time()                 if cv2.waitKey(25) & 0xFF == ord('q'):            cv2.destroyAllWindows()            break
And voila!
screen_capture([edgeprocessed])
As you see, there are still more tweaks and fine tuning that can be done. But it is an awesome application as an introductory learning project!
The fun part is, the entire thing works live! You can play the game and detect lanes simultaneously as you keep the code running. This gives us a host of possibilities, like saving images for ML, game controls etc. which I will explore in future posts.
That’s it, folks! I hope you managed to implement lane detection in these 5 simple steps. The Jupyter notebook can be found in this link.
Feel free use and adapt the code and let me know if you have any interesting tweaks, results or ideas. Happy learning!