Robot following a walkway using image segmentation | by Constantin Toporov | Towards Data Science

In my previous story, I was teaching a Raspberry-powered robot-tank to navigate on a walkway on his own. The bottleneck was a road recognition — I used a simple approach with color filtering via OpenCV and the results were not reliable.
Generally speaking, this a task of image segmentation. There is a good article describing the most popular methods. They are simple and usually do not produce perfect results on real-life photos. The same thing as my color-based implementation.
Looking for a way to improve I decided to use a semantic segmentation. Neural networks are getting used for this task only recently but demonstrated impressive results. This great article from PyImageSearch gave me inspiration and pushed from theory to the practice. This article describes the image segmentation via E-net neural network.
Turned out, E-net is relatively young and not well-known network. It was developed based on the success of more famous U-net architecture.
U-net is a first and most-famous neural network used for semantic segmentation. Initially it was developed for medical needs in Germany. The original document is available here.
People pretty quickly figured out this approach could be used for wider purposes than X-ray reading. There are some articles how to train and use U-net for road segmentation on city images:
One
Two
Three
There is a lot of interesting and new information, but I was looking for a ready network to use with my tank right away. Unfortunately, the search did not produce quick results.
So switched back to E-net.
E-net was based on ideas generated by U-net. According to the source, it was originally designed for city images segmentation. Some open datasets were used to train on road scenes:
Cambridge streets dataset aka CamVid
German and Swiss streets dataset aka Cityscape
There is an introductory article about E-net.
If someone wants to go deeper there is a detailed document about how to train and use E-net network. I would say it is a mandatory step — to tailor the network for your own needs and requirements.
But I was too excited to get immediate results and decided to use the ready-to-go network from PyImageSearch as mentioned above.
So far my aim was to recognize only walkway instead of 20 classes of objects the network was trained to detect. For that purpose I edited the file with colors to ignore all classes but roads.
Then read the model from the filesystem:
def load_segment_model():    try:        classes = None        with open(PiConf.SEGMENT_CLASSES) as f:            classes = f.read().strip().split("\n")        colors = None        with open(PiConf.SEGMENT_COLORS) as f:            colors= f.read().strip().split("\n")        colors = [np.array(c.split(",")).astype("int") for c in colors]        colors = np.array(colors, dtype="uint8")        print("[INFO] loading model...")        net = cv2.dnn.readNet(PiConf.SEGMENT_MODEL)        return net, classes, colors    except Exception as e:        logging.exception("Cannot load segment model")    return None, None, None
Do the segmentation:
def segment_image(image_path, seg_net, seg_classes, seg_colors):    image0 = cv2.imread(image_path)    image = cv2.resize(image0, (1024, 512), interpolation=cv2.INTER_NEAREST)    blob = cv2.dnn.blobFromImage(image, 1 / 255.0, (1024, 512), 0, swapRB=True, crop=False)    seg_net.setInput(blob)    start = time.time()    output = seg_net.forward()    end = time.time()    print("[INFO] inference took {:.4f} seconds".format(end - start))    (numClasses, height, width) = output.shape[1:4]    classMap = np.argmax(output[0], axis=0)    mask = seg_colors[classMap]    mask = cv2.resize(mask, (image0.shape[1], image0.shape[0]), interpolation=cv2.INTER_NEAREST)    classMap = cv2.resize(classMap, (image0.shape[1], image0.shape[0]), interpolation=cv2.INTER_NEAREST)    gmask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)    gmask = cv2.resize(gmask, (128, 64), interpolation=cv2.INTER_NEAREST)    gmask = gmask[0:64,32:96]    output = ((0.6 * image0) + (0.4 * mask)).astype("uint8")           return output, gmask
The first problem is seen in the code at once it an input image size. The network was trained on pretty big images 1024x512. My Raspberry camera is set on 320x240 in order to save resources for the processing.
After the code was ported it was time to handle photos the tank already made on walkways.
Only left part of the walkway is recognized.
Reduce the picture and cut its center 64x64 (this mask is expected by another neural network deciding about the direction).
The direction network (it is a 3-class classifier) commands to drive left. Not the best decision but bearable.
The similar situation, the right bottom corner is not recognized again. Probably the wet surface confused the network. But the biggest part of the road is recognized correctly. The mask looks consistent:
And the direction network decides to go forward.
This is a failed state — the robot located in the middle of a walkway and needs to go right.
The road is recognized pretty much completely.
And the direction classifier commands to go right. That is it.
So the road recognition results are much better than just simple color filtering as in the old implementation. Let's run it on the real tank.
Unfortunately, the noticed input image size caused problems right away. The Raspberry is too slow to handle so big images. One segmentation takes about 6 seconds — it is unacceptable to real-time vehicle. The neural network worked good but when it was working the tank went off the road.
So the only solution is to train my own E-net network with smaller pictures.
Also probably it is a good idea to use external power to make the segmentation, like Intel Neural Computer Stick or something similar.
The last picture demonstrates that even the road decisions are valid they have been made too late.
Previous attempt to drive on walkway — with color-based segmentation
The tank firmware with E-net segmentation on GitHub
PyImageSearch article which pushed me to make this step
A guide how to train your own E-net implementation
The tank site where you can get more information and buy the tank DIY kit