Smart Sprinkler — More Accurate Lawn Watering Using an Image Segmentation Model Paired with an Arduino Based Sprinkler System
By: Fazal Mittu
Tags: Data, Arduino, Artificial Intelligence, Machine Learning, Lawn Care

Smart Sprinkler — More Accurate Lawn Watering Using an Image Segmentation Model Paired with an Arduino Based Sprinkler System Fazal Mittu Follow Apr 26 · 12 min read

Source: Dvaires on Flickr

The Problem

Imagine you are strolling through your neighborhood past beautiful green lawns. You step in between the puddles of water collecting near your feet. On your left, a large lawn is being showered in water from a dozen sprinklers in the ground. Despite the miniature water park soaking up the grass, it is still covered with small patches. Many of us don’t see a problem with this, and will happily walk over these puddles and move on with our day. However, the issue is that even though so much water is being used, the lawn is still unhealthy and unable to thrive. This problem is more severe than you may think. In America, 30–60% of urban fresh water is used to water lawns and the sad fact, is that about 50% of this water is wasted due to inefficient and inaccurate watering.

Lawns are wasteful. They are heavily resource dependent on commodities like fertilizers, composts, and most importantly, water. This article will list a detailed version of my developed solution as well as the steps that I took to plan, prototype, and deploy it.

The Goal

The engineering goal for this experiment was to develop a 3-part system including an image segmentation model, a motor value script, and an Arduino controlled sprinkler with the intent of accurately watering patches in a grass lawn, saving significant amounts of water and time.

First Thoughts

At the beginning of this project, I lacked the experience with computer vision, and also any formal plan.

At first glance, there is a noticeable color difference between the lighter brown patches of grass and the green of the healthy grass. My first approach to this project was to look at the RGB values of different areas of the image. If I could determine areas that are “less green” and find a way to threshold it, I could pinpoint the patches. However, it wasn’t as simple as anticipated. The main issue was the numerous external factors that could affect my results. For example, lighting can change the RGB value of a patch in an image. With so many extraneous factors, it would lack the consistency of a practical system.

I definitely needed a better solution.

Image Segmentation

The new solution came in the form of a concept called image segmentation. Image Segmentation is a computer vision process that divides images into “segments” based on the similarity of pixels. It is mainly used to detect objects within images (object detection) and is used in autonomous vehicles, medical image processing, and satellite imaging.

Using image segmentation, I could detect the patches in the lawn. I began to research creating image segmentation models using Tensorflow.

During my research, I found a wonderful library called ImageAI written by Olafenwa Moses. ImageAI gives users the option for custom object detection: the ability to use a custom, user created dataset. ImageAI would allow me to assemble a curated dataset, best suited for my model.

First Dataset/Test

Data enables machine learning and without accurate, meaningful, and plentiful data, it isn’t possible to effectively train practical models to make predictions about the world around us. If we wish to teach a student well, we must provide them the right resources to learn efficiently.

Many websites and applications contain free to use, public datasets that engineers can use for their own projects. However, there were no existing datasets for this project, meaning I would have to make my own. As it would turn out, building the dataset proved to be a big challenge over the course of this project.

An image segmentation dataset has two parts to it: images and annotations. There are many ways to annotate images, or mark where the object is located in the image. I used the Pascal VOC format which saves the annotations in .xml files. Essentially, if my dataset contained 50 images, I would have to annotate each individual image to create 50 .xml files annotating their corresponding images.

Initially, I had many questions about the dataset: how many images I needed, how do I annotate an image, how do I use it in code, how do I know if an image is good, and so on. Thanks to Olafenwa Moses, who luckily wrote an informative article detailing how to create a custom dataset and use it with his library ImageAI.

The first step was acquiring images, which proved to be significantly more difficult than anticipated. There weren’t many high quality images that I could find online. I googled “patchy grass” and manually downloaded just 65 images for my first dataset. To put this in perspective, most image datasets contain thousands of images to train on.

After collecting the images, I annotated them using a software called Label IMG. I identified the directory to get images from, and the directory to save the annotations in. Then, I annotated each image by boxing any patches. Below is a screenshot of the Label IMG software I used to annotate the images.

Label IMG Software

I finally had a complete dataset of annotations and images and was ready to train my model. I faced many difficulties with my dataset and downloaded libraries and after troubleshooting, I trained my first model over 15 epochs with a batch size of 4.

Dataset 1 Results

After testing the model on an image of my front yard, this was the result. It had taken me so long to get a clean run without an error that I was ecstatic to even get a box to show up. It hardly crossed my mind that this result was horrible and useless for my overall system. The model vaguely marks the general area of the bad grass. However, it’s very inaccurate and practically useless.

Datasets 2–4/Further Testing

After evaluating my results, it was obvious that I lacked enough images to properly develop my model. I decided that I needed a much larger dataset, and manually downloading my images was inefficient.

After some research I discovered web scrapers. A web scraper is a tool that can extract data and content from websites, and download those files locally to your computer. This was just what I needed, and after following a short tutorial, I had a small web scraper downloading images with the keywords “patchy grass”, “bad grass”, and “bad lawn”. Using this web scraper, I accumulated a folder of 180 images, my second dataset.

After removing any unusable images, I had a third dataset of 160 images. After training the model with the third dataset, the results remained inconsistent and inaccurate. Again, I suspected the main issue was a lack of images in the dataset.

After reaching out for some help and doing some research, I discovered image augmentation, a process through which a dataset can be expanded by modifying existing images. This includes changing the orientation of the image and the RGB values of pixels.