ML approach on how to make German ‘Green Sauce’
By: Pia Scharf
Tags: Neural Networks, Artificial Intelligence, Tradition, Machine Learning, Fastai

Our problem is a classification problem. So we want to know if the herbs we find are relevant for making the green sauce. We intend to classify them as parsley, chives, chervil, borage, sorrel, garden cress and salad burnet. As input for our little algorithmic helper we select pictures. Once we are in the fields, we can take a photo of the plant or leaves we want to identify and “show” it to our program. In short, we want to build a model (a mathematical abstraction) that, by classifying an input image, tells us which of the classes the image corresponds to.

The environment we’ll work in is a jupyter notebook. Fastai is a library (library = collection of tools/code parts) sitting on top of Pytorch — a very powerful machine learning library, working with Python code. If you’ve never heard of one of these things: Python is the language we’re writing our code in, Pytorch and fastai are the tools we are using and the jupyter notebook helps us organize our code. For further information about the setup visit the fastai website.

We start our jupyter notebook and type in:

from fastai import *

from fastai.vision import *

This way we have all the image recognition functions of the fastai library. Next we want to be sure that we can train our network. By providing data to the neural network, we want to build a mathematical representation (model) with which we can then categorize new photos of discovered plants/leaves.

Our model does not know what parsley is, it has never seen the green sauce or anything similar. To train a model, we show it a series of photos with correct labeling. That’s why we first have to create seven folders with seven different labels: Parsley, chives, chervil, borage, sorrel, garden cress and salad burnet.

Step 1: Getting out dataset

We, therefore, open our chrome browser, go to google image search and type in: e.g. ‘parsley’. We then get a large number of pictures of parsley and scroll down. Then we hit Shift+Control+j to open up the Javascript Consol of our browser. We type in:

urls = Array.from(document.querySelectorAll('.rg_di .rg_meta')).map(el=>JSON.parse(el.textContent).ou);

window.open('data:text/csv;charset=utf-8,' + escape(urls.join('

')));

When we hit enter, we download a file with all the urls of the parsley google-Images. We save it under ‘parsley.txt’ and continue with the other six herbs. Great! Out dataset links are prepared. We put all the text files in a folder called ‘seven_herbs’ and make sure that it lays in the data folder in our home directory (fastai won’t find it otherwise).

Step 2: Downloading the image files

Next, we download the data directly to the server, we’re working on. Fastai needs to know the path, where our dataset lies. This step might be a little different in some environments — for example if you’re working in kaggle’s kernels. If you struggle, you can visit my starter kernel on kaggle.

path = Path('data/seven_herbs')

Afterwards we set the names of our herbs as classes.

classes = ['borage', 'chervil', 'chives', 'garden_cress', 'parsley', 'salad_burnet', 'sorrel']

This next steps seems a little bit tricky, in fact it’s very simple. It just takes the text files and makes two things: it creates and names a folder for each class (aka herb) and then fills this folder with the pictures defined in the equally named text file. (In the kaggle starterset you don’t need to do these steps, because I already attached the dataset to the kernel.)

for x in range(0,7):

dest = path/(classes[x])

dest.mkdir(parents=True, exist_ok=True)

download_images(path/(classes[x]+str('.txt')), dest, max_pics=200)

We then review our files to make sure they are not damaged or the links are not buggy. The following code will erase those that fail to open or download correctly.

for c in classes:

print(c)

verify_images(path/c, delete=True, max_workers=8)

Step 3: Creating an ImageDataBunch

Now it’s getting serious. We start processing our data by creating an ImageDataBunch object. This way we can manipulate the data and train our model later. The np.random.seed(13) defines which images are randomly selected from our training data. (We only need this line of code if we want to train the model again in the same way.) Our ImageDataBunch is created by passing the path of the folder with our photo data in it. We define a validation set that represents 20% of the data and define a size for our images.

np.random.seed(13)

data = (ImageList.from_folder(path)

.split_by_rand_pct(0.2)

.label_from_folder()

.transform(tfms, size=128)

.databunch())

Next we also need to assign our classes to to ImageDataBunch.

data.classes

If we type in the following code, the ImageDataBunch shows up and we can have a look at our data.

data.show_batch(rows=3, figsize=(7,8))

Step 4: Training the model

Let the training begin! First, we load in the vision.learner, a module that holds the cnn_learner: imagine this to be as a pretrained neural net, that is trained on a large image dataset. Next, we’re using ‘transfer learning’ as a technique — we’re showing our dataset to the cnn_learner, so that it learns the specifics of the seven herbs.

from fastai.vision.learner import create_cnn,models

from fastai.vision import error_rate

Finally: Let’s create our model — our mathematical representation of the correlations between labeling and pictures.

learn = create_cnn(data, models.resnet34, metrics=error_rate)

So we downloaded the pretrained model from the Fastai library. Next, we want to adjust the model to the data. To make sure that we can keep our results for later, we save the model directly after our four training cycles.

learn.fit_one_cycle(4)

learn.save('green-sauce-stage-1')

After this process we end up with a result, that shows an error rate of 0.33, that means that in 33% of the cases, the model will still mess up. We want to have a closer look on what that means:

learn.unfreeze()

learn.lr_find()

learn.recorder.plot()

This plot shows us the learning rate and the loss of our trained model. For further explanation, please have a look e.g. here. In short, the loss function shows us how well we are doing with our model. For each prediction, the graph shows us the absolute difference between the prediction that our model makes and the actual label that the photo holds. (Remember: all our training data is labeled). As a rule of thumb, we always want to look for a downhill path in the plot. If we look at the part from 1e-5 to 1e-3, it seems as if the model there is still getting better. Shortly after 1e-3 the gradient is rising again very quickly, which means that the error rate is also increasing again. We can improve our model by training it further.

learn.fit_one_cycle(2, max_lr=slice(1e-5,1e-3))

learn.save('green-sauce-stage-2')

Step 4: Interpreting the results

After all this training, we want to find out how well we’re doing. We interpret our results.

interp= ClassificationInterpretation.from_learner(learn)

interp.plot_confusion_matrix()

The confusion matrix shows us the relationship between the predictions made on the basis of the model and the actual labels of the images. As we can see, in most cases the model makes the right decisions: In particular, the chives images were correctly recognized in 33 cases. The losers are the chervil and the sorrel: The chervil, an herb that is less easy for the human eye to identify than the long and smooth chives, is less surprising, whereas the sorrel, at least to my surprise, has been mistaken by the neuronal network for garden cress on several occasions. The most confused lists also points this out:

interp.most_confused()

We can see from this that things went wrong during training. Overall, our model doesn’t come off so badly, although we could of course still do a lot to improve our results, e.g:

Improving our dataset by deleting images that have appeared in the image search, but that do not exactly match our intended result (e.g. FileDeleter widget of the fastai library does.).

Make our model even fit better by training more/with more data/with other data

Design a better validation set. Read more here

Right now, we’ll leave it at that and announce: We are ready! We have built our neural network. You might think: ‘Ok, great, but how do I photograph a herb and show it to my trained network?’

Test: I want to show this picture to my network — Photo by Pia Scharf

Step 5: Building a website

Remember, the goal is to have a website where we can upload a picture. Now there is a need for an interface design. (We won’t spent time on the design itself in this tutorial, but maybe in the upcoming ones.) For simplicity’s sake, we want to create a graphical user interface that can only receive one image. We want our service to be accessible via the Internet at all times, so we make a simple homepage with only one interaction option, namely to have a photo analyzed. To make our small application available, we use render. This is a cloud provider that you can use to deliver apps and websites.

First, we need to export our model. That’s what this looks like on kaggle kernels:

learn.load('green-sauce-stage-2')

learn.export(file = Path("/kaggle/working/export.pkl"))

This adds the file ‘export.pkl’ in our data structure. We download the file to our hard drive. Next we push it to a google drive and make sure that it is free to read for anyone (otherwise the following command will not work). We use this direct link generator, so downloading our file automatically starts with a click on the link.

In order to use Render, we need to upload the project we want to ‘render’ to github. In case you don’t know github: its basically a place where people share code. The great thing about github is, that you’ll find a lot of nice projects which you can use as a starting point for your own projects.

In this case, we use the repository of anurag as a basis and make own out of it. If you click on ‘fork’ at anurags repository, you’ll find his repository in your github account. You can then modify it (rename, type in your code) and ‘commit’ the changes (don’t worry- this will only change your ‘branch’ — in other words: your version of the repository). If you want to know how github works in detail, I strongly recommend to read the github guide.

So after forking anurags repository, you might rename it to ‘green-sauce’ and we need to do two important things to make this code work:

We need to edit the file ‘server.py’ (green-sauce/app/server.py) so that our app finds the direct download link to our model. We also modify the classes-section and type in the names (aka classes) of our seven herbs. We then hit ‘commit changes’.

export_file_url = 'https://YOUR-DIRECT-DOWNLOAD-LINK' export_file_name = 'export.pkl' classes = ['borage', 'chervil', 'chives', 'garden_cress', 'parsley', 'salat_burnet', 'sorrel']

We modify the index.html (green-sauce/app/view/index.html). We can change the heading of our web page by changing the text between the div-brackets (e.g. <div class=’title’>this text</div> ) and we can write a little description text between the <p>-brackets. We also ‘commit’ the changes

If we want to change colors, fonts or other visual elements, we can modify the css-file. As soon as we’re finished, we move on to render.

At the dashboard of render we click on the ‘new web service’-icon. There we type in our repositories github-address. You can find my github repository for this medium article at https://github.com/piaoyapia/Seven-Herbs. If you want to render your own app, leave the environment dropdown at ‘Docker’. Thats it. Render will take care of the rest!

Step 6: Testing the website

Finally we are finished and we can test our website! If you followed along, you probably have your own version by now. Otherwise just test mine.

So I uploaded a photo of me holding a parsley. Then I clicked on “analyze”. The result is not so spectacular optically, but at least it is correct: The network recognizes the herbs as parsley. Yeah!🎉