Fruit Yield Assessment from Photos with Machine-Learning Scikit-image
By: Li-Ting Liao
Tags: Machine Learning, Scikit Learn, Agriculture, Birch, Data Science

Programming Process

Import required libraries:

Import image I’m going to use and convert it into color pixels arrays. I found this picture from a person’s blog:

I was really curious about how this picture looks like as I knew some pictures will be 2-D or 3-D arrays. Used the following codes to check it out:

What I found out is that there are 3 layers i.e. 3 dimensions of my chosen picture. X axis is in the 1st layer, Y axis is in the 2nd. For the 3rd layer I will be using the first item only as using the other 2 items seems indifferent versus using just the first item. This structure will be more important in the later part of code.

Then we start to click on the reddish ripen mangoes on the imported picture. When we click on the picture, our click’s x-coordinates and y-coordinates will be stored in a list:

The click event uses matplotlib event connections to create a function that stores points clicked on the image. Once clicked, coordinates are added to the list:

The clicked points will show its details on the top left corner of the subplot:

Let’s say I have clicked on 6 mangoes. The corresponding coordinates can be printed like below and the length of clicked list (puntosinteres) is 6:

Then if I want to click on more mangoes, i.e. adding upon what I just clicked. The code snippet looks like this:

From the above code, I got the clicked ones shown as red dots on the picture. Now I can click on other mangoes other than those clicked 6 dotted ones:

Next step is to take a closer look at my selected templates i.e. clicked mangoes pictures. This is to make sure I click on the right things:

Since I clicked on 3 more, so now I have 9 template pictures:

Once I double checked all the templates, if I want to delete one of the picture, I can use the below code. Also check the length of clicked list (puntosinteres) again:

Here comes the most exciting part. Start to train our model with our template and original pictures using color pixel arrays!

Here I’m using sklearn’ s match_template library. I noticed that before putting into match_template function, the template picture’s x-axis and y-axis coordinates need to be reversed (similar thing I saw from people who are using OpenCV library):

result variable will give us a matrix of all normalized cross-correlation between template and original picture, showing how well the template matched to different locations in the original image; then we set a threshold as 0.8. Only those matched higher than this score will be appended to our result list (listaresultados). Eventually our result list (listaresultados) show arrays of color pixels that are closet to our clicks on the original picture.

Take a special note that when we need those coordinates from the outcome, we need to reverse it back again to get the right coordinates for both axes.

Let’s plot the mangoes coordinates from our machine learning model directly on the original picture:

Then we can see from below that several pixel from our result list (listaresultados) are actually from the same mango, which could lead to double counting:

So it’s why we need a cluster analysis here to help us solve this issue.

What is Birch — in sklearn.cluster library there’s a BIRCH method, which is a memory-efficient clustering method for large datasets. Its algorithm creates Clustering Features (CF) Tree for a given dataset and CF contains the number of sub-clusters that holds only a necessary part of the data. Thus the method does not require to memorize the entire dataset.

The usage process includes define the birch method and fit it with our input data.

Converting result list (listaresultados) into datalist to fit in birch model later on:

The most important arguments of birch model is :

1. threshold: sets the limit between the sample and sub-cluster.

So we create a birch model and fit datalist into it:

Then we will see the model now can identify 15 mangoes as ripen ones.

Watch out the birch models’ threshold argument — I notice that when its number becomes smaller, the model will split its cluster more, and this will make it more likely to have multiple clusters at nearby areas i.e. redundantly mark over the same mangoes.

And it seems that we only use 9 templates to make the model able to recognize 6 more ripen fruits.

We see there are still some limitations of this procedure such as there are still many more mangoes left unidentified. Also, I believe there’s a way to give the model to several mangoes’ pictures, and then we take those as templates and ask the model to identify how many mangoes on the tree we have above.

Other helpful materials I’ve used for this article:

Also this article is my first baby step of using machine learning in agriculture. I was fully inspired by this wonderful material:

That’s it. Thanks for reading!