Let me count the leaves for you — A Deep Learning Case Study
By: Rajat Malviya
Tags: Keras, TensorFlow, Image Processing, Segmentation, Deep Learning

Regression Models based on Segmentation

These models leverage the segmentation masks generated by the Segmentation model in order to predict the leaf count for the image.

Image taken from Leaf Counting with Deep Convolutional and Deconvolutional Networks

Combined Regression Model

In this model , the input to the regression model is a 4 channel SRGB input where S stands for the segmentation mask that has the output probabilities for each pixel obtained from the segmentation model.

The data preprocessing and augmentation for the model is the same as what we did for the segmentation model. The only difference here is that we would be concatenating the segmentation mask generated from segmentation model for the augmented image with it as shown below.

Model Architecture:

Regression Model architecture(Image taken from Leaf Counting with Deep Convolutional and Deconvolutional Networks)

As shown in the above architecture, the regression model contains blocks of Convolution layers, Layer Normalizations and ReLU activation layers which ultimately squeezes the input image into a flattened vector followed by some fully connected layers to finally get the leaf count.

Loss Function:

As mentioned in the reference paper, Smooth L1 loss(also known as Huber Loss) was used as the loss function(for all the regression models). Smooth L1 loss can be interpreted as a combination of L1 and L2 losses.

Huber Loss(Image taken from Tensorflow documents)

In the above image, x is the difference between actual and predicted values, and d is the fixed delta value.

2. Combined Regression Model(Only leaves)

The model architecture for this model is the same as above. The only difference here is that the binary segmentation mask(a threshold is set on each pixel output probability to get 1 or 0 valued pixels) obtained from Segmentation Model and the input RGB image are multiplied to get a masked image. So the input would be a 3 channel masked image.

The rest of the model architecture remains the same along with the training and loss functions.

3. Combined Regression Model(cropped leaves)

The masked image that is obtained as shown in the previous model is further cropped on the leaves and resized to focus more on the leaves part. This image is then used as an input to the regression model. This implemented in code as part of data augmentation and preprocessing as shown below..

The rest of the model architecture remains the same along with the training and loss functions.

Transfer Learning Models

Since the dataset is very small, the other approach that can be very useful here is Transfer Learning. Hence images features were obtained from pretrained models and then the model was trained further on the data to get the predicted leaf count.

The data augmentation for these models was done using Image Data Generator in Tensorflow Keras. This was not possible for the regression models that used segmentation masks because the segmentation masks had to be augmented accordingly as well. Hence, those models required a proper pipeline with a custom implementation of data augmentation.

This above mentioned Transfer Learning approach was tried with three pretrained models..

DenseNet121 ResNet50 VGG16

An example of the above mentioned models is as shown below:

The loss function and the training part remains the same as previous models.

Results Analysis

Since our evaluation set contains a very small amount of images, the best way to evaluate the model performance is to test their predictions across all directories in the dataset. Since, each directory contains different types or combinations of plant images, the results across each directory will give a better indication on which model would be most effective for a specific type of plant. This is in accordance with the results and evaluation of the model in the Referenced Research paper.

Performance of the models across directories