<figure><img src="image_1.png" /></figure>

HAL
open science

# Identification and counting of sorghum panicles using artificial intelligence based drone field phenotyping

Modou Mbaye, Alain Audebert

► To cite this version:

Modou Mbaye, Alain Audebert. Identification and counting of sorghum panicles using artificial intelligence based drone field phenotyping. Advances in Artificial Intelligence and Machine Learning, 2021, 1 (3), pp.234-240. 10.54364/AAIML.2021.1115. hal-05180323

HAL Id: hal-05180323

https://hal.science/hal-05180323v1

Submitted on 23 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Identification and Counting of Sorghum Panicles Using Artificial Intelligence Based Drone Field Phenotyping

**Modou Mbaye**

Centre d'Etude Régional pour l'Amélioration de l'Adaptation à la Sécheresse (CERAAS), Institut Sénégalais de Recherches Agricoles (ISRA), Route de Khombole, Thiès BP 3320, Senegal

**Alain Audebert**

UMR AGAP Institut, Université Montpellier, CIRAD, INRAE, Institut Agro, F-34398 Montpellier, France

**Corresponding Author:** Modou Mbaye.

MY.GANDHY@GMAIL.COM

ALAIN.AUDEBERT@CIRAD.FR

**Copyright** © 2021 Modou Mbaye and Alain Audebert. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

## Abstract

One of the most promising and difficult challenges for field phenotyping is accurate and reliable counting of sorghum panicles using drone imagery both from RGB and multispectral cameras. In this paper, we present a hybrid Machine Learning method for sorghum panicle identification and counting. The methodology first consists in building a Machine Learning classifier following the two most used methods in the literature for drone and agriculture applications: Support Vector Machine Learning (SVM) and, Artificial Neural Networks (ANN). The present dataset includes 5300 images, and 60% of the dataset were used for training and 20% for testing and validation. Following the results obtained from these models, image segmentation using super-pixel affinity propagation and k-means clustering was used based on simple linear iterative clustering. With an accuracy of 99%, SVM gave a superior performance also in terms of precision and kappa when compared to the ANN model whose accuracy was 98%. Concerning the SVM, a radial basis kernel was used, and the sigma parameter was kept constant at a value of 5.6 determined analytically.

**Keywords:** UAVs, Machine Learning, Artificial Neural Networks, Phenotyping, Sorghum.

## 1. INTRODUCTION

The use of Unmanned Aerial Vehicles (UAV) technology is becoming increasingly popular in the field of agriculture [1-8]. Coupled with sensors, UAVs have recently been proposed as a cost-effective option for high-throughput phenotyping noninvasive tools. Many studies have shown that UAVs can be used to monitor crop health, plant biomass, nitrogen content and plant water need estimates [9-13].

Machine Learning (ML) has shown its potential for automating the needs of breeders in agriculture. By combining ML and UAV, several agro-morphological and physiological plant traits can be studied at the field level. Thus, this combination of technologies can greatly help breeding programs
to evaluate agro-morphological traits that are challenging to measure with traditional phenotyping techniques which are often expensive, labor-intensive, and time-consuming [14]. The application of high-resolution UAV sensors for plant monitoring has resulted in significant volumes of image data that needed to be processed to extract valuable information. Deep Learning (DL) has received great success in agriculture (diseases and plant recognition [15-17]) and specially for high throughput phenotyping (HTPP). For example, Neupane et al. [18] proposed to detect and count banana plants by using a Convolutional Neural Network (CNN). Jin et al. [19] proposed a Faster R-CNN DL model to accurately segment individual maize from terrestrial Lidar. Recently, Kong et al. [20] introduced a rice panicle segmentation algorithm based on Mask R-CNN. Zhou et al. [21] used a high-definition RGB camera combined three popular DL architectures such as AlexNet, VggNet and Inception-V3 for rice panicle counting. Baweja et al. [22] developed a DL pipeline for HTPP using a ground-robot based on Faster R-CNN and semantic segmentation. A review on recent progress in semantic image segmentation and deep learning application in agriculture can be found in [23-27]. The work presented in this research is focused on the development of a rapid phenotyping technique for sorghum counting using UAVs based on artificial intelligence (AI).

## 2. MATERIALS & METHODS

In this research images were obtained with a hexacopter UAV at 50 meters height at a constant speed of 4.5 meters per second. The unmanned aerial vehicle (UAV) (FeHexaCopterV2, MikroKopter Company, Germany) may fly remotely or autonomously using GPS. The program that implements a flight plan, monitors the flight, and allows information such as drone position (Mikrokopter tools, Mikrokopter firm, Germany). A thermal infrared camera with an Airphen multispectral camera with six spectral bands, blue(450nm), green(532nm), green-edge(568nm), red(675nm), red-edge(730nm), NIR(850nm) were used. Additionally, an RGB SONY ILCE-6000 digital camera with a 6000 × 4000 pixel sensor and a 60 mm focal length lens were used. We limited data collection missions to clear, cloudless days and conducted them in the middle of the morning to reduce the impacts of ambient light. Both the RGB and the Airphen multispectral cameras acquired images

<figure><img src="image_3.png" /><figcaption>Figure 1: Detailed description of the steps of the approach used. Regarding phenotyping in RGB mode, we calculated various vegetation indices from the literature such as the BI (Brightness Index), the SCI (Soil Color Index) the GLI (Green Leaf Index) and the PSRI (Plant Senescence Reflectance Index). All these steps are fully automatic.</figcaption></figure>
continuously at a rate of 1 Hz. The precise locations of these GCPs were determined using GPS GNSS technology with a 2 cm precision. FIGURE 1 shows the field-based UAV aerial phenotyping pipeline to perform sorghum panicle counting.

### 3. RESULTS AND DISCUSSION

To perform automatic identification and counting, we sampled images to create training data to be used to train the machine learning model. This method is entirely based on the identification of pixels of the same class, either a panicle or not. We divided the data into training, test, and validation datasets and applied a reduced and centered transformation to the variables to remove the asymmetry and the different weights due to the different scale of the variables. The next step in the process of building the classification model is to identify the variables that have high correlations and eliminate the variables so that no absolute correlation per pair of predictor variables is greater than a predefined level. In this case, we chose to eliminate variables that have pairwise correlations greater than 0.7.

FIGURE 2 illustrates our method of identifying and detecting the number of panicles. The variable X represents the pixels of the Sorghum raster training data. In total, our network consists

<figure><img src="image_4.png" /><figcaption>Figure 2: Artificial neural network model followed by segmentation for panicle identification and counting. Xᵢ are the pixels images of subsample of the orthomosaic drone images and Yᵢ are panicle or no panicle classes.</figcaption></figure>
of seven layers. At the output of the artificial neural network, an automated identification of the sorghum panicles is performed, then the result of the identification is passed to a segmentation.

FIGURE 3a represents the input variable selection step of the ML model. The indices are matched for the RGB UAV, and they are all calculated from the red and green combination, the green band corresponding to the NIR band in the multi spectral images. The variables selected are DVI (green, red), GEMI (green, red) and green band. These important variables in the construction of the identification and counting model depend on the phenology of the plant. For example, from the beginning of flowering other vegetation indices are most important. In our example of important flowering, it is normal that the green component and the index of the DVI (Green, Red) is important. FIGURE 3b shows a comparison of two machine learning models among the three models used in this paper. The SVM achieved a better result in terms of accuracy and kappa compared to the neural networks but in application of the segmentation the two models were similar. A radial basis kernel was used for the SVM, and the “sigma” parameter was kept constant at a value of 5.6 determined analytically. The accuracy was used to select the best performing model using the largest value. The final values used for the model are σ = 5.6 and cost C = 16.

<figure><img src="image_5.png" /><figcaption>Figure 3: (a) Illustration of all the RGB-like vegetation indices of the multi-correlation matrix which defines the basis for variable selection of the model. (b) Results between two machine learning models for identifying panicles of sorghum. Accuracy and kappa are the two metrics used to assess the performance of the model.</figcaption></figure>

For this identification and counting, we have chosen an example of an RGB image of a sorghum field. A more detailed examination of this image shows mainly a more uncluttered area in yellow on the image of FIGURE 4a, where the flowering of plants is very well developed compared to the blue area where the flowering is important at a less mature stage and with dense foliage, the area in white is in the same state of flowering but the panicles are more widely dispersed between them. These are very sensitive to the accuracy of the model. The FIGURE shows the detection and counting output: it is noticeable that the yellow zone presents a better counting result 98% as well as in the white zone, however in the areas with very dense foliage and early flowering the performance of
<figure><img src="image_6.png" /><figcaption>Figure 4: Results of the machine learning model followed by segmentation for sorghum grain panicle counting. (a) denote the input image to the algorithm and (b) and (c) represent the results.</figcaption></figure>

the model is less accurate. In summary, our method can automatically count panicles well when the flowering is mature, or the panicles are small and at the right distance from each other.

We have noted some limitations to the method proposed here. The most important limitation is when the sorghum is in the late flowering phase. In this situation, there is a decreased detection of sorghum panicles. A second limitation is noted when the drone flies beyond 50m during field images acquisition.

## 4. CONCLUSION

In this paper, we demonstrated the application of artificial intelligence for sorghum phenotyping using drone imagery. Experimental results in this paper show the artificial neural networks combined with image segmentation have the potential to perform accurate panicle detection and counting. The model used produced an average accuracy and kappa of 0.98. Our research shows promising steps towards the use of AI computer vision in-field phenotyping to assist plant breeders.

## 5. CONFLICT OF INTEREST

We have no conflicts of interest to disclose.
# 6. ACKNOWLEDGEMENTS

This research was supported by Centre d'Etude Régional pour l'Amélioration de l'Adaptation à la Sécheresse (CERAAS). The author thanks Dr. Alain Audebert, Dr. Gregory Beurier and Dr. Ndjido Kane, who initiated the 2D and 3D phenotyping approaches study of morpho-physiological traits under the Grain Legumes and Dryland Cereals program (GLDC).

# References

[1] Milics G. Application of UAVs in Precision Agriculture. International Climate Protection. Springer, Cham. 2019:93-97.

[2] Gago J, Douthe C, Coopman RE, Gallego PP, Ribas-Carbo M, et al. UAVs Challenge to Assess Water Stress for Sustainable Agriculture. Agricultural water management. 2015;153:9-19.

[3] Grenzdörffer GJ, Engel A, Teichert B. The Photogrammetric Potential of Low-Cost UAVs in Forestry and Agriculture. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. 2008;1207-1214.

[4] Niu H, Zhao T, Wang D, Chen Y. Estimating Evapotranspiration With UAVs in Agriculture: A Review. 2019 ASABE Annual International Meeting. 2019;1-12.

[5] Simelli I, Tsagaris A. The Use of Unmanned Aerial Systems (UAS) in Agriculture. HAICTA. 2015;730-736.

[6] Cerro JD, Cruz Ulloa C, Barrientos A, de León Rivas J. Unmanned Aerial Vehicles in Agriculture: A Survey. Agronomy. 2021;11:203.

[7] Maddikunta PK, Hakak S, Alazab M, Bhattacharya S, Gadekallu TR, et. al. Unmanned Aerial Vehicles in Smart Agriculture: Applications, Requirements, and Challenges. IEEE Sensors Journal. 2021;12-27.

[8] Torres-Rua AF. Challenges and Opportunities of Unmanned Aerial Vehicles in Agriculture in the Next Decade. Autonomous Air and Ground Sensing Systems for Agricultural Optimization and Phenotyping VI. 2021;11-74.

[9] Reinecke M, Prinsloo T. The Influence of Drone Monitoring on Crop Health and Harvest Size. 2017 1st International conference on next generation computing applications. (NextComp).2017;5-10.

[10] Bruscolini M, Suttor B, Giustarini L, Zare M, Gaffinet B, et al. Drone Services for Plant Water-Status Mapping. 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. 2021; 8527-8530.

[11] Adeluyi O, Harris A, Foster T, Clay GD. Exploiting Centimetre Resolution of Drone-Mounted Sensors for Estimating Mid-Late Season Above Ground Biomass in Rice. European Journal of Agronomy. 2022;132:126411.

[12] Gano B, Dembele JS, Ndour A, Luquet D, Beurier G, et. al. Using UAV Borne, Multi-Spectral Imaging for the Field Phenotyping of Shoot Biomass, Leaf Area Index and Height of West African Sorghum Varieties Under Two Contrasted Water Conditions. Agronomy. 2021;11:850.
[13] Dembele JS, Gano B, Kouressy M, Dembele LL, Doumbia M, et. al. Plant Density and Nitrogen Fertilization Optimization on Sorghum Grain Yield in Mali. Agronomy Journal. 2021;113; 4705-4720.

[14] Luvisi A. Electronic Identification Technology for Agriculture, Plant, and Food. A Review. Agronomy for sustainable development. 2016;36:13.

[15] Roy AM, Bhaduri J. A Deep Learning Enabled Multi-Class Plant Disease Detection Model Based on Computer Vision. AI. 2021;2:413-428.

[16] Xiong J, Yu D, Liu S, Shu L, Wang X, et. al. A Review of Plant Phenotypic Image Recognition Technology Based on Deep Learning. Electronics. 2021;10:81.

[17] Hati AJ, Singh RR. Artificial Intelligence in Smart Farms: Plant Phenotyping for Species Recognition and Health Condition Identification Using Deep Learning. AI. 2021;2:274-289.

[18] Neupane B, Horanont T, Hung ND. Deep Learning-Based Banana Plant Detection and Counting Using High-Resolution Red-Green-Blue (RGB) Images Collected From Unmanned Aerial Vehicle (UAV). PloS one. 2019;14: e0223906.

[19] Jin S, Su Y, Gao S, Wu F, Ma Q, et.al. Separating the Structural Components of Maize for Field Phenotyping Using Terrestrial Lidar Data and Deep Convolutional Neural Networks. IEEE Transactions on Geoscience and Remote Sensing. 2019;58:2644-2658.

[20] Kong H, Chen P. Mask R-CNN-Based Feature Extraction and Three-Dimensional Recognition of Rice Panicle CT Images. Plant Direct. 2021;5:e00323.

[21] Zhou C, Ye H, Hu J, Shi X, Hua S, et. al. Automated Counting of Rice Panicle by Applying Deep Learning Model to Images From Unmanned Aerial Vehicle Platform. Sensors. 2019;19:3106.

[22] Baweja HS, Parhar T, Mirbod O, Nuske S. StalkNet: A Deep Learning Pipeline for High-Throughput Measurement of Plant Stalk Count and Stalk Width. In: Hutter M., Siegwart R. (eds) Field and Service Robotics. Springer Proceedings in Advanced Robotics. 2018: 5:5-18.

[23] Liu X, Deng Z, Yang Y. Recent Progress in Semantic Image Segmentation. Artificial Intelligence Review.2019;52:1089-1106.

[24] Ren C, Kim DK, Jeong D. A Survey of Deep Learning in Agriculture: Techniques and Their Applications. Journal of Information Processing Systems. 2020;16:1015-1033.

[25] Zhang Q, Liu Y, Gong C, Chen Y, Yu H. Applications of Deep Learning for Dense Scenes Analysis in Agriculture: A Review. Sensors. 2020;20:1520.

[26] Santos L, Santos FN, Oliveira PM, Shinde P. Deep Learning Applications in Agriculture: A Short Review. Springer, Cham. InIberian Robotics conference 2019; 20:139-151.

[27] Yang X, Sun M. A Survey on Deep Learning in Crop Planting. In IOP Conference Series: Materials Science and Engineering.2019;490.