# Towards a Deep Learning-Powered Herbarium Image Analysis Platform

Youcef Sklab‡, Hanane Ariouat‡, Youssef Boujydah‡, Yassine Qacami‡, Edi Prifti‡, Jean-daniel Zucker‡, Régine Vignes Lebbe§, Eric Chenin‡

‡ IRD, Sorbonne Université, UMMISCO, F-93143, Bondy, France

§ Institut de Systématique, Evolution, Biodiversité (ISYEB), Muséum national d'Histoire naturelle, CNRS, Sorbonne Université, EPHE, Université des Antilles, Paris, France

Corresponding author: Youcef Sklab (youcef.sklab@ird.fr)

Received: 27 Aug 2024 | Published: 28 Aug 2024

Citation: Sklab Y, Ariouat H, Boujydah Y, Qacami Y, Prifti E, Zucker J-daniel, Vignes Lebbe R, Chenin E (2024)

Towards a Deep Learning-Powered Herbarium Image Analysis Platform. Biodiversity Information Science and Standards 8: e135629. https://doi.org/10.3897/biss.8.135629

## Abstract

Global digitization efforts have archived millions of specimen scans worldwide in herbarium collections, which are essential for studying plant evolution and biodiversity. *ReColNat* hosts, at present, over 10 million images. However, analyzing these datasets poses crucial challenges for botanical research. The application of deep learning in biodiversity analyses, particularly in analyzing herbarium scans, has shown promising results across numerous tasks (Ariouat et al. 2023, Ariouat et al. 2024, Groom et al. 2023, Sahraoui et al. 2023).

Within the *e-Col+project* (ANR-21-ESRE-0053), we are developing multiple deep learning models aimed at identifying plant morphological traits. We have developed pipelines and models for cleaning, analyzing, and transforming herbarium images, including models for: i) detecting non-vegetal elements, such as barcodes, envelopes, labels, etc.; ii) detecting plant organs, including leaves, flowers, fruits, etc.; and iii) segmenting to recognize plant parts for image cleaning. We are also developing models for classification tasks related to various morphological traits.

To validate these models, improve their generalization, and make them easily usable by end-users, deploying them within a generic platform is crucial. The generic platform
called PlantAI, currently under development by the e-Col+ project, should enable easy deployment during development for testing and allow users to load annotations for new traits in order to train a model and add it to the existing catalog. The platform is based on a microservice architecture, allowing users to upload images, create custom datasets, and access various AI models for image analysis.

The platform is composed of four main modules, as illustrated in Fig. 1. The first module is the collaborative workspace manager, which allows users to create projects and image datasets and invite other users to collaborate on a project. The second module is the navigation interface and dashboards. This module integrates a search engine using metadata and AI annotations, a navigation interface between projects, datasets, and specimens, as well as dashboards for analysis across datasets, specimens, and AI models.

<figure><img src="image_1.png" /><figcaption>Figure 1. Main components of the PlantAI platform.</figcaption></figure>
The third module is the dataset manager, which handles metadata and annotations associated with the specimens. These annotations can be produced either by expert users or by AI models. The fourth module is the AI models management module, so that models can be used to generate AI annotations of specimen. During the development lifecycle of an AI model, users can create datasets and annotate them with AI models. These annotations can be in two possible states: validated by experts and non-validated. Users collaborating on a project can indicate errors in the model predictions and leave comments to explain their evaluations. These corrections made by experts can be used to retrain the models and thus improve their performance.

This platform, will be highly beneficial for botanists, enhancing the efficiency and effectiveness of biodiversity analyses from herbarium scans. We aim to provide users with a catalog of AI models through this platform and allow them to import their own datasets with their own annotations regarding traits of their choice. Users will be able to select a model from the AI model catalog and train it using their dataset. Ultimately, the model obtained from this training will be automatically deployed to be available for AI annotation. The annotations produced by this model will be automatically available in the filtering and navigation interface, thus allowing for dynamic and automatic integration of the AI annotations into the navigation interface.

## Keywords

herbarium scans, AI annotation, navigation interface

## Presenting author

Youcef Sklab

## Conflicts of interest

The authors have declared that no competing interests exist.

## References

* Ariouat H, Sklab Y, Pignal M, Vignes Lebbe R, Zucker J, Prifti E, Chenin E (2023) Extracting Masks from Herbarium Specimen Images Based on Object Detection and Image Segmentation Techniques. Biodiversity Information Science and Standards 7 https://doi.org/10.3897/biss.7.112161

* Ariouat H, Sklab Y, Pignal M, Jabbour F, Vignes Lebbe R, Prifti E, Zucker J, Chenin E (2024) Enhancing YOLOv7 for Plant Organs Detection Using Attention-Gate Mechanism. Advances in Knowledge Discovery and Data Mining223-234. https://doi.org/10.1007/978-981-97-2253-2_18
* Groom Q, Dillen M, Addink W, Ariño A, Bölling C, Bonnet P, Cecchi L, Ellwood E, Figueira R, Gagnier P, Grace O, Güntsch A, Hardy H, Huybrechts P, Hyam R, Joly A, Kommineni VK, Larridon I, Livermore L, Lopes RJ, Meeus S, Miller J, Milleville K, Panda R, Pignal M, Poelen J, Ristevski B, Robertson T, Rufino A, Santos J, Schermer M, Scott B, Seltmann K, Teixeira H, Trekels M, Gaikwad J (2023) Envisaging a global infrastructure to exploit the potential of digitised collections. *Biodiversity Data Journal* 11 https://doi.org/10.3897/bdj.11.e109439

* Sahraoui M, Sklab Y, Pignal M, Vignes Lebbe R, Guigue V (2023) Leveraging Multimodality for Biodiversity Data: Exploring joint representations of species descriptions and specimen images using CLIP. *BISS* 7 https://doi.org/10.3897/biss.7.112666