# Extracting Masks from Herbarium Specimen Images Based on Object Detection and Image Segmentation Techniques

Hanane Ariouat, Youcef Sklab, Youcef Sklab, Marc Pignal, Régine Vignes

Lebbe, Jean-Daniel Zucker, Edi Prifti, Eric Chenin

<figure><img src="image_1.png" /></figure>

## To cite this version:

Hanane Ariouat, Youcef Sklab, Youcef Sklab, Marc Pignal, Régine Vignes Lebbe, et al.. Extracting Masks from Herbarium Specimen Images Based on Object Detection and Image Segmentation Techniques. Biodiversity Information Science and Standards, 2023, 7, 10.3897/biss.7.112161. hal-04608858

HAL Id: hal-04608858

https://hal.science/hal-04608858v1

Submitted on 23 Jul 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Conference Abstract

# Extracting Masks from Herbarium Specimen Images Based on Object Detection and Image Segmentation Techniques

Hanane Ariouat†, Youcef Sklab†, Marc Pignal§, Régine Vignes Lebbe§, Jean-Daniel Zucker‡,¶, Edi Prifti‡,ł, Eric Chenin‡

† IRD, Sorbonne Université, UMMISCO, F-93143, Bondy, France

§ Institut de Systématique, Évolution et Biodiversité, ISYEB – Muséum national d’Histoire naturelle, Centre national de la Recherche scientifique, Sorbonne Université, École pratique des Hautes Études, Université des Antilles, Paris, France

| Sorbonne Université, INSERM, Nutrition et Obesities; systemic approaches, NutriOmique, AP-HP, Hôpital Pitié-Salpêtrière, Paris, France

Corresponding author: Hanane Ariouat (hanane.ariouat@ird.fr)

Received: 04 Sep 2023 | Published: 06 Sep 2023

Citation: Ariouat H, Sklab Y, Pignal M, Vignes Lebbe R, Zucker J-D, Prifti E, Chenin E (2023) Extracting Masks from Herbarium Specimen Images Based on Object Detection and Image Segmentation Techniques. Biodiversity Information Science and Standards 7: e112161. https://doi.org/10.3897/biss.7.112161

## Abstract

Herbarium specimen scans constitute a valuable source of raw data. Herbarium collections are gaining interest in the scientific community as their exploration can lead to understanding serious threats to biodiversity. Data derived from scanned specimen images can be analyzed to answer important questions such as how plants respond to climate change, how different species respond to biotic and abiotic influences, or what role a species plays within an ecosystem. However, exploiting such large collections is challenging and requires automatic processing. A promising solution lies in the use of computer-based processing techniques, such as Deep Learning (DL). But herbarium specimens can be difficult to process and analyze as they contain several kinds of visual noise, including information labels, scale bars, color palettes, envelopes containing seeds or other organs, collection-specific barcodes, stamps, and other notes that are placed on the mounting sheet. Moreover, the paper on which the specimens are mounted can degrade over time for multiple reasons, and often the paper's color darkens and, in some cases, approaches the color of the plants.
Neural network models are well-suited to the analysis of herbarium specimens, while making abstraction of the presence of such visual noise. However, in some cases the model can focus on these elements, which eventually can lead to a bad generalization when analyzing new data on which these visual elements are not present (White et al. 2020). It is important to remove the noise from specimen scans before using them in model training and testing to improve its performance. Studies have used basic cropping techniques (Younis et al. 2018), but they do not guarantee that the visual noise is removed from the cropped image. For instance, the labels are frequently put at random positions into the scans, resulting in cropped images that still contain noise. White et al. (2020) used the Otsu binarization method followed by a manual post-processing and a blurring step to adjust the pixels that should have been assigned to black during segmentation. Hussein et al. (2020) used an image labeler application, followed by a median filtering method to reduce the noise. However, both White et al. (2020) and Hussein et al. (2020) consider only two organs: stems and leaves. Triki et al. (2022) used a polygon-based deep learning object detection algorithm. But in addition to being laborious and difficult, this approach does not give good results when it comes to fully identifying specimens.

In this work, we aim to create clean high-resolution mask extractions with the same resolution as the original images. These masks can be used by other models for a variety of purposes, for instance to distinguish the different plant organs. Here, we proceed by combining object detection and image segmentation techniques, using a dataset of scanned herbarium specimens. We propose an algorithm that identifies and retains the pixels belonging to the plant specimen, and removes the other pixels that are part of non-plant elements considered as noise. A removed pixel is set to zero (black). Fig. 1 illustrates the complete masking pipeline in two main stages, object detection and image segmentation.

<figure><img src="image_3.png" /><figcaption>Figure 1. Overview of the masking pipeline.</figcaption></figure>
In the first stage, we manually annotated the images using bounding boxes in a dataset of 950 images. We identified (Fig. 2) the visual elements considered to be noise (e.g., scale-bar, barcode, stamp, text box, color pallet, envelope). Then we trained the model to automatically remove the noise elements. We divided the dataset into 80% training, 10% validation and 10% test set. We ultimately achieved a precision score of 98.2%, which is a 3% improvement from the baseline. Next, the results of this stage were used as input for image segmentation, which aimed to generate the final mask. We blacken the pixels covered by the detected noise elements, then we used HSV (Hue Saturation Value) color segmentation to select only the pixels with values in a range that corresponds mostly to a plant color. Finally, we applied the morphological opening operation that removes noise and separates objects; and the closing operation that fills gaps, as described in Sunil Bhutada et al. (2022) to remove the remaining noise. The output here is a generated mask that retains only the pixels that belong to the plant. Unlike other proposed approaches, which focus essentially on leaves and stems, our approach covers all the plant organs (Fig. 3).

<figure><img src="image_4.png" /><figcaption>Figure 2.
Annotation of noise elements.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 3.
Examples of image segmentation.</figcaption></figure>

Our approach removes the background noise from herbarium scans and extracts clean plant images. It is an important step before using these images in different deep learning models. However, the quality of the extractions varies depending on the quality of the scans, the condition of the specimens, and the paper used. For example, extractions made from samples where the color of the plant is different from the color of the background were more accurate than extractions made from samples where the color of the plant and background are close. To overcome this limitation, we aim to use some of the obtained extractions to create a training dataset, followed by the development and the training of a generative deep learning model to generate masks that delimit plants.

**Keywords**

image noise, deep learning, herbarium mounting sheet

**Presenting author**

Youcef Sklab
Presented at

TDWG 2023

Conflicts of interest

The authors have declared that no competing interests exist.

References

* Hussein BR, Malik OA, Ong W, Slik JWF (2020) Semantic Segmentation of Herbarium Specimens Using Deep Learning Techniques. In: Lecture Notes in Electrical Engineering (Ed.) Computational Science and Technology. 603. https://doi.org/10.1007/978-981-15-0058-9_31

* Sunil Bhutada, Nakerakanti Yashwanth, Puppala Dheeraj, Kethavath Shekar (2022) Opening and closing in morphological image processing. World Journal of Advanced Research and Reviews 14 (3): 687-695. https://doi.org/10.30574/wjarr.2022.14.3.0576

* Triki A, Bouaziz B, Mahdi W, Hamed H, Gaikwad J (2022) Deep learning based approach for digitized herbarium specimen segmentation. Multimedia Tools and Applications 81 (20): 28689-28707. https://doi.org/10.1007/s11042-022-12935-8

* White A, Dikow R, Baugh M, Jenkins A, Frandsen P (2020) Generating segmentation masks of herbarium specimens and a data set for training segmentation models using deep learning. Applications in Plant Sciences 8 (6).

* Younis S, Weiland C, Hoehndorf R, Dressler S, Hickler T, Seeger B, Schmidt M (2018) Taxon and trait recognition from digitized herbarium specimens using deep convolutional neural networks. Botany Letters 165: 377-383. https://doi.org/10.1080/23818107.2018.1446357