# SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification

Youcef Sklab, Hanane Ariouat, Eric Chenin, Edi Prifti, Jean-daniel Zucker

► To cite this version:

Youcef Sklab, Hanane Ariouat, Eric Chenin, Edi Prifti, Jean-daniel Zucker. SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification. IET Computer Vision, 2025, 19, 10.1049/cvi2.70036. hal-05184161

HAL Id: hal-05184161

https://hal.science/hal-05184161v1

Submitted on 24 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

ORIGINAL RESEARCH

OPEN ACCESS

# SIM-Net: A Multimodal Fusion Network Using Inferred 3D Object Shape Point Clouds From RGB Images for 2D Classification

Youcef Sklab¹ | Hanane Ariouat¹ | Eric Chenin¹ | Edi Prifti¹,² | Jean-Daniel Zucker¹,²

¹IRD, Sorbonne Université, UMMISCO, Bondy, France | ²INSERM, Nutrition et Obesities, Systemic Approaches, NutriOmique, AP-HP, Hôpital Pitié-Salpétrière, Paris, France

**Correspondence:** Youcef Sklab (youcef.sklab@ird.fr)

**Received:** 12 December 2024 | **Revised:** 23 June 2025 | **Accepted:** 29 June 2025

**Handling Editor:** Wenguan Wang

**Funding:** This study was supported by Agence Nationale de la Recherche (ANR), e-Col+ Project (Grant ANR-21-ESRE-0053), GENCI, HPC and Storage Resources (Grant 2023-A0150114385).

**Keywords:** botany | computer vision | image classification | image processing | image representation | learning (artificial intelligence) | neural net architecture | neural nets | object recognition

## ABSTRACT

We introduce the shape-image multimodal network (SIM-Net), a novel 2D image classification architecture that integrates 3D point cloud representations inferred directly from RGB images. Our key contribution lies in a pixel-to-point transformation that converts 2D object masks into 3D point clouds, enabling the fusion of texture-based and geometric features for enhanced classification performance. SIM-Net is particularly well-suited for the classification of digitised herbarium specimens—a task made challenging by heterogeneous backgrounds, nonplant elements, and occlusions that compromise conventional image-based models. To address these issues, SIM-Net employs a segmentation-based preprocessing step to extract object masks prior to 3D point cloud generation. The architecture comprises a CNN encoder for 2D image features and a PointNet-based encoder for geometric features, which are fused into a unified latent space. Experimental evaluations on herbarium datasets demonstrate that SIM-Net consistently outperforms ResNet101, achieving gains of up to 9.9% in accuracy and 12.3% in F-score. It also surpasses several transformer-based state-of-the-art architectures, highlighting the benefits of incorporating 3D structural reasoning into 2D image classification tasks.

## 1 | Introduction

Over the past two decades, deep learning has made profound advancements leading to the development of numerous innovative architectures that have significantly advanced the field of artificial intelligence [1-10]. In particular, the field of computer vision has seen the emergence of many high-performing models, especially for image classification [1-5, 10]. However, traditional classification methods often struggle with complex backgrounds, occlusions, and noisy environments—challenges

that are especially pronounced in biological datasets such as digitised herbarium specimens,¹ which have become vital resources for botany and biodiversity research [11-17]. Models such as ResNet [5] and Vision Transformers [10], exhibit varying degrees of robustness to noise and differences in foreground and background sensitivity [18, 19].

Analysing digitised herbarium specimens remains challenging due to varying textures and presence of numerous nonplant elements, such as scale bars, labels, and extraneous artefacts

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

© 2025 The Author(s). *IET Computer Vision* published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology.
(cf., Figure 1). These elements, often scattered randomly across herbarium sheets, can obscure crucial information within the specimens.

Further complicating the task, the quality of the paper used for herbarium sheets varies significantly, with some fixed-on newspapers and others on paper that has degraded and darkened over time, often to a hue like that of the plants themselves. This degradation makes it increasingly difficult for image classification models to distinguish the specimen from the background. Noise, in this context, refers to any irrelevant or extraneous information that obscures or distorts the herbarium specimen. This, in turn, complicates the task of feature classification.

In a previous study [16], we explored herbarium image classification for detecting the presence of thorns on plants using ResNet on two datasets: one with raw images and another with segmented images where all background elements were removed, leaving only the plant. We found that while ResNet achieved better training and validation performance on the raw image dataset (95% accuracy vs. 94% on the segmented dataset), the model trained on segmented images demonstrated superior generalisation. When tested on a set of 60 new images (30 with and 30 without thorns), the accuracy of the ResNet model trained on raw images dropped to 28.8%. In contrast, the model trained on segmented images showed a smaller decline, achieving 58.8% accuracy. These results suggest that removing background noise enabled the model to focus more effectively on relevant plant features, thereby improving its ability to generalise to unseen data. Figure 2 illustrates the decisive regions identified by ResNet in unsegmented and segmented images. The top example shows an incorrect prediction where the model focused on the background instead of the plant. The bottom example demonstrates a correct prediction, with the model properly focussing on the plant itself. This contrast highlights the value of background removal in enhancing a model's ability to identify key features, emphasising the importance of targeted focus in deep-learning-based plant

classification. Our study highlights the critical role of dataset preprocessing in developing models with improved generalisation for botanical image classification.

Although dataset preprocessing is crucial for improving classification, traditional 2D image classification models face a more fundamental limitation: they rely solely on colour, texture, and spatial patterns. This reliance makes them highly sensitive to background noise and occlusions, reducing their ability to generalise in complex settings. A promising solution lies in multimodal learning, which integrates complementary data representations to enhance classification. Multimodal architectures have transformed many AI fields, particularly in 3D object detection and vision-language models [20], as demonstrated by pioneering models such as CLIP [9]. In image classification, combining 2D texture information with 3D shape representations can yield richer features and greater robustness to background variations. Fusing 2D and 3D information to enhance accuracy has thus become an increasingly active area of research [21].

<figure><img src="image_4.png" /><figcaption>**FIGURE 2** | The decisive parts in prediction on unsegmented and segmented images.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**FIGURE 1** | Examples illustrating the diversity in paper colour, quality and the nonplant elements in herbarium sheets (nonexhaustive).</figcaption></figure>
This paper introduces SIM-Net—Shape-Image Multimodal Network—(Figure 3), a novel approach for 2D image classification that incorporates 3D object shape information extracted from 2D images. Our approach employs a multimodal fusion strategy, combining standard 2D images with a structured 3D representation of object shapes in the form of point clouds. These point clouds preserve essential properties such as an unordered, irregular format and permutation invariance. Unlike traditional classification models that rely solely on 2D textures and spatial arrangements within images, SIM-Net integrates geometric and structural information by transforming segmented 2D object masks into 3D point clouds. We begin by generating masks of the targeted objects—in this case, plant specimens—using segmentation techniques, and subsequently transform these masks into 3D object shape point clouds. Specifically, a point cloud is derived from an image through a pixel-to-point transformation process: each pixel, initially defined by its position coordinates (x, y) and colour values (r, g, b) in the RGB colour space, is converted into a 3D point. This transformation maps the pixel's spatial and colour information into three coordinates (x, y, z) in the point cloud, effectively translating 2D image data into a 3D representation. The resulting pairs of 2D images and corresponding 3D point clouds are processed through a multimodal fusion architecture (SIMI-Net) composed of two networks: a CNN network (ResNet [5]) and a PointNet network [7]. The 3D shape representations provide geometric and spatial information not readily discernible in 2D images, thereby enriching the feature space available to the learning model. The 3D point cloud representation enables the network to learn plant morphology independently of background noise, lighting variations, and paper degradation—key factors that hinder the performance of purely 2D-based classifiers.

ResNet [5], with its deep convolutional layers, excels in extracting hierarchical and detailed features from 2D images, such as textures, colours, and edges. However, its capabilities are limited to the information present in two dimensions. PointNet [7] processes 3D point clouds, capturing the spatial geometry of objects. By understanding the arrangement and

relationship of points in the three-dimensional space, it can glean insights about the shape and structure of objects, aspects that 2D images alone cannot provide. When these two networks are fused in a merging architecture, they complement each other, creating a more robust and comprehensive feature set for classification.

Unlike existing multimodal approaches that primarily focus on 3D object detection, such as LiDAR-camera fusion in autonomous driving, SIM-Net bridges the gap between 2D image classification and 3D structural learning, enhancing object recognition by creating a more expressive feature space. Unlike previous models that rely on sensor-based 3D data, such as LiDAR or RGB-D cameras, our approach directly infers point clouds from 2D images. Additionally, while traditional CNNs primarily leverage colour and texture cues, SIM-Net enhances classification by integrating shape-based reasoning, resulting in a feature-rich multimodal representation that improves generalisation to unseen samples.

A prime application of our approach is the classification of digitised herbarium specimens. We conducted extensive experiments using diverse datasets comprising a wide range of plants. Our model demonstrated a significant improvement in classification accuracy compared to ResNet alone.

In summary, our main contributions in this paper are as follows:

* We introduce a pixel-to-point transformation pipeline that systematically converts segmented 2D objects into 3D point clouds.
* We propose a multimodal feature fusion network designed as a dual-branch architecture, with an image encoder and a point cloud encoder. These two complementary feature sets are fused into a unified latent space, enabling the model to learn both texture-based and shape-based characteristics simultaneously.

<figure><img src="image_6.png" /><figcaption>**FIGURE 3** | 2D image to 3D point cloud data transformation pipeline and the shape-image multimodal fusion network.</figcaption></figure>
* We demonstrate that transforming 2D images into point clouds allows for the infusion of additional knowledge into the classification process.
* We show that point cloud-based architectures can surpass more complex traditional 2D classification models when applied to 2D image classification tasks.

The paper is organised as follows: Section 2 reviews related work. Section 3 details the methodology and processing techniques. Section 4 presents the experimental setup, data collections, findings and a discussion. Finally, Section 5 concludes with a summary of the results and potential directions for future research.

## 2 | Related Work

### 2.1 | 2D Image Classification

2D image classification has advanced considerably over the last decade. AlexNet [2], introduced key innovations such as the ReLU activation function, dropout layers to mitigate overfitting, and overlapping pooling. VGGNet [3], emphasised the depth, utilising 16 convolutional layers. GoogLeNet/Inception [4] introduced the Inception module, which concatenates feature maps from filters of different sizes to capture multi-scale information efficiently. ResNet [5] introduced residual connections, allowing gradients to flow directly through networks exceeding 100 layers in depth. DenseNet [22] connects each layer to every other layer in a feed-forward fashion, maximising information flow throughout the network. Transformer Models [6], originally developed for natural language processing, have been adapted for image classification through self-attention mechanisms that support parallelisation and capture long-range dependencies. Vision Transformers [10] treat image patches as sequences for this purpose. Despite their general effectiveness, traditional 2D image classification techniques face substantial challenges when applied to digitised herbarium specimens, largely due to their sensitivity to background noise [18]. Figure 1 illustrates the wide variation in paper colour, quality, and the presence of nonplant elements in herbarium images. Models such as ResNet and Vision Transformers display varying degrees of robustness to background noise and sensitivity to foreground-background contrast [18, 19].

### 2.2 | 3D Point Cloud Classification

A point cloud is a collection of data points in space. Processing point clouds is challenging due their irregular structure, which requires networks that are invariant to point order, density variations, and spatial transformations. PointNet [7] addresses this challenge with a neural network specifically designed to process point clouds while maintaining permutation invariance, offering a streamlined and efficient approach. Despite its simplicity, PointNet achieves notable efficiency and effectiveness. PointNet++ [23] extends this architecture by applying PointNet recursively on a nested partitioning of the input point set, leveraging metric space distances to learn local features at progressively larger contextual scales. The Point Transformer

[24] further enhances point cloud processing by extracting both local and global features through a novel local-global attention mechanism. PointMLP [25] adopts a pure residual MLP (multi-layer perceptron) architecture without complex local geometric extractors yet delivers competitive performance. PointNeXt [26] builds upon PointNet++ by incorporating improved training strategies, an inverted residual bottleneck design, and separable MLPs, enabling efficient and scalable point cloud processing.

### 2.3 | 2D-3D Multimodal Architectures

Numerous studies have explored strategies for fusing 2D images with 3D point clouds, employing a variety of approaches. However, most of them focus on improving neural network performance in 3D domains, such as 3D object detection [27, 28] or 3D segmentation [29, 30]. Poliyapram et al. [29] proposed the point-wise LiDAR and image multimodal fusion network (PMNet) for 3D semantic segmentation of aerial point clouds by integrating aerial image features with LiDAR data. PMNet's architecture comprises two main components: a PointNet-based [7] backbone for point cloud feature extraction and a CNN Encoder-Decoder for image feature extraction. The point-wise fusion method combines features from both networks using a spatial correspondence table (XY coordinate index from the original point cloud data). This ensures that the extracted point-wise image features precisely mirror the point cloud structure in terms of the number of patches, points (N), and feature dimensions. The network merges 128 features from 2D aerial images and 128 features from the point cloud before feeding them into an MLP for final processing. Pradelle et al. [30] proposed a segmentation technique that leverages both 2D images and 3D point cloud data, eliminating the need for 3D ground truth and relying solely on 2D labels. This approach addresses the challenges of 3D ground truth labelling, which is labour-intensive and complex, in contrast to the simpler process of labelling 2D images. Their method employs a lightweight encoder to process features from a 3D point set, enhancing 2D segmentation results. The network accepts a point cloud, an image, and the corresponding projection matrix as inputs. It processes the point cloud with a 3D encoder to extract features for each visible point, which are then projected onto the image to create a 2D geometric feature map. This map is combined with the RGB image using a single-layer MLP and fed into a 2D segmentation network.

Yang et al. [31] introduced the multimodal interlaced transformer (MIT) for weakly supervised 3D point cloud segmentation using only scene-level annotations. Their architecture employs two separate encoders: one for 2D multi-view images and another for 3D point clouds. The decoder performs interlaced 2D-3D cross-attention without requiring camera poses or depth maps. Notably, MIT alternates the roles of queries and key-value pairs across decoder layers to enable iterative fusion, enhancing each modality with features from the other. Zhao et al. [32] introduced SAFNet (similarity-aware fusion network), a late fusion framework for 3D semantic segmentation that adaptively fuses 2D image features and 3D point cloud data. Unlike traditional projection-based fusion methods, SAFNet addresses key challenges such as local mismatching and varying
point density by computing geometric and contextual similarities between original and back-projected point clouds. These similarity metrics guide the fusion process, allowing the model to dynamically adjust the contribution of each modality per point.

Guo et al. [27] presented the deep multi-scale and multi-modal fusion (DMMF) for 3D object detection in autonomous driving, where cameras and LiDAR gather 3D spatial information. DMMF involves projecting the point cloud onto a bird's eye view (BEV) and extracting features from both the BEV map and the RGB image. These multi-modal features are then fused using a deep multi-scale fusion technique, followed by their application in a position regression and classification network for precise object classification and positioning. This method has shown impressive results on the KITTI dataset, especially in detecting cars and pedestrians, and has demonstrated significant accuracy improvements for more challenging data. Gu et al. [28] introduced PointSee, a lightweight and flexible multimodal fusion solution to improve 3D object detection (3OD) networks. It integrates LiDAR point clouds with scene images through a unique architecture comprising a hidden module (HM) that decorates LiDAR point clouds using 2D image information in an offline fusion manner. This leads to minimal or no adaptations of existing 3OD networks. A seen module (SM) enriches the LiDAR point clouds by acquiring point-wise representative semantic features, leading to enhanced performance of existing 3OD networks. This approach streamlines fusion without significant network modifications, offering a plug-and-play module for existing 3OD frameworks. Extensively validated on outdoor (KITTI) and indoor (SUN-RGBD) datasets, PointSee demonstrates marked improvements in performance, particularly when combined with 3DSSD. Its contributions lie in its novel modules (HM and SM), enhancing 3D object detection while maintaining network simplicity and efficiency. Similarly, SAMFusion [33] addresses 3D object detection in harsh environments (night, fog, and snow) using RGB, NIR gated cameras, radar, and LiDAR. Its architecture incorporates depth-aware attention and a transformer operating in BEV space. SAMFusion targets the autonomous driving domain and relies entirely on physical sensors.

Y. Chen et al. [34] tackle the task of depth completion from RGBD inputs by proposing a novel architecture that learns joint 2D-3D representations. Their core contribution is a modular 2D-3D fusion block composed of two domain-specific sub-networks: one operating in the 2D image space via traditional convolutions, and another in the 3D space using continuous convolutions on sparse point clouds. These branches extract appearance and geometric features independently, which are then fused in the image domain. By stacking these blocks, the network builds a hierarchical fusion of features across both modalities. In ref. [35], the authors utilise 2D datasets to construct 3D facial data using a method that incorporates physical depth, which differs from our approach. Our objective is not to create a 3D representation of plants but to generate an alternate representation using point clouds. Distinctive aspects of our methodology include the representation of plant shapes in point clouds using pixel (x, y) coordinates and variations in colour intensity, rather than physical depth. Furthermore, we address the unique challenges presented by herbarium datasets,

as opposed to facial data, and employ specialised architectural and fusion techniques tailored to plant morphology. Additionally, works like [36] focus on hyperspectral image classification using combined 2D and 3D CNNs. These models operate on volumetric data (spatial + spectral), not on inferred point clouds, and are therefore not directly comparable to our pixel-to-point transformation pipeline.

In summary, the vast majority of recent 2D-3D fusion approaches are tailored towards 3D-centric tasks such as semantic segmentation or object detection, and they typically rely on explicit 3D input from physical sensors like LiDAR or RGB-D cameras. These methods are often designed to process entire scenes or multiple views and focus on leveraging depth or spatial priors grounded in real-world geometry. In contrast, our method introduces a novel perspective by performing 2D image classification using a synthetic 3D representation inferred directly from a single 2D image. Rather than leveraging real 3D sensor data, we transform segmented image regions into partial point clouds through a pixel-to-point conversion that encodes spatial coordinates and colour information. This object-centric and sensor-free approach enables geometric reasoning without requiring depth supervision, multi-view consistency, or full-scene understanding. Our method remains distinct in both its objectives and input data modality, bridging a novel space between 2D vision and geometric reasoning for classification tasks.

Recent work such as ClusterFormer [37] highlights the effectiveness of clustering-based mechanisms for learning generalisable visual representations across tasks. By reformulating attention as a recurrent clustering process with feature dispatching, ClusterFormer enables hierarchical reasoning at multiple levels of granularity. It operates entirely on 2D inputs, internally encoding structure through clustering rather than relying on geometric representations. In contrast, our method explicitly models 3D structure via synthetic point clouds derived from 2D images. Integrating clustering-based attention into our multimodal setup could improve the joint representation of texture and geometry. Similarly, the 3D Gaussian Splatting framework [38] introduces an expressive representation for modelling shape and density using anisotropic Gaussian primitives. Unlike our sparse point-based approach designed for classification, Gaussian Splatting focuses on dense 3D reconstruction for photo-realistic rendering. It typically requires multi-view inputs or depth supervision, while our method is single-view and sensor-free. Nevertheless, adopting Gaussian primitives in our pipeline could enhance shape expressivity and improve multimodal fusion.

## 3 | Approach

Our goal is to improve image classification by fusing 2D images and 3D point cloud data in a multimodal architecture to identify the characteristics of objects of the same nature, such as herbarium images. The core of our approach involves transforming 2D images into 3D point clouds that represent the targeted objects. To achieve this, we first use segmentation techniques to extract masks, which we transform into 3D object shape point clouds.
The original 2D images and their corresponding 3D object shape point clouds form pairs of (image, object shape point cloud) that are integrated into a multimodal fusion architecture (SIM-Net) comprising two encoders: an image encoder (ResNet) and a point cloud encoder (PointNet). The 3D shape representations provide crucial geometric and spatial information not readily discernible in 2D images, thereby enriching the feature space accessible to the learning model. For herbarium images, segmentation enables the extraction of the plant from the herbarium sheets while removing background elements.

## 3.1 | Image to Point Cloud Data Transformation

The transformation pipeline for converting herbarium sheet images into 3D point clouds of 1024 points, as illustrated in Figure 3, begins with a segmentation phase. The original herbarium image is processed using the U-Net [39] network, selected for its proven ability to extract detailed features from complex backgrounds. The U-Net training dataset was prepared using a colour segmentation approach, detailed in four primary stages as described in Ariouat et al. [40], and employed YOLOv7-ag [17] for detecting nonplant elements. After reducing noise by detecting and removing nonplant elements, generating masks via colour segmentation, and applying morphological refinements, the final masks are used to extract plant pixels from the original images. This systematic process has resulted in a curated dataset of 2225 images and their corresponding masks, which was used to train U-Net.

The U-Net model segments each herbarium image into a binary mask, precisely isolating the plant material from the background. This mask is then projected onto the original image to selectively extract plant pixels while preserving the morphology of the specimen. Nonplant pixels are set to black by assigning their red, green and blue (R, G, B) colour values to (R, G, B) = (0, 0, 0).

An image $I \in \mathbb{R}^{H \times W \times C}$ is a tensor, where two dimensions (the height $H$ and the width $W$) represent the spatial coordinates $(x, y)$ of a pixel, and the third dimension (the number of colour channels $C$) represents its colour $(R, G, B)$. Each pixel (tensor element) $I(x, y) = (R_{xy}, G_{xy}, B_{xy})$ is a triplet of colour intensities $(R_{xy}, G_{xy}, B_{xy})$, typically ranging from 0 to 255. For each pixel $I(x, y)$ with $(R_{xy}, G_{xy}, B_{xy}) \neq (0, 0, 0)$, the transformation to a 3D point starts by a mapping function that maps the pixel to a new point $p(x_p, y_p)$, where $x_p = x$ and $y_p = y$, in a two-dimensional space, using the spatial coordinates.

This transformation generates a large point cloud $P$, where $|P| < H \times W$, in a two-dimensional space from the image pixels, using the spatial coordinates of each eligible pixel. The resulting point cloud is depicted in blue (cf., Figure 3). During the downsampling stage, the point cloud is pruned to retain only 1024 points. Given a point cloud $P = \{p_1, p_2, ..., p_n\}$, where $|P| = n$, we use a farthest point sampling (FPS) algorithm for downsampling $P$ to a smaller subset $P^m = \{p'_1, p'_2, ..., p'_m\}$, where $m = 1024$, that best represents the original set $P$ in terms of spatial distribution. Each new point is chosen based on its maximum distance from the previously selected points

$\{p'_1, p'_2, ..., p'_{j-1}\}$, ensuring that each new addition is the farthest from the existing subset with respect to the remaining points. This method of sampling distinctly contrasts with random sampling, as FPS achieves a more comprehensive coverage of the entire set of points, maintaining effectiveness even with a limited number of centroids. Furthermore, the FPS algorithm adapts to the data distribution, ensuring that the reduced subset retains the essential features of the original point cloud. This downsampling is pivotal to reduce computational complexity and to focus on significant morphological features, facilitating the deployment of deep learning models like PointNet. The final stage involves adding the third dimension $z_p$ to each point $p(x_p, y_p)$ in the point cloud. This is achieved by translating the colour information of each corresponding eligible pixel $I(x, y)$ with $(R_{xy}, G_{xy}, B_{xy}) \neq (0, 0, 0)$ into a spatial dimension $z_p$. Specifically, $z_p$ is assigned as the average of the pixel's colour values:

$$p(x_p, y_p, z_p) \text{ where } x_p = x_p, y_p = y_p \text{ and } z_p = \frac{R_{xy} + G_{xy} + B_{xy}}{3}.$$

From a conceptual standpoint, the rationale for transforming 2D image pixels into 3D point cloud representations lies in the objective of capturing the global shape structure of the target object rather than its real appearance alone. While convolutional neural networks are highly effective at extracting localised texture-based features from 2D images, they remain inherently sensitive to background clutter, lighting variations, and pose. In contrast, architectures like PointNet are specifically designed to learn from unordered point sets and are good at modelling the overall geometric configuration of an object—its shape—independently of colour or background context. In our case, the inferred point cloud is not intended to represent true physical depth, but rather to encode shape cues derived from the spatial distribution and intensity of pixels within the targeted object (the plant).

## 3.2 | Image Encoder

Image encoders play a critical role in multimodal architectures due to their ability to process and interpret complex data types. Encoders such as ResNet [5] are particularly effective for processing 2D images and extracting features such as edges and textures. In our approach, we employ the ResNet101 encoder $E^I$ (see Figure 3) within the SIM-Net architecture to extract detailed point-wise image features from 224 × 224 images. These features are encoded into an image feature vector $I^{fv}$, where $|I^{fv}| = 2048$, as illustrated in Figure 3.

## 3.3 | Point Cloud Encoder

Point clouds are sets of points in 3D space. Each point is defined by its $x, y$, and $z$ coordinates and may include additional attributes such as colour or intensity. Networks specifically designed for 3D data, such as PointNet [7], are required to process the derived object shape point clouds. In SIM-Net, we leverage the classification component of the PointNet network as a point cloud encoder $E^P$. This encoder takes 1024 points as input,
processes each point individually through input and feature transformations and fully connected layers and, aggregates point features by using max pooling to produce an order-invariant global feature vector $P^{gfv}$, where $|P^{gfv}| = 1024$:

$$P^{gfv}(p_1, p_2, \dots, p_m) = g(h(p_1), h(p_2), \dots, h(p_m)),$$

where $m = 1024$ is the number of points in $P$, $h$ is an MLP function, $g$ is a global max-pooling function.

The difference between 2D images and point clouds in terms of the features captured by neural networks like ResNet [5] and PointNet [7] is quite significant due to their inherent nature. Unlike 2D images, point clouds capture the spatial structure and shape of objects in three-dimensional space, providing information about geometry and volume. PointNet is capable of recognising patterns in the spatial arrangement of points. It can extract insights about the shape and structure of objects, aspects that 2D images alone cannot provide.

## 3.4 | Fusing RGB Features With 3D Global Features

Fusion strategies are central to the effectiveness of multimodal architectures, as they enable the integration of complementary information from different data modalities. These strategies typically combine the outputs of two specialised neural networks to achieve a unified prediction task [27–30]. In SIM-Net, the fusion function $F$ is implemented by concatenating the image feature vector $I_i^{fv} = E^l(i)$ of an image $i$, where $E^l$ is the image encoder function, and the point cloud feature vector $P_i^{fv} = MLP^P(P_i^{gfv})$ of a point cloud $P_i = T(i)$ derived from the image $i$, where MLP$^P$ is an MLP function, $P_i^{gfv} = E^P(T(i))$ is the

point cloud global feature vector, $E^P$ is the point cloud encoder function and $T(i)$ is the 2D-to-3D image transformation function.

$$F_i^{fv} = F(I_i^{fv} \oplus P_i^{fv}).$$

$F_i^{fv}$ represents the fusion feature vector with a size of 2056. This vector is fed into the MLP$^F$ MLP function for classification. MLP$^F$ is an MLP that contains three fully connected layers with ReLU activation function, batch normalisation and dropout function. Finally, a sigmoid activation function for binary classification was carried out as illustrated in the SIM-Net architecture (Figure 3). The combination of ResNet global features with PointNet features allows the model to exploit both detailed surface information and 3D spatial relationships. This fusion creates a more robust and comprehensive feature representation, enabling the model to better classify complex structures that require the integration of surface details and geometric context.

## 4 | Experiments

In this section, we present a set of experiments designed to evaluate the effectiveness of the PointNet, PointNet++ and SIM-Net models using the point clouds derived from herbarium 2D images, with ResNet serving as the baseline for comparison. The first set of experiments (cf., Tables 1 and 2) assesses the performance of PointNet [7] and PointNet++ [23] relative to ResNet101 [5] trained from scratch. PointNet++ extends the original PointNet through a hierarchical architecture that applies its core principles recursively across nested partitions of the input point set.²

We further evaluate how point quality impacts model performance by conducting tests on both a curated set of high-quality

<table><caption>TABLE 1 | Accuracy (from scratch) using unsegmented selected images and selected point cloud datasets.</caption><thead><tr><th rowspan="2">Trait</th><th colspan="5">Point cloud with 3 dimensions (x, y, z)</th><th colspan="5">Point cloud with 6 dimensions (x, y, z, r, g, b)</th></tr><tr><th>Images ResNet101</th><th>PointNet</th><th>Δ</th><th>PointNet++</th><th>Δ</th><th>PointNet</th><th>Δ</th><th>PointNet++</th><th>Δ</th></tr></thead><tbody><tr><td>Thorns</td><td>75.46</td><td>61.03</td><td>-14.43</td><td>83.05</td><td>+7.59</td><td>85.73</td><td>+10.27</td><td>92.90</td><td>+17.44</td></tr><tr><td>Fruits</td><td>54.09</td><td>54.01</td><td>-0.08</td><td>59.34</td><td>+5.25</td><td>54.71</td><td>+0.62</td><td>58.58</td><td>+4.49</td></tr><tr><td>Leaves with acuminate tips</td><td>58.82</td><td>59.72</td><td>+0.9</td><td>68.61</td><td>+9.79</td><td>64.72</td><td>+5.9</td><td>73.61</td><td>+14.79</td></tr><tr><td>Infructescence</td><td>54.87</td><td>56.11</td><td>+1.24</td><td>63.33</td><td>+8.46</td><td>57.22</td><td>+2.35</td><td>59.72</td><td>+4.85</td></tr><tr><td>Leaves with an acute base</td><td>59.86</td><td>56.52</td><td>-3.34</td><td>65.83</td><td>+5.97</td><td>67.29</td><td>+7.43</td><td>71.45</td><td>+11.59</td></tr></tbody></table>

<table><caption>TABLE 2 | Accuracy (from scratch) using unsegmented complete images and complete point cloud datasets.</caption><thead><tr><th rowspan="2">Trait</th><th colspan="5">Point cloud with 3 dimensions (x, y, z)</th><th colspan="5">Point cloud with 6 dimensions (x, y, z, r, g, b)</th></tr><tr><th>Images ResNet101</th><th>PointNet</th><th>Δ</th><th>PointNet++</th><th>Δ</th><th>PointNet</th><th>Δ</th><th>PointNet++</th><th>Δ</th></tr></thead><tbody><tr><td>Thorns</td><td>72.00</td><td>59.08</td><td>-12.92</td><td>86.23</td><td>+14.23</td><td>84.36</td><td>+12.36</td><td>93.76</td><td>+21.76</td></tr><tr><td>Fruits</td><td>53.62</td><td>53.50</td><td>-0.12</td><td>56.50</td><td>+2.88</td><td>58.33</td><td>+4.71</td><td>59.50</td><td>+5.88</td></tr><tr><td>Leaves with acuminate tips</td><td>63.00</td><td>57.89</td><td>-5.11</td><td>75.11</td><td>+12.11</td><td>68.21</td><td>+5.21</td><td>73.96</td><td>+10.96</td></tr><tr><td>Infructescence</td><td>55.63</td><td>52.04</td><td>-3.59</td><td>62.03</td><td>+6.4</td><td>56.25</td><td>+0.62</td><td>68.87</td><td>+13.24</td></tr><tr><td>Leaves with an acute base</td><td>59.20</td><td>55.20</td><td>-4.00</td><td>66.34</td><td>+7.14</td><td>61.30</td><td>+2.10</td><td>68.49</td><td>+9.29</td></tr></tbody></table>
point clouds and a broader dataset that may include lower-quality point clouds. The quality of the point clouds depends directly on the image segmentation. Additionally, we explore the benefits of enriching point clouds by appending the three RGB colour values to each point, thereby increasing the dimensionality of the point cloud representation from three to six.

We then present a series of experiments evaluating the performance of point clouds within the SIM-Net multimodal architecture, which uses ResNet101 as the image encoder and PointNet as the point cloud encoder (cf., Tables 3 and 4). We benchmark SIM-Net against ResNet101 pretrained on the ImageNet-1k dataset. In this second series of experiments, we examine how segmentation quality—and consequently, point cloud quality—influences SIM-Net performance. We compare model results across datasets containing various plant traits, using both well-segmented samples and datasets that may include lower-quality segmentations.

## 4.1 | Experiments Settings

In this study, we evaluated four distinct models: ResNet101, PointNet, PointNet++, and SIM-Net. The experiments were conducted using four Nvidia A100 GPUs. For the ResNet101 and SIM-Net models, we employed a range of batch sizes [8, 16, 32, 64, 128] across all training sessions. PointNet and PointNet++ were trained using a fixed batch size of 24. All models were trained for 100 epochs, employing the Adam optimiser with an initial learning rate of 0.001 and a weight decay set at 1e-4. We applied a StepLR scheduler to systematically reduce the learning rate by 30% every 20 epochs. The cross-entropy loss function was used for network optimisation.

To mitigate overfitting in the PointNet and PointNet++ models, we applied several data augmentation techniques, including random rotations around the z-axis and the addition of random noise to the point cloud data. For the ResNet model, we used three augmentation strategies, namely random horizontal flip, random vertical flip, and random rotation set at 90°. These techniques were chosen to increase variability and complexity within the training datasets.

Our evaluation metrics were focused on accuracy and F1-score to ensure a thorough assessment of each model's performance. All implementations were carried out using the Pytorch framework.

## 4.2 | Datasets

In our study, we used an initial set of 4005 images annotated for the presence or absence of five distinct plant traits: thorns (sharp structures on branches or stems), presence of fruits, leaves with acuminate tips, infructescence (arrangement of elementary fruits, like grapes), and leaves with an acute base. These images were processed through a segmentation pipeline, resulting in a second dataset of 4005 segmented images. This segmented dataset was subsequently used for 2D-to-3D transformation, producing a corresponding dataset of 4005 point clouds. However, we faced two major challenges: significant class imbalance in trait annotations and inconsistent segmentation quality across some images. To address class imbalance, we created a *Complete Datasets* series, consisting of two balanced datasets per trait: one containing segmented images and one containing unsegmented images. For each image dataset, we also generated a corresponding point cloud dataset. To study the impact of segmentation quality, we selected well-

<table><caption>TABLE 3 | Evaluation metrics on unsegmented images with point cloud datasets.</caption><thead><tr><th rowspan="3">Trait</th><th colspan="6">Selected image point cloud (SIPD)</th><th colspan="6">Complete image point cloud (CIPD)</th></tr><tr><th colspan="2">SIM-Net</th><th colspan="2">ResNet101</th><th colspan="2">Δ</th><th colspan="2">ResNet101</th><th colspan="2">SIM-Net</th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>95.75</td><td>95.74</td><td>95.53</td><td>94.02</td><td>+0.22</td><td>+1.72</td><td>95.00</td><td>95.01</td><td>95.20</td><td>93.47</td><td>-0.20</td><td>+1.54</td></tr><tr><td>Fruits</td><td>66.56</td><td>69.65</td><td>61.58</td><td>66.08</td><td>+4.98</td><td>+3.57</td><td>64.34</td><td>69.52</td><td>64.66</td><td>67.22</td><td>-0.32</td><td>+2.3</td></tr><tr><td>Leaves with acuminate tips</td><td>79.84</td><td>79.19</td><td>77.94</td><td>73.02</td><td>+1.9</td><td>+6.17</td><td>81.40</td><td>81.44</td><td>77.00</td><td>73.92</td><td>+4.40</td><td>+7.52</td></tr><tr><td>Infructescence</td><td>70.19</td><td>71.35</td><td>60.29</td><td>66.67</td><td>+9.9</td><td>+4.68</td><td>73.05</td><td>74.24</td><td>69.37</td><td>69.10</td><td>+3.68</td><td>+5.14</td></tr><tr><td>Leaves with an acute base</td><td>74.67</td><td>75.84</td><td>70.71</td><td>71.61</td><td>+3.96</td><td>+4.23</td><td>71.80</td><td>73.46</td><td>71.40</td><td>72.66</td><td>+0.40</td><td>+0.80</td></tr></tbody></table>

<table><caption>TABLE 4 | Evaluation metrics on segmented images with point cloud datasets.</caption><thead><tr><th rowspan="3">Trait</th><th colspan="6">Selected segmented image point cloud (SSIPD)</th><th colspan="6">Complete segmented image point cloud (CSIPD)</th></tr><tr><th colspan="2">SIM-Net</th><th colspan="2">ResNet101</th><th colspan="2">Δ</th><th colspan="2">SIM-Net</th><th colspan="2">ResNet101</th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>92.61</td><td>92.55</td><td>93.4</td><td>91.17</td><td>-0.79</td><td>+1.38</td><td>95.9</td><td>95.79</td><td>94.2</td><td>92.86</td><td>+1.7</td><td>+2.93</td></tr><tr><td>Fruits</td><td>64.88</td><td>67.86</td><td>60.95</td><td>62.33</td><td>+3.93</td><td>+5.53</td><td>64.84</td><td>67.03</td><td>60.17</td><td>67.28</td><td>+4.67</td><td>-0.25</td></tr><tr><td>Leaves with acuminate tips</td><td>81.82</td><td>81.4</td><td>70.59</td><td>70.62</td><td>+11.23</td><td>+10.78</td><td>82.27</td><td>82.33</td><td>79.4</td><td>78.88</td><td>+2.87</td><td>+3.45</td></tr><tr><td>Infructescence</td><td>66.35</td><td>69.51</td><td>63.72</td><td>57.23</td><td>+2.63</td><td>+12.28</td><td>70.2</td><td>72.32</td><td>70.83</td><td>72.87</td><td>-0.63</td><td>-0.55</td></tr><tr><td>Leaves with an acute base</td><td>73.87</td><td>74.92</td><td>71.68</td><td>74.43</td><td>+2.19</td><td>+0.49</td><td>71.84</td><td>73.56</td><td>70.0</td><td>69.42</td><td>-1.16</td><td>+4.14</td></tr></tbody></table>
segmented images from the Complete Datasets and generated their corresponding point clouds, forming the *Selected Datasets* series. Figure 4 provides a visual comparison of image segmentation quality and its impact on point cloud generation. The top row shows an example of high-quality segmentation resulting in a well-formed point cloud, which represents the standard we aim to maintain in our Selected Datasets series.

Table 5 presents the size of training and validation sets of the Complete and Selected datasets series. In contrast, the bottom row illustrates an example of lower-quality segmentation, which leads to a correspondingly lower-quality point cloud transformation. Such cases are representative of the broader range of image qualities found in the Complete Datasets collection.

We created a collection of 40 unimodal datasets, each comprising image data and point cloud data. These datasets are derived from two distinct series: Complete and Selected. For each of the five traits studied, we generated eight distinct datasets—four from the Complete series and four from the Selected series—consisting of unsegmented images, segmented images, point clouds generated from unsegmented images, and point clouds generated from segmented images. Each series thus contributes 20 unique datasets, yielding a total of 40 unimodal datasets. In addition, we generated 20 multimodal datasets, resulting in an additional 20 datasets. The composition and naming conventions for each multimodal dataset across the five traits are described below:

1. *Complete Image Point cloud Dataset (CIPD)*: Combines the full set of original unsegmented images with the point clouds generated from their corresponding segmented images.

2. *Complete Segmented Image Point cloud Dataset (CSIPD)*: Pairs the entire collection of segmented images with the point clouds derived from the same segmented images.

3. *Selected Segmented Image Point cloud Dataset (SSIPD)*: Merges the selected segmented images with the point clouds generated from these accurately segmented images.

4. *Selected Image Point cloud Dataset (SIPD)*: Consists of the selected set of original unsegmented images that correspond to the well-segmented images, along with their associated point clouds from the well-segmented counterparts.

Additionally, we generated two enriched point cloud datasets for each trait—one within the selected series and one within the complete series. These datasets consist of six-dimensional point clouds in which the RGB colour values of each corresponding pixel are appended to the original (x, y, z) coordinates of each point. This integration transforms each pixel into one point with (x, y, z, r, g, b) coordinates. We use these enriched datasets to investigate the effects of augmenting the original point clouds with additional information. Altogether, this expansion yields a total of 70 distinct datasets used in our experiments, providing a rich foundation for comprehensive analysis.

## 4.3 | Discussion

The experimental results are presented in Tables 1–4 and A6. Tables 1 and 2 report the accuracy achieved by each model (ResNet101, PointNet, and PointNet++), with the delta (Δ) columns indicating the accuracy difference between the point cloud models (PointNet and PointNet++) and ResNet101.

<table><caption>TABLE 5 | Datasets distribution for complete and selected datasets.</caption><thead><tr><th rowspan="2">Trait</th><th colspan="2">Complete dataset</th><th colspan="2">Selected dataset</th></tr><tr><th>Train</th><th>Val</th><th>Train</th><th>Val</th></tr></thead><tbody><tr><td>Thorns</td><td>2426</td><td>500</td><td>1786</td><td>380</td></tr><tr><td>Fruits</td><td>2776</td><td>580</td><td>1812</td><td>380</td></tr><tr><td>Leaves with acuminate tips</td><td>2360</td><td>500</td><td>1648</td><td>340</td></tr><tr><td>Infructescence</td><td>2340</td><td>480</td><td>1570</td><td>340</td></tr><tr><td>Leaves with an acute base</td><td>2308</td><td>500</td><td>1360</td><td>280</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>FIGURE 4 | Transformation examples showing accurate point cloud generation (top) and less precise point cloud generation (bottom).</figcaption></figure>
Tables 3 and 4, in contrast, present the highest accuracy (Acc) and F1-score (F1) values achieved by both SIM-Net and ResNet101. It is important to note that the Acc and F1-score values listed for a given model on the same row may originate from different training runs, as we selected the highest accuracy, and the highest F1-score obtained across three training iterations for each model on each dataset. The delta columns in these tables indicate the performance gain or loss of SIM-Net relative to ResNet101.

An initial observation from Tables 1 and 2 concerns the accuracy of ResNet101, PointNet, and PointNet++ models trained from scratch. We observe that PointNet consistently underperforms relative to ResNet101 across both the Selected and Complete dataset series. However, PointNet++ achieves up to a 14% performance improvement over ResNet101 for the Thorns trait (see Table 2). This suggests that with large 3D point cloud datasets derived from 2D images—comparable in scale to ImageNet-1K—models such as PointNet and PointNet++ can surpass traditional models like ResNet.

A key finding from this first series of experiments is the performance boost obtained by enriching point clouds with additional information—specifically, by appending RGB colour values to the (x, y, z) coordinates. This enhancement results in a 17% performance gain for PointNet++ over ResNet101 for the Thorns trait in the Selected datasets (Table 1), and an even larger 21% improvement in the Complete datasets (Table 2). These results highlight the potential of embedding additional image-based information into the point cloud representation.

In the second series of experiments (Tables 3 and 4), we observe that the multimodal architecture (SIM-Net) consistently achieves higher accuracy in 80% of the test cases (16 out of 20) and higher F1-score in 90% of the cases (18 out of 20). The first major insight is that SIM-Net outperforms ResNet101 pretrained on ImageNet-1K in nearly all scenarios. This holds true for both segmented and unsegmented images, across both the Selected and Complete datasets, with only 5 exceptions out of 40 cases. It is important to note that ResNet101 serves as a unimodal baseline that relies solely on 2D image data, whereas SIM-Net integrates both 2D images and their corresponding inferred 3D point clouds. This comparison demonstrates the added value of incorporating 3D geometric features into the classification process.

A second key observation is that SIM-Net performs better on the selected datasets series than on the Complete series, particularly for unsegmented images (see Table 3). This improvement can be attributed to the superior quality of the point clouds in the selected datasets compared to those in the complete series. This trend remains consistent even for segmented images (see Table 4), reinforcing the conclusion that point cloud quality—driven by segmentation accuracy—plays a critical role in enhancing the model's performance.

From a theoretical perspective, the performance improvement observed with the 2D-to-3D transformation can be attributed to the complementarity between 2D visual features and 3D geometric representations. Traditional 2D image architectures primarily capture local texture, colour, and edge-based features

within a fixed receptive field. These features are highly sensitive to background context, lighting conditions, and specimen placement, which can vary significantly across images. In contrast, transforming segmented 2D images into 3D point clouds produces a spatial representation of object shape that abstracts away background variation while preserving morphological consistency. This structural representation encodes topological and geometric relationships between object parts, which are invariant to appearance-level noise. In a multimodal setting, the 2D and 3D branches provide orthogonal yet complementary feature spaces: the image encoder captures discriminative texture-based cues, while the point cloud encoder encodes global and local shape priors. This fusion enhances the model's capacity to distinguish fine-grained inter-class differences and generalise across varying background conditions—especially when classification relies on subtle morphological traits.

Despite SIM-Net's promising results, several challenges present opportunities for improvement. Segmentation quality remains crucial, as errors in segmentation directly affect point cloud generation and, consequently, classification accuracy. Enhancing segmentation models through self-supervised learning or attention mechanisms could improve robustness. Additionally, multimodal fusion increases computational complexity; optimising architectures with lightweight feature extractors or more efficient fusion strategies could help reduce resource demands. Another open challenge is generalisation beyond herbarium datasets. Domain adaptation techniques may be required to enhance performance across diverse object categories. Scalability is another concern, as the 2D-to-3D transformation process introduces computational overhead. Applying efficient downsampling techniques or hybrid processing strategies could enhance efficiency for large-scale datasets. Furthermore, while point clouds encode spatial structure, they lack true depth information. Exploring monocular depth estimation or benchmarking SIM-Net against alternative multimodal fusion techniques could yield valuable insights. In summary, improvements in segmentation, computational efficiency, generalisation, and depth representation could further enhance SIM-Net's scalability and applicability, making it a more versatile classification solution.

## 5 | Conclusion

The foundation of our approach lies in transforming the same dataset into multiple representations to leverage the strengths of different architectures for feature extraction. In this study, we convert 2D herbarium images into point clouds, enabling the use of a multimodal architecture that combines a convolutional neural network for detailed pixel-level analysis with a point cloud-based neural network for interpreting geometric structures. This methodology significantly improves the classification performance of herbarium images, particularly in complex and nuanced scenarios where traditional 2D methods might struggle.

A crucial aspect of our approach is its dependence on the quality of both the segmentation and the resulting point clouds. Higher
segmentation quality consistently leads to improved perfor-
mance. This relationship demonstrates the importance of pre-
cise and accurate initial data processing within our pipeline.

Looking ahead, several avenues could further enhance the
performance of our model. One promising direction is the
integration of advanced point cloud encoders such as Point-
NeXt, PointMLP, and Point Transformer. These models are
particularly effective in processing point cloud data. Addition-
ally, optimising the data transformation pipeline presents a
significant opportunity for improvement. By incorporating
additional knowledge into the point clouds, we can enrich the
data and potentially achieve more nuanced and accurate clas-
sification results. This could involve advanced techniques in
data augmentation techniques or the integration of supple-
mentary features to help the model better capture complex
patterns.

From a theoretical perspective, it would be intriguing to un-
derstand why the addition of 3D modality enables the creation
of more efficient representations, given that the initial infor-
mation is entirely 2D and no new external information is
added-apart from the explicit inclusion of pixel position co-
ordinates. This raises interesting questions about how the 2D-to-
3D transformation enhances the informational richness of the
representation.

Author Contributions

**Youcef Sklab:** conceptualization, data curation, formal analysis, fund-
ing acquisition, methodology, project administration, resources, soft-
ware, supervision, validation, visualization, writing – original draft,
writing – review and editing. **Hanane Ariouat:** data curation, re-
sources, software, writing – original draft. **Eric Chenin:** data curation,
funding acquisition, project administration. **Edi Prifti:** funding acqui-
sition, resources, supervision, writing – review and editing. **Jean-
Daniel Zucker:** conceptualization, funding acquisition, supervision,
writing – review and editing.

Acknowledgements

This work was part of tasks within the e-Col+ project (ANR-21-ESRE-
0053). This project was provided with computing HPC and storage re-
sources by GENCI at IDRIS thanks to the Grant 2023-A0150114385 on
the supercomputer Jean Zay's A100 partition.

Conflicts of Interest

The authors declare no conflicts of interest.

Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon request and will be released publicly upon
acceptance. The source code used in this study is available from the
corresponding author upon request and will be released publicly upon
acceptance.

Endnotes

References

1. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” *Proceedings of the IEEE* 86, no. 11 (1998): 2278–2324, https://doi.org/10.1109/5.726791.

2. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classifi-
cation With Deep Convolutional Neural Networks,” *Communications of
the ACM* 60, no. 6 (2017): 84–90, https://doi.org/10.1145/3065386.

3. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv Preprint arXiv:1409.1556 (2015), https://arxiv.org/abs/1409.1556.

4. C. Szegedy, W. Liu, Y. Jia, et al., “Going Deeper With Convolutions,” in *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)* (IEEE Computer Society, 2015), 1–9, https://doi.ieeecomput ersociety.org/10.1109/CVPR.2015.7298594.

5. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)* (2016), 770–778, https://doi.org/10.1109/CVPR.2016.90.

6. A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention Is All You Need,” arXiv Preprint arXiv:1706.03762 (2023), https://arxiv.org/abs/1706.03762.

7. R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)* (2017), 77–85, https://doi.org/10.1109/CVPR.2017.16.

8. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-
Training of Deep Bidirectional Transformers for Language Under-
standing,” arXiv Preprint arXiv:1810.04805 (2019), https://arxiv.org/abs/
1810.04805.

9. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning Transferable Visual Models From Natural Language Supervision,” arXiv Preprint arXiv:2103.00020 (2021), https://arxiv.org/abs/2103.00020.

10. A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., “An Image Is Worth 16 × 16 Words: Transformers for Image Recognition at Scale,” arXiv Preprint arXiv:2010.11929 (2021), https://arxiv.org/abs/2010.11929.

11. T. Lorieul, K. D. Pearson, E. R. Ellwood, et al., “Toward a Large-Scale and Deep Phenological Stage Annotation of Herbarium Specimens: Case Studies From Temperate, Tropical, and Equatorial Floras,” *Applications in Plant Sciences* 7, no. 3 (2019): e1233, https://doi.org/10.1002/aps3.1233.

12. S. Younis, M. Schmidt, C. Weiland, S. Dressler, B. Seeger, and T. Hickler, “Detection and Annotation of Plant Organs From Digitised Herbarium Scans Using Deep Learning,” *Biodiversity Data Journal* 8 (2020): e57090, https://doi.org/10.3897/BDJ.8.e57090.

13. A. Abdelaziz, B. Bassem, and W. Walid, “A Deep Learning-Based Approach for Detecting Plant Organs From Digitized Herbarium Specimen Images,” *Ecological Informatics* 69 (2022): 101590, https://doi.org/10.1016/j.ecoinf.2022.101590.

14. Y. Sklab, H. Ariouat, Y. Boujydah, et al., “Towards a Deep Learning-
Powered Herbarium Image Analysis Platform,” *Biodiversity Information
Science and Standards* 8 (2024): e135629, https://doi.org/10.3897/biss.8.
135629.

15. Y. Sklab, H. Ariouat, E. Prifti, J.-D. Zucker, and E. Chenin, “Iden-
tification of Non-Plant Elements in Herbarium Images Using YOLO,” in
*17th African Conference on Research in Computer Science and Applied

<footnote n="1">¹ Natural history collections can preserve several hundred years of in-
formation on the evolution of biodiversity and the environment. In-
terest in these collections is on the rise, as they offer valuable insights
for understanding and alleviating severe threats to biodiversity.</footnote>

<footnote n="2">² PointNet falls short in recognizing local structures within the metric space in which the points reside, which restricts its capacity to discern intricate patterns and adapt to multifaceted scenes. By leveraging metric space distances, PointNet++ is adept at capturing and learning local features at progressively larger contextual scales.</footnote>

<footnote n="3">³ http://host.robots.ox.ac.uk/pascal/VOC/.</footnote>
Mathematics (CARI, 2025), https://doi.org/10.1007/978-3-031-88226-5_10.

16. H. Ariouat, Y. Sklab, E. Prifti, J.-D. Zucker, and E. Chenin, "Enhancing Plant Morphological Trait Identification in Herbarium Collections Through Deep Learning-Based Segmentation," *Applications in Plant Sciences* 13, no. 2 (2025): e70000, https://doi.org/10.1002/aps3.70000.

17. H. Ariouat, Y. Sklab, M. Pignal, et al., "Enhancing YOLOv7 for Plant Organs Detection Using Attention-Gate Mechanism," in *Pacific-Asia Conference on Knowledge Discovery and Data Mining* (Curran Associates, 2024), https://doi.org/10.1007/978-981-97-2253-2_18.

18. M. Moayeri, P. Pope, Y. Balaji, and S. Feizi, "A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022)* (IEEE, 2022), 19065-19075, https://doi.org/10.1109/CVPR52688.2022.01850.

19. J. T. Leksut, J. Zhao, and L. Itti, "Learning Visual Variation for Object Recognition," *Image and Vision Computing* 98 (2020): 103912, https://doi.org/10.1016/j.imavis.2020.103912.

20. M. Sahraoui, Y. Sklab, M. Pignal, R. Vignes Lebbe, and V. Guigue, "Leveraging Multimodality for Biodiversity Data: Exploring Joint Representations of Species Descriptions and Specimen Images Using CLIP," *Biodiversity Information Science and Standards* 7 (2023): e112666, https://doi.org/10.3897/biss.7.112666.

21. J. Zhao, Y. Wang, Y. Cao, et al., "The Fusion Strategy of 2D and 3D Information Based on Deep Learning: A Review," *Remote Sensing* 13, no. 20 (2021): 4029, https://doi.org/10.3390/rs13204029.

22. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely Connected Convolutional Networks," in *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017)* (IEEE, 2017), 4700-4708, https://doi.org/10.1109/CVPR.2017.243.

23. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space," in *Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS 2017)* (2017), 5105-5114.

24. N. Engel, V. Belagiannis, and K. Dietmayer, "Point Transformer," *IEEE Access* 9 (2021): 134826-134840, https://doi.org/10.1109/ACCESS.2021.3116304.

25. X. Ma, C. Qin, H. You, H. Ran, and Y. Fu, "Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework," in *International Conference on Learning Representations (ICLR 2022)* (2022), https://openreview.net/forum?id=3Pbra_u76D.

26. G. Qian, Y. Li, H. Peng, et al., "PointNeXt: Revisiting PointNet++ With Improved Training and Scaling Strategies," in *Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 2022)* (2022).

27. R. Guo, D. Li, and Y. Han, "Deep Multi-Scale and Multi-Modal Fusion for 3D Object Detection," *Pattern Recognition Letters* 151 (2021): 236-242, https://doi.org/10.1016/j.patrec.2021.08.028.

28. L. Gu, X. Yan, P. Cui, et al., "PointSee: Image Enhances Point Cloud," *IEEE Transactions on Visualization and Computer Graphics* 30, no. 6 (2024): 6291-6308, https://doi.org/10.1109/TVCG.2023.3331779.

29. V. Poliyapram, W. Wang, and R. Nakamura, "A Point-Wise LiDAR and Image Multimodal Fusion Network (PMNet) for Aerial Point Cloud 3D Semantic Segmentation," *Remote Sensing* 11, no. 24 (2019): 2961, https://doi.org/10.3390/rs11242961.

30. O. Pradelle, R. Chaine, D. Wendland, and J. Digne, "Lightweight Integration of 3D Features to Improve 2D Image Segmentation," *Computers & Graphics* 114 (2023): 326-336, https://doi.org/10.1016/j.cag.2023.06.004.

31. C.-K. Yang, M.-H. Chen, Y.-Y. Chuang, and Y.-Y. Lin, "2D-3D Interlaced Transformer for Point Cloud Segmentation With Scene-Level

Supervision," in *Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV)* (IEEE, 2023), 977-987, https://doi.org/10.1109/ICCV51070.2023.00096.

32. L. Zhao, J. Lu, and J. Zhou, "Similarity-Aware Fusion Network for 3D Semantic Segmentation," in *Proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)* (IEEE, 2021), 1585-1592, https://doi.org/10.1109/IROS51168.2021.9636494.

33. E. Palladin, R. Dietze, P. Narayanan, M. Bijelic, and F. Heide, "SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weather," in *European Conference on Computer Vision* (Springer, 2024), 484-503, https://doi.org/10.1007/978-3-031-73030-6_27.

34. Y. Chen, B. Yang, M. Liang, and R. Urtasun, "Learning Joint 2D-3D Representations for Depth Completion," in *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)* (IEEE, 2019), 10022-10031, https://doi.org/10.1109/ICCV.2019.01012.

35. T. S. Ly, N.-T. Do, S.-H. Kim, H.-J. Yang, and G.-S. Lee, "A Novel 2D and 3D Multimodal Approach for In-the-Wild Facial Expression Recognition," *Image and Vision Computing* 92 (2019): 103817, https://doi.org/10.1016/j.imavis.2019.10.003.

36. Z. Ge, G. Cao, X. Li, and P. Fu, "Hyperspectral Image Classification Method Based on 2D-3D CNN and Multibranch Feature Fusion," *Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing* 13 (2020): 5776-5788, https://doi.org/10.1109/JSTARS.2020.3024841.

37. J. C. Liang, Y. Cui, Q. Wang, T. Geng, W. Wang, and D. Liu, "ClusterFormer: Clustering as a Universal Visual Learner," in *Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS)* (2023), https://arxiv.org/abs/2305.13095.

38. G. Chen and W. Wang, "A Survey on 3D Gaussian Splatting," *arXiv Preprint arXiv:2401.03890* (2025), https://doi.org/10.48550/arXiv.2401.03890.

39. O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," *arXiv Preprint arXiv:1505.04597* (2015), https://doi.org/10.48550/arXiv.1505.04597.

40. H. Ariouat, Y. Sklab, M. Pignal, et al., "Extracting Masks From Herbarium Specimen Images Based on Object Detection and Image Segmentation Techniques," *Biodiversity Information Science and Standards* 7 (2023): e112161, https://doi.org/10.3897/biss.7.112161.

41. H. Bao, L. Dong, S. Piao, and F. Wei, "BEiT: BERT Pre-Training of Image Transformers," *arXiv Preprint arXiv:2106.08254* (2022), https://doi.org/10.48550/arXiv.2106.08254.

42. Z. Dai, H. Liu, Q. V. Le, and M. Tan, "CoAtNet: Marrying Convolution and Attention for All Data Sizes," *arXiv Preprint arXiv:2106.04803* (2021), https://doi.org/10.48550/arXiv.2106.04803.

43. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A ConvNet for the 2020s," *arXiv Preprint arXiv:2201.03545* (2022), https://doi.org/10.48550/arXiv.2201.03545.

44. S. Woo, S. Debnath, R. Hu, et al., "ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders," *arXiv Preprint arXiv:2301.00808* (2023), https://doi.org/10.48550/arXiv.2301.00808.

45. S. d'Ascoli, H. Touvron, M. L. Leavitt, A. S. Morcos, G. Biroli, and L. Sagun, "ConViT: Improving Vision Transformers With Soft Convolutional Inductive Biases," *Journal of Statistical Mechanics: Theory and Experiment* 2022, no. 11 (2022): 114005, https://doi.org/10.1088/1742-5468/ac9830.

46. Z. Liu, Y. Lin, Y. Cao, et al., "Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows," *arXiv Preprint arXiv:2103.14030* (2021), https://doi.org/10.48550/arXiv.2103.14030.

47. M. Ding, B. Xiao, N. Codella, P. Luo, J. Wang, and L. Yuan, "DaViT: Dual Attention Vision Transformers," *arXiv Preprint arXiv:2204.03645* (2022), https://doi.org/10.48550/arXiv.2204.03645.
# Appendix 1

In this appendix, we provide additional information regarding four additional tests we conducted to evaluate the effectiveness of incorporating x, y, z coordinates into point clouds in our approach.

## Extensive Evaluation With State-of-the-Art Models

Table A1 presents a comparative evaluation of eight state-of-the-art deep learning models (BEiT [41], CoAtNet [42], ConvNeXt [43], ConvNeXt2 [44], ConViT [45] ViT [6], Swin [46] and DaViT [47]), assessed in two experimental settings: (1) single-modality classification using only 2D images and (2) multimodal fusion, where each model was integrated into our architecture (SIM-Net) as an image encoder in place of the originally used ResNet. The evaluation covers the same five botanical traits: Leaves with acuminate tips, Leaves with an acute base,

Infructescence, Fruits, and Thorns. For each model and each setting, we conducted three independent runs from scratch, and the values presented in the table correspond to the best accuracy obtained across these runs. We run the experiments on an Nvidia A100 card, with a batch size set to 128. Each model underwent training for 100 epochs, employing the Adam optimiser with an initial learning rate of 0.001 and a weight decay set at 1e-4. A StepLR scheduler was applied, systematically reducing the learning rate by 30% every 20 epochs. The cross-entropy loss function was used for network optimisation. The Delta column in the table indicates the performance gain (or loss) achieved by integrating the models into our multimodal fusion network (SIM-Net) compared to its performance in a single-modality setting.

Our analysis demonstrates that SIM-Net consistently improves classification accuracy across a wide range of visual traits, particularly for Leaves with acuminate tips, Leaves with an acute base, and

<table><caption>**TABLE A1** | Comparative performance of eight state-of-the-art deep learning models in single-modality (2D images) and multimodal (2D + 3D fusion) settings for botanical trait classification.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Leaves with acuminate tips</th><th colspan="3">Leaves with an acute base</th><th colspan="3">Infructescence</th><th colspan="3">Fruits</th><th colspan="3">Thorns</th></tr><tr><th>Single</th><th>SIM-Net</th><th>Δ</th><th>Single</th><th>SIM-Net</th><th>Δ</th><th>Single</th><th>SIM-Net</th><th>Δ</th><th>Single</th><th>SIM-Net</th><th>Δ</th><th>Single</th><th>SIM-Net</th><th>Δ</th></tr></thead><tbody><tr><td>BEiT</td><td>63.59</td><td>64.12</td><td>+00.53</td><td>61.29</td><td>62.37</td><td>+01.08</td><td>54.57</td><td>55.75</td><td>+01.18</td><td>65.62</td><td>57.52</td><td>-08.10</td><td>73.57</td><td>73.46</td><td>-00.11</td></tr><tr><td>CoAtNet</td><td>50.59</td><td>58.53</td><td>+07.94</td><td>50.54</td><td>59.50</td><td>+08.96</td><td>50.15</td><td>54.18</td><td>+04.03</td><td>50.92</td><td>57.78</td><td>+06.86</td><td>59.89</td><td>75.20</td><td>+15.31</td></tr><tr><td>ConvNeXt</td><td>57.65</td><td>61.76</td><td>+04.11</td><td>50.54</td><td>62.37</td><td>+11.83</td><td>53.10</td><td>54.28</td><td>+01.18</td><td>70.71</td><td>57.78</td><td>-12.93</td><td>67.01</td><td>73.09</td><td>+06.08</td></tr><tr><td>ConvNeXt2</td><td>52.23</td><td>59.41</td><td>+07.18</td><td>50.18</td><td>60.93</td><td>+10.75</td><td>51.34</td><td>54.28</td><td>+02.94</td><td>51.72</td><td>56.73</td><td>+05.01</td><td>56.99</td><td>68.60</td><td>+11.61</td></tr><tr><td>ConViT</td><td>55.59</td><td>59.71</td><td>+04.12</td><td>58.78</td><td>60.02</td><td>+01.24</td><td>53.98</td><td>54.87</td><td>+00.89</td><td>67.81</td><td>57.52</td><td>-10.29</td><td>69.39</td><td>68.34</td><td>-01.05</td></tr><tr><td>ViT</td><td>59.71</td><td>61.18</td><td>+01.47</td><td>60.44</td><td>61.29</td><td>+00.85</td><td>55.16</td><td>56.64</td><td>+01.48</td><td>70.46</td><td>58.05</td><td>-12.41</td><td>74.41</td><td>65.80</td><td>-08.61</td></tr><tr><td>Swin</td><td>53.24</td><td>60.00</td><td>+06.76</td><td>50.18</td><td>63.80</td><td>+13.62</td><td>52.51</td><td>54.87</td><td>+02.36</td><td>52.31</td><td>57.52</td><td>+05.21</td><td>68.07</td><td>61.48</td><td>-06.59</td></tr><tr><td>DaViT</td><td>51.42</td><td>56.76</td><td>+05.34</td><td>50.54</td><td>60.93</td><td>+10.39</td><td>51.15</td><td>53.69</td><td>+02.54</td><td>51.13</td><td>56.20</td><td>+05.07</td><td>58.35</td><td>72.30</td><td>+13.95</td></tr></tbody></table>

<table><caption>**TABLE A2** | Comparison of SIM-Net and cross-attention variants on the selected image point cloud dataset (SIPD).</caption><thead><tr><th rowspan="3">Trait</th><th colspan="2">SIM-Net</th><th colspan="4">Point cloud as query—Image as key/ value</th><th colspan="4">Image as query—Point cloud as key/ value</th></tr><tr><th rowspan="2">Acc</th><th rowspan="2">F1</th><th colspan="2">SIM-Net<sub>CA</sub></th><th colspan="2">Δ</th><th colspan="2">SIM-Net<sub>CA</sub></th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>92.61</td><td>92.55</td><td>94.46</td><td>94.43</td><td>+1.85</td><td>+1.88</td><td>94.46</td><td>94.52</td><td>+1.85</td><td>+1.97</td></tr><tr><td>Fruits</td><td>64.88</td><td>67.86</td><td>60.69</td><td>66.78</td><td>-4.19</td><td>-1.08</td><td>60.16</td><td>66.78</td><td>-4.46</td><td>-1.08</td></tr><tr><td>Leaves with acuminate tips</td><td>81.82</td><td>81.4</td><td>76.76</td><td>75.08</td><td>-5.06</td><td>-6.32</td><td>72.35</td><td>74.26</td><td>-9.47</td><td>-7.14</td></tr><tr><td>Infructescence</td><td>66.35</td><td>69.51</td><td>61.65</td><td>66.67</td><td>-4.7</td><td>-2.84</td><td>64.03</td><td>67.5</td><td>-2.32</td><td>-2.01</td></tr><tr><td>Leaves with an acute base</td><td>73.87</td><td>74.92</td><td>69.53</td><td>74.32</td><td>-4.34</td><td>-0.6</td><td>71.33</td><td>72.4</td><td>-2.54</td><td>-2.52</td></tr></tbody></table>

<table><caption>**TABLE A3** | Comparison of SIM-Net and cross-attention variants on the complete image point cloud dataset (CIPD).</caption><thead><tr><th rowspan="3">Trait</th><th colspan="2">SIM-Net</th><th colspan="4">Point cloud as query—Image as key/ value</th><th colspan="4">Image as query—Point cloud as key/ value</th></tr><tr><th rowspan="2">Acc</th><th rowspan="2">F1</th><th colspan="2">SIM-Net<sub>CA</sub></th><th colspan="2">Δ</th><th colspan="2">SIM-Net<sub>CA</sub></th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>95.20</td><td>93.47</td><td>94.00</td><td>94.05</td><td>-1.20</td><td>+0.58</td><td>95.00</td><td>94.99</td><td>-0.20</td><td>+1.52</td></tr><tr><td>Fruits</td><td>64.66</td><td>67.22</td><td>58.79</td><td>66.82</td><td>-5.87</td><td>-0.40</td><td>58.79</td><td>66.59</td><td>-5.87</td><td>-0.63</td></tr><tr><td>Leaves with acuminate tips</td><td>77.00</td><td>73.92</td><td>75.40</td><td>75.15</td><td>-1.60</td><td>+1.23</td><td>76.00</td><td>76.74</td><td>-1.00</td><td>+2.82</td></tr><tr><td>Infructescence</td><td>69.37</td><td>69.10</td><td>64.58</td><td>66.86</td><td>-4.79</td><td>-2.24</td><td>64.17</td><td>67.64</td><td>-5.20</td><td>-1.46</td></tr><tr><td>Leaves with an acute base</td><td>71.40</td><td>72.66</td><td>69.00</td><td>71.40</td><td>-2.40</td><td>-1.26</td><td>69.60</td><td>69.97</td><td>-1.80</td><td>-2.69</td></tr></tbody></table>
<table><caption>**TABLE A4** | Comparison of SIM-Net with simple concatenation (SIM-Net) versus SIM-Net with bidirectional cross-attention (SIM-NetBCA), evaluated on two datasets: CIPD and SIPD.</caption><thead><tr><th rowspan="3">Trait</th><th colspan="2">SIM-Net</th><th colspan="4">SIPD</th><th colspan="4">CIPD</th></tr><tr><th colspan="2">SIM-NetBCA</th><th colspan="2">Δ</th><th colspan="2">SIM-NetBCA</th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>92.61</td><td>92.55</td><td>92.35</td><td>92.18</td><td>-0.26</td><td>-0.37</td><td>95.20</td><td>95.24</td><td>+2.59</td><td>+2.69</td></tr><tr><td>Fruits</td><td>64.88</td><td>67.86</td><td>60.42</td><td>67.03</td><td>-4.46</td><td>-0.83</td><td>62.24</td><td>66.75</td><td>-2.64</td><td>-1.11</td></tr><tr><td>Leaves with acuminate tips</td><td>81.82</td><td>81.40</td><td>72.06</td><td>73.42</td><td>-9.76</td><td>-7.98</td><td>76.40</td><td>76.89</td><td>-5.42</td><td>-4.51</td></tr><tr><td>Infructescence</td><td>66.35</td><td>69.51</td><td>62.83</td><td>67.47</td><td>-3.52</td><td>-2.04</td><td>65.83</td><td>69.41</td><td>-0.52</td><td>-0.10</td></tr><tr><td>Leaves with an acute base</td><td>73.87</td><td>74.92</td><td>69.53</td><td>73.68</td><td>-4.34</td><td>-1.24</td><td>70.00</td><td>69.49</td><td>-3.87</td><td>-5.43</td></tr></tbody></table>

<table><caption>**TABLE A5** | Accuracy evaluation with ablation on the point clouds of the SIM-Net model on the selected unsegmented image point cloud dataset (SSIPD).</caption><thead><tr><th>Trait</th><th>1024<br>points</th><th>410<br>points (40%)</th><th>102<br>points (10%)</th></tr></thead><tbody><tr><td>Fruits</td><td>66.56</td><td>62.01</td><td>61.64</td></tr><tr><td>Leaves with acuminate tips</td><td>79.84</td><td>76.47</td><td>71.47</td></tr><tr><td>Infructescence</td><td>70.19</td><td>66.08</td><td>64.01</td></tr><tr><td>Leaves with an acute base</td><td>74.67</td><td>70.97</td><td>69.18</td></tr></tbody></table>

Infructescence. In most cases, SIM-Net outperforms the single-modality (2D-only) baselines. The most notable improvements are observed with CoAtNet, ConvNeXtV2, and Swin, for all the traits with accuracy gains reaching up to +13.62% for Leaves with an acute base in the Swin model. For the Thorns and Fruits traits, SIM-Net achieves better performance in 4 out of 8 models. Significant gains are recorded for CoAtNet and DaViT on Thorns, with improvements of +15.31% and +13.95%, respectively. However, not all results are positive. A slight performance drop is observed for BEiT on Thorns (−0.11%), and more notably for ViT (−8.61%). These decreases may be attributed to segmentation errors during the point cloud generation process, which can affect model input quality. In the Fruits category, SIM-Net also shows mixed outcomes, with some models such as ConvNeXt and ViT experiencing substantial drops of −12.93% and −12.41%, respectively. This drop in performance may stem from the fact that fruits are often light-coloured and visually like the background (e.g., paper), leading to segmentation errors. In such cases, portions of the fruit may be misclassified as background, resulting in incomplete or noisy point clouds and degraded classification accuracy. Despite these challenges, SIM-Net consistently provides substantial performance gains across several traits and architectures. The structural traits, in particular, benefit the most from multimodal fusion. Models such as ConvNeXt (on Leaves with an acute base, +11.83%) and CoAtNet (on Leaves with acuminate tips, +7.94%) showcase the added value of incorporating 3D geometric cues alongside 2D visual features.

To further investigate the benefits of more advanced multimodal fusion mechanisms, we extended the SIM-Net architecture by integrating a cross-attention mechanism. In the adapted architecture, both point cloud and image features are first projected into a 512-dimensional embedding space through linear transformations and reshaped as single-token sequences. We explored three cross-attention configurations: (i) in the first variant (SIM-NetBCA, Table A2), the point cloud features act as the query, and the image features serve as key and value; (ii) in the second variant (SIM-NetCA, Table A3), the image features act as the query, and the point cloud features serve as key and value; (iii) in the third variant (SIM-NetBCA, Table A4), we implemented a bidirectional cross-attention mechanism, where both directions of attention (point cloud to image and image to point cloud) are computed independently, and their outputs are concatenated before fusion. The

attention outputs in each case are passed through a fully connected layer for classification. Batch normalisation and dropout are also applied for regularisation. We evaluated these cross-attention variants under the same experimental conditions as SIM-Net and ResNet101, using two dataset configurations: SIPD and CIPD.

The results are reported in Tables A2–A4. Overall, the use of cross-attention introduces variable effects across traits and datasets. While the single-direction cross-attention variants (Tables A2 and A3) occasionally lead to improvements on specific traits such as Thorns, they tend to degrade performance on several others, particularly for Fruits and Leaves with acuminate tips. Among these two single-direction variants, the configuration where the image acts as the query generally performs slightly better than the reverse. The bidirectional cross-attention variant (Table A4), which fuses the outputs of both attention directions, also shows a mixed impact: it achieves improvements on some traits and datasets (notably Thorns on CIPD) but again does not consistently outperform the baseline SIM-Net across all traits. These observations suggest that while cross-attention can enhance multimodal fusion in certain cases, its effectiveness is highly trait- and dataset-dependent and further tuning or more adaptive attention mechanisms may be required to obtain robust gains. Moreover, as in earlier experiments, the relatively small dataset size may limit the full potential of more sophisticated attention-based mechanisms.

More broadly, our results confirm that the integration of point cloud data consistently improves classification accuracy for traits with complex geometric structures. Notably, ResNet101 remains a strong image encoder across all settings. As shown in Table A1, SIM-Net equipped with ResNet101 outperforms alternative variants using other backbones, reinforcing the idea that deep convolutional networks offer excellent generalisation in low-data regimes—even compared to more recent attention-based approaches—when combined with complementary 3D representations.

## Ablation Studies

### Remove Points From the Point Clouds

To investigate the impact of point density on the model performance, we conducted an ablation study where we randomly removed 60% and 90% of the points from the point clouds associated with the images. The results showed that the accuracy performance decreased as the number of points decreased (cf., Table A5), highlighting the importance of a sufficient number of points for better classification.

### Point Clouds Without z Coordinates

To assess the impact of integrating the RGB colour information into the point cloud data on classification performance, we conducted an ablation study where the z-coordinate was set to zero in the four herbarium datasets (SPID, CIPD, SSIPD and CSIPD), effectively removing the depth information. We then compared the results to those obtained
using point clouds where the z-coordinate was calculated as the average of the colour intensities. The results, as illustrated in Tables A6 and A7, demonstrates a significant decrease in accuracy and F1-score performance when the z-coordinate was excluded, with differences of up to 5.29% in accuracy on the SIPD dataset, 3.60% on the CIPD dataset, 3.98% on the SSIPD dataset, and 7.7% on the CSIPD dataset. This highlights the importance of depth information in capturing geometric relationships and improving robustness to variations in specimen colour and preservation.

## Colour Encoding of Pixels Coordinates

To demonstrate the importance of explicitly incorporating x, y, z coordinates in the form of point clouds and processing them with PointNet-like architectures, we encoded the same spatial information into images referred to as colour-coded coordinates masks (cf., Figure A1). We used then these masks in a dual ResNet101 encoder architecture, that we call, a colour-coded mask and image fusion network (CMI-FNet).

<table><caption>TABLE A6 | Evaluation of SIM-Net model performance on point clouds with and without z coordinate using unsegmented images.</caption><thead><tr><th rowspan="3">Trait</th><th colspan="6">Selected image point cloud (SIPD)</th><th colspan="6">Complete image point cloud (CIPD)</th></tr><tr><th colspan="2">With z</th><th colspan="2">Without z</th><th colspan="2">Δ</th><th colspan="2">With z</th><th colspan="2">Without z</th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>95.75</td><td>95.74</td><td>94.20</td><td>94.05</td><td>+1.55</td><td>+1.69</td><td>95.00</td><td>95.01</td><td>91.60</td><td>91.49</td><td>+3.40</td><td>+3.52</td></tr><tr><td>Fruits</td><td>66.56</td><td>69.65</td><td>64.12</td><td>67.88</td><td>+2.44</td><td>+1.77</td><td>64.34</td><td>69.52</td><td>62.24</td><td>68.26</td><td>+2.10</td><td>+1.26</td></tr><tr><td>Leaves with acuminate tips</td><td>79.84</td><td>79.19</td><td>77.94</td><td>77.63</td><td>+1.90</td><td>+1.56</td><td>81.40</td><td>81.44</td><td>77.80</td><td>77.35</td><td>+3.60</td><td>+4.09</td></tr><tr><td>Infructescence</td><td>70.19</td><td>71.35</td><td>64.90</td><td>67.90</td><td>+5.29</td><td>+3.45</td><td>73.05</td><td>74.24</td><td>71.67</td><td>72.69</td><td>+1.38</td><td>+1.55</td></tr><tr><td>Leaves with an acute base</td><td>74.67</td><td>75.84</td><td>69.89</td><td>72.39</td><td>+4.78</td><td>+3.45</td><td>71.80</td><td>73.46</td><td>69.60</td><td>71.50</td><td>+2.20</td><td>+1.96</td></tr></tbody></table>

<table><caption>TABLE A7 | Evaluation of SIM-Net model performance on point clouds with and without z coordinate using segmented images.</caption><thead><tr><th rowspan="3">Trait</th><th colspan="6">Selected segmented image point cloud (SSIPD)</th><th colspan="6">Complete segmented image point cloud (CSIPD)</th></tr><tr><th colspan="2">With z</th><th colspan="2">Without z</th><th colspan="2">Δ</th><th colspan="2">With z</th><th colspan="2">Without z</th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>92.61</td><td>92.55</td><td>90.24</td><td>90.49</td><td>+2.37</td><td>+2.06</td><td>95.90</td><td>95.79</td><td>88.20</td><td>88.80</td><td>+7.7</td><td>+6.99</td></tr><tr><td>Fruits</td><td>64.88</td><td>67.86</td><td>62.27</td><td>65.52</td><td>+2.61</td><td>+2.34</td><td>64.84</td><td>67.03</td><td>59.48</td><td>66.44</td><td>+5.36</td><td>+0.59</td></tr><tr><td>Leaves with acuminate tips</td><td>81.82</td><td>81.40</td><td>79.29</td><td>79.27</td><td>+2.53</td><td>+2.13</td><td>82.27</td><td>82.33</td><td>80.80</td><td>80.42</td><td>+1.47</td><td>+1.91</td></tr><tr><td>Infructescence</td><td>66.35</td><td>69.51</td><td>64.01</td><td>67.75</td><td>+2.34</td><td>+1.76</td><td>70.20</td><td>72.32</td><td>69.79</td><td>71.63</td><td>+0.41</td><td>+0.69</td></tr><tr><td>Leaves with an acute base</td><td>73.87</td><td>74.92</td><td>69.89</td><td>73.75</td><td>+3.98</td><td>+1.17</td><td>71.84</td><td>73.56</td><td>71.16</td><td>69.97</td><td>+0.68</td><td>+3.59</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>FIGURE A1 | Colour-coded coordinates masks generation pipeline and the colour-coded mask and image fusion network (CMI-FNet).</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>FIGURE A2 | Examples of herbarium scans with corresponding masks, colour-coded masks and point clouds.</figcaption></figure>

<table><caption>TABLE A8 | Performance evaluation of the SIM-Net model compared to a colour-coded mask and image fusion network (CMI-FNet) using pairs of (unsegmented images, colour-coded masks) applied on the—Selected image point cloud dataset (SIPD).</caption><thead><tr><th rowspan="2">Trait</th><th colspan="2">SIM-Net</th><th colspan="2">CCM-IFNet</th><th colspan="2">Δ</th></tr><tr><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th><th>Acc</th><th>F1</th></tr></thead><tbody><tr><td>Thorns</td><td>95.75</td><td>95.74</td><td>93.67</td><td>93.51</td><td>+2.08</td><td>+2.23</td></tr><tr><td>Fruits</td><td>66.56</td><td>69.65</td><td>62.27</td><td>66.78</td><td>+4.29</td><td>+2.87</td></tr><tr><td>Leaves with acuminate tips</td><td>79.84</td><td>79.19</td><td>74.71</td><td>75.60</td><td>+5.13</td><td>+3.59</td></tr><tr><td>Infructescence</td><td>70.19</td><td>71.35</td><td>67.55</td><td>70.44</td><td>+2.64</td><td>+0.91</td></tr><tr><td>Leaves with an acute base</td><td>74.67</td><td>75.84</td><td>70.25</td><td>74.12</td><td>+4.42</td><td>+1.72</td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>FIGURE A3 | Distribution of the five selected classes in the prepared VOC dataset.</figcaption></figure>
## Colour-Coded Coordinates Masks Generation Pipeline

The transformation pipeline of herbarium sheet images into colour-coded coordinates masks, as illustrated in Figure A1, begins with a segmentation phase that removes the background elements. The obtained image $I \in R^{H \times W \times C}$ is then transformed into another image $I'$, termed 'colour-coded coordinate masks'. This transformation is achieved by converting each pixel $I(x, y)$ with $(R_{xy}, G_{xy}, B_{xy}) \neq (0, 0, 0)$ to a new pixel $I'(x, y)$ with $(R'_{xy}, G'_{xy}, B'_{xy}) = y, B'_{xy} = \frac{R_{xy} + G_{xy} + B_{xy}}{3}$. The values $(R'_{xy}, G'_{xy}, B'_{xy})$ are normalised to the interval [0, 256], assuming the pixel colours values $R'G'B'$ are scaled appropriately within this range.

## Fusing RGB Features With Colour-Coded Masks Features

The fusion function involves concatenating the RGB image feature vector, of size 2048, with the colour-coded coordinates mask feature vector, of size 256 (cf., Figure A1). In this network, we use ResNet101 as the image encoder. To train the model, we used the same parameters as

<figure><img src="image_11.png" /><figcaption>**FIGURE A4** | Distribution of the 20 classes in the prepared VOC dataset.</figcaption></figure>

in SIM-Net training, except for the batch size, which we set to 64. Figure A2 shows examples of herbarium images, their masks, colour-coded masks, and the associated point clouds, respectively.

We compared the results to our original architecture SIM-Net and we observed that the performance of the SIM-Net model is better than those obtained with the colour-coded coordinates mask and image fusion network, with improvements of up to more than 5% for accuracy and more than 3.5% for the F1-score (cf., Table A8).

## Generalisability

To study the performance of our model on more general datasets with varied labels, we used the public dataset—The PASCAL Visual Object Classes (PASCAL VOC).³ The dataset contains 2913 images with their associated masks. However, some images contain multiple segmented objects. Consequently, using this dataset as it is, does not suit our study case in this paper since, as mentioned, we are interested in images targeting a single object. Therefore, we selected a subset of images containing a single object of interest and constructed a dataset of 1389 images containing a single segmented object. The dataset is labelled with 20 different classes. We created two datasets. The first dataset, with 5 classes ('train', 'dog', 'cat', 'bird', 'aeroplane'), contains 700 images with a balanced distribution across the classes (Figure A3). The second dataset, with 20 classes, includes the 1389 selected images but has a significant imbalance between the classes (Figure A4). For each image, we used its mask to generate the equivalent point clouds. Figure A5 shows examples of images and their masks from VOC dataset, with their corresponding generated point clouds.

Table A9 demonstrates that our method is more effective when we utilise ResNet101 built from scratch in both architectures. However, when ResNet101 is pre-trained on ImageNet 1K, our approach significantly underperforms compared to using ResNet101 alone, with discrepancies reaching up to 20%. This performance gap is likely influenced by the fact that our method focuses on classifying objects of a similar nature, such as herbarium images, whereas the VOC dataset comprises objects of a vastly different nature. Furthermore, in the VOC dataset, target objects are often partially occluded or incomplete, with segmentation results producing object shapes that do not accurately

<figure><img src="image_12.png" /><figcaption>**FIGURE A5** | Examples of images and their masks from VOC dataset, with their corresponding generated point clouds.</figcaption></figure>
<table><caption>TABLE A9 | Accuracy evaluation on VOC-selected dataset.</caption><thead><tr><th></th><th>Nº classes</th><th>ResNet101</th><th>SIM-Net</th></tr></thead><tbody><tr><td rowspan="2">ResNet101 from scratch</td><td>5</td><td>35.51</td><td>47.83</td></tr><tr><td>20</td><td>17.52</td><td>22.63</td></tr><tr><td rowspan="2">ResNet101 pretrained</td><td>5</td><td>90.38</td><td>70.62</td></tr><tr><td>20</td><td>39.78</td><td>24.45</td></tr></tbody></table>

reflect their real-world geometry—unlike herbarium specimens, which are typically fully visible and free from such occlusions. Additionally, the close similarity between the VOC and ImageNet datasets may contribute to significant variances when generalising results.