# Optimizing Plankton Image Classification With Metadata-Enhanced Representation Learning

Mojtaba Masoudi, Sarah L.C. Giering, Noushin Eftekhari, Miquel Massot-Campos, Jean-Olivier Irisson, Blair Thornton

► To cite this version:

Mojtaba Masoudi, Sarah L.C. Giering, Noushin Eftekhari, Miquel Massot-Campos, Jean-Olivier Irisson, et al.. Optimizing Plankton Image Classification With Metadata-Enhanced Representation Learning. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 17, pp.17117 - 17133. 10.1109/jstars.2024.3424498. hal-04792218

HAL Id: hal-04792218

https://hal.science/hal-04792218v1

Submitted on 20 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Optimizing Plankton Image Classification With Metadata-Enhanced Representation Learning

Mojtaba Masoudi ID, Sarah L.C. Giering ID, Noushin Eftekhari ID, Miquel Massot-Campos ID, Jean-Olivier Irisson ID, and Blair Thornton ID

*Member, IEEE*

**Abstract**—Automated camera-based sensors are widely used in vessel-based research to monitor plankton and marine particles. However, current methods suffer from the costly and time-consuming requirement of annotating data for fully supervised learning, especially in plankton grouping tasks characterized by long-tailed datasets. In response, we propose a novel self-supervised learning framework that significantly reduces reliance on expensive human annotations by leveraging crucial metadata such as water depth and location. The method comprises three major steps: self-supervised training, innovative sampling, and final classification. It identifies key sample subsets from an unlabeled dataset using a hierarchical clustering approach and incorporates an innovative balancing representative subsampling strategy that addresses the challenge of dataset imbalance and enhances generalizability across diverse plankton classes. Our approach prioritizes discerning representation features observed in images that exhibit correlations with the patterns found in their associated metadata. Furthermore, our method introduces a novel grouping based on the visual perspective selection method, enabling the identification of balanced subset views that depart from traditional class-based categorization. Our experimental results showcase a significant enhancement in image classification accuracy, with a 23% improvement over methods that do not utilize metadata, and attains a macro F1-score of 54% for ten populated species from a severely long-tailed dataset. This is achieved with a mere 0.3% of the entire dataset used for annotation.

**Index Terms**—Convolutional neural network (CNN), marine imaging, plankton, representation learning, self-supervised learning (SSL).

Manuscript received 14 February 2024; revised 5 June 2024; accepted 2 July 2024. Date of publication 11 July 2024; date of current version 30 September 2024. This work was supported in part by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme through ANTICS project Under Grant 950212 and in part by the European Union's Horizon 2020 research and innovation programme (TechOceanS) under Grant 101000858. (Corresponding author: Sarah L.C. Giering.)

Mojtaba Masoudi and Sarah L.C. Giering are with the Department of Data, Science and Technology (DST), National Oceanography Centre, SO14 3ZH Southampton, U.K. (e-mail: mojmas@noc.ac.uk; s.giering@noc.ac.uk).

Noushin Eftekhari is with the Alan Turing Institute, NW1 2DB London, U.K. (e-mail: neftekhari@turing.ac.uk).

Miquel Massot-Campos and Blair Thornton are with the Faculty of Engineering and Physical Science, University of Southampton, SO16 7QF Southampton, U.K. (e-mail: miquel.massot-campos@soton.ac.uk; b.thornton@soton.ac.uk).

Jean-Olivier Irisson is with the Laboratoire d'Océanographie de Villefranche, Sorbonne Université, 06230 Villefranche-sur-Mer, France (e-mail: jean-olivier.irisson@imev-mer.fr).

This article has supplementary downloadable material available at https://doi.org/10.1109/JSTARS.2024.3424498, provided by the authors.

Digital Object Identifier 10.1109/JSTARS.2024.3424498

## I. INTRODUCTION

THE critical role of plankton as the foundation of aquatic food webs underscores the necessity of their monitoring [1], [2]. Traditionally (and still prevalent today), monitoring involves manual sampling with plankton nets and subsequent microscopic analysis, which is time-consuming and labor-intensive. These methods often result in years between sample collection and data availability, and sampling itself can be limited by weather conditions and access to sampling sites. To overcome these limitations and to make plankton monitoring more efficient and effective, significant efforts have been dedicated to developing automated quantitative imaging equipment (see references in [3] and [4]) such as the underwater vision profiler (UVP) [5], which provides detailed optical information on individual plankton, enhancing the efficiency and scope of monitoring efforts.

Despite these technological advances, the processing and interpretation of data from imaging systems are fraught with challenges, as illustrated in Fig. 1, where the ambiguity in many images demonstrates the difficulty of plankton classification. Plankton images are frequently of poor quality due to the microscopic size of the organisms, optical water distortions, and movement from both the camera and the plankton, resulting in unclear images difficult for even skilled taxonomists to classify. Moreover, the presence of numerous nonplankton objects such as “marine snow” (detritus particles) creates a significant imbalance, with dominant classes overshadowing rare species that may be critical for ecosystem assessment. To efficiently handle the rapidly growing volumes of plankton imagery, there is an increasing reliance on computer-aided workflows that streamline the annotation process, enhancing the accuracy of plankton identification [6], [7].

This article introduces a novel learning framework employing self-supervised learning (SSL) techniques to enhance understanding of plankton data by leveraging crucial metadata such as water depth and location. The framework is designed to minimize human annotation reliance, integrating advanced machine learning strategies specifically tailored for plankton image recognition. By incorporating metadata directly into the learning process, the model significantly improves its ability to generate low-dimensional latent representations, aiding in the semantic interpretation of marine biodiversity. This approach not only ensures sensitivity to ecological variations across different marine environments but also facilitates a deeper and more robust classification of diverse and rare plankton species.
Our model addresses the challenge of imbalanced datasets by employing a novel balancing representative subsampling approach, which minimizes the need for extensive manual annotations. It utilizes a semisupervised workflow that leverages both labeled and unlabeled data, with the latter being augmented by pseudolabels generated through the model's self-learning capabilities. Through the strategic use of metadata and innovative hierarchical techniques, our framework establishes a new standard for automated plankton classification, offering substantial improvements in efficiency and accuracy while effectively tackling the challenges posed by the vast diversity and imbalance found in marine species datasets.

Here, the concept of “views”—different visual perspectives or characteristics within the same category—is pivotal. By strategically selecting representative subsamples from each view, the model enhances its ability to generalize across the vast and heterogeneous plankton classes more effectively than traditional methods. This is particularly critical given the high degree of intraclass variability and the presence of rare species, which often complicate classification tasks. In summary, our proposed method makes the following key contributions.

1) *Enhanced SSL with metadata regularization:* We explore the effectiveness of SSL metadata regularization in learning meaningful representations of plankton images. This process involves reconstructing images while embedding prior information to group visually similar patterns together, thus enhancing the robustness of the learned features.

2) *Semantic mapping and retrieval applications:* The learned features are utilized in semantic mapping applications, including hierarchical clustering and content-based image retrieval. This approach provides a richer contextual understanding of the images, enabling more nuanced interpretations and classifications.

3) *Innovative representative sampling for imbalance management:* We introduce a novel representative sampling selection strategy designed to address significant imbalances in the data. This method prioritizes discerning representation without extensive human-supervised labeling, substantially streamlining the annotation process and reducing the workload on human taxonomists.

4) *Handling rare and diverse species:* Our framework excels in identifying rare species and managing the substantial diversity and feature overlap inherent in plankton images.

5) *Visual perspective selection for improved classification:* We leverage novel grouping based on visual perspective selection by introducing the concept of “view” to achieve more accurate classification performance.

## II. BACKGROUND

### A. Conventional Methods

Traditional research in automatic plankton and particle classification has predominantly employed handcrafted methods that focus on low-level visual attributes such as size, morphology, SIFT, and LBP, assuming these are key for distinguishing classes [8], [9], [10], [11], [12], [13], [14], [15]. The rise of deep

learning (DL) has, however, questioned the efficiency of these methods by introducing algorithms that transform raw data into feature vectors, capturing essential invariances that handcrafted features often miss [16], [17], [18]. DL's ability to learn abstract representations and adapt across diverse datasets without domain-specific expertise marks a significant shift toward more flexible and generalizable methods [19]. Supervised DL methods are particularly favored for their robust capacity to learn and distinguish complex visual features. These methods streamline the classification process by integrating data representation and classification into a single end-to-end workflow, thus bypassing the need for extensive feature engineering and parameter tuning [18], [20], [21], [22], [23], [24], [25]. Cui et al. [20] applied ResNet-32 to improve throughput and efficiency in plankton image classification. Meanwhile, Py et al. [21] conducted an extensive exploration of various convolutional neural network (CNN) configurations, integrating an inception layer designed to handle multisize input images. Dai et al. [22] proposed ZooplanktoNet, testing the effects of data augmentation and the number of convolutional layers. Guo et al. [23] introduced a model using multiple image views, combining the original image with versions processed through Gaussian filters to slightly improve accuracy.

However, supervised ML models demonstrate poor performance, generalization failures, and biases to the extremely imbalanced dataset [26]. Many efforts, such as augmentation strategies, try to alleviate some of the problems but still, highly correlated data lead to unacceptable model performance [24]. Addressing class imbalance, Lee et al. [24] used a simplified CNN derived from AlexNet, pretraining with class-normalized samples to boost performance. In addition, CGAN-like architectures [27] were employed to augment underrepresented classes, and a two-stage training process was used where a CNN pretrained on the least represented categories was later integrated into a full dataset training setup [25]. Walker and Orenstein [28] used a method called background resampling, which involves hard negative mining to downsample background data, creating a more balanced dataset for training.

Despite advances in supervised learning, its application in plankton research faces significant challenges. A primary challenge is the lack of extensive annotated datasets. Researchers typically work in isolation, on relatively small datasets collected with different instruments or instrument settings, and following different taxonomic choices (e.g., differing taxonomic resolution or naming conventions). Key species can furthermore be rare [29], resulting in low numbers of training data for these critical groups. Moreover, in many real-world scenarios, it is impractical to create extensive labeled training datasets because of the vast diversity and unpredictability of marine environments. To address the constraints on human resources, incorporating unlabeled data into the training process has become a common strategy.

Overall, these challenges—severe imbalanced data, constraints on human resources, and the complexities of dynamic marine environments—underscore the necessity for innovative approaches that can more effectively utilize existing data and enhance generalization across diverse conditions and datasets.
## B. Innovative Methods for Reducing Annotation

Research into learning representations using unlabeled and “few labeled” data has led to the development of several research fields, including unsupervised, semisupervised, and self-supervised [30]. These fields aim to bypass the laborious task of data annotation by developing representations that can generalize across different learning tasks [31]. Particularly, SSL leverages preexisting or inherently available labels, eliminating the need for explicit human annotations. In SSL, features are acquired through a simple pretext or proxy task defined for the network to solve during training [32]. Typically, models such as CNNs or increasingly vision transformers [33] are trained on a large corpus of unlabeled data by optimizing a self-generated objective. This process allows the model to capture high-level data representations without manual labels, which are then adaptable for supervised tasks in practical applications. Wang et al. [34] categorize SSL approaches into three main types: generative, context-based, and contrastive methods.

Generative methods, including autoencoders [35] and generative adversarial networks [27], focus on recreating or generating the input data, and have been applied to various data types such as multispectral and hyperspectral images [36], [37]. Context-based methods leverage the contextual features within images, with techniques designed to exploit aspects such as context similarity [38] and spatial structure [39]. A pioneering method in this category involved predicting the relative positions of image patches to understand their spatial relationships [40]. Foundation models such as SpectralGPT [41], which are crafted for processing spectral data, can combine features of both generative and context-based methods within the SSL framework [42].

Contrastive methods improve a model's ability to identify similarities among semantically related inputs without relying on specific single pretext tasks. These methods train models by comparing semantically identical inputs, such as two augmented views of the same image, and encouraging similar representation in the embedding space. A notable advance in this field is the development of SimCLR [43], a simple framework for contrastive learning of visual representations, which has shown promising results in hyperspectral image classification with minimal labels [44].

In addition, clustering-based SSL methods group similar features together in the embedding space using clustering algorithms such as K-means, as demonstrated in DeepCluster [45]. This approach generates pseudolabels that help in training models to predict these labels effectively. Techniques such as training autoencoders with additional loss functions to enhance clustering in the latent space have also been developed, showcasing their utility in applications such as seafloor imaging [46], [47], [48].

Metadata can offer contextual information that enhances data categorization by emphasizing relevant features and patterns [49]. For example, in conventional microscopy-based taxonomy, details such as the water depth and collection location of an organism assist in its classification. In [46], the researchers introduced a location-guided autoencoder that utilizes horizontal location information to regulate learning.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Visual examples of plankton classes: Copepoda, Tuff-like, and Fiber-like. Given the low image quality, distinguishing across classes will be difficult.</figcaption></figure>

Some researchers remain unclear whether the high-level representations shaped by category-level influences are comparable to those our visual system uses to recognize and distinguish the myriad of objects we encounter daily [50]. The specific knowledge governing these visual representations in the human brain is still not fully understood [51], [52], [53], leading some to question the suitability of category-level forces as proxies in understanding these representations and to consider alternative theories. Our analysis of plankton data supports this perspective, which reveals greater similarities between images of plankton and marine particles across different classes than within the same class. Consequently, we explore whether self-supervised grouping of a representative subset of images could improve classification accuracy. Throughout this article, we refer to these groups as “views” within each class, where “view” in computer vision usually refers to the visual perspective or angle of image capture. This terminology is vital for our research as it aids in categorizing images by their visual presentation. Our model’s hierarchical structure is devised to focus on distinguishing features within each plankton view, rather than forcing it to recognize features across all classes [54], [55].

## III. METHODOLOGY

We developed a metadata-driven representation learning workflow using SSL to pinpoint key samples for understanding category distinctions of interest and apply this to plankton image classification problems, as illustrated in Fig. 2. Initially, inputs are passed through an encoder-decoder network, leveraging a convolutional backbone to learn the representations, detailed in Section II-A. Next, a data sampling technique is employed on these latent representations to refine the number of dominant classes into a manageable number of representative samples, ensuring diverse coverage that includes a variety of types, particularly infrequent species, as described in Section III-B. These features are then organized into clusters to distinguish among various class views. An unsupervised clustering algorithm selects *m* key samples from these clusters, where *m* ∈ (1...*N*<sub>c</sub>) and *N*<sub>c</sub> represents the total number of samples per cluster. We demonstrate that this approach can yield effective results even with smaller sample sizes. Subsequently, based on these key samples, pseudolabels are assigned. These annotations are then
<figure><img src="image_3.png" /><figcaption>Fig. 2. Overview of the study design. (a) Following data collection, our augmented data were processed through an encoder-decoder network, enabling regularization with the metadata. The KL divergence was assessed by evaluating the mutual information between the latent space's similarity distribution and the metadata space. (b) Here, the data distribution is depicted. This step categorizes the representation into three groups, reducing the impact of the predominant detritus class, which constitutes 80% of the data. One group is entirely nondetritus, another is a borderline samples that is difficult to discern, and the third is composed solely of detritus samples. (c) Selected representative samples were annotated by human experts, and pseudolabels were assigned to all unlabeled samples. (d) These annotations and pseudolabels were then incorporated into the CNN architecture for fine tuning.</figcaption></figure>

used in the final phase of the downstream task, as outlined in Section III-C.

## A. Self-Supervision Using Metadata-Guided Autoencoder

The concept of autoencoders was initially presented in [35] as a neural network designed to learn the reconstruction of its input. The primary mechanism of an autoencoder involves two key processes: encoding and decoding. In the encoding phase, the network transforms the input data into a compressed latent space representation, and in the decoding phase, it attempts to reconstruct the input data from this compressed representation. The ultimate goal of an autoencoder is to minimize the reconstruction error, making the output as close as possible to the original input. Formally, as described in [56], the objective of an autoencoder can be encapsulated by the functions $A: \mathbb{R}^n \to \mathbb{G}^p$ and $B: \mathbb{G}^p \to \mathbb{R}^n$, where $A$ is the encoder and $B$ is the decoder. Here, $\mathbb{R}$ and $\mathbb{G}$ represent the sets from which the data vectors and their encoded representations are drawn, respectively, with $n$ and

$p$ being positive integers such that $0 < p < n$. The effectiveness of these functions is measured through the minimization of the expected value of a dissimilarity function $\Delta$, which quantifies the difference between the original input and its reconstruction

$$ \arg\min_{A,B} E[\Delta(\mathbf{B} \circ \mathbf{A}(x), x)]. \quad (1) $$

In this equation, $x$ represents the input data, $h = A(x)$ is the encoded latent representation, and $x_{\text{rec}} = B(h)$ is the reconstructed data. The function $E$ denotes the expectation over the distribution of $x$, and $\Delta$ is the dissimilarity function, typically a loss function such as mean squared error, that measures how well the autoencoder is performing in terms of reconstructing the original input from the encoded representation. While autoencoders primarily focus on accurate input reconstruction, it is equally crucial for the low-dimensional representation to encompass meaningful and generalizable features. Utilizing an auxiliary data-driven target function in the SSL context can accelerate the learning process within the embedding space [57].
For instance, in environmental monitoring, integrating geo-referenced information into the loss function acts as an effective regularization strategy, enhancing the model performance [46]. The overall loss function in such a scenario is designed to minimize both the reconstruction loss and a regularization term. Specifically, it is defined as follows:

$$L_{\text{all}} = L_{\text{rec}} + \lambda \text{KL}(P \parallel Q). \quad (2)$$

Here, $L_{\text{rec}}$ represents the reconstruction loss, and the term involving the Kullback-Leibler (KL) divergence functions as the regularization component. The KL divergence measures the discrepancy between two probability distributions, $P$ and $Q$, derived from the encoded latent space and the associated metadata, respectively. This regularizer aims to align the distribution $P$, characterizing the similarity between data points in the latent space, with the distribution $Q$, which is informed by external metadata.

An effective feature learner aims to minimize the distance between similar data points' embeddings, denoted as $h_i$ and $h_j$, within the latent representation space. This means that if the original data, $x_i$ and $x_j$, are similar, their corresponding embeddings should be positioned closer together. However, considering the similarity of just $x_i$ and $x_j$ might not fully represent their relationship; the associated metadata $y_i$ and $y_j$ also play a crucial role. Therefore, it is common to employ the Student's t distribution [58] as a kernel for quantifying the affinity or similarity between data points in a transformed, lower dimensional space. This kernel helps ensure that points closer in the original space are also proximate in the embedded space, thus maintaining topological fidelity. The probability distribution $P_{ij}$, representing the relationship between embeddings of samples $i$ and $j$, can be formally expressed as follows

$$P_{ij} = \frac{(1 + \|h_i - h_j\|^2 / \alpha)^{-\frac{\alpha+1}{2}}}{\sum_{i'} \sum_{j'} (1 + \|h_{i'} - h_{j'}\|^2 / \alpha)^{-\frac{\alpha+1}{2}}}. \quad (3)$$

For the metadata-driven distribution $Q_{ij}$, the following formulation is used to capture the similarity between metadata samples $i$ and $j$:

$$Q_{ij} = \frac{(1 + \text{sim}(y_i, y_j)^2 / \alpha)^{-\frac{\alpha+1}{2}}}{\sum_{i'} \sum_{j'} (1 + \text{sim}(y_i, y_{j'})^2 / \alpha)^{-\frac{\alpha+1}{2}}}. \quad (4)$$

In this setup, $\alpha$ is fixed at 1, and $\text{sim}(y_i, y_j)$ represents the similarity measure between metadata samples, computed here using the Euclidean distance [59] due to its simplicity and interpretability, aligning well with the type of metadata available.

The autoencoder described is designed to compress the visual appearance of images while also regularizing patterns based on associated metadata. This process is guided by minimizing a specific loss function, referred to as (2). Once trained, the encoder component of the autoencoder functions as a robust feature extractor. To combat the risk of overfitting, a strategic sampling method is employed. Specifically, a 1:1 sampling ratio is used to balance between images that meet certain user-defined metadata criteria and those that are randomly selected from the entire dataset. For example, within the scope of user-defined criteria, similarity among images is measured using Euclidean distance.

Let us consider a scenario where the metadata criterion is depth, and the maximum allowable similarity distance is set at 10 m. In practical terms, this means selecting the first image randomly, then choosing 50% of subsequent samples from within a 10-m distance of first image, and the remaining 50% from across the entire dataset. This sampling strategy ensures a diverse range of samples in the affinity matrices $P$, effectively preventing excessive regularization due to skewed sample distributions. It also allows for equal consideration of both similar and dissimilar images during each batch iteration. By maintaining this balance, the model can learn from a varied set of examples, improving its ability to generalize and avoid biased representations.

## B. Reduction of Large Class Imbalance

In this part, we utilize latent representations derived from the previous encoder-decoder network, forming the foundation for our clustering and sampling strategies. These strategies are essential for creating a balanced dataset crucial for the subsequent analysis phases and the success of the supervised taxonomist-led training phase (see Section III-C). A balanced dataset is crucial to mitigate overfitting and to ensure that the learning process is not skewed by the overrepresentation of any single class. In our study, the dataset is predominantly composed of the detritus class, which accounts for 80% of the images. This imbalance can adversely affect the performance of clustering algorithms, leading them to disproportionately select representative samples from the detritus class. Such a scenario often results in the misclassification of the majority class instances as minority classes, creating clusters of uniform size rather than clusters that reflect the true distribution of the data. This issue is known as the “uniform effect” in k-means clustering [60], presenting significant challenges in adequately representing rarer classes, such as specific types of plankton which are of particular interest in our study. To tackle this imbalance, we have developed a customized data sampling strategy. Considering the shape heterogeneity and the overlap between the detritus class and other plankton classes [61], [62], our approach employs an innovative undersampling method. This method strategically eliminates noisy and less informative examples from the detritus class. Our methodology comprises several steps [illustrated in Fig. 2(b)], which are as follows.

1) **Unsupervised clustering:** We apply hierarchical k-means clustering (H-kmeans) to the latent data generated in Section III-A to group similar images based on their visual and metadata characteristics. We set the number of clusters ($k$) to twice the number of expected classes ($k = 46$), as this has shown to enhance the clustering algorithm's sensitivity to nuanced patterns within each class based on preliminary tests (see Section IV-B2).

2) **Expert annotation:** The most representative image from each subcluster is selected for annotation by expert taxonomists. This step is crucial for accurately depicting the variability within each cluster, especially when the clustering resolution is challenged by ambiguous class boundaries. This process further divides each of the $k$ clusters into smaller subclusters using a hierarchical approach,
calculated by the ratio $[M_1/k]$ where $M_1$ equals 500, the total number of images designated for annotation. From these, the most central image in each subcluster is selected for detailed examination by expert taxonomists, ensuring representative samples are used, especially in cases with ambiguous class boundaries.

3) *Reduction of dominant class samples:* To refine the representation of nondetritus classes, we reduce the number of detritus images by retaining only a small proportion (10% as determined in our tests) of images from near the center of each detritus-dominated subcluster.

Subsequently, each of the *k* clusters is categorized into one of three groups based on the composition of the images they contain.

1) *Group A (Detritus):* Clusters composed exclusively of detritus class samples.

2) *Group B (Ambiguous):* Clusters containing a mix of detritus and nondetritus samples, reflecting transitional characteristics that may blur the distinction between detritus and plankton, especially when image quality affects the clarity of class features.

3) *Group C (Nondetritus):* Clusters distinctly separate from the detritus class, exclusively containing nondetritus (plankton) samples.

This categorization strategy effectively addresses the overwhelming presence of detritus in our dataset. Initially, most clusters fall into Group A, which is heavily dominated by detritus. This dominance restricts the adequate representation of Groups B and C. To manage this, we adopt a targeted undersampling strategy for Group A, retaining only a small portion of images from the central region of each heavily detritus-laden subcluster. By increasing the number of clusters and selectively reducing detritus samples, we improve the resolution of nondetritus classes, enabling the clustering algorithm to better distinguish previously overlooked nondetritus groups.

### C. Human-Led Annotation for Pseudolabeling and Final Classification

In the latter stages of our methodology, after mitigating the dominance of the detritus class, we recalibrate and select a fresh set of $M_2$ representative images. The number of images is set to $M_2 = 2000$ based on experiments (see Section IV-B3). Expert taxonomists annotate these images to provide key insights that are crucial for reorganization of the dataset. For images that remain unannotated, we implement a nearest neighbor technique to assign pseudolabels by matching them with the most similar annotated samples.

Next, we introduce a structured framework of “views” within our methodology. Every cluster, redefined postannotation, is regarded as a unique “view.” This structure helps address challenges such as class imbalance and the presence of subclasses with subtle visual differences. The “multiviews” strategy reorganizes the data, aligning it according to visual similarities and dissimilarities, both within and between classes. This is particularly important when some views in a class visually resemble other classes more than their own, necessitating a model that

prioritizes distinction over broad categorization. This nuanced approach not only helps in identifying subtle features but also significantly boosts classification accuracy. While the “multiview” setup is advantageous, it necessitates careful management to avoid overfitting. The number of views is determined through empirical testing and ongoing evaluation of the model’s performance during the fine-tuning phase. This carefully structured approach informs our final step of model training. We proceed to train a CNN classifier on this pseudolabeled dataset using five classifiers for comparison: Random Forest (RF), Decision Tree (DT), Support Vector Machine (SVM), XGBoost (XG), and CNN (see Section IV-B4).

## IV. EXPERIMENTS

In this section, we conduct a series of comprehensive comparative experiments using a benchmark dataset of plankton images collected by the UVP6. After describing the dataset, we present a detailed analysis of the performance of our proposed workflow across various experimental parameter settings, aiming to establish the effectiveness of our approach. Each configuration was evaluated individually to assess its efficacy. We then briefly delve into a discussion of the obtained results, exploring the connections between our findings and the field of ocean ecology. This analysis will shed light on the insights gained from our research and their implications. Finally, we will identify potential avenues for further research and exploration in this domain.

### A. Dataset

A large image dataset collected using the UVP6 camera system [5] was used to develop and test the suggested method. The UVP6 system utilizes a 5-Megapixel CMOS monochrome image camera sensor (Sony IMX264) where objects are illuminated by a collimated light beam positioned in front of the lens. To ensure accurate object detection, the processing unit software incorporates a zone-specific gain correction method that adjusts the gain settings of different image regions based on their specific lighting conditions. Subsequently, an automatic background subtraction and thresholding-cropping technique is applied. The camera system has been especially designed to cover a broad variety of particle sizes, ranging from about 100 µm to around 50 mm in diameter.

The geographical locations of our dataset cover a broad range of oceanic regions, including the Mediterranean Sea, equatorial Atlantic Ocean, equatorial Pacific Ocean, and polar regions [see Fig. 3(a)]. The dataset comprises images from 980 dives between 0 and 5000 m depth [see Fig. 3(b)], with the highest data density within the upper 1000 m of depth. A total of 451 379 images were recorded.

In order to examine the size distribution of the imaged particles and accurately categorize them, we employed a scaling factor calculation for each individual object as part of the metadata. This scaling factor is determined by comparing the original size of the specimen, measured in millimeters per pixel, to a standardized value such as a fixed patch size of the image (i.e.,
<figure><img src="image_4.png" /><figcaption>Fig. 3. Charting locations and analyzing depth-size distributions. (a) Global heatmap revealing dive locations across the globe. (b) Exploring the relationship between depth and scaling factor with a bivariate and histogram analysis.</figcaption></figure>

227 pixels). This calculation plays a crucial role in differentiat-
ing between small and large plankton, which would otherwise
be grouped together based solely on visual observations. We
can calculate the size of each item by including the scaling
factor; bigger plankton are indicated by greater scaling factor
values. Scaling factors range from 0.1 to 17.5, indicating that
we imaged plankton that spans two orders of magnitude in size
[see Fig. 3(b)].

Expert plankton taxonomists annotated all images in the
dataset across 55 classes, with varying image counts per class
ranging from 46 to 509 k. To streamline the computer-led classi-
fication, we condensed the number of human-annotated classes
to 23 by merging closely related and/or underrepresented classes
(e.g., combining the visually almost identical order of *Collo-
daria* and family of *Aulosphaeridae* to clade level, Rhizaria).
It is noteworthy that these annotations may contain a degree
of human error in the classification (see discussion in [63] and
[64]).

B. *Experimental Settings Description*

For the self-supervised model, the autoencoder is structured
using the Alexnet [65] architecture modified to incorporate batch
normalization at each layer, serving as the encoder. The tradi-
tional AlexNet is adapted by altering the final fully connected
layer to output a compressed 64-D latent representation of the
input image, which measures 227 × 227 pixels. The decoder,
designed as the inverse of the encoder, employs transposed
convolutional layers to reconstruct the original image from its
latent representation, effectively mirroring the encoding process.
The training of this autoencoder model leverages the Adam
optimizer [66], chosen for its efficiency with sparse gradients and
adaptability to different data distributions. To ensure stability
and precision in the learning process, a low learning rate of
1e-5 is used alongside a batch size of 256. The self-supervised
model is trained for 200 epochs when weights are initialized with
the value of AlexNet pretrained on the ImageNet dataset. Each

image in the dataset has been carefully cropped to feature only a
single object against a dark background, enhancing the object’s
visibility. The images are stored with an 8-bit depth in two chan-
nels, providing a simplified yet effective data structure. To filter
out irrelevant data, images captured before the device reaches
the sea surface, which often contain artifacts and negative depth
values, are automatically removed from the dataset. In addition,
conventional data transformation methods, such as rotating, flip-
ping, and shifting are used as well as random image cropping and
Gaussian noise. For example, the image is randomly flipped up
and down, left and right, and then, randomly, a zero-mean Gaus-
sian noise is added to the actual ROI. During the training phase,
further refinement of the input data includes the normalization
of all metadata, with location data converted into the Universal
Transverse Mercator coordinate system to facilitate uniformity
and precision in spatial computations. This normalization is
crucial for the subsequent calculation of Euclidean distances
between data points. A strategic approach to sampling is adopted
to enhance the model’s learning efficacy; half of the images
in each training batch are selected based on their proximity in
the metadata-defined space to a randomly chosen sample. This
selection is governed by a Gaussian distribution with varying
sigma values, ensuring a diverse yet representative sample pool.
The remaining images are selected randomly from across the
entire dataset. Since classification performance is affected when
dealing with a severely skewed dataset, we adopted an enhanced
F1-measure evaluation for multiclass classification tasks, fol-
lowing the methodology of [67]. Specifically, we calculate F1
scores for each category using a one-versus-rest approach [68],
culminating in a macroaverage that does not weigh classes by
their sample size. This approach ensures that no class is deemed
more significant than another regardless of its frequency in the
dataset, promoting fairness in model evaluation. To rigorously
test the generalizability of our model, we employed stratified
fivefold cross validation (CV). This method maintains the rel-
ative distribution of each class across all folds, ensuring that
our model is tested under varied conditions that closely mimic
real-world distributions. Furthermore, our experimental setup includes a comprehensive exploration of key parameters likely to affect feature learning and classification performance. We systematically varied the following.

1) The regularizer, defined by user-selected metadata criteria such as collection location, depth, or object size, each at four distinct levels.

2) The clustering algorithm, choosing between k-means, H-kmeans, or a random assignment approach.

3) The number of annotations for generating pseudolabels, tested at three different scales (500, 1000, and 2000) to evaluate robustness against varying amounts of training data.

4) The classifier type, where we tested several robust models including SVM, DT, RF, XG, and CNN, to identify which performs best across diverse scenarios.

These steps ensure that our findings are not only statistically valid but also broadly applicable, enhancing the credibility of our results in real-world applications. Through this meticulous experimental design, we aim to demonstrate the robustness and scalability of our proposed solution in the face of varying data characteristics and analytical conditions.

1) Regularizer: In the feature extraction phase, each sample is passed through a deep CNN backbone and projected into a 64-D latent point, incorporating regularization with metadata information. The representation learning objective involves two key components. First, we enhance the similarity between these embedded points by adjusting them in the direction of the average representation among them. Simultaneously, we give more weight to data points with more confident assignments through an auxiliary target distribution (i.e., the metadata). To examine whether this auxiliary information reveals any critical structural similarity in latent space, we quantify how much the loss regularizer function contributes to classification success. The results are compared with those from an identical network architecture without the regularizer. We set different experiments with user-defined metadata criteria as follows: location distances as *l* = 10 m, 100 m, 1 km, and 10 km, different depths as *d* = 1 m, 5 m, 10 m, and 100 m, and different scale parameters as *s* = 0.2, 0.4, 0.6, and 0.8. We only used one user-defined metadata criteria at a time as preliminary results on a smaller subset of the data suggested no improvement or even worsening of the results when multiple criteria are combined (data not shown), because of time limitations, and because discussion of the results is more straightforward. The score value results with these parameters for all experimental settings (including choice of clustering algorithm, number of annotations, and classifier) are reported in Tables I and II.

Based on our observation, incorporating the metadata auxiliary distribution as a regularizer in our deep autoencoder can generally enhance the learning process, resulting in accelerated classification performance compared to the absence of this regularizer (see Table I). In terms of classification performance, the depth regularizer exhibited superior results compared to the location, scale, and no regularizer approaches. Specifically, in one equal configuration G3, C3, K3, and M3, the depth regularizer achieved an impressive accuracy of 46.9%.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Evaluation of classification performance based on the regularization. Difference in F1-score of the different regularizer configurations (using H-kmeans and $M_2 = 2000$) compared to no regularization. Panels show regularization criteria: location (*l*), depth (*d*), and scale (*s*). Colors indicate classifier as shown in the legend.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. Evaluation of classification performance based on the clustering algorithm. Difference in F1-score of the non-DL classifiers with either depth regularization or no regularization ($M_2 = 2000$) compared to the random control (same regularization and number of annotated images). Panels show classifiers: DT, RF, SVM, and XG. Colors indicate regularization level as shown in the legend.</figcaption></figure>

In contrast, the location, scale, and no regularizer approaches achieved accuracies of 32.1%, 31.2%, and 30.0%, respectively. The impact of the regularizer across different metadata variations is depicted in Fig. 8(a) through a box plot. The average F1-score for varying depth values consistently surpassed the performance of other regularizers. Notably, when the depth was specifically configured to 10 m, it achieved the highest average F1-score of 50.5%. In terms of location, the performance demonstrated improvement as the distance increased. The highest efficiency was achieved with a distance of 1 km, resulting in a 35.9% F1-score. However, the overall mean for scale parameters negatively affected performance across most configurations when compared to not using scale regularization. The highest F1-scores achieved with scale regularization were 36.2% and 32.0%, respectively. Overall, these findings highlight the advantageous impact of the depth regularizer, while also showcasing the potential benefits of considering location in optimizing the model's performance. However, caution should be exercised when utilizing scale parameters, as their inclusion often results in diminished performance.

Based on our observations, incorporating the metadata auxiliary distribution as a regularizer in our deep autoencoder resulted in a mixed response in classification performance depending on the criteria, level and classifier used. For the following, reported values are based on the H-kmeans algorithm and an annotation number of $M_2 = 2000$. The depth regularizer enhanced the classification performance at any level (1, 5, 10, and 100 m) and for all classifiers, with an average F1-score increase of 11.6
<table><caption>TABLE I
PERFORMANCE OF VARIOUS NON-DL METHODS ON THE UVP6 BENCHMARK DATASET: F1-SCORES AND STANDARD DEVIATIONS (%)</caption><thead><tr><th rowspan="2">config</th><th rowspan="2">Regularisation</th><th rowspan="2">Classifier</th><th colspan="3">Random</th><th colspan="3">k-means</th><th colspan="3">H-kmeans</th></tr><tr><th>500</th><th>1000</th><th>2000</th><th>500</th><th>1000</th><th>2000</th><th>500</th><th>1000</th><th>2000</th></tr></thead><tbody><tr><td>A1</td><td>Location:</td><td>DT</td><td>16.5±5.5</td><td>17.5±4.1</td><td>17.2±3.9</td><td>20.1±2.6</td><td>21.7±5.1</td><td>22.3±6.1</td><td>21.6±4.4</td><td>21.5±4.9</td><td>23.1±2.3</td></tr><tr><td>A2</td><td>10m</td><td>RF</td><td>19.4±4.1</td><td>20.6±3.8</td><td>20.1±4.1</td><td>22.4±7.7</td><td>24.2±4.9</td><td>25.2±2.4</td><td>20.4±5.3</td><td>22.3±3.2</td><td>22.1±4.1</td></tr><tr><td>A3</td><td></td><td>SVM</td><td>21.4±5.2</td><td>21.6±5.4</td><td>22.7±4.3</td><td>25.5±6.1</td><td>26.6±7.5</td><td>28.2±2.0</td><td>27.4±3.3</td><td>30.2±4.1</td><td>32.4±1.7</td></tr><tr><td>A4</td><td></td><td>XG</td><td>20.9±4.6</td><td>21.3±3.4</td><td>21.8±3.1</td><td>24.1±4.2</td><td>25.4±5.7</td><td>27.3±1.5</td><td>25.1±5.2</td><td>26.9±2.8</td><td>29.1±3.3</td></tr><tr><td>B1</td><td>Location:</td><td>DT</td><td>18.7±2.8</td><td>19.2±3.2</td><td>19.0±5.2</td><td>21.9±6.7</td><td>20.4±6.2</td><td>20.9±3.8</td><td>23.2±7.2</td><td>26.3±4.5</td><td>26.1±2.3</td></tr><tr><td>B2</td><td>100m</td><td>RF</td><td>18.1±3.0</td><td>18.5±3.8</td><td>19.1±4.4</td><td>21.3±5.3</td><td>21.4±4.1</td><td>21.1±4.5</td><td>25.7±3.9</td><td>26.8±6.2</td><td>28.8±4.5</td></tr><tr><td>B3</td><td></td><td>SVM</td><td>26.6±3.7</td><td>27.1±3.5</td><td>27.5±5.7</td><td>30.3±7.3</td><td>32.1±1.7</td><td>32.6±3.8</td><td>29.5±2.4</td><td>30.6±6.0</td><td>35.5±3.6</td></tr><tr><td>B4</td><td></td><td>XG</td><td>27.5±4.4</td><td>27.8±4.7</td><td>28.3±4.2</td><td>31.2±4.1</td><td>32.5±2.7</td><td>33.2±1.9</td><td>27.6±5.2</td><td>28.2±5.1</td><td>28.1±3.7</td></tr><tr><td>C1</td><td>Location:</td><td>DT</td><td>19.2±5.1</td><td>19.9±2.7</td><td>20.2±2.4</td><td>23.3±2.7</td><td>23.6±4.1</td><td>27.2±3.3</td><td>28.8±7.1</td><td>30.2±5.8</td><td>31.6±1.0</td></tr><tr><td>C2</td><td>1km</td><td>RF</td><td>18.5±4.5</td><td>19.8±3.4</td><td>19.4±2.2</td><td>22.4±5.2</td><td>26.5±6.2</td><td>26.6±4.6</td><td>24.7±6.1</td><td>25.9±6.5</td><td>27.2±3.4</td></tr><tr><td>C3</td><td></td><td>SVM</td><td>25.0±3.1</td><td>26.1±4.1</td><td>25.7±3.3</td><td>28.5±2.9</td><td>31.2±3.5</td><td>33.8±1.9</td><td>32.1±2.9</td><td>34.2±3.2</td><td>35.9±2.5</td></tr><tr><td>C4</td><td></td><td>XG</td><td>25.2±5.2</td><td>25.9±3.2</td><td>25.4±3.7</td><td>29.1±6.8</td><td>33.2±4.0</td><td>33.7±4.1</td><td>33.8±3.5</td><td>35.1±3.8</td><td><strong>36.2±1.6</strong></td></tr><tr><td>D1</td><td>Location:</td><td>DT</td><td>19.5±3.6</td><td>20.6±3.5</td><td>21.0±4.2</td><td>22.6±2.5</td><td>26.5±5.7</td><td>29.4±4.0</td><td>27.8±3.5</td><td>29.1±2.8</td><td>31.2±0.5</td></tr><tr><td>D2</td><td>10km</td><td>RF</td><td>17.3±3.7</td><td>18.9±3.4</td><td>18.6±2.7</td><td>21.2±1.4</td><td>23.6±3.2</td><td>23.9±1.7</td><td>25.3±2.6</td><td>25.9±4.7</td><td>28.8±2.2</td></tr><tr><td>D3</td><td></td><td>SVM</td><td>24.5±2.9</td><td>24.7±3.1</td><td>25.7±3.6</td><td>28.1±6.2</td><td>30.1±5.3</td><td>31.2±2.7</td><td>31.8±4.4</td><td>31.8±1.5</td><td>32.1±4.1</td></tr><tr><td>D4</td><td></td><td>XG</td><td>24.1±4.1</td><td>24.2±2.8</td><td>23.4±2.8</td><td>26.3±5.7</td><td>26.8±4.0</td><td>28.1±3.8</td><td>32.2±1.2</td><td>33.8±5.0</td><td>34.1±2.7</td></tr><tr><td>E1</td><td>Depth:</td><td>DT</td><td>23.1±4.8</td><td>24.2±2.8</td><td>23.4±2.8</td><td>28.2±1.3</td><td>29.5±2.4</td><td>32.1±1.6</td><td>30.3±4.1</td><td>36.5±1.2</td><td>39.3±0.8</td></tr><tr><td>E2</td><td>1m</td><td>RF</td><td>23.8±3.9</td><td>24.4±2.8</td><td>25.2±2.2</td><td>26.8±2.5</td><td>28.4±4.1</td><td>30.4±1.2</td><td>25.6±3.1</td><td>29.1±1.3</td><td>31.8±1.5</td></tr><tr><td>E3</td><td></td><td>SVM</td><td>33.1±4.4</td><td>34.5±1.9</td><td>34.6±2.1</td><td>36.5±5.9</td><td>40.7±1.9</td><td>41.4±1.3</td><td>44.1±2.6</td><td>46.3±2.8</td><td>49.5±2.6</td></tr><tr><td>E4</td><td></td><td>XG</td><td>32.4±5.1</td><td>33.5±2.6</td><td>33.2±3.1</td><td>35.4±4.7</td><td>38.1±2.8</td><td>39.7±1.9</td><td>39.6±1.7</td><td>40.1±5.1</td><td>42.8±0.2</td></tr><tr><td>F1</td><td>Depth:</td><td>DT</td><td>24.9±3.6</td><td>26.3±3.1</td><td>25.8±2.9</td><td>28.4±2.4</td><td>28.2±5.0</td><td>30.7±2.9</td><td>32.8±3.9</td><td>34.7±2.2</td><td>36.8±1.9</td></tr><tr><td>F2</td><td>5m</td><td>RF</td><td>26.3±4.2</td><td>27.4±2.7</td><td>27.0±3.2</td><td>29.5±3.3</td><td>30.2±2.1</td><td>33.9±0.2</td><td>28.3±3.2</td><td>28.5±3.8</td><td>29.6±2.7</td></tr><tr><td>F3</td><td></td><td>SVM</td><td>30.2±3.8</td><td>30.6±3.0</td><td>30.7±2.7</td><td>33.2±6.0</td><td>36.1±4.2</td><td>37.6±4.4</td><td>36.5±6.9</td><td>39.1±7.1</td><td>42.2±3.9</td></tr><tr><td>F4</td><td></td><td>XG</td><td>30.6±2.5</td><td>31.7±2.6</td><td>32.0±2.5</td><td>34.2±5.1</td><td>35.2±1.7</td><td>35.6±5.3</td><td>35.1±5.1</td><td>38.2±3.0</td><td>42.3±1.1</td></tr><tr><td>G1</td><td>Depth:</td><td>DT</td><td>29.3±2.8</td><td>30.1±2.4</td><td>30.4±2.6</td><td>33.4±5.4</td><td>38.2±2.7</td><td>39.1±0.4</td><td>39.8±5.4</td><td>41.9±3.3</td><td>43.2±1.7</td></tr><tr><td>G2</td><td>10m</td><td>RF</td><td>27.4±3.4</td><td>28.4±3.8</td><td>29.2±3.4</td><td>30.5±5.1</td><td>33.8±4.2</td><td>34.2±1.5</td><td>34.6±4.2</td><td>36.2±3.6</td><td>39.9±2.1</td></tr><tr><td>G3</td><td></td><td>SVM</td><td>38.4±2.8</td><td>38.8±4.1</td><td>39.5±2.6</td><td>41.5±2.4</td><td>44.4±3.4</td><td>45.2±2.0</td><td>46.9±3.2</td><td>47.2±2.0</td><td><strong>50.5±1.8</strong></td></tr><tr><td>G4</td><td></td><td>XG</td><td>36.9±1.7</td><td>38.6±2.8</td><td>38.0±1.1</td><td>40.2±5.1</td><td>41.1±1.1</td><td>42.0±0.4</td><td>45.6±4.5</td><td>46.2±1.5</td><td>47.1±2.5</td></tr><tr><td>H1</td><td>Depth:</td><td>DT</td><td>26.5±4.3</td><td>27.3±3.7</td><td>28.1±3.1</td><td>30.1±2.5</td><td>32.0±1.9</td><td>32.1±2.2</td><td>37.1±4.7</td><td>38.2±3.6</td><td>38.2±1.2</td></tr><tr><td>H2</td><td>100m</td><td>RF</td><td>29.1±2.8</td><td>29.8±2.6</td><td>30.7±2.9</td><td>32.5±3.1</td><td>33.5±2.0</td><td>34.1±2.6</td><td>35.4±3.6</td><td>36.2±2.9</td><td>37.1±2.2</td></tr><tr><td>H3</td><td></td><td>SVM</td><td>29.3±4.6</td><td>31.1±3.6</td><td>29.9±3.5</td><td>32.8±6.2</td><td>35.2±5.1</td><td>36.5±2.3</td><td>37.1±6.2</td><td>39.2±5.2</td><td>41.8±4.3</td></tr><tr><td>H4</td><td></td><td>XG</td><td>26.3±2.8</td><td>27.0±2.4</td><td>27.4±3.2</td><td>30.1±7.1</td><td>31.8±4.9</td><td>33.7±3.9</td><td>38.9±4.3</td><td>39.3±2.8</td><td>40.1±2.4</td></tr><tr><td>I1</td><td>Scale:</td><td>DT</td><td>15.8±2.9</td><td>16.9±2.8</td><td>16.3±4.2</td><td>18.3±2.4</td><td>17.7±5.1</td><td>18.5±2.8</td><td>20.1±3.1</td><td>23.6±2.4</td><td>24.6±1.5</td></tr><tr><td>I2</td><td>0.2</td><td>RF</td><td>12.8±3.5</td><td>13.6±3.8</td><td>13.4±3.4</td><td>16.2±6.8</td><td>18.9±3.0</td><td>19.4±1.7</td><td>18.3±2.0</td><td>20.9±2.5</td><td>22.4±0.7</td></tr><tr><td>I3</td><td></td><td>SVM</td><td>19.8±4.2</td><td>21.2±2.8</td><td>20.3±2.8</td><td>23.2±4.6</td><td>24.1±1.6</td><td>25.2±1.8</td><td>24.4±5.1</td><td>25.8±2.9</td><td>25.5±3.1</td></tr><tr><td>I4</td><td></td><td>XG</td><td>17.2±3.5</td><td>18.2±5.2</td><td>18.0±4.3</td><td>21.8±1.5</td><td>21.5±2.0</td><td>22.0±1.7</td><td>25.5±6.2</td><td>25.1±4.1</td><td>26.6±3.3</td></tr><tr><td>J1</td><td>Scale:</td><td>DT</td><td>15.5±2.7</td><td>16.8±2.1</td><td>17.1±3.6</td><td>18.3±3.2</td><td>20.7±1.2</td><td>20.5±1.3</td><td>19.4±2.9</td><td>19.6±1.1</td><td>20.0±1.2</td></tr><tr><td>J2</td><td>0.4</td><td>RF</td><td>12.7±4.1</td><td>13.8±4.1</td><td>13.9±2.7</td><td>16.6±2.3</td><td>18.6±2.4</td><td>20.8±1.6</td><td>16.2±1.4</td><td>17.9±1.5</td><td>19.6±0.9</td></tr><tr><td>J3</td><td></td><td>SVM</td><td>18.6±2.8</td><td>19.2±2.8</td><td>20.9±3.9</td><td>21.4±5.6</td><td>22.3±2.6</td><td>22.2±2.4</td><td>23.1±2.5</td><td>25.1±3.1</td><td>25.7±2.7</td></tr><tr><td>J4</td><td></td><td>XG</td><td>20.6±2.9</td><td>21.5±3.1</td><td>21.3±3.3</td><td>22.4±1.1</td><td>23.1±3.0</td><td>24.8±2.5</td><td>20.2±5.1</td><td>21.3±3.5</td><td>22.5±0.9</td></tr><tr><td>K1</td><td>Scale:</td><td>DT</td><td>19.8±2.2</td><td>21.7±2.6</td><td>20.9±2.9</td><td>22.5±1.7</td><td>26.5±1.1</td><td>26.2±0.5</td><td>24.5±2.3</td><td>26.8±2.2</td><td>28.0±1.6</td></tr><tr><td>K2</td><td>0.6</td><td>RF</td><td>19.8±3.7</td><td>21.1±1.9</td><td>21.0±3.3</td><td>23.2±2.8</td><td>25.2±2.3</td><td>28.9±2.2</td><td>26.1±2.8</td><td>29.2±4.2</td><td>30.4±1.4</td></tr><tr><td>K3</td><td></td><td>SVM</td><td>27.6±5.2</td><td>29.1±3.5</td><td>29.7±2.5</td><td>30.2±5.2</td><td>31.3±4.5</td><td>32.8±0.8</td><td>31.2±7.8</td><td>33.2±6.1</td><td><strong>36.2±2.1</strong></td></tr><tr><td>K4</td><td></td><td>XG</td><td>21.9±2.7</td><td>24.4±4.4</td><td>25.5±4.0</td><td>26.1±2.3</td><td>26.6±3.4</td><td>29.2±2.1</td><td>28.6±1.8</td><td>30.5±2.1</td><td>31.5±3.2</td></tr><tr><td>L1</td><td>Scale:</td><td>DT</td><td>20.5±4.2</td><td>21.0±3.9</td><td>21.6±2.6</td><td>23.2±2.9</td><td>25.3±1.1</td><td>26.5±1.3</td><td>22.6±3.9</td><td>26.3±1.8</td><td>29.7±2.4</td></tr><tr><td>L2</td><td>0.8</td><td>RF</td><td>19.5±2.6</td><td>21.0±2.7</td><td>19.9±3.7</td><td>21.6±3.4</td><td>22.9±3.1</td><td>24.6±2.2</td><td>22.3±1.5</td><td>24.7±1.2</td><td>26.1±1.3</td></tr><tr><td>L3</td><td></td><td>SVM</td><td>23.4±1.3</td><td>24.4±2.6</td><td>25.5±3.5</td><td>25.7±6.0</td><td>27.1±3.4</td><td>29.1±1.6</td><td>24.4±3.5</td><td>26.7±5.2</td><td>26.1±2.6</td></tr><tr><td>L4</td><td></td><td>XG</td><td>23.4±2.8</td><td>26.1±3.8</td><td>25.2±2.4</td><td>25.5±5.2</td><td>26.8±3.1</td><td>28.2±0.2</td><td>25.9±6.3</td><td>28.5±3.5</td><td>30.2±1.9</td></tr><tr><td>M1</td><td>No</td><td>DT</td><td>22.2±4.3</td><td>23.1±3.8</td><td>22.7±3.6</td><td>26.1±3.0</td><td>26.9±2.2</td><td>27.2±2.1</td><td>27.2±2.5</td><td>26.7±2.7</td><td>30.3±0.8</td></tr><tr><td>M2</td><td>regularisation</td><td>RF</td><td>20.0±3.5</td><td>20.3±4.2</td><td>20.5±3.1</td><td>24.1±2.8</td><td>24.8±2.3</td><td>25.8±2.5</td><td>25.2±2.9</td><td>25.5±2.1</td><td>26.8±1.6</td></tr><tr><td>M3</td><td></td><td>SVM</td><td>22.7±4.2</td><td>23.7±3.6</td><td>23.5±2.6</td><td>26.9±3.5</td><td>28.4±4.1</td><td>29.5±2.6</td><td>30.0±6.2</td><td>31.7±3.2</td><td><strong>32.0±1.6</strong></td></tr><tr><td>M4</td><td></td><td>XG</td><td>23.9±3.8</td><td>24.8±2.7</td><td>24.9±3.2</td><td>26.2±3.9</td><td>26.5±4.3</td><td>27.0±0.2</td><td>29.0±3.3</td><td>29.4±4.7</td><td>31.0±3.0</td></tr></tbody></table>

Best performer for each metadata is highlighted in Bold. All results are based on multiview grouping technique. DT represents Decision Tree, RF for Random Forest, SVM for Support Vector Machines, and XG for XGBoost.

units (range: 2.8–20.3%pt). On average, level 10 m resulted in the best classification results (score increase of 15.9 ± 2.8 units). Specifically, for the non-DL classifiers, it achieved the highest average F1-score (50.5%) when configured to 10 m. For the CNN classifier, it also achieved the highest average F1-score (54.4%) albeit for a configuration with *d* = 1 m. For the location regularizer, there was generally a positive trend between location distance level and F1-score improvement, although consistent F1-score improvements across all classifiers were only achieved for levels 1 and 10 km (average 3.4 ± 2.5 and 2.7 ± 2.9 units improvement, respectively). The scale regularizer generally

worsened the classification performance with a mean F1-score difference of –3.15 units (range –10.3–6.0%pt). The impact of the regularizer across different metadata variations is depicted in Fig. 8(a) through a box plot. Overall, these findings highlight the advantageous impact of the use of metadata-based regularizers in optimizing the model’s classification performance. However, caution should be exercised when choosing the metadata parameters as they can also—as shown by the scale criteria—result in diminished classification performance. Potential ecological reasons for the observed change in classification performance when using the different criteria are discussed later (see Section V).
<table><caption>TABLE II
RESULTS OF THE CNN METHOD USING SINGLE OR MULTIPLE VIEWS PER CLASS WITH HIERARCHICAL K-MEANS DATA SELECTION ON THE UVP6 BENCHMARK DATASET: F1-SCORES AND STANDARD DEVIATIONS (%)</caption><thead><tr><th rowspan="2">config</th><th rowspan="2">Regularization</th><th rowspan="2">classifier</th><th colspan="4">Single view</th><th colspan="3">Multi-view</th></tr><tr><th>500</th><th>1000</th><th>2000</th><th>500</th><th>1000</th><th>2000</th></tr></thead><tbody><tr><td>N1</td><td>Location: 10m</td><td>CNN</td><td>29.7±1.8</td><td>33.6±3.8</td><td>36.0±0.6</td><td>31.7±2.2</td><td>36.5±1.3</td><td>38.3±2.4</td></tr><tr><td>N2</td><td>Location: 100m</td><td>CNN</td><td>30.8±2.1</td><td>33.4±2.6</td><td>36.4±1.2</td><td>32.6±1.2</td><td>35.7±2.2</td><td>39.1±2.3</td></tr><tr><td>N3</td><td>Location: 1km</td><td>CNN</td><td>33.5±1.1</td><td>35.6±1.5</td><td>38.4±0.3</td><td>35.3±2.7</td><td>37.6±2.6</td><td>40.3±2.4</td></tr><tr><td>N4</td><td>Location: 10km</td><td>CNN</td><td>34.9±2.6</td><td>35.1±1.1</td><td>39.0±0.1</td><td>36.4±2.4</td><td>37.3±2.5</td><td><b>41.6±1.2</b></td></tr><tr><td>O1</td><td>Depth: 1m</td><td>CNN</td><td>45.6±1.2</td><td>46.3±2.2</td><td>52.9±1.4</td><td>48.3±2.3</td><td>49.2±1.1</td><td><b>54.4±0.5</b></td></tr><tr><td>O2</td><td>Depth: 5m</td><td>CNN</td><td>34.9±1.4</td><td>39.5±1.3</td><td>39.3±0.2</td><td>37.3±0.5</td><td>41.6±2.4</td><td>42.3±2.1</td></tr><tr><td>O3</td><td>Depth: 10m</td><td>CNN</td><td>46.8±1.4</td><td>47.0±1.6</td><td>50.7±0.8</td><td>49.3±1.5</td><td>49.5±1.7</td><td>53.9±1.6</td></tr><tr><td>O4</td><td>Depth: 100m</td><td>CNN</td><td>36.3±3.2</td><td>40.0±2.1</td><td>44.4±0.6</td><td>39.2±2.6</td><td>42.3±1.8</td><td>46.6±1.5</td></tr><tr><td>P1</td><td>Scale: 0.2</td><td>CNN</td><td>26.4±2.2</td><td>27.5±3.4</td><td>29.1±1.5</td><td>28.7±3.5</td><td>29.3±2.2</td><td>30.3±0.2</td></tr><tr><td>P2</td><td>Scale: 0.4</td><td>CNN</td><td>25.2±2.3</td><td>25.0±0.2</td><td>25.9±0.8</td><td>25.2±2.1</td><td>26.7±1.3</td><td>28.7±1.2</td></tr><tr><td>P3</td><td>Scale: 0.6</td><td>CNN</td><td>30.3±3.4</td><td>34.8±2.3</td><td>38.8±1.2</td><td>34.5±1.5</td><td>37.2±2.5</td><td><b>40.1±2.3</b></td></tr><tr><td>P4</td><td>Scale: 0.8</td><td>CNN</td><td>24.3±1.2</td><td>27.6±1.2</td><td>27.7±1.4</td><td>26.4±2.1</td><td>30.1±2.6</td><td>29.7±1.5</td></tr><tr><td>Q1</td><td>No regularization</td><td>CNN</td><td>30.9±2.4</td><td>32.4±1.7</td><td>33.3±1.5</td><td>31.9±2.5</td><td>33.4±1.4</td><td>34.1±1.6</td></tr></tbody></table>

Best performing metrics are highlighted in bold.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Evaluation of classification performance based on the number of manually annotated images ($M_2$). (a) Difference in F1-score of all CNN + H-kmeans configurations compared to the CNN + H-kmeans control (no regularization, 500 annotated images). Panels show regularization criteria depth ($d$), location ($l$), no regularization (N) and scale ($s$). Colors indicate regularization level as shown in the legend. (b) Top performing model configurations in terms of overall F1-score. Colors indicate number of annotated samples as red = 500, yellow = 1000, and blue = 2000. Configuration details are stated on the y-axis. Error bars show the standard deviation of model reruns.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7. Effect of the multiview setting on the classification performance. Difference in F1-score of all CNN + H-kmeans configurations with multiview compared to single-view for regularization criteria depth ($d$; blue), location ($l$; red), no regularization (N; gray), and scale ($s$; yellow).</figcaption></figure>

2) *Clustering Algorithm*: We undertook a comprehensive comparison of different methods for selecting representative samples from our dataset. Specifically, we explored the effectiveness of k-means clustering, H-kmeans clustering, and random selection. Our primary objective was to identify a specific data group suitable for fine tuning while ensuring that all classes present in the entire dataset were included. To do that, we took into account the taxonomy tree of plankton and focused on 23 coarse classes that represented the entire dataset. Instead of performing a computationally expensive grid search considering all possible values, we opted for a fixed number of 46 clusters (twice the number of classes) to provide the algorithm with enough flexibility to consider smaller subsets.

For the reduction technique aimed at decreasing the number of majority-class samples (i.e., detritus images), we determine the number of images to retain from each majority-class cluster (group A) by testing a range (ten values between 1 and 50%) of different ratios between the number of samples belonging to group A (Size$_A$) and the combined size of groups B and group C (Size$_{BC}$)

$$R^i = \frac{Size_A^i}{Size_{BC}^i} \qquad (5)$$

where $i$ denotes the iteration. A larger value of $R^i$ indicates a cluster with more samples from group A and fewer from group BC, aligning it with the majority class. Conversely, a smaller $R^i$ suggests a cluster with more Group-BC class samples, deviating from majority class characteristics. Our goal is to identify a tradeoff value for $R^i$ that balances these considerations [69]. To quantify the tradeoff, we continue reducing until all our ten target classes are found in representative samples. The selected reduction value of 90% was determined through experimentation and yielded the best tradeoff based on the number of nondetritus samples found in representative samples selected by H-kmeans. This 90% reduction value is consistently applied across all configurations in CV tests to ensure a fair comparison.

To ensure fairness and draw conclusive comparisons about which clustering algorithm performs best, we maintained consistent configurations for our data selection algorithms. For instance, we utilized the same set of 64-D features extracted from a
<figure><img src="image_9.png" /><figcaption>Fig. 8. Effect of the input parameters in box plots where the first quartile is at the bottom, and the third quartile is at the top: (a)–(c) Box plots of the F1-score for various input regularizers, classifiers, and data selection methods, respectively.</figcaption></figure>

pretrained deep autoencoder across all approaches. In the case of k-means, both the hierarchical and nonhierarchical variants, we applied identical parameter settings. To account for the potential impact of initialization, we executed each algorithm ten times, employing different centroid seeds in each run. In addition, we took into consideration the class frequencies of the inputs to balance the weighting for clustering. By adhering to consistent configurations, running multiple trials, and accounting for class frequencies, we aimed to minimize any potential sources of bias and obtain reliable and conclusive comparisons between the different data selection algorithms.

Both clustering algorithms, k-means and H-kmeans, improved classification accuracy compared to the random selection with an average improvement of 5.1 ± 1.9%pt and 8.0 ± 3.4%pt, respectively (all configurations, *n* = 52). H-kmeans yielded better classification performance in 45 out of the 52 tested configurations, with an average performance difference of 2.8 ± 2.9%pt. The highest classification performance gains were observed for the regularization with depth (4.6 ± 2.9%pt compared to k-means and 10.4 ± 3.3%pt compared to random, all levels), specifically for the level *d* = 1 m for the classifiers SVM and DT (respectively, +8.1%pt and +7.2%pt compared to k-means) (see Fig. 5). The gains for the other regularizers, compared, respectively, to random and k-means, were 7.8 ± 3.2%pt and 2.3 ± 3.2%pt for location, 5.9 ± 2.6%pt and 1.6 ± 2.5%pt for scale, and 7.1 ± 1.1%pt and 2.6 ± 1.3%pt for no regularization.

3) *Number of Annotated Samples:* Before running predictions on the unlabeled data, manual human-led annotations were performed on a subset of representative sample views, referred to as *M*₂-selected samples. F1-scores are reported for different numbers of annotated samples, specifically *M*₂ = 500, 1000,

2000 (see Tables I and II). Increasing the number of annotated samples generally results in higher scores: a change from 500 to 2000 samples (H-kmeans and all model configurations) resulted in an average increase of 3.7%pt (range 0.3–16.1%pt; Tables I and II). However, for some model configurations, the additional 1000 images from *M*₂ = 1000 to *M*₂ = 2000, which effectively doubles the workload for the human expert taxonomist, starts to have a diminishing return [e.g., for “No regularization,” “*d* = 5 m,” “*l* = 10 m,” and “*s* = 0.8” using the CNN classifier; see Fig. 6(a)]. For the best model configuration with the CNN classifier (*d* = 1 m and 10 m), more annotation would likely be a good investment of time as there is no tailing off yet, indicating that further annotation could improve the classification accuracy. Nevertheless, our ultimate objective is to achieve satisfactory performance while minimizing the number of annotations required. While employing 2000 annotations yields a high score, certain configurations outperform others even with a smaller number of annotations. For example, in the top ten configurations in terms of classification performance (out of 585 tested configurations), two configurations used only 500 manual annotations [CNN + hierarchical + *d* = 10 m and SVM hierarchical + *d* = 10 m; see Fig. 6(b)], hence outperforming the majority of configurations with *M*₂ = 2000. These findings underscore that, with an optimized configuration, even with very limited input from human expert taxonomists, our semisupervised metadata-guided model achieves high classification performance.

4) *Classifier:* We conducted a comparative study involving various well-known classic machine learning techniques, which are commonly employed to advance machine learning models. These methods can be broadly categorized into two distinct groups.
a) *Group 1:* The first group encompasses commonly used methods that operate directly on latent space vectors. We utilized SVM, a popular algorithm that learns a nonlinear model using the kernel trick [70]. To ensure effective classification boundaries, we specifically chose the radial basis function (RBF) kernel with a large degree. We incorporated DT, which make predictions by traversing from the root node to a leaf node based on feature conditions [71]. DTs are known for their interpretability and can capture complex decision boundaries. Another approach employed in our study is RF, which involves constructing an ensemble of DTs that are randomly generated [72]. This randomness and aggregation of multiple trees often lead to improved predictive accuracy compared to a single decision. Finally, we employed a gradient-boosting algorithm called XG. This powerful classification model combines the predictions of multiple base estimators to enhance overall robustness and performance [73]. XG leverages the greedy boosting strategy to iteratively refine the model's predictions.

Overall, we observed a trend where the two SVM and XG classifiers tended to outperform the two DT and RF classifiers (Fig. 3 in Appendix). For the specific configuration with a depth-regularization of $d = 1$ m (E configuration), hierarchical clustering, and $M_2 = 500$ samples, we observed notable variations in performance among the classifiers. In this setup, SVM outperformed the other three approaches, achieving an F1-score of 44.1%. In comparison, the DT obtained an F1-score of 30.3%, RF scored 25.6%, and XG achieved a score of 39.6% (see Table I). In the comparison between SVM and XG, an SVM consistently demonstrated slightly superior performance compared to XG. However, there were specific setups where XG exhibited an average advantage of approximately 1%pt in mean scores. For instance, when considering the specified configuration for clustering and the number of samples, the SVM outperformed 8 out of 13 samples across all metadata variations. An example of the superiority of XG can be observed in the C configuration (location regularization $l = 1$ km), where XGBoost achieved an F1-score of 33.8%, while the SVM obtained a slightly lower F1-score of 32.1% (see Table I and Fig. 8(b) for detailed results).

b) *Group 2:* The second group comprises methods that are based on deep convolutional networks. In our study, we employed the ResNet18 architecture for fine tuning our model. This architecture introduces skip connections to preserve the gradient flow, leading to a significant performance boost [74]. By comparing the single and multiview suggested approaches, we showcased the classification performance of various configurations utilizing the CNN model within the N-Q setup. Overall, the most remarkable performances are observed in configurations O1 and O3, corresponding to a CNN configuration with depths of 1 m (with an F1-score of 54.4 ± 0.5%) and 10 m (with an F1-score of 53.9 ± 1.6%), respectively. Notably, the performance difference between these configurations is minimal and within the standard errors.

Finally, we investigated the effect of multiple views compared to a single-view setting for the CNN classifier (using H-kmeans). Across all settings with various regularizers and numbers of annotated images, multiview outperformed single view by an average of 2.1 ± 0.8%pt (range: 0-4.2%pt; Fig. 7). In addition,

we demonstrated the benefits of utilizing multiple views for specific classes by examining the performance per class (see Fig. 1 in Appendix). Notably, the utilization of multiview data has resulted in improved average scores for the detritus and *Rhizaria* classes, which exhibit a wide range of variability. As a result, the overall average score has seen an increase of approximately 2-3%pt. This finding highlights the advantage of incorporating multiple views in the classification process, particularly for classes that exhibit greater variation within their samples.

## V. DISCUSSION

### A. Use of Metadata to Help Classification

For traditional microscope-based zooplankton identification, metadata provides the taxonomist with vital information that aids the correct species identification, such as the size of the organism and the location and water depth where it was found. In contrast, however, the usefulness of metadata for our unsupervised classification framework was mixed (see Fig. 4). We suspect that size was not useful information as our classification categories are too broad, leading to a considerable overlap in the sizes of different categories. For example, copepods range in size from <0.1 to 18 mm, hence spanning four orders of magnitude, while the size range of amorphous detritus is even larger, practically spanning from microscopic to several cm. Hence, at broad classification levels such as ours, size appears to hinder rather than help the classification. In settings with higher taxonomic resolution that include classes with distinct size ranges (e.g., species level), however, we expect size to increase classification accuracy. Location appeared to be more useful at larger search distances (>1 km radius), possibly reflecting the patchiness of plankton in the ocean in terms of geographical distribution. Different to traditional taxonomy, we did not use absolute location (e.g., equatorial Pacific versus North Atlantic) to aid classification but rather the relative location of two objects to each other. Such an approach should be sufficiently sensitive to capture geographical changes in plankton distribution caused by, e.g., climate change [75], [76], without enforcing strict location constraints that could conceal distribution changes. Finally, depth information greatly improved classification accuracy likely because plankton of the same species often swarm, colocating in relatively narrow depth bands of just a few meters (e.g., [77], [78], and [79]). A class-based analysis shows that depth information is particularly useful for aiding the classification of copepods and *foraminifera* (see Fig. 9), both of which are often found in distinct depth layers [79], [80]. For detrital particles, their shape and type typically also change with depth (e.g., [81] and [82]) as particles are reworked and become more refractory the further they are away from the surface ocean, from where they originate. An interesting future addition in metadata would be the combination of the sampling depth and the time of day when sampling occurred (relative to sunset/sunrise) as many zooplankton species undergo diel vertical migration, feeding near the surface during night and resting at depth (often >500 m depth) during the day. Yet, as shown by size, any metadata criteria used for regularization has to be chosen with care and checked for ecological meaning.
<figure><img src="image_10.png" /><figcaption>Fig. 9. Boxplots showing the F1-score from all model configurations for the ten classes. Boxplots indicate median (line), upper and lower quantile (box), minimum and maximum (error bars), and individual data points (solid dots). (a)–(j) Comparison of F1-score with regularization for location, depth, scale, and no regularization. Note the difference in the y-axis scale. The F1-scores for the best model, under the O1 configuration, are as follows: Detritus = 75.2, Puff = 73.1, Rhizaria = 48.5, FI (Fiber-Filament) = 77.6, Copepoda = 46.6, Artifact = 92.5, Eumalacostraca = 27.2, Chaetognatha = 29.3, Foraminifera = 36.1, and Salpida = 37.8. (k) Comparison of the F1-score of all classes using all model configurations.</figcaption></figure>

## B. Reduced Need for Human Expert Annotation

Our model framework dramatically reduced the work for human expert annotation to 1000 images ($M_1 + M_2 = 500 + [500, 1000, 2000]$) for a dataset of 450 000 images. The question is whether the resulting classification is useful for ecological purposes. In all configurations, our self-supervised model algorithms detected ten common classes (detritus, fibre/filaments, *copepoda*, artifact, puff, *rhizaria*, *eumalacostrata*, *chaetognatha*, *foraminifera*, and *salipda*). These common classes are among the 13 most abundant classes in our dataset. Certain rare classes, such as *ostracoda*, *actinopterygii* (ray-finned fish), or *appendicularia* (which were only represented by 28 samples), were also detected in specific configurations. Our best-performing configuration (O1) found 16 of the 24 classes (it also found *larvacean* houses, *aulacanthidae*, *ostracoda*, *Creseis acicula*, *actinopterygii*, and *Aulattractus*). Hence, the SSL appears adept in recognizing the main structure of the plankton community across our dataset. The F1-scores for individual classes varied widely between model configurations and classes [see Fig. 9(k)]. Notably, the class “ artifact,” attained the highest average F1-score of 75%, underscoring its utility in data quality control and in scenarios where the data are

employed for class-independent analyses, such as community size spectra [83].

To demonstrate how well our model configurations perform, we calculated the F1-score for two classification options: 1) the classifier assigns a label at random with an equal chance for all classes, or 2) the classifier assigns detritus to all images. For the random labeling, the F1-score ranges from 0.01 to 8.3% per class with a mean of 1.2%. For the all-detritus, although the F1-score for the detritus class is 90.8%, the mean score is 3.9%. Hence, both models perform much worse than our best-performing configuration, O1 (mean 54.4%; albeit having a mean of 34.7% if we consider the F1-score for undetected classes as 0%).

The ability to detect classes and the overall classification performance are dependent on the number of images for each class in the representative samples for human expert annotation ($M_1$ and $M_2$). $M_1$ and $M_2$ are selected by H-kmeans, whereby the algorithm tries to find key features for detecting and distinguishing classes, and their composition is, hence, less linked to the relative abundance of the individual classes in the entire dataset. As a result, many of the configurations showed the beginning of “leveling off” as $M_2$ increases, e.g., the control configuration (no regularization; Fig. 6). For our best-performing model configuration (O1 with $M_2 = 2000$),
<figure><img src="image_11.png" /><figcaption>Fig. 10. Comparison of object processing speeds across different annotation methods. This plot illustrates the processing speeds of human manual classification, AI-assisted manual classification (Ecotaxa [6]), semisupervised (MorphoCluster [7]), and our proposed method, showcasing the average number of objects processed per human-hour and the range for each method. Error bars indicate the minimum and maximum throughput.</figcaption></figure>

we found a weak relationship between the overall abundance and F1-score for each class in the entire dataset (linear regression: $R^2 = 0.35$, $p = 0.055$, and $n = 9$, when not including detritus), indicating the importance of selecting representative samples that capture the diversity of features across classes. When $M_2$ was increased further, from 2000 to 3000, the F1-score only slightly improved (data not reported), hence showing the “leveling off” associated with the model finding more of the relevant representative samples. Overall, we demonstrate that SSL is effective at learning the key features relevant for plankton classification and can, hence, dramatically reduce the required input from human expert taxonomists.

In the rapidly evolving domain of automated annotation, the effectiveness of different methodologies can be quantitatively compared by their ability to save human-time and enhance throughput. Traditional manual annotation techniques typically process between 300 and 1000 objects per hour (pers. comm. and [7]). Although supervised platforms such as EcoTaxa [6] improve on this by allowing for a sorting speed ranging from 300 to 15 000 objects per hour, depending on the level of automation and manual validation involved [7], these numbers are still modest when compared to more sophisticated methods. For instance, the interactive semisupervised approach used by MorphoCluster [7] markedly increases efficiency, reaching speeds of approximately 17 000 objects per hour of human annotation. Our method, however, represents a significant leap forward, demonstrating the ability to process around 200 000 to 450 000 objects per hour of human annotation, as illustrated in Fig. 10. In terms of class management, large number of classes (e.g., see MorphoCluster [7]) often include the same class in different orientations, or mixed classes (in between two pure classes, such as copepods and detritus), which are often subsequently merged by the plankton researcher for their analysis. In our approach, such “pseudoclasses” would be merged, which is not only computationally efficient but also aligns closely with the needs of final ecological interpretation, where such distinctions are often unnecessary and combined for broader ecological insights.

The methodology aims to diminish the reliance on extensive human annotation, which is particularly beneficial for large-scale monitoring projects where manual annotation of vast amounts of data is impractical. By automating this process, resources can be reallocated to other critical tasks within marine research. The model is well suited for deployment on edge devices commonly used in marine environments, such as AUVs, floats, or gliders. These devices benefit from the model’s real-time processing capabilities, which reduce the need for extensive data transmission back to shore-based systems, allowing for immediate decision making directly in the field. However, it is critical to recognize that the model’s operational efficiency comes with a tradeoff in the granularity of data classification. Operating with a minimal amount of data embedding—while advantageous for conserving bandwidth and reducing data transmission over satellite—inevitably leads to a reduction in the resolution of data classification. This resolution reduction predominantly affects the classification of less common species, which could be a significant limitation for ecologists requiring detailed taxonomic analysis across a diverse range of plankton types. However, the use of more powerful platforms that can afford higher computational resources and energy, could be considered. Such devices would enable the use of more complex models without the constraints of lesser-equipped platforms, thereby enhancing both the depth and breadth of marine ecological studies.

In our study, variability in plankton image classification primarily stems from fluctuations in environmental conditions such as lighting, water clarity, or sensor discrepancies, similar to spectral variability observed in many remote sensing scenarios [84]. Such variability significantly impacts the performance of traditional image classification models by altering the visual appearance of plankton, leading to inconsistencies and inaccuracies in species identification and quantification. To address this, our approach integrates metadata to enhance representation learning, enabling the model to contextualize each image and adapt its classification decisions to specific environmental conditions. This methodology not only increases the model’s resilience against variable conditions but also ensures effectiveness under diverse and challenging scenarios. Applied to datasets characterized by high levels of variability, we anticipate that the model would demonstrate superior resilience compared to traditional methods, as the use of environmental metadata facilitates effective normalization of imaging differences. Looking forward, investigating the impacts of additional environmental variables such as water temperature, salinity, and light intensity on classification accuracy presents a promising avenue for future research. Furthermore, developing adaptive algorithms that dynamically adjust to real-time environmental changes could significantly advance the field of aquatic bioimaging, providing robust solutions for accurate plankton classification under varying conditions.

## VI. CONCLUSION

In this article, we present a novel representation learning model designed for plankton image classification, taking inspiration from metadata information that serves as valuable guidance for class grouping. Our approach involves multiple stages to
achieve efficient representation learning and improve classification performance. First, we employ an encoder-decoder network that optimizes the reconstruction loss and employs a regularized loss function integrated with location, depth, and scale information. This fusion of metadata information helps in effectively grouping species, enhancing the model's understanding of the underlying patterns. Second, we focus on data reduction for the majority class and select representative samples from the latent space data. These selected representative samples are then labeled and grouped based on different class views, as suggested by our encoder-decoder architecture. This approach enables us to handle data with class imbalances and efficiently leverage the labeled samples for training. Finally, we utilize a CNN network to fine tune the model for the downstream classification task, leveraging the representative annotated samples. This fine tuning step further refines the model's capabilities and enhances its classification performance.

While our suggested framework demonstrates promising results, there is substantial scope for advancing its performance through more sophisticated representation learning techniques. A notable direction involves integrating stronger constraints, such as cosine similarity in contrastive learning, could enhance the network's capability to fully utilize metadata information without relying heavily on human annotations. The scalability and efficiency of this approach are particularly valuable for deployment in real-time systems on autonomous vehicles and remote sensing platforms, where rapid and reliable image analysis is critical. These methods could potentially reduce the risk of overfitting, particularly when the data include noise or extraneous details. As a result, the model becomes more adaptable to various scenarios. Furthermore, they are efficient in environments needing quick similarity computations, such as real-time recommendation engines or interactive systems. Future studies could explore these aspects, potentially transforming the way we process and utilize marine imagery in dynamic, resource-constrained environments. By advancing these techniques, we aim to minimize reliance on extensive human annotations, further automating and enhancing the accuracy of ecological monitoring and conservation efforts.

In addition, it is important to acknowledge that our current dataset is limited to a specific instrument and exhibits narrow variability (i.e., primarily concentrated near the coast). This constraint hinders our model's ability to uncover broader patterns and insights about the data. To overcome this limitation, future investigations could focus on datasets with greater variability, providing the model with more diverse and comprehensive data for learning. Comprehensive testing on newer version camera systems could also be considered to assess performance across diverse datasets. Through these endeavors, future studies aim to verify whether the model remains effective and agnostic to different systems, sustaining its effectiveness in plankton image classification tasks across various setups.

## ACKNOWLEDGMENT

The output reflects only the author's view and the Research Executive Agency (REA) cannot be held responsible for any use that may be made of the information contained therein.

## REFERENCES

[1] G. Beaugrand, A. McQuatters-Gollop, M. Edwards, and E. Goberville, "Long-term responses of North Atlantic calcifying plankton to climate change," *Nature Climate Change*, vol. 3, no. 3, pp. 263–267, 2013.

[2] K. Kalloniati et al., "Long-term warming and human-induced plankton shifts at a coastal Eastern Mediterranean site," *Sci. Rep.*, vol. 13, no. 1, 2023, Art. no. 21068.

[3] F. Lombard et al., "Globally consistent quantitative observations of planktonic ecosystems," *Front. Mar. Sci.*, vol. 6, 2019, Art. no. 196.

[4] S. L. C. Giering et al., "Sinking organic particles in the ocean—Flux estimates from in situ optical devices," *Front. Mar. Sci.*, vol. 6, 2020, Art. no. 834.

[5] M. Picheral et al., "The underwater vision profiler 6: An imaging sensor of particle size spectra and plankton, for autonomous and cabled platforms," *Limnol. Oceanogr. Methods*, vol. 20, no. 2, pp. 115–129, 2022.

[6] M. Picheral, S. Colin, and J.-O. Irisson, "EcoTaxa, a tool for the taxonomic classification of images," 2017. [Online]. Available: http://ecotaxaobs.vlfr.fr

[7] S.-M. Schröder, R. Kiko, and R. Koch, "MorphoCluster: Efficient annotation of plankton images by clustering," *Sensors*, vol. 20, no. 11, 2020, Art. no. 3060. [Online]. Available: https://www.mdpi.com/1424-8220/20/11/3060

[8] F. Zhao, F. Lin, and H. S. Seah, "Binary sipper plankton image classification using random subspace," *Neurocomputing*, vol. 73, no. 10/12, pp. 1853–1860, 2010.

[9] P.-H. Liu, S.-F. Su, M.-C. Chen, and C.-C. Hsiao, "Deep learning and its application to general image classification," in *Proc. Int. Conf. Informative Cybern. Comput. Social Syst.*, 2015, pp. 7–10.

[10] G. Gorsky et al., "Digital zooplankton image analysis using the zooscan integrated system," *J. Plankton Res.*, vol. 32, no. 3, pp. 285–303, 2010.

[11] H. M. Sosik and R. J. Olson, "Automated taxonomic classification of phytoplankton sampled with imaging-in-flow cytometry," *Limnol. Oceanogr. Methods*, vol. 5, no. 6, pp. 204–216, 2007.

[12] J. Ellen, H. Li, and M. D. Ohman, *Quantifying California Current Plankton Samples With Efficient Machine Learning Techniques*. Piscataway, NJ, USA: IEEE, 2015.

[13] P. F. Culverhouse et al., "Automatic classification of field-collected dinoflagellates by artificial neural network," *Mar. Ecol. Prog. Ser.*, vol. 139, pp. 281–287, 1996.

[14] M. B. Blaschko et al., "Automatic in situ identification of plankton," in *Proc. IEEE 7th Workshop Appl. Comput. Vis.*, 2005, pp. 79–86.

[15] Z. Liu, J. Watson, and A. Allen, "Efficient affine-invariant Fourier descriptors for identification of marine plankton," in *Proc. OCEANS 2017–Aberdeen*, 2017, pp. 1–9.

[16] J. Y. Luo et al., "Automated plankton image analysis using convolutional neural networks," *Limnol. Oceanogr. Methods*, vol. 16, no. 12, pp. 814–827, 2018.

[17] P. Gonzalez, A. Castaño, E. E. Peacock, J. Díez, J. J. Del Coz, and H. M. Sosik, "Automatic plankton quantification using deep features," *J. Plankton Res.*, vol. 41, no. 4, pp. 449–463, 2019.

[18] E. C. Orenstein and O. Beijbom, "Transfer learning and deep feature extraction for planktonic image data sets," in *Proc. IEEE Winter Conf. Appl. Comput. Vis.*, 2017, pp. 1082–1088.

[19] Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 35, no. 8, pp. 1798–1828, Aug. 2013.

[20] J. Cui et al., "Texture and shape information fusion of convolutional neural network for plankton image classification," in *Proc. Oceans-MTS/IEEE Kobe Techno-Oceans*, 2018, pp. 1–5.

[21] O. Py, H. Hong, and S. Zhongzhi, "Plankton classification with deep convolutional neural networks," in *Proc. IEEE Inf. Technol. Netw. Electron. Autom. Control Conf.*, 2016, pp. 132–136.

[22] J. Dai, R. Wang, H. Zheng, G. Ji, and X. Qiao, "Zooplanktonet: Deep convolutional network for Zooplankton classification," in *Proc. OCEANS 2016-Shanghai*, 2016, pp. 1–6.

[23] B. Guo et al., "Automated plankton classification from holographic imagery with deep convolutional neural networks," *Limnol. Oceanogr. Methods*, vol. 19, no. 1, pp. 21–36, 2021.

[24] H. Lee, M. Park, and J. Kim, "Plankton classification on imbalanced large scale database via convolutional neural networks with transfer learning," in *Proc. IEEE Int. Conf. Image Process.*, 2016, pp. 3713–3717.

[25] C. Wang, Z. Yu, H. Zheng, N. Wang, and B. Zheng, "CGAN-plankton: Towards large-scale imbalanced class generation and fine-grained classification," in *Proc. IEEE Int. Conf. Image Process.*, 2017, pp. 855–859.
[26] G. Menardi and N. Torelli, "Training and assessing rare classification rules with imbalanced data," *Data Mining Knowl. Discov.*, vol. 28, pp. 92–122, 2014.

[27] I. Goodfellow et al., "Generative adversarial nets," in *Proc. Adv. Neural Inf. Process. Syst.*, vol. 27, 2014.

[28] J. L. Walker and E. C. Orenstein, "Improving rare-class recognition of marine plankton with hard negative mining," in *Proc. IEEE/CVF Int. Conf. Comput. Vis.*, 2021, pp. 3672–3682.

[29] S. L. Giering, P. F. Culverhouse, D. G. Johns, A. McQuatters-Gollop, and S. G. Pitois, "Are plankton nets a thing of the past? An assessment of in situ imaging of zooplankton for large-scale ecosystem assessment and policy decision-making," *Frontiers in Marine Science*, vol. 9, 2022, Art. no. 986206.

[30] X. Zhai, A. Oliver, A. Kolesnikov, and L. Beyer, "S4L: Self-supervised semi-supervised learning," in *Proc. IEEE/CVF Int. Conf. Comput. Vis.*, 2019, pp. 1476–1485.

[31] G.-J. Qi and J. Luo, "Small data challenges in Big Data era: A survey of recent progress on unsupervised and semi-supervised methods," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 44, no. 4, pp. 2168–2187, Apr. 2022.

[32] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit.*, 2016, pp. 2536–2544.

[33] A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in *Proc. Int. Conf. Learn. Representations (OpenReview.net)*, 2020.

[34] Y. Wang, C. M. Albrecht, N. A. A. Braham, L. Mou, and X. X. Zhu, "Self-supervised learning in remote sensing: A review," *IEEE Geosci. Remote Sens. Mag.*, vol. 10, no. 4, pp. 213–247, 2022.

[35] D. E. Rumelhart, J. L. McClelland, and PDP Research Group, *Parallel Distributed Processing: Explorations in The Microstructure of Cognition*. Cambridge, MA, USA: MIT Press, 1986.

[36] L. Madhuanand, F. Nex, and M. Y. Yang, "Self-supervised monocular depth estimation from oblique UAV videos," *ISPRS J. Photogrammetry Remote Sens.*, vol. 176, pp. 1–14, 2021.

[37] R. Kemker and C. Kanan, "Self-taught feature learning for hyperspectral image classification," *IEEE Trans. Geosci. Remote Sens.*, vol. 55, no. 5, pp. 2693–2705, May 2017.

[38] D. Hong, N. Yokoya, G.-S. Xia, J. Chanussot, and X. X. Zhu, "X-modalNet: A semi-supervised deep cross-modal network for classification of remote sensing data," *ISPRS J. Photogrammetry Remote Sens.*, vol. 167, pp. 12–23, 2020.

[39] D. Hong, J. Chanussot, N. Yokoya, U. Heiden, W. Heldens, and X. X. Zhu, "Wu-net: A weakly-supervised unmixing network for remotely sensed hyperspectral imagery," in *Proc. IEEE Int. Geosci. Remote Sens. Symp.*, 2019, pp. 373–376.

[40] C. Doersch, A. Gupta, and A. A. Efros, "Unsupervised visual representation learning by context prediction," in *Proc. IEEE Int. Conf. Comput. Vis.*, 2015, pp. 1422–1430.

[41] D. Hong et al., "SpectralGPT: Spectral remote sensing foundation model," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 46, no. 8, pp. 5227–524, Aug. 2024.

[42] M. R. S. B. DATA, "Multimodal artificial intelligence foundation models: Unleashing the power of remote sensing Big Data in Earth observation," *Innovation*, vol. 2, no. 1, 2024, Art. no. 100055.

[43] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in *Proc. Int. Conf. Mach. Learn.*, 2020, pp. 1597–1607.

[44] L. Zhao, W. Luo, Q. Liao, S. Chen, and J. Wu, "Hyperspectral image classification with contrastive self-supervised learning under limited labeled samples," *IEEE Geosci. Remote Sens. Lett.*, vol. 19, pp. 1–5, Mar. 2022.

[45] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, "Deep clustering for unsupervised learning of visual features," in *Proc. Eur. Conf. Comput. Vis.*, 2018, pp. 132–149.

[46] T. Yamada, A. Prügel-Bennett, and B. Thornton, "Learning features from georeferenced seafloor imagery with location guided autoencoders," *J. Field Robot.*, vol. 38, no. 1, pp. 52–67, 2021.

[47] T. Yamada, M. Massot-Campos, A. Prügel-Bennett, S. B. Williams, O. Pizarro, and B. Thornton, "Leveraging metadata in representation learning with georeferenced seafloor imagery," *IEEE Robot. Autom. Lett.*, vol. 6, no. 4, pp. 7815–7822, Oct. 2021.

[48] T. Yamada, M. Massot-Campos, A. Prügel-Bennett, O. Pizarro, S. B. Williams, and B. Thornton, "Guiding labelling effort for efficient learning with georeferenced images," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 45, no. 1, pp. 593–607, Jan. 2023.

[49] J. S. Ellen, C. A. Graff, and M. D. Ohman, "Improving plankton image classification using context metadata," *Limnol. Oceanogr., Methods*, vol. 17, no. 8, pp. 439–461, 2019.

[50] T. Konkle and G. A. Alvarez, "A self-supervised domain-general learning framework for human ventral stream representation," *Nature Commun.*, vol. 13, no. 1, 2022, Art. no. 491.

[51] J. V. Haxby, M. I. Gobbini, M. L. Furey, A. Ishai, J. L. Schouten, and P. Pietrini, "Distributed and overlapping representations of faces and objects in ventral temporal cortex," *Science*, vol. 293, no. 5539, pp. 2425–2430, 2001.

[52] N. Kanwisher, "Functional specificity in the human brain: A window into the functional architecture of the mind," *Proc. Nat. Acad. Sci. USA*, vol. 107, no. 25, pp. 11163–11170, 2010.

[53] H. P. O. de Beeck, I. Pillet, and J. B. Ritchie, "Factors determining where category-selective areas emerge in visual cortex," *Trends Cogn. Sci.*, vol. 23, no. 9, pp. 784–797, 2019.

[54] B. Long, C.-P. Yu, and T. Konkle, "Mid-level visual features underlie the high-level categorical organization of the ventral stream," *Proc. Nat. Acad. Sci. USA*, vol. 115, no. 38, pp. E9015–E9024, 2018.

[55] Y. Wu and K. He, "Group normalization," in *Proc. Eur. Conf. Comput. Vis.*, 2018, pp. 3–19.

[56] P. Baldi, "Autoencoders, unsupervised learning, and deep architectures," in *Proc. ICML Workshop Unsupervised Transfer Learn.*, 2012, pp. 37–49.

[57] D. Pathak, R. Girshick, P. Dollár, T. Darrell, and B. Hariharan, "Learning features by watching objects move," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit.*, 2017, pp. 2701–2710.

[58] L. Van der Maaten and G. Hinton, "Visualizing data using t-SNE," *J. Mach. Learn. Res.*, vol. 9, no. 11, 2008.

[59] A. Darmochwał, "The Euclidean space," *Formalized Math.*, vol. 2, no. 4, pp. 599–603, 1991.

[60] H. Xiong, J. Wu, and J. Chen, "K-means clustering versus validation measures: A data distribution perspective," in *Proc. 12th ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining*, 2006, pp. 779–784.

[61] M. Zimmer, "Detritus," in *Encyclopedia of Ecology*, S. E. Jørgensen and B. D. Fath, Eds. Oxford, U.K.: Academic, 2008, pp. 903–911. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780080454054004754

[62] M. Stukel, K. Mislan, M. Décima, and L. Hmelo, "Detritus in the pelagic ocean," in *Proc. Eco-DAS IX Symp.*, Waco, TX, USA, 2014, pp. 49–76.

[63] P. F. Culverhouse, R. Williams, B. Reguera, V. Herry, and S. González-Gil, "Do experts make mistakes? A comparison of human and machine identification of dinoflagellates," *Mar. Ecol. Prog. Ser.*, vol. 247, pp. 17–25, 2003.

[64] P. F. Culverhouse, N. Macleod, R. Williams, M. C. Benfield, R. M. Lopes, and M. Picheral, "An empirical assessment of the consistency of taxonomic identifications," *Mar. Biol. Res.*, vol. 10, no. 1, pp. 73–84, 2014.

[65] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," *Commun. ACM*, vol. 60, no. 6, pp. 84–90, 2017.

[66] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in *Proc. 3rd Int. Conf. Learn. Representations*, 2015.

[67] S. H. Khan, M. Hayat, M. Bennamoun, F. A. Sohel, and R. Togneri, "Cost-sensitive learning of deep feature representations from imbalanced data," *IEEE Trans. Neural Netw. Learn. Syst.*, vol. 29, no. 8, pp. 3573–3587, Aug. 2018.

[68] R. P. Espíndola and N. F. Ebecken, "On extending F-measure and G-mean metrics to multi-class problems," *WIT Trans. Inf. Commun. Technol.*, vol. 35, pp. 25–34, 2005.

[69] S.-J. Yen and Y.-S. Lee, "Cluster-based under-sampling approaches for imbalanced data distributions," *Expert Syst. Appl.*, vol. 36, no. 3, pp. 5718–5727, 2009.

[70] C. M. Bishop and N. M. Nasrabadi, *Pattern Recognition and Machine Learning*. Berlin, Germany: Springer, 2006, vol. 4, no. 4.

[71] B. Kamiński, M. Jakubczyk, and P. Szufel, "A framework for sensitivity analysis of decision trees," *Central Eur. J. Oper. Res.*, vol. 26, no. 1, pp. 135–159, 2018.

[72] T. K. Ho, "Random decision forests," in *Proc. 3rd Int. Conf. Document Anal. Recognit.*, 1995, pp. 278–282.

[73] J. H. Friedman, "Greedy function approximation: A gradient boosting machine," *Ann. Statist.*, vol. 29, pp. 1189–1232, 2001.

[74] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proc. IEEE Conf. Comput. Vis. Pattern Recognit.*, 2016, pp. 770–778.
[75] B. Gregory, L. Christophe, and E. Martin, "Rapid biogeographical plankton shifts in the North Atlantic ocean," *Glob. Change Biol.*, vol. 15, no. 7, pp. 1790-1803, 2009.

[76] E. Villarino et al., “Modelling the future biogeography of North Atlantic zooplankton communities in response to climate change,” *Mar. Ecol. Prog. Ser.*, vol. 531, pp. 121–142, 2015.

[77] K. Bandara, S. L. Basedow, G. Pedersen, and V. Tverberg, “Mid-summer vertical behavior of a high-latitude oceanic zooplankton community,” *J. Mar. Syst.*, vol. 230, 2022, Art. no. 103733.

[78] H. Ueda, A. Kuwahara, N. Tanaka, and M. Azeta, “Underwater observations on copepod swarms in temperate and subtropical waters,” *Mar. Ecology Prog. Ser.*, vol. 11, pp. 165–171, 1983.

[79] J. W. Ambler, "Zooplankton swarms: Characteristics, proximal cues and proposed advantages," *Hydrobiologia*, vol. 480, pp. 155-164, 2002.

[80] T. Pados and R. F. Spielhagen, "Species distribution and depth habitat of recent planktic foraminifera in Fram Strait, Arctic Ocean," *Polar Res.*, vol. 33, no. 1, 2014, Art. no. 22483.

[81] S. E. Wilson, D. K. Steinberg, and K. O. Buesseler, “Changes in fecal pellet characteristics with depth as indicators of zooplankton repackaging of particles in the mesopelagic zone of the subtropical and subarctic north pacific ocean,” *Deep Sea Res. Part II: Topical Stud. Oceanogr.*, vol. 55, no. 14/15, pp. 1636–1647, 2008.

[82] S. L. Giering, B. Hosking, N. Briggs, and M. H. Iversen, “The interpretation of particle size, shape, and carbon flux of marine particle images is strongly affected by the choice of particle detection algorithm,” *Front. Mar. Sci.*, vol. 7, 2020, Art. no. 564.

[83] T. Platt and K. Denman, “Organisation in the pelagic ecosystem,” *Helgoländer Wissenschaftliche Meeresuntersuchungen*, vol. 30, pp. 575–581, 1977.

[84] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu, “An augmented linear mixing model to address spectral variability for hyperspectral unmixing,” *IEEE Trans. Image Process.*, vol. 28, no. 4, pp. 1923–1938, Apr. 2019.

<figure><img src="image_12.png" /></figure>

**Mojtaba Masoudi** received the B.Eng. degree in software engineering from the Shahrood University of Technology, Shahrud, Iran, in 2014, and the M.Sc. degree in artificial intelligence and robotics from the School of Computer Science and Engineering, Ferdowsi University of Mashhad, Mashhad, Iran, in 2017.

He is currently serving as a Research Scientist with the National Oceanography Centre, Southampton, U.K. His research interests include the field of Ocean AI.

<figure><img src="image_13.png" /></figure>

**Sarah L.C. Giering** received the B.Sc. (Hons.) degree in marine biology from the University of Aberdeen, Aberdeen, U.K., in 2009, and the Ph.D. degree in marine biogeochemistry from the University of Southampton, Southampton, U.K., in 2013.

She is a Marine Biogeochemist. She leads an interdisciplinary research group that aims to improve state-of-the-art technologies and methodologies to fill the biggest knowledge gaps in our understanding of the ocean carbon cycle. Her research interests include the role of plankton and particles in the ocean carbon

cycle, recognized for having made key discoveries that are critical to understanding Earth system functioning.

<figure><img src="image_14.png" /></figure>

**Noushin Eftekhari** received the Ph.D. degree in artificial intelligence from Teesside University, U.K., in 2023.

she specializes in developing deep learning models to explore personalized medicine.

<figure><img src="image_15.png" /></figure>

**Miquel Massot-Campos** received the M.Eng. degree in industrial engineering from BarcelonaTech, Barcelona, Spain in 2011, the M.Sc. degree in information technologies and the Ph.D. degree from the University of the Balearic Islands, Palma, Spain, in 2013 and 2019, respectively.

He is currently with the University of Southampton, Southampton, U.K. He is a Member of the Centre for In Situ and Remote Intelligent Sensing. His research focuses on the scalability of autonomous underwater vehicle's missions.

<figure><img src="image_16.png" /></figure>

**Jean-Olivier Irisson** received graduation degree from the École Normale Supérieure, Paris, France, in 2001, and the master's degree in the behavior of tropical fish larvae and its implication for population connectivity and the Ph.D. degree in ecology from the École Pratique des Hautes Études, Perpignan, France, in 2003 and 2008 respectively.

He is currently a Computational Ecologist. Since 2009, he has been an Associate Professor with Laboratoire d'Océanographie de Villefranche, Sorbonne Université, Paris. He leads the plankton ecology team.

His research interests include long-term monitoring programs as well as research on plankton ecology, at the interface between environmental and computer sciences.

<figure><img src="image_17.png" /></figure>

**Blair Thornton** (Member, IEEE) received the B.Eng. degree in naval architecture and the Ph.D. degree in underwater robotics from Southampton University, Southampton, U.K., in 2002 and 2006, respectively.

In 2003, he joined the Underwater Robotics and Application Laboratory, Institute of Industrial Science, UTokyo, before rejoining Southampton University in 2016, where he is currently a Professor of marine autonomy. He has spent more than 450 days at sea deploying robotic systems and is dedicated to generating data and insight in marine science through

improved sensing and autonomy.