# Copepods Image-Skeleton Pairing with N-Pair Loss

Martinel Anthonin, Benzinou Abdesslam, Nasreddine Kamal, Foulon

Valentin, Catherine Borremans, Daniela Zeppilli

► To cite this version:

Martinel Anthonin, Benzinou Abdesslam, Nasreddine Kamal, Foulon Valentin, Catherine Borremans, et al.. Copepods Image-Skeleton Pairing with N-Pair Loss. 25th International Conference on Digital Signal Processing (DSP), Jun 2025, Pylos, Greece. pp.1-5, 10.1109/DSP65409.2025.11075170. hal-05244497

HAL Id: hal-05244497

https://hal.univ-brest.fr/hal-05244497v1

Submitted on 8 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Copepods Image-Skeleton Pairing with N-Pair Loss

MARTINEL Anthonin*, BENZINOU Abdesslam*, NASREDDINE Kamal*, FOULON Valentin*, BORREMANS Catherine† and ZEPPILLI Daniela†

*ENIB, UMR CNRS 6285 LabSTICC, 29238 Brest, France

Email: martinel@enib.fr

†Ifremer, 29280 Plouzané, France

**Abstract**—In this study, we propose a novel method to determine whether a copepod image belongs to the same morphological case as a one-channel binary hand-drawn skeleton. This method can be used for indexing and two-modal data matching, allowing hand-drawn skeletons to be aligned with specific functional trait representations. Our problem is particularly challenging due to the small dataset size, morphological case imbalance, and morphological ambiguities. To address these issues, we employ two convolutional neural networks to encode images and skeletons into vector representations. We then refine these encodings using N-Pair Loss, training two fully connected layers to compare the encoded image with skeletons from different morphological cases. This contrastive learning approach leverages additional comparative information, helping the model distinguish subtle morphological variations more effectively. For experiments we used an image dataset of 1969 copepod specimens. The results demonstrated that our method outperforms traditional training approach, effectively addressing the challenges of small, imbalanced, and complex datasets and offering a more robust latent space representation. Moreover, it can be extended to datasets with additional morphological cases and different input types, making it applicable to taxonomic studies, particularly those focusing on functional traits.

**Index Terms**—Machine Learning, Meiofauna, Functional Traits, Dual Modal Matching, N-Pair Loss

## I. INTRODUCTION

Meiofauna (or Meiobenthos) are small animals living at the bottom of the sea in the sediments. Their size ranges from 20µm to about 1mm, smaller than Macrofauna but larger than Microfauna (see Figure 1). This range follows the standard size of sieves used for sorting marine sediments and specimens. In this work, we focus on copepods, the second most abundant group in the meiofauna, which are considered as bioindicators of ecological issues, pollution and global changes. In recent years, taxonomists have explored functional traits-based approaches for animal classification [1]. Functional traits are defined by Violle et al. [2] as “morphophysio-phenological traits which impact fitness indirectly via their effects on growth, reproduction and survival, the three components of individual performance”. In the field of marine environment, Martini et al. [1] proposed a framework for the study of functional traits, consolidating the terminology and cataloging datasets and methods. The use of machine learning to extract and characterize functional traits remains an emerging area. Orenstein et al. [3] recently explored this

approach for plankton images but challenges remain when datasets are small, imbalanced, or involve subtle morphological differences. To address these challenges, we propose a novel multimodal approach that matches an image of copepod with a binary hand-drawn skeleton. In fact, taxonomists commonly use simplified hand-drawn schematics to describe morphological and functional traits as illustrated in Figure 1 for the shape of antennas as a functional trait. Our method aims to determine whether a copepod image and a given skeleton belong to the same morphological case, defined by the appearance in the image of key traits such as antennas and legs. This framework is versatile and can be extended for other applications, such as distinguishing active versus inactive copepods based on antenna positioning [4] or indexing large datasets with input schematic traits.

The relationships between two-modal inputs have been extensively studied in various fields of computer vision such as for natural language processing [6][7] and medical analysis [8]. The CNN-based approaches often rely on a joint embedding representation or a similarity learning network [9]. In the joint embedding methods, two branches each encodes an input type and projects it into a shared latent space to learn a cross-modal embedding representation [6][7][8]. Inspired by these approaches, our method uses two neural networks to encode images and skeletons in two feature vectors. We leverage the N-Pair loss [10], a generalization of the triplet loss introduced by Schroff et al. [11], to allow joint comparison between these two feature vectors. This brings the matching instances closer to each other, while keeping apart the non-matching ones. While the original N-Pair loss was not designed for multimodal data, we adapted it to ensure effective comparison

<figure><img src="image_2.png" /><figcaption>Fig. 1: Examples of copepods real images from BOLD dataset [5] and corresponding drawings [4]. Each drawing shows antennas position, a functional trait. Deployed antennas (left figure) are associated with diapause (i.e. the delay in the copepod development) termination.</figcaption></figure>

<footnote>This work is supported by Equinor-Norway and EUR ISblue (Interdisciplinary graduate School for the Blue Planet).
979-8-3315-1213-2/25/$31.00 ©2025 IEEE</footnote>
between image and skeleton representations. For experimental evaluation, We will use a copepod image dataset of 1969 copepod specimens collected from BOLD plateform (The Barcode of Life Data System dataset [5]). Skeleton are hand-drawn before defining matching and non-matching pairs.

The proposed model is presented in Section II. The dataset and metrics are described, respectively, in Sections III and IV. The results of our method are discussed in the Section V. Finally, we draw conclusion and perspectives in Section VI.

## II. PROPOSED METHOD

Our objective is to determine whether a copepod image and a given skeleton belong to the same morphological case, defined based on the visibility of key morphological features such as antennae and legs. To achieve this, we employ two Convolutional Neural Networks (CNNs) to encode the image and skeleton into two vector representations before projecting them into a joint latent space (see Figure 2). Training is conducted in multiple stages, leveraging the N-Pair loss [10] in the final phase to optimize similarity learning. The following subsections provide a detailed explanation of our approach.

### A. Proposed model

We assume that there are $N$ morphological cases based on the presence of studied key morphological features. For each case, we have samples of representative binary one-channel hand-drawn skeletons. Given a pair $(x, s)$, where $x$ is an RGB image and $s$ is a binary one-channel hand-drawn skeleton, our goal is to develop a model that classifies whether the pair belongs to the same morphological case ("Match") or not ("Non-Match"). To address this issue, we propose a model composed of two CNN encoders, one for the image, one for the skeletons. Instead of comparing the image embedding $f$ to the skeleton embedding $g$, we also use the skeleton encoder to embed skeletons sampled from other morphological cases. The model predicts which skeleton is most likely to belong to the same morphological case as the image using a cosine similarity metric. Based on this choice, we can then predict whether the original pair $(x, s)$ belongs to the same morphological case ("Match") or not ("Non-Match").

### B. Training Strategy

The training process consists of three key stages:

1. **Pretraining the Encoders:** We first pretrain the image encoder on the full copepod dataset to classify taxonomic orders. Then, we fine-tune it on the Calanoida images to distinguish the studied morphological cases. This hierarchical approach ensures that the model captures general morphological patterns before focusing on finer distinctions. The skeleton encoder is initially trained to classify skeleton images into the same morphological cases.

2. **Fine-Tuning the Skeleton Encoder:** Once trained, the final layer of skeleton encoder is replaced with a fully connected layer to produce feature embeddings of the desired dimension, and is further trained using the original N-Pair loss to structure the latent space effectively. Once the skeleton encoder has learned a well-structured latent space, we freeze its weights to serve as a stable reference for aligning the image encoder.

3. **Aligning the Image Encoder with the Skeleton Encoder:** The last layer of the image encoder is trained using a modified N-Pair loss that enforces similarity between matching image-skeleton pairs while maximizing dissimilarity between non-matching pairs. This step ensures that the embeddings from both encoders reside in a shared latent space, improving retrieval performance.

### C. Loss Function and Similarity Computation

Given an $(N+1)$-tuple $\{x, s^+, s_1, \dots, s_{N-1}\}$ where $x$ represents the image, $s^+$ is its corresponding skeleton, and $\{s_i\}_{i=1}^{N-1}$ are skeletons from other morphological cases, we define the feature embeddings as follows:

$$f(x, \theta) = f \qquad (1)$$

$$g(s^+, \theta') = g^+ \qquad (2)$$

$$\forall i \in [1, N-1], \quad g(s_i, \theta') = g_i \qquad (3)$$

Our modified N-Pair loss is expressed as:

$$\mathcal{L}(\{x, s^+, \{s_i\}_{i=1}^{N-1}\}; f; g) = \log\left(1 + \sum_{i=1}^{N-1} (f^T g^+ - f^T g_i)\right) \qquad (4)$$

Here, $f^T g^+$ represents the similarity between the image and its matching skeleton, while $f^T g_i$ penalizes similarity with non-matching skeletons. To prevent exploding norms in the embeddings, $\mathcal{L}_2$ regularization is applied.

For inference, we follow the same approach: we encode the image and the skeleton from the $(x, s)$ pair, and skeletons from other morphological cases using our image and skeleton encoders. We compute cosine similarity scores between the image embedding $f$ and the skeletons embeddings $\{g^+, g_1, \dots, g_{N-1}\}$. The skeleton with the highest similarity score is selected, and the pair $(x, s)$ is classified as a "Match" if their labels align; otherwise, it is classified as a "Non-Match".

### D. Architectures and Attention Mechanisms

We evaluate three widely used CNN architectures:

* **AlexNet** [12] serves as a baseline to compare against deeper architectures.

* **ResNet** [13] and **DenseNet** [14] are chosen for their superior feature extraction capabilities and ability to mitigate vanishing gradients.

* To enhance global context capture, we integrate **attention mechanisms** into a ResNet model, following the guidelines of Wang et al. [15].

Loss functions are weighted to account for morphological cases imbalances and ensure stable convergence during training.
<figure><img src="image_3.png" /><figcaption>Fig. 2: Diagram illustrating the proposed model. Two CNNs encode images and skeletons into a shared latent space. Cosine similarity is used for classification.</figcaption></figure>

### III. EXPERIMENTAL DATA

Our copepod images dataset is a collection of BOLD data [5]. We selected only images displaying the whole animal with a resolution of at least 256 x 256 pixels. The images are resized to this shape for consistency. Copepods are identified by their taxonomic order. The dataset is composed of 1969 copepod specimens distributed as follows: 902 Calanoida, 766 Cyclopoida, 150 Harpacticoida and 151 Siphonostomatoida. These are used to pretrain the image encoder. For subsequent experiments, we focused on non ambiguous Calanoida images belonging to four morphological cases: "Nothing", "Legs", "Antenna" and "AntennaANDLegs". The classes are defined based on the visibility of key morphological features, in other words, if we clearly see the antenna and/or the legs of the animal. These features are critical for differentiating subtle taxonomic functional variations among specimens. We obtain the following distribution: 141 *Nothing*, 99 *Legs*, 302 *Antenna* and finally 968 *AntennaANDLegs*.

Skeletons are hand-drawn using the sorted image dataset as templates as taxonomists do to describe morphological and functional traits ensuring consistent representation. Skeletons are binary one channel images. We obtain the following distribution: 115 *Nothing*, 152 *Legs*, 226 *Antenna* and finally 476 *AntennaANDLegs*. Examples of images and skeletons are given in Figure 3 and 4.

To train the model, we generated pairs of “Match” and “Non-Match” samples by associating each image with skeletons from the dataset. Given an image, one skeleton from its morphological case is paired as the “Match”, and skeletons from other morphological cases are randomly selected as “Non-Matches”. For each image, this process is repeated 4 times in the validation and testing sets to reduce variance and allow comparison across models. This results in 8(N - 1) pairs per image. The final dataset consists of 25,296 pairs for training, 5,496 pairs for validation, and 5,496 pairs for testing. When splitting the dataset, we make sure that one image or skeleton cannot be found in multiple sets.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Examples of Calanoida images. From left to right, the animal belongs to the morphological case: *AntennaANDLegs*, *Legs*, *Antenna* and *Nothing*, respectively.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4: Examples of handdrawn skeletons. From left to right, the animal belongs to the morphological case: *AntennaANDLegs*, *Legs*, *Antenna* and *Nothing*, respectively.</figcaption></figure>

### IV. EVALUATION METRICS

To evaluate the performances of the pretrained networks and our model, we compute accuracy, precision, and recall. To account for the four morphological cases data distribution, precision and recall are micro-averaged, i.e. we add up all the true positives, false negative, etc, from each morphological cases before computing a global metric score.

We extract the characteristics of the images and skeletons using the two pretrained network, one for the image, the other one for the skeletons. Using cosine similarity, we determine which skeleton vector is most similar to the extracted image vector. The model’s prediction is considered correct if the selected skeleton belongs to the same morphological case as the image. Based on these predictions, we compute accuracy, precision, and recall to evaluate model performance.
# V. RESULTS AND DISCUSSION

## A. Pretraining Results

The image encoding model undergoes a two-stage pretraining process. First, it is trained for classification using all BOLD copepod images. Then, it is fine-tuned on our four-class (Antenna, Legs, Antenna and Legs, Nothing) Calanoida dataset, enabling the model to learn shared morphological features of copepods. To improve generalization, we apply data augmentation techniques such as flipping, rotation, cropping, translation, brightness adjustment, and pixel-wise Gaussian noise. The best-performing models are reported in Table I, where DenseNet201 achieves the highest accuracy, precision, and recall on the test dataset. All reported precision and recall values are micro-averaged.

<table><caption>TABLE I: Pretraining results for the image classifier. ResNetAtt refers to the ResNet model with Attention.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Accuracy</th><th colspan="2">Precision</th><th colspan="2">Recall</th></tr><tr><th>Train</th><th>Test</th><th>Train</th><th>Test</th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>AlexNet</td><td>81,31</td><td>69,16</td><td>81,31</td><td>69,16</td><td>81,31</td><td>69,16</td></tr><tr><td>ResNet50</td><td>87,57</td><td>83,26</td><td>87,57</td><td>83,26</td><td>87,57</td><td>83,26</td></tr><tr><td>ResNetAtt</td><td><strong>94,40</strong></td><td>81,50</td><td><strong>94,40</strong></td><td>81,50</td><td><strong>94,40</strong></td><td>81,50</td></tr><tr><td>DenseNet201</td><td>87,57</td><td><strong>83,70</strong></td><td>87,57</td><td><strong>83,70</strong></td><td>87,57</td><td><strong>83,70</strong></td></tr></tbody></table>

Similarly, the skeleton encoding model is pretrained for classification across the four morphological cases. The best-performing networks are selected, and their classification layers are removed to produce a skeleton encoder for the matching network. Data augmentation is also applied, including rotation, cropping, and translation. Table II presents the results, where ResNet with Attention achieves the highest performance, reaching 94.52% accuracy, precision, and recall on the test set.

<table><caption>TABLE II: Pretraining results for the skeleton classifier. ResNetAtt refers to the ResNet model with Attention.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Accuracy</th><th colspan="2">Precision</th><th colspan="2">Recall</th></tr><tr><th>Train</th><th>Test</th><th>Train</th><th>Test</th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td>AlexNet</td><td>85,99</td><td>81,51</td><td>85,99</td><td>81,51</td><td>85,99</td><td>81,51</td></tr><tr><td>ResNet50</td><td>89,23</td><td>88,36</td><td>89,23</td><td>88,36</td><td>89,23</td><td>88,36</td></tr><tr><td>ResNetAtt</td><td><strong>94,25</strong></td><td><strong>94,52</strong></td><td><strong>94,25</strong></td><td><strong>94,52</strong></td><td><strong>94,25</strong></td><td><strong>94,52</strong></td></tr><tr><td>DenseNet201</td><td>89,38</td><td>93,15</td><td>89,38</td><td>93,15</td><td>89,38</td><td>93,15</td></tr></tbody></table>

## B. Final Results

To evaluate the effectiveness of our method, we establish a baseline model for comparison. This baseline consists of two neural networks predicting the image and skeleton morphological case independently. A match is predicted if both networks assign the same morphological case label.

We then propose an alternative approach to structure the latent space more efficiently. After pretraining, we add a layer at the end of both encoders, trained using the N-Pair Loss. We experiment with different vector sizes (64, 256, and 512), observing similar results across all configurations. The smallest vector size (64) is selected to minimize computational

<figure><img src="image_6.png" /><figcaption>Fig. 5: T-SNE visualization of the embedding space of the image and skeleton DenseNet201 encoders</figcaption></figure>

complexity. The final results are reported in Table III, where DenseNet201 achieves the best performance on the test set. Figure 5 shows T-SNE visualization of the embedding space of the image and skeleton DenseNet201 encoders. We see that images and skeletons belonging to the same morphological case are well separated and clustered. AntennaANDLegs morphological case is less clustered, revealing broader diversity of images.

<table><caption>TABLE III: Final results: comparison between the baseline (b) and proposed method (pm). The final vector size is 64.</caption><thead><tr><th rowspan="2">Model</th><th rowspan="2"></th><th colspan="2">Accuracy</th><th colspan="2">Precision</th><th colspan="2">Recall</th></tr><tr><th>Train</th><th>Test</th><th>Train</th><th>Test</th><th>Train</th><th>Test</th></tr></thead><tbody><tr><td rowspan="2">AlexNet</td><td>b</td><td>81,13</td><td>72,76</td><td>89,40</td><td>81,96</td><td>70,64</td><td>58,37</td></tr><tr><td>pm</td><td><strong>85,23</strong></td><td><strong>76,41</strong></td><td><strong>91,23</strong></td><td><strong>84,21</strong></td><td><strong>77,96</strong></td><td><strong>65,01</strong></td></tr><tr><td rowspan="2">ResNet50</td><td>b</td><td>85,90</td><td>81,98</td><td>93,31</td><td>89,23</td><td>77,34</td><td>72,72</td></tr><tr><td>pm</td><td>79,19</td><td>77,53</td><td>86,85</td><td>85,21</td><td>68,81</td><td>66,63</td></tr><tr><td rowspan="2">ResNetAtt</td><td>b</td><td>92,57</td><td>84,82</td><td>95,42</td><td>90,38</td><td>89,44</td><td>77,94</td></tr><tr><td>pm</td><td><strong>94,13</strong></td><td><strong>86,47</strong></td><td><strong>96,81</strong></td><td><strong>92,29</strong></td><td><strong>91,28</strong></td><td><strong>79,59</strong></td></tr><tr><td rowspan="2">DenseNet201</td><td>b</td><td>86,05</td><td>85,70</td><td>92,90</td><td>91,12</td><td>78,05</td><td>79,11</td></tr><tr><td>pm</td><td><strong>90,12</strong></td><td><strong>88,71</strong></td><td><strong>94,45</strong></td><td><strong>93,52</strong></td><td><strong>85,24</strong></td><td><strong>83,19</strong></td></tr></tbody></table>

# VI. CONCLUSION AND FUTURE WORK

In this work, we present a novel approach to determine whether a three-channel image matches a binary one-channel hand-drawn skeleton, meaning whether they belong to the same morphological case. Our study focuses on the morphology of Calanoida, a group of copepods, and we evaluate different deep learning architectures, including AlexNet, ResNet, ResNet with Attention, and DenseNet. Among these, DenseNet201 achieved the best performance, reaching an accuracy of 88.71% on the test dataset. This method significantly improves upon the naive approach by better leveraging feature extraction and representation learning. While our study focuses on four morphological cases, future work could extend this framework to a broader range of morphologies, meeting the needs of biologists studying these organisms. Additionally, replacing hand-drawn skeletons with alternative encoded representations of other functional trait descriptors could further enhance the model applicability across different biological contexts.
REFERENCES

[1] S. Martini, F. Larras, A. Boye, E. Faure, N. Aberle et al., "Functional trait-based approaches as a common framework for aquatic ecologists," *Limnology And Oceanography*, vol. 66, no. 3, pp. 965-994, 2021.

[2] C. Violle, M.-L. Navas, D. Vile, E. Kazakou, C. Fortunel et al., "Let the concept of trait be functional!" *Oikos*, vol. 116, no. 5, pp. 882-892, 2007.

[3] E. c. Orenstein, S. Ayata, F. Maps, E. c. Becker, F. Benedetti et al., "Machine learning techniques to characterize functional traits of plankton from image data," *Limnology And Oceanography*, vol. 67, no. 8, pp. 1647-1669, 2022.

[4] P. H. Lenz and V. Roncalli, "Diapause within the context of life-history strategies in calanid copepods (calanoida: Crustacea)," *The Biological Bulletin*, vol. 237, no. 2, pp. 170-179, 2019, pMID: 31714852.

[5] S. Ratnasingham and P. Hebert, "Bold: The barcode of life data system (www.barcodinglife.org)," *Molecular ecology notes*, vol. 7, pp. 355-364, 06 2007.

[6] L. Wang, Y. Li, and S. Lazebnik, "Learning deep structure-preserving image-text embeddings," 2016. [Online]. Available: https://arxiv.org/abs/1511.06078

[7] Y. Zhang and H. Lu, "Deep cross-modal projection learning for image-text matching," in *Proceedings of the European Conference on Computer Vision (ECCV)*, September 2018.

[8] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, "Contrastive learning of medical visual representations from paired images and text," 2022. [Online]. Available: https://arxiv.org/abs/2010.00747

[9] X. Gao, T. Mu, J. Y. Goulermas, and M. Wang, "Attention driven multi-modal similarity learning," *Information Sciences*, vol. 432, pp. 530-542, 2018.

[10] K. Sohn, "Improved deep metric learning with multi-class n-pair loss objective," in *Advances in Neural Information Processing Systems*, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds., vol. 29. Curran Associates, Inc., 2016.

[11] F. Schroff, D. Kalenichenko, and J. Philbin, "Facenet: A unified embedding for face recognition and clustering," in *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*. IEEE, Jun. 2015, p. 815-823.

[12] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in *Advances in Neural Information Processing Systems*, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., vol. 25. Curran Associates, Inc., 2012.

[13] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016, pp. 770-778.

[14] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," 2018. [Online]. Available: https://arxiv.org/abs/1608.06993

[15] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li et al., "Residual attention network for image classification," 2017. [Online]. Available: https://arxiv.org/abs/1704.06904