# The point is the mask: scaling coral reef segmentation with weak supervision

Matteo Contini, Victor Illien, Sylvain Poulain, Serge Bernard, Julien Barde, Sylvain Bonhommeau, Alexis Joly

► To cite this version:

Matteo Contini, Victor Illien, Sylvain Poulain, Serge Bernard, Julien Barde, et al.. The point is the mask: scaling coral reef segmentation with weak supervision. ICCV 2025 - Joint Workshop on Marine Vision, Oct 2025, Honolulu, Hawaii, United States. hal-05457113

HAL Id: hal-05457113

https://inria.hal.science/hal-05457113v1

Submitted on 14 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# The point is the mask: scaling coral reef segmentation with weak supervision

Matteo Contini¹,², Victor Illien¹, Sylvain Poulain⁴, Serge Bernard³, Julien Barde⁴, Sylvain Bonhommeau¹, and Alexis Joly²

¹IFREMER, ²INRIA, ³CNRS, and ⁴IRD

<figure><img src="image_2.png" /><figcaption>Figure 1. Workflow of the proposed weakly supervised semantic segmentation (WSSS) multi-scale coral reef segmentation approach. Underwater images are classified using *DinoVdeau* model (step 1), generating spatialized predictions that are interpolated (step 2) to create continuous rasters. These rasters are used as coarse annotations for training an Unmanned Aerial Vehicle (UAV)-based segmentation model (step 3) using the SegFormer model. Model predictions are then refined using SAMRefiner (step 4) and used to retrain Segformer (step 5), improving segmentation accuracy.</figcaption></figure>

## Abstract

Monitoring coral reefs at large spatial scales remains an open challenge, essential for assessing ecosystem health and informing conservation efforts. While drone-based aerial imagery offers broad spatial coverage, its limited resolution makes it difficult to reliably distinguish fine-scale classes, such as coral morphotypes. At the same time, obtaining pixel-level annotations over large spatial extents is costly and labor-intensive, limiting the scalability of deep learning-based segmentation methods for aerial imagery. We present a multi-scale weakly supervised semantic segmentation framework that addresses this challenge by transferring fine-scale ecological information from underwater imagery to aerial data. Our method enables large-

scale coral reef mapping from drone imagery with minimal manual annotation, combining classification-based supervision, spatial interpolation and self-distillation techniques. We demonstrate the efficacy of the approach, enabling large-area segmentation of coral morphotypes and demonstrating flexibility for integrating new classes. This study presents a scalable, cost-effective methodology for high-resolution reef monitoring, combining low-cost data collection, weakly supervised deep learning and multi-scale remote sensing.
# 1. Introduction

Coral reefs are vital marine ecosystems that support biodiversity, protect coastlines and sustain fisheries and tourism industries [5, 16, 17, 28]. However, they are increasingly threatened by climate change, ocean acidification, pollution and human activities, leading to widespread degradation and habitat loss [19, 20, 34]. Effective reef monitoring at large spatial and temporal scales is necessary for conservation and management efforts. Although traditional diver-based surveys provide high-resolution and quality data, they are limited in coverage, labour-intensive and costly [6]. Recent advances in remote sensing technologies, particularly unmanned aerial vehicles (UAVs) and autonomous surface vehicles (ASVs), offer scalable solutions for large-scale coral reef monitoring [13, 25]. UAVs enable rapid aerial assessments of extensive reef areas [3], while ASVs provide fine-scale underwater observations [15]. Additionally, spatial interpolation of ecological indicators from plot-based surveys has been proposed as a cost-effective strategy for rapid biodiversity assessment [4]. However, integrating data across these different scales remains a key challenge.

Deep learning-based image analysis has transformed monitoring applications, allowing automated classification of complex environments [2, 23, 32, 33]. Semantic segmentation is a computer vision task that assigns a class label to each pixel in an image, allowing for detailed spatial mapping of objects or habitats. Fully supervised semantic segmentation models require pixel-wise annotations, but producing such detailed labels for large-scale reef monitoring is labour-intensive and time-consuming [30, 31]. Moreover, while broad habitat classes such as sand, coral, or sea-grass can be reliably identified [14, 27, 40], distinguishing between different coral morphotypes is challenging due to the limited resolution of UAV imagery that often does not provide enough discriminative detail to differentiate fine-scale morphological variations. This challenge demands alternative approaches that reduce annotation efforts while preserving annotation accuracy.

Weakly supervised semantic segmentation (WSSS) applied to remote sensing images offers a promising solution by leveraging weak labels, such as image-level annotations, bounding boxes, or rough segmentation masks, instead of manually labelled pixel-wise ground truth masks [18, 35]. This technique reduces the need for extensive manual annotation by training models on approximate labels (human or machine-generated) and learning to predict masks from these weak annotations. However, to our knowledge, WSSS approaches have not yet been applied in multi-scale contexts, where cross-modal knowledge transfer (e.g., from ASV-based underwater classification to UAV-based aerial segmentation) can be used to improve the quality and scalability of segmentation models.

In this study, we introduce a multi-scale WSSS approach

that bridges the gap between fine-scale coral reef monitoring based on underwater images and large-scale mapping based on aerial images. The workflow of the proposed method is illustrated in Fig. 1.

Instead of manually annotating UAV images, we propose using classification-based probability maps derived from underwater images to generate segmentation annotations for aerial imagery. Our method spatially interpolates ASV-derived class predictions, producing rough segmentation masks that serve as training labels for SegFormer, a transformer-based semantic segmentation model [36]. The segmentation masks output of the initial SegFormer model are then refined using the SAMRefiner algorithm [24], a universal and efficient approach that adapts Segment Anything Model (SAM) to the mask refinement task [22]. The refined masks are then used to retrain the UAV model, improving final segmentation accuracy.

This approach:

* Transfers fine-scale information from ASV images to UAV segmentation models, reducing the need for manual labelling to the simpler task of labelling underwater images at the image level.

* Improves the spatial representation of coral morphotypes and habitats in aerial imagery, by passing from image classification to semantic segmentation.

* Enables large-scale coral mapping with high-resolution detail, supporting scalable conservation monitoring.

# 2. Method

This study presents a multi-scale method for generating fine-scale segmentation rasters of coral morphotypes and habitats on UAV imagery. The approach relies on collecting and automatically classifying dense underwater images over a portion of the study area to provide fine-scale details for aerial model training.

First, a deep learning model (the teacher model) is used to classify coral morphotypes in ASV images. These predictions are then linearly interpolated to generate continuous probability maps across the surveyed area. To obtain high-resolution rasters, a high density of images is required in the surveyed zone. For this reason, an ASV is ideal as it allows transects to be spaced less than one metre apart, ensuring complete spatial coverage of the area. This enables the generation of reliable, fine-grained rasters representing coral morphotype distributions. Since ASV images are georeferenced, these rasters can be aligned with aerial images (see [12, 13] for more information on georeferencing), serving as weak segmentation annotations for training a second model (student) on UAV imagery. Finally, the student model predictions are corrected through a mask refinement model, improving the quality of the segmentation masks and enabling the retraining of the model for better accuracy. To sum up, this new methodology follows the
following key steps:

1. Underwater data acquisition and image processing: collect high-density georeferenced ASV images and classify them using *DinoVdeau* model (step 1 in Fig. 1).

2. Interpolation & rasterization: create continuous spatial rasters from image classification outputs of the teacher model [9] (step 2 in Fig. 1).

3. Aerial data acquisition and image processing: build a high-resolution aerial orthophoto from UAV images, ensuring map overlay between ASV spatialised annotation rasters and UAV orthophoto [10, 11].

4. Aerial segmentation dataset construction: split fine-scale annotation rasters and aerial orthophoto into tiles, giving more weight to rare classes.

5. Aerial model training: use the generated annotations to train a student model for UAV-based coral and habitat semantic segmentation (step 3 in Fig. 1).

6. Mask refinement and model retraining: refine student model predictions and retrain a new model to improve segmentation accuracy (steps 4 and 5 in Fig. 1).

We provide the methodological details of each step in the following subsections.

## 2.1. Underwater data acquisition and image processing

Underwater images were collected using an ASV equipped with a GoPro Hero 8 camera and a differential GPS Emlid Reach M2 mounted on a waterproof case. This ASV design builds on the version developed in [15]. ASV missions were conducted along predefined transects, spaced between 0.5 and 1 metre apart (Fig. 2)<fnref n="1" />.

Each ASV image (extracted from raw videos) was enriched with metadata, including latitude, longitude and attitude metadata (roll, pitch and yaw angles), so that the images could be georeferenced and orientated correctly in the survey area (Fig. 2a). Time synchronisation between the camera and GPS clocks, as well as metadata correction, was performed following the procedure described in [12]. To obtain fine-scale predictions of coral morphotypes and habitat types, the publicly available model *DinoVdeau* [9] trained on the open source dataset *Seatizen Atlas image dataset* [8] is used as a teacher model. The model outputs probability scores for 31 classes described in [13], indicating the likelihood of each class being present in a given image (Fig. 2b). Since image classification identifies whether a class is present in an image but does not generate segmentation masks of the class's exact location, no information about their spatial coverage or location within the frame is provided.

The teacher model was applied to underwater images collected during six ASV sessions conducted in the *Trou*

<figure><img src="image_3.png" /><figcaption>Figure 2. Example of the rasterization process on an ASV data collection event in the *Trou d'eau* lagoon in Réunion Island: (a) GPS coordinates of ASV images along the session. The high density of images allows for fine-scale spatial interpolation. (b) Pointwise predictions of *Acropora Branching* presence in each ASV image. (c) Interpolated raster representing the probability of *Acropora Branching* presence at each grid point. (d) Zoomed-in view of the raster and corresponding underwater predictions. The raster provides a continuous spatial representation of coral morphology distributions, enabling the generation of rough segmentation masks that can be used for UAV model training.</figcaption></figure>

lagoon, Réunion Island, generating predictions over a total surveyed area of 16,904 m².

## 2.2. Interpolation & rasterization

To generate dense training labels for UAV image segmentation, ASV pointwise predictions are transformed into continuous spatial rasters.

To ensure a fine spatial resolution, the grid spacing for rasterization was set to the median distance between two consecutive ASV images along the survey (Fig. 2a). Depending on the survey, this distance varied between 0.3 m and 0.35 m. A uniform grid was then created over the survey area, with each grid point representing a pixel in the final raster. Raster generation was performed using linear interpolation.

These rasters are aligned with UAV orthophotos to provide approximate segmentation masks for UAV model training. Figure 2c shows the raster generated from ASV predictions for the *Acropora Branching* class and in Fig. 2d a zoomed-in view of the raster and the corresponding underwater prediction points.

For each ASV data collection event and for each class predicted by the teacher model, a probability raster was generated, representing the likelihood of its presence at every grid point.

<footnote n="1">¹A larger high-resolution version of this figure can be found in the Supplementary Material S1 file.</footnote>
## 2.3. Aerial data acquisition and image processing

Aerial drone images are taken with a DJI Mavic 2 Pro drone. A high-resolution UAV orthophoto is constructed by mosaicking individual UAV images. Since the drone is not equipped with a differential GPS, once images were taken and the Structure from Motion (SfM) model was built, the orthophoto positioning is refined by collecting ground control points (GCPs) on the surveyed area using a differential GPS.

Two missions were carried out in the lagoon of Réunion Island: one in the *Saint-Leu* lagoon [10] and the other in the *Trou d'eau* lagoon [11], measuring 204,748 m<fnref n="2" /> and 189,682 m<fnref n="2" /> respectively. The ground sampling distance (GSD) of the resulting orthophotos was 0.9 cm for *Trou d'eau* and 1.6 cm for *Saint-Leu*, reflecting differences in flight altitude and mission setup. The model is trained on the *Trou d'eau* lagoon and tested on the *Saint-Leu* lagoon, in order to evaluate its ability to generalize across geographically distinct sites without site-specific tuning.

## 2.4. Aerial segmentation dataset

### 2.4.1. Classes selection

To simplify the segmentation task, we retained only classes that are clearly distinguishable in aerial images. Indeed, while the underwater teacher model was trained on a broader set of classes, many of them cannot be reliably identified from UAV imagery due to their size (*Algae* classes, small coral colonies like *Acropora Digitate*) spectral similarity (*Dead coral* and *Rubble*) or simply lack of resolution in the aerial images<fnref n="2" />. Thus, for the purpose of a proof of concept, we selected the following subset of classes, which exhibit clear visual patterns in aerial images:

1. *Acropora Branching*

4. *Other Corals*

2. *Acropora Tabular*

5. *Sand*

3. *Non-acropora Massive*

The first two classes are common coral morphotypes of the genus *Acropora* that can be found in the lagoons of Réunion Island. *Acropora Branching* is a branching coral easily distinguishable from aerial images due to its particular colour (dark brown) and is often found in large colonies. *Acropora Tabular* is a tabular coral, slightly more difficult to identify on aerial images but still distinguishable due to the particular layer shape of its colonies. The third class *Non-acropora Massive* is a stony coral with a light colour, often found alone surrounded by sand. The fourth class *Other Corals* encompasses many different species of corals that can range from small colourful colonies of the genus *Pocillopor* to large stony corals that are darker in colour

than *Non-acropora Massive* corals. Finally, the only retained habitat class was *Sand*. Indeed differentiating it from other habitat classes, such as *Rubble* or *Rock*, is a complex task that requires a better resolution than that provided by the UAV images collected.

### 2.4.2. Annotation generation

In the semantic segmentation task, a model is trained to predict a class label for each pixel in an image. To achieve this, a set of images must be labelled so that each pixel corresponds to one and only one class. Let *C* be the set of classes, *I* be the set of images and *N* the cardinality of *C*.

In our case, for an image $i \in I$, we have *N* continuous rasters representing the probability of presence for each class $c \in C$. Thus, we need a function that, for each pixel, maps these probabilities to a class label. A straightforward approach is to assign to each pixel the label of the class with the highest probability of presence.

However, this approach has limitations. Because the underwater teacher model is a multi-label classifier (trained with the binary cross-entropy loss function), the predicted presence/absence probabilities of each class are independent and not necessarily calibrated in the same way.

For common classes such as *Sand*, the teacher model tends to assign very high probabilities to true positives and low values to true negatives, resulting in well-separated bimodal distributions. In contrast, for rare classes such as *Acropora Digitate*, the model shows a lower confidence, with true positive predictions distributed more uniformly across the probability range. This leads to a systematic underestimation of rare classes when using naïve maximum-probability labelling.

To address this, we implement a class-specific quantile normalisation strategy. Following the principles outlined in [38], we normalise the predicted probability maps for each class independently using their empirical percentiles. Specifically, for each class *c*, we aggregate all predicted probabilities generated by the teacher model across all ASV sessions and compute the 1st and 99th percentiles. We then linearly scaled the per-pixel probability values within this range. This ensures that for each class, the normalised values reflect relative confidence within that class's distribution, rather than across the entire set.

Formally, let $p_c(x, y)$ denote the predicted probability of presence for class *c* at pixel location $(x, y)$ in the ASV-derived raster. Let $q_1(c)$ and $q_{99}(c)$ be the empirical 1st and 99th percentile values of the predicted probability distribution for class *c* across all sessions. The normalised probability $\hat{p}_c(x, y)$ is computed as:

$$ \hat{p}_c(x, y) = \text{clip} \left( \frac{p_c(x, y) - q_1(c)}{q_{99}(c) - q_1(c) + \epsilon}, 0, 1 \right) \quad (1) $$

where $\epsilon$ is a small constant added for numerical stability and the clip function ensures that values remain within the valid

<footnote n="2">²Supplementary Material S2 illustrates examples of these limitations, highlighting how ecologically relevant classes identifiable in underwater imagery cannot be reliably distinguished in aerial orthophotos.</footnote>
probability range. This normalisation method improves the naïve maximum selection by taking into account the class-specific prediction dynamics and mitigating the bias caused by imbalanced raw probability distributions. The final label assigned to each pixel is then:

$$L(x, y) = \arg \max_{c \in C} \hat{p}_c(x, y) \qquad (2)$$

This quantile-based normalisation is conceptually similar to class-specific normalisation strategies explored in gene expression data preprocessing, where non-uniform expression distributions across classes can lead to biased analyses if left uncorrected [38]. Applying this principle to semantic segmentation in ecology helps rectify inter-class imbalance, leading to a more equitable representation of rare classes in the resulting annotations.

Finally, since the aerial orthophoto and the ASV annotation rasters do not share the same resolution and since semantic segmentation models require a one-to-one pixel correspondence between the image and annotation, we upsample the ASV annotation rasters to the resolution of the orthophoto by replicating nearest-neighbour pixel values.

## 2.5. Aerial deep learning model (student model)

WSSS provides a framework for training models using approximate labels instead of manually annotated ground truth masks [7]. In our case, the annotations are derived from ASV-based probability maps, which are inherently coarse and may contain noise. These weak labels, combined with the challenges of UAV-based segmentation (such as variable lighting conditions, water clarity differences and limited spatial resolution) introduce significant uncertainty into the training process. As a result, the segmentation model must be robust to label noise and able to generalise to different spatial conditions and environmental contexts.

To address these requirements, we used SegFormer, a transformer-based segmentation architecture that offers several advantages over conventional convolutional neural networks [29, 36]:

* Hierarchical feature representation: SegFormer processes images at multiple scales, allowing it to capture both global contextual information and fine-grained details. This is particularly useful for UAV imagery, where coral morphotypes may have highly variable spatial patterns.

* Robust to weak annotations: unlike models that rely on precise pixel-wise labels, SegFormer effectively learns from weak labels by employing an efficient self-attention mechanism, improving segmentation accuracy even when trained on approximate masks.

* Lightweight and efficient: unlike traditional vision transformers, SegFormer does not require heavy computational resources, making it feasible for large-scale aerial image processing.

Choosing an appropriate loss function is critical in WSSS, where training annotations are coarse and often affected by noise. Region-based losses, such as Dice loss, are especially suitable in these scenarios because they evaluate the overlap between predicted and annotated regions as a whole, rather than focusing on individual pixels or precise edges [1]. Unlike pixel-wise losses, which are sensitive to misaligned labels, or boundary-based losses, which require accurate contour annotations, region-based losses are more tolerant of uncertainty in object boundaries, making them a better fit for learning from approximate masks. Given the nature of this problem, we opted to minimise the region-based Dice loss with an Adam optimiser [21], more details can be found in the Supplementary Material S3 file.

The input and output resolutions are chosen to be 512 x 512 pixels, with a batch size of 16 images. The initial learning rate was initially set to $10^{-5}$ and decayed by a factor of 10 whenever the validation loss plateaued for five consecutive epochs. Further details are available in the GitHub repository.

## 2.6. Mask refinement and model retraining

The final step in our methodology involves refining the initial segmentation masks generated by the student model. A common technique in computer vision for improving model performance without additional supervision is Self Knowledge Distillation (SKD), where the model is retrained using its own predictions as pseudo-labels [37]. In this configuration, the same model assumes the roles of both teacher and student, iteratively improving by distilling and refining its previously generated outputs.

In our case, we explored both this technique and a variant in which the masks are first refined before retraining using a mask refinement model. The chosen model is SAMRefiner, a universal and efficient approach that adapts SAM to the mask refinement task [24].

## 2.7. Test zone and model evaluation

Due to the lack of human annotations, the deep learning model cannot be evaluated on weakly generated annotations using standard computer vision metrics on the test set. For this reason, we selected two test zones: one within the *Trou d'eau* lagoon covering 84 m² and the other in the *Saint-Leu* lagoon covering 194 m². We then manually annotated all coral morphotypes and habitats present in these areas.

These zones were chosen due to their diverse composition of coral morphotypes, habitats and other marine organisms, representing a challenging environment for model validation.
# 3. Results

## 3.1. Evaluation of model variants during pipeline development

To justify the contribution of each component in the proposed pipeline (as illustrated in Fig. 1), we evaluated the segmentation performance of multiple model variants on manually annotated ground-truth data from the test zones in the *Trou d'eau* and *Saint-Leu* lagoons (see Sec. 2.7). The following model configurations were tested to isolate the effect of each step in the pipeline:

*   **Coarse model:** Segformer trained solely on coarse annotations derived from ASV predictions (step 3 in Fig. 1). No mask refinement or self-distillation is applied.

*   **Coarse refined:** coarse annotations are first refined using the SAMRefiner model. Segformer is then trained on these refined masks. No self-distillation is applied.

*   **Coarse self-distilled:** Segformer is initially trained on coarse annotations and then retrained using its own predictions as supervision (self-distillation), without involving SAMRefiner.

*   **Coarse refined + distilled:** full pipeline. Segformer is first trained on coarse annotations, whose predictions are then refined using SAMRefiner. A final Segformer is trained via self-distillation on these refined predictions. This is the final proposed model (step 5 in Fig. 1).

Tab. 1 presents total pixel accuracy, mean Intersection over Union (IoU) and per-class IoU for each model.

<table><caption>Table 1. Comparison of total accuracy and mean IoU across different model variants, along with IoU per class.</caption><thead><tr><th>Model</th><th>Accuracy</th><th>Mean IoU</th><th>Acropora B.</th><th>Acropora T.</th><th>Non-acro M.</th><th>Other Corals</th><th>Sand</th></tr></thead><tbody><tr><td>Coarse</td><td>0.8250</td><td>0.4625</td><td>0.5109</td><td>0.2087</td><td>0.2877</td><td><strong>0.4201</strong></td><td>0.8854</td></tr><tr><td>Coarse Refined</td><td>0.8467</td><td>0.4658</td><td>0.4676</td><td>0.1532</td><td>0.3721</td><td>0.4078</td><td><strong>0.9285</strong></td></tr><tr><td>Coarse Self-Distilled</td><td>0.8300</td><td>0.4745</td><td>0.5175</td><td>0.2114</td><td>0.3395</td><td>0.4190</td><td>0.8852</td></tr><tr><td>Coarse Refined + Distilled</td><td><strong>0.8572</strong></td><td><strong>0.5010</strong></td><td><strong>0.5282</strong></td><td><strong>0.2313</strong></td><td><strong>0.4095</strong></td><td>0.4111</td><td>0.9249</td></tr></tbody></table>

The results show that although the *Coarse* and the *Coarse refined* model variants achieve a slightly better IoU for *Other Corals* and *Sand* classes respectively, only the full pipeline offers a more balanced trade-off between pixel-level accuracy and spatial precision, as indicated by its superior accuracy and mean IoU scores. This progressive evaluation validates the effectiveness of the proposed weakly supervised multi-scale training framework.

## 3.2. Model performance on annotated reef zones

We now focus on the performance of the final model (i.e., the *Coarse refined + distilled* variant) on the same annotated test zones used in the previous section, namely the *Trou d'eau* and *Saint-Leu* lagoons (see Sec. 2.7). This assessment aims to characterise the spatial accuracy and ecological relevance of the predicted segmentation masks at the UAV scale.

Per-zone and per-class metrics are presented in Tab. 2.

<table><caption>Table 2. IoU per class, total accuracy and mean IoU by zone.</caption><thead><tr><th>Zone</th><th>Accuracy</th><th>Mean IoU</th><th>Acropora B.</th><th>Acropora T.</th><th>Non-acro M.</th><th>Other Corals</th><th>Sand</th></tr></thead><tbody><tr><td>Trou d'eau</td><td>0.9053</td><td>0.4493</td><td>0.2354</td><td>0.3623</td><td>0.3295</td><td>0.3707</td><td>0.9484</td></tr><tr><td>Saint-Leu</td><td>0.8160</td><td>0.5793</td><td>0.5641</td><td>/</td><td>0.4318</td><td>0.4242</td><td>0.8972</td></tr><tr><td>Total</td><td><strong>0.8572</strong></td><td><strong>0.5010</strong></td><td><strong>0.5282</strong></td><td><strong>0.2313</strong></td><td><strong>0.4095</strong></td><td><strong>0.4111</strong></td><td><strong>0.9249</strong></td></tr></tbody></table>

The total pixel accuracy reached 85.72%, while the mean IoU was 50.10%. The confusion matrix, showing the percentage of true class pixels predicted for each class, along with misclassification errors, is presented in Fig. 3.

<figure><img src="image_4.png" /><figcaption>Figure 3. Normalized confusion matrix computed across both test zones (*Trou d'eau* and *Saint-Leu*). Values represent the percentage of true class pixels predicted for each class.</figcaption></figure>

## 3.3. Predicted habitat composition across reef zones

To assess the spatial consistency and ecological relevance of the segmentation predictions across large reef areas, we applied the trained UAV model to the entire UAV orthophotos of the *Trou d'eau* and *Saint-Leu* lagoons, measuring 189,682 m² and 204,748 m² respectively.

In Fig. 4 the model predictions over the entire orthophoto of the *Saint-Leu* lagoon are displayed, accompanied by a pie chart summarising the percentage cover of each predicted class<fnref n="3" />.

# 4. Discussion

The results of the multi-scale upscaling process demonstrate the feasibility of training a UAV-based coral segmentation model using weak annotations derived from ASV-based image classification. As shown in Fig. 4, the benthic coverage for each class can be quantified while highlighting

<footnote n="3">³A larger high-resolution version of this figure, along with predictions for the *Trou d'eau* lagoon and a zoomed-in view of a specific area, are provided in the *Supplementary Material S1* file.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 4. Results of the UAV model applied to the entire or-
thophoto of the *Saint-Leu* lagoon. The model predicts the pres-
ence of coral morphotypes and habitats across the entire reef area.
Piecharts can then be computed to quantify the predicted habitat
composition.</figcaption></figure>

spatial patterns throughout the reef scape. Although sur-
face area is a useful metric for many benthic classes, it may
be less informative for others (e.g., *Non-acropora Massive*
or mobile species) where the number of detected individ-
uals serves as a more ecologically relevant indicator. As
demonstrated in Sec. 4.2, the proposed method also enables
individual-level detection and counting for such classes.
*Supplementary Material S1* file presents a zoomed-in view
of a specific area in the *Trou d'eau* lagoon and the cor-
responding segmentation mask. While minor misclassifi-
cation errors can be observed, such as isolated spurious
patches of *Non-acropora Massive*, the overall quality of the
segmentation is encouraging. The predicted masks accu-
rately reflect the structure and extent of the main coral mor-
photypes, with clear and coherent shapes that align well
with the visible habitat patterns in the aerial imagery. This
qualitative assessment highlights the potential of the model
to support spatial analyses of reef composition and guide
targeted conservation actions.

**4.1. Aerial model evaluation**

With a total accuracy of 85.72% and a mean IoU of 50.10%.
(see Tab. 2), the model effectively captures the spatial dis-
tribution of coral morphotypes and habitats in aerial im-
agery. The difference between the two metrics indicates that
the model is biased towards the most common class, *Sand*,
which is easily recognisable and segmentable in aerial im-
ages. Its distinct spectral signature and broad spatial cover-
age in aerial imagery likely contributed to the high recall of
93.6% (see Fig. 3). The model also performed well on the
three coral classes *Acropora Branching*, *Acropora Tabular*
and *Non-acropora Massive*, correctly identifying 79.6%,

85.5% and 83.6% of the annotated pixels for each class re-
spectively.

The lowest performance is encountered for the *Other Corals* class. This class, grouping various coral morphotypes, is characterised by high intra-class variability and spectral ambiguity, leading to a lower performance of 47.3%. For example the *Acropora Digitate* morphotype, which is a small coral colony being part of this class, can be easily confused on aerial images with *Acropora Branching* or *Acropora Tabular* morphotypes, which explains the high misclassification rates of 29.2% and 15.6% respectively, as shown in Fig. 3. Furthermore, the *Other Corals* class groups a wide variety of coral morphotypes that differ greatly in appearance and structure. For instance, it includes both large, dome-shaped colonies that resemble massive corals and also flat, spreading forms like encrusting corals, which are morphologically and spectrally distinct. This high intra-class variability makes it difficult for the model to learn coherent and consistent features, reducing its classification performance. While the current model was not trained to recognize the full spectrum of coral morphotypes, the methodology itself is generic and has been validated for its ability to incorporate new classes as needed (see Sec. 4.2). Although subdividing the *Other Corals* category into more specific morphotypes would likely improve accuracy, this was not pursued in the present study due to the need for more detailed annotations, an expanded underwater dataset and higher-resolution UAV imagery to ensure reliable training.

**4.2. Expanding to new ecological classes**

The presented methodology can be easily extended to in-
clude additional benthic classes, such as specific types of
seagrass or other coral morphotypes not found in Reunion
Island. To incorporate new classes, it is sufficient to col-
lect both underwater and aerial images in a region where
the new class is present and to train the student model using
the ASV-derived annotations following the same upscaling
methodology. It is important to note that the method re-
mains applicable regardless of the specific architecture of
the underwater model, as long as it provides reliable class-
level predictions (whether from monolabel or multilabel
classifiers). This flexibility ensures the continued relevance
and adaptability of the approach as AI models evolve, while
also enabling its transferability to different reef systems for
broader regional monitoring without requiring changes to
the core pipeline.

Another way to incorporate new species, including mo-
bile ones such as sea cucumbers or turtles, is to manually
annotate a small number of instances directly in the aerial
imagery. For slow-moving species such as sea cucumbers,
the collection of fine-scale ASV images and UAV images
simultaneously can provide the spatial and temporal align-
ment required to propagate class labels from underwater predictions to aerial images [12]. However, weather conditions or logistical constraints may prevent synchronised data collection. In addition, for fast-moving species such as turtles, simultaneous acquisition is not feasible. In these situations, a few manual annotations can be added on top of ASV-generated masks to introduce new classes to the training dataset. The introduction of SAM in 2023 [22] has greatly simplified the annotation process. Tools such as the *Geo-SAM* plugin for QGIS [39] allow users to create object masks by clicking on points or drawing bounding boxes, reducing the time required for detailed labelling. For instance, using Geo-SAM, we manually annotated 338 sea cucumbers in a 13,840 m² area of the *Trou d'eau* lagoon in approximately 55 minutes. These annotations were integrated with the existing model predictions in the training zone of Sec. 2.1 by overlaying them onto the predicted masks. The student model was then retrained from scratch to include the new *Sea cucumber* class. Given that sea cucumbers are small organisms occupying a limited number of pixels in aerial images, their representation in the training data was proportionally low. To address this imbalance and ensure the model could effectively learn to detect them, we adjusted the loss function by applying a weighted Dice loss, assigning a higher penalty to errors involving sea cucumbers. This strategy increased the model's sensitivity to the new class, but reduced the overall accuracy, see *Supplementary Material S4* for more details. An example of the final segmentation output is shown in Fig. 5.

<figure><img src="image_6.png" /><figcaption>Figure 5. Example of the segmentation output for the *Sea cucumber* class. (a) Zoomed-in view of the UAV orthophoto of the *Trou d'eau* lagoon showing the presence of sea cucumbers. (b) Corresponding predicted segmentation mask, showing the spatial distribution of sea cucumbers.</figcaption></figure>

Fig. 5a illustrates a zoomed-in view of the UAV orthophoto of the *Trou d'eau* lagoon showing the presence of sea cucumbers and in Fig. 5b the corresponding predicted segmentation mask. Some key ecological information can be extracted from such maps, including the spatial density of individuals per square metre and estimates of the size of the organism. For instance, the average length of each detected sea cucumber can be approximated by measuring the longest axis within each predicted segmentation

mask. In the back reef depression of the *Trou d'eau* lagoon, this method yielded an average length of 19.6 ± 0.24 cm, calculated over 742 predicted individuals. These estimates are consistent with *in situ* measurements reported in [26], where average lengths of 24.6 ± 0.5 cm and 12.5 ± 0.7 cm were recorded for the sea cucumber species *Holothuria leucospilota* and *Stichopus chloronotus*, respectively. Based on 72 individuals per species (144 in total), this yields a combined average length of 18.5 cm, in close agreement with our estimate.

## 5. Conclusion

This study introduces a multi-scale WSSS framework for coral reef monitoring that transfers fine-scale information from underwater imagery to large-scale aerial assessments. Using dense ASV image collections and a deep learning classification model, we generated spatially interpolated probability maps that serve as weak annotations for training UAV-based segmentation models, eliminating the need for pixel-level labels.

Despite coarse supervision, the method enables accurate segmentation of key benthic classes and captures ecologically relevant spatial patterns. Region-based loss functions and model refinement via self-distillation further improve segmentation performance.

The approach is flexible and extensible. New classes can be incorporated either by collecting a small number of additional ASV sessions in areas where these classes are present or by manually annotating a limited number of instances directly in aerial imagery, as demonstrated for sea cucumbers.

This framework offers a practical and cost-effective solution for ecological monitoring in complex and changing marine environments. By reducing annotation requirements and leveraging multi-scale data sources, it contributes to the development of robust, scalable tools for automated coral reef assessment.

## 6. Acknowledgements

The authors gratefully thank the anonymous reviewers for their constructive comments, which helped improve the clarity and quality of the manuscript considerably. This work was supported by: Seatizen (Ifremer internal grant), Plancha (supported by the Contrat de convergence et de transformation 2019-2022, mesure 3.3.1.1 de la Préfecture de la Réunion, France), IOT project (funded by FEDER INTERREG V and Prefet de La Réunion: grant #20181306-0018039 and the Contrat de Convergence et de Transformation de la Préfecture de La Réunion), Ocean and Climate Priority Research Programme, FISH-PREDICT project (funded by the IA-Biodiv ANR project: ANR-21-AAFI-0001-01) and G2OI FEDER INTERREG V (grant #20201454-0018095).
## References

[1] Reza Azad, Moein Heidary, Kadir Yilmaz, Michael Hüttemann, Sanaz Karimijafarbigloo, Yuli Wu, Anke Schmeink, and Dorit Merhof. Loss functions in the era of semantic segmentation: A survey and outlook. *arXiv preprint arXiv:2312.05391*, 2023. 5

[2] Marc Besson, Jamie Alison, Kim Bjerge, Thomas E Gorochowski, Toke T Høye, Tommaso Jucker, Hjalte MR Mann, and Christopher F Clements. Towards the fully automated monitoring of ecological communities. *Ecology Letters*, 25 (12):2753–2775, 2022. 2

[3] Angel Borja, Torsten Berg, Hege Gundersen, Anders Gjørwad Hagen, Kasper Hancke, Samuli Korpinen, Miguel C Leal, Tiziana Luisetti, Iratxe Menchaca, Ciaran Murray, et al. Innovative and practical tools for monitoring and assessing biodiversity status and impacts of multiple human pressures in marine systems. *Environmental Monitoring and Assessment*, 196(8):694, 2024. 2

[4] Léo Broudic, Mathieu Pinault, Romain Claud, Touria Bajjouk, Tévamie Rungassamy, Natacha Nikolic, Estelle Crochelet, Camille Mazé, and Benjamin Bergerot. Mapping benthic biodiversity indicators of coral reefs using spatial interpolation. *Coral Reefs*, 2025. 2

[5] Andrew W Bruckner. Life-saving products from coral reefs. *Issues in Science and Technology*, 18(3):39–44, 2002. 2

[6] Jennifer A Cardenas, Zahra Samadikhoshkho, Ateeq Ur Rehman, Alexander U Valle-Pérez, Elena Herrera-Ponce de León, Charlotte AE Hauser, Eric M Feron, and Rafiq Ahmad. A systematic review of robotic efficacy in coral reef monitoring techniques. *Marine Pollution Bulletin*, 202:116273, 2024. 2

[7] Lyndon Chan, Mahdi S Hosseini, and Konstantinos N Plataniotis. A comprehensive analysis of weakly-supervised semantic segmentation in different image domains. *International Journal of Computer Vision*, 129(2):361–384, 2021. 5

[8] Matteo Contini, Julien Barde, Sylvain Bonhommeau, Victor Illien, and Alexis Joly. Seatizen atlas image dataset, 2024. 3

[9] Matteo Contini, César Leblanc, and Victor Illien. Dinovdeau-large-2024_04_03-with_dataAug_batch-size32_epochs150_freeze – Hugging Face. https://doi.org/10.57967/hf/2947, 2024. 3

[10] Matteo Contini, Sylvain Bonhommeau, Serge Bernard, and Julien Barde. Aerial images collected by an unmanned aerial vehicle in st-leu, réunion - 2023-12-08 (processed.data), 2025. 3, 4

[11] Matteo Contini, Sylvain Bonhommeau, Serge Bernard, and Julien Barde. Aerial images collected by an unmanned aerial vehicle in trou-deau, réunion - 2023-12-02 (processed.data), 2025. 3, 4

[12] Matteo Contini, Victor Illien, Julien Barde, Sylvain Poulain, Serge Bernard, Alexis Joly, and Sylvain Bonhommeau. From underwater to drone: A novel multi-scale knowledge distillation approach for coral reef monitoring. *Ecological Informatics*, page 103149, 2025. 2, 3, 8

[13] Matteo Contini, Victor Illien, Mohan Julien, Mervyn Ravitchandirane, Victor Russias, Arthur Lazennec, Thomas

Chevrier, Cam Ly Rintz, Léanne Carpentier, Pierre Gogendeau, et al. Seatizen atlas: a collaborative dataset of underwater and aerial marine imagery. *Scientific Data*, 12(1):67, 2025. 2, 3

[14] Michaela Doukari and Konstantinos Topouzelis. Overcoming the uas limitations in the coastal environment for accurate habitat mapping. *Remote Sensing Applications: Society and Environment*, 26:100726, 2022. 2

[15] Pierre Gogendeau, Sylvain Bonhommeau, Hassen Fourati, Mohan Julien, Matteo Contini, Thomas Chevrier, Anne Elise Nieblas, and Serge Bernard. An autonomous surface vehicle for acoustic tracking, bathymetric and photogrammetric surveys. *Ocean Engineering*, 331:121201, 2025. 2, 3

[16] Deny Hidayati et al. The importance of the sustainable use of fishery resources to improve the livelihoods of fishermen on the islands of sumatra and sulawesi, Indonesia. In *Proceedings of the 5th Conference on Agribusiness, Green Energy, Environment, and Sustainable Development (CAGEES-V5)*, pages 120–141, 2022. 2

[17] Ove Hoegh-Guldberg, Peter Mumby, A.J. Hooten, R.S. Steenneck, Paul Greenfield, Erick Gomez, Catherine Harvell, Peter Sale, Alasdair Edwards, Ken Caldeira, Nancy Knowlton, C. Mark Eakin, Roberto Iglesias-Prieto, Nyawira Muthiga, Roger Bradbury, Alfonse Dubi, and M Hatziolos. Coral reefs under rapid climate change and ocean acidification. *Science* (*New York, N.Y.*), 318:1737–42, 2008. 2

[18] Liwei Huang, Bitao Jiang, Shouye Lv, Yanbo Liu, and Ying Fu. Deep-learning-based semantic segmentation of remote sensing images: A survey. *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, 17: 8370–8396, 2024. 2

[19] Terry P Hughes, Andrew H Baird, David R Bellwood, Margaret Card, Sean R Connolly, Carl Folke, Richard Grosberg, Ove Hoegh-Guldberg, Jeremy BC Jackson, Janice Kleypas, et al. Climate change, human impacts, and the resilience of coral reefs. *science*, 301(5635):929–933, 2003. 2

[20] Terry P Hughes, James T Kerry, Mariana Álvarez-Noriega, Jorge G Álvarez-Romero, Kristen D Anderson, Andrew H Baird, Russell C Babcock, Maria Beger, David R Bellwood, Ray Berkelmans, et al. Global warming and recurrent mass bleaching of corals. *Nature*, 543(7645):373–377, 2017. 2

[21] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014. 5

[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 4015–4026, 2023. 2, 8

[23] César Leblanc, Lukas Picek, Rémi Palard, Benjamin Deneu, Maximilien Servajean, Pierre Bonnet, and Alexis Joly. Mapping biodiversity at very-high resolution in europe. In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 2349–2358, 2025. 2

[24] Yuqi Lin, Hengjia Li, Wenqi Shao, Zheng Yang, Jun Zhao, Xiaofei He, Ping Luo, and Kaipeng Zhang. Samrefiner: Taming segment anything model for universal mask refinement. *arXiv preprint arXiv:2502.06756*, 2025. 2, 5
[25] Benjamin Misiuk and Craig J. Brown. Benthic habitat mapping: A review of three decades of mapping biological patterns on the seafloor. *Estuarine, Coastal and Shelf Science*, 296:108599–2024. 2

[26] Joséphine Pierrat, Léa Urbistondoy, Alexandre Modi, Betsy Viramoutou, and Patrick Frouin. Searching for drivers of the patchy distribution of sympatric deposit-feeding sea cucumbers: A multi-scale monitoring study. *Limnology and Oceanography*, 69(9):2057–2070, 2024. 8

[27] Liang Qu, Xiaoli Song, Mengmeng Zhang, Juan Wang, Ruobing Wen, and Shengke Wang. Gan-based defogging and multiscale fusion approach for uav-based seagrass bed imagery semantic segmentation in challenging marine environments. In *International Conference of Pioneering Computer Scientists, Engineers and Educators*, pages 55–72. Springer, 2024. 2

[28] Alice Rogers, Julia Blanchard, and Peter Mumby. Fisheries productivity under progressive coral reef degradation. *Journal of Applied Ecology*, 55, 2017. 2

[29] Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du. Learning affinity from attention: End-to-end weakly-supervised semantic segmentation with transformers. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 16846–16855, 2022. 5

[30] Jonathan Sauder, Guilhem Banc-Prandi, Anders Meibom, and Devis Tuia. Scalable semantic 3d mapping of coral reefs with deep learning. *Methods in Ecology and Evolution*, 15 (5):916–934, 2024. 2

[31] Jonathan Sauder, Viktor Domazetoski, Guilhem Banc-Prandi, Gabriela Perna, Anders Meibom, and Devis Tuia. The coralscapes dataset: Semantic scene understanding in coral reefs. *arXiv preprint arXiv:2503.20000*, 2025. 2

[32] Justine Talpaert Daudon, Matteo Contini, Isabel Urbina-Barreto, Brianna Elliott, François Guilhaumon, Alexis Joly, Sylvain Bonhommeau, and Julien Barde. Geoai for marine ecosystem monitoring: a complete workflow to generate maps from ai model predictions. In *The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences*, pages 223–230. Copernicus GmbH, 2023. 2

[33] Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank Van Langevelde, Tilo Burghardt, et al. Perspectives in machine learning for wildlife conservation. *Nature communications*, 13(1):1–15, 2022. 2

[34] Joost W van Dam, Andrew P Negri, Sven Uthicke, and Jochen F Mueller. Chemical pollution on coral reefs: exposure and ecological effects. In *Ecological Impacts of Toxic Chemicals*, pages 187–211. Bentham Science Publishers, 2011. 2

[35] Junling Wang, Yupeng Wang, Liping Liu, Hengfu Yin, Ning Ye, and Can Xu. Weakly supervised forest fire segmentation in uav imagery based on foreground-aware pooling and context-aware loss. *Remote Sensing*, 15(14):3606, 2023. 2

[36] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In

*Advances in Neural Information Processing Systems*, pages 12077–12090. Curran Associates, Inc., 2021. 2, 5

[37] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. *arXiv preprint arXiv:2402.13116*, 2024. 5

[38] Yaxing Zhao, Limsoon Wong, and Wilson Wen Bin Goh. How to do quantile normalization correctly for gene expression data analyses. *Scientific reports*, 10(1):15534, 2020. 4, 5

[39] Zhuoyi Zhao, Chengyan Fan, and Lin Liu. Geo SAM: A QGIS plugin using Segment Anything Model (SAM) to accelerate geospatial image segmentation, 2023. 8

[40] Kristina Øie Kvile, Hege Gundersen, Robert Nøddebo Poulsen, James Edward Sample, Arnt-Børre Salberg, Medyan Esam Ghareeb, Toms Buls, Trine Bekkby, and Kasper Hancke. Drone and ground-truth data collection, image annotation and machine learning: A protocol for coastal habitat mapping and classification. *MethodsX*, 13:102935, 2024. 2