Nacala-Roof-Material: Drone Imagery for Roof Detection, Classification, and Segmentation to Support Mosquito-borne Disease Risk Assessment

## Abstract

Abstract As low-quality housing and in particular certain roof characteristics are associated with an increased risk of malaria, classification of roof types based on remote sensing imagery can support the assessment of malaria risk and thereby help prevent the disease. To support research in this area, we release the Nacala-Roof-Material dataset, which contains high-resolution drone images from Mozambique with corresponding labels delineating houses and specifying their roof types.
The dataset defines a multi-task computer vision problem, comprising object detection, classification, and segmentation.
In addition, we benchmarked various state-of-the-art approaches on the dataset. Canonical U-Nets, YOLOv8, and a custom decoder on pretrained DINOv2 served as baselines. We show that each of the methods has its advantages but none is superior on all tasks, which highlights the potential of our dataset for future research in multi-task learning. While the tasks are closely related, accurate segmentation of objects does not necessarily imply accurate instance separation, and vice versa.
We address this general issue by introducing a variant of the deep ordinal watershed (DOW) approach that additionally separates the interior of objects, allowing for improved object delineation and separation.
We show that our DOW variant is a generic approach that improves the performance of both U-Net and DINOv2 backbones, leading to a better trade-off between semantic segmentation and instance segmentation.

## 1 Introduction

Mosquito-borne diseases refer to a group of infectious illnesses transmitted by the bite of mosquitoes. Malaria is a mosquito-borne disease caused by single-celled parasites of the Plasmodium group spread through bites of infected female Anopheles mosquitoes. It ranks among the world’s most severe public health problems and is a leading cause of mortality and disease in many developing countries. It is therefore crucial to improve prevention, control, and surveillance measures of malaria, particularly in sub-Saharan Africa (Venkatesan, ; WHO, ).
Low-quality housing built of natural materials, for example, having a thatched roof of grass or palm and having cane, grass, shrub, or mud as internal and external walls, is associated with an increased risk of malaria infection (Dlamini et al., ).
Sub-standard housing has more mosquito entry points and most malaria transmissions in sub-Saharan Africa occur inside dwellings while the inhabitants are asleep (Tusting et al., , ; Jatta et al., ; Tusting et al., ).
Houses with metal roofs are hotter in the daytime than houses with thatched roofs. This may reduce mosquito survival and inhibit parasite development within the mosquito in metal roof houses. On this basis, the proliferation of modern construction materials in sub-Saharan Africa may have contributed decisively to the reduction of malaria cases (Tusting et al., ).
Classification of roof characteristics thus holds potential to support malaria surveillance and control programs.
Roof characteristics, such as geometry, material, and condition can be monitored using remote sensing imagery to advance risk assessment of mosquito-borne diseases and guide mitigation strategies, especially when detailed health and socioeconomic data are scarce.

Here, we introduce the Nacala-Roof-Material drone-imagery dataset to support the development of machine learning algorithms for automated building *and* roof type mapping in low-income areas prone to malaria risk.
Our dataset is based on high-resolution drone imagery ($\approx 4.4$ cm) of peri-urban and rural settlements in Nacala, Mozambique.
The Mozambican NGO #MapeandoMeuBairro delineated 17 954 buildings and categorized them according to five roof types, and the authors again carefully verified all annotations.
We define three tasks on the Nacala-Roof-Material dataset,
building detection, multi-class roof type classification, and pixel-level building segmentation.

While these tasks are related, closer inspection reveals a misalignment between their objectives.
Accurate segmentation as measured by the intersection over union (IoU) does not necessarily imply accurate object separation, and vice versa.
For accurate detection and classification, it would be sufficient to only detect the interior of an object as long as the segmented area allows to correctly classify the type.
If the roofs of two buildings are (almost) touching, then some segmentation may have a high IoU but could make it difficult to separate buildings for counting. This is also a common issue in other applications, e.g., when studying cells in medical images (Ronneberger et al., ) or trees from satellite images (Brandt et al., ; Mugabowindekwe et al., )).

We benchmark three conceptually different state-of-the-art approaches on our multi-task dataset.
First, we evaluate YOLOv8 (Jocher et al., ) developed for object detection, classification, and instance segmentation.
Second, we build a segmentation model based on DINOv2 (Oquab et al., ), a state-of-the-art pretrained vision transformer.
Lastly, we evaluate U-Net (Ronneberger et al., ) a fully-convolutional encoder-decoder architecture, designed for semantic segmentation.
To address the potential conflicts between pixel-level segmentation and correct object separation as outlined above, we propose a simple approach based on the recent work by cheng2024scattered, which we refer to as the Deep Ordinal Watershed (DOW) method.
We extend both U-Net and DINOv2 to produce an additional output map that predicts the interior of objects.
While the original exterior segmentation map maximizes the IoU, we show that the interior map supports object separation.

The main contributions of our work are the following:

- 1.
We provide the Nacala-Roof-Material dataset containing drone imagery from peri-urban and rural areas in a sub-Saharan African region. The dataset contains accurate segmentation labels for buildings, categorized into five roof types.
- 2.
Based on the dataset, we define a multi-task machine learning benchmark for binary and multi-class object detection and semantic segmentation.
We implemented and benchmarked different carefully adopted baseline methods, reflecting three different approaches to address these tasks.
- 3.
We propose a general and simple approach to extend models for semantic segmentation to yield good segmentation *and* object separation results.

The data and code for reproducing the experiments are made freely available at [https://mosquito-risk.github.io/Nacala](https://mosquito-risk.github.io/Nacala).

The next section presents the Nacala-Roof-Material data, provides some background about roof types and risk of vector-borne diseases, and briefly discussed related datasets.
Section 3 describes the deep learning models we evaluated with an emphasis on deep watershed methods. Experimental results are presented in Section 4 before we conclude.

## 2 Nacala-Roof-Material Data

#### Background: Housing conditions and risk of mosquito-borne diseases.

In sub-Saharan Africa, housing conditions, health outcomes, and socioeconomic status of the residents are interrelated (gram-hansen2019mapping; abraham2019; tusting2020). As poverty is widespread, diseases are more prevalent, and data are scarce in this region, automatic profiling of housing conditions based on analysis of satellite imagery holds the potential to estimate the socioeconomic status of the inhabitants and assess the risk of disease. This may in turn support targeted public health interventions.

Mosquitoes are vectors for diseases such as malaria, dengue, Zika, West Nile fever, Chikungunya, and Yellow fever.
In 2022, more than $600\,000$ deaths occurred due to malaria globally and out of the approximately 249 million documented cases, around 233 million occurred within the WHO African Region, accounting for roughly around 94% of the total documented cases. The economic impact of malaria in Sub-Saharan Africa not only impedes progress towards achieving Sustainable Development Goal 3 (Good Health and Well-being) but also undermines efforts to attain SDG 1 (No Poverty) and SDG 8 (Decent Work and Economic Growth) by compromising economic productivity. Extreme weather conditions caused by climate change will likely exacerbate problems with mosquito-borne diseases in sub-Saharan Africa, as floods are expected to increase in frequency and have been linked to outbreaks of malaria in Africa (10665-268220).

Low-quality housing increases the risk of transmission of diseases by mosquitoes, as sub-standard houses have more mosquito entry points and thereby increase human exposure to infection in the home (tusting2015; nomcebo_2017). Mosquito survival is lower in metal-roof houses compared to thatched-roof houses due to higher daytime temperatures (tusting2015). Most malaria transmissions in sub-Saharan Africa occur indoors at night, and poor climatic performance of housing has been linked to increased malaria risk (JATTA2018e498). This is because elevated indoor temperatures can cause discomfort for inhabitants, which may result in decreased use of mosquito nets during the night. Roof materials, geometry, and conditions are critical for indoor climate, as roofs comprise the primary surface exposed to the sun. Automatic classification of roof characteristics thus holds potential for informing risk assessment of malaria and support targeted interventions.

#### The Nacala-Roof-Material dataset.

We gathered drone imagery of the Nacala region in Mozambique.
The burden of malaria in Mozambique is approximately 10-fold the world average (number of documented cases compared to the total population, VENKATESAN2024e214).
The data covers three informal settlements of Nacala, a city of $350\,000$ inhabitants on the northern coast of Mozambique.
Aerial imagery was collected using a DJI Phantom 4 Pro drone and processed using AgiSoft Metashape software.
All data was recorded between October and December 2021, under a development project led by #MapeandoMeuBairro and supported by Nacala Municipal Council.
The image resolution is $\approx 4.4$ cm, and we made all raw imagery available in OpenAerialMap (openaerialmap).
The total number of buildings in the study areas is $17\,954$.
We distinguished five major types of roof materials in Nacala, namely metal sheet, thatch, asbestos, concrete, and no-roof, and their counts are $9776$, $6428$, $566$, $174$, and $1010$, respectively.
The region is mostly dominated by metal sheets and thatch roofs.

Figure: Figure 1: (a) Visualisation of the training, validation and test sets with reference to longitude and latitude; (b) Drone imagery with labels; (c) Instance counts for each class in all sets.
Refer to caption: extracted/5651730/figures/data5CI.png

From the three informal settlements, see Figure 1, the first two areas were split into training $\mathcal{D}_{\text{train}}$, validation $\mathcal{D}_{\text{val}}$, and test $\mathcal{D}_{\text{test}}$ using stratified sampling. We created a square grid of 225 meters and counted the roof types in these cells. Then we partitioned the cells into three sets based on the class counts to achieve a similar class distribution in each set, where we prioritized the distribution of minority classes (i.e., concrete and asbestos).
We defined that a building only belongs to a specific grid cell if its centroid falls into the cell.
If a building area falls into two grid cells and those two cells belong to two different sets (e.g., training and test set), we choose to have data pixels in the set where the centroid of the building is placed. The remaining part of the building in the other set was masked to avoid data leaking between sets.

Although objects in training, validation, and test sets are from different cells, they stem from the same two areas.
To evaluate the generalization to a new area without adjacent training data, we hold out the third settlement as a second test set referred to as $\mathcal{D}_{\text{ext}}$.

#### Related datasets.

The project “Mapping Informal Settlements in Developing Countries using Machine Learning with Noisy Annotations and Multi-resolution Multi-spectral Data” (helber2018generating; gram-hansen2019mapping) is most closely related to our work.
They used freely available 10m/pixel resolution imagery from the Sentinel-2 satellite and obtained labels for three roof types (metal, shingles, thatch)
from geo-located survey data provided by Afrobarometer(^1^11[www.afrobarometer.org](www.afrobarometer.org)). These labels are very noisy in space and time. The labels are often not aligned with buildings because the geo-located coordinates were distorted for privacy reasons. Furthermore, the survey questions and satellite image observations may not be aligned in time.
While the low spatial resolution of the Sentinel-2 imagery might allow to cover large geographic regions, it makes roof type classification challenging (helber2018generating).

There are many datasets that contain remote sensing imagery with building labels, which, however, typically do not distinguish roof types.
In particular, *Open Buildings* is a freely available continental-scale building dataset covering the whole of Africa
(sirko2021continental). In comparison, Nacala-Roof-Material is much more focused, providing significantly higher resolution images, more accurate delineations, and in particular roof type classifications.

alidoost18 distinguish between roof types in aerial images. However, they map a rather high-income town in Germany, where they distinguish between
three roof shapes common in that region (flat, gable, and hip). Another dataset for classifying roof geometry is provided by persello2023, who distinguish 12 fine-grained details of roof geometry.

## 3 Benchmarked Methods

This section presents the approaches we benchmarked on the Nacala-Roof-Material data set.
The goal is to accurately segment the buildings (as assessed by metrics based on the IoU), separate individual buildings, and classify the roof materials.
As baselines, we considered U-Net (ronneberger2015unet), YOLOv8 (yolov8), and a model performing segmentation based on DINOv2 (oquab2024dinov).
Furthermore, we extend the U-Net and the DINOv2 based systems with the deep ordinal watershed method recently proposed by cheng2024scattered.
These approaches are compared in two settings.
In the *two-stage* setting, we first solved the building segmentation and separation tasks and afterwards classified the roof material for each detected building. In the *end-to-end* setting, segmentation and classification were done in parallel.

### 3.1 Baseline Models

Figure: Figure 2: Baseline (top) and DOW (bottom) variants of our systems using either ResNet35 (in the case of the U-Net architectures) or DINOv2 as encoders. When using DOW, The watershed algorithm takes two segmentation masks as input, the predicted objects (level 1) and their interiors (level 2). In the two-stage approach, the classifier shown in Figure 3 is using the binary building segmentation (left). In the end-to-end setting, the roof material is predicted directly with a multi-class segmentation approach (right).
Refer to caption: x1.png

#### U-Net.

The U-Net is arguably the most common architecture
for semantic segmentation (ronneberger2015unet).
We utilized a ResNet34 (resnet) encoder pretrained on ImageNet and a decoder similar to the original U-Net, except that we used nearest-neighbor upsampling instead of transposed convolutions (nn-upsampling), see Figure A.5 in the Appendix.

To identify individual instances in the semantic segmentation output map, the connected components in the map were determined (brandt:20).
To better separate individual buildings, we used a pixel-wise weight map during training that puts more emphasis on the space between buildings as already suggested by ronneberger2015unet
(see Appendix A.1 for details) and commonly used in remote sensing (e.g. brandt:20).
However, this is not sufficient to separate buildings that are very close to each other or touch each other.
Thus, we modified the target segmentation masks during training: Some border pixels were relabeled as background to ensure that there is a minimum gap of $n_{\text{gap}}=7$ pixels between roofs. This modification of the target masks was only applied during training, before computing the weight map but not when calculating any performance metrics.

#### YOLOv8.

We trained YOLOv8 (yolov8), which is among the state-of-the-art methods for instance segmentation.
We fine-tuned a model pretrained on the COCO dataset.
While the original YOLO architecture was designed for object detection, YOLOv8 allows for instance segmentation by integrating concepts from YOLACT (bolya2019yolact).

#### DINOv2.

We benchmarked an approach based on DINOv2 (oquab2024dinov), a state-of-the-art pretrained vision transformer. It uses the DINOv2 *Base* model as an encoder, which is extended by a convolutional decoder.
The DINOv2 output, a patch embedding with the shape of $\mathbb{R}^{1024\times 768}$, is reshaped into feature maps of size $\mathbb{R}^{32\times 32\times 768}$.
Then convolutional and linear upsampling layers are used on top of these feature maps as a decoder (see Appendix A.3).
We used the same loss function, weighting function, training label adjustment, and training strategy as for U-Net. We froze the encoder weights and only the convolutional decoder was trained.

### 3.2 Deep Ordinal Watershed

U-Nets and the DINOv2 based method described above try to classify each pixel as accurately as possible.
However, for proper separation of objects it is sufficient – and typically preferable – if only the interior of an object is segmented.
If the border of a building can be classified as background, even touching buildings can be separated.
This reasoning leads to the deep ordinal watershed (DOW) model introduced by cheng2024scattered.

In the watershed approach, each pixel is assigned a height and the image is viewed as a topological map (soille1990automated).
A DOW architecture does not only predict a single segmentation mask but $n_{\text{lev}}$ feature maps
for $n_{\text{lev}}+1$ discrete height levels, $\{0,1,\dots,n_{\text{lev}}\}$, where 0 0 corresponds to the highest and $n_{\text{lev}}$ to the lowest elevation. Background pixels are assumed to have level 0. The Euclidean distance transformation is computed for each object, and
the distances are discretized into the remaining $n_{\text{lev}}$ height levels.
Target feature map $m\in\{1,\dots,n_{\text{lev}}\}$ marks all pixel with a distance level of
$m$ or higher.
That is, the objects in the target feature maps get smaller with increasing $m$
(if $n_{\text{lev}}=1$ we recover the standard U-Net).
Learning the discrete height levels of pixels this way solves an ordinal regression task (frank2001simple; cheng2008neural).
Given the pixel heights, the watershed algorithm can be applied as a post-processing step for instance segmentation (soille1990automated).
Local minima in the elevation map define basins, each of which defines a distinct object.
Adopting a flooding metaphor, the watershed algorithm now floods the basins until basins attributed to different starting points meet on watershed lines. Pixels attributed to the same basin belong to the same object.

cheng2024scattered employ a DOW U-Net for individual tree segmentation, however, without a comparison with a standard U-Net or exploring different numbers of levels. For our task, we hypothesize
that a minimal number of $n_{\text{lev}}=2$ different non-background heights is sufficient.
In this setting, the system outputs two masks representing the full object and its interior, respectively. Let $n_{\text{pix}}$ denote the difference in distance between two levels.
The smallest building in our data set has size $1.463\text{\,}\mathrm{m}^{2}$. Thus, for the given image resolution, the number of pixels per side is approximately $\sqrt{1.46}\big{/}0.044$. This suggests to define the levels such that $n_{\text{pix}}<13$, and we picked $n_{\text{pix}}=10$.

We empirically evaluated DOW variants of both our U-Net and DINOv2 based systems,
see Figure 2.
We describe the U-Net extension in more detail in Appendix A.2, the DINOv2 based systems were modified analogously. The DOW U-Net network architecture U-NetDOW used in our study is illustrated in Figure A.6 in the appendix. For a comparison with a DOW U-Net with $n_{\text{lev}}=6$ we refer to Appendix A.2 and Appendix B.

Although the approaches are related, we would like to stress the DOW method is conceptually different from *deep level sets*, where deep neural networks learn a (continuous) level set function, the zero-set of which defines object boundaries
(hu2017deep; Hatamizadeh20), as well as from predicting interior and border of an object as, for instance, done by girard2021polygonal.

### 3.3 Two-stage vs. End-to-end

All the neural network architectures described above can directly classify the roof types of detected buildings by predicting multi-class segmentation masks.
However, encouraged by good classification results using DINOv2 features, we also studied an alternative two-stage approach: First we segmented and separated the buildings using the algorithms described above ignoring the roof material information.
That is, we reduced the multi-class problem to a binary task.
After that, we predicted the roof material of each detected building.
We used DINOv2 to processes a $448\times 448$ patch centered around each building,
see Figure 3.
The output of DINOv2, a patch embedding with the shape of $\mathbb{R}^{1024\times 768}$ was reshaped into feature maps of $\mathbb{R}^{32\times 32\times 768}$.
These feature maps were then upsampled to the input patch size, masked with a target binary building mask, and average pooling was applied to obtain the final feature vector for the building.
Standard machine learning classifiers were applied to this embedding to predict the roof material, where linear probing gave the best results (see Appendix B.2 for a comparison of different classifiers).

The two-stage methods are referred to as U-Net, DINOv2, U-NetDOW, and DINOv2DOW, and the corresponding end-to-end methods are denoted by U-NetMulti, DINOv2DOW-Multi, U-NetDOW-Multi, and DINOv2DOW-Multi,
see Figure 2 for an overview.

Figure: Figure 3: The architecture of the DINOv2 based roof material classifier used in the two-stage setting. A classifier (e.g., logistic regression) is applied to the resulting feature vector.
Refer to caption: extracted/5651730/figures/dinovc.png

## 4 Experiments and Results

### 4.1 Experimental Setup

All models, except for YOLOv8 where we followed its original training protocol, were trained using cross-entropy loss with pixel-wise weighting.
We employed the AdamW optimizer (adamw) with an initial learning rate of $0.0003$.
All models were trained for 300 epochs, utilizing a learning rate scheduler that decreased the learning rate by a factor of $10$ every $50$ epochs.
The final weight configuration and hyperparameters for each model were selected based on the highest IoU score achieved on the validation dataset.
The hyperparameters of the U-Net were chosen by observing results on the validation data set in an iterative process.
The high training speed of YOLOv8 allowed for more systematic model selection: We applied the genetic algorithm that comes as part of the YOLOv8 framework for hyperparameter optimization (yolov8).
The input patch sizes for the U-Net variants, YOLOv8, and DINOv2 models were 512, 640, and 448, respectively.

### 4.2 Evaluation Metrics

The semantic segmentation performance was evaluated by the IoU.
We considered both the IoU of the binary building segmentation and the mean IoU for class-specific roof segmentation.
The roof materials concrete and asbestos are very rare. While $\mathcal{D}_{\text{train}}$, $\mathcal{D}_{\text{val}}$, and $\mathcal{D}_{\text{test}}$ are stratified samples containing all classes, the spatially distinct data $\mathcal{D}_{\text{ext}}$ does not contain any example of the two roof types, see Figure 1.
To allow for a better comparison between the two test sets and to see the effect of the rare classes on the macro-averaged mean IoU, we provide the mean IoU of the three main classes (mIoU^3) alongside with the mean IoU of all five classes (mIoU^5).

Instance segmentation was assessed using the AP50 score, that is, the
average precision evaluated at an IoU threshold of $0.5$ (pascal-voc-2007; lin2014microsoft).
We evaluated the AP for both the predictions of building instances and the predictions of multi-class roof type instances (i.e., in the latter case an object is only detected if the roof material is correctly identified).
Similar to IoU, mAP${}_{\text{50}}^{\text{3}}$ and mAP${}_{\text{50}}^{\text{5}}$ denote the mean AP50 over three and five classes.
To estimate the average precision, a confidence score is required for each building segment.
The confidence score of binary and multi-class segmentation models was obtained by
interpreting the neural networks’ outputs as probability distributions over classes and calculating the mean probability of belonging to the predicted class over all pixel within a predicted segment.
The exception was YOLOv8, which provides its own confidence score.
When a classifier using DINOv2 features was used on top of binary segmentation models, the confidence score was derived from the canonical probability score of the classifier.
Additional metrics, AP50-95 and TPs, are shown in Appendix B.
Information on the computer resources is provided in Appendix A.4.

### 4.3 Results and Discussion

**Table 1: Benchmarking results on the Nacala-Roof-Material dataset. The table reports averages over five trials $\pm$ standard deviations. The upper five models were trained in the two-stage setting. The lower half of the models was trained in the end-to-end setting, where multi-class classification is performed together with the segmentation as indicated by the subscript *Multi*. Models that used the DOW extension are indicated by the subscript *DOW*. IoU and AP50 were computed on the binary output, where the predictions of multi-class models were binarized. mIoU and mAP50 are macro averages, the superscipts indicate whether the averaging was done over all five classes or over the three frequent roof types. Results for individual roof types can be found in Appendix B.**
|  | $\mathcal{D}_{\text{test}}$ | $\mathcal{D}_{\text{ext}}$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | pixel level | object level | pixel level | object level |  |  |  |  |  |  |
| Model Name | IoU | mIoU^3 | mIoU^5 | AP50 | mAP${}_{\text{50}}^{\text{3}}$ | mAP${}_{\text{50}}^{\text{5}}$ | IoU | mIoU^3 | AP50 | mAP${}_{\text{50}}^{\text{3}}$ |
| YOLOv8 | 0.866<br>$\pm$ 0.012 | 0.713<br>$\pm$ 0.019 | 0.568<br>$\pm$ 0.015 | 0.941<br>$\pm$ 0.003 | 0.815<br>$\pm$ 0.011 | 0.698<br>$\pm$ 0.018 | 0.896<br>$\pm$ 0.002 | 0.761<br>$\pm$ 0.006 | 0.963<br>$\pm$ 0.005 | 0.846<br>$\pm$ 0.008 |
| DINOv2 | 0.833<br>$\pm$ 0.002 | 0.755<br>$\pm$ 0.004 | 0.562<br>$\pm$ 0.003 | 0.882<br>$\pm$ 0.004 | 0.789<br>$\pm$ 0.006 | 0.683<br>$\pm$ 0.008 | 0.905<br>$\pm$ 0.000 | 0.747<br>$\pm$ 0.011 | 0.919<br>$\pm$ 0.005 | 0.806<br>$\pm$ 0.008 |
| DINOv2DOW | 0.884<br>$\pm$ 0.001 | 0.763<br>$\pm$ 0.002 | 0.565<br>$\pm$ 0.004 | 0.930<br>$\pm$ 0.005 | 0.836<br>$\pm$ 0.002 | 0.725<br>$\pm$ 0.004 | 0.905<br>$\pm$ 0.001 | 0.852<br>$\pm$ 0.007 | 0.956<br>$\pm$ 0.001 | 0.852<br>$\pm$ 0.007 |
| U-Net | 0.895<br>$\pm$ 0.003 | 0.757<br>$\pm$ 0.024 | 0.570<br>$\pm$ 0.016 | 0.910<br>$\pm$ 0.005 | 0.810<br>$\pm$ 0.008 | 0.688<br>$\pm$ 0.014 | 0.909<br>$\pm$ 0.001 | 0.748<br>$\pm$ 0.007 | 0.929<br>$\pm$ 0.000 | 0.787<br>$\pm$ 0.011 |
| U-NetDOW | 0.895<br>$\pm$ 0.002 | 0.775<br>$\pm$ 0.013 | 0.577<br>$\pm$ 0.009 | 0.935<br>$\pm$ 0.001 | 0.836<br>$\pm$ 0.005 | 0.730<br>$\pm$ 0.011 | 0.911<br>$\pm$ 0.002 | 0.764<br>$\pm$ 0.006 | 0.947<br>$\pm$ 0.004 | 0.812<br>$\pm$ 0.008 |
| YOLOv8Multi | 0.824<br>$\pm$ 0.023 | 0.708<br>$\pm$ 0.010 | 0.550<br>$\pm$ 0.017 | 0.910<br>$\pm$ 0.005 | 0.816<br>$\pm$ 0.009 | 0.597<br>$\pm$ 0.007 | 0.885<br>$\pm$ 0.002 | 0.785<br>$\pm$ 0.006 | 0.948<br>$\pm$ 0.003 | 0.849<br>$\pm$ 0.015 |
| DINOv2Multi | 0.880<br>$\pm$ 0.002 | 0.774<br>$\pm$ 0.004 | 0.699<br>$\pm$ 0.012 | 0.899<br>$\pm$ 0.003 | 0.820<br>$\pm$ 0.010 | 0.689<br>$\pm$ 0.025 | 0.899<br>$\pm$ 0.002 | 0.818<br>$\pm$ 0.005 | 0.946<br>$\pm$ 0.001 | 0.880<br>$\pm$ 0.011 |
| DINOv2DOW-Multi | 0.885<br>$\pm$ 0.001 | 0.786<br>$\pm$ 0.006 | 0.734<br>$\pm$ 0.006 | 0.918<br>$\pm$ 0.003 | 0.824<br>$\pm$ 0.011 | 0.702<br>$\pm$ 0.013 | 0.902<br>$\pm$ 0.001 | 0.819<br>$\pm$ 0.006 | 0.950<br>$\pm$ 0.005 | 0.875<br>$\pm$ 0.010 |
| U-NetMulti | 0.879<br>$\pm$ 0.012 | 0.783<br>$\pm$ 0.010 | 0.634<br>$\pm$ 0.024 | 0.924<br>$\pm$ 0.004 | 0.850<br>$\pm$ 0.011 | 0.716<br>$\pm$ 0.018 | 0.903<br>$\pm$ 0.002 | 0.805<br>$\pm$ 0.020 | 0.943<br>$\pm$ 0.010 | 0.844<br>$\pm$ 0.039 |
| U-NetDOW-Multi | 0.892<br>$\pm$ 0.001 | 0.777<br>$\pm$ 0.012 | 0.672<br>$\pm$ 0.042 | 0.928<br>$\pm$ 0.002 | 0.829<br>$\pm$ 0.011 | 0.671<br>$\pm$ 0.022 | 0.904<br>$\pm$ 0.002 | 0.794<br>$\pm$ 0.014 | 0.942<br>$\pm$ 0.005 | 0.812<br>$\pm$ 0.021 |

Our experimental results on $\mathcal{D}_{\text{test}}$ and $\mathcal{D}_{\text{ext}}$ are presented in Table 1, additional details can be found in Appendix B. All metrics on the test sets were computed on raw images instead of patches to avoid artifacts when splitting images.
We report averages over five trials on the corresponding standard deviations.
The methods reached AP50 and IoU values on the spatially separated test set of up to $0.963$ and $0.880$, respectively. Thus
the tasks can be solved with an accuracies high enough for subsequent analysis while still leaving room for improvement. Detecting thatch roofs is particularly relevant, as they are associated with an increased malaria risk (Tusting2019MappingCI), and these roofs can be identified particularly well, see Table B.3 in the Appendix.

When comparing the different approaches, we find that there is no method that was better than the others across all metrics.
The U-Nets and YOLOv8 did well on their home grounds: YOLOv8 gave good object detection results (e.g., the best AP50 scores), while the U-Nets performed well for semantic segmentation as measured by IoU. DINOv2 combined with a simple decoder was also competitive.
Exemplary results are shown in Figure 4.
As could be expected, classifying the minority roof types asbestos and especially concrete (which resembles concreted background areas)
was most difficult, in particular for end-to-end YOLOv8, see Table B.3.
YOLOv8 had the tendency to produce
artefacts when applied to the larger images.
This is one of the reasons for its lower IoU score.

In general, the DOW extension improved
both U-Nets and DINOv2 based architectures.
Comparing DINOv2 with DINOv2DOW and U-Net with U-NetDOW, the DOW variants were better in all ten performance indices (except for IoU on $\mathcal{D}_{\text{test}}$ where U-Net and U-NetDOW gave the same result).
Comparing DINOv2Multi with DINOv2DOW-Multi, the latter was better in all indicators except mAP${}_{\text{50}}^{\text{3}}$ on $\mathcal{D}_{\text{ext}}$. Only for U-NetDOW-Multi the results were mixed, using DOW gave lower values for five indices and higher values for the other half.
Overall, the DOW extension had a statistically significant positive effect on the object separation as intended. If we pool all 20 DOW trials and compare with the corresponding trials predicting a single mask, then the AP50 improved significantly (two-sided Wilcoxon rank sum test, $p<0.001$) while the difference in IoU was not significant ($p>0.05$).

Figure: Figure 4: Exemplary predictions on $\mathcal{D}_{\text{test}}$ by different models. The predictions are polygonized and colored by class. The roof types with few training examples, asbestos and concrete, are particularly difficult, see bottom row.
Refer to caption: extracted/5651730/figures/results4.png

#### Limitations.

The Nacala-Roof-Material dataset is not a large-scale data set by current standards and it is restricted to a single region. However, considering the proliferation of low-cost drone technologies, high-resolution geospatial surveying is becoming increasingly affordable and common in sub-Saharan Africa. Accordingly, similar but unlabelled data will likely become available in the coming years at large scale, which makes it important to develop methods to make good use of these data now.
The Nacala-Roof-Material dataset covering informal settlements is a good example for the target areas of our risk disease monitoring and prevention research. In this context, Mozambique is particularly relevant because the country suffers from a high malaria incidence rate (VENKATESAN2024e214).
The second test set allows for testing generalization in an area geographically separated from the main training/test/validation data.
In general, we would argue that there is a need for medium size
benchmark data sets such as the Nacala-Roof-Material data to support equity in machine learning research, as we need benchmarks that can be utilized by researchers with limited compute resources.

## 5 Conclusions

The Nacala-Roof-Material dataset contains high-resolution drone imagery from informal settlements in Mozambique, where buildings and their roof material were carefully annotated.
We curated the dataset as part of an intercontinental and interdisciplinary research project on risk assessment of mosquito-borne diseases, especially malaria, with the goal to predict risk maps and to develop and support measures for risk reduction.
From a methodological perspective, the dataset defines a multi-task problem.
We are interested in accurate semantic segmentation to determine the roof areas and also in identifying the individual buildings and classifying their roof types. Thus, the dataset adds to the landscape of computer vision benchmarks by providing a relevant resource for the development and evaluation of frameworks that strive at solving semantic segmentation as well as object detection and classification simultaneously with a high accuracy.
For example, working on the Nacala-Roof-Material data has led us to the proposed deep ordinal watershed (DOW) approach, a reduced variant of the method described by cheng2024scattered. This variant method first segments objects along with their interiors into two elevation levels and then performs a watershed segmentation to separate objects. The DOW idea is applicable beyond the Nacala-Roof-Material data, on which it improved both the standard U-Net architectures as well as a system based on DINOv2 features for segmentation.
Implementations of all algorithms are made publicly available together with the data ([https://mosquito-risk.github.io/Nacala](https://mosquito-risk.github.io/Nacala)).
With the Nacala-Roof-Material dataset, we invite the machine learning community to develop new approaches for interpreting high-resolution drone images that can ultimately support risk assessments of vector-borne diseases.

## Acknowledgments and Disclosure of Funding

We acknowledge support through the project
*Risk-assessment of Vector-borne Diseases Based on Deep Learning and Remote Sensing* (grant number NNF21OC0069116) by the Novo Nordisk Foundation.

## Appendix A Details on Models and Training Procedure

### A.1 U-Net

Figure: Figure A.5: Basic U-Net architecture
Refer to caption: extracted/5651730/figures/unet.png

The basic U-Net architecture we used is shown in Figure A.5.

During training, the loss of each background pixel $x$ is multiplicatively weighted
by $w(x)$ defined as

$$ $w(x)=w_{0}\cdot\exp\left(-\frac{(d_{1}(x)+d_{2}(x))^{2}}{2\sigma^{2}}\right)$ (A.1) $$

following ronneberger2015unet.
Here, $d_{1}(x)$ denotes the distance to the border of the nearest segment, and $d_{2}(x)$ is the distance to the border of the second nearest segment. We set $w_{0}=10$ and $\sigma=5$ according to ronneberger2015unet.

During training, we modified the target masks to ensure that $d_{1}(x)+d_{2}(x)\geq n_{\text{gap}}=7$ for each background pixel $x$ before we computed the weights $w(x)$.

### A.2 Deep Ordinal Watershed U-Nets

We considered a stripped down version of the DOW U-Net proposed by cheng2024scattered and set the number of elevation levels to $n_{\text{lev}}=2$.
The architecture of the resulting DOW network is depicted in Figure A.6, which extends the basic U-Net architecture shown in Figure A.5.
In contrast to the original U-Net, the DOW model has two heads. One is predicting an object’s area, while the other predicts its interior. The interior is defined by removing pixels within a 10-pixel distance from the border of the building segment.
Each head comprises a convolutional layer, batch normalization, ReLU activation, and finally a pointwise convolutional layer with outputs equal to the number of classes.
While the first head had filters of size $3\times 3$ in its first convolutional layer, the second head for the interior used 64 filters. The class label of an object was derived from the second head. If no interior was predicted, which can happen in the case of small objects, the output from the first head defined the class.

Figure: Figure A.6: U-NetDOW architecture producing two output maps, segmenting objects and their interiors, respectively. The architecture differs from the baseline U-Net only in the output heads.
Refer to caption: extracted/5651730/figures/heads2.png

We compared this DOW variant, referred to as U-NetDOW,
to the original DOW with several elevation levels, in which the levels are added to the standard U-Net architecture (Figure A.5) simply by increasing the number of output masks.
We considered $n_{\text{lev}}=6$ discrete height levels and accordingly refer to the model as U-NetDOW-6.
The pixel margin $n_{\text{pix}}$ for each height level was determined experimentally by testing $n_{\text{pix}}\in\{1,3,5,7,9,11,13,15\}$ on validation data, leading to $n_{\text{pix}}=5$ for U-NetDOW-6.
An experimental comparison of U-NetDOW and U-NetDOW-6 can be found in the extended results in Section B in the appendix.

### A.3 Segmentation and Classification Using DINOv2

The segmentation architecture based on DINOv2 is illustrated in Figure A.7.
We refer to it simply as DINOv2. From this architecture, we derived DINOv2DOW in the same way as we extended U-Net to U-NetDOW .

Figure: Figure A.7: DINOv2 architecture
Refer to caption: extracted/5651730/figures/dinoseg.png

### A.4 Compute resources

All experiments were conducted on AMD MI250X GPUs
with 64 GB VRAM provided by LUMI(^2^22[https://lumi-supercomputer.eu](https://lumi-supercomputer.eu)). A total of 8550 GPU hours were used for the project, including preliminary experiments not included in the paper. The computation time for training semantic segmentation model was approximately 20 hours for 300 epochs when the entire data were loaded to GPU memory.

## Appendix B Additional Results

### B.1 Detailed Results for Different Roof Materials

Additional results on $\mathcal{D}_{\text{test}}$ are presented in Table B.2 and Table B.3. The tables report the IoU scores for the individual roof material classes. They also show the true positive rates TPs in addition to the AP50 the AP50-95.
The AP50-95 is defined as the mean AP over IoU thresholds from $50\,\%$ to $95\,\%$ with an interval of $5\,\%$.
The mean of AP50-95 over all classes is mAP50-95.
TPs are the number of segments that overlap with ground truth segments with a minimum IoU of $0.5$, we used this metric to assess the counting of buildings.

Beyond the performance metrics already discussed, we have included the results for U-NetDOW-6 as described in Section A.2 in the appendix, showing that the two DOW architectures perform on par.

The corresponding results on $\mathcal{D}_{\text{ext}}$ are given in Table B.4 and Table B.5
The mean IoU in Table B.4, and mAP50 and mAP50-95 in Table B.5 estimated on only four classes as there are no asbestos roofs in $\mathcal{D}_{\text{ext}}$. Also, there are only two buildings of concrete found in $\mathcal{D}_{\text{ext}}$ and these two buildings were not identified from any of the experimental models, so results for the concrete class were not added to both tables.

**Table B.2: Pixel-level accuracies on $\mathcal{D}_{\text{test}}$. IoU refers to the IoU computed on the binary outputs, where the predictions of multi-class models were binarized. mIoU^5 refers to the macro average of the IoUs for the individual classes. The subscript *Multi* indicates the end-to-end setting.**
|  | IoU-Score of each class |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model Name | Metal<br>Sheet | Thatch | Asbestos | Concrete | No<br>Roof | mIoU^5 | IoU |
| YOLOv8 | 0.807<br>$\pm$ 0.003 | 0.852<br>$\pm$ 0.038 | 0.450<br>$\pm$ 0.023 | 0.250<br>$\pm$ 0.027 | 0.480<br>$\pm$ 0.034 | 0.568<br>$\pm$ 0.015 | 0.866<br>$\pm$ 0.012 |
| DINOv2 | 0.804<br>$\pm$ 0.003 | 0.854<br>$\pm$ 0.003 | 0.349<br>$\pm$ 0.010 | 0.196<br>$\pm$ 0.004 | 0.608<br>$\pm$ 0.013 | 0.562<br>$\pm$ 0.003 | 0.883<br>$\pm$ 0.002 |
| DINOv2DOW | 0.810<br>$\pm$ 0.003 | 0.867<br>$\pm$ 0.001 | 0.348<br>$\pm$ 0.009 | 0.188<br>$\pm$ 0.015 | 0.613<br>$\pm$ 0.005 | 0.565<br>$\pm$ 0.004 | 0.884<br>$\pm$ 0.001 |
| U-Net | 0.813<br>$\pm$ 0.009 | 0.881<br>$\pm$ 0.002 | 0.408<br>$\pm$ 0.012 | 0.171<br>$\pm$ 0.021 | 0.577<br>$\pm$ 0.073 | 0.570<br>$\pm$ 0.016 | 0.895<br>$\pm$ 0.003 |
| U-NetDOW | 0.824<br>$\pm$ 0.005 | 0.879<br>$\pm$ 0.010 | 0.384<br>$\pm$ 0.042 | 0.174<br>$\pm$ 0.010 | 0.623<br>$\pm$ 0.028 | 0.577<br>$\pm$ 0.009 | 0.895<br>$\pm$ 0.002 |
| U-NetDOW-6 | 0.824<br>$\pm$ 0.006 | 0.887<br>$\pm$ 0.002 | 0.424<br>$\pm$ 0.055 | 0.160<br>$\pm$ 0.026 | 0.591<br>$\pm$ 0.057 | 0.577<br>$\pm$ 0.011 | 0.888<br>$\pm$ 0.009 |
| YOLOv8Multi | 0.750<br>$\pm$ 0.030 | 0.824<br>$\pm$ 0.004 | 0.405<br>$\pm$ 0.021 | 0.223<br>$\pm$ 0.059 | 0.549<br>$\pm$ 0.026 | 0.550<br>$\pm$ 0.017 | 0.824<br>$\pm$ 0.023 |
| DINOv2Multi | 0.821<br>$\pm$ 0.003 | 0.862<br>$\pm$ 0.002 | 0.490<br>$\pm$ 0.026 | 0.682<br>$\pm$ 0.031 | 0.640<br>$\pm$ 0.014 | 0.699<br>$\pm$ 0.012 | 0.880<br>$\pm$ 0.000 |
| DINOv2DOW-Multi | 0.839<br>$\pm$ 0.002 | 0.870<br>$\pm$ 0.002 | 0.542<br>$\pm$ 0.009 | 0.773<br>$\pm$ 0.009 | 0.649<br>$\pm$ 0.015 | 0.734<br>$\pm$ 0.006 | 0.885<br>$\pm$ 0.001 |
| U-NetMulti | 0.819<br>$\pm$ 0.012 | 0.880<br>$\pm$ 0.004 | 0.514<br>$\pm$ 0.025 | 0.306<br>$\pm$ 0.091 | 0.650<br>$\pm$ 0.029 | 0.634<br>$\pm$ 0.024 | 0.879<br>$\pm$ 0.012 |
| U-NetDOW-Multi | 0.827<br>$\pm$ 0.011 | 0.887<br>$\pm$ 0.002 | 0.511<br>$\pm$ 0.044 | 0.290<br>$\pm$ 0.105 | 0.636<br>$\pm$ 0.013 | 0.630<br>$\pm$ 0.026 | 0.889<br>$\pm$ 0.009 |

**Table B.3: Object-level accuracy on $\mathcal{D}_{\text{test}}$. We report the AP for each roof type, and $mAP\textsubscript{50}$ and $mAP\textsubscript{50-95}$ are macro averages over the roof types. The rightmost three columns give the results when we discard the roof type information and just consider building detection. The TPs columns count true positives, where TPs are the number of objects that overlap with ground truth objects with a minimum IoU of $0.5$. The total number of ground truth objects in the $\mathcal{D}_{\text{test}}$ is $2527$.**
|  | AP50 of each class | average over classes | ignoring roof type |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model Name | Metal<br>Sheet | Thatch | Asbestos | Concrete | No<br>Roof | mAP50 | mAP50-95 | TPs | AP50 | AP50-95 | TPs |
| YOLOv8 | 0.841<br>$\pm$ 0.003 | 0.945<br>$\pm$ 0.008 | 0.505<br>$\pm$ 0.032 | 0.542<br>$\pm$ 0.055 | 0.661<br>$\pm$ 0.026 | 0.698<br>$\pm$ 0.018 | 0.548<br>$\pm$ 0.010 | 2262.2<br>$\pm$ 7.386 | 0.941<br>$\pm$ 0.003 | 0.798<br>$\pm$ 0.002 | 2405.0<br>$\pm$ 5.514 |
| DINOv2 | 0.807<br>$\pm$ 0.005 | 0.885<br>$\pm$ 0.006 | 0.470<br>$\pm$ 0.011 | 0.579<br>$\pm$ 0.025 | 0.673<br>$\pm$ 0.015 | 0.683<br>$\pm$ 0.008 | 0.531<br>$\pm$ 0.005 | 2135.6<br>$\pm$ 7.761 | 0.882<br>$\pm$ 0.004 | 0.733<br>$\pm$ 0.003 | 2261.6<br>$\pm$ 9.351 |
| DINOv2DOW | 0.852<br>$\pm$ 0.002 | 0.940<br>$\pm$ 0.001 | 0.517<br>$\pm$ 0.016 | 0.600<br>$\pm$ 0.028 | 0.715<br>$\pm$ 0.006 | 0.725<br>$\pm$ 0.004 | 0.573<br>$\pm$ 0.008 | 2238.4<br>$\pm$ 5.238 | 0.930<br>$\pm$ 0.005 | 0.781<br>$\pm$ 0.005 | 2376.2<br>$\pm$ 7.194 |
| U-Net | 0.826<br>$\pm$ 0.005 | 0.924<br>$\pm$ 0.006 | 0.499<br>$\pm$ 0.016 | 0.511<br>$\pm$ 0.042 | 0.679<br>$\pm$ 0.015 | 0.688<br>$\pm$ 0.014 | 0.578<br>$\pm$ 0.014 | 2191.2<br>$\pm$ 11.25 | 0.910<br>$\pm$ 0.005 | 0.797<br>$\pm$ 0.003 | 2323.0<br>$\pm$ 6.033 |
| U-NetDOW | 0.855<br>$\pm$ 0.005 | 0.946<br>$\pm$ 0.005 | 0.545<br>$\pm$ 0.019 | 0.596<br>$\pm$ 0.049 | 0.707<br>$\pm$ 0.012 | 0.730<br>$\pm$ 0.011 | 0.614<br>$\pm$ 0.007 | 2249.4<br>$\pm$ 4.128 | 0.935<br>$\pm$ 0.001 | 0.819<br>$\pm$ 0.003 | 2383.6<br>$\pm$ 5.314 |
| U-NetDOW-6 | 0.851<br>$\pm$ 0.006 | 0.943<br>$\pm$ 0.004 | 0.551<br>$\pm$ 0.011 | 0.587<br>$\pm$ 0.049 | 0.687<br>$\pm$ 0.022 | 0.724<br>$\pm$ 0.007 | 0.606<br>$\pm$ 0.005 | 2243.2<br>$\pm$ 3.487 | 0.929<br>$\pm$ 0.004 | 0.818<br>$\pm$ 0.002 | 2374.4<br>$\pm$ 5.783 |
| YOLOv8Multi | 0.849<br>$\pm$ 0.006 | 0.923<br>$\pm$ 0.007 | 0.467<br>$\pm$ 0.035 | 0.070<br>$\pm$ 0.027 | 0.676<br>$\pm$ 0.020 | 0.597<br>$\pm$ 0.007 | 0.481<br>$\pm$ 0.003 | 2195.6<br>$\pm$ 15.383 | 0.910<br>$\pm$ 0.005 | 0.751<br>$\pm$ 0.007 | 2328.2<br>$\pm$ 9.988 |
| DINOv2Multi | 0.869<br>$\pm$ 0.003 | 0.923<br>$\pm$ 0.005 | 0.474<br>$\pm$ 0.043 | 0.508<br>$\pm$ 0.100 | 0.669<br>$\pm$ 0.033 | 0.689<br>$\pm$ 0.025 | 0.512<br>$\pm$ 0.021 | 2231.6<br>$\pm$ 2.332 | 0.899<br>$\pm$ 0.003 | 0.733<br>$\pm$ 0.004 | 2311.4<br>$\pm$ 2.728 |
| DINOv2DOW-Multi | 0.888<br>$\pm$ 0.004 | 0.937<br>$\pm$ 0.004 | 0.536<br>$\pm$ 0.018 | 0.504<br>$\pm$ 0.056 | 0.647<br>$\pm$ 0.028 | 0.702<br>$\pm$ 0.013 | 0.558<br>$\pm$ 0.011 | 2270.4<br>$\pm$ 9.308 | 0.918<br>$\pm$ 0.003 | 0.766<br>$\pm$ 0.002 | 2366.0<br>$\pm$ 7.266 |
| U-NetMulti | 0.883<br>$\pm$ 0.009 | 0.940<br>$\pm$ 0.010 | 0.531<br>$\pm$ 0.022 | 0.498<br>$\pm$ 0.051 | 0.728<br>$\pm$ 0.040 | 0.716<br>$\pm$ 0.018 | 0.603<br>$\pm$ 0.011 | 2262.4<br>$\pm$ 6.119 | 0.924<br>$\pm$ 0.004 | 0.797<br>$\pm$ 0.007 | 2358.4<br>$\pm$ 9.091 |
| U-NetDOW-Multi | 0.864<br>$\pm$ 0.001 | 0.918<br>$\pm$ 0.006 | 0.533<br>$\pm$ 0.024 | 0.537<br>$\pm$ 0.048 | 0.702<br>$\pm$ 0.018 | 0.711<br>$\pm$ 0.010 | 0.609<br>$\pm$ 0.012 | 2216.2<br>$\pm$ 7.305 | 0.903<br>$\pm$ 0.004 | 0.786<br>$\pm$ 0.003 | 2308.0<br>$\pm$ 5.044 |

**Table B.4: Pixel-level accuracies on $\mathcal{D}_{\text{ext}}$. IoU refers to the IoU computed on the binary outputs, where the predictions of multi-class models were binarized. mIoU^5 refers to the macro average of the IoUs for the individual classes. The subscript *Multi* indicates the end-to-end setting.**
|  | IoU-Score of each class |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model Name | Metal<br>Sheet | Thatch | No<br>Roof | IoU<br>(Mean) | IoU<br>(Binary) |
| YOLOv8 | 0.888<br>$\pm$ 0.003 | 0.879<br>$\pm$ 0.003 | 0.516<br>$\pm$ 0.011 | 0.761<br>$\pm$ 0.006 | 0.896<br>$\pm$ 0.002 |
| DINOv2 | 0.867<br>$\pm$ 0.014 | 0.853<br>$\pm$ 0.004 | 0.523<br>$\pm$ 0.022 | 0.747<br>$\pm$ 0.011 | 0.905<br>$\pm$ 0.000 |
| DINOv2DOW | 0.891<br>$\pm$ 0.003 | 0.880<br>$\pm$ 0.002 | 0.560<br>$\pm$ 0.018 | 0.777<br>$\pm$ 0.007 | 0.905<br>$\pm$ 0.001 |
| U-Net | 0.896<br>$\pm$ 0.005 | 0.883<br>$\pm$ 0.005 | 0.463<br>$\pm$ 0.017 | 0.748<br>$\pm$ 0.007 | 0.909<br>$\pm$ 0.001 |
| U-NetDOW | 0.905<br>$\pm$ 0.002 | 0.895<br>$\pm$ 0.003 | 0.493<br>$\pm$ 0.018 | 0.764<br>$\pm$ 0.006 | 0.911<br>$\pm$ 0.002 |
| U-NetDOW-6 | 0.900<br>$\pm$ 0.008 | 0.889<br>$\pm$ 0.002 | 0.452<br>$\pm$ 0.031 | 0.747<br>$\pm$ 0.009 | 0.902<br>$\pm$ 0.003 |
| YOLOv8Multi | 0.890<br>$\pm$ 0.006 | 0.860<br>$\pm$ 0.006 | 0.606<br>$\pm$ 0.019 | 0.785<br>$\pm$ 0.006 | 0.885<br>$\pm$ 0.002 |
| DINOv2Multi | 0.905<br>$\pm$ 0.002 | 0.875<br>$\pm$ 0.004 | 0.674<br>$\pm$ 0.018 | 0.818<br>$\pm$ 0.005 | 0.899<br>$\pm$ 0.002 |
| DINOv2DOW-Multi | 0.912<br>$\pm$ 0.001 | 0.881<br>$\pm$ 0.002 | 0.663<br>$\pm$ 0.017 | 0.875<br>$\pm$ 0.010 | 0.902<br>$\pm$ 0.001 |
| U-NetMulti | 0.913<br>$\pm$ 0.005 | 0.884<br>$\pm$ 0.003 | 0.617<br>$\pm$ 0.061 | 0.805<br>$\pm$ 0.020 | 0.903<br>$\pm$ 0.002 |
| U-NetDOW-Multi | 0.921<br>$\pm$ 0.001 | 0.888<br>$\pm$ 0.002 | 0.613<br>$\pm$ 0.033 | 0.807<br>$\pm$ 0.011 | 0.909<br>$\pm$ 0.002 |

**Table B.5: Object-level accuracies on $\mathcal{D}_{\text{ext}}$. We report the AP for each roof type, and $mAP\textsubscript{50}$ and $mAP\textsubscript{50-95}$ are macro averages over the classes. The rightmost three columns give the results when we discard the roof type information and just consider building detection. TPs are the number of objects that overlap with ground truth objects with a minimum IoU of $0.5$. The total number of ground truth objects in the $\mathcal{D}_{\text{ext}}$ is $1541$.**
|  | AP50of each class | Objects with Classes | Only Building Objects |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model Name | Metal<br>Sheet | Thatch | No<br>Roof | mAP50 | mAP50-95 | TPs | AP50 | AP50-95 | TPs |
| YOLOv8 | 0.928<br>$\pm$ 0.001 | 0.947<br>$\pm$ 0.000 | 0.661<br>$\pm$ 0.023 | 0.846<br>$\pm$ 0.008 | 0.428<br>$\pm$ 0.002 | 1447.2<br>$\pm$ 4.534 | 0.963<br>$\pm$ 0.005 | 0.838<br>$\pm$ 0.002 | 1493.8<br>$\pm$ 3.826 |
| DINOv2Multi | 0.898<br>$\pm$ 0.004 | 0.885<br>$\pm$ 0.009 | 0.635<br>$\pm$ 0.024 | 0.484<br>$\pm$ 0.005 | 0.393<br>$\pm$ 0.005 | 1381.6<br>$\pm$ 7.172 | 0.919<br>$\pm$ 0.005 | 0.786<br>$\pm$ 0.006 | 1428.8<br>$\pm$ 7.305 |
| DINOv2DOW-Multi | 0.932<br>$\pm$ 0.002 | 0.942<br>$\pm$ 0.005 | 0.681<br>$\pm$ 0.02<br>0 | 0.852<br>$\pm$ 0.007 | 0.423<br>$\pm$ 0.003 | 1441.8<br>$\pm$ 4.400 | 0.956<br>$\pm$ 0.001 | 0.828<br>$\pm$ 0.003 | 1486.2<br>$\pm$ 1.939 |
| U-Net | 0.915<br>$\pm$ 0.006 | 0.921<br>$\pm$ 0.006 | 0.520<br>$\pm$ 0.027 | 0.590<br>$\pm$ 0.006 | 0.407<br>$\pm$ 0.004 | 1399.4<br>$\pm$ 4.758 | 0.929<br>$\pm$ 0.000 | 0.836<br>$\pm$ 0.002 | 1438.4<br>$\pm$ 4.499 |
| U-NetDOW | 0.932<br>$\pm$ 0.003 | 0.946<br>$\pm$ 0.004 | 0.559<br>$\pm$ 0.027 | 0.812<br>$\pm$ 0.008 | 0.528<br>$\pm$ 0.003 | 1429.0<br>$\pm$ 6.229 | 0.947<br>$\pm$ 0.004 | 0.858<br>$\pm$ 0.004 | 1468.6<br>$\pm$ 6.499 |
| U-NetDOW-6 | 0.935<br>$\pm$ 0.001 | 0.940<br>$\pm$ 0.004 | 0.509<br>$\pm$ 0.022 | 0.795<br>$\pm$ 0.008 | 0.518<br>$\pm$ 0.005 | 1421.0<br>$\pm$ 3.688 | 0.939<br>$\pm$ 0.000 | 0.851<br>$\pm$ 0.004 | 1458.8<br>$\pm$ 4.118 |
| YOLOv8Multi | 0.949<br>$\pm$ 0.004 | 0.934<br>$\pm$ 0.007 | 0.664<br>$\pm$ 0.044 | 0.849<br>$\pm$ 0.015 | 0.423<br>$\pm$ 0.008 | 1446.0<br>$\pm$ 4.899 | 0.948<br>$\pm$ 0.003 | 0.808<br>$\pm$ 0.005 | 1477.2<br>$\pm$ 3.655 |
| DINOv2Multi | 0.955<br>$\pm$ 0.004 | 0.935<br>$\pm$ 0.007 | 0.749<br>$\pm$ 0.030 | 0.880<br>$\pm$ 0.011 | 0.539<br>$\pm$ 0.005 | 1454.4<br>$\pm$ 3.878 | 0.946<br>$\pm$ 0.001 | 0.801<br>$\pm$ 0.003 | 1468.8<br>$\pm$ 2.926 |
| DINOv2DOW-Multi | 0.956<br>$\pm$ 0.001 | 0.943<br>$\pm$ 0.005 | 0.727<br>$\pm$ 0.026 | 0.875<br>$\pm$ 0.010 | 0.521<br>$\pm$ 0.047 | 1460.8<br>$\pm$ 3.868 | 0.950<br>$\pm$ 0.005 | 0.820<br>$\pm$ 0.002 | 1478.8<br>$\pm$ 3.311 |
| U-NetMulti | 0.956<br>$\pm$ 0.004 | 0.926<br>$\pm$ 0.008 | 0.651<br>$\pm$ 0.107 | 0.844<br>$\pm$ 0.039 | 0.548<br>$\pm$ 0.017 | 1439.0<br>$\pm$ 16.358 | 0.943<br>$\pm$ 0.010 | 0.838<br>$\pm$ 0.006 | 1463.6<br>$\pm$ 13.063 |
| U-NetDOW-Multi | 0.951<br>$\pm$ 0.004 | 0.920<br>$\pm$ 0.006 | 0.632<br>$\pm$ 0.029 | 0.834<br>$\pm$ 0.010 | 0.458<br>$\pm$ 0.005 | 1426.8<br>$\pm$ 5.418 | 0.947<br>$\pm$ 0.001 | 0.854<br>$\pm$ 0.004 | 1472.6<br>$\pm$ 2.417 |

### B.2 Performance of Different Classifiers

In the two-stage approach, we used a classifier based on DINOv2 features, as described in Section 3.3 and illustrated in Figure 3.
The input representation was fixed and was processed by standard classification algorithms. We compared linear probing based on logistic regression with $L_{2}$-regularization
and k-nearest neighbours (kNN) classification trained on our data.
For evaluating the classifiers and tuning their hyperparameters, we combined the training and validation data and performed 10-fold cross-validation (CV) with F1-score as performance metric.
The best CV results gave logistic regression with $L_{2}$-regularization, and this model was used for all subsequent two-stage experiments, see Table B.2.

**Table B.6: Cross-validation accuracies on combined training and validation data of k-nearest neighbour classification (kNN) and logistic regression applied to the DINOv2 features. The baseline is the architecture depicted in Figure 3, *w/o mask* refers to omitting the masking and averaging the DINOv2 features across the whole input patch, and *w/o upsampling* did not upsample the DINOv2 features but downsampled the building mask instead.**
|  | F1-Score |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | baseline | w/o mask | w/o upsampling |  |  |  |
| Classifier | Mean | Std | Mean | Std | Mean | Std |
| Logistic Regression | 0.770 | 0.063 | 0.573 | 0.077 | 0.768 | 0.067 |
| kNN | 0.734 | 0.045 | 0.389 | 0.029 | 0.733 | 0.051 |

We also performed an ablation study to show the importance of the masking and the upsampling in our architecture shown in Figure 3. The results are also depicted in Table B.2. When we omitted the masking and considered all features, the results got considerably worse.
If we omitted the upsampling of the DINOv2 output and downsampled the masks instead, the performance also slightly dropped.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How well would the models generalize to datasets from different geographical areas with varying building styles and materials?

2. Are there plans to extend the dataset to include more regions to enhance generalization and applicability across sub-Saharan Africa or similar environments?

3. What measures were taken to ensure that the manual annotations are free from bias, and how might annotation inconsistencies affect the benchmark results?

4. How does the DOW method compare against other state-of-the-art post-processing techniques for object delineation beyond watershed-based methods?

5. Are there specific public health interventions or organizations already interested in or collaborating on using these findings?
soundness: 4
strengths: __Significance and Novelty:__

The paper addresses a crucial public health challenge by linking housing conditions with mosquito-borne disease risk assessment. This intersection of computer vision and epidemiology is novel and holds significant societal impact, especially in resource-limited regions. By providing a high-resolution drone imagery dataset focused on roof material classification, the research fills a gap left by existing lower-resolution datasets that do not distinguish between specific roof types.

__Soundness of the Claims:__

Theoretical grounding is evident as the authors effectively combine concepts from computer vision with epidemiological insights. The benchmarked models are carefully selected, and the inclusion of the DOW method demonstrates a thoughtful approach to improving segmentation outcomes. The experimental setup is robust, with a detailed analysis of various metrics, including intersection over union (IoU) and AP scores, ensuring empirical validity.

__Empirical Evaluation:__

The extensive empirical evaluation demonstrates that no single model is superior across all tasks, providing valuable insights for future researchers. By comparing models on multiple segmentation and classification metrics, the study offers a nuanced understanding of model performance, emphasizing practical limitations and advantages of each approach. The statistical analysis adds credibility, and the publicly available dataset promotes reproducibility.
summary: This paper introduces the Nacala-Roof-Material dataset, a high-resolution drone imagery dataset collected from informal settlements in Nacala, Mozambique. The dataset is designed to assist in research on roof detection, classification, and segmentation, with a specific focus on mitigating mosquito-borne disease risks, such as malaria, which are influenced by housing quality and roof materials. The authors define three primary tasks using this dataset: building detection, multi-class roof type classification, and pixel-level building segmentation. Several state-of-the-art models, including U-Net, YOLOv8, and DINOv2, are benchmarked to evaluate performance on these tasks. Additionally, the paper proposes the Deep Ordinal Watershed (DOW) method, which improves segmentation accuracy by separately delineating building exteriors and interiors. By releasing this dataset and associated benchmarks, the authors aim to advance research on using remote sensing for public health applications.
weaknesses: __Generalization Concerns:__

Although the dataset provides a meaningful resource, it is limited to a single geographic region, which may affect the generalizability of the findings. The paper does not explore how well models trained on this dataset might perform in different regions with varying architectural styles or environmental conditions, which is a crucial aspect for real-world applications.

__Limited Baseline Diversity:__

While the models benchmarked are popular, the study could benefit from including more diverse architectures or emerging models from the latest research to better assess the dataset’s full potential. Additionally, the DOW method is compared mainly against traditional segmentation methods without an extensive exploration of other advanced post-processing techniques.

__Computational Resource Requirements:__

The paper mentions that training the models requires significant computational power, which could be a barrier for adoption, particularly in the regions where this research is most needed. This limitation reduces the accessibility of the proposed methods, especially for small organizations or researchers with limited resources.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: I appreciate the writing style. However, due to the lack of insights and innovation in the method, submission to a remote sensing journal is suggested.
soundness: 2
strengths: 1.This paper is written in a clear and concise way, making it enjoyable to read and easy to digest. 
2. The topic is interesting and valuable.
summary: This paper presents the Nacala-Roof-Material dataset, which includes drone imagery from a sub-Saharan African region, designed for building segmentation and roof-type classification. The authors apply a deep ordinal watershed model to accomplish this task, with potential applications in risk assessments for vector-borne diseases.
weaknesses: 1. Dataset. The design of the extend test dataset is inadequate, which does not contain any example of two roof types.
2. Related works. Missing discussion on relevant methods, such as multi-task/multi-resolution building extraction techniques. 
3. Technical novelty. The approach relies heavily on established methods.

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 3
questions: The concrete link between mosquitos and roof materials did not become clear to me: Are there some studies relating exactly these two data sources, or is roof material mainly a proxy for wealth and slum mapping? 
Concretely, is the connections with mosquitos merely motivational, or do you have data collected (e.g., through surveys or disease occurrence rates) that supports this connection?
soundness: 3
strengths: It is an interesting problem and a well-curated dataset with several existing baseline models
summary: The paper proposes a dataset to classify and segment roof materials motivated by a correlation of certain roof materials with mosquito risks. It is certainly an interesting dataset and problem. The dataset is certainly impactful, as it has been created with substantial manual effort by local university students and NGO members from Nacala. Also, the distinction between a spatial in-distribution D_text and spatially distinct D_ext, some classes missing in the training sets, is certainly a challenging setup.  
However, I evaluate the impact of this paper for a machine learning conference is limited. In terms of methodology, the paper uses existing (pre-trained) vision models and proposes a watershed postprocessing step, "DOW," inspired by  Cheng et al. (2024). In my opinion, it would fit better to an ICLR workshop (e.g., Machine Learning for Remote Sensing) or in a Remote Sensing Journal/Conference where the impact of the paper could be better evaluated and where it also would have more impact overall.
weaknesses: * My central concern is the limited relevance of this dataset to a machine learning audience, as it is mainly relevant to remote sensing researchers. I did not see any aspects of this dataset that would open a particular targeted challenge/problem to develop new machine learning models on. It's contribution is rather in the environmental/social impact than building the basis for new model developments. Hence, it would fit a dedicated workshop or a remote sensing journal better where it can be evaluated properly.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: see Weaknesses
soundness: 3
strengths: 1. The roof material segmentation and detection dataset is valuable for autormatic house quality estimation.
2. The paper is writtn well and easy to follow.
3. The visual foundation model, e.g., DINOv2, is considered into their study.
summary: This paper presented a root material segmentation and detection dataset based on drone images. Authors also benchmarked YOLOv8, Unet, DINOv2 models on their dataset at both pixel-level and object-level.
weaknesses: 1. The benchmark is not comprehensive. There are many good object detection, semantic segmentation, and instance segmentation models can be evaluated on this new dataset for more insights.
2. The analysis is insufficient, e.g., what are the main error sources in this task? how well does the larger vision foundation models (e.g., SAM) do? The authors should give more insights to guide future readers in exploring roof material classification problems.
3. A comparison between conventional datasets and your dataset should be included.