MESSI: A Multi-Elevation Semantic Segmentation Image Dataset of an Urban Environment
Authors: Barak Pinkovich
Boaz Matalon
Ehud Rivlin
Hector Rotstein
Venue: Accepted by TMLR ()

Abstract: This paper presents a Multi-Elevation Semantic Segmentation Image (MESSI) dataset. A reduced version of the dataset has been published at https://github.com/messi-dataset/ for reviewing purposes (due to the anonymity requirement). The full dataset will be made available at the time of the decision. MESSI comprises 2525 images taken by a drone flying over dense urban environments. MESSI is unique in two main features. First, it contains images from various altitudes (both with horizontal and vertical trajectories), allowing us to investigate the effect of depth on semantic segmentation. Second, it includes images taken from several different urban regions (at different altitudes). This is important since the variety covers the visual richness captured by a drone's 3D flight, performing horizontal and vertical maneuvers. MESSI contains images annotated with location, orientation, and the camera's intrinsic parameters. It can be used to train a deep neural network for semantic segmentation or other applications of interest. This paper describes the dataset and provides annotation details. It also explains how semantic segmentation was performed using several neural network models and shows several relevant statistics. MESSI will be published in the public domain to serve as an evaluation benchmark for semantic segmentation using images captured by a drone or similar vehicle flying over a dense urban environment.

Published in Transactions on Machine Learning Research (08/2025)
MESSI: A Multi-Elevation Semantic Segmentation Image Dataset of an Urban Environment
Barak Pinkovich
barakp haapppammmtssttetioocc.....
Deportment of Computer Science Technson-Isre Institute of Technology
Boaz Matalon
mbonz3technnion..ckk
Department Computer Science
Technion-Isse Israel Institute of Technology
Ehud Rivlin
eluddress Hbudr@css.echrrtiee wec..cioo.c....c.c...oo
Department of Computer Science
Techmson-Israel Institute of Technology
Hector Rotstein
hectorStechrion..c.
Department of Computer Science Technion--srael Institute of Technology
Reviewed on OpenReview: https: // openrevie.. net/ form?? id= axggi/ztuugoooeeeetttt------B- date c
Abstract
This paper presents Multi-Elevation Semantic Segmentation Image (MESSI) dataset comprising 2525 images taken by a drone flying over dense urban environments. MESSI is unique in two main features. First, contains images from various altitudes, allowing us to investigate the effect of depth on semantic segmentation. Second, includes images taken from several different urban regions (at different altitudes). This important since the variety covers the visual richness captured by a drone's 3D flight, performing horizontal and vertical maneuvers. MESSI contains images annotated with location, orientation, and the camera's 's intrinsic parameters and can be used to train deep neural network for semantic segmentation other applications of interest (e.g., localization, navigation, and tracking). This paper describes the dataset and provides annotation details. It also explains how semantic segmentation was performed using several neural network models and shows several relevant statistics. MESSI will be published the public domain to serve as an evaluation benchmark for semantic segmentation using images captured by drone or similar vehicle flying over dense urban environment.
1 Introduction
In recent years, the interest in using unmanned aerial vehicles (UAVs) or drones to perform various tasks like package delivery, construction site mapping, traffic analysis, structure integrity survey, police search, acrial taxi services, and change detection analysis, has been gaining momentum and attracted the interest of many researchers Mohsan al. (2023): Han al. (2022). The successful and safe operation of these tasks on completely autonomous UAV requires robust perception of the environment, particularly when sensing is limited to monocular images from downward-looking camera. Two different but connected research activities have evolved from the motivation above. First, an increasing number of works have appeared focusing on exploring alternatives for semantic segmentation of aerial images using deep neural networks, given the substantial improvement this method offers compared to traditional ones. Second, datascts of
The dataset has been published https://isl wccchiin.....//eaereeeeeeetseaee

---

Published in Transactions on Machine Learning Research (08/2025)
drone images are being created to train and benchmark potential semantic segmentation solutions. For instance, the web page Papers with Code contains references to 326 datasets when filtered for Semantic Segmentation. Among them, it is possible to find the now classical KITTI Geiger et al. (2012); Liao et al. (2022) or Cityscapes Cordts et al. (2016) which have been extensively used for different learning tasks, inc including ing semantic segmentation. A Among them, only a few contain aerial views, whereas some are unsuitable
for urban scenes, c.g.; Gupta et al. (2019) Van Etten al. (2021)
In addition, although drones enable flexibility in terms of data collection, this same ficxibility may pose new challenges to the semantic segmentation task. For instance, the review Han et al. (022 enumerates some of the challenges arising when using drone images for perception:
1. Uneven object distribution and size, resulting in acquiring images from different altitudes at constant focal length.
2. Multiple viewpoints and occlusions due to drone mobility in three dimensions. For instance, seg- mentation may fail to segment objects from the background when objects are far from the camera.
3. Limited power supply and computation hardware the sogmentation scheme to be implemented on a relatively small platform.
Although these challenges are quite general, they become especially acute when a drone flies over a dense urban environment and altitude changes are dictated by regulatory constraints or exploited to achieve specific objectives. More specifically, and following Han et al. (2022), this paper proposes the MESSI Dataset for evaluating semantic segmentation algorithms applied to the emergency landing problem as the background application while addressing the problem at different scales (spatial resolution) and multiple viewpoints when collecting images and performing semantic segmentation while flying at different clevations in an urban area proposed in Pinkovich el al. (2022)
When using deep neural networks to solve the semantic segmentation problem, a sufficiently rich and labeled dataset is nceded for training to achieve satisfactory results in real scenarios. The recent work Nigam et al.) (2018) considers some of the properties such a database must. have to serve as a benchmark for semantic segmentation of drone images. Since UAVs may change their altitude during flight, objects may appear in various sizes. Additionally, UAVs are generally equipped with a wide-angle lens. Thus, objects can be viewed from various angles. Motivated by these observations, the Eletti-Elevation Semantic Segmentation Images (MESST) dataset was constructed using analyzed and labeled aerial images captured by a high-resolution downward-looking camera on a drone flying over a dense urban environment. This paper aims to present MESSI and show how can be used to train deep neural nets to perform semantic segmentation. To summarize, MESSI's main contributions are:
1. Provide a dataset of images from an urban region captured by drone at different altitudes. The drone views every area on multiple scales and in different spatial resolutions. The dataset can be used to train a semantic segmentation model that will capture the richness of a 3D flight and thus improve accuracy at inference.
2. Include location and orientation data per image and the camera's intrinsic parameters to extend MESSI to other rescarch topics such as localization, navigation, and tracking (e.g., Chen et al. (018b)
3. Enable researchers to improve their autonomous tasks by modeling phenomena such as the effect of how training a network at single or multiple altitudes improves or degrades model performance (see Fig.
2
Related Work
The motivation for building the MESSI dataset was to provide data for training and comparing deep-learning semantic segmentation algorithms using images captured by the downward-looking monocular camera of
2

---

Published in Transactions on Machine Learning Research (08/2025)
drone flying over dense urban environment. The number of existing datasets more less related to
MESST is quite large. The characteristics of the most relevant ones are summarized in Table 1 to guide the reader before delving into detailed description. For example, openDD Breuer et al. (2020) presents
Table 1: Main characteristics of existing datasets relevant to MESSI
Name
Foatures
Main Application
Source
openDD
Urban
Traffic
Drone
DroneDeploy
Unstructured. No overlap. Single altitude
6 classes segmentation
Satellite
Rural Scape
Rural arca. Multiple altitudc
Video segmentation
HF video
Mid-Air
Unstructured. Low altitude
Autonomous flight
IFF simulated
UAVid
Urban. Single altitude. 45" University. 2 altitudes
classes segmentation
UAV
UDD
classes segmentation
Drone
VDD
Urban. 2 altitudes
classes segientattin
Drone
ICG
Urban. Low multiple altitudes
20 classes auonomous flight
Drone
AcroSpaces
Single low altitude. No overlap
12 classos segmentation
Drone
CityScapes
Urban. Stereo video
Ground level
Road vehicle
understanding
ManipalUAVid
University, Video. Single altitude
4 classes segmentation
Drone
MESSI
Urban. Multiple altitudes & revisiting Includes pose
16 classes segmentation.
Drone
Search sensor
a large collection of dronc-captured images an urban environment, focusing on traffic, and Blaga & Nedevschi (2020) evaluates three acrial datasets: DroneDeploy Droneeepooo (2019), RuralScape Marcu et al. (2020), and Mid-Air Fonder & Van Droogenbroeck (2019). Notice that these three Dataasess-DooneDeeee DroneDeploy, RuralScape, and Mid-Air deal with unstructured environments, and Mid-AN Air specifically deals with high- fidelity simulated images.
A more relevant dataset is UAVid Lyu et al. 2020), containing 300 images that are densely labeled with eight classes, collected from different areas in the urban scene, and aimed at developing algorithms for scene understanding. In UAVid, the camera captures each area from a single altitude with a slant angle of 45 degrees. Consequently, it fails to capture the size and occlusion issues from observing the same environment from different altitudes. Therefore, studying the effect of different altitudes semantic segmentation is of interest since it may shed light on the overall behavior of an algorithm.
Two related datasets are UDD Chen et al. (2018b) and VDD Cai et al. (2023), where VDD is new dataset extended with UDD and new annotations from UAVid. The UDD dataset contains 141 images labeled with six classes taken from 60 to 100 meters. UDD's goal is to use semantic constraints for Structure from Motion (SFM), where only a sample of the images was used for semantic information. VDD contains 400 images labeled with seven classes taken from 50 to 120 meters. MESSI stands out from the latter with its unique feature of organically capturing different scenes from different altitudes, allowing the rescarch community to study the changes in information as function of altitude and potentially infer from one altitude to another. In contrast, UDD and VDD may contain some images taken from different altitudes, but not by design, so as to allow a systematic study.
The ICG Drone Dataset Mostegel et al. 2019 contains 400 images labeled with 20 classes for training and 200 private images for testing. All images are taken from a bird's eye view at an altitude of to 30 meters above the ground. This low altitude range does not provide substantial spatial resolution or scale variations.
Another well-known dataset is AeroScapes Nigam et al. 2018 a collection inspired by the impact of Cityscapes Cordts et al. (2016), a widely used large-scale dataset consisting of urban street seenes on which the instances of each class appear in various scales. AeroScapes is an aerial semantic segmentation dataset containing 3269 images with 11 classes acquired from 141 video sequences from a varying altitude of 5 to meters. AcroScapes includes several objects, some relevant to the applications we are interested in, some not (e.g., sky, boat). AcroScapes lacks the revisiting from different altitudes, the location and orientation per image, and the camera's intrinsic parameters, which are probably the main features of MESSI.
3

---

Published in Transactions on Machine Learning Research (08/2025)
An additional dataset, ManipalUAVid Girisha et al (2019) constructed to evaluate semantic segmen- tation algorithms. It comprises 667 images captured in six different locations on closed university campus. Images were taken from an altitude of approximately 25 meters, and four different classes were annotated. Again, this dataset lacks the richness required for performing semantic segmentation in an urban environ- ment.
Finally, additional datasets like Cordts et al. (0166) or Ballesteros et al. (2022) include different viewpoints but are less relevant since the former consists of street-level images, while the latter also uses DSM of the environment to enrich the available information.
3 The MESSI Dataset
MESSI is unique in several aspects when compared to existing databases. It consists of a collection of images captured by drone at different altitudes. Regions are revisited in multiple scales and spatial resolutions, capturing the visual richness observed by a drone performing horizontal and vertical maneuvers.
The drone was flown in horizontal trajectories in Agamim and Ir Yamim neighborhoods in Netanya and Ha- Medinah Square in Tel Aviv. The Agamim sequences consist of three horizontal trajectory sections (paths A to C), each at four different altitudes: 30, 50, 70, and 100 meters. The number of images was roughly inversely proportional to the altitude as the size of the footprint increased. There is approximately 75% overlap between consecutive frames. The Ir Yamim sequence consists of a horizontal full-coverage trajectory at the same four altitudes. Compared to Agamim, Ir Yamim has different scene composition and class population, but some classes have visual resemblances. The Ha-Medinah Square sequence consists of images taken while flying in a horizontal trajectory at a single altitude of 60 meters. This sequence has a considerable domain summarizes the number and main characteristics of the images sequences. Samples of the horizontal trajectories are available in where the same area is revisited from different altitude.
shift compared to the previous. Table 2
collected for each of the Appendix Sec. A.5 Tables 10 11
MESSI also includes fifteen vertical trajectory sequences on selected locations in the Agamim neighborhood, consisting of images taken by the drone when descending from 120 to 10 meters, with decreasing overlap between images. Each sequence contains either 100 or 125 images. These sequences partly overlap with Agamim's 's horizontal trajectory scenarios. Table. 7 in the Appendix Sec. [A.2 summarizes the number and main characteristics collected at each location. This rather unusual collection can be potentially used to investigate the correlation between semantic segmentation results at different altitudes. An example of the vertical trajectorics is available in Appendix Sec. A.6 Table 12 where Agamim Descend 100 0041 is sampled roughly every 25 meters.
All images were collected using a remotely controlled DJ1 MAVIC 2 PRO. Due to regulation limitations, the drone was flown at linc-of-sight over streets by ALTA INNOVATION a licensed operator as mandated by law. Images were laken by stabilized downward-looking RGB camera while an RTK INS/GPS algorithm was used achieve high navigation accuracy. For every image in the dataset, the drone's degreos of freedom (i.e., the location and orientation of the drone and gimbal) are shared in CSV files.
Table 2: Horizontal trajectory: number of images per altitude and flight length per area
Number of Images per Flight. Altitude
Horizontal Flight
30 im]
50 mm]
60
[m] 70
100 [m]
Total
Length/ Coverr/Cveeeeeee
Agamin Path A
40
25
13
96
195 [u]]
Agamin Path B
51
31
16
121
320 [
Agamin Path C
45
28
20
15
108
340 [
Ir Yauum
166
73
39
20
208
145 [m] x 215 [m]
Ho-Medinahh
52
52
300 [m]]
Square
htttpe://alta.cemm
4

---

Published in Transactions on Machine Learning Research (08/2025)
Figure 1: Google Earth views. Left: 'Ha-Medinah Square' trajectory (magenta). Middle: 'Ir Yamim" coverage trajectory (orange) over the central square. Right: 'Agamim" neighborhoods paths A (red), B (green), and C (blue).
The trajectories described above can be combined to train semantic segmentation neural networks in several ways. Fig. shows a Google Farth 3D model and the ground projection of the horizontal trajectories of the areas Ir Yamim and Agamim, in which the images were taken.
3.1 Annotations Details
The dalabase comprises 2525 frames with an image resolution of 5472x3684 pixels. Pixel-wise annotations with double-independent checking in our quality control process were done using V7 Darwin automated annotation tool. This tool was chosen for several reasons:
The tool had a unique feature: polygon tracking. This feature was essential for us because of the horizontal and vertical trajectories. The images the horizontal trajectories overlap by 50%, so annotations done on the current image can be transferred the next one while adjusting the mask in the following image with click of button. This feature even more critical in the vertical trajectories, where most objects reappear in the following image but with different scales and orientations. Overall, this feature significantly shortened the annotation process.
V7 also provided professional subcontractor annotation companies. We defined the object classes, annotation process, and workflow. The workflow and quality control were rigorous. The workflow was defined and monitored online and contained three main phases. The first phase was the amno- lation. In the second phase, we defined the percentage of mask images that other annolators would check. In the last phase, we defined the percentage of mask images to be checked by us. At each quality control phase, remarks or errors are written to the annotator on the image, and the process goes back to the annotation phase. In fact, we rechecked nearly 100% of the mask images in the third phase. All the quality control was monitored online the tool.
A class taxonomy table included Sec. A.3 of the Appendix. Several relaxation steps were taken without contradicting the overall objective. One such step annotating buildings whole, mcaning that any visible objects (e.g., chairs, stairs, objects on the balcony) inside building were not labeled.
3.2 Statistics
A dataset can be partitioned into training, validation, and test sets in many different ways. Since the trajectories of Ir Yamim and Ha-Medinah Square do not overlap with those of Agamim, they were combined to form the test set to evaluate the models' out-of-listribution properties. Test images will be released as an online benchmark with undisclosed ground truth annotations. The descent scenarios Agamim were selected as the training set. In order to limit training time, only one out of five images was taken. Additionally, three descend scenarios were excluded from the training (i.e., 1, 35, and 36) due to their significant similarity with other scenarios.
77labs.
5

---

Published in Transactions on Machine Learning Research (08/2025)
x10
10[m] <Altitude [m]
14
10
Population
trares lefr
walking lorr.
vegelation
buikiig
rough tarr.
soft lorr.
vahicle
farce
shied
offer
pale
atairs
person
water
Alcycoee
void
Category
Figure 2: Statistics of the various categories in the combined training and validation sets
Table 3: The selected composition of the training, validation, and test sets
Training
Validation
Test set
Images
set
Agamin Descend every fifth
295
image (sets 1, 35, 36 excluded)
Agamin Path A
+
96
Agamin Path B
+
121
Agamin Path C
+
108
Ir Yamim
298
Ha-Modinah Square
52
Paths A to were defined as the validation set. Table. D summariizes how the dataset was divided. As done this work, this set may be used for lyprraarameter search and can be added later to the training set in order to improve overall results. The class population histogram of the combined training and validation sets is shown in Fig. 2 There is large imbalance between the highly populated and classes (compare Transporlation Terrain or Building with Person or Bicycle) and can be challenging for semantic segmentation algorithm, giving poor performance by mostly ignoring some of the rare classes. The next section shows how selecting an appropriate weighting policy can partly overcome this difficulty.
vold
buildinng
lother
pole
shad
stalrs
vegetation
walking terr.
heleccle
person
rough tarr.
water
1x00
Population
[0im)sAitttuee3330
33mm
60lm)
Figure 3: Training and validation set pixel categories statistics per attitude interval
In addition, we investigate the class distribution as function of altitude by considering four altitude intervals (10, 33), (33, 60), (60, 88), and (88, 125] meters. Fig. 3 shows the four distributions consistent. with these

---

Published in Transactions on Machine Learning Research (08/2025)
intervals. Notice that the class distribution changes with the altitude intervals, and that three groups of categories change their distribution when the altitude increases. In the first group, the Transporlation terrain and Soft terruin are always the highest and one of the lowest populated classes, respectively, although the distribution within the classes changes as a function of the altitude. In the second group, Vehicle and Pole are the most and least populated classes, respectively. To further investigale the class population trends, a y-axis log-scaled spaghetti plot is presented Fig. 1 We can notice, for instance, that the Buiiding, Shed, Stairs, and Water populations increase as the altitude interval increases, while conversely, the Transporlation terrain and Person populations decrease.
MESSI enables to study the effect of images taken from different altitudes on semantic segmentation al- gorithms, & property that is hard to benchmark using existing datasets. MESSI can be used to train the network either on a specific altitude interval or all altitudes, improving the accuracy when testing images with different spatial resolutions taken different altitudes.
trone
ter:
vegetation
fenoe
other
PHRR0
WWW
MoNt-AMluue--uun
Figure 4: Class over altitude intervals spaghetti plot
4
Experiments
This section aims to illustrate how MESSI can be used to train and test a semantic segmentation algorithm.
4.1
Implementation Details
The open-source semantic segmentation toolbox MMsegmenntttio Contributors test various semantic segmentation models. Performance was assessed using the Jaccard index, commonly known as the Intersoction-ovee-Uniio Union (IoU) metric, both per category and averaged over all categories (mloU).
Contiibbttniiinnotiiiuuttttto (2020) was used to train and
One method of dealing with imbalanced datasets is to increase samples' weight from rare categories. As a result, segmentation accuracy improves in the sense that these categories are not filtered out due to their small representation the overall set. Three different schemes were tested: 1) uniform weight for all classes (referenced as 'Equal'), 2) a weight inversely proportional the representation of each class in the training set ('Prop'), ), and 3) a weight inversely proportional the square-root of the representation of each class in the training set ('Sqrt'). Note that the class weights are normalized by their total sum to prevent the need to modify some optimization hyper-parameters and the learning rate in particular.
As well-known in the literature, the generalization of a neural net can be enhanced by using a pre-trained model for initializing the optimization process. Since MESSI contains images from multiple heights, the size of an observed object may vary significantly due to its distance from the camera. Hence, an initial model trained on datasets with similar characteristies may prove advantageous. Cityscapes Cordts et al. (2016) is large and diverse dataset in which instances of each class (e.g., vehicles, poles, buildings) appear in various
7

---

Published in Transactions on Machine Learning Research (08/2025)
scales. For this reason, and similarly to UAVid Lyu et al. (2020). Cityscapes was selected as the dataset for initialization.
Image augmentations during training are another standard way to facilitate generalization. Since a drone can hover at any altitude during the test time, resizing the images may be a good way to improve the segmentation results. Thus, random resizing by up to 15% was employed during training. In addition, a photometric distortion augmentation was performed on the training images to accommodate the model for the slightly changing appearance of surfaces and objects. It includes small random modifications to the images' brightness, saturation, contrast, and hue. Finally, random horizontal flipping was also employed.
The main experiments were performed on an RTX 3090 graphics card with 24 GB memory. Unfortunately, running inference of most medium-sized models (e.g.AA. A.1 on a single 5472x 3684 image was found unfeasible, let alone training with such large images. Therefore, and similarly to the approach followed by UAVid Lyu ct al. (2020), each image was divided into 3x3 overlapping tiles of 2048 by 1366 pixcls, maintaining ratio of 3:2 as the original image. Two adjacent tiles, either 1712 (horizontally) or 1141 (vertically) pixels apart, should be placed to occupy the entire image. These tiles are fed to the model individually during testing, and the prediction inside overlapped areas is averaged. During training, which requires extra memory due to gradient calculation, cach image in the mini-batch (of size two) is a random crop of 1024 by 1024.
Three different main models for semantic segmentation were used on the MESSI dataset. DeepLabV3+ Chen et al. (2018a) is a state-of-the-art convolutional net combining an atrous spatial pyramid that captures multi-scale features with an encoder-decoder structure that gradually refines the boundary between objects. SegFormer Xie et. al. (2021) is relatively efficient Transformer-based segmentation model. Tts encoder is hicrarchical Transformer that does not require positional coding, while its lightweight MLP decoder simply aggregates information from layers with different scales. Bi Yu et al. (2018), which was also employed in PPinkovich et al. (0022), fuses a context path with a spatial path. The context allows information from distant pixels to affect a pixel's classification at the cost of reduced spatial resolution. In contrast, the spatial path maintains fine details by limiting the number of down-sampling operations. Two variants from each main model were chosen: the lightest and the heaviest, as long as a batch size of two fits into the GPU memory during training.
Finally, Mask2Former Cheng et al. (0022). state-of-the-art end-to-end architecture with a key component of masked attention, which extracts localized features by constraining cross-attention within predicted mask regions, was also tested. MMsegmentation does not support the overlapped tiles method in the Mask2Forrerr implementation, so the entire image was needed; therefore, to try to achiove the best results, Mask2Former's largest model whose inference can occupy an A6000 graphics card with 48 GB memory was used. Its training peper-parameters were set to the same values as all other models here, e.g. each mini-batch (of size two) is random crop of 1024 by 1024.
The number of training epochs for all models was set at 320, the maximal default value suggested by MMssemennatiin. Other than those already mentioned, all hyper-parameters were set to the default values used by MMsegmentation on Cityscapes.
A performance table depicting the running time per crop (and on the entire image for Mask2Former) and memory utilization during inference for all the variants chosen this paper is presented in the Appendix Sec. A.I Table 6 As readily observed, BiSeN has the fastest runtime, while SegFormer- B3 and are the slowest.
As mentioned above, MESSI is unique in that it captures arcas at different altitudes; thus, another experiment was performed to examine the effect of this feature on performance. Specifically, the SegFormer-Bi B3 model with Sqrt weighting was trained on all 'Agamim Path" and 'Ir Yamim" scenarios contained within a defined set of altitudes (horizontal trajectories). Then, the model was tested on the entire collection of Descend scenarios (vertical trajectories) without exclusions. All typer-parametery of the models were chosen as described above, except for the number of epochs that were reduced to 160. The results of these experiments, measured by average accuracy per pixel, are presented in the next section (Fig. 7
8

---

Published in Transactions on Machine Learning Research (08/2025)
4.2 Results
Fig. 5 shows a sample image from Ha-Medinah Square, its ground-truth annotation, and its corresponding prediction using a SegFormer-B3 model. observed, a large proportion of the image is accurately segmented and classified. The more frequent cases of confusion are between Soft and Rough Terrain, Transportation and Walking Terrain, and Poles and Other Objects.
One of the more noticeable errors appears in the rightmost building, where part of the roof was misclassified Shed. Although formally error, it is indeed shaded area the roof which, as mentioned in Sec. 3 was filled during the annotation process. Consequently, whenever building contains objects or surfaces that otherwise should have been annotated, they might be misclassified, as was the case here. This problem can be sat relaxed by ignoring the Building class both in training and lesting and therefore, each experiment was repeated with that category omitted. More sample predictions are available in the Appendix, Sec, A.7
Tables 13
14
(a) Sample test image
(b) Ground truth
(c) Inference: SegFormer-Bn BB
Figure 5: A visual example of Ha-Medinal Square test set (see table for color legend)
Table 4: Performance on the full test set
Weighting Method
All Categorics
w/o Building
Model Type
mAce
mIoU
mAce
mloU
BiSeNetV1-13
29.0
17.9
30.8
19.4
BiSeNetV1-50
31.8
20.9
32.4
20.6
DeeplabV3+118
51.9
39.8
55.6
44.4
Equal
DeepLab33+50
57.0
45.7
59.2
47.9
SegFormer-B0
56.8
43.7
57.6
16.7
ScgFormer-B3
63.4
50.3
63.3
52.9
Musk2Forreer-Swinn-
37.7
45.7
57.8
38.8
BiSeNetV1-18
54.7
35.0
55.3
39.7
BiSeNetV1-50
54.4
35.8
55.9
40.1
DeepIabV3+18
37.5
42.7
61.3
45.7
Sqrt
DecpLab V3 50
59.9
44.7
62.8
48.5
SegFormer-BD
62.5
44.1
63.8
48.8
SegFormer-B2
67.7
52.6
66.6
54.5
Mask2Former-SSua Swin-
57.1
44.5
57.3
40.4
BiSeNetV1-18
27.1
6.6
30.8
7.1
BiSeNettV-550
20.0
3.3
21.5
4.0
DeepLabV++10
58.8
37.1
61.6
43.1
Prop
DeepLab
60.9
41.4
63.2
43.8
SegFormer-Bii
63.0
40.1
65.1
44.8
SegFormer-B3
47.9
67.7
51.8
Mask2Former-Swin.
48.0
35.8
60.9
40.7
inference on the entire image.

---

Published in Transactions on Machine Learning Research (08/2025)
Table 4 shows the performance of the various models on the entire test set. Although the principal measure is mloU (defined in the previous subsection), mAcc (mean Accuracy) is also presented for completeness. The best result for each of the six selections is boldfaced. Note that weighting methods experiments are not normally conducted when publishing a new dataset (ef. Cordts et al.] (2016) Lyu et al. (2020) Nigam et al. 2018) and more). The table shows that: 1) as could be expected, the larger the model, the better it performs; 2) the Sqrt weighting method usually outperforms the others, especially for the least successful models; and 3) almost unanimously, discarding the Building category improves the quantitative results.
Table 5: IoU per category the full test set when using SegFormer-B3 model
Weighting Method
Equal
Sqrt
Prop
Inc. Building?
Yes
No
Yes
No
Yes
No
building
56.1
61.2
55.7
trans. terr.
74.3
74.8
76.7
74.8
71.6
71.6
walking terr.
80.7
83.9
82.2
84.2
80.2
82.1
stairs soft terr.
41.5
26.3
41.2
30.1
25.4
33.6
71.0
86.8
70.9
88.1
69.2
86.0
rough terr.
44.2
39.1
44.9
43.7
42.6
43.6
vegetation
76.9
77.3
76.9
77.5
76.9
77.4
water
0.3
6.3
2.1
1.8
0.7
9.7
pole
19.7
22.0
24.7
26.8
23.5
21.7
shed
51.2
64.7
63.1
71.1
52.3
67.1
fence
52.1
58.4
54.0
59.2
49.0
53.2
vehicle
90.6
91.1
89.5
90.8
86.5
87.2
bicycle
19.2
23.5
26.5
27.5
21.7
18.5
person
46.5
46.8
42.8
45.1
32.9
34.3
other object
30.3
39.8
31.7
42.5
30.3
38.5
mloU with water
50.3
52.9
52.6
54.5
47.9
51.8
mIoU without water
53.9
56.5
56.2
58.6
51.3
55.0
Mask2Former did not perform anticipated. As mentioned in Sec. 4.1 the training conditions were the same those for the other models (except for using GPU with larger memory size to train the second- largest model). However, due to MMssenenntttion's implementation, the inference was performed on the entire image and not on overlapped tiles, which was thought to achieve better rosults. Mask2Former is region-based, which could explain why it did not perform as well as SegFormer on small objects. Similar results were reported in Wang et al. (2023), where Fask2Former was trained and tested on aerial images. The IoU per category for Mask2Former is presented in the Appendix Sec. A.4 Table where evidently it performed significantly worse classes such "bicycle' and 'stairs" than SegFormer-B3 (see below).
The ToU per category for SegFormer-B3 (the best-performing model) is presented in Table 5 where the highest score for each category is boldfaced. One can observe that, for the most part, the trends of the overall mloU are maintained here using the Sqrt weighting method and discarding the Building category improve performance. Another observation that the rare categories (e.g., Water, Bicycle, Person) generally get the worst scores. Notable exceptions are Shed and Vehicle, where the latter actually has the best score among all categories. A possible explanation is that the Vehicle category is quite distinct in appearance.
Another way to interpret the results is by looking at the confusion matrix in Fig. 6 This matrix calculated for the SegFormer-B3 model (with Sqrt weighting and for all categories). As readily seen, the least populated categories (e.g., Water, Pole, Bicycle, Slairs) have significant off-diagonal components. In particular, for the Water category, some of its instances are located on buildings' roofs, which explains the significant confusion between them and why it was omitted from Table 5 and from the mIoU calculations.
In addition, Pole is often misclassified Other Objects since low-enough poles were labeled as the latter. A Another observation is that the Building category is occasionally confused with Soft Terrain. This stems
10

---

Published in Transactions on Machine Learning Research (08/2025)
from having some unusual buildings with grass-covered rooftops in the Ir Yamim images. Finally, Rough Terrain is frequently misclassified as Vegetation since, looking from above, it may be difficult. to distinguish between the two.
parionnaaionnnnooo Harrain
Mion terran
Dieydte
Oeter
okyde
bakding
65.00
1.87
0.01
0.01
0.00
2.41 33.13 0.7 0.11
0.11
6.59
0.8
fance
4.37
95.74
1338
0.0
0.31
2.12
0.36 1.66
0.58
D39
200
6.33
0/8
50
other o0jects
person
0.49
0.26
17.36
83.2
1.36 0.07
0.07 0.00
0.76 0.16
1.39
0.8
Ground Truthh Label
2,42
6.39
40.04
6.45
$2.00
0.51
2.92 0.4
0.07
50b
1.36
9:28
0/0
Hough tomaln
shod
0.01
6.35
0.76
6.78
6.03
0.75
0.57
16
18
0.24
0.03
0.38
0.21
0.02
9.74
0.0
scft terteertt
0.02
0.1
0.21
0.0
0.05
0:04
93.07
0.29
2.8
3.26
0/0
stars
transportation berrats
0.9
6.74
0.46
0.29
6.03
0.05
0.08
0.02 0.48
0.0
89
0.23
0.68
7.49
0.0
5.23
0.56 0.33 6.02 0.05 2.72 0.22 1.4
0.27
33.43
0.22
0.0
20
Walcing berrets 0.02
1.94
1.27
0.67 5.04 0.06
0.29 0.29 0.33 0.12 4:88 111 0.04
99.31
68
07.73
2.41 2 0.0 0.0 1.67 2.22 9.45 0.07 2:25
3.67
6.08
2.00
Prediction Label
Figure 6: Normalized confusion matrix on the test set for the SegFormer-B3 (All categorics, Sqrt weighting)
MESSI was mainly constructed to enable the study of the effect of images laken from different altitudes on semantic segmentation algorithms, a property that is hard to benchmark using previous datasets. Fig. 7 stresses the significant comparative advantage of using MESSI. The figure shows how training the best- performing model SegFormer-B3 with Sqrt weighting scheme horizontal trajectory at different altitudes (i.e., 'Agamim path' and 'Tr Yamim" data) improves degrades the accuracy when testing on images from the vertical trajectories (i.e., "Agamim descend' data). Fig. 7a shows the accuracy degradation when training at higher altitudes but desconding to lower altitudes where no training was performed. For instance, training on images from all 'path" altitudes (i.e., 30, 50, 70, 100 meters) has little or no effect on the "descend" test accuracy. However, the average accuracy when training on a limitod set of altitudes (like 100 meters Fig. 7al results in lower performance as the elevation decreases. Fig, 7b shows the accuracy degradation, but this time, the training images are taken from lower altitudes, and the drone ascends to higher altitudes where no training was performed. At first glance, training lower altitudes and extrapolating to higher altitudes seems better. However, this precisely emphasizes the tradeoff:
1. Lower altitude: higher spatial resolution, smaller ground footprint, longer trajectories, more obsta- cles
2. Higher altitude: lower spatial resolution bigger ground footprint, shorter trajectories, fewer obstacles
Tt is worth stressing that accuracy was caleulated on the overlapping field of view from all altitudes in each trajectory of the 'descend" data area: whereas at the lowest altitude, the accuracy was calculated on the entire image, as ascending, a smaller area of the image participated in the calculation.
5 Summary and Conclusions
This paper presents new dataset called 'MESSI,' designed to serve as an evaluation benchmark for elevation- based semantic segmentation algorithms. MESSI consists of rich collection of aerial images at. different
11

---

Published in Transactions on Machine Learning Research (08/2025)
1.00
0.95
0.95
0.90
0.90
0.85
0.85
0.80
0.75
0.75
0.70
0.70
0.85
0.85
0.60
0.55
0.55
0.50
0.50
0.40
0.40
0.35
0.35
0.30
0.30
Horizontal trajectory training (100)
Horizontal trajectory training (30)
0.20
0.20
0.15
Horizontal trojectory training altitudes [70, 100)
0.15
Horizontal trojectory training altitudes [30. 50]
0.10
Horizontal trajeztory training altitudes |50, 70, 100)
0.10
Horizontal trajeztory training altitudes [30, 50, 70)
Horizontal trajectory training altitudes (30, 50, 70, 100)
Horizontal trajectory training altitudes (30. 50, 70, 100)
0.00
0.00
120
120
Vertical trajectory test altitude[m]
Vertical trajectory test altitude(m)
(a) Training with descending horizontal trajectories
(b) Training with ascending horizontal trajectories
Figure 7: Training altitude impact on semantic segmentation accuracy
altitudes over dense urban environment and is unique several aspects. The dataset was annotated using Darwin, the V7 pixel-wise tool, with 15 different classes. Extensive effort was made make the dataset error-free and consistent. A statistical study of the resulting semantic maps shows that there is a large variability in population between classes, challenge for current semantic segmentation algorithms.
MMSogmentation Contributors (0200) was used to train and test various semantic segmentation models, and an Intersection-over-UUU Uninn metric, both per category and averaged, was used to assess performance. Three different weighting policies were used to overcome the population imbalance. Overall, SegFormer MiT-B3 provided the best results over six tested alternatives using three weighting methods.
The results in the paper show that MESSI can indeed be used to train deep neural network for performing semantic segmentation on aerial images over dense urban environment. The fact that MESSI contains annotated images from both horizontal and vertical altitudes can be used to study the effect of altitude on semantic segmentation algorithms and properties that are hard to benchmark using existing datasets. Finally, MESSI is published in the public domain to help improve the result presented in this paper, by potentially utilizing other network structures or weighting policies.
References
John R. Ballesteros, German Sanchez- sorchhez-Toress,.. and John W. Branch-Bedoya, Hagdavs: augght-a geo- located dataset for detection and semantic segmentation of vehicles in drone aerial orthomosaics. Data, 7 (4), 2022. ISSN 2306-5729. doi: 10.3390/datta74400000 URL htttp::/wwww.mpi..oo wttpppp..coo//300577////////////
Bianca-Cerasella -eelaala-eeai Blaga and Sergiu Nedevschi. A critical evaluation of aerial datasets for semantic segmentation. In 2020 IEEE 16th International Conference Intelligent Computer Communication and Processing (ICCP), pp. 353-360, 2020. doi: 0.110//1CCP5500022222..9229
Antonia Breuer, Jike Termöhlen, Silviu Homoceanu, and Tim Fingscheidt. opendd: A large-scale roundabout drone dataset. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp. 1-6, 2020. doi: LLLL09//TSCC5512220000.22222
Wenxiao Cai, Ke Jin, Jinyan Hou, Cong Guo, Letian Wu, and Wankou Yang. Vdd: Varied drone dataset for semantic segmentation. arXiv preprint 1XXi::3355.1808 2023.
Siang-Chich Chen, Yukum Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European con- ference on computer vision (ECCV), pp. 801-818 2018a.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Yu Chen, Yao Wang, Peng Lu, Yisong Chen, and Guoping Wang. Large-scale structure from motion with semantic constraints of aerial images. In Pattern Recognition and Computer Vision: Pirst Chinese Conference, PRCV 2018, Guengzhou, China, November 23-26, 2018, Proceedings, Part 1 1, pp. 347 359. Springer, 2018b.
Bowen Cheng, Ishan Misra, Alexander Schwing, Alexander Kirillov, and Rohit Girdhar. Maskod-attention mask transformer for universal image segmentation. In Proceedings of the IBEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp.
2022.
MLMSegmentation Contributors, MMSSegmentttion: Openmmlab semantic segmentation toolbox and bench- 2020. Apache 2.0 License.
mark. https:///g.thub..cmm com http:///......aaaaaaaaaaaaaaaaaa
Marius Cordts, Mohamed Omran, Scbastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cilyscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3213 3223, 2016.
DronoDeploy.
Segmentation dd-mi 1d-m1-=a femmmm--eeeeee well-segmeettiiin aeemmentationn-bechharz. benchmark
Benchmark, 2019.
Michaël Fonder and Mare Van Droogenbroeck. Mid-air: A multi-modal dataset for extremely low altitude drone flights. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation (CVPR), pp. 553 562, 2019. doi: (0.1109/CVPRM 201..000119000000
A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
pp.
S Girisha, Manohara MM Pai, Ujjwal Verma, and Radhíka M Pai. Performance analysis of semantic seg- mentation algorithms for finely annotated new uav aerial video dataset (manipaluavid). IEEE Access, 7: 019239-136253, 2019.
Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xbd: A dataset for assessing building damage from satellite imagery. arXiv preprint arxiv:191 1.09296, 2019.
Yuqi Han, Huaping Liu, Yufeng Wang, and Chunlei Liu. A comprehensive review for typical applications based upon unmanned aerial vehicle platform. IEEE Journal of Selected Topics in Applied Earth Obser- vations and Remote Sensing, 15:9654-966 9666, 2022.
Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43():: 3292 3310, 2022.
Ye Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing, 165:108- 119, 2020.
Alina Marcu, Vlad Licaret, Dragos Costea, and Marius Lcordeanu. Semantics through time: Semi-supervised segmentation of aerial videos with ilerative label propagation. In Proceedings of the Asian Conference on Computer Vision, 2020.
Syed Agha Hassnain Mohsan, Nawaf Qasem Hamood Othman, Yanlong Li, Mohammed H Alsharif, and Muhammad Asghar Khan. Unmanned aerial vehicles (uavs): Practical aspects, applications, open chal- lenges, security issues, and future trends. Intelligent Service Robotics, 16(1):109-137, 2023.
Christian Mostegel, Michael Maurer, Nikolaus Heran, Jesus Pestana Puerta, and Friedrich Fraundorfer. Icg drone dataset. httpp:://droneataasst .ig.tii.g...... tugraz. at 2019.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Ishan Nigam, Chen Huang, and Deva Ramanan. Ensemble knowledge transfer for semantic segmentation. In 2018 IEEE Winter Conference on Applications of Compuler Vision (WACV), pp. 1499 1499-1508, 2018. doi: N0.1109/WACK 2018.00168.
Barak Pinkovich, Boaz Matalon, Ehud Rivlin, and Hector Rotstein. Finding landing site in urban area:
A multi-rosolution probabilistic approach. Sensors, 22(24):9807, 2022.
Adam Van Etten, Daniel Hogan, Jesus Martinez Manso, Jacob Shermeyer, Nicholas Weir, and Ryan Lewis. The multi-temporal urban development spacenet datasct. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognilion. (CVPR), pp. 6398 6407, 2021.
Di Wang, Jing Zhang, Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. Samrs: Scaling-up remote sensing segmentation dataset with segment anything model.
Oh, T. Naumann, A. Globerson, K. Sacnko, M. Hardt, and Levine (ods.), Advances in Neural Information Processing Systems, volume 36, pp. 8815 8827. Curran Associates, Inc., 2023. URL httttt://pptooooo httpps//pproceeiinng.. wtwpeiiiinnnn.nuuiiineeee www.ieeuriicoom cttps.cc/ppppeee wttpcccpppppffiaaeaec sampeerrfflese/eeeeeeeeeeeeesrrr files/ wawaddBenhmaarss..e.... pdf
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Aduances Neural Information Processing Systems, 34:12077 12090, 2021.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang, Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 325 341,,,, 2018.
14

---

Published in Transactions on Machine Learning Research (08/2025)
A Appendix
A.1 Inference Time and Memory Usage
Table presents the running time and memory utilization during inference for all the variants chosen this paper. The performance for all models measured per crop, while for Mask2Former on the entire image. As can be seen, Mask2Former is the largest and slowest model.
Table 6: Inference time and memory usage for the different models
Model Type
Encoder Backbone
Inf. Time (ms./crop)
Inf. Memory
(GB)
BiSeNet
ResNet-18
8.6
6.07
BiScNet V1
ResNct-50
87.5
6.65
DeepLab V3+
V3+
ResNet-18
19.8
5.31
DeepLabV3+
ResNet-50
160.1
7.97
SegFormer
MiT-BO
104.9
10.5
SegFormer
MiT-B3
316.5
10.76
Mask2Former
Swin-B
1550
31.317
inference on the entire image.
A.2 Vertical Trajectories Scenarios
MESSI includes 15 vertical trajectory sequences in selected locations in the Agamim neighborhood, consisting of images laken by the drone when descending from 120 to 10 meters, with decreasing overlap between images. Each sequence contains either 100 125 images.
Table 7: Vertical trajectory: number of images per location and flight length per area
Number of Images
Vertical Flight
Agamim Location
per Flight Altitude
Length
100 0000
125
106 m
100_0002
125
107 m
100_0008
100
97 m
100 0004
125
101
100 0005
125
104 m
100_0006
125
107 m
125
110 m
100_0035
125
104 m
125
108
100 0037
125
106 m
100 0038
125
108 m
100 00-00
125
103 m
100 100_0041
125
106 m
100. 0042
125
101
100 0003
125
101 m
15

---

walking ter- Published in Transactions on Machine Learning Research (08/2025)
A.3 Segmentation Taxonomy
The segmentation taxonomy was designed for finding safe, obstacle-free ground areas suitable for landing while considering human, property, and drone safety. Table g shows the selected class laxonomy used in MESSI.
Table Classes with brief description
Class Name
Description
GT
Color
building
without including their ground-floo yards road, parking lot, bicycle lanes
transportation
terrain
sidewalk, walking lanes, playground, basketball court, concrete, deck, porch, etc.
rain
stairs
stairs
soft terrain
grass, soil (with no bushes), gravel, yard, etc. soil mixed with low bushes, rocks, junk, construction ma- terials, and bleachers
rough terrain
vegetation
bushes, frees
waler
any water deposit, including pools
pole
lighting poles (inc. base), traffic signs, etc.
shed
shed, gazebo, sun shades, shaded bus stations fence, wall
fence
vehicle
any vehicle with four or more wheels
bicycle
bicycles, motorbikes, scooters and other small vehicles either walking standing (not in a vehicle)
person
other object
including trash bins, benches, phone booths, statues unlabeled pixels (not actual class)
void
A.4 IoU Per category The Full Test Set When Using Mask2Former Model
Table 5 presents the ToU per category on the complete test set when using the SegFormer-BS model. A similar table containing the IoU per category on the Mask2Former model is presented in Table g Please note that Fask2Former performs significantly worse in classes such as "bicycle" and "stairs." Table 9: IoU per category the full test set when using Fask2Former model
Weighting Method
Equal
Sqrt
Prop
Inc. Building?
Yes
No
Yes
No
Yes
No
building
44.45
49.58
28.59
trans. terr.
79.81
71.35
77.15
66.72
55.06
74.69
walking terr.
84.16
82.41
79.61
76.14
70.79
79.39
stairs
5.21
1.04
0.34
0.17
2.5
2.39
soft. terr.
69.48
69.45
67.74
60.33
67.
61.06
rough terr.
41.33
37.97
39.82
26.1
28.84
27.64
vegetation
76.05
75.94
75.64
75.16
72.63
63.87
water
0
0
0
0
0
pole
30.31
28.1
21.49
27.49
25.1
29.29
shed
31.38
26.18
27.86
35.32
16.19
38.18
fence
48.24
46.19
50.35
45.04
47.97
35.65
vehicle
91.26
27.54
87.83
72.96
88.03
77.96
bicycle
9.89
0.71
2.66
person
41.39
44.75
52.71
50.73
4.84
39.56
other object
32.25
31.92
31.34
29.14
28.65
27.83
mloU with water
45.7
38.8
44.5
40.4
35.8
40.7
mloU without water
48.94
41.81
47.65
43.48
38.37
43.84
16

---

Published in Transactions on Machine Learning Research (08/2025)
A.5 Sample Images of Horizontal Trajectories
The trajectory is performed at different elevations (30, 50, 70, and 100 meters) in Ir Yamim and Agamim Path A, B, and C. Out of the entire path, a single area has been selected to emphasize the difference in information captured at different altitudes. In contrast, Ha-Medinah Square of the test set captured only at 60 meters; therefore, various areas of the trajectory are presented. Table 10: Horizontal trajectories sampling the same location at all elevations
A sample of images of horizontal trajectories is presented in Tables 10
30
50
70 m
100 m
Agamim Path A
Agamim Path B
Agamim Path C
Ir Yamim
Table 11: Ha-Medinah Square horizontal trajectory sampling the trajectory at 60 meters
60m
Ha- Medinah
Square
17

---

Published in Transactions on Machine Learning Research (08/2025)
A.6 Sample Images Of Vertical Trajectories
Agamim Descend 100 0040 is presented in Table 12 in which the drone descends vertically from 125 to 10 meters. The images are sampled roughly every 25 meters.
Table 12: Agamim vertical trajectories sampling Agamim Descend 100
117m
100m
85m
60m
34.5m
11m
18

---

Published in Transactions on Machine Learning Research (08/2025)
A.7 Sample prediction images with SegFormer MIT-B3
Some Ir Yamim and Ha-Medinah Square prediction examples using SegFormer MIT-B3 are presented in Tables 13 and 14 respectively. In Ir Yamim, although the "soft terrain' and "other objects" on the roof of the building were not annotated in the ground truth, the model actually predicted them. Morcover, both Ir yamim and Ha-Medinah Square, the model occasionally confuses between 'soft terrain' and "rough terrain".
19

---

Published in Transactions on Machine Learning Research (08/2025)
Table 13: Ir Yamim: Prediction examples with SegFormer MIT-B3
Image
Ground truth
Inference
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 14: Ha-Medinah Square: Prediction examples with SegFormer MIT-B3
Image
Ground truth
Inference
21


Reviews:
[{"review_id": "frcYo7013z", "reviewer": "Reviewer_GNHy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. A new dataset is introduced for evaluating semantic segmentation algorithms.  It contains the visually rich images captured by a drone flying over several areas in different horizontal and vertical trajectories. This would be of interest to several people in the TMLR community.    \n2. Recent SOTA segmentation models with three different weighting policies have been tested on MSSEI.     \n3. This paper is easy to read and uses many tables and figures for an intuitive visualization.    \n\n## Weaknesses\n\n1. The real work section is quite chaotic and lacks clear organization. The related works should be presented in a clear sequence, either by publication date or relevance to your work.\n2. Some expressions are unclear and confusing.\n3. Missing out on some crucial content and details, such as the composition of the training set on Page 5 and the view of “Ha-Medinah Square” in Figure 1.\n4. The papers mentioned that “The Ha-Medinah Square sequence has a considerable domain shift”, but the papers did not explain how to overcome the domain shift in Ha-Medinah Square. If possible, domain shift can also be used as a research direction for this dataset, so as to increase the contributions of this work.\n5. When discarding the Building category, the mIoU of models in Table 2 normally increases, but the mIoU of Mask2Former-Swin-B becomes lower using Equal and Sqrt, meanwhile, its mAcc w/o Building is higher than the mAcc with all categories. Why does this happen?\n6. Why is the mIoU of Segformer-B3 in Table 2 inconsistent with that in Table 3?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new dataset called MESSI, taken by a drone in an urban environment. MSSEI contains images from various altitudes and has a large class imbalance, which can serve as a challenging evaluation benchmark for semantic segmentation algorithms.  Classical segmentation models and three different weighting policies have been evaluated on MESSI.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tThe realized works section needs to be reorganized. It is better to introduce related work in a certain order, such as publication date or relevance. Alternatively, the deficiencies of the current datasets can be categorized and summarized into several categories, and finally, it would be claimed that this work has overcome or compensated for these deficiencies.\n2.\tSome expressions are unclear and confusing:    \n   (1) In “To mention two”, what does “two” refer to?  As there are several datasets following this sentence.   \n   (2) Similarly, in “Notice that these three datasets deal with”, what does “these three” refer to?    \n   (3) “Further motivation for this problem may also be found in Pinkovich et al. (2022)”. It is better to explain the motivations in this paper, rather than let the readers find them in other papers.   \n   (4) In “models;and 3) almost unanimously”, there is a missing space.    \n   (5) The model name of SegFormer-B3 should be fixed, instead of sometimes “SegFormer-B3” and sometimes “Segformer-B3”.   \n3.\tMissing out on some crucial content and details.   \n(1) In Figure 1, “Left: the trajectory (red)”, but there is no red instead of orange in the left subfigure. In addition, why is the view of “Ha-Medinah Square” absent? It is better to add it in Figure 1.  \n(2) In Page 5, as the paper states that Ir Yamim and Ha-Medinah Square are test set and Paths A to C are the validation set, what is the composition of training set?  It is suggested to draw a table to clearly explain how to divide these sets and the number of samples for each set, so as to form a unified standard for the latter researchers to evaluate their models.  \n4.\tPlease explain what has been done to deal with the domain shift in Ha-Medinah Square, especially when it is a part of the test set.  \n5.\tPlease explain why the mIoU of Mask2Former-Swin-B becomes lower using Equal and Sqrt, meanwhile its mAcc w/o Building is higher than the mAcc with all categories in Table 2.   \n6.\tPlease explain why the mIoU of Segformer-B3 in Table 2 is inconsistent with that in Table 3.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXXgfFZfg8", "reviewer": "Reviewer_sAFp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\nS1: The paper introduces an innovative and practical concept by providing real-world data of the same scene captured at different altitudes, potentially offering valuable insights for in depth analysis.\n\n## Weaknesses:\n\nW1: The paper claims multiple times that the MESSI dataset enables the study of the effect of images taken from different altitudes on semantic segmentation algorithms, a property that is hard to benchmark using existing datasets. However, to analyze these effects in depth, the frames need to be aligned. Given Image 1 at an altitude of 30m and Image 2 at 50m, one would need the transformations to determine which areas in Image 2 are covered in Image 1, assuming a quasi-static scene. Analyzing class IoUs or mIoU for different altitudes is a first step but cannot provide in-depth insights. While this is mentioned briefly on page 10, it is not described in detail in the paper. The authors should provide more insights into this, as it is the foundation of the work. More importantly, this feature is barely utilized in the analysis. What is the detailed influence of having data at different flight altitudes on performance?\n\nW2: The authors realized throughout the process that the \"building\" class is “falsely annotated” for rooftops with other objects on top, as shown in the example in Fig. 5. The proposed solution is to simply ignore the building class: “This problem can be somewhat relaxed by ignoring the Building class both in training and testing, and therefore, each experiment was repeated with that category omitted.” To summarize, the dataset is poorly annotated for a major class in urban scenarios, and the solution to ignore the class is inadequate.\n\nW3: The paper presents an image dataset. The data and concept of the dataset should be visualized by showing the images, the alignment based on drone altitude, and the ground truth in the main paper, rather than only in the supplemental material.\n\nW4: The writing is rough and partly unscientific. For example, phrases like “the nice review” (Introduction, second paragraph, second sentence) are overly informal. The paper would benefit from clearer writing and a more prominent storyline. In particular, the introduction would benefit from a clearer motivation and narrative. For example, the novel aspects of the dataset and the rationale for having data for the same location at different altitudes are hidden in the text and should be more prominent. Additionally, the overview of the paper’s sections is unnecessary in a research paper and could be omitted.   \n\n## Minor Remarks:\n- Including the footnote text in the OpenReview abstract is confusing.\n- Abstract: \"can be used … for semantic segmentation or other applications of interest\" — this is not scientific. Which specific applications can be addressed with the provided data?\n- Sec. 2, first sentence: \"The motivation for building the MESSI was to provide data to train and baseline deep-learning semantic segmentation algorithms\" — should this be \"the MESSI dataset\"? Also, what does it mean to \"baseline deep-learning semantic segmentation algorithms\"?\n- Table 1: Formatting is broken. The word “Coverage” overlaps with the table content.\n- Figure 2: Could be visualized in log scale, with bars colored according to their respective class colors in the annotations (see [A]).\n- Include the number of images in the train, validation, and test splits.\n- Fig. 4: Difficult to interpret. What is shown on the x-axis?\n- Should be \"Cityscapes\" instead of \"CityScapes\" [A].\n- Beginning of page 6: \"each image was divided into 3×3 of 2048 by 1366 overlapped tiles\" — this is not a complete sentence.\n- Fig. 6: Numbers are hard to read due to suboptimal coloring.\n- Sec. 5: \"This paper presents a new dataset called 'MESSI,' designed to serve as a baseline for semantic segmentation algorithms.\" — how can a dataset serve as a baseline?\n- Fig. 7: Appears in the middle of the abstract.\n- References: Inconsistent formatting, e.g., \"Cordts et al. In Proceedings of the IEEE conference on computer vision and pattern recognition\" vs. \"Geiger et al., In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).\" Venue names are inconsistently abbreviated. Additionally, capitalization in paper titles is inconsistent.\n\n[A] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213–3223, 2016.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Multi-Elevation Semantic Segmentation Image (MESSI) dataset, which consists of images captured by a drone flying over four distinct urban scenes at varying altitudes. The study presents the dataset and evaluates multiple state-of-the-art semantic segmentation methods using it.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The work needs to be carefully revised. The authors should address the inconsistent annotations present on the roofs of buildings. Additionally, the presentation, and particularly the writing, needs improvement. This should begin with a clear and well-structured motivation in the introduction section. The authors must also explain in detail how the dataset was created, including how images are aligned across different altitudes. Furthermore, the paper needs to propose a clear methodology for assessing the influence of altitude on the segmentation performance of different methods.", "broader_impact_concerns": "The paper introduces a dataset consisting of images captured by a drone flying over four different urban scenes at varying altitudes. This requires a clearer motivation and an ethical statement, as the dataset could potentially be misused for military research.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H1qD2x9aMY", "reviewer": "Reviewer_9e8q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nS1: MESSI provides images from a drone at a larger variety of altitudes than previous datasets and includes rich metadata about how the data was collected, such as camera intrinsics.\n\nS2: The discussion of related works is extensive and detailed. As someone who is not familiar with these datasets, it gave me insight into exactly how MESSI is novel compared to previous work.\n\nS3: There are extensive experiments utilizing an existing semantic segmentation benchmarking tool. Multiple metrics and results across class weighting schemes are reported along with a detailed investigation of the behavior of the best performing model.\n\nS4: Figure 7 provides a unique insight into the effects of altitude on model performance.\n\n**Weaknesses**\n\nW1: The description of the annotation process lacks detail. Were annotations fully generated by a deep learning model then checked by human annotators? Do the authors have any relationship to V7 that should be disclosed?\n\nW2: Section 3 focuses on horizontal trajectories, lacking a table like Table 1 for the vertical trajectories. Despite this, the vertical trajectories are used for the training set. How many images are in the training set and is that currently listed in the paper?\n\nW3: Is there any concern about data distribution shift due to using vertical trajectories for train and horizontal trajectories for validation and test that should be discussed? Was there a motivation for this shift?\n\nW4: I interpret the last paragraph of Section 4.1 as saying that the Segformer-B3 model used a different training and test set (what happened to validation?) than the other models. What motivated comparing across splits with different model architectures rather than a separate ablation where the same architecture is trained with different splits?\n\nW5: I interpret the second paragraph of Section 4.2 as saying that it is fine for the model to label the shaded part of the roof as a shed, but do not understand why. On a related note, does the dataset contain inaccurate labels? If so, this should be included in Section 3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose MESSI: a dataset composed of drone images of urban environments captured at varying altitudes with semantic segmentation labels. The dataset includes metadata for how and where the data was captured. MESSI captures data from a larger variety of altitudes than previous datasets. The dataset is split into training and validation sets that share a location and a test set composed of two different locations to out-of-distribution generalization. Experiments utilizing MMsegmentation evaluate 7 model architectures across 3 class weighting schemes. Performance across classes is investigated for the best performing model. The effect of altitude variety on model performance is also examined.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "R1: Remove the new lines in the OpenReview abstract. (strengthen the work)\nR2: The text above Table 3 is missing a space before “and 3)”. (strengthen the work)\nR3: Highlight Figure 7 in your third main contribution in the Introduction. (strengthen the work)\nR4: Code for loading the dataset and executing experiments should be shared. (critical for recommendation)\nR5: The model and class weighting scheme used for Figure 7 should be stated. (critical for recommendation)", "broader_impact_concerns": "This dataset is clearly relevant to military interests, but I do not have any immediate ethical concerns with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)