LADI v2: Multi-label Dataset and Classifiers for Low-Altitude Disaster Imagery

## Abstract

Abstract ML-based computer vision models are promising tools for supporting emergency management operations following natural disasters. Arial photographs taken from small manned and unmanned aircraft can be available soon after a disaster and provide valuable information from multiple perspectives for situational awareness and damage assessment applications. However, emergency managers often face challenges finding the most relevant photos among the tens of thousands that may be taken after an incident. While ML-based solutions could enable more effective use of aerial photographs, there is still a lack of training data for imagery of this type from multiple perspectives and for multiple hazard types. To address this, we present the LADI v2 (Low Altitude Disaster Imagery version 2) dataset, a curated set of about 10,000 disaster images captured in the United States by the Civil Air Patrol (CAP) in response to federally-declared emergencies (2015-2023) and annotated for multi-label classification by trained CAP volunteers. We also provide two pretrained baseline classifiers and compare their performance to state-of-the-art vision-language models in multi-label classification. The data and code are released publicly to support the development of computer vision models for emergency management research and applications. † † DISTRIBUTION STATEMENT A. Approved for public release. Distribution is unlimited.
This material is based upon work supported by the Department of the Air Force under Air Force Contract No. FA8702-15-D-0001. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of the Air Force.
© 2024 Massachusetts Institute of Technology.
The software/firmware is provided to you on an As-Is basis
Delivered to the U.S. Government with Unlimited Rights, as defined in DFARS Part 252.227-7013 or 7014 (Feb 2014). Notwithstanding any copyright notice, U.S. Government rights in this work are defined by DFARS 252.227-7013 or DFARS 252.227-7014 as detailed above. Use of this work other than as specifically authorized by the U.S. Government may violate any copyrights that exist in this work.

## 1 Introduction

Rapid and accurate assessment of post-disaster conditions is critical for effective disaster response and recovery operations. Low altitude aerial imagery, such as aerial photographs collected by the Civil Air Patrol (CAP) in the United States or images from small Unmanned Aerial Systems (sUAS), can provide valuable information about the extent and severity of damage caused by disasters. However, the large quantity of images collected during these missions can present a significant challenge for analysts tasked with identifying actionable information in a timely manner. To address this challenge, we introduce the Low Altitude Disaster Imagery v2 (LADI v2) dataset, a multi-label image classification dataset designed to facilitate the development of computer vision models for identifying useful post-disaster aerial images. LADI v2 builds upon existing work in image-based damage assessment [, , , , , ] by providing a diverse, multi-hazard dataset that includes oblique and nadir aerial imagery from various locations and disaster declarations across the United States.

The technical contributions of LADI v2 are twofold. First, we present a curated dataset of post-disaster aerial imagery from multiple perspectives for multiple hazard types, with high quality, operationally-relevant labels annotated by trained CAP volunteers for multi-label classification. Second, we provide two pretrained baseline reference classifiers. We compare them to state-of-the art vision-language models (VLM) on open-vocabulary classification as a benchmark. We outperform the open source VLM on nearly all classes and are competitive with the commercial VLM in test set, and broadly outperform both VLMs in the validation set. This demonstrates the continued need for open, domain-specific training data for specialized applications such as disaster response.

LADI v2 also offers unique characteristics that can contribute to the advancement of machine learning research. The dataset features a realistic distribution shift between the training and test sets, representing annual variation in disaster incident types, as well as changes due to new operational procedures and technologies. This characteristic makes LADI v2 a valuable benchmark for evaluating domain adaptation techniques in the context of disaster response.

The dataset, classifiers, and associated documentation are made openly available on GitHub [] and Hugging Face [, ], enabling researchers and practitioners to build upon this work and adapt the models to their specific needs. Through this contribution, we aim to streamline the process of identifying useful post-disaster aerial images, ultimately supporting more efficient and effective disaster response efforts while providing a valuable resource for the broader machine learning community.

The paper is structured as follows: Section 2 covers related work in disaster imagery datasets and necessary background on vision-language models; Section 3 provides the details of the dataset; Section 4 discusses the pretrained baseline classifiers; and Section 5 concludes with a summary and comments on limitations and future directions.

## 2 Related Work

### 2.1 Natural Disaster Imagery Datasets

There is existing work on using imagery and computer vision to facilitate post disaster damage assessment [, , , , , ]. We highlight a few relevant examples and comment on the gap that LADI v2 addresses. The xBD dataset [] provides $23000$ labeled nadir-perspective satellite images annotated with bounding boxes for building damage for multiple locations across the globe, covering multiple hazard types. FloodNet [] and RescueNet [] each provide segmentation masks, classification labels, and visual question-answering captions for high resolution, low altitude aerial imagery from UAVs, but are limited to single incidents each: Hurricanes Harvey and Michael, respectively. Incidents1M [] provides a large multi-label dataset classifying 43 incident types in 49 outdoor locations from nearly 1 million images scraped from the web. These images feature a variety of perspectives and locations, but are primarily ground-based, and only identify the type and location of the incident. Compared to the existing literature, there is still a lack of training data to support both oblique and nadir low-altitude aerial imagery—which is increasingly common in disaster response applications with UAVs and small manned aircraft—across multiple geographies, event types, and damage and infrastructure categories.

Version 1 of the Low Altitude Disaster Imagery dataset (LADI v1) began to address these gaps by providing annotations for low altitude, multi-perspective imagery from multi-hazard and multi-geographic events []. It was included in a number of NIST TRECVID challenges [, , ]. However, these labels were created by untrained crowdsourced workers, and the term “damage" was not clearly defined. Instead, LADI v2 was labeled by a team of volunteer annotators from the Civil Air Patrol who have been trained in the FEMA damage assessment process. Damage labels follow FEMA’s criteria for preliminary damage assessments (PDAs), which articulate five levels of damage: unaffected, affected, minor damage, major damage, and destroyed. Furthermore, the label set and annotator training materials were developed in conjunction with FEMA’s Response Geospatial Office to ensure compatibility with their procedures. For these reasons, LADI v2 offers a different label set than LADI v1. Nevertheless, we still believe LADI v1 offers some value, since it features a larger number of annotated images, and thus may serve as a suitable pretraining task for classifiers trained on LADI v2.

### 2.2 Vision-Language Models

Recent advances in vision-language models represent extremely promising developments toward generalizable solutions for computer vision problems []. Vision-language models typically include an image encoder and text encoder for each respective data modality. These models are trained on tasks to promote alignment between the image encoding and text encoding; such tasks include contrastive image-text learning, popularized by [], as well as input reconstruction such as masked language modeling [] and masked image modeling []. These vision-language models can be used to address various computer vision tasks, including open vocabulary classification, object detection, and segmentation []. We discuss a few vision language models that are relevant to this paper, particularly CLIP [], LLaVA-NeXT [], and GPT-4o [].

CLIP (Contrastive Language-Image Pretraining) [] is pretrained on 400 million image-text pairs from the internet. The model is given a batch of images and captions, and is trained to pair the associated image to its respective caption. In doing so, the model learns to align the encoded image representation to the respective encoded caption text representation. As a result, images with similar textual descriptions tend to be closer together in the CLIP image embedding space, and dissimilar images are further apart. We use this property to characterize the distribution of our validation and test sets in section 4.2.1.

LLaVA (Large Language and Vision Assistant) [] and its refinements LLaVA 1.5 [] and LLaVA-NeXT [] build upon CLIP by incorporating a large language model decoder, such as Vicuna [] or Mistral [], to allow it to be trained additional tasks, such as visual question answering. Commercial offerings such as GPT-4o [] offer similar multimodal capabilities. We compare our models trained on LADI v2 against zero-shot classification using LLaVA-NeXT and GPT-4o in Section 4.3.

## 3 LADI v2 Dataset

Figure: Figure 1: A sample of images from our training set. LADI v2 contains images with both positive and negative examples of damage from a range of altitudes, perspectives, geographies, and lighting conditions
Refer to caption: extracted/5644187/img/example-ladi-imgs.png

### 3.1 Data collection and annotation

LADI v2 images are sourced from the FEMA (Federal Emergency Management Agency) Civil Air Patrol Image Uploader Repository, publicly hosted in an Amazon Web Services s3 bucket [], which contains aerial photographs collected in support of federally declared disasters from 2009 onward, as well as CAP training missions. Figure 1 shows a sample of images in the dataset. Image metadata includes timestamp and location information. To ensure LADI v2 contains only images collected during disasters, we compared image metadata against disaster declaration data from the OpenFEMA Disaster Declarations API [] to identify images taken within 14 days of the start of a declared federal disaster and within an affected county’s boundaries.

We consulted emergency management professionals at the FEMA Response Geospatial Office to develop the set of labels and labeling instructions. Labels were chosen to help emergency managers identify the most relevant images when conducting initial damage assessments, which can support disaster declarations and assistance grants. The label set is provided as the “v2" set in Table 2(a). The various damage levels for the buildings—affected, minor, major, and destroyed, listed in increasing order of severity—are determined based on the FEMA preliminary damage assessment criteria [].

Images were annotated by a team of 46 Civil Air Patrol volunteers who had been previously trained in the FEMA preliminary damage assessment process []. Each image was shown to three volunteers, and labels were assigned by majority vote when there was disagreement between the annotators.

Figure: (a) The v2 and v2a set of labels.
Refer to caption: extracted/5644187/img/train-ex-per-state.png

After initial experiments with training classifiers on the “v2" label set, we found that there were very few positive examples of “bridges_damage". In addition, we found that performance was poor when distinguishing between various levels of building damage (“buildings_affected", “buildings_minor", “buildings_major", and “buildings_destroyed"). FEMA staff advised that it was less important to classify building damage categories than to determine whether buildings that had sustained any level of damage were present in image. In light of this, we removed the “bridges_damage" class and combined the building damage categories into “buildings_affected_or_greater" and “buildings_minor_or_greater". This revised set of labels is called the “v2a" label set, as shown in Table 2(a), and is what we report our results on. The “v2a" label set contains 12 labels, categorized into 5 non-damage-related (“bridges_any", “buildings_any", “roads_any", “trees_any", and “water_any") and 7 damage-related (“buildings_affected_or_greater", “buildings_minor_or_greater", “debris_any", “flooding_any", “flooding_structures", “roads_damage", and “trees_damage").

### 3.2 Dataset statistics and characteristics

Our dataset consists of 9,963 images, split into 8,030 train examples, 892 validation examples, and 1,041 test examples. The training and validation examples are drawn from disaster declarations between 2015-2022, and the test examples are drawn from disaster declarations in 2023.

*Disaster type distribution.*
Since the training and validation examples are drawn from the same set of disaster incidents, the distributions of hazard types for those two splits are quite similar, whereas the test set has comparatively more images from severe storms, floods, and fires, and many fewer images from hurricanes. The hazard type distributions of the splits are visualized in Figure 2(c).

Figure: (a)
Refer to caption: extracted/5644187/img/label-label-set-2pane.png

*Label distribution and co-occurrence.* Figures 3(a) and 3(b) give the label-label co-occurrence matrices and label-hazard type co-occurrence matrices for the training, validation, and test sets. As the training and validation sets are randomly drawn from the same disaster incidents, they do not have substantially different statistical distributions and are combined in these figures for the sake of brevity. We observe a substantial difference between the training/validation co-occurrence matrices and the test co-occurrence matrices. In the label-hazard type co-occurrence matrix, this is easily explained by the fact that different incident hazard types were more common in the 2023 test set. The label-label matrices require slightly more in-depth explanation.

The label-label matrices indicate that images showing damage occur less frequently in the test set. We believe this is due to a combination of factors. First, CAP has recently expanded their use of the WaldoAir camera system []. This system takes images at regular time intervals in a grid pattern [], whereas images taken from handheld cameras tend to focus on pre-selected targets or areas that the photographer selects, which often are areas with prominent damage. As a consequence, a much lower percentage of the WaldoAir images contain damage. The train and validation sets consist of about 50% Waldo images, while the test set consists of 65% Waldo images.

*Distribution shift.* The distribution shifts between the train/validation and test sets represent challenges in disaster applying for machine learning. The distribution of hazard types, severity, and locations change year to year based on cyclical weather patterns [] and climate change []. Furthermore, changes in operational procedures and technology, such as the increased adoption of the WaldoAir system, can lead to distribution variation even among disasters resulting from the same type of hazard. This underscores one of the key domain-specific challenges of applying machine learning solutions for disaster response applications. To our knowledge, no other benchmark disaster imagery dataset explicitly addresses the shift in data distribution from year to year.

## 4 Pretrained Classifiers

### 4.1 Architecture and training details

To support research and deployment applications, we provide two pretrained reference classifiers, LADI-v2-classifier-small-reference and LADI-v2-classifier-large-reference,(^1^11We provide four classifiers in the repository. The “reference” versions, discussed in his paper, are trained only on the train set. The “main” versions are trained on all splits and intended for deployment and downstream applications.) hereby referred to as the “small" and “large" classifiers for brevity. The small classifier is based on the Big-Transfer (BiT-50) architecture [], pretrained on ImageNet-1k [], while the large classifier is based on Swin v2 Large [], pretrained on ImageNet-21k [] and finetuned on ImageNet-1k []. Standard random augmentations of resizing, cropping, horizontal flipping, affine transformations, and color jitter were applied to the training images.

These architectures were the best performers among a number of available models. We first finetuned twenty five pretrained classifiers, available on Hugging Face, on LADI v2 using default settings. We chose the top two performing candidates and performed additional optimization through hyperparameter tuning and pretraining on LADI v1 []. Hyperparameter tuning was done via random search on optimizer type, learning rate, and learning rate scheduler. We considered the AdamW [] and Adafactor [] optimizers, initial learning rates of $2\times 10^{-4}$, $1\times 10^{-4}$, and $5\times 10^{-5}$, and LR schedulers with exponential LR decay ($\gamma$: 0.5 or 0.9) or with reduce LR on plateau ($\gamma$: 0.5 or 0.9, patience: 5 or 10 epochs). Results for the top two architectures are shown in ablation in Table 2; we evaluate using mean Average Precision (mAP). Hyperparameter tuning provides significant benefit for both architectures. Pretraining on LADI v1 provides modest benefits in the test mAP for the model based on Swin v2, but degrades performance for the model based on BiT-50. The final configurations for LADI-v2-classifier-small and LADI-v2-classifier-large are indicated in Table 2 with ${}^{\text{\textdagger}}$ and ${}^{\text{\textdaggerdbl}}$ respectively.

**Table 1: Selected hyperparameters for our reference small and large models.**
| Name | Architecture | LR Scheduler | Optimizer | Initial LR | $\gamma$ | Epochs |
| --- | --- | --- | --- | --- | --- | --- |
| Large | Swin v2 | Exponential | AdamW | $5*10^{-5}$ | 0.9 | 50 |
| Small | BiT-50 | Exponential | AdamW | $1*10^{-4}$ | 0.9 | 50 |

**Table 2: Classifier performance ablation with hyperparameter tuning and LADI v1 pretraining. The small model configuration is indicated with ${}^{\text{\textdagger}}$ and the large model with ${}^{\text{\textdaggerdbl}}$.**
| Architecture | Tuning | v1 Pretrain | Validation mAP | Test mAP |
| --- | --- | --- | --- | --- |
| BiT-50 | No | No | 89.6 | 89.0 |
| Swin v2 | No | No | 88.7 | 86.0 |
| ${}^{\text{\textdagger}}$BiT-50 | Yes | No | 93.3 | 91.1 |
| Swin v2 | Yes | No | 93.8 | 92.3 |
| BiT-50 | Yes | Yes | 87.9 | 85.9 |
| ${}^{\text{\textdaggerdbl}}$Swin v2 | Yes | Yes | 93.8 | 92.6 |

### 4.2 Performance Analysis

For the sake of brevity, we report the results only from the “large" model.

*Incident hazard type analysis.* The performance between the test and validation sets is comparable for most hazard types, except for fire (see Figure 4(a)). The difference in performance for fire incidents is likely due to the relative lack of fire data in the training and validation sets compared to the test set.

*Geographic analysis.*
The classifier performs robustly across various geographies. Figure 4(b) shows the mAP of the classifier for states in the test set, which are those states in which CAP collected images for disasters in 2023. While only 10 states are represented in the test set, the classifier achieved an mAP between 80 and 96 for all of them. The state of Hawaii, with the lowest mAP at 80, had only one CAP mission in the test set, the August 2023 Hawaii wildfires. Relatively low performance is likely due to the relative lack of fire events in the training data, as well as potential differences due to geography. We thus caution practitioners and researchers against using models trained on LADI v2 for applications not well represented in the training set and recommend supplemental data collection and training.

Figure: (a) mAP by event type for validation and test sets.
Refer to caption: extracted/5644187/img/mAP-set-inc.png

#### 4.2.1 Semantic Similarity Analysis

In addition to analyzing by incident hazard type and geography, we attempt to quantify “out-of-sample-ness" by using distance in CLIP space []. CLIP embeddings align images with similar textual descriptions, such that images with semantically similar content will be nearby in CLIP space even if they are not necessarily visually similar in their pixel representations. In this way, we can use distance in CLIP space as a proxy for semantic similarity between two images, where similar images are closer in CLIP space. We use the Euclidean distance between normalized vectors as the distance metric, $d(\theta)=\sqrt{2(1-\cos{\theta})}$, where $\theta$ is the angle between the two vectors.
For each image in the validation and test sets, we compute the CLIP distance between it and its nearest neighbor in the training set. We also compute the $L^{1}$ norm of the error vectors (the difference between the post-sigmoid/pre-threshold prediction and ground-truth vectors) for each image in the validation and test sets.

We visualize the joint distribution of the $L^{1}$ error norm and distance to nearest training point for each image in the validation (blue) and test (orange) set in Figure 5. Kernel density estimates of the marginal distributions are visualized along the top and right hand axes. We can see that the test set is on average further away in CLIP space and has larger error norms. There appears to be a positive relationship between the distance to the nearest training example and the average error norm, as well as the variance in the distribution of the error norm. This approach could be used characterize how out-of-sample a given set of images is, as well as estimate the potential expected degradation of performance associated with that distribution shift.

Figure: Figure 5: Error vector $L^{1}$ norm vs. the distance from a point in an evaluation set to its nearest neighbor in the train set in CLIP space. Validation data is plotted in blue and test data in orange.
Refer to caption: extracted/5644187/img/errorCLIPdist.png

### 4.3 Comparison with open-vocabulary classification (LLaVA and GPT-4o)

To demonstrate the need for open datasets and classifiers for post-disaster imagery, we compare our model to a number of recent vision-language models. We first compare against the recently released open 7.5 billion parameter LLaVA-NeXT model [] on a zero-shot classification task. For each class, the LLaVA model saw an image in the test or validation set, followed by a prompt such as, “Does this image contain class_name? Answer with ‘yes’ or ‘no.’" For the classes involving FEMA preliminary damage assessment categories, we included a summary of the damage category criteria based on the FEMA Preliminary Damage Assessment Pocket Guide [] in the prompt. The model outputs were converted to binary labels and used to compute the $F_{1}$ score and shown in Table 3. In the test set, our method outperformed LLaVA-NeXT on all labels, including all damage-related classes, except in three categories: “bridges_any", “roads_any", and “trees_any", in which our model comes within 3% of LLaVA-NeXT. In the validation set, our model outperforms LLaVA-NeXT in all categories.

We also compare our model to a commercial multimodal model, GPT-4o, using the same prompt format as above. The results are shown in Table 3. On the test set, our model is competitive with GPT-4o, beating or tying its performance on 5 of the 12 classes. On the validation set, we outperform GPT-4o in all but the “water_any" class, showing strong in-sample performance. However, it is notable that the performance of GPT-4o is relatively consistent between validation and test set.

Compared to the open source LLaVA-NeXT, GPT-4o broadly outperforms. We believe that this indicates a gap in openly available labeled disaster-related aerial imagery. Though GPT-4o clearly highlights the potential of multimodal vision-language models for computer vision tasks—particularly in its demonstrated performance across the different distributions of the validation and test sets—its closed-source nature and restrictive licensing makes it difficult to build derivative works. GPT-4o also suffers from practical challenges for operational use. It is only accessible over the internet via an API, and we found that the API frequently timed out when trying to download large high-resolution images, such as those captured by CAP. This required the images to be downloaded, resized, re-encoded, and uploaded in small batches. As such, it is not suitable for large-scale or time-critical tasks, nor for usage in offline environments. In comparison, our pretrained classifiers run much quicker than both VLMs, and do not require an internet connection like GPT-4o.

**Table 3: Comparison of $F_{1}$ scores of our method, LLaVA-NeXT, and GPT-4o on the test and validation sets across classes.**
|  | Test | Validation |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Class | Ours | GPT-4o | LLaVA | Ours | GPT-4o | LLaVA |
| bridges_any | 0.53 | 0.52 | 0.56 | 0.65 | 0.59 | 0.44 |
| buildings_any | 0.94 | 0.94 | 0.88 | 0.96 | 0.93 | 0.94 |
| buildings_affected_or_greater | 0.50 | 0.66 | 0.45 | 0.76 | 0.74 | 0.56 |
| buildings_minor_or_greater | 0.59 | 0.55 | 0.21 | 0.68 | 0.50 | 0.38 |
| debris_any | 0.46 | 0.40 | 0.40 | 0.66 | 0.55 | 0.46 |
| flooding_any | 0.53 | 0.61 | 0.35 | 0.79 | 0.73 | 0.50 |
| flooding_structures | 0.43 | 0.39 | 0.11 | 0.78 | 0.72 | 0.28 |
| roads_any | 0.90 | 0.92 | 0.92 | 0.94 | 0.90 | 0.92 |
| roads_damage | 0.17 | 0.18 | 0.07 | 0.44 | 0.18 | 0.18 |
| trees_any | 0.93 | 0.97 | 0.95 | 0.96 | 0.96 | 0.96 |
| trees_damage | 0.45 | 0.50 | 0.26 | 0.75 | 0.54 | 0.55 |
| water_any | 0.79 | 0.84 | 0.82 | 0.91 | 0.93 | 0.89 |

## 5 Conclusion

*Summary of Contributions.* In this paper, we introduce the LADI v2 dataset, a curated collection of post-disaster aerial images from multiple perspectives, hazard types, and geographies across the United States. The dataset addresses the need for high-quality, diverse training data to support the development of computer vision models for disaster response applications. To facilitate research and implementation efforts, we provide the dataset and two pretrained reference classifiers as open-source resources.

One of the key strengths of LADI v2 is the quality of its annotations, which were provided by trained Civil Air Patrol volunteers using label sets and training materials developed in collaboration with FEMA. This ensures that the labels are consistent with the standards used by emergency management professionals, enhancing the practical utility of the dataset.

Furthermore, LADI v2 features a realistic distribution shift between the training, validation, and test splits, capturing the year-to-year variability in disaster events as well as changes in operational procedures and technology, such as the increased adoption of the WaldoAir system by the Civil Air Patrol. This characteristic makes the dataset a valuable benchmark for evaluating domain adaptation techniques in the context of disaster response.

The pretrained classifiers demonstrate strong performance on the LADI v2 test set, outperform state of the art open source open-vocabulary classification from LLaVA-NeXT, and are competitive with commercial offerings such as GPT-4o on the most relevant damage classes. This comparison highlights the value of open source domain-specific training data for specialized applications and underscores the potential impact of the LADI v2 dataset on the broader machine learning community.

*Limitations and potential improvements.* While LADI v2 represents a step forward in the availability of high-quality annotated disaster imagery datasets, there are some limitations to consider. First, certain hazard types and geographies are under-represented due to multiple factors, including the likelihood of those hazard types occurring, the likelihood that a federal disaster declaration is issued, and whether CAP is tasked to collect images. For example, imagery from California is comparatively underrepresented in the dataset relative to the state’s size, population, and exposure to hazards because its state-level emergency management agency is relatively well-equipped, reducing the likelihood of FEMA-supported CAP missions in the area.

Second, LADI v2 is specific to the United States, which means that the architecture, biomes, disaster types, and infrastructure represented in the dataset is not representative of the rest of the world. Researchers and practitioners working on disaster response applications outside of the US should augment the dataset with additional imagery from the application domain.

Finally, LADI v2 currently supports only multi-label classification tasks. Applications requiring finer-grained localization or segmentation may require additional effort to adapt the dataset or models to their specific needs.

Despite these limitations, LADI v2 offers a valuable resource for the machine learning community and has the potential to support the development of more effective and efficient disaster response tools. Future work could expand the dataset to include a wider range of geographies and disaster types, as well as provide additional annotation classes to support other vision tasks.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Questions:
[Q1] What is the GSD of the imagery? If the GSD is small, has care been taken to ensure the residents' privacy?
soundness: 3
strengths: [S1] Expert involvement during development: The construction of the dataset involves heavy expert involvement (from annotating images to deciding classes), ensuring the practicality of the benchmark. 
[S2] Focus on low-altitude imagery: As described in the related work, prior datasets on low-altitude imagery are limiting 
[S3] Extensive analysis: The domain gap between val and test is large (table 4) enough that it could be an interesting machine learning challenge to figure out how to bridge the domain gap.
summary: This paper introduces a new multi-label image classification benchmark for natural disasters. During the benchmark construction, the authors extensively involved experts to ensure the practicality of the dataset. Extensive analyses are conducted to show the usefulness of the dataset.
weaknesses: [W1] Open-vocabulary classification: The authors make a comparison to several SOTA generalist VLMs. However, since this is a highly specialized domain, it would be useful to compare to VLMs that are developed for remote sensing [1, 2]


[1] Mall, Utkarsh, Cheng Perng Phoo, Meilin Kelsey Liu, Carl Vondrick, Bharath Hariharan, and Kavita Bala. "Remote sensing vision-language foundation models without annotations via ground remote alignment." arXiv preprint arXiv:2312.06960 (2023).

[2] Liu, Fan, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. "Remoteclip: A vision language foundation model for remote sensing." IEEE Transactions on Geoscience and Remote Sensing (2024).

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: In the co-occurrence matrix, it is hard to understand what these numbers are a percentage of. neither the rows, not the column, nor the whole matrix sums up to 100. How the normalization is done and what these numbers mean should be explained in the paper. 

Minor suggestions:
In Figure 2a some of the text is covered by Figure 2b. This should be fixed.
The maps in general are hard to see/understand/sometimes misleading:
   * In Fig 2b the states not considered can be grayed out like 4b. This makes it seem like all states have some images which might not be true.
   * Hawaii can be cropped and zoomed in on more.
   * Puerto Rico can also be placed in an inset and zoomed in more.
   * The figures need improvement. Many of the fonts are hard to read/understand.


The quotations look weird in many places such as line 100: in latex use '' and `` for left and right quotes.
soundness: 2
strengths: The dataset has the potential to be very relevant for the use of disaster reporting. It covers diverse types of events, object types, time period, and locations.
summary: The paper proposes a new dataset with around 10k images of disaster events with multi-label classes. The paper improves on a past dataset by collecting labels from experts in the area called civil air patrollers. The dataset is divided into a training and testing set based on the year of acquisition. Two different sets of models are tested on the dataset one that is fine-tuned to perform classification as well as zero-shot multimodal LLMS.
weaknesses: Not a challenging benchmark:
   * The numbers in the results suggest that the benchmark is not very challenging. Most simple fine-tuning methods can reach up to 90% mAP on the test set. Even zero-shot methods can perform well. This suggests that while the benchmark might be useful for the disaster reporting community, it is not very useful for ICLR, since all the proposed methods have already saturated the performance. Some ways to make the dataset more challenging could be adding extreme disaster events, or adding other tasks such as few-shot learning.
   * The dataset only contains classification labels which are fairly well explored in the learning community. Other types of labels such as detection, segmentation, or captioning would have created a more challenging benchmark for the community. However, the good performance of zero-shot and fine-tuned models suggests the benchmark is not very challenging.

Potential improvements in benchmarking:
   * The number of examples in the test/training set while around 1k/10k is not huge. Therefore, it might be better to train models multiple times and provide an error bound around the numbers. With 1k images in the test set, there is a possibility that the errors are huge.
   * It is not clear how much time and effort it takes for CAPs to label the entire dataset. When creating a new dataset this cost should be mentioned, so that future researchers have an estimate of cost to them in case of collecting more data.
   * A difference between LADI v1 and v2 is that v1 was collected by a layperson while v2 was collected by experts. Expert labels might matter for domains where images are not well represented on the internet. The experiments with GPT and LLAVA suggest that these images are well-represented. Secondly, costwise collecting expert labels is typically more expensive than a layperson. Therefore, it might be useful to do the same cost analysis, i.e. with a larger set of data collected with layperson labels, to truly show the effectiveness of this method.
    * More qualitative examples should be shown. For example, Figure 1 can be labeled with the true labels. Qualitative examples can be shown where certain models fail and others do not.
    * Some scaling law analysis is also needed: How many images are needed to be collected to get to a good performance? is it less than 10k or a lot more?

Many arguments in the paper are presented but not confidently resolved:
   * For example, the reason for the distribution shift (line 275) is attributed to both a change in technology (waldoair) as well and changes in event types. While such distribution shifts should be present in challenging benchmarks, the shifts should be studied in more detail. For example, one way to study this would could be to create a test set from events of some other year, when Waldoair was not introduced.

   * Line 456, suggest a significant performance difference between test and val set for ZS methods. However, it is not directly evident from the table. To me it seems like ZS methods indeed generalize better to the test set compared to the fine-tuning methods.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Given the performance of GPT-4o, could LADI v1 be improved (and combined with LADI v2) for improved results?
soundness: 2
strengths: Rapid response to disasters is an important area where ML can contribute, so having a sound dataset for model development and training is valuable to the community.
summary: The authors propose a low-altitude dataset of ~10,000 post-disaster imagery.  They also pretrain two baseline classifiers.
weaknesses: The exact goal that the dataset is enabling is a bit unclear.  The authors state that when a mission is flown, there are a lot of images, but not all of them have relevant (i.e. damage) in them.  However, what that enables (and therefore what type of data is needed and what performance is required) is not clear- are they trying to find a specific building to visit on the ground (in that case isn't more precision needed), are they trying to estimate the magnitude of the damage (in that case doesn't the imagery need to be de-dupped)? 

The authors mention that these annotators were more experienced than those for LADIv1, but additional information around training and (dis)agreement between annotators is necessary to speak to the quality of the annotations.

Some kind of analysis showing the performance out of distribution is important given that this data is limited to certain areas.  Showing the generalization capabilities is important for establishing how it could be used in a new setting.  

GPT-4o, with no knowledge of this dataset, does a reasonably good job, if not better.  Therefore the contribution from the models is somewhat limited.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What are the mAP scores for the VLMs? 
2. Why are results for the 25 trained models mentioned in the paper not included? 
3. What are the properties of this dataset that are not covered by other related datasets? 
4. Why was the validation set IID instead of temporal? Why not use a temporal split like for the test set? 
5. What are the use cases for multi-label compared to finer annotations like segmentation? Is one a definitively better cost-benefit proposition for the end users? 
6. Can you specify the result in Beyer (2022) that supports the pretraining observations (L338). It wasn't obvious on a quick skim, but I may have missed it.
soundness: 3
strengths: * Important problem. Making post-disaster imagery rapidly searchable sounds like an excellent use case for computer vision. 
* Good labeling protocol. The labels are provided by an end-user (FEMA / Civil Air Patrol) who provides their annotators with standardized training. Multiple labelers review each image.
* The temporal split fairly models the real use-case: training a model with data up to time $T$ and applying it for $t > T$. 
* Comparison between supervised models and vision-language (GPT-4o and LLaVA) models are interesting (vision-language models like GPT-4o achieve reasonable but generally worse performance). 
* Clear hyperparameter tuning procedures. 
* Related work seems to be covered well, though I am not deeply familiar with the area and will defer to more knowledgeable reviewers on this point.
summary: This work introduces a dataset of post-disaster aerial images with multi-label annotations (12 categories, e.g. flooding, damaged bridges, destroyed buildings). The images vary in viewing angle, geographical location, and disaster type. The intended use of this data is to train computer vision models to rapidly filter post-disaster imagery for actionable information. The paper also evaluates two transformer models as baselines for the multi-label classification task. Two publicly available vision-language models (LLaVa, GPT-4o) are also evaluated. 

Overall, this seems like a nice dataset for an important problem. With some additional work, I think it could be a solid contribution. In its current form, the technical contributions are modest.
weaknesses: The machine learning content of this work is modest. While there are some interesting experiments (e.g. the vision-language model comparisons), the benchmarking consists of only two supervised models and two off-the-shelf VLMs. Further, mAP values are not reported for the VLMs so one cannot compare them to the supervised models directly on the primary benchmark metric. Even then, only one of the two supervised models is evaluated in the latter half of the paper. 

The authors mention that 25 models were trained, but those results are not provided. This is unfortunate, since a comprehensive analysis of those models would significantly strengthen the paper if it provided technical insights into distribution shift, hyperparameter sensitivity, VLM strengths/weaknesses, etc. 

Section 4.2 claims to "assess the marginal benefit of using trained annotators over crowdsourced workers" - however, there seem to be many confounding differences between the two datasets (not least of which is that the two datasets consist of almost entirely different images). How could it be possible to attribute differences in model performance to the label quality?