Scale-aware Recognition in Satellite Images under Resource Constraints

## Abstract

Abstract Recognition of features in satellite imagery (forests, swimming pools, etc.) depends strongly on the spatial scale of the concept and therefore the resolution of the images. This poses two challenges:
Which resolution is best suited for recognizing a given concept, and where and when should the costlier higher-resolution (HR) imagery be acquired?
We present a novel scheme to address these challenges by introducing three components: (1) A technique to distill knowledge from models trained on HR imagery to recognition models that operate on imagery of lower resolution (LR), (2) a sampling strategy for HR imagery based on model disagreement, and (3) an LLM-based approach for inferring concept “scale". With these components we present a system to efficiently perform scale-aware recognition in satellite imagery, improving accuracy over single-scale inference while following budget constraints. Our novel approach offers up to a 26.3% improvement over entirely HR baselines, using 76.3 % fewer HR images.

## 1 Introduction

Figure: Figure 2: System overview. First, we determine which resolution is best suited for the search concept based on its scale (sec. 3.3). Then, we analyze the search area to find which regions would benefit the most from higher resolution inference (sec. 3.5). We sample the best suited regions while staying within a user specified budget. Based on this guidance we perform inference using one of three models, a high resolution satellite model, a low resolution satellite model, and a low resolution satellite model with knowledge distilled from its high resolution counterpart (sec. 3.4). This knowledge distilled model allows us to infer finer details using low resolution satellite imagery alone.
Refer to caption: extracted/5970003/images/new_Directed_Pipeline.png

The ever-increasing number of earth observation satellites (now more than 1500) offer us a unique opportunity to understand changes at the planetary scale, be it tracking the ecological degradation of forests and coral reefs [, ], the destruction of cultural heritage [], or economic development and well-being [].
A key to all these downstream applications is the ability to recognize accurately a broad vocabulary of concepts: a major computer vision challenge.

An important aspect of recognition in satellite imagery is the notion of *scale*:
Satellite image resolution is dependent on the particular satellite/sensor and is characterized by the GSD, or ground sampling distance: the distance corresponding to a single pixel.
For example, Sentinel-2 images are fairly low resolution (1 pixel = 100 m^2) [], while images captured by the NAIP program are fairly high resolution (1 pixel = 1 m^2) [].
The image scale matters because many concepts of interest also have a characteristic physical size/scale.
An olympic-sized swimming pool, typically about 50 m long, would be barely a speck in Sentinel-2 images.
In contrast, a lake may be hundreds of square kilometers, necessitating an exceptionally large image to capture its extent in terms of NAIP imagery, which current models would not be equipped to do.

The right resolution may also depend on the underlying geography.
Dense urban areas with many geographical features may require high resolution data to analyze.
In contrast, large swathes of uninhabited deserts can probably be analyzed accurately even with low-resolution imagery.

Accuracy notwithstanding, we must also consider cost and availability.
Low-resolution (LR) imagery is free, abundant, and covers the planet densely over space and time [, ].
In contrast, high resolution (HR) imagery typically comes from drones or low flying satellites that are commissioned as needed, and therefore come with a high cost and inconsistent temporal and spatial coverage [, ].

Scientists in many application domains, be it ecology, archaeology, and urban planning [, , , ], today manually reason about these different aspects, namely, the scale of the concept, the nature of the underlying geography, and the cost of acquired imagery, to achieve the best recognition that they can afford with their budget.
We need an approach that can automate this tradeoff and automatically identify when to acquire more expensive high resolution imagery, taking into account the cost, the scale of the concept and the geography.
While there is past work on the impact of scale on satellite image recognition, much of it is focused on accuracy and disregards costs [, , ].
Prior work that has looked at costs focuses on the geography but ignores the scale of the concept itself [].
A holistic treatment of scale to achieve accurate recognition under a budget is missing.

In this paper, we address this gap by proposing:

- •
A knowledge distillation technique which allows LR models to improve significantly in recognizing finer concepts by learning from HR.
- •
A novel approach that leverages the semantic understanding of LLMs to determine the scale of each concept.
- •
A new approach for determining which geographical regions require higher resolution analysis by predicting when low and high resolution models might disagree, and
- •
A unified framework that combines these ideas to yield the most accurate retrieval results for a range of concepts while adhering to a strict budget constraint.

Finally, we note that while our experiments are on LR and HR satellite imagery, these techniques are more general and can generalize to any problem where there are two or more modalities with different costs and accuracy tradeoffs.

We evaluate our approach with multiple recognition models (supervised and open-vocabulary) and multiple satellite modalities.
We find that compared to simply using high resolution images always, our framework *improves* accuracy up to 13 points while reducing the number of HR images used by 5$\times$.
Our approach also significantly outperforms (by more than 25 points) other prior work that trades off between accuracy and cost. In sum, our results demonstrate that our holistic reasoning of scale leads to significantly higher accuracy with large reductions in cost.

## 2 Related Works

### 2.1 Multi-scale Recognition in Satellite Imagery

Remotely sensed images or satellite images are inherently multi-scale.
Depending on the sensor heights and properties, the physical distance between two adjacent pixels in remotely sensed images (known as Ground Sample Distance/GSD) could vary from 0.3m to 1km.
These multi-scale images provide complementary information for various applications [, , , ].

Few prior works have explicitly investigated the effect of different scales in visual recognition in satellite imagery. For instance, [] shows that it is crucial to build scale-specific representations for images with different GSD. However, they do not explicitly address the high acquisition cost of high-resolution satellite imagery. Works on image super-resolution [, , ] seek to sidestep the problem of acquisition cost by upsampling low-resolution to their high-resolution counterparts. But as shown in [], these models are prone to hallucination and would produce high-resolution imagery with limited fidelity. In this work, we consider a more realistic setup where we allow a small budget for acquiring high-resolution images during inference. This allows us to tap into the stronger performance of scale-aware models such as [, , ] while avoiding the hallucination problem in image super-resolution.

### 2.2 Resource Constraints in Visual Recognition

Various resource constraints could be considered during the development or deployment of visual recognition models.
Active learning approaches [] consider a budget for acquiring annotations during development and seek to develop the best-performing models by selecting the most informative set of training examples to annotate.
This could be done by selecting examples that the model is most uncertain about [, , ], examples that are most representative of the unlabeled data [, ], or a combination of both [].
Prior works have also considered compute constraints for developing and deploying visual recognition models.
These works satisfy the compute constraints either through developing specialized model architectures [] or compressing/pruning existing models [, ].
Our work considers a different type of constraint inherent to remote sensing, i.e., the acquisition cost for high-resolution images. While others have looked into optimizing high-resolution image acquisition [, ], we provide a solution to produce the performant visual recognition models under a fixed acquisition budget with a semantic understanding of the concept. To the best of our knowledge, this is an uncharted problem domain.

### 2.3 Knowledge Distillation/Learning with Privileged Information

To recognize fine-grained concepts in LR images, we construct a query model by distilling the logits generated by another model that is trained on HR imagery.
This approach bears a resemblance to knowledge distillation [, , , , , ], which seeks to compress a large-scale teacher model by training a compute-efficient student model to mimic the output of the teacher model.
Different from conventional knowledge distillation, our query student model and teacher model operate on different inputs. Specifically, our query model operates on LR imagery, whereas our teacher model operates on HR imagery.
Our approach can also be viewed as an instance of learning with privileged information [, , ], in which additional information that is available during test time (e.g., high-resolution satellite imagery) is used for training a model.
This paradigm has enabled the development of better-performing models in various applications, such as autonomous driving [], image super-resolution [], and so on.
In remote sensing, prior works [, ] have attempted to use additional sensor data as privileged information for improving land-cover classification.
Our work leverages high-resolution satellite image data as privileged information for improving recognition in low-resolution imagery: a different scenario.

## 3 Methodology

### 3.1 Problem Setup

We are interested in defining a framework for optimizing accuracy/cost tradeoffs when performing recognition in satellite imagery.
This framework should reason about not only the scale of the concept being recognized but also the underlying geography of each location.

We concretize the problem as follows.

##### Setup.

We frame our problem as a *retrieval task*: given a concept $c$ (e.g., “tennis courts”) and a large set of locations $\mathcal{I}$ (e.g., the state of New York), we wish to identify which of these locations have the concept.

For each location, we have one LR image and a set of HR images that tile the same area. The size of this set, $K$, depends on the difference in GSD between the two modalities. For example, if LR images come from Sentinel-2 (GSD=10m) and HR images come from NAIP (GSD=1m), $K$ would be 100.

We assume that we have special purpose multi-label classification models for each resolution, denoted by $M^{HR}$ and $M^{LR}$ respectively.
Given any concept $c$ and any location $i$, we can use these models to score the presence of this concept at that location.
For LR imagery, this is straightforward:

$$ $\displaystyle s_{c}^{LR}(i)=[M^{LR}(I_{i}^{LR})]_{c}$ (1) $$

For HR imagery, we simply take the maximum over model scores for all the images at that location:

$$ $\displaystyle s_{c}^{HR}(i)=\max_{j=1}^{K}[M^{HR}(I_{ij}^{HR})]_{c}$ (2) $$

The max here is used because we want $s_{c}^{HR}(i)$ to be high (denoting the presence of the concept) if *any* of the $K$ HR images indicate the presence of the concept.
We call these scores LR scores and HR scores respectively.

As discussed in the introduction, HR imagery is expensive. We assume a budget in terms of the maximum number of HR images we can acquire, $B$.
Our goal is, given a concept $c$ and a set of locations $\mathcal{I}$, retrieve all locations $i\in\mathcal{I}$ that have concept $c$ while staying within the HR budget.

##### Training / hyperparameter validation.

To allow for algorithms to train any models or validate hyperparameters, we assume access to a fixed set of locations $\mathcal{I}^{train}$ where we have already acquired (a) images from both resolutions, and (b) annotations for the presence/absence of a set of “seen" concepts $\mathcal{C}^{seen}$.
For fairness, these locations are distinct from the locations used for evaluation.
Our set of evaluation concepts are also kept distinct (“unseen" classes); however, we also report performance on the seen classes.
A full list of the concepts, seen and unseen, is in Appendix 6.1.

### 3.2 System Overview

Our overall system to tackle this problem is shown in Figure 2 and has three steps:
Given a search concept, our system uses an LLM-based approach to predict which resolution it is best suited towards (Section 3.4).
If the search concept was found to be better suited to HR imagery, we would ask the user for a budget and utilize our Model Disagreement based sampling technique to optimize the budget in selecting locations for HR imagery (Section 3.5). We describe how this technique can be performed using LR imagery alone via a Knowledge-Distilled LR model and how such a model allows us to perform improved inference with LR imagery alone as well (Section 3.4).

### 3.3 Inferring concept “scale" using LLMs

The question of whether HR imagery is needed or not depends on the concept.
For some concepts, the low-resolution scores $s^{LR}$ are better, while for others, the HR scores $s^{HR}$ are more accurate.
This is, unfortunately, difficult to predict without some background knowledge about the concept.
To capture this background knowledge, we propose to leverage large language models (LLMs).

Unfortunately, while LLMs have generally learned various kinds of background knowledge, it is unlikely that they have seen or reasoned about the different modalities of satellite imagery.
As such, while the LLM might know that a lake is bigger than a swimming pool, this may not be enough to figure out which of the two modalities is better.

We address this knowledge gap with *in-context learning* [].
Concretely, we first use the available training locations $\mathcal{I}^{train}$ and the annotations therein to evaluate both the HR scores and the LR scores for each concept in $\mathcal{C}^{seen}$.
Thus, for each of these seen concepts, we know which modality yields higher accuracy.
We then give these seen concepts, as well as the corresponding modality, as in-context examples to the LLM and ask the LLM to infer the right modality for all other (unseen) concepts. The prompts to the LLM can be found in sec. 6.2 in the appendix.

### 3.4 Knowledge Distillation from HR to LR

Suppose we know apriori that a concept is better detected in HR imagery. Given the limited acquisition budget of $B$, we do not have full coverage of the locations in $\mathcal{I}$. Thus we need a model that consumes LR images and approximates the predictions made by $M^{HR}$. To build such a model $M^{LR}_{KD}$, we minimize the MSE between the model’s prediction and the corresponding HR scores:

$$ $\displaystyle\sum_{i}\sum_{c\in\mathcal{C}^{seen}}||[M^{LR}_{KD}(I_{i}^{LR})]_ {c}-s_{c}^{HR}(i)||_{2}^{2}$ (3) $$

By minimizing the loss, we essentially distill the HR knowledge [] in the $M^{HR}$ into a model that consumes LR images $M^{LR}_{KD}$.
This formulation also bears a resemblance to learning with privileged information []. In this case,
when training the model $M^{LR}_{KD}$, we use privileged HR information to guide the $M^{LR}_{KD}$ to detect smaller concepts at LR imagery.

### 3.5 Acquiring HR imagery based on Model Disagreement

When detecting a small concept, $M^{LR}_{KD}$ might not be perfect since an LR image at the test location $l$, $I_{l}$ might not have enough visual signatures.
In this case, we would have to leverage our budget of $B$ to acquire HR imagery and run predictions using $M^{HR}$.
A simple criterion to select a location $l$ to acquire HR imagery is by looking at the disagreement between the LR scores and HR scores. The disagreement at $l$ can thus be defined as:

$$ $\displaystyle\delta(l)=\sum_{{c\in\mathcal{C}^{seen}}}|s_{c}^{HR}(l)-s_{c}^{LR }(l)|$ (4) $$

A location $l$ with higher $\delta(l)$ is more likely to have concepts that $M^{HR}$ could detect where $M^{LR}$ could not (or vice versa).
However, to compute $\delta$ we would have to already possess HR imagery over the whole region, which is exactly what we want to avoid acquiring. Instead to approximate $\delta$, we leverage $M^{LR}_{KD}$ as a stand in for $M^{HR}$ and compute the following disagreement criterion for ranking various locations:

$$ $\displaystyle\delta^{\prime}(k)=\sum_{{c\in\mathcal{C}^{seen}}}|[M^{LR}_{KD}(I _{k}^{LR})]_{c}-s_{c}^{LR}(k)|$ (5) $$

### 3.6 Scale-Aware Recognition under Resource Constraints

Given a concept $c$, our system uses an LLM-based approach to predict which modality would yield better accuracy irrespective of cost (i.e, whether HR score, $s_{c}^{HR}$, or LR score, $s_{c}^{LR}$ , is better) (Section 3.3). For a concept best suited to LR imagery, no HR imagery is acquired and inference is carried out using the LR model. For concepts for which HR scores are predicted to be better, our system further decides which locations to acquire HR imagery for. We prompt the user for a budget for HR imagery and calculate our disagreement criterion as described in eq. 5. With this criterion and the budget, we select those locations with the maximum disagreement for HR imagery. We then perform inference with the HR model on the acquired HR images. All other locations are scored using LR imagery and the improved KD model, $M^{LR}_{KD}$.
The end result is accurate retrieval where HR imagery is used sparingly, only when it is really needed.

## 4 Experiments and Results

We evaluate the effectiveness of our scale-aware concept recognition approach by assessing the impact of knowledge distillation, model-disagreement-based acquisition, and LLM-based scale inference. We first evaluate our whole system then the three components individually.

### 4.1 Experimental Setup

#### 4.1.1 Pre-trained Models

We perform our experiments with two sets of low-resolution and high-resolution models.

- •
GRAFT is a *zero-shot* vision-language model that can be used for performing open-vocabulary recognition on satellite imagery [], similar to CLIP []. We use the released model for Sentinel-2 and NAIP and train our own model for the NICFI basemaps. The model architecture is ViT-B-16.
- •
We finetune a *fully-supervised* ResNet-50 pre-trained on ImageNet on both LR (Sentinel-2) and HR (NAIP) data with Open Street Maps annotations on our training set.

#### 4.1.2 Benchmark

For our experiments, we require aligned LR and HR imagery to fairly compare the performance of LR and HR models. However, this is not required in our problem setup or approach. We curate the following two benchmarks. Both benchmarks use Sentinel-2 as the low-resolution modality, but differ in the high-resolution modality:

##### Sentinel-2/NAIP.

In this benchmark, we use HR imagery (GSD=1m) captured by the National Agriculture Imagery Program []. This program only captures data for the US, so this benchmark is restricted to the US. Here, one Sentinel-2 image corresponds to 100 NAIP images. The cost of acquiring similar resolution imagery ranges from $1.00–$6.00 per square km [].

We created a training dataset and validation dataset using images from the following states and regions: Arkansas, Delaware, Idaho, Maine, Rhode Island, Wyoming, and US Virgin Islands. These datasets are comprised of 45,885 Sentinel-2 images for LR, and 4,588,500 NAIP images for HR in the training dataset, and 4,938 and 493,800 images respectively for the validation dataset. Our testing imagery is comprised of images from D.C., Puerto Rico, and Hawaii. The test dataset is comprised of 5,015 Sentinel-2 and 505,100 NAIP images. We chose these locations as they represent a diverse set of demographic and climatic regions within the US.

##### Sentinel-2/NICFI.

Our second benchmark leverages the NICFI Satellite Data Program Basemaps for Tropical Forest Monitoring [] which provides HR imagery (GSD=5m) for regions of Central/South America, Africa, and Asia. Similar imagery has prices ranging from $0.60–$2.0 per square km []. Here one Sentinel-2 image corresponds to 4 NICFI images.

All our data was downloaded via Google Earth Engine []. To handle time discrepancies we retrieved the LR image for a set of HR within the same month that the HR image was collected to avoid major disagreements between the two sets of images. This data will be released publicly along with the download scripts so that others may download data for additional locations. For both benchmarks, we follow [], and use [] to obtain ground truth annotations for 40 concepts (listed in Appendix 6.1).

All classes are seen for supervised techniques, while all classes are unseen for zero-shot baselines.
We reserve 30 of these classes as seen classes and the other 10 are unseen classes when evaluating the knowledge-distilled zero-shot techniques.

#### 4.1.3 Metrics

Since the scope of this method is retrieval, we evaluate our results with mAP@k averaged over all classes. We also look at the amount of HR imagery utilized, as well as runtime in seconds to perform inference on the imagery. The runtime is calculated on the images determined by each technique; for LR models our test set has 5,015 images, and for our HR models the test set has 501,500 images. Techniques utilizing HR sampling methods, such as our own or Patchdrop [] use a combination of these two datasets. All inference is performed using a batch size of 32 on a single Nvidia RTX A6000 GPU.
For our experiments we set a budget of 1000 locations to acquire HR imagery, with each location covering roughly 5 sq. km. We report our results in sq. km., so we can compare to other techniques that do not sample by location.

### 4.2 Results

We examine our method by testing individual components.
We first evaluate the entire system (sec 4.2.1).
We then evaluate our LR Knowledge Distillation Models in sec. 4.2.2.
After that, we examine our model disagreement scheme for selecting modalities in sec. 4.2.3).
Finally, we assess the usage of LLMs for search concept scale determination (sec 4.2.4).

#### 4.2.1 Overall System Performance

We assess the performance of our entire system (Table 2) by comparing the following baselines:

- •
HR: Simply using the HR data and HR models for all locations.
- •
LR: Only using LR data and LR model.
- •
KD: Only using LR data, but with our KD model.
- •
model dis.: Using our model disagreement approach to sample locations for HR imagery, but without any LLM-based inference of whether the concept needs HR data.
- •
LR + LLM: Using an LLM to decide which concepts need HR data, but then acquiring HR images for all locations and using the LR model for other concepts.
- •
KD + LLM: Using an LLM to decide which concepts need HR data, but then acquiring HR images for all locations and using the KD model for other concepts.
- •
Patchdrop: Using Patchdrop [] to determine HR sample locations.
- •
Ours full: Using our full system, from LLM to determine the best suited resolution, with our model disagreement technique to sample HR imagery with a constrained budget, and using our KD model to perform inference for HR-suited concepts in out-of-budget area. Which would be the same as KD + LLM + model dis.
- •
nl. sampling: Using sampling strategies put forth in IS-count [] (via-night lights) and compare it to our own model disagreement technique.

We find that our overall system performs the best on both classes of pre-trained models, even outperforming the baseline of using all HR data for all locations. With 26.3% and 24.6% improvement on zero-shot technique GRAFT, and 6% and 8.3% improvement on supervised techniques in mAP^100 and mAP^20 respectively using only 23.6% of the HR images.

We see our system perform more efficiently with regard to runtime as our system can handle retrievals for multiple concepts at once. If all of the concepts are best suited for LR, inference using the LR model occurs at most once. If some concepts are better suited for HR imagery then inference happens at most twice using LR imagery and at most once using HR imagery. So irrespective of the number of concepts, we can retrieve multiple concepts in about the same time.

**Table 1: Full system performance under various settings for retrieval under budget for zero-shot techniques.^†:Techniques utilizing our contributions. *: Classes seen during knowledge distillation, for all other zero-shot baselines all classes are unseen. For models using both HR and LR data, we assign a budget of 1000 locations ($\sim$ 5 sq. kms each). Our full system performs much better than the baselines with improved HR data constraints.**
|  |  | Seen Classes* | Unseen Classes | # HR images | Inf. Time |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Data | mAP^100 | mAP^20 | mAP^100 | mAP^20 | sq. km. | s |
| GRAFT HR | HR | 0.501 | 0.513 | 0.541 | 0.574 | 25,163 | 1539 |
| GRAFT LR + LLM^† | LR,HR | 0.530 | 0.561 | 0.549 | 0.573 | 25,163 | 1575 |
| GRAFT KD + LLM^† | LR,HR | 0.557 | 0.576 | 0.541 | 0.574 | 25,163 | 1570 |
| GRAFT LR | LR | 0.482 | 0.507 | 0.379 | 0.471 | 0 | 17 |
| GRAFT KD^† | LR | 0.534 | 0.559 | 0.490 | 0.503 | 0 | 17 |
| RemoteCLIP liu2024remoteclip [] | LR | 0.461 | 0.493 | 0.413 | 0.440 | 0 | 17 |
| CLIP-RSICD arutiunian2021fine [] | LR | 0.393 | 0.441 | 0.311 | 0.303 | 0 | 17 |
| OpenCLIP cherti2023reproducible [] | LR | 0.481 | 0.505 | 0.392 | 0.447 | 0 | 16 |
| GRAFT LR + nl. sampling meng2022count [] | LR,HR | 0.379 | 0.354 | 0.440 | 0.492 | 5,954 | 323 |
| GRAFT KD + model dis.^† | LR,HR | 0.621 | 0.628 | 0.461 | 0.494 | 5,954 | 340 |
| Graft (Ours full)^† | LR,HR | 0.633 | 0.639 | 0.502 | 0.564 | 5,954 | 372 |

**Table 2: Full system performance of various fully-supervised techniques for retrieval under budget.^†:Techniques utilizing our contributions. For models using both HR and LR data, we assign a budget of 1000 locations ($\sim$ 5 sq. kms each). Our full system performs much better than the baselines with improved HR data budget constraints.**
| Model | Data | mAP^100 | mAP^20 | # HR images in sq. km. | Inf. Time in s |
| --- | --- | --- | --- | --- | --- |
| Supervised HR | HR | 0.695 | 0.735 | 25,163 | 1585 |
| Supervised LR + LLM^† | LR,HR | 0.689 | 0.731 | 25,163 | 1610 |
| Supervised KD + LLM^† | LR,HR | 0.696 | 0.735 | 25,163 | 1609 |
| Supervised LR | LR | 0.451 | 0.473 | 0 | 14 |
| Supervised KD^† | LR | 0.570 | 0.606 | 0 | 14 |
| Resnet 50 (Multi-Res) | LR | 0.421 | 0.447 | 0 | 21 |
| SatMAE cong-22 [] | LR | 0.351 | 0.388 | 0 | 67 |
| Cross-Scale MAE tang2023cross [] | LR | 0.412 | 0.422 | 0 | 91 |
| Supervised LR + nl. sampling meng2022count [] | LR,HR | 0.463 | 0.480 | 5,954 | 319 |
| Supervised + Patchdrop uzkent2020learning [] | LR,HR | 0.445 | 0.470 | 6,776 | 360 |
| Supervised KD + model dis.^† | LR,HR | 0.733 | 0.791 | 5,954 | 328 |
| Supervised (Ours full)^† | LR,HR | 0.736 | 0.796 | 5,954 | 338 |

#### 4.2.2 Detection in Low Resolution

We compare the performance of several models for multi-label classification in Tables 3 and 4 on 40 classes (as listed in the appendix 6.1) using LR imagery alone. We see improvements in performance through our knowledge distillation process for both zero-shot and fully-supervised techniques. For each corresponding training data type and model architecture, we also show the performance of the HR model (in gray).

We also compare our zero-shot KD Model with other zero-shot baselines CLIP-RSICD [] and RemoteCLIP [] in Table 3.
We compare our fully supervised KD Models with other fully supervised baselines ScaleMAE [], SatMAE [], and Cross-Scale MAE [] as well as a Resnet-50 fine-tuned on imagery of multiple resolutions (Multi-Res) in Table 4.

While the performance of our KD models reaches closer to that of HR models there still is a gap (e.g., more than 5 percentage points in mAP). This suggests that knowledge distillation from HR data alone is not enough to build better recognition systems. Therefore, a framework like ours that works on multiple resolutions is needed, since not all concepts are favored by HR imagery.

#### 4.2.3 Model Disagreement for Modality Selection

Following the methodology in sec. 3.5, we describe how the disagreement between HR and LR models can act as a good indicator for which locations would benefit from HR imagery. We demonstrate how well the KD model performs as a stand in for the HR model, so that we may calculate this disagreement without requiring HR imagery.
Figure 3 visualizes images ranked by degree of disagreement between the LR and HR model (top) and between the LR and KD model (bottom).
We see that both approaches yield similar rankings.
Therefore the disagreement between the LR and KD model is a good proxy for locations where HR data is needed.

Figure: Figure 3: Images ranked according to disagreement between the LR and HR model (top) and the LR and KD model (bottom). Both rankings are similar, even though the latter only uses LR images.
Refer to caption: extracted/5970003/images/re_spaced_min_to_max_disagreement.png

We use this model disagreement along with the KD and HR model to show how well we perform retrieval under an HR-data budget.
Figures 4 illustrate the improvement in retrieval with our HR data sampling technique.
The figure on the left shows precision@K at a fixed HR data budget (1000 locations) and the figure on the right shows precision@1000 when varying the data budget for the GRAFT model on unseen categories.
Our HR data sampling strategy is significantly better than randomly sampling locations or model uncertainty-based methods. This also shows that sampling HR images leads to improvement over just using LR-data with KD models.

We compare the performance of our sampling technique against the sampling technique presented in [], in which night lights are used as an indicator to sample more densely. The results of this sampling technique are included in Table 2.

Figure: Figure 4: Performance when using our model disagreement-based sampling strategy. Our approach consistently yields higher precision across all budgets and across all values of $K$.
Refer to caption: extracted/5970003/images/precision_for_varying_k_budget_1000.png

#### 4.2.4 Concept Scale Inference

We compared our LLM-based approach for inferring the best modality per concept with other baseline approaches. These baselines include either always choosing the HR modality, always choosing the LR modality, or selecting between the two based on the average area of each concept as measured in the Open Street Maps (OSM) annotations.
For each approach, we evaluate its accuracy in terms of determining the right modality.

We evaluate this on 10 unseen classes as the seen concepts were given to the LLM as examples as described in sec. 6.2.
The LLM approach selected the correct resolution 100% of the time. Both the OSM average area approach and choosing only HR imagery were correct  90% of the time whereas selecting solely LR imagery was correct only  10% of the time. This result is interesting as the LLM outperformed using the average area of the concepts (OSM), suggesting that it is not just the size of the concept that is an important consideration, but also a semantic understanding of the concept and its features. We include a tabulated version of these results in the appendix, sec. 5.

Figure: Table 3: Performance of the unsupervised models for recognition at low resolution. Our knowledge-distilled models perform better on both seen and unseen concepts (corresponding HR models in grey; references included in Tab. 1).

Limitations.
While our technique works well to cover all concepts regardless of scale, it performs in a setting in which satellite imagery is split into two groups, low and high resolution. However, in reality, resolution is not discrete; recognizing exactly which resolution is best suited to a concept is something we have not yet determined.

We also utilize OSM data to train and evaluate our models. We acknowledge that this data contains noise, due to its crowd-sourced nature.
Unfortunately, there are not many sources for fine-grained labeling of satellite imagery.
Therefore, we follow past work in using OSM for training and evaluation [, ].
Moreover other standard sources of labeled satellite imagery such as Microsoft’s building footprint dataset work with similar noise levels [].
Additionally, to show robustness to this noise, we performed an experiment using the GRAFT LR model, wherein we constructed bootstrap samples by sub-sampling within our original test set 1000 times.
The results show that the 90% confidence interval is within 2.43% of the reported mAP.

Our technique also does not allow for the segmentation of concepts, i.e., segmenting specific regions within lower-resolution images that are worth looking into at a higher resolution. We leave the exploration of this direction to future work.

## 5 Conclusion

Conclusions. We introduce a new approach to scale-aware recognition in satellite imagery under resource constraints. Our approach allows one to accurately detect various concepts using a fixed budget of HR imagery, outperforming entirely HR baselines by more than 26% mAP in zero-shot techniques, and more than 8% mAP in supervised techniques using 5$\times$ fewer HR images.

Broader Impacts.
Our system offers a cost-efficient way to perform recognition in satellite imagery by maximizing the use of HR satellite imagery while simultaneously cutting the costs. This can have a significant positive impact on a wide range of scientists, anthropologists, archaeologists, NGOs, and human rights organizations, who operate on limited budgets, but who greatly benefit by the use of satellite imagery in their work.

## 6 Appendix

### 6.1 Multi-Label Classes

The classes we use for labeling our data as well as for testing are the following.

’tennis’, ’skate’, ’american football’, ’swimming’, ’cemetery’, ’garage’, ’golf’, ’roundabout’, ’parking lot’, ’supermarket’,
’school’, ’marina’, ’baseball’, ’fall’, ’pond’, ’airport’, ’beach’, ’bridge’, ’religious’, ’residential’, ’warehouse’,
’office’, ’farmland’, ’university’, ’forest’, ’lake’, ’nature reserve’, ’park’, ’sand’, ’soccer’, ’equestrian’, ’shooting’,
’ice rink’, ’commercial area’, ’garden’, ’dam’, ’railroad’, ’highway’, ’river’, ’wetland’

In the case of the unsupervised techniques, the first 30 are used as the seen classes, while the latter 10 are treated as unseen classes.

### 6.2 Prompts to LLM used when Inferring Concept Scale

The prompt was “Please act as a binary classifier for the following concepts to determine if they are better suited for ’LR’ (low resolution) or ’HR’ (high resolution) imagery. I will first give you some examples with the correct response. Then given a concept you are to return ’lr’ or ’hr’”

This was followed by the list of ‘seen’ concepts and their optimal resolution in the format “concept:resolution”. After this the LLM only responded with either ‘lr’ or ‘hr’ when given a concept.

### 6.3 Tabulated Concept Scale Inference Results

**Table 5: Comparison of concept recognition performance (accuracy, precision, recall) between baseline models and LLM selected models.**
|  | Seen Classes | Unseen Classes |
| --- | --- | --- |
| Technique | Accuracy | Accuracy |
| LLM (ours) | 100% | 100% |
| OSM | 73.33% | 90% |
| Always HR | 83.33% | 90% |
| Always LR | 16.67% | 10% |

We compared our LLM-based approach for inferring the best modality per concept with other baseline approaches. For each approach, we evaluate its accuracy in terms of determining the right modality. These baselines include either always choosing the HR modality and always choosing LR modality, or selecting between the two based on the average area of each concept, as provided via OpenStreetMaps (OSM) [].

We evaluate on the 10 unseen classes as the seen concepts were given to the LLM as examples as described in sec. 6.2.

The LLM approach selected the correct resolution 100% of the time. Both the OSM average area approach and choosing only HR imagery were correct  90%. Finding that the LLM outperformed using the average area of the concepts (OSM), suggests that it is not just the size of the concept that is an important consideration, but also a semantic understanding of the concept itself.

### 6.4 Results Per Concept

Table 6 contains results for each class, allowing one to see which models performed best between the supervised HR model and the novel technique presented in this paper. One can see that for most classes, our model manages to out perform the HR model, without needing nearly as many HR images, and for some concepts with no HR images at all.

**Table 6: Performance of various algorithms for various concepts. One can see that our usage of HR images is fully concept aware, and we are able to even outperform models that use solely HR imagery.**
|  | Technique |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Supervised HR | Supervised Ours Full |  |  |  |  |
| Concept | mAP^100 | mAP^20 | # HR imgs | mAP^100 | mAP^20 | # HR imgs |
| Tennis | 0.617 | 0.701 | 25,163 | 0.969 | 1.0 | 5,954 |
| Skate Park | 0.277 | 0.300 | 25,163 | 0.163 | 0.345 | 5,954 |
| American Football | 0.449 | 0.663 | 25,163 | 0.265 | 0.305 | 5,954 |
| Swimming | 0.789 | 0.781 | 25,163 | 0.912 | 0.879 | 5,954 |
| Cemetery | 0.294 | 0.308 | 25,163 | 0.615 | 0.817 | 0 |
| Garage | 0.696 | 0.835 | 25,163 | 0.964 | 1.0 | 5,954 |
| Golf | 0.483 | 0.194 | 25,163 | 0.974 | 1.0 | 0 |
| Roundabout | 0.401 | 0.471 | 25,163 | 0.823 | 0.984 | 5,954 |
| Parking Lot | 1.0 | 1.0 | 25,163 | 1.0 | 1.0 | 5,954 |
| Supermarket | 0.928 | 0.978 | 25,163 | 0.773 | 0.756 | 5,954 |
| School | 0.824 | 0.717 | 25,163 | 0.913 | 0.956 | 5,954 |
| Marina | 0.389 | 0.516 | 25,163 | 0.189 | 0.294 | 5,954 |
| Baseball | 0.672 | 0.747 | 25,163 | 0.658 | 0.713 | 5,954 |
| Pond | 0.921 | 0.967 | 25,163 | 0.888 | 0.923 | 5,954 |
| Airport | 0.558 | 0.843 | 25,163 | 0.590 | 0.934 | 5,954 |
| Beach | 0.970 | 0.995 | 25,163 | 0.910 | 0.921 | 5,954 |
| Bridge | 0.986 | 1.0 | 25,163 | 0.976 | 1.0 | 5,954 |
| Religious Buildings | 0.933 | 1.0 | 25,163 | 0.980 | 1.0 | 5,954 |
| Residential | 1.0 | 1.0 | 25,163 | 1.0 | 1.0 | 5,954 |
| Warehouse | 0.901 | 0.969 | 25,163 | 0.945 | 0.927 | 5,954 |
| Office | 0.955 | 0.995 | 25,163 | 0.961 | 0.914 | 0 |
| Farmland | 0.859 | 0.883 | 25,163 | 0.854 | 0.883 | 0 |
| University | 0.505 | 0.448 | 25,163 | 0.652 | 0.920 | 0 |
| Forest | 0.977 | 0.995 | 25,163 | 0.933 | 1.0 | 0 |
| Lake | 0.206 | 0.458 | 25,163 | 0.257 | 0.496 | 5,954 |
| Nature Reserve | 0.938 | 0.997 | 25,163 | 0.915 | 0.995 | 5,954 |
| Park | 1.0 | 1.0 | 25,163 | 0.983 | 0.997 | 0 |
| Sand Pits | 0.847 | 0.811 | 25,163 | 0.983 | 1.0 | 5,954 |
| Soccer | 0.497 | 0.440 | 25,163 | 0.122 | 0.053 | 5,954 |
| Equestrian | 0.395 | 0.497 | 25,163 | 0.083 | 0.200 | 5,954 |
| Shooting Range | 0.01 | 0.05 | 25,163 | 0.01 | 0.05 | 5,954 |
| Commercial Area | 0.483 | 0.425 | 25,163 | 0.895 | 0.934 | 5,954 |
| Garden | 0.815 | 0.839 | 25,163 | 0.901 | 0.956 | 0 |
| Dam | 0.062 | 0.227 | 25,163 | 0.062 | 0.227 | 5,954 |
| Railroad | 0.964 | 0.974 | 25,163 | 0.900 | 0.974 | 5,954 |
| Highway | 1.0 | 1.0 | 25,163 | 1.0 | 1.0 | 0 |
| River | 1.0 | 1.0 | 25,163 | 0.967 | 1.0 | 5,954 |
| Wetland | 0.827 | 0.925 | 25,163 | 0.824 | 0.957 | 5,954 |

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1.Quote" A location l with higher δ(l) is more likely to have concepts that MHR could detect where MLR  could not (or vice versa)." Since it is the absolute value, it is difficult to understand.
2. How is the LLM trained? What's the confidence for the determination of LLM?
3. Can it be understood as that the low resolution images taken as large as possible because the cost constraints is not imposed?
4. Quote"...Remote CLIP(Liuetal.,2024) in Table 3." But the table is not the detection result.
soundness: 3
strengths: 1. A novel idea of scale determination is proposed for targer recognition using reasonable resolution remote sensing images.
2. Low resolution detection model using distillation strategy is taken as the replacement of high resolution detection model.
3. Two benchmark datasets are constructed for research.
summary: To provide a holistic treatment of scale to achieve accurate recognition under a budget, the paper tries to give solutions to which resolution is the best and where and when  the costlier higher-resolution (HR)imagery should be acquired. The main contributions are: (1)knowledge distillation is adopted to allow Low resolution models to improve significantly in  recognizing finer concepts by learning from High resolution ones. (2) Semantic understanding of LLM is used to determine the  scale of each concept.(3) A sampling strategy for HR imagery based on model disagreement is proposed. The process is frameworked as remote sensing image retrieval task. Experiments show some promising success for the proposed idea.
The idea is very novel and interesting.
weaknesses: 1. The budget constraints seems actually impose no impact on the final result.
2. The processes such as distillation, concept scale LLM training are not provided the details.
3. The retrieval process has no systematic description to include all the subprocesses.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The system is based on many ideal assumptions. Whether deployed online or offline, it will encounter foreseeable issues such as data storage and retrieval, online search, and download stability. Additionally, the overall process time and reliability are concerns. Although I like the idea, there seems to be a large gap between the proposed method and practical application.
soundness: 3
strengths: 1. This paper presents a novel framework to perform the tradeoff between cost and accuracy. 
2. The topic is interesting and valuable. 
3. The writing is clear. 
4. It is good that the authors plan to release their data and download scripts.
summary: This paper introduced a new approach to scale-aware recognition in satellite imagery under resource constraints. The proposed system leverages LLMs and knowledge distillation technique. The proposed system outperformed the baselines  more than 26% mAP in zero-shot techniques, and more than 8% mAP in supervised techniques using 5× fewer HR images.
weaknesses: 1. It is unclear how the system handles noisy labels using OpenStreetMap annotations. 
2. Can the proposed system retrieve multiple targets simultaneously? 
3. How to deal with the time discrepancy between high- and low-resolution images, which can lead to disagreements between the LR and HR models. Did the authors consider the impact of this issue? 
4. No mention of runtime in Table 3.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: The paper distills a model from paired HR and LR images to predict whether HR images are necessary based on prediction differences. Why not directly train a super-resolution model on paired HR-LR images, use it to generate HR images for any LR input, and then perform recognition on both the generated HR and original LR images? If the recognition results differ significantly, HR images could be invoked. What are the advantages of the proposed, more complex method over this approach?

Impact of resolution ratio on the proposed method is suggested to be discussed. Does the resolution ratio between high and low-resolution images affect the performance of the proposed method? Could you please provide some analysis or experiments to explore how this ratio influences the model’s efficiency and accuracy?
soundness: 3
strengths: This paper focuses on a highly interesting problem with significant industrial application potential. 

The proposed knowledge distillation method and model disagreement-based sampling strategy are both innovative and effective. The distillation enhances the performance of LR models, while the sampling strategy efficiently balances the trade-off between cost and accuracy.
summary: This paper addresses the challenge of feature recognition in satellite imagery, focusing on the optimal use of image resolution to balance accuracy and resource constraints. The proposed system introduces three key components: LLM-based scale inference, knowledge distillation, model disagreement-based sampling. The proposed approach has been demonstrated to improve recognition accuracy while reducing the need for HR imagery.
weaknesses: The approach heavily relies on the examples provided in the prompt for the LLM. The model’s understanding of the relationship between manually defined categories and image resolutions is largely based on these examples. This dependence raises concerns about the generalizability of the method to niche or less common satellite imagery concepts.

The paper does not include experiments comparing the proposed method with techniques that generate HR images via super-resolution, then combine these images with LR data for recognition. Such comparisons, focusing on both computational efficiency and accuracy, would provide a more comprehensive evaluation of the proposed approach.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Does Equation 5 require knowledge of all the information for all seen categories at a specific location to be computed? Why are the HR score and LR score related to location? Does this limit the model to only locations that appeared in the training set? How would the model handle unknown locations?

2. Is using LLM-based in-context learning to determine the scale reliable? Has the accuracy been tested on unseen categories in the validation set? Since the context is determined by the HR and LR scores, why not directly train a network using the HR and LR scores? It is well known that LLMs are very resource-intensive, and using an LLM just to determine the scale seems like overkill. Furthermore, it contradicts the premise of resource efficiency—if one cannot afford HR images, they likely don't have the computational resources to run an LLM either.
soundness: 3
strengths: The authors address the real-world issue where high-resolution (HR) remote sensing images are paid, while low-resolution (LR) images are free, by proposing a framework that effectively improves scene classification performance under limited funding. This framework can be used to analyze which high-resolution images are more beneficial to purchase, helping researchers make informed decisions,
summary: A scene classification task under resource-constrained conditions is proposed, combining image retrieval and scene classification. Knowledge distillation techniques and an active learning-like sample selection strategy are used to choose HR samples, and an LLM is employed to determine the scale of concepts.
weaknesses: 1. The description of the LLM section is not very clear. The authors did not specify which LLM was used, how the hyperparameters were set, or provide comparative experiments on the performance of different LLMs. This weakens the contribution of the LLM. It seems that the authors are merely using it as a tool for invocation.

2. In Section 4.2.3, the discussion of Figure 3, which uses only five images to demonstrate ranking similarity, is not convincing. The authors need to provide a quantitative analysis to support this argument.

Meta-review: The paper proposes a method for scale-aware scene recognition in satellite imagery. This method utilizes knowledge distillation to transfer knowledge from models trained on high-resolution images to models trained on low-resolution images. The goal is to enable the latter to work alongside low-resolution models to identify regions that might require high-resolution imagery based on low-resolution images only. Model disagreements are employed as a decision-making mechanism to determine whether high-resolution imagery is necessary for a given geographical region based solely on low-resolution images.

In this context, it is debatable whether the use of large language models (LLMs) is necessary to identify the scale of a concept. Model disagreement alone can serve as an effective decision mechanism, making scale information from LLMs redundant. While the reviewers recognize the overall idea as interesting and acknowledge the value of model disagreement in determining whether cost-intensive high-resolution imagery is needed, it is questionable whether this approach constitutes a novel contribution.

Decision: Accept (Poster)