Scale-aware Recognition in Satellite Images under Resource Constraints

Authors: Shreelekha Revankar, Cheng Perng Phoo, Utkarsh Mall, Bharath Hariharan, Kavita Bala
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Satellite Imagery, Resolution

Abstract:
Recognition of features in satellite imagery (forests, swimming pools, etc.) depends strongly on the spatial scale of the concept and therefore the resolution of the images. This poses two challenges:
Which resolution is best suited for recognizing a given concept, and where and when should the costlier higher-resolution (HR) imagery be acquired? 
We present a novel scheme to address these challenges by introducing three components: (1) A technique to  distill knowledge from models trained on HR imagery to recognition models that operate on imagery of lower resolution (LR), (2) a sampling strategy for HR imagery based on model disagreement, and (3) an LLM-based approach for inferring concept "scale". With these components we present a system to efficiently perform scale-aware recognition in satellite imagery, improving accuracy over single-scale inference while following budget constraints. **Our novel approach offers up to a 26.3\% improvement over entirely HR baselines, using 76.3 \% fewer HR images.** Resources are available at https://www.cs.cornell.edu/~revankar/scale_aware.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730079613262

thread_type:
official_review

<thread_object>
creator: reviewer_cazh

<content>
summary:
A scene classification task under resource-constrained conditions is proposed, combining image retrieval and scene classification. Knowledge distillation techniques and an active learning-like sample selection strategy are used to choose HR samples, and an LLM is employed to determine the scale of concepts.

strengths:
The authors address the real-world issue where high-resolution (HR) remote sensing images are paid, while low-resolution (LR) images are free, by proposing a framework that effectively improves scene classification performance under limited funding. This framework can be used to analyze which high-resolution images are more beneficial to purchase, helping researchers make informed decisions,

weaknesses:
1. The description of the LLM section is not very clear. The authors did not specify which LLM was used, how the hyperparameters were set, or provide comparative experiments on the performance of different LLMs. This weakens the contribution of the LLM. It seems that the authors are merely using it as a tool for invocation.

2. In Section 4.2.3, the discussion of Figure 3, which uses only five images to demonstrate ranking similarity, is not convincing. The authors need to provide a quantitative analysis to support this argument.

questions:
1. Does Equation 5 require knowledge of all the information for all seen categories at a specific location to be computed? Why are the HR score and LR score related to location? Does this limit the model to only locations that appeared in the training set? How would the model handle unknown locations?

2. Is using LLM-based in-context learning to determine the scale reliable? Has the accuracy been tested on unseen categories in the validation set? Since the context is determined by the HR and LR scores, why not directly train a network using the HR and LR scores? It is well known that LLMs are very resource-intensive, and using an LLM just to determine the scale seems like overkill. Furthermore, it contradicts the premise of resource efficiency—if one cannot afford HR images, they likely don't have the computational resources to run an LLM either.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730628098603

thread_type:
official_review

<thread_object>
creator: reviewer_kyd3

<content>
summary:
This paper addresses the challenge of feature recognition in satellite imagery, focusing on the optimal use of image resolution to balance accuracy and resource constraints. The proposed system introduces three key components: LLM-based scale inference, knowledge distillation, model disagreement-based sampling. The proposed approach has been demonstrated to improve recognition accuracy while reducing the need for HR imagery.

strengths:
This paper focuses on a highly interesting problem with significant industrial application potential. 

The proposed knowledge distillation method and model disagreement-based sampling strategy are both innovative and effective. The distillation enhances the performance of LR models, while the sampling strategy efficiently balances the trade-off between cost and accuracy.

weaknesses:
The approach heavily relies on the examples provided in the prompt for the LLM. The model’s understanding of the relationship between manually defined categories and image resolutions is largely based on these examples. This dependence raises concerns about the generalizability of the method to niche or less common satellite imagery concepts.

The paper does not include experiments comparing the proposed method with techniques that generate HR images via super-resolution, then combine these images with LR data for recognition. Such comparisons, focusing on both computational efficiency and accuracy, would provide a more comprehensive evaluation of the proposed approach.

questions:
The paper distills a model from paired HR and LR images to predict whether HR images are necessary based on prediction differences. Why not directly train a super-resolution model on paired HR-LR images, use it to generate HR images for any LR input, and then perform recognition on both the generated HR and original LR images? If the recognition results differ significantly, HR images could be invoked. What are the advantages of the proposed, more complex method over this approach?

Impact of resolution ratio on the proposed method is suggested to be discussed. Does the resolution ratio between high and low-resolution images affect the performance of the proposed method? Could you please provide some analysis or experiments to explore how this ratio influences the model’s efficiency and accuracy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638816114

thread_type:
official_review

<thread_object>
creator: reviewer_jocu

<content>
summary:
This paper introduced a new approach to scale-aware recognition in satellite imagery under resource constraints. The proposed system leverages LLMs and knowledge distillation technique. The proposed system outperformed the baselines  more than 26% mAP in zero-shot techniques, and more than 8% mAP in supervised techniques using 5× fewer HR images.

strengths:
1. This paper presents a novel framework to perform the tradeoff between cost and accuracy. 
2. The topic is interesting and valuable. 
3. The writing is clear. 
4. It is good that the authors plan to release their data and download scripts.

weaknesses:
1. It is unclear how the system handles noisy labels using OpenStreetMap annotations. 
2. Can the proposed system retrieve multiple targets simultaneously? 
3. How to deal with the time discrepancy between high- and low-resolution images, which can lead to disagreements between the LR and HR models. Did the authors consider the impact of this issue? 
4. No mention of runtime in Table 3.

questions:
The system is based on many ideal assumptions. Whether deployed online or offline, it will encounter foreseeable issues such as data storage and retrieval, online search, and download stability. Additionally, the overall process time and reliability are concerns. Although I like the idea, there seems to be a large gap between the proposed method and practical application.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708038430

thread_type:
official_review

<thread_object>
creator: reviewer_nagq

<content>
summary:
To provide a holistic treatment of scale to achieve accurate recognition under a budget, the paper tries to give solutions to which resolution is the best and where and when  the costlier higher-resolution (HR)imagery should be acquired. The main contributions are: (1)knowledge distillation is adopted to allow Low resolution models to improve significantly in  recognizing finer concepts by learning from High resolution ones. (2) Semantic understanding of LLM is used to determine the  scale of each concept.(3) A sampling strategy for HR imagery based on model disagreement is proposed. The process is frameworked as remote sensing image retrieval task. Experiments show some promising success for the proposed idea.
The idea is very novel and interesting.

strengths:
1. A novel idea of scale determination is proposed for targer recognition using reasonable resolution remote sensing images.
2. Low resolution detection model using distillation strategy is taken as the replacement of high resolution detection model.
3. Two benchmark datasets are constructed for research.

weaknesses:
1. The budget constraints seems actually impose no impact on the final result.
2. The processes such as distillation, concept scale LLM training are not provided the details.
3. The retrieval process has no systematic description to include all the subprocesses.

questions:
1.Quote" A location l with higher δ(l) is more likely to have concepts that MHR could detect where MLR  could not (or vice versa)." Since it is the absolute value, it is difficult to understand.
2. How is the LLM trained? What's the confidence for the determination of LLM?
3. Can it be understood as that the low resolution images taken as large as possible because the cost constraints is not imposed?
4. Quote"...Remote CLIP(Liuetal.,2024) in Table 3." But the table is not the detection result.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731872444823

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to Reviewers

comment:
We thank the reviewers for their time and feedback. Here we answer common questions, we address individual concerns in the comments.

1. **Overall Retrieval Process**  Reviewers nAgQ, JoCu, and Cazh had questions about details of our retrieval process. Given a concept, our LLM determines which resolution is best suited. If the best suited resolution is LR, we perform inference using the LR model and all LR imagery only. If the best suited resolution is HR, we ask the user for their HR budget. Then we perform model disagreement using LR imagery alone and the LR and KD models. With these disagreement scores and the user budget, we then sample the top locations, obtain HR images and perform inference using the HR model for these locations. The retrieval results for the remaining area, that it is preferable to use HR for, but cannot be fit in the budget,  is scored using the KD model on LR imagery.

2. **LLM Component** Reviewers nAgQ, Kyd3, and CazH had questions about the LLM component of our technique. We use an off-the shelf LLM, specifically ChatGPT 3.5, and provide in-context examples as described in section 6.3 (in the supplementary material). We discuss testing the LLM on **unseen** classes in both sec. 4.2.4 on page 9 as well as more in depth on page 2 of the supplementary materials. Here we compare our LLM approach to other baselines, and we include additional results using other off the shelf LLMs (Gemini and Claude).

| Technique | Seen Classes Accuracy | Unseen Classes Accuracy |
|:-----------:|:---------------------:|:----------------------:|
| LLM ChatGPT 3.5 (ours) | **100%** | **100%** |
| LLM Gemini (ours) | **100%** | **100%** |
| LLM Claude (ours) | **100%** | **100%** |
| OSM | 73.33% | 90% |
| Always HR | 83.33% | 90% |
| Always LR | 16.67% | 10% |


We compared our LLM-based approach for inferring the best modality per concept with other baseline approaches. For each approach, we evaluate its accuracy in terms of determining the right modality. These baselines include either always choosing the HR modality and always choosing LR modality, or selecting between the two based on the average area of each concept, as provided via OpenStreetMaps (OSM). Here one can see that the LLM performs with 100% accuracy for our set of concepts, but it is also capable of easily being extended to far more concepts.

3. **Impact of Different Budgets** Reviewer nAgQ inquired about the impact of various budgets on performance. We are able to get an improved accuracy on a tight budget since we only sample HR images when necessary. The reason we are able to do so is (a) some concepts are better in LR and (b) for the remaining concepts we are able to recognize which regions require HR. 

We demonstrate this in the table below with results using our method with varying budgets.
| Model | Seen mAP || Unseen mAP || HR images in Mill. | runtime in s |
|-------|:---:|:---:|:---:|:---:|:--------------------------------------:|:-------------:|
| | @20 | @100 | @20 | @100 | | |
| GRAFT HR | 0.501 | 0.513 | **0.541** | **0.574** | 25,163 | 1539 |
| GRAFT LR | 0.482 | 0.507 | 0.379 | 0.471 | 0 | 17 |
| GRAFT (Ours Full), Budget 100 | 0.557 | 0.605 | 0.439 | 0.521 | **595** | 48|
| GRAFT (Ours Full), Budget 500 | 0.601 | 0.626 | 0.480 | 0.542 | 2,977 | 170 |
| GRAFT (Ours Full), Budget 750 | 0.614 | 0.617 | 0.495 | 0.557 | 4,466 | 247 |
| GRAFT (Ours Full), Budget 1000  | **0.633** | **0.639** | 0.502 | 0.564 | 5,954 | 372 |

5. **Use of Super Resolution model for Model Disagreement Score** Reviewer Kyd3 suggested using a super resolution (SR) model in place of the KD model for model disagreement.

Here we present the results using the SR images with the HR model to calculate the model disagreement score to sample HR imagery. LR + SR + LLM denotes the use of both the LLM and model disagreement components of our work, (using the SR model disagreement), without the use of any KD model. KD + SR + LLM denotes results with our full system, but with the model disagreement scores being obtained via the SR imagery.

With the use of our KD model, SR does offer improvements in mAP for unseen classes. However, the total inference time alone is ~**34 times longer than our technique**, and **this improvement only comes with the use of our KD model.**  Overall this significant jump in inference time makes using any superresolution model in place of our KD model less favorable.

| Model | Seen mAP || Unseen mAP || HR images in Mill. | runtime in s |
|:-------:|:---:|:---:|:---:|:---:|:---------------------:|:--------------:|
| | @20 | @100 | @20 | @100 | | |
| GRAFT HR | 0.501 | 0.513 | 0.541 | 0.574 | 25,163 | 1,539 |
| GRAFT LR + SR + LLM| 0.527 | 0.583 | 0.512 | 0.547 | **5,954**| 12,867 |
| GRAFT KD + SR + LLM | 0.554 | 0.610 | **0.556** | **0.589** | **5,954** | 12,884 |
| GRAFT (Ours Full) | **0.633** | **0.639** | 0.502 | 0.564 | **5,954** | 372 |

Thank you all for your time and we are happy to answer any further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a method for scale-aware scene recognition in satellite imagery. This method utilizes knowledge distillation to transfer knowledge from models trained on high-resolution images to models trained on low-resolution images. The goal is to enable the latter to work alongside low-resolution models to identify regions that might require high-resolution imagery based on low-resolution images only. Model disagreements are employed as a decision-making mechanism to determine whether high-resolution imagery is necessary for a given geographical region based solely on low-resolution images.

In this context, it is debatable whether the use of large language models (LLMs) is necessary to identify the scale of a concept. Model disagreement alone can serve as an effective decision mechanism, making scale information from LLMs redundant. While the reviewers recognize the overall idea as interesting and acknowledge the value of model disagreement in determining whether cost-intensive high-resolution imagery is needed, it is questionable whether this approach constitutes a novel contribution.