InstructSAM: A Training-free Framework for Instruction-Oriented Remote Sensing Object Recognition
Authors: Yijie Zheng
Weijie Wu
Qingyun Li
Xuehui Wang
Xu Zhou
Aiai Ren
Jun Shen
Long Zhao
Guoqing Li
Xue Yang
Venue: NeurIPS 2025 poster ()
Keywords: object recognition
remote sensing
vision-language model
Primary area: applications
TL;DR: This paper presents InstructSAM, a training-free framework that counts, detects, and segments objects in remote sensing imagery based on tailored instructions.

Abstract: Language-guided object recognition in remote sensing imagery is crucial for large-scale mapping and automated data annotation. However, existing open-vocabulary and visual grounding methods rely on explicit category cues, limiting their ability to handle complex or implicit queries that require advanced reasoning.
To address this issue, we introduce a new suite of tasks, including Instruction-Oriented Object Counting, Detection, and Segmentation (InstructCDS), covering open-vocabulary, open-ended, and open-subclass scenarios. We further present EarthInstruct, the first InstructCDS benchmark for earth observation. It is constructed from two diverse remote sensing datasets with varying spatial resolutions and annotation rules across 20 categories, necessitating models to interpret dataset-specific instructions.
Given the scarcity of semantically rich labeled data in remote sensing, we propose InstructSAM, a training-free framework for instruction-driven object recognition. InstructSAM leverages large vision-language models to interpret user instructions and estimate object counts, employs SAM2 for mask proposal, and formulates mask-label assignment as a binary integer programming problem. By integrating semantic similarity with counting constraints, InstructSAM efficiently assigns categories to predicted masks without relying on confidence thresholds. Experiments demonstrate that InstructSAM matches or surpasses specialized baselines across multiple tasks while maintaining near-constant inference time regardless of object count, reducing output tokens by 89\% and overall runtime by over 32\% compared to direct generation approaches. We believe the contributions of the proposed tasks, benchmark, and effective approach will advance future research in developing versatile object recognition systems. The code is available at https://VoyagerXvoyagerx.github.io/InstructSAM.

InstructSAM: A Training-Free Framework for Instruction-Oriented Remote Sensing Object Recognition
Yijie Zheng, Weijie Wu Aerospace Information Research Institute, Chinese Academy of Sciences University of Chinese Academy of Sciences Ichengyijie22, wwweijie (eeeiijieee))mmaiii ucas ac.
Qingyun Li Harbin Institute of Technology 21b9050030stu. wit. edu...d.... cn
Xuehui Wang
Xu Zhou, Aiai Ren, Jun Shen University Wollongong 1x2572, jaaere44,,ssseeena)oe wuw.aaaaennoollllccmmmm edu...edda...
Shanghai Jiao Tong University
wangxuehui 8s ju.. edu.
Long Zhao, Guoqing Li
Xue Yang
Aerospace Information Research Institute, Chinese Academy of Sciences (zhaolong, liggaaaraa.. ac.
SAIS, Shanghai Jiao Tong University yangxue-201 a-1 tuβsjtu. edu.cn
Abstract
Language-gus guided object recognition in remote sensing imagery is crucial for large- scale mapping and automated data annotation. However, existing apen-vocabulary and visual grounding methods rely on explicit category cues, limiting their ability to handle complex or implicit queries that require advanced reasoning. To address this issue, we introduce new suite of tasks, including instruction-Oriented Object Counting, Detection, and Segmentation (InstructCDS), covering open- open-vocabulary, open-ended, and open-subelass scenarios. We further present EarthInstruct, the first InstructCDS benchmark for earth observation. It is constructed from two diverse remote sensing datasets with varying spatial resolutions and annotation rules across 20 categories, necessitating models to interpret datasct-specific in- structions. Given the scarcity of semantically rich labeled data in remote sensing, propose InstructSAM, training-fiee free framework for cnstruction-driver object recognition. InstructSA.M leverages large vaniun-e models to interpret user instructions and estimate object counts, employs SAM2 for mask proposal, and formulates mask- label assignment as a binary integer programming prob- lem. By integrating semantic similarity with counting constraints, InstructSAM efficiently assigns categories to predicted masks without relying on confidence thresholds. Experiments demonstrate that InstructSAM matches surpasses spe- cialized baselines across multiple tasks while maintaining near-constant inference time regardless of object count, reducing output tokens by 89% and overall runtime by over 32% compared to direct generation approaches. We believe the contribu- tions of the proposed tasks, benchmark, and effective approach will advance future research in developing versatile object recognition systems. The code is available at htttp::/Nooygeeexvoyaeerr github. io/ Inssuruututts/////
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Close-set
(b) Open-vocabulary
(c) Open-ended
(d) Open-subelass
Figure 1: Comparison of four task settings in object detection and segmentation. The sample is from DIOR Dataset [34] (a) Close-set: Standard annotation with categories defined in the DIOR dataset. (b) vpen-vocabulary: Instruction specifies which categories to detect and segment (e.g.. "soccer field", "football field", "parking lot"). (c) Open-ended: Instruction requires detection and segmentation of all visible objects without category specification. (d) Open-subclass: Instruction targets all objects within super-category (e.g., "sports fields").
1
Introduction
Object recognition in remote sensing imagery captures vast array of objects and phenomena across diverse environments, providing rich information for supporting achieving the Sustainable Development Goals issued by the United Nations (59 3, such as wildlife monitoring [2 [66]
poverty estimation 50 and disaster response [8. The recent advent of powerful vision-language models (VLMs), such as CLIP [6]]. has ushered in new era of remote sensing oriented open- vocabulary object recognition algorithms (e.g. detection [8] 56 and segmentation (2I)). However, existing open-vocabulary approaches predominantly rely explicit category cues, which restrict their capability to handle complex or implicit queries that demand advanced reasoning and contextual understanding. In other words, the rich diversity of visible objects in remote sensing images, due to the bird's s-eye view, predicates that any predefined, fixed list of categories is inevitably incomplete, limiting its practicality for open-ente ended analysis in the real world.
To address this issue, we expand the cnernctiio--oiennted object detection task [60] and intro- duce novel suite of tasks Instr-ntnttuttnnnnrieeeee (InstructCDS) that encompasses apen-vocabulary. open-ended, and apen-subelass settings, as illus- trated in Figure [ The InstructCDS task entails a more flexible and scalable interpretation beyond fixed category sets and comprehends the complex users' task requirements. We further present Earth- Instruct, the first benchmark for InstructCDS in earth observation. The benchmark is constructed from two generic remote sensing object datasets covering 20 categories with different annotation rules and spatial resolution. EarthInstruct guides models to comprehend complex user instructions beyond the predefined three settings.
Object Counting, Detection, and Segmentation
Recent advancements in VLMs have demonstrated impressive performance in object detection 45 40
60 57, semantic segmentation 88 94], visual grounding 2 / and reasoning- based segmentation
within the natural image domain. However, transferring these methods to remote sensing imagery presents several challenges. First, direct inference leads to significant accuracy degradation due to the domain gap between natural and aerial images
666 29
[14]. Second, most existing remote sensing open-vocabulary detection [38 39 90, 31 and segmentation [91 88 21 methods are trained on datasets with only limited number of categories, restricting their generalization to diverse unseen categories. Third, conventional detectors [35 [1] rely on a threshold to filter predicted bounding boxes, which is not obtainable in zero-shot scenarios.
To tackle these challenges, we decompose the instmaction-oriented object detection and segmentation tasks into several tractable steps and propose framework without task- specifcc training, Instruct- SAM. First, large vision-lant language model (LVLM) is employed to interpret user instructions and predict object categories and counts, with prompts systematically designed to maximize model's capability. In parallel, SAM2
63 is utilized to automatically generate mask proposals. Next, CLIP model pre-trained on remote sensing images computes the semantic similarity between the predicted objects categories and mask proposals. We then formulate the object detection and segmentation mask-label matching problem, assigning predicted categories to mask proposals

---

by integrating semantic similarity and global counting constraints. By inherently integrating three powerful foundation models, InstructSAM achieves superior performance across multiple tasks compared to both generic and remote -ppecfc- VLMs trained on large-scale object recognition data. Notably, the inference time of InstructSAM remains nearly constant with respect to the number of predicted objects, reducing output tokens by 89% and overall runtime by 32% compared to directly generating bounding boxes using Qwen2.5-VV VL [2] in the open-ended setting. Our work paves the way for scalable, instruction-driven remote sensing object detection and segmentation, eliminating the need for costly pre-training or manual threshold tuning. Furthermore, the training-free paradigm allows InstructSAM to recognize objects in natural images when equipped with generie CLIP models.
In summary, our contributions are as follows:
We introduce the InstructCDS task, which challenges models to interpret user-provided natural language instructions and infer the number and locations relevant objects.
We construct EarthInstruct to benchmark InstructCDS in earth observation, covering open- vocabulary, open geeneended, and open-subelass settings and counting, detection, and segmentation tasks.
We develop InstructSAM, a training-free and confidence- framework achieving near-constant inference time for the InstructCDS task.
Experiments on the benchmarks demonstrate that InstructS matches close-set models object counting and surpasses generic and remote sensing-specific models in open-vocabulary and open- ended object recognition.
2 Related work
2.1 Instruction-oriented object detection and segmentation
Instruction-oriented methods Instruction-orientee object detection (IOD), first introduced in (60.
includes four instruction settings: category-specified (open-vocabulary [80] detection, detection of all objects (open-ended [40]), detection within super-category (we name it open-subelass), and detection to achieve certain goals. Ins-DeiCLIP [60] trains a detector to identify foreground objects and passes their features large language model (LLM) to generate categories based on user instructions. In addition to models specifically designed for the IOD task, Qwen2.5-Vi VL (2]. trained multi-task instruction data, also showcases the ability of dense object detection. However, both approaches require extensive task-specific training data, and their inference times increase substantially with the number of objects.
Open-vocabulary methods vocce-vobbulary object detection and segmentation datasets 70
766 and visual grounding datasets 277
23
enable various open-vocabulary learning approaches,
including knowledge distillation [16] and region-text pre-training (26] 52 45 35 J Self-training with
confidence threshold-fitered pseudo labels from image-text pairs (e.g.. CC3M (. WebLI2B 6
further boosts the performance [51 45
93 87]. However, the quality of pseudo labels is highly
sensitive to the chosen threshold (51). and these methods require predefined object categories, limiting
their ficxibility in diverse scenarios.
Open-ended methods GenerateU first formulates the open-ended object detection (OED) problem. Concurrent works such as DetCLIPv3 [77, Florence-2 [83]. and DINO-X 655 introduce generative frameworks that jointly predict object categories and bounding boxes using language models. However, constructing large-scale datasets with bounding box and caption pairs is resource- intensive. VI-SAM [1 proposes a training-free approach via attention as prompts, but iterative mask refinement and multi-prompt ensemble strategies are computationally expensive.
2.2 Instrnetion-orienneed remote sensing object detection and segmentation
have also enabled vocabuaayyl learning the remote sensing domain. Diverse semantic tags from OpenStreetMap [19 and those generated by LVLMSs drive the development of contrastive language-image pre-training remote sensing images 77 Following the generic open- vocabulcyy learning frameworks, methods for remote sensing open-vocabulary
Recent advances VLMs 61 44

---

detection [38 79 56 22 and segmentation 19 88 21 32
methods emerge. However, their human-
annotated training data, being limited a few dozen categonies 34 to onttr--distributionn or zero-shot scenarios. Although some remote sensing LVLMs could support object recognition tasks like single-class object detection
78
47, hinder generalization
25 visual grounding [54 58)
referring expression segmentation [57, grounded conversation generation [68], and scene graph generation [49], they failed to follow complex reasoning instructions, such as open-vocabulary and open-subelass object detection. To annotate vast-vocabularg training data for remote sensing object detection, LAE-Label [56] employs a generic LVLM [ to predict categories for cropped mask proposals. However, this approach loses global context for accurate category classification.
In contrast, our InstructSA. adopts a confidence- paradigm, requires no task-specific pre-training or finetuning. and maintains near-constant inference time regardless of object counts.
(a)
(b)
(d)
Figure 2: Annotation samples from NWPU-VHR-10 (a) and DIOR (b-d) illustrating rules that might differ from common sense. (a) Ships and vehicles unannotated in low-resolution images (NWPU- VHR-10). (b) Expressway service areas separated by a road are treated as distinct instances (DIOR).
(c. d) Airports annotated only when fully visible (DIOR).
3 EarthInstruct, a benchmark for InstructCDS in remote sensing
3.1 Instruction setup
To enable practical applications in remote sensing such as large-scale mapping and image annotation, define three distinet settings for InstructCDS:
1) Open-vocabulary:
Counting, detection, and segmentation with user-specified categories.
2) Open-ended: Counting, detection, and segmentation of all objects without specifying categories.
3) Open-subclass: Counting, detection, and segmentation of objects within super-category.
We construct EarthInstruct using NWPU-VHR-I [9] and DIOR 34 datasets, selected for their widespread use and diverse sensors, resolutions, and annotation rules. Critically, these dataset- specific annotation rules might deviate from common sense (e.g., excluding low-resolution vehicles) exhibit semantic ambiguities (e.g., "bridge" "'verppsss") reflecting the original annotators' specific goals (Figure 2 Consequently, simple instructions like "count vehicles" would fail to capture the nuances required by dataset conventions user intent. EarthInstruct therefore necessitates models to interpret detailed instructions that clarify target definitions and handle dataset-specific rules (e.g., "do count vehicles in images with spatial resolution lower than lm."). To ensure fair evaluation aligned with dataset conventions and user requirements, prompts are designed accordingly, but image-specific prompts prohibited to maintain scalability for large-area applications where prior content knowledge for each image is unavailable.
Evaluation metrics
Multi-class object counting Standard counting metrics, such as Mean Absolute Error (MAE)
and Root Mean Squared Error (RMSE), used in benchmarks like FSC- 147 (22 and RSOC [5]] inadequately capture nuanced multi-class evaluation. They cannot distinguish between over-counting and under-counting errors. Additionally, being unnormalized, they allow categories with larger counts to disproportionately skew the overall score when being averaged across classes.

---

To address these issues, we adopt precision, recall, and Fi-score, whereby offering normalized, per-class insights. We define per-image, per-class counting components follows: let Cgt denote the ground truth count and Cpred the predicted count for class in an image. Then, True Positives (TP)=
min(Cyts Cpred), False Positives (FP) = max (0, Cpred Cgt) ) for over-counting. and False Negatives (FN) ma Cgt Cprea) for under-counting. These definitions enable standard calculation of precision, recall, and Fi-score, aggregated across images per class and then averaged for final score.
Rethinking metries for confidence-free detectors Evaluating generative models like Florence- 2 [83] or Qwen2.5-VL (2). which output detections without confidence scores, poses a challenge for standard metrics. Average Precision (AP) [12] relies on confidence scores to rank predictions and generate precision-readt curves. Without such ranking, standard AP is ill-defined. Furthermore, practical applications often involve filtering predictions with fixed threshold, treating all remaining detections equally 51)
To resolve these issues and ensure fair comparison, we adopt confidence-free metries: mean Fi Fs-sc (mF1) and mean Average Precision with no confidence (mAP me [36]. mF, measures performance at single operating point, suitable for fixed-threshold deployment. mAP ne adapts AP by assigning maximum confidence to all predictions. For confidence- models, these metrics are computed directly (Results in Table TOp. For conventional detectors (e.g., [64] 4) that provide scores, when the confidence threshold is swept from (step 0.02), the threshold maximizing mF, (using an IoU threshold of 0.5) across categories is selected, and the corresponding cusp score is reported.
Evaluating open-ended and open-subelass settings In open-ended and open-subelass settings, LVLMs may generate category names (e.g.. "car"" that differ textually from ground truth labels (e.g.. "vehicle"). To handle this synonymy during evaluation, we adopt semantic similarity matching by following established protocols [40, [60]. Specifically,
encode generated categories and ground truth categories using the GeoRSCLIP [95 text encoder with the template "' satellite image of {category>"' A generated category name is deemed equivalent to ground truth category if their embedding cosine similarity exceeds 0.95. This allows predicted objects associated with the generated name to be accurately evaluated against the matched ground truth category.
4 InstructSAM
Addressing the challenges of instruction-follovii domain gaps, and threshold sensitivity in remote sensing object recognition, we propose a training-free framework named InstructSAM. It decomposes InstructCDS into three synergistic steps: instruction based object counting using an LVLM, class- agnostic mask proposing via SAM2, and novel counting-constrained constrained mask-label matching procedure. This approach avoids the costly model training and threshold tuning, offering efficient and robust performance.
4.1 Instruction-oriented object counting with LVLM
As illustrated in Section 3, accurately interpreting user intent in remote sensing requires handling dataset-specific rules and semantic ambiguities, which the simple category prompts could fail to capture. We leverage state-of-the--e LVLMs (e.g.. GPT-4o (24,, Qwen2.5-Ve VL. (2] for this task. Inspired by (20), utilize structured prompts in JSON format, which allows easy integration of dataset-specific instructions alongside the core Task (detailed in Appendix Given an image and detailed prompt P, the LVLM acts counter, outputting the target categories {cal,) } and their
M corresponding counts (nwm,) ) present in the image: {cat,j, num, ) j= WWWM-Coonterr. P).
4.2 Class-agnostic mask proposing
Concurrent with counting, SAM2 (63] is employed to generate high-quality, class-agnosti object masks for its strong generalization to remote sensing imagery [75] 81]. Using its automated mask generation mode prompted by regular point grid, we obtain dense set of masks proposals {mask;) p To enhance recall for small objects, mask generation also applied to image crops (detailed in Appendix C.4).
5

---

Counting Constralnts (sun,))
Visual Information
Semantic Information
Quanbiative Information
Structured Counting Prompt
LMLM
Counter
Object Counts
bascbaal
Template
[atom,
Fai
&
CLIP
Stane
Pruposer
Matched Pains
Imago
Masi
[mask,]'>
X (2-sy)
[[muask,.ott out,) xxy 1]
Proposals
Crup
[vyliva
Figure 3: The InstructSAM framework. Given an input image and a structured counting prompt, the LVLM Counter extracts target categories {eat,} (semantic info) and their counts {num,}) } (quantitative info). Concurrently, the Mask Proposer generates mask proposals (masky (visual info), CLIP model computes the similarity matrix S between mask embeddings (from scaled crops) {v;} and category embeddings {ty}- Finally, the Binary Integer Programming (BIP) Solver optimally assigns categories to masks by maximizing summed similarity, subject the counting constraints, yielding the final recognition results.
4.3 Mask-label matching with counting constraints
A key innovation of InstructS AM is reframing object detection and segmentation constrained mask-label matching problem, by integrating the outputs from previous steps. Rather than using fragile confidence thresholds [10]51), utilize global counts {numy) derived from the LVLM to constrain the assignment semantic labels {caty} to visual mask proposals {mask;).
Given N mask proposals and M target categories with counts, we compute semantic similarity ma- trix S E NMM where 8y represents the cosine similarity between the CLIP (95] image embedding of patch cropped around mask; (scaled by 1.2 for context) and the text embedding of cat, (using the template "a satellite image of {category)")) We then seek binary assignment matrix X E {0, 1].xM, where Tij assigns mask; to catj. , by solving the Binary Integer Programming (BIP) problem:
min
([-8i)) Fij
(I)
i= j=1
M
s.t.
Σ
Tij S 1,
ViE (1....,
(2)
j=1
M
Σ
Dij = reumjs
Vj E (l....,
M}.
if
Σ j=1
num,
(3)
Fíj N,
N
numj
j=1
j=1
where constraint 22) ensures each mask is assigned to at most one category. Constraint 33) enforces that the number of assigned masks for each category matches the counts provided by the LVLM. Constraint (4) handles cases where the number of proposals is less than the total target count, ensuring all proposals are assigned.
As depicted in Figure this BIP formulation elegantly fuses visual information, semantic information, and quantitative information. The visual information comes from the CLIP embeddings of mask proposals (u; which contribute to sij. Semantic information is derived from the categories' sentence embeddings } also contributing to 8ij. Quantitative information from object counts (nuwn serves as constraints in 3 The problem is efficiently solvable using open-source BIP solvers like Pul.P [53] The resulting non-zero entries in X define the final set of recognized objects (mask; cat,) | 1}.

---

Table 1: Zero-shot performance comparison on EarthInstruct under open open- vocabulary setting.
NWPU-VHR-10
DIOR
Method
Cnt Fj
Box Fi
Mask F₁
IoU
Cnt F
Box Fi
Mask Fj
IoU
Grounding DINO 45
14.9
14.0
10.7
6.0
OWLv2 [5]
39,4
27.2
23,4
14.3
Qwen2.5-VL 2
68.0
36.4
52.0
27.8
GSNet [88]
1.3
6.4
0.0
0.3
SegFarth-On
3.5
12.1
1.2
6.7
InstructSAM-Qwen InstructSAM-GP440
73.2
38,9
23.7
12.1
59.3
24.0
18.5
83.0
41.8
26.1
14.8
79.9
29.1
28.1
20.2
5
Experiments
5.1
Implementation
We implement InstructSAM using GPT-4o- 2022--11-22 (24) (short for InstruceSAM-GPT40
Qwen2.5-VL-7B [2] (short for InstructSAM-- as the LVLM counter, AAM2-hiera-largo [63 for mask proposal, and GeoRSCLIP-VTT 95 for similarity computation. For the open- open- -occbuaayy setting, we follow previous works 77 21 90 to split base and novel classes, and report mF1, mean Insert-over-nnion (mloU), mAP ac. For open-subclass setting, set two parent class "means of transport" and "sports fields". We compare InstructSAM with a wide range of models, whose training data and abilities are listed in Table 6
5.2 Results EarthInstruct
Open-vocabulary setting We report mean metrics across classes for generic methods [45] 51
and remote sensing open-vocabulary segmentation models [8 32 trained on broader vocabularies
in [ Zero-shot performance on novel classes for models trained on base classes [90] [79 21]
appears Table [0] Models using novel class images trained entire detection datasets evaluated on two additional datasets (Table IT..
Table shows InstructS AM (esp. with GPT-4o) leading in all tasks with top counting metrics. novel classes (Table TO), InstructSAM -weenn achieves superior or competitive mAP, against specialized models. This underscores InstructSAM's training-free free, couting--onstrriin matching advantage over traditional or fine-tuned approaches.
Open-ended setting Table sum marizes the results under open-ended setting. InstructSAM consistently achieves equivalent or higher Fi- scores than remote sensing-specific methods, including those trained on grounded description tasks (49 68)
Table 2: Performance comparison EarthInstruct under open-ended setting.
NWPU-VHR
DIOR
Method
Mask Fi
Cut Fi
Mask
Qwen2
2
48.6
32.0
36.6
21.7
GPT-4o
OWL
51
32.6
24.0
30.6
21.0
Notably, InstructSAM surpasses LAE-
SkySeneGPT
34.9
1.7
39.2
6.5
Label (56) by leveraging global view
FarthDial 71
47.6
LAE-Label 56
46.2
27.3
24.1
23.3
11.5
10.4
of the image to accurately predict ob- ject categories. While the absence of class-specific instructions in this set- ting limits further gains, InstructSA. still demonstrates robust performance (Figure
GooPixel
68
40.8
29.9
29.1
21.4
13.8
14.7
InstnccSAM-Qwen InsmicSAAM-PP44
55.2
29.6
28.5
33.2
14.3
14.4
Open-subclass setting Table shows that InstructSAM outperforms or matches Qwen2.5-V -L across both parent classes. When prompted with categories identified by GPT-4o, OWLv2 performs strongly for "means of transport" but struggles with "sports fields". likely due to the prevalence of transport-related categories in natural image datasets. These findings are consistent with the open-vocabulary results, where generic detectors, such as Grounding DINO and OWI, struggle with remote sensing categories except for airplane, vehicle, and ship.
7

---

Table 3: Performance comparison on EarthInstruct under open-subelass setting. 'S' denotes the parent category "sports field",, and 'T' denotes "means of transport".
IWPU-VHR-10
DIOR
Method
Cnt
Box F;
Mask
Cnt
Box Fj
Mask
S
T
S
T
S
T
S
T
S
T
S
T
Qw
2
54.1
48.9
32.4
42.2
52.2
51.8
34.0
39.2
GPT-4o 24
OWL [ST]
40.3
68.0
19.8
65.9
41.5
73.6
27.6
70.9
InstructS AM- Qwen
50.9
55.3
33.5
41.9
33.0
39.7
40.4
67.4
22.2
49.3
22.6
49.5
InstructSAM 40
84.2
60.6
46.9
44.2
45.8
41.6
82.2
53.6
40.9
38.3
40.2
38.6
Ground Trub
SMM
OWLy2
Qwen2.5-V 5-VL
(a) Open-vocabulary setting
InstrucdSAM
Qwen25-VL
LAb-Label
In
Owen2 5-VL
(b) Open-ended setting
(c) Open-subclass setting
Figure 4: Qualitative comparison of object detection and segmentation methods on DIOR dataset. InstructSAM consistently outperforms other methods in generating accurate bounding boxes and segmentation masks across various settings. More qualitative results can be found in Appendix F
GeoPixel R7w0.43
Court ng Time Proporing
SkysenseGPT P
Qwer2 V
ReaFart
Liblabd
Inference Time (secands)
80
N'=0.99
60c
LAE-Labal p'=0.99
Earth Dial 20.8 tw D.
InstructSAM R-+0.01
Number of Predicted Objects
Figure 5: Inference time as a function of bounding box count for InstructSAM- -wen and baseline methods. Solid lines indicate lincar regressions, and scatter points represent individual samples. The shaded regions for InstructSAM illustrate the time composition of different processing steps. Experiments conducted on RTX 4090 GPU.

---

Inference time analysis Figure compares inference times for open-ended methods using 7B LLMs. Mask proposal dominates the runtime of InstructSAM, whereas the PuLP optimizer solves the BIP problem in 0.07 seconds. InstructSAM maintains nearly constant inference time, while other methods scale linearly with the number of objects. Unlike representing bounding boxes as natural language tokens, InstructSAM reduces output tokens by 89% and total inference time by 32% compared to Qwen2.5VVL. This advantage will grow more pronounced as the model size scales up, highlighting the efficiency of our framework.
Table 4: Open vocabulary counting performance (Precision/Recal//F-soo
Table 5: Ablations on model scaling, MP recall is class-agnostic recall of mask proposals.
Method
Add Instr
NWPU
NWPU
DIOR
LVLM
Mask Proposal
MP Recall
Box
vehicle
CLIP
Faster RCNN
15/49/23
77/77/33
91/74/81
GPT-4o
SAM241
DFN2R-1 L
82.4
41.3
GPT-4o
SAM2-L
RemoteCLIP-1
82.4
45.7
Qwen2.5VL
11/75/19
80/72/72
73/53/66
GPT-4o
SAM2-L
SkyCLIP-L
82.4
45.8
Qwen2.5VL
28/70/41
82/71/73
83/51/59
GPT-4
SAM2-L
SkyCLIP-B
45.2
GPT-4o
1/80/20
68/79/67
87/65/72
GPT-4o
SAM2-S
SkyCLIP-B
77
79.1
44.1
GPT-4o
75/68/71
86/75/80
Qwen2.5VL
SAM2-S
SkyCLIP-B
77
79.1
40.6
5.3
Ablation studies
Prompt design Taalle how additional instructions enhance object counting, particularly for categories with ambiguous or dataset-specific annotation rules. Initially, DIOR-trained Faster-RCNN and LVLM counters exhibit low vehicle precision on NWPU-VHR-10. Explicit annotation rules in instructions significantly boost vehicle precision for Qwen2.5-VIL and GPT-4o, and also improve mF by 3% and 8%, respectively, on DIOR. Contrary to 992 these results show that capable foundation models with precise, instruction-driven prompts truly empower LVLMs to match or exceed closed-set model performance.
Model generalization and scaling To assess InstructSAM's generalization and scalability, we ablate LVLM counters, mask proposers, and CLIP models for the open-vocabulary detection (OVD) task on NWPU- VHR-U0 3). InstructSAM consistently benefits from CLIP models fine-tuned on remote sensing data [43 over generic CLIP 133. whereby yielding higher Box Fi Fcoscss.. Performance improves with larger model components, demonstrating the framework's 's scalability. Notably, even with a smaller SAM2-S and SKyCLIP-B, InstructS coupled with Qwen2.5-V VI. (40.6 Box Fi) outperforms direct detection using Qwen2.5-VI. alone (36.4 Box Fi), underscoring the efficacy of our approach.
Mask-label matching with counting constraints Using fixed thresholds to filter CLIP predic- tions 10] has inherent limitations. Figure the green and blue solid curves show InstructS AM without matching, where predictions are filtered by similarity score (CLIP score) threshold before evaluation. Optimal thresholds vary widely across categories. As a result, single global value
metrics InstructSAM
Par-class F1-scare of InstructSAM w/o Matching
molfre Iilnnsss MiMuzeng
80
HeAP IootuttSAN we Saiching
trdt Hathin
45
MAPnc/mF1 (9%
60
Fi-score (%)
witicce
30
80
20
0.200
0.225
0.250
0.275
0.300
0.200
0.225
0.250
0.275
0.300
Threshold
Threshold
Figure 6: Threshold sensitivity analysis in opca-uocayulary setting. Left: Impact on mean metrics. Right: Category-specifee F, Scores.. Dashed lines indicate optimal thresholds maximizing mF,

---

performs poorly, and the results are highly shreshold-eensitvve,, which is consistent with the findings matching removes this dependency by
[3]. In contrast, InstructSAM's dynamically assigning predictions according to the estimated counts, yielding stronger performance in multi-class and open-world settings.
5.4 Error analysis of OVD task
Error identification reveals distinct error patterns across methods (Figure 7 OWLv2 primarily suffers from classification errors, while Qwen2.5-VL. shows improved classification but struggles with missed detections. InstructSA.M-On 4o benefits from SAM2's s localization capabilities, though the background confusion persists due to GeoRSCLIP's scene-focuse" training, which prioritizes broader contexts over individual objects.
Miss
Loc
Cls
Loc
Miss
Miss
Bkg
Loc
Bkg
Bkg
(a) OWLv2
(b) Qwen2.5-VL (c) InstructSAM
Figure 7: Error distribution of OVD task on WWPU-VHR- 10 dataset. Pie charts show proportions of
classification (Cls), localization (Loc), background (Bkg), and missed detection (Miss) errors models.
6 Limitations
InstructSAM builds on pre-trained foundation models, and performance inherits their capabilities and biases. For instance, SAM2 can miss the full extent of objects with intricate geometry (e.g.. basketball court in Figure 4 CLIP models finetuned on scene-level remote sensing image- text pairs perform suboptimally when aligning object-level crops with semantic cues. More generally, foundation models such as GPT-4o and SAM are trained primarily on optical imagery and struggle to count segment objects in synthetic aperture radar (SAR) images.
Future work can mitigate these limitations by (i) incorporating foundation models trained on semanti- cally diverse, multi-modal remote sensing data and (ii) leveraging advanced class-agnostic region proposers [84). Extending InstructSAM to semantic segmentation would further benefit the earth observation community. Although InstructSAM is only first step, we aim to use it as automated labeling engine to annotate at scale, enabling large, semantically rich object-recognition datasets.
7 Conclusion
In this paper, we introduce InstructCDS for instruction-driien object counting, detection, and seg- mentation, along with EarthInstruct, the first benchmark for this task in remote sensing domain. Our training-fie free InstructSAM framework integrates LVLMs, SAM2, and domain-specific CLIP to handle instruction-oriented scenarios with counting constrained mask-label matching. Experiments demonstrate that InstructSAM outperforms specialized baselines while maintaining near-constant inference time regardless of object counts. As the first approach extending instruction-oriented detection to the broader InstructCDS paradigm, InstructS will benefit from advancements in both remote sensing foundation models 442
85 86 67 69 and general-purpose models [74] 98 48)
paving the way for more scalable, instruction-driven earth observation data analysis.
Acknowledgement
This work was partly supported by the National Farth Observation Data Center Research Project (Grant No. 8020A), National Natural Science Foundation of China (62506229), and Natural Science Foundation of Shanghai (252R1402268).
10

---

References
[1] Kumar Ayush, Burak Uzkent, Kumar Tanmay, Marshall Burke. David Lobell, and Stefano Ermon. Efficient poverty mapping from high resolution remote sensing images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12-20, 2021.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenhin Ge, Sibo Song. Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. /3923, 2025.
[3] Marshall Burke, Anne Driscoll, David B Lobell, and Stefano Ermon. Using satellite imagery to understand and promote sustainable development. Science, xe566e55,,,,,,6, 2021.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tal tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-338 3368-,56,.222 2021.
[6] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni. Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language- image model. arXiv preprint arXiv:2209.067. 2022.
[7] Zhe Chen, Weiyun Wang, Yue Cao. Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye. Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and time scaling. arXiv preprint arXiv:2412.05271, 2024.
[8] Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye. Zhangwei Gao, Erfci Cui, Wenwen Tong, Kongzhi Hu, Jiapeng I Luo, Zheng Ma, al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(1))22001 2024.
[9] Gong Cheng, Junwei Han, Peicheng Zhou, and Lei Guo. Multi-class geospatial object detection and geographic image classification based on collection of part detectors. ISPRS Journal of Photogrammetry and Remote Sensing, 98:11 19-132, 2014.
[10] Tianheng Cheng, Lin Song. Yixiao Ge. Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901-16911, 2024.
[11] Seokju Cho, Heeseong Shin, Sunghwan Hong, Amurag Arnab, Paul Hongsuck Seo, and Seungryong Kim. Cat-seg: Cost aggregation for open- vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 4113-4123, 2024.
[12] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303 -38, 2010.
[13] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. In The Twelfth International Conference on Learning Representations, 2024.
[14] Yuqian Fu, Yu Wang, Yixuan Pan, Lian Huai, Xingyu Qiu. Zeyu Shangguan, Tong Liu, Yanwei Fu, Luc Van Gool, and Xingqun Jiang. Cross-domain few-shot object detection via enhanced open-set object detector. In European Conference on Computer Vision, pages 247-264. Springer, 2024.
[15] Guangshuai Gao, Qingjie Liu, and Yunhong Wang. Counting from sky: A large-scale data set for remote sensing object counting and benchmark method. IEEE Transactions on geoscience and remote sensing, 59(5):3642-3655, 2020.
[16] Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1 13921, 2021.
[17] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 5356-5364, 2019.
[18] Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, and Matthew Gaston. Creating xbd: A dataset for assessing building damage from satellite imagery. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 10 10-17,, 2019.

---

[19] Mordechai Haklay and Patrick Weber. Openstreetmap: User-generated street maps. IEEE Pervasive computing, 7(4):12-18, 2008.
[20] Jia He, Mukund Rungta, David Koleczek, Arshdeep Sekhon, Franklin X Wang. and Sadid Hasan. Does prompt formatting have any impact on Ilm performance? arXiv preprint arXiv:241 10541, 2024.
[21] Shiqi Huang, Shuting He, and Bihan Wen. ZoRI: Towards discriminative zero-shot remote sensing instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3724-3732, 2025.
[22] Ziyue Huang, Yongchao Feng, Shuai Yang, Ziqi Liu, Qingjie Liu, and Yunhong Wang. Openrsd: Towards open-prompts for object detection in remote sensing images. arXiv preprint arXiv:2503.061 2025.
[23] Drew A Hudson and Christopher D Manning. Gqæ: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[24] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXh: 2024.
[25] Jeremy Andrew Irvin, Emily Ruoyu Liu, Joyce C. Chen, Ines Dormoy, Jinyoung Kim, Samar Khanna, Zhuo Zheng, and Stefano Ermon. TEOChat: A large lisoon-lagguage assistant for temporal carth obscrvation data. In The Thirteenth International Conference on Learning Representations, 2025.
[26] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1780-1790, 2021.
[27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-74 --3,, 2017.
[28] Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abbijit Das, Salman Khan, and Fa- had Shahbaz Khan. Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 27831-27840, 2024.
[29] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.
[30] Chunyuan Li, Haotian Liu, Liunian Li. Pengchuan Zhang. Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al, Flevater: A benchmark and toolkit for evaluating language-augmentee visual models. Advances Neural Information Processing Systems, 35:9287-9301, 2022.
[31] Jiahao Li, Jiancheng Pan, Yuze Sun, and Xiaomeng Huang. Semantic-aware ship detection with vision- language integration. arXiv preprint arXiv:2508.159) 2025.
[32] Kaiyu Li, Ruixun Liu, Xiangyong Cao, Xueru Bai, Feng Zhou, Deyu Meng. and Zhi Wang. Segearth- ov: Towards training-fir free open- voen-uocabylary segmentation for remote sensing images. arXiv preprint arXiv:2410.01766 2024.
[33] Kaiyu Li, Zepeng Xin, Li Pang, Chao Pang, Yupeng Deng, Jing Yao, Guisong Xia, Deyu Meng. Zhi Wang. and Xiangyong Cao. Segearth-rl: Geospatial pixel reasoning via large language model. arXiv preprint arXiv:2504.0 09644, 2025.
[34] Ke Li, Gang Wan, Gong Cheng, Liqiu Meng. and Junwei Han. Object detection in optical remote sensing images: A survey and new benchmark. ISPRS journal of photogrammetry and remote sensing, 159:296-307, 2020.
[35] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965-10975. 2022.
[36] Qingyun Li. Yushi Chen, Xinya Shu, Dong Chen, Xin He, Yi Yu, and Xue Yang. A simple aerial detection baseline of multimodal language models. arXiv preprint arXiv:2501.0972) 2025.
12

---

[37] Xiang Li, Jian Ding, and Mohamed Elhoseiny. VRSBench: A versatlee vision-language benchmark dataset for remote sensing image understanding. In Advances in Neural Information Processing Systems, volume 37. pages 3229-3242 2024.
[38] Yan Li, Weiwei Guo, Xue Yang, Ning Liao, Dunyun He, Jiaqi Zhou, and Wenxian Yu. Toward open vocabulary aerial object detection with clip-activated stdent-teacher learning. In European Conference on Computer Vision, pages 431 448. Springer, 2024.
[39] Yan Li, Weiwei Guo, Xue Yang, Ning Liao, Shaofeng Zhang. Yi Yu, Wenxian Yu, and Junchi Yan. Exploiting unlabeled data with multiple expert teachers for open vocabulary acrial object detection and its orientation adaptation. arXiv preprint 2oXi::4411.00053 2024.
[40] Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13958-13968, 2024.
[41] Zhiwei Lin, Yongtao Wang, and Zhi Tang. Training- Free Open- Ended Object Detection and Segmentation via Attention as Prompts. In Advances in Neural Information Processing Systems, volume 37, pages 19588-696088 2024.
[42] Chenyang Liu, Keyan Chen, Rui Zhao, Zhengxia Zou, and Zhenwei Shi. Text2carth: Unlocking text-driven remote sensing image generation with a global-scale dataset and a foundation model. arXiv preprint 1rXiv:2501.00893 2025.
[43] Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Remoteclip: A vision language foundation model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[44] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023.
[45] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38 55. Springer, 2024.
[46] Sihan Liu, Yiwei Ma. Xiaoqing Zhang, Haowei Wang, Jiayi Ji, Xiaoshuai Sun. and Rongrong Ji. Rotated multi-scale interaction network for referring remote sensing image segmentation. In P Proceedings of the IEFF/VV Conference Computer Vision and Pattern Recognition, pages 26658-26666 2024.
[47] Yanxing Liu, Jiancheng Pan, and Bingehen Zhang. Control copy-paste: Controllable diffusion- augmentation method for remote sensing few-shot object detection. arXiv preprint arXiv:2507.2/816 2025.
[48] Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono-intervll: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training arXiv preprint arXiv:2410.08202 2024.
[49] Junwei Luo, Zhen Pang. Yongjun Zhang, Tingzhu Wang, Linlin Wang. Bo Dang. Jiangwei Lao, Jian Wang. Jingdong Chen, Yihua Tan, et al. Skysensegpt: A fine- gininee instruction tuning dataset and model for remote sensing vision-language understanding. arXiv preprint arXiv:2406.1011 2024.
[50] Chenlin Meng, Enci Liu, Willie Neiswanger, Jiaming Song. Marshall Burke, David Lobell, and Stefano Ermon. Is-count: Large-scale object counting from satellite images with covariate- based importance sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12034- 12042, 2022.
[51] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling ooca-vocyy object detection. Advances in Neural Information Processing Systems, 36:72983-73007. 2023.
[52] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection. In European conference on computer vision, pages 728-758-77 Springer, 2022.
[53] Stuart Mitchell, Michael OSullivan, and lain Dunning. Pulp: linear programming toolkit for python. The University of Aucklond, Auckland, New Zealand, 65:25, 2011.
13

---

[54] Dilxat Muhtar, Zhenshi Li, Feng Gu, Xueliang Zhang, and Pengfeng Xiao. Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. In European Conference on Computer Vision, pages 440-457. Springer, 2024.
[55] Ruizhe Ou, Yuan Hu, Fan Zhang, Jiaxin Chen, and Yu Liu. Geopix: Mudti-modal large language model for pixel-leved image understanding in remote sensing, arXiv preprint arXiv:2501. 068288, 2025.
[56] Jiancheng Pan, Yanxing Liu. Yugian Fu, Muyuan Ma. Jiahao Li, Danda Pani Paudel, Luc Van Gool, and Xiaomeng Huang. Locate anything on earth: Advancing open-vocabulary object detection for remote sensing community. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6281-6289) 2025.
[57] Jiancheng Pan, Yanxing Liu, Xiao He, Long Peng, Jiahao Li, Yuze Sun, and Xiaomeng Huang. Enhance then search: An augmenntaiion-earchh strategy with foundation models for cross-domain few-shot object detection. Proceedings of the Computer Vision and Puttern Recognition Conference, pages 1548-1556, 2025.
[58] Chao Pang. Xingxing Weng, Jiang Wu, Jiayu Li, Yi Liu, Jiaxing Sun, Weijia Li. Shuai Wang. Litong Feng. Gui-Song Xia, al. Vhm: Versatile and honest vision language model for remote sensing image analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6381-6388, 2025.
[59] Claudio Persello, Jan Dirk Wegner, Ronny Hänsch, Devis Tuia, Pedram Ghamisi, Mila Koeva, and Gustau Camps-Valt Valls. Deep learning and carth observation to support the sustainable development goals: Current approaches, open challenges, and future opportunities. IEEE Geoscience and Remote Sensing Magazine, 10(2):172-200, 2022.
[60] Renjie Pi, Lewei Yao, Jianhua Han, Xiaodan I. Liang, Wei Zhang, and Hang Xu. Ins-detclip: Aligning detection model to follow human language instruction. In The Twelfth International Conference on Learning Reprexentations, 2024.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-88 8763. PmLR, 2021.
[62] Viresh Ranjan, Udbhay Sharma, Thu Nguyen, and Minh Hoai. Learning count everything. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3394-3403, 2021.
[63] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloc Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint urXiv:2408.000 007144 2024.
[64] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. IEEE transuctions on pattern analysis and machine intelligence, 39(6):1137-1 1149, 2016.
[65] Tianhe Ren. Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, al. Dino-x: A unified vision model for open- object detection and understanding. arXiv preprint arXiv:241 14347, 2024.
[66] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojic Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 26374-26383, 2024.
[67] Rajat Sahay and Andreas Savakis. On aligning sam remote sensing data. In Geospatial Informatics XIV, volume 13037, pages 10-18. SPIE, 2024.
[68] Akashah Shabbir, Mohammed Zumri, Mohammed Bennamoun, Fahad S Khan, and Salman Khan. Geopixel: Pixel grounding large multimodal model in remote sensing. arXiv preprint 200.22501.13925. 2025.
[69] Zhe Shan, Yang Liu, Lei Zhou, Cheng Yan, Heng Wang, and Xia Xie. Ros-sam: High-quality interactive segmentation for remote sensing moving object. arXiv preprint arXiv:2503.1200 12006, 2025.
[70] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A lcale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pages 8430 (8000883,,000 2019.
14

---

[71] Sagar Soni, Akshay Dudhane, Hiyam Debary. Mustansar Fiaz, Muhammad Akhtar Munir. Muhammad So- hail Danish, Paolo Fraccaro, Campbell D Watson, Levente Klein, Fahad Shahbaz Khan, et al. Earthdial: Turning multi-sensory earth observations to interactive dialogues. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14303-14313, 2025.
[72] Hao Su, Shunjun Wei, Min Yan, Chen Wang, Jun Shi, and Xiaoling Zhang. Object detection and instance segmentation in remote sensing imagery based on precise mask r-cnn. In IGARSS 2019-2019 IEEE International Geoxcience and Remote Sensing Symposium, pages 1454-1457. IEEE, 2019.
[73] Xian Sun, Peijin Wang, Zhiyuan Yan, Feng Xu, Ruiping Wang, Wenhui Diao, Jin Chen, Jihao Li, Yingehao Feng, Tao Xu, et al. Fairlm: A benchmark dataset for fine-grained ohject recognition high-resolution remote sensing imagery. ISPRS Journal of Photogranmetry and Remote Sensing, 184:116-1300 130, 2022.
[74] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard. Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 250: 19786, 2025.
[75] Di Wang, Jing Zhang. Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. Samrs: Scaling- up remote sensing segmentation dataset with segment anything model. Advances in Neural Information Processing Systems, 36:8815-8827, 2023.
[76] Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Conghui He, and Dahua Lin. V3det: Vast vocabulary visual detection datasct. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11844-19854 2023.
[77] Zhecheng Wang. Rajanie Prabha. Tianyuan Huang. Jiajun Wu, and Ram Rajagopal. Skyscript: A large and semantically diverse on Artificial Intelligence, volume 38, pages 5805-5813, 2024.
dataset for remote sensing. In Proceedings of the AAAI Conference
[78] Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui Gui-Song Xia, and Xiang Bai. isaid: A large-scale dataset for instance segmentation in aerial images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 28-37, 2019.
[79] Guoting Wei, Xia Yuan, Yu Liu, Zhenhao Shang, Kelu Yao, Chao Li, Qingsen Yan, Chunxia Zhao Haokui Zhang, and Rong Xiao. Ova-detr: Open vocabulary aerial object detection using image-text alignment and fusion. arXiv preprint arXiv:2408. 12246, 2024.
[80] Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary learning: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7):5092 (595),5092-5113, 2024.
[81] Qiusheng Wu and Lucas Prado Osco. samgeo: python package for segmenting geospatial data with the segment anything model (sam). Journal of Open Source Software, 8(89):5663,, 2023.
[82] Zijing Wu, Ce Zhang, Xiaowei Gu, Isla Duporge, Lacey F Hughey, Jared A Stabach, Andrew K Skidmore, Grant C Hoperaft, Stephen Lee, Peter M Atkinson, al. Deep learning enables satellite-based monitoring of large populations of terrestrial mammals across heterogeneous landscape. Nature communications, 14(1):3072, 2023.
[83] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety vision tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 48 18-9,,99, 2024.
[84] Jianlin Xie, Guanqun Wang, Tong Zhang. Yikang Sun, He Chen, Yin Zhuang, and Jun Li. Llama- unidctector: A Illma- universal framework for open- vocabulayy object detection in remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 2025.
[85] Zhitong Xiong, Yi Wang, Weikang Yu, Adam J Stewart, Jie Zhao, Nils Lehmann, Thomas Dujardin, Zhenghang Yuan, Pedram Ghamisi, and Xiao Xiang Zhu, Geolangbind: Unifying earth observation with agglomerative vision-language foundation models. arXiv preprint arXiv:2503.062: 2025.
[86] Zhiyuan Yan, Junxi Li, Xuexue Li, Ruixue Zhou, Wenkai Zhang, Yingchao Feng, Wenhui Diao, Kun Fu, and Xian Sun. Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images. IEEE Transactions on Geoscience and Remote Sensing, 61:1-16, 2023.
[87] Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detclipy3: Towards versatile generative open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2744117701, 2024.
15

---

[88] Chengyang Ye, Yunzhi Zhuge, and Pingping Zhang. Towards open-vocabulary remote sensing image semantic segmentation. In Proceedings the AAAI Conference on Artificial Intelligence, volume 39, pages 9436-9444, 2025.
[89] Qihang Yu, Ju He, Xueqing Deng. Xiaohui Shen, and Liang-Chicll Chen. Convolutions die hard: Open- vocabulary segmentation with single frozen convolutional clip. Advances Neural Information Processing Systems, 01522215-32234 2023.
[90] Zhengging Zang, Chenyu Lin, Chenwei Tang, Tao Wang, and Jiancheng Lv. Zero-shot aerial object detection with visual description regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6926 6934, 2024.
[91] Valérie Zermatten, Javiera Castillo-Navarro, Diego Marcos, and Devis Tuia. Learning transferable land cover semantics for open vocabulary interactions with remote sensing images. ISPRS Journal of Photogrammetrry and Remote Sensing, 220:621-636, 2025.
[92] Chenhui Zhang and Sherrie Wang. Good at captioning bad at counting: Benchmarking gpt-4v on earth observation data. In Procedinggs of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 7839-7849, 2024.
[93] Haotian Zhang, Pengehuan Zhang, Xiaowei Hu, Yen- Cen-Chun Chen, Liunian Li, Xiyang Dai. Lijuan Wang. Yuan, Jenq-Nee Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- laion-language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022.
[94] Mingjin Zhang, Qian Xu, Yuchun Wang, Xi Li, and Haojuan Yuan. Mirsam: multimodal vision-language segment anything model for infrared small target detection. Visual Intelligence, 3(1):1-13, 2025.
[95] Zilun Zhang, Tiancheng Zhao, Yulong Guo, and Jianwei Yin. Rs5m and georsclip: A large scale vision- language dataset and a large vision-language model for remote sensing. IEEE Transactions on Geoscience and Remote Sensing, 2024.
[96] Yijie Zheng, Jinxuan Yang. Yu Chen, Yaxuan Wang, Yihang Lu, and Guoqing Li. Beluga whale detection from satellite imagery with point labels. arXiv preprint arXiv:2505. /2066, 2025.
197] Yue Zhou, Mengcheng Lan, Xiang Li, Yiping Kc, Xue Jiang, Litong Feng, and Wayne Zhang. Ge- oground: A unified large vision-language model. for remote sensing visual grounding. arXiv preprint arXiv:2411.11900 2024.
[98] Jinguo Zhu, Weiyun Wang, The Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Interwl3:: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 1044:99944.0499 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outlined the challenge in language- guided remote sensing object recognition and the paper's contribution of introducing the InstructCDS task, EarthInstruct benchmark, and InstructSAM framework.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [NA]
Justification:
This paper focuses on the experimental demonstration of the proposed frame- work, and does not include theoretical results proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The InstructS framework is clearly described in Section 4 and the imple- mentation details to reproduce the main experimental results are thoroughly outlined in
Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The source codes of InstructSAM, and evaluation scripts are provided in the supplemental materials zip file. The complete source code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wwtpcc/pccppcpcccceooooooop http:/gguuddes/ooeeuubbbiiiii weneiisssonPoPiioio
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The dataset split of the EarthInstruct benchmark is specified in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance analysis on open-vocabulary counting task in NWPU- VHR-10 10 is presented in Appendix [D.3 Error bars for other experiments are not reported due to the inference cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The comparison of inference time with other methods is presented in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The paper fully conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Both positive societal impacts and potential negative societal impacts are discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all used datasets and pre-trained models. The EarthInstruct benchmark is built upon datasets with CC BY-NC 4.0 license.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The codes, along with the documents of the proposed benchmark and methods is provided in the supplemental materials.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The LLM is key component of the proposed InstructSAM framework. The usage of LLMs is described in Section 4.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
23

---

Technical Appendices
A
Comparison with related work
Remote sensing imagery captures vast, diverse range of contexts. Exemplified by the SkyScript dataset [77), curated from Google Earth and OpenStreetMap [19], featuring 29,000 distinct semantic tags. Despite this richness, annotated training datasets for remote sensing object recognition typically span only a few dozen categories (Table 6 This lexical scarcity hinders effective alignment of semantic and visual information compared generic models, which are typically trained millions of human-annotated images and near-infinite vocabularies sourced from the Internet. Similarly, instruction-followe following datasets for remote sensing, such FIT-RS (49) and GeoPixelID [88. also
built upon manually annotated data, often inheriting these vocabulary-specife limitations.
Regarding model capabilities, remote sensing LVLMs typically fail to follow complex object recogni- tion instructions as illustrated in Figure Nevertheless, SkysenseGPT [49] and GeoPixel [68], trained scene graph generation pixel-grounde conversation generation tasks, deliver comprehensive object detection or segmentation outputs, effectively serving a similar purpose of open-ended object recognition.
Table 6: Comparison of segguage-guideds object recognition methods. We summarize the model, publication, training data, dense segmentation capability, and support for instruction-oriente object detection -occbuvoyay detection (OVD), open-ended detection (OED), and open-subclass detec- tion (OSD)). Green v indicates support, red X indicates not supported.
Object Recognition Training Data
Dense Segmentation
Method
Publication
OVD OED OSD
Remote sensing-sppecified methods
DesReg 900
2AAI'24
DIOR
X
X
OVA-DETR 79
arXiv'24
DIOR+DOTA+*
X
X
CASTDet
38
ECCV'24
DIOR
X
X
X
LAE-DINO [56]
AAAI'25
LAE-IM
X
V
X
X
ZORI 21
AAAI'25
NWPU/DIOR
V
X
X
GSNet
AAAI'25
LandCover-40k
SegEarth-OV
32
CVPR'25
Image-level Training
SkysenseGPT
49
arXiv'24
FIT-RS
X
GeoPixel
68
ICML'25
GeoPixelD
X
Generic methods
GroundingDINO
ECCV'24
FivODs++GoldG+CapAM
X
X
X
OWLv2 51
NIPS'22
WebLI2B
X
X
Florence-2 83
CVPR'24
FLA-5B
Qwen2.5-VL. 2
arXiv'25
Various sources
X
InsDerCLIP 660
ICLR'24
Object365
VL-SAM 41
NeurIPS'24
Training-free
/
InstructSA.
Ours
Training-free
B Benchmark details
The details of the datasets comprising EarthInstruct follows:
a 10-class dataset for very-high-resolution (VHR) remote sensing object detection. It originally comprises 650 positive images (each containing at least one target object) and 150 negative images (containing no target objects). For constructing FarthInstruct, we utilize the 650 positive images. Instance-level mask annotations were subsequently provided by (72).
NWPU-VHR-I 10

---

User
Provade Provide the coortinatas of all the baseball flelds ground track fields, and tonns courts in the image
There baseball field, 1 ground track, Reidd teanis courts in the crbmaa <ft (<19770--000 45o-00 31s-d 5,0440550000)) (418,70=000 (<19. 7a:.733=0000 25x2200022222111 21 1119 50>/400>), (<19.73xd100
SkysensGGP
53>-dil>))
[1, [1.2.13,44] 141 [1, 26, [1, 11,26,55 19,19699..11 421 13,43,1.558 12,55,197711, [1,57.19,83) 83). 11.98,1992].
TEOChat
[1,04,19,100 (1,8,,11,,,, 100), 11,76,19,94 19,921, 1L.67, 19,83), EL 19.55..99991..... 19.711, 43,1995581, (1.26.1942)
Image Input
(1,10,19,14] (1.10,1111 19,42), A1, 19,59,9 18,711, (167,1988),
Detect objects the image and provide subclass names and their coordinates.
There and buildings, 11 cans, tomni cuurts, 1 groundtracficldd hasehalffiddd building, car, caz, I cac, car: car, car, I car, car, car, car, car, 1 cae cac car, oar, car, car, cac. cac car, car, oar, car, 1 car, 1 car. cac, cac
car,
car,
groundtmecrfield at groundtrckffild center, basebollfield top left. bassballfield top right, groundtraciceeld the bottom left, groundtrckhheld at the bottoe right, I prundtrackfied at the center, 1 groundtractield at the Jeft. yroundlrackcieedd the right, grouddrrcckiied groundrackfield top left...
TBOClat
User
beloug
"sports field"
image. Provide the subelass correspcading coontinates.
SKysenseGPT
objeect coordinates: <tbor((<=10 (=10:022000002211199 9.80211772770000000000 <ttb)
TEOChat
Figure 8: Examples illustrating that SKysenseGPT [49] and TEOChat [25] fail to produce meaningful responses for open-vocate vora-booab,,r, open-ended, and open-subclass prompts. Their responses either lack category name outputs or exhibit looped generation.
DIOR 34 large-scale benchmark for object detection in optical remote sensing images, encompassing 20 object categories. The dataset is split into 5,862 training images, 5,863 validation images, and 11,725 test images. For the EarthInstruct benchmark, we utilize the validation set of DIOR. Mask annotations provided by the SAMRS (75) dataset.
Table provides summary of these datasets, including their spectral bands, spatial resolution ranges, and the specific category splits within EarthInstruct.
Table 8 compares EarthInstruct with other language-guides object recognition datasets and bench- marks. Most current benchmarks [60] 46 37] generate prompts using LLMs or templates based on specific objects in the image, resulting in few negative samples. In addition, referring segmenta-
Table 7: Details of the datasets used in EarthInstruct, including spectral bands, resolution, and category splits for base, novel, and open-subelass (sports field, means of transport) settings. "Res" denotes spatial resolution. "NWPU" stands for NWPU-VHR-10.
Means of Transports
Dataset
Band
Res
Base Class
Novel Class
Sports Field
NWPU
RGB/ Color
0.08-2m
airplane, storage tank, baseball field. tennis_court, track_field, bridge, vehicle
ship, basketball. court, harbor
baseball field, tennis court, basketball_coure court, track_fickl ficld
airplane, ship. vehicle
Infrared
DIOR
RGB
0.3-30m airplane, baseball field, bridge, chimney. expressway service area, expressway toll station, dam, golf field, harbor, overpass, ship, stadium, storage tank, tennis court, train station, vehicle
airport, basketball court, ground track field, windmill
baseball field, basketball court, golf field, ground track field, tennis court, stadium
airplane, ship. vehicle
25

---

Table 8: Comparison between EarthInstruct and other language-guii object recognition benchmark datasets. "OV", "OE", "OS" denote open- voccboocabu,ar, open-ended, and open-subclass, respectively. Goal-Oriented denotes instructions detect objects that achieve certain goals. Content Prior Free means the benchmark does not require image content prior for prompt construction.
Goal- Oriented Instructions
Content
OV Instructions
OE Instructions
OS Instructions
Class
Dataset
Publication
Prior
Num
Free
10D-Bench 60
ICLR'24 24
X
RRSIS-D 46
CVPR'24
x
X
20
VRSBench 37
NeurIPS'24
X
FIT-RS (OD) 49
arXiv'24
X
FIT-RS (SGG) 49
arXiv'24
X
X
v
48
GeoPixelID 68
ICML'25
X
X
V
15
EarthReason
33
arXiv'25
X
28
EarthInstruct
Ours
X
20
tion [4)] and visual grounding [37] benchmarks typically support only single-class singlc-object queries, restricting efficiency in large-scale applications.
EarthInstruct addresses these issues by supporting open- vona-ullabaar..... open-ended, and open- puenlsubelass settings, and uses dataset- dltaset-specifie prompts that encode user intent without relying on image content priors. This enables more comprehensive and efficient evaluation for instruction-drie object recognition.
C Implementation details
This section provides further details the implementation of our InstructSAM framework and other baselines. A critical component of leveraging LVLMs for the InstructCDS tasks is the design of effective prompts. We employ structured prompts in JSON format to systematically guide the LVLM counter. This structured approach allows for clear delineation of the model's Persona, the specific Task to be performed, detailed Instructions for execution, the desired Output format, and illustrative Examples. The Instructions field enables the incorporation of dataset- specific annotation rules and disambiguation of category definitions, which crucial for accurate interpretation the user intension, as discussed in Section 3.1 The subsequent subsections detail the specific prompts used for each setting within the EarthInstruct benchmark.
Here we further clarify how the categories that might have bewildered the model. For example, an "overpass" is defined as "a bridge, road, railway similar structure that is over another road or railway" while a "bridge" is "a structure built to span physical obstacle (such as a body of water, valley, road, railway) without blocking the path underneath It is possible that the LVLM counter will recognize overpasses as bridges since overpass is a subset of bridge. Adding clarification such as "bridge in this dataset refers structure spanning body of water" to the Instructions field will guild the LVLM to count accurately.
C.1 Prompts in open-vocahulary setting
Prompt for NWPU-VHR. dataset:
{
"Persona": "You are advanced AI model capable of understanding and analyzing aerial images.",
"Task"; "Given an input satellite imagery, count the number of objects from specific categories. Provide the results in JSON format where the keys are the category names and the values are the corresponding counts.", "Instructions":
https://en, httpe////eewwwiiedde..... wtt.wwwi//wwiigg
26

---

"Do not count ships vehicles that hard to annotate in the annotated due to the small "The 10 categories in the dataset are: l'airplane', 'ship', "storage_ tank', 'basebal_ 'tennis_court" 'basketball.court' track_field? 'harbor', 'bridge', 'vehicle']", "The spatial resolution of the imagery in the dataset ranges from 0.08 m to m..,
relatively low-resolution images they
size.",
"Harbor is defined as pier to dock ships. If miltiple harbors are visible in the image, count each distinct pier separately."
].
"Dutput format": "[V "[""aateoory\"; counti, \caateoory count2, ... }", "Exampee":
{ ""aipplane:: 2, "ship": 0, "storage_ tank": 3, "baseball_fiee 1, "ttorage_tank": 0, strack_fiedd" 0, "harbor": 6,
"tennis_ 0, "basketball_courr" "bridge": 0, "vehicle": },
"airplane": 5, "ship": 2, "storage_tande 0, "baseball bbasebal__feedd.. field"; 0, "tennis_court": 1, (basketball_courte O, "track 0, "harbor"; 0, "bridge": 1, "vehicle": 10
Prompt for DIOR dataset:
{
"Persona": "You are an advanced AI model capable of understanding and analyzing remote sensing images."
"Task"; "Given input satellite imagery, count the number of objects from format the keys are
specific categories. Provide the results the category names and the values are the corresponding counts.", "Instructions":
"The 20 categories in the dataset are: ['airplane', 'airport', 'baseball field', 'basketball court', 'bridge', 'chimney', 'expressway service area', 'expressway toll station', 'dam', 'golf field', 'ground track field', 'harbor', 'overpass', 'ship', 'stadium', 'storage tank', 'tennis court', 'train station', 'vehicle', 'windmill'j",
"The spatial resolution of the images is 0. 0.3m-30m.",
large area of land where aircraft can take off land. It includes runways and other facilities. Do not count airport if the it is not compeletely visible in the image.",
"Airport
pier to dock ships. If multiple harbors are visible in the image, count each distinct pier separately.". "Expressvay toll station is toll booth at entrance the expressway and spans the road.",
"Harbor is defined
"Expressway service area is rest area along expressway. If it exsits the both sides of the expressway, count them separately.", "Overpass is road crossing over another road. Bridge road spanning river. Distinguish them carefully.",
"If the overpass or bridge is composed of parallel, separate sections (for example, different lanes directions of traffic), each section should be counted individually.",,
"Count every ship and vehicle carefully, even resolution low and the objects are small and dense.",
"If none of the objects among the categories is visible, output JSON object with all categories set to 0"
].
"Dutput format": "{ "category1": count1, "category2": count2, ... }", EExamples; "{ "airplane": 2, "airport"; 0, "baseball field": 0, "basketball court";
o, "bridge": 1, ... }"
"I "airplane": "airport": 0, "baseball field": 2, "basketball court":
6, "bridge": 0, }"
}
27

---

C.2 Prompts in open-ended setting
Prompt for NWPU-VHR-10 dataset:
{
advanced AI model capable of understanding and analyzing remote sensing images.",
"Persona": "You
"Task"; ; "Given an input satellite imagery, count the number of all the visible remote sensing objects. Provide the results in JSON format where the keys the category names and the values are the corresponding counts.", "Instructions":
"The spatial resolution of the imagery in the dataset ranges from 0.08 m to 2 m.",
"Do not count ships vehicles that are too samll and are hard to imagery. category is not visible, do not include it in the output." ],
annotate in the relatively lov-resolution images.",
"Only count objects that are clearly visible
"Dutput format": "[ "category1"; counti, "category2"; count2, }",
" AAsswer::
"Ensure the category names are in singular form", "Provide the counts as integers."
}
Prompt for DIOR dataset:
{
"Persona": "You are an advanced model capable understanding and analyzing remote sensing images.",
"Task" "Given input satellite imagery, count the number of all the visible remote sensing objects scenes. Provide the results in JSON format where the keys are the category names the values are corresponding counts.",
"Instructions": [
"The spatial resolution of the imagery in the dataset ranges from 0.3 m.". resolution is too limited or the scene too dense to accurately count certain objects, exclude those objects from the reaulta.", "Only count objects that are clearly visible in the imagery."
},
"Output format": "[ "category1"; count1, "category2"; count2, ... }", AAsswer;
"Ensure the category names are in singular form", "Provide the counts integers."
C.3 Prompts in open-subelass setting
Prompt for counting "sports field" in NWPU-VHR-10 dataset:
{
"Persona": "You are an advanced AI model capable of understanding and analyzing remote sensing images."
"Task" : "Given input satellite imagery, count the number of objects that belong the parent category **sports field*.. Provide the results JSON format where the keys are the names of the subcategories and the values are the corresponding counts.",
"Instructions"
"The spatial resolution of the images 0.08m-2m." not count objects that are
annotate in the relatively low-resolution images as they are not annotated due to the small size.", ",
"If none of the objects belong to the parent category is visible, output empty JSON object like Y{ *
28

---

J,
"Dutput format": "{ "subcategory1": counti, "subcategory2": count2, ... }", "Answer":
"Ensure the category names are in singular form", "Provide the counts as integers."
}
Prompt for counting "means of transport" in NWPU-VHR-IE dataset:
(
"Persona": "You are an advanced model capable understanding and analyzing remote seneing images."
"Task": "Given an input satellite imagery, count the number of objects that belong to the parent category **means of traneport**. Provide the results in JSON format where the keys the of the subcategories and the values are the corresponding counts.", "Instructions:
"The spatial resolution of the imagery in the dataset ranges from 0.08 m to 2 m.",
count boats land vehicles that hard annotate the relatively res-resolution images as they are not annotated due to the small size.",
"If none of the objects belong to the parent category is visible, output empty JSON object like M{ \
"Output format": "[ "subcategory1": counti, "subcategory2": count2, }", "Answwer;
"Ensure the category names are in singular form", "Provide the counts integers."
Prompt for counting "sports field" in DIOR dataset:
{
"Persona": "You are an advanced AI model capable of understanding and analyzing remote sensing images."
"Task": "Given input satellite imagery, objects that belong to the parent category **sports field**. Provide the results in JSON format where the keys are the names of the subcategories and the values are the corresponding counts.", "Instructions": [
the number of visible
"The spatial resolution
images 0.3m-30m.",
"Return only the categories and counts that meet the visibility and resolution criteria. If none of the objects belong to the parent category is visible, output empty JSON object like Vf \)"
].
"Output format"; "[ "subcategory1"; counti, "subcategory2": count2, ... }", "Answwr";
"Ensure the category names are in singular form", "Provide the counts integers."
Prompt for counting "means of transport" in DIOR dataset:
"Persona": "You are an advanced AI model capable of understanding and analyzing remote sensing images."
"Task": "Given input satellite imagery, count the number of visible objects that belong the parent category the results JSON format where the keys are the names of the subcategories and the values are the corresponding counts.",
of transport**. Provide
"Instructions"
"The spatial resolution of the images is 3.33-30m.
29

---

"Return only the categories and counts that meet the visibility and resolution criteria. If of the objecta belong the parent category is visible, output a empty JSON object like \
,,
"Output format": "[ "subcategory!": countl, "subcategory2": count2, )", "Answwr": :
in singular form", "Provide the counts as integers."
"Ensure the category
]
}
C.4 Hyperparameters of InstructSAM
422 class-agnostic mask proposals generated using SAM22hiera-large [63 in its automatic mask generation mode. The model was configured with the parameters detailed in Table[aacooss both NWPU-VHR-I0 and DIOR. These settings balance the proposal quality and computational efficiency, ensuring a dense set of high-quality proposals, especially for small objects.
Mask proposer As described in Section
Table 9: Hyperparameters SAM2.
Parameter
Value
pred_iou ie__jiouttheer tho_th
0.75
stability_ Secllysssseeeeeeeeee _erreshhrshh
0.75
points_per_sine side
24
crop_ layers
box nms thresh 0.5
LVLM counter For Qwen2.5-VI. and GPT-4o, the temperature is set 0.01 and top_p set at to reduce randomness.
C.5 Implementations of baseline methods
To ensure rigorous and fair comparisons against InstructSAM, baseline methods are implemented or adapted as detailed below. Unless otherwise specified, publicly available pre-trained models and official codebases are utilized, adhering their recommended configurations.
Open- Voen-Vocabulary semantic segmentation methods For SegEarth-OV
32 and GSNet 88 we
follow their prescribed inference procedures. This involves providing the target categories along with an explicit "background" category as input. To derive instance-level proposals from the pixel-level segmentation maps produced by these models, we apply a standard connected component labeling algorithm to the per-class binary masks. This allows evaluation using instance-base -baee metries (Mask Fj).
Open-ended methods
SkysenseGPT 44): We utilize the prompt "[grounding Analyze and describe every detail you identify in the image," which is representative of its training data, and set the maximum output token limit 5000.
GeoPixel [68]: Following the illustrative examples provided in their publication, we employ the prompt "Can you give a thorough description of this image, including interleaved segmentation masks to highlight key objects?"
with the same hyperparameters as InstructSA.M (see Appendix (C.4). For label generation, we replace LAE-Label's 's original InterVL2- 8B and reasoning prompt with the newer Qwwe2.55VVL-7B. guided by a structured prompt for open-ended category identification without explicit reasoning. This change of prompt improves mF, by 4.3% on NWPU-VHR-10 and reduces inference time by 92%. The prompt we use as follows:
LAE-Label [56]: For mask proposing, we use
30

---

{
"Persona"; "You are an advanced model capable understanding and analyzing remote sensing images."
"Task"; "Given input satellite imagery, identify the most likely object
class visible in the image. Provide the result single class name." " "Instructions"
"Consider this is region of interest cropped from a larger remote sensing image.",
"Focus identifying the main object class visible in the cropped region.",
"Be specific with your answer, using single class name."), "Dutput format": "\"class, "\"class_name>""
"Answer"; ("Provide only the class name in quotes, without additional explamations. If it is not recognized, output \"Unrecogmizedd" "]
}
Qwen2.5-VL Given Qwen [2 extensive pre-training on object detection tasks, we request Qwen2.5-VL VL generate prompt template to perform IOD, and specify the output format following the examples in their paper. Taking DIOR dataset as an example, the prompts for each task are as follows.
Open- vocabuoocyylary detection:
**Prompt:**
I need assistance with performing remote sensing object detection task using the provided category names: ['airplane', 'airport', 'baseball field', basketball court', 'bridge', 'chimmey', 'expressway service area', expressway toll station', 'dam', 'golf field', 'ground track field', 'harbor', 'overpass', 'ship', 'stadium', 'storage tank', 'tennis court', 'train station', 'vehicle', 'windmill'].
Please provide the results in JSON format, where each object represented follows:
Each object should include a label (category name) and its corresponding bounding box coordinates (in the format [x1, y1, x2, y2]).
**Example Output:**
[
("label": rcategory_name" "bbox_ _bbo [x1, y1, x2, y2l),
("label": "category_nanc" name", abbo__dd". [x1, yl, x2, y21),
Open-ended detection:
**Prompt:**
Please detect the visible objects the satellite image provide results JSON format, where each object is represented as follows: Each object should include a label (category name) in singular form and its corresponding bounding box coordinates (in the format [x1, yl, x2, y2]). +xEample Output:**
[
("label": "category1", "bbox_2d" [x1, y1, x2, y2]), ("label": "category2", "bbox_2d" [x1, y1, x2, y2]),
Open-subelass detection (e.g., for "sports field""):
**Prompt***
Please detect all the visible objects in the satellite image that belong to the parent category **sports field=* and provide the results in JSON format, where each object represented follows:
31

---

Each object should include label (subcategory name) in singular form and its corresponding bounding box coordinates (in the format [x1, yl, x2, y2]). none of the objects belong to the parent category visible, output empty empty list like [].
* E*Eample Output:**
[
{"label": "subcatagory1", "bbox 2d": [x1, yl, x2, y2]3, {"label": "subcategory2", "bbox_ [x1, y1, x2, y2]3,
While incorporating detailed dataset-specific instructions into prompts enhances Qwen2.5-VL's performance for object counting, such elaborations does not benefit direct detection. In fact, such instructions reduce mF, by 1% on NWPU- SWPUUVHR-ID and have little effect on DIOR. The main paper reports open-vocabulary detection results without these instructions.
D
Additional experiments
D.1
Comparison with zero-shot remote sensing object recognition methods on FarthInstruct
Direct comparison with some zero-shot remote sensing detection and segmentation methods (c.g.. DesReg 190), OVA-DETR 79), ZoRI (2 is difficult due to differing dataset splits, evaluation met- rics, and limited reproducibility. Table T0 shows their reported mAP scores alongside InstructSAM's mAPne on novel classes. Notably, InstructS AM-nwwn matches or outperforms these methods on their respective benchmarks (e.g., ZoRI on NWPU-VHR-I0, OVA-DETR on DIOR). While these baselines use confidence-based AP, InstructSAM's fonfidence-free APne results underscore its robustness.
Table 10: Comparison with zero-shot remote sensing object detection and segmentation methods on novel classes. '-' means data missing due to limited reproducibility. ( indicates model lacking the segmentation ability. APne is reported for InstructSAM, while values in gray for other methods are AP reported in their original papers.
NWPU-VHR-10
DIOR val
DIOR test
Method
Box AP
Mask AP
Box AP
Mask AP
Box AP
Mask AP
DesReg 90
0
0
7.9
0
OVA-DETR 79
0
7.1
0
0
ZoRI 21
12.3
8.5
InstructSAM-OO -wQQen
24.6
24.1
7.6
6.3
4.9
4.3
InstructSAM-GPTR
26.8
26.5
11.2
8.9
6.6
5.4
D.2 Comparison with open-vocabulary detection methods on out-of-distribution datasets
To compare with open-vocabulary methods that include DIOR into training, compare the zero-shot performance on two out-of-distribution (OOD) datasets:
xBD [88 is large-scale building damage assessment dataset with spatial resolution below 0.8m. The test set includes 933 pre- and post-disaster image pairs with instance-level building masks. We use pre-disaster images to evaluate object counting and detection with the single category "building".
Aerial Maritime Drone Large 30 is drone-based object detection dataset with 74 aerial maritime images and 1,151 bounding boxes. Categories include "docks", "boats", "lifts", "jetskis"" and "cars". The entire dataset is used for zero-shot evaluation.
Table IT shows that LAE-DINO, trained on LAE-IM (including "building"), achieves the highest detection F Fi-c (50.6) on xBD. On Aerial Maritime Drone, however, LAE-DINO detects only boats, even struggling with docks and cars despite semantic similarity to harbor and vehicle categories in training. Moreover, the confidence-based detectors (OWL, CASTDet, LAE-DINO) require dataset-
32

---

Table 11: Comparison with open open-vocabulary detection methods on xBD and Aerial Maritime Drone
Large datasets.
xBD
Aerial Maritime Drone Large
Method
Best Thr
Cnt mFi
Det mFi
Best Thr
Cnt mFi
Det mFi
OWL 51
0.02
53.0
31.5
0.26
28.3
21.5
CASTDet 38
0.00
0.0
0.0
0.40
13.3
10.8
LAE-DINO
56
0.08
79.6
50.6
0.30
13.0
12.5
InstructSAM-GP11
65.9
37.2
44.5
22.5
specific thresholds to optimize zero-shot results, highlighting InstructSAM's advantage in eliminating threshold tuning.
D.3 Selection of prompt format
We conduct five independent experiments evaluating Qwen2.5-VL and GPT-4o's open-vocabulaty counting ability on NWPU-VHR-IE using Markdown and JSON prompts. Table 12 shows the mean accuracies and standard deviations. Qwen2.5-VL exhibits only 1.2% difference in mean accuracy between Markdown and JSON, with zero standard deviation for both. GPT-4o shows slight variability: 82.38% + 0.94 (Markdown) and 82.68% + 0.39 (ISON). These results indicate that these LVLM counters maintain relatively stable performance across different prompt formats.
Table 12: Mean accuracy and I-sigma standard deviation for different prompt formats and models.
Model
Prompt Format
Mean Accuracy (%)
1-Sigma (Stand Deviation)
Qwen2.5-Vi
Markdown
72.0
0.00
Qwen2.5-VL
JSON
73.2
0.00
GPT-4o
Markdown
82.4
0.94
GPT-4o
JSON
82.7
0.39
E
Broader impacts
InstructSAM demonstrates strong ability recognize objects in remote sensing imagery with versatile instructions. It accelerates large-scale mapping applications, further supporting public agencies and humanitarian organizations critical areas such as poverty mapping, disaster response. Its training-free free paradigm and near-constant inference speed lower computational costs, decreasing carbon emissions and broadening access remote sensing applications resource-limited settings.
InstructS capabilities also raise privacy concerns as the spatial resolution of satellite imagery is growing higher. Hallucinations misclassificatioss, stemming from biases in pre- are-trained models may produce unreliable outputs, require user verification to ensure accuracy.
F Qualitative results
This section presents qualitative results InstructSAM across open-vocabulary (Figure 1) open- ended (Figure 10), and open-subclass settings (Figure IT). Additionally, we showcase its versatility handling instructions achieve certain goals in Figure 12 and its generalization natural images
in Figure [3
33

---

F1
Qualitative results in open-vocabulary setting
Ground Trubs
InstrudSAM
OWLv2
Qwen2 5 VL
SeyEanb-OW
Ground Thuth
InstradAAM
Ow1v2
Qwen2.5-V1
Seg Barth-OV
Ground Trab
InstrucESA
OWLv2
Qwen2.5
Soe SopEards-ON
Figure 9: Qualitative results in open-vocabulary setting. While OWLv2 struggles to distinguish remote sensing objects beyond vehicles, and SegFarth-OV fails to separate foreground objects from the background, InstructSAM demonstrates superior performance in segmenting remote sensing objects.
34

---

F.2
Qualitative results in open-ended setting
Reueeeee Amnotation
InstrudSAM
Gooffaxll
SkysmseGPP
Qwen 2.5VL
(a)
Refernee Ammotatioo
InstrusSAM
GeoPixel
SkymnsGPT
Qwan2.5-V1 S-L
(b)
Referencs Annotazion
InstructSAM
GooPicel
GPy
QwenZ.5 Dwenn..-VVL
(c)
Figure 10: Qualitative results in open-ended setting. Unlike GeoPixel and SkysenseGPT, which fail to detect classes outside their training set, InstructSAM demonstrates its ability to recognize diverse objects (e.g., pavilion in (a), tree in (c)) and provides more accurate bounding boxes and less fragmented masks.
35

---

F.3
Qualitative results in open-subelass setting
Reference Annolation
InstructSAM
Qwen2.5 5LVL
GPT-40-OWL
aa) Detecting the subclass of "means of transport"
Reference Annotation
InstructSA.
Qwen2.5-V6
GPT-4o-OWL
(b) Detecting the subclass "sports field"
Reference Annotation
InstructSAM
Qwen2 5-VL
GPT-4o-OWL
(c) Detecting the subclass of "sports field"
Figure 11: Qualitative results in subn-subelass setting. InstructSAM effectively identifies objects within parent categories. In contrast, Qwen2.5-VL. struggles with dense objects, and OWLv2 faces challenges in classifying sports fields from a bird' med'seeye view.
36

---

F44
Qualitative results on other instructions
Ground Truth
InstretSAM
Qwen2.5 5-VL.
infrrsrrccture ary onsum thst ambulaross
land elfectirely the midst ubbna- 1985 enerrescies
Tisko Detect recreational spaces would elfective hasti- a highly intersotive community spo. githering
Teeku What would be ideal vesue swingnng both safedy easy
aocess?
Figure 12: Qualitative results on following versatile instructions. Samples are from EarthReason 33 dataset. InstructSAM successfully recognizes objects based implicit cues.
F5 Qualitative results in natural images
InstructSAM
Qwen2,5-VL
Tadec Delect all the electranic co- Expouenls image.
100
100
100
94V-0
94V-0
Tasle Detect dioce whose Jet- betore pucture.
0
Q
I
I
N
Figure 13: Qualitative results on natural images. When equipped with generic CLIP models (DFN2B-
CLIP 13) InstructSAM effectively recognizes objects in natural images.
37


Reviews:
[{"review_id": "J0QRFs2U4W", "reviewer": "Reviewer_QWiC", "summary": "The authors formulate a new set of problems for Remote Sensing i.e Instruction-Oriented Object Counting, Detection, and Segmentation in open-vocabulary, open-ended, and open-subclass settings. They then proceed by providing the first benchmark on this suite of problems named Earth-Instruct built on top of two existing datasets. Finally, they propose a framework that excels in these tasks build upon existing large models like GPT-4o and SAM2.", "questions": "I believe the main drawback of this work is the diversity of the benchmark itself both in terms of the classes it encompasses as well as in terms of its input --> Sensors, domains of interest (land, marine etc.) , Ground sampling distances etc.\n \n- How were these datasets selected? The explanation given in L129 is a bit vague.\n- How can the benchmark be (potentially) extended?\n- Extending the evaluation of InstructSAM to standard RS segmentation datasets outside of the benchmark and comparing its performance with standard baselines would be interesting and could help us identify its current limits. \n  - For example, how would it perform on pixel based land cover land use classification e.g on FLAIR[1]. \n  - What is the impact of the granularity of the target classes? \n  - How does it adapt to varying environmental conditions?\n- Some experiments on other popular EO modalities like SAR and Multispectral data could showcase the current limitations of adapting this training-free methodology. (Example SAR segmentation dataset for flood mapping: Kuro Siwo [2])\n\n[1] Garioud, Anatol, et al. \"FLAIR: a country-scale land cover semantic segmentation dataset from multi-source optical imagery.\" Advances in Neural Information Processing Systems 36 (2023): 16456-16482.\n\n[2] Bountos, Nikolaos Ioannis, et al. \"Kuro Siwo: 33 billion $ m^ 2$ under the water. A global multi-temporal satellite dataset for rapid flood mapping.\" Advances in Neural Information Processing Systems 37 (2025): 38105-38121.", "limitations": "The authors address the limitations of their work in the appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strenghts**\nThe authors introduce a novel set of problems, along with a respective benchmark to complement the tasks. Their benchmark is built on top of 2 diverse detection datasets. The authors propose a novel, training-free, method that performs well on these tasks setting a strong baseline for future methods. The out-of-the-box performance of InstructSAM is particularly interesting. Overall the paper is clear and well written.\n\n\n**Weaknesses**\n\n***Minor Weaknesses***\n\nClass legends in figures are very small. One has to zoom in a lot to actually read them\n\n***Major Concerns***\n\n\n\n- The benchmark is built upon two pre-existing RGB and RGB+NIR datasets, at varying Ground Sampling Distances (GSDs) spanning a few cm to 30m. The included classes are very specific and do not cover major areas of interest in the Earth Observation community e.g land use land cover, crops, floods, burnt areas, tree species. Many of these categories have hierarchical structures e.g forest type, tree genus, tree species etc often resulting to finegrained classification tasks. Solving them would typically require varying sensors and Ground Sampling Distances as input. Including more diverse data could make the benchmark more representative for EO. Nevertheless this work is an important first step towards this direction.\n- It is not very clear why the benchmark was limited to these two datasets. Obviously, there are not many detection datasets for Earth Observation but there are definitely more than two. Even more, segmentation datasets could be transformed to detection tasks, incorporating greater class and GSD diversity\n- Relying on SAM2 for mask proposal may be restricting when switching to sensors highly different from optical images (e.g Synthetic Aperture Radar (SAR) and Interferometric SAR (InSAR)). In such cases the benefits of the proposed methodology could diminish completely. An experiment studying this could add value to the paper.", "was_revised": true, "final_justification": "The authors addressed all of my concerns and even introduced new experiments, further solidifying their paper. The authors promised to extend the limitations discussion including the sensitivity to target class granularity, environmental variability, and performance constraints of new modalities e.g SAR, improving the reader's understanding of the scope of this work. Overall this is a good paper, and I stick to my initial recommendation for acceptance.", "rebuttal": "**Minor Concerns:**\n\n> Class legends in figures are very small. One has to zoom in a lot to actually read them.\n\nThank you for pointing this out. We will improve the display of legends in the final version.\n\n**Major Concerns and questions:**\n\n> The included classes are very specific and do not cover major areas of interest in the Earth Observation community e.g land use land cover, crops, floods, burnt areas, tree species. Many of these categories have hierarchical structures e.g forest type, tree genus, tree species etc often resulting to finegrained classification tasks. Solving them would typically require varying sensors and Ground Sampling Distances as input. Including more diverse data could make the benchmark more representative for EO. Nevertheless, this work is an important first step towards this direction.\n\nThank you for acknowledging our contribution as an important first step. Our primary focus is on object recognition tasks, but we agree that extending InstructSAM to segmentation datasets is valuable for addressing broader EO challenges. To this end, we have conducted additional experiments on FloodNet and FLAIR datasets to explore its potential for land cover and flood segmentation tasks.\n\n> How were these datasets selected? The explanation given in L129 is a bit vague.\n\nWe appreciate the opportunity to clarify our dataset selection process:\n\n* **Diversity in resolution and modality:** Current object detection datasets in remote sensing are predominantly high-resolution and RGB. NWPU-VHR-10 includes NIR bands, while DIOR spans a wide range of resolutions (0.3m to 30m), representing diverse GSDs.\n* **Annotation rules and user requirements:** NWPU-VHR-10 is a subset of DIOR in terms of classes, but their annotation rules differ significantly, reflecting varying user requirements. For example, NWPU excludes vehicles in low-resolution images, while DIOR includes them. This makes these datasets ideal for evaluating a model's ability to follow user-specific instructions beyond simple category matching.\n* **Comparability with prior work:** Both datasets have been widely used in previous zero-shot object detection and segmentation studies, such as ZoRI (AAAI'25) and DesReg (AAAI'24). We provide a detailed comparison with these methods in Appendix D1, Table 10.\n\n> It is not very clear why the benchmark was limited to these two datasets.  Obviously, there are not many detection datasets for Earth Observation but there are definitely more than two.\n\nWe agree that there are other datasets available. However, we have found that NWPU-VHR-10 and DIOR are sufficient to demonstrate both the strengths and limitations of InstructSAM in optical object recognition:\n\n* **Strengths:** InstructSAM excels at following diverse instructions and operates in a training-free paradigm.\n* **Limitations:** Its performance is constrained by the capabilities of SAM and CLIP models, as detailed in Appendix E.\n\nThat said, we acknowledge the value of extending the benchmark to other modalities and tasks. This is an exciting direction for future work.\n\n> How can the benchmark be (potentially) extended?\n\nThe benchmark can be extended in several ways:\n\n* **Increase in class diversity:** Adding more fine-grained categories, such as tree species or specific aircraft types.\n* **Incorporate additional modalities:** Including SAR data to evaluate cross-modality generalization.\n* **Expand task scope:** Extending to land cover classification and change detection tasks.\n\n> Extending the evaluation of InstructSAM to standard RS segmentation datasets outside of the benchmark and comparing its performance with standard baselines would be interesting and could help us identify its current limits. For example, how would it perform on pixel based land cover land use classification e.g on FLAIR.\n\nThank you for this suggestion. To adapt InstructSAM for land cover tasks, we made the following key modifications:\n\n1. **Area-based prediction:** Since segmentation tasks involve \"uncountable\" objects, we used the LVLM to predict the proportion of each class in the image (e.g., 30% building, 70% brushwood).\n2. **Modified objective function:** We added a penalty term to minimize the difference between the predicted and assigned areas for each class. The updated objective function is:\n\n$$\n\\min_{X} \\frac{1}{N} \\sum_{i=1}^N \\sum_{j=1}^M (1 - s_{ij}) x_{ij} + \\lambda \\sum_{j=1}^M \\left| \\sum_{i=1}^N a_i x_{ij} - t_j \\right|\n$$\n\n$$\ns.t. \\sum_{j=1}^M x_{ij} = 1, \\quad \\forall i \\in {1, ..., N}\n$$\n\nwhere $t_j$ is the target area for class $j$ predicted by LVLM, $a_i$ is the area of mask $i$, and $\\lambda$ is the weight of the penalty term.\n\nThe zero-shot performance on **FLAIR** dataset was promising. InstructSAM-GPT4.1 outperformed the SegEarthOV baseline. The penalty significantly improved the accuracy of mask-label assignments.\n\n| Model                 | building | pervious surface | impervious surface | bare soil | water | coniferous | deciduous | brushwood | vineyard | herbaceous vegetation | agricultural land | plowed land | swimming pool | mIoU |\n| ----------------------- | ---------- | ------------------ | -------------------- | ----------- | ------- | ------------ | ----------- | ----------- | ---------- | ----------------------- | ------------------- | ------------- | --------------- | ------ |\n| SegEarth-OV [1]       | 47.6     | 6.5              | 0.1                | 13.7      | 49.6  | 10.6       | 3.1       | 4.2       | 34.5     | 8.8                   | 16.9              | 7           | 5.8           | 16   |\n| InstructSAM ($λ=0$)    | 25.8     | 9                | 4.4                | 28.8      | 45.3  | 11.4       | 21.7      | 14.1      | 46.9     | 10.9                  | 22.7              | 28.4        | 2.5           | 20.9 |\n| InstructSAM ($λ=0.05$) | 33.1     | 14.1             | 23.9               | 34.7      | 47.7  | 13.5       | 21.1      | 13.5      | 44.1     | 19                    | 27.2              | 29.4        | 3.3           | 25   |\n| InstructSAM ($λ=0.1$)  | 32.2     | 13.7             | 23.5               | 34.7      | 46.4  | 13.6       | 20.9      | 13.6      | 43.9     | 18.9                  | 26.8              | 29          | 3.3           | 24.6 |\n\n> The impact of the granularity of the target classes?\n\nInstructSAM performs well for coarse-grained categories (e.g., \"means of transport\") but struggles with fine-grained distinctions (e.g., Boeing 777 vs. Airbus A320, or hockey field vs. soccer field). This limitation arises because current RS-specific CLIP models (e.g., RemoteCLIP, GeoRSCLIP, SkyCLIP) are not trained on semantically rich datasets. Incorporating high-quality, diverse training data (e.g., from OpenStreetMap tags) could significantly enhance their capabilities.\n\n> How does it adapt to varying environmental conditions?\n\nThis is an excellent point. The appearance of objects can vary significantly across geolocations and seasons. Adding geolocation and seasonal information to the prompt could improve instruction-oriented object counting. We plan to explore this in future work.\n\n> Extending the evaluation of InstructSAM to standard RS segmentation datasets outside of the benchmark and comparing its performance with standard baselines would be interesting and could help us identify its current limits.\n\nWe conducted additional experiments on the SARDet-100k [2] dataset to evaluate InstructSAM's performance on SAR imagery. These results highlight the challenges of applying current LVLMs and SAM2 to SAR data. Generic LVLMs struggle to interpret SAR imagery, and SAM2 fails to generate meaningful masks. This underscores the need for modality-specific adaptations in future work.\n\n| Model              | Counting mF1 | Object Recall |\n| -------------------- | -------------- | --------------- |\n| InstructSAM-Qwen   | 0.3          | 1.1           |\n| InstructSAM-GPT4.1 | 6.0          | 1.1           |\n\n**References:**\n\n[1] Li, Kaiyu, et al. \"SegEarth-OV: Towards training-free open-vocabulary segmentation for remote sensing images.\" ​*Proceedings of the Computer Vision and Pattern Recognition Conference*​. 2025.\n\n[2] Li, Yuxuan, et al. \"SARDet-100k: Towards open-source benchmark and toolkit for large-scale sar object detection.\" *Advances in Neural Information Processing Systems.* 2024.\n\nWe thank you for great comments and hope our response clarified the paper's contribution to this community."}, {"review_id": "f3vjGJTRZj", "reviewer": "Reviewer_apuM", "summary": "The paper introduces InstructSAM, a training-free framework for instruction-driven object recognition in remote sensing imagery. The authors first propose InstructCDS, a suite of tasks covering open-vocabulary, open-ended, and open-subclass scenarios for object counting, detection, and segmentation. To benchmark these tasks, they construct EarthInstruct and incorporate dataset-specific annotation rules (e.g., excluding low-resolution vehicles). InstructSAM leverages three components: An LVLM to interpret instructions and predict object categories/counts. SAM2 to generate class-agnostic mask proposals. CLIP (domain-tuned) to compute semantic similarity, followed by binary integer programming (BIP) to assign categories to masks under counting constraints. Experiments show InstructSAM matches or surpasses specialized baselines, maintains near-constant inference time regardless of object count, and reduces output tokens by 89%. The framework eliminates task-specific training and confidence thresholds, enhancing scalability and robustness.", "questions": "1. Scalability with Larger LVLMs: InstructSAM uses GPT-4o (proprietary) and Qwen2.5-VL (open). How does it affect the performance with other large open LVLMs (e.g., LLaMA-3-70B, Gemini 1.5)? \n\n2. SAM2 Alternatives: SAM2 is central to mask proposals. Did you explore other lightweight alternatives (e.g., MobileSAM, FastSAM) for edge deployment? Also quantify trade-offs (speed vs. recall)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Strong empirical results across diverse tasks. InstructSAM-GPT4o outperforms remote sensing-specific models. 2. Ablations validate design choices (e.g., prompt engineering, model scaling). 3. Error analysis reveals clear failure modes (e.g., background confusion due to CLIP’s scene-level bias).\n\nWeakness:\n1) Reliance on pre-trained models (SAM2, CLIP, LVLMs) introduces inherited biases. Sometimes SAM2 struggles with complex geometries and only provide patches in these complex geometries.\n2) Limited validation on some goal-oriented instruction prompts. For example, given the prompts \"detect flooded buildings\", it's hard to find flooded building validation dataset. Therefore its hard to validate model performance for such kind of prompt.", "was_revised": true, "final_justification": "While some of my key concerns have been addressed, I improve my original borderline accept rating.", "rebuttal": "**Weekness**\n\n> Reliance on pre-trained models (SAM2, CLIP, LVLMs) introduces inherited biases. Sometimes SAM2 struggles with complex geometries and only provide patches in these complex geometries.\n\nWe acknowledge this limitation, as it is an inherent drawback of using a generic SAM2 for segmenting EO images. Recent research has developed SAM variants tailored for EO imagery [1, 2], though their pre-trained weights are not yet publicly available. We believe InstructSAM will benefit from any pre-trained models (SAM2, CLIP, LVLMs) models for EO images, thanks to its training-free design. This adaptability ensures that future advancements in EO-specific foundation models can be seamlessly integrated into our framework.\n\n> Limited validation on some goal-oriented instruction prompts. For example, given the prompts \"detect flooded buildings\", it's hard to find flooded building validation dataset. Therefore, it's hard to validate model performance for such kind of prompt.\n> We agree the evalution of goal-oriented instruction prompts would be valuable. Here we further tested InstructSAM on FloodNet [3] (0.02m GSD, incorporating 9 classes covering diverse object of interests in remote sensing community). The LVLM is the key to understanding goal-oriented instructions. With structured prompt design and given the annotation rules, InstructSAM counts objects at a very high accuracy, while SegEarth-OV and OWLv2 failed to comprehend the subtle difference between flooded buildings and non-flooded buildings. However, given the clear texture presented in such high-definition drone images, SAM2 may segment an object into patches, reducing the recall of mask proposals and detection performance. This is consistent with the limitations we found in Appendix E.\n\n| model              | Metric | Building-flooded | Building-non-flooded | Pool | Vehicle | Road-flooded | Road-non-flooded | Water | Tree | Grass | mIoU/F1 |\n| -------------------- | -------- | ------------------ | ---------------------- | ------ | --------- | -------------- | ------------------ | ------- | ------ | ------- | --------- |\n| SegEarth-OV        | IoU    | 0                | 37.5                 | 19   | 13      | 8.8          | 51.6             | 53.7  | 55   | 77.6  | 35.1    |\n| InstructSAM-GPT4.1 | IoU    | **13.8**             | 36.7                 | 14.9 | 15.7    | 9.2          | 19.6             | 44.3  | 36   | 61.6  | 28      |\n| OWLv2              | Cnt F1 | 11.8             | 48.7                 | 64.5 | 78.5    | -            | -                | -     | -    | -     | 50.9    |\n| InstructSAM-GPT4.1 | Cnt F1 | **78.5**             | **71.3**               | **84.2** | **80.6**    | -            | -                | -     | -    | -     | 78.7    |\n| OWLv2              | Det F1 | 0.8              | 24                   | 37.1 | 29.2    | -            | -                | -     | -    | -     | 22.8    |\n| InstructSAM-GPT4.1 | Det F1 | **2.8**              | 5.8                  | 22.2 | 6.5     | -            | -                | -     | -    | -     | 9.32    |\n\n**Questions**\n\n> Scalability with Larger LVLMs: InstructSAM uses GPT-4o (proprietary) and Qwen2.5-VL (open). How does it affect the performance with other large open LVLMs (e.g., LLaMA-3-70B, Gemini 1.5)?\n\nWe evaluated InstructSAM with other large open LVLMs, such as Gemma 3 and Mistral-Small, to assess scalability of InstructSAM. These results show that other open LVLMs, such as Gemma 3 and Mistral-Small, also achieve competitive performance. Notably, Qwen2.5-VL-72B significantly outperformed its 7B counterpart, demonstrating good scalability with larger models.\n\n| Model                 | Counting mAcc | Counting mRecall | Counting mF1 |\n| ----------------------- | --------------- | ------------------ | -------------- |\n| Gemma3-12B            | 73.2          | 76.4             | 72.8         |\n| Mistral-Samll-3.2-24B | 84.5          | 65.6             | 70.8         |\n| Qwen2.5-VL-7B         | 81.9          | 70.5             | 73.2         |\n| Qwen2.5-VL-72B        | 81.2          | 83.9             | 79.5         |\n\n> SAM2 Alternatives: SAM2 is central to mask proposals. Did you explore other lightweight alternatives (e.g., MobileSAM, FastSAM) for edge deployment? Also quantify trade-offs (speed vs. recall)?\n\nThank you for this valuable suggestion. We evaluated several variants of SAM2 to analyze the trade-offs between model size, inference time per image, and mask proposal recall. SAM2 Small strikes a good balance between model size and recall, making it a viable option for resource-constrained scenarios.\n\n| Model      | Size (M) | Inference Time (s) | Mask Proposal Recall |\n| ------------ | ---------- | -------------------- | ---------------------- |\n| SAM2 Large | 224.4    | 3.52               | 82.4                 |\n| SAM2 Base+ | 80.8     | 3.38               | 81.4                 |\n| SAM2 Small | 46       | 3.11               | 79.1                 |\n| SAM2 Tiny  | 38.9     | 3.01               | 74.1                 |\n\n**References**\n\n[1] Yan, Zhiyuan, et al. \"RingMo-SAM: A foundation model for segment anything in multimodal remote-sensing images.\" *IEEE Transactions on Geoscience and Remote Sensing*. 2023.\n\n[2] Shan, Zhe, et al. \"ROS-SAM: High-quality interactive segmentation for remote sensing moving object.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.\n\n[3] Maryam, Rahnemoonfar et al. \"FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding\" *IEEE Access*. 2021.\n\nWe thank you for great comments and hope our response clarified the paper's contribution to this community."}, {"review_id": "fvqvEDZP0S", "reviewer": "Reviewer_xfUj", "summary": "The paper investigates zero shot object recognition in the remote sensing domain. \n\nIt introduces the InstructSAM model, which first using a VLM (Qwen 2.5 or GPT-4o) to predict the categories and number of objects for each category in the scene. The SAM2 model is used to predict object masks in the scene using a regular grid of point prompts (Generating many potential segmentations which are assumed to cover any objects of interest). The next step is to assign a subset of the masks to counted objects from the VLM. The category text and masks are encoded with CLIP and then compared. A binary integer program solves the assignment problem subject to counting constraints.\n\nThis method is tested on EarthInstruct, which is created from two existing datasets: NWPU-CHR-10 and DIOR. And defines 3 distinct task settings: open-vocab, open-ended, and open-subclass.\n\nThe technique is also evaluated in terms of inference time, and claims to be nearly constant with respect to the number of boxes.\n\nAblations cover prompt design and model selection. Experiments also cover error analysis.", "questions": "1. Have you tried using the predicted object classes from the LVLM in your SAM2 prompt? I'm wondering what the different in scores would be if you primed SAM based on the VLM output rather than just using a grid of prompt points.\n\n2. What is the maximum size of the inputs to the binary integer program? You state that it is solvable efficiently, but this is wrong in general (unless P=NP). I think it is important to justify that the problem size won't become pathological - or if it does, state the limit at which that will happen. The PuLP solver is good, but it doesn't scale that far.\n \n3. I don't understand why your runtime is constant (wrt to number of boxes), but others are not. Why does inference time increase with the number of objects in instruction-oriented models?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n* The construction of the EarthInstruct dataset with the awareness of not counting objects in an image subject to spatial resolution is well thought out. \n\n* The model selection ablation is informative as to how performance varies as a function of the backend models.\n\n* The mask-label ablation is impressive. If I'm reading this right, then if you just filter boxes by confidence score the version with the BIP matcher outperforms all operating points.\n\n* Combining existing trained models in a novel way to perform well on a zero shot problem is both impressive and practical.\n\n\nWeakness:\n\n* The prompt design section of the ablation study is unclear. I don't understand what is being varied. \n\n* While the mask-label ablation is very interesting, I think the reason that BIP matching is able to outperform all operating points it that SAM2 is prompted with a regular grid of points, producing many more candidates than needed. The process isn't trained to produce the correct boxes around all objects, its being asked to produce boxes where there may not be a valid object. \n\n* Being confidence free isn't necessarily a good thing. If the final boxes were assigned a score, the ability to filter / prioritize them could be beneficial for some applications. Can this method be imbued with a confidence score?\n\n\nComments:\n\nA recent paper: Vision Language Models are Biased (Vo et al 2025) https://arxiv.org/abs/2505.23941\nshowed a strong bias and reliance on context cues in counting VLMs. This is\nrecent, so I'm not counting it against the paper, but followup work should\nconsider this.\n\n\n[Line 66] Why is bounding box generation with qwen so slow?\n\n[Line 77] Is confidence free actually a pro? Could that also be a con due to no ability to filter based on a score? Can your method be imbued with a confidence score?\n\n[Line 160] Strength: The method for computing AP without confidence values is correct.\n\n[Line 163] Using the maximum F1 might not be fair. It is a \"best possible\" view and cannot be used to say a candidate model is \"better\" than another.\n\n[Line 171] Performing a random sample on the selected pairs at cosine similarity > 0.95 and then quantifying the error rate would be valuable to justify this threshold choice.\n\n[Line 201] The 1.2x scale factor seems arbitrary, was it tuned? Not not a big deal if it wasn't.\n\n[Line 222] Clarify, I find this wording hard to follow: For open subclass setting, we set two parent class \"means of transport\" and \"sports field\".\n\n[Figure 6] Is the Yellow line adding to the key takeaway of this plot? I'm also confused how mAPnc of InstructSAM w/o matching has different values at different thresholds. Am I missing something here? Please double check these numbers and if the takeaway is that BIP matching outperforms all operating points, state that explicitly or add clarifying text so readers don't make that mistake.", "was_revised": true, "final_justification": "I'm raising my score due to the addition of new experiments especially the one that shows this does NOT work on SAR. I believe this paper is a significant step forward in how we in the vision and AI community think about using using these models to solve problems, and by showing the limitations of where it does not work, it lays groundwork for more progress to be made.", "rebuttal": "**Strengths**\n\n> The mask-label ablation is impressive. If I'm reading this right, then if you just filter boxes by confidence score the version with the BIP matcher outperforms all operating points.\n\nThank you for recognizing this. Indeed, Figure 6 compares the per-class F1-score of InstructSAM with filtering solely by a single confidence score across all images and categories. Each category requires a different optimal threshold, making a single threshold ineffective across classes. In contrast, the counting-constrained matching process dynamically assigns labels without relying on a fixed threshold, ensuring optimal assignments for each image.\n\n**Weakness**\n\n> The prompt design section of the ablation study is unclear. I don't understand what is being varied\n\nWe appreciate your feedback. The \"Add Instr\" column in Table 4 refers to additional instructions, specifically dataset-specific annotation rules (e.g., NWPU’s “harbor” as distinct piers, excluding small vehicles). See lines 645–651 and 678–693 for details. These instructions guide the LVLM counter to better align with user intent, significantly reducing inherent biases.\n\n> While the mask-label ablation is very interesting, I think the reason that BIP matching is able to outperform all operating points is that SAM2 is prompted with a regular grid of points, producing many more candidates than needed. The process isn't trained to produce the correct boxes around all objects, its being asked to produce boxes where there may not be a valid object.\n\nThis is an insightful observation. Indeed, SAM2 often generates more mask proposals than the actual number of objects in the image. Constraint Equation (4) at Line 204 guarantees that all mask proposals are assigned when they are fewer than the predicted object counts.\n\n\n> Being confidence free isn't necessarily a good thing. If the final boxes were assigned a score, the ability to filter / prioritize them could be beneficial for some applications. Can this method be imbued with a confidence score?\n\nGreat point again! Each matched mask-label pair in InstructSAM already has a semantic similarity score, $s_{ij}$, which is automatically saved in the final predictions. This score can be used for filtering or prioritization, similar to conventional detectors. For example, in the open-subclass setting, threshold filtering based on $s_{ij}$ improved box mF1 by up to 8.4%.\n\n**Comments and Questions**\n\n> A recent paper: Vision Language Models are Biased (Vo et al. 2025) showed a strong bias and reliance on context cues in counting VLMs. Followup work should consider this.\n\nThank you for pointing this out. This paper focuses on counterfactual scenarios (e.g., counting legs on a three-legged chicken). In remote sensing, annotation rules may vary across datasets but rarely contradict common sense. Adding annotation rules significantly improves counting accuracy. For example, when distinguishing \"bridge\" from \"overpass,\" the model often recognizes overpasses as bridges due to their hierarchical relationship. However, annotation rules improved bridge accuracy and overpass recall by large margins:\n\n| Prompt Type| Bridge (Acc / Recall / F1)| Overpass (Acc / Recall / F1) |\n| - | -- | -- |\n| w/o annotation rules | 44.1 / 44.5 / 44.3| 75.9 / 16.7 / 27.4|\n| w/ annotation rules  | 66.3 / 50.3 / 57.2| 63.4 / 64.9 / 64.2|\n\n> [Line 66] Slow Qwen bounding box generation. Why your runtime is constant (wrt to number of boxes), but others are not. Why does inference time increase with the number of objects in instruction-oriented models?\n\nRepresenting every bounding boxes as text requires more tokens. Experimental results show that inference time is proportional to the number of output tokens. In contrast, counting requires far fewer tokens, making InstructSAM-Qwen faster than Qwen2.5-VL when the number of predicted objects exceeds five.\n\n> [Line 77] Is confidence free actually a pro? Could that also be a con due to no ability to filter based on a score?\n\nConfidence-free methods are advantageous in zero-shot or OOD scenarios, where bounding box quality is highly sensitive to threshold selection. However, if a validation dataset is available for tuning thresholds, confidence scores can be useful.\n\n> [Line 163] Using the maximum F1 might not be fair. It is a \"best possible\" view and cannot be used to say a candidate model is \"better\" than another.\n\nWe agree. A fairer approach is to tune thresholds on training data. InstructSAM can also be evaluated for \"best possible\" performance by enabling confidence filtering, as shown in the table above.\n\nThe impact of threshold filtering varies across different settings. For tasks like OVD and OED, which involve a large number of categories, a single threshold is ineffective at balancing performance across all classes. However, in scenarios with fewer classes, such as subclass-specific tasks, threshold filtering becomes more effective and can significantly improve performance.\n\n| Model| Threshold filtering | OVD    | OED | OSD/sports field | OSD/transport |\n| --- | --- | --- | --- | --- | --- |\n| Qwen2.5-VL        | ×| 36.4| 32.0| 32.4| 42.2|\n| InstructSAM-Qwen  | ×| 38.9| 29.6| 33.5| 41.9|\n| InstructSAM-GPT4o | ×| 41.8| 31.3| 46.9| 44.2|\n| GPT-4o+OWL        | √| 37.4| 24.0| 19.8| 65.9|\n| InstructSAM-Qwen  | √| 38.9| 29.8 (0.2↑) | 34.2 (0.7↑)| 48.9 (7↑)|\n| InstructSAM-GPT4o | √| 41.9 (0.01↑) | 31.3| 49.3 (2.4↑)| 52.6 (8.4↑)  |\n\n> [Line 160] Strength: The method for computing AP without confidence values is correct.\n\nThanks, we hope this method can raise the awareness of fair comparison bewteen confidence-based and confidence-free methods in generic object detection domain.\n\n> [[Line 171] Performing a random sample on the selected pairs at cosine similarity > 0.95 and then quantifying the error rate would be valuable to justify this threshold choice.\n\nThank you for the suggestion. We manually evaluated open-ended counting results on NWPU using Qwen2.5-VL and GPT-4o. The error rate was 3% (1 out of 32), with the only error being \"sports\\_field\" mapped to \"baseball\\_field.\" Most synonyms were correctly mapped, but some misses (e.g., \"tank\" vs. \"storage tank\") highlight the need for better RS-specific CLIP models.\n\n> [Line 201] The 1.2x scale factor seems arbitrary, was it tuned? Not not a big deal if it wasn't.\n\nIt was not tuned initially. We hypothesized that adding context would improve accuracy since CLIP is trained on full images rather than object crops. Experiments on NWPU in the OVD setting confirmed this, with 1.3x yielding the best results:\n\n| Model| Crop Scale | Box AP |\n| -- | --- | -- |\n| InstructSAM-GPT4o | 1.1×| 41.0|\n| InstructSAM-GPT4o | 1.2x| 41.8|\n| InstructSAM-GPT4o | 1.3x| 42.7|\n| InstructSAM-GPT4o | 1.4x| 41.9|\n\n> [Line 222] Clarify, I find this wording hard to follow: For open subclass setting, we set two parent class \"means of transport\" and \"sports field\".\n\nThe subclasses for \"means of transport\" and \"sports field\" are listed in Table 7 (Appendix B). We will add a reference to this table in the main text for clarity.\n\n> [Figure 6] Is the Yellow line adding to the key takeaway of this plot? I'm also confused how mAPnc of InstructSAM w/o matching has different values at different thresholds. Am I missing something here? Please double check these numbers and if the takeaway is that BIP matching outperforms all operating points, state that explicitly or add clarifying text so readers don't make that mistake.\n\nThe yellow line represents InstructSAM without matching, where predictions are filtered by a similarity score threshold before evaluation. That's the reason the value differs. We will clarify this in the revised manuscript. The key takeaway is that BIP matching outperforms all operating points, and we will explicitly state this to avoid confusion.\n\n> Have you tried using the predicted object classes from the LVLM in your SAM2 prompt? I'm wondering what the difference in scores would be if you primed SAM based on the VLM output rather than just using a grid of prompt points.\n\nThe official implementations of SAM and SAM2 do not support text prompts, though this capability was discussed in the original paper [1]. Additionally, prior work on text-prompted segmentation for remote sensing (e.g., GeoPixel [2]) has shown limited generalization to classes outside the training set (see Figure 10, Appendix G.2). For these reasons, we use gridded point prompts in our experiments.\n\n> What is the maximum size of the inputs to the binary integer program? You state that it is solvable efficiently, but this is wrong in general (unless P=NP). I think it is important to justify that the problem size won't become pathological - or if it does, state the limit at which that will happen. The PuLP solver is good, but it doesn't scale that far.\n\nThank you for this insightful question. You are correct that BIP is NP-hard in general; our claim refers to it being **tractable in practice** for the problem scales encountered in our framework.\n\nIn our experiments, the problem size was modest. For example, on NWPU, the maximum number of mask proposals (N) was 200 and categories (M) was 8, with the solver finding an optimal solution in under 0.07s.\n\nTo further justify this, a multiple linear regression of the solution time against N and M yielded the equation: `Time = -0.0052 + 0.00013*N + 0.00535*M`. This model's high coefficient of determination (R² = 0.84) confirms that the solution time scales predictably and near-linearly within our operational range, ensuring practical efficiency.\n\n**References**\n\n[1] Kirillov, Alexander, et al. \"Segment anything.\" CVPR. 2023.\n\n[2] Shabbir, Akashah, et al. \"Geopixel: Pixel grounding large multimodal model in remote sensing.\" ICML. 2025.\n\nWe thank you for great comments and hope our response clarified the paper's contribution to this community."}, {"review_id": "vVWlx6KKCK", "reviewer": "Reviewer_te26", "summary": "The paper introduces InstructSAM, a training-free pipeline that turns a natural-language instruction into object counts, masks and labels in remote-sensing images. It pairs a vision-language model for counting with SAM2 for mask proposals. The authors also release EarthInstruct, a small benchmark to test this “instruction counting-detection-segmentation” task.", "questions": "1.\tRS domain: Why did you focus your experiments on RS when you are using models from the natural domain? Why not evaluating on natural images as well?\n2.\tBroader evaluation:  As your paper focus on RS, additional experiments with other datasets would improve the paper, specifically, low-res datasets (e.g. S-2) or dataset with more diverse class types (e.g. from land-use or agriculture).\n3.\tOut-of-distrubution: You are testing InstructSAM on two further datasets in the appendix. Why are they called “out-of-distribution”? You are using generic models (SAM2 and CLIP) that are most likely trained on these categories as well.\n4.\tTask novelty claim: As you clarify in the related work, InstructCDS does not fundamentally differs from existing instruction-oriented tasks and your claim “new suite of tasks” might be a bit overstated.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Training-free approach.\n- Works with different VL models and reduces runtime compared to other models.\n- The BIP problem removes hand-tuned thresholds for label assignment.\n- Covers multiple tasks (conting, detection, segmentation) in multiple settings.\n- Very good results compared to benchmark models.\n\nWeaknesses\n- Tested only on two datasets covering two classes categories. Only high-res data is evaluated most public RS data is medium-res (e.g. Sentinel 2 with 10m).\n- Task novelty overlaps earlier instruction-based RS works; “first” claim overstated.\n- Section headers do not follow the style guidelines.\n- Real world applicability is limited due to very specific architecture design and limited generalization experiments.", "was_revised": true, "final_justification": "Technically solid paper, main concerns regarding dataset diversity and task novelty were adressed during the rebuttal.", "rebuttal": "**Weaknesses**\n\n> Tested only on two datasets covering two classes categories. Only high-res data is evaluated most public RS data is medium-res (e.g. Sentinel 2 with 10m).\n\nThank you for this observation. Most object recognition datasets in remote sensing are with high-resolution. DIOR was chosen for its wide GSD range (0.3m to 30m), which includes medium-resolution images. Many categories, such as \"ship,\" \"airport,\" \"expressway service area,\" and \"ground track field,\" frequently occur in medium-resolution images (as shown in Figure 2(d)). Additionally, EarthInstruct is a multimodal (NIR+RGB) and multi-resolution (0.08m–30m) benchmark, making it representative of diverse remote sensing scenarios.\n\n> Task novelty overlaps earlier instruction-based RS works; “first” claim overstated.\n\nWe appreciate your feedback. While there are overlaps with earlier works like EarthDial [1] and TEOChat, we believe InstructCDS and EarthInstruct introduce significant novelties:\n\n* **Broader instruction scope:** Existing works are limited to counting or detecting **one object type per query**. In contrast, EarthInstruct benchmark is the first benchmark that supports **open-vocabulary**,**​ open-ended**, and **open-subclass** settings in EO, enabling much more complex and diverse instructions.\n* **Structured outputs:** Earlier works fail to generate structured outputs for multi-class tasks. InstructSAM provides structured, interpretable results, aligning with complex user-defined instructions.\n\n> Section headers do not follow the style guidelines.\n\nThank you for pointing this out. We will ensure all section headers follow the style guidelines, with only the first word and proper nouns capitalized.\n\n> Real world applicability is limited due to very specific architecture design and limited generalization experiments.\n\nWe acknowledge this concern and have added model generalization and scaling experiments to address it. InstructSAM demonstrates good generalization across various open LVLMs, as shown below.\n\n| Model                 | Counting mAcc | Counting mRecall | Counting mF1 |\n| ----------------------- | --------------- | ------------------ | -------------- |\n| Gemma3-12B            | 73.2          | 76.4             | 72.8         |\n| Mistral-Samll-3.2-24B | 84.5          | 65.6             | 70.8         |\n| Qwen2.5-VL-7B         | 81.9          | 70.5             | 73.2         |\n| Qwen2.5-VL-72B        | 81.2          | 83.9             | 79.5         |\n\nAdditionally, InstructSAM's \"out-of-box\" design allows it to benefit from any improved open-source or proprietary foundation models, enhancing its applicability. The use of LVLM allows InstructSAM to interpret diverse instructions.\n\n**Questions**\n\n> RS domain: Why did you focus your experiments on RS when you are using models from the natural domain? Why not evaluate natural images as well?\n\nOur focus is on remote sensing (RS) due to its unique challenges, such as the lack of semantically diverse training data. In contrast, the natural domain benefits from abundant datasets like Object365, LVIS, and V3Det, making it relatively straightforward to achieve InstructCDS using models like GPT4o+OWLv2 or by training a generative model such as Ins-DetCLIP [1].\n\nAlso, we observed that SAM struggles to segment complex objects (e.g., person, car) as a single entity in natural images. However, this issue is less pronounced in satellite images due to the different texture features with natural images.\n\nThat said, we conducted qualitative experiments on natural images, such as electronic components and dice detection (Fig. 13). In these cases, InstructSAM outperformed generic detectors like OWLv2 and Qwen2.5-VL. Quantitative evaluations on natural images remain an interesting direction for future work.\n\n> Broader evaluation: As your paper focus on RS, additional experiments with other datasets would improve the paper, specifically, low-res datasets (e.g. S-2) or dataset with more diverse class types (e.g. from land-use or agriculture).\n\nThank you for this suggestion. While DIOR already includes medium-resolution images, we agree that additional experiments on low-resolution datasets or those with diverse class types (e.g., land-use or agriculture) would strengthen the paper. Here we demonstrate InstructSAM's superior zero-shot ability in land cover classification on FLAIR dataset compared to the recent open-vocabulary segmentation method. Broader evaluation is an exciting direction for future work.\n\n| Model                 | building | pervious surface | impervious surface | bare soil | water | coniferous | deciduous | brushwood | vineyard | herbaceous vegetation | agricultural land | plowed land | swimming pool | mIoU |\n| ----------------------- | ---------- | ------------------ | -------------------- | ----------- | ------- | ------------ | ----------- | ----------- | ---------- | ----------------------- | ------------------- | ------------- | --------------- | ------ |\n| SegEarth-OV       | 47.6     | 6.5              | 0.1                | 13.7      | 49.6  | 10.6       | 3.1       | 4.2       | 34.5     | 8.8                   | 16.9              | 7           | 5.8           | 16.0  |\n| InstructSAM-GPT4.1| 33.1     | 14.1             | 23.9               | 34.7      | 47.7  | 13.5       | 21.1      | 13.5      | 44.1     | 19                    | 27.2              | 29.4        | 3.3           | 25.0   |\n\n> Out-of-distrubution: You are testing InstructSAM on two further datasets in the appendix. Why are they called “out-of-distribution”? You are using generic models (SAM2 and CLIP) that are most likely trained on these categories as well.\n\nWe apologize for the confusion. You are correct that these datasets are not truly \"out-of-distribution\" for InstructSAM. The term \"out-of-distribution\" refers to datasets outside the training distribution of open-vocabulary detection models like OWLv2, CASTDet, and LAE-DINO. These models showed significant performance degradation on these datasets, whereas InstructSAM, equipped with more capable foundation models, performed better.\n\n> Task novelty claim: As you clarify in the related work, InstructCDS does not fundamentally differs from existing instruction-oriented tasks and your claim “new suite of tasks” might be a bit overstated.\n\n- **Instruction-Oriented Object Counting and Segmentation:** While we followed the settings of previous work on Instruction-Oriented Object Detection [1]. We add object counting and object segmentation to form a new suite of tasks, in other words we are the first to implement the comprehensive tasks altogether. This is a significant advancement, particularly for object counting, as current benchmarks like CountBenchVQA [2] are limited to single-class counting with explicit class names in the query. In contrast, InstructCDS requires counting based on implicit instructions.\n\n-  **Novel evaluation framework:** We defined F1-score for multi-class object counting, providing a more informative evaluation metric (line 141).\n\n**References**\n\n\n[1] Pi, Renjie, et al. \"INS-DetCLIP: Aligning detection model to follow human-language instruction.\" ​*The Twelfth International Conference on Learning Representations*​. 2024.\n\n[2] Paiss, Roni, et al. \"Teaching clip to count to ten.\" ​*Proceedings of the IEEE/CVF International Conference on Computer Vision*​. 2023.\n\nWe thank you for great comments and hope our response clarified the paper's contribution to this community."}]

Author rebuttal:

Meta-review: Following the discussion phase, all reviewers recommended acceptance (1 Strong Accept, 3 Accepts), citing the paper's clear writing, novelty, and impressive results. They highlighted the introduction of a new benchmark with potential value to the community, as well as the thorough evaluation. The rebuttal effectively addressed many of their concerns—for example, through the addition of extensive experiments on new datasets, as well as generalization and scaling analyses. Accordingly, the ACs have decided to accept the paper. Please incorporate the reviewers' feedback when preparing the camera-ready version.
Decision: Accept (poster)