GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Authors: Akashah Shabbir
Mohammed Zumri
Mohammed Bennamoun
Fahad Shahbaz Khan
Salman Khan
Venue: ICML 2025 poster (2025)
Keywords: Large Multimodal Models (LMMs)
Remote Sensing (RS)
Vision Language Models (VLMs)
Computer Vision
Segmentation
Pixel Grounding
Primary area: applications->computer_vision
TL;DR: GeoPixel is a high-resolution remote sensing multimodal model enabling pixel-level grounding and GeoPixelD is introduced, a dataset curated via semi-automated, scalable pipeline integrating prior-informed visual prompting with state-of-the-art LMMs.

Abstract: Recent advances in large multimodal models (LMMs) have recognized fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such representation in LMMs are limited to the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehension. Moreover, the development of the grounding conversation capability of LMMs within RS is hindered by the lack of granular, RS domain-specific grounded data. Addressing these limitations, we propose GeoPixel - the first end-to-end high-resolution RS-LMM that supports pixel-level grounding. This capability allows fine-grained visual perception by generating interleaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To support the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies validate the effectiveness of each component in the overall architecture. Our code and data will be publicly released.

GeoPixel
Pixel Grounding Large Multimodal Model in Remote Sensing
Akashah Shabbir Mohammed Zumri! Mohammed Bennamoun Fahad Shahbaz Khan 13 Salman Khan 14
Abstract
Con you thoroush
aaage, including intericoeool
nanks highliigt key abjacte?
Recent advances in large multimodal models (LMMs) have recognized fine-gra fine-grained grounding as an imperative factor of visual understanding and dialogue. However, the benefits of such rep- resentation in LMMs are limited the natural image domain, and these models perform poorly for remote sensing (RS). The distinct overhead viewpoint, scale variation, and presence of small objects in high-resolution RS imagery present a unique challenge in region-level comprehen- sion. Moreover, the development of the ground- ing conversation capability of LMMs within RS is hindered by the lack of granular, RS domain- specific grounded data. Addressing these lim- itations, we propose GeoPixel the first end- to-end high-resolution RS-LMM that supports pixel-level grounding. This capability allows fine- grained visual perception by generating inter- leaved masks in conversation. GeoPixel supports up to 4K HD resolution in any aspect ratio, ideal for high-precision RS image analysis. To sup- port the grounded conversation generation (GCG) in RS imagery, we curate a visually grounded dataset GeoPixelD through a semi-aus automated pipeline that utilizes set-of-marks prompting and spatial priors tailored for RS data to methodically control the data generation process. GeoPixel demonstrates superior performance in pixel-level comprehension, surpassing existing LMMs in both single-target and multi-target segmentation tasks. Our methodological ablation studies val- idate the effectiveness of each component the overall architecture. https://githubb. htnp://gihhub.ccommbbuuu- com/mbzuai- oryx/GeoPixel.
The seriat viei sperts eetieing poll. The rnck aut whttapee wardess sparts. the aurrurniss area spaces. the
the
mitti tase uresin trick enstecting sperts, is tla recrassslar aed tilled wite
by sea at different sports conplie reatures rectoreala or by causing trmck inage gloes slacis geople woving anjecia ard viaable.
end,
Figure I. An example of visually grounded detailed descriptions generated by the proposed GeoPixel. highlighting its ability to in- terpret and segment high-resolution remote sensing imagery with fine-grained precision. The model applies distinct masks key objects (ground track field, swimming pool, soccer field) and se- mantic mask smaller objects (vehicles). effectively identifies spatial positions (center. top) and relationships (within the sports complex) while distinguishing between the global context (build- ings, roads, green spaces) and localized structures.
1. Introduction
Recent large multimodal models (LMMs) (Liu et al., 2024a; Dai et al., 2023; Bai et al., 2023b; Chen et al., 2024b) have utilized the foundational capabilities of Large Lan- guage Models (LLMs) (Touvron et al., 2023; Chiang et al., 2023; Javaheripi et al.: Bai et al., 2023a) and success- fully expanded their horizon to the visual modality with promising capabilities. These L.MSs can not only per- form visual recognition, but also excel in advanced per- ception and reasoning required for dision-language tasks such as visual question answers, image captioning, visual grounding, and referring expression segmentation. Ground- ing LMMs (Rasheed et al., 2024; Ma et al., 2025; Zhao et al., 2023) have further advanced the fine-graines context- aware interpretation of complex visual information by allow- ing textual outputs to be associated with object instances. Facilitated by large-scale data in the natural images do- main, grounding multimodal models pre-trained on exten- sive datasets have shown impressive capabilities, achieving performance levels comparable to specialist models.
'Mohamed bin Zayed University of Artificial Intelligence "The University of Western Australia 'Linköping University "Australian National University. Correspondence Akashah Shabbir <akashah. shabbir @ mbzuai.ac.ae>, Mohammed Zumri <mohammed.zumrie @ mbzuai.ac.ac>>.
However, with increasing granularity of vision and language understanding, these general domain models exhibit signif- icant limitations in adequately supporting complex earth observation tasks. The performance degradation is influ-
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
by accommodating up to 4k resolution.
Table Comparison of remote sensing large multimodal models (RS- (RS-LMMs), focusing on their grounding capabilitiess The 'Re- gion Output" column highlights the model's ability to associate objects with specifie spatial regions. Existing models primarily utilize LLMs to generate bounding box coordinates for object grounding. However, none of the current RS-LMMs possess the capability for grounding', i.c., generating detailed segmenta- tion masks, which are crucial for fine-grained spatial interpretation.
We create GeoPixelD, a multi-modal grounded conver- sation generation (GCG) dataset comprising 53,816 grounded phrases linked to 600,817 object masks, specifically tailored for RS image understanding. Ex- tensively granular annotations are created with seg- mentation masks through a semi-automated, scalable pipeline that integrates prior-informed visual prompt- ing with state-of-the-art LMMs and ensures quality via rigorous verification and filtering steps.
MODELS
RESOLITTIO
IMAGE
OUTPUT DECOBER GROUNDING
REGION
PixeL
RSGPT (HL EY AL., 2023)
x 224
HZRSVLM (PANG ET AL., 2024)
336 x33
LLAA (BAZ) ET 2024)
336 335
We introduce comprchensive benchmark for evalu- ating RS LMMs in fine-grained visual understanding, assessing models' ability to interpret complex, spa- tially grounded information. It comprises 5,427 manu- ally validated referring expressions and segmentation masks, covering 61,384 annotated objects.
GEOCHAT (KUCKRETA BT AL., 2024)
SkyFNIGPT (ZMAN Mr., 2024)
448
X
EARTIGPT (ZHANG 17 Al., 2024c)
LHRS- Bor (MUITAA ET At., 2014)
224x224
SKYSENSEGPT (LuO 2024)
x 504
enced not only by the unique vantage point inherent to re- mote sensing (RS) images but also by large variations in the objects' size and orientation. Moreover, in high-resolution (HR) remote sensing imagery, objects of interest may exhibit challenging-to-segee -esemmentt spatial footprints, such as narrow bridges that connect urban landscapes and play a critical role in city traffic planning, adding further complexity to the task.
2. Related Work
Large Multimodal Models (LMMs): LMMs build on the success of LLMs to acquire vision capabilities, Pioneer works such as LLaVA (Liu et al., 2024b), MiniGPT-4 (Zhu et al., 2023), InstructBI.IP (Dai et al., 2023) and mPL.UG- Owl (Ye al., 2023b) aligned visual features with language representations through a vision language connector, en- hanced by instruction tuming to improve multimodal integra- tion. Improving beyond image-level understanding, models such as GPT4Rol (Zhang et al., 2023), InternGPT (Liu et al., 2023b) and RegionGPT (Guo et al., 2024) introduce regional understanding by allowing inputs such as points, masks, and bounding boxes. Some models feed image co- ordinates directly into the language model, while others employ additional feature extraction modules to represent specific image regions' features effectively.
Existing vision language models in RS (Luo et al., 2024; Zhang et al., 2024b; Kuckreja et al., 2024) use quantized coordinates in the form of bounding boxes to localize and ground objects in their response. Such a representation structure is not adequate to associate correct object seman- ties and also adds a computational burden to the LLM that scales with the number of distinguishable objects. More- over, monitoring the geospatial environment and its entities demands a broader spatial perspective, now increasingly achievable through advancements in RS technologies that provide HR imagery. However, despite the availability of such rich data, current LMMs in RS struggle to fully ex- ploit this spatial detail. These models often struggle with suboptimal resolution capabilities, hindering their ability to capture the intricate patterns present in high-resolution RS images. In addition, existing RS datasets often lack fine- tne-grrined spatial association between objects and their corresponding linguistic descriptions.
Region-level comprehension is fur- ther expanded by models such as Kosmos-2 (Peng et al., 2024), Ferret (You et al., 2023), Shikra (Chen et al., 2023), Pink (Xuan et al., 2024) and LION (Chen et al., 2024a) that allow for the precise location of objects in their out- puts based textual descriptions, capability known as grounding. These models localize objects on coarse scale using bounding boxes. Recent models (Lai et al., 2024; Rasheed et al., 2024; Xia et al., 2024; Ren et al., 2024; Zhang et al., 2024d: Liu et al., 2023a) focus on achieving more fine- fine-grained visual and linguistic semantic alignment, by exploring pixel grounding. LISA (Lai et al., 2024), Pix- elLM (Ren et al., 2024) and GLaMM (Rasheed et al., 2024) incorporate a [SEG] token into the LLM's vocabulary, lever- aging its corresponding token embedding as a conditioning input for SAM (Kirillov et al., 2023) to enable segmentation. Additionally, GSVA (Xia et al., 2024) introduces a [REJ] token to explicitly learn to reject specified targets. Whereas
Grounding LMMs:
To address these issues, we present GeoPixel, a model that can generate a detailed natural language response for high- resolution RS image with corresponding geospatial object segmentation masks. Our contributions are as follows:
Our proposed LMM, GeoPixel, is explicitly designed for high-resolution RS image analysis with advanced multi-target pixel grounding capability. Our model adaptively divides the input images into local and global regions, enabling efficient encoding and analysis
2

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Llava-plus (Liu et al., 2023a) employs LLMs as agents to assign tasks to the segmentation expert.
3. Method
In the current remote sensing landscape, LMMs face sig- nificant limitations in terms of grounding and resolution capabilities (as seen in Table 1). Specifically, the outputs generated by these models lack precise spatial and semantic association with the imagery, leading to either ungrounded or only coarsely grounded text. Furthermore, most LMMs operate on relatively low-resolution data, which restricts their ability to perform fine-scale analysis essential for RS tasks such as detailed land use and transportation network extraction, infrastructure mapping, damage assessment, and environmental monitoring. To address these limitations, we present GeoPixel, model designed to interpret high- resolution remote sensing images and generate finely de- tailed, pixel-grounded outputs that encompass multiple tar- get objects.
Our work aligns with pixel-grounding approaches, such as those in (Lai et al., 2024; Ren et al., 2024; Rasheed et al., 2024). However, these models do not interpret the dis- tinct top-down perspective and cannot differentiate complex spatial arrangements of remote sensing (RS) imagery. In addition, the models' restricted input size, typically limited to dimensions such as 224x224, exacerbates this issue by constraining the field of view and spatial perception.
High- Righ-Reesolutinn Understanding: Vision encoders, such as CLIP ViT (Radford et al., 2021), are widely utilized for various vision tasks but are typically constrained by low resolution (e.g. 224x224) restricting their applicabil- ity in HR scenarios. To address this limitation, some ap- proaches (Dosovitskiy al., 2021; Bai et al., 2023b; Li et al., 2023) scale positional encodings within the CLIP model through interpolation to accommodate larger input sizes, while others such as CogAgent (Hong et al., 2024) and Vary (Wei et al., 2025), employ an additional HR branch. Models such as Monkey (Li et al., 2024b). SPHNIX (Lin et al., 2023), Llava-Next (Liu et al., 2024a). IXC2.5 (Zhang et al., 2024a), Textmonkey (Liu et al., 2024d) and Ure- ader (Ye et al., 2023a) divide the image into grids to enhance performance on HR text-centric tasks.
3.1. GeoPixel Architecture Overview
GeoPixel primarily consists of 5 components (see Figure 2). (1) Adaptive Image Divider (2) Vision Encoder (3) Large Language Model (4) Grounding Vision Encoder (5) Pixel Decoder. The first three components are discussed in Sec- tion 3.2, while the latter two in Section 3.3. Jointly, these modules enable high-resolution perception, fine- semeigraindd in- terpretation, and grounding, as detailed below.
Remote Sensing (RS) LMMs: RSGPT (Hu et al., 2023) pioneered RS-based natural language conversation and gen- erated detailed captions, followed by GeoChat (Kuckreja al., 2024) which supported region-specific inputs and vi- sual grounding through oriented bounding box coordinates. SkyEyyGPPT (Zhan et al., 2024) extended its functional- ity to RS video captioning. while EarthGPT (Zhang et al., 2024c) and EarthDial (Soni et al., 2024) integrated mul- tisensor RS interpretation. RS-LLaVA (Bazi et al., 2024) and H2RSVLM (Pang et al., 2024) further improved the RS data interpretation, with H2RSVLM uniquely recognizing and rejecting unanswerable questions. SkySenseGPT (Luo et al., 2024) enables image-level scene graph generation and relation reasoning, while LHRS-Bot (Muhtar et al., 2024) enhances multilevel vision-language alignment and TeoChat (Irvin et al., 2025) converses on temporal se- quences. However, these models operate at low resolution and lack pixel-level understanding and grounding.
3.2. High Resolution Understanding
For high resolution, we adopt the dynamic image partition- ing strategy of IXC-2.5 (Zhang et al., 2024a). Initially, the adaptive image divider processes the input image Zing. with dimensions [h; x wy). by up- up-scaling and padding it to align with the closest grid size denoted as [9h. x gwl.
9h ki B, s.t., k.t, k2 N, k1 x k2 P
Iv k2 B,
(1)
where B is the base resolution of the vision encoder and P is the number of maximum allowable image patches. Subse- quently, the image is divided into k l2 non-overlapping patches &pa.' where p = 0, 1, 2, (ki x k2 - 1), and i denote the row and column indices of each patch in the grid.
We employ the scaled CLIP ViT-L/14 (Zhang et al., 2024a) as our vision encoder (Z), with base resolution of B = 560, facilitating large patches for enhanced visual representation. Furthermore, a global view Iotob is generated by resizing Timg fixed dimension of 560 x 560, aligned with the base resolution B. Feature embeddings of patches Sp., are appended with a learnable token at the end of each row before flattening and merging (Dong et al., 2024b). Finally, global features folos and patch features fp are concatenated (I) with a special separator (sy) inserted between them
GeoGround (Zhou et al., 2024), RSUniVLM (Liu & Lian, 2024), and GeoPix (Ou et al., 2025) are works concurrent to ours and share similarities. GeoGround and RSUniVLM support lexel-level grounding by converting masks into text sequences, adding a computational burden to the LLM that scales with the number of distinguishable objects. Whereas, GeoPix incorporated both referring expressions segmenta- tion and visual grounding/detection using box coordinates in text, which leads to task confusion.
3

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
serial
Putfe wire caras 1o Hh
area
Large Language Model
takang
Grouddd Go Coreesoin
11gh thehheallrrr Fnriie
PLORA
with
suppesiing fenus dn sparts ant commartyy grarn roroodd by y rurmi toa rells conser tt the see aurrewiee b tarpee apiit
Can you give a thorough descraption of this lmoge, soprontation maaks to high Light key abjects?
2a aatteeee le: The it
ainnorre cotarrr grwesy
vielble the nartor left
Visian Projection
Text Projection
arevides rotural coundayy and dass to the
Adaptive Image
rocation.
Divider
Ioput
Grounding Vision Encoder
Vision Encoder
Pixel Decoder
Resize/ Trans/orm
Local Images
Global Image
Figure 2. Overview of GeoPixel Architecture: Left: High-resolution RS images are dynamically partitioned into local patches and a resized global view, encoded by frozen vision encoder. The encodings projected into the language domain with separator tokens. Middle: Vision tokens, combined with text, are input into the I.M,, where pl.oRA is applied to vision tokens for efficient and effective multimodal alignment. Right: The corresponding embeddings for the [SEG] tokens are passed decoder through text projector, along with vision embeddings from the grounding vision encoders, to generate precise segmentation masks.
(Ding et al., 2019), effectively integrating global semantics with tine-grained local details.
without altering the language token outputs xı, thus achiev- ing tailored cross- modal integration.
x, Poffooo |5
3.3. Pixel Grounding
S.L. fglob = I(Iglob), fp.. = I(Iptss
To establish grounding in LMM, we initialize the ground- ing vision encoder (I,)) with a pre-trained SAM-2 (Ravi et al., 2024) encoder together with dedicated pixel decoder module (D). The SAM2 visual encoder is a Masked Au- toencoder (MAE) (He et al., 2022) pre-trained Hiera (Ryali et al., 2023) image encoder having a hicrarchical structure that allows the use of multiscale features during decoding. The tokenizer's vocabulary is expanded by incorporating an additional <SEG> token, with its corresponding last-layer embedding (E) mapped to the decoder through text pro- jection layer Pt. The text projection is two-layer MLP that receives embeddings of dimension 4096 and transforms them into the input space of the pixel decoder, which has a dimensionality of 256.
We project the final unified image features onto the LLM, In- temLL2 7B model (Cai et al., 2024), denoted as L. through a two-layer MLP as a vision projector P... InternLM22 is an LLM designed to process sequences of text tokens, where its input consists of discrete embeddings derived from tex- tual data. These embeddings correspond to either natural language tokens or special placeholders inserted to represent external modalities. The placeholder <TMAGE> in the input text query V₁ is a special token that represents the position of the image within the input sequence. When processing multimodal input, this placeholder is replaced with visual features F,, extracted from the image, and projected into the same embedding space using P..
The pixel decoder processes the image features from the frozen grounding vision encoder, along with projected LLM embeddings, to generate segmentation masks (M). The grounding vision encoder (SAM-2) is already pre-trained on large-scale datasets, making it highly effective at extracting robust, generalized image features for segmentation. Freez- ing the encoder ensures that these pretrained features are preserved. However, the tight-weight pixel decoder and pro- jection layer are trained to adapt pretrained vision features for segmentation tasks in GeoPixel.
Partial Low-Rank Adaptation (LoRA) (Dong et al., 2024a) is then applied to ensure efficient alignment of the vision tokens. Partial LoRA is a modality plug-in mod- ule designed align features from a new modality with LLM, preserving the model's inherent capabilities while enriching it with modality-specifee insights. By applying low-rank adaptations selectively to visual tokens, Partial LoRA enhances alignment efficiency while reducing the computational cost. Formally, it introduces low-rank ma- trices WA A E CCx and WB e RCo.xX within each LLM linear layer, modifying the visual token outputs Fe

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Cluster/Crowden Annotation
Give brast, aisgle-sentence descriantos Ecanegary. seen) at the (pos] to the paven verial imag,, without
info;
rurerous
categary, paan
vehic les
pos teft side
"A roc of Dis pereeee Lakely
Imsage
Imege + SOM
Image + SOM Prices
Masis
personsl cars, are parked stong
lrage, od] acent to the sports facillsy."
Holistic Image Annotation
Individual Instance Annotation
List attrabutes
marked (eark_runter)
Descrize the image in four stort sentences.
short settence. Bo not lestion the sarked ruiter in output-
"The Image is an seriat view of a ssorts corptex with wuttiple courts. ternis currts, There are surfaces, inficating cifferent blue surrace and others tosketsall court is sise visible, with pcople playing on it, The surrounding area int ludes parking sseces and sress, sugges ting the part larger recreational commnnity setting.""
category
court,
eategory
category
left, sarter_
bottom, mar Jumber
marker sumber
blue Slayang surface and white
the batton, with greup
top of courts located next to The blue basketball court."
bourdary Tires.
people gattered near are side."
Figure 3. The GeoPixelD Annotation Pipeline provides detailed multi-tier descriptions of remote sensing imagery with object phrases aligned precisely with manually annotated masks. begins with Holistic Image Annotation (bottom left), where an LMM generates concise scene descriptions. Individual Instance Annotation (bottom right) uses (patial({pos) and categorical (f (<catagorory. sarrry_nmme) }) priors with SOM (( (mark__nl mnane_) prompting to describe key objects. Clusterr Annotation (top right) organizes smaller or dense objects using refined grids for precise spatial analysis.
els, where IXC consistently outperformed its counterparts in terms of qualitative performance. Open-ended de- scriptions are restricted to a limited length, integrated in prompts like "<image> Describe the image in four short sentences" " (Figure 3 (bottom left)). Thus, redundancy effectively minimized in subsequent annotations, and the model is driven to provide holistic, context-rich depiction of each image.
M = (Zimg)) P((E))
(3)
Given the variable length of the input image tokens, result- ing from adaptive image partitioning, the output embedding mask for <SEG> tokens dynamically adjusted to align with these variations. This configuration ensures accurate detection of the <SEG> token and its associated embedding.
4. GeoPixelD-RS Pixel Grounding Dataset
4.2. Individual Instance Annotation
Next, we identify key objects for depiction and employ set- of-mark (SOM) prompting (Yang et al., 2023), which over- lays distinct visual markers over specific image regions, to provide auxiliary information for visually grounded output. However, directly employing this method for aerial imagery, characterized by expansive views and diverse objects within single frame, poses challenges such as hallucinated mark- ers and incorrectly associated details (see Figure 7). To enhance object description accuracy in complex RS images, we spatially guide the model by introducing prior knowl- edge in the query in the form of category name and location, along with a marked number. Thus, effectively guiding the model to generate a precise and comprehensive description.
Remote sensing imagery captures intricate semantic infor- mation and complex inter-object relationships across di- verse spatial scales. To enable LMMs to acquire a detailed comprehension ability, it is essential to integrate broad contextual views with object-level distinction. Address- ing the current deficit in datasets capable of facilitating tine-grained understanding of top-down perspectives, we introduce GeoPixelD, a dataset established to provide hier- archical descriptions derived through automated multilevel image analysis. GeoPixeID structures its descriptions at three primary levels: (1) holistic scene representation, (2) individual instance observations, and (3) densely populated object groups annotations (as depicted in Figure 3).
Specifically, we partition each image into 3x3 grid and lo- calize the objects by measuring their overlap with the quad- rants to determine their positional reference. This quadrant- based localization, combined with categorical labels and marked numbers, serves as positional and categorical priors for LMM, enhancing focus on the target objects, a process that proves effective given the densely packed and spatially complex nature of RS imagery, where objects often vary in
4.1. Holistic Image Annotation
Initially, we generated descriptive captions for RS im- ages using robust open source model, IXC (Zhang et al., 2024a), to capture comprehensive and diverse image details. We chose the IXC model (Zhang et al., 2024a) based on a comparative study con- ducted with other state-of-the-art vision language mod-
5

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
scale, orientation, and proximity.
(utilzing iSAID validation set images) undergoes meticu- lous manual curation, requiring "350 man-hours to ensure annotation completeness. We employed robust validation protocol where each geg description in the test set was metic- ulously verified by expert annotators. The process includes correcting for any omissions, inaccuracies, or partial annota- tions, including adjustments to object attributes that do not align with the corresponding image, thereby establishing a high-quality evaluation benchmark.
We evaluated various open-source and proprictary models for prior-informed SOM prompting in RS imagery (see Figure 8), comparing combined and individual querying approaches. ChatGPT (OpenAI, 2023) generated detailed descriptions with inferred information, while Gemini (Team et al.. 2023) and InterVVL (Chen al., 2024b) exhibited repetitive output as the target objects increased. InternL-- XComposer (Zhang et al., 2024a) achieved performance comparable to ChatGPT in terms of accuracy and diversity.
5. Experiments
4.3. Cluster/Crowd Annotation
Here, we explain the implementation details, present a com- parative performance analysis on Remote Sensing Grounded Conversation Generation (RS-GCG) and Referring Remote Sensing Image Segmentation (RRSIS), and include an abla- tion study to assess the impact of key components.
After identifying and annotating prominent large objects, the remaining objects are grouped or identified along with their spatial properties through a structured three-stage positional analysis. First, the image is divided into a xx3 grid, with each grid cell assigned a unique identifier. To enhance alignment with human perceptual tendencies, the central region of the grid is given a larger spatial weight. Next, 2 grid is considered for the localization of more dispersed objects. Finally, a half-image grid (1 x2 and 2x 1) assigns broader positional information. This gridding provides a systematic framework for the localization of clusters as well as large groups of objects. LMM then describes the group attributes using quantitative and positional information.
5.1. Implementation Details
The model is initialized with pre-trained Intern XComposer-2.55 model (IXC-2.5) with 7B parameters, using LoRA for efficient fine-tuning. A fixed CLIP ViT-L vision encoder with resolution of 560 is employed, along with a grounded vision encoder initialized from SAM2 weights. The trainable components include a pixel decoder (D), LoRA parameters (a = 8), a vision projector P... and language projector P.. The adaptive image divider uses a maximum patch number P to 9 for training. In our training process, with effective batch size of 20 over 10 epochs, the learning rate increases linearly to maximum value of 3 x 10-4 over the initial 100 training steps, followed by a gradual cosine decay. We train GeoPixel on the GeoPixeID dataset for the GCG task on two NVIDIA A6000-48GB GPUs, which takes around 3 days.
4.4. Unifying Annotations and Language Marking
For the preprocessed iSAID (Waqas Zamir et al., 2019) train dataset (Appendix A), we derive 16,795 holistic, 36,793 instance, and 17.023 group annotations, collectively encom- passing 600,817 objects. The annotations were rigorously filtered to eliminate aerial perspective inconsistencies, arti- facts such as marker identifiers, fore/background references, depth cues, and inconsistent descriptors.
5.2. Baselines
The key noun chunk corresponding to the object category in individual- and group-level annotations is tagged with unique identifiers (" "phrase-number") each linked to an in- stance or semantic mask, a process termed text marking. These marked annotations are combined with holistic scene representations into a single descriptive narrative. We em- ploy a Llama-3. 1. -instruct 8B (Dubey et al., 2024) LLM to paraphrase concatenated annotations while preserving their semantic integrity (see Figure 9). The L.LM processes the concatenated text under strict constraints to retain all marked phrases unchanged, ensuring a consistent link to associated visual masks. The outputs undergo iterative paraphrasing if any marked phrases are not preserved. By adopting this language marking strategy, the GeoPixeID dataset achieves a robust framework to generate high-quality, context-rich GCG descriptions precisely aligned with visual elements.
To rigorously evaluate the efficacy of GeoPixel, we intro- duce three robust baselines for comparative analysis on the GeoPixelD benchmark. First baseline LISAÍ, an enhanced LISA (Lai et al., 2024) model, incorporates multitarget seg- mentation masks within its output pipeline and an updated tokenizer that includes phrase tokens (<p> and < </p>) for the GCG task. The second baseline PixelI.M derived from PixeILM (Ren et al., 2024), is configured without the SAM encoder, the codebook image feature scaling factor of 2, three segmentation tokens, and the vision tower resize pa- rameter of 448. Phrase tokens are added, and <SEG> token is replaced with multiple codebook tokens. The third base- line, GLaMM, specifically GLaMM- GCG, model tailored for the GCG task. LISAÍ, PixelI and GI M-MMff mod- els are initialized from pretrained L-SA-7 PixeILM- 7B and GLaMM-GCG (7B), respectively, and additionally trained on GeoPixeID data.
Following similar procedures, test set GCG descriptions
6

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Table Performance Comparison on RS-GCG task. LISAf and PixeILMt denote pretrained LISA and PixelLM models adopted for
RS RS-GCG and finetuned on GeoPixelD training data. GLaMM represents the zero-shot performance, whereas GLaMM-FT refers to the pretrained model finetuned on GeoPixelD. GeoPixel outperforms other models across all metrics.
UNI-TARGET
MULTI-TARGET
OVERALL
MODEL
CIDER METEOR
CLAIR
AP50
MIOU
RECALL
AP50
MIOU RECALL
AP50
MIOU RECALL
GLAMM (CVPR'24)
0.1
5.8
43.11
1.2
18.1
14.8
0.5
16.5
6.3
0.5
16.9
7.1
LISAT (CVPR'24)
14.6
22.3
68.96
9.5
41.7
43.1
8.3
43.1
27.5
8.5
42.7
29.0
PIXELLMI (CVPR220)
18.3
22.5
73.93
13.5
41.2
44.0
10.4
42.9
28.1
10.5
42.4
29.6
GLAMM-FT (CVPR'20)
15.7
23.0
71.74
18.8
44.4
48.5
12.4
47.1
31.1
12.5
46.4
32.8
GEOPIXEL
21.6
24.0
77.50
25.5
50.8
55.6
18.0
52.9
37.0
19.0
52.3
38.8
Table 3. Performance Comparison of GeoPixel in Referring Ex- pression Segmentation on RRSIS-D dataset. The segmentation accuracy based on referring expressions is expressed through the Precision at IoU threshold of 0.5 (P@O.5), Overall Intersection- over- Union (oloU) and Mean Intersection-over Union (mloU).
Table 4. Performance Comparison on Referring Expression De- tection Task, reporting Acc@0.5 Acc@0.7 across categories: unique, non-unique and overall. GeoPixel demonstrated substan- tial gains over existing methods in referring expression detec- tion/locllization.
VALIDATION
SET
UNIQUE
NON-UNIQUE
METHOD
P@0,
oloU MIOU P@0.
oloU MIoU
MODEL
A.c@0.5 Acc@0.7
Acc@0,5 Acc@0.7
AcC@0.5 AcC
RRN (L1 AL., 2018)
51.09
66.53
46.06
51.07
66.43
45.64
MINIGPT-V2
40.7
32.4
15.2
35.8
CSMA (YE ET 2019)
55.68
69.68
48.85
55.32
69.39
48.54
LLAVA-L5
51.1
16.4
34.8
11.5
41.5
13.6
LSCM (Hur AL., 2020)
57.12
69.28
50.36
56.02
69.05
49.92
MINI-GEMNI
41.1
4.9
30.1
CMPC (HUANG CT 2020)
57.93
70.15
50.41
55.83
69.22 49.24
GUOCHAT
57.4
22.6
44.5
18.0
49.8
19.9
BRINET (Hu ET AL., 2020)
58.79
70.73
51.14
56.90
69.88
49.65
GEOPIX
57.0
22.7
44.8
18.2
49.8
20.0
CMPC+ (LIV AL., 2022)
59.19
70.14
51.41
57.65
68.64 50.24
GROPIXEL-FT
70.37
41.54
65,80
40.32
67.70
40.83
LGCE (YUAN ET AL., 2024)
68.10
76.68
60.16
67.65
76.34 59.37
LAVT (YANG ET AL., 2024)
69.54
77.59
61.46
69.52
77.19 61.04
RMSIN (LIU ET AL.. 2024c)
74.66
78.27
65.10
74.26
77.79
64.20
imagery guided by textual descriptions. The input prompt used is: "Could you provide segmentation in this image?" The model generates the response, "Sure, it is <SEG>. " where the corresponding embeddings of <SEG> token is subsequently decoded produce the segmentation mask. To address this task, we fine-tune the GeoPixel model on the RRSIS-D (Liu et al., 2024c) dataset. The resulting GeoPixel-ft model demonstrates superior performance compared to recent approaches, as shown by results on the RRSIS-D test and validation sets in Table 3. The qualitative results are provided in Figure 5.
GEOPIXEL-FT
80,00
81.77
67.99
83.33
84.90
67.30
mask for [referring.expressicee
5.3. Results
Remote Sensing Grounded Conversation Generation: A comparative analysis of the performance of various models on the RS-GCG task is evaluated across different metrics, including CIDEr, METEOR, CLAIR (Chan et al., 2023), AP50, mloU, and recall, segmented into Uni-Targed Target, Multi- Target, and Overall categories. CL CLAIR is an M.LM-based sim- ilarity metric (GPT-40 in our case) that better aligns with human judgments. GeoPixel demonstrates superior perfor- mance in all metrics compared to the baselines, showing bet- ter fluency and text relevance in textual outputs while main- taining strong performance in more complex multi-target scenarios. In contrast, LISAt struggles with segmentation- based tasks, as evidenced by its low AP50 scores in all categories. PixelLMt shows a moderate improvement over LISAf, benefiting from better image feature scaling and seg- mentation token adjustments. GLaMM-ft exhibits improved outcomes due dedicated grounding encoder and GCG pre-training, however, its performance remains inferior to that of GeoPixel (as detailed in Table 2). Qualitative results are presented in Figure 4.
Referring Remote Sensing Image Detection: We for- mulate referring expression detection (RED) post- processing task on GeoPixel's predictions, deriving horizon- tal (HBBs) and oriented bounding boxes (OBBs) from out- put masks, for consistent evaluation. Table 4 compares per- formance on VRSBench (Li et al., 2024a), a dataset based on DOTAv2 and DIOR. To prevent data leakage, GeoPixel (trained on GeoPixeID) is finetuned on VRSBench with- out using any data from RRSIS-D (DIOR-based dataset), Moreover, GeoPixelD and VRSBench use DOTA's training set for training and validation set for testing. We further compare GeoChat and GeoPixel using OBBs (Table 5) with both models finetuned on VRSBench, highlighting the im- portance of pixel alignment in LMMs. Figure 10 illustrates the qualitative results.
Referring Remote Sensing Image Segmentation: This task focuses on segmenting specific regions in aerial
7

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
The residestial strial photagramn of surroatsed by trees. It teatures too vater, suggesting areainity to lake positarrin of the houses indicate to actersing
The view of spurts comples, festuring ruminng track with red rimning surface and green surrounding acditional storts focilities, The the center
suriel view perking cesignated
The lot with runerous cars parked in indicating arranged orderty well
sorking system, The porking lot aspears larger fscalaty. supple, tudestted the prevence of trees and other structures. the imagu large parsing area inctuding and possibly coaches, sarked a coganized conrer. There are mntepl
The iroge on acrial photograph of rural area with road cottang through it, aspeardnn to tao lane hishway it.
adouraar setting, with straight, perpersiiaaa tee elmpstee, straight, ant seteeds tre the spructores or osjeets on it, with blus vater, and surrarreed by sam deck,
part
shape, surrounded
with trees, and the soarn 2als at the right well rotntsined grassy with vistble aorkings The stractures, which could be seating areas or other facilities, are visible trees and
into the the end
vebicles traeeling surrounding lardseape is predneinsnt dry and sparsely vegetated, indicatioo read, there are for 233 con a The scone has ratural and undevelosed aspeorance, no visible suildings or than
spaces supgeats that the coeplex. is designed for ousdsor astivities
conee drivesay at the lottos left of the Lage, adjacert to greseery and the aboetce conmentaa butlstings hin this
regiona Hhe abvence of people the smape could iolly that the photo ws taken daring time lov activity or polnt individiats are not easily discernible.
itself.
possibly comunisy and serene steesphere to the absence people in the scene.
strictures qutet, peacraly afrtuent
Figure 4. Qualitative results of GeoPixel on GCG. Contextually rich descriptions of RS imagery with grounded object annotations. Depending object scale and density, it employs instance masks for precise delineation of individual objects (right and middle- right images) while semantic masks capture broader categories, such as large clusters of vehicles or small objects (middle-left and left images).
Table 5. Visual grounding performance on VRSBench dataset us- ing oriented bounding boxes for referring object localization. The observed performance improvement underscores the critical role of precise pixel-level aligmment.
Table 6. Effect of Inference Resolution. Reported metrics show the relationship between resolution and overall performance.
TRAINING INFERENCE CIDER METEOR AP50
MIOU RECALL
PATCHES PATCHES
MODEL.
UNIQUE
NON-UNIQUE
OVERALL
P
1
14.6
23.1
12.9
47.8
32.2
Acc@
ACc@0.7
Acc@0.5
ACc@0.7
A.c@0.5
0.5
ACc@0.7
P=
P
P=0
17.7
23.9
16.6
51.8
37.1
p
20.5
24.3
17.6
52.1
37.4
GEOCHAT
32.3
12.6
18.5
5.7
24.3
8.6
GEOPIXEL-FT
54.48
24.87
60.51
58.00
28.42
Table 7. Effect of Annotation Complexity. Avg. Len is the average
5.4. Ablation Study
character length of captions.
Inference Resolution Effect: Increasing inference patches improves all evaluation metrics, reflecting better compre- hension of visual content (Table 6). At P = 9, CIDEr increases from 14.6 to 20.5, and METEOR improves from 23.1 to 24.3, indicating enhanced semantic understanding as image tokens scale up. The moderate mAP and mloU gains suggest that while HR inference contributes to supe- rior localization accuracy, competitive performance can still be maintained at lower resolutions when the model is pre- trained at HR. The superior results at (P- = 9) underscore the critical role of incorporating fine-grained spatial details during the training phase for generalized feature learning.
DATA
OBJECTS PHRASES AVG. LEN
MIOU RECALL
INSTANCES ONLY
1,740
1,740
634
58.4
48.8
SEMANTIC ONLY
21,483
698
518
44.1
37.7
MIX DATA
38,161
2,989
737
50.9
33.3
and semantic mask representationssas seen in Figure 1). Table 7 shows that this annotation complexity impacts per- formance, with mixed annotations yielding the lowest recall.
Remote sensing images often contain visually similar ob- jects with subtle variations in appearance, spatial layout, and proximity, yet exhibit significant scale variations across images. This complexity challenges the model's ability to distinguish between object presence, quantity, and the appro- priate annotation type required (e.g., instance or semantic). The challenge is pronounced in the semantic-only category, where the model exhibits the lowest mloU scores, indicat- ing two key challenges: the model's ability to ensure full coverage of all instances within a category and grouping
Annotation Complexity Effect: GeoPixel adjusts its mask- ing based on object size and distribution (Figure 4), using instance masks for precise identification, while semantic masks for broader categories (clusters/small objects). In scenarios requiring both granularity and generalization, the model employs hybrid annotations, blending instance-level
8

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
could you provide segnentation nask for tennis court on the right in this image?
Could you provide segnentation mask for red vehicte this image?
Could you provide segnentation mack for on oval ground track tield in this inage?
Could you provide a segrentation mask for a windmill on the Lawer right in this trage?
Could you provide a segsentation mask for the beseball field on the Lower right of the bascball field on the top in this inage?
Could you previde a segnentation rask for a ground track rield above the airport in the niddle in this image?
Figure 5. Qualitative results of GeoPixel's capability referring remote sensing expression segmentation. The figure highlights Geopixel's s ability to interpret referring expressions of varying lengths and generate precise segmentation masks, adapting to scale variations, as shown the ground track fields. Spatial descriptors (e.g "right", "lower right"), and object characteristics (e.g "red") are interpreted with precision to achieve accurate segmentation.
insights into its ability to generalize across different scales and segmentation strategies. The results indicate that the inclusion of more complex annotation (Set- 1B) enhances segmentation accuracy and descriptive detail, as the model is trained with more diverse mask configurations.
Table Effect of Data Complexity and Training Vision Projection (VP) Layer. T stands for Trainable and F for Frozen.
TrainInG DATA VP CIDER METEOR AP50 MTOU RECALL
SET-1A SET-1B
Next we study the effect of training the vision projection layer by comparing the performance when the vision projection layer is fixed trainable during the fine-tuning stage. Table 8 summarizes the results. Training the vision projection layer results in an improvement in some metrics, highlighting the role of feature aligmment.
Vision Projection:
19.3
23.6
18.2
48.0
33.6
4
T
20.5
24.0
17.8
51.7
36.7
7
18.7
24.4
15.3
51.6
35.1
objects under a unified mask instead of individual instances. The low mask recall in mixed data further suggests that gen- eralizing masking decisions in dense scenes is difficult due to scale and distributional variability of geospatial entities.
6. Conclusion
Role of Data Complexity: Table 8 compares GeoPixel's 's performance across data partitions with varying masking complexity. S is less complex, with no intraclass segmentation differences. Each instance of a single class is individually masked or represented using a semantic mask. Se introduces higher complexity by assigning instance masks to larger objects while grouping smaller ones under semantic mask within the same class. For example, larger boats may be individually described, while smaller boats in the image could be grouped under a single semantic description. This ablation evaluates the model's s ability to handle varying levels of annotation granularity, providing
We present GeoPixel, large multimodal model (LMM) designed specifically for the unique challenges of high- resolution remote sensing (RS) image analysis. GeoPixel in- troduces a robust end-to-end architecture capable of adaptive image partitioning and pixel-level grounding, enabling the precise interpretation and generation of geospatially aware descriptions in RS imagery. By addressing key limitations of current LMMs, such as low-resolution constraints and coarse object-grounding, GeoPixel provides ine-grained visual understanding that bridges the gap between language and high-resolution RS data.
9

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Impact Statement
Bai, J., Bai, S., Yang. S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C.,, and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint mrXiv:2308.12966 1(2):
GeoPixel, large multimodal model, is designed to enhance the fine-grained spatial understanding of high-resolttion remote sensing (RS) imagery. Our work contributes to the advancement of machine learning by improving pixel-level object grounding, a critical capability for geospatial appli- cations such as urban planning, environmental monitoring, disaster response, and infrastructure assessment.
3, 2023b.
Bazi, Y., Bashmal, L., Al Rahhal, M. M., Ricci, R., and Melgani, F. Rs-llava: A large vision-language model for joint captioning and question answering in remote sensing imagery. Remote Sensing, 16(9):1477, 2024.
From an ethical perspective, GeoPixel operates on publicly available remote sensing datasets, ensuring transparency and reproducibility in research. However, the use of high- resolution imagery raises privacy considerations, particu- larly in applications that involve urban environments. Al- though our model does not process personally identifi- able information (PII) or real-time surveillance data, we acknowledge that future extensions incorporating private datasets should adhere to strict data governance and privacy- preserving policies.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H., et al. Internlm2 technical report, 2024.
Chan, D., Petryk, S., Gonzalez, J. E.,, Darrell, T., and Canny, J. Clair: Evaluating image captions with large language models. arXiv preprint arXiv:2310.1297. 2023.
Chen, G., Shen, L., Shao, R., Deng, X., and Nie, L. Lion: Empowering multimodal large language model with dual- level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 26540-26550, June 2024a.
Regarding social impact, the ability of GeoPixel to gener- ate detailed geospatially grounded descriptions can help in disaster relief efforts by improving situational awareness from satellite imagery. It can also improve climate change monitoring by facilitating precise land use classification and deforestation tracking. We emphasize that GeoPixel is devel- oped for scientific and humanitarian purposes and encourage ethical deployment in accordance with open research prin- ciples. Overall, this work aligns with the broader goal of advancing machine learning in remote sensing, promoting open and responsible Al applications, and fostering positive societal impact while remaining mindful of potential ethical considerations.
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F. and Zhao, R. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint wxXi:2336.151999 2023.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q.. Zhu, X., Lu, L., Li, B., Luo, P., Lu, T.. Qiao, Y., and Dai, J. Internvl: Scaling up vision foun- dation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24185-24198, June 2024b.
Acknowledgement
The computations were enabled by resources provided by NAISS at Alvis, partially funded by the Swedish Research Council through grant agreement no. 2022-06725, LUMI hosted by CSC (Finland) and LUMMI consortium, and by Berzelius resource provided by the Knut and Alice Wallen- berg Foundation the NSC
Chiang. W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng. L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, F. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:/ hws./mmasssssoosao. org/boog 2023-03-2
References
Dai, Li, D., Tiong, A., Zhao, J., Wang. W., Li, B., Fung, P. N., and Hoi, S. Instructblip: Towards general- purpose vision-language models with instruction tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 49250- 49267. Curran Associates, Inc., 2023.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P. Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang. S., Yao, Y., Yu, B., Yuan, H. Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Ding, G., Khan, S., Tang, Z., Zhang, J., and Porikli, F. Towards better validity: Dispersion based clustering for unsupervised person re-identification. arXiv preprint arXiv:196.00/000 2019.
10

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., Zhang, W. Li, Y., Yan, H., Gao, Y., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang, X., Qiao, Y., Lin, D., and Wang, J. Internin-xcomppsser2 Mastering free-form text-image composition and comprehension in Isiounlanguage large model. arXiv preprint wrXiv22401.16200 2024a.
Huang, S., Hui, T., Liu, S., Li, G., Wei, Y., Han, J., Liu, L., and Li, B. Referring image segmentation via cross- modal progressive comprehension. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. (0488-10497, 2020.
Hui, T., Liu, S., Huang, S., Li, G., Yu, S., Zhang, F., and Han, J. Linguistic structure guided context modeling for referring image segmentation. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 59-75. Springer, 2020.
Dong, X., Zhang, P., Zang, Y., Cao. Y., Wang. B., Ouyang, L.,, Zhang, S., Duan, H., Zhang, W., Li, Y., Yan, H., Gao, Y. Chen, Z., Zhang, X., Li, W., Li, J., Wang, W., Chen, K., He, C., Zhang, X., Dai, J., Qiao, Y., Lin, D., and pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512, 2024b.
Wang, J. Internlm-xcomee howppll-xcoooose22--e1
Irvin, J. A., Liu, E. R., Chen, C., Dormoy, I., Kim, J., Khanna, S., Zheng, Z., and Ermon, S. Teochat: A large vision-languasn assistant for temporal earth observation data. In International Conference on Learning Represen- tations, 2025.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani. M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https:// openrevi ew, wwwpeeewwwwwweeeeeeeeeeeeeeeeeee forum? d=Y cbFáNTTy.
Javaheripi, M., Bubeck, S., al. Phi-2: the surprising power of small language models (2023). URL https://www. microsoft. pfwer-of- saaas--mllll-aaaaa
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead. S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Guo, Q.. De Mello, S., Yin, H., Byeon, W., Cheung, K. C.,, Yu, Y., Luo, P., and Liu, S. Regiongpt: Towards region un- derstanding vision language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR), pp. 13796-13800 13806, June 2024.
Kuckreja, K., Danish, M. S., Naseer, M., Das, A., Khan, S., and Khan, F. S. Geochat-grounded large vision-language model for remote sensing. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 27831-27848 2024. doi: 10.1 10.110999VPR552733.20000 02629.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979-15988, 2022. doi: D0.100//CVPR7768222222.055
Lai, X., Tian, Z. Chen, Y., Li, Y., Yuan, Y., Liu, S., and Jia, J. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579 9589, 2024.
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. Cogagent: A visual language model for gui agents. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-1211 -44900, 2024.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-imen -mage pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023,
Hu, Y., Yuan, J., Wen, C., Lu, X., and Li, X. Rsgpt: A remote sensing vision language model and benchmark. arXiv preprint 2rXiv:2307.15260 2023.
Li, R., Li, K., Kuo, Y.C.,, Shu, M., Qi, X., Shen, X., and Jia, J. Referring image segmentation via recurrent refinement networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5745-5753, 2018. doi: 10.1109/CVPR 8.0000..
Hu, Z., Feng, G., Sun, J.. Zhang, L., and Lu, H. Bi- directional relationship inferring network for referring image segmentation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4423 4432,,43,220 2020. doi: 10.1109/CVPR42660.2222 00448.
Li, X., Ding, J., and Elhoseiny, M. Vrsbench: A versatile Aision-language benchmark dataset for remote sensing image understanding. In Globerson, A., Mackey, L., Bel- grave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C.

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
(eds.), Advances in Neural Information Processing Sys- tems, volume 37, pp. 3242. Curran Associates, Inc., 2024a. URL ptttps::/prroceediing.s neurips. cc
Liu, Z., He, Y., Wang, W., Wang, W., Wang. Y., Chen, S., Zhang, Q. Lai, Z., Yang, Y., Li, Q., et al. Interngpt: Solv- ing vision-centric tasks by interacting with chatgpt be- yond language. arXiv preprint mmXiv22355.0660 2023b.
cc/paper paper htgpeer_iiieee/ppprr///////////
Luo, J., Pang, Z.., Zhang, Y., Wang, T., Wang, L., Dang, B., Lao, J., Wang. J.. Chen, J., Tan, Y., et al. Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding. arXiv preprint M.ii::2406.101000 2024.
and Hed_BBenchmrrk_Traeee
Li, Z., Yang, B., Liu, Q.. Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., and Bai, X. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 26763-26677 2024b.
Ma, C., Jiang, Y., Wu, Yuan, Z., and Qi, X. Groma: Local- ized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pp. 417-435. Springer, 2025.
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C.. Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575. 2023.
Muhtar, D., Li, Z., Gu, F., Zhang, X., and Xiao, P. Lhrs- bot: Empowering remote sensing with vgi-enhanced large multimodal language model. arXiv preprint arXiv:2402.02544, 2024.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava-y wwtp:/wwwww.giw...o.... wtatthu.../blloo. 2024-01-30-1aav
OpenAL. Chatgpt: Language model for dialogue applica- tions. http:::/t///t /tppnappnaaa///////ee//ee com/ chatgpt, 2023. Ac- cessed: 2024-12-31.
Ou, R., Hu, Y., Zhang, F,, Chen, J., and Liu, Y. Geopix: Multi-modal large language model for pixel-level im- age understanding in remote sensing. arXiv preprint arXiv:2501.06825 2025.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Pang, C., Wu, J., Li, J., Liu, Y., Sun, J., Li, W., Weng, X., Wang, S., Feng, L., Xia, G.-S., et al. H2rsvlm: Towards helpful and honest remote sensing large vision language model. arXiv preprint arXiv:2403.20213 2024.
Liu, S., Hui, T., Huang, S., Wei, Y., Li, B., and Li, G. Cross-modal progressive comprehension for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 440))4461-4775, 2022. doi: TO..O//PAAL..001 1LL10/TTAMM.2221.00799999999
Peng, Z., Wang, W., Dong, L., Hao, Y., Huang. S., Ma, S., Ye, Q., and Wei, F. Grounding multimodal large lan- guage models to the world. In The Twelfth International Conference on Learning Representations, 2024.
Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.0544 2023a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. $748-8763 PMLR, 2021.
Liu, S., Ma, Y., Zhang, X., Wang, H., Ji, J., Sun, X., and Ji, R. Rotated multi-scale interaction network for re- ferring remote sensing image segmentation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26648-26653 2024c. doi: 10.100/CCVPP37733.2222.0011
Rasheed, H., Maaz, M., Shaji. S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E.,, Yang, M.-H., and Khan, F. S. Glamm: Pixel grounding large multimodal model. In 2024 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 13009-13018, 2024. doi: 10.
Liu, X. and Lian, Z. Rsunivlm: A unified vision language model for remote sensing via gramularit--oriented mixture of experts. arXiv preprint arXiv:2412.05567. 2024.
Ravi, N., Gabeur, V., Hu, Y.-T,, Hu, R., Ryali, C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, L., Mintun, E., Pan, J., Alwala, K. V., Carion, N. Wu, C.-Y,, Girshick, R. Dollár, P., and Feichtenhofer, C. Sam 2: Segment anything in images and videos. arXiv
Liu, Y., Yang, B., Liu, Q. Li, Z., Ma, Z., Zhang, S., and Bai, X. Textmonkey: An ocr-free large multi- modal model for understanding document. arXiv preprint arXiv:2403.0 044733 2024d.
12

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
preprini arXiv:2408.0077 00714, 2024. arxiv. abs / 2400 00714.
URL https://
Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, J. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310.11441. 2023.
Ren, Z., Huang, Z., Wei, Y., Zhao, Y., Fu, D., Feng, J., and Jin, X. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 26374-26383 June 2024.
Yang, Z., Wang. J., Ye, X., Tang, Y., Chen, K., Zhao, H., and Torr, P. H. L.nguage-aware vision transformer for referring segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-18, 2024. doi: 10. 10.1109/PAMM.220444446688
Ryali, C., Hu, Y.-T., Bolya, D., Wei, C., Fan, H., Huang, P.-Y., Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoff- man, J., Malik, J., Li, Y., and Feichtenhofer, C. Hiera: A hierarchical vision transformer without the bells-and- whistles. ICML, 2023.
Ye, J., Hu, A., Xu, H., Ye, Q. Yan, M., Xu, G., Li, C., Tian, J., Qian, Q. Zhang, J., et al. Ureader: Uni- versal ocr-free visually-situated language understanding with multimodal large language model. arXiv preprint 23X00:2310.055 /22, 2023a.
Soni, S., Dudhane, A., Debary, H., Fiaz, M., Munir, M. A., Danish, M. S., Fraccaro, P., Watson, C. D., Klein, L. J., Khan, F. S., et al. Earthdial: Turning meltorsensory earth observations to interactive dialogues. arXiv preprinl arXiv:2412.15199, 2024.
Ye, L., Rochan, M., Liu, Z., and Wang, Y. Cross-modal self-attention network for referring image segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10502-10511, 2019.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K. et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint M3Xx::2304.11178 2023b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.1 13971, 2023.
You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F,, and Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.0770. 2023.
Waqas Zamir, S., Arora, A., Gupta, A., Khan, S., Sun, G., Shahbaz Khan, F., Zhu, F., Shao, L., Xia, G.-S., and Bai, X. isaid: A large-scale dataset for instance segmentation in aerial images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 28-37, 2019.
Yuan, Z., Mou, L., Hua, Y.. and Zhu, X. X. Rrsis: Referring remote sensing image segmentation. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Zhan, Y., Xiong, Z., and Yuan, Y. Skyeyegpt: Unify- ing remote sensing vision- laiiunla tasks via instruc- tion tuning with large language model. arXiv preprint DoXiv:240109777 2024.
Wei, H., Kong, L., Chen, J., Zhao, L., Ge, Z., Yang. J., Sun, J. Han, C., and Zhang. X. Vary: Scaling up the vision vocabulary for large vision-language model. In European Conference on Computer Vision, pp. 408-424. Springer, 2025.
Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., Guo, Q.. Duan, H., Wang, B., Ouyang, L., Zhang, S., Zhang, W., Li, Y., Gao, Y., Sun, P., Zhang, X., Li, W., Li, J., Wang, W., Yan, H., He, C., Zhang, X., Chen, K., Dai, J., Qiao, Y., Lin, D., and Wang. J. Internlm-xcomposer 2.5: A versatile large vision language model support- ing long-contextual input and output. arXiv preprint AXXV:2407.033200 2024a.
Xia, Z., Han, D., Han, Y., Pan, X., Song, S., and Huang, G. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 3858-3869, June 2024.
Zhang, S., Sun, P., Chen, S.,, Xiao, M., Shao, W., Zhang, W., Liu, Y., Chen, K., and Luo, P. Gpt4roi: Instruction tuning large language model on region-of-interess. arXiv preprint arXiv:2307.0360) 2023.
Xuan, S., Guo, Q. Yang, M., and Zhang, S. Pink: Unveiling the power of referential comprehension for multi-modal llms. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 13838-13845 June 2024.
Zhang, W., Cai. M., Zhang, T., Zhuang, Y., and Mao, X. Earthgpt: universal multimodal large language model
13

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
for multisensor image comprehension in remote sens- ing domain. IEEE Transactions Geoscience and Re- mote Sensing, 62:1-20, 2024b. doi: 10.1109/GRS.22244 3409624.
Zhang, W., Cai, M., Zhang, T., Zhuang, Y., and Mao, X. Earthgpt: A universal multi-modal large language model for multi-sensor image comprehension in remote sensing domain. IEEE Transactions on Geoscience and Remote Sensing, 2024c.
Zhang, Y. Ma, Z.., Gao. X., Shakiah, S.. Gao, Q. and Chai, J. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14227-14238, June 2024d.
Zhao, Y., Lin, Z., Zhou, D., Huang, Z., Feng, J., and Kang, B. Bubogpt: Enabling visual grounding in multi-modal Ilms. arXiv preprint 2xiv:2307.0888/ 2023.
Zhou, Y., Lan, M., Li, X., Ke, Y., Jiang, X., Feng, L., and Zhang, W. Geoground: A unified large vision- language model. for remote sensing visual grounding. arXiv preprint arX1v:2411.11904, 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 2023.
14

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
A. Limitations and Challenges
While GeoPixel has demonstrated significant advances in pixel-level grounding for high-resolution RS images, several challenges remain. These challenges are particularly evident in the following failure cases (illustrated in Figure 6). The model occasionally produces erroneous masks due ambiguities in the masking strategy, particularly in determining object presence and quantity, well deciding whether semantic segmentation instance-level annotation is appropriate. An incorrect decision in this regard can result in repetitive descriptions of visually similar objects, leading to inconsistencies in the generated output. Furthermore, such errors may manifest as fragmented or overlapping masks, introducing confusion in object delineation and undermining the overall segmentation quality. Moreover, the model often confuses instance masks within the same spatial location, particularly in densely populated or crowded images.
Future work may foeus on addressing these challenges by incorporating more robust masking strategies and dynamic resolution adjustment techniques improve segmentation accuracy in complex scenes. Additionally, extending GeoPixel's capabilities to integrate multimodal data, such as Synthetic Aperture Radar (SAR) or infrared imagery, could significantly enhance its ability to analyze diverse remote sensing datasets. GeoPixel is significant step forward in leveraging the potential of LMMs for remote sensing, opening new avenues for research in this critical domain.
The image and wertal multapic sirport,
areas and clas naway. There
the some static araniized Horsaly,
Lang commrrcaa
staooming cetse arronganent of ara pattom. with sariess notl end pesuiity
visabic
the Bitran the tao, them commirrene the the right, tor the ue.
bitton ragtt trggume pecit
the
peeele evvire ceticiee. sitanie perpesticatar
tw "istt aroo, tabidegge vitleger
crand winh onter
the le cigst dhorectouee to its preeenee spperte
abiste Eifferst lecetarn. cras extrveeet' loyest and deston.
metsagale
worecs coter and artivity
the laece earting setiee la wcliesty.
Figure 6. Failure case due to incorrect mask association (left) and wrong instance segmentation in the same spatial region (right).
B. GeoPixeID dataset
Preprocessing and Marking: We utilize the instance-level annotated dataset, iSAID (Waqas Zamir et al., 2019), to generate grounded conversations through our annotation pipelines. The images undergo a preprocessing step in which they are cropped into 800 x 800 pixel patches. Objects for instance annotations are selected based on an area threshold to ensure their reasonable size, therefore preventing the marker from obscuring a significant portion of the object and maintaining its distinguishability. A 14 x 14 pixels fixed size marker is used, regardless of the actual dimensions of the object. However, the marker's 's placement is determined based on the segmentation mask's area and shape. For large objects, the marker is positioned at the center of the mask if the caleulated center falls within the mask boundaries; otherwise, it is adjusted to the nearest point on the object's border. For small objects, the center of the bounding box is aligned with point on the polygon mask boundary, which typically results in an average marker overlap of 50% with the object.
In addition, multiple marking techniques were explored, including bounding boxes, masks, contours, and numerical markers, to determine their impact on accuracy and object fidelity. Our findings reveal that bounding boxes and contours add superfluous visual information that can obscure the fine details of the object. In contrast, simple numerical markers placed
15

---

driving on GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
directly on the object effectively signal its presence without compromising visual clarity introducing noise, thereby preserving the integrity of object details for enhanced model performance.
Set-of-Mark Prompting
Set-of-Mark Prompting with Prior
narked by [sarkers, str] wallyring rap- The description ion should enurerated in the rollowing forat: 1. <descristions 2. -descraptions
Describe objectrebjects of
down (category cams at the (pos that number Number) short sentence. not rention the marked number cutsut.
the
rountabout at
of intersection star-shaped sculsture, How.
roundabout at the conter distinctive pattern,
2. To the north of the roundatout, there is aulti-story huileing with flat ronf, which appears to be an Institurions rocal or corsercial structure.
2. The vehicle at the top right is derk-colored sedan.
To building in the north.
another nulti- roundabout, architectural
whiclee the top seall, derk-colored car,
The
4. To the south of the roundabeut, there is a sailler
dark-colared sodan,
bustding with a flat roof, which could be an office or building.
The vahicle at the top right is sna sark-colored car parked perpendicatar
parking with 5. There is a grassy area the southeast of the roundabout, which provides green space anidst the urban ewirerment.
curb, adjacent
several
indicating
public racility nearbyy
The vehicle top blse parked in designated parking spot.
Figure 7. Comparative effectiveness of SOM prompting methods, highlighting the critical role of priors. Without priors, SOM relies solely the VLM to detect and describe marked objects independently, resulting inaccurate descriptions and hallucinated markers in complex remote sensing scenes. contrast, SOM with priors utilizes explicit marker positions ((pos)) }) and predefined object categories (caaegory_naane) as priors, providing structured prompts that reduce ambiguity and guide the VLM produce precise and reliable descriptions. Incorrect parts are noted in red whereas correct parts are noted in green.
List down all attributes vehicte left one short sentence. Do not irenton the rarked nurber the output.
narked number
appears conpact car, is traveling curved adjacent parking storage shipping containers.
vehicle
The vehicle curved road.
The vehicle at the left is seni-truck with blue cab and an attached orange container.
The vehicle is a blue truck.
The plane in the center is parked at a gate, has a white fuselage with wings extended, apprars to be a coneercial sir Lire, and casts shadow on the tarnac due to sighight frot the tog left.
1. plane at
stripe and parked at gate. 2. plane the right
blue stripe and is parked at gate,
2. The plane on the right is parked at gate, has white fuse with extenced wings, appears to be comsercial airliner, and casts shadow to the lower right side due to the sunlight fron the top left.
3. The plane at the center is white with
a red stripe and is parked at gate.
3. The plane in the center is parked
gate, has white Tuselage with extended wings, appears shadou toward the sunlight coming fros the upper
1. The plane at the center is large comnrccil aircraft with predoninantlly white body and blue engines,
be
connercial eirliner, lower left side left.
2. The plane at the right is large connercial aircraft with predoninantly white body. and blue tnes.
The plane
parked
3. The plane the center is large comnercial aircreft with engines positioned terminal gate.
plane right and tarrec. parked on
The plane center tarsac.
facing
Figure Comparison open-source and proprietary models prior-informed set marks (SOM) prompting for RS imagery. Incorrect parts are noted in red whereas correct parts are noted in green.
16

---

GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Prospt Tesplate: Paraphrase the tollovire description of image viewed fror satellite in sing le pareoraph while ensuring that all the vords foreground, or canera position. Trage
enclosed in single quetes are included and rresereed exactly as they are. Rerove any mention of Description: (Holistic Image Annotation) (0: last) (Individual Instance Annctation] (Cluster/trowded Annotation) (Holistic Inage mmotatioo]]asst
Preept: Paraptrase the tollovirg sescristion of on wille esering that all the viris anc losed paogrash quetes
Freept: Paraptrace the fellocirg deseriaties of taag returing thst all the vares seclnted single gotes inslated ard presereed pectly tyy arc. lomore ississer. trage duseriation: feregrasre urage
they art. any senticr of aidgrrnst feregraste casers thetograph
preserete
lesse Image Srecriation: with dacke are cersprtteion use. The airoodda land teatures aessibly afflucts tsigitorhood, The
vie corrts are prronge in gras pattern, eich with its precscte, unbieeting the presmi
state, The sppeart sucoettirg recrestionsl with
trees
at the
with bls staying cirthcat, carted for slny, sarroueded atjocnt people virinle courte, sugpecting that play
strecturs that includes recf are sppears to case best ond Enniiall
structure extesting over the sater, arto
Description: serual shologrash residential oner drsks and buss, cats
crd, odlerting ints water consciol vodes rectoog far clear
conrectss
the loft clongite
wits Tu-tark, ard ie cecraogutar. the right is the right white with a center sest with sumple sclbord eater, dicked pases
smal:
The
pearetil, aseetbly sirgle teat
the
with stractum including out case wood. Herry, the
resrselion. tremaporaat isc, ladd gueasery private proserty, inttrating relgnartoon..
ceeerr
carmunteed
The bottes
stracture estending zer the sater, cornected to dacking relll structure at ite ent, extendiig ints the sater and eutaring ptels, sorressated desk man with lrange ure rectargs
with longe steies, The
veseen pier. The
repoors scodes the
anclun mist eaier Risutall Rovuiii wiit toal. white best with single with sargle Hoasen piers, stull wator perpestirutar The scene is trapgell, with no visible scrivity.
serinu basts ribille
ccs Descriptize: The Joage is rise of Teant tornas turmasteed crweeery sejaowir to parking petters, with snidet greasery, indicating the presence of tress or landscaping.
seppodittllr Mislc wily sargle cakin and awcked woden There clstate trangut arnocatere.
the
all
playtro
play,
the
and markings, astusted
Figure 9. Unifying Annotations through LLM Paraphrasing and Text Marking to track associated masks. Objects are indexed numerically (c.g.. "object-N"), and holistic (blue), individual (tcal), and cluster (green) annotations are concatenated into single image description. Paraphrasing instructions with combined description produce concise, consistent GCG description that eliminates redundancy while preserving object-mask associations, even with reordering.
fould you poouide a segnastation mack
you srauide secaentat fon axsk for the frane at the mandie right aids?
Costs you praciie segnentatize mask for the beyord the visible barde-?
of the snaper
Guild you provide a segnastacion nack for right Imago?
Guild YOU provide sopnentation on naok for the
Could provide segeentation mask for the saall vetsete located in the bettos- left part
Figure 10. Qualitative results of GeoPixel's capability in Referring Expression Detection on VRSBench. Segmentation masks generated by GeoPixel are used to derive horizontal bounding boxes (shown in bluc) and oriented bounding boxes (shown in red).
17


Reviews:
[{"review_id": "TmRMcSvlgO", "reviewer": "Reviewer_Amru", "summary": "The paper presents GeoPixel, a novel large multimodal model (LMM) that advances high-resolution remote sensing (RS) image analysis by integrating pixel-level grounding with textual understanding, addressing limitations in existing RS-LMMs. Its architecture features an adaptive image divider for processing 4K-resolution RS images through global-local patch fusion, combined with vision encoders (CLIP ViT-L and SAM-2), the InternLM2 language model, and a pixel decoder for mask generation. GeoPixel demonstrates excellent performance on RS-GCID tasks and RRSIS segmentation tasks; additionally, a key contribution is the semi-automatically constructed dataset GeoPixelD, which serves as an effective resource for pixel-level understanding of RS imagery.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Amru,  \n\nThank you for your detailed review and constructive comments regarding our submission on GeoPixel. We appreciate your taking the time to analyze our work and the insights you provided. Below, we address your comments and concerns:  \n\n**Clarity on Adapted Baselines:** In our experiments, LISA+ extends LISA which couldn't handle multiple instances and was enhanced to include multitarget segmentation masks in its output pipeline and phrase tokens (<p> and </p>) are added for the GCG task.  PixelLM+ builds on PixelLM, where phrase tokens are added, and <SEG> token is replaced with multiple codebook tokens. These changes ensure a fair comparison and are detailed in Section 5.2 (Baselines) of our manuscript.  \n\n**Related Work:** GeoGround, RSUniVLM, and GeoPix (which was published in January 2025) are works concurrent to ours and share similarities. Comparative analysis can enrich the understanding of GeoPixel's unique contributions. GeoGround and RSUniVLM support pixel-level grounding by converting masks into text sequences, adding a computational burden to the LLM that scales with the number of distinguishable objects. GeoPixel resolves this limitation through end-to-end training with a dedicated mask decoder. Moreover, GeoPixel not only outperforms GeoGround on RRSIS-D data **+11.11**(Acc0.5 val), **+15.83**(Acc0.5 test), **+6.89**(mIoU val), and **+6.8**(mIoU test) but also outperforms specialist models (Table 4) showcasing strong referring expression segmentation capability.  \n\n**Training and Inference Efficiency:** All models, including GeoPixel and the baseline models (LISA+, PixelLM+, and GLaMM-ft), were trained for the same epochs with similar computational resources, ensuring performance gains stem from model design, not training or hardware differences. Furthermore, to provide a comparison of inference efficiency, we additionally report average runtime per sample: LISA+: 25.28s, PixelLM+: 91.44s, GLaMM-ft:35.08s, and GeoPixel: (P=1) 46.48s for RS-GCG task.\n\n**Adaptive Partitioning and Multi-scale feature fusion MSFF:** Adaptive partitioning is not merely an alternative but a complementary strategy to MSFF. RS imagery exhibits significant variations in spatial coverage (e.g. 800x800 pixels can cover diverse areas such as 1 km² or 10 km²). In this context, image resolution is crucial to determine the granularity and clarity of details within each square meter. High resolution (HR) enhances the visibility of finer features, essential for accurate identification and detailed analysis. MSFF can also be valuable by allowing information integration from various scales. Balancing both strategies presents a promising direction for future exploration, potentially improving RS accuracy and efficiency.\n\n**Necessity and Advantages of GCG in RS:** GCG is immensely important in RS, as traditional supervised segmentation tasks, although precise, are limited to fixed categories and offer minimal interactivity. GCG adds an intuitive, interactive layer, enabling users to explore and query data more flexibly, broadening the scope beyond predefined labels. GeoPixel demonstrates specialized, high-precision segmentation capability, but by integrating GCG, it can expand its usability, empowering more dynamic and user-driven data exploration.  \n\n**Cross-Dataset Robustness:** We evaluated GeoPixel on VRSBench referring expression detection task, reporting Acc\\@0.5 / Acc\\@0.7 across 3 categories: unique (U), non-unique(NU) and overall (O)\n\n**MiniGPT-v2** scored 40.7/18.9 (U), 32.4/15.2 (NU), 35.8/16.8 (O).  \n\n**LLaVA-1.5** achieved 51.1/16.4 (U), 34.8/11.5 (NU), 41.6/13.6 (O).  \n\n**Mini-Gemini** showed lower performance with 41.1/9.6 (U), 22.3/4.9 (NU), 30.1/6.8 (O). \n \n**GeoChat** reached 57.4/22.6 (U), 44.5/18.0 (NU), 49.8/19.9 (O).  \n\n**GeoPix** performed similarly with 57.0/22.7 (U), 44.8/18.2 (NU), 49.8/20.0 (O).  \n\n**GeoPixel** achieved the best results: **70.37/41.54** (U), **65.80/40.32** (NU), **67.70/40.83** (O).  \n\nWe further compared **GeoChat** and **GeoPixel** on VRSBench using orientated bounding boxes. \n\n**GeoChat** scored 32.3/12.6 (U), 18.5/5.7 (NU), 24.3/8.6 (O).  \n\n**GeoPixel** significantly outperformed, achieving **54.48/24.87** (U), **60.51/30.97** (NU), **58.00/28.42** (O).  \n\nTo avoid data leakage, GeoPixel (trained on GeoPixelD) is finetuned on VRSBench without using any data from RRSIS-D (a DIOR-based dataset). GeoPixelD and VRSBench rely on DOTA's training set for training and validation set for testing. These steps ensure no leakage from either DOTA or DIOR. We plan to extend our evaluations using additional datasets like xView to test GeoPixel's applicability to broader RS scenarios. The DIOR base results indicate promising adaptability, which we'll detail along with qualitative results in the revised submission.  \n\nWe hope our responses adequately address your concerns. Thank you for your recommendations, which have undoubtedly helped improve our work.", "claims_and_evidence": "Claim: GeoPixel supports high-resolution RS imagery.\nEvidence: Dynamic partitioning (up to 4K) and experiments showing performance gains with higher patch counts (Table 4).\n\nClaim: Pixel-level grounding improves RS understanding.\nEvidence: Comparisons with bounding-box-based RS-LMMs (Table 1) and superior metrics in multi-target segmentation (Table 2).\n\nClaim: GeoPixelD enables fine-grained comprehension.\nEvidence: Dataset construction details (Section 4) and ablation on annotation complexity (Table 5).", "methods_and_evaluation_criteria": "Methods: Adaptive partitioning, dual vision encoders, and a pixel decoder address RS-specific challenges, constructing a pipeline to achieve image understanding and pixel-level segmentation tasks.\n\nEvaluation: Metrics (CIDEr, mIoU) and benchmarks (RS-GCG, RRSIS) align with task objectives. Baselines (LISA+, PixelLM+) are adapted fairly but require clarity regarding their modifications.", "theoretical_claims": "The paper does not make any theoretical claims that require rigorous mathematical proof. Its core contributions focus on method design and empirical verification.", "experimental_designs_or_analyses": "Yes, Baselines (LISA, PixelLM) are adapted for RS via fine-tuning on GeoPixelD, ensuring fair comparison. The RRSIS benchmark (Table 3) uses established metrics (P@0.5, mIoU), but cross-dataset generalization (e.g., on non-iSAID data) is untested.", "relation_to_broader_scientific_literature": "GeoPixel advances RS-LMMs by introducing pixel grounding, addressing a gap in prior works (e.g., RSGPT, GeoChat) limited to bounding boxes. It aligns with natural-image pixel-grounding models (LISA, PixelLM) but is applied to remote sensing scenarios.", "essential_references_not_discussed": "There are some MLLM with Segmentation Capability works [1][2][3].\n[1] RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-Oriented Mixture of Experts.\n[2] GeoGround: A Unified Large Vision-Language Model. for Remote Sensing Visual Grounding.\n[3] GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing.\nDiscussing the differences in tasks and methods can help readers better understand this article.", "other_strengths_and_weaknesses": "Strengths:\n1. First RS-LMM to achieve pixel-level grounding, addressing a critical gap in existing literature.\n2. GeoPixelD’s semi-automated annotation pipeline ensures high-quality, granular data for RS-specific tasks.\n3. The paper is well-presented and well-written.\n4. It is good that the author considered the limitations and challenges\n\nWeaknesses:\n1. It is unclear what GeoPixel has done with remote sensing images and how effective they are.\n2. There is a lack of discussion with some related work.(Please see Essential References Not Discussed *)\n3. No discussion of training/inference efficiency, especially for the additional SAM2 decoder.", "other_comments_or_suggestions": "Please see Weaknesses. The experimental results of the article are good, the workload is sufficient, and the quality of the data set is also high. My focus may be on the effectiveness and significance of the task itself.", "questions_for_authors": "1. The paper mentions adaptive partitioning, but the multi-scale feature fusion method is widely used in MLLM, such as S2-Wrapper[1]. Is there any special design for remote sensing? Or is this design more helpful for remote sensing images?\n[1] Shi B, Wu Z, Mao M, et al. When do we not need larger vision models?\n\n2. Do we need GCG tasks for remote sensing images? The current mainstream supervised segmentation tasks often target limited categories in remote sensing images. In comparison, what are the advantages of GEOPIXEL, which is also oriented to a single task?\n\n3. Experiments rely heavily on GeoPixelD/iSAID; cross-dataset robustness (e.g., DIOR, xView) is untested. Can GeoPixel be extended to more general remote sensing scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wOKXSy03M7", "reviewer": "Reviewer_zikq", "summary": "This paper introduces GeoPixel and GeoPixelD. \n\nGeoPixel is a combination of models that receives high-res RGB satellite imagery and outputs text (e.g., a description in natural language) and a dense segmentation map. The combination consists of a frozen vision encoder that \"tokenizes\" the image and feeds these tokens to a pretrained LLM. The LLM outputs text, which is fed to a pixel decoder along with visual tokens. And the pixel decoder outputs the segmentation map. GeoPixel performs well on both image captioning / describing (in natural language) and image segmentation. \n\nGeoPixelD is a dataset that consists of matching high-res RGB satellite imagery and natural language descriptions, constructed via a semi-automated pipeline.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zikq, \n \nThank you for your detailed review and insights on our submission, GeoPixel. We appreciate the opportunity to address your concerns and clarify aspects of our research. \n \n**Evaluation on Other RS Image-to-Text Datasets:** You raised an important question regarding why GeoPixel was not evaluated on additional RS image-to-text datasets, such as those used in GeoChat, EarthGPT, etc. The primary reason for this was the specific focus of GeoPixel on high-resolution satellite imagery coupled with text-guided image segmentation. While valuable, the datasets used in GeoChat EarthGPT often encompass varied tasks that do not align directly with the dense segmentation focus of GeoPixel. As the stated models do not support referring expression segmentation, we formulate referring expression detection as a post-processing task applied to the outputs of GeoPixel. Specifically, segmentation masks predicted by GeoPixel are utilized to derive horizontal bounding boxes (HBBs) and oriented bounding boxes (OBBs), enabling a consistent basis for comparative evaluation on VRSBench. \n\nGeoPixel outperforms GeoChat in referring expression detection in HBB and OBB, reporting Accuracy\\@0.5 / Accuracy\\@0.7 across unique, non-unique, and overall cases. The gain in performance is summarized as follows: HBB-based Detection Unique: **+12.97 / +18.94** Non-Unique: **+21.30 / +22.32** Overall: **+17.90 / +20.93** OBB-based Detection Unique: **+22.18 / +12.27**, Non-Unique: **+42.01 / +25.27**, Overall: **+33.70 / +19.82**.  \nBoth models were evaluated after finetuning on VRSBench; the gain in performance clearly shows the importance of pixel-level alignment in multimodal models.  \n\n**Models Finetuning Clarification:** The models LISA+ and PixelLM+ were indeed modified and finetuned on the GeoPixelD dataset. The designation \"FT\" was specifically used for GLaMM to distinguish between its original and finetuned versions, as it was not modified. We will ensure that future iterations of the manuscript clearly articulate these distinctions to avoid confusion regarding the experimental setups and model adaptation status.  \n\n**Methodological Choices and Justifications:** Typically employed for tasks like region-specific captioning, explicit encoders are not used in our approach. Instead, we utilize a Set of Marks (SOM) alongside spatial priors, as detailed in our data annotation pipeline, to target and delineate specific regions precisely. This method ensures accurate regional descriptions and functions independent of model frameworks and offers an effective alternative. Table 4 presents the ablation study on the impact of tiling high-resolution images, where P=1 denotes the use of a single image patch, effectively implying no tiling.   \n\n**Relation to Broader Scientific Literature and Technical Novelty:** We appreciate the observation regarding the conceptual similarity between GeoPixel and models like GLaMM. However, remote sensing (RS) imagery demands high spatial resolution to capture fine structural details of ground objects. GLaMM's input resolution (336×336) is inadequate for representing RS data's rich spatial context and intricate features. GeoPixel addresses this by combining high-resolution visual understanding with pixel-level grounding, making it well-suited for the complexities of RS imagery. Additionally, our semi-automated data annotation pipeline, specifically tailored for RS imagery, extracts regional information through a set of marks and spatial priors, which also constitutes a key contribution.  \n\nIn the revised manuscript, we will discuss RS-specific foundation models (e.g., SatMAE) to contextualize our work further within the broader literature.  \n\nWe hope that these clarifications address your concerns adequately. Thank you for your constructive feedback, which is invaluable in improving our work.", "claims_and_evidence": "Yes. The paper claims to outperform the SOTA on text-guided image segmentation and image description tasks. I believe the results on page 7 support this.", "methods_and_evaluation_criteria": "There are many RS benchmarks that this paper does not use to evaluate GeoPixel. I believe this is because GeoPixel is designed for two specific tasks (RS image descriptions in natural language and text-guided image segmentation) for which few datasets exist.\n\nCan the authors please clarify why they did not evaluate GeoPixel on other RS image to text datasets, like those used in GeoChat, EarthGPT, etc.?", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Overall, the experiments look sound. According to the end of section 5.2, all LMM models are finetuned on GeoPixelD. However, in table 2, GLaMM has two entires: \"GLaMM\" and \"GLaMM-FT\". LISA and PixelLM do not have the \"FT\" suffix, were these _not_ finetuned on GeoPixelD?", "relation_to_broader_scientific_literature": "GeoPixel demonstrates a method to leverage pretrained models as components in a system that can be finetuned successfully on another domain (remote sensing). This research question is relevant to many ML applications. That being said, the GeoPixel method itself is very similar to GLaMM (which was not applied to RS).", "essential_references_not_discussed": "This paper does not discuss RS-specific foundation models, such as SatMAE, etc.", "other_strengths_and_weaknesses": "As discussed, GeoPixel is quite similar to GLaMM. I see the differences between GeoPixel and GLaMM as:\n\n1. GeoPixel does not use an explicit region encoder (see top left of Figure 2 in GLaMM). Instead, GeoPixel tiles each high-res image and encodes them independently and globally (via downsampling). \n2. GeoPixel uses PLoRA, which applies LoRAs to the vision tokens inside of the LLM.\n\nTo me, #1 makes sense in RS since we may not have a strong region encoder to extract salient regions. Tiling a high-res image seems like a sensible choice. #2 also seems reasonable to me, as the LLM likely needs to be adjusted a bit. However, I do not see ablations on these choices that empirically justify them.\n\nOverall, I expect more experiments from an applied submission with limited technical novelty.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mEnTHLzC9m", "reviewer": "Reviewer_Vj3H", "summary": "This work introduces a multi-modal Grounded Conversation Generation (GCG) dataset that includes grounded descriptions for high-resolution remote sensing images, along with a benchmark featuring human-verified annotations. By leveraging recent advances in large vision-language models, the authors propose a model aimed at achieving fine-grained visual understanding in remote sensing imagery.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Vj3H,  \n\nThank you for your thorough review and insightful comments regarding our submission. We appreciate the opportunity to address the issues raised and clarify aspects of our research methodology and dataset.  \n\n**Method Novelty and Framework Comparison:** While our method framework resembles existing MLLMs like LISA and VisionLLM-v2, our approach is specifically tailored to the unique challenges of remote sensing (RS) imagery. The novelty lies in integrating high-resolution image comprehension with pixel-level grounding capabilities. This cohesion is critical for RS, as it enhances the model's capacity to decipher complex visual information at a granular level, thereby providing a more refined understanding and interaction with diverse geospatial objects. Moreover, our data annotation pipeline is uniquely designed for RS imagery, leveraging spatial priors and region-specific markers to extract and represent regional information effectively.  \n \n**Dataset Collection and Annotation Methodology:** Thank you for raising concerns about annotation accuracy with 3×3 grids. To address this, we would like to clarify the robustness of our dataset annotation methodology: In instance annotation, precise object assignment is ensured through a pixel overlap ratio, rather than just bounding box or object centers, to ensure accurate placement of objects. Furthermore, in scenarios where ambiguity arises, each individual annotation is already supplemented with a distinct set of markers. These markers aid in distinguishing between instances, particularly when locational ambiguity exists. Further reinforced through the integration of category priors, this proves to be a highly effective strategy for instance-specific annotations. In comparison, situations involving cluster annotations where markers are absent, we have adopted an enhanced localization mechanism with a multi-grid hierarchical system. This adjustment significantly mitigates ambiguities by refining the spatial grid at which annotations are applied, ensuring comprehensive and precise coverage across intersecting grids.  \n\n**Pretraining Discrepancy and Evaluation Metrics:** Regarding the pretraining discrepancy mentioned, I would like to clarify that all the models in Table 3 are trained on GeoPixelD data under the same training conditions, parameters, and epochs. Evaluation Metric: We acknowledge that CIDEr may not fully capture the nuances of detailed image captioning. To address this, we conducted additional evaluations using the CLAIR score [1], an LLM-based metric (GPT-4o in our case) that better aligns with human judgments. The results are GLaMM: 43.11, LISA+: 68.96, PixelLM+: 73.93, GLaMM-ft: 71.74, and GeoPixel **77.50**. Notably, GeoPixel outperforms all other models in this evaluation and across ROUGE-1, ROUGE-2, and ROUGE-L scores.  [1] Chan, D., Petryk, S., Gonzalez, J. E., Darrell, T., & Canny, J. (2023). CLAIR: Evaluating image captions with large language models. arXiv preprint arXiv:2310.12971. \n\n**Example of CLAIR score and reasoning:** \"score\": 0.25, \"explanation\": \"The candidate and reference sets both describe an aerial view involving docks and a body of water with a dark appearance. However, there are notable discrepancies in the details: the candidate mentions four piers with specific structures on them, while the reference describes only two piers. The description of the boats also differs, with the candidate mentioning a solitary boat moored at a dock, while the reference describes two boats in different positions.\"  \n\n**Ethical Review and Annotation Accuracy:** In addressing ethical review concerns regarding the accuracy and use of automatically generated annotations, it is important to emphasize the rigorous validation process implemented in this research. We employed a robust validation protocol where each gcg description in the test set was meticulously verified by expert annotators. These specialists, trained in recognizing and correcting discrepancies in automated text, manually corrected any data discrepancies found. Moreover, training set annotations were also rigorously filtered to eliminate aerial perspective inconsistencies, artifacts such as marker identifiers, fore/background references, depth cues, and inconsistent descriptors. This validation process underscores our commitment to ethical research practices.  \n\n**Related Work:** We thank you for pointing out several relevant works. We will update our manuscript to include discussions on RS-specific LMMs like SkyEyeGPT, VRSBench, Popeye, MMM-RS, and RS-MoE, providing a more comprehensive overview of the landscape and situating our contributions within it.  \n\nWe hope that these clarifications and planned improvements address your concerns and demonstrate our commitment to advancing the field of RS through responsible and innovative research practices.  Thank you for your constructive feedback, invaluable in refining our work.", "claims_and_evidence": "Method novelty. The method framework has no significant difference with existing MLLMs, such as LISA and VisionLLM-v2. The idea of dividing the input images into local and global regions is widely explored in large vision-language models, such as Llava-Next.", "methods_and_evaluation_criteria": "The proposed method and evaluation make sense for the target problem.", "theoretical_claims": "The work does not include any theoretical proofs.", "experimental_designs_or_analyses": "The experimental design is mostly good, except for minor unfairness in Table 3.", "relation_to_broader_scientific_literature": "This work lies in the scope of MLLMs in remote sensing. There are already many attempts in this direction, with only a few focusing on pixel-level tasks. This work mainly differs in the newly collected grounded detailed dataset.", "essential_references_not_discussed": "Some Remote Sensing (RS) LMMs are missing in the related work section, such as, SkyEyeGPT, VRSBench, Popeye, MMM-RS, and RS-MoE.", "other_strengths_and_weaknesses": "1. Method Novelty\nThe framework does not significantly differ from existing MLLMs, such as LISA and VisionLLM-v2. The strategy of partitioning input images into local and global regions is a well-explored concept in large vision-language models (e.g., Llava-Next).\n\n2. Dataset Collection\nGrid-Based Annotation: In the individual pipeline, images are divided into 3×3 grids, with annotations generated for each grid. This approach raises concerns when objects are positioned at grid boundaries or span across multiple grids, potentially leading to incomplete or inaccurate annotations.\nAnnotation Accuracy: The reliance on automatically generated annotations by vision-language models may lead to inaccuracies, even when using state-of-the-art proprietary models.", "other_comments_or_suggestions": "No further suggestions.", "questions_for_authors": "Pretraining Discrepancy: In Table 3, the proposed GeoPixel model is pretrained on the iSAID-based GeoPixelD dataset, whereas the comparison methods are trained solely on the target dataset, which creates an imbalance in the experimental setup.\nEvaluation Metric: For the detailed captioning task, the use of CIDER as the evaluation metric may not be entirely appropriate. This limitation should be discussed in the manuscript.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethical_review_concerns": "Annotation Accuracy: The reliance on automatically generated annotations by vision-language models may lead to inaccuracies, even when using state-of-the-art proprietary models.", "ethics_expertise_needed": ["Responsible Research Practice (e.g., IRB, documentation, research ethics, participant consent)"]}}, {"review_id": "4x8mk099VA", "reviewer": "Reviewer_fxwn", "summary": "The paper introduces GeoPixel, a remote sensing multimodal LLM for pixel level understanding and reasoning in high resolution aerial images. The authors present GeoPixelD, a new dataset with detailed, spatially-aware annotations for grounded conversations in remote sensing. The authors develop an adaptive image divider module for high resolution image understanding within the MLLM. Experimental results demonstrate GeoPixel's superior performance in generating grounded conversations and segmenting referred objects in remote sensing data, highlighting its advancements in understanding and interpreting visual information.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fxwn,  \n\nThank you for your comprehensive review of our submission. We appreciate your insights and the opportunity to clarify and expand upon aspects of our work.  \n\n**Performance Benchmarking:** We appreciate your acknowledgment of GeoPixel's superior performance in handling high-resolution aerial images compared to models like PixelLM and RSMIN. Regarding your suggestion to compare GeoPixel with other multimodal models in remote sensing (e.g., GeoChat, SkyEyeGPT, TeoChat), we would like to clarify that GeoPixel is specifically designed as a segmentation-centric model. In contrast, current RS multimodal language models do not support segmentation outputs .\n\nTo enable a fair comparison, we formulated referring expression detection as a post-processing task over GeoPixel's segmentation predictions. Specifically, the segmentation masks generated by GeoPixel are used to derive horizontal bounding boxes (HBBs) and oriented bounding boxes (OBBs), providing a consistent evaluation framework across models. This approach was applied on VRSBench, allowing us to benchmark GeoPixel against a range of existing models under comparable detection metrics. \n\nGeoPixel demonstrated substantial gains over existing methods in referring expression detection/localization, measured by Accuracy\\@0.5 and Accuracy\\@0.7 across unique, non-unique, and overall categories:  \n\n**MiniGPT-v2** scored 40.7/18.9 (unique), 32.4/15.2 (non-unique), and 35.8/16.8 overall.  \n\n**LLaVA-1.5** achieved 51.1/16.4 (unique), 34.8/11.5 (non-unique), and 41.6/13.6 overall.  \n\n**Mini-Gemini** showed lower performance with 41.1/9.6 (unique), 22.3/4.9 (non-unique), and 30.1/6.8 (overall).  \n\n**GeoChat** reached 57.4/22.6 (unique), 44.5/18.0 (non-unique), and 49.8/19.9 (overall).  \n\n**GeoPix** performed similarly with 57.0/22.7 (unique), 44.8/18.2 (non-unique), and 49.8/20.0 (overall). \n \n**GeoPixel** achieved the best results: **70.37/41.54** (unique), **65.80/40.32** (non-unique), and **67.70/40.83** (overall).  \n\nWe further compared **GeoChat** and **GeoPixel** on VRSBench using orientated bounding boxes for referring expression detection.  \n\n**GeoChat** scored 32.3/12.6 (unique), 18.5/5.7 (non-unique), and 24.3/8.6 (overall).  \n\n**GeoPixel** significantly outperformed, achieving **54.48/24.87** (unique), **60.51/30.97** (non-unique), and **58.00/28.42** (overall).   \n \n\n**Relation to Broader Scientific Literature:** Thank you for acknowledging GeoPixel's contribution as an early work in remote sensing MLLMs. It sets a foundation for subsequent research in this rapidly evolving domain. Also, we appreciate your reference to the concurrent work TeoChat. We will discuss it in the revised manuscript to contextualize GeoPixel's contributions within the broader scientific dialogue.  \n\n\n**Further Contributions:** Your recognition of the GeoPixelD dataset as a valuable resource for the community is encouraging. We believe that both GeoPixelD will help propel forward the capabilities of RS-MLLM models.  \n\nWe are preparing a revision that incorporates these insights and will provide additional data and qualitative results to address the gaps you've highlighted. Thank you once again for your thorough evaluation and constructive criticism.", "claims_and_evidence": "1. GeoPixel is designed to handle and reason about high resolution aerial images which is confirmed by performance comparison with models like PixelLM and RSMIN.\n2. GeoPixel is designed to provide pixel grounded outputs and its capability is demonstrated on the RS-GCG dataset.\n3. Although the impact statement mentions that GeoPixel can be used for urban planning and disaster response, the paper does not showcase any such applications. Instead, it focuses solely on demonstrating the performance of GeoPixel on benchmark datasets.", "methods_and_evaluation_criteria": "1. There is no comparison of GeoPixel shown against other remote sensing based MLLM such as GeoChat, SkyEyeGPT or TeoChat in tasks like remote sensing image captioning or object detection. Although they are mentioned in the related works and Table-1, it might be interesting to see a comparison with these models.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments shown in the paper are adequate to evaluate the pixel-level understanding and reasoning capability of GeoPixel.", "relation_to_broader_scientific_literature": "GeoPixel is one of the early works in remote sensing that developed a MLLM for grounded conversations and pixel-level reasoning. While such models exist for other domains, GeoPixel outperforms these generalist models, particularly for high-resolution aerial images. As mentioned in the impact statement, GeoPixel can be utilized in remote sensing applications like urban planning and disaster response (although not shown in the paper).", "essential_references_not_discussed": "TeoChat: Irvin, J. A., Liu, E. R., Chen, J. C., Dormoy, I., Kim, J., Khanna, S., ... & Ermon, S. (2024). Teochat: A large vision-language assistant for temporal earth observation data. ICLR.", "other_strengths_and_weaknesses": "1. The paper introduces GeoPixelID, a dataset for grounded conversation in remote sensing. The paper also introduces a benchmark dataset to evaluate RS-LLM models on the task of referring segmentation.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: After the discussion phase, the majority of reviewers recommended acceptance (Accept, 2x Weak Accept, Weak Reject), finding the paper to be well written, the introduced dataset valuable, and the proposed method to have novelty in its formulation. The remaining negative rating was due to concerns about the technical novelty (i.e., similarity to an existing method) and the lack of ablation studies to better highlight the contributions. The rebuttal addressed many of the reviewers' concerns, including those related to dataset annotation details.  Ultimately, the AC found the review by R-Amru to be the most compelling. This paper falls under the applications track, tackles an interesting problem, proposes a method with several novel elements, introduces a new dataset, and has compelling results. As such, an accept decision was reached. The AC does agree, however, that the ablations could be more robust. Please take the reviewer feedback into account when preparing the camera-ready version.
Decision: Accept (poster)