GeoPixel: Pixel Grounding Large Multimodal Model in Remote Sensing
Authors: name: Akashah Shabbir, name: Mohammed Zumri, name: Mohammed Bennamoun, name: Fahad S. Khan, name: Salman Khan

Recent advances in large multimodal models (LMMs) have recognized
fine-grained grounding as an imperative factor of visual understanding and
dialogue. However, the benefits of such representation in LMMs are limited to
the natural image domain, and these models perform poorly for remote sensing
(RS). The distinct overhead viewpoint, scale variation, and presence of small
objects in high-resolution RS imagery present a unique challenge in
region-level comprehension. Moreover, the development of the grounding
conversation capability of LMMs within RS is hindered by the lack of granular,
RS domain-specific grounded data. Addressing these limitations, we propose
GeoPixel - the first end-to-end high resolution RS-LMM that supports
pixel-level grounding. This capability allows fine-grained visual perception by
generating interleaved masks in conversation. GeoPixel supports up to 4K HD
resolution in any aspect ratio, ideal for high-precision RS image analysis. To
support the grounded conversation generation (GCG) in RS imagery, we curate a
visually grounded dataset GeoPixelD through a semi-automated pipeline that
utilizes set-of-marks prompting and spatial priors tailored for RS data to
methodically control the data generation process. GeoPixel demonstrates
superior performance in pixel-level comprehension, surpassing existing LMMs in
both single-target and multi-target segmentation tasks. Our methodological
ablation studies validate the effectiveness of each component in the overall
architecture. Our code and data will be publicly released.