Title: REMSA: An LLM Agent for Foundation Model Selection in Remote Sensing
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: LLM Agent, Remote Sensing, Foundation Models, Benchmarking, Strcutured Database

Abstract: Foundation Models(FMs) are increasingly integrated into remote sensing(RS) pipelines for tasks such as environmental monitoring, disaster assessment, and land use mapping. These models encompass both unimodal vision encoders and multimodal vision-language architectures, each pretrained on domain-specific data modalities, including synthetic aperture radar(SAR), multispectral, and hyperspectral imagery, across varying spatial, spectral, and temporal resolutions. FMs are adapted to support a range of downstream tasks, including semantic segmentation, image classification, change detection, and visual question answering, depending on their pretraining objectives and architectural design. However, selecting the most suitable remote sensing foundation model(RSFM) for a downstream task remains challenging due to scattered documentation, heterogeneous formats, and complex deployment constraints. To address this, we first introduce the RS Foundation Model Database(FMD), the first structured and schema-guided resource covering over 150 RSFMs trained using various data modalities, associated with different spatial, spectral, and temporal resolutions, considering different learning paradigms. Built on top of FMD, we further present REMSA(Remote-sensing Model Selection Agent), the first LLM agent for automated RSFM selection from natural language queries. REMSA combines structured FM metadata retrieval with a task-driven agentic workflow. In detail, it interprets user input, clarifies missing constraints, ranks models via in-context learning, and provides transparent justifications. Our system supports various RS tasks and data modalities, enabling personalized, reproducible, and efficient FM selection. To evaluate REMSA, we introduce a benchmark of 75 expert-verified RS query scenarios, resulting in 900 task-system–model configurations under a novel expert-centered evaluation protocol. REMSA outperforms multiple baselines, including naive agent, dense retrieval, and unstructured retrieval augmented generation based LLMs, showing its utility in real decision-making applications. REMSA operates entirely on publicly available metadata of open source RSFMs, without accessing private or sensitive data. Our code and data will be publicly released.