NaOHjiang/GeoLens

Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing

Fengxiang Wang^1,*, Jiangnan Huang^2,*, Mingshuo Chen¹, Yueying Li¹,
Yang Shi¹, Junwei Luo², Haoyu Wang^3,†, Yansheng Li², Jing Zhang²,
Haiyan Zhao^3,†, Wenjing Yang<sup>1

1</sup> National University of Defense Technology   ² Wuhan University   ³ Tsinghua University
^* Equal contribution   ^† Corresponding authors

📃 Paper (Coming Soon) | 🤗 GeoLens Model | 🤗 GeoMTVR Dataset | 🏆 XLRS-Bench

---

📚 Contents

• 📚 Contents
• 🔥News
• 📜Abstract
• 🔍Method
  • Supervised Fine-Tuning (SFT)
  • Tool-Attention-Focused Reinforcement Learning (RTAL)
• 🚀Reproduction
  • Environment
  • SFT
  • RTAL
  • XLRS-Bench
  • RSHR
  • LRS-GRO-eval
• 📊Results
• 🔗Citation
• 🤝Acknowledgement

🔥News

• [Coming Soon] Paper will be released on arXiv.
• [2026.06] Model weights and code released.
• [2026.06] GeoMTVR dataset released.

📜Abstract

Ultra-high-resolution (UHR) remote-sensing (RS) imagery provides fine-grained Earth-observation evidence over city-scale scenes, but poses a fundamental challenge for multimodal large language models (MLLMs): task-relevant evidence is often sparse, local, and spatially dispersed across extremely large visual contexts. A natural solution is to equip MLLMs with zoom-in tools for active local inspection. However, through a pilot study on XLRS-Bench, we find that zoom-in is only partially effective: it resolves easy and medium-level tasks with locally recoverable evidence, but saturates on hard cases requiring global search, multi-region comparison, path planning, or dispersed-evidence reasoning.

Motivated by this finding, we move beyond single-tool zoom-in and introduce GeoMTVR, a large-scale Geospatial Multi-Tool Visual Reasoning dataset built from wide-area satellite imagery. GeoMTVR contains 13K UHR VQA samples with interleaved reasoning trajectories, diverse visual tool calls, and returned visual observations, enabling models to learn question decomposition, tool selection, regional inspection, object-level grounding, auxiliary visual reasoning, and cross-tool evidence integration.

Beyond supervised fine-tuning, we propose a tool-attention-focused reinforcement learning algorithm (RTAL) that concentrates optimization on critical tool-use decisions, including when to invoke tools, which tool to select, where to apply it, and how to interpret tool outputs.

By combining SFT on GeoMTVR with our RL algorithm, we develop GeoLens, a multi-tool visual reasoning MLLM for UHR RS. Experiments show that GeoLens consistently outperforms direct reasoning and single-tool zoom-in baselines, achieving stronger accuracy, better evidence grounding, and more efficient tool-use trajectories.

Model Weights

Model	Description	Link
GeoLens	Final multi-tool visual reasoning model (SFT + RTAL)	🤗 NaOHjiang/GeoLens
GeoMTVR	SFT training dataset (13K UHR VQA samples with tool trajectories)	🤗 NaOHjiang/GeoMTVR

Repository Structure

GeoLens/
  sft/                         # SFT configs and data-building scripts
  rtal/RTAL_dl_final/          # RTAL RL training code (dual JSD loss)
  evaluation/
    EVAL_by_myself/            # XLRS-Bench eval & score recomputation
    RSHR_eval/                 # RSHR benchmark evaluation
    LRSGRO_eval/               # LRS-GRO-eval evaluation
  scripts/                     # Reproduction wrappers (SFT, RTAL, eval)
  docs/                        # Source manifest and notes

🔍Method

Supervised Fine-Tuning (SFT)

We fine-tune Qwen2.5-VL-7B-Instruct on the GeoMTVR dataset, which contains 13K UHR VQA samples with interleaved tool-use trajectories. The SFT stage teaches the model to follow tool-use instructions, perform regional inspections, and integrate multi-tool evidence.

Tool-Attention-Focused Reinforcement Learning (RTAL)

We propose RTAL (Reinforced Tool-Attention Learning), a novel RL algorithm that applies a dual JSD (Jensen-Shannon Divergence) loss with key-mask and valid-mask to concentrate optimization on critical tool-use decisions:

• When to invoke tools: The model learns to decide whether current visual evidence is sufficient or additional tool calls are needed.
• Which tool to select: The model learns to choose the appropriate tool (zoom-in, region inspection, object grounding, etc.) based on the task requirement.
• Where to apply it: The model learns to localize the correct spatial region for tool application.
• How to interpret tool outputs: The model learns to integrate tool-returned visual observations into its reasoning chain.

🚀Reproduction

Environment

git clone https://github.com/NaOH3021/GeoLens.git
cd GeoLens
python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -e rtal/RTAL_dl_final/verl

SFT

The SFT stage fine-tunes Qwen2.5-VL-7B-Instruct on the GeoMTVR dataset to obtain the SFT checkpoint.

Step 1: Download Dependencies

Dependency	Source	Notes
Qwen2.5-VL-7B-Instruct	🤗 Qwen/Qwen2.5-VL-7B-Instruct	Base model
GeoMTVR dataset	(To be released)	Multi-tool visual reasoning dataset
LLaMA-Factory	pip install llamafactory	Training framework used by the SFT script

Place the base model at a local path, e.g., /path/to/models/Qwen2.5-VL-7B-Instruct.

Step 2: Prepare Dataset

The GeoMTVR dataset should be organized in LLaMA-Factory format under a single dataset directory:

/path/to/geomtvr-data/
  super_accepted_or_route_3img_summary_cold_start/
    train.json
    images/
  lrs_gro_zoom_strict_superstyle_cold_start/
    train.json
    images/
  route_visual_accepted_388_superstyle_cold_start/
    train.json
    images/
  dataset_info.json

Refer to LLaMA-Factory dataset preparation for the exact format.

Step 3: Run SFT

BASE_MODEL=/path/to/models/Qwen2.5-VL-7B-Instruct \
DATASET_DIR=/path/to/geomtvr-data \
OUTPUT_DIR=/path/to/output/my_model_SFT_13078 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
bash scripts/run_sft.sh

The script automatically injects your paths into the config template at sft/cold_start_qwen25vl_8gpu/config_cold_start_super_plus_lrs_8gpu_20260408.yaml. If using a different number of GPUs or DeepSpeed config, modify the following in the template:

• model_name_or_path → your BASE_MODEL path
• dataset_dir → your DATASET_DIR path
• output_dir → your OUTPUT_DIR path
• deepspeed → path to your DeepSpeed config (ZeRO-3 recommended for full-parameter tuning)

Note: We do not release the SFT checkpoint. The SFT step must be run locally before proceeding to RTAL. Training on 8×A100-80GB takes approximately 2 days.

RTAL

RTAL (Reinforced Tool-Attention Learning) starts from the SFT checkpoint and uses key-mask + valid-mask dual JSD loss with rollout n=8. The training data consists of parquet files with the following schema: prompt (with tool-call format), response, and reward scores.

Step 1: Prepare Data

Convert the GeoMTVR dataset into the parquet format expected by the verl framework:

/path/to/rtal_data/
  train.parquet          # Training set with tool-interleaved prompts and responses
  val.parquet            # Validation set

Each parquet row must contain at least prompt and response columns. Refer to the data bundle structure in rtal/RTAL_dl_final/repro_supervqa_rl748/data_bundles/ for the exact schema.

Step 2: Run RTAL

BASE_MODEL=/path/to/output/my_model_SFT_13078 \
TRAIN_PARQUET=/path/to/rtal_data/train.parquet \
VAL_PARQUET=/path/to/rtal_data/val.parquet \
OUTPUT_DIR=/path/to/rtal_outputs \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
PYTHON_BIN=/path/to/python \
bash scripts/run_rtal.sh

Key hyper-parameters (config: rtal/RTAL_dl_final/repro_supervqa_rl748/config_supervqa_rl748_sft13078_rtal_dualloss_washed_super8066_lrsgro1913.yaml):

• attention_jsd_loss_coef_key_mask=0.01
• attention_jsd_loss_coef_valid_mask=0.01
• actor_rollout_ref.rollout.n=8
• max_prompt_length=8192
• max_response_length=1024

Note: If you prefer to skip SFT and directly run RTAL, you can use our released GeoLens checkpoint as a starting point: set BASE_MODEL to the downloaded GeoLens model path.

Evaluation

XLRS-Bench

DATASET_GLOB='/path/to/XLRS-Bench-lite_part*.jsonl' \
PYTHON_BIN=/path/to/python \
bash scripts/run_eval.sh /path/to/hf_model

Default generation uses:

max_new_tokens=8, temperature=0, do_sample=false, choice_constrained=true, choice_allow_space=true

Score recomputation:

bash scripts/recompute_xlrs_scores.sh \
  --samples_jsonl /path/to/*_samples_xlrs-lite-local-custom.jsonl \
  --results_json /path/to/*_results.json \
  --model_name MODEL_NAME \
  --summary_only

RSHR

Evaluation code is in evaluation/RSHR_eval/. Download the benchmark images first:

cd evaluation/RSHR_eval
python download_xhrbench_images.py --save_dir /path/to/rshr_images

Run evaluation on a HuggingFace model:

python eval_qwen25vl_xhrbench.py \
  --model_path /path/to/hf_model \
  --save_dir /path/to/rshr_images \
  --output_dir results/

This evaluates MCQ, Open-Ended VQA, and Image Captioning tasks. See evaluation/RSHR_eval/README.md for details.

LRS-GRO-eval

Evaluation code is in evaluation/LRSGRO_eval/. First download the ZoomEarth-3B monitor:

cd evaluation/LRSGRO_eval
bash download_zoomearth3b_with_monitor.sh

Generate inference results:

bash infer.sh /path/to/hf_model /path/to/output/inference.jsonl

Then compute accuracy:

bash eval.sh /path/to/output/inference.jsonl

See evaluation/LRSGRO_eval/run_scripts/ for batch evaluation scripts on multiple models.

📊Results

GeoLens achieves state-of-the-art performance on three benchmarks:

Benchmark	Metric	GeoLens
XLRS-Bench	Macro-13 Avg	54.2%
GRO-eval	Accuracy	60.7%
RSHR	Accuracy	42.54%

Results demonstrate that GeoLens consistently outperforms direct reasoning and single-tool zoom-in baselines, with stronger accuracy, better evidence grounding, and more efficient tool-use trajectories.

🔗Citation

If you find our work helpful, please consider citing:

@article{wang2025geolens,
  title={Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing},
  author={Fengxiang Wang and Jiangnan Huang and Mingshuo Chen and Yueying Li and Yang Shi and Junwei Luo and Haoyu Wang and Yansheng Li and Jing Zhang and Haiyan Zhao and Wenjing Yang},
  journal={arXiv preprint},
  year={2026}
}

🤝Acknowledgement

• Qwen2.5-VL: The base model for GeoLens.
• XLRS-Bench: Evaluation benchmark for ultra-high-resolution remote sensing.
• verl: The RL training framework used for RTAL.
• GeoLLaVA-8K: Prior work on UHR RS MLLMs.