AeroPilot-v1

Open-vocabulary person detection for drone search and rescue.

AeroPilot-v1 is a fine-tune of Meta's SAM 3.1. It finds 9 in 10 people in real wilderness search-and-rescue imagery, works on night thermal and open water, and still answers any text prompt.

91.7%	94.4%	10 / 10
recall on the HERIDAL SAR benchmark (base SAM 3.1: 79.5%)	recall on night thermal people (base SAM 3.1: 17.5%)	held-out test sets improved over base SAM 3.1

Why

From a drone at search altitude, a lost hiker is a 20–60 pixel smudge in a 4K frame of rock, scrub and shadow. General-purpose detectors are trained on photos taken at eye level, so they miss many of these people. Off the shelf, SAM 3.1 finds 65% of the people in HERIDAL.

AeroPilot-v1 adapts SAM 3.1's vision side to the overhead viewpoint and keeps its text encoder frozen. Prompts such as person, boat, life jacket or cattle work as before, and they now work from the air.

Results

Full results table (recall / precision, %)

Benchmark	Domain	Test images	SAM 3.1	AeroPilot-v1	AeroPilot-v1, 2×2 tiles
HERIDAL test	wilderness SAR, 4K	101	65.0 / 74.7	87.5 / 72.7	91.7 / 75.9
HERIDAL unseen sites	sites absent from training	22	69.9 / 67.0	87.1 / 71.7	91.4 / 73.9
LADD	forest, field, snow SAR	202	70.6 / 91.1	92.4 / 90.6	96.8 / 86.0
SARD	wilderness SAR, low-res	570	79.1 / 90.2	91.4 / 88.8	–
WiSARD unseen flight	forest SAR, flight absent from training	112	63.3 / 51.9	75.1 / 62.5	82.0 / 59.7
HIT-UAV	night thermal	288	17.5 / 75.1	94.4 / 76.6
SeaDronesSee	open-water rescue	600	70.1 / 52.3	90.6 / 87.1
TinyPerson	tiny people, beach and sea	600	57.2 / 74.9	68.2 / 69.3
WAID	aerial wildlife and livestock	600	66.8 / 79.1	95.2 / 88.9
COCO val2017	everyday photos, 80 classes	600	78.0 / 71.3	80.5 / 65.0

Precision is measured at a fixed confidence of 0.5. AeroPilot-v1 is more confident overall, so at that threshold it trades a little precision for much higher recall. At equal recall it is more precise. When AeroPilot-v1 is set to match base SAM 3.1's HERIDAL recall, its precision is 87.0% against 74.7%. TinyPerson's labels omit many visible people, which understates its precision.

Quickstart

pip install git+https://github.com/facebookresearch/sam3.git huggingface_hub

import torch
from PIL import Image
from huggingface_hub import hf_hub_download
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

ckpt = hf_hub_download("AeroPilot/AeroPilot-v1", "aeropilot_v1.pt")
model = build_sam3_image_model(checkpoint_path=ckpt, load_from_HF=False)
processor = Sam3Processor(model, confidence_threshold=0.5)

image = Image.open("frame.jpg").convert("RGB")
with torch.autocast("cuda", dtype=torch.bfloat16):
    out = processor.set_text_prompt("person", processor.set_image(image))

for box, score in zip(out["boxes"].tolist(), out["scores"].tolist()):
    print(f"person {score:.2f} at {[round(v) for v in box]}")  # x0, y0, x1, y1 in pixels

aeropilot_v1.pt is a drop-in replacement for sam3.1_multiplex.pt. Every SAM 3 image API works unchanged, and no gated download is needed.

4K frames: for frames wider than about 2,700 px, run a 2×2 grid of tiles with 15% overlap, then merge the boxes with NMS. On HERIDAL this raises recall from 87.5% to 91.7%. Don't tile low-resolution frames; it adds false alarms without adding recall.

Tiled inference helper

from torchvision.ops import nms

def detect_tiled(processor, image, prompt, n=2, overlap=0.15):
    W, H = image.size
    tw, th = int(W / (n - (n - 1) * overlap)), int(H / (n - (n - 1) * overlap))
    boxes, scores = [], []
    for y in torch.linspace(0, H - th, n).int().tolist():
        for x in torch.linspace(0, W - tw, n).int().tolist():
            with torch.autocast("cuda", dtype=torch.bfloat16):
                out = processor.set_text_prompt(prompt, processor.set_image(image.crop((x, y, x + tw, y + th))))
            boxes.append(out["boxes"].float() + torch.tensor([x, y, x, y], device=out["boxes"].device))
            scores.append(out["scores"].float())
    boxes, scores = torch.cat(boxes), torch.cat(scores)
    keep = nms(boxes, scores, 0.5)
    return boxes[keep], scores[keep]

Training

The whole SAM 3.1 detector was fine-tuned (vision backbone, fusion encoder, decoder and heads) with box-only losses. The text encoder stayed frozen, which keeps the prompt space and with it open-vocabulary detection. Two kinds of negatives teach the model to return nothing when the target is absent: prompts for categories missing from an image, and frames with no people.

Dataset	Domain	Images per epoch	Prompts
HERIDAL	Mediterranean wilderness SAR	6,843	person
LADD	forest, field and snow SAR	6,088	person
SARD	wilderness SAR	5,183	person
WiSARD	US forest and mountain SAR	6,000	person
SeaDronesSee	open-water rescue	6,000	person, boat, jet ski, life jacket, buoy
HIT-UAV	night thermal	4,016	person
TinyPerson	tiny people, beach and sea	3,201	person
WAID	aerial wildlife and livestock	5,000	sheep, cattle, seal, camel, donkey, zebra
COCO train2017	everyday photos (replay against forgetting)	1,500	80 classes

4K frames were used both whole and as 2×2 tiles.

Recipe. 2 epochs (12,082 steps) at an effective batch of 8, on 2× H100 in about 6 hours. AdamW, peak learning rate 4e-5 (1.25e-5 for the vision backbone, with 0.9 layer decay), inverse-square-root schedule with warmup and cooldown, bf16 at 1008 px.

Data hygiene.

• 157 HERIDAL training images that duplicate or nearly duplicate test images were removed.
• Frames labeled "no person" were screened with base SAM 3.1, and 158 frames containing unlabeled people were dropped.
• Whole HERIDAL sites and a WiSARD flight were held out to measure generalization to unseen terrain.

Evaluation protocol

• Base SAM 3.1 and AeroPilot-v1 are scored by the same harness; only the weights differ.
• A detection is correct at IoU ≥ 0.5, with one-to-one greedy matching, at confidence ≥ 0.5.
• COCO crowd regions are ignored.
• Tiled runs use a 2×2 grid with 15% overlap and NMS at IoU 0.5.
• Every benchmark is a test split that was never used for training.

License and attribution

AeroPilot-v1 is a derivative of SAM 3.1 and is released under the SAM License. Built with SAM 3.1 by Meta (github.com/facebookresearch/sam3).

Each training dataset remains under its own license; review those terms before commercial use. Example images: HERIDAL (CC BY 3.0) and HIT-UAV.

@misc{aeropilot2026,
  title  = {AeroPilot-v1: Open-Vocabulary Person Detection for Drone Search and Rescue},
  author = {AeroPilot},
  year   = {2026},
  url    = {https://huggingface.co/AeroPilot/AeroPilot-v1}
}