NexusDwin/sailswarm-lraspp-thermal

SailSwarm LRASPP thermal student (water/sky/obstacle segmentation on a FLIR Lepton 3, 160×120)

lraspp_mobilenet_v3_large (torchvision, ~3.2 M params) trained to segment obstacle / water / sky in long-wave infrared frames from a FLIR Lepton 3.0 (PureThermal, 160×120, ~57° HFOV) mounted on the SailSwarm autonomous-sailboat obstacle-detection box (University of Konstanz, Lake Constance). Companion to the RGB model NexusDwin/sailswarm-lraspp-student; same architecture, same 3-class convention {0 obstacle, 1 water, 2 sky}.

The point of this model: thermal is the night-time primary sensor for the boat, and no public maritime LWIR segmentation ground truth exists at this resolution. The labels are cross-modal: the RGB student's masks on the co-mounted fisheye camera, warped into the thermal frame (bearing-for-bearing under the measured linear thermal bearing model + a horizon-fit vertical offset; the two lenses share a rigid mount with a ~6 cm baseline, so parallax is <1° beyond 2.5 m).

What is in this repo

Folder	Generation	Data	Pinned holdout (under-pier clip 2026-07-08 16-44-56, n=752)
v1_day_2026-08-22/	JOINT recipe, day-only thermal	33,467 gray-Konstanz fisheye frames (six missions) + gray LaRS + ~10k warped real thermal frames, 60 epochs	acc 0.9090 / 0.9069 / 0.9080 (seeds 0/1/2), obstacle IoU 0.89, water IoU 0.46, sky 0.74
v2_night_2026-08-28/	same JOINT recipe, first dusk/night thermal (2026-08-26 outing: pontoon → open water → harbour dusk → night)	68,350 gray-Konstanz native frames (whole 2026 corpus) + gray LaRS + 114,501 total training pairs incl. ~50k warped thermal frames from 63 clips, 60 epochs	acc 0.9136 / 0.9139 / 0.9131 (seeds 0/1/2 → 0.9135 ± 0.0003), obstacle IoU 0.90, water IoU 0.54–0.55, sky 0.69

Per seed: thermal_joint_s<k>_best.pth (torchvision state dict) and thermal_joint_s<k>_160x120.onnx (fp32, opset 17, fixed 160×120, input image NCHW, output logits 1×3×120×160). train_all_seeds.log is the full training log. Seed variance ≤ ±0.001 in both generations — pick any; seed 0 is the one we evaluate against. Use v2_night_2026-08-28/ unless you specifically need the day-only model: it is +0.005 acc on the pinned holdout and is the only generation that has seen thermal frames captured after civil dusk. Water IoU +0.10 / sky −0.06 vs v1 on that holdout — the night frames shift where the model draws the water boundary under the pier; treat the sky drop as a known property of this eval scene (almost no sky visible), not a regression measurement. warp_fit_report.json is the per-clip fisheye→thermal warp fit used to make v2's labels.

Why "JOINT" — the lesson this model encodes

Trained on thermal alone (~1k frames of one dock), this architecture memorised scene layout (holdout 0.38–0.58 across seeds). Grayscale RGB from the same lake, trained jointly in the same batches as the thermal frames, is a stronger structural prior than more thermal data: 10k gray frames of one dock → 0.61 ± 0.02; 33k gray frames across six missions → 0.908 ± 0.001. Sequential fine-tuning from a gray-pretrained init does not reach this; joint batches do. Augmentations: random polarity inversion (p 0.5 — the Lepton's AGC flips warm/cold contrast between day and night) and ±12 px vertical shift (breaks horizon-row memorisation).

Preprocessing (must match)

Input is the upright, colourised 8-bit Lepton stream collapsed to luminance and replicated to 3 channels, then ImageNet-normalised — exactly what the --gray training flag did:

import onnxruntime as ort, numpy as np, cv2
sess = ort.InferenceSession("v1_day_2026-08-22/thermal_joint_s0_160x120.onnx",
                            providers=["CPUExecutionProvider"])
frame = cv2.imread("thermal.png")                       # 160x120 BGR from the PureThermal stream
g = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255.0
x = np.stack([g, g, g], 0)                              # gray x3, CHW
x = (x - np.array([0.485, 0.456, 0.406])[:, None, None]) / np.array([0.229, 0.224, 0.225])[:, None, None]
cls = sess.run(None, {"image": x[None].astype(np.float32)})[0][0].argmax(0)   # 120x160 class map

Our capture rotates the thermal frame 180° at the box (the module is mounted inverted); feed the model the upright frame. Repair dead sensor rows before inference if your Lepton has them (ours has row 21 stuck hot).

Limits — read before using

• v1 is day-only; v2 has one dusk/night outing (2026-08-26, ~4 h of Lepton footage, pseudo-labelled from the RGB model while the RGB model could still see). Frames after full dark are labelled from an RGB teacher operating near its own luminance floor — those labels are the weakest in the set. The pinned holdout is a daytime clip, so the numbers above do not measure night performance; audited night thermal ground truth does not exist yet.
• Water IoU 0.46 on the holdout is a property of the holdout scene (an under-pier clip with almost no visible water), not a measurement of open-water quality; there is no independent thermal ground truth yet — every label is a warped RGB pseudo-label, so systematic RGB-model errors (e.g. reflections read as obstacle) are inherited.
• fp32 only. Static int8 quantisation fails our pixel-agreement gate on this model (per-frame agreement min 40% on seed 0, worse on seeds 1–2) — recipe-independent fragility of weak thermal logits under quantisation. At 160×120 the fp32 ONNX is fast enough on a Raspberry Pi 4 CPU.
• Not validated for closed-loop navigation.

Provenance / license

Backbone: torchvision lraspp_mobilenet_v3_large (BSD-3, ImageNet-pretrained). Training data: LaRS (Žust et al., ICCV 2023; research/non-commercial — derived weights inherit its terms) as grayscale, plus Lake-Constance fisheye and Lepton footage captured by the SailSwarm project (University of Konstanz, Cluster of Excellence in Collective Behaviour) with pseudo-labels from NexusDwin/sailswarm-lraspp-student. Trained with scripts/gpu_seg/train_student.py (--gray --aug-polarity 0.5 --aug-vshift 12 --size 160x120) from the SailSwarm obstacle-detection repository.