NexusDwin/sailswarm-lraspp-student

SailSwarm LRASPP student (real-time water/sky/obstacle segmentation for Raspberry Pi 4)

lraspp_mobilenet_v3_large (torchvision, ~3.2 M params) distilled for 3-class maritime semantic segmentation — obstacle / water / sky — for the SailSwarm autonomous-sailboat obstacle-detection module (University of Konstanz). Trained on LaRS ground truth plus 10,731 Lake-Constance fisheye frames labelled by the eWaSR teacher (NexusDwin/sailswarm-ewasr-lars).

The point of this model: the teacher costs 3.9 µs/pixel int8 on a Pi 4 — quantisation-saturated. The student costs **1.1 µs/pixel**, making on-device segmentation real-time.

Files

File	What
student512t_best.pth / train_512.log	512×384-trained weights (val acc 0.9944, IoU obst/water/sky 0.947/0.995/0.991)
student864t_best.pth / train_864.log	864×648-trained weights (val acc 0.9955, IoU 0.957/0.996/0.993) — crisper thin structures at full resolution
student_512x384.onnx · student_640x480.onnx	fp32 exports of the 512-trained model (opset 17, fixed size, single image input NCHW)
student864t_864x648.onnx	fp32 export of the 864-trained model
*.int8.onnx	static per-channel QDQ int8 (calibrated on Lake-Constance frames; 99.1–99.3% pixel agreement vs fp32), ~3 MB each

Measured performance (Raspberry Pi 4, onnxruntime CPU, 2 threads, unthrottled)

input	int8 latency	fps	async worker (with free-space derivation)
512×384	217.6 ms	4.60	3.00 Hz — one mask per 3 fps camera frame
640×480	327.9 ms	3.05	—
864×648 (864-trained)	594 ms	1.68	1.30 Hz

Scales to 5.75 fps @512 with 4 threads. Quality vs the teacher on-domain: 99.50% mean pixel agreement over 10,731 frames (p5 99.01%, min 93.7%).

Usage (onnxruntime)

import onnxruntime as ort, numpy as np, cv2
sess = ort.InferenceSession("student_512x384.int8.onnx", providers=["CPUExecutionProvider"])
img = cv2.imread("frame.jpg")[:, :, ::-1]                       # RGB, undistorted
x = cv2.resize(img, (512, 384)).astype(np.float32) / 255.0
x = ((x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)[None].astype(np.float32)
cls = sess.run(None, {"image": x})[0][0].argmax(0)              # {0 obstacle, 1 water, 2 sky}

Intended use & limits

• Input: undistorted RGB frames (Kannala–Brandt-rectified fisheye in our deployment). Masks upsample to full frame with nearest-neighbour.
• Daytime/dawn RGB only. On near-black night frames the model produces confidently wrong output (measured: 86% "obstacle" on a black frame) — our runtime refuses frames below a mean-luminance floor of 25/255 and defers to thermal + radar. Replicate that gate.
• Trained for a lake-surface camera ~0.27 m above the waterline; expect a domain gap for high-mounted or open-sea cameras.
• Not validated for closed-loop navigation.

Provenance / license

Backbone: torchvision lraspp_mobilenet_v3_large (BSD-3, ImageNet-pretrained). Training data: LaRS (research/non-commercial — derived weights inherit its terms) + teacher pseudo-labels from NexusDwin/sailswarm-ewasr-lars (itself LaRS-derived). Verify all licences before any commercial or redistribution use.

Citation

@InProceedings{Zust2023LaRS,
  title={LaRS: A Diverse Panoptic Maritime Obstacle Detection Dataset and Benchmark},
  author={{\v{Z}}ust, Lojze and Per{\v{s}}, Janez and Kristan, Matej},
  booktitle={International Conference on Computer Vision (ICCV)},
  year={2023}
}