Painting Vision — robot-ready wall-paint coverage segmentation

A frontier research preview for autonomous painting and surface-finishing robots: from pixels to an executable paint plan.

Painting Vision reads a single camera frame of a wall and returns far more than a mask. It segments the paintable surface, the fixtures and openings that must not be painted, and the substrate — then hands a downstream planner a wall boundary, corners, a two-tool stroke plan, machine-readable waypoints, and a live paint-progress grid, independently cross-checked against depth/LiDAR.

It is built for the hard part of construction robotics: knowing exactly where the wall is, where the keep-outs are, and when the coat is complete.

Status: research preview. The segmentation architecture, dataset tooling, geometry planner and depth validator are implemented and unit-tested; the model weights are trained by the open train.py in this repo on session-separated, real capture data.

Why this model exists

General segmentation models tell you a wall is present. Painting robots need to know:

which pixels are paintable wall versus trim, skirting, openings and fixtures;
the exact boundary and corners of the wall, so a roller or spray head does not cross an edge;
a stroke path that respects a physical clearance margin and lifts over obstacles;
the substrate (e.g. drywall) so paint behaviour is not confused with surface type;
remaining coverage across repeated observations, so the robot knows when to stop.

Painting Vision is designed end-to-end around those questions.

Capabilities

1. Ten-class semantic segmentation + substrate head

A configurable backbone (--arch: MobileNetV3-Large, ResNet-50/101 DeepLabV3, or a modern SegFormer) drives a ten-class semantic head and an independent drywall-material head, so paint state and substrate are never conflated:

other · wall_unpainted · wall_painted · wall_uncertain · skirting · switch/outlet · AC unit · door · window · wall_obstacle

Painted coverage is reported as wall_painted / (wall_painted + wall_unpainted) with lower/upper bounds that include uncertain wall area — a confidence-aware estimate, not a single false-precise number.

2. Window detection you can trust around

Windows are the most safety-critical keep-out in exterior painting. The window_postprocess stage merges mullion-split fragments, fits an oriented minimum-area rectangle with rotating calipers (so a window stays a window at any viewing angle), and filters specks and non-window shapes by fill ratio and aspect ratio — producing complete, regularised keep-out geometry.

3. Robot paint planner

robot_planner.py converts the masks into an executable plan:

ordered boundary polygon and corners (Moore-neighbour contour tracing + Douglas-Peucker — it follows real rooflines and voids, not a convex hull);
optional wall-plane rectification with edge lengths in millimetres;
a two-tool plan: roller passes for open areas plus trim/brush passes for edges and around fixtures the roller cannot reach;
machine-ready waypoints with paint on/off flags and travel distance;
paint-progress tracking (--state) that targets only what remains and tells the robot when the wall is done.

A documented real-facade run lifted planned coverage from 68% (roller only) to 97.5% (roller + trim) on the same wall — the residual is the physically unreachable remainder, reported explicitly.

4. Depth / LiDAR validation on top of vision

depth_validation.py independently checks the vision output in metric 3D: RANSAC wall-plane fitting, protrusion/recess detection (missed pipes or windows), recess-versus-window agreement, and a metric millimetres-per-pixel scale derived from depth instead of guessed — so the planner works in real units.

5. Balanced-data tooling

data_balance.py and audit_dataset.py score coverage and balance across indoor/exterior, substrate, lighting, weather, paint stage and classes, flag wall/session leakage and thin classes, and recommend exactly how many more samples each gap needs. train.py supports class-, wall- and hybrid-balanced sampling so the balanced collection survives into the optimizer.

Quickstart

python3 -m pip install -r requirements.txt

# 1. Segment a frame
python3 predict.py frame.jpg --checkpoint artifacts/best.pt \
    --paintable-mask paintable_wall_mask.png \
    --keepout-mask fixture_keepout_mask.png \
    --json result.json --tta

# 2. Plan the paint path (optional wall-plane rectification + mm scale)
python3 robot_planner.py --paintable paintable_wall_mask.png \
    --keepout fixture_keepout_mask.png \
    --corners "80,60 1180,55 1190,700 70,710" \
    --stroke-width 230 --trim-width 60 --overlap 0.2 --margin 40 --mm-per-unit 3.0 \
    --state wall_014.npz --plan plan.json --overlay plan.png

# 3. Validate against depth / LiDAR if available
python3 depth_validate.py --depth depth.npy --intrinsics "fx,fy,cx,cy" \
    --mask painting_mask.png --report validation.json

Repository layout

File	Role
`models.py`	architecture factory: DeepLabV3 (MobileNet/ResNet) + SegFormer, semantic + drywall heads
`train.py`	training, balanced sampling, window loss boost, boundary/coverage metrics
`predict.py`	inference: masks, coverage, geometry, regularised windows
`robot_planner.py`	boundary/corners, two-tool stroke plan, waypoints, progress state
`window_postprocess.py`	fragment merge + oriented rectangle fitting for windows
`depth_validation.py`	LiDAR/depth plane fit, protrusion/recess checks, metric scale
`data_balance.py`	coverage/balance analytics
`audit_dataset.py`	dataset audit: schema, leakage, class and domain balance
`fetch_real_data.py`	downloads and converts a real labeled dataset for testing
`smoke_test.py`	end-to-end audit → train → predict → plan
`benchmark.py`	head-to-head vs a generic baseline on the shared wall/window/other subset
`provenance.py`	content-addressed dataset identity, integrity + licence ledger
`safety_gate.py`	fail-closed approve/reject deployment gate
`flywheel.py`	ingest corrected captures with content + perceptual dedupe
`active_learning.py`	rank which frames to label next (novelty + domain gaps + uncertainty)
`run_registry.py`	append-only, hash-linked ledger of training/eval runs
`test_*.py`	geometry, window, depth and architecture regression tests
`kaggle/`	Kaggle training: notebook, dataset/kernel metadata, guarded push helper

Evaluation & status

This preview ships the full pipeline and its tests; model accuracy numbers are deliberately not claimed here and are produced by train.py against a session-separated, real paint dataset. Reported training metrics include per-class IoU, drywall precision/recall, window precision/recall, wall boundary F1, bottom-edge error, painted-coverage error and uncertainty fraction.

Intended use and limitations

Intended: assistive perception and path planning for painting/finishing robots; dataset curation and annotation QA for construction vision.

Not intended: unattended safety-critical control without calibrated geometry, clearance validation and a human-approved deployment; judging finish quality (streaking, thin coat, runs) — that needs dedicated defect labels and reference captures after the coat cures.

RGB alone cannot always reveal what is under a smooth surface; the model scores drywall only where visible evidence supports it and uses unknown otherwise. Always validate on held-out physical walls with the production camera.

Citation

@misc{constructelligence_painting_vision,
  title  = {Painting Vision: robot-ready wall-paint coverage segmentation},
  author = {Constructelligence},
  year   = {2026},
  note   = {Research preview}
}

Keywords

wall painting robot · autonomous painting · paint coverage estimation · construction AI · building facade segmentation · drywall detection · skirting detection · window detection · semantic segmentation · LiDAR validation · depth sensing · robot path planning · paint progress tracking · construction robotics · BIM · surface finishing automation

Downloads last month: -; Downloads are not tracked for this model. How to track