VLAlert — Observe Before You Alert: Belief-State Driver Alerting with Vision–Language Models (CoRL 2026)

This repository holds the model weights and code behind the CoRL 2026 paper Observe Before You Alert: Belief-State Driver Alerting with Vision–Language Models (Yuhang Wang, Lingyao Li, Hao Zhou). The benchmark, annotations, and all result files live in the companion dataset repository: https://huggingface.co/datasets/HenryYHW/VLAlert

arXiv version of the paper: https://arxiv.org/abs/2609.08130

VLAlert casts dashcam driver alerting as a tri-action policy over SILENT / OBSERVE / ALERT. A LoRA-adapted Qwen3-VL-4B generates a structured safety-evidence span <|BELIEF|> ... </|BELIEF|>; the hidden states inside that span are pooled into a compact belief vector that drives a DangerHead (risk) and a PolicyHeadV2 (three-way action).

Contents

Path	What it is
sft_x_v3_lora/	Stage-1 LoRA adapter for Qwen/Qwen3-VL-4B-Instruct (PEFT, r=32, α=64, targets q/k/v/o/gate/up/down), tokenizer files with the added `<
heads/armB/danger_c1last_seed{1..5}/best.pt	DangerHead checkpoints (5 seeds) trained on the source-balanced 11,640-tick subset; these heads produce the Table 2 / test-split numbers in the paper.
heads/armB/policy/best.pt	PolicyHeadV2 (three-way SILENT/OBSERVE/ALERT) trained on the same subset.
heads/armB/danger_full_seed1/best.pt	Full-size DangerHead variant used together with the policy head for the qualitative demo videos.
code/lkalert/	Model components under models/ (belief_vlm.py span pooling, danger_head.py, policy_head_v2.py, adaptive_window.py) and benchmark label rules under labels/.
code/training/SFT/	Stage-1 belief-format supervised fine-tuning (LoRA).
code/training/Policy/, code/training/danger/	Belief caching, head training, policy evaluation and post-processing.
code/tools/	Evaluation and paper scripts, e.g. build_paper_final_v3.py (Table 2 protocol, DAUS), compute_daus_v1_val.py, eval_badas_v1_val.py, score_v1_val_gemini.py, score_v1_val_baselines.py, demo_score_v3_new.py, render_demo_videos.py.
code/colab/train_heads_colab.py	Head training script used for the camera-ready runs (5 seeds, validation-only selection).
code/requirements.txt, code/setup.py, code/configs/	Environment and configs.

Scripts reference local paths of the original experiments (e.g. /home/henry/Desktop/LKAlert/...); point them at your own copies of the benchmark files from the dataset repository.

Pipeline

1. Belief generation. For each 1 Hz tick the VLM receives the last 8 frames and emits <|BELIEF|> evidence text </|BELIEF|> followed by an action token.
2. Belief pooling. Hidden states between the two delimiters are mean-pooled per frame and the last four transformer layers are concatenated, giving a 10,240-d belief vector per frame; the hidden state at the closing delimiter is kept as a decision register.
3. Heads. DangerHead scores visual risk from the 8 frame beliefs; PolicyHeadV2 outputs the SILENT/OBSERVE/ALERT distribution. P(ALERT) is thresholded at 0.3394, selected on the validation split by maximizing balanced accuracy and then frozen for the test split.

Results (VLAlert-Bench, full denominators)

	AUROC	AP_tick	Recall_v	F1_t	BalAcc	DAUS
Validation (11,220 ticks)	0.680	0.229	0.914	0.321	0.638	0.5001
Test (23,661 ticks, threshold frozen on validation)	0.679	0.254	0.952	0.349	0.634	—
BADAS-Open re-scored on identical frames, test	0.519	0.158	0.935	0.266	0.523	—

Paired bootstrap over test videos: ΔAUROC = +0.159 [+0.146, +0.172]. On 221 ADAS-TO-Critic takeover clips VLAlert reaches R@5s 88.7% vs. 74.2% for BADAS-Open. Per-tick score files that reproduce every number are in the dataset repository under results/.

Quickstart (sketch)

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel
import torch

base = "Qwen/Qwen3-VL-4B-Instruct"
model = AutoModelForImageTextToText.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained("HenryYHW/VLAlert", subfolder="sft_x_v3_lora")  # tokenizer with BELIEF tokens
model.resize_token_embeddings(len(processor.tokenizer))
model = PeftModel.from_pretrained(model, "HenryYHW/VLAlert", subfolder="sft_x_v3_lora")

# 1) generate the BELIEF span for 8 frames (see code/training/Policy/make_belief_cache_v2.py),
# 2) pool hidden states between <|BELIEF|> and </|BELIEF|> over layers 20/24/28/32 (code/lkalert/models/belief_vlm.py),
# 3) run heads/armB/danger_c1last_seed1/best.pt and heads/armB/policy/best.pt (code/lkalert/models/danger_head.py, policy_head_v2.py),
# 4) alert when P(ALERT) >= 0.3394.

code/tools/demo_score_v3_new.py runs the complete streaming pipeline on an mp4 and writes per-tick action probabilities; code/tools/render_demo_videos.py renders the side-by-side demo videos.

License

Adapter weights, head checkpoints and code: Apache-2.0. The base model Qwen3-VL-4B-Instruct is released by Alibaba under Apache-2.0. Benchmark data and annotations are in the dataset repository under CC-BY-4.0; third-party source videos are not redistributed.

Citation

CoRL 2026 camera-ready:

@inproceedings{wang2026vlalert,
  title     = {Observe Before You Alert: Belief-State Driver Alerting with Vision--Language Models},
  author    = {Wang, Yuhang and Li, Lingyao and Zhou, Hao},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026}
}

arXiv preprint (submitted under the earlier title):

@misc{wang2026observe_arxiv,
  title         = {Observe Before You Alert: Adaptive Driver Alerting with Vision-Language Models},
  author        = {Wang, Yuhang and Li, Lingyao and Zhou, Hao},
  year          = {2026},
  eprint        = {2609.08130},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.08130}
}