Brusnicki/SAVANT-scene-description-lora

SAVANT Scene Description Model (LoRA Adapter)

This repository contains the LoRA adapter for the scene description model (Phase 1) described in the SAVANT paper, currently under peer review.

This repository is provided for peer-review purposes only. After the review process, the model will be made publicly available through the authors' main account.

Model Description

LoRA adapter for Qwen/Qwen2.5-VL-7B-Instruct, fine-tuned to generate structured scene descriptions from driving scene images. This is Phase 1 of the SAVANT two-phase pipeline.

Given a front-camera image, the model produces a structured JSON description across four semantic layers:

• Street layer: geometry, topology, surface condition, lane markings
• Infrastructure layer: traffic lights, signs, cones, barriers, construction sites
• Movable objects layer: vehicles, pedestrians, other dynamic objects
• Environmental layer: weather, visibility, lighting conditions

Training Details

• Base model: Qwen/Qwen2.5-VL-7B-Instruct
• Method: LoRA (Low-Rank Adaptation)
• Dataset: 4,260 samples with structured scene descriptions
• Epochs: 3
• Learning rate: 1e-4 (cosine schedule)
• Precision: bfloat16 with Flash Attention 2

LoRA Configuration

Parameter	Value
Rank (r)	16
Alpha	32
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, fc1, fc2, qkv, mlp.0, mlp.2

Usage

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
import torch

base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "u94fmn391j/SAVANT-scene-description-lora")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

Limitations

• Trained on the CODA dataset; generalization to other driving domains not evaluated
• Single-frame analysis only (no temporal context)