csicosmos3nanoquality — Cosmos3-Nano LoRA for vial-filling line inspection

• UW Milwaukee Connected Systems Institute Factory line quality detection fine tuning model
• https://uwm.edu/csi/
• Advanced optical Inspection - Custom model generation
• Predictive Maintence on visual damages.

LoRA adapter for nvidia/Cosmos3-Nano (reasoner, Qwen3-VL-8B layout) fine-tuned with NVIDIA TAO's cosmos-rl backend to inspect a vial-filling line from a side-view camera. Given one image and the manufacturing plan as text, the model returns a JSON report with each vial's content, fill level, status, the deviating positions, and an overall pass/fail.

Contents

path	what
adapter/	PEFT-style LoRA adapter (r=16, α=32, q/k/v/o projections of all 36 language layers), ~45 MB
base/ (optional)	nvidia/Cosmos3-Nano reasoner + vision tower re-keyed as a plain Qwen3-VL checkpoint (bf16, ~17 GB). Required to run the adapter; see below if absent
eval/	metrics.json + predictions.jsonl for the fine-tuned model and the zero-shot base on 300 held-out questions
report.html	training + evaluation report

Results (300 held-out synthetic questions, 60 per type)

Question type	Base model	Fine-tuned
Content of one vial	68.3 %	100 %
Vial count	100 %	100 %
Does vial N match the plan	76.7 %	100 %
List all deviating positions	10.0 %	88.3 %
Full JSON inspection report	0.0 %	95.0 %
Overall	51.0 %	96.7 %

Report-field accuracy (fine-tuned): valid JSON 100 %, per-vial content 99.7 %, per-vial status 99.2 %, pass/fail verdict 100 %. Remaining misses are under-detected over-fill, mostly on clear liquid.

Training

• Data: 800 synthetic side-view images × 5 questions (content, count, match, deviation list, JSON report), generated by dataset/generate_tube_dataset.py in the cosmos3tao repo
• 5 epochs, 625 steps, batch 8 × 4 GPUs, AdamW lr 2e-5, bf16, FSDP on 1 node × 4 GB200
• Train loss 0.290 → 0.033; validation loss 0.0971 → 0.0353 (best, epoch 5); 24 minutes

Usage

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel
from PIL import Image

REPO = "Balab2021/csicosmos3nanoquality"
base = f"{REPO}"            # if base/ is included: subfolder="base"; otherwise convert nvidia/Cosmos3-Nano with
                            # cosmos3tao/cluster/convert_omni_to_qwen3vl.py and point to that directory
processor = AutoProcessor.from_pretrained(base, subfolder="base")
model = Qwen3VLForConditionalGeneration.from_pretrained(base, subfolder="base", dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(model, REPO, subfolder="adapter").merge_and_unload()   # or merge manually, see repo

CONTEXT = ("You are a quality inspector watching a vial filling line from the side. Each wheeled carrier on the "
           "track holds one clear vial with a printed label. Vials are numbered by position from left to right. "
           "A correctly filled liquid vial is filled to roughly one third of its height.")
plan = "position 1 (VIAL 0019): orange liquid; position 2 (VIAL 0020): blue liquid"
question = ("Produce the inspection report as compact JSON with keys: vials (list of {position, vial_id, planned, "
            "actual, fill_pct, status}), deviating_positions, pass. status is one of OK, wrong_color, underfill, "
            "overfill, empty, wrong_content. Output JSON only.")
img = Image.open("plant.jpg").convert("RGB")
msgs = [{"role": "user", "content": [{"type": "image", "image": img}, {"type": "text", "text": f"{CONTEXT}\nManufacturing plan: {plan}\n{question}"}]}]
text = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[img], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=768, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Vocabulary: contents orange, blue, yellow, red, green, purple, clear, cubes, empty; statuses OK, wrong_color, underfill, overfill, empty, wrong_content. Keep the prompt structure above; the adapter was trained on it.

Limitations

Trained on synthetic renders of one camera geometry. On a real plant photo it identified 7 of 8 vials correctly (missed a partly occluded cube stack). Fill-level judgement of clear liquid is the weakest skill. Mix labeled real frames into training before relying on it in production.

License

Derivative of nvidia/Cosmos3-Nano, distributed under the NVIDIA Open Model License. The adapter and any redistributed base weights inherit that license.