Adithya-Rajendran/cosmos3-edge-kitti-lora

Cosmos 3 Edge KITTI LoRA adapters (2D detection by text)

Two LoRA adapters that teach nvidia/Cosmos3-Edge to list every car, pedestrian, cyclist, truck and tram in a KITTI driving frame as boxes in text. They come from the CMPE 249 (SJSU, Fall 2026) report One Flag and Two Gaps: What Zero-Shot COCO-to-Driving Detection Results Hide; code, metric files and the paper are at github.com/Adithya-Rajendran/HW1-2D-Detection (tag v1.0).

Subfolder	Output format	Use it for
json/	the JSON the zero-shot model already emits	best accuracy
compact/	one label x1 y1 x2 y2 line per object, nearest first	half the tokens, about 2x faster

Coordinates are normalized to 0 to 1000 in both formats.

Results (KITTI 2D object val, Chen et al. split)

AP is COCO AP@[.5:.95] over five classes with the KITTI devkit's ignore rules (DontCare, Van and Person_sitting as neighbor classes). The VLM returns no scores, so each box is scored by the geometric-mean probability of its tokens; those scores are not calibrated like a detector's, so F1 (IoU 0.5, every box) is given alongside.

Model	AP, 300-frame subset	F1, subset	AP, full val (3,769)	F1, full val	Tokens / answer	s / frame, Orin Nano 8 GB	s / frame, GB10
Cosmos 3 Edge, zero-shot	14.4	0.66	n/a	n/a	271	19.1	5.47
+ json/ adapter	32.4	0.85	30.1	0.83	180	11.6	3.67
+ compact/ adapter	31.5	0.82	28.6	0.82	91	6.3	1.92
YOLO26m fine-tuned on the same frames (reference)	27.6	0.84	26.5	0.82	n/a	0.018 (TensorRT)	n/a

Orin Nano latency is the mean over the first 50 subset frames, BF16, Hugging Face Transformers, in Docker on JetPack 6. KITTI moderate AP|R40 on full val, json/ adapter vs. fine-tuned YOLO26m: car 81.5 vs. 90.6, pedestrian 62.4 vs. 63.3, truck 47.7 vs. 5.6, cyclist 49.4 vs. 34.1, tram 21.4 vs. 18.7. The adapters are much stronger on the rare classes and weaker on cars. They have not been evaluated outside KITTI.

Usage

Transformers must include Cosmos 3 support (the adapters were trained and evaluated with commit 60f49549e0db78f022ee30aa1571d9ab6e4a15bf) and peft 0.21.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = "nvidia/Cosmos3-Edge"
processor = AutoProcessor.from_pretrained(base)
model = AutoModelForImageTextToText.from_pretrained(base, dtype=torch.bfloat16, device_map={"": 0})
model = PeftModel.from_pretrained(model, "Adithya-Rajendran/cosmos3-edge-kitti-lora", subfolder="compact")
model = model.merge_and_unload().eval()

prompt = ("Locate every instance that belongs to the following categories: car, pedestrian, cyclist, truck, tram. "
          "Answer one object per line as: label x1 y1 x2 y2, with coordinates from 0 to 1000, nearest objects first. "
          "Answer none if there are none.")
# json/ adapter prompt: "Locate every instance that belongs to the following categories: car, pedestrian,
# cyclist, truck, tram. Report bbox coordinates in JSON format."
messages = [{"role": "user", "content": [{"type": "image", "path": "000123.png"}, {"type": "text", "text": prompt}]}]
inputs = processor.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_dict=True,
                                       return_tensors="pt", enable_thinking=False).to(model.device, torch.bfloat16)
with torch.inference_mode():
    out = model.generate(**inputs, do_sample=False, max_new_tokens=1024)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Multiply coordinates by width/1000 and height/1000 for pixels. The repository's det_eval.vlm_grounding has the parsers for both formats and the token-probability scoring.

Training

• Data: 3,412 KITTI training frames (the 3,712-frame train half minus 300 held out for checkpoint selection); targets are the KITTI boxes for car, pedestrian, cyclist, truck and tram. Van and Person_sitting were background.
• LoRA rank 16, alpha 32, on the language model's q_proj, k_proj, v_proj, o_proj, fc1, fc2 (vision encoder frozen); AdamW, lr 1e-4 with warmup and cosine decay, gradient accumulation 8, 4 epochs (1,708 optimizer steps); best held-out loss at step 1,200 (JSON 0.298, compact 0.593), which is the checkpoint here.
• Hardware: ASUS Ascent GX10 (NVIDIA GB10), about 2.9 s per step, about 18 GB.
• Commands: python -m det_eval.vlm_finetune train --epochs 4 --grad-accum 8 --val-every 200 --n-val 100 (add --format compact --order size for the compact adapter).

Limitations

• One training run per adapter, one dataset (KITTI, one camera in one city). Off-domain behavior is untested; a full fine-tune of YOLO26m on the same frames lost most of its accuracy on nuImages.
• Token-probability scores are not calibrated; compare by F1 at a fixed IoU when mixing with detectors.
• At 6 to 12 s per frame on an Orin Nano this is a labeler or teacher, not a real-time perception module.
• Capping max_new_tokens below about 128 for the compact adapter cuts off far objects (recall halves at 64).

License and attribution

• The adapters are derived from NVIDIA Cosmos 3 Edge, released under the OpenMDW License Agreement 1.1, a copy of which is in LICENSE. Built on NVIDIA Cosmos.
• They were trained on KITTI (Geiger, Lenz and Urtasun, CVPR 2012), which is licensed CC BY-NC-SA 3.0. Use these adapters for non-commercial research only.

Citation

@misc{rajendran2026oneflag,
  author = {Adithya Rajendran},
  title  = {One Flag and Two Gaps: What Zero-Shot COCO-to-Driving Detection Results Hide},
  year   = {2026},
  note   = {CMPE 249 report, San Jos\'e State University},
  url    = {https://github.com/Adithya-Rajendran/HW1-2D-Detection}
}