SamHung/astrovision-lora

AstroVision LoRA — astronomy image captioning

LoRA adapter for Llama-3.2-11B-Vision-Instruct, trained on 250 astronomy image/caption pairs on a single free-tier T4.

Full model card, training code, evaluation and limitations: Samuelsunshine/astrovision-lora-debugging

The repository is two things: this pipeline, and a written audit of five defects in the published pipeline it reproduces — one of which made that project's public demo serve base weights instead of its fine-tune, silently. Read the audit.

Licence

Llama 3.2 Community License. This adapter is a derivative of Llama-3.2-11B-Vision-Instruct and inherits that licence — not Apache 2.0, which is what the auto-generated card originally said here. See the licence.

The training and evaluation code in the linked repository is MIT. The dataset's source imagery has unestablished provenance; check before downstream use.

Results

Tesla T4, 2026-08-21, 25 held-out images, greedy decoding. Raw log committed in the repository.

Metric	Value
BLEU	0.0722
ROUGE-1 / ROUGE-2 / ROUGE-L	0.3735 / 0.1405 / 0.3361
length_ratio	0.9561
Best validation loss	0.7232 (at the final step)

Validation loss fell monotonically with its minimum at step 30, so the model is undertrained at this budget rather than overfit. 30 steps was chosen to match the compute of the pipeline being reproduced.

BLEU is a weak fit for captioning and these numbers are not a quality claim. An image has many correct descriptions, and two correct captions with disjoint vocabulary both score near zero. They are reported because they are reproducible and comparable across this project's own runs.

Limitations — please read before trusting a caption

This is not an astronomy tool. It was trained on 250 images across five loosely-defined subjects (Earth from orbit, Mars, Mars rovers, the Milky Way, Hubble) for 30 optimizer steps. It reproduces the writing style of that corpus. It has no astronomical knowledge beyond what the base model already had.

It is confidently wrong, demonstrated. From the held-out set, unedited:

Reference: A close-up photograph of a Mars rover undergoing calibration tests inside a NASA facility on Earth. Prediction: A photograph of the Mars rover exploring the Martian terrain under the reddish-orange sky.

A laboratory on Earth described as the surface of Mars, fluently, with no hedging. Two of five logged samples make errors of this kind.

Do not use it for identification of objects, missions, instruments or locations; measurement or any scientific claim; accessibility descriptions, where a confident wrong caption is worse than none; or education without review.

Out of domain it degrades quietly. Given a photograph that is not astronomy imagery it will still produce an astronomy-flavoured caption, because that is the only register it was trained in. Fluency here is unrelated to correctness.

Usage

from peft import PeftModel
from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Llama-3.2-11B-Vision-Instruct-bnb-4bit",
    load_in_4bit=True,
    use_gradient_checkpointing="unsloth",
)
model = PeftModel.from_pretrained(model, "SamHung/astrovision-lora")
FastVisionModel.for_inference(model)

Use PeftModel.from_pretrained to load a trained adapter. FastVisionModel.get_peft_model(model, lora_adapter=...) reads as though it would do the same and does not — it constructs a new adapter and ignores the keyword, and because LoRA initialises B to zero the result is mathematically identical to the base model, with no error raised. That failure is Finding 4 in the audit, and scripts/verify_adapter_loading.py exists to catch it: it checks the artifacts are present, that ‖lora_B‖ > 0, and that enabling the adapter actually changes the output.

Inference should go through generate_caption() from the repository, which slices the prompt off in token space and always states do_sample explicitly — decoding the full generate() output echoes the instruction into every prediction and roughly doubles measured length.

Training

LoRA r=16, alpha=16, dropout=0, bias="none", on vision and language layers, attention and MLP. 30 steps at an effective batch of 8, lr=2e-4, adamw_8bit, fp16, 4-bit NF4 base. 200 train / 25 validation / 25 test, seed=42, split before any training dataset was built and asserted disjoint at runtime. 67,174,400 trainable parameters. 20.1 minutes on a T4, peak 9.227 GB of 14.563 GB.

Provenance

Coursework reproduction. The dataset is AIOmarRehan/space-multimodal-dataset and the pipeline reproduced is AIOmarRehan/Unsloth_Llama_3.2_11B_Vision_Instruct_Astronomy (MIT, © Omar Rehan). Three data/evaluation defects in that pipeline were found and fixed here, and two further findings documented; the audited code and every execution log cited as evidence are the original author's.

Trained with Unsloth and TRL.