BonsaiRobotics/bonsai-vineyard-sample-lora

Bonsai Vineyard Sample LoRA (Cosmos3-Nano)

A rank-32 LoRA on nvidia/Cosmos3-Nano that turns a seed image, a segmentation video, a depth video, or segmentation + depth together into a 720p, 15 fps video of a farm robot driving through vineyard rows. It is the checkpoint produced by the Cosmos3-Nano multi-control transfer LoRA recipe in NVIDIA/cosmos-framework (docs/transfer_multicontrol_lora_nano.md), trained for 150 iterations on the 50 training clips of BonsaiRobotics/bonsai-vineyard-sample.

This repository holds the exported checkpoint (base weights plus the LoRA tensors), in the format cosmos_framework.scripts.inference --checkpoint-path loads. One checkpoint serves all four modes.

Use

From a cosmos-framework checkout (see the recipe for the pinned commit and setup):

uvx hf@latest download BonsaiRobotics/bonsai-vineyard-sample-lora --local-dir examples/checkpoints/bonsai-vineyard-sample-lora
python -m cosmos_framework.scripts.inference --parallelism-preset=latency --seed 2026 \
    -i <spec>.json -o outputs/bonsai-vineyard-sample-lora \
    --checkpoint-path examples/checkpoints/bonsai-vineyard-sample-lora

The recipe's examples/transfer_multicontrol_lora_nano/make_specs.py writes the specs used for its results (seed image -> video: 149 frames, guidance 6; control modes: 61 frames, guidance 3, control guidance 0.9, 35 steps, shift 10). The Wan2.2 VAE is fetched automatically on first use.

Results

Held-out day of the dataset, 6 clips, one seed, against the base checkpoint with identical inputs (mean over clips):

Mode	PSNR (dB), base -> LoRA	LPIPS, base -> LoRA
Seg + depth -> video	11.82 -> 15.65	0.463 -> 0.323
Seg -> video	12.13 -> 15.30	0.466 -> 0.332
Depth -> video	12.31 -> 15.32	0.476 -> 0.334
Seed image -> video	19.27 -> 19.39	0.333 -> 0.318

The gain is mostly appearance: the LoRA renders the site's bright, dry daylight where the base model renders a generic low-sun vineyard. Full tables, the before/after comparison and a check on orchard and strawberry clips are in the recipe.

Limitations

• Trained on a small sample dataset: 50 ten-second clips from one camera rig in one vineyard, daylight only. The results show what a small dataset can do, not the best this recipe can reach.
• Objects the segmentation marks but the training clips rarely show (buildings at the row end, people) are not drawn.
• Where the segmentation leaves part of the frame unassigned, such as a distant horizon, the LoRA fills it with the training site's background (dry hills, a fence).
• In the control modes, leaves near the camera change more from frame to frame than in real video.

Licence

The fine-tuned checkpoint is released under the Creative Commons Attribution-NonCommercial 4.0 International licence (CC BY-NC 4.0), the same terms as the dataset it was trained on: non-commercial use, with credit to Bonsai Robotics.

It is derived from nvidia/Cosmos3-Nano, which NVIDIA releases under OpenMDW-1.1; the base model's terms and notices continue to apply to the parts of the weights that come from it.

Citation

@misc{bonsai_vineyard_sample_lora_2026,
  title        = {Bonsai Vineyard Sample LoRA (Cosmos3-Nano)},
  author       = {Zachariah, Sandeep Sam and {Bonsai Robotics}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/BonsaiRobotics/bonsai-vineyard-sample-lora}}
}