Hugging Face's logo

Ravenh97
/
roborender_image3f

Robotics
sim2real
image-generation
wan
lora
Model card Files
xet
 Community

RoboRender Image3F (rgb030-depth030 step-5000)

Per-frame 3-view robot-scene renderer: one request = prompt + 3x(depth, robot mask, previous RGB) -> three synchronized 416x240 views. Used to relabel observations inside a DAgger loop at ~192 ms/frame.

What is here: the fine-tuned LoRA adapter plus the deployment source (runtime, export and engine-build scripts, vendored pipeline, tokenizer, model card).

What is not, and why:

  • Foundation weights — pull PAI/Wan2.1-Fun-V1.1-1.3B-Control separately and point ROBORENDER_MODEL_BASE at it.
  • TensorRT engines / ONNX — per GPU arch and TRT version. Build locally: bin/create_baremetal_env.sh -> bin/doctor.py --stage pre-export -> bin/export_onnx.sh -> bin/build_engine.sh -> bin/doctor.py --stage runtime -> bin/calibrate_teacache.sh -> bin/smoke_test.sh.

Contract: batch 1, 5 denoising steps, CFG 1.0, views ext1/ext2/wrist. Conditioning is fixed-lag-256; in practice hold one teacher-rendered anchor per episode (FT-FIXED). Chaining on the model's own outputs collapses around frame 20. More denoising steps make it worse here — 50 steps drifts off the anchor's colours (17.9 dB at 5 steps vs 14.5 at 50).

Anchors: Ravenh97/roborender_anchors · Teacher that renders them: Ravenh97/roborender_teacher_lora

Gotchas: the tree is hash-verified by doctor (never leave edits in it), PYTHONDONTWRITEBYTECODE=1 is required, and assets/generated/ must be empty at package stage.

RoboRender Image3F accelerated 4090 source bundle

This is the portable, build-on-target release for the rgb030-depth030 RoboRender Image3F LoRA at fine-tuning update 5000. It contains the adapter, tokenizer, a fixed-lag-256 validation fixture, Raven source, and the scripts needed to export and compile the accelerated five-step path on the deployment GPU.

This archive is intentionally thin. It contains none of the following:

  • Raven/Wan foundation-model weights;
  • ONNX graphs or ONNX external-data files;
  • TensorRT .engine or .plan files;
  • TensorRT timing caches;
  • serialized or captured CUDA Graph state; or
  • target-local generated build and inference outputs.

Those exclusions are part of the portability contract, not missing files. Point the bundle at an authorized Raven foundation-model directory, then export ONNX and build TensorRT on the exact RTX 4090 and software stack that will run the model.

Start with HANDOFF.md for an operator-ready sequence.

Release status

Item Status
Adapter Included and pinned by size and SHA-256
Foundation weights Required externally through ROBORENDER_MODEL_BASE
Adapter-specific ONNX Generated locally; not distributed
TensorRT engine and timing cache Generated locally; not distributed
CUDA Graph Disabled by default; no capture is distributed
Five-step + TeaCache source path Adapter-specific B200 validation passed at threshold 0.3
RTX 4090 correctness, memory, and latency certification Not yet completed
200 ms / 5 Hz claim on RTX 4090 No claim until target certification passes

The default TeaCache threshold is 0.3. Adapter-specific B200 validation measured 186.34 ms mean full-RGB latency at CFG 1 with PSNR 38.77 dB and SSIM 0.9904 versus the five-step PyTorch path. Those results select a conservative algorithm setting; they are not performance claims for an RTX 4090.

Model and conditioning contract

  • Architecture: RoboRender Image3F / Wan-Fun-Control 1.3B.
  • Adapter: adapter/rgb030-depth030-step-5000.safetensors.
  • Adapter SHA-256: c68de259ce5ee8e4df5a45296be8328242b3ebba351990df1d86f4e3a7a2d588.
  • Initialization: parent LoRA update 228270, followed by 5000 fine-tuning updates.
  • View order: ext1, ext2, wrist.
  • Per-view shape: RGB 416x240; batch size 1; one generated frame per view.
  • Conditions: language, three depth views, three masks, and three lagged RGB reference views.
  • Reference policy: for target frame t < 256, use RGB frame 0; for t >= 256, use RGB frame t-256.
  • Training dropout: RGB and depth use independent dropout triggers at probability 0.3. When a modality triggers, exactly one of its three views is selected for dropping. Masks are never dropped. RGB dropout is disabled for the frame-0 bootstrap prefix.
  • Sampling: fixed five-step sparse schedule, CFG 1.0.
  • Acceleration: real-valued RoPE, SDPA/flash attention, static and exact-prompt caches, TeaCache, and TensorRT for the 30-block DiT stack.

The five-step sampler is a coarse inference schedule applied to this LoRA. It is not a separately trained KD, self-forcing, or FastGen student model.

What Raven must provide

Set ROBORENDER_MODEL_BASE to a directory containing:

PAI/Wan2.1-Fun-V1.1-1.3B-Control/
├── diffusion_pytorch_model.safetensors
├── Wan2.1_VAE.pth
├── models_t5_umt5-xxl-enc-bf16.pth
└── models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth

Exact byte sizes and SHA-256 values are locked in config/foundation_models.lock.json. The package does not download those files and must not silently substitute another model revision.

Bare-metal quick start

Run from the extracted package root:

export ROBORENDER_MODEL_BASE=/absolute/path/to/raven-model-root
export CUDA_VISIBLE_DEVICES=0

python3 bin/doctor.py --stage package
./bin/create_baremetal_env.sh
export PYTHON="$PWD/.venv/bin/python"

"$PYTHON" bin/doctor.py \
  --stage pre-export --model-root "$ROBORENDER_MODEL_BASE"
PYTHON="$PYTHON" ./bin/export_onnx.sh
PYTHON="$PYTHON" ./bin/build_engine.sh
"$PYTHON" bin/doctor.py \
  --stage runtime --model-root "$ROBORENDER_MODEL_BASE"
PYTHON="$PYTHON" ./bin/calibrate_teacache.sh
PYTHON="$PYTHON" ./bin/smoke_test.sh

build_engine.sh calls the exporter when a matching export is absent. The explicit export command is shown so handoff operators can distinguish export failures from TensorRT build failures.

Generated files are written only under assets/generated/:

assets/generated/
├── onnx/<adapter-and-base-lineage-key>/
├── engines/<target-and-software-key>/
├── engines/current -> <target-and-software-key>
└── timing/

Do not add this generated directory to the release ZIP.

Run the fixed-lag fixture

After the target engine passes doctor:

"$PYTHON" runtime/image3f_deploy.py \
  --model-root "$ROBORENDER_MODEL_BASE" \
  --memory-mode auto \
  --vae-mode untiled \
  run \
  --request assets/fixtures/lag256/request.json \
  --output-type rgb \
  --warmup 1 \
  --iterations 5 \
  --output-dir outputs/lag256-smoke

Use --output-type latent to omit VAE decoding. Report latent-only and full-RGB latency separately. Runtime timing does not include model startup, sensor capture, filesystem/IPC transport, robot policy execution, or actuation.

DAgger integration

Keep one Image3FSession alive. Model loading, TensorRT deserialization, and prompt embedding are startup costs and should not occur per frame. The caller must retain at least 257 synchronized RGB triplets:

reference(t) = RGB(0)       when t < 256
reference(t) = RGB(t - 256) when t >= 256

If the deployment evaluates autoregressive drift, RGB(t-256) must be the model-generated RGB stored at that earlier step. If it evaluates a teacher-conditioned path, use the captured RGB from that earlier step and label the result accordingly. Never substitute t-1; doing so changes the training contract and can encourage copying.

Warm the exact production prompt before signaling the robot loop ready. Apply backpressure rather than queuing stale observations. Independently measure the complete observation-to-action loop before claiming real-time operation.

Package layout

Path Purpose
adapter/ Fine-tuned LoRA and immutable training provenance
assets/tokenizer/ Pinned tokenizer files
assets/fixtures/lag256/ Exact t / t-256 three-view build and smoke fixture
assets/generated/ Target-local output; absent or empty in the ZIP
config/deployment.json Machine-readable release and runtime contract
config/foundation_models.lock.json External Raven weight identities
runtime/ Persistent inference, lineage, and TensorRT runtime code
bin/export_onnx.sh Adapter-specific ONNX export
bin/build_engine.sh Target-native TensorRT build and correctness gate
bin/calibrate_teacache.sh Matched CFG-1/CFG-5 target TeaCache sweep
bin/doctor.py Package, foundation, export, and runtime verification
vendor/raven-image3f-joint/ Raven source snapshot used for export/runtime
validation/b200_rgb030_depth030_20260730/ Small retained B200 reports and representative comparison images; no generated engine/ONNX
docs/RTX4090_RUNBOOK.md Target qualification and acceptance gates

Non-negotiable operational rules

  • Verify the package before export and verify the external foundation hashes.
  • Build and run in the same final GPU/software environment.
  • Re-export after changing the adapter, base DiT, exporter, or fixed input profile.
  • Rebuild after changing the GPU model, driver, CUDA-bearing PyTorch stack, TensorRT version, ONNX export, or builder settings.
  • Do not copy a B200, RTX PRO, or another 4090's engine into this bundle.
  • Keep CUDA Graph disabled unless it independently passes correctness, teardown, endurance, and latency gates on the final target.
  • Qualify TeaCache and five-step quality on representative DAgger trajectories, including closed-loop error accumulation.
  • Treat the RTX 4090 as unvalidated until the checks in docs/RTX4090_RUNBOOK.md pass.
Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
Robotics
loading