R3D2

One-step diffusion for realistic lighting and shadows around inserted 3D assets. This is the tiny VAE version of R3D2.

Paper · Code · Project · R3D2-big

Non-commercial use only. This model is released under the Waymo Dataset License Agreement. All downstream use or modification is subject to its terms, including Section 4. Full agreements and attribution: LICENSE.md, NOTICE.

Use with Diffusers

Tested with PyTorch 2.6.0, Diffusers 0.40.0 and Transformers 5.17.0. A CUDA GPU is required for the example below:

pip install torch==2.6.0 diffusers==0.40.0 transformers==5.17.0 peft

import torch
from diffusers import DiffusionPipeline
from PIL import Image

pipe = DiffusionPipeline.from_pretrained(
    "bertaveira/R3D2",
    trust_remote_code=True,
    torch_dtype=torch.float16,
).to("cuda")

pipe(Image.open("input.png").convert("RGB")).images[0].save("output.png")

Use with the original repository

Inside zenseact/R3D2, after uv sync:

uv run huggingface-cli download bertaveira/R3D2 --local-dir checkpoints/R3D2
uv run r3d2/scripts/eval.py --model-path checkpoints/R3D2 \
    --image-path input.png --output-path output.png

The repository CLI uses FP32. For lower GPU memory use, use the FP16 example above.

Credits

Built on SD-Turbo and TAESD, using the Waymo Open Dataset. Powered by Stability AI. For citation, see the paper's repository.