OzzyGT/idol_custom_blocks

Idol Modular Diffusers

Generated with the code below (seed 42, 50 steps, 262,144 Gaussians).

Modular Diffusers blocks for speridlabs/idol: generate a 3D Gaussian splat from a single image. The model code ships with these blocks; the converted weights are in OzzyGT/idol_diffusers.

Requires PyTorch, diffusers==0.42.1, transformers==5.14.1, accelerate>=1.14, safetensors>=0.8, numpy and pillow.

Built with DINOv3.

Licensing

The blocks are Apache-2.0, with the model classes, image encoding, octree sampling and weight conversion adapted from speridlabs/idol-sdk under Apache-2.0. See NOTICE. This repository contains no model weights.

Usage

The example uses trust_remote_code=True, which loads these blocks and the model classes that ship with them. The weights repository holds no code.

Note: trust_remote_code=True runs Python code downloaded from the Hub. Only use it with repositories you trust, and review their code before passing a different one.

import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image


pipe = ModularPipeline.from_pretrained("OzzyGT/idol_custom_blocks", trust_remote_code=True)
pipe.load_components(dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("https://huggingface.co/datasets/OzzyGT/diffusers-examples/resolve/main/idol/nb-dragon-anime_input.png")
result = pipe(
    image=image,
    generator=torch.Generator("cuda").manual_seed(42),
    output=["gaussians", "camera_latents"],
)
gaussians = result["gaussians"]
camera_latents = result["camera_latents"]

To save the result as a standard 3DGS .ply file that any Gaussian splat viewer opens, use the Idol recipe from diffusers-recipes.

The input is a prepared image: 1024x1024, the object cropped with a margin and composited on black, with no automatic background removal or crop. num_gaussians sets the size of the splat, from 32,768 to 1,048,576 in multiples of 32.

The output is Gaussians, not a file: gaussians holds the activated xyz, features_dc, scaling, rotation and opacity, and camera_latents the predicted input camera. Coordinates are normalized and Z-up.

Converting the weights

convert_idol_to_diffusers.py converts the original checkpoint into the transformer, octree_decoder and gaussian_decoder folders of OzzyGT/idol_diffusers. It imports the model classes next to it, so download it together with them:

hf download OzzyGT/idol_custom_blocks convert_idol_to_diffusers.py transformer_idol.py octree_decoder_idol.py gaussian_decoder_idol.py --local-dir idol_convert
cd idol_convert
python convert_idol_to_diffusers.py --output_dir ./idol_diffusers

It downloads the checkpoint from speridlabs/idol, or reads a local copy with --checkpoint /path/to/idol.