ZIL: Zero-shot Image-to-LiDAR Registration

This repository hosts the public inference checkpoint for ZIL, a foundation model for zero-shot, non-synchronized image-to-LiDAR registration. The model estimates the camera pose in an unaligned LiDAR point cloud without dataset-specific fine-tuning.

Implementation, training, preprocessing, and evaluation code live in the ZIL GitHub repository. This model repository contains artifacts only.

Try the interactive Gradio Space or visit the project page.

Usage

Install the CUDA stack described by the GitHub repository, then install ZIL:

pip install git+https://github.com/ZijunLi7/ZIL.git

import numpy as np
from zil import ZIL

K = np.array([
    [721.5, 0.0, 609.6],
    [0.0, 721.5, 172.9],
    [0.0, 0.0, 1.0],
])

model = ZIL.from_pretrained("ZijunLi/ZIL")
result = model.register("image.png", "points.bin", K)
print(result.transform)

The returned matrix is T_lidar_from_camera for column vectors, with translation in metres. Input LiDAR points must use FLU axes: x forward, y left, z up. A .bin file is interpreted as float32 N x 4 by default; only XYZ is used.

Files

• zil-inference.safetensors: model weights with the zil.pose.v1 metadata schema.
• config.json: input conventions and the public inference recipe.

Limitations

• The released implementation requires an NVIDIA CUDA GPU because its sparse-convolution backbone has no supported CPU path. FlashAttention is optional; PyTorch SDPA is used when that extension is unavailable.
• The model is intended for outdoor camera/LiDAR systems. Performance outside the training distribution should be independently validated.
• Pose quality depends on correct image intrinsics, metric point coordinates, and the documented axis convention.

License

The checkpoint is released under CC BY-NC 4.0, including non-commercial restrictions inherited from its VGGT-1B and Utonia initialization. Source-code licensing and all third-party notices are documented in the GitHub repository.