Lucasgitguy-co/image3d-student

Image3D Student v0.1.0 — first public prototype

Train a compact image-to-Gaussian-splat student on cached teacher outputs, using Apple Metal or CPU.

First release: v0.1.0 · Apache-2.0. See Windows, Linux and macOS instructions, all publisher model guides, license, third-party scope, and release notes.

This is working research code, not a pretrained general-purpose image-to-3D model. The included demonstration checkpoint learned only procedural ellipsoids. It has not learned from Idol, InstantMesh or Hi3D and should not be used as evidence of real-image quality.

What is implemented

Component	Status
Compact image encoder + coordinate decoder	Implemented, CPU and Metal training tested
Training, object-level validation, metrics and checkpoints	Implemented
Import/export of degree-zero Gaussian PLY	Implemented; preserves position, anisotropic scale, rotation, colour and opacity
Idol supervision	Import its PLY and the exact prepared input image
InstantMesh supervision	Import an aligned coarse triangle mesh as an auxiliary surface-occupancy target
Hi3D supervision	Import an aligned refined triangle mesh as a second auxiliary surface-occupancy target
Running Idol on Mac	Experimental staged adapter; tiny architecture components tested separately; full pretrained pipeline unverified
Running InstantMesh or Hi3D on Mac	Not implemented; official GPU implementations need porting
Binary/1-bit student	Not implemented; this small baseline uses FP32
Differentiable rendering / multiview photometric loss	Not implemented in this first baseline

The three-teacher training interface is present, but a fully local way to generate all three teachers' data is still unresolved. Importing an existing output does not mean the corresponding teacher can run on this Mac.

Start

On the Mac where this project was built, the dependencies are already installed in the working directory. From this delivered folder, bash run-local.sh doctor reuses that environment. Pass any command described below to bash run-local.sh instead of image3d. On a different machine, use the setup steps below.

Use Python 3.11–3.13. From this project directory:

bash setup.sh
source .venv/bin/activate
image3d doctor

setup.sh installs the student dependencies only. It does not download teacher weights. doctor checks this process's Metal access. Sandboxed terminals can report Metal unavailable even when a normal macOS Terminal can use it. Choose --device mps to require GPU execution; auto may fall back to CPU.

Run the synthetic pipeline test

image3d demo-data --out data/demo --count 12 --grid 8
image3d train --manifest data/demo/manifest.jsonl --out runs/demo \
  --device mps --width 64 --steps 100 --lr 0.001 --eval-every 25 --allow-synthetic
image3d generate --checkpoint runs/demo/best.pt \
  --image data/demo/images/fixture-007.png --out runs/demo/student.ply --device mps
image3d preview --ply runs/demo/student.ply --out runs/demo/turntable.png
python -m pytest -q

The data split is seeded and groups all samples with the same object ID together. The commands refuse to overwrite an existing run or exported PLY. Use a new directory for a new experiment. Ctrl-C saves completed optimizer steps to last.pt. This version does not resume optimizer training from a checkpoint.

With validation enabled, best.pt selects the lowest held-out distillation loss. last.pt always contains the final completed step. With --validation-fraction 0, there is no best.pt and no generalization measurement.

Train with real teacher data

1. Supply an image of one centered object and obtain its Idol PLY.
2. Cache the exact input framing seen by Idol, ideally the *_input.png saved by Idol.
3. Optionally attach aligned coarse/refined meshes for the same object and image.
4. Repeat for a dataset of different objects. Keep related views in the same --object-id.
5. Train and judge unseen objects, not only training loss.

image3d cache --image teacher/item01/input.png --idol-ply teacher/item01/teacher.ply \
  --id item01-view01 --object-id item01 --out data/teachers --grid 16

image3d train --manifest data/teachers/manifest.jsonl --out runs/teachers-v1 \
  --device mps --steps 1000 --batch-size 1

image3d generate --checkpoint runs/teachers-v1/best.pt \
  --image your-centered-cutout.png --out results/object.ply --device mps

The student does not perform automatic background removal. Use already-framed cutouts, with the object scale and orientation matching the training data. RGBA is composited on black, matching Idol's prepared images. An arbitrary photo with clutter or different framing is outside the baseline's input contract.

Attach all three teachers

image3d cache --image teacher/item02/input.png --idol-ply teacher/item02/teacher.ply \
  --coarse-mesh teacher/item02/instantmesh.obj --coarse-transform transforms/coarse.json \
  --fine-mesh teacher/item02/hi3d.glb --fine-transform transforms/fine.json \
  --id item02-view01 --object-id item02 --out data/teachers --grid 16

Each transform file is a JSON 4×4 affine matrix mapping mesh coordinates into the exported Idol PLY coordinate frame. Idol changes its internal z-up frame when exporting PLY; do not apply that rotation twice. Only use an identity transform if the assets are already aligned. Example identity:

[[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]]

Targets must fit inside [-0.5,0.5]^3. Out-of-bounds targets are rejected, not silently normalized. Check rotation, scale, translation and camera framing visually before combining teachers. Different teachers can hallucinate different unseen geometry. This baseline does not automatically align shapes or resolve their disagreements.

Geometry losses supervise auxiliary occupancy heads sharing the student features; they do not guarantee that final Gaussian surfaces exactly follow the refined mesh. Joint rendering-based supervision and confidence filtering are future work.

The manifest stores source labels, transforms, input/target hashes, object IDs and compression counts. A missing teacher is masked out of the loss. Do not label fabricated outputs as real teacher supervision.

Experimental local Idol runner

The adapter uses the official Idol architecture at pinned SDK commit 4702a7f00850e34b6e83a2170cc123c3e9570744. It loads background removal, conditioning, denoising, octree decoding and Gaussian decoding in stages, deleting models between stages. It replaces complex rotary embeddings with the upstream real-valued equivalent and chunks attention along the query dimension while retaining all keys/values. CUDA compilation and batched classifier-free guidance are disabled.

This reduces simultaneous working memory but does not establish that full Idol fits in 16 GB. The encoders, their operators, loading peaks and full-size attention still need real-weight testing. The runner has no silent CPU operator fallback, so unsupported operations surface as errors.

Install code dependencies separately:

uv pip install --python .venv/bin/python -e '.[teacher]'

Idol requires access to these gated encoders:

• DINOv3 ViT-H+
• FLUX.2-dev VAE

Accept their terms using your own Hugging Face account. Set the local cache consistently before logging in:

export HF_HOME="$PWD/.cache/huggingface"
hf auth login

# Explicitly allows model downloads. Start with a transparent single-object PNG.
image3d teacher-idol --image your-cutout.png --out teacher/item01 \
  --cache "$HF_HOME" --allow-download --device mps --steps 20 --gaussians 32768

The download command requires 16 GiB of free disk space as a conservative safety margin, not a measured model minimum. At build time this Mac had approximately 7.4 GB available before dependency installation, so no model weights were downloaded. With a complete existing cache, omit --allow-download: the runner then uses local files only. 20 steps is a reduced-cost experiment; it is not the teacher's default 50-step quality setting. Use --steps 50 once execution and memory have been verified.

Architecture and losses

The baseline is deliberately small: a four-stage convolutional image encoder produces a shared feature vector. A coordinate MLP predicts one Gaussian per cell in a normalized object cube, plus coarse/fine geometry logits. The default 16³ grid has at most 4,096 Gaussians; grid 32 has at most 32,768. Grid 8 is for smoke tests. An image encoder trained from scratch is not expected to generalize from a handful of examples.

When several Idol Gaussians occupy a cell, the cache retains the highest-opacity one. This is lossy compression, not a fidelity-preserving replacement for the full teacher. It can lose fine geometry, transparency and colour. Teacher colours outside display RGB are clipped for student supervision.

Training uses balanced cell-occupancy BCE, cell-scaled position error, colour error, log-scale error, quaternion sign-invariant rotation error, opacity error, and optional geometry occupancy BCE. It uses direct cached-target regression; no CUDA rasterizer is needed for student training. The preview renderer is an orthographic CPU diagnostic renderer, not a calibrated quality evaluator.

The model's speed comes from its small one-pass architecture, not from copying InstantMesh's speed through a loss. Teacher-level quality, real-photo reconstruction, mesh topology and 1-bit compression are not established.

Reproducibility

requirements-tested.txt records the environment used for verification; pyproject.toml specifies installable dependencies. See VERIFICATION.md and examples/synthetic-smoke/ for the delivered test evidence. No external GPU or cloud training service was used. Source and Python package downloads did use the internet.

Upstream references: InstantMesh, Hi3D, Idol SDK.