appautomaton/sam-3d-objects-mlx

SAM 3D Objects MLX for mlx-spatial

Run SAM 3D Objects on Apple Silicon through mlx-spatial, using an MLX-ready safetensors bundle instead of local PyTorch checkpoint conversion.

This bundle is for users who want masked single-image object reconstruction on a Mac: download the model, provide an image plus object mask, and generate SAM3D Gaussian or mesh artifacts with mlx-spatial-sam3d. No CUDA is required.

Quick Start: Masked Image to 3D Object

Install mlx-spatial:

pip install mlx-spatial

Download this model bundle:

hf download appautomaton/sam-3d-objects-mlx \
  --local-dir weights/sam-3d-objects-mlx

Validate the local layout:

mlx-spatial-sam3d validate weights/sam-3d-objects-mlx
mlx-spatial-sam3d inspect weights/sam-3d-objects-mlx

Generate a Gaussian-splat PLY:

mlx-spatial-sam3d reconstruct weights/sam-3d-objects-mlx image.png \
  --mask mask.png \
  --output outputs/sam3d/object/gaussians.ply \
  --trace-output outputs/sam3d/object/trace.json

Generate a GLB mesh as well:

mlx-spatial-sam3d reconstruct weights/sam-3d-objects-mlx image.png \
  --mask mask.png \
  --output outputs/sam3d/object/gaussians.ply \
  --glb-output outputs/sam3d/object/object.glb \
  --trace-output outputs/sam3d/object/trace.json

The trace records quality diagnostics such as sparse-structure occupancy, geometry range, opacity, selected mask, and output paths.

What This Model Bundle Provides

This Hugging Face repository contains the converted SAM 3D Objects checkpoint bundle expected by mlx-spatial:

checkpoints/pipeline.yaml
checkpoints/ss_generator.safetensors
checkpoints/slat_generator.safetensors
checkpoints/ss_decoder.safetensors
checkpoints/slat_decoder_gs.safetensors
checkpoints/slat_decoder_gs_4.safetensors
checkpoints/slat_decoder_mesh.safetensors
checkpoints/conversion_metadata/
conversion_manifest.json
weight-audit-source-vs-mlx.json

It also includes the converted MoGe ViT-L pointmap dependency used by the default SAM3D preprocessing path:

moge/model.safetensors
moge/conversion_metadata/model.yaml

The bundled MoGe checkpoint lets the normal mlx-spatial-sam3d reconstruct command run from one model repository. Advanced users can still pass a different MoGe root or provide an external pointmap.

Best For

• Apple Silicon MLX inference experiments.
• Masked single-image object reconstruction.
• SAM3D Gaussian Splat PLY generation with mlx-spatial.
• SAM3D mesh or GLB export workflows.
• Researchers and developers who need SAM 3D Objects weights in safetensors format.

Current Limitations

• This is an unofficial converted derivative bundle, not an official Meta or MoGe release.
• The upstream facebook/sam-3d-objects Hugging Face repository is gated. Users should have access to the upstream model and accept the upstream terms before using this conversion.
• Reconstruction requires an input image and a useful binary object mask.
• Standard 3D Gaussian viewers may use different coordinate conventions than SAM 3D Objects' native output convention.
• This is not an int8, 4-bit, or otherwise quantized model.
• CUDA is not required and is not used by mlx-spatial SAM3D inference.

Conversion Fidelity

The converted checkpoint bundle was audited against the original SAM 3D Objects checkpoint files.

Role	Tensors	Missing	Extra	Shape mismatches	Nonzero numeric diffs	Max abs diff
ss_generator	1,741	0	0	0	0	0.0
slat_generator	1,225	0	0	0	0	0.0
ss_decoder	74	0	0	0	0	0.0
slat_decoder_gs	101	0	0	0	0	0.0
slat_decoder_mesh	120	0	0	0	0	0.0
slat_decoder_gs_4	101	0	0	0	0	0.0

Total compared SAM3D tensors: 3,362.

Some decoder tensors are stored as float32 in this safetensors bundle even when the source checkpoint tensor was float16. This is lossless for value preservation. The numeric audit compares values after float32 materialization and found zero difference.

See weight-audit-source-vs-mlx.json for the audit summary.

Conversion Details

This bundle was produced from the original SAM 3D Objects checkpoint layout with:

mlx-spatial-sam3d convert weights/sam-3d-objects \
  --output-root weights/sam-3d-objects-mlx \
  --moge-root weights/moge-vitl \
  --moge-output-root weights/sam-3d-objects-mlx/moge \
  --max-archive-gb 16

The conversion rewrites checkpoint references in pipeline.yaml from PyTorch checkpoint files to .safetensors files. It does not quantize the model or change the architecture.

Project Links

• Runtime package: mlx-spatial
• mlx-spatial PyPI package: https://pypi.org/project/mlx-spatial/
• mlx-spatial source: https://github.com/appautomaton/mlx-spatial
• This model repo: https://huggingface.co/appautomaton/sam-3d-objects-mlx

Upstream Source and License

This bundle is based on Meta's SAM 3D Objects release and includes a converted MoGe dependency:

• Upstream SAM 3D Objects model: https://huggingface.co/facebook/sam-3d-objects
• Upstream SAM 3D Objects code: https://github.com/facebookresearch/sam-3d-objects
• SAM License: https://github.com/facebookresearch/sam-3d-objects/blob/main/LICENSE
• MoGe dependency: included as converted moge/model.safetensors

The original SAM 3D Objects checkpoints and code are licensed by Meta under the SAM License. The included MoGe dependency follows its own upstream license and terms.

This repository is not an official Meta or MoGe release. Users are responsible for complying with the upstream SAM 3D Objects and MoGe license, access, and use requirements.

If you use this conversion, cite the original SAM 3D Objects work and link to the upstream model and code.