Tasks

Text-to-3D

Text-to-3D models take in text input and produce 3D output.

Inputs
Prompt

a cat statue

Text-to-3D Model
Output
Result

text-to-3d-3d-output-filename.glb

About Text-to-3D

Use Cases

Text-to-3D models can be used in a wide variety of applications that require 3D, such as games, animation, design, architecture, engineering, marketing, and more.

Text-to-3D Thumbnail

This task is similar to the image-to-3d task, but takes text input instead of image input. In practice, this is often equivalent to a combination of text-to-image and image-to-3d. That is, the text is first converted to an image, then the image is converted to 3D.

Generating Meshes

Meshes are the standard representation of 3D in industry.

Generating Gaussian Splats

Gaussian Splatting is a rendering technique that represents scenes as fuzzy points.

Inference

Inference for this task typically leverages the Diffusers library for inference, using Custom Pipelines.

These are unstandardized and depend on the model. More details can be found in each model repository.

import torch
import requests
import numpy as np
from io import BytesIO
from diffusers import DiffusionPipeline
from PIL import Image

pipeline = DiffusionPipeline.from_pretrained(
    "dylanebert/LGM-full",
    custom_pipeline="dylanebert/LGM-full",
    torch_dtype=torch.float16,
    trust_remote_code=True,
).to("cuda")

input_prompt = "a cat statue"
result = pipeline(input_prompt, None)
result_path = "/tmp/output.ply"
pipeline.save_ply(result, result_path)

In the code above, we:

  1. Import the necessary libraries
  2. Load the LGM-full model and custom pipeline
  3. Define the input prompt
  4. Run the pipeline on the input prompt
  5. Save the output to a file

Output Formats

Meshes can be in .obj, .glb, .stl, or .gltf format. Other formats are allowed, but won't be rendered in the gradio Model3D component.

Splats can be in .ply or .splat format. They can be rendered in the gradio Model3D component using the gsplat.js library.

Useful Resources

Compatible libraries

Text-to-3D demo

No example widget is defined for this task.

Note Contribute by proposing a widget for this task !

Models for Text-to-3D
Browse Models (34)

Note Generative 3D gaussian splatting model.

Datasets for Text-to-3D
Browse Datasets (7)

Note A large dataset of over 10 million 3D objects.

Note Descriptive captions for 3D objects in Objaverse.

Spaces using Text-to-3D

Note Text-to-3D demo with mesh outputs.

Note Text/image-to-3D demo with splat outputs.

Metrics for Text-to-3D

No example metric is defined for this task.

Note Contribute by proposing a metric for this task !