Text-to-3D
Text-to-3D models take in text input and produce 3D output.
Prompt
a cat statue
Result
text-to-3d-3d-output-filename.glb
About Text-to-3D
Use Cases
Text-to-3D models can be used in a wide variety of applications that require 3D, such as games, animation, design, architecture, engineering, marketing, and more.
This task is similar to the image-to-3d task, but takes text input instead of image input. In practice, this is often equivalent to a combination of text-to-image and image-to-3d. That is, the text is first converted to an image, then the image is converted to 3D.
Generating Meshes
Meshes are the standard representation of 3D in industry.
Generating Gaussian Splats
Gaussian Splatting is a rendering technique that represents scenes as fuzzy points.
Inference
Inference for this task typically leverages the Diffusers library for inference, using Custom Pipelines.
These are unstandardized and depend on the model. More details can be found in each model repository.
import torch
import requests
import numpy as np
from io import BytesIO
from diffusers import DiffusionPipeline
from PIL import Image
pipeline = DiffusionPipeline.from_pretrained(
"dylanebert/LGM-full",
custom_pipeline="dylanebert/LGM-full",
torch_dtype=torch.float16,
trust_remote_code=True,
).to("cuda")
input_prompt = "a cat statue"
result = pipeline(input_prompt, None)
result_path = "/tmp/output.ply"
pipeline.save_ply(result, result_path)
In the code above, we:
- Import the necessary libraries
- Load the
LGM-full
model and custom pipeline - Define the input prompt
- Run the pipeline on the input prompt
- Save the output to a file
Output Formats
Meshes can be in .obj
, .glb
, .stl
, or .gltf
format. Other formats are allowed, but won't be rendered in the gradio Model3D component.
Splats can be in .ply
or .splat
format. They can be rendered in the gradio Model3D component using the gsplat.js library.
Useful Resources
Compatible libraries
No example widget is defined for this task.
Note Contribute by proposing a widget for this task !
Note Text-to-3D mesh model by OpenAI
Note Generative 3D gaussian splatting model.
Note A large dataset of over 10 million 3D objects.
Note Descriptive captions for 3D objects in Objaverse.
Note Text-to-3D demo with mesh outputs.
Note Text/image-to-3D demo with splat outputs.
No example metric is defined for this task.
Note Contribute by proposing a metric for this task !