license: apache-2.0
Lumina-T2I
Lumina-T2I is a model that generates images based on text conditions, supporting various text encoders and models of different parameter sizes. With minimal training costs, it achieves high-quality image generation by training from scratch. Additionally, it offers usage through CLI console programs and Web Demo displays.
Our generative model has LargeDiT as the backbone, the text encoder is the LLaMa 7B model, and the VAE uses a version of sdxl fine-tuned by stabilityai.
- Generation Model: Large-DiT
- Text Encoder: LLaMA2-7B
- VAE: sdxl-vae
📰 News
- [2024-4-1] 🚀🚀🚀 We release the initial version of Lumina-T2I for text-to-image generation
🎮 Model Zoo
More checkpoints of our model will be released soon~
| Resolution | Flag-DiT Parameter | Text Encoder | Prediction | Download URL |
|---|---|---|---|---|
| 1024 | 5B | LLaMA2-7B | Rectified Flow | hugging face |
Installation
Before installation, ensure that you have a working nvcc
# The command should work and show the same version number as in our case. (12.1 in our case).
nvcc --version
On some outdated distros (e.g., CentOS 7), you may also want to check that a late enough version of
gcc is available
# The command should work and show a version of at least 6.0.
# If not, consult distro-specific tutorials to obtain a newer version or build manually.
gcc --version
Downloading Lumina-T2X repo from github:
git clone https://github.com/Alpha-VLLM/Lumina-T2X
1. Create a conda environment and install PyTorch
Note: You may want to adjust the CUDA version according to your driver version.
conda create -n Lumina_T2X -y
conda activate Lumina_T2X
conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
2. Install dependencies
pip install diffusers fairscale accelerate tensorboard transformers gradio torchdiffeq click
or you can use
cd lumina-t2i
pip install -r requirements.txt
3. Install flash-attn
pip install flash-attn --no-build-isolation
4. Install nvidia apex (optional)
While Apex can improve efficiency, it is not a must to make Lumina-T2X work.
Note that Lumina-T2X works smoothly with either:
- Apex not installed at all; OR
- Apex successfully installed with CUDA and C++ extensions.
However, it will fail when:
- A Python-only build of Apex is installed.
If the error
No module named 'fused_layer_norm_cuda'appears, it typically means you are using a Python-only build of Apex. To resolve this, please runpip uninstall apex, and Lumina-T2X should then function correctly.
You can clone the repo and install following the official guidelines (note that we expect a full build, i.e., with CUDA and C++ extensions)
pip install ninja
git clone https://github.com/NVIDIA/apex
cd apex
# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
# otherwise
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Inference
To ensure that our generative model is ready to use immediately, we provide a user-friendly CLI program and a locally deployable Web Demo site.
CLI
- Install Lumina-T2I
pip install -e .
- Prepare the pre-trained model
⭐⭐ (Recommended) you can use huggingface_cli to download our model:
huggingface-cli download --resume-download Alpha-VLLM/Lumina-T2I --local-dir /path/to/ckpt
or using git for cloning the model you want to use:
git clone https://huggingface.co/Alpha-VLLM/Lumina-T2I
- Setting your personal inference configuration
Update your own personal inference settings to generate different styles of images, checking config/infer/config.yaml for detailed settings. Detailed config structure:
/path/to/ckptshould be a directory containingconsolidated*.pthandmodel_args.pth
- settings:
model:
ckpt: "/path/to/ckpt" # if ckpt is "", you should use `--ckpt` for passing model path when using `lumina` cli.
ckpt_lm: "" # if ckpt is "", you should use `--ckpt_lm` for passing model path when using `lumina` cli.
token: "" # if LLM is a huggingface gated repo, you should input your access token from huggingface and when token is "", you should `--token` for accessing the model.
transport:
path_type: "Linear" # option: ["Linear", "GVP", "VP"]
prediction: "velocity" # option: ["velocity", "score", "noise"]
loss_weight: "velocity" # option: [None, "velocity", "likelihood"]
sample_eps: 0.1
train_eps: 0.2
ode:
atol: 1e-6 # Absolute tolerance
rtol: 1e-3 # Relative tolerance
reverse: false # option: true or false
likelihood: false # option: true or false
sde:
sampling_method: "Euler" # option: ["Euler", "Heun"]
diffusion_form: "sigma" # option: ["constant", "SBDM", "sigma", "linear", "decreasing", "increasing-decreasing"]
diffusion_norm: 1.0 # range: 0-1
last_step: Mean # option: [None, "Mean", "Tweedie", "Euler"]
last_step_size: 0.04
infer:
resolution: "1024x1024" # option: ["1024x1024", "512x2048", "2048x512", "(Extrapolation) 1664x1664", "(Extrapolation) 1024x2048", "(Extrapolation) 2048x1024"]
num_sampling_steps: 60 # range: 1-1000
cfg_scale: 4. # range: 1-20
solver: "euler" # option: ["euler", "dopri5", "dopri8"]
t_shift: 4 # range: 1-20 (int only)
ntk_scaling: true # option: true or false
proportional_attn: true # option: true or false
seed: 0 # rnage: any number
- model:
ckpt: lumina-t2i checkpoint path from huggingface repo containingconsolidated*.pthandmodel_args.pth.ckpt_lm: LLM checkpoint.token: huggingface access token for accessing gated repo.
- transport:
path_type: the type of path for transport: 'Linear', 'GVP' (Geodesic Vector Pursuit), or 'VP' (Vector Pursuit).prediction: the prediction model for the transport dynamics.loss_weight: the weighting of different components in the loss function, can be 'velocity' for dynamic modeling, 'likelihood' for statistical consistency, or None for no weightingsample_eps: sampling in the transport model.train_eps: training to stabilize the learning process.
- ode:
atol: Absolute tolerance for the ODE solver. (options: ["Linear", "GVP", "VP"])rtol: Relative tolerance for the ODE solver. (option: ["velocity", "score", "noise"])reverse: run the ODE solver in reverse. (option: [None, "velocity", "likelihood"])likelihood: Enable calculation of likelihood during the ODE solving process.
- sde
sampling-method: the numerical method used for sampling the stochastic differential equation: 'Euler' for simplicity or 'Heun' for improved accuracy.diffusion-form: form of diffusion coefficient in the SDEdiffusion-norm: Normalizes the diffusion coefficient, affecting the scale of the stochastic component.last-step: form of last step taken in the SDElast-step-size: size of the last step taken
- infer
resolution: generated image resolution.num_sampling_steps: sampling step for generating image.cfg_scale: classifier-free guide scaling factorsolver: solver for image generation.t_shift: time shift factor.ntk_scaling: ntk rope scaling factor.proportional_attn: Whether to use proportional attention.seed: random initialization seeds.
- Run with CLI
inference command:
lumina infer -c <config_path> <caption_here> <output_dir>
e.g. Demo command:
cd lumina-t2i
lumina infer -c "config/infer/settings.yaml" "a snow man of ..." "./outputs"
Web Demo
To host a local gradio demo for interactive inference, run the following command:
# `/path/to/ckpt` should be a directory containing `consolidated*.pth` and `model_args.pth`
# default
python -u demo.py --ckpt "/path/to/ckpt"
# the demo by default uses bf16 precision. to switch to fp32:
python -u demo.py --ckpt "/path/to/ckpt" --precision fp32
# use ema model
python -u demo.py --ckpt "/path/to/ckpt" --ema