thanks to Alpha-VLLM ❤

README.md

@@ -0,0 +1,170 @@
+---
+license: apache-2.0
+tags:
+- text-to-audio
+- music
+library_name: transformers
+---
+
+# Lumina Text-to-Music
+
+We will provide our implementation and pretrained models as open source in this repository recently.
+
+- Generation Model: Flag-DiT
+- Text Encoder: [FLAN-T5-Large](https://huggingface.co/google/flan-t5-large)
+- VAE: Make an Audio 2, finetuned from [Makee an Audio](https://github.com/Text-to-Audio/Make-An-Audio)
+- Decoder: [Vocoder](https://github.com/NVIDIA/BigVGAN)
+
+## 📰 News
+
+- [2024-06-07] 🚀🚀🚀 We release the initial version of `Lumina-T2Music` for text-to-music generation.
+
+## Installation
+
+Before installation, ensure that you have a working ``nvcc``
+
+```bash
+# The command should work and show the same version number as in our case. (12.1 in our case).
+nvcc --version
+```
+
+On some outdated distros (e.g., CentOS 7), you may also want to check that a late enough version of
+``gcc`` is available
+
+```bash
+# The command should work and show a version of at least 6.0.
+# If not, consult distro-specific tutorials to obtain a newer version or build manually.
+gcc --version
+```
+
+Downloading Lumina-T2X repo from github:
+
+```bash
+git clone https://github.com/Alpha-VLLM/Lumina-T2X
+```
+
+### 1. Create a conda environment and install PyTorch
+
+Note: You may want to adjust the CUDA version [according to your driver version](https://docs.nvidia.com/deploy/cuda-compatibility/#default-to-minor-version).
+
+```bash
+conda create -n Lumina_T2X -y
+conda activate Lumina_T2X
+conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
+```
+
+### 2. Install dependencies
+
+>[!Warning]
+> The environment dependencies for Lumina-T2Music are different from those for Lumina-T2I. Please install the appropriate environment.
+
+Installing `Lumina-T2Music` dependencies:
+
+```bash
+cd .. # If you are in the `lumina_music` directory, execute this line.
+pip install -e ".[music]"
+```
+
+or you can use `requirements.txt` to install the environment.
+
+```bash
+cd lumina_music # If you are not in the `lumina_music` folder, run this line.
+pip install -r requirements.txt
+```
+
+### 3. Install ``flash-attn``
+
+```bash
+pip install flash-attn --no-build-isolation
+```
+
+### 4. Install [nvidia apex](https://github.com/nvidia/apex) (optional)
+
+>[!Warning]
+> While Apex can improve efficiency, it is *not* a must to make Lumina-T2X work.
+>
+> Note that Lumina-T2X works smoothly with either:
+> + Apex not installed at all; OR
+> + Apex successfully installed with CUDA and C++ extensions.
+>
+> However, it will fail when:
+> + A Python-only build of Apex is installed.
+>
+> If the error `No module named 'fused_layer_norm_cuda'` appears, it typically means you are using a Python-only build of Apex. To resolve this, please run `pip uninstall apex`, and Lumina-T2X should then function correctly.
+
+You can clone the repo and install following the official guidelines (note that we expect a full
+build, i.e., with CUDA and C++ extensions)
+
+```bash
+pip install ninja
+git clone https://github.com/NVIDIA/apex
+cd apex
+# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
+# otherwise
+pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
+```
+
+## Inference
+
+### Preparation
+
+Prepare the pretrained checkpoints.
+
+⭐⭐ (Recommended) you can use `huggingface-cli` downloading our model:
+
+```bash
+huggingface-cli download --resume-download Alpha-VLLM/Lumina-T2Music --local-dir /path/to/ckpt
+```
+
+or using git for cloning the model you want to use:
+
+```bash
+git clone https://huggingface.co/Alpha-VLLM/Lumina-T2Music
+```
+
+### Web Demo
+
+To host a local gradio demo for interactive inference, run the following command:
+
+1. updated `AutoencoderKL` ckpt path
+
+you should update `configs/lumina-text2music.yaml` to set `AutoencoderKL` checkpoint path. Please replace `/path/to/ckpt` with the path where your checkpoints are located (<real_path>).
+
+```diff
+  ...
+        depth: 16
+        max_len: 1000
+
+    first_stage_config:
+      target: models.autoencoder1d.AutoencoderKL
+      params:
+        embed_dim: 20
+        monitor: val/rec_loss
+        - ckpt_path: /path/to/ckpt/maa2/maa2.ckpt
+        + ckpt_path: <real_path>/maa2/maa2.ckpt
+        ddconfig:
+          double_z: true
+          in_channels: 80
+          out_ch: 80
+  ...
+```
+
+2. setting `Lumina-T2Music` and `Vocoder` checkpoint path and run demo
+
+Please replace `/path/to/ckpt` with the actual downloaded path.
+
+```bash
+# `/path/to/ckpt` should be a directory containing `music_generation`, `maa2`, and `bigvnat`.
+
+# default
+python -u demo_music.py \
+    --ckpt "/path/to/ckpt/music_generation" \
+    --vocoder_ckpt "/path/to/ckpt/bigvnat" \
+    --config_path "configs/lumina-text2music.yaml" \
+    --sample_rate 16000
+```
+
+## Disclaimer
+
+Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.

bigvnat/args.yml

@@ -0,0 +1,63 @@
+resblock: '1'
+num_gpus: 0
+batch_size: 64
+num_mels: 80
+learning_rate: 0.0001
+adam_b1: 0.8
+adam_b2: 0.99
+lr_decay: 0.999
+seed: 1234
+upsample_rates:
+- 4
+- 4
+- 2
+- 2
+- 2
+- 2
+upsample_kernel_sizes:
+- 8
+- 8
+- 4
+- 4
+- 4
+- 4
+upsample_initial_channel: 1536
+resblock_kernel_sizes:
+- 3
+- 7
+- 11
+resblock_dilation_sizes:
+- - 1
+  - 3
+  - 5
+- - 1
+  - 3
+  - 5
+- - 1
+  - 3
+  - 5
+activation: snakebeta
+snake_logscale: true
+resolutions:
+- - 1024
+  - 120
+  - 600
+- - 2048
+  - 240
+  - 1200
+- - 512
+  - 50
+  - 240
+mpd_reshapes:
+- 2
+- 3
+- 5
+- 7
+- 11
+use_spectral_norm: false
+discriminator_channel_mult: 1
+num_workers: 4
+dist_config:
+  dist_backend: nccl
+  dist_url: tcp://localhost:54341
+  world_size: 1

bigvnat/best_netG.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:970ca75ee4d5ce583e9396a4534acb14971ea2b4f1c22e038f476680c868a789
+size 449217313

maa2/maa2.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7621db6654b1a96398cf20edd6bc783ba8b7d4bc074e2ac42d609f0426480f19
+size 7308070914

music_generation/119.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:988e12b6ac963968a9214a5ec6a91abb1904b299639c54173a6432ea1f3631c8
+size 4592631803