Commit
β’
235491a
1
Parent(s):
9e947a4
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,166 @@
|
|
1 |
-
---
|
2 |
-
license: apache-2.0
|
3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
tags:
|
4 |
+
- text-to-audio
|
5 |
+
---
|
6 |
+
|
7 |
+
# Lumina-T2Music
|
8 |
+
|
9 |
+
We will provide our implementation and pretrained models as open source in this repository recently.
|
10 |
+
|
11 |
+
- Generation Model: Flag-DiT
|
12 |
+
- Text Encoder: [FLAN-T5-Large](https://huggingface.co/google/flan-t5-large)
|
13 |
+
|
14 |
+
## π° News
|
15 |
+
|
16 |
+
- [2024-06-07] πππ We release the initial version of `Lumina-T2Music` for text-to-music generation.
|
17 |
+
|
18 |
+
## Installation
|
19 |
+
|
20 |
+
Before installation, ensure that you have a working ``nvcc``
|
21 |
+
|
22 |
+
```bash
|
23 |
+
# The command should work and show the same version number as in our case. (12.1 in our case).
|
24 |
+
nvcc --version
|
25 |
+
```
|
26 |
+
|
27 |
+
On some outdated distros (e.g., CentOS 7), you may also want to check that a late enough version of
|
28 |
+
``gcc`` is available
|
29 |
+
|
30 |
+
```bash
|
31 |
+
# The command should work and show a version of at least 6.0.
|
32 |
+
# If not, consult distro-specific tutorials to obtain a newer version or build manually.
|
33 |
+
gcc --version
|
34 |
+
```
|
35 |
+
|
36 |
+
Downloading Lumina-T2X repo from github:
|
37 |
+
|
38 |
+
```bash
|
39 |
+
git clone https://github.com/Alpha-VLLM/Lumina-T2X
|
40 |
+
```
|
41 |
+
|
42 |
+
### 1. Create a conda environment and install PyTorch
|
43 |
+
|
44 |
+
Note: You may want to adjust the CUDA version [according to your driver version](https://docs.nvidia.com/deploy/cuda-compatibility/#default-to-minor-version).
|
45 |
+
|
46 |
+
```bash
|
47 |
+
conda create -n Lumina_T2X -y
|
48 |
+
conda activate Lumina_T2X
|
49 |
+
conda install python=3.11 pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y
|
50 |
+
```
|
51 |
+
|
52 |
+
### 2. Install dependencies
|
53 |
+
|
54 |
+
>[!Warning]
|
55 |
+
> The environment dependencies for Lumina-T2Music are different from those for Lumina-T2I. Please install the appropriate environment.
|
56 |
+
|
57 |
+
Installing `Lumina-T2Music` dependencies:
|
58 |
+
|
59 |
+
```bash
|
60 |
+
cd .. # If you are in the `lumina_music` directory, execute this line.
|
61 |
+
pip install -e ".[music]"
|
62 |
+
```
|
63 |
+
|
64 |
+
or you can use `requirements.txt` to install the environment.
|
65 |
+
|
66 |
+
```bash
|
67 |
+
cd lumina_music # If you are not in the `lumina_music` folder, run this line.
|
68 |
+
pip install -r requirements.txt
|
69 |
+
```
|
70 |
+
|
71 |
+
### 3. Install ``flash-attn``
|
72 |
+
|
73 |
+
```bash
|
74 |
+
pip install flash-attn --no-build-isolation
|
75 |
+
```
|
76 |
+
|
77 |
+
### 4. Install [nvidia apex](https://github.com/nvidia/apex) (optional)
|
78 |
+
|
79 |
+
>[!Warning]
|
80 |
+
> While Apex can improve efficiency, it is *not* a must to make Lumina-T2X work.
|
81 |
+
>
|
82 |
+
> Note that Lumina-T2X works smoothly with either:
|
83 |
+
> + Apex not installed at all; OR
|
84 |
+
> + Apex successfully installed with CUDA and C++ extensions.
|
85 |
+
>
|
86 |
+
> However, it will fail when:
|
87 |
+
> + A Python-only build of Apex is installed.
|
88 |
+
>
|
89 |
+
> If the error `No module named 'fused_layer_norm_cuda'` appears, it typically means you are using a Python-only build of Apex. To resolve this, please run `pip uninstall apex`, and Lumina-T2X should then function correctly.
|
90 |
+
|
91 |
+
You can clone the repo and install following the official guidelines (note that we expect a full
|
92 |
+
build, i.e., with CUDA and C++ extensions)
|
93 |
+
|
94 |
+
```bash
|
95 |
+
pip install ninja
|
96 |
+
git clone https://github.com/NVIDIA/apex
|
97 |
+
cd apex
|
98 |
+
# if pip >= 23.1 (ref: https://pip.pypa.io/en/stable/news/#v23-1) which supports multiple `--config-settings` with the same key...
|
99 |
+
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
|
100 |
+
# otherwise
|
101 |
+
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./
|
102 |
+
```
|
103 |
+
|
104 |
+
## Inference
|
105 |
+
|
106 |
+
### Preparation
|
107 |
+
|
108 |
+
Prepare the pretrained checkpoints.
|
109 |
+
|
110 |
+
ββ (Recommended) you can use `huggingface-cli` downloading our model:
|
111 |
+
|
112 |
+
```bash
|
113 |
+
huggingface-cli download --resume-download Alpha-VLLM/Lumina-T2Music --local-dir /path/to/ckpt
|
114 |
+
```
|
115 |
+
|
116 |
+
or using git for cloning the model you want to use:
|
117 |
+
|
118 |
+
```bash
|
119 |
+
git clone https://huggingface.co/Alpha-VLLM/Lumina-T2Music
|
120 |
+
```
|
121 |
+
|
122 |
+
### Web Demo
|
123 |
+
|
124 |
+
To host a local gradio demo for interactive inference, run the following command:
|
125 |
+
|
126 |
+
1. updated `AutoencoderKL` ckpt path
|
127 |
+
|
128 |
+
you should update `configs/lumina-text2music.yaml` to set `AutoencoderKL` checkpoint path. Please replace `/path/to/ckpt` with the path where your checkpoints are located (<real_path>).
|
129 |
+
|
130 |
+
```diff
|
131 |
+
...
|
132 |
+
depth: 16
|
133 |
+
max_len: 1000
|
134 |
+
|
135 |
+
first_stage_config:
|
136 |
+
target: models.autoencoder1d.AutoencoderKL
|
137 |
+
params:
|
138 |
+
embed_dim: 20
|
139 |
+
monitor: val/rec_loss
|
140 |
+
- ckpt_path: /path/to/ckpt/maa2/maa2.ckpt
|
141 |
+
+ ckpt_path: <real_path>/maa2/maa2.ckpt
|
142 |
+
ddconfig:
|
143 |
+
double_z: true
|
144 |
+
in_channels: 80
|
145 |
+
out_ch: 80
|
146 |
+
...
|
147 |
+
```
|
148 |
+
|
149 |
+
2. setting `Lumina-T2Music` and `Vocoder` checkpoint path and run demo
|
150 |
+
|
151 |
+
Please replace `/path/to/ckpt` with the actual downloaded path.
|
152 |
+
|
153 |
+
```bash
|
154 |
+
# `/path/to/ckpt` should be a directory containing `music_generation`, `maa2`, and `bigvnat`.
|
155 |
+
|
156 |
+
# default
|
157 |
+
python -u demo_music.py \
|
158 |
+
--ckpt "/path/to/ckpt/music_generation" \
|
159 |
+
--vocoder_ckpt "/path/to/ckpt/bigvnat" \
|
160 |
+
--config_path "configs/lumina-text2music.yaml" \
|
161 |
+
--sample_rate 16000
|
162 |
+
```
|
163 |
+
|
164 |
+
## Disclaimer
|
165 |
+
|
166 |
+
Any organization or individual is prohibited from using any technology mentioned in this paper to generate someone's speech without his/her consent, including but not limited to government leaders, political figures, and celebrities. If you do not comply with this item, you could be in violation of copyright laws.
|