chore: replace Vevo v1 weights with Vevo2 (RMSnow/Vevo2 inference subset)

Wipe the misfiled Vevo v1 layout and upload the inference-only files from RMSnow/Vevo2: AR (posttrained), FM (fm_emilia101k_singnet7k_repa), Vocos vocoder, and content-style + prosody tokenizers. README rewritten to match. Training artifacts and the _text/pretrained variants are intentionally dropped.

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+contentstyle_modeling/posttrained/tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -1,22 +0,0 @@
-MIT License
-
-Copyright (c) 2024 OpenMMLab (Amphion)
-Mirrored by AEmotionStudio under original license terms.
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.

README.md

@@ -5,75 +5,74 @@ tags:
   - voice-conversion
   - singing-voice
   - speech-synthesis
+  - text-to-speech
   - vevo2
   - amphion
   - safetensors
   - maestraea
 pipeline_tag: audio-to-audio
-base_model: amphion/Vevo
+base_model: RMSnow/Vevo2
 ---
 
 # Vevo2 Models (Mæstræa Mirror)
 
-**Singing Voice Synthesis, Conversion & Editing**
+**Speech & Singing Voice Synthesis · Conversion · Editing · Style Transfer · Melody Control**
 
-[Original Model](https://huggingface.co/amphion/Vevo) by [OpenMMLab / Amphion](https://github.com/open-mmlab/Amphion) · MIT License
+[Original Weights](https://huggingface.co/RMSnow/Vevo2) by [RMSnow](https://huggingface.co/RMSnow) · [Source Code](https://github.com/open-mmlab/Amphion/tree/main/models/svc/vevo2) by [OpenMMLab / Amphion](https://github.com/open-mmlab/Amphion) · MIT License
 
-> This is a mirror of the Vevo2 model weights for use with [Mæstræa AI Workstation](https://github.com/AEmotionStudio/Maestraea). All credits go to the original authors.
+> Mirror of the inference-only files from `RMSnow/Vevo2`, packaged for use with the [Mæstræa AI Workstation](https://github.com/AEmotionStudio/Maestraea). Training artifacts (`optimizer.pt`, `scheduler.pt`, `rng_state_*.pth`, `trainer_state.json`, `training_args.bin`, the `_text` FM variant, and the `pretrained` AR baseline) are dropped to keep the download lean. All credit for the model itself goes to the upstream authors.
 
 ## What's in This Repo
 
 | Path | Description | Size |
 |------|-------------|------|
-| `contentstyle_modeling/PhoneToVq8192/model.safetensors` | AR model (Qwen2.5-0.5B, ~500M params) | ~2.5 GB |
-| `contentstyle_modeling/Vq32ToVq8192/model.safetensors` | Style transfer model | ~1.5 GB |
-| `acoustic_modeling/Vq8192ToMels/model.safetensors` | Flow matching model (~350M params) | ~1.4 GB |
-| `acoustic_modeling/Vocoder/model*.safetensors` | Vocos vocoder (~250M params) | ~1 GB |
-| `tokenizer/vq32/` | HuBERT tokenizer (pickle + config) | ~1.3 GB |
-| `tokenizer/vq8192/model.safetensors` | VQ8192 tokenizer | ~200 MB |
+| `contentstyle_modeling/posttrained/model.safetensors` | AR transformer (Qwen2.5-0.5B post-trained) | ~970 MB |
+| `acoustic_modeling/fm_emilia101k_singnet7k_repa/model.safetensors` | Flow-Matching transformer (~350M params) | ~1.4 GB |
+| `acoustic_modeling/fm_emilia101k_singnet7k_repa/whisper_stats.pt` | Per-channel mean/std for normed Whisper features | ~12 KB |
+| `vocoder/model*.safetensors` | Vocos vocoder (~250M, sharded) | ~1.2 GB |
+| `tokenizer/contentstyle_fvq16384_12.5hz/model.safetensors` | Content-style tokenizer (FVQ16384 @ 12.5 Hz) | ~234 MB |
+| `tokenizer/prosody_fvq512_6.25hz/model.safetensors` | Prosody tokenizer (FVQ512 @ 6.25 Hz) | ~261 MB |
+| `contentstyle_modeling/posttrained/{tokenizer.json, vocab.json, …}` | AR text tokenizer + configs | ~22 MB |
+| `*/config.json`, `amphion_config.json`, etc. | Per-component configs | small |
 
-**Total: ~8 GB**
+**Total: ~4 GB**
+
+> Whisper-medium (~1.5 GB, used by the content-style tokenizer at inference time) is **not** mirrored here — `openai-whisper` will pull it to `~/.cache/whisper` on first run.
 
 ## What Vevo2 Does
 
-Vevo2 is a state-of-the-art voice conversion and singing voice synthesis system from the Amphion toolkit. It supports:
+Vevo2 is a unified, controllable speech-and-singing voice generation system from the Amphion toolkit. The Mæstræa panel exposes six task tabs that all route to the same backend pipeline:
 
-- **Voice Conversion** — Transform vocals to a target voice/timbre
-- **Singing Voice Synthesis** — Generate singing from text + melody
-- **Speech Editing** — Modify speech content while preserving speaker identity
-- **Zero-Shot TTS** — Generate speech in any voice from a short reference
+- **Convert** — voice/timbre conversion, FM-only (fastest)
+- **TTS** — zero-shot text-to-speech / text-to-singing from a short reference clip
+- **Edit** — rewrite words while preserving voice, melody, prosody, and style
+- **Style** — singing style transfer (e.g. breathy → vibrato, pop → opera) preserving voice + melody
+- **Melody** — sing target lyrics over a humming, whistled, or instrumental melody
+- **SVC** — full singing voice conversion via the AR + FM pipeline (deeper than Convert)
 
 ### Architecture
 
-- **AR Model** (Qwen2.5-0.5B) — Autoregressive content-style modeling
-- **FM Model** (~350M) — Flow matching for acoustic generation
-- **Vocos Vocoder** (~250M) — High-quality waveform synthesis
-- **Total: ~1.1B parameters**
+- **AR Model** (Qwen2.5-0.5B post-trained) — autoregressive content-style modeling
+- **Flow-Matching Transformer** (~350M) — acoustic generation
+- **Vocos Vocoder** (~250M) — high-quality 24 kHz waveform synthesis
+- **Content-style + Prosody Tokenizers** — FVQ codecs over Whisper / chromagram features
 
 ### VRAM Requirements
 
-| Reference Length | VRAM |
-|-----------------|------|
-| 15s | ~8 GB |
-| 30s | ~10 GB |
-| 45s | ~12 GB |
+| Reference Length | VRAM (GPU, FP16) |
+|------------------|------------------|
+| 15 s             | ~6 GB            |
+| 30 s             | ~10 GB           |
+| 45 s             | ~12 GB           |
 
-Recommended: Keep reference audio to 15–45 seconds.
+Recommendation: keep the timbre reference between 15–45 s. Longer references buy more identity fidelity at a real VRAM cost.
 
-## Usage with Mæstræa
+## Usage in Mæstræa
 
-These models are automatically downloaded by the Mæstræa AI Workstation backend. Place in:
+The Mæstræa runner clones [open-mmlab/Amphion](https://github.com/open-mmlab/Amphion) to `~/.maestraea/libs/amphion/` on first model load and imports the pipeline from `models.svc.vevo2.vevo2_utils.Vevo2InferencePipeline`. The download manager pulls these weights to `~/.maestraea/models/vevo2/` and the runner resolves checkpoints from that directory at the paths shown in the table above.
 
-```
-~/.maestraea/models/vevo2/
-```
+For the standalone Amphion path, see the [upstream Vevo2 README](https://github.com/open-mmlab/Amphion/blob/main/models/svc/vevo2/README.md).
 
 ## License
 
-MIT — same as the original Amphion/Vevo2 release.
-
-## Credits
-
-- **Model**: [Amphion Vevo2](https://github.com/open-mmlab/Amphion/tree/main/models/vc/vevo2)
-- **Paper**: See [Amphion repository](https://github.com/open-mmlab/Amphion) for citation
-- **Mirror by**: [AEmotionStudio](https://huggingface.co/AEmotionStudio)
+MIT, inherited from upstream. Commercial use of generated audio is permitted. Don't clone someone's voice without their consent.

acoustic_modeling/fm_emilia101k_singnet7k_repa/config.json

@@ -0,0 +1,62 @@
+{
+    "model_type": "FlowMatchingTransformer",
+    "preprocess": {
+        "hop_size": 480,
+        "sample_rate": 24000,
+        "n_fft": 1920,
+        "num_mels": 128,
+        "win_size": 1920,
+        "fmin": 0,
+        "fmax": 12000,
+        "mel_var": 8.14,
+        "mel_mean": -4.92,
+        "f0_fmin": 50.0,
+        "f0_fmax": 1100.0,
+        "load_phone": false,
+        "load_chromagram": true,
+        "load_semantic_features": true,
+    },
+    "model": {
+        "flow_matching_transformer": {
+            "mel_dim": 128,
+            "hidden_size": 1024,
+            "num_layers": 16,
+            "num_heads": 16,
+            "cfg_scale": 0.2,
+            "use_cond_code": true, // false means Hidden features
+            // "cond_dim": 1024, // HuBERT features dimension
+            "cond_codebook_size": 16384, // VQ Codebook Size
+            "cond_scale_factor": 4, // 1 means not use ReTrans. 4 means 12.5Hz * 4 = 50Hz. This should be aligned with the frame rate with Mels
+            "sigma": 1e-5,
+            "time_scheduler": "cos",
+            "whisper_perturb": false,
+            "repa": {
+                "layer_index": 5, // Use the Wav2Vec2Bert features to align. 5 means the 6th layer.
+                "output_dim": 1024, // The dimension of the Wav2Vec2Bert features.
+                "loss_type": "cos", // "cos" or "l1". By default is "l1"
+            },
+        },
+        "cond_sample_rate": 16000, // whisper: 16000
+        "coco": {
+            "coco_type": "content_style", // content, style, or content_style
+            "downsample_rate": 4, // The original frame rate is 50 Hz, downsample to 12.5 Hz
+            "codebook_size": 16384,
+            "hidden_size": 1024, // Representations Dim
+            "codebook_dim": 8,
+            "encoder": {
+                "vocos_dim": 384,
+                "vocos_intermediate_dim": 2048,
+                "vocos_num_layers": 12,
+            },
+            "decoder": {
+                "vocos_dim": 384,
+                "vocos_intermediate_dim": 2048,
+                "vocos_num_layers": 12,
+            },
+            "use_normed_whisper": true,
+            "whisper_stats_path": "models/svc/vevosing/config/whisper_stats.pt",
+            "whisper_dim": 1024,
+            "chromagram_dim": 24,
+        },
+    },
+}

acoustic_modeling/{Vq8192ToMels → fm_emilia101k_singnet7k_repa}/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:750f013ac1485855bfbe992ffec8ed5f625e6070b8bc52ce71a6f9ae0229c5c4
-size 1350803704
+oid sha256:ef3733b3f92cf8f38e32f6d161f58751247b199cffc15cc1562e76c1289a7186
+size 1451496488

tokenizer/vq32/hubert_large_l18_mean_std.npz → acoustic_modeling/fm_emilia101k_singnet7k_repa/whisper_stats.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:84115dd2ad3a44526b8e75e173956e41805ba540704acd735f302889a83067b4
-size 8692
+oid sha256:6117052b3e23e9075a79cc208ecc24328ae2f71a7dd4c9793db7c90a88b4a519
+size 9215

config.json

@@ -1,5 +0,0 @@
-{
-    "download_tracking": {
-        "query_files": ["config.json", "*.safetensors"]
-    }
-}

contentstyle_modeling/Vq32ToVq8192/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:83e4984695487d6feeba9664c95375f9487e3326059b7239db1fe220e1d49b1d
-size 1925991368

contentstyle_modeling/posttrained/amphion_config.json

@@ -0,0 +1,73 @@
+{
+    "preprocess": {
+        "hop_size": 480,
+        "sample_rate": 24000,
+        "n_fft": 1920,
+        "num_mels": 128,
+        "win_size": 1920,
+        "fmin": 0,
+        "fmax": 12000,
+        "mel_var": 8.14,
+        "mel_mean": -4.92,
+        "f0_fmin": 50.0,
+        "f0_fmax": 1100.0,
+        "wav_code_frame_rate": 18.75, // Vevo2: 12.5 (Content-Style Code) + 6.25 (Prosody Code) = 18.75
+        "min_dur": 1,
+        "max_dur": 30,
+        "drop_prosody_id_prob": -1, // Dropping prosody ids means the Text-to-CS, while not dropping means the Text+Note-to-CS,
+        "pad_token_id": 151643, // <|endoftext|> for Qwen2.5-0.5B-Instruct,
+        "eos_token": "<|im_end|>",
+        "eos_token_id": 151645, // <|im_end|> for Qwen2.5-0.5B-Instruct,
+        // "tokenizer_path": "/mnt/data4/zhangxueyao/SpeechGenerationYC_ckpts/ckpts/vevo2/pretrained/Qwen2.5-0.5B-Instruct-add_prosody_contentstyle"
+    },
+    "model": {
+        // "pretrained_model_path": "/mnt/data4/zhangxueyao/SpeechGenerationYC_ckpts/ckpts/vevo2/pretrained/Qwen2.5-0.5B-Instruct-add_prosody_contentstyle", // Qwen2.5 Model
+        // "rl_init_model_path": "/mnt/data4/zhangxueyao/SpeechGenerationYC_ckpts/ckpts/vevo2/llm_dpo/dpo_qwen0.5B_intp2_highsim_3e-5/checkpoint_backup/epoch-0023_step-0027000_loss-0.000961", // DPO Model
+        "use_intelligibility_reward": true,
+        "use_chromagram_reward": true,
+        "use_target_length_reward": true,
+        "reward_combination_strategy": "advantage_first", // "reward_first" or "advantage_first"
+        "coco_style": {
+            "coco_type": "style", // content, style, or content_style
+            "downsample_rate": 8, // The original frame rate is 50 Hz, downsample to 6.25 Hz
+            "codebook_size": 512,
+            "hidden_size": 1024, // Representations Dim
+            "codebook_dim": 8,
+            "encoder": {
+                "vocos_dim": 384,
+                "vocos_intermediate_dim": 2048,
+                "vocos_num_layers": 12,
+            },
+            "decoder": {
+                "vocos_dim": 384,
+                "vocos_intermediate_dim": 2048,
+                "vocos_num_layers": 12,
+            },
+            "use_normed_whisper": true,
+            "whisper_stats_path": "models/svc/vevosing/config/whisper_stats.pt",
+            "whisper_dim": 1024,
+            "chromagram_dim": 24,
+        },
+        "coco_content_style": {
+            "coco_type": "content_style", // content, style, or content_style
+            "downsample_rate": 4, // The original frame rate is 50 Hz, downsample to 12.5 Hz
+            "codebook_size": 16384,
+            "hidden_size": 1024, // Representations Dim
+            "codebook_dim": 8,
+            "encoder": {
+                "vocos_dim": 384,
+                "vocos_intermediate_dim": 2048,
+                "vocos_num_layers": 12,
+            },
+            "decoder": {
+                "vocos_dim": 384,
+                "vocos_intermediate_dim": 2048,
+                "vocos_num_layers": 12,
+            },
+            "use_normed_whisper": true,
+            "whisper_stats_path": "models/svc/vevosing/config/whisper_stats.pt",
+            "whisper_dim": 1024,
+            "chromagram_dim": 24,
+        },
+    },
+}

contentstyle_modeling/posttrained/config.json

@@ -0,0 +1,29 @@
+{
+  "_name_or_path": "epoch-0023_step-0027000_loss-0.000961",
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 896,
+  "initializer_range": 0.02,
+  "intermediate_size": 4864,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 21,
+  "model_type": "qwen2",
+  "num_attention_heads": 14,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.47.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 168565
+}

contentstyle_modeling/posttrained/generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.1,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "4.47.1"
+}

contentstyle_modeling/{PhoneToVq8192 → posttrained}/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:033248ccc7879456474134b6c77ea0d82a0e78ba7a09ba1ae7e1d99943d4eeaf
-size 2973225456
+oid sha256:b056f8b4f0616dd5eb8598a0cc3fe5a4bd9dcdffc2dbe344156a16691496cf16
+size 1017897016

tokenizer/vq32/hubert_large_l18_c32.pkl → contentstyle_modeling/posttrained/tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7d13c6f6e34a1ef43cd7d67cef9814df0ca99e9c5b4610c0a0115bb2b1d76045
-size 235174528
+oid sha256:2b4afa3f042a3dd5ddf573c035c68f88810d01d714fe3341054e9688c0eebef7
+size 13891825

tokenizer/{vq8192 → contentstyle_fvq16384_12.5hz}/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:660bd48b023e637a786a9c78f404cb979ef9a5d1c93ce24837e0bec942352c4d
-size 177183712
+oid sha256:bebdcea39e2d0134dbcce193aed0e7b0d393a866346317cd196389e40e9151b0
+size 244781696

tokenizer/prosody_fvq512_6.25hz/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:754ba5928dfe4dd3c4b6e91ce396e2a50cff52937053c0e3e4ac3a854a4b8ae4
+size 273642504

tokenizer/vq32/hubert_large_l18_c32.yaml

@@ -1,30 +0,0 @@
-bias: true
-code_dim: 1024
-codebook_num: 1
-codebook_size: 32
-dec_block_dilations:
-- 1
-- 1
-dec_block_kernel_size: 3
-dec_kernel_size: 3
-dec_ratios:
-- 1
-- 1
-dec_strides:
-- 1
-- 1
-decode_channels: 1024
-enc_block_dilations:
-- 1
-- 1
-enc_block_kernel_size: 3
-enc_kernel_size: 3
-enc_ratios:
-- 1
-- 1
-enc_strides:
-- 1
-- 1
-encode_channels: 1024
-input_channels: 1024
-output_channels: 1024

vocoder/config.json

@@ -0,0 +1,84 @@
+{
+    "model_type": "Vocoder",
+    "preprocess": {
+        "hop_size": 480,
+        "sample_rate": 24000,
+        "max_length": 36000,
+        "n_fft": 1920,
+        "num_mels": 128,
+        "win_size": 1920,
+        "fmin": 0,
+        "fmax": 12000,
+        "mel_var": 8.14,
+        "mel_mean": -4.92,
+        "load_phone": false,
+        "load_chromagram": false
+    },
+    "model": {
+        "vocos": {
+            "input_channels": 128,
+            "dim": 1024,
+            "intermediate_dim": 4096,
+            "num_layers": 30,
+            "n_fft": 1920,
+            "hop_size": 480,
+            "padding": "same"
+        },
+        "period_gan": {
+            "max_downsample_channels": 1024,
+            "channels": 64,
+            "channel_increasing_factor": 2
+        },
+        "spec_gan": {
+            "stft_params": {
+                "fft_sizes": [
+                    128,
+                    256,
+                    512,
+                    1024,
+                    2048
+                ],
+                "hop_sizes": [
+                    32,
+                    64,
+                    128,
+                    256,
+                    512
+                ],
+                "win_lengths": [
+                    128,
+                    256,
+                    512,
+                    1024,
+                    2048
+                ],
+                "window": "hann_window"
+            },
+            "in_channels": 1,
+            "out_channels": 1,
+            "channels": 64,
+            "kernel_sizes": [
+                5,
+                3
+            ],
+            "max_downsample_channels": 1024,
+            "down_scales": [
+                2,
+                2,
+                2
+            ],
+            "use_weight_norm": true,
+            "use_complex": false
+        }
+    },
+    "loss": {
+        "mel_loss": {
+            "sample_rate": 24000
+        },
+        "disc_loss_weight": 1.0,
+        "mel_loss_weight": 10.0,
+        "adv_loss_weight": 2.0,
+        "fm_loss_weight": 2.0,
+        "spec_fm_loss_weight": 1.0
+    },
+}

{acoustic_modeling/Vocoder → vocoder}/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7670d180569fdae986fbf94ede07d6fc4ce8bfcf406cd1aadbe33e08581b5f6a
+oid sha256:5b5d1a46b19351c9a71bd8a5a59dd16be0be2ddefe70d3a0b4915d9a425e56d3
 size 1020206416

{acoustic_modeling/Vocoder → vocoder}/model_1.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:164c3db2a9f41a994ac1c7a7a57aa15aadb97dd3f94a4af9bef95d2f7edfad34
+oid sha256:850799d78699134b969056183fc9d490c51f8d8154d0ed00fae3e738b6b30af6
 size 69768280

{acoustic_modeling/Vocoder → vocoder}/model_2.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:03785c5a3ed7181b22fae11a266d1438cb59333726dfbe8429f00d5543dd8c95
+oid sha256:56130fd13d5fbe828e56d61edb0049d35700db0472a866b8167d1d217d2687f8
 size 180693296