language: en
tags:
- audio
- music-generation
- sample-generation
- piano
- fine-tuning
- stable-audio
datasets:
- custom
model_name: Royal Cities Vocal Textures (SAO Finetune)
base_model: stabilityai/stable-audio-open-1.0
license: other
license_name: stabilityai-community-license
license_link: https://stability.ai/license
Royal Cities Vocal Textures (SAO Finetune)
Introduction
- Male Vocals
- Female Vocals
- Ensemble Vocals (Combination of Male and Female)
Model Features
- Multiple Types of Stem Generation: Outputs three types of voicings with a focus on Chord Progressions only,
- Tonal Versatility: Generates stems in any key across the 12-tone chromatic scale, in both major and minor scales.
- Simplified Scale Notation: Scales are written using sharps only in the following format:
Minor Scales A minor, A# minor, B minor, C minor, C# minor, D minor, D# minor, E minor, F minor, F# minor, G minor, G# minor Major Scales A major, A# major, B major, C major, C# major, D major, D# major, E major, F major, F# major, G major, G# major
For more details on the VSTs and gear used in sample creation, refer to the Gear section below.
Training Methodology
This model was designed in conjunction with a future public dataset with a focus on only generating vocal chord progressions across the 12-tone chromatic scale in either Female, Male or Ensemble outputs.
Vocals / Choirs have very long attacks so it was trained with metadata simply classifying each sample as "Chord Progression" along with the necessary key, bpm and bar information. More details can be found in the dataset section.
Usage Guide
Supported GitHub Interfaces
This model works in both the RC Stable Audio Gradio and the original Stable Audio GitHub.
To use the model simply place both the .ckpt file and the config .json inside their own sub-folder within the "models" folder and launch the gradio.
VST Support
This model has direct VST compatibility in the Audialab Engine.
Prompt Structure
To ensure the best results, use the following format for your prompts:
[Vocal Type], Chord Progression, [Key], [BPM], [Bar Count]
Vocal Type Prompts
["Male Vocal Texture"]
["Female Vocal Texture"]
["Ensemble Vocal Texture"]
Examples
Male vocal prompt with model output.
"Male Vocal Texture, chord progression, D# minor, 128BPM, 8 bars,"
Female vocal prompt with model output.
"Female Vocal Texture, chord progression, B major, 120BPM, 8 bars,"
Ensemble Vocal Prompt with model output.
"Ensemble Vocal Texture, chord progression, C major, 140BPM, 8 bars,"
BPMs/Bars
The BPMs ranged from as low as 100BPM up to 150BPM. The main denominations are 100BPM, 110BPM, 120BPM, 128BPM, 130BPM, 140BPM, 150BPM.
There are 2 bar settings: 4 bars and 8 bars.
Dataset Breakdown
Overview
- Total .wav files: 5040
- Total duration: 980.64 minutes
- Average Length: 11.67 seconds
- Total Size: 14.50 GB
- Sample Rate: 44100 Hz
This dataset was meticulously designed to maintain a perfect balance between major and minor chord progressions, ensuring high accuracy in key generation.
By including an equal number of samples for all 12 tones in both major and minor scales (across 4 and 8 bars), the model achieves precise results. It reliably produces chord progressions in either major or related minor scales, effectively handling the nuances where major and minor scales share the same notes.
For example, in music theory, the F major scale and D minor scale share the same notes, but the tonal center of the chord progression drives the classification. After training, the model accurately generates the correct scale based solely on the prompt.
"Male Vocal Texture, chord progression, D minor, 150BPM, 8 bars,"
"Male Vocal Texture, chord progression, F major, 150BPM, 8 bars,"
Dataset Details
| Vocal Type | Major Progression (4 bars) | Major Progression (8 bars) | Minor Progression (4 bars) | Minor Progression (8 bars) |
|---|---|---|---|---|
| Male | 420 | 420 | 420 | 420 |
| Female | 420 | 420 | 420 | 420 |
| Ensemble | 420 | 420 | 420 | 420 |
The dataset will be released in the future for researchers and music producers who want to get started in finetuning.
Technical Specifications
- Platform: Runpod
- Monitoring Tool: Weights and Biases
- Epoch: 23
- Steps: 1800
- Learning Rate: 5e-5
- Optimizer: AdamW
- Scheduler: InverseLR
- Batch Size: 32
- Hardware: 2x NVIDIA A40 GPUs
See config file for further details.
Limitations and Biases
The Model has high accuracy when it comes to staying in key due to the balance in the dataset. The metadata however was designed in such a way that the model is mainly designed to generate chord progressions only - as opposed to the piano model which also generated melodies. This is due to Vocal choir samples often having very long attacks so often sit at the back of a mix or to fill out the frequency space.
I have noticed some light artifacting in the outputs - in particular the ensemble progressions. It almost sounds like the model is trying to add other instrumentation in. I think this may be due to the base model primarily being trained on music stems rather than vocals but I cannot say for certain - this will need to be corrected in a future model or when there is far more vocal data.
Best use case may be to add some light reverb or post-processing for best results / use in a song.
Gear Used
- DAW: FL Studio (Image-Line)
- Vocals: Multiple Choir Libraries
- EQ: PRO-Q3 (FabFilter)
- Additional Gear: Fruity Reeverb 2, Shaperbox 3 (Compressor), Soothe2
License
This model is licensed under the Stability AI Community License. It is available for non-commercial use or limited commercial use by entities with annual revenues below USD $1M. For revenues exceeding USD $1M, please refer to the LICENSE for detailed terms.