VoiceTuner: Self-Supervised Pre-training and Efficient Fine-tuning For Voice Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 13:06:07

Abstract: Voice large language models (LLMs) cast voice synthesis as a language modeling task in a discrete space, and have demonstrated significant progress to date. Despite the recent success, the current development of voice LLMs in low-resource applications is hampered by data scarcity and high computational cost. In this work, we propose VoiceTuner, with a self-supervised pre-training and efficient fine-tuning approach for low-resource voice generation. Specifically, 1) to mitigate data scarcity, we leverage large-scale unlabeled dataset and pre-train VoiceTuner-SSL without pre-defined applications, which can be fine-tuned in downstream tasks; 2) to further reduce the high training cost in complete fine-tuning, we introduce a multiscale transformer adapter to effectively update only around 1% parameters as a plug-and-play module. Experimental results demonstrate that VoiceTuner-SSL presents strong acoustic continuations, and VoiceTuner achieves state-of-the-art results in rich-resource TTS evaluation compared with competitive baseline models. Low-resource (1h, 10h, 30h) downstream applications including zero-shot TTS, instruction TTS, and singing voice synthesis present VoiceTuner's superior audio quality and style similarity with reduced data requirement and computational cost. Audio samples are available at https://VoiceTuner.github.io

Main paper: 
 related works 2.1 generative voice models: 
 Text-guided voice synthesis (text-to-speech and singing voice synthesis) typically converts input text into mel-spectrogram (e.g., Tacotron [41] , FastSpeech [34] ), which is then transformed to waveform using a separately trained vocoder [14, 21] , or directly generate waveform from text (e.g., EATS [8] and VITS [20] ). In zero-shot scenarios, when the distributions of style prompts deviate from the training data, the quality of the synthesized voice often suffers degradation due to distribution mismatches: Meta-StyleSpeech [29] generally adopts a speech encoding network for multi-speaker synthesis. GenerSpeech [16] leverages multi-level style adaptors for the global and local stylization of the custom utterance. YourTTS [4] is built upon VITS with several novel modifications for zero-shot multi-speaker and multilingual training.Self-supervised learning (SSL) [2, 11] has been shown to achieve remarkable advances in recent years, opening up a wide array of applications that leverage their power by adapting models. AudioLDM 2 [27] leverages AudioMAE [13] and performs self-supervised audio generation learning with a latent diffusion model conditioned on audio tokens. UniAudio [43] trains on different generative tasks to obtain prior knowledge in the inter-relationship between audio and other modalities and support new audio generation tasks after simple fine-tuning. Spear-TTS [50] is pre-trained on a BERTlike [7] denoising pretext task, where the model is provided with a corrupted version of an original semantic token sequence with the goal to produce the corresponding uncorrupted token sequence. Differently, we pre-train LLMs (namely VoiceTuner-SSL) in a nexttoken prediction task without supervision, which has not been investigated in voice synthesis task.Efficient fine-tuning aims to reduce data and device requirements in downstream generation tasks. SpeechFlow [26] achieve better performance utilizing low-rank adaptation (LoRA), which adds the linear input projection to each self-attention layer. Au-dioBox [39] include two-stage full fine-tuning to improve model fidelity and quality where all parameters are optimized together. In this work, we introduce a multiscale transformer adapter for parameter-efficient adaptation, which updates only around 1% of the parameters on top as a lightweight plug-and-play module. 
method: 
 In this section, we first overview the motivation, and introduce generative self-supervised pre-training with follow-up fine-tuning approach with discrete voice representation. Next, we propose a lightweight, plug-and-play adapter for parameter-efficient finetuning. In the following, we introduce the scalable global and local architecture in Section 3.3.Current voice large language models (LLMs) [3, 19, 40, 50] cast voice synthesis as a language modeling task in a discrete representation space. However, these voice LLMs are trained from scratch using web-scale data, replicating this success in low-resource scenarios is hampered by two major challenges: 1) data scarcity: large-scale training data are lacking in low-resource scenarios; and 2) high computational cost: training voice LLMs from scratch are computationally intensive and time-consuming, and the inefficient attention mechanism in transformer further challenges model in modeling long codec sequence.To alleviate data scarcity, we pre-train the next-token prediction model (VoiceTuner-SSL) in the large-scale unlabeled dataset, which can be fine-tuned in downstream generation tasks with reduced data and device requirements. To address the high computational cost in low-resource scenarios, we propose VoiceTuner to respectively decrease data requirements and learnable parameters in fine-tuning with multiscale transformer adapter.Audio codec models such as SoundStream [45] and Encodec [6] have recently shown that encoder-decoder architecture excels at learning acoustic information in a self-supervised manner, where the representation can be used in a variety of generative tasks.The acoustic codec model typically consists of an audio encoder, a residual vector-quantizer (RVQ), and an audio decoder: 1) The audio encoder 𝐸 consists of several convolutional blocks with a total downsampling rate of 320 and generates continuous representations at every 20-ms frame in 16kHz. 2) The residual vector-quantizer 𝑄 produces discrete representations 𝑎 𝑞 with a codebook size of 𝐾 2 , using a vector quantization layer [37] . 3) The audio decoder 𝐺 reconstructs the signal ŷ, from the compressed latent representation 𝑎 𝑞 . In the end, a speech utterance 𝑦 is represented as acoustic tokens with [𝑎 1 , 𝑎 2 , . . . , 𝑎 𝑇 ] , 𝑎 𝑖 ∈ {0, 1, . . . , 𝐾 2 -1}, ∀1 ≤ 𝑖 ≤ 𝑇 , where 𝑇 is the number of frames.With powerful models, large language models have recently exhibited high-quality samples in natural language processing. To make audio modeling more tractable, recent studies propose to represent audio signals as multiple streams of discrete tokens representing the same signal and flatten these codes [1, 22] . It comes at the high computational cost of modeling extremely long sequences, because of the quadratic cost of self-attention and large feedforward networks per-position.Our model (denoted as 𝜃 𝐴𝑅 ) predicts long sequences with endto-end differentiable multiscale transformers or state space models similar to Yang et al. [43] , Yu et al. [44] . This enables sub-quadratic calculation, unlocking better performance at reduced cost for both training and generation. As illustrated in Figure 1(c o = 𝜃 local 𝐴𝑅 (L 0:K-1 i + G 1:K o ).Our model presents the improvements from scaling attention layers' depth and width without the requirement of scattered modelspecific methodologies. As expected, scaling the model size (160M (base), 420M (medium), and 1.1B (large) parameter) results in better scores. We refer the reader to Section 8.1 for our findings.Most voice LLMs rely on web-scale training data and cast voice synthesis as a language modeling task, while the data shortage hampers its application in low-resource scenarios. To alleviate it, we leverage unlabeled corpus and pre-train LLMs (namely VoiceTuner-SSL) in a next-token prediction task without supervision, where we hypothesize that a generative model without pre-defined application can be applied to different downstream tasks, reducing data requirement in low-resource application.VoiceTuner-SSL is pre-trained on arbitrary voice, which contains many speakers with various accents, diverse demographics, and heterogeneous recording conditions. Next, we fine-tune VoiceTuner-SSL to align speech and text modalities utilizing supervised data in downstream voice generation applications, where we find that the self-supervised pre-training stage offers a distinct gain in both rich and low-resource scenarios. We expect our VoiceTuner-SSL to keep the speaker identity, prosody, and recording conditions of the prompt and produce new content. We refer the reader to Section 5 for our findings. After pre-training VoiceTuner-SSL on unlabeled speech corpus, we fine-tune the model in downstream tasks with supervised data. In this section, we introduce an efficient plug-and-play module, i.e., a multiscale transformer adapter, where we have separated fine-tuning strategies respectively for global and local transformers to update only around 1% parameters. Specifically,• In a local transformer, we include low-rank adaptation (LoRA) [12] in the linear input projection of each layer in attention blocks, where only the LoRA parameters are optimized. • In a global transformer, a set of learnable prompts with gates [49] are added to the input, which learn to adaptively inject new instructions (conditions) into the pre-trained model and avoid disturbing speech tokens at the beginning of training.Suppose we have condition representation (i.e., task-specific prompts) 𝐼 ∈ R 𝐾 ×𝐶 with length 𝐾 and feature dimension 𝐶. For instruction TTS, we use pre-trained Flan-T5-XL [33] and freeze the weights to derive condition representation; For zero-shot TTS and SVS, we use the token embedding matrix to obtain the representation of acoustic and pitch tokens from speaker and MIDI prompt, which are then pad to a fix length 𝐾 = 150. We initialize learnable adaption prompt {𝑃 𝑙 } 𝐿 𝑙=1 for 𝐿 layers, where we have each layer's prompt 𝑃 𝑙 ∈ R 𝐾 ×𝐶 and speech tokens 𝑇 𝑙 ∈ R 𝑀 ×𝐶 . Then, the adaption prompt is conducted an elementwise addition with condition representation:𝑃 𝑙 = [𝑃 𝑙 + 𝐼 ] ∈ R 𝐾 ×𝐶 .Suppose the model is processing with the speech tokens 𝑇 𝑙 and condition 𝑃 𝑙 , The attention score related to learnable prompt is calculated as 𝑆 𝑙 , which represents how much information the learnable prompt contributes. Initialized by zero, 𝑔 𝑙 can first eliminate the influence of under-fitted prompts and then increase its magnitude to provide more instruction semantics.To conclude, the adaptation enjoys efficient training efficiency with only around 1% learnable parameters. As a lightweight plugand-play module, this enables us to fine-tune voice LLMs on cheap devices.We train a unit-based neural vocoder from scratch for the acoustic unit to waveform generation. Inspired by BigVGAN [24] , the synthesizer includes the generator and multi-resolution discriminator (MRD). The generator is built from a set of look-up tables (LUT) that embed the discrete representation and a series of blocks composed of transposed convolution and a residual block with dilated layers. The transposed convolutions upsample the encoded representation to match the input sample rate. Details are included in Appendix B.2.  
training and evaluation 4.1 dataset: 
 For self-supervised pre-training, we utilize large-scale datasets with Librilight [18] and WenetSpeech [47] , where we have ∼160K hours of 16 kHz audio that greatly increases the domain coverage.We fine-tuning VoiceTuner-SSL to align speech and text modalities utilizing TTS data such as LibriTTS [46] , VCTK [38] and Prompt-Speech [9] , resulting in rich-resource VoiceTuner. For text sequence, we tokenize it into the phoneme sequence with an open-source grapheme-to-phoneme conversion tool [36] . To evaluate Voice-Tuner in low-resource scenarios, we construct paired data (1h, 10h, 30h) with three application tasks: instruction-guided TTS, zeroshot TTS, and singing voice synthesis, respectively generalizing to unseen instruction, speaker, and modality. We have attached detailed information on data configuration in Figure 1 .Speech intelligibility. We report word error rate (WER) to evaluate the intelligibility of speech by transcribing it using a whisper [32] ASR system following [40] .Style similarity. SIM assesses the coherence of the generated speech in relation to the speaker's characteristics, and we employ the speaker verification model WavLM-TDNN [5] to evaluate the speaker similarity. F0 Frame Error (FFE) measures the prosody similarity of synthesized and reference audio.For pitch, speaking speed, and volume attributes accuracy, considering that the values of generated singing may slightly deviate from the boundaries used for categorization, we adopt a soft-margin mechanism for accuracy calculation. Specifically, we take the accuracy of data falling within the correct range as 100, and calculate the accuracy with 100 * exp (-𝑘𝜖) for data outside the correct range, where 𝜖 is the error between the data value and the boundary, and 𝑘 is a hyper-parameter controlling the decay rate of accuracy at the margins, with larger 𝑘 corresponding to faster decay. We illustrate the accuracy curves in Figure 5 in Appendix C.1.Subjective evaluation. We also conduct a crowd-sourced human evaluation via Amazon Mechanical Turk, which is reported with 95% confidence intervals (CI), and analyze two aspects: style similarity (speaker, emotion, and prosody) and audio quality (clarity, high-frequency), respectively scoring SMOS and MOS. Our subjective evaluation tests are crowd-sourced and conducted by 20 native speakers via Amazon Mechanical Turk on a 1-5 Likert scale.The MOS (mean opinion score) tests explicitly instruct the raters to "(focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion, and prosody).)". For style similarity evaluation, we explicitly instruct the raters to "(focus on the similarity of the style (timbre, emotion, and prosody) to the reference, and ignore the differences of content, grammar, or audio quality.)". More information has been attached in Appendix C.For acoustic tokens, we train the SoundStream model with 12 quantization levels, each with a codebook of size 1024 and the same downsampling rate of 320. We take three quantization levels as the acoustic tokens, representing each frame as a flat sequence of tokens from the first, second, and third quantization layers. We trained three sets of VoiceTuner, with 160M (base), 459M (medium), and 1.1B (large) parameters. As for the unit-based vocoder, we use the modified V1 version of BigVGAN. A comprehensive table of hyperparameters is available in Appendix B. Except explicitly stated, we use our 459M (medium) model for downstream evaluation.During training, we pre-train VoiceTuner-SSL for 100K steps using 8 NVIDIA A100 GPUs with a batch size of 6000 tokens for each GPU on the publicly-available fairseq framework [30] , and fine-tune VoiceTuner for 10K steps using 1 NVIDIA A100 GPU. Adam optimizer is used with 𝛽 1 = 0.9, 𝛽 2 = 0.98, 𝜖 = 10 -9 . The unit-based vocoder is optimized with a segment size of 8192 and a learning rate of 1 × 10 -4 until 500K steps using 4 NVIDIA V100 GPUs. For sampling, we employ top-p [10] sampling with p = 0.25.We compare the generated audio samples with other systems, including 1) GT, the ground-truth audio; 2) GT (voc.), where we first convert the ground-truth audio into tokens and then convert them back to audio using BigVGAN; 3) YourTTS [4] : a zero-shot multispeaker TTS model which is built upon VITS [20] ; 4) VALL-E [40] and Spear-TTS [19] , recently proposed Speech LLMs for English zero-shot TTS. For easy comparison, the results are compiled and presented in the following sections. We expect our generative foundation model VoiceTuner-SSL to keep the speaker identity, prosody, and recording conditions of the prompt and produce new content in next-token prediction. Specifically, we generate continuations of 5 seconds for each 3-second prompt, where the prompts are obtained by cropping samples from Librispeech test-clean. In the following, we run the speaker, style, emotion, and speed classifier on the sampled continuations (excluding the prompts) and report the results. 
self-supervised pre-training results: 
 The evaluation results are presented in Table 2 , and we have the following observations: 1) VoiceTuner-SSL can preserve the speaker, style, emotion, and speaking speed in the prompt with a high recognition accuracy at a zero-shot setting, even if the model is not fine-tuned in downstream datasets; Informally, VoiceTuner-SSL is optimized in a large amount of self-supervised data, which contains many speakers with various accents and diverse demographics to improve robustness and generalization; and 2) as shown in the demo page, in a noisy environment, VoiceTuner also presents the acoustic consistency and maintain the noise conditions from the prompt. 
quantitative findings: 
 Our proposed self-supervised pre-training and follow-up fine-tuning approach are essential for the early-stage training stability and final generation capacity. To demonstrate the rich-resource performance, we fine-tune VoiceTuner-SSL to align speech and text modalities in downstream TTS or frame level TTS (FTTS) tasks, respectively taking phone or duration-expanded phone sequences as inputs.We plot the loss/accuracy curves in Figure 4 and present results in Table 3 , and have the following observations: 1) the model with pre-training converges faster and reaches lower loss bounds than the model trained from scratch; and 2) For the intelligibility of the generated speech, VoiceTuner (with pre-training) has achieved To compare VoiceTuner with several baselines in the benchmark rich-resource zero-shot TTS tasks, we train our model in the Librilight dataset and assess the audio quality and style similarity, using a small-scale test set with the examples provided on the demo page. We also score MOS and SMOS for subjective evaluation, rated from 1 to 5, and reported with 95% confidence intervals (CI). The results are compiled and presented in the following table. 1) For audio quality, VoiceTuner has achieved the highest MOS with scores of 4.03 compared with the baseline models; 2) Regarding style similarity, VoiceTuner presents the SIM of 0.85, surpassing baseline models in transferring the style of custom voices. To conclude, self-supervised pre-training on an arbitrary voice corpus offers distinct gains in downstream fine-tuning, and Voice-Tuner's direct text-to-acoustic generation avoids the cascaded error [50] 's cascaded semantic and acoustic tokens). We will include this discussion in the revised version of the paper, and hope this will resolve your concerns. 
low-resource fine-tuning results: 
 We hypothesize that a generative foundation model can be applied to different downstream tasks, reducing data requirements and computational cost, especially in low-resource scenarios. Though it would be more helpful to investigate the real low-resource languages, it would be challenging to train a low-resource language model due to 1) the lack of stable lexicon or grapheme-to-phoneme (G2P) tools, and 2) the expensive and substantial amount of manpower needed for collecting and labeling data, which are beyond our scope. As such, we choose English/Chinese as the target languages, and construct the 1/10/30 hours data to simulate the low-resource languages following previous low-resource speech systems [25, 35] .To present the capability of VoiceTuner in low-resource scenarios, we construct (1h, 10h, 30h) hours of data for three application tasks: instruction-guided TTS (ITTS), zero-shot TTS (ZS-TTS), singing voice synthesis (SVS), respectively generalizing to unseen instruction, speaker, and modality. For training efficiency, we investigate full training from scratch (Full-s), full fine-tuning from VoiceTuner-SSL (Full-p), and efficient fine-tuning with a multiscale transformer adapter (Adapter). In this section, we fine-tune VoiceTuner in a zero-shot TTS task, where we generate the speech conditioned on the acoustic tokens of the 3-second enrolled recording and the phoneme prompt, which constrain the speaker and content information respectively. The results in the zero-shot TTS task are presented in Table 6 , and we have the following observations:1) As training data is reduced in the low-resource scenario, a distinct degradation in speech quality and similarity could be witnessed. For example, VoiceTuner (Adapter) presents a distinct drop in TTS WER of 6.9 → 7.5 → 9.6 when reducing training data from 30 to 1 hours. 2) As the amount of trainable parameters is decreased to 1% in multiscale adapter, only a slight quality drop is observed. As such, effective fine-tuning with the proposed multiscale adapter enables us to fine-tune voice LLMs on limited data and cheap devices.3) It is worth mentioning that in extremely low-resource scenarios, VoiceTuner (Full-s) cannot converge when training from scratch. As expected, a generative model (namely VoiceTuner-SSL) without a pre-defined application can be applied to different downstream tasks, reducing data requirements in low-resource applications.In this section, we fine-tune VoiceTuner in the instruction-guided TTS task, where we take a prompt with both style and content descriptions as input to synthesize the corresponding speech. In this way, users are able to create speech from a prompt, resulting in style control without the requirements for acoustic knowledge or reference speech. The results are presented in Table 5 , and we have the following observations:1) With 30-hour training data, VoiceTuner (full-p) presents high perceptual quality with outperformed subjective and objective evaluation results, which shows that the model can synthesize speech in a consistent style with the intention of style prompts. 2) In extremely low resource scenarios (i.e., 1-hour data), Voice-Tuner (Adapter) presents an outperformed quality compared to full-parameter fine-tuning models (Full-p), and VoiceTuner (Full-s) even cannot converge when training from scratch. To conclude, a complete fine-tuning of large-scale voice LLMs can lose the general ability of the foundation model (e.g., gender continuations), especially in low-resource scenarios. 3) Speech volume is relatively less distinguishable than gender and speed attributes, and thus the speaking volume classifier presents an accuracy of 78.9% in ground truth data. which is lower than gender (96.6%) and speed (86.9%). In this section, we fine-tune VoiceTuner in the singing voice synthesis task, where we generate the singing voice conditioned on the acoustic tokens of the 3-second enrolled recording, F0 prompt (from MIDI), and the phoneme prompt, which constrain the speaker, pitch, and content information respectively. Detailed information on the MIDI-to-F0 converter has been included in Appendix B.1. The results are presented in Table 7 , and we have the following observations:1) In all data usage settings, VoiceTuner (Full-s) cannot converge when training from scratch. Besides, a distinct quality drop can be witnessed when decreasing data usage, which is more significant than those in zero-shot TTS or instruction TTS tasks. To sum up, singing voice synthesis resembles the prosodic style of the F0 prompt and requests a precise pitch reconstruction, and thus it can be more sensitive to data scarcity in low-resource scenarios; and 2) Regarding computational cost, though full parameter fine-tuning systems demonstrate better results in most cases, the multiscale transformer adapter has still achieved the comparable results (e.g., FFE and SIM of 0.61, 0.78 in 1-hour SVS). It indicates that the adapter enjoys high-fidelity generation with only around 1% learnable parameters, which enables us to fine-tune voice LLMs on cheap devices; 
analysis and ablation studies: 
 To verify the capabilities of VoiceTuner, we conduct ablation studies on model scalability and few-shot adaptation, and discuss key findings as follows.As illustrated in Table 8 , we report results for different model sizes, namely 160M (base), 459M (medium), and 1.1B (large) parameter To enable few-shot learning without losing the general abilities, we fine-tune VoiceTuner in 10-hour instruction TTS data, and compare the results among different adaptation methods. Illustrated in Table 9 , as a lightweight plug-and-play module, the proposed multiscale transformer adapter enjoys superior training efficiency with only around 1% parameters in contrast to full fine-tuning, demonstrates the 9.2% WER drop and outperformed attributes accuracy (gender, speed, and pitch) compared to Lora [12] . This enables us to fine-tune voice LLMs on cheap devices. 
conclusion: 
 In this work, we propose VoiceTuner with a pre-training and efficient fine-tuning approach for low-resource voice generation. To mitigate the data scarcity and high computational cost for training voice LLMs, we 1) leveraged large-scale unlabeled dataset and pre-trained VoiceTuner-SSL in a next-token prediction task, which could be fine-tuned in downstream tasks with reduced data; 2) introduced an efficient multiscale transformer adapter to fine-tune only around 1% parameters in downstream applications, further eliminating the computational cost. Experimental results demonstrated that VoiceTuner-SSL presented strong speech continuations. VoiceTuner achieves state-of-the-art results in rich-resource TTS evaluation compared with competitive baseline models. VoiceTuner exhibited superior quality and style similarity with reduced data requirement and computational cost in three low-resource (1h, 10h, 30h) voice generation tasks, including zero-shot TTS, instruction TTS, and singing voice synthesis. We envisage that our work serves as a basis for future low-resource voice synthesis studies.  
: 
 introduces a multi-scale transformer to enable sub-quadratic selfattention, unlocking better performance at a reduced cost for training and generation. A line of works [1, 3, 19] introduces the hierarchical approach that combines semantic and acoustic audio tokens to decrease supervision in model training.Despite the success achieved, the current development of voice LLMs in low-resource scenarios (training data with limited labels) is hampered by two major challenges: 1) data scarcity: most existing models rely on web-scale training data, which are lacking in low-resource scenarios; and 2) high computational cost: training voice LLMs from scratch are computationally intensive and timeconsuming, and the inefficient attention mechanism in transformer further challenges model in modeling long codec sequence.In this work, we propose VoiceTuner, with a self-supervised pretraining and efficient fine-tuning approach for low-resource voice generation. To alleviate data scarcity, we pre-train the next-token prediction model (VoiceTuner-SSL) in the large-scale unlabeled dataset, which can be fine-tuned in downstream generation tasks with reduced data and device requirements. To further reduce computational cost and avoid losing the general abilities of VoiceTuner-SSL, we introduce a multiscale adapter with separated fine-tuning strategies for global and local transformers, effectively fine-tuning only around 1% parameters in downstream applications.VoiceTuner is pre-trained on ∼160K hours of unlabeled voice data without supervision, followed by rich or low resource (1h, 10h, and 30h) adaptation in downstream applications including zeroshot TTS, singing voice synthesis, and instruction TTS, respectively generalizing to unseen speaker, modality, and instruction. Experimental results demonstrate that VoiceTuner-SSL keeps acoustic continuations, maintaining speaker identity, emotion, and speaking speed from prompts. VoiceTuner exhibits superior audio quality and style similarity, unlocking the ability to generate voice samples in low-resource scenarios.The key contributions are as follows:• We present VoiceTuner, with a self-supervised pre-training and fine-tuning approach to alleviate data scarcity in lowresource applications. • We introduce a lightweight multiscale adapter with separated fine-tuning strategies for global and local transformers, to efficiently fine-tune only around 1% parameters and reduce the computational cost. • Experimental results present VoiceTuner-SSL's ability to keep acoustic continuations, and demonstrate VoiceTuner's achieves state-of-the-art results in rich-resource evaluation. • Low-resource downstream applications including zero-shot TTS, singing voice synthesis, and instruction TTS present VoiceTuner's superior audio quality and style similarity with reduced data requirement and computational cost. 
 
 Appendix:  


Decision: The paper presents VoiceTuner, a method leveraging self-supervised pre-training and efficient fine-tuning for voice generation. Reviewers appreciate that "the paper shows the pipeline of pre-training and fine-tuning auto-regressive models in LLMs can also work in the speech domain" and that "The techniques presented in the paper, such as self-supervised pre-training and efficient fine-tuning, offer valuable insights for advancing voice synthesis in the audio community." Despite concerns about novelty in methodology and comparison breadth, the overall contribution to advancing voice synthesis is acknowledged by most reviewers.

Reviews: review: paper_summary: paper_summary: The paper proposes VoiceTuner, a method for voice generation. It addresses data scarcity and high computational cost using self-supervised pre-training and efficient fine-tuning. VoiceTuner achieves state-of-the-art results in TTS and exhibits superior audio quality in low-resource applications.
strength_weakness: strength_weakness: This paper introduces a pretrain+finetune pipeline for voice generation tasks, which has shown success in the field of large language models (LLMs). The techniques presented in the paper, such as self-supervised pre-training and efficient fine-tuning, offer valuable insights for advancing voice synthesis in the audio community.
limitations: limitations: Missing comparison and discussion to related work such as SpeechFlow, Spear-TTS, and Audiolm.

In Audiolm, the authors propose exploring both semantic tokens and acoustic tokens for pretraining. They find that using only acoustic tokens leads to inconsistencies in linguistic content. However, the authors of this paper still employ acoustic tokens for pretraining. I am curious to know if this approach of using only acoustic tokens with the multi-scale transformer is a better method for audio pretraining compared to the combination of semantic and acoustic tokens. Specifically, I would like to know if your pretraining method can outperform Audiolm in terms of audio quality and long-term consistency for unconditional audio generation.

In Spear-TTS, they utilize a semantic-to-acoustic pretraining approach similar to Audiolm. They also achieve impressive results using only 15 minutes of labeled data and claim to address low-resource TTS. Therefore, I believe it would be valuable for the authors to compare their approach with Spear-TTS under low-resource scenarios.

Additionally, SpeechFlow also adopts a pretrain+finetune pipeline and also utilizes techniques such as LORA and full parameter fine-tuning. It would be beneficial for the authors to discuss the advantages of their strategy compared to SpeechFlow.

Including a comparison and discussion with these related works would provide a comprehensive analysis of the proposed method's effectiveness and highlight its unique contributions in the field of audio pretraining and fine-tuning.
paper_summary: paper_summary: This paper propoese a self-supervised pre-training and fine-tuning method for low-resource voice generation: 1) large scale unlabeled data are used for model pre-training; 2), a multi-scale transformer adapter is introduced to update a small part of parameters as a plug-and-play modeule. Experimental results show significant improvement achieved.
strength_weakness: strength_weakness: 1. speech pre-training and fine-tuning mehtod proposed to alleviate data scarcity in low resource speech generation.
2. a ligntweight multiscale adapater is used for global and local transformers.
3. experiments are conducted for zero-shot TTS, instruction text-to-speech and singing voice synthesis.
limitations: limitations: 1. pre-training and fine-tuning methodology is not novel in this area.
2. local and global network structure is not novel for speech generation, as already been used in uniaudio.
4. Is table 4 the result of rich-resource fine-tuining resutls? I think the title of table 4 is wrong.
5. Are the results of other models, such as YourTTS, VALL-E and Spear-TTS from their papers or based on your re-implementation ? 
6. What is the result compared with diffusion based method, such as voicbox? 
7. The resutls of other models are not compared in the low-resouce setting, why is that?
paper_summary: paper_summary: The paper introduces a novel approach utilizing self-supervised pretraining alongside parameter-efficient fine-tuning to address data scarcity in voice synthesis. It features a lightweight multi-scale adapter, enabling fine-tuning with just 1% of the model's parameters. The experiments demonstrate state-of-the-art (SOTA) performance across both resource-rich and resource-poor settings, assessed through objective and subjective methods. The model is adept at handling a variety of low-resource downstream tasks such as zero-shot TTS, instructional TTS, and singing voice synthesis.
strength_weakness: strength_weakness: - The paper conducts extensive experiments. Both the effect of self-supervised pretraining and finetuning was demonstrated. The proposed model was compared with SOTA TTS models, including YourTTS, VALL-E, and Spear-TTS. Ablation studies were conducted to confirm the advantage of design choices. The advantage of computation cost is also confirmed by comparing with the full-finetune counterpart.
- The proposed model achieves strong performance in both well-resourced and limited-resource environments, shown by both objective and subjective metrics. 
- The analysis and discussion of the result is comprehensive.
- The fine-tuning phase is innovatively designed to incorporate various types of control signals, including MIDI for pitch and duration, as well as audio for acoustic and semantic information.
- The paper is well-organized and clearly written, making it accessible and easy to comprehend.
limitations: limitations: The reviewer has not identified any major issues within the paper.
paper_summary: paper_summary: This paper attempts the task of voice generation. Specifically, the authors propose a two-stage approach, named as VoiceTuner, encompassing a stage of self-supervised pre-training and a stage of fine-tuning. The first stage takes advantage of the abundance of unlabelled speech data to improve the model performance on low-resource scenarios. In the second stage, the authors propose using both LoRA and learnable prompts for efficient fine-tuning. The results demonstrate VoiceTuner could achieve the state-of-the-art performance for low-resource zero-shot TTS. Through the evaluation on zero-shot text-to-speech, instruction text-to-speech, singing voice synthesis, the authors show that proposed multi scale-transformer adapter can achieve competitive performance comparing to full-fine tuning with very few learnable parameters.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow. 
2. The results are strong with better performance than several baselines. Comprehensive ablation studies have been conducted to demonstrate the effectiveness of proposed efficient fine-tuning strategy. 
3. The paper shows that the pipeline of pre-training and fine-tuning auto-regressive models in LLMs can also work in the speech domain.
limitations: limitations: 1. The authors only include the comparisons with previous models using a "small-scale test set with the examples provided on the demo page". I think the evaluation is limited and it is better to include more comparisons to justify the effectiveness of VoiceTuner. 
2. The proposed pipeline is of limited novelty, which I think is a minor weakness considering that showing a LLM pipeline can also work in speech domain is also valuable. Moreover, the proposed efficient fine-tuning seems to be a combination of learnable prompts and LoRA, which are not new to me. 
3. A minor question is that the arrangement of Table 9 is a little confusing. Please correct me if I am wrong, multi-scale transformer adapter is equal to adapted prompts + Lora. So "Lora" in the third row means only using Lora without adapted prompts, right?