SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement

## Abstract

Abstract Singing voice conversion (SVC) aims to convert a singer’s voice to another singer’s from a reference audio while keeping the original semantics. However, existing SVC methods can hardly perform zero-shot due to incomplete feature disentanglement or dependence on the speaker look-up table. We propose the first open-source high-quality zero-shot SVC model SaMoye that can convert singing to human and non-human timbre. SaMoye disentangles the singing voice’s features into content, timbre, and pitch features, where we combine multiple ASR models and compress the content features to reduce timbre leakages. Besides, we enhance the timbre features by unfreezing the speaker encoder and mixing the speaker embedding with top-3 similar speakers. We also establish an unparalleled large-scale dataset to guarantee zero-shot performance, which comprises more than 1,815 hours of pure singing voice and 6,367 speakers. We conduct objective and subjective experiments to find that SaMoye outperforms other models in zero-shot SVC tasks even under extreme conditions like converting singing to animals’ timbre. The weights, code, dataset, and documents of SaMoye are publicly available on https://github.com/CarlWangChina/SaMoye-SVC .

## 1 Introduction

SVC aims to convert the timbre in a given song to the reference audio without disrupting the original content. This technique has wide applications such as virtual singers Nakano & Goto (); Hong et al. (); Kaewtip et al. (), and music production Turk et al. (); Ijiga et al. (). In the SVC task, most singing contents and required timbres are unseen data, which brings huge challenges to SVC models. In recent years, researchers have investigated a variety of approaches to SVC. These studies disentangle the audio into timbre and content features Liu et al. (); Guo et al. (); Zhang et al. (). The content features are fused with timbre features from the reference audio and are input into the decoder to generate the audio with timbre converted.

Zero-shot SVC requires the model to perform SVC from short reference audio without finetuning. However, existing SVC models still need minutes to hours of singing clips to fine-tune the model for high-quality singing voice conversion, which vastly constricts these models’ usageHuang et al. (). There are three possible reasons for this limitation. First, Some studies introduce a speaker lookup table for timbre features Fernandez-Martín et al. (); Qian et al. (), making it hard to expand the table for unseen speakers. Other studies drop the speaker lookup table and train a speaker encoder to extract the speaker embedding from a given audio. But they confront the second problem called timbre leakage in content features. The content features are extracted by pretrained automatic speech recognition (ASR) models like Hubert  Hsu et al. (), Whisper Radford et al. (), or ContentVec Qian et al. (). Despite those ASR models keeping the semantic information while reducing the timbre information in the content features, the timbre leakage can result in the converted audio being more similar to the original instead of reference audio when facing the unseen data. The last reason may be the limited training data. SVC data can be classified as parallel and unparalleled data on whether multiple singers sing the same song. Zero-shot SVC requires the model to see as much data as possible to enhance its generalizing capability. Due to the limited number of parallel data, existing models are normally trained using unparalleled data. However, there is still a low quantity of data for these models to perform zero-shot SVC.

Figure: Figure 1: Figure (a) illustrates the functional demonstration of the SaMoye model, which possesses zero-shot sing voice conversion capability to transform both human and non-human timbres like those of cats or dogs. Figure (b) is a specific case analysis of the audio after SaMoye model’s conversion, where one can observe that the mel-spectrogram and formant of the converted audio closely resemble those of the animal timbre.
Refer to caption: extracted/5934164/pics/fig_0.png

We establish a large-scale unparalleled SVC dataset including 1,815 hours of pure songs with 6,367 speakers by collecting online audio and open-source datasets, and we manually check the datasets to ensure their quality. Besides, we base on Whisper-VITS-SVC(^1^11https://github.com/PlayVoice/whisper-vits-svc) to modify the feature disentanglement and inference strategy and propose a zero-shot high-quality SVC model SaMoye can convert human and even non-human timbres. We argued that compressing the content features may improve the timbre leakage and we take multiple approaches to improve the feature disentanglement. Specifically, we evaluate the different combinations of pretrained ASR models including HubertSoft Hsu et al. (), Whisper Li et al. (), and ContentVec Qian et al. (), where we compress the features using vector quantization or k-means. Instead of freezing the speaker encoder in training, we train the speaker encoder together to enhance the timbre information in the speaker embedding. We evaluate SaMoye through objective and subjective experiments on the timbre of humans. As shown in Figure 1, we also assess SVC models on non-human timbres like cats and dogs as the extreme condition, for non-human timbres are absolute unseen data and are effective in assessing the zero-shot ability. The results show that SaMoye can perform SVC with high quality on human and non-human timbre.

To summarize, our contribution includes:

- •
We establish a large-scale open-source dataset on SVC and propose the first open-source high-quality zero-shot SVC model SaMoye that can even perform SVC for non-human timbre.
- •
We designed and evaluated the effect of different content features by combining existing ASR models and compression methods like vector quantization and feature clustering.
- •
We conduct objective and subjective experiments on converting songs to human and non-human timbre to prove that SaMoye can perform high-quality zero-shot SVC.

## 2 Related Work

### 2.1 Singing Voice Conversion

The core objective of SVC is to convert the singing voice of a source singer into that of a target singer while preserving the musical content, including melody and rhythm. SVC methods predominantly rely on the recognition-synthesis framework, which involves recognizing the content of the singing voice and then synthesizing it in the target voice.  Nercessian () use a pretrained LSTM to extract the speaker embedding and concatenate it with the original speaker’s phoneme and loudness embedding in the decoder to generate the converted audio. PitchNet Deng et al. () introduces a singer prediction network and pitch regression network to control the timbre and pitch stability. The embeddings from the two networks are fed into the decoder with the output from the encoder to generate the audio.  Luo et al. () use separate encoders to extract singer and techniques embedding for singer and techniques classification tasks respectively and are concatenated before feeding into the decoder and refinement network to generate the converted audio.  Polyak et al. () takes as input the speech features by a pre-trained automatic speech recognition (ASR) model Wav2Letter, the F0 feature by Crepe, and the loudness feature from the power spectrum. The speaker embedding from the target singer is included in the generator for converted audio generation.  Li et al. () introduce F0 features, PPGs features as content features, and the Mel-spectrogram as the timbre features, which is enhanced through singer classification and reconstruction. FastSVC  Liu et al. () leverages sine-excitation signals and loudness features and uses a Conformer model to extract content features. These studies introduce HiFi-GAN Kong et al. () and BigVGAN  gil Lee et al. () as the vocoder to generate the converted audio.

### 2.2 Singing Feature Disentanglement

Feature disentanglement is a crucial technique in SVC, as it separates different aspects of the singing voice, such as content, timbre, and pitch, into distinct representations. This separation is essential for achieving flexible and accurate voice conversion. However, few studies have focused on more complete feature disentanglement. Most studies directly apply Hubert Hsu et al. (), Whisper Qian et al. (), or ContentVec Qian et al. () to extract content features without modification. These models are trained on ASR tasks, resulting in more content and less timbre information in their embeddings. Some studies like Whisper-VITS-SVC introduce auxiliary tasks like speaker recognition to enhance the timbre features. However, few studies have focused on reducing the timbre leakage in the content features. In this paper, we evaluate different content features and apply compression methods like k-means and vector quantization to reduce timbre leakages.

## 3 Methods

### 3.1 Overview of the SaMoye

We base on Whisper-VITS-SVC and propose a zero-shot SVC model SaMoye shown in Figure 2. Considering the importance of pitch in singing, we use RMVPE Wei et al. () to extract pitch features. We use GE2E Wan et al. () as the speaker encoder to extract speaker embedding from the audio as the timbre features. For content features, we evaluate various compression methods and combinations of existing ASR models, which will be detailed in Section 3.2. We get the prior distribution from these features using a Flow model, while the posterior distribution is obtained from the original waveform and the speaker embedding using another posterior encoder. SaMoye is trained on audio reconstruction from the posterior distribution aligned with the prior distribution by minimizing their KL divergence. We also introduce a multi-scale discriminator for adversarial learning. In the inference stage, we extract content features, pitch features from the original audio, and timbre features from the reference audio to get their prior distribution through the Flow model and generate the converted results.

Figure: Figure 2: The overall framework of SaMoye. $u_{q}$ and $log_{q}$ are of the posterior distribution, and $u_{p}$ and $log_{p}$ are of the prior distribution. $z_{q}$, $z_{p}$, $z_{t}$, $z_{t}$ are sampled from their corresponding latent space, among which $z_{t}$ is from the forward-process of the Flow model and $z_{f}$ is from the inverse process.
Refer to caption: extracted/5934164/pics/fig1.png

Figure: Figure 3: The process of extracting the speaker embedding (left panel) and content features (right panel). We applied at least one of the pre-trained models for the ASR model, including HubertSoft, ContentVec, and Whisper.
Refer to caption: extracted/5934164/pics/fig2.png

### 3.2 Feature Disentanglement and Enhancement

We improve feature disentanglement by reducing content feature timbre leakages and enhancing the speaker encoder’s timbre information. As is depicted in Figure 3, we evaluate various combinations of HubertSoft, Whisper, and ContentVec using concatenation and try to compress their features using k-means or vector quantization. Specifically, we implement vector quantization by a residual vector quantization through a set of codebooks. For timbre features, we unfreeze the speaker encoder during training.

### 3.3 SaMoye (retrieval)

We are inspired by retrieval-based voice conversion to introduce another inference strategy, namely SaMoye (retrieval). SaMoye (retrieval) keeps the same architecture as SaMoye but retrieves the top-3 most similar speaker embeddings from the speaker embedding database and averages them with the reference speaker embedding in the inference stage. We build the speaker embedding database from the dataset to compute the cosine similarity between the reference audio and the embedding in the database. Then, in the inference stage, we can retrieve the top three similar speaker embeddings and average them with the reference speaker embedding as the timbre features; this method may reduce the timbre similarity of the converted audio but improve the zero-shot performance by exploiting the seen speaker embedding to fill the gap brought by the unseen reference audio that can result in bad cases like mute voice.

### 3.4 Adversarial Learning

We follow Whisper-VITS-SVC and introduce an adversarial training strategy to train the generator to synthesize the converted audio and a discriminator to discriminate between the generated and real audio. For the generator, the Mel-spectrogram and speaker embedding pass through the posterior encoder to get the posterior latent variables $z_{q}$. We apply the decoder in gil Lee et al. () to generate the audio from the latent variables. Meanwhile, the pitch and content features pass another encoder to get the latent variables $z_{p}$. Inspired by gil Lee et al. (), we use the speaker embedding as the condition for a Flow model to generate the prior latent variables $z_{t}$ to be aligned with the posterior latent space. We compute the Kullback–Leibler divergence(KL) between the $\mu$ and variance ${\sigma}^{2}$ of $z_{t}$ and $z_{q}$ as well as $z_{f}$ and $z_{p}$ to sum up as the $\mathcal{L}_{kl}$. We also sum up the L1 and L2 loss for waveform as $\mathcal{L}_{wav}$ and the mel-spectrogram as $\mathcal{L}_{mel}$ for audio reconstruction. We also use the $\mathcal{L}_{stft}$ following  Takaki et al. ().

$$ $\mathcal{L}_{dis}=E\left[D(a)^{2}+(1-D(a^{\prime}))^{2}\right]$ (1) $$

$$ $\mathcal{L}_{adv}=E\left[(1-D(a^{\prime}))^{2}\right]$ (2) $$

For the discriminator, we use the multi-scale and multi-period discriminator in our study, which takes the generated and real audio as input to compute the L1 loss between their feature maps in the discriminator as $\mathcal{L}_{fmap}$. The loss for the discriminator is shown in equation 1 and the adversarial loss for the generator is described in equation 2, where $D$ is the discriminator and $a$ and $a^{\prime}$ is real and generated audio.

The final loss for the generator is shown below, where we set the $\alpha$ to 1.0, $\beta$ to 0.2, and $\gamma$ to 9.0 during training.

$$ $\begin{split}\mathcal{L}_{gen}&=\mathcal{L}_{wav}+\mathcal{L}_{mel}+\beta* \mathcal{L}_{kl}\\ &+\mathcal{L}_{adv}+\mathcal{L}_{fmap}+\gamma*\mathcal{L}_{stft}\end{split}$ (3) $$

## 4 Experiment

### 4.1 Datasets

We have utilized a dataset comprising 6,367 speakers and over 1,815 hours of data for training purposes. This dataset includes 36.5 hours of online music and over 1,815 hours of open-source singing and speech data. We separate the pure human voice from these music clips using Demucs Défossez et al. () for online music. Consequently, the obtained data are manually checked to remove all the errors in the recognition results. The specifics of these datasets are detailed in Table 1.

### 4.2 Experiment Setup

**Table 1: The statistics of the used datasets.**
| Datasets | Speakers | Duration(hours) |
| --- | --- | --- |
| JSUT-SongSonobe et al. () | 1 | 0.41 |
| PJSKoguchi & Takamichi () | 1 | 0.60 |
| KiSingShi et al. () | 1 | 0.88 |
| Jvs MusicTamaru et al. () | 100 | 4.00 |
| CSDChoi et al. () | 1 | 4.86 |
| OpencpopHuang et al. () | 1 | 5.20 |
| DSD100Liutkus et al. () | 100 | 6.99 |
| PopcsLiu et al. () | 117 | 5.89 |

The models are trained on the established datasets. All the audio sampling rates are united to 32k. We use a filter length of 1024, a hop length of 320, and a Hanning window with a length of 1024 when converting the waveform to the Mel-spectrogram with 80 filters.

We select five song clips with a length of 20 seconds and 11 speakers (6 humans, and 5 non-humans) to evaluate the models’ performance on zero-shot SVC including the extreme condition for non-human timbre. The five songs cover the full range from bass to treble. The human timbres contain 4 males and 2 females, while the 5 non-humans include 3 cats and 2 dogs.

For the subjective experiment, we invite 20 music-professional participants to listen to the original music and timbre and evaluate the converted song clips.

### 4.3 Metrics

We use several objective and subjective metrics to evaluate the quality of the converted audio and how similar the timbre is to the original audio.

#### 4.3.1 Subjective Metrics

The subjective metrics include:

- •
Mean Opinion Score on Similarity(MOS-S): MOS-S is based on a 5-score Likert scale, where 5 means the same timbre and 1 for a completely different timbre.
- •
Mean Opinion Score on Quality(MOS-Q): MOS is a widely-used audio or video quality evaluation standard based on expert evaluation. The score of MOS is from 1 to 5, where a higher score means higher quality.

#### 4.3.2 Objective Metrics

The following various objective metrics are also included to evaluate the converted audio from different perspectives:

- •
Perceptual Evaluation of Speech Quality (PESQ): PESQ computes multiple perspectives like temporal alignment and perceptual filtering. PESQ score is from -0.5 to 4.5, where a higher score stands for better perceptual quality.
- •
Short-Time Objective Intelligibility (STOI): STOI represents how well the audio can be comprehended, and the STOI score is from 0 to 1. The higher STOI score means that the audio is easier to understand.
- •
Non-Intrusive Speech Quality Assessment (NISQA) (^2^22https://github.com/gabrielmittag/NISQA): NISQA is a non-intrusive metrics based on a pre-trained deep learning model. Given audio, the NISQA predicts MOS for audio quality, NOI for noise degree, DIS for audio coherence, COL for timbre quality, and Loud for loudness. All these predicted scores are the higher the better.
- •
Speaker Encoder Cosine Similarity (SECS): We use CAM++  Wang et al. () to extract speaker embedding from generated and original audio and compute their cosine similarity. SECS score is from 0 to 1, and higher SECS means higher timbre similarity.

**Table 2: Objective and subjective results of SaMoye and baseline models (Human).(All subjective metrics exhibit statistically significant differences with p < 0.03.)**
| Model | Objective Metrics | Subjective Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PESQ$\uparrow$ | STOI$\uparrow$ | NISQA$\uparrow$ | SECS$\uparrow$ | MOS-S$\uparrow$ | MOS-Q$\uparrow$ |  |
| Sovits-SVC | 1.818 | 0.572 | 2.893 | 0.257 | 2.98 | 3.50 |
| FreeVC | 0.200 | 0.466 | 3.807 | 0.236 | 1.36 | 1.25 |
| GPT-Sovits | 0.332 | 0.447 | 1.755 | 0.201 | 1.52 | 1.28 |
| SaMoye (retrieval) | 2.053 | 0.622 | 3.193 | 0.192 | 3.24 | 3.80 |
| SaMoye | 2.343 | 0.640 | 2.278 | 0.266 | 3.17 | 3.64 |

**Table 3: Objective and subjective results of SaMoye and baseline models (Non-human).(All subjective metrics exhibit statistically significant differences with p < 0.05.)**
| Model | Objective Metrics | Subjective Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PESQ$\uparrow$ | STOI$\uparrow$ | NISQA$\uparrow$ | SECS$\uparrow$ | MOS-S$\uparrow$ | MOS-Q$\uparrow$ |  |
| Sovits-SVC | 2.085 | 0.608 | 3.056 | 0.208 | 2.80 | 3.47 |
| FreeVC | 0.269 | 0.475 | 4.112 | 0.178 | 1.25 | 1.27 |
| GPT-Sovits | 0.642 | 0.454 | 1.507 | 0.356 | 1.45 | 1.24 |
| SaMoye (retrieval) | 2.057 | 0.627 | 3.221 | 0.164 | 3.09 | 3.88 |
| SaMoye | 2.392 | 0.652 | 2.762 | 0.241 | 2.95 | 3.47 |

Figure: (a) Fixed speaker encoder.
Refer to caption: extracted/5934164/pics/fig3_a.png

### 4.4 Comparison

We compare SaMoye with several SVC or VC models to evaluate its performance. These models are trained on our datasets and their details are as follows:

- •
SoVITS-SVC(^3^33https://github.com/svc-develop-team/so-vits-svc): The flow model based Sovits-SVC, which uses Contentvec as its representations.
- •
FreeVC Li et al. (): FreeVC is a speech voice-conversion model, which disentangles content information by imposing an information bottleneck to WavLM features, and introduces the spectrogram-resize-based data augmentation to improve the purity of extracted content information.s
- •
GPT-SoVITs(^4^44https://github.com/RVC-Boss/GPT-SoVITS): GPT-SoVITs is originally a TTS-model. We transfer GPT-SoVITS to singing voice conversion by replacing the phoneme embedding in GPT-SoVITS with F0 embedding and training on our datasets.

The results are shown in Table 2. SaMoye outperforms other models in PESQ, STOI, and SECS. We also noticed that FreeVC achieves the highest NISQA. The reason may be that NISQA is evaluated by a model trained on speech datasets to predict MOS scores, while there is a gap between singing and speech. FreeVC is a voice-conversion model, in which the converted audio sounds like speech instead of singing. For subjective evaluation, we observed that SaMoye (retrieval) performs the best in both MOS-S and MOS-Q because SaMoye (retrieval) mixes the top three similar speaker embeddings, which fill the information gap in contrast with just using single speaker embedding; this is proved through the lowest SECS for SaMoye (retrieval) because the mixture impact some of the timbre features extracted from the reference audio. Furthermore, we conclude that audio quality can significantly influence the timbre similarity, for models with low MOS-Q can hardly achieve MOS-S. Despite the speaker embedding difference introduced by SaMoye (retrieval), listeners are insensitive to the influence when the audio quality is high.

#### 4.4.1 Evaluation on Animal Timbres

We use the animal voice as the extreme condition to perform zero-shot singing voice conversion for they are complete unseen data for all SVC models. The results are shown in Table 3. The results are nearly the same as the comparison experiment. However, we find that GPT-Sovits achieves the highest SECS. This may be attributed to that the animal voice is still unseen data for CAM++ which is trained on human speech datasets. Because of the inaccuracy brought by these unseen data, the SECS is less reliable under this extreme condition. This is proved by the subjective metrics, where GPT-Sovits results in low MOS-S and MOS-Q scores. The results further explain the conclusion in the human voice cases that SaMoye (retrieval) can fill the blank information in the speaker embedding, especially for the animal cases where the timbre information blank is huger.

**Table 4: Objective results for different feature representations.**
| Setting | Model | Objective Metrics |  |  |  |
| --- | --- | --- | --- | --- | --- |
| PESQ$\uparrow$ | STOI$\uparrow$ | NISQA$\uparrow$ | SECS$\uparrow$ |  |  |
| #1 | Whisper | 2.073 | 0.613 | 2.833 | 0.270 |
| #2 | HubertSoft | 2.013 | 0.522 | 2.889 | 0.278 |
| #3 | ContentVec (Layers 9) | 1.032 | 0.478 | 2.776 | 0.187 |
| #4 | ContentVec (Layers 12) | 0.957 | 0.483 | 2.620 | 0.205 |
| #5 | HubertSoft+k-means(900, Tensor) | 0.497 | 0.376 | 2.499 | 0.215 |
| #6 | HubertSoft+k-means(900, Codes) | 0.272 | 0.214 | 1.726 | 0.165 |
| #7 | HubertSoft+RVQ(Tensor) | 0.847 | 0.352 | 2.131 | 0.189 |
| #8 | HubertSoft+RVQ(Codes) | 1.071 | 0.230 | 2.483 | 0.278 |
| #9 | HubertSoft+Whisper | 2.343 | 0.640 | 2.278 | 0.266 |

**Table 5: The results of fixed and unfrozen speaker encoder. (All subjective metrics exhibit statistically significant differences with p < 0.05.)**
| Speaker Encoder | Objective Metrics | Subjective Metrics |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PESQ$\uparrow$ | STOI$\uparrow$ | NISQA$\uparrow$ | SCES$\uparrow$ | MOS-S$\uparrow$ | MOS-Q$\uparrow$ |  |
| Fixed | 2.105 | 0.614 | 2.796 | 0.250 | 3.05 | 3.35 |
| Unfrozen | 2.343 | 0.640 | 2.278 | 0.266 | 3.17 | 3.63 |

### 4.5 Evaluation on Feature Disentanglement

We investigate the most used content feature extractors including HubertSoft, Whisper, and ContentVec, all trained on ASR tasks. We also evaluate the features from the different layers of ContentVec. Besides, we introduce different compression methods including k-means and RVQ (Residual Vector Quantization) to reduce the timbre leakage in the content features. We use Tensor methods to input the cluster center of k-means or codebook embeddings into the model, while for Codes we use the cluster index for k-means or codebook’s index for RVQ instead. The results are shown in Table 4. We noticed that Hubert and Whisper perform better than ContentVec. Setting #2, #7, and #8 prove that HubertSoft contains less timbre leakage of the original audio and results in higher similarity in the converted audio. Setting #6, and #8 that use codes have worse performance than those using embeddings; the reason may be that the codes learned from k-means and RVQ lose too much information, which makes it harder for the model to understand the original content. Considering the overall performances, we use setting #9 as the best model for further experiments.

We also evaluate the effect of unfreezing the speaker encoder during training. The results are shown in Table 5 that unfreezing the speaker encoder during training enhances the performance in all metrics except NISQA, which may be because the NISQA model is pretrained on speech datasets where a gap exists between the singing and speech. In addition, we use t-SNE to reduce the speaker embedding to two dimensions for visualization as shown in Figure 4. We select 328 speakers from speech datasets including Aishell, KSS, and VCTK, and 6,299 speakers in our singing datasets. The results show training the speaker encoder on the SVC tasks makes a clearer difference between the speaker embedding from speech and singing. It also proved that the timbre in singing is more complex and different from speech, which can be caught through the unfrozen speaker encoder.

## 5 Conclusion

This paper proposes an open-source large-scale dataset and a high-quality zero-shot SVC model that can convert a song to human and even non-human timbre. We investigate multiple methods for disentangling the content features by combining different ASR models and introducing K-means and vector quantization to compress these features. We also enhance the timbre feature by training the speaker encoder together. We conduct objective and subjective experiments to find that SaMoye can perform high-quality SVC on human and non-human timbre.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: I guess the main question is, how are the findings in the paper significant for the broader community? While it is true that the method proposed involves steps that have not been explored in this specific topic, I am interested in seeing if there are resuable insights for the general audience.
soundness: 3
strengths: The paper's objectives are very clear and the figures do a good job of explaining how the model architecture works. The paper does a thorough evaluation of their proposed ideas in the ablation studies showing the validity of each of their contribution.
summary: The paper presents SaMoye, a method for singing-voice conversion in the zero-shot setting. The method extends existing approaches for SVC in a few different ways:
1. The method attempts to disentangle timbre, content and pitch by combining features from multiple ASR models
2. The method involves an enhanced set of timbre features by training the speaker encoder in conjunction with the model instead of using a pre-trained and frozen speaker encoder or a speaker embedding table. 
3. The timbre features are also enhanced by mixing the top-3 most similar speakers' embeddings.

The paper involves a large collation of singing voice datasets and the authors perform both subjective and objective tests to evaluate their method.
weaknesses: The insights from this paper do not seem very significant for the community at large. 
The main idea is to incorporate more data, and add more conditioning variables to disentangle timbre, content and pitch. However, even then, the contributions in the paper seem very narrow. The authors have stated that previous work has utilized various speech encoders to try to disentangle content. Here, they used a combination of encoders instead. 

Another complaint is that the paper seems not very polished and needs a round of proofreading. It also seems like the template is different from the ICLR paper template. The bottom margin has a lot of space and the text width seems wider.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: --
soundness: 2
strengths: 1. The problem statement is clear.
2. The baseline, with so-VITS and FreeVC, is sufficient.
summary: The author proposes a zero-shot singing voice synthesis model that aims to achieve better feature space disentanglement than the current state-of-the-art. It claims to achieve this goal using the following:

1. A large-scale singing voice dataset including both human and non-human voices.
2. Use of the VITS architecture, along with multiple ASR modules for improved content feature extraction to reduce timbre leakage.
3. Inference-time speaker embedding retrieval using cosine similarity.
weaknesses: 3. The theoretical explanation is poor. In particular:

	3.1 This is essentially a re-explanation of the VITS architecture, with no significant novelty found.
	3.2 This section should present the main theoretical novelty of the paper; however, the algorithm, including the use of k-means and quantization, remains unclear.
	Figure 3(a) suggests that the speaker encoders are directly interfacing with the spectrograms, which implies that unfreezing them during training could allow timbre leakage.
	Figure 3(b) is unclear. How is the final vector produced from the clusters and codebooks? Are Q1, Q2, etc., referring to the codebooks? Do you use both methods simultaneously, or only select one?
	3.3 Mapping to the closest speaker embeddings in the training set "may reduce the timbre similarity of the converted audio." However, since one of the main goals stated in the abstract is to avoid timbre leakage, what is the rationale behind using this technique?
	3.4 The loss components presented are not novel, as all of them are already found in the HiFi-GAN and/or FreeVC papers.

	- Regarding the dataset, while it is noted as a novel aspect, this is left unexplained in Section 3.

4. Experiments/VisualizationThe experimental details are lacking. Specifically:
	4.2 Where are the training parameters? The only parameters specified are those in the loss function equation (3). Did you re-train the baselines on the proposed dataset as well? Furthermore, with such a large-scale dataset, why would you only evaluate on 11 speakers for 100 seconds each? For the non-human part, are you testing zero-shot conversion from humans to non-humans, non-humans to humans, or within each group only?
   
	Tables 2 and 3: Although seemingly complete and acceptable metrics-wise, these contain two issues:
		- Since the primary goal is reducing timbre leakage, why is there only a trivial gain on SECS in both human and non-human cases? For example, in Table 1, your gain above the original so-VITS-SVC is only 0.009 — is that really statistically significant?
		- The retrieval method you proposed has a much lower SECS. How, then, does it achieve the highest MOS? Does that mean a user would rate 5/5 if I presented them with unconverted audio?


	Section 4.1: The Tensor/Codes method and Table 4 do not make sense without adequate explanations in Section 3.2.

	Figure 4: What is the purpose of this figure? In the text, it is mentioned that "The results show that training the speaker encoder on the SVC tasks makes a clearer distinction between the speaker embeddings from speech and singing." But isn't the primary goal to demonstrate improvements in SVC performance?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: The authors should strengthen their literature review of the broader speech synthesis field, including VC and TTS, and reconsider the contributions of this study in that context.
soundness: 2
strengths: 1. Timbre leakage is indeed a critical issue in the fields of zero-shot VC and SVC. Finding ways to reduce speaker characteristics embedded within pretrained content features is a crucial area requiring further exploration and resolution.

2. The authors have invested considerable effort into experimental investigations of various content-based representations, including both continuous hidden features and discrete codecs.

3. The mixture strategy based on the top-3 similar speaker embeddings is a good engineering trick.
summary: This paper aims to address the issue of timbre leakage in zero-shot singing voice conversion (SVC). The authors leverage content features obtained from various content-based pretrained models and utilize a mixed speaker embeddings approach to mitigate timbre leakage. They conduct experimental investigations using a 1.8k-hour singing voice corpus, drawing data from academic speech/singing corpora as well as relatively in-the-wild sources like the DAMP singing voice corpus.
weaknesses: Overall, I find this paper to be an experimental exploration in the zero-shot SVC task. However, it resembles more of an engineering-focused technical report rather than an innovative research paper. Specifically:

1. Regarding novelty: (1) Existing SVC literature has already explored combining various content-based pretrained models (e.g., Whisper, ContentVec) [1]; (2) Using vector quantization to remove speaker information was first suggested in the VQ-VAE paper [2], where the authors noted its potential application in voice conversion. This approach has also been implemented in works like VQ-VC [3]. K-means clustering, similarly a form of vector quantization, is already a well-known method for addressing timbre leakage in the VC field.

2. Some of the authors' terminology is misleading and lacks rigor. For example: (1) HuBERTSoft and ContentVec should not be labeled as ASR models—they are self-supervised models; (2) The authors claim that "we establish an unparalleled large-scale dataset," but the term "establish" is highly misleading. In reality, the authors mainly aggregated multiple existing datasets for their experiments, with only 36.5 hours of Online Music being newly added in this study; (3) In Table 5, "Fixed" should correspond with "Unfixed," while "Unfrozen" should pair with "Frozen."

3. Despite utilizing numerous engineering tricks, the final audio sample quality remains subpar. The authors should at least compare with a diffusion-based SVC baseline.

4. The emphasis on mimicking the timbre of a dog or cat is not a unique contribution of this paper. Any SVC model can conduct inference with dog or cat samples.

> [1] Leveraging Diverse Semantic-based Audio Pretrained Models for Singing Voice Conversion. IEEE SLT 2024.
>
> [2] Neural discrete representation learning. NIPS 2017.
>
> [3] One-shot voice conversion by vector quantization. ICASSP 2020.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. **Why use SECS (Speaker Similarity) for non-human audio?** The effectiveness of speaker embeddings for non-human audio remains uncertain, raising questions about their reliability in such contexts.  

2. **Incorporate a speaker classification model for enhanced evaluation.** Introducing a speaker classification model would provide a more robust assessment of performance, ensuring a comprehensive evaluation beyond speaker similarity metrics.
soundness: 2
strengths: 1. The paper presents a flow-based architecture designed specifically for singing voice conversion, demonstrating improved performance in transforming vocal timbres while maintaining naturalness and expressiveness.
2. The study introduces a novel method for converting non-human voices into human-like voices, addressing key challenges in voice synthesis and bridging the gap between synthetic and natural speech.
3. The paper releases a new dataset, providing valuable resources for further research in voice conversion and fostering advancements in related fields through the availability of high-quality data.
summary: The paper introduces a novel zero-shot singing voice conversion method. The proposed approach leverages the Flow model in conjunction with speaker embeddings and audio codes, aiming to achieve high-quality voice conversion. The model is further optimized using a GAN-based loss function.

Key contributions of the paper include:
1. The release of an open-source dataset to facilitate future research.
2. A comprehensive evaluation of different content features, employing ASR models, Residual Vector Quantization (RVQ), and k-Nearest Neighbors (KNN).
3. Objective and subjective experiments conducted to assess the performance of the proposed singing voice conversion framework.
4. A novel attempt to extend the model's capabilities by converting non-human timbre into human timbre, addressing a unique challenge in voice synthesis research.
weaknesses: 1. The audio outputs are not sufficiently convincing, making them challenging to listen to and assess effectively.

2. The concept of singing voice conversion has been extensively studied in the literature, particularly through the use of flow-based and diffusion models. Notable examples include:
Real-Time and Accurate: Zero-shot High-Fidelity Singing Voice Conversion with Multi-Condition Flow Synthesis [https://arxiv.org/abs/2405.15093]

SingVisio: Visual Analytics of Diffusion Model for Singing Voice Conversion [https://arxiv.org/pdf/2402.12660]

3. There is a potential issue of timbre leakage from the automatic speech recognition (ASR) model, which could affect the conversion quality.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. How do we evaluate the similarity of converting the timbre into animals? Current speaker verification models are trained using human speech, is there a mismatch?
soundness: 2
strengths: 1. The authors extend the timbre reference from the unseen speaker to animals, which is an interesting exploration.
2. The authors provide a dataset for singing voice conversion investigation.
summary: This paper studies zero-shot sing voice conversion. The authors follow a disentanglement framework to disentangle the singing voice's features into content, timbre, and pitch features.  A combination of multiple ASR models and joint training with speaker encoder are introduced to reduce the timbre leakage and improve the speaker similarity.
weaknesses: 1. While the singing voice conversion is an interesting task, the contribution of this paper is limited. The combination of multiple ASR or SSL models to disentangle linguistic content is widely employed in many VC works. The joint training between the backbone model and the pre-trained speaker encoder is also not a new approach. The authors should strengthen their claims by providing a clearer distinction between their work and prior models.
2. Section 3.2 of feature disentanglement and enhancement does not provide sufficient detail. A more thorough explanation of how the combination of content features operates and the details of k-means or vector quantization, would enhance understanding.
3. The experimental results are strange, it makes me confused as to why the PESQ of FreeVC and GPT-Sovits is about 0.2 and 0.3, respectively. Meanwhile, the speaker similarity of all systems is significantly lower than normal values. Please clarify this issue. 
4. There are fewer demos to display the performance and the existing demos of animal conversion have an undesirable similarity. It would be nice to provide more audio samples.
5. Some trivial issues:
 - SVC -> Singing Voice Conversion (SVC) in Line 29
 - Miss blank between usage and Huang in Line 40
 - Huber and ContentVec are self-supervised models instead of ASR models