VoiceMixer: Adversarial Voice Style Mixup
Authors: Sang-Hoon Lee
Ji-Hoon Kim
Hyunseung Chung
Seong-Whan Lee
Venue: NeurIPS 2021 Poster (2021)
Keywords: Voice Conversion
Style Transfer
Speech Processing

Abstract: Although recent advances in voice conversion have shown significant improvement, there still remains a gap between the converted voice and target voice. A key factor that maintains this gap is the insufficient decomposition of content and voice style from the source speech. This insufficiency leads to the converted speech containing source speech style or losing source speech content. In this paper, we present VoiceMixer which can effectively decompose and transfer voice style through a novel information bottleneck and adversarial feedback. With self-supervised representation learning, the proposed information bottleneck can decompose the content and style with only a small loss of content information. Also, for adversarial feedback of each information, the discriminator is decomposed into content and style discriminator with self-supervision, which enable our model to achieve better generalization to the voice style of the converted speech. The experimental results show the superiority of our model in disentanglement and transfer performance, and improve audio quality by preserving content information.


Reviews:
[{"review_id": "ve1dKkHS3y", "reviewer": "Reviewer_H8yZ", "summary": "The paper presents VoiceMixer for voice conversion, which can decompose the content and voice style in a self-supervised manner.  More specifically, the paper introduces a self-supervised representation learning method to extract the content embedding and the voice style embedding, without the necessity to condition on external features (such as additional text transcriptions).  Furthermore, the paper also proposes an adversarial voice style mixup method which contains two separate discriminators for disentangled content and style representation learning respectively.\n\nCompared with existing self-supervised method like AutoVC, one advantage of the proposed method is there is no need to find the proper size of the information bottleneck (e.g. down-sampling in channel and temporal, from Auto VC) heuristically.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality]\n\n(1) The paper poses a similarity-based information bottleneck with self-supervised representation learning framework, in which the content embeddings are downsampled according to the cosine similarity between adjacent phonetic content embeddings.  Furthermore, to decide the dynamic bottleneck size, Gaussian downsampling is adopt with reference to Shen et al. 2020. (Line 116-117) by assuming that the center of same content has the largest information.\n\n(2) To increase the similarity between adjacent content embeddings, the paper also introduces contrastive loss from wav2vec (Schneider et al, 2020) as well as the adversarial speaker classification.  The method sounds technically reasonable.\n\nSchneider et al. 2020. Wav2vec: Unsupervised Pre-training for Speech Recognition.\n\n(3) Different from Cycle-GAN, the paper proposed separate content and style discriminators for adversarial feedback by following the autoencoder based reconstruction for model training.\n\n[Quality and Clarity]\n\nMost of the claims are well supported.  More importantly, in Section 3, the paper has been written in a clear way, including the similarity-based information bottleneck, the use of several different losses including the GAN losses.\n\nDetailed structures of the proposed model as well as the code has been given in the supplementary materials.  It would be easy for other researchers of the community to follow the work.\n\nFor some detailed comments, please find it in the following “Limitations” part.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(1) Line 283-285: it is said “…when trained without any information bottleneck or all of the auxiliary losses, these models are not able to covert any voice, but only reconstruct source speech”.  Would the authors please specify which line of Table 3 corresponds to this conclusion?\n\n(2) Line 9/64/275/307/314: the authors have mentioned several times “… with only a small loss of content information”.  Would the authors please clarify this conclusion by adding some experimental results to support this and also giving more explanation about this why there is such a small loss?\n\n(3) In the work of AUTOVC (Qian et al., 2019), it is said that “The down-sampling can be regarded as dimension reduction along the temporal axis, which, together with the dimension reduction along the channel axis, constructs the information bottleneck.”  The downsampling operation should be conducted both along temporal axis and channel axis.  However, it seems the proposed “similarity-based information bottleneck” by the paper only considers the downsampling along the temporal axis.  The authors should either explain why no downsampling along channel axis is not considered, or add more experiments for downsampling along channel axis.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "ui9gjnVDDLg", "reviewer": "Reviewer_NNWU", "summary": "This work expands on recent work in autoencoder-based voice conversion models by using a novel variable-length similarity-based information bottleneck trained with contrastive loss inspired by self-supervised representation learning, and avoids complications around choosing a fixed downsampling ratio. In addition, it introduces a pair of adversarial discriminators for better disentangling of content versus speaker. It can be trained without any paired data or additional information such as text transcriptions or pitch contours, and achieves significantly higher objective and subjective scores for naturalness and speaker similarity compared to previous work for both many-to-many and zero-shot voice style transfer.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality \nThis work builds on ideas from related work and combines them in a novel way for the target task. As far as I am aware, the variable-length similarity-based information bottleneck is novel, and the combination of autoencoder and disentangled GAN is likely to be novel as well.\n\n# Quality\nThe submission is high quality with multiple baselines and relevant metrics both subjective and objective. Audio samples and code are included, as well as a detailed description of model architecture and hyperparameters.\n\nThe claim that the proposed model results in only a small loss of content information (with the implied claim that baseline models have a larger loss of content information) could be better supported with comparing ASR metrics, as the groundtruth transcripts are available for VCTK.\n\nI personally would have also preferred some more (subjective) analysis on other aspects of \"style\" beyond speaker identity, for example pacing, pause lengths, and accents.\n\n# Clarity\n\nThere are parts of the submission that are not very clear, but overall it is understandable.\n\n# Significance\n\nI believe that the proposed variable-length similarity-based content autoencoder is a significant contribution, not just for VST, but potentially for other speech tasks such as TTS or ASR. The ability to obtain phoneme durations from just raw audio can be important. It would be interesting to see if this approach can be improved when text transcriptions are available, and whether the result can be more accurate than alignments from FastSpeech2.\n\n# Misc\n\n* Section 1 Introduction\n  * line 19 \"such a voice spoofing\"\n  * line 56 \"Self-supervised learned similarity makes the information bottleneck disentangle the content and style without effort to find the proper down sampling size.\"\n    * This line is a bit confusing to read. Can you try to rephrase it?\n  * Consider adding a reference to _Jia, Ye, et al. \"Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis.\" Advances in Neural Information Processing Systems 31 (2018)._\n* Section 2 Background\n  * personally I find it a bit distracting that $f_c, f_s$ are given in (content, style) order yet $X_{1,A}$ is given in (style, content) order.\n  * line 85/86 \"too narrow\"/\"too wide\"\n    * Can you confirm my understanding here? A narrow bottleneck means a large value for $\\tau$ and fewer downsampled feature frames. I would have thought that this would lead to lower reconstruction quality. \n  * \"In the process of separating content and style information, some content information is lost even with proper bottleneck size. Therefore, missing some content information in converted voice is inevitable.\"\n    * I'm not sure I agree with this conclusion theoretically.\n* Section 3 VoiceMixer\n  * Section 3.2 Similarity-based information bottleneck\n    * What is the value of $\\mathbf{c}_{T+1}$?\n    * \"where $\\mathbf{d}_n$ is cumulative sum until the similarity $q_t$ is under the average similarity\"\n      * This part is not very clear. Can you clarify it?\n  * Section 3.3 Auxiliary losses for similarity\n    * What is $k$? Is it a constant? If it is a constant then no sampling is happening. What about content embeddings near the end of the sequence?\n  * Figure 3\n    * \"Recontructed Speech\"\n  * Section 4.6 Content and speaker disentanglement\n    * line 296/297 \"speaker identify\"\n* Appendix A\n  * Figure 5\n    * \"SoftPuls\"\n  * Figure 6\n    * \"Discrminator\"\n  * Table 5\n    * \"Spaker\"\n* Audio demos\n  * The audios starting from Zero-shot VST do not load", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The submission acknowledges that some loss of content information is unavoidable with their approach. The submission acknowledges that voice style transfer can be used maliciously. However, the submission does not propose any mitigation strategies.", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"], "time_spent_reviewing": "5"}, "extra_text": {"ethical_concerns": "2. Raise safety or security concerns. For example: is there a risk that applications could cause serious accidents or open security vulnerabilities when deployed in real-world environments?\n\nVoice conversion can exploit speaker id based security setups.\n\n7. Deceive people in ways that cause harm. For example: could the approach be used to facilitate deceptive interactions that would cause harms such as theft, fraud, or harassment? Could it be used to impersonate public figures to influence political processes, or as a tool of hate speech or abuse?\n\nVoice conversion can exacerbate the issue of fake news.\n\n\nThe ethical concerns have not been addressed except for a single line about the potential for voice spoofing in the introduction. However, this work is building on previous work, and improves the speaker verification equal-error-rate metric from 14.0% (AUTOVC (τ=32)) to 12.5%, which is likely not a significant enough improvement to unlock any new avenues of attack. The improvement in Naturalness for the zero-shot VST may be a cause for concern. I have marked this as needs ethics review purely based on the quality of the zero-shot VST in the audio samples, which sound alarmingly real."}}, {"review_id": "XO-xh9Srewp", "reviewer": "Reviewer_fx4p", "summary": "This paper proposes two ideas to tackle voice conversion task.\n\nFirst, the authors propose a new information bottleneck approach that aggregates similar content embeddings using similarity-based downsampling&upsampling method.\n\nSecond, the authors use adversarial training method to improve the quality of the output. More specifically, the authors propose to use self-learned embeddings to provide conditioning signals to two discriminators, content discriminator and style discriminator.\n\nThe proposed VoiceMixer model was compared with previous zero-shot VST baseline models.\nExperiment results show that the proposed model shows superior performance than the baseline models.\n\nIn Table3, the authors also conduct ablation experiments to see which method is mostly contributing to the superior performance of the proposed model. Experiment results show that the contrastive learning method for content embedding is contributing the most to the performance.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Comments on the originality:\n\nThis paper proposes several ideas to tackle the VC task.\nThe quality of the produced speech from this model sounds plausible.\n\nHowever, I have a few comments on the originality and novelty of this work.\n\n1. Too many methods are used together making the readers hard to follow what is actually contributing to the performance. For example, the authors use Instance Normalization in the content encoder. It is well known that Instance Normalization works as sort of a bottleneck and help remove speaker information from speaker embedding [1]. Therefore, it is unclear how much the proposed information bottleneck approach in 3.4 is actually contributing to the final results in the absence of Instance Normalization. Additionally, the authors proposed to use repulsion feature matching loss, which sounds uncommon to me. Is this really contributing to the final performance at all?\n2. More significantly, I have doubt on the originality of this work because very similar ideas were proposed before.\nIn 3.4, the authors propose to use contrastive learning method and use similarity between adjacent content embeddings to found boundaries. This is actually very similar to the method proposed in [2] where the they also use contrastive learning method and then find the boundary based on similarities between the adjacent embeddings frames.\nIn addition, the idea of reusing the self-learned embeddings for conditioning discriminator in 3.4 was also proposed previously in [3], where they proposed to learn speaker embedding using contrastive learning and then reuse it to condition the discriminator.\n\n[1] Chou, J. C., Yeh, C. C., & Lee, H. Y. (2019). One-shot voice conversion by separating speaker and content representations with instance normalization. Interspeech 2019.\n\n[2] Kreuk, F., Keshet, J., & Adi, Y. (2020). Self-supervised contrastive learning for unsupervised phoneme segmentation. Proc. Interspeech 2020.\n\n[3] Choi, H. S., Park, C., & Lee, K. (2020). From inference to generation: End-to-end fully self-supervised generation of human face from speech. Proc. ICLR 2020.\n\n\n\nComments on writings:\n\n- Pros\n1. The proposed method is easy to follow.\n\n- Cons\n1. It seems like the authors tried to address the negative societal impact by mentioning in the Introduction section very shortly as written below.\n\n17 VC is also called voice style transfer (VST),\n18 and it shares a long history with the objective to clone someone’s voice. There is even a potential\n19 risk of usage in crime such a voice spoofing (Kinnunen et al., 2012),\n\n This deteriorates the flow and logical structure of the writing. I recommend the authors to address this issue in the Conclusion section for better Introduction.\n\n\n\nSome other comments & questions:\n\n1. Why is Similarity(%) results in zero-shot VST experiment better than that of many-to-many experiment? Maybe there was something wrong in the experiment setting?\n2. Is it okay to conduct human evaluation without IRB? (as written in checklist)\n3. The authors argue that the alignment found by the proposed method is actually related to \"phoneme alignment\". However, I'm not sure if the proposed method is actually related to \"phoneme-alignment\". I don't think Fig4. (c) and Fig4. (d) are similar. In addition, I don't think simply showing figures (in Fig.4) support the argument.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the potential negative societal was commented in the Introduction, it was addressed too shortly.\nI ask the authors to address this more on Conclusion section and mention what would be the possible solutions to tackle the possible negative impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "U2zKtEmRSr", "reviewer": "Reviewer_Yn2G", "summary": "This paper proposes a new information bottleneck design to disentangle speaker style information from the content information in speech for voice conversion. The information bottleneck downsamples the hidden representation based on representation similarity (instead of uniform downsampling as in the AutoVC baseline). The proposed voice conversion system also integrates some auxiliary loss to further improve the performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros of this paper:\n\n1. The idea of similarity-based bottleneck is novel and intuitively plausible.\n2. This paper evaluates the performance of voice conversion systems through very comprehensive objective and subjective metrics.\n3. The audio demo sounds very compelling. It is among the highest-quality demos for non-parallel unsupervised voice conversion systems. The demos for the baselines also sound pretty nice.\n\nCons of this paper:\n\n1. On top of the proposed novel information bottleneck design, this paper introduces a lot of loss terms, most of which are like simple superpositions of known techniques. For the generator, there are four losses during reconstruction, and four losses during conversion. It is the most complicated voice conversion system I have seen so far.\n\nWhile the complication can be partially justified by the good performance, there are some outstanding issues. First, this paper does not discuss in detail how the hyper-parameter tuning is performed. Considering the number of loss terms, tuning the loss weights looks like a big challenge. Second, the ablation study was not performed thoroughly. In particular, the main novelty in this paper is an improved bottleneck design that disentangles speaker style information, but a lot of the loss terms also aim to disentangle speaker info, Ladv, Lpos, Lneg, L*adv, L*style+, and L*style-. It is thus unclear how much the improved bottleneck design plays a role. In Table 3, some ablation studies are performed, but a more thorough ablation study would be desirable, considering the large number of loss terms. For example, I would be interested in seeing an AutoVC baseline with only the bottleneck replaced with the similarity-based one.\n\n2. On a related note, as the major innovation of this paper, the information bottleneck design is not fully studied by the experiment. For example, the paper claims that similarity-based IB frees the need to tune the hidden representation sampling rate, and thus achieves a better trade-off between speaker style disentanglement and content loss. I would love to see this point better validated. For example, one possible experiment is to plot the disentanglement and content loss (choose one of the many objective metrics in the paper for each) against the physical bottleneck dimension. If the authors' claim holds, AutoVC should exhibit a very clear trade-off as bottleneck dimension changes, but the proposed algorithm should not be as sensitive.\n\nMinor issues:\n1. The authors seem to disambiguate between the adversarial loss for the discriminator and the generator using only the argument difference, i.e. Ladv(Dc, Ds; G) v.s. Ladv(G; Dc, Ds). This is not quite a standard notation. Try L^D_{adv}(Dc, Ds; G) v.s. L^G_{adv}(G; Dc, Ds).\n\n2. Recently, [1] proposed a similar similarity-based information bottleneck. The general goals of the IB design are different in these two papers (one is for disentangling rhythm, and this work is for better disentangling speaker style), but the practice is similar. It is worthwhile to mention this work in the related work section.\n\n[1] Qian, Kaizhi, et al. \"Global Prosody Style Transfer Without Text Transcriptions.\" International Conference on Machine Learning. PMLR, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion about limitations and negative societal impact is adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision was reviewed and paper accepted.

----

This paper was discussed at length between the SACs, ACs, ethics reviewers, ethics review chairs, and program chairs.  In the end, a decision was made to conditionally accept the paper.  The list of conditions for acceptance is as follows:

1. Meaningful broader impacts statement. Moving beyond superficial discussion of obvious harms into a more detailed and thorough reflection on ethical issues, especially possible broader impact and potential for misuse. 

2. Restricted release of model through some type of licensing or form-restricted access (ie. private repo accessed via request, model code and data use restricted by licenses, etc.)

3. Discussion of possible theoretical and practical mitigation strategies for minimizing harm of such technologies in the future. If this is not possible to discuss, include a clear articulation of the limits of such models in the absence of mitigation approaches. It is not necessary to implement the mitigation strategies discussed though some current work in this area should be highlighted by authors in the main text. 

The original meta-review from the AC follows.

---

This paper proposes a similarity-based information bottleneck with self-supervised representation learning and adversarial training to decompose the content and style for voice conversion. Reviewers have many discussion on the pros and cons of this work. Although some of the techniques used in the paper have been proposed before, the integration of these techniques is non-trivial and novel. Some reviewers have concerns on the experiment results, and suggest the authors could add more detailed ablation studies to verify the effectiveness of each single component in the method, which is expected to be done in the camera ready paper if being accepted. Ethics reviewer also had some concerns on ethical issues but the authors have explained that well. Overall, it is a good paper and I recommend to accept as a poster.
Decision: Accept (Poster)