Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed NANSY++, a unified framework of synthesizing and manipulating voice signals from analysis features. The backbone network of NANSY++ is trained in a self-supervised way. After the backbone network is pre-trained, different voice applications can be adapted by modeling the analysis features required for each task. Experiments show that the proposed NANSY++ framework has the advantages of controllability, data efficiency, fast convergence, and high quality synthesis.
strength_and_weaknesses: It's impressive that NANSY++ can use one unified framework to handle four different speech tasks while achieving good performance in audio quality, data efficiency and controllability. The framework is well designed in a modularized way and can disentangle various speech representations. One drawback of this paper, however, is limited ablation study. Since the proposed NANSY++ framework requires a variety of modules and augmentations, it's hard to know what are the core factors that contribute to the model performance.

Several detailed questions:

In section 2.2, for the linguistic feature disentanglement, what's the exact equations of breathiness perturbation?

In section 2.3, are the global and time-varying timbre embeddings also trained in a self-supervised way or trained jointly with the waveform reconstruction? If they are jointly trained, how to guarantee the time-varying timbre embeddings don't contain other information such as pitch and linguistic?

In section 4.2.2, what's the problem definition here for zero-shot TTS? Does the mean the model trained on TTS task can be adapted to a new dataset or the pre-trained backbone model can be used directly for TTS task?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and technically solid. However, ablation study is largely missing. 
summary_of_the_review: The proposed unified voice synthesis framework NANSY++ achieved impressive performance and can serve as a general speech pre-processing and pre-training pipeline.


=================

Updates after reading the author's rebuttal: the author addressed my concerns well, so I updated my rating to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper describes a learned analyzer and synthesizer for speech that can be trained end-to-end, and demonstrates their utility in several speech and singing synthesis tasks: voice conversion to match a target individual, voice conversion to match target characteristics, text to speech, and singing voice synthesis. In listening tests for the targeted voice conversion, NANSY++ exceeded YourTTS  by 0.56 MOS points (from 1-5 range) in quality and 0.38 points (1-5 range) in speaker similarity. For TTS, compared with Glow-TTS, the proposed approach performs better in quality and character error rate of a pre-trained recognizer at various amounts of training data from a single speaker (5 mins, 10 mins, 30 mins, and 30 hours). For singing voice synthesis, the proposed approach out performed diffsinger in terms of subjective quality with various amounts of training data. And for voice conversion to match target characteristics, the proposed approach was able to generate 320 speakers that achieved an objective voice diversity similar to that of 240 actual different speakers.
strength_and_weaknesses: Strengths:
* Well executed experiments: the experiments are all clearly described, well performed, and test key claims of the paper
* Approach works well. The results of those experiments show that the proposed method does provide superior performance to the various baseline approaches.
* Model provides general utility. As demonstrated by the variety of experiments conducted, the proposed model is useful in a large number of situations without requiring a large amount of data or a large amount of time to retrain. For example, for TTS, between 30 minutes and 30 hours of data, the model improves in MOS from 3.64 to 4.07 and in CER from 2.20 to 1.68%. These are meaningful improvements, but the model trained on 30 minutes is still performing quite well and fairly close to the model trained on 30 hours.
* General clarity of presentation. The paper is generally well presented, except for the issue of certain claims being slightly over stated mentioned below under weaknesses.

Weaknesses:
* The captions of the figures and tables are too brief and should be expanded so that figures are clearly interpretable by themselves without referring to the text.
* Certain claims are slightly overstated, although these claims are also not especially necessary for the work to have impact. In particular, I am referring to claims that the model is disentangling various voice characteristics and that it is self-supervised. While both of these may be technically true, it is thanks to careful inductive biases baked into the model architecture. Thus it is not especially surprising that the model has these characteristics, but it's not like they just popped out of the data. 
clarity,_quality,_novelty_and_reproducibility: See above, but generally the paper is clearly written and high quality experiments. One minor comment is that I would recommend avoiding the acronym "VD" for "voice designing" as there is already an unpleasant meaning to this acronym.

In terms of novelty, the approach is based upon an earlier work, NANSY (Choi et al, 2021b), but adds the trainable transformations in the analysis block, which enables the end-to-end training. This paper also applies it to a number of new tasks, which were not necessarily easy to target in the earlier version.

On reproducibility, the paper provides details of all of the architectures as well as hyperparameters in the appendix. Datasets are publicly available. Code is not released because of concerns about the potential of the model to be misused.
summary_of_the_review: End-to-end trainable voice analysis-synthesis pipeline that cleverly separates various voice characteristics through relative contrasts. Experiments show that it is useful and data efficient in several different tasks. Minor issues with clarity of captions and with certain claims being slightly overstated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a unified framework for human voice synthesis based on neural analysis and synthesis modules. The analysis part extracts disentangled and controllable voice features based on domain knowledge (pitch, periodicity/aperiodicity and timbre). Each of them is trained independently with a dedicated neural analysis module. The timbre analysis module divided the feature into a global timbre embedding vector and time-varying timbre tokens.  The synthesis part take the voice features and generates waveforms with frame-level and sample-level synthesis modules. The framework is evaluated in terms of F0 estimation and reconstruction error as a vocoder, showing superior results.  The framework is applied to four voice synthesis tasks including voice conversion, TTS, singing voice synthesis, and voice design, along with additional encoder modules. In each task, the proposed framework outperform previous state-of-the-arts. 



strength_and_weaknesses: Strengths
- The proposed framework is highly modular and flexible, being plugged in various voice synthesis tasks
- The framework is interpretable as it is designed to be similar to DSP-based parametric vocoders. 
- They achieve impressive results in all tasks, outperforming compared models.   

Weakness
- The paper includes too much content. Due to the space issue, all details are moved to the appendix.  
- Although the proposed framework is well-structured, it is extended from the authors' previous work (NANCY) and many new components are adopted from other previous work (self-supervised pitch estimation, linguistic representation, parallel WaveGAN, and so on). 
- While the framework achieves better performances than previous work in many tasks, the experiment condition is not completely fair because the NANCY framework was pretrained with a large-scale dataset (10,571 hours from 6,176 speakers and 624 singers).  
- The time-varying timbre embedding is a great idea but it was not validated through an ablation study. 

clarity,_quality,_novelty_and_reproducibility: Clarity 
- The paper is easy to read in general. But, all details are located in the appendix section. I am not sure this type of writing structure is appropriate as a conference paper. 
- Since the framework include a lot of components, the authors should clarify which part is novel and how they affect the model performance.   

Quality 
- The model quality was well evaluated on the backbone experiment and each application task with MOS, SSIM, CER and more. 
- F0 estimation was evaluated only with a listening test for the reconstruction quality. It could be also evaluated in terms of F0 estimation accuracy. 

Novelty 
- While a lot of components are borrowed from previous work, there are also some new ideas such as time-varying timbre embeddings. But, the novel part was not clearly validated. For example, the time-varying timbre embeddings could be evaluated in the reconstruction experiment (section 3.2) as an ablation study.

Reproducibility
- The model architecture is well delineated in the appendix section. 
- However, the detail of the training data is missing. Information on gender, age, language, professionality (they include voice actors or professional singers, etc.) would be helpful to understand the model training. 

Minor comments 
- (page 2) the sinusoidal waveform equation ( x[t] = Ap[t] sin (...)) has the summation inside the sine function. Is it correct? The summation should be moved outside the sine function?  

- (page 3) In equation (1), "cossim" could be replaced with a short symbol such as "d". In page 9, they use the symbol "d" to denote the cosine distance. Please make them consistent. 

- (page 4) what is "simple spectrogram"?  Is it the spectrogram with the linear frequency scale?

 
summary_of_the_review: This paper proposes a flexible voice synthesis framework with high-performance. The framework include robust F0 estimation and high-quality reconstruction as a vocoder.  The flexible structure was applied to voice conversion, TTS, singing voice synthesis and new voice design. I believe that the framework is highly well designed and engineered. 

One main issue of this paper is the writing structure. The main body is a summary of long manuscripts on "a large system" and all details are moved to the appendix section. I feel like it would be better for the authors to submit this paper to a journal which allows many pages. 

In addition, the novel part such as time-varying timbre embedding was not thoroughly validated, which might be probably due to the space issue. 

Lastly, the high-performance of the framework is attributed to not only the model architecture but also the large-scale training data. This should be clearly indicated when compared to other models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - Authors propose a unified framework to learn many voice synthesis tasks. The unified framework includes task-independent self-supervised training of a backbone network, which does not require any labelled data, and then task specific training with labelled data.
- Authors claim that this unified approach offers advantages such as controllability of synthesized speech, data efficiency for downstream tasks, and fast training convergence — all this without sacrificing the quality of synthesized speech.
- Authors assume that the most of the voice synthesis tasks can be defined synthesizing and controlling our aspects of voice, that is, pitch, amplitude, linguistic, and timbre.
- For estimating pitch, in an unsupervised fashion, authors use the property of CQT input representation, and the transforming the representation appropriately to get a known shift in the fundamental frequency value. While this will bias learning of fundamental frequency like value, it is not clear how authors obtain absolute fundamental frequency. This method is inspired by “SPICE: Self-supervised Pitch Estimation”.
- For getting linguistic features, authors propose to use a contrastive learning approach where they modify the signals in a way to keep the linguistic information intact, while changing other properties of the signal — and use this property learn features which maximizes similarity if the source audio (i.e. before transformations) is the same i.e positive samples in the contrastive learning approach are two different transformations of the same signal and negative samples are the transformations of two different signals. Negative samples clearly have different linguistic information, but it is not clear that they will share the same non-linguistic features — maybe that is not necessary for this approach to work. Authors extract wav2vec features from transformed signal, which is transformed via another learnable network. I am curious about the importance of this contrastive learning approach vs directly using wav2vec features.
- Authors propose learnable vector encoding of input signal — one global vector and then time varying tokens — to represent timbre. It is not clear why these learnable encodings will necessarily encode timbre, and will necessarily be disentangled from pitch and linguistic representation.
- Authors propose to use Parallel WaveGan trained via reconstruction loss  on mel-spectrogram, and adversarial and feature matching loss to train the synthesizer network.
- Experiments:
    - Authors shows that replacing fundamental frequency extracted from the analysis network by the fundamental frequency extracted from four other approaches, and show that the synthezised audio using their synthesizer is assessed to match the original signal before doing the f0 replacement. It is not clear why this implies that their analyser learns better f0 — it is totally possible and likely that even if their analyser is not learning the correct f0, but since the representation is trained end-to-end along with synthesizer, reconstructions with this representation are closer to original, and using a more correct (or even a different f0) will throw the model off.
    - Authors show better reconstruction quality than a mel2wav architecture namely HiFiGAN. Since the input representations are different, specifically authors choice of input is much more rich than mel-spectrogram used in mel2wav setup. It is not clear why having better reconstruction is a good metric to measure in this case.
    - Authors show clear improvement in zero-shot voice conversion, when compared with the baseline.
    - Authors show better performance than the baseline on TTS and Singing voice synthesis. The baseline for full data seem far from real audio, which probably points to the baseline being weaker than SOTA TTS.
    - Authors show good results on Voice Designing.
strength_and_weaknesses: - Overall:
    - Authors claim about disentangled representation learning of f0, timbre, linguistic information might be misleading. I wonder if authors will consider changing the wording to showcase what design choices they make that promote learning disentangled representation, and they verify experimentally that some of it is achieved.
    - Experimental results on downstream tasks seems very good. The impact of the work can be maximized by making sure that the baseline performance is replicated from original work e.g. baseline TTS on full data, seems to be very far from real data MOS, which does not reflect the SOTA on TTS tasks.

Strengths:
- Extensive evaluation
- Impressive Results

Weakness:
- Not so great scientific rigor
- Not so-great choice of baselines
- Potentially misleading claims or just confusing to understand what is claimed vs what is expected.
clarity,_quality,_novelty_and_reproducibility: The paper's writing can be improved by clearly stating assumptions -- revisiting this assumption in the experimental sections and verifying them. Stating what is claimed, vs which choices are made to encourage learning specific kind of features, even when they are not guaranteed to happen, will be useful.

The paper re-uses components from other papers and cites them. Paper has limited novelty in terms of methods proposed but seems to be a useful well-engineered system.
summary_of_the_review: Overall, the authors should reconsider claims about disentangled representation learning. Should consider doing ablations for various choices, and improve clarity of the writing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: NANSY++ describes a learned analyzer and synthesizer for speech that can be trained end-to-end in a self-supervised way and demonstrates their utility in several speech synthesis tasks: voice conversion to match a target individual, voice conversion to match target characteristics, text-to-speech, and singing voice synthesis. The results of those experiments show that the proposed method does provide superior performance to the various baseline approaches. The results are impressive. 

In terms of novelty, using the analyzer and synthesizer on various synthesis tasks is novel. The learning of analyzer and synthesizer is based upon an earlier work, NANSY (https://arxiv.org/abs/2110.14513). Many new components are adopted from other previous work (self-supervised pitch estimation, linguistic representation, parallel WaveGAN, and so on). The training difference between the backbone self-supervised models of NANSY and NANSY++ includes: a. NANSY++ can be end-to-end trained, b. NANSY++ directly generates waveforms, and c. time-varying timbre embeddings, etc. The authors should specify the main difference between NANSY and NANSY++ in the introduction.

A weakness of the paper is that which components in NANSY++ make it has outstanding performance is not validated. The authors' feedback does not address the issue. The authors claimed they do not have sufficient computing resources to do all the ablations. NANSY and NANSY++ have been conducted both on VC, so at least they can be compared on VC, but the authors did not do that.  

The reproducibility of the paper is low. Although the authors provide details of all the architectures and hyperparameters in the appendix, the dataset for pre-training is not publicly available, and the code is not released (because of concerns about model misuse). Since NANSY++ is pre-trained from a vast high-quality in-house dataset (in downstream tasks, the authors did use public datasets), it is hard to tell the outstanding performance of NANSY++ is from the proposed approach or simply a good dataset. I suggest the authors pre-train NANSY++ on the public dataset and report the results. 

I suggest the authors carefully check their baseline performances. For example, the MOS of YourTTS is close to the real speech in the original paper (https://arxiv.org/abs/2112.02418). But its MOS is far away from the real data in this paper.

justification_for_why_not_higher_score: The major problem of NANSY++ is that the paper needs to include ablation studies of each component. The authors proposed a set of inventions, and all of them together achieved an outstanding performance. However, we need to find out which innovation influences the results in which ways. 

In addition, the performance of TTS usually highly depends on data. NANSY++ used a large in-house dataset for pre-training (the authors did use public data sets in downstream tasks, but I am talking about pre-training here). The excellent performance can even come from the enormous in-house dataset for pre-training, not related to the invention in NANSY++. Therefore, it is unclear whether the proposed approaches in NANSY++ are truly helpful or not. 

From this point of view, the experiments have not yet confirmed the paper's technical contributions. So I am at the border between acceptance and rejection. "Accept (poster)" is the best I can recommend.
justification_for_why_not_lower_score: I am hesitating between acceptance and rejection. NANSY++ achieves outstanding performance. However, due to the lack of an ablation study, we do not know which innovation in NANSY++ is helpful, and we need to know whether it is better than its previous version, NANSY, but I cannot find the comparison in the paper. However, using a single self-supervised model to achieve a wide range of speech synthesis tasks is novel from my viewpoint. I think it is good to let the world know that a general framework for speech synthesis based on self-supervised learning is feasible. In the end, I recommend the paper as accepted. However, I am okay with the decision if the senior meta reviewer thinks it should be rejected.
Decision: Accept: poster