Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the problem of zero-shot style transfer for text-to-speech (TTS) synthesis of out-of-domain (OOD) custom voice.  To address the problem, the paper proposes GenerSpeech, a generalizable text-to-speech model, which models and controls the style-agnostic (linguistic content) and style-specific (speaker timber, emotion and prosody) speech variations respectively.  More specifically, mix-style layer normalization (MSLN) is proposed to eliminate the style attributes in the linguistic content representation; multi-level style adaptor is adopted for modeling the style representations at global level speaker and emotion characteristics, and the local level (utterance, phoneme and word-level) prosody representations.  Extensive experiments demonstrate the effectiveness of the proposed method in zero-shot style transfer.
strengths_and_weaknesses: Strengths:

1) Towards zero-shot style transfer of OOD voice, the paper proposes a method by decomposing the speech variations into style-agnostic and style-specific parts.  The idea is a kind of speech representation disentanglement which is quite useful for voice cloning and/or style transferring.

2) The paper incorporates several techniques to improve the generalization ability of the proposed method, including mix-style layer normalization (MSLN), multi-level style adaptor, and the flow-based post-net.

3) The samples in the provided demo webpage do show the superiority of the proposed method over the other comparison methods.  For both parallel style transfer and non-parallel transfer, the synthesized results well demonstrate the effectiveness of the proposed method in transferring speaker timbers, emotions, and prosody variations.

Weaknesses:

I don't think this paper has major flaws.  But please refer to further comments for detailed questions.

questions: 1) It is unclear how the phonetic representation $H_c$ is input to different local style encoders as the query $Q$.  Is $H_c$ phoneme level (before length regulator) or frame level (after length regulator)?  From Fig. 1(a), it seems $H_c$ is phoneme level.  If it is the case, how could the output of Style Adaptor (phoneme level) be added to the output of Content Adaptor (frame level) as the input to Mel Decoder?  The authors need to make this clearer.

2) In Fig. 1(d), it would be better if the authors could explicitly give the additional input of $Q$ (i.e. phonetic representation $H_c$) for Utterance / Phoneme / Word-level local style encoders.

3) In Section 3.3, the paper proposes mix-style layer normalization (MSLN) by considering not only the original style vector $w$ but also the shuffled style vector $ \tilde{w}$.  Experiments also validates the effectiveness of such design.  It would be better the authors give more explanations about MSLN and why it can be adopted for deriving style-agnostic representations.

4) In Section 3.4.1, from Appendix A.2.1, averaging pooling is adopted followed by two sperate full connection (FC) layers to derive the global speaker and emotion information respectively.  It is expected that the averaging pooling of the wav2vec model outputs should carry such speaker and emotion styles.  But it is unclear why wav2vec could be adopted to capture the global style characteristics.  The authors could give more explanations about such design.

5) Although I can guess what does the $S_u$ mean, the authors do not give the definition of $S_u$ in Section 3.4.2.

limitations: The authors address the impacts and limitations in Appendix G.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed GenerSpeech, which decomposed the speech variation into the style-agnostic and style-specific parts by introducing a multi-level style adaptor modeling both global speaker and emotion characteristics and the local fine-grained prosodic representations, and a generalizable content adaptor with Mix-Style Layer Normalization to eliminate style information in the linguistic content representation. The evaluations on style transfer demonstrated that GenerSpeech could synthesize high-quality speech in terms of audio quality and style similarity.

Update: the authors addressed my concerns well. I changed my ratings.
strengths_and_weaknesses: The introduced GenerSpeech model achieved better model generalization with several techniques to learn both the style-agnostic and style-specific variations in speech separately. The multi-level style adaptor could model and transfer various style attributes, including the speaker and emotion global characteristics, and the fine-grained utterance-level, phoneme-level, and word-level prosodic representations. The Mix-Style layer normalization was able to eliminate the style information in linguistic representations for improving the model generalization. GenerSpeech showed good performance in zero-shot style transfer results for OOD text-to-speech synthesis compared to other baseline methods. However, there are missing analyses about the effectiveness of the proposed style-agnostic and style-specific modules.

First, the author used shuffle operation in the Mix-Style Layer Normalization (MSLN) layer to eliminate the style information from linguistic content representation. But the study of the effectiveness of shuffle operation is missing, then how to guarantee the style information is fully disentangled and removed from the linguistic content representation by using shuffle in MSLN? Second, what's the latent representation extracted by the Style Adaptor from reference audio, such as different levels (utterance, word, phoneme) of latent representations? Do these local modules effectively capture the desired style information?

Minor: in Figure 1(e), it's helpful to mark where the Q(query) in "Style-to-Content Alignment" module comes from. Since 1(d) shows that the "Local style encoder" only takes mel-spectrogram as input.
questions: In section 3.6.1, is the pitch reconstruction loss L_p only used to train the parameters in pitch predictors, or the gradient from pitch reconstruction loss is  also used to train the rest part of the model?

In section 4.4, when performing model adaptation using different amount of data, what's the detailed fine-tuning settings for the multi-level style adaptor? For example, what are the optimization configuration, and loss functions (if there are any changes).
limitations: The authored mentioned future research directions in section 5 and potential negative societal impacts in Appendix G.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposed a non-autoregressive TTS model with good style transfer under out of domain conditions. It includes 1) a multi-level style adaptor for global styles (speaker and emotion) and local styles (utterance, phoneme and word) 2) a generalizable content adaptor with mix-style layer-normalization 3) a flow based post-net. The experimental results show the proposed method could outperform baselines compared with and demonstrate the efficacy of the proposed method.
strengths_and_weaknesses: Strengths:
1. Proposing a layer-normalized mix-style for better generalization
2. A style-specific module to integrate different style information. 
3. Including a flow-based post-net on the top of the fast speech 2 model
4. Detailed comparison with different baselines.

Weaknesses:
1. Despite the authors' lengthy appendix, which provides more information about the experiments,  some critical details of the proposed method are still missing. For example, what is the shape of $Su$ after VQ? Is it a vector or a sequence of embedding vectors? If it is a sequence of embedding vectors, is there any constraint applied to make them identical or close?
2. Some abbreviations are used without definition, such as USE, PSE and WSE in Table 3.
questions: 
 After listening the sample audio clips, one impression I have the style transfer from the expressive FS2 is as good as the proposed GenerSpeech. One difference I noticed is the synthesized audio quality. Could we add another experiment to compare the expressive FS2 with a flow-based post-net?

What is the shape of $Su$ after VQ? Is it a vector or a sequence of embedding vectors? If it is a sequence of embedding vectors, is there any constraint applied to make them identical or close?

The local multi-layer style encoders share the common architecture, are they all use VQ code size=128? How do we choose this number?

What's the purpose to include positional encoding embedding to the style representation before they are fed into the attention module?
limitations: The authors have addressed that in section 5.
review_title: null

Meta-review: metareview: All 3 reviewers agree that the paper is novel, technically strong and experimentally convincing. THis paper should be accepted.
confidence: Certain
Decision: Accept