Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary

AdaSpeech is a paper on practical TTS custom voice adaptation with the aim of reducing the amount of adapted parameters per voice to allow cloud serving of a large number of custom voices while maintaining high adaptation quality and similarity. The novel piece that enables this is the conditioning of layernorm in the model on the speaker embedding. The grammar reads slightly awkwardly in places, but the paper is understandable and well structured. Descriptions of the model, experiments, and analysis of results are well done.

### Recommendation 

**Weak Reject** 

I believe this paper is not novel enough / too applied / focused on experimental results for this conference. There is little discussion on the theoretical side of the acoustic condition modelling, such as how the authors are able to determine that the utterance-level and phoneme-level vectors are modelling things like room condition. Instead, the strengths of this paper are entirely through the strong numerical results. I think this paper would be a solid accept for a more specialized conference like ICASSP or Interspeech.

### Positives

1. Well written, great analysis of results and ablation studies.

### Negatives

1. What is the loss used to train the phoneme level acoustic predictor? MSE?

1. How is it determined that acoustic conditions such as loudness or room conditions are actually captured by the utterance- and phoneme-level acoustic condition modelling? My intuition would be that your phoneme-level predictor is trained only with phoneme hiddens (textual information only) (do these phoneme hiddens include speaker embedding information?), so at most it models some pitch or prosody information. The utterance-level can definitely model the rest, but where is the evidence? It could end up modelling only one very specific dimension and still improve the MOS.

1. Similarly, I highly doubt the utterance-level acoustic condition modelling does not also capture speaker information. What happens when using a speaker embedding with the utterance-level vector extracted from a reference speech for a different speaker?

1. The paper would be better with a discussion on controllability. As a reference utterance needs to be provided, does this mean the synthesized speech will take on the prosody in the reference? What happens if you want to synthesize a prosody for a speaker that's not present in any of that speaker's reference utterances? I understand this is a big topic with ongoing research which is why it would be a big bonus if this paper can make any kind of progress in that area.

1. I am curious how your phoneme-level predictor would compare with a VAE-based setup, although I understand this can be difficult to set up so no action is required here.

### Misc

**2.3 Pipeline of AdaSpeech**: "we do not use the two matrices in each conditional layer normalization" -- which two matrices?

**4.2 Method Analysis**: What does it mean to remove conditional layer normalization? Then you don't have any adaptation parameters, so is it not equivalent to Baseline (spk emb) case?


review_title: Weak Reject: Well written paper, but too applied and slightly lacking in novelty

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes AdaSpeech, a Transformer-based TTS architecture derived from FastSpeech, but multi-speaker, and focussed on the task of low-resource, robust, and low-dimensional speaker adaptation. The tactic for speaker modelling is that the speaker conditions only the scale and bias terms in the decoder. I don't have a clear intuition for exactly what kind of effect this would have on the phoneme embeddings and their mapping to spectral features, given that there are several non-linearities involved, but it certainly is a strong restriction. A global acoustic embedding conditions the decoder in addition to speaker embeddings, in the hopes of accounting for recording conditions, and, I suppose, timbre, which should then be disentangled from the linguistic information from the text in the decoder during pretraining and adaptable to new recordings at fine-tuning/inference. There is also a phoneme-level acoustic embedding which is used in the same way, which at inference is taken from random sentences (why not in training?), and, I guess, is supposed to cover phoneme-level idiosyncrasies of the speaker, although this isn't clear to me. However notice that, if I'm not mistaken, these acoustic embeddings are used zero-shot; it is *only* the speaker embedding that is the input to fine-tuning, and this only via the normalization parameters. (Both the normalization parameters *and* the speaker embedding itself are fine-tuned.) I am not sure what the speaker-embedding is left to do with all this acoustic-level input, but OK. The authors assert in section 2.2 that zero-shot is not enough, but they do not cite a paper that does exactly what they did. This would be useful, or, even more welcome, an ablation study with the acoustic embeddings but not the speaker embedding. Looking at Figure 4b just underscores this point for me. The result is that, within the margin of error, this method is just as good in terms of speaker similarity as fine-tuning the entire decoder. Having listened to the examples they gave, I do find that there are speakers for which it is clearly not as good, but this is not reflected in the evaluator's results.

Overall, this is very exciting work, as it not only promises space-efficient voice cloning, but, in doing so, suggests better disentanglement of speaker and phoneme properties in multi-speaker synthesis.
review_title: AdaSpeech review

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors present AdaSpeech, a TTS system that can adapt to a custom voice with a high quality output and a low number of additional parameters. The model is based on the TTS model in FastSpeech 2, with several additional components. The authors show that AdaSpeech has improved results over other baselines. They also provide an interesting ablation study.

Overall, the model architecture is interesting and results seem to show its validity. However, I was wondering why the authors didn't compare their results to other known multi-speaker systems (e.g. multispeech or deepvoice 2 which were mentioned in the paper). 
review_title: Interesting paper, additional comparison of results might be necessary

final_score: rating: 7: Good paper, accept
review_content: The authors propose an interesting text-to-speech adaptation method for high quality and efficient customization of new voice. The proposed method consists of two-stage modeling : multi-phonetic-level acoustic condition modeling and conditional layer normalization. In the first stage modeling, the authors proposed a new phoneme-level acoustic condition modeling in addition to the speaker and utterance-level approaches. In the second stage modeling, they employ conditional layer normalization for efficient adaptation.

Overall, I vote for ACCEPTING. The idea of TTS adaptation for customization of new voice is appealing to me. The proposed approach seems new and technically decent. Their experimental results are good and meaningful. The overall structure of the paper is systematic and well-written despite its minor errors and contains plenty of solid experimental results and discussion.

Pros
- This paper takes one important issue of current speech synthesis area: TTS adaptation to new voice.
- Its multi-phonetic-level acoustic condition modeling approach seem technically new and interesting,
- Its comprehensive experimental results well showed the effectiveness of the proposed approach.

Cons

This paper still has some issues that are conceptually not so convincing, which need to be clarified in the rebuttal session.

- In paper, it is said that the phoneme-level acoustic encoder uses phoneme-level Mel features as its input. In the inference, the phoneme-level acoustic predictor uses phoneme hiddens as its input to predict phoneme-level vectors. I think the Mel features used in phoneme-level acoustic encoder contain personal voice information. On the contrary, the phoneme hiddens used in phoneme-level acoustic predictor do not seem to contain any personal voice information because they are resulted from the phoneme encoder that uses text information only as its input in Fig 1.  Please clarify this issue.
- The overall structure of acoustic condition modeling in Figure 2 (a) is not so clear. For higher reproducibility, authors need to describe it in more detail. The addition of three, that is, speaker, utterance, and phoneme level vectors to the output of phoneme encoder can be done either in element-by-element or by concatenation.
- In section 2.3, authors described that during inference, they do not use the two matrices, but it seems difficult to understand how gamma and beta variables could be calculated using eq. 1 without them?
  
Some typos: 

- In pages 2 and 7, describe full representation of MOS, SMOS, and CMOS, respectively.
- In page 3, random chosen  -->  randomly chosen
- In pages 2, 4, while ensure -->  while ensuring

review_title: AdaSpeech: Adaptive Text to Speech for Custom Voice

Meta-review: The paper is about adapting a voice generation model to new speakers with minimal amount of training data. The key insight in this paper is that the voice can be adapted using a small set of variables -- the bias and the variance associated with the layer that normalizes the mel-spectrogram associated with the decoder. Additionally, they characterize voice at the utterance level to capture stationary factors like background acoustic conditions and at the phoneme level to capture factors such as prosody, though there are no explicit constraints to force such representation.

The strength of the paper are:
+ Simplicity of the approach
+ Empirical evaluation that demonstrates its effectiveness

The weakness of the paper are:
- analysis of what the crucial parameters of the model represent
- lack of clarity that is obvious from several back-and-forths between the reviewers and the author.

A few examples include:
- “There is also a phoneme-level acoustic embedding which is used in the same way, which at inference is taken from random sentences (why not in training?), and, I guess, is supposed to cover phoneme-level idiosyncrasies of the speaker, although this isn't clear to me.”
- “ it is only the speaker embedding that is the input to fine-tuning, and this only via the normalization parameters. (Both the normalization parameters and the speaker embedding itself are fine-tuned.) I am not sure what the speaker-embedding is left to do with all this acoustic-level input, but OK.”
Decision: Accept (Poster)