Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces ProsodyBERT for learning prosody representations to strengthen the prosodic aspects in unsupervised text-to-speech synthesis (TTS). Similar to HuBERT (Hsu et al, 2021), ProsodyBERT is a masked encoder model trained to predict K-means cluster labels, except in this case the K-means is performed on domain-informed features (such as pitch, energy, and dynamics, etc) to capture the long-term temporal structure of prosody. These learned hidden features then become targets for training a prosody predictor, and then downstream a duration predictor. To evaluate the effectiveness of ProsodyBERT, the paper demonstrates expressive TTS, interpolation between speaker’s prosodic styles, and how ProsodyBERT features improve emotion recognition in speech. 

strength_and_weaknesses: Strengths

This paper provides a targeted and effective solution for modeling prosody, which is crucial to synthesizing expressive and natural speech. The overall design of the modeling, training and inference pipeline is well-thought-out, leveraging strong prior work as building blocks, such as HuBERT (Hsu et al, 2021) and SpanBERT (Joshi et al, 2020) for prosody representation learning and UTTS (Lian et al, 2022a,b) as the unsupervised text-to-speech synthesis, and demonstrates how ProsodyBERT representations can be added to strength prosody generation. The provided synthesis examples are compelling, and the improvement on the emotion recognition task shows that the choice of features and learned representations are useful.

Weaknesses 

Since the main contribution of the paper is on learning prosody representations and predicting prosodic features for synthesis, it could be helpful to provide more grounding on what prosody is, and how it motivates subsequent design choices in modeling. Informally, prosody is both a general phenomenon, could also be a person’s signature way of speaking. What is the dependency structure between style, speaker, prosody, duration? How does the paper choose its current ordering of conditioning and predicting them? In the appendix, zero-shot speaker synthesis was mentioned as a motivating factor for prosody features to be not dependent on speaker. Perhaps this is a design choice inherited from prior work that focuses on the zero-shot setup that is also well-suited for the current setting? 

For comparison to prior work, it could be helpful to have a paragraph discussing how the baselines are chosen, what other models are considered but may not be as relevant, or practically not feasible to compare to. In what ways it may or may not be an apples-to-apples comparison. For example, what are the model sizes involved, which may be a result of the different design objectives of the prior models? 

Some examples seem to be missing from the demo page, for example “emotion control” examples, and there’s only one example for “expressiveness control”. For the “expressive TTS” examples, would it be informative to include a few examples for each text to show the range of expressiveness that is possible. It seems all the text sentences are from the original dataset. How does the approach perform on unseen text?

clarity,_quality,_novelty_and_reproducibility: Clarity

The paper is overall well-written. As it builds on a lot of prior work, at times it becomes understandably hard to be self-contained, and some components of the modeling are only mentioned in passing. For example, what is “style”, in addition to speaker? What is z_align? At times, it took a while to figure out what was proposed in prior work and what was new. For Figure 2, would it be possible to visually distinguish which modules are new and which modules and pipeline are building on prior work. The current dotted-lined box in Figure 2(a) is helpful. Is Figure 2(c) a zoomed in illustration of the “shared predictor” module in Figure 2(b)? 

Clarification questions on the K-means procedure: What is the dimensionality of the extracted features, and how are they weighted when performing K-means? Was there a need to explore different weightings or dimensionality reduction? How are the interdependencies between the different features accounted for? What is the actual dimensionality that the K-means was performed in? 

Quality

The proposed approach is well-thought-out. The experiments include both quantitative, qualitative and subjective evaluations. Figures such as Figure 4 helps the reader understand the differences in what the approaches are able to capture, which is very helpful. More discussion on comparison to prior work and samples could strengthen the quality of the work.

Novelty 

In contrast to prior work that used mostly pitch and energy when considering prosody, this paper proposes to use more expressive domain-informed features to better capture prosody, and to model it using a HuBERT-like approach, making it possible to learn general prosody from large amounts of (unpaired) speech audio.

Reproducibility 

The description of ProsodyBERT is detailed, and the TTS pipeline builds on prior published work. The code for ProsodyBERT is included in the supplementary material, and it was mentioned in the paper that the checkpoint will be released at a later point. 

summary_of_the_review: The paper brings to attention the importance of prosody, and provides an effective approach to modeling it and adding it to unsupervised TTS systems. The approach is sound, and the results sound compelling. More discussion on the evaluation and comparison to prior work could help position this work and its contributions. A more complete demo page would also strengthen the paper’s contributions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work explores utilizing self-supervised pretraining on speech data for improving prosody modeling in TTS and emotion recognization tasks. The proposed model, ProsodyBERT, is similar to HuBERT / SpanBERT, but with more input features added (f0, energy, NCCF). Experiments are conducted on two tasks: TTS (on DailyTalk + VCTK datasets) and emotion recognition (on IEMOCAP dataset).
strength_and_weaknesses: Strength:
 - The idea of this work is sound
 - The experimental results looks interesting

Weaknesses
 - The novelty is limited.
 - Technical correctness issues, as explained in the next section.
clarity,_quality,_novelty_and_reproducibility: Clarity:
 - Writing style needs improvement. For example, the correct use of `\citep{}` and `\citet{}`
 - It's not clear how "6.1.2 EXPRESSIVENESS CONTROL" is conducted.

Quality: 
 - I have concerns on the technical correctness. For examples:
   - This work claims to learn prosody representation. However, key prosody components, e.g. pace / rhythm, are not concerned in the proposed model.
   - The TTS experiment used FastSpeech 2 as one of the baseline, and claimed that used "the official FastSpeech 2 checkpoint". However, there is no such "official FastSpeech 2 checkpoint" ever released.
   - Even if the authors used a non-official third-party FastSpeech 2 checkpoint, it's unlikely that the checkpoint was trained on the same dataset as used in this work (DailyTalk), which is not a popular TTS dataset. As such, the comparison in Table 1 is not fair at all.
   - "The most commonly used prosody representations include acoustic attributes like fundamental frequency (F0), energy, and duration." Please provide references.
   - Fig 2 doesn't make well sense -- it looks like the TTS prediction only depends on z_speaker, z_align, z_prosody but nothing else. How about content?
 - I also have other concerns on the experiment setup. 
   - For the TTS experiment -- are the improvement from using extra features extracted by another model pre-trained with extra data, or from using extra features extracted by ProsodyBERT pre-trained with extra data? A comparison between UTTS + HuBERT and UTTS + ProsodyBERT would be required to show the improvement are from ProsodyBERT.
 
Novelty:
 - Limited. The proposed model is similar to HuBERT / SpanBERT, except for adding more input features (f0, energy, NCCF).
summary_of_the_review: This paper tackle on an important problem (self-supervised prosody modeling). However, the contribution from the works is limited, and there are issues on the technical correctness.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a BERT-based architecture for extracting prosody embeddings from speech. The pre-training of the model is performed following a self-supervised strategy similar to that of HuBERT but based on prosody features to get the cluster labels. The performance of the proposed approach is evaluated in an unsupervised Text-To-Speech Task proposed in previously published work.
strength_and_weaknesses: The paper exploits recent ideas in speech processing and text-to-speech technologies, to propose a new and promising idea to incorporate prosodic information into speech synthesis/generation tasks. The proposed approach is theoretical well supported, and the authors provided experiments and results with enough empirical evidence of the improvements achieved by their approach. Nice work.
clarity,_quality,_novelty_and_reproducibility: The paper is clear even though the complexity of the proposed approach is high. The novelty is clear, both in terms of the methods and performance improvement in unsupervised TTS. The proposed approach is not easily reproducible due to the inherent complexity of the proposed approach, the multiple stages of the pipeline, and the complexity of the task under study. The authors included a statement with a pledge to share checkpoint models and code with hyperparameters values and source code for training. Either way, the stage related to phone and word alignment could be better explained. 
summary_of_the_review: The paper proposed a BERT-based architecture and a self-supervised training strategy for prosodic-related embedding representation extraction from speech signals. The prosodic learned features are used in an unsupervised TTS task yielding a real improvement in terms of Mean Opinion Score and an ASR Intelligibility metric (Word Error Rate). The proposed approach is built on top of recent advances in speech processing and unsupervised TTS technologies and proposed new ideas to incorporate prosodic features to enhance the natural and expressive of synthetic speech.

This is a great work, congratulations to the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The following work proposes a self-supervised representation for learning. Unlike prior methods, the proposed framework relies only on raw audio. Feature vectors are extracted based on a combination of NCCF, log F0, delta log F0, and average energy per frame. The resulting N-D feature vector is k-means clustered to achieve discrete codes. Self-supervised training objectives from HuBERT and SPANBert are then used to learn the final representation. The resulting features are then applied to downstream unsupervised TTS and emotion recognition.
strength_and_weaknesses: Strengths:
- Fairly simple setup to achieve explicit incorporation of prosodic features
- Demonstrated applicability in several downstream tasks

Weaknesses:
- While I have no doubt their setup works in practice, I do have some concerns regarding the K-means clustering.
  - Not all frames contain valid F0 values. It's not clear how this is being handled, but constants and interpolated values will introduce devations from the true data distribution
  - It's also not clear how to best combine these individual attributes. Scaling individual features can have drastic effects on which one dominates the clustering task.
- I don't think there's an ablation on the SpanBert objective
- Error bars missing in Table 1. Furthermore, MOS is not a good metric for comparing fine grained differences between various models compared to performing pairwise preference testing.
- While there is no known good way to truly measure the quality of synthesized F0 and energy, I would be very careful about making claims backed up by a single qualitative comparison in Figure 4.
- I'm not sure I follow the logic behind the claim that wave2vec, hubert, and waveLM filter out long-range prosodic information.
- It's not fully clear how necessary the large transformer architecture with the self supervised objectives matter at all for some of the downstream tasks. I suppose an even simpler baseline would've been to individually quantize each prosodic feature into K bins using K-means (separately per feature). Then for each downstream task, pair each 1-hot encoding with its own embedding matrix to map to a lower dimensional representation. Alternatively, perhaps for the emotion recognition task, one could even just directly condition on the raw values like in Mellotron (Valle 2020), appending the user-normalized raw values with HuBERT features. 

Questions for clarity:
  - Can you clarify how EER is computed?
clarity,_quality,_novelty_and_reproducibility: Overall, the technical portion of this work was fairly easy to follow. I generally trust that improvements have been made. I can certainly hear some improvements in the provided audio samples.

I don't think the novelty of this work is very high. Algorithmically, vector quantization (k-means) has been explored and used for resynthesis in [1]. This work takes it a bit further by incorporating additional features such as log F0, energy, and NCCF.

[1] Polyak et al.  Resynthesis from Discrete Disentangled Self-Supervised Representations. Interspeech 2021
  
summary_of_the_review: Overall, easy to read paper with some promising results. However, I think the novelty is somewhat low and the complexity of the solution (large transformers with self supervised learning objectives) might not be necessary for some of the downstream tasks demonstrated in this work. However, I'm certainly happy to change my review if the authors can demonstrate otherwise.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper uses low-frequency Mel bins, f0, energy as input to train a HuBERT, and term the model ProsodyBERT. The paper claims the learned representation is disentangled and contains prosody information. To validate the claim, the paper includes experimental results on TTS and emotion recognition.

Overall, the reviewers are not convinced that experimental results support the claim that the representation is disentangled and contains prosody information. The paper does not have a definition of disentanglement, nor does it evaluate disentanglement. The paper also does not evaluate what information is in the representation. An improvement in emotion recognition might be attributed to something else, either in the representation or due to the modification of the system. This is questioned by reviewer 925W for the lack of grounding.

The TTS results are further questioned by reviewer J32w. The system is not a mainstream TTS system, and there are no proper baseline for the TTS system (for example, anchoring on other representation such as the one from HuBERT).

Both reviewers J32w and CWFe are concerned about the design decisions and how they impact the results.

The paper can be improved by having fine-grained probing tasks for evaluating the representation, experimenting on a more mainstream TTS system, and doing careful ablation studies on the design decisions.

Reviewer PaY1 gives a high score, but does not include sufficient evidence why the paper warrants the high score.
justification_for_why_not_higher_score: The claims of the paper (even iterated by the authors several times in their replies as well) are not justified by their experiments.
Decision: Reject