Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposed a solution to the practical problem of contextual speech synthesis to address TTS model limitation for highly phonetically contextual languages such as Chinese. The paper employs the use of conformer architecture baseline and introduces 3 main changes to enable paragraph-level expressiveness - segment-level memory reuse using RNN based on Transformer-XL architecture,  text-based contextual embedding for both text and statistical features, as well as linearized self-attention to reduce transformer computational complexity. For the contextual text embedding, the context incorporates both the previous and future sentences in order to improve synthesis quality. Experimental results shows that the proposed contextual TTS performs better than the ConformerTTS baseline in terms of MOS score and performs close to the original recording as judged by human evaluators. Analysis also shows that the proposed architecture shows significant MOS improvement for paragraphs with both extra- long and short sentences with the extra-long sentences showing better improvement. The use of of inter-sentence memory reuse is also shown to significantly reduce the latency by 4x - 10x. Ablation results also show that each of the additional changes to the conformer TTS baseline contributes to the improved synthesis quality. 
strength_and_weaknesses: The paper is well motivated as it's attempting to solve practical problem of speech synthesis for highly phonetically contextual languages. The architectural changes are intuitive and the experimental results in terms of improved synthesis and reduced computational complexity support the changes.

The main weakness is the writing style as it gets winding sometimes. There are several repetitions of the same concept, perhaps to remind the readers about the authors' contributions but it doesn't detract from the technical contribution of the paper. Also, using pairwise comparison (PMOS) has been shown to be a better comparison for speech synthesis models than just MOS. It might also be good if another phonetically contextual languages is evaluated. English might be a good choice given the availability of datasets although it might not be as phonetically contextual as Chinese.
clarity,_quality,_novelty_and_reproducibility: The paper's contributions are novel and the experimental and ablation results are interesting. However, I would recommend that the authors trim things down a bit to improve the clarity and flow of the article.
summary_of_the_review: The paper addressed a practical problem facing speech synthesis for highly phonetically contextual languages such as Chinese. The proposed architectural changes are intuitive and novel, and the experimental and ablation results supports the proposed changes.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents ContextSpeech, a text-to-speech synthesis model that additionally takes the embedding of the previous sentence as input to improve prosody modeling for paragraph reading. The authors also utilize efficient attention to improve efficiency.
strength_and_weaknesses: Strengths
- This paper tackles an important problem that has not received enough attention yet.
- The proposed method outperforms a simple baseline that do not have previous sentence as input

Weakness
- Presentation of this paper requires improvement. The current draft is barely readable
  - `\citep` should be used instead of `\citet` in most of the places to improve readability.
  - the diagram is not very clear or useful. It is unclear what Fig 1 (b) corresponds to in Fig 1 (a), and how “paragraph text” is different from “previous segment”. It is also unclear why there is a connection from “previous segment” to “current segment” in Fig 1 (b). 
  - “where $t$ is the index of current sequence” is ambiguous. Does $t$ index tokens within a segment, or does it index segments?
  - It is unclear where the previous segment is used in Eq 1, which reads like a skip connection instead.
  - Fig 2 is too complicated and again unclear where it refers to in Fig 1 (a) / (b). How are F0-F5 used? What does “F0 = $i_{t_s}$ / $n_{t_s}$ = the i-th token / the number tokens in a sentence” mean? 
  - Does “concat” and “Average Pooling” operate on $[E_{t-c}, \cdots E_{t+c}]$ or just E_t?
- This paper does not compare with any prior work that addresses context modeling, such as Wang et al. (2020). Instead, it only compares  with a model that does not have context from previous segments as input. The experiments are not sufficient

clarity,_quality,_novelty_and_reproducibility: It requires substantial improvement for readability and clarity of the proposed method. Novelty also seems limited considering this is an ensemble of existing techniques such as efficient attention. Reproducing the experiments would be hard given the lack of details and clarity.

summary_of_the_review: The proposed methods are not described clearly. Novelty is limited. Experiments are insufficient.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of synthesizing extra-long text inputs such as a paragraph-level inputs for TTS tasks. The proposed method relies on the Transformer-XL like approach to capture the inter-sentence information and integrates BERT-based context information for both training and synthesis. The authors further employ a linearized self-attention module to reduce the overall memory cost and computational cost. Experimental results on an internal dataset confirms the effectiveness and efficiency of the proposed method.
strength_and_weaknesses: Strength:
The topic this paper studies is very interesting. I definitely agree that the context learning of TTS is not yet well-developed, and making progress on context learning will shed light on the some promising directions for natural speech synthesizing.

Weaknesses:
In section 2.1, "But the expressiveness is limited due to teacher-student distillation",  FastSpeech 2 requires no distillation.

The vocoder is not presented, I am curious whether the vocoder has a similar recurrent mechanism like Transformer-XL over the decoding process? If not, how do you join the output audios together?

The backbone of the proposed network is not well-presented. Is this model based on FastSpeech, or FastSpeech 2? The main body of the paper and Figure 1 clearly state that this model is based on FastSpeech. However, in the second paragraph of section 4.1, the authors present that the variance adapter is included which indicates that this model is based on FastSpeech 2. I am very curious whether the baseline model also uses FastSpeech 2 for a fair comparison?

The autoregressive models such as Tacotron 2 should be included in the comparison. Despite that AR models often suffer from relatively slower inference speed, it is computationally cheaper and suitable for synthesizing very long text.

The novelty of the method is limited since the Transformer-XL based TTS model has been discussed before.

The baseline model is weak, some modern TTS models such as FastSpeech 2, GlowTTS, VITS should be included in the comparison.

Typo: in the second paragraph of section 4.1 and the first paragraph of section 4.4, "ContexSpeech" should be  "ContextSpeech".

Missing a most relevant reference:
L. Xue, F. K. Soong, S. Zhang and L. Xie, "ParaTTS: Learning Linguistic and Prosodic Cross-Sentence Information in Paragraph-Based TTS," in IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2854-2864, 2022.

clarity,_quality,_novelty_and_reproducibility: The contributions are somewhat new. Aspects of the contributions exist in prior work. Some important details are missing.

summary_of_the_review: While the paper’s premise is interesting, the submission suffers from poor presentation quality. and I’m not entirely convinced by the authors' claims when considering that some modern TTS models are not included for comparison. I also have concerns with the novelty of this work since the Transformer-XL based models and BERT-based context learning have been discussed from the prior works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes ContextSpeech, a text-to-speech (TTS) system that models the contextual information in a paragraph for coherence and expressiveness, which constructs text-based semantic information in a hierarchical structure. At the same time, ContextSpeech doesn't largely increase the computation or memory cost by introducing a memory-cached recurrence mechanism. ContextSpeech improves the paragraph-level speech quality and prosody expressiveness in terms of subjective and objective evaluation metrics.
strength_and_weaknesses: ContextSpeech is a nice combination of engineering and research efforts. By making use of caching mechanism and linearized self-attention module with corresponding permute-based relative position encoding, ContextSpeech can capture previous information with reduced computation cost and memory consumption. For the context awareness, ContextSpeech uses BERT embedding combined with pre-defined statistical information from contextual scripts, which broadens the model receptive field.

ContextSpeech has improved the paragraph-level prosody expressiveness compared to the baseline model. It generates high quality speech samples in terms of naturalness, intonation, stress and style. Also, the adapted speech rate makes extra-short utterance sound much more natural and close to human reading.
clarity,_quality,_novelty_and_reproducibility: The paper is technically solid and well written.

Some minor typos:
1. section 3.2: we use a BERTDevlin et al. -> we use a BERT Devlin et al.
2. “character” (fullwidth) -> "character" (halfwidth)

summary_of_the_review: ContextSpeech achieves good performance in paragraph reading, including audio quality and inference speed. And the paper is well written. However, in consideration of the fact that most of the modules and algorithms in the pipeline are from previous work, my rating is also adjusted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: # Summary
This paper is focused on improving the quality of speech synthesis at the paragraph level. It is well known that synthesizing paragraphs sentence-by-sentence does not lead to acceptable results due to inconsistent expressiveness in the synthesis of successive sentences, missing paragraph level context leading to poor renditions of prosody, and trouble synthesizing especially short or especially long sentences. This paper attempts to improve paragraph-scale TTS by providing a segment-level recurrent memory (similar to Transformer-XL) in the encoder and decoder, which provides previous segment context to shape the synthesis of the current segment, and also text encodings that capture characteristics of the surrounding sentence context. To reduce the computational costs of introducing longer context into the synthesis process, a linearized self-attention mechanism with a permutation-based relative positional encoding are used. Experiments on a Chinese audiobook collection illustrate the advantages of the proposed approach.

# Strengths
- The paper is addressing an important research question, and the approach explored in the paper is well motivated.
- The engineering optimizations used in the paper (e.g., linearized self-attention and the specific relative positional encoding) are potentially useful in other TTS systems.
  
# Weaknesses
- VITS, as the current state-of-the-art in TTS, should have been included in the evaluation for comparison. Because ConformerTTS is less well known, it is difficult for readers to judge the significance of the improvements over that particular baseline.
- The writing needs to be improved: ConformerTTS is based on FastSpeech2, but the paper repeatedly says "FastSpeech," the vocoder used in the tested TTS system is never specified, and the formatting of the citations makes it difficult to read sections of the paper with many references.

justification_for_why_not_higher_score: The empirical evaluation needs to be strengthened. Minimally, this would be done by including a state-of-the-art TTS model such as VITS in the evaluation. Ideally, the authors would apply their ideas to a second TTS model that is closer to the mainstream to show that their ideas generalize beyond ConformerTTS.

The writing of the paper needs to be improved. The formatting of the citations should have been corrected after the reviewers mentioned it, the paper should be clear that ConformerTTS is based on FastSpeech2 (not FastSpeech), and the paper should specify which vocoder is used.

The experimental results should provide variances on the MOS scores.
Decision: Reject