Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: - To enhance the quality of style-controlled generation, especially in an unsupervised manner and non-parallel setting, this paper proposes a "style equalization" mechanism to prevent the content leakage problem. In the style equalization module, the style of a sample is transformed to be the same as the style of ground truth. The authors assumed that content information is time-dependent whereas the style can be time-independent so that the authors employ time-average pooling to learn the global style. Then the style difference is added to the inputs style features. At each time step, a content attended feature queries and attends appropriate style equalized feature via the multihead attention module. The entire model is optimized to maximize the ELBO. The proposed method is demonstrated on speech synthesis and hand-writing synthesis tasks.
main_review: Strong points

- The problem this paper tackles is crucial and practical in controllable generations. In particular, the unsupervised learning and non-parallel setting are realistic. Also, the proposed method and the motivation behind it are simple but effective. The method is also technically sound.
- Overall, this paper is neat, clearly written, and well-organized.
- The experimental results are much improved in the automated evaluations and human evaluation. In particular, I was hard to distinguish the synthesized speech examples whether it is ground-truth or generated. The low WER of generated speech is very impressive for me.
- The authors provide sufficient experimental details for reproducibility and mentioned several training techniques.

Weak points

- Please clarify that, in Sec. 5.2, which data are compared in the 'cos-sim' metric?
- The implementation codes are not submitted. Although the authors mentioned the details of training, publish the source codes would be helpful for the community. 

Questions

- Please refer to the weak points above.
summary_of_the_review: Recommendation

- I vote for "accept" with the reasons that the proposed method is simple and somewhat novel, and also it shows strongly effective results. I think the results are able to contribute to the style-controlled synthesis communities.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors argue that the typical training algorithms for controllable sequence generative models suffers from the 'training-inference mismatch'. Therefore, to address such a problem, they introduce a style transformation module that is called 'style equalization'. Such a module is designed to enable training using different content and style samples and thereby mitigate the training-inference mismatch problem. To demonstrate the generality of the proposed approach, the 'style equalization' is applied to two tasks of TTS and text-to-handwriting synthesis on three datasets. On both tasks. the models show good results. 

Controllable sequential generative models have been studies for years. One of the most fundamental problem is how to effectively capture the content information and style information, respectively. It is a critical while very challenging research problem, because the 'content' and 'style' are entangled in the training samples, and ones must carefully design the training objective such that each of these factor can be learned in a controllable way. The idea of learning the 'style equalization' is interesting, and achieves promising results on tasks in different application scenarios, i.e. TTS and text-to-handwriting synthesis. Despite that, the paper is easy to follow, and the demos show in the project page qualitatively demonstrate the proposed approach.
main_review: Strengths:
Controllable sequential generative models have been studies for years. One of the most fundamental problem is how to effectively capture the content information and style information, respectively. It is a critical while very challenging research problem, because the 'content' and 'style' are entangled in the training samples, and ones must carefully design the training objective such that each of these factor can be learned in a controllable way. The idea of learning the 'style equalization' is interesting, and achieves promising results on tasks in different application scenarios, i.e. TTS and text-to-handwriting synthesis. Despite that, the paper is easy to follow, and the demos show in the project page qualitatively demonstrate the proposed approach.

Weakness:
1.My main concern is the training of 'style equalization' module. Based on the design, the training is to approximate the posterior p(z|x, c) with q(z|M(x', phi(x, x)), c). In this such a design, the expectation is that the style encoder only capture the style information from the given samples x and x'. However, it might be possible that the network is trained to find a shortcut that conv(x) -> x_content+x_style, and M & phi are just trained to diminish x'. In this way, the training loss is still very small, however, the conv(.) does not trained to encode pure style information as expected.  I am wondering, is there any penalties or training tricks are used to avoid such shortcuts? As I do not find implementation details, and either ablations or analysis on the latent space that inferred by the style encoder conv(.), so I am not fully convinced.
2. To deal with the 'training-inference mismatch' issue, there are another line of works hat encourage disentanglement of 'content'
 and 'style'. For example, in [1], the model are also trained with paired and un-paired text-speech by disengaging the content and style information in each encoders. While, I do not see comparisons and discussions about this line of work. I am very curious to see what are the differences when applying 'disentanglement' and 'style equalization'. 
3. There lack of implementation and experimental details.
4. The comparing SOTA are quite a few. On TTS, the model is only compared with Tacotron-based models, e.g. GST and other versions. On text-handwriting, is only Graves.
5. There lack of comprehensive ablation studies. Especially on the style latent space. I am curious to see differences of the latent space directly inferred by the style encoder conv(.), and the after by applying M( ., phi(.)).
6. No source code is submitted, it raises doubles on the reproducible ability. 

[1] S. Ma, D. McDuff, Y. Song. NEURAL TTS STYLIZATION WITH ADVERSARIAL AND COLLABORATIVE GAMES
summary_of_the_review: This paper is well motivated, the idea of applying 'style equalization' is interesting. Also, the showed experimental results are impressive.

Upon the weakness I've pointed, I would suggest the authors:
1) demonstrate how M & phi can be trained in the expected way, which avoids the model collapse to shortcuts.
2) add discussions and comparisons with the very related line of work, i.e. content-style disentanglement
3) adding ablation studies and comparisons with more SOTA approaches

Minors:
By looking into the paper, I am not clear about the training and evaluation datasets. For example, on TTS, what is the training dataset and what is the evaluation dataset? Are they trained on VoxCeleb, and then evaluated on VCTK & LibirTTS. Or they trained on VCTK and evaluated on VCTK, and the same for LibriTTS?  If the latter, are the evaluation performed on 'seen' or 'unseen' speakers?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper:  - Proposed an unsupervised style transferring framework based on VRNN;
 - Conducted experiments on TTS and handwriting synthesis to demonstrate the effectiveness of the proposed method.
main_review: Strengths:
 - Proposed an interesting method with nice experimental results.

Weaknesses:
 - The connection to related works is not well discussed.
 - The benefit of the proposed method over other similar methods is not well justified.

Detailed comments:
 1. The main idea of this paper is based on VRNN (Chung et al., 2015). Such connection, as well as the connection to other methods based on VRNN (e.g. Aksan et al., 2018), is not clearly discussed in this paper. Other major approaches on style transferring are not well discussed either, such as VAE-based approaches (Akuzawa et al., 2018, Henter et al, 2018, Sun et al., 2020).
 2. This paper claims "state-of-the-art". However, the experiments are not compared to the state-of-the-art approaches in the field.
    - The TTS experiments focus on speaker similarity, but is compared to the Global Style Tokens (GST) model, which is not good at voice modeling due to its limited dictionary. VAE-based approach, such as Hsu et al., 2019, are better baselines. For the Tacotron baseline, it's also worth to include the result conditioned on speaker ID.
    - On handwriting synthesis, the baseline used in the paper, Graves, 2013, is not a strong baseline either. More recent works, such as Aksan et al., 2018, Kotani et al., 2020, are better baselines.
 3. The main novelty of this paper is Eq (2). The idea is: to generate sample $x$, the model is conditioned on both a content $c$ and a style $z$, where $z$ is from a function $M(x, x')$. Here $x'$ is a sample unrelated to $x$. A natural question is: is $x'$ really used by the model, or is it actually ignored? This is neither theoretically nor empirically answered in the paper.
 4. To address the above question empirically, I'd suggest running two ablation studies:
    - replace $M(x, x’)$ with $M(x)$
    - replace $x’$ (more precisely $f'$) with a learned prior
    - (interesting but not required) replace $x’$ (more precisely $f'$) with a random noise
 5. There are quite some redundancy in the statement of the mismatch problem. The key idea of this paper is not presented until paper 5. It would be nice to save some space from them, and use it for the connection to related works.

References:
 - Chung et al., A Recurrent Latent Variable Model for Sequential Data, NeurIPS 2015.
 - Aksan et al., DeepWriting: Making Digital Ink Editable via Deep Generative Modeling, 2018.
 - Akuzawa et al., Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder, Interspeech 2018.
 - Henter et al., Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis, 2018.
 - Sun et al., Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis, ICASSP 2020.
 - Hsu et al., Hierarchical Generative Modeling for Controllable Speech Synthesis, ICLR 2019.
 - Kotani et al., Generating Handwriting via Decoupled Style Descriptors, ECCV 2020.
summary_of_the_review: While this paper proposes an interesting idea for style transferring with nice experimental results, the benefit of the proposed idea over reasonable baselines is not clearly exhibited. The connection to related works is also not well discussed.
review_title: null

Meta-review: This work aims to improve style transfer in the unsupervised non-parallel case. It does this by proposing a style equalization approach to prevent content leakage and assuming that content information is time-dependent whereas style information is time-dependent. This is an important problem to solve and lots of prior work in the area exists. The work is well-organised with good experimental results. However, there are strong claims in the paper and there is insufficient experimental comparison to similar related work such as Hsu et al. 2019 and Ma et al. 2018 to back that up. If there's no comparison with the current state of the art (e.g. due to a private implementation or dataset) then it's hard to justify calling a new work a new state of the art. Even though an implementation may be private, it can be worth spending time to reproduce a paper or asking the authors for an implementation. Finally task and metric selection could be improved to better highlight the performance of the approach. The reviewers thank the authors for the rebuttal but it was insufficient to change their decision.
Decision: Reject