Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes VAE latent embeddings to extract subspaces that relate to pitch (f0) and formant frequencies (f1 through fN). This is done through first training a frame-synchronous  IS-VAE model from clean speech data. The authors pass controlled synthesized speech through the model and obtain the embeddings corresponding to the synthesized data with randomly varying one fj component and keeping others constant. Each embedding corresponds to a frame of spectral data. The set of embeddings are then analyzed by PCA to find the principal eigenvectors corresponding to each fj explaining 80% of the total energy. The mapping from f0, f1, .. fN values and the subspace coefficients is done through a linear regression mapping independently for each fj which is also learned from synthetic data. Through simple examples, the authors show that they can change individual components (f0, f1, .., fN) by modifying the embeddings correspond to a signal, to some extent. They compare their results with rule-based vocoders (WORLD, TD-PSOLA) and a VAE baseline that does not use subspaces.
main_review: The paper is interesting in analyzing the subspaces of the latent space that correspond to each formant and the pitch in a trained VAE. The subspaces were obtained using synthesized speech which is not ideal and it is not clear whether the method would work well on all speech data. Another possible issue is that the model performance is analyzed with very simplistic data: English vowels. It was not tried on continuous real speech utterances. Also, I am not totally convinced that this latent space post-analysis way is the best way to obtain disentangled latent information. It could be possible to have multiple embeddings that correspond to each fj from the start. How would this approach compare to the post-analysis approach? The subspaces are found to be "mostly" orthogonal but not exactly, so there may be some leakage between subspaces.

Some itemized questions and recommendations:
1. Dimension D of x_i is not defined before it was mentioned in line 7 of Section 2.1. Maybe instead of {\cal X}, you can directly indicate the vector space for the features.
2. In Section 3.3, the justification for linear regression for predicting the subspace coefficients is not clear to me. Why would the subspace coefficients be a scalar multiple of the fj value? Would it not work better if you used a multi-layer nonlinear prediction?
summary_of_the_review: The analysis looks interesting, but the training and eval data used for this analysis looks too simplistic. A direct disentangled representation could also be possible with an easy way to make smaller embeddings corresponding to each fj.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method for utilizing labeled synthetic data in order to characterize and control the latent space of a VAE trained on individual frames of speech spectrograms.
Key properties of the data which one might want explicit control over are identified, i.e., pitch and formant frequencies, and a parametric speech synthesizer is used to generate synthetic datasets for each property, where the property in question is varied but all others are kept fixed.
These labeled data are used to identify subspaces of a VAE latent which correspond to each property, essentially by a principal components analysis of the latent vectors from each point in the synthetic dataset for that property.
The degree of disentanglement of the latent representation can be characterized by how orthogonal the bases are across subspaces.
Furthermore, individual properties can be directly controlled in isolation by learning a simple linear regression model mapping from the quantity in question (e.g., fundamental frequency in Hertz) to the subspace basis using supervision from the corresponding synthetic data.
main_review: Strengths:
 - Clear and well written introduction to the source-filter model of speech production.
 - Clever method for identifying and controlling disentangled subspaces given supervised labels for them.
 - Convincing quantitative and qualitative demonstration in Section 4 that the proposed approach 1) learns a representation which is well disentangled and 2) can independently control each component. 
 
Weaknesses:
 - Experiments are all on a relatively simple domain of speech spectrogram frames (in fact, restricted only to vowels, speech sounds which are most easily modeled using a source-filter decomposition).  The quantitative results (sec 4.2) are further restricted to *isolated* vowels, more closely matching the domain of the labeled synthetic data and not covering the dynamic space of variation in natural speech.
   - At the very least the paper could use a detailed discussion about the how the proposed approach might be applied to other domains of interest to the generative modeling community, e.g., to natural images or at least to more complex speech tasks.
     Such generalizations seems difficult since a key requirement for the technique to work is the existence of datasets which capture variation in a single attribute independently of others (or a synthesizer/generator which can create such datasets), i.e., it requires data which explicitly demonstrates disentanglement to use as supervision.  This requirement seems like it would be difficult to fulfill in other less restrictive domains.

 - Synthetic data was generated using a synthesizer based on the same source-filter decomposition that the proposed approach is designed to identify and recover.  The learned bases $U_i$ capture the space of variation in the synthetic data, but it's unclear how representative this is of real speech spectra.
  - Overall it is unclear how well the approach might generalize to real data which does not perfectly match the synthetic labeled datasets?  E.g., speech in the presence of background noise or overlapping speakers.

 - The fact that a VAE trained on clean speech spectrogram frames naturally disentangles this structure is of moderate interest to practitioners in that domain.
   But it's also not very surprising, as the problem of decomposing speech into separate source and filter components has been well understood for decades.
   See e.g., slides 15-18 of https://www.ee.columbia.edu/~dpwe/e6820/lectures/L05-speechmodels.pdf and references in https://en.wikipedia.org/wiki/Mel-frequency_cepstrum#History which describe how the cosine transform commonly used in cepstral analysis approximates the principal components of log-spectrum.
   Qualitatively, the described transformations in Sec 4 (transforming to constant f0, or "whispered" speech) are all fairly straightforward to implement without any machine learning at all using signal models from the 1980s, e.g., LPC or cepstral analysis, by modifying the source components while keeping the filter components fixed.
   Training a VAE and using a complex generative model such as WaveGlow for time-domain audio reconstruction is overkill for these simple transformations.

Specific comments:
- Sec 1, top of page 2: There have been other recent approaches to speech synthesis based on the source-filter model which are worth referencing in addition to LPCNet.  For example https://ieeexplore.ieee.org/abstract/document/8915761.
- Sec 4:  It's interesting that $M_0$  is as high as four, when the fundamental frequency should be easily captured by a scalar.  Similar for $M_2$ and $M_3$.  Is there an explanation for this? 
- Sec 4.2.2: The low average NISQA scores of 2.6 (out of 5) on the original speech signals suggests a potential mismatch between the data that model was trained on (presumably real speech?), and the isolated vowel sounds used for evaluation.  Overall the large confidence intervals for NISQA in Table 1 further call the utility of this metric for this task into question.
- Sec 5, top of page 9: The papers cited as "voice conversion methods" all describe text-to-speech synthesis techniques, not approaches to pure text-independent voice conversion, an area which has been well studied for many years.  Some recent deep-learning based examples include the sequence-to-sequence-based models (e.g., https://arxiv.org/abs/1810.0686), or the CycleGAN-VC (http://www.kecl.ntt.co.jp/people/kameoka.hirokazu/publications/Kaneko2018EUSIPCO08_published.pdf) or StarGAN-VC (https://arxiv.org/abs/1907.12279) model families
summary_of_the_review: Although the paper is well written, it's main findings seem likely to be of interest to only a small segment of the community.  Unless the proposed technique is more broadly applicable to more complicated domains, it seems like this work might be a better fit for a more specialized speech venue.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the latent representations of speech spectrograms learned by unsupervised variational autoencoders (VAEs) and discovers that the VAE learns to model the variation of fundamental frequencies (F0/source) and formant frequencies (filters) using orthogonal subspaces. Based on the discovery, the authors further propose to build a regression model that maps target value of a property to the factor loading coefficients of the subspace corresponding to that property, which enable control of properties independently.
main_review: Strengths
- This paper could be seen as an extension of Hsu et al. (2017a), both of which found that unsupervised VAEs learn to model speech properties such as formants in orthogonal subspaces. To transform speech, both consider an affine transformation that subtracts the component of the original property ($U_i U_i^T z$ in this paper and $\mu_{src}$ in Hsu et al. (2017)) and then add the component of the target property ($U_i g_{\eta_i} (y)$ in this paper and $\mu_{tgt}$ in Hsu et al. (2017)). There are two methodological improvements in this paper. First, the authors find the subspace modeling for each property U using labeled data, such that the subtraction can be done for an unlabeled instance. Second, a regression model is built to predict the coefficients of the target value for each property, such that this method can in principle generalize to unseen values for a property.
- Empirically, the proposed method demonstrates better source and filter parameter control compared to the VAE baseline (Hsu et al., 2017).

Weaknesses
- The main contribution is determining latent subspaces for F0 and formants and building a regression model to predict the coefficients of the components for each subspace. It offers limited real world applications since the performance still lags behind traditional signal-processing based methods if these attributes (source and filter parameters) were to be modified. The novelty is also limited and might have limited interest for the machine learning community. Specifically, the orthogonal subspace property has been shown in computer vision and speech modeling, and the authors have only demonstrated that the proposed method can be used to control a limited set of speech properties.

summary_of_the_review: This paper presents a nice extension from the literature. However, the novelty and applicability is limited and the generality of the proposed method for controlling other attributes is unclear. Hence, this could be of limited interest to the audience of the conference
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper shows that the fundamental frequency and formant frequency information is encoded in a speech VAE model.
This can be found by using artificially controlled/generated dataset.
After finding how to manipulate the latent space, one can control arbitrary speech samples in a desirable way, such as controlling fundamental frequency or formant frequencies.
The authors also show that the harmonic parts can be removed from the original speech and reconstructed as a whispered voice using only the spectral envelope part.
Experiment results show that the proposed method can control formant frequencies and show that it can control better than the method proposed by previous work.
The proposed method show on par or worse performance on fundamental frequency control experiment when compared to traditional DSP vocoders, such as TD-PSOLA or WORLD.

main_review: strengths:

The fact that one can control the fundamental frequency and formant frequencies of speech by manipulating the latent space is an interesting finding.
Finding the subspaces of latent the VAE latent space using artificially controllable dataset is an interesting idea.


weaknesses:

It's interesting work to someone who is working in the speech signal processing field.
However, I wonder how much implication this work could give to general audiences.
The fact that the VAE latent space can be manipulated using an artificially controllable dataset is interesting but at the same time, it is a disadvantage because we cannot find the subspaces if the artificially controllable dataset cannot be acquired.

summary_of_the_review: This paper shows interesting findings on speech VAE.
The proposed method is straightforward and I think it can draw some attention from the speech signal processing field.
I recommend [6: marginally above the acceptance threshold].

Questions:
1. Why was IS-VAE chosen to model speech?
2. On Page 3, Section 3, second paragraph, "Di denote a dataset of artificially- generated speech vectors (more precisely short-term power spectra) synthesized by varying only fi, all other factors {fj , j ̸= i} being arbitrarily fixed", does this mean that when making D_0, f1, f2, and f3 is fixed? If that's true what values for f1, f2, and f3 are chosen?
3. Can we still find subspaces when we do not have the artificial generator? (e.g., by manually selecting speech samples we would like to take control of?)

References:
Some papers the authors can consider to add in the reference:
[1] shows that one can disentangle source and filter in a supervised manner.
Recently [2] showed that the source and filter parts can be disentangled even in a self-supervised setting. 

[1] https://arxiv.org/abs/1908.01919
[2] https://arxiv.org/abs/2110.14513
review_title: null

Meta-review: This work shows that the source-filter model of speech production naturally arises in the latent space of a variational autoencoder (VAE). It is interesting that the fundamental frequency and formant frequencies are encoded in orthogonal subspaces of the VAE latent space -- this opens up a possible way of easily controlling these.

The key motivation/goal of the paper has caused some confusion. The abstract highlights an observation about VAE’s learned representation. In retrospection, some reviewers have not found the findings very surprising. On the other hand, the authors also do not attempt at developing and evaluating a speech generation method. As is, the paper seems to be much more suitable to a specialized workshop on speech. Alternatively, the paper could be extended to other modalities to show steerability of a representation using a synthetic dataset. However, the current scope seems to be somewhat limited hence I am not able to recommend the current manuscript for acceptance.
Decision: Reject