Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Update after discussion with authors**
I want to thank the authors for their incredibly detailed responses and engaging so actively in the discussion. Some of my criticism could be addressed, while other issues are still somewhat open. If the main merit of the paper is to make the "sequential VAE community" aware of issues that have been discussed and addressed before, then I think the paper does an OK job at that (though I'm not entirely sure what community that is, the issues have been discussed before in the fields of vision and VAEs with autoregressive decoders). 

I want to strongly encourage the authors to be as precise as possible when describing the novelty - maximizing the mutual information that a representation carries w.r.t. some relevance variable while simultaneously minimizing information that it carries w.r.t. to another variable is NOT novel. What is novel is the application of that principle to separating "global" from "local" information in sequential data (and how to actually perform this originally intractable optimization in practice). I also want to encourage the authors to state what's known and what's new as clearly as possible and improve the quality and clarity of the "educational" review of why maximizing mutual information is not enough as much as possible.

Viewing the paper as "showing how a known problem also appears when separating global from local information, and how to apply known solution-approaches to the problem in this specific context", shifts the relative importance of the issues raised by me. Essentially, that view emphasizes the paper as mostly an application paper (rather than a novel theoretical contribution). Accordingly (but please make sure that that shift in view is also clear in the final paper), I am weakly in favor of accepting the paper and have updated my score accordingly.

---

**Summary**
The paper tackles the problem of separating ‘global’ from ‘local’ features in unsupervised representation learning. In particular, the paper tackles a common problem in autoencoders where the decoder (generative model) is autoregressive and conditioned on a variable. Ideally, the latter variable captures all global information (such as e.g. speaker identity) whereas the autoregressive model deals with generating local structure (such as e.g. phonemes). As the paper points out, capturing only global information (and all of it) in the conditioning variable alone is notoriously difficult with standard variational autoencoder objectives, and several solutions have been proposed in the past. In this paper the idea is to add an explicit penalty for statistical dependence (mutual information) between the global and the local random variable. This intractable objective is simplified with a series of approximations, leading to a novel training objective. Results are shown for speech data, and MNIST/FashionMNIST, where the proposed training objective outperforms a beta-VAE objective and an objective that explicitly aims to maximize information on the global variable.

---
**Contributions, Novelty, Impact**

1) Analysis of shortcomings of mutual-information maximization to regularize latent representations into capturing ‘global’ features. This topic has been widely discussed in the literature before, typically in the context separating nuisance factors from relevant variations in the data, or more broadly: separating relevant from irrelevant information (which is canonically addressed in the information bottleneck framework of course). Most of this previous discussion was aimed at supervised learning ([2]), but there is a considerable body of work in unsupervised learning as well ([1] discusses the same issue but with more clarity), and some recent, very relevant work targeting VAEs with autoregressive decoders as well ([3] is among the state-of-the-art models). The paper provides a recap of this literature, but misses some key references, and the clarity of the writing (pages 1-4) could be improved (see my comments on clarity below). Therefore I would rate the impact of this contribution as low.

2) Proposal of a novel regularizer. The main idea behind the regularizer Eq. (8) is good, but certainly not novel - it has been broadly discussed in the literature and implemented in various ways. The merit is thus in the particular derivation and approximations that lead to the objective in Eq. (13) and (14). To me the derivation seems correct, though the precise motivation is somewhat unclear (what shortcomings of alternative approaches are addressed here, e.g. using the density ration trick?). I personally think that there is sufficient novelty, but in the current manuscript it is hard to assess whether the novel method has benefits compared to strong competitor methods (which are unfortunately missing from the experiments).

3) Experiments on a speech dataset (using a state-space-model decoder), and MNIST/FashionMNIST (using a PixelCNN). Results indicate that the extracted latent space does capture global features slightly better than a beta-VAE, or (quite a bit better than) a MI-maximizing VAE. There is also some indication that local features capture less global information with the proposed method compared to a beta-VAE. These results are promising, but not surprising since beta-VAE and MI-VAE were not designed to solve the shortcomings that the method is trying to address. For results to be more convincing and stronger, it would be good to compare against alternative approaches that have the same objective, such as e.g. [1] and [3]. Additionally more control experiments and ablations as well as reporting more metrics (l(x;z) and I(z;s), or proxies/approximations thereof) would strengthen the findings and thus the potential impact (see my comments on improvements below).

(I am not an author of any of these)
[1] Moyer et al. Invariant Representations without Adversarial Training, 2018
[2] Jaiswal et al. Discovery and Separation of Features for Invariant Representation Learning, 2019
[3] Razavi et al. Generating Diverse High-Fidelity Images with VQ-VAE-2, 2019

---
**Score and reasons for score**
The paper addresses an important problem that has received attention in the literature for at least two decades (the InfoBottleneck framework lays the theoretical foundations here). The particular application to: (i) unsupervised learning, and (ii) global-conditioned autoregressive (VAE) models is very timely and has received less attention in the literature (but there are some papers). 

My main issue is that the paper addresses two problems: (A) separating global from local information, (B) avoiding that autoregressive decoders ignore the global latent variable. Clearly stating both problems, reviewing the literature for each of them, and then showing how the paper solves both of them (and showing experimental results for both of them) would really help with clarity and readability of the paper. It would also help flesh out the novel contributions made by the paper. Additionally, it is not entirely clear how well the proposed objective actually addresses (A) and (B) in the experiments. There is some good indication for (A), but it is not directly measured (e.g. by estimating I(x:z) and I(s;z)), the effect is only shown indirectly via AoLR and mCAS (or Err(z) and Err(s)). The same is true for (B): there is some that the generative model does not ignore the global latent code via the mCAS experiments, but it is quite indirect (also looking at the generated examples in appendix K raises some doubts about diversity of the generative model). 

Overall I think the ingredients for a good paper are there, but they are not quite coming together yet. A deeper look into the empirical results (control experiments, additional metrics), and a comparison against strong competitor methods are needed. My recommendation would also be to really focus on the new objectives (Eq. 13 and 14) and discuss in more detail how they differ from competitor approaches and what the theoretical/empirical advantages of these differences are (for instance I am personally not yet fully convinced that using the density-ratio-trick with a neural network classifier will always work well in practice). If all of that were in place, I think the paper would be significantly stronger and could potentially have wide impact. I thus recommend a major revision of the work, which is not possible within the rebuttal period. Below are concrete suggestions for improvements, and I will of course take into account the other reviews and authors’ response.

---
**Strengths**

1) The problem (separation of global and local info in variational unsupervised representation learning with autoregressive decoders) is timely and important, and has been somewhat neglected in the representation learning community (though there is work out there, and the same problem has been discussed extensively in a related context, such as e.g. supervised learning).

2) The Method builds on a body of previous great work and applies it in an interesting context (global vs. local features).

---
**Weaknesses**

1) Merits of the method somewhat unclear - the motivation/derivation when going from Eq. 8 to Eq. 13, 14  is a bit ad-hoc. What alternatives are there to the choices/approximations made? What are the advantages/disadvantages of these? Answering this might also involve some control experiments and ablations.

2) The experiments show somewhat indirectly that the goals were achieved. There is some empirical evidence that the method is working to some degree, but it remains unclear whether e.g. the learned z capture only global information (and all of it), and whether s captures only local information (and how much of it). What’s needed here are additional results/experiments.

3) The current writing is ok but could be improved. I think it would be helpful to clearly state the problems and previous approaches of solving them (and the issues with these previous approaches). This would make it easier to see how the proposed method fits into the wider picture and which specific problem it addresses/improves upon. I think Alemi 2018 (cited in the paper) and [1] mentioned above do a very good job of describing the overall problem..

---
**Correctness**
The derivation of the method looks ok to me, though it would be nice to justify the approximations made and attempt to empirically verify that they do not have a severe impact on the solution. The conclusions drawn from the experiments are broadly ok, but since the evaluation measures the desired properties in a quite indirect way, the generality of the findings and the extent to which the method solves the problem (quantitatively) remain somewhat unclear.

---
**Clarity**
It took me a bit longer to follow the main line of arguments than it should have (which might of course be my fault). It’s a bit hard to pinpoint to a specific paragraph, but perhaps the following suggestions are helpful for improving readability. It might be worth clearly stating the main problems (denoted (A) and (B) further up in ‘Score and Reasons for Score’) and separately discussion how they have been addressed (and what the remaining issues are) and how the paper addresses them. Currently this is entangled in the derivation of the method.

It would probably also help to have a short paragraph that summarises the novel contributions clearly (which makes it clear what’s novel and what’s been proposed before).

---
**Improvements (that would make me raise my score) / major issues**

1) Comment on all assumptions made when going from Eq (8) to (13), (14). Are these assumptions justified in practice? Would there be alternative choices, and if yes what are the downsides of these alternative choices? Some of the assumptions will then lead to further control experiments that would ideally be included in the paper. One example (perhaps the most important one) is below in 2)

2) Neural network classifiers are notoriously known to be ill-calibrated (typically having over-confident probability estimates). This could be problematic in the DRT approximation since the discriminator’s output probability crucially matters! Is the discriminator well calibrated in practice? How robust is the method against calibration issues? Is the problem expected to get worse when scaling to more complex data and bigger network architectures? This needs to be discussed, but ideally some points are also verified empirically.

3) beta-VAE and MI-VAE are ok baselines, but are not sufficient to show that the method performs very well. These two baseline methods have not been designed to address the main issue (separating global from local information) - it is thus not too surprising that the proposed method performs well. What’s needed is comparisons against strong baselines, e.g. a (hierarchical) VQ-VAE2 (ref [3] further up). Given that the method only slightly outperforms beta-VAE on the metrics shown (which has no explicit incentive to capture global information) this is important.

4) Report additional metrics (for each experiment it would be good to also report: reconstruction error, estimates of I(x;z) and I(z;s)). As \gamma is varied, does the method lead to consistent increase in I(x;z) and decrease in I(z;s)? Are the values for the latter two significantly better than when using beta-VAE/MI-VAE?

5) Reporting AoLR and mCAS with a logistic regressor/classifier is ok, because it says something about latent-space geometry which could be interesting. But for the paper it is more important to capture the exact amount of global information captured by z and s. Therefore it would be good to show additional results for AoLR and mCAS where the regressor/classifier is a powerful nonlinear model (a deep neural net).

6a) Alemi et al. 2018 gets cited quite a bit in the paper, but is not very well represented in the paper. In particular: the paper proposes a quantitative measure as a diagnostic to see how much information is captured by the latent variable and how much of that is used/ignored by the decoder (which leads to the definition of several operating regimes, such as the “auto-decoding” regime). Why not report the same measure in the current method?

6b) Alemi et al. 2018 actually propose a modified objective to target a specific rate. They empirically observe that a beta-VAE with beta<1 *in their experiments* leads to the VAE operating in the desired regime. As far as I understand they do not propose this as a general solution to fix decoders ignoring latent codes. This should be mentioned in the paper. As a consequence 6a) becomes even more important, or without any verification beta-VAE becomes an even weaker baseline, meaning that comparison against strong methods becomes more important.

---
**Minor comments**

a) Eq. (10) should be an inequality, because I(z;s) is upper bounded on r.h.s.?

b) How was it determined that “alpha=1 works reasonably”, is this based on some control experiments?

c)  Eq (13). Why this particular mixing in of the KL-term, why not multiply KL(s) with (1-\gamma) as well?

d) Table 1: report the reconstruction error. In particular, for high \gamma is there still reasonable reconstruction performance (and thus separation into global z and local s), or is all information except global information discarded and s essentially does not capture much meaningful information anymore, making good reconstruction impossible?

e) Fig 3a - is the x-axis ELBO or KL?

f) Fig 3, Table 1: ideally report multiple repetitions with error bars.

g) For \gamma=0.6 in appendix K, there seems to be very little diversity in samples drawn from either model. This should be mentioned more clearly in the main text.

review_title: Good main idea, but hard to judge whether approximations are justified and whether method would work well on more complex data

final_score: rating: 7: Good paper, accept
review_content: Summary
Observing the deficiency of existing MI-based sequential VAEs, where through the learning objective design, the local and global features may become disentangled, the authors propose adding a regularizer to explicitly disentangle local and global features. To address the computation tractability issue of adding the new regularizer, a density-ratio based approximation is adopted and a classifier is trained to approximate the density-ratio term, which during training is learned alternatively with the VAE objective.
Relation between the proposed method and Conditional MI, beta-VAE, and domain adversarial training is discussed. Then the proposed method is empirically evaluated on two sequential VAE based tasks: speech voice manipulation (DSAE) and image generation (PixelCNN-VAE). The authors designed experiments that show that the proposed approach can improve global representation learning.

Quality

*Pros: 
The paper is overall of good quality: the context of the problem is well explained with adequate diagrams/plots to aide understanding. The approach is well motivated and the derivation of the approximation method is mostly easy to follow. Experiments are well designed and details are provided.

*Cons:
The density-ratio technique to approximate D(q(z, s) || p(z)q(s)) should be better explained in the paper: I had to look into the reference (Sugiyama et al 2012) to understand the derivation in Equation (11).

Clarity
The paper is clearly written.

Originality
The paper is based on existing work of MI-VAE families of neural networks. Instead of proposing a completely new architecture/method, the authors spot a gap in the current literature, i.e., that local and global feature representations can be disentangled using the current learning objective. The paper addresses exactly this gap.

Significance
The paper and the proposed method should be of some significance to the VAE and domain-adaptation community and inspire future works.
review_title: Good paper, accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update after rebuttal:
I agree with Reviewer 5 that this paper has good ingredients, and the discussion and update of the draft clarifies the novelty and provides better review on the related work. However, the experiments presented in this paper are not very comprehensive, particularly the baselines and the ablation/alternative studies. I am not fully convinced by the authors response of "because MI-VAE performed worse than $\beta$-VAE in PixelCNN-VAEs ... we expected that a similar tendency would be observed." PixelCNN-VAE uses an autoregressive decoder, which are known to exhibit issues that are not observed from non-autoregressive ones like DSAE. This explanation of why MI-VAE was emitted seems slightly hand-wavy. I decided to decrease the rating to 6 to reflect the insufficiency in experiments, but hope that this experiment can be added if the paper is accepted.

Summary:
This paper aims to learn representations that capture global features in structured data, such as the speaker information within speech or the digit class in an image. The authors argue that previous work regularizing the representation $z$ by maximizing its mutual information $I(x; z)$ with the data $x$ has the side effect of simultaneously maximizing the mutual information between $z$ and local features $s$. This may cause the global feature $z$ to encode unwanted local information or vice versa. To address this issue, the authors propose to regularize $z$ through maximizing $I(x;z) - I(z;s)$, which is a lower bound of the conditional MI $I(x;z|s)$. The proposed regularization is further estimated using the density ratio trick, which employs a discriminator to estimate the ratio $\dfrac{q(z,s)}{p(z)q(s)}$ via a binary classification task and provide training signals to the encoder. The proposed regularization is applied to DSAE and PixcelCNN-VAE to demonstrate its effectiveness.

Pros:
- The paper is well written and easy to follow. Motivations of this work are clear, and related studies are also properly described to contrast the difference from this work.
- The formulation of the regularization seems novel to me. Derivation and approximations also make sense.
- Experiments conducted on multiple domains (speech and images) demonstrate superior performance compared to the baseline methods (no regularization or MI-based regularization). Multiple metrics are adopted for evaluation of different aspects (e.g., AoLR/EER evaluates linear separability, mCAS indirectly evaluates diversity)

Cons/Questions:
- The experiments on images compared the proposed method with both beta-VAE and MI-VAE. I am curious why the authors only compare with beta-VAE for the speech experiments but not MI-VAE. The EER reported in Hsu et al. (2017) is much lower than the results in this paper, and as mentioned by the authors, that work regularizes the representations with a discriminative objective that approximates $H(x|z)$ and therefore can be seen as a form of MI-VAE. The authors should also compare with such regularization for the speech experiments since it’s shown effective in the previous work.
- The authors state that approximating both $I(x;z)$ and $I(z;s)$ may complicate optimization, and avoid doing so by forcing alpha in Eq.8 to be 1 to enable rearrangement in Eq.10. However, it would be informative to show the results of approximating both terms with the density ratio trick and see how much it would affect the performance. By doing so, the strength of regularizing $I(z;s)$ can be independently tuned.

review_title: Review

Meta-review: This paper presents a representation method for time series data in the sequential VAE, where the global feature z and local features  s are better disentangled. The intuition behind learning z is to maximize the mutual information between z and input x, while minimizing the mutual information between z and s. The second mutual information is estimated with a discriminator in the DRT framework. Overall, the methodology can be seen as reasonable applications of the disentanglement principle to sequential data. The authors have shown that z and s learned in this way is better disentangled as compared to beta-VAE. In the end, the reviewers feel that while there is good intuitions/technical ingredients, the derivations in Section 3 are not very smooth, and several approximations/choices are not very carefully justified (e.g., choice of alpha, choice of DRT vs. other MI estimators), and perhaps stronger baselines than beta-VAE can be used.

The reviewers rate this paper to be borderline.
Decision: Reject