Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Contributions:

The main contribution of this paper lies in the proposed OSPOT-VAE for semi-supervised classification. The proposed model is (i) an one-stage framework that unifies the generation and classification loss in one ELBO, (ii) achieves a tighter ELBO by applying optimal transport to the distribution of the latent variables, and (iii) achieves SOTA semi-supervised learning results without sacrificing generative performance on many benchmarks. 

Strengths:

(1) Novelty: This paper contains novelty in terms of two aspects. First, the derivation of Eqn. (13) unifies both generation loss and classification loss naturally. The derivation seems different from a standard semi-supervised VAE. Second, the use of optimal transport (OT) estimation to tighten the ELBO results in improved classification performance. The use of OT seems novel in this context. 

(2) Writing: The paper is generally well written. However, I also found section 3 is hard to follow, with details in the Weaknesses section.

(3) Experiments: The results look impressive. OSPOT-VAE achieves better performance than other generative approaches. 

Weaknesses:

(1) Clarity: My biggest concern lies in the clarity of section 3, which I think is confusing in its current shape. After several rounds of reading, below is my understanding. In section 3.1, the authors first derive the objective in Eqn. (13). This objective is derived based on the use of mutual information (Eqn. (8)). By the end, no special cross-entropy loss term like in Eqn. (7) is needed.  Second, in order to tighten the bound, section 3.2 introduces optimal transport estimation. Combining these two losses together (see in Algorithm 2) results in the final algorithm. However, generally, I feel the presentation of section 3 is confusing. One may need to present the big picture (framework) first, and then dig into details.

Besides that, I have some questions as below. 

a) There are many hyper-parameters in the final objective, for example, the mutual information I_z, I_c is also considered as hyper-parameters. How all these hyper-parameters are selected?

b) Is there any intuition on why Eqn. (13) is already enough without introducing the empirical cross-entropy loss like in Eqn. (7)? It seems to me adding this additional cross-entropy loss will not hurt performance. 

c) In Eqn. (12), how is the last term D_{KL}(p(c|X)||q_{\phi}(c|X)) calculated, since we cannot get the true p(c|X)?

d) As shown in Table 4, the use of optimal transport seems to be the key. However, section 3.2 is hard to follow. For example, In Algorithm 1, Line 6 & 7 considers the true posterior of z and c is a Gaussian and a multinomial distribution, respectively. Why doing this? Supported by theory? I also feel the calculation of L_{Rz} and L_{Rc} in Line 8 & 9 is hard to follow. 

e) How reliable is this optimal transport estimation is? And how it contributes to the final classification performance? Since this loss term is not a classification loss, then is there any intuition on why it serves as a regularization to help the final performance so much?

f) Results in Table 5 on Cifar10 seem not imply OSPOT-VAE tightens the ELBO. Adding results on MNIST will be better, since the ELBO on MNIST is widely benchmarked. 

(2) Minor:

a) I am not sure why section 2.1 is needed. Also, the feature matching GAN paragraph in section 2.2 seems also unnecessary. If you really want to discuss the use of GAN for semi-supervised learning, then there is a lot of other work that needs discussion.

b) There is a missing right bracket in the second term of Eqn. (9).

Overall, I think the results look quite impressive. However, the presentation of the method part is unclear to me, and could be much improved. 




review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes to combine a VAE model with the Optimal Transport to approximate some components of the model. The authors evaluate their approach on semi-supervised problems and claim to obtain very competitive results compared to literature. Unfortunately, the paper is very unclear and hard to follow. The authors make some claims that are not true, for instance, learning so called M1+M2 architecture end-to-end is hard, however, there are papers that successfully train such model. Moreover, the authors reported their results as SOTA, however, they missed many other papers with much better scores.

Remarks:
- The authors claim in the introduction that training a two-level VAE with a classifier (so called M1+M2 architecture) is not robust. However, there are papers that were able to train such model without any reported problems, for instance:
* Louizos, C., Swersky, K., Li, Y., Welling, M., & Zemel, R. (2015). The variational fair autoencoder. arXiv preprint arXiv:1511.00830.
* Davidson, T. R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J. M. (2018). Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891. (published at UAI 2018)
* Ilse, M., Tomczak, J. M., Louizos, C., & Welling, M. (2019). DIVA: Domain Invariant Variational Autoencoders. arXiv preprint arXiv:1905.10427.

 - The authors report SOTA results on MNIST (100 labels). However, there are papers that report better scores, for instance 2.6 in:
Davidson, T. R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J. M. (2018). Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891. (published at UAI 2018)
In this paper they used a VAE model (the M1+M2 architecture) that was trained end-to-end.

- A rather minor remark, but I do not fully see a reason why the authors spent a lot of space on Section 2. I believe most of this text could be included in the Appendix.

- Equation 8: The authors claim that this decomposition was introduced in (Zhao et al., 2017), however, it was done earlier:
Hoffman, M. D., & Johnson, M. J. (2016, December). ELBO surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS.

- Equation (16) is very vague and hard to follow. Moreover, the idea of using Optimal Transport is also hard to follow.  The authors present a very generic description of the Optimal Transport, and then refer to an algorithm (a pseudocode). It is very unreadable.

===== AFTER REBUTTAL =====
I would like to thank the authors for the rebuttal. I read the comments carefully. However, I am not still convinced by claims of the paper. It is still vague to me why the proposed training procedure is necessary. Therefore, I decided to keep my original score.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper aims to solve the existing problems in semi-supervised learning by employing optimal transport theory and proposes a one-stage training VAE, OSPOT-VAE, which has a tighter ELBO and demonstrates better performance on benchmark datasets like CIFAR-10 and CIFAR-100 than benchmark methods. One main contribution in the paper is to use optimal transport to achieve a tighter ELBO in the proposed framework.

My main concern is that it appears in the paper the authors are trying to use a linear transport to approximate the intractable posterior distribution p(z|X) and p(c|X). I am not sure how this can be achieved and there is no discussions in the paper about this or how to reduce the approximation errors.


review_title: Official Blind Review #2

Meta-review: The paper proposes to combine a VAE model with the Optimal Transport to approximate some components of the model. The authors evaluate their approach on semi-supervised problems and claim to obtain very competitive results compared to literature. Unfortunately, the paper would benefit substantially from revisions to make it easier to follow. For this reason, the paper is not ready for publication in this venue at this time.
Decision: Reject