Good Semi-supervised VAE Requires Tighter Evidence Lower Bound
Authors: Haozhe Feng
Kezhi Kong
Tianye Zhang
Siyue Xue
Wei Chen
Venue:  (2020)
Keywords: VAE
Semi-supervised Learning
ELBO
Generative Model

Abstract: Semi-supervised learning approaches based on generative models have now encountered 3 challenges: (1) The two-stage training strategy is not robust. (2) Good semi-supervised learning results and good generative performance can not be obtained at the same time. (3) Even at the expense of sacrificing generative performance, the semi-supervised classification results are still not satisfactory. To address these problems, we propose One-stage Semi-suPervised Optimal Transport VAE (OSPOT-VAE), a one-stage deep generative model that theoretically unifies the generation and classification loss in one ELBO framework and achieves a tighter ELBO by applying the optimal transport scheme to the distribution of latent variables. We show that with tighter ELBO, our OSPOT-VAE surpasses the best semi-supervised generative models by a large margin across many benchmark datasets. For example, we reduce the error rate from 14.41% to 6.11% on Cifar-10 with 4k labels and achieve state-of-the-art performance with 25.30% on Cifar-100 with 10k labels. We also demonstrate that good generative models and semi-supervised results can be achieved simultaneously by OSPOT-VAE.

Under review as a conference paper at ICLR 2020
GOOD SEMI-SUPERVISED VAE REQUIRES TIGHTER EVIDENCE LOWER BOUND
Anonymous authors Paper under double-blind review
ABSTRACT
suni-supervised learning approaches based on generative models have now en- countered 3 challenges: (1) The two-stage training strategy is not robust. (2) Good semi-supervised learning results and good generative performance can not be ob- tained at the same time. (3) Even the expense of sacrificing generative perfor- mance. the semi-supervised classification results are still not satisfactory. To ad- dress these problems, we propose One-stage Semi-suPervised Optimal Transport VAE (OSPOT-VAB) fies the generation and classification loss in one ELBO framework and achieves tighter ELBO by applying the optimal transport scheme to the distribution of latent variables. We show that with tighter ELBO, our OSPOT-VAE surpasses the best semi-supervised generative models by a large margin across many bench- mark datasets. For example, we reduce the error rate from 14.41 %% to 6.11% % on Cifar-10 with 4k labels and achieve state-of-the-ee performance with 25.30% on Cifar-100 100 with 10k labels. We also demonstrate that good generative models and semi-supervised results can be achieved simultaneously by OSPOT-VAE.
one-stage deep generative model that theoretically uni-
INTRODUCTION
The rise of deep neural networks has led breakthroughs in computer vision, natural language processing, and many other domains. Most of these models are trained on large labeled datasets supervised learning. However, in many scenarios, although it is easy acquire a large amount of the original data, obtaining corresponding labels is often very costly or even infeasible. Semi- supervised learning (Thomas) 2009) is proposed to address this problem by training classifiers with sufficient unlabeled data and a small fraction of labeled data.
Recent works on semi- supervspe learning can be grouped into three categories: (1) disagreement based learning via data perturbation (Miyato al. 2019) and consisteney enforcing (Verma et al., 2019)
(2) metric learning (Wu et al., (218)), (3) generative approaches via generative adversar- ial network (GAN) (Springenbergl (016) and variational autoencoder (VAE) (Kingma al. 2014). Compared with the first two categories, generative approaches have great advantages in interpretabil- ity. Based the latent variable assumption (Doersch) 2016), the generative model has explicit variational inference form, so it can learn the marginal probability distribution of the raw data as well the conditional distribution of the latent variables given the input data, which makes pre- dictions more reasonable. Besides, generative approaches not only learn the required classification representations, but also capture the semmanics-disentaaglee factors that generate the data, making easier to generalize to different tasks (Narayanaswamy et al. 2017).
However, in practice, sumi-supervised generative approaches often encounter three major chal- lenges: (1) The two-stage training process is not robust. Semi-supervised VAE (Kingma et al.) 2014] needs to trained carefully with a two-stage hierarchical strategy. while the training pro- cess of GAN is a two-stage adversarial game (Chrysos et al., 2019). (2) Good semi-supervised learning results and good generative performance can not be obtained at the same time. In GAN, good semi-supervised learning performance will lead to a mismatch between the generated results and the real data distribution (Dai et al. 2017). While in VAE, the evidence lower bound (ELBO) objective is irrelevant to the classification loss, making it difficult to leam from the labels directly Naaayanaswamyy et al. 2017). (3) Even at the expense of sacrificing generative performance, the semi-supervised classification results still not satisfactory. In practice, disagreement- based meth-

---

Under review as a conference paper at ICLR 2020
X
Up(eix) o)
Pa c)
9e(alx:)
<olelk.)
X2
X2
4,
optimal (CaKK))
temesport
inference
sample
9p(elX;))
Φ
infermee
Ma
margin
(e Mull(e;))
%
Dur (110(le0)))
sampled D2
(a) Ote-stage SSL VAE
(b) Optimal Transport Estimation
Figure The schematic of OSPOT-VAE
ods (Xie et al. 2019 Berthelot et al. 2019) have dramatically improved the state-of-the-ar results several standard datasets, surpassing generative approaches by large margin. These challenges naturally raise a question: What limits the performance of generative approaches in semi-supervised learning?
In this work, we propose One-stage Semi-suPervisee Optimal Transport VAE (OSPOT-VAE) to address these challenges, which consists of improvements: (I) a one-stage semi-supervised
VAE model that unifies the generation and classification loss in one ELBO framework. (2) an estimation of the margin between true log-likelihood and the ELBO that exports a tighter evidence lower bound by applying optimal transport (Ambrosio & Gigli 2013) scheme to the distribution of latent variables.
Our model has the following contributions:
We show that OSPOT-VAB can be well trained with direct one-stage strategy.
We show that OSPOT- VAE can achieve both good generative performance and semi- supervised learning results simultaneously on a series of benchmark datasets.
We point out that it is the large margin between the ELBO and the log-likeldoodd of the input data that limits the performance of comi-supervised VAE. Besides, we evaluate this assumption across many standard datasets and show that with the proposed tighter ELBO,, -SPOT-VAE surpasses the best coni-supervised generative models by a large margin and achieves state-of-the-ar art performance on Cifar-100 with 10k labels.
SEMI-SUPERVISED LEARNING METHODS
that makes the correct inference y for unseen samples from p(X). In supervised learning (SL),
facing with training data that appears input-target pairs (X,y) E DD, sampled from an unknown distribution y). Our goal is to learn function f(X; c) parameterized
While in semi-supervised learning (SSL), we can obtain an extra collection of unlabeled data X Dy sampled from the same distribution p(X). We hope leverage the data from both D and By to achieve more accurate model than what would have been obtained by only using D),,
In this section, we review some existing methods for SSL. We mainly focus on those who have reached sfate-of-the--rt results, well as generative approaches which are strongly connected with our model; the more comprehensive overview beyond the scope of this paper, we refer readers to (Oliver al. 2018).
2.1 DISAGREEMENT BASED LEARNING
Disagreement-basee learning refers to the general approaches of imposing disagreement among multiple learners on the same task or multiple predictions from single learner. By eliminating the disagreement, we can enforce the generalization of the model on unseen data. A common technique for creating disagreement is data augmentation, which applies transformations or perturbations on the input data and leaves class semantics unchanged. For X E Dy, loss term can be derived as
[lf(ggugeet(()) - fXx; +)11
where the Augment(XX X)
a stochastic function which can be obtained by image transformation Xic al. 2019), virtual adversarial training (Miyato al. 2019), mixup method (Verma et 2018
2

---

Under review as a conference paper at ICLR 2020
Berthelot al. 2019). Another disagreement construction technique is to train multiple learners the same dataset and utilize the loss
to enforce the predictive consistency of different models, for example, "Mean Teacher" (Tarvainen) & Valpola 2017) and "Teacher Graph" (Luo et al., 2018). The generalization the models gets enhanced.
2.2 GENERATIVE APPROACHES
In generative approaches, input X is supposed to have corresponding continuous and discrete latent variables, which we denote by z and c respectively.
Feature matching (FM) GANs (Salimans et al. 2016 Dai et al. 2017) apply GANs to semi- supervised learning on K-classification tasks by specifying a (K+1)a (++1)-class objective for the discrim- inator. Instead of binary classification, true samples are classified into the first K classes respec- tively and fake samples are classified into the (K+1)-th class. This target function achieves strong empirical results by matching the generator distribution with true data distribution and improves semi-supervised classification performance.
Semi-supervised VAEs (Kingma et al. 2014 Narayanaswamy et al. (017) construct a probabilistic model parameterized by and Φ that respectively describe the generation and inference process between X and latent variables c. The generation process of X by and is:
p(z) = (2; 0, I); p(c) = Mult(c; r); (o c) [(X:
where Mult(K, π) is the multinomial distribution with class K and parameter π. f(X; z, c, 0) is suitable likelihood function, e.g. Bernoulli or Gaussian distribution, parameterized by a non-linear transformation of the latent variables z and c. The class label y is treated as if given. For the inference process, with the following hypothesis
c( = go (z X) (c) X);
p(z, c(X) = p(z p(c|Xi)
p(z, = p(z)p(c) (4)
evidence lower bound (ELBO) used as objective to predict the posterior distribution of latent variables as follows (see Appendix A. for proof):
log p( x) 2 Eggla x) <golog X 1z, c) - (k(((o((/X)]ll(z) - Dku(go(c/X)]pp()) = ELBO (S)
For the likelihood log (X) is infeasible, VAE maximizes its evidence lower bound instead, which derives the negative ELBO loss function L(X; Φb 0) -ELBO. The predictions for the classifica- tion label y can be obtained from the inferred posterior distribution 1s X). When class label y is not given, missing label sampling technique is used to sample from g (c X) as
Σ gol X)
(6)
Here Yk represents one-hot vector with in k-th dimension. Utilizing this sampling method, the algorithmic complexity of VAE is proportional to K so it is computationally inefficient. Note that in objective (4), the label predictive distribution (x(c) only contributes to the generative performance. To remedy this, existing models simply add cross-entropy loss to the negative BLOO loss such that the distribution qs (c()) can also learn classification rules from the labeled data. The extended objective loss is
min BainEx-u EX~De L(X; o. 0) + EX (C(,),/,,L((,,CCC y 0.0
y: 0) - log q0 (y|X))
(7)
Two-stage Training Strategy: In practice, (Kingma et al. 2014) finds that directly training the one-stage objective (7) will lead to bad semi-supervised learning result, so a two-stage training strategy is proposed to improve the model. The two-stage training strategy consists of two parts, M1 and M2. MI means to learn a new continuous latent representation Z1 first, and M2 means to train a semi-supervised model (7) with the embedding 2 from MI instead of the raw data X. This MI+M2 strategy builds deep VAE with two layers of random variables: Pe (X, 2 Z2, c) the inference 9 (c)x but is not robust in training. Moreover, GAN's training process can also be considered as two-stage with generator and discriminator competing with each other, and the two-stage adversarial game adds the difficulty in training.
Pe(X (z )e Z Zz, c)p(zz))(()) which can dramatically improve the performance
3

---

Under review as a conference paper at ICLR 2020
3 ONE-STAGE SEMI-SUPERVISED OPTIMAL TRANSPORT VAE
In this section, we introduce our semi-supervised VAE framework, OSPOT-VAF Firstly, we derive one-stage loss function that unifies the generation and classification loss under one ELBO without introducing any additional auxiliary loss items like (7). Then, we analyze a phenomenon that good ELBO values do not guarantee good semi-supervised performance and propose the optimal transport estimation to deal with it. At last, combining the two parts, we give the detailed algorithm of OSPOT- VAE and discuss
problems in model optimization.
3.1 ONE-STAGE SEMI-SUPERVISED VAE
Following the notations and assumptions (3.4) in Section 2.2 we derive our one-stage semi-
supervised VAE. With the empirical distribution Pempl X: D)
1x=x', we utilize the X'ED
decomposition in (Zhao et al. 2017 and rewrite the second part of (5) into (proof in Appendix A.2 P (x) DKL (1 X ())p(())) Lap (X; + (1 (z)]lp(z) 2 Ino (X:z)
(8)
where los (z) ID|
1
Σ (z(x))) and 2o (X: is the mutual information between X and The XED
left part of (8) equals to 0 when X and are independent. This is undesirable. so Ing (X; can be regarded as the lower bound of controlled mutual information. The continuous variables in (8) can be easily extend to discrete variables c. We can use L, and Ic to denote the controlled information capacity and derive the objective for the unlabeled dataset TDy
LTy (X: 0, Φ) =
loo log [X l2, c)] + Dx|Dxr (10 Iz]
+ Rel DKL. (1 () [p(c)) Lel
where B, I, I are all hyper-parameters forcing the KL divergence term to match the mutual infor- mation capacities of z and c.
For the labeled subset D3.,, instead of directly employing class label sampled we view it the parameter of the true posterior distribution, i.e. p(c[X) = Mult(e; K,y) and derive the following one-stage ELBO form:
p(X, z, c)
pl p(X, z,
log p( X) log R. Be (z) X X)p(c(X[ (xsla(X).((( log
l (z
p(z)p(c) E p(x log p(
[tx log ( ()
(10)
p(c)
+ Ex())x log
(x(c
Notice that DRL. vector. In this respect, the margin between x(c[X) and 4s (c(c)) can be significantly small when the suitable optimization method is chosen. This allows us utilize the approximation p(c[X) se
is equal to the common cross-entropy loss for y is one-hot
p(c) in (10), resulting in consist ELBO with LT (X; 0 Φ):
d (c) to modify Ep(c(X) log (c
Ep(e x) log p(c) ~ (Whea (c) ( log p(c)
= (xt. (11)
9. (c
9 (c))
Combining the ELBO form (10) of Dz with the mutual information decomposition (8) and the approximation (11), the objective for semi-supervised VAE is:
CD, XX y: 0, Φ) = (x x) log pe (X|2, c)] + (xD I (12)
+ DeL (go (c) X lIp(c)) - Iel + (xL
With (9) and (12), the objective for the entire dataset is now
min Ex-pemp (X.D) Lo, X; 0, Φ)) + ((x.)) (X, 0, 6.0
(13)
This one-stage objective with simple approximate transformation (9) unifies the generation loss as well as the target of SSL and results in improved performance of semi-supervised learning, which demonstrate in Section 4.1
4

---

Under review as a conference paper at ICLR 2020
Algorithm Optimal transport estimation ingests batch of observation X as well as the represen- tation U (z) X), T (c]) inferred from the original VAE and returns the estimation of the margin DKL x ( (pz(((() and DKL (9s(cc (p(c()))
Input:
Batch of observation X sampled from Pemple (X);
Inferred parameter (1e, diag(o²)) of qo (z|x) N(z ; μ., diag(o")))
Inferred parameter of ( Mult(c; K, nT);
Hyperparameter a for mixup vicinal distribution Pmizup
Output:
X sampled from Pmntrup X);
Estimation Las, of the margin (9 (p)z)))())
Estimation LM. of the margin DKL (9 (c X) ( (pp((())
X', n , x² RandomPerrmttation X, π. u1 0²)
2: X = > x (1 A) x B(a, a)
3: π OptimalTransporC((,, m' A)
4: (2, = Oprimaa//ransporr(((44 o²), (1',, )!2),
5: 2o (z))) (x((())) = VAB(X)
6: f(2[X) -N(z; ju, diag(
7: p(c[X) Mult(c; K, T)
8: LM2 (
9: LM. DKL
10: return X, LM Lice
3.2 OPTIMAL TRANSPORT ESTIMATION
To summarize the above, VAE aims to Iearn the usefuu representation go (z X) and qs Xx) by reducing the KL divergence between the empirical distribution Pemp (X) and the model marginal p(X) Sx p())p(z X))pc(()dzzze Instead of minimizing DKL (X)elmpXX)l(())) directly,
VAE models use the expected ELBO mentioned in (5) as target via the following inequality
DKL. (Pemp X))I((X) (X)) < [Pemp (X))
(X) ELBO
(14)
However, one phenomenon is that good ELBO values do not imply accurate inference. typical example has been discussed in (Zhao et al.) 2017). Here we mainly focus the cause of this phenomenon and propose optimal transport estimation to alleviate this problem in semi-supervised learning. Following the work in (Rezende et al. 2014), we write down the closed form of the expected margin between true log- likeliilddood and ELBO as (proof in Appendix AA.3)
E Ex [log (X)) ELBO] Expen. [DKL (zl + (c (Ip((())
(15)
Combined with the decomposition (8), training the expected ELBO target can only reduce the dif- ference between marginal distributions 4o (Z) and p(c).p(z). It means that even with good ELBO, the margin E E (x) DxL (ya [x)()()(X)) and E (X) DKL (ga(c|X) l(p(e))))((())
(15)
can still be large. In this scenario, the consistent optimization of ELBO will contribute more to the semi-supervised classification performance. However, optimizing the margin in (15) directly impossible, for p)c[X) and p(z] X) are unknown. To remedy this, extend the empirically effective approximation in (Zhang et al. 2018) to our VAE framework with the form
E Epmicep (X) DKL (go [p X)) E Epenyy (x) DKL X (p(z) + 0)
(16)
E Emicar (x) D (go (p(c Ep (x) DKL (gs (c |P(c + +)
where Pmizup X) is the mixup vicinal distribution (Zhang et al. 2018 and a is the re- lated parameter. Then we propose optimal transport estimation to construct the estimations of Epmerop(( DKL( (z) X) [)(((()())) as well as (pmarup(() DKL (40 (z) X) [|p(z ()) by applying op- timal transport scheme to latent variables and c. The computation steps of optimal transport estimation are provided in Algorithm and we present the details of the optimal transport scheme
in the rest of this section.
5

---

Under review as a conference paper at ICLR 2020
Algorithm 2 OSPOT-VAB training process with epoch 6.
Input:
Batch of labeled pairs Vr.) Da,, of unlabeled examples X Dy;
ELBO hyperparameters: S2. Se. I,, Le = ELBBOScheduler()):
Optimal transport estimation weights: W.,,, WM. WeightSchheuuler(():
Model parameters: ((t-1), (((-1); (((-)))
Model optimizer: SGD
Output:
Updated parameters: ((() (6(1)
1: Lt = Lmz Yz; (1111) . (2(1=11; Ba, Be: 12, Ic)
2: Lu -Lv (Xu; ((1-1), (b(t-1); 1); BB, Be, I,, Io)
3: LMN,LAG
OptimaTTrassorreStimatte (x
(Xu, 10
4: L = L-L Lu + WM, LM,, + WMe Life
5: 0(0) .b = SGD(O( (t11) aL OL
00 0Φ
6: return 0(0) (10)
Optimal Transport Scheme: The mixup vicinal distribution can be understood as applying linear transport between the points X, X' D, extending the original dataset with new points falling one straight line X À + (1 A) X', X E [0, 1]. For X. it natural thought that this linear transformation could associate with the shortest-path transport the latent space. Based on this, we calculate the distributions ((2[X), p(c] X) of z, c and consider them as the estimation of the true posterior distributions. Following the work of (Ambrosio & Gigli 2013), the norm-2 based optimal transport scheme yxx, y) between two distributions p(x) and ply) satisfy:
inf J. 11 ||1x-x117(x y)dxdy ((x.))
(17)
s.t. / (xx = p(x); = (
yxx = p
For the continuous variable N(u, )iag(o?)) and discrete variable Mult( (, π), the fol- lowing 2 propositions are proposed to calculate the shortest-path based on optimal transport scheme (see Appendix [A.4 for proof).
Proposition 3.1. The shortest-path derived from optimal transport scheme (17) between z ~ N(U1, diag(ot)) and z2 N diag(oz) with X [0, 1]
Ap1 + (1 A)p2
(18)
== + A)o2
Proposition 3.2. The shortest-path derived from KL divergence based optimal transport scheme between C1 ~ Mult( K, TT) and ~ Mult( K, TT2 with X E [0, is
#= Am + 2)
(19)
Algorithm yields the optimal transport estimation of the margin in (15), which leads tighter ELBO. In Section 4.2) we demonstrate that with this tighter ELBO, the inference performance of semi-supervised VAE significantly improved on many benchmark datasets.
3.3 OPTIMIZATION OF OSPOT- VAE
Combining one-stage semi-supervised VAE and optimal transport estimation, we can get the com- plete OSPOT-VAR VAE model. The full OSPOT-VAE algorithm provided in Algorithm 2 and schematic is shown in Figure [ Note that the conditions for the approximations used in Algo- rithm 12 satisfy (1) qo (c Rs pfcc and (2) the VAE model has already achieved good ELBO. Therefore, the warm-up schedule (Higgins et al. 2017) is used
parameters I, 82, 3, and WM, WM.- We list the details of "HLBOOScheduler(t)) and "WeightScheduler() in Appendix A.5
In Algorithm 2 apply stochastic gradient descent (SGD) optimizer, which needs to cal- culate the gradient Vo.LL. The target loss L consists of KL divergence and the expected log- likelihood. The derivation of KI. divergence part has closed form, while calculating the gradient

---

Under review as a conference paper at ICLR 2020
BackBone
Method
MNIST(100 labels)
SVHN(1I labels)
Disentangled VAE
9.71(=0.911
38.91(#1.06)
Narayanaswamy et al. 2017)
Same with MI+M2
MI (KKngma al. 2014)
11.97(#1.71)
54.33(=0.11)
M1+M2(Kmggna et al. 2014
+0.14)
36.02(=0.10)
One-stage VAE
3.14
27.38
Table 1: One-stage VAE error rate in MNIST and SVHN.
BackBone
Model category
Model
Cifar10(4k labels)
Temporal Ensembling(TE)
16.37
Laine & Aila 2017
Disagreement
Mean Teacher larvainen & Val Valpola
2017
15.87
WRN-28-2
VAT Miry et
2019
13.13
Mix.Mtcch et al.
2019
6.37
GS- GS-BadGAN "Li et al.
2019)
17.11
Generative
OSPOT-VAL
8.51(=0.32)
AutoAugmentt Cubuk et al.
Cubukk
2019
14.1
Disagreement
Temporal Ensembingd Laine
Larne & Aila
2017)
12.16
MixMatch" (Berthelot
2019
4.95
WRN-28-10
GS-BadGAN (Li et al. 2019)
14.41
Generative
GAN combine TE (Wei al. 2018
9.98
OSPOT-
6. 6.11(=0.34)
Table 2: Error rate in Cifarl0. denotes the best semi-supervised generative approach result. denotes the model ensemble two categories. denotes the corresponding backbone is not exactly WideResNet (Zagoruyko & Komodakis 2016). but belongs to one kind of its variations with comparable amount of parameters.
of E E gafielX) log pe X (z, c) is difficult. To this end, we follow the work of (Rezende al. 2014 and (Jang el al. 2017). using the reparameterization trick as
Fo (z) X) log X (z)
(e. V ( xo
log + +6 Vo log po (X(c(T 0)
(20)
(..1) Vo.s log Po (X X Softmax
T
Note that with (0 the algorithmic complexity of one-stage semi-supervised VAE is independent with the class number K, making it easier to extend to large-scale classification tasks.
EXPERIMENTS
In this section, we demonstrate the contributions of our OSPOT-VAE model with sufficient experiments on 4 standard SSL benchmark datasets, that is, MNIST, SVHN, Cifar10, and Ci- for100. Section 4.1 we show the validity of our one-stage semi-supervised VAE objective (13) by comparing with other one-stage and two-stage VAE models. Then, evaluate the per- formance of OSPOT-VAE under "WideResNNet (Zagoruyk & Komodakis 2016) backbone and compare with other state-of-the-art SSL models mentioned in Section Besides, We provide an ablation study to verify the contribution the optimal transport estimation. As an addi- tional application,
show that good generative models and semi-supervised results be ob- tained at the same time by OSPOT-VAL (Section 4.3). The source code is available at https:] 7/github. cww htt.c///aaapeeooeeeeemsssssooooo pendix A.6
VAE, more details are available in Ap-
4.1 ONE-STAGE
VAE
We evaluate the effectiveness the one-stage semi-supervised VAE objective on standard bench- marks, MNIST and SVHN. As for baseline models, we consider two VAE-based SSL models, which are ste-stage disentangled VAE (Narayanaswamy et al. 2017) and two-stage VAE(MI+M2) (Kinmma et al. 2014). For fairness, except the target loss functions, all models use the same struc- ture as is used in MI+M2 (Kingma et al.] 2014). The results are presented in Table and our model achieves the best performance.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Error in Cifar100. f and have the same meaning as described in Table 2
BackBone
Model
Cifar100(4k labels)
Cifar100(10k labels)
II Laiee & Aila 2017)
39.19
GS- US--BaGANN"
(Li et al. 2019)
45.11
37.16
WRN-28-2
LP* [Iscen al. 2019)
43.73
35.92
OSPOT-VAB
40.58
00448)
31.41(+00.21)
MixMatch" (Berthelot et al.
2019)
25.88
WRN-28-10
OSPOT-VAE
33.76(
0.53)
25.30 (+.31)
Table 4: Ablation study with SVHN, Cifar10, and Cifar100.
Methods
SVHN(IK labels)
Cifar10(4k labels)
Cifar100(10k labels)
One-stage VAE
10.11(
(...61)
38.62(10.67)
Optimal transport estimation (with encoder only)
6.54(++0622
10.71(#0.44)
36.21(10.29)
OSPOT-VAE
5.79(+0.15)
8.51(=0.32)
31.41(#0.21)
4.2 OSPOT-VAE
We compare the results of OSPOT-VAE with two categories of state-of-the-art models mentioned in Sectior2, In all experiments, we use the "WideResNet-28" model or other deep models with comparable amount of parameters
the backbone. The results in Table 2 demonstrate that our model outperforms most of the existing methods and surpasses state-of oftee-of-hhearrr art semi- remi-supereised generative models (Dai al. 2017) by a large margin. Notice that recently, fata--uugmentttion based method, MixMatch, (Berthelot et al. 2019) achieves the absolute state-of-the-ar results in all benchmarks. uses pre-designed sophisticated data augmentation strategies for different datasets and outperforms our model. We list its results fairly as comparison, while OSPOT-VAL VAE surpasses it in C0f dataset.
The OSPOT-VAE model consists of two parts: (1) one-stage VAE objective and (2) an optimal transport estimation. In ablation study, we analyze the effect each component in model with the backbone 2WideResNet-228- 2"" To study the independent effects of transport timation, we combine it with the encoder part of OSPOT- VAE to build a classifier with loss function T.M.- The improved classification error rates in Table show that, with optimal transport estima- tion, the posterior inference qo (c) X) gets closer to the true distribution p(c(X), X). It indicates that our optimal transport estimation does reduce the gap between ELBO and the log-likelihood of the input data and yield a tighter ELBO, which leads to a better semi-supervised performance.
Ablation Study:
Table 5: Generative performance measured by ELBO with ELBO < log (X)
Model
Cifar10
Cifar100
Pure VAE -226.25(+14.25) -12.2.91(+1110)
OSPOT-VAE -237.62(+6.27) -1271.82(224.15)
4.3 GENERATIVE PERFORMANCE
Eeeny (X) ELBO measures the margin between the true data distribution and the distribution learned by generation models (Doersch 2016). By comparing the Ex)mp(X) value of pure unsupervised VAE and our supesiueee VAF model under the same "WideResNet-28-2 backbone, we demonstrate that good generative models and semi-supervised results can be obtained at the same time in OSPOT- VAE. The results in Table 5 show that the data generative distribution learned by OSPOT-VAE model is as good as the pure VAE model. Further generated results are available in Appendix A.7
CONCLUSION
In this work, we pointed out that it was the large margin between ELBO and the true log likelihodd of the raw data that limits the performance of semi-supervised VAE. To this end, we introduced OSPOT-VAE, a one-stage generative model that unified the classification and generation objec- tive and achieved tighter ELBO by optimal transport estimation. We demonstrated our asser- tion through extensive experiments, and our sumi-supereised results significantly outperform former state-of-the- afrt generative SSL methods by large margin Cifar10 and 10f
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Luigi Ambrosio and Nicola Gigli. A users guide to optimal transport. 2013.
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach semi-supervised learning. CoRR, abs/1995.02222 2019.
C2249
Grigorios G. Chrysos, Jean Kossaifi, and Stefanos Zafeiriou. Robust conditional generative adver- sarial networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: https////ppereeeeeeeeom net Pooormmiii Byg0DsCqYQ
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation strategies from data. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Zihang Dai, Zhilin Yang, Fan Yang. William W. Cohen, and Ruslan Salakhutdi- nov. Good semi-supervised learning that requires bad GAN. In Advances Annual Conference on Neu- 4-9 December 2017, Long Beach,
in Neural Information Processing Systems 30:
ral Information Processing Systems 2017.
CA. USA. pp. 6510-6520, 2017. URL http::/papers nips cc/paper/
7229 good semi supervised Learning that requires bad gan
Carl Doersch. Tutorial on variational autoencoders. CoRR, abs/1606.05508,, 2016. URL http: www.ii.org.ab///eeeee......eee/e 05908
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April24-266 24-26, 2017, Conference Track Proceedings, 2017. URL https://cpenreviin net/ forum So
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi- supervised learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5070-5079, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzttoon with gumbel-softmax, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24- 26, 2017, Conference Truck Proceedings, 2017. URL/tttps::/openeeiiee net//orumm forum?
id 1d=rkE3y85ee
Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi- supervised learning with deep generative models. In Advances in Neural Information Processing
Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3581-3588 2014. URL ppppppppp/ppppes..nnnpp cc/ Saper/53522 semi superviii sed-
ea
Max Kuang and Esteban G. Tabak. Preconditioning of optimal transport. SIAM J. Scientific Computing, 39(4), 2017. doi: 10.1137/16M10749953, URL
16M1074953
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https:// http::///eppeeeeeeeeemee / net. forum?
id=BJ6o0fqgel
Wenyuan Li, Zichen Wang, Jiayun Li, Jennifer Polson, William Speier, and Corey W. Arnold. Semi- supervised learning based on generative adversarial network: comparison between good GAN and bad GAN approach. CoRR, abs/1905.06484, 2019. URL http:// htttp://rxiiv org/ abs/
i 1905. 06484

---

Under review as a conference paper at ICLR 2020
Yucen Luo, Jun Zhu, Mengxi Li, Yong Ren, and Bo Zhang. Smooth neighbors on teacher graphs for semi-supervised learning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018. Salt Lake City. UT. USA, June 18-22, 2018. pp. 8896-8905, 2018. doi: 10.1109) CVPR.2018.00922 URL httphtpp: htttt::///penaccessss theccf.....ccc www htt.c/////ootttcecceeece cwt.oottttttp__eeee____ 2018
html httt://uuo....oooo....eeeorrrrrp
wew_h_Neeghhh___oreee CVPR 2 parpp
paper. html
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A
regularization method for supervised and semi-supervised learning. JEEE Trans. Pattern Anal.
Mach. Intell., 41(8):1979-1993, 2019. doi: 10. 10. 1..1099/AMM.2208885588888
URL https://
doi.org/10, 10 1ww111199PPAAMMM 2018 2018.2858821 2858821
Siddharth Narayanaswamy, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank D. Wood, and Philip H. S. Torr. Learning disentangled rep- resentations with supervipe deep generative models. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 2017.
December 2017, Long Beach, CA, USA, pp.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian Goodfellow. Realistic evaluation of semi-supervised learning algorithms. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track net/ Beet//oruum1dd-BC225777
Proceedings, 2018. URL httpt::t/tpppeeeeeeeeeeeeeeeeeee
Danilo Jimenez. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 3/th International Con- ference on Machine Learning, ICML 2014, Beijing, China, 21 June 2014, pp. 1278-1286, 2014. URL
mlr pww.s/essseee...... https//////eeeeeeeeeeeeeeeee rww.ndezeeeeeeeeeee htmll
Tim Salimans, lan Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2234-2242. Curran Associates, Inc., 2016. URL
nips. ct..p.ccc/ppprr
Jost Tobias Springenberg. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:
//arxiv.orq wwt.o////aa//a......... 06310000000
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con-
sistency targets improve semi-supervised deep learning results. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track
Proceedings, 2017. URL https:/ http:://ppereeeiieeemmem net www.:neet/oornoa forum? pttprrmmmidrryyy2111
Philippe Thomas. Semi-supervised learning by olivier chapelle, bernhard schölkopf, and alexander zien (review). JEEE Trans. Neural Nenworks, 20(3):542, 2009. doi: 10. 2009/TNN.0099.211..
URL http::tss////o http///i.oogg/1/// 1101 TNN. 2009. 2015974
Vikas Verma, Alex Lamb, Christopher Beckham, Aaron C. Courville, Ioannis Mitliagkas, and Yoshua Bengio. Manifold mixup: Encouraging meaningful on-manifold interpolation as a regu- larizer. CoRR, abs/ 806.05236, 2018. URL /atttp://arxxe http://aarxxv..or///////////// wrgg 05236
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty-Eighih Inter- national Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 3635-3641, 2019. doi: 0.24966/jjcai.2019955 34963/ijcai. wttp://aaaa......... 2019/504
URL https://doi. wwr.//rr.....
Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved train- ing of wasserstein gans: consistency term and its dual effect. In 6th International Con- ference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL fttps::/openzeviees net net/forum forum?
id=SJx9GQb0-
10

---

Under review as a conference paper at ICLR 2020
Zhirong Wu, Yuanjun Xiong, Stella X. Yu. and Dahua Lin. Unsupervised fea- ture learning via non-parametric instance discrimination. In 2018 IEEE Confer- on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City,
UT, USA, June 18-22, 2018, pp- 3733-3742, 2018. doi: 10.1109/CVPR.0018 00393. http:/ htt.tp:p//ppeaccssssssss.s thecvf. com content_ovppppppeeeee evpr. 2018/htm /We. Unsupervised Feature Learning CVPR 2018 paper.htm.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoe V. Le. Unsupervised data augmentation. CoRR, 11/11944.1248,, 2019, URL
12848
Sergey Zagonuyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016.
URL http:://axxiv.or//bbbss.. abs/1bb5.... 1tt./11115000066666
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In 6th International Conference Learning Representationss ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL httpss://pperevieew.nrt net forum 1t
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational au- 02262
toencoders. CoRR, 7tnp//706.0222,, 2017. URL
A APPENDIX
A.I BASIC INEQUALITY OF ELBO
Proposition A.1. The Basic inequality of ELBO is
log (X) 2 E (x) plogoo (||2, c)] - DKL (2x(2) (e
proof
P(X, 2) 2s ( (z. c(
p(X, 2, c) (z x(X)
(XX log
U (())
log E (x,e
p(X, 2,
2 (x)(x log
20 Zz c X(X)
p(z, c)
/ 20(2, (ogpoo 2.0
g0(z, c log
2s (z, c(x)
2.C
DKL. (go (z, c X [lp(2, c)) + E [1/
with assumption (3.4) E (x) [log Po (Xiz, c)] DKL (z) (xa(oo(c))lp((e)
A.2 DECOMPOSITION OF ELBO IN INFO-VAE
Proposition A.2. The expected ELBO with empirical distribution satisfies
Epenp (x) DKL (x) Igo (X;z) + DKL (xL(go(z))pp(z)
proof
Epemp (x) (z) = Pemp (X) / ()) (x(2|X)
dzdx
p(z)
- Pempp (X) /f Gob (z)(2 (z(
dzddX Pempl()) / (z)() 20( dzdX
2 (z)
p(z)
[x 90( X) log
X)
dzdx + Dku(go(2)(pp(z)
4ol (z))pepp()) (X)
Igp (X;2) + DKL(Us (2) [[((2)

---

Under review as a conference paper at ICLR 2020
A.3 THE EQUATION FORM OF ELBO
Proposition A.3. The expected margin between the true log-likelihood pxemp(X) log p(X) and
E peemp (X) EELBO ()ELBO
E Pemp (X) (x) [lgg((X - ELBO) = E Eeemp (x) (PxLlg + Drs.((x((c/X))]xcxxXx))
proof
We just need to prove the following equation
p(X) ELBO (x/ugszz/)))p/zzll Dxx(go(c/X)(p((xx
and the proof under assumption (3, 4) is
log x(X) = / 4 (44 (c) X)
log pl X dzdc / 90(2/X))o(c( log
p z. c)
p(z O((cX))(c((X) dzdc
XX 2, c)
/ (olexX)ao(ci))oog
20 (g
(xx log
9Φ (z) qo (c()
p(z(X)p(c|X()
=ELBO+ + (zl( X)) (xx((o(c(X)lp(xxx
A.4
OPTIMAL TRANSPORT SCHEME
Proposition A.4.
The shortest-path derived from optimal transport scheme (17) between 21 NN diag(( (a}))
and 22 ~ (N2,, diag(o?)) with À E [0, 1] is
j Ap1 + (1 11-Mi 22)p2
(A.1)
σ = Ao + (1 A)oz
proof
Utilizing the conclusions in Kuang & Tabak (2017). the closed form of optimal transport from one
multi-normal distribution N (zz: M1. Σn) another normal distribution N ((zz u2, Eg)
T(z) = μ2 + T(z (4);; T x イ
Utilize the diag matrix assumption, the optimal transport scheme with À is
n = (1 A)21 + (Tz)
(A.2)
T
Utilize (A.2), we can get (A.1) ) as
Zx (ñ, doag(
Ap1 + (1 A)p
ã= to (1- \)oz
2. The shortest-path derived from KL divergence based optimal transport scheme between C₁ ~ Mult( K, T)) and C2 Mult( T2) with À [0, is
मे = \m + (1 2)
(A.3)
proof
As the definition (17) has no closed-form solution for multinomial distribution, we use KL diver- gence instead. The KL divergence based optimal transport target ~ Mult( K, ボ) between two multinomial distribution e₁ ~ Mult(K, πι) and C2 ~ Mult(K, T2) with À E [0, 1] satisfy
min AD ( n [T)
(1 A)DKL
8.1.
#
(A.4)
The Lagrange multiplier form of (A.4) is
K +t* π 1)
C(+,t) = ADKL #) + (1 A)DKL(TTll
12

---

Under review as a conference paper at ICLR 2020
Table 6: Schedule Parameters
MNIST
SVHN
Cifar10
Cifar100
Amaz
Ima
Aymar
Emar
Rymax
Imaz
Amaz
Tmax
Ayaa
Emar
30
50
1
175
le-3
150
le-3
150
le-1
150
Dc
30
50
1
175
1
150
1e-3
150
le-3
150
Ix
17.5
50
50
175
1280
150
200
150
1280
150
Ic
17
50
50
175
2.3
150
2.3
150
4.6
150
WM,
le-3
150
1e-3
150
le-1
150
WM
400
280
1
280
and the related KKT conditions are
aC(T,,t t)
AT1 + (1 - A)TT
t
0
π
K
(A.5)
t
n. - 1)
Solve the equation (A.5), we can get the closed form of स
म Am + (1 A)T22
LI
A.5
SCHEDULE ANALYSIS
10
0.8
0.6
h(t)
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
10
epoch/epochmax
Figure The Exponential Function-Baan -Based Scheduler
The warm-up scheduler aims to slowly increase the parameters until they reach their maximum. For a certain hyperparameter h, there are parameters control its warm-up process, the target value hmaz and the total epoch tmar reach the target value. We use the exponential function get the middle value h as
h = hmaz exp (-5 (1 minn11,4(mmm)))
(A.6)
The curve (A.6) shown Figure and we list the scheduler parameters of 4 benchmark datasets, i.e. MNIST, SVHN, Cifar10, Cifar100, in Table
13

---

Under review as a conference paper at ICLR 2020
Table Details of Training Process
MNIST
SVHNonee-tagge)
SVHN
Cifar10
Cifar100
Latent Dim(z/c) c)
10/I0
32/10
128/I0
128/I0
128/I00
Mutual Info(z/o) /c)
17.5/17.0
50/50
1280/2.3
200/2.3
1280/4.6
loss of - log Pe X(c,
BCE
BCE
MSE 2
MSE
BCE
of Pmizup X)
2
2
optimizer
Adam
SGD
learning rate
5e-4
le-3
0.1
every 50 epoch after 200-th(0.5)
Ir scheduler(decay ratio)
[500,600,650(0.22)
weight decay
0
0
5e-4
A.6 DETAILS OF TRAINING PROCESS
Here we list some important items need to set in the training process for different process. We classify these items into categories: (1) items related to the loss function and (2) items related to the optimization strategy. The details are as follows and we list the exact value in Table 7
Items related to the loss function
Latent dim
The latent dim of discrete variable is the same the number of classifications, that is, K. For continuous variable z, the latent dim is determined by experiments.
Mutual information
We find the value of continuous mutual information will affect the generative per- formance, but have little impact semi-supervised learning results, so we choose suitable value to get the best generative performance, For discrete information, we choose the value in the ideal scene, that is, I. log(K).
Calculation of
log Po (X
PX c, z) has two forms: (1) normal distribution with (f, and (2) multi- nomial distribution with Mult(dim ddmm fo z)). For the two forms, the loss func- tion of log Pa [X|c, is mean square error(MSE) and binary cross entropy(BCE) respectively. We use reparameterization trick (20) to approximate expectation, and the sampling frequency 1.
of Pmiiuup (X)
The B(a, a) for mixup vicinal distribution will strongly affect SSL performance, set it to in all experiments.
2. Items related to the optimization strategy
Optimizer
In one-stage VAE, we use Adam. OSPOT-VAE, we use SGD with momentum 0.9.
Learning
We set the initial learning rate to 0.1 in OSPOT-VAE, which obtains best SSL perfor- mance.
The scheduler of adjusting learning rate We decay the learning ratio in some milestones with the specified decay rate.
Weight decay
Weight decay controls the strength of L2 regularization.
We also list some standard training curves on benchmark datasets in Figure 3 and
14

---

Under review as a conference paper at ICLR 2020
100
0.92
80
0.88
60
0.84
10
20
0.76
0
100
200
300
400
500
600
500
600
Figure 3: The training curves of Cifar10.
Left: classification performance. Right: (ex - log Pe ( (X(c, 2)
M
200
0.65
160
0.55
80
0:35
0
100
200
300
400
500
600
200
300
500
600
Figure 4: The training curves of Cifar100.
Left: classification performance. Right: (e., log po (epipoo((eee
2)
A.7
GENERATIVE PERFORMANCE
Following 5 show the generation performance of OSPOT-VAE.
0
0
0
0
0
0
0
0
O
0
0
/
/
1
1
1
1
\
1
2.
2
2
2
2
2.
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
5
54
5
5
6
6
6
6
6
6
6
6
6
6
6
6
6
6
6
7
7
7
7
7
7
7
7
7
7
7
7
7
7
.
8
8
8
8
8
8
8
8
8
8
8
8
8
8
9
9
9
9
9
9
9
8
9
9
9
9
(a) MNIST
(b) SVHN
Figure 5: The generative performance of OSPOT-VAE in MNIST and SVHN
15

---

Under review as a conference paper at ICLR 2020
(a) Pure VAE
OSPOT-VAE
Figure 6: Compare the generative performance of pure VAE and OSPOT- VAE in Cifar10
(a) Pure VAE
OSPOT- VAE
Figure 7: Compare the generative performance of pure VAE and OSPOT-VAL in Cifar
16


Reviews:
[{"review_id": "HkeHLmrhFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to solve the existing problems in semi-supervised learning by employing optimal transport theory and proposes a one-stage training VAE, OSPOT-VAE, which has a tighter ELBO and demonstrates better performance on benchmark datasets like CIFAR-10 and CIFAR-100 than benchmark methods. One main contribution in the paper is to use optimal transport to achieve a tighter ELBO in the proposed framework.\n\nMy main concern is that it appears in the paper the authors are trying to use a linear transport to approximate the intractable posterior distribution p(z|X) and p(c|X). I am not sure how this can be achieved and there is no discussions in the paper about this or how to reduce the approximation errors.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJgqv01pYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions:\n\nThe main contribution of this paper lies in the proposed OSPOT-VAE for semi-supervised classification. The proposed model is (i) an one-stage framework that unifies the generation and classification loss in one ELBO, (ii) achieves a tighter ELBO by applying optimal transport to the distribution of the latent variables, and (iii) achieves SOTA semi-supervised learning results without sacrificing generative performance on many benchmarks. \n\nStrengths:\n\n(1) Novelty: This paper contains novelty in terms of two aspects. First, the derivation of Eqn. (13) unifies both generation loss and classification loss naturally. The derivation seems different from a standard semi-supervised VAE. Second, the use of optimal transport (OT) estimation to tighten the ELBO results in improved classification performance. The use of OT seems novel in this context. \n\n(2) Writing: The paper is generally well written. However, I also found section 3 is hard to follow, with details in the Weaknesses section.\n\n(3) Experiments: The results look impressive. OSPOT-VAE achieves better performance than other generative approaches. \n\nWeaknesses:\n\n(1) Clarity: My biggest concern lies in the clarity of section 3, which I think is confusing in its current shape. After several rounds of reading, below is my understanding. In section 3.1, the authors first derive the objective in Eqn. (13). This objective is derived based on the use of mutual information (Eqn. (8)). By the end, no special cross-entropy loss term like in Eqn. (7) is needed.  Second, in order to tighten the bound, section 3.2 introduces optimal transport estimation. Combining these two losses together (see in Algorithm 2) results in the final algorithm. However, generally, I feel the presentation of section 3 is confusing. One may need to present the big picture (framework) first, and then dig into details.\n\nBesides that, I have some questions as below. \n\na) There are many hyper-parameters in the final objective, for example, the mutual information I_z, I_c is also considered as hyper-parameters. How all these hyper-parameters are selected?\n\nb) Is there any intuition on why Eqn. (13) is already enough without introducing the empirical cross-entropy loss like in Eqn. (7)? It seems to me adding this additional cross-entropy loss will not hurt performance. \n\nc) In Eqn. (12), how is the last term D_{KL}(p(c|X)||q_{\\phi}(c|X)) calculated, since we cannot get the true p(c|X)?\n\nd) As shown in Table 4, the use of optimal transport seems to be the key. However, section 3.2 is hard to follow. For example, In Algorithm 1, Line 6 & 7 considers the true posterior of z and c is a Gaussian and a multinomial distribution, respectively. Why doing this? Supported by theory? I also feel the calculation of L_{Rz} and L_{Rc} in Line 8 & 9 is hard to follow. \n\ne) How reliable is this optimal transport estimation is? And how it contributes to the final classification performance? Since this loss term is not a classification loss, then is there any intuition on why it serves as a regularization to help the final performance so much?\n\nf) Results in Table 5 on Cifar10 seem not imply OSPOT-VAE tightens the ELBO. Adding results on MNIST will be better, since the ELBO on MNIST is widely benchmarked. \n\n(2) Minor:\n\na) I am not sure why section 2.1 is needed. Also, the feature matching GAN paragraph in section 2.2 seems also unnecessary. If you really want to discuss the use of GAN for semi-supervised learning, then there is a lot of other work that needs discussion.\n\nb) There is a missing right bracket in the second term of Eqn. (9).\n\nOverall, I think the results look quite impressive. However, the presentation of the method part is unclear to me, and could be much improved. \n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxNP6tnKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to combine a VAE model with the Optimal Transport to approximate some components of the model. The authors evaluate their approach on semi-supervised problems and claim to obtain very competitive results compared to literature. Unfortunately, the paper is very unclear and hard to follow. The authors make some claims that are not true, for instance, learning so called M1+M2 architecture end-to-end is hard, however, there are papers that successfully train such model. Moreover, the authors reported their results as SOTA, however, they missed many other papers with much better scores.\n\nRemarks:\n- The authors claim in the introduction that training a two-level VAE with a classifier (so called M1+M2 architecture) is not robust. However, there are papers that were able to train such model without any reported problems, for instance:\n* Louizos, C., Swersky, K., Li, Y., Welling, M., & Zemel, R. (2015). The variational fair autoencoder. arXiv preprint arXiv:1511.00830.\n* Davidson, T. R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J. M. (2018). Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891. (published at UAI 2018)\n* Ilse, M., Tomczak, J. M., Louizos, C., & Welling, M. (2019). DIVA: Domain Invariant Variational Autoencoders. arXiv preprint arXiv:1905.10427.\n\n - The authors report SOTA results on MNIST (100 labels). However, there are papers that report better scores, for instance 2.6 in:\nDavidson, T. R., Falorsi, L., De Cao, N., Kipf, T., & Tomczak, J. M. (2018). Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891. (published at UAI 2018)\nIn this paper they used a VAE model (the M1+M2 architecture) that was trained end-to-end.\n\n- A rather minor remark, but I do not fully see a reason why the authors spent a lot of space on Section 2. I believe most of this text could be included in the Appendix.\n\n- Equation 8: The authors claim that this decomposition was introduced in (Zhao et al., 2017), however, it was done earlier:\nHoffman, M. D., & Johnson, M. J. (2016, December). ELBO surgery: Yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS.\n\n- Equation (16) is very vague and hard to follow. Moreover, the idea of using Optimal Transport is also hard to follow.  The authors present a very generic description of the Optimal Transport, and then refer to an algorithm (a pseudocode). It is very unreadable.\n\n===== AFTER REBUTTAL =====\nI would like to thank the authors for the rebuttal. I read the comments carefully. However, I am not still convinced by claims of the paper. It is still vague to me why the proposed training procedure is necessary. Therefore, I decided to keep my original score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to combine a VAE model with the Optimal Transport to approximate some components of the model. The authors evaluate their approach on semi-supervised problems and claim to obtain very competitive results compared to literature. Unfortunately, the paper would benefit substantially from revisions to make it easier to follow. For this reason, the paper is not ready for publication in this venue at this time.
Decision: Reject