Title: GOOD SEMI-SUPERVISED VAE REQUIRES TIGHTER EVIDENCE LOWER BOUND Anonymous

Abstract: Semi-supervised learning approaches based on generative models have now encountered 3 challenges: (1) The two-stage training strategy is not robust. (2) Good semi-supervised learning results and good generative performance can not be obtained at the same time. (3) Even at the expense of sacrificing generative performance, the semi-supervised classification results are still not satisfactory. To address these problems, we propose One-stage Semi-suPervised Optimal Transport VAE (OSPOT-VAE), a one-stage deep generative model that theoretically unifies the generation and classification loss in one ELBO framework and achieves a tighter ELBO by applying the optimal transport scheme to the distribution of latent variables. We show that with tighter ELBO, our OSPOT-VAE surpasses the best semi-supervised generative models by a large margin across many benchmark datasets. For example, we reduce the error rate from 14.41% to 6.11% on Cifar-10 with 4k labels and achieve state-of-the-art performance with 25.30% on Cifar-100 with 10k labels. We also demonstrate that good generative models and semi-supervised results can be achieved simultaneously by OSPOT-VAE.

1. INTRODUCTION:

The rise of deep neural networks has led to breakthroughs in computer vision, natural language processing, and many other domains. Most of these models are trained on large labeled datasets via supervised learning. However, in many scenarios, although it is easy to acquire a large amount of the original data, obtaining corresponding labels is often very costly or even infeasible. Semisupervised learning (Thomas, 2009) is proposed to address this problem by training classifiers with sufficient unlabeled data and a small fraction of labeled data.

Recent works on semi-supervised learning can be grouped into three categories: (1) disagreement based learning via data perturbation (Miyato et al., 2019) and consistency enforcing (Verma et al., 2019) , (2) metric learning (Wu et al., 2018) , (3) generative approaches via generative adversarial network (GAN) (Springenberg, 2016) and variational autoencoder (VAE) (Kingma et al., 2014) . Compared with the first two categories, generative approaches have great advantages in interpretability. Based on the latent variable assumption (Doersch, 2016) , the generative model has an explicit variational inference form, so it can learn the marginal probability distribution of the raw data as well as the conditional distribution of the latent variables given the input data, which makes predictions more reasonable. Besides, generative approaches not only learn the required classification representations, but also capture the semantics-disentangled factors that generate the data, making it easier to generalize to different tasks (Narayanaswamy et al., 2017) .

However, in practice, semi-supervised generative approaches often encounter three major challenges: (1) The two-stage training process is not robust. Semi-supervised VAE (Kingma et al., 2014) needs to be trained carefully with a two-stage hierarchical strategy, while the training process of GAN is a two-stage adversarial game (Chrysos et al., 2019) . (2) Good semi-supervised learning results and good generative performance can not be obtained at the same time. In GAN, good semi-supervised learning performance will lead to a mismatch between the generated results and the real data distribution (Dai et al., 2017) . While in VAE, the evidence lower bound (ELBO) objective is irrelevant to the classification loss, making it difficult to learn from the labels directly (Narayanaswamy et al., 2017) . (3) Even at the expense of sacrificing generative performance, the semi-supervised classification results are still not satisfactory. In practice, disagreement-based meth- ods (Xie et al., 2019; Berthelot et al., 2019) have dramatically improved the state-of-the-art results on several standard datasets, surpassing generative approaches by a large margin. These challenges naturally raise a question: What limits the performance of generative approaches in semi-supervised learning?

In this work, we propose One-stage Semi-suPervised Optimal Transport VAE (OSPOT-VAE) to address these challenges, which consists of two improvements:

(1) a one-stage semi-supervised VAE model that unifies the generation and classification loss in one ELBO framework.

(2) an estimation of the margin between true log-likelihood and the ELBO that exports a tighter evidence lower bound by applying optimal transport (Ambrosio & Gigli, 2013) scheme to the distribution of latent variables.

Our model has the following contributions:

• We show that OSPOT-VAE can be well trained with a direct one-stage strategy.

• We show that OSPOT-VAE can achieve both good generative performance and semisupervised learning results simultaneously on a series of benchmark datasets. • We point out that it is the large margin between the ELBO and the log-likelihood of the input data that limits the performance of semi-supervised VAE. Besides, we evaluate this assumption across many standard datasets and show that with the proposed tighter ELBO, OSPOT-VAE surpasses the best semi-supervised generative models by a large margin and achieves state-of-the-art performance on Cifar-100 with 10k labels.

2. SEMI-SUPERVISED LEARNING METHODS:

In supervised learning (SL), we are facing with training data that appears as input-target pairs (X, y) ∈ D L sampled from an unknown distribution p(X, y). Our goal is to learn a function f (X; φ) parameterized by φ that makes the correct inference y for unseen samples from p(X).

While in semi-supervised learning (SSL), we can obtain an extra collection of unlabeled data X ∈ D U sampled from the same distribution p(X). We hope to leverage the data from both D L and D U to achieve a more accurate model than what would have been obtained by only using D L .

In this section, we review some existing methods for SSL. We mainly focus on those who have reached state-of-the-art results, as well as generative approaches which are strongly connected with our model; the more comprehensive overview is beyond the scope of this paper, we refer readers to (Oliver et al., 2018) .

2.1. DISAGREEMENT BASED LEARNING:

Disagreement-based learning refers to the general approaches of imposing disagreement among multiple learners on the same task or multiple predictions from a single learner. By eliminating the disagreement, we can enforce the generalization of the model on unseen data. A common technique for creating disagreement is data augmentation, which applies transformations or perturbations on the input data and leaves class semantics unchanged. For X ∈ D U , loss term can be derived as

f (Augment(X); φ) -f (X; φ) 2 2 (1)

where the Augment(X) is a stochastic function which can be obtained by image transformation (Xie et al., 2019) , virtual adversarial training (Miyato et al., 2019) , or mixup method (Verma et al. 2018; Berthelot et al. 2019) . Another disagreement construction technique is to train multiple learners on the same dataset and utilize the loss f (X; φ 1 ) -f (X; φ 2 ) 2 2

(2) to enforce the predictive consistency of different models, for example, "Mean Teacher" (Tarvainen & Valpola, 2017) and "Teacher Graph" (Luo et al., 2018) . The generalization of the models gets enhanced.

2.2. GENERATIVE APPROACHES:

In generative approaches, input X is supposed to have corresponding continuous and discrete latent variables, which we denote by z and c respectively.

Feature matching (FM) GANs (Salimans et al., 2016; Dai et al., 2017) apply GANs to semisupervised learning on K-classification tasks by specifying a (K+1)-class objective for the discriminator. Instead of binary classification, true samples are classified into the first K classes respectively and fake samples are classified into the (K+1)-th class. This target function achieves strong empirical results by matching the generator distribution with true data distribution and improves semi-supervised classification performance.

Semi-supervised VAEs (Kingma et al. 2014; Narayanaswamy et al. 2017 ) construct a probabilistic model parameterized by θ and φ that respectively describe the generation and inference process between X and latent variables z, c. The generation process of X by z and c is :

EQUATION

) where Mult(K, π) is the multinomial distribution with class K and parameter π. f (X; z, c, θ) is a suitable likelihood function, e.g. a Bernoulli or Gaussian distribution, parameterized by a non-linear transformation of the latent variables z and c. The class label y is treated as c if given. For the inference process, with the following hypothesis

EQUATION

) evidence lower bound (ELBO) is used as objective to predict the posterior distribution of latent variables as follows (see Appendix A.1 for proof):

log p(X) ≥ E q φ (z,c|X) [log p θ (X|z, c)] -D KL (q φ (z|X) p(z)) -D KL (q φ (c|X) p(c)) = ELBO

(5) For the likelihood log p(X) is infeasible, VAE maximizes its evidence lower bound instead, which derives the negative ELBO loss function L(X; φ, θ) = -ELBO. The predictions for the classification label y can be obtained from the inferred posterior distribution q φ (c|X). When class label y is not given, missing label sampling technique is used to sample from q φ (c|X) as

EQUATION

Here y k represents a one-hot vector with 1 in k-th dimension. Utilizing this sampling method, the algorithmic complexity of VAE is proportional to K so it is computationally inefficient. Note that in objective (4), the label predictive distribution q φ (c|X) only contributes to the generative performance. To remedy this, existing models simply add a cross-entropy loss to the negative ELBO loss such that the distribution q φ (c|X) can also learn classification rules from the labeled data. The extended objective loss is min This M1+M2 strategy builds a deep VAE with two layers of random variables:

EQUATION

p θ (X, z 1 , z 2 , c) = p θ (X|z 1 )p θ (z 1 |z 2 , c)p(z 2 )p(c)

, which can dramatically improve the performance of the inference q φ (c|X) but is not robust in training. Moreover, GAN's training process can also be considered as two-stage with generator and discriminator competing with each other, and the two-stage adversarial game adds the difficulty in training.

3. ONE-STAGE SEMI-SUPERVISED OPTIMAL TRANSPORT VAE:

In this section, we introduce our semi-supervised VAE framework, OSPOT-VAE. Firstly, we derive a one-stage loss function that unifies the generation and classification loss under one ELBO without introducing any additional auxiliary loss items like (7). Then, we analyze a phenomenon that good ELBO values do not guarantee good semi-supervised performance and propose the optimal transport estimation to deal with it. At last, combining the two parts, we give the detailed algorithm of OSPOT-VAE and discuss some problems in model optimization.

3.1. ONE-STAGE SEMI-SUPERVISED VAE:

Following the notations and assumptions (3, 4) in Section 2.2, we derive our one-stage semi-

supervised VAE. With the empirical distribution p emp (X; D) = 1 |D| X ∈D

1 X=X , we utilize the decomposition in (Zhao et al., 2017) and rewrite the second part of (5) into (proof in Appendix A.2)

EQUATION

where q φ (z) = 1 |D| X∈D q φ (z|x) and I q φ (X; z) is the mutual information between X and z. The left part of (8) equals to 0 when X and z are independent. This is undesirable, so I q φ (X; z) can be regarded as the lower bound of controlled mutual information. The continuous variables in (8) can be easily extend to discrete variables c. We can use I z and I c to denote the controlled information capacity and derive the objective for the unlabeled dataset

EQUATION

where β, I z , I c are all hyper-parameters forcing the KL divergence term to match the mutual information capacities of z and c.

For the labeled subset D L , instead of directly employing class label y as sampled c, we view it as the parameter of the true posterior distribution, i.e. p(c|X) = Mult(c; K, y) and derive the following one-stage ELBO form:

log p(X) = log E q φ (z|X),p(c|X) p(X, z, c) q φ (z|X)p(c|X) ≥ E q φ (z|X),p(c|X) log p(X, z, c) q φ (z|X)p(c|X) = E q φ (z|X),p(c|X) [log p(X|z, c) + log p(z)p(c) q φ (z|X)p(c|X) ] = E q φ (z|X),p(c|X) log p(X|z, c) -D KL (q φ (z|X) p(z)) -D KL (p(c|X) q φ (c|X)) + E p(c|X) log p(c) q φ (c|X) (10)

Notice that D KL (p(c|X) q φ (c|X)) is equal to the common cross-entropy loss for y is a one-hot vector. In this respect, the margin between p(c|X) and q φ (c|X) can be significantly small when the suitable optimization method is chosen. This allows us to utilize the approximation p(c|X) ≈ q φ (c|X) to modify E p(c|X) log p(c) q φ (c|X) in (10), resulting in a consist ELBO with L D U (X; θ, φ):

E p(c|X) log p(c) q φ (c|X) ≈ (when p(c|X)≈q φ (c|X)) E q φ (c|X) log p(c) q φ (c|X) = D KL (q φ (c|X) p(c)) (11)

Combining the ELBO form (10) of D L with the mutual information decomposition (8) and the approximation (11), the new objective for semi-supervised VAE is:

EQUATION

With (9) and (12), the objective for the entire dataset is now min

EQUATION

This one-stage objective with a simple approximate transformation (9) unifies the generation loss as well as the target of SSL and results in improved performance of semi-supervised learning, which we demonstrate in Section 4.1.

Algorithm 1 Optimal transport estimation ingests a batch of observation X as well as the representation q φ (z|X), q φ (c|X) inferred from the original VAE and returns the estimation of the margin D KL (q φ (z|X) p(z|X)) and D KL (q φ (c|X) p(c|X)).

Input::

Batch of observation X sampled from p emp (X); Inferred parameter (µ, diag(σ 2 )) of q φ (z|X) = N (z; µ, diag(σ 2 )); Inferred parameter π of q φ (c|X) = Mult(c; K, π); Hyperparameter α for mixup vicinal distribution p mixup (X) Output:

X sampled from p mixup (X);

Estimation L Mz of the margin D KL (q φ (z| X) p(z| X)); Estimation L Mc of the margin D KL (q φ (c| X) p(c| X)) 1: X , π , µ , σ 2 = RandomPermutation(X, π, µ, σ 2 ) 2: X = λ * X + (1 -λ) * X , λ ∈ β(α, α) 3: π = OptimalTransportC(π, π , λ) 4: ( μ, σ2 ) = OptimalTransportZ((µ, σ 2 ), (µ , σ 2 ), λ) 5: q φ (z| X), q φ (c| X) = VAE( X) 6: p(z| X) = N (z; μ, diag( σ2 )) 7: p(c| X) = Mult(c; K, π) 8: L Mz = D KL (q φ (z| X) p(z| X)) 9: L Mc = D KL (q φ (c| X) p(c| X)) 10: return X, L Mz , L Mc

3.2. OPTIMAL TRANSPORT ESTIMATION:

To summarize the above, VAE aims to learn the useful representation q φ (z|X) and q φ (c|X) by reducing the KL divergence between the empirical distribution p emp (X) and the model marginal p(X) = z c p(X)p(z|X)p(c|X)dzdc. Instead of minimizing D KL (p emp (X) p(X)) directly, VAE models use the expected ELBO mentioned in (5) as target via the following inequality

D KL (p emp (X) p(X)) ≤ H(p emp (X)) -E pemp(X) ELBO (14)

However, one phenomenon is that good ELBO values do not imply accurate inference. A typical example has been discussed in (Zhao et al., 2017) . Here we mainly focus on the cause of this phenomenon and propose optimal transport estimation to alleviate this problem in semi-supervised learning. Following the work in (Rezende et al., 2014), we write down the closed form of the expected margin between true log-likelihood and ELBO as (proof in Appendix A.3):

E pemp(X) [log p(X) -ELBO] = E pemp(X) [D KL (q φ (z|X) p(z|X)) + D KL (q φ (c|X) p(c|X))]

(15) Combined with the decomposition (8), training the expected ELBO target can only reduce the difference between marginal distributions q φ (c),q φ (z) and p(c),p(z). It means that even with a good ELBO, the margin E pemp(X) D KL (q φ (z|X) p(z|X)) and E pemp(X) D KL (q φ (c|X) p(c|X)) in (15) can still be large. In this scenario, the consistent optimization of ELBO will contribute no more to the semi-supervised classification performance. However, optimizing the margin in (15) directly is impossible, for p(c|X) and p(z|X) are unknown. To remedy this, we extend the empirically effective approximation in (Zhang et al., 2018) to our VAE framework with the form

E pmixup(X) D KL (q φ (z|X) p(z|X)) ≈ E pemp(X) D KL (q φ (z|X) p(z|X))(α → 0) E pmixup(X) D KL (q φ (c|X) p(c|X)) ≈ E pemp(X) D KL (q φ (c|X) p(c|X))(α → 0) (16)

where p mixup (X) is the mixup vicinal distribution (Zhang et al., 2018) and α is the related parameter. Then we propose optimal transport estimation to construct the estimations of E pmixup(X) D KL (q φ (z|X) p(z|X)) as well as E pmixup(X) D KL (q φ (z|X) p(z|X)) by applying optimal transport scheme to latent variables z and c. The computation steps of optimal transport estimation are provided in Algorithm 1, and we present the details of the optimal transport scheme in the rest of this section. Updated parameters:

θ (t) , φ (t) 1: L L = L D L (X L , y L ; θ (t-1) , φ (t-1) ; β z , β c , I z , I c ) 2: L U = L D U (X U ; θ (t-1) , φ (t-1) ; β z , β c , I z , I c ) 3: L Mz , L Mc = OptimalTransportEstimation(X U , q φ (z|X U ), q φ (c|X U )) 4: L = L L + L U + w Mz L Mz + w Mc L Mc 5: θ (t) , φ (t) = SGD(θ (t-1) , φ (t-1) , ∂L ∂θ , ∂L ∂φ ) 6: return θ (t) , φ (t)

Optimal Transport Scheme: The mixup vicinal distribution can be understood as applying linear transport between the points X, X ∈ D, extending the original dataset with new points falling on one straight line

X = λ * X + (1 -λ) * X , λ ∈ [0, 1].

For X, it is a natural thought that this linear transformation could associate with the shortest-path transport in the latent space. Based on this, we calculate the distributions p(z| X), p(c| X) of z, c and consider them as the estimation of the true posterior distributions. Following the work of (Ambrosio & Gigli, 2013) , the norm-2 based optimal transport scheme γ(x, y) between two distributions p(x) and p(y) satisfy: 

For the continuous variable z ∼ N (µ, diag(σ 2 )) and discrete variable c ∼ Mult(K, π), the following 2 propositions are proposed to calculate the shortest-path based on optimal transport scheme (see Appendix A.4 for proof). Proposition 3.1. The shortest-path derived from optimal transport scheme (17

) between z 1 ∼ N (µ 1 , diag(σ 2 1 )) and z 2 ∼ N (µ 2 , diag(σ 2 2 )) with λ ∈ [0, 1] is μ = λµ 1 + (1 -λ)µ 2 σ = λσ 1 + (1 -λ)σ 2 (18)

Proposition 3.2. The shortest-path derived from KL divergence based optimal transport scheme between c 1 ∼ Mult(K, π 1 ) and

c 2 ∼ Mult(K, π 2 ) with λ ∈ [0, 1] is π = λπ 1 + (1 -λ)π 2 (19)

Algorithm 1 yields the optimal transport estimation of the margin in (15), which leads to a tighter ELBO. In Section 4.2, we demonstrate that with this tighter ELBO, the inference performance of semi-supervised VAE is significantly improved on many benchmark datasets.

3.3. OPTIMIZATION OF OSPOT-VAE:

Combining one-stage semi-supervised VAE and optimal transport estimation, we can get the complete OSPOT-VAE model. The full OSPOT-VAE algorithm is provided in Algorithm 2, and a schematic is shown in Figure 1 . Note that the conditions for the approximations used in Algorithm 1,2 satisfy (1) q φ (c|X) ≈ p(c|X) and (2) the VAE model has already achieved a good ELBO. Therefore, the warm-up schedule (Higgins et al., 2017) is used to set parameters I z , I c , β z , β c and w Mz , w Mc . We list the details of "ELBOScheduler(t)" and "WeightScheduler(t)" in Appendix A.5.

In Algorithm 2, we apply stochastic gradient descent (SGD) as optimizer, which needs to calculate the gradient ∇ θ,φ L. Mean Teacher (Tarvainen & Valpola, 2017) 15.87 VAT+EntMin (Miyato et al., 2019) 13.13 MixMatch (Berthelot et al., 2019) 6.37

Generative GS-BadGAN (Zagoruyko & Komodakis, 2016) , but belongs to one kind of its variations with a comparable amount of parameters.

of E q φ (z|X),q φ (c|X) log p θ (X|z, c) is difficult. To this end, we follow the work of (Rezende et al., 2014) and (Jang et al., 2017) , using the reparameterization trick as ∇ θ,φ E q φ (z|X) log p θ (X|z) = E N ( ;0,I )∇ θ,φ log p θ (X|µ + σ • ) E Gumbel( ;0,1) )∇ θ,φ log p θ (X|Softmax(

EQUATION

Note that with (20), the algorithmic complexity of one-stage semi-supervised VAE is independent with the class number K, making it easier to extend to large-scale classification tasks.

4. EXPERIMENTS:

In this section, we demonstrate the 3 contributions of our OSPOT-VAE model with sufficient experiments on 4 standard SSL benchmark datasets, that is, MNIST, SVHN, Cifar10, and Ci-far100. In Section 4.1, we show the validity of our one-stage semi-supervised VAE objective (13) by comparing with other one-stage and two-stage VAE models. Then, we evaluate the performance of OSPOT-VAE under "WideResNet" (Zagoruyko & Komodakis, 2016 ) backbone and compare with other state-of-the-art SSL models mentioned in Section 2. Besides, We provide an ablation study to verify the contribution of the optimal transport estimation. As an additional application, we show that good generative models and semi-supervised results can be obtained at the same time by OSPOT-VAE (Section 4.3). The source code is available at https: //github.com/PaperCodeSubmission/OSPOT-VAE; more details are available in Appendix A.6.

We evaluate the effectiveness of the one-stage semi-supervised VAE objective on 2 standard benchmarks, MNIST and SVHN. As for baseline models, we consider two VAE-based SSL models, which are one-stage disentangled VAE (Narayanaswamy et al., 2017) and two-stage VAE(M1+M2) (Kingma et al., 2014) . For fairness, except the target loss functions, all models use the same structure as is used in M1+M2 (Kingma et al., 2014) . The results are presented in Table 1 , and our model achieves the best performance. (2) an optimal transport estimation. In ablation study, we analyze the effect of each component in our model with the backbone "WideResNet-28-2". To study the independent effects of transport estimation, we combine it with the encoder part of OSPOT-VAE to build a classifier with loss function L Mc . The improved classification error rates in Table 4 show that, with optimal transport estimation, the posterior inference q φ (c|X) gets closer to the true distribution p(c|X). It indicates that our optimal transport estimation does reduce the gap between ELBO and the log-likelihood of the input data and yield a tighter ELBO, which leads to a better semi-supervised performance. 

4.3. GENERATIVE PERFORMANCE:

E pemp(X) ELBO measures the margin between the true data distribution and the distribution learned by generation models (Doersch, 2016) . By comparing the E pemp(X) value of pure unsupervised VAE and our semi-supervised VAE model under the same "WideResNet-28-2" backbone, we demonstrate that good generative models and semi-supervised results can be obtained at the same time in OSPOT-VAE. The results in Table 5 show that the data generative distribution learned by our OSPOT-VAE model is as good as the pure VAE model. Further generated results are available in Appendix A.7.

5. CONCLUSION:

In this work, we pointed out that it was the large margin between ELBO and the true log-likelihood of the raw data that limits the performance of semi-supervised VAE. To this end, we introduced OSPOT-VAE, a one-stage generative model that unified the classification and generation objective and achieved a tighter ELBO by optimal transport estimation. We demonstrated our assertion through extensive experiments, and our semi-supervised results significantly outperform former state-of-the-art generative SSL methods by a large margin on Cifar10 and Cifar100. The warm-up scheduler aims to slowly increase the parameters until they reach their maximum. For a certain hyperparameter h, there are 2 parameters control its warm-up process, the target value h max and the total epoch t max to reach the target value. We use the exponential function to get the middle value h t as h t = h max × exp [-5 * (1 -min(1, t/t max )) 2 ] (A.6)

The curve of (A.6) is shown in Figure 2 , and we list the scheduler parameters of 4 benchmark datasets, i.e. MNIST, SVHN, Cifar10, Cifar100, in Table 6 . 

:



Figures and Tables:

Figure 1: The schematic of OSPOT-VAE

Two-stage Training Strategy: In practice, (Kingma et al., 2014)  finds that directly training the one-stage objective (7) will lead to a bad semi-supervised learning result, so a two-stage training strategy is proposed to improve the model. The two-stage training strategy consists of two parts, M1 and M2. M1 means to learn a new continuous latent representation z 1 first, and M2 means to train a semi-supervised model (7) with the embedding z 1 from M1 instead of the raw data X.

y)dy = p(x);x γ(x, y)dx = p(y)

Figure 2: The Exponential Function-Based Scheduler

Figure 3: The training curves of Cifar10.Left: classification performance. Right: E q φ (z|X),q φ (c|X) -log p θ (X|c, z)

Figure 5: The generative performance of OSPOT-VAE in MNIST and SVHN

Algorithm 2 OSPOT-VAE training process with epoch t. Batch of labeled pairs (X L , y L ) ∈ D L ,Batch of unlabeled examples X U ∈ D U ; ELBO hyperparameters: β z , β c , I z , I c = ELBOScheduler(t); Optimal transport estimation weights: w Mz , w Mc = WeightScheduler(t);

Table content: <table><tr><td>Input:</td></tr><tr><td>Model parameters: θ (t-1) , φ (t-1) ;</td></tr><tr><td>Model optimizer: SGD</td></tr><tr><td>Output:</td></tr></table>

The target loss L consists of KL divergence and the expected loglikelihood . The derivation of KL divergence part has a closed form, while calculating the gradient One-stage VAE error rate in MNIST and SVHN.

Table content: <table><tr><td>BackBone</td><td/><td>Method</td><td colspan="2">MNIST(100 labels) SVHN(1k labels)</td></tr><tr><td/><td/><td>Disentangled VAE (Narayanaswamy et al., 2017)</td><td>9.71(±0.91)</td><td>38.91(±1.06)</td></tr><tr><td colspan="2">Same with M1+M2</td><td>M1(Kingma et al., 2014)</td><td>11.97(±1.71)</td><td>54.33(±0.11)</td></tr><tr><td/><td/><td>M1+M2(Kingma et al., 2014)</td><td>3.33(±0.14)</td><td>36.02(±0.10)</td></tr><tr><td/><td/><td>One-stage VAE</td><td>3.14(±0.19)</td><td>27.38(±0.78)</td></tr><tr><td>BackBone</td><td colspan="2">Model category</td><td>Model</td><td>Cifar10(4k labels)</td></tr><tr><td/><td/><td colspan="2">Temporal Ensembling(TE) (Laine &amp; Aila, 2017)</td><td>16.37</td></tr><tr><td/><td colspan="2">Disagreement</td><td/></tr><tr><td>WRN-28-2</td><td/><td/><td/></tr></table>

Error rate in Cifar10. † denotes the best semi-supervised generative approach result. ‡ denotes the model ensemble two categories. * denotes the corresponding backbone is not exactly WideResNet

Table content: <table><tr><td>† *  (Li et al., 2019)</td><td>17.11</td></tr><tr><td>OSPOT-VAE</td><td>8.51(±0.32)</td></tr></table>

Error in Cifar100. † and * have the same meaning as described in Table2.

Table content: <table><tr><td>BackBone</td><td>Model</td><td colspan="2">Cifar100(4k labels) Cifar100(10k labels)</td></tr><tr><td/><td>Π -Model(Laine &amp; Aila, 2017)</td><td>\</td><td>39.19</td></tr><tr><td>WRN-28-2</td><td>GS-BadGAN  † *  (Li et al., 2019) LP  *  (Iscen et al., 2019)</td><td>45.11 43.73</td><td>37.16 35.92</td></tr><tr><td/><td>OSPOT-VAE</td><td>40.58(±0.48)</td><td>31.41(±0.21)</td></tr><tr><td>WRN-28-10</td><td>MixMatch  *  (Berthelot et al., 2019) OSPOT-VAE</td><td>\ 33.76(±0.53)</td><td>25.88 25.30(±0.31)</td></tr></table>

Ablation study with SVHN, Cifar10, and Cifar100.We compare the results of OSPOT-VAE with two categories of state-of-the-art models mentioned in Section2. In all experiments, we use the "WideResNet-28" model or other deep models with a comparable amount of parameters as the backbone. The results in Table2,3 demonstrate that our model outperforms most of the existing methods and surpasses state-of-the-art semi-supervised generative models(Dai et al., 2017) by a large margin. Notice that recently, data-augmentation based method, MixMatch,(Berthelot et al., 2019) achieves the absolute state-of-the-art results in all benchmarks. It uses pre-designed sophisticated data augmentation strategies for different datasets and outperforms our model. We list its results fairly as a comparison, while OSPOT-VAE surpasses it in Cifar100 dataset.

Table content: <table><tr><td>Methods</td><td colspan="3">SVHN(1k labels) Cifar10(4k labels) Cifar100(10k labels)</td></tr><tr><td>One-stage VAE</td><td>10.53(±0.17)</td><td>18.26(±0.51)</td><td>38.62(±0.67)</td></tr><tr><td>Optimal transport estimation (with encoder only)</td><td>6.54(±0.62)</td><td>10.71(±0.44)</td><td>36.21(±0.29)</td></tr><tr><td>OSPOT-VAE</td><td>5.79(±0.15)</td><td>8.51(±0.32)</td><td>31.41(±0.21)</td></tr><tr><td>4.2 OSPOT-VAE</td><td/><td/><td/></tr></table>

Generative performance measured by ELBO with ELBO ≤ log p(X)

Table content: <table><tr><td>Model</td><td>Cifar10</td><td>Cifar100</td></tr><tr><td>Pure VAE</td><td colspan="2">-226.25(±14.25) -1292.91(±1.10)</td></tr><tr><td colspan="3">OSPOT-VAE -237.62(±6.27) -1271.82(±24.15)</td></tr></table>

Schedule Parameters MNIST SVHN(one-stage) SVHN Cifar10 Cifar100 h max t max h max t max h max t max h max t max h max t max β

Table content: <table><tr><td>z</td><td>30</td><td>50</td><td>1</td><td>175</td><td>1e-3</td><td>150</td><td>1e-3</td><td>150</td><td>1e-1</td><td>150</td></tr><tr><td>β c</td><td>30</td><td>50</td><td>1</td><td>175</td><td>1</td><td>150</td><td>1e-3</td><td>150</td><td>1e-3</td><td>150</td></tr><tr><td>I z</td><td>17.5</td><td>50</td><td>50</td><td>175</td><td>1280</td><td>150</td><td>200</td><td>150</td><td>1280</td><td>150</td></tr><tr><td>I c</td><td>17</td><td>50</td><td>50</td><td>175</td><td>2.3</td><td>150</td><td>2.3</td><td>150</td><td>4.6</td><td>150</td></tr><tr><td>w Mz</td><td>\</td><td>\</td><td>\</td><td>\</td><td>1e-3</td><td>150</td><td>1e-3</td><td>150</td><td>1e-1</td><td>150</td></tr><tr><td>w Mc</td><td>\</td><td>\</td><td>\</td><td>\</td><td>1</td><td>400</td><td>1</td><td>280</td><td>1</td><td>280</td></tr><tr><td colspan="4">and the related KKT conditions are</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td>∂L( π, t) ∂ π</td><td>= t -</td><td colspan="2">λπ 1 + (1 -λ)π 2 π</td><td>= 0</td><td/><td/><td/></tr><tr><td/><td/><td/><td/><td>t  *  (</td><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
Contributions: The main contribution of this paper lies in the proposed OSPOT-VAE for semi-supervised classification. The proposed model is (i) an one-stage framework that unifies the generation and classification loss in one ELBO, (ii) achieves a tighter ELBO by applying optimal transport to the distribution of the latent variables, and (iii) achieves SOTA semi-supervised learning results without sacrificing generative performance on many benchmarks. Strengths: (1) Novelty: This paper contains novelty in terms of two aspects. First, the derivation of Eqn. (13) unifies both generation loss and classification loss naturally. The derivation seems different from a standard semi-supervised VAE. Second, the use of optimal transport (OT) estimation to tighten the ELBO results in improved classification performance. The use of OT seems novel in this context. (2) Writing: The paper is generally well written. However, I also found section 3 is hard to follow, with details in the Weaknesses section. (3) Experiments: The results look impressive. OSPOT-VAE achieves better performance than other generative approaches. Weaknesses: (1) Clarity: My biggest concern lies in the clarity of section 3, which I think is confusing in its current shape. After several rounds of reading, below is my understanding. In section 3.1, the authors first derive the objective in Eqn. (13). This objective is derived based on the use of mutual information (Eqn. (8)). By the end, no special cross-entropy loss term like in Eqn. (7) is needed. Second, in order to tighten the bound, section 3.2 introduces optimal transport estimation. Combining these two losses together (see in Algorithm 2) results in the final algorithm. However, generally, I feel the presentation of section 3 is confusing. One may need to present the big picture (framework) first, and then dig into details. Besides that, I have some questions as below. a) There are many hyper-parameters in the final objective, for example, the mutual information I_z, I_c is also considered as hyper-parameters. How all these hyper-parameters are selected? b) Is there any intuition on why Eqn. (13) is already enough without introducing the empirical cross-entropy loss like in Eqn. (7)? It seems to me adding this additional cross-entropy loss will not hurt performance. c) In Eqn. (12), how is the last term D_{KL}(p(c|X)||q_{phi}(c|X)) calculated, since we cannot get the true p(c|X)? d) As shown in Table 4, the use of optimal transport seems to be the key. However, section 3.2 is hard to follow. For example, In Algorithm 1, Line 6 & 7 considers the true posterior of z and c is a Gaussian and a multinomial distribution, respectively. Why doing this? Supported by theory? I also feel the calculation of L_{Rz} and L_{Rc} in Line 8 & 9 is hard to follow. e) How reliable is this optimal transport estimation is? And how it contributes to the final classification performance? Since this loss term is not a classification loss, then is there any intuition on why it serves as a regularization to help the final performance so much? f) Results in Table 5 on Cifar10 seem not imply OSPOT-VAE tightens the ELBO. Adding results on MNIST will be better, since the ELBO on MNIST is widely benchmarked. (2) Minor: a) I am not sure why section 2.1 is needed. Also, the feature matching GAN paragraph in section 2.2 seems also unnecessary. If you really want to discuss the use of GAN for semi-supervised learning, then there is a lot of other work that needs discussion. b) There is a missing right bracket in the second term of Eqn. (9). Overall, I think the results look quite impressive. However, the presentation of the method part is unclear to me, and could be much improved.