Learning Consistent Deep Generative Models from Sparse Data via Prediction Constraints
Authors: Gabriel Hope
Madina Abdrakhmanova
Xiaoyin Chen
Michael C Hughes
Erik B Sudderth
Venue:  (2021)
Keywords: Semisupervised learning
deep generative models
variational autoencoders

Abstract: We develop a new framework for learning variational autoencoders and other deep generative models that balances generative and discriminative goals. Our framework optimizes model parameters to maximize a variational lower bound on the likelihood of observed data, subject to a task-specific prediction constraint that prevents model misspecification from leading to inaccurate predictions. We further enforce a consistency constraint, derived naturally from the generative model, that requires predictions on reconstructed data to match those on the original data. We show that these two contributions -- prediction constraints and consistency constraints -- lead to promising image classification performance, especially in the semi-supervised scenario where category labels are sparse but unlabeled data is plentiful. Our approach enables advances in generative modeling to directly boost semi-supervised classification performance, an ability we demonstrate by augmenting deep generative models with latent variables capturing spatial transformations. 

Under review as a conference paper at ICLR 2021
LEARNING CONSISTENT DEEP GENERATIVE MODELS FROM SPARSE DATA VIA PREDICTION CONSTRAINTS
Anonymous authors Paper under double-blind review
ABSTRACT
We develop new framework for learning variational autoencoders and other deep generative models that balances generative and discriminative goals. Our frame- work optimizes model parameters to maximize variational lower bound on the likelihood of observed data, subject to a task-specific prediction constraint that prevents model misspecification from leading to inaccurate predictions. We further enforce a consistency constraint, derived naturally from the generative model, that requires predictions on reconstructed data match those on the original data. We show that these two contributions prediction constraints and consistency constraints lead promising image classification performance, especially in the semi-supervised scenario where category labels are sparse but unlabeled data is plentiful. Our approach enables advances in generative modeling to directly boost remi-supervised classification performance, an ability we demonstrate by augment- ing deep generative models with latent variables capturing spatial transformations.
INTRODUCTION
We develop broadly applicable methods for learning flexible models of high- sigh-dimensional data, like images, that are paired with (discrete or continuous) labels. We are particularly interested in semi- supervised learning (Zhu 2005 Oliver et al., 2018) from data that is sparsely labeled, a common situation in practice due to the cost or privacy concerns associated with data annotation. Given large and sparsely labeled dataset, we seek single probabilistic model that simultaneously makes good predictions of labels and provides high-quality generative model of the high high-dimenssioal input data. Strong generative models are valuable because they can allow incorporation of domain knowledge, address partially missing or corrupted data, and can be visualized to improve interpretabilltyy
Prior approaches for the superviidd learning of deep generative models include methods based on variational autoencoders (VAEs) (Kingma et al. [2014: Siddharth al. 2017). generative adversarial networks (GANs) (Dumoulin et all][201] Kumar et a (017). and hybrids of the two (Larsen et al., 2016 de Bem al.! 2018 Zhhng al. 2019). While these all allow sampling of data, a major shortcoming of these approaches is that they do not adequately use labels to inform the generative model. Furthermore, GAN-based approaches lack the ability to evaluate the learned probability density function, which can be important for tasks such as model selection and anomaly detection.
This paper develops framework for training prediction constrained variational autoencoders (PC-VAEs) that minimize poplication-motivattt loss functions in the prediction of labels, while simultaneously learning high-quality generative models of the raw data. Our approach is inspired by the prediction-constraine
framework recently proposed for learning supervised topic models of "bag of words" count data (Hughes al. 2018), but differs in four major ways. First, we develop scalable algorithms for learning much larger and richer family of deep generative models. Second, we capture uncertainty in latent variables rather than simply using point estimates. Third, we allow more flexible specification of loss functions. Finally, show that the generative model structure leads to a natural consistency constraint vital for semi-sus eemi-supervised learning from very sparse labels.
Our experiments demonstrate that consistent rrediction-constraine (CPC) VAE training leads to prediction performance competitive with oftte--f-thh-att discriminative methods on lable-labeled datasets, and excels over these baselines when given semi-supervised datasets where labels are rare.

---

Under review as a conference paper at ICLR 2021
VAE-then-MLP
PA-VAE
CPC-VAE
M2
M2 (14)
CPC-VAE (14)
77.9%
78.1%
98.4%
98.19 1%
80.6%
98.5%
83.8%
98. 2%
98.4%
98.
96.4%
98.1 %6
Figure 1: Predictions from SSL VAE methods on half- moon binary classification task, with accuracy in lower corner. Each dot indicates a 2-dim. feature vector, colored by predicted binary label. Top: 6 labeled examples (diamond markers), 994 unlabeled. Bottom: 100 labeled, 900 unlabeled. First 4 columns use C' 2 encoding dimensions, last use C = 14. M2 (Kingma et al.) (014) classification accuracy deterioriates when increasing model capacity from 2 to 14, especially with only labels (drop from 98. 1 80.6% accuracy). In contrast, our CPC VAF approach reliable at any model capacity, as it better aligns generative and discriminative goals.
2 BACKGROUND: DEEP GENERATIVE MODELS AND SEMI-SUPERVISION
We now describe VAEs deep generative models and review previous methods for semi-supervised learning (SSL) of VAEs. highlighting weaknesses that we later improve upon. We assume all SSL tasks provide two training datasets: an unsupervised (or unlabeled) dataset D of N feature vectors x, and a supervised (or labeled) dataset D containing M pairs (z, y) of features x and label y e y Labels are often sparse (N > M) and can be discrete or continuous.
2.1 UNSUPERVISED GENERATIVE MODELING WITH THE VAE
The variational autoencoder Kingmaa & Welling 2014) is an unsupervised model with two compo- nents: a generative model and an inference model. The generative model defines for each example a joint distribution (z, 2) over "features" (observed vector x E R) and "encodings" (hidden vector Rº). The "inference model" of the VAE defines approximate posterior (a(z x), which trained to close to the true posterior (gø(z x) R po(z x)) but much easier to evaluate, As
in Kingma& & Welling 2014), we assume the following conditional independence structure:
(I) The likelihood F is often multivariate normal, but other distributions may give robustness to outliers. The (deterministic) functions μ4 and do. with trainable parameters 0, define the mean and covariance of the likelihood. Given any observation 2, the posterior of is approximated normal with mean μe and (diagonal) covariance as parameterized by ø. These functions can be represented as multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), or other (deep) neural networks.
po 2) = (( 0,1c) F(x (uo(z),oo=)))
(=(= x) = (( (o(x)),a(<)))
We would ideally learn generative parameters 0 by maximizing the marginal likelihood of features x, integrating latent variable 2. Since this is intractable, we instead maximize variational lower bound:
max 8.
EPED L VAE (x: 0, o),
L VAE (x; 0, Φ) = (x(x)) [log
Po (z,2) (=(=)))
S log (o.)..
(2)
This expectation can be evaluated via Monte Carlo samples from the inference model (=(x)) Gradi- ents with respect to 0, & can be similarly estimated by the reparameterization "trick" of representing (s(= 2)) linear transformation of standard normal variables (Kingma & Welling 2014).
Throughout this paper, we denote variational parameters by Φ. Because the factorization of changes for more complex models, we will write ø= to denote the parameters specific to factor q(z(z))
2.2 TWO-STAGE SSL: MAXIMIZE FEATURE LIKELIHOOD THEN TRAIN PREDICTOR
One way to employ the VAE for a semi-supervised task is a two-stage "VAE-then theE-thhn-MLP. First, train VAE to maximize the unsupervised likelihood ( of all observed features æ (both labeled D and unlabeled D Second, we define a aabel-from-code predictor (2) that maps each learned code representation
predicted label y ). We use MLP with weights w. though any predictor could do. Let (g ÿ) be a loss function, such as cross-entropy, appropriate for the prediction task. We train the predictor minimize the loss: min,, Ex.e (x (Es Gw (z))]. Importantly, this second stage only the small labeled dataset and relies fixed parameters from stage one.
2

---

Under review as a conference paper at ICLR 2021
While "VAE- Then-MMLP"iss is simple common baseline (Kingma et al. 2014), it has a key disadvantage: Labels are only used in the second stage, and thus a misspeciifidd generative model in stage one will likely produce inferior predictions. Fig. Tillustrates this weakness.
2.3 SEMI-SUPERVISED VAEs: MAXIMIZE JOINT LIKELIHOOD OF LABELS AND FEATURES
To overcome the weakness of the two-stage approach, previous work by Kingma et al. (2014) presented a VAE-inspired model called "M2" focused on the joint generative modeling of labels y and data z. M2 has two components: a generative model pe (I, 2) and inference model qs (v, z). Their generative model is factorized to sample labels (with frequencics т) first, and then features x:
(1, y, 2) A(= 0, Ic) Cat(y T) F(z [o 2), ao z)).
(3)
The M2 inference model sets (y, = x) Uøv (v 1x x))gsi>>x
2). where e = (ebble df
To train M2, Kingma et al. 2014) maximize the likelihood of all observations (labels and features): EXNEDS L (xx, (x.,;;8 y; 0- + EzEDU Ld 0, d0 s=lm.)).
max 0,00l soll
(4)
The first, "supervised" term in Eq. (4) is a variational bound for the feature-and-labed joint likelihood:
( 0, d=l..)) ( log po(x..p=)) 9gelte (3/2.3]
S (o,,
(5)
The second, "unsupervised" term is a variational lower bound for the features-only likelihood log px() 2 L" where L Boolu.sl)) loog [a(a))))) can be simply expressed in terms of L
L (x;;; 0, 0,0 (=-1..0)- = EvEy Tgvi= (r) (C. (x,, 0, dell..) log Javi= (y x))
(6)
As with the unsupervised VAE, both terms in the objective can be computed via Monte Carlo sampling from the variational posterior, and gradients can estimated via the reparameterization trick.
M2's prediction dilemma and heuristic fix. After training parameters 0, d. we need to predict labels y given test data æ. M2's structure assumes we make predictions via the inference model's discriminator density gaxi= (y w²). However, the discriminator's parameter dool is only informed by the unlabeled data when using the objective above (it is not used to compute LS). We cannot expect accurate predictions from parameter that does not touch any labeled examples in the training set.
To partially overcome this issue, Kingma et al. 2014) and later work use a weighted objective:
Σ Z.EDS
(a
Jasi= lagigg (v) x)) XL (c. y, 0, (flww) + E DEDU
c (x;; ci d=12..)
max 9,6
This objective biases the inference model's discriminator to do well on the labeled set via an extra loss term (weighted by hyperparameter a >> 0). We can further include À >> 0 to balance the supervised and unsupervised terms. Originally, Kingma et al. (2014) fix = and tune a to achieve good performance. Later, Siddharth et al. (2017) tuned X to improve performance. Maaløe et al. (2016) used this same a log g 2) term for labeled data to train VAEs with auxiliary variables.
Disadvantage: What Justification? While the E and L terms in Eq. have rigorous jus- tification as maximizing the data likelihood under the assumed generative model, the first term (a log 4(y ar)) is not justified by the generative or inference model. In particular, suppose the training data were fully labeled: we would ignore the C terms altogether, and the remaining terms would decouple the parameters 6, ş from the discriminator parameters dala This is deeply unsatisfying: We want single model guided by both generative and discriminative goals, not two separate models. Even in partially-labeled scenarios, including this a term does not adequately balance generative and discriminative goals, as we demonstrate in later examples. An overly flexible yet misspecified generative model may go astray and compromise predictions.
Disadvantage: Runtime Cost. Another disadvantage is that the computation of L in Eq. 6 is expensive. If labels are discrete, computing this sum exactly is possible but requires a sum over all L= | possible class labels, computing a Monte Carlo estimate of LS for each one. While Monte Carlo approximations can avoid the explicit sum in Eq. (6). they may make gradients too noisy.
Extensions. Siddharth et al. (017) showed how LS and " could be extended to any desired conditional independence structure for q 2 z). generalizing the habel-then-code factorization gsly (= xc, y) of Kingma al. (2014). While importance sampling leads to likelihood bounds,
3

---

Under review as a conference paper at ICLR 2021
VAE-then-MLP
Supervised VAE
PC-VAE
CPC-VAE
M2
54.9%
66.2
74.1%
81.
69.1%
Figure 2: Supi-supe learning of 2-dim. encodings MNIST digits, with accuracy in lower corner. All methods use 100 labeled examples and 49,900 unlabeled examples. Each observed image r mapped to its most likely 2-dim. encoding = and colored by true label Labeled examples are emphasized. Where applicable. we also show class decision boundaries. Baselines (from left): 2-stage unsupervised VAE-then-MLP (Sec. 2.2 and a "supervised" VAE maximizing joint likelihood log p(x, y) (a special case of PC method with À
Sec. Sec.B.1 3.I methods: Prediction constrained VAE VAE with X 25, Sec. 3.1 and consistent prediction constrained VAE (CPC- VAE, Sec. (.2). Competitors: M2 from Kingma et al (2014). which intentionally
decouples lahel from "style" 2, has limited accuracy due to imbalance of discriminative and generative goals.
the overall objective still has two undesirable traits. First, it is expensive, requiring either marginal- ization y to compute L in Eq. ( or marginalization of z to compute ((y[>)) = 9s [o(y, (y. 2 d2z) Second, the approach requires the heuristic inclusion of the discriminator loss a log q(y 2). While recent parallel work by Gordon & Hernández-Lobale (2020) also tries to improve SSL for VAEs, their approach couples discriminative and generative terms only distantly through joint prior over parameters and still requires expensive sums over labels when computing generative likelihoods.
3 PREDICTION-CONSTRAIII LEARNING WITH CONSISTENCY
We now highlight two experiments that demonstrate disadvantages of prior SSL methods, and contrast them with our new approaches. In Fig. [ we show the predictive accuracy of several SSL methods on the widely-used "half-moon" task, where the goal is to to predict a binary label y given 2-dimensional features æ. We focus on the top row, which shows results given only labeled examples (3 of each class) but hundreds of unlabeled examples. Notably, while M2 has 98. % accuracy with small encoding space (C = 2). if the generative model is too flexible (C = 14) it learns overly complex structure that does not help label- habel-from-fcatur predictions, dropping accuracy to only 80.6%. In contrast, our consistent prediction constrained (CPC) VAE gets over 98% accuracy with either
or C 14. We have verified it maintains 98% even C 50, while M2 shows further instability.
Second, in Fig. 2 we show SSL methods for classifying images of MNIST digits (LeCun et al. 2010). given only 10 labeled examples per digit. We seek models with highly accurate fabel-from-featteee predictions, as well as interpretable relationships between the encoding = and these predictions. When forced to
2-dimensional latent space, M2 has worse accuracy and (by design) apparent relationship between encoding and label y. In contrast, our CPC approach offers noticeable advantages over all baselines in both accuracy and interpretability of the encoding space.
3.1 PREDICTION CONSTRAINED TRAINING FOR VAES
We develop framework for jointly learning strong generative model of features x, and making senel-ivve-featurr predictions j(x) of uncompromised quality, by requiring predictions to meet user-specified quality threshold. Our prediction constrained training objective enables end-to- end estimation of all parameters while incorporating the same task- prediction rules and loss functions that will be used in heldout evaluation ("test"") scenarios. Our goals are similar to previous work on end-to-end approximate inference for task-specific losses with simpler probabilistic models LLacstt--uulics et al. 2011 Stoyanov et al.) 201I), but our approach yields simpler algorithms.
Generative model. Our generative model does not include labels y. only features and encodings Their joint distribution pe (z) 2)) factorizes as the unsupervised VAE of Eq. (I). and we also use the inference model qs(2 .:) defined in Eq. [ While M2 included the labels y in its generative predictions, but we are not interested in label marginals or other distributions over y that do not condition æ.
model (Kingma et al. (2014), our goals are different: we wish to make bobel-given--eare
prediction. To predict labels y from features x. we use a predictor similar to the two-stage method of Sec. 2.2. We first sample encoding (=(r)) from the learned inference
4

---

Under review as a conference paper at ICLR 2021
model, and then transform this encoding to a label via the predictor function Bw (2) with parameter w. By sharing random variable 2, the generative model is involved in wel---fro-ffttttte predictions.
Constrained PC objective. Unlike the two-stage model, our approach does not do post-hoc predic- tion with a previously learned generative model. Instead, we train the predictor simultancously with the generative model via new, prediction-constrainnel (PC) objective:
max
E UDS
(=; (x; 01 subj. to:
1 M
Σ E.
(eotslal S. (8)
0.
P(2.3100115.00
The constraint requires that any feasible solution achieve average prediction loss less than e on the labeled training set. Both the loss function and scalar threshold needs (c.g., classification must have a certain false positive rate or overall accuracy). The loss function may be any differentiable function, and need not equal the oogliielihood of discrete labels assumed by previous work specialized to supervision of topie models (Hughes et al. 2018).
can be set to reflect task-specific
Unconstrained PC objective. Using the KKT conditions, we define an equivalent unconstrained objective that maximizes the unsupervised likelihood but penalizes inaccurate label predictions:
ErEDuUDS L (z; (1 - >EE, EDS P(x; Vū 6fle,, w.).
D88 0,671o,0 au
Here À is positive Lagrange multiplier chosen to ensure that the target prediction constraint is achieved; smaller loss tolerances e require larger penalty multipliers A. This PC objective, and gradients for parameters 0, o, w, can be estimated via Monte Carlo samples from 90(2 zz).
Justification. While the PC objective of Eq. ( may look superficially similar to Eq. 7 we emphasize two key differences. First, objective couples generative likelihood and prediction loss via the shared variational parameters œf* This makes both generative and discriminative performance depend on the same learned encoding z. (Later we show how to partition & so some entries are discriminative, while others affect generative "style" only.) In contrast, the M2 objective uses a sea--ivvee-featrre conditional to make predictions that does not share any of its parameters øbi= with the supervised likelihood CS. Second, our objective is more affordable: no term requires an expensive marginalization over labels. This is key to scaling to big unlabeled datasets, and also enables tractable learning from datasets whose labels are continuous or
Hyperparameters. The major hyperparameter influencing PC training is the constraint multiplier 2 O. Setting À = 0 leads to unsupervised maximum likelihood training (or MAP training, given priors on 0) of a classic VAE. Setting À= = 1 and choosing probabilistic loss log p(y 2) produces a "supervised VAE" that maximizes the joint likelihood po (z. y). But as illustrated in Fig. 2 because features æ have much higher dimension than labels y, the resulting model may have weak predictive performance. Satisfying the strong prediction constraint of Eq. (8) typically requires À > 1, and in practice we use validation data select the best of several candidate À values. If a task motivates concrete tolerance e, we can test an increasing sequence of X values until the constraint is satisfied.
We emphasize that although Eq. (9) is easier to optimize, we prefer to think of the constrained problem in Eq. (8) as the "primary" objective, because our applied goals are to satisfy discriminative quality first; a generative model that predicts poorly is not plausible. Furthermore, the constrained objective is far more natural for semi-supervised learning. The choice of need not be concerned by the relative sizes of the labeled and unlabeled datasets. In contrast, either [D"| or |DS] changes, the value of À may need to change dramatically to reach the same prediction quality.
3.2 ENFORCING CONSISTENT PREDICTIONS FROM GENERATIVE MODEL RECONSTRUCTIONS
While the PC objective effective given sufficient labeled data, may generalize poorly when labels very sparse (see Fig. D.. This fundamental problem arises because in the PC objective of Eq. 8 the parameters w of the predictor gn (z) are only directly informed by the labeled training data.
Revisiting the generative model, let po(- z²) and ã ~ po |2' be two observations sampled from the same latent code z'. Even if the true label y of x is uncertain, we know that for this model to be useful for predictive tasks, ø must have the same label x. We formalize this relationship via consistency constraint requiring label predictions for common-code data pairs (z, σ) to approximately match (see Fig. 3. As we show, this regularization may dramatically boost performance.
5

---

Under review as a conference paper at ICLR 2021
2
L
w
π
Labeled Data
N Unlabeled Data
Inference Net. (,(2lx)
Decision
Rendom
Variable
ELBO
Constraint
Prediction
Consistency Constraint
Constraint
Aggregate
VAE as a decision network (Cowell et al. 2006). Circular nodes are random variables, including the latent VAE code z that generates observed features æ. Shaded nodes are observed, including the class labels y for some data (left). Square decision nodes indicate predictions tu of class labels that depend the amortized inference network ( x). Diamonds indicate losses (negative utilities) that influence the variational prediction of labels and latent variables. Generative likelihood: Like standard VAEs, our generalizations choose generative model parameters 9 and variational posteriors qe to maximize the ELBO C (orange). Prediction accuracy: Unlike previous semi-supervised VAEs, we do not directly model the probabilistic dependence of labels y on and/or We instead treat label prediction as decision problem, with upplication-motivatee (2|xx) (and therefore the encodings and generative model). Prediction consistency: For unlabeled data (right), we cannot directly evaluate the quality of predictions. However, we do know that if two observations I and are generated from the same latent code they should have identical labels; otherwise, the model cannot have high accuracy. The loss fc (blue) enforces the consistency of such predictions, Aggregate consistency: By the law of large numbers, frequencies observed labeled data. The loss La (green) enforces this constraint, and penalizes degenerate predictors that satisfy &e by predicting the same label f.. for most or all of the unlabeled data.
Figure Formalization our consistency-constraie
loss Es (red), that constrains the approximate VAE posterior
also know that aggregate label frequencies for unlabeled data should be close to the
Given features x, our method predicts labels by sampling posterior, and then applying our predictor Uar (z). Alternatively, given i we can first simulate alternative features E with matching code by sampling from the inference and generative models, and then predict the label associated with ã. We constrain the label predictions for æ, and ğ for I. to be similar via consistency penalty function (o 7). For the classification tasks considered below, we use a cross- values of the following consistency costs unlabeled and labeled examples, respectively:
gø(2 x) from the approximate
consistency penalty &c. Given this penalty, we constrain the maximum
C² (z: 0, ó. w) A Ex,(= [p]())=) [Bq((H))) [fw (2), (2)II),
(10)
C (z. y: o, w) (=]x)) (p)()) [EIa) [f o (3))i]]
(11)
Consistent PC: Unconstrained objective. To train parameters, we apply our consistency costs to unlabeled and labeled feature vectors, respectively. The overall objective becomes:
Σ
LAA (x: 0,
Σ DED"
JC (x; (x;
+ Σ -XP(x,
cc (z. 0, w),
mAx
8,0,0
REDUUDS
a.EEDS
where L is the unsupervised likelihood, P is the predictor loss, and C are the consistency constraints.
Here, 7 > 0 is scalar Lagrange multiplier for the consistency terms, with similar interpretation as X.
Aggregate Label Consistency. For SSL applications, find is also useful regularize our model with an aggregate label consistency constraint, which forces the distribution of label predictions for unlabeled data be aligned with a known target distribution æ. This discourages predictions on ambiguous unlabeled examples from collapsing to a single value. We define the aggregate consistency loss as: (TT, DV [hn (=))). and again use a cross-entropy penalty. If the target distribution of labels is unknown, we set it to the empirical distribution of the labeled data.
Related work on consistency. Recently popular SSL image classifiers focused on discriminative goals will train the weights of CNN to minimize a modified objective that penalizes both label accuracy and a notions of consistency or smoothness on unlabeled data. Examples include consistency under adversarial perturbations (Miyato et al.) 2019), label-in iibbriiivaiaat transformations (Laine & Aila 2017), and when interpolating between training features (Berthelot et al.) 2019). This regularization can deliver competitive discriminative performance, but does not meet goal of generative modeling.

---

Under review as a conference paper at ICLR 2021
Recently, Unsupervised Data Augmentation (UDA, Xie al. (2200) achieved state-of-the-ar vision and text SSL classification by enforcing label consistency on augmented samples of unlabeled features. UDA relies on the availability of vell-engineered augmentation routines for specific domains (e.g. image processing library transforms for vision or transsraanl for text). In contrast, we learn generative model that produces feature vectors for which predictions need to be consistent. Our approach is more applicable to new domains where advanced augmentation routines are not available.
In broader machine learning, "cycle-consistency" has improved generative adversarial methods for images (Zhu et al. 2017; Zhou et al., (016) or biomedical data (McDermott et al. 2018). Others have developed sycle-consisten objectives for VAEs (Jha al. 2018) which focus on consistency in code vectors 2. In contrast, our work focuses on semi--sueervsed learning and enforces cycle consistency in labels y. Recently, MMlllrr et al. (2019) developed discriminative regularization for VAFs. Their objective is not designed for SSL and uses a direct feature-to-label prediction model that must be consistent with reconstructed predictions. Our approach uses code-to-label prediction and SSL.
3.3 IMPROVED GENERATIVE MODELS: ROBUST LIKELIHOODS AND SPATIAL TRANSFORMERS
As our approach is applicable to any generative model, we can incorporate prior knowledge of the data domain to improve both generative and discriminative performance. We consider two examples: likelihoods that model noisy pixels, and explicit affine transformations to model image deformations.
Robust Likelihoods. Instead of normal (or other common) likelihood we use a "Noise-Normal" likelihood to model images more robustly. We assume that pixel intensities have values in the interval [-1, 1] and rescale our datasets to match. Our Noise-Normal likelihood is defined as a 2-component mixture of truncated Normal and uniform "noise" distribution, with pixel-specific mixture weights. Define the standard normal PDF as Φ(.) and standard normal CDF as (()). We write the probability density function of the Noise-Normal distribution with parameters (p. J4, σ)
+ - p) ())
F(x P. 0) =
(12)
Following Eq. ( our (unsupervised) VAE now uses the revised generative and inference models: (0,, 2) N( 0, Ic) -F( po(=), oz())) 90(2 2) = N/2 (z) az(z)) (13) This approach allows model to avoid sensitivity to outliers and noise in the observed images, and boosts the performance of our CPC method for SSL.
Spatial Transformer VAE. Our spatial transformer VAE retains the structure of a standard VAE, but reinterprets the latent code z as two components. We denote the first latent dimensions as 2t, and associate these with 6 affine transformation parameters capturing image translation, rotation, scaling, and shear. The generative model maps each value into a fixed range and creates an affine transformation matrix, M,, by applying the transformations in fixed order. The remainder of the latent code, 2,, is used to generate parameters for independent, per-pixel likelihoods. Assuming normal likelihoods, the output parameters for the pixel with coordinates (i.j) are po(z))ij. 08(2))
We re-orient our per- per-pixel likelihoods according to the affine transform M,. The parameters of the likelihood for pixel (i, will use the decoder outputs at coordinate (i', where M, defines the affine mapping from (2',5") )') to (i,j). We apply this transformation in (sub) differentiable way via a spatial transformer layer (Jaderberg et al. 2015) that takes as input M, and the appropriate parameter maps ua (z.). a=(2)). and outputs a final set of parameters for the individual pixel likelihoods. As (f',)") may not correspond to integer coordinates, we use bilinear interpolation over an appropriate representation of the likelihood parameters, and appropriately pad the size of the decoder output.
We further account for this special structure in our prediction and consistency constraints. For many applications, we have prior knowledge that small affine transforms should not affect the class of an image, and thus we can define consistency constraints that condition on 2, but not 2t.
4 EXPERIMENTS
We assess our consistent prediction-constramme (CPC) VAE on two key goals: accurate prediction of labels y given features x (especially when labels are rare) and useful generative modeling of a.. We compare to ablations of our own method (without consistency, without spatial transformations) and to external baselines. We report each method's mean and standard deviation in classification accuracy
7

---

Under review as a conference paper at ICLR 2021
Source
Method
MNIST (100)
SVHN (1000)
NORB (1000)
Tables 1-2 of Kingma al. 2014)
MI M2
96.67% (10014)
63.98% (10.10)
Table of Maaloc al. 2016
ADGM
99.04% (+0.02)
77.14%
89.94% (=0.05)
Table 2 of Maaloc ct al.
2016
SDGM
98.68% (=0.07)
83.39% (=0.24)
90.60% (=0.04)
Gordon & Hernandez L Lobato
2020
Blended M2
93.05%
(+0.73)
Tables 3-4 of Miyato et al. (2019)
VAT
98.64%
(+0.08)
94.23% (+0.32)
ours, using labeled-set only ours
WRN
73.91% (+1.15)
(+02)) 94.22% (+0.62)
87.7%
86.7 7% (+133 92.00% (+1.21)
CPC VAE
98.29% (+0.50)
Table I: Test accuracy of SSL methods. Our results show mean (std. dev.) across 10 samples of the labeled set. The labeled-set-only discriminative neural net (WRN) has roughly the same size as our CPC VAE.
Method
MNIST (100)
Method
MNIST (100)
CPC (2 layer)
96.68% (10.54)
M1 + M2 Kingma al. 2014
96.67% (100.14)
CPC (2 layer, w/o aggregate loss)
94.27% (1.3.78)
M2 (1 layer, = 0.1)
Kingmaa
2014
88.03% (11.71)
CPC (2 layer, w/o transforms)
91.93% (=1)
M2 (2 layer,
0.1)
83.32% (=5)2
CPC (4 layer, w/o transforms)
93.78%
(#:225)
M2 (4 layer, a 0.1)
47.05% (+8.13)
PC (2 layer)
80.49% (+3331)
M2 (4 layer, tuned to a = 10)
68.15% (#3.43)
VAE MLP
72.90% (4:1.8)
M2 (1 layer, a = 0.1, Noise-Normal)
73.93% (48.12)
Table 2: Ablation study for SSL on MNIST using dense MLP matching M2's network architecture. Trials used 100 labels and encoding size C 50. Unless cited, all results come from our implementation, where encoder and decoder have 1000 units per hidden layer. All our introduced techniques improve accuracy. M2's accuracy deteriorates when its networks are overparameterized. even after tuning a on validation set instead of using default from Kingmma et al. (2014), while our method remains stable.
across 10 labeled subsets. We trained using ADAM (Kingma & Ba 2014), with each minibatch containing 50% labeled and 50% unlabeled data. Hyperparameter search used Optuna (Akiba et al., 2019) to maximize accuracy on validation data. Supervised baselines used either MLP or wide residual nets (WRN, Zagoruyko & Komodakis (2016)). Reproducible details are in appendices.
SSL classification on MNIST with thorough internal comparisons. In Table 2 we compare several variations of our CPC methods and the M2 model on an SSL version of MNIST (LeCun et al. 2010
10 classes, S 100,[D = 49900, 10000 validation, 10000 test).
SSL classification on SVHN and NORB. Table I compares our methods on two standard SSL tasks:
Street-View Housing Numbers (SVHN) (Netzer et al. 2011 10 classes, |DS| = 1000, |D4| = 62257.
10000 validation, 26032 test) and the NYU Object Recognition Benchmark (NORB) (LeCun et al.
2004 5 classes, [D = 1000, |D = 21300, 2000 validation, 24300 test).
SSL classification on CelebA. We ran additional experiments on a variant of the CelebA dataset (Luu et al.) 2013). For these trials we created a classification problem with 4 classes based on the combination of gender (woman/man) and facial expression (neutral/smiling). (4 classes, |D$ 1000,
|D = 159770, 2000 validation, 19962 test). We report our results in figure
Across all evaluations, we can conclude:
Both consistency and prediction constraints are needed for high accuracy. In Table PC alone gets 80% accuracy on 100-label MNIST, while adding consistency yields 97% for CPC. The benefits of CPC over PC in both accuracy and latent interpretability are visible in Figs.
Our aggregate label consistency improves robustness, reducing the variance in CPC accuracy (from 3.78² to 0.54²).
CPC training delivers strong improvements in SSL prediction quality over baselines. In Table [] our CPC achieves 94.22% and 92.0% on the challenging 1000-label SVHN and NORB benchmarks, which surpasses by >1.4% all reported baselines while being reliable across runs. The M1+M2 baseline (Kingma et al. 2014) is not a coherent generative model, but rather a discriminative ensemble of multiple models. It performs well on MNIST, but very poorly the more challenging SVHN.
CPC delivers better generative performance; it is not all about prediction. We improve on unsupervised VAEs by explicitly learning latent representations informed by class labels (Fig. 22) In Fig. 4 Fig. 5] and Fig. we show essuall-plausible cans-conniti samples from our best CPC models. Additional visuals from learned VAE and CPC-VAE models are in the supplement.
8

---

Under review as a conference paper at ICLR 2021
1 1
22
3,00 9
0
3
444
555
5
6 6
6
77
7
8
6
8
9 9 9
3
6
7
7
8
8
8
$
3
2
& 6
4
701
8
Figure 4: Sampled reconstructions used to compute the consistency loss during training. Top: Original image.
Middle:
Sampled reconstructions using a "Noise-Normal" likelihood. Bottom: Sampled reconstructions with spatial affine transformations sampled from the prior.
20
2
35
SS
60
30
202
2:
E
5
55
26
73 73 75
6 18
199
2:212
15
141447
88
9 999 9
20
20
12
25252
32 32
1
16
727777
1 10 18
8
801307 907
3:13113
55
15 15
46
16
37
17 37
201
281 28
0
07
0000900
4
60
22
80 80
8 19
50
10
10
212 2
78
78
7
18118
(a) Prior samples
(b) Reconstructions
Figure 5: Visualizations generative model performance on SVHN for a prediction and consistency constrained VAF incorporating latent affine transformations, trained on the SVHN dataset with all labels. (5a) Samples from
the learned generative model conditioned on class (more are shown in supplement B Samples are chosen via rejection sampling in the latent space with threshold of 95% confidence in the target class. (56) Reconstructions images the held-out test set. Each triplet shows the original image (left), the reconstruction (middle), and an aligned reconstruction (right) obtained by fixing the learned affine transform variables to the global mean.
Woman, neutral
CPC VAE
80.73%
Model
Mon, neutral
Woman, smiling
WAE MIP
28.06%
20
Man, smiling
Accurscy
Figure 6: CelebA dataset results, Left: Test accuracy on 4-class CelebA SSL task (1000 labeled training images, 159,770 unlabeled: error bars show std. dev. of 10 labeled set samples). CPC- VAE improves on the labeled-set only WRN, and especially beats the unsupervised VAE which poorly separates the classes in the latent space. Right: Class-conditional samples of the 4 possible classes from our semi-supervised CPC VAE.
With improved generative models, CPC can improve predictions. Fig. 5 shows that including spatial transformations allows learning a canonical orientation and scale for each digit. This generative improvement boosts classifier accuracy (e.g., MNIST improves from 91.9% to 96.7% in Table
CONCLUSION
We have developed new optimization framework for semi-supervised VAEs that can balance discriminative and generative goals. Across image classification tasks, our CPC-VAE method delivers superior accuracy and label-informed generative models with visually-plausibl samples. Unlike previous efforts to enforce constraints on latent variable models, such as expectation constraints (Mann & McCallum 2010). posterior regularization (Zhu et al. 2014 2014|2012 2012), posterior constraints (Ganchey et al.) 2010), prediction constraints for topic models (Hughes al.) 2018), our approach is the only one that coherently and simultaneously treats uncertainty latent variables 2, applies to flexible "deep"" non-conjugate models, and offers scalable training and test evaluation via amortized inference.
further contribution is demonstrating the necessity of consistency for improving discrimination. Our CPC approach is an antidote to model misspecification: the constraints prediction quality and consistency prevent the model from learning a generative model that is unaligned with the classification task or that overfits with more flexible generative models (as M2 is vulnerable to do). As we show with spatial transformers, our work lets improvements in generative model quality directly improve sepi-super label prediction, helping realize the promise of deep generative models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. MixMatch: A Holistic Approach to Semi-Supervised Learning. In Advances in Neural Information Processing Systems, 2019. URL
org abs//9/s...990 02249
Robert G Cowell, Philip Dawid, Steffen L Lauritzen, and David Spiegelhalter. Probabilistic
networks and expert systems: Exact computational methods for Bayesian networks. Springer Science & Business Media, 2006.
Rodrigo de Bem, Arnab Ghosh, Thalaiyasingam Ajanthan, Ondrej Miksik, N. Siddharth, and Philip Torr. A Semi-supervised Deep Generative Model for Human Body Analysis. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018.
URL http: hptpp//cpppc..c.pp/ppppppp/. thecve www..eeeweewceeeeeeieeeeeeee
thecvf..... wtt.:/ooc///cneee _ccv 2018 workshops/we /w11 html / de A Semi supervised su Deep Generative Modelfor Human Body
Ana Analysis__ccvW__2 Bww.EECcCW____11___00 2eeec111_p_ppe___1 www.ppppppp...
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially Learned Inference. In International Conference on Learning Representations (ICLR), 2017.
Michael Figurnov, Shakir Mohamed, and Andriy Mnih, Implicit reparameterization gradients. In
Advances in Neural Information Processing Systems, 2018.
Kuzman Ganchev, João Graça, Jennifer Gillenwater, and Ben Taskar. Posterior Regularization for Structured Latent Variable Models. Journal of Machine Learning Research, 11:2001-2049, 2010.
Jonathan Gordon and José Miguel Hernández-Lobato. Combining deep generative and discriminative models for Bayesian semi-supervised learning. Pattern Recognition, 100, 2020.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS'04, pp. 529-5366 Cambridge, MA, USA, 2004. MIT Press.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. bae:-vae: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations (ICLR), 2017.
Michael C. Hughes, Gabriel Hope, Leah Weiner, Thomas H. McCoy, Roy Perlis, Erik Sudderth, and Finale Velei-Velez. Semi-Supervised Prediction-Constraiin Topic Models. In Artificial Iniel- ligence and Statistics, 2018.
mlr. http://ss.....eeee//////tteeeeee
html
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In Advances in Neural Information Processing Systems, 2015. URL htttp://papers
pdf
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and V. S. R. Veeravasarapu. Disentangling
Factors of Variation with Cycle-Consistent Variational Auto-encoders. In European Conference Computer Vision (ECCV). Springer International Publishing, 2018.
Diederik Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. arXiv:1412.6980 [cs], 2014. URL http:://awx....r////eaaaea 111. 6980
Diederik P. Kingma and Max Welling, Auto Auto-EEcoding Variational Bayes In International Conference Learning Representations, 2014. URL ht..tutttt/ttt//aaaaaaa htt.:/tcpp.//axi...oo//aaaa////
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In Advances in Neural Information Processing Systems, 2014. URL https: wwwppaapp.ppppp... oc wwtppppppp//2..... sem---emiiiie- -ep--eeer-s-seeeddd
learni na- wi th- deep generat -ve- mode-models pddf
Abhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Semi-supervised Learning with
GANs: Manifold Invariance with Improved Inference. In Advances in Neural Information
Processing Systems, 2017. URL
nips httppssccpppppe/1//aeeep11 ii
supervised leareeernnni ng wig ta- inference.pdf
Simon Lacoste-Julien, Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the
loss-calibrated bayesian. In Artificial Intelligence and Statistics, 2011.
Samuli Laine and Timo Aila. Temporal Ensembling for Semi Supervise Learning. In International Conference on Learning Representations, 2017. URL http::ttppeeeeeiiieeeee net/pdf?
[dd-BJooOfqge)
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Au- toencoding beyond pixels using a learned similarity metric. In International Conference on Machine Learning, pp. 1558-1566, 2016. URL BRR/attpp//prreeedngg/ wtt..mess......//////.
harsen16.html
LeCun, F. J. Huang, and L. Bottou. Learning Methods for Generic Object Recognition with Invariance to Pose and Lighting. In JEEE Computer Vision and Pattern Recognition (CVPR), 2004.
URL http::ttpa/nn..atttmmm lecun. com/ wtt.e//cw.....e///.peeeeeea/aa/e hwtp://pbbl///////eeeeeeeaa pd.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database, 2010. URL
http://yann www.neeecncccc wwccuncco//eeeeecoom wwwp:/ww//c/tocceoooooct
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary Deep Generative Models. arXiv:1602.05473 fcs, stat], 2016. URL
1602. 05473
Gideon S Mann and Andrew McCallum. Generalized expectation criteria for semi-supervised learning with weakly labeled data. Journal of Machine Learning Research, 11(Feb):95-998, 2010.
Matthew A McDermott, Tom Yan, Tristan Naumann, Nathan Hunt, Harini Suresh, Peter Szolovits, and Marzyeh Ghassemi. Semi-Supervised Biomedical Translation with Cycle Wasserstein Re- gression GANs. In Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pp. 8. 2018. URL https: /wwww/wwww.//// aaai. wwtp///////oo/eeeeeeeeer/eeeeeee phtp://AAT/AAAI/////aaaap viewF FeewFilel/e 0ee//1698//5550
Andrew Miller, Ziad Obermeyer, John Cunningham, and Sendhil Mullainathan. Discriminative Regularization for Latent Variable Models with Applications Electrocardiograapyy In Interna- tional Conference on Machine Learning, pp. 10, 2019. URL BuRl/https::/prrceeddngg mlr
press/v97/milleen 9tt.mammlmml. httpmmmleee199.pp.p.
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual Adversarial Train- ing: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 1199))1979-1993, 2019. URL https
//ieeexplore. org
cocument/841799 htw.:eee.rr//ccccmmee////1111330
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. URL
eduu
housenumbers)
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic Evaluation of Deep Semi- Supervieee Learning Algorithms. arXiv: [cs, stat], 2018.
wtt.vvorrga/bbssseeee 09170

---

Under review as a conference paper at ICLR 2021
N. Siddharth, Brooks Paige, Jan- Willem de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, and Philip H. S. Torr. Learning Disentangled Representations with Semi-Superrised Deep Generative Models. In Advances in Neural Information Processing Systems, 2017. URL /ttp//ttp:///r//../or///////////
70006 00400
Veselin Stoyanov, Alexander Ropson, and Jason Eisner. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In Artificial Intelligence and Statistics, 2011.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87.
doi. 1ttp::///.......11111100..0000 5244:///...544/4... 30.87
Xiang Zhang, Lina Yao, and Feng Yuan. Adversarial Variational Embedding for Robust Semi- supervised Learning. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 139-147, Anchorage AK USA, 2019, ACM.
Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A. Efros. Learning Dense Correspondence via 3D-Guided Cycle Consistency. In 2016 IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pp. 117-126, Las Vegas, NV, USA, 2016. IEEE.
Jun Zhu, Amr Ahmed, and Eric Xing. MedLDA: Maximum margin supervised topic models. The Journal of Machine Learning Research, 13(1):2237-222 2012.
Jun Zhu, Ning Chen, and Eric P Xing. Bayesian inference with posterior regularization and ap- plications to infinite latent SVMs. Journal of Machine Learning Research, 15()):1799-1847, 2014.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired Image-to-Image Transla- tion Using Cycle-Consistent Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242-2251, Venice, 2017. IEEE.
Xiaojin Zhu. Semi-Supervised Learning Literature Survey. Technical Report Technical Report 1530. Department of Computer Science, University of Wisconsin Madison., 2005.
12

---

Under review as a conference paper at ICLR 2021
A CODE AVAILABILITY
During the review process, anonymized code for our methods is available. After review, we will publish all code under permissive open source license.
Browse in your web browser http://ssss....oooooooooooo//osa
Download a ZIP file here:
REVIEW RELEASE. z. -ippdl=1
B DETAILS AND VISUALIZATIONS OF GENERATIVE MODELS
B. NOISE-NORMAL LIKELIHOOD
As discussed in Sec. 4.3, we use "Noise-Normal" distribution as the pixel likelihood for many of experiments. We define this distribution be parameterized two-component mixture of a truncated-normal distribution and a uniform distribution. We will use p to denote the mixture probability of the Normal component, and μ and o to denote the mean and standard deviation of the truncated-normal, respectively. The generative model (or decoder) predicts distinct outlier probability p) for each pixel. We assume that pixel intensities are defined on the domain [- 1] and rescale our datasets to match. We can write the probability density function of the Noise-Normal distribution via the standard normal PDF ()-), and standard normal CDF Φ(-), follows:
(-= 1-
f(x P.14,)
+1 p)
(14)
+
(-=T))
$
We can similarly express the cumulative distribution function of the Noise-Normal distribution as:
④0 -- + (=!z4) +
p. =
+ + p)
(15)
In order to propagate gradients through the sampling process of the noise- normal distribution, use the implicit reparameterization gradients approach of Figurnov et al. (2018). Given a sample æ drawn from this distribution, compute the gradient with respect the parameters p. 14, and a
P.poT -V F(x (.H..))
(16)
f(x p,
When fitting the parameters of this distribution using gradient descent, we enforce the constraints that E [0, 1], f E 1-1, 1], and a > 0. To do this, we optimize unconstrained parameters P.. Ha, T., and then define p = sigmoid(p.) ). μ = tanh(u.). and a softplus(a.).
B.2 SPATIAL TRANSFORMER VAE
We now describe how to sample affine transformations Md, for use in our generative model of images. As described in Sec. 4.3, the latent transformation code has real-valued dimensions, each corresponding to one of the following 6 affine transformation parameters:
horizontal Iranslation,
vertical translation,
rotation,
shear,
(8) horizontal scale,
vertical scale.
To constrain our transformations to fixed range of plausible values, we construct M; using parame- ters 2(()) (anh(z(() that first mapped the interval [1], and then linearly rescaled an appropriate range via hyperparameters (1) (6) Figure illustrates that the induced prior for
13

---

Under review as a conference paper at ICLR 2021
-0.75 -0.50
-0.25
Figure 7: Prior distribution for latent parameters z(()
tanh(2 used to represent affine transformations.
2t is heaviest for extreme values, encouraging aggressive augmentation when sampling from the prior. The mapping function could be changed to modify this distribution for other applications.
(1)
Given these latent transformation parameters, we define an affine transformation matrix M, follows:
0 a0)a00
cos a 19 ")) sin a (()
sin(a wia 10
M,
0
a s(a)
cos(a stll
0
(17)
To determine the parameters of the likelihood function for the pixel at coordinate (i.j), the generative model (or decoder) output at the pixel (i',5") for which
M
3'
(18)
1
This corresponds to applying horizontal and vertical scaling, followed by rotation and shear, followed by translation. We the spatial transformer layer proposed by Jaadeberg al. (2015) with bilinear interpolation to apply this transformation with non-integer pixel coordinates. For the Noise-Normal distribution we independently interpolate the p. μr, and g² parameters.
B.3 CLASS-CONDITIONAL SAMPLING
A standard VAE generates data by sampling (O, I). and then sampling (Noo((), oo(2)), or an alternative like the Noise-Normal likelihood. For the PC-VAE or CPC-VAE, we can further sample images conditioned on particular class label. As labels are not explicitly part of the generative model, we accomplish this by sampling images that would be confidently predicted as the target class. We use rejection sampler, repeatedly sampling 2 (O, I) until sample meets the e, for some target threshold e. We typically use = 0.05 in our experiments.
criteria: P.r 2)
MNIST digit samples for models with a 2-D latent space. Fig. 2 in the main text shows dimensional latent space encodings of the MNIST dataset using several different models. We provide a complementary visualization of generative models in Fig. [ where we compare class-conditional samples for three of these models. The unsupervised VAE's encodings of some classes (e.g.. 2's and 4's and 8's and 9's) are not separated, and samples thus frequently appear to be the wrong class. Model M2 (Kingma al. 2014) explicitly encodes the class label as latent variable, but nevertheless many sampled images do not visually match the conditioned class. In contrast, for our CPC-VAE model almost all samples are easily recognized as the target class.
C SENSITIVITY TO CONSTRAINT MULTIPLIERS
We compare the test accuracy for our consistency-constriin model for MNIST over a range of values for both À (the prediction constraint multiplier) and (the consistency constraint multiplier) in
14

---

Under review as a conference paper at ICLR 2021
0
/ 6
5
9
5
7
8
9
O
1
2
3
4
5
6
7
8
9
0
f
2
3
4
5
6
7
8
9
0
1
2
5
9
5
6
7
9
0
1
3
3
5
6
8
0
f
2
3
4
5
6
7
8
9
0
2
5
9
6
7
Q
2
6
8
0
9
2
3
4
5
6
7
8
9
Q
b
5
9
7
4
5
6
7
8
9
0
F
2
3
4
5
6
7
/
9
Q / 3
3
4
5
6
8
9
0
I
2
3
4
5
6
7
8
9
0
9
2
3
4
5
7
8
9
6 | b
3
7
5
6
7
5
4
0
2
3
4
5
6
7
8
9
0
|
2
3
4
5
6
7
3
9
0 / 2
3
9
5
5
7
0
2
3
5
6
8
0
9
2
3
5
6
7
8
9
0
3
3
9
7
9
6
9
0
9
2
3
4
]
6
7
8
9
0
3
7
5
7
8
9
2
4
5
7
8
9
0
f
2
3
4
5
7
8
9
0
/
2
5
7
5
6
7
8
0
/
2
3
4
5
6
7
8
9
0
9
2
3
4
5
6
7
8
9
(a) Unsupervised VAE
(b) CPC-VAE
(c)
M2
Figure 8: Class-conditions samples of the 10 possible digit classes in the MNIST dataset. Each column shows multiple samples from one specific digit class. From left right, each panel shows samples from a standard unsupervised VAE, our CPC- VAE, and model M2 (Kingma et al. (014). All models use 2-dimensional latent code, and are trained on the MNIST dataset with 100 labeled examples (10 per class).
5
24
3
4
7
50
3
7
J0
2
3
4
5
6
10
M
2
3
4
5
63
T7
9
10
11
5
6
18
9
20
2
13
4
6
6
8
9
10
2
35
4
5
6
7
18
19
10
3
8
10
9
Figure 9: Class coneisonniiional samples of the 10 possible digit classes in the SVHN dataset, extending figure 5a The generative model was trained on fully labeled SVHN dataset with prediction and consistency constraints. Samples were chosen via rejection sampling in the latent space with threshold 95% confidence in target. class.
14
02
Altamodal
(a) Sensitivity to
(b) Sensitivity to
Figure 10: Sensitivity of test accuracy to the constraint (lagrange multiplier) hyperparameters À and
figure 10 All runs used our best consistency- constrriie model for MNIST using dense networks.
kept all hyperparameters identical to the previous results (see section D) changing only the value of interest for each run.
We see that the resulting test accuracy smoothly varies across several orders of magnitude, with the optimal result being at near the values chose for our experiments.
15

---

Under review as a conference paper at ICLR 2021
D EXPERIMENTAL PROTOCOL
Here we provide details about models and experiments which did not fit into the primary paper.
D.1 HYPERPARAMETER OPTIMIZATION
The hyperparameter search for all models, including the CPC-VAE and various baselines, used Optuna (Akiba et al. 2019) to achieve the best accuracy labeled validation set. For our 2-layer and 4-layer M2 experiments, we used our own implementation (available in our code release) and followed the hyperparameters used by the original authors. For the 4-layer variant, different settings of a, ranging from 0.05 to 50, reporting both the the result using the original suggested value for (0.1) and the best value for we found for the setting (10). For M2, also dynamically reduced the learning rate when the validation loss plateaued.
tested 10
D.2 NETWORK ARCHITECTURES
For our PC-VAE and CPC-VAE models the MNIST data, we use fully-connected encoder and decoder networks with two hidden layers, 1000 hidden units per layer, and softplus activation functions. Like the M2 model (Kingma et al. 2014). we use 50-dimensional latent space. The original M2 experiments used networks with single hidden layer of 500 units. We compare this to replications with networks matching ours, as well as 4-layer networks.
For the SVHN and NORB datsets, we adapt the wide-residual network architecture (WRN-28-22 ZZgoruyko Komodakis, (016) that was proposed as a standard for semi- remi-supervised deep learning research in Oliver al. (2018). In particular, we use this architecture for our encoder with two notable changes: We replace the final global average pooling layer with dense layers with 1000 hidden units, and add a final dense layer that outputs means and variances for the latent space. We find that the dense layers provide the capacity needed for accomplishing both generative and discriminative tasks with single network. For the decoder network we use "mirrored" version of this architecture, reversing the order of layer sizes used, replacing convolutions with transposed convolutions, and removing pooling layers. We maintain the residual structure of the network. Our best classification results with this architecture were achieved with a latent space dimension of 200.
D.3 BETA- VAE REGULARIZATION
As an additional form of regularization for our model, we allow our hyperparameter optimization adjust a weight on the KL-divergence term in the variational lower bound, which we call in previous work (Higgins et al. 2017):
VAE [ 0, σ) (=tx)) [log (x =)] + 3 (=1==) log p(z)
(19)
( xx)
This allows us to encourage (z x)) to more closely conform to the prior, which may be necessary balance the scale of the objective, depending on the likelihoods used and the dimensionality of the dataset.
D.4 PREDICTION MODEL REGULARIZATION
We add two standard regularization terms to the prediction model used in our constraint, Ow (v/ 2). The first is & regularizer the regression weights, ||w]|2 to help reduce overfitting. The second is an entropy penalty. As (z) defines a categorical distribution over labels, we compute this as: -Eg... (y =) [log In (y =)]. which has been shown to be helpful for semi-supervised learning in Grandvalet & Bengio (204) and was used as part of the standardized training framework of Oliver al. (2018). We allowed our hyperparameter optimization approach to select appropriate weights for both terms.
D.5
IMAGE PRE-PROCESSING
For all of our image datasets, we rescale the inputs to the range [-1, 1]. For our NORB classification results, downsample each image to 48x48 pixels. For our SVHN classification results, convert
16

---

Under review as a conference paper at ICLR 2021
images to greyscale to reduce the representational load generative model. Before the grayscale conversion, we apply contrast normalization to better disambiguate the colors within each image.
For the SVHN and NORB results, follow the recommendation of recent survey of semi- supervised learning methods (Oliver et al. 2018) and apply a single data augmentation technique: random trans up to 2-pixels in each direction. For generative results, retained the original color images and trained with full labels.
D.6 LIKELIHOODS
For all of our image datasets, we use the Noise-Normal likelihood for our CPC methods. For all experiments toy data (e.g. half-moon), we used normal likelihood.
For our implementation of M2 for extensive experiments on MNIST we retained the Bernoulli likelihood used by the original authors (Kingma et al., 2014). That is, we rescaling each pixel's s numerical intensity value to the unit interval [0,1], and then sampled binary values from Bernoulli with probability equal to the intensity.
D.7 SUMMARY OF HYPERPARAMETER SETTINGS FOR FINAL RESULTS
Table 3 below provides all hyperparameter settings used in our experiments.
17

---

Under review as a conference paper at ICLR 2021
Hyperparameter
MNIST (100)
SVHN (1000)
NORB (1000)
Encoder/decoder
FC layers
WRN-28-2 + FC
WRN-28-2 + FC
Fully connected layer size
1000 units
1000 units
1000 units
Network activations
Softplus
Leaky ReLU
Leaky ReLU
Latent dimension
50
200
200
Pixel likelihood
Noise-
Noise-Normal
Moisel
Prediction multiplier À
25
140
80
Consistency multiplier Y
4.25X
1.25A
41
Aggregate consistency penalty
0.1A
0.2A
0.2X
3-VAE weight
1.3
2
Predictor reg. (I]w1()
1
Entropy reg. (ul=) llog Pw (y]=))
0.5A
0.5X
0.5A
Translation range (a (2))
0.2 (image-width)
0.2 x (image-width)
0.2 (image-width)
Rotation range (3)
0.4 rad
0.5 rad
0.4 rad
Shear range (a!
0.2 rad
0.2 rad
0.2 rad
Scale range (a. = a (6)
1.5
1.5
1.5
Optimizer
ADAM
ADAM
ADAM
Learning rate
x 10
10
3 x 10
Table 3: Hyperparameter settings for semi-supervised learning experiments with our CPC-VAE.
E
DATASET DETAILS
For each dataset considered in our paper, we provide more detailed overview of its contents and properties.
E.1 MNIST
Overview. We consider 10-way exclusive categorization task for MNIST digits. We 28-by-28 pixel grayscale images.
Public availability. We will make code to extract our version available after publication.
Data statistics.
Statistics for MNIST are shown in Table
split
num. examples
label distribution
labeled train
100
[0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]
unlabeled train
49900
[0.1 0.11 0.1 0.1 0. 0.09 0.1 0.1
0.1]
labeled valid
10000
[0.1 0.11 0.1 0.1 0.1 0.09 0.1 0.1 0.1 0.1]
labeled test
10000
[0.1 0.11 0.1 0.1 0.1 0.09 0.1 0.1 0.1]
Table MNIST dataset.
E.2 SVHN
Overview. We consider 10-way exclusive categorization task for SVHN digits. We use 32x32 pixel grayscale images.
Public availability. We will make code extract version available after publication.
18

---

Under review as a conference paper at ICLR 2021
Data statistics.
Statistics for SVHN are shown in Table
split
num. examples
label distribution
labeled train
1000
00000010...000000000000000000000
unlabeled train
62257
0..0000.....01122000000000000000
0.06]
labeled valid
10000
[0.07 0.19 0. 14 0.0.0.0....0000000000000 0.08 0.08 0.07 0.06]
labeled test
26032
[0.07 0.20 0.16 0.11 0.10 0.08 0.06 0.06]
Table 5: SVHN dataset.
E.3 NORB
Overview.
We use 48x48 pixel grayscale images.
Public availability. We will make code extract our version available after publication.
Data statistics.
Statistics for NORB are shown in Table 6
split
num. examples
label distribution
labeled train
1000
[0.2 0.2 0.2 0.2
unlabeled train
21300
[0.2 0.2 0.2 0.2 0.2]
labeled valid
2000
[0.2 0.2 0.2 0.2]
labeled test
24300
[0.2 0.2 0.2 0.2 0.2]
Table 6: NORB dataset.
E.4 CELEBA
Overview.
We use 64x64 pixel grayscale images. Images were cropped to square from the CclebA aligned variant and downscaled our 64x64 resolution for computational efficiency. Labels were generated from the provided attributes. Our dataset used classes: woman/neutral face, man/neutral face, woman/smiling, man/smiling.
Public availability. We will make code to extract our version available after publication.
Data statistics.
Statistics for CelebA are shown in Table 7
split
num. examples
label distribution
labeled train
1000
[0.25 0.25 0.25 0.25]
unlabeled train
21300
[0.25 0.25 0.34 0.16]
labeled valid
2000
[0.25 0.25 0.25 0.25]
labeled test
24300
[0.27 0.23 0.35 0.15]
Table 7: CelebA dataset.
19


Reviews:
[{"review_id": "oz0zdgE6H-M", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a new VAE framework for semi-supervised problems, which uses the latent representation \\\\(z\\\\) to reconstruct input image \\\\(x\\\\) and to serve as the features for the classification of the label of \\\\(x\\\\). Based on this framework, the paper also proposes additional \"cycle\" losses, where the label prediction based on \\\\(\\overline{z}\\\\) of \\\\(\\overline{x}\\\\) is close to the true label (for data with supervisions) or the label of \\\\(x\\\\) (for data without supervisions). In addition, the paper also introduces another loss term of \"aggregate label consistency\" and applies other techniques including noise likelihood and STN. The proposed approach outperforms M1 and M2 of Kingma et al. 2014 on the synthetic dataset and shows more stability than M1 and M2. It seems that the performance advantage of the proposed method over others is not very significant on real datasets.\n\nPros:\n- The idea of using  \\\\(z\\\\) to reconstruct input image \\\\(x\\\\) and to serve as the features for the classification of the label of \\\\(x\\\\) is simple and straightforward (in a good way). The consistent PC loss is also intuitive and adds more credits to the technical depth of the paper.\n\n- The released code is a plus of the reproducibility of the paper.\n\n- The writing of the paper is clear in general.\n\nCons:\n\n- The demonstration of the motivations of the paper is a bit confusing to me. The half-moon data in Fig 1 is simple, therefore C=2 is good enough for the models to fit. The proposed model is more stable when C increases. But it's less motivated to increase C in this case given the data simplicity. So C=14 might not be a good setting in this case. Also in Fig 2, MNIST is clearly more complex than the half-moon data. Therefore, C=2 might not be a good setting in this case. The settings of Fig 1 and 2 are less intuitive to demonstrate the motivations.\n\n- One of the main claims is the high running cost of M2. The basic framework (PC) of the paper might have an advantage in running cost. But I'm wondering if it still has when added with the consistent PC loss, aggregate label consistency, and also STN. With those components, the complexity of the proposed model clearly increases and it seems that without those components the proposed model has no clear advantage over others. The paper didn't provide any empirical comparison of the running cost.\n\n- There seems to be no clear advantage of the proposed method over others in terms of performance. There are wins and loses across different datasets. I also have a concern about the fairness of the comparison. For example, it is reasonable to expect that with STN, the performance of M1/M2/others might be also improved. In the comparison, CPC is with STN but the numbers of M1/M2 are from their original papers. Therefore, it is unclear where the performance gain comes from.\n\n-  As the proposed method involves several components. Some of them are not detailed enough. For example, it's a bit unclear to me how the noise likelihood is employed given a short paragraph of description.\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nThe author response addresses some of my concerns. So I have updated my rating from 4 to 5. I recognise the novelty of the proposed framework but I feel that the main framework has not shown a clear performance advantage given several other components are added. Therefore, I am unable to give a clear recommendation for acceptance. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "EW_hEfm_qeS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n### Summary\nThe paper presents a novel methodology for semi-supervised learning with variational autoencoders.\nThe discriminative part is added via prediction constraints to the standard VAE, resulting in PC-VAE. \nThe model is further extended with \"consistency constraints\" (an extra loss component that improves the prediction quality). The final model, CPC-VAE, is experimentally shown to perform better of similar to the competitors. \n\n##########################################################################\n### Reasons for score\n Overall, I give this paper a weak accept. The methodology is novel (to my knowledge), and is shown experimentally to outperform the competitors. However, there is no sufficient motivation behind the consistency constraints. \n The discussion of its effect is also lacking.\n \n##########################################################################\n### Pros \n1) The paper is well written and is easy to understand.\n2) A good introduction into the problem is given, with comprehensive background on unsupervised\nand semi-supervised VAE.\n3) The first proposed model, PC-VAE, is well motivated.\n4) The experiments are thorough. Enough details are given (in the supplement) to ensure reproducibility.\n\n\n##########################################################################\n### Cons\n\n1) Related work is scattered throughout the paper (intro, background, section 3.2), which makes\nit hard to quickly place the work in perspective.\n2) The hyperparemeter \\lambda seems to play crucial role in balancing the generative and predictive parts. However,\nthe analysis of the relationship between \\epsilon and \\lambda is not thorough. There are no specific recommendations provided for the choice of \\lambda based on the desired \\epsilon.\n3) The second model, CPC-VAE, is different from the PC-VAE in 2 ways: the consistency constraints  (eq 10-11)\nand the aggregate label consistency constraint. Neither of these additions is motivated by the model or inference scheme and are heuristic regularisers. At the same time, the unmotivated heuristic regularizer used by [Kingma 2014] (and further works) was one of the big motivation points for this work.\n4) The paper does not discuss reasons for the bad performance of the first proposed model (PC-VAE) in Table 2,\nand only focuses on its modification (CPC-VAE). My intuition is that the discriminator part in PC-VAE overfits.\nThe issue seems to be fixed by the consistency constraints. This should be discussed and more intuition \nshould be provided as to how these consistency constraints fix the PC-VAE model.\n\n\n##########################################################################\n### Some typos/ minor comments\n\nFig. 2: The plot for the M2 model is unnecessary and can be misleading because the latent space\nin M2 only provides auxiliary information to the class,\nis not supposed to encode the class, and is not used to infer the class.\n\nP.6 \"[...], which forces the distribution of label predictions for unlabeled data is align with a known target distribution \\pi.\"\nBad sentence/incorrect grammar. Rephrase, please, e.g. \"[...], which forces the distribution of label predictions for \nunlabeled data to be aligned with a known target distribution \\pi.\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ap4ql6A2ebL", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework for semi-supervised settings to leverage both unlabeled data and (limited) labeled data where VAEs are trained subject to regularization terms from label information. More specifically, the proposed method trains a VAE and a NN classifier simultaneously by optimizing an objective that consists of the usual (unsupervised) variational lower bound, classification error for the labeled data based on the latent space, and consistency term for all data encouraging the same prediction for latent representations corresponding to the original and reconstructed version of a data point. The proposed method is compared with a few other deep generative semi-supervised learning methods on three image datasets.\n\n+:\nThe paper is well written. It mentions some of the shortcomings of the previous approaches and motivates the proposed method and puts it into context. It is nice to see the details of the hyperparameter values used in the experiments. The results show reasonable performance in terms of classification accuracy, and demonstrate the effectiveness of using unlabeled data, predictive loss regularization, consistency loss regularization, and aggregate label consistency regularization.\n\n-:\nThe proposed method involves several pieces that are put together. The regularization term from prediction loss, the regularization term from consistency loss, aggregate label consistency term, entropy regularization term for predictions, “noise-normal” likelihood, considering dimensions accounting for transformations in the latent space, KL scaling, …. The importance and contribution of some pieces to the performance is not clear, like the likelihood and entropy regularization. Especially that some like the likelihood, KL scaling, and transforms seem to be applicable to other VAE-based semi-supervised methods. Additionally, as all these terms have an associated multiplier, investigating sensitivity to these hyperparameters is important and will be helpful. The results in Table 1 and 2 are not very promising in terms of outperforming other existing methods.\n\nOverall, the paper adds another VAE-based semi-supervised learning method that tries to address some of the limitations of previous approaches, but due to the above points, in my opinion in its current format is at a marginal level.\n\nPost-rebuttal: Thanks for the authors’ response. After reading the responses and other comments and checking the updates to the paper, I retain the score and my recommendation at weak accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-ALso4D_yp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors developed two constraints (and other techniques like using Spatial transformer, aggregate label consistency) that can better balance the generative and discriminative goals when learning VAEs and other generative models. The paper is easy to follow, the proposed methods look interesting and sound to me. According to Figure 1 and 2 in the paper, the proposed methods do have some benefits over other prior approaches. However, I have the below concerns which make me hesitate to accept this paper.\n\nRegarding the overall performance:\nOne contribution, as highlighted in the abstract, is that the proposed method PC VAE and CPC VAE can boost semi-supervised classification performance. However, the major results shown in Table one and Table two seem do not strongly support this claim. In the Table one, the proposed approach only clearly outperforms other methods in NORB. Actually, many counterpart methods this paper compared with do not have results for NORB.\nIn the Table 2, clear benefits are from cycle-consistency rather than the predictive constraint, and the CPC also does not show clear benefits over M1+M2.\n\nRegarding predictive constraints:\nThe PC VAE is very well motivated. However, it finally falls into a multitask objective (equation 9) for handling both labeled and unlabeled data. It is not clear that how PC itself seriously improve over prior VAE-based semi-supervised learning approaches. \nIn the Table two, PC does not outperform M2 and M1+M2. Also, in the Table two, can we compare PC directly with its deterministic counterpart?\n\nRegarding consistent PC:\nThe proposed approach is a little bit similar to the ``\"Unsupervised data augmentation for consistency training”. According the Table two, this technique improves the performance a lot. What if this consistency loss (and or other tricks like aggregate label consistency) was added to loss terms like (7)?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a semi-supervised model (named CPC-VAE) that trains a variational autoencoder (VAE) and a NN classifier simultaneously. The method maximizes an ELBO subject to a task-specific prediction constraint and a consistency constraint. The constraints are defined as some expectations of the variational posteriors. Such constraints are known as posterior regularization. Though the consistency constraint seems to be new, the prediction constraint has been well-examined under deep generative models (see e.g., max-margin deep generative models for (semi-)supervised learning, IEEE TPAMI, 2018). The paper needs more a thorough analysis and comparison. 
Decision: Reject