Variational Classification
Authors: Shehzaad Zuzar Dhuliawala
Mrinmaya Sachan
Carl Allen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Latent priors
classification

Abstract: Classification tasks, ubiquitous across machine learning, are commonly tackled by a suitably designed neural network with a softmax output layer, mapping each data point to a categorical distribution over class labels. 
We extend this familiar model from a latent variable perspective to variational classification (VC), analogous to how the variational auto-encoder relates to its deterministic counterpart. We derive a training objective based on the ELBO together with an \textit{adversarial} approach for optimising it.

Within this framework, we identify design choices made implicitly in off-the-shelf softmax functions and can instead include domain-specific assumptions, such as class-conditional latent priors. We demonstrate benefits of the VC model in image classification. We show on several standard datasets, that treating inputs to the softmax layer as latent variables under a mixture of Gaussians prior, improves several desirable aspects of a classifier, such as prediction accuracy, calibration, out-of-domain calibration and adversarial robustness.

Under review as a conference paper at ICLR 2023
VARIATIONAL CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Classification tasks, ubiquitous across machine learning, are commonly tackled by a suitably designed neural network with a sofimax output layer, mapping each data point to categorical distribution over class labels. We extend this familiar model from a latent variable perspective to variational classification (VC), analogous to how the variational auto-encoder relates to its deterministic counterpart. We derive a training objective based on the ELBO together with an adversarial approach for optimising it. Within this framework, we identify design choices made im- plicitly in off-the-shelf softmax functions and can instead include domain-specific assumptions, such as caasscooniiional latent priors. We demonstrate benefits of the VC model in image classification. We show on several standard datasets, that treating inputs to the softmax layer as latent variables under a mixture of Gaussians prior, improves several desirable aspects of classifier, such as prediction accuracy, calibration, out of domain calibration and adversarial robustness.
INTRODUCTION
Classification is central to much of machine learning, not only in its own right, c.g. to categorise every day objects (Klasson et al. 2019). make medical diagnoses (Adem et al. 2019 Mirbabaie et al. 2021] or detect potentially life-supporting planets (Tiensuu al. 2019), but also an important component in other learning paradigms, e.g. to select actions in reinforcement learning, distinguish positive and negative samples in contrastive learning or within the attention mechanism of large language models. Recently, it has become all but default to tackle classification tasks with domain-specific neural networks with a sigmoid or softmax output layer[] The neural network deterministically maps each data point (in domain X) to a real vector (x)) which the last layer maps to the parameter of discrete distribution po(y)(z) (z) over class labels y e V, defined by point on the simplex AP', e.g.
The
exp(f((x) Wy by) po(u)æ) = softmax (2; f) Eyey exp g(2, y( 0) Evey cxp Wy + by.
exp g(z, y: 0)
(1)
Despite frequently outperforming alternatives and their widespread use, softmax classifiers are not without issue. The overall mapping from X' to AD' is learned numerically by minimising a loss function over a finite set of training samples. The result is poorly understood in general, remaining in many respects a "black box" with predictions hard to rationalise. A trained classifier may make accurate predictions for the training set, but predictions for other data points, e.g. test data, are determined by fa FF, from a class of functions chosen to be highly flexible in the hope of approximating the unknown true mapping f(x) {p(y/ beey. With sufficient ficxibility, (i) F may contain many, possibly infinite, functions (for different w) that give accurate training set predictions, but dissimilar and hence uncertain predictions elsewhere; and (ii) predictions
change materially for imperceptible changes in the data (adversarial examples). Lastly, where po(y/z>) fails to reflect the true label distribution p(y|ır) it fails reflect the frequency with which classes are expected to occur, making the classifier miscalibrated.
A standard softmax classifier also lacks several desirable properties. For example, under certain conditions, it is known that a softmax classifier learns to approximate p(y|z) and so captures stochastic, or aleatoric, uncertainty in the data. However, in practice, predictions for some regions of X' may be more reliable than others and it can be important to understand the confidence or epistemic
Since the softmax function generalises the sigmoid function to more than two classes, we refer to softmax throughout, but all arguments be applied to the sigmoid case.

---

Under review as a conference paper at ICLR 2023
Figure 1: Distributions of softmax inputs 4s (z)g), Z R², under the three VC training objectives (MNIST dataset): (left) standard softmax Cross Entropy "MLE" treatment; (centre) with class- conditional Gaussian priors po (zlpy)- "MAP" treatment; (right) and entropy "Bayesian" treatment. (Note: softmax inputs have been artificially restricted to 2-dimensional for visualisation purposes.)
uncertainty of predictions. Also, beyond making positive classifications from set of labels, it can be useful to know when a sample is "none of the above", or out of distribution. In such cases, softmax might output uniform distribution over all classes but that is equally appropriate for a fully in in-distribution sample that occurs with all labels, and so ambiguous without further assumptions.
Overall, classification accuracy, model calibration and adversarial robustness depend on how well model Po (y (r) approximates the distribution ply|z)) in particular how it interpllaeesccttrpolatte from the training set to X. Meanwhile, prediction confidence and out-of-sample detection depend on how new data samples compare those seen at training time. This suggests that to improve softmax classifier in general requires (i) modelling p(ylar) more accurately, e-g. by obtaining more data or constraining fw useful, perhaps domain -speifc-peecffic way; and (ii) developing measure of familiarity between data samples, e.g. by learning a distribution over x, similar.
We approach this by generalising the mechanics of the softmax classifier guided by two key observa-
tions: (i) the parallel between softmax po (o(l()) = Ey exppl eirp(g((.:))] and Bayes' rule p(yl (z) Ey p(
exp(n(z..))
m(a,ay)
and (ii) that each layer of a neural network classifier is a function of random variable (the data x) and so can be treated as a latent random variable The first hints at a probabilistic interpretation of the softmax function. The second also underpins the relationship between the deterministic auto-encoder and the variational auto-encoder (VAE) (Kingma & Welling 2014] Rezende et al.) 2014), the latter of which both generalises and constrains the former. a similar way, we propose the variational classifier (VC) that introduces latent variable z into the (Markov) prediction model po(y|æ>= (<dyl>>=//) Je (y( )p (z(z). with marginal p(z). This offers a promising, principled way to address the issues outlined if p(z) both usefully constrains the model improve accuracy; and, when evalu- ated for latent variables associated with observed data, indicates their "familiarity", We develop an analog of the ELBO to train a VC and propose an "adversarial trick" for its optimisation.
We show that the standard softmax classifier falls within the VC framework under distributional assumptions that equate to implicit design choices. By identifying such choices, alternatives can be introduced where appropriate, such domain-specifi latent priors. This also shows that the VC framework does not "complicate matters" by requiring difficult distribution choices to be made, rather it exposes that default assumptions are made in softmax classifiers that may not be optimal. A regular softmax classifier is also seen to concentrate class- coiss-ondiiionall latent distributions q (2lyy)= [y)= Jx (2(2()))) (r)))))))))))).)))))))))))))))))
single point, akin to maximium likelihood point estimate, whereas VC fits (2ly)) to class-conde condicooniiional prior pe (z ly) mirroring more Bayesian treatment (see Figure I
On a series of image classification experiments, we demonstrate that a VC outperforms a regular softmax classifier in many of the ways outlined, such calibration, including for out of domain samples, adversarial robustness, and modestly improves accuracy, more notably if data is scarce.
We believe the VC framework offers deeper interpretation of softmax classification and takes a step towards more fully understanding these familiar models, potentially enabling their further improvement and/or integration with other latent variable models, e.g. VAEs contrastive learning.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND (THE VARIATIONAL AUTO-ENCODER)
The proposed generalisation from softmax to variational classification is highly analogous to how the deterministic auto-encoder relates to the variational auto-encoder, which we briefly summarise.
Maximising the likelihood of the data to estimate parameters of the latent variable model pe (x)
S. is intractable in general, so the evidence lower bound (ELBO) is maximised instead:
f p(x) log (x)) p(z) / co(2)1)) log po(x)2) log
de(alz) polz)
po(e) f + log (o(2)))
}
> J p(z)) { log /.
(=(=)ar) log ga(sla)
} = ELBO (2)
po(2)
Maximising the ELBO in equation
2 is equivalent to minimising
DxL (xl[p(x)) ([(p(()]]]) pe (1r) + Ey [Pxa ( [=<(==
where Pka,(p(>)) [(x))= p(x)) log g(z)
p(z) is the Kullback- -ellback-Leibler (KI.) divergence. This shows that
maximising the ELBO fits the model (I) to the data distribution p(z), whilst fitting the approximate
posterior qs (z]z) to the implied posterior under the model po(z|z)) p=(=)=)pu(=), or ensuring that
po(#)
the two modelled distributions qo (=|wr) and po(x]2> are convistent under Bayes' rule.
The Variational Auto-Encoder (VAE) (Kingma & Welling 2014 Rezende al. 2014) implements
the ELBO with the parameters of pe and Is defined by flexible neural networks. Restricting the variance of qæ towards zero such that go tends delta distribution, the first term of equation tends to the loss function of a deterministic auto-encoder. As such, the VAE can be seen to generalise the deterministic case, allowing for uncertainty or stochasticity in the latent variables while constraining their marginal distribution p(z) with the second "regularisation" term.
3 VARIATIONAL CLASSIFICATION
We now present the the (latent) variational classifier (VC) as generalisation of the softmax classifier.
standard softmax classifier is deterministic function that maps each data point E X through a sequence of intermediate representations to prediction po(y)æ). a categorical distribution over labels E V, defined by point on the simplex AD', Treating each data sample as a realisation of a random variable x with distribution p(x). any intermediate representation 2 =g()) can be considered the realisation of latent random variable z with distribution p(z) defined implicitly by sampling pxx). Assuming the Markov property x - + y, the latent VC model is given by:
pe(y|æ) pa (v]2))0(<((r)
(4)
While equation 4 is more general, we keep softmax classification mind with a running example where is the input to the softmax layer; (=(2)=)=0= (a). delta distribution parameterised by the neural network up to the softmax layer [-): and (y(y)=) (=) is categorical label distribution defined by the softmax layer. We now derive an objective to learn parameters of the VC model.
Similarly to the latent variable model for (x( (section 2). the VC model cannot generally be learned by likelihood maximisation, hence we compute a lower bound comparable to the ELBO (eq. 2
[ p(.,, y) log (o()x) 2.9
J /p(or,00 (0(2)))) log log (y)l,,))) go (=(2)))) po(a|z,0)
go(ola)
[ pla w\ (0(2|1)) logpo(vlz, - [[ (pla,)) w) / go(2|2)) po (5)
where Po(y|æ, 2) ())=2 in the last step (by Markov). A strict analogue of the ELBO would replace by y in equation 2 and condition throughout on (new) such that the variational distribution qa depends on both x and y (c.g. see Tang & Salakhutdinov, 2013). Instead, 9s is chosen to depend
only on æ, hence the KI. term Daaa [ (2(2)))) [ po(2 y) (dropped in line is minimised but, in
3

---

Under review as a conference paper at ICLR 2023
dog
ADI
Nird
Ply=cat|zz Ply=doglz))
pelyla) f
car
pobxly = hird)
P(zly=cat) P(zly=dog)
pu(xly hind)
X
Figure 2: The Variation Classier: (l) (z(z)z) stochastically maps data the latent domain, where class-conditions distributions (zly) are fitted to priors po (zly) with known analytical form; class predictions po(y|2) 2) are "read off"" by Bayes' rule. (r-bottom) images from line in the data domain X' that passes through two data samples (dog and cat images); (r-middle) p(z) for the corresponding latent variables = [w)); (r-top) corresponding output predictions po (y(z)) Note that output predictions may seem confident even where p(z) indicates lower confidence should be attributed.
general, may not vanish when maximising the lower bound (final expression). However, do have Deal go (=|==]] po(2)2, y)] = 0 in the limit that z sufficient statistic for y given x, i.e. z contains all information about y in x Hence maximising the lower bound and so minimising [ (2[xx)) po(2|z; ax, y)] pushes z towards a sufficient statistic.
So far, the training objective (last term, eq. closely resembles cross- coss-entropy, which it reduces to (=fxx) is set to delta function. However, now, the modelled distribution pa (yle (2) and the implied distribution (=ly)) Se go (z(x)p(<1))) yy) must be consistent under Bayes' rule, analogous to po (x|2) pa(xlu)po())
and qo (2)2)) ) in the ELBO (section 2 Hence. by modelling pe (yylz)== Σy po(=ly')po(r)) we should
have (2(y)) qo (zly), or Dxul 9o (z(1)) (=(x)))]= o To clarify, in the general case, for a class y: (zly)) is implicit distribution defined by putting samples px>((xy) through the neural network parameterising go (z/2z and sampling therefrom; whereas po (zly) is class-conditional latent prior defined analytically. In the running softmax example, g (zly) is given directly by putting samples from class through the neural network: 2 = ( p(x
Including the KI. constraint and model of the class distribution Psr (y), we maximise (VC objective):
[ pla. f 2.9
wi go log pogl2())))
go (2ly) log po(aly)
(s(z)y)
(v)
(6)
Taken incrementally, terms of the VC objective involving (note the KL term has 2) can be seen to treat the latent from a maximum likelihood, maximum a posteriori and Bayesian perspective:
maximising [=]=) log (y(lz) gives ( 2n = arg max, po(l)=)
[MLE]
(ii) maximising the prior J f. (2 ly) log (o((21)) (=/v) alters the above point estimate 2n
[MAP]
maximising entropy J (0 log g
(= =>yy) has qs "fill" the prior po(yly)
[Bayesian]
"p(2/z, v)p(y)z) = = 2)p(= = p(2|æ, -)= ( + p(y)))==(pll>>)> =)) (by Markov) NNote that the analogous term in the ELBBO Daal (=(r))) (=(2))) (conditioned on z rather than y). is minimised implicitly when maximiising the EIBOO,, whereas minimise Dxul (s(zl))] (2lv)) explicitly.
4

---

Under review as a conference paper at ICLR 2023
Comparing to the KL form of the ELBO (equation 3 maximising the VC objective minimises:
Ex (pkulp((pl=)] (t]lz)) + Bz,u Do (z (e y)]]
+ Eyy D go pe P Pa Pa(v]]]
(7)
revealing how the central objective of modelling ply(z) by pe (z) is constrained. Figure gives an overview of the VC model. Components of variational classifier can be interpreted as follows:
the neural network up to the (generalised) softmax layer (fw) effectively transform mixture of analytically unknown class-conditional distributions p(x|y) over X, to a mixture of analytically defined distributions po (z. (z]y) over Z;
with inputs following anticipated mixture of class-conditional distributions, the (generalised) softmax layer effectively "reads off" class predictions by Bayes' rule.
3.1 RELATIONSHIP TO SOFTMAX
Training a Variational Classifier by maximising its objective (eq. 6) requires 4s (z)x) and pa (zly) to be defined (p,, (y) assumed categorical for classification). While the VC objective applies more generally. as in the rumning example we now treat softmax layer inputs as samples of and go (z|zz)
delta function parameterised by the neural network: = ==== (ar); go (=)xr))= - (w)- Under these
assumptions, the first term of the VC objective equates to softmax cross-entropy (SCE)
[ (0(2)x) log
po po(zly)Do(v)
(x) Wy + by}
Po(y)z) log
(8)
Eyey Po
Σ Evey (xp(Su(x)) wy + by.
if po (2(p) = exp(2 w,, + 1/ + c(2)}, Vy where c(z) may depend on but not y, and Uú absorbs po(y). This shows that the standard softmax cross-entropy loss fits the "MLE"" form of the VC objective (i.e. the first term only), under the implicit assumptions (g(2(x) is delta distribution (Vx), and latent variables follow a mixture of (similar) class condiiiniii exponential family distributions. e.g. equivariate Gaussians. We note a close analogy to how deterministic auto-encoder fits within the variational auto-encoder framework.
In corresponding to the MLE form of the VC objective, softmax cross-entropy (SCE) loss accom- modates the softmax input distribution qo (zly) (i.e. z= (z)), x>>p(xl)) following the class prior po(yly), but does not encourage samples to fit Indeed, SCE is optimised ga (7ly) "collapses" to single point maximising po (y) 22) (see (i), Section 3). Figure shows the softmax input distributions (2(g)) as each term of the VC objective included: the prior has clear impact (left centre), whereas the entropy effect is more subtle (centre to right). We also see that, in practice, 4s (zly)) for each class y does not fully collapse to point, likely due to constraints on fas e.g. Eg regularisation.
Since softmax classification is special case of the VC framework, the latter does not add unnecessary complexity by requiring difficult distributional assumptions be made, rather exposes that unscrutinised assumptions lie within softmax classifiers. By generalising the softmax case, the variational classifier enables such assumptions to varied, e.g. to use domain-specifie priors po (zly), and extends the MIEE latent variable model to fuller Bayesian-like treatment.
softmax classifier is readily extended to VC by: (i) optimising the full VC objective (eq. 6 such that softmax inputs for class y, i.e. Z go are encouraged to fit pa (zly); and (ii) relaxing the constraint that class-conditional priors are similar (e.g. equivariant) exponential family distributions, e.g. letting pe (zly) for each class y be multivariate Gaussian with parameters learned from the data.
3.2 OPTIMISING THE VC OBJECTIVE
Of the VC objective (eq.6)
the first term can be calculated by sampling (z(z)z) (using the "reparameterisation trick" if necessary (Kingma & Welling 2014)) and computing po (ylz) by Bayes' rule:
the third term standard multinomial cross-entropy; but
the second term less immediate since go (zll) an implicit distribution, which cannot be evaluated only sampled, by sampling g [/]x), defined by the neural network, for z~~pl pxxl).
5

---

Under review as a conference paper at ICLR 2023
Fortunately, we only require log ratios, log (e( Vy E V/ which can each be approximated by training a binary classifier to distinguish samples of (o(zly) from po (zly). This contrastive "trick" has become increasingly common and underpins learning methods such as Noise Contrastive Estimation (Gutmann & Hyvärinen 2010) and Contrastive Learning (Oord et al. 2018) and has been used in a similar way train variants of the VAE (Makhzani al. 2015 Mescheder al. 2017).
[o(ely)
Specifically, we maximise an auxiliary objective w.r.t. the parameters x/r of a set of binary classifiers:
| (o(=lv) log (T)(
(og))
where a is the logistic sigmoid function (x) = (0=))=(1+e77) (1+ (x) = wy 2 by and セ= (wy, by- It is easy to show that equation 9 is optimised if (2) (x(=lp)
(=(=))) Vy hence when all binary classifiers are trained, {TH (2) (z)]we vey approximate the log ratios required by the VC objective.
This approach adversarial since the VC objective is maximised when the log ratios are minimsed, i.e. (a(=ly) = po(z)y) and samples produced by the neural network are indistinguishable from those of the prior; whereas the auxiliary objective is maximised when the log ratio maximised and the two distributions discriminated. Any distributional differences that the auxiliary binary classifiers identify are removed under the main objective by bringing the distributions closer together. until the distributions match. Similar to Generative Adversarial Networks (GANs) (Goodfellow al. 2014). the neural network fw can be considered generator and each binary classifier a discriminator, however, require one discriminator per class that each distinguish generated samples from a learned rather than fixed reference distribution; and whereas a GAN discriminator typically distinguishes between highly complex distributions, each VC discriminator simply compares a Gaussian to an approximate Gaussian, requiring only logistic regression.
In principle, require gradients of the log ratios w.r.t parameters 0, ó of the VC objective. However, the gradient w.r.t. the occurrence of é in the log ratio is zero (Mescheder et al., (217)) and that w.r.t. can be computed directly from equation and no gradients from the binary classifiers required.
3.3 INTERPRETING VARIATIONAL CLASSIFICATION
Variational classification might be interpreted in one of two ways:
Well-specified generative model: Assume data X' is generated according to hierarchical model: y - z > x, where p(y) is categorical; p(zly) are analytically known distributions with z of manageable dimension, e.g. N (Py Ey); : and x = h((z) for arbitrary invertible function h Z - X (if X' is of higher dimension than Z, assume maps one-to-one to a manifold in X'). Hence p(x) is mixture of unknown distributions. By choosing po (2|y) to have the form of the true p(zly), variational classification effectively aims to invert h and learn the parameters of the true generative model. In practice, the model parameters including h may only be identifiable up to certain equivalences, however the learned latent variables may reflect true latent variables and be semantically meaningful.
Miss-specified model: Assume data is generated as above, but with of vast, possibly uncountable dimension with complex onter-depeneencies, e.g. determining every blade of grass in a landscape or every hair on a cat. an impossible task to learn all such latent variables with lower dimensional model, in general. As such, the VC latent space might learn a complex function of many true latent variables.
The former scenario appeals since the model may Iearn disentangled, semantically meaningful features of the data. However, that requires well-specified distributions and the number of true latent variables may make it impossible. For natural data with many latent variables, the second case seems more likely and choosing (z(z))) to be Gaussian may be justified by the Central Limit Theorem.
RELATED WORK
Noting that the softmax denominator satisfies Evey exp g(,, v) x (x)) Grathwohl et al. (0019) view the softmax classifier as energy-based (i.e. un-normalised probabilistie) model. Generative aspects aside, our work is comparable in the sense of taking an abstracted view of softmax classifica- tion to improve aspects of it. However, the benefits they obtain, e.g. calibration and adversarial

---

Under review as a conference paper at ICLR 2023
robustness, come at cost to the core aim of accurate classification. Further, the MCMC step they require for normalisation reportedly slows and destabilises training, whereas we use tractable probability distributions with little overhead.
Several previous works adapt the standard ELBO, used to learn a model of p(r), to a conditional analog for learning p(y|ar) (Tang & Salakhutdinov 2013 Sohn al. 2015). However, such works focus on generative scenarios rather than classification, e.g. x being a face image and y/æ being the same face in a different pose determined by latent 2; or x being part of an image and yla its completion given latent content z. The Gaussian stochastic neural network (GSNN) model (Sohn) (et al., 2015) takes further step towards our own by conditioning q(2 2.,9) y) only on z, however none of these models considers the class-level latent distributions ((=ly) in variational classification.
Variational classification effectively subsumes many works that add a regularisation term to the softmax cross-entropy loss function, interpretable as prior over latent variables in the "MAP" VC model. For example, several remi-supervised learning models can be interpreted as treating softmax oulputy as latent variables, where the prior guides the outputs of unlabelled data (e.g. see AAlen et al., 2020). More closely related to variational classification, several works can be seen to treat the softmax imput latent variable with regularisation term applied to encourage certain prior beliefs. For example, label predictions can be encouraged to be deterministic (all probability mass on a single class) by enforcing large margin between clusters in the latent space (Liu et al. 2016 Wen et al. 2016
Wan et al. 2018 2022).
5 EMPIRICAL VALIDATION
We now aim show empirically that the latent structure imposed by the variational classifier objective improves various useful properties relative to the standard softmax classifier. In principle, the VC model is expected to be applicable wherever softmax classifier is used, if useful distributional assumptions can be made. We choose the visual domain and demonstrate performance of the variational classifier across a range tasks on familiar datasets.
For fair comparison, we make minimal changes to adapt standard softmax classifier to variational classifier. As described in section 3.T. we train on the VC training objective (equation (6) under the following assumptions: Ja (z(a)))) is delta distribution parameterised by neural network Jw X - 2; caas-conndiiinal priors pe (zly) are mulli-variate Gaussians with parameters Icarned from the data (we use diagonal covariance for simplicity). To examine the effect of each component of the VC objective, we compare classifiers trained to maximise three objective functions (see section 3):
JCE = Ja,,, p(z, v) [.= (0(2(z) log (a(())) + log Ps (v)}.
This is equivalent to standard softmax cross-entropy under the above assumptions and corresponds to the MI.E form of the VC objective (section (i) We refer to this model CE.
JGM = JCE + Se,y p(x, y) Sg (2)))) log (o(2(y)
This includes class-conditional priors and corresponds to the MAP form of the VC objective (section 2 [i)). We refer to this model as GM.
Jvc = JGas - Ja,yp(e y) (=(()) log qs (2ly)
This includes entropy of the generated latent class-conditional distributions and corresponds to the Bayesian form of the VC objective (section (in).. We refer to this model as VC.
5.1 ACCURACY AND CALIBRATION
We first look to compare the classification accuracy and calibration of each model on two stan- dard benchmarks (CIFAR-10 10 and CIFAR-100), across two standard ResNet model architectures
CIFAR-I 10
CIFAR- 100
CE
GM
VC
CE
GM
VC
WideResNNe-28-110
96.11
95.02
96.27
80.21
79.53
80.44
ResNet-50
93.76
92.97
93.28
73.19
74.31
73.42
Table 1: Classification Accuracy (%)
7

---

Under review as a conference paper at ICLR 2023
CIFAR-TO
CIFAR-T00
CE
GM
VC
CE
GM
VC
WideRexNeet28-10
3.10
3,51
2.06
10.23
19.58
4.81
R0
3.72
4.13
3.17
8.72
10.60
5.21
Table 2: Expected Calibration Error (%) (lower is better).
(WideResNee-228 28-10 10 and ResNet-50). Calibration is evaluated in terms of the Expected Calibration Error (ECE) (uo et al. 2017) (sce Appendix (A).
Table T shows that the VC model achieves slight performance improvement for the more powerful WideResNer-228
model, while per- forming competitively in general. However, Ta- ble shows that the VC model is by far the most calibrated. We note that this calibration not performed post hoc and requires no ex- ternal calibration set, unlike approaches such as Platt's scaling and temperature scaling (Plati et al. 1999] Guo et al. (017). Reliability dia- grams comparing the VC and CE models are plotted in Figure
58
Figure 3: Reliability Diagrams: (I) CE, (r) VC
5.2 OUT OF DISTRIBUTION GENERALIZATION
We first test the ability detect OOD examples by computing the AUROC when model is trained
Model (c)(yl Pvc(ulz)
SVHN
C-100
CelebA
0.92
0.88
0.90
on CIFAR-10 10 and evaluated on the validation
of CIFAR-10 10 mixed (in turn) with SVHN, CIFAR-100
0.93
0.86
0.89
and CELEBA (Goodfellow et al. 2013 Liu et al. (013). We compare the VC and CE models using the probabil- ity of the predicted class arg max, pe (yl (z) as a means of identifying OOD samples.
Table 3: AUROC for the OOD detection
task. Models trained CIFAR- 100 and evaluated on in and out-of-domain samples.
Table shows that VC does not perform better than the standard model. For the VC model, we also tried p(z) as metric to detect OOD samples and found it perform comparably. Our results are fairly consistent with those of Grathwohl et al. (2019). We conclude that although the VC model more structured latent space, this does not extend OOD data, which are mapped randomly and hence we perform comparably to CE.
learss map training samples
When deployed in real-world settings, machine learning models encounter distribution shift relative to the training data. It can be imperative to know when the model output is reliable and be trusted, requiring that models be calibrated OOD data and know when they do know. To test performance under distribution shift, we look to the robustness benchmarks proposed by Hendrycks & Dietterich/ (2019): they simulate distribution shift by adding varying intensities of different corruptions to a dataset. We compare the CE to VC models on two proposed benchmarks: CIFAR-10- and CIFAR- 100-C.
Model
0.40
Model
0.35
LVC
LVC
0.5
0.30
0.3
0.15
0.10
0.1
0.05
2.0
3.0
4.0
5.0
Slift Intersity
Intersity
Figure 4: Calibration under distribution shift (corruption). (1) CIFAR- (r) CIFAR-100-C
8

---

Under review as a conference paper at ICLR 2023
100
LVC CE
100
LVC
CE
Accuracy
Accuracy
60
40
60
50
20
0.0
011
0.2
0.4
0.0
0.1
0.3
0.5
Epsilon
Epilion
Figure 5: Prediction accuracy as FGSM adversarial attacks increase (l) MNIST; (r) CIFAR-10
Both models perform comparably in terms of classification accuracy (see Appendix), which accords with the results above. However, as shown in Figure VC has consistently lower calibration error as the corruption intensity increases (left to right). We note that the benefit of the VC model is more noticeable in the more challenging CIFAR- 100-C task (right plot).
5.3 ADVERSARIAL ROBUSTNESS
Here, we look to understand regularising a classifier's latent structure improves adversarial robust- We test model robustness by measuting performance on adversarially generated images using the common Fast Gradient Sign Method (FGSM) of adversarial attack. Perturbations are generated as P X sign (C(z, y)). where [(x, y) is the model loss for datapoint and correct class y, and is the magnitude of the adversarial attack. We compare all models trained on MNIST and CIFAR-10 against adversarial FGSM attacks of different magnitudes.
Fig 5 shows the degradation in accuracy as the attack is intensified. For both datasets, CE performs consistently poorly, while VC shows consistent increased robustness to the adversarial attacks.
5.4 Low DATA REGIME
Lastly, we briefly investigate model performance when data is scarce on the hypothesis that imposing prior over the latent space may enable model to generalise better from fewer samples. Each model is trained on only 500 training samples from the MNIST dataset.
CE
GM
VC 94.2 + 0.4
93.1 + 0.5
94.4 1 0.3
Table 4: Classification accuracy with low data (mean and standard error over 5 runs)
The results in Table conffrrm that the prior, present both the GM and VC models, improves model accuracy in low data regime.
CONCLUSION
We have presented Variational Classification (VC), generalisation of the softmax classifier, mirroring the relationship between the variational auto-encoder and the deterministic auto-encoder. We show that the softmax classifier is special case of the VC model under specific assumptions that effectively taken for granted when using the softmax output layer. We present a training objective train VC analogous to the ELBO, together with an adversarial optimisation regime. series of experiments show that with little computational overhead, a variational image classifier outperforms the standard softmax in several ways, in particular in terms of calibration, adversarial attacks and when data is scarce, without degrading and potentially even improving classification accuracy.
The VC model opens up several interesting future directions. For example, q(2[z) might be modelled as a stochastic distribution rather than a delta function and be trained to reflect uncertainty in the latent variables. Also, having a prior over latent variables may enable semi- cupi-super learning similar to other methods that work by implicitly imposing latent prior.
In terms of limitations, we have focused on particular aspect of softmax classifier, how inputs to the softmax layer can be manipulated to make better predictions. A key outstanding question is what the rest of the neural network f.. does up to that point, or indeed, should do.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
For our experiments, we adapt our implementations for our ResNer-50 and WideResNet-28-/0 models from a public GitHub repository We use their implementation in the basic setting which we keep consistent throughout the CE, G.,, and VC training approaches. Our experiments with MNIST were performed using 6-layer CNN. For our discriminator, we experimented between using a single- and 2-layer neural network. We treated this choice hyperparameter.
We will release our code and models upon acceptance.
ETHICS STATEMENT
We foresee no major ethical considerations for this work. The datasets used in this work do not contain any sensitive information the best our knowledge.
REFERENCES
Kemal Adem, Serhat Kiliçarslan, and Onur Cömert, Classification and diagnosis of cervical cancer with stacked autoencoder and softmax classification. Expert Systems with Applications, 115: 557-564, 2019.
Carl Allen, Ivana Balažević, and Timothy Hospedales. probabilistic model for discriminative and neuro-symbolic cemi-supervised learning. arXiv preprint arXiv:2006.058880 2020.
lan Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Amnud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv:1312.6083 2013.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint 1903 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations, 2014.
Mareus Klasson, Cheng Zhang. and Hedvig Kjellström. A hierarchical grocery store image dataset with visual and semantic labels. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
2019.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In International Conference on Machine Learning. 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
httpp::/ggithbb.com http:///ttotoocoooeeeeeeeeeeeeee
10

---

Under review as a conference paper at ICLR 2023
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:151 2015.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In International conference on machine learning, pp. 2391-2400. PMLR, 2017.
Milad Mirbabaie, Stefan Stieglitz, and Nicholas RJ Frick. Artificial intelligence in disease diagnostics: critical review and classification on the current state research guiding future direction. Health and Technology, 11(4):693-731, 2021.
Jishnu Mukhoti, Andreas Kirsch, Joost Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty: A simple baseline. arXiv e-prints, pp. arXiv-2102, 2021.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /807:0077737 2018.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wienstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
Charlie Tang and Russ R Salakhutdinov. Learning stochastic feedforward neural networks. Advances in Neural Information Processing Systems, 2013.
Jacob Tiensuu, Maja Linderholm, Sofia Dreborg, and Fredrik Öm. Detecting exoplanets with machine learning: A comparative study between convolutional neural networks and support vector machines, 2019.
Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. Rethinking feature distribution for loss functions image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9117-9126, 2018.
Weitao Wan, Jiansheng Chen, Cheng Yu, Tong Wu, Yuanyi Zhong, and Ming- Hsuan Yang. Shaping deep feature space towards gaussian mixture for visual classification. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pp. 499-515. Springer, 2016.

---

Under review as a conference paper at ICLR 2023
A CALIBRATION METRICS
One way to measure if a model is calibrated is to compute the expected difference between the confidence and expected accuracy of model.
Ep(aa)) F(a aal a(P(a))) p)
(10)
This is known expected calibration error (ECE) (Nacini et al. 2013). Practically, ECE is estimated by sorting the predictions by their confidence scores, partitioning the predictions in M equally spaced bins (B1 ... BM) and taking the weighted average of the difference between the average accuracy and average confidence of the bins. In our experiments we use 20 equally spaced bins.
M
Bml
ECE
lacc(
conf(Bmi
(11)
m=1
A.0.1 RELIABILITY DIAGRAMS
Another common tool to visualize model calibration is a reliability diagram. A reliability diagram plots sample accuracy function of confidence for each bin. If a model is perfectly calibrated, the confidence and accuracy bars should be identical.
B OOD DETECTION
80
CIFAR-10
60
CIFAR-10
20
-20
20
-0
-10
-60
-80
-60
-80
Figure 6: t-SNE plots of the feature space for a classifier trained on CIFAR-10. (I) Trained using CE. (r) Trained using VC. We posit that similar to CE, VC model is unable to meaningfully represent data from an entirely different distribution.
C CLASSIFICATION UNDER DOMAIN SHIFT
comparison of accuracy between VC and CE under domain shift can be found in Figure 7 We find that VC performs comparably well as CE.
D SEMANTICS OF THE LATENT SPACE
In effort to try understand what semantics the latent space capture, MNIST model on the Ambiguous MNIST dataset (Mukhoti et al., (021). We then interpolate the ambiguous 7's that spill close the I's and 2's Gaussian. We can qualitatively from the mean of the 7's Gaussian to the I's, the ambiguous 7's start to look more like I's. Our latent space representation can be seen Fig g
pre-trained
that as we traverse
12

---

Under review as a conference paper at ICLR 2023
0.9
0.7
Accuracy
0.5
0.4
0.5
0.3
Model
Model
0.1
0.2
LVC
LVC
0.0
Shift Intersity
Shift Intersity
Figure 7: Classification accuracy under distributional shift, CIFAR- (r) CIFAR-100-C
Figure 8: Interpolating in the latent space: Ambiguous MNIST when mapped on the latent space. (1)
VC, (r) CE
13


Reviews:
[{"review_id": "m7zbQ8n0sC", "reviewer": "Reviewer_aooA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a Variational Classification (VC), which generalizes the softmax classifier, and the authors claim that it mirrors the relationship between the variational auto-encoder and the deterministic auto-encoder. The main contribution is to include the latent variable to softmax classifier and then VC objective analogous to the ELBO is designed. The experimental results show that the VC classifier outperforms the standard softmax in several ways, in particular in terms of calibration, adversarial attacks and when data is scarce. \n", "summary_of_the_review": "The paper adds a latent variable to classifier and impose constraints (class conditional priors) in the objective function.\nOverall it is a good paper, but not ready for publication.", "strength_and_weaknesses": "Quality/Clarity: the paper is well written and the techniques presented are ok to follow. Its motivation is clear, which is to generalize softmax and design a better model: (1) modeling p(y|x) (2) measure of\nsimilarity between data samples. On a technical level, I do not see how VC objective in Eq. 6 related to ELBO. The authors need to give more details derivation from Eq.5 to Eq. 6.\n\nOriginality/significance: the idea is interesting, which introduces latent variable and adds more constraints to generalize the softmax function. However, it gives readers an impression that the VC objective in Eq. 6 is a heuristic result, not from ELBO. If the authors can provide the detailed derivation to bridge the gap as well as show more experimental results, it should be an accepted paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i7OcgwWqxW5", "reviewer": "Reviewer_Lvkv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces variational classification (VC) by treating the inputs to the softmax layer of a classification model as latent variables with a mixture of Gaussians prior. To achieve this probabilistic interpretation of the softmax classifier, they derived an objective to be minimized, which is similar to the ELBO used in VAEs. The resulting VC model generalizes the softmax classifier (similar to VAE vs deterministic autoencoder) and enables incorporating class-conditional priors. A derivation of the training objective is provided with some practical design choices to optimize it. The evaluations are performed on several datasets (CIFAR-10, CIFAR-100, SVHN, CelebA, MNIST) in terms of accuracy, calibration, OOD generalization, adversarial robustness, and performance on low data regime. ", "summary_of_the_review": "While the paper makes some interesting points, I think it is still a borderline one as the performance is somewhat mixed and gains are mostly incremental. I would be happy to revise my score if the above concerns are addressed sufficiently.", "strength_and_weaknesses": "Strengths:\n\n- The generalization of the softmax classifier with a Bayesian interpretation is interesting and it can potentially have impacts on the several desired properties for classification models such as calibration and OOD generalization. \n\n- The authors provide a derivation for the VC objective with nice connections to VAE. I also liked that they provided practical implementation points of the VC using the commonly employed tools like the reparameterization trick. However, I believe it could be useful to provide an algorithm table or pseudo-code for better clarity. Could the authors also clarify if there are any instability issues faced during the adversarial optimization?\n\n- The experiments were selected to validate different aspects of the proposed VC. More on that below.\n\nWeaknesses: My main concerns are about the experimental results.\n\n- From the results, it is hard to say VC is doing notably better than GM or CE methods, except model calibration (even for that, it is sometimes worse than CE, see Fig. 4. left). Could this be due to optimization difficulties, e.g. due to adversarial objective, or are there other reasons?\n\n- For calibration, adding temperature scaling as another baseline could be helpful to gauge the improvements.\n\n- Can this method be applicable to more complex tasks like ImageNet classification and more strong adversarial attacks like PGD? Also for low data regime, are there any benefits for datasets other than MNIST where the performance is more or less saturated? What happens to calibration under low data regime?\n\n- Demonstration of the experimental results are somewhat inconsistent. For example, why is there no standard deviation in Table 1 from multiple runs while they are provided for Table 4? What do the bars and whiskers represent in Fig. 4?\n\n- Is there a substantial increase in the training time compared to standard training? An analysis on this could be useful for practitioners.\n\n- Outlining the experimental protocol, e.g. used augmentations and hyperparameters, could increase the reproducibility of the work.\n\n- I found Sec. 3.3 to be somewhat confusing. Did the authors observe whether the model learned any disentangled and semantically meaningful representations? \n\nMinor: There are several typos throughout the paper. Also, the font size in figures are too small.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GIDx7BGOdDk", "reviewer": "Reviewer_qvVd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a variational classifier (VC). Typical machine learning classifiers use sigmoid or softmax to deterministically map last layer feature vector to class label predictions. This paper revisits the MLE, MAP and Bayesian under a variational framework. This work designs a novel objective based on ELBO and adversarial/contrastive technique. Experiments show that VC has some desired properties in interpolation, prediction confidence, out-of-sample detection, etc.", "summary_of_the_review": "This paper is a more theory-oriented paper with empirical experiments support. The idea is interesting, though resembling several prior works. This paper should better shape its relations with literature and elaborate on the model details and experiments. ", "strength_and_weaknesses": "Pros:\n1. Compared to deterministic methods, probabilistic ones like variational model can also smooth the predictions and alleviate overfitting issues. That's why this method can potentially help with calibration, interpolation, OOD, etc.\n2. The paper explains the details very thoroughly, and carefully compare the difference and similarity between existing and proposed methods.\n3. The experiments explored multiple domains/tasks.\n\nCons:\n1. From a machine learning perspective, I think another view of understanding this paper is the embedding learning. Both input feature x and label y are mapped to a latent embedding z. The label learnt probabilistic space is the prior space and the input learnt probabilistic space is the posterior space. You kinda try to match the two spaces. While probabilistic method often brings smoothness, the performance loss is also concerned. In general, I feel this work still doesn't jump out of this framework and maybe suffer from the same issues?\n2. Though this type of variational learning is relatively novel, the use of VAE in classification has been quite ubiquitous. The latent variable models also learn and align latent subspaces [1, 2]. I think you could discuss these works.\n3. While this paper has experiments on multiple setups/scenarios, they look a bit thin to me. First, the major datasets are cifar-10, cifar-100, etc. In the computer vision domain, these are just too small. It's less convincing to only use these datasets. Also, you only tested WideResNet-28-10 and ResNet-50. More and larger models would give readers a more comprehensive view on the performance.\n4. You can give some derivation steps for Eq. 5 and 6, at least in Appendix. \n5. Figure 2: Variation Classier -> Variational Classifier\n\n[1] Bai, J., Kong, S. and Gomes, C.P., 2022, June. Gaussian mixture variational autoencoder with contrastive learning for multi-label classification. In ICML (pp. 1383-1398). PMLR.\n\n[2] Bai, J., Kong, S. and Gomes, C., 2021, January. Disentangled variational autoencoder based multi-label classification with covariance-aware multivariate probit model. In IJCAI (pp. 4313-4321).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject