Variational Classification: A Probabilistic Generalization of the Softmax Classifier
Authors: Shehzaad Zuzar Dhuliawala
Mrinmaya Sachan
Carl Allen
Venue: Accepted by TMLR ()

Abstract: We present a latent variable model for classification that provides a novel probabilistic interpretation of neural network softmax classifiers. We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders, that generalises the cross-entropy loss used to train classification models. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency between their anticipated distribution, required for accurate label predictions to be output, and the empirical distribution found in practice. We augment the variational objective to mitigate such inconsistency and encourage a chosen latent distribution, instead of the implicit assumption in off-the-shelf softmax classifiers. Overall, we provide new theoretical insight into the inner workings of widely-used softmax classification. Empirical evaluation on image and text classification datasets demonstrates that our proposed approach, variational classification,  maintains classification accuracy while the reshaped latent space improves other desirable properties of a classifier, such as calibration, adversarial robustness, robustness to distribution shift and sample efficiency useful in low data settings.

Published in Transactions on Machine Learning Research (12/2023)
Variational Classification: A Probabilistic Generalization of the Softmax Classifier
Shehzaad Dhuliawala
shehaoad. Sethaanddddaaoooooaaiii.........
Deportment of Computer Science, ETH Zurich, Switzerland
Mrinmaya Sachan
Department of Computer Science, ETH Zurich, Switzerland
Carl Allen
carlaalenn@at ethz.ch
Centre, ETH Zurich, Switzerlond
Abstract
We present a latent, variable model for classification that provides a novel probabilistic interpretation of neural network softmax classifiers. We derive a variational training objoctive, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders, that generalises the cross-entropy loss used train classification models. Treating inputs to the softmax layer samples of latent variable, our abstracted perspective reveals potential inconsistency between their anticipated distribution, required for accurate label predictions to be outpul, and their empirioal distribution found in practice. We augment. the variational objective to mitigate such inconsistency and encourage chosen latent distribution, instcad of the implicit assumption found a standard softmax layer. Overall, we provide new theoretical insight into the inner workings of widely-used softmax classifiers. Empirical evaluation on image and text classification datasets demonstrates that our proposed approach, variational classification maintains classification accuracy while the reshaped latent space improves other desirable properties of classifier, such calibration, adversarial robustness, robustness to distribution shift and sample efficiency useful in low data settings.
1 Introduction
Classification central task in machine learning, used to categorise objects (Klasson et al. 2019 provide medical diagnoses (Adem al., 2019 Mirbabaie al. 2021), or identify potentially supeorupp planets (Tiensuu et al. 2019). Classification also arises in other learning regimes, e.g. select actions in reinforcement learning, distinguish positive and negative samples in contrastive learning, and pertains to the attention mechanism in transformer models (Vaswani et al.) 2017). Classification is commonly tackled by training a neural network with a sigmoid or softmar output layer Each data sample x is mapped deterministically by an encoder fw (with weights w) to a real vector 2 fz), which the softmax layer maps to a distribution over class labels y yey:
exp(= T Wy + by) po(y| = Eyey exp(=T wwy + bby
(1)
Softmax classifiers have achieved impressive performance (e.g. Krizhevsky al. 2012), however they are known suffer from several issues. For example: such classifiers are trained to numerically minimise a loss function over random dataset and their resulting predictions are hard to explain; model predictions may accurately identify the correct class by their mode but less accurately reflect meaningful class distribution p(yla), known as miscalibration; predictions can vary materially and erroncously for imperceptible changes in the data (adversarial ezamples); and highly flexible neural nerwsrks are often used in order to achieve accurate predictions, which tend to require considerable labelled data to train.
!Code: con wtwpa/sdddd/vaiiattt-----aaeeeee
Review: www. www.openreviev.co net//forumIid-BNNsXGG055
2e refer throughout sollmax function since generalises sigmoid multiple classes, arguments apply

---

Published in Transactions on Machine Learning Research (12/2023)
Figure 1: Empirical distributions of imputs to the output layer (o(2))) for classifiers trained under incremental
components of the VC objective (Eqn. 7 on MNIST (ef the central Z-plane in figure 2p. (1) "MLF" objective softmax cross-entropy; (c) "MAP" objective MLE + Gaussian class priors po(zly) (in contour); (r) VC
objective = MAP + entropy of (x(v))) Colour indicates class y: Z= for visualisation purposes.
In order to better understand softmax classification and ideally mitigate some of its known shortcomings, we take latent perspective, introducing latent variable z in graphical (Markov) model y + This model can be interpreted generatively as first choosing a sample's class, or what it is (v); then parameters defining its attributes, e.g. size, colour (z); which determine the observation (x), subject to any stochasticity, e.g. noise or natural variation. Class labels can be inferred by learning to reverse the process: predicting # from x, and y from 2, integrating over all 2: po.o (y)z) (o(y)=) (2|=:) It is generally intractable to learn parameters (8, of this predictive model by maximising the log likelihood, Je,y,(z p(z, y) log Pe.o(ylaz Instead a lower bound on the log likelihood can be maximised, comparable to the evidence lower bound (ELBO) used to train a variational auto-encoder (VAE) (Kingma & Welling 2014 Rezende el. al. 2014).
We show that training softmax classifier under cross entropy loss (SCE) is, in fact, a special case of training this generalised latent classification model under the variational objective, in which the input to the softmax layer (z of Equ. 1 is treated as the latent variable, the encoder paramelerises (o(z|z)) and the sofumax layer computes (v(lz). In other words, the latent variable model and its training objective provide an interpretable generalisation of softmax classification. Probing further, the softmax layer can be interpreted pole polelp)po(y) assuming that latent variables follow exponential family
as applying Bayes' rule, po(u)z)= Σ. (vr)
class-conditional distributions po(z)y) for true class distributions to be output. Meanwhile, the distribution that latents actually follow, 90 (2lv)) = I (=())))((=))) is defined by the data distribution and the encoder. We refer to these two descriptions of p(zly) as the anticipated and empirical latent distributions, respectively, and consider their relationship. We show, both theoretically and empirically, that in practical settings these distributions can materially differ. Indeed, optimising the SCE objective may cause each empirical distribution (zly) to collapse to a point rather than fit the anticipated po(zly). This essentially overfits to the data and loses information required for estimating confidence or other potential downstream tasks, limiting the use of as a representation of x. To address the potential discrepancy between (x(y))) and po(zly), so that the softmax layer receives the distribution it expects, we minimise the Kullback-Leible (KL) divergence between them. This is non-trivial since (2(2))) can only be sampled not evaluated, hence we use the density ratio trick (Nguyen et al. 2010 Gutmann & Hyvärinen 2010), as seen elsewhere (Makhzani et al. 2015 Mescheder al. 2017). to approximate the required log probability ratios an auxiliary lask.
The resulting Variational Classification (VC) objective generalises softmax cross-entropy classification from latent perspective and fits empirical latent distributions (z)v)) to anticipated class priors pe (zly). Within this more interpretable framework, latent variables learned by a typical softmax classifier can be considered maximium likelihood (MLE) point estimates that marimise po(u)z). By comparison, the two KL components introduced variational classification, respectively lead to mazimrum a posteriori (MAP) point estimates; and a Bayesion treatment where latent variables (approximately) fit the full distribution po (z(y)) (Figure D Since Variational Classification serves to mitigate over-fitting, which naturally reduces with increased samples, VC is anticipated to offer greatest benefit in low data regimes.
We the notation Os distinguish distributions, will be made clear. 4Terms of the standard ELRO can be interpreted similarly.
2

---

Published in Transactions on Machine Learning Research (12/2023)
Through series of experiments vision and text datasets, we demonstrate that VC achieves comparable accuracy to regular softmax classification while the aligned latent distribution improves calibration, robust- ness to adversarial perturbations (specifically FGSM "white box"), generalisation under domain shift and performance in low data regimes. Although many prior works target any one of these pitfalls of softmax classification, often requiring extra hyperparameters to be tuned on held-out validation sels, VC simullaneously improves them all, without being tailored towards any or nceding further hyperparameters or validation data. Overall, the VC framework gives novel mathematical insight and interpretability to softmax classification: the encoder maps mixture of unknown data distributions p(zly) to mixture of chosen latent distributions po(=)y), which the coftmax/outpal layer "flips" by Bayes' rule. This understanding may enable principled improvement of classification and its integration with other latent variable paradigms (e.g. VABs).
2
Background
The proposed generalisation from softmax to variational classification ($3) is analogons to how deterministic auto-encoder relates to a variational aulo-encoder (VAE), as briefly summarised below.
Estimating parameters a latent variable model of the data po(x) = S. pa (x)z))o(z)) by maximising the likelihood, Jn p(x) log (z)),, is often intractable. Instead, one can maximise the evidence lower bound (ELBO):
[ p(x) log po()) = -s (((t)) (o(221)) log
log po(2)2) - log (o(s)m) po(=) + log (=(x)))
(a(2)))
> [ p(x)) (00(2 (oorr)))oogoopee (ogpo())))) - log
()(=)=) po(2)
= ELBO,
(2)
where (o(2(ar) is the approcimate posterior and the term dropped in the inequality is Kullback-Leibler (KL) g(z)
divergence, ( p())]= ((2) log p(z) 2 0. The VAE (Kingma & Welling 2014; Rezende et al. 2014
uses the ELBO as training objective with po]]))) and qa(2/z) assumed to be Gaussian parameterised by neural networks. Setting the variance of (z)xr) to zero, i.e. each (z(xr) to delta distribution, the first ("reconstruction") term of Equ. 2 equates to the training objective of deterministic auto-encoder, which the VAF can be interpreted to probabilistically generalise, allowing for uncertainty or stochasticity in (x(2/z) constrained by the second ("rogularisation") term.
Maximising the ELBO directly equates to minimising D [p(x)] [=)] + Ex,
dog
(=(==)]]] and so fits the model p=(x)) to the data distribution p(x:) and
A
(=(2)x) to the model posterior Pe (=(2x) = p=(2)=)pu(2)
po(w)
Equivalently, the modelled distributions (0(2(z) and 2=(2)) are made consistent under Bayes' rule.
3 Variational Classification
Z
Classification Latent Variable Model (LVM): Consider data x E X and labels E y as samples of random variables x, y jointly distributed p(x,y) y). Under the (Markov) generative model in Figure 2 (left),
Pelx' - hind)
X
p(x) = -f p(cl))p((al))p())
x
labels can be predicted by reversing the process,
pe(u)a) =
= p (p(vl=))o(<(o))
(4) Figure 2: Variational Classification, reversing the generative process: go(2|z) maps data z to the latent space Z, where empirical distributions (z(y) are fitted to class priors po(zly); top layer computes Po(y)z) by Bayes' rule to give class prediction p(y]æ).
A neural network (NN) softmax classifier a deter- ministic function that maps each data point x, via sequence of intermediate representations, point on
3

---

Published in Transactions on Machine Learning Research (12/2023)
the simplex A that parameterises categorical label distribution (y]zz)) Any intermediate representation (xx can be considered a sample of a latent random variable x from conditional distribution p(<[z) =
Proposition: a NN softmax classifier is a special case of Equ. 4
Proof: Define (i) the input the softmax layer latent variable z; (ii) po(=l=)= = delta distribution paramcterised by fw; the NN up to the softmax layer (the encoder); and (iii) po(y)2) by the softmax layer (as defined in RHS of Eqn.
3.1 Training& a Classification LVM
Similarly to the latent variable model for po (x) ($2) parameters of Equ. cannot in general be learned by directly maximising the likelihood. Instead we can maximise a lower bound:
[ p( v) logpo(ulz) a.y
= S u) / go( (0()))) log po (vlz, - log 20 a)
log ga(=]2) po(= (o(2]2.))
}
a.
2 ppa..w) 20(2/x) log N.
ELBOve
(5)
Here, z)) (y|z) by the Markov model, and the (freely chosen) variational posterior qs is assumed to depend only on and set equal to po(2|z) (eliminating the second term) J
The derivation of Equ. follows analogously to that of Eqn. 2 conditioned on x; alternative derivation follows from Jensen's 's inequality.
Unlike for the standard ELBO, the "dropped" KL term DRr Ig [ (2|x)] po(2|z, w)] (minimised implicitly as ELBOve is maximised) may not minimise to zero - except in the limiting case Po(y]æ, 2) = pp(()))) That is, when z is a sufficient statistic for y given x, intuitively meaning that z contains all information contained in x about y Hence, maximising ELBOve implicitly encourages to learn a sufficient statistic for l/..
Proposition: softmax cross-entropy (SCE) loss is special case of ELBOvC-
Proof: In Equ. let (i) (0(2(x) = (=)) and (ii) pa (z(p)) = h(z) exp "yy + byy, V, for constants Wy: by, arbitrary positive function h: 2 +
RR and by b/ + log po()):
[ pl.. v) f f (oolel))lgg ( p(z, /p(z,y)oog yoog po fyl (x)) (Bages) 2.9
p(z, yloog po(s=fu(ell))poot
Ey pul== (1)) (1)))
p( (Jr,y)log a.y
A Atstepp (x Twtbby = SCE. (6) Σ, Ma) (X whr
Corollary: A NN softmax classifier outputs true label distributions p(ylær) if inputs to the softmax layer, 2, follow anticipated class-conditional distributions po(2ly) of (equi-scale) exponential family form.
3.2 Anticipated vs Empirical Latent Distributions
Defining an LVM for classification (Eqn. 4 requires specifying po(ul=). In the special case of softmax classification, po(v)2) is effectively encoded by Bayes" rule assuming exponential lamily po(=lu), i.e. distributions over softmax layer imputs for class y (Eqn. 6). More generally, one can choose the parametric form of po(2]g) and compute (o(l2) by Bayes' rule in a classifier's output layer (generalising the standard softmax layor), thereby encoding the distribution latent variables are anticipated to follow for accurate label predictions p(ylæ) to be output. natural question then is: do latent variables of a classification LVM empirically follow the anticipated distributions p=()
Empirical latent distributions are not fixed, but rather defined by (o(2|)) = (=21)))(==l) i.c. by sampling go(=(xr) (parameterised by the encoder fw) given class samples ~ p(x v). Since FLBOve is optimised w.r.t.
parameters o, if optimal parameters are denoted ø², the question becomes: does qa- (2(y) po(2)))?
It can be seen that ELBOve is optimised w.r.t @ if go (2)z)=ō--2-r for 2g = arg max, Eyイ- [log pe [y]=)] (see appendix A.1 Tn practice, true label distributions p(y(ar) are unknown and we have only finite samples
We use the notation "gs" " by analogy to the VAE and to later distinguish qa(zly), derived from (=(x)), from po(z v). Prooff: from p(ala, v)p(yla) =))(x,))p((l)) and Markovianity, we see that Dxa. (2lx)] v)] = po(z|a, + sufficient statistic for yla. We assume the parametric family ds sufficiently flexible closely approximate the analytic maximiser of ELBOvC-
(s)a)) + po (tll:)))=((((,)==ppp)))
4

---

Published in Transactions on Machine Learning Research (12/2023)
from them. For continuous data domain X, e.g. images or sounds, any empirically observed is sampled twice with probability wero and so is observed once with a single label y(r). A similar situation arises (for any X) if as a property of the data- every I has only one ground truth label y(z), i.e. labels are mutually exclusive and partition the data 8 In either case, the expectation over labels simplifies and, for a given class y, Ze = arg max, Po (y))z),, meaning the optimal latent distribution is- (=/r)) is identical for all samples æ of class y Letting By denote the optimal latent variable for all z of class y, optimal class-level distributions are simply q6- - and ELBOve is maximised if all latent representations of class, and hence (=(z)y), "collapse" to the same point, irrespective of the anticipated po(2ly).
Since softmax classification is a special case, this reveals the potential for softmax classifiers to learn over- concentrated, or over-confidenl, latent distributions relative to anticipated distributions (subject to the data distribution and model flexibility). In practical terms, the softmax cross-entropy loss may be minimised when all samples of a given class are mapped (by the encoder fw) to the same latent variableereoreeennatiion, regardless of differences in the samples' probabilities or semantics, thus disregarding information that may be useful for calibration reamm lasks. We note that the Information Bottleneck Theory (Tishby et al.) 2000: Tishby & Zaslavsky 2015 Shwartz-ZZiv & Tishby 2017) assumess that such "loss of information" is beneficial, but as we sec below, it is wnnecessary for classification and may be undesirable in general.
3.2.1 Aligning the Anticipated and Empirical Latent Distributions
We have shown that the ELBOyc objective, generalisation of SCE loss, effectively involves two versions of the latent class conditional distributions, po(zly) and ga(zly), and that a mismatch between them may have undesirable consequences in terms of information loss. We therefore propose to aligu po(zly) and go(zly), or, equivalently, for (e(ylz) and (z(p) to be made consistent under Bayes' rule (analogous to pe(z|z) and (0(2(r) in the ELBO, $2 Specifically, we minimise Dxa (zlp))) W)], VyEy. Including this constraint (weighted by B>0) 0) and learning required class distribution pr (y) defines the full VC objective:
-Lvc /pla, w [ (o(2/1)) log Ey pol=ly'len (p) B
po(ely)ps (v)
{ (=(()l)) log
(o(=lw) po(=ly) + log logr+(u)}
(7)
Taken incrementally, 9s terms of Lve can be interpreted as treating the latent variable z from marimum likelihood (MLE), mazimum posteriori (MAP) and Bayesian perspective:
(i) maximising [_g0(=|x)) log po(ul=) may overfit (=(2))) = 62-2, (as above):
[MLE]
(ii) adding class priors 1 (2 y) (o changes the point estimates Zy;
[MAP]
(iii) adding entropy - (2 log do (2ly) encourages qo (2ly) to "fill out" po(y)y).
Bayesian]
Figure shows samples from empirical latent distributions (2lv) for classifiers trained under incremental terms of the VC objective. This cmpirically confirms that softmax cross-cntropy loss does not impose the anticipated latent distribution encoded in the output. layer (left). Adding class priors po(zly) changes the point at which latents of a class concentrate (centre). Adding entropy encourages class priors to be "filled out" (right), relative to previous point estimates/ diaattess/d-isstiiuttions As above, if each has a single label (e.g. MNIST), the MLE/MAP training objectives are optimised when class distributions 4o (zly) collapse to a point. We note that complete collapse is not observed in practice (Figure [ left, centre), which we conjecture is due to strong constraints on for in particular continuity and l2 regularisation and early stopping based on validation loss. Compared to the KL form of the ELBO ($2) maximising Eqn. is equivalent to minimising:
En [[(](()))))))) + E. (=)[)) (=|a,, w)]] + +E,, [Dxa. (=lp))) (1 (ooe)))] + (xx.(p())) px (8)
showing the extra constraints over the core objective of modelling p(y|z) by Po(ulæ) (underlined).
As in popular image datasets, e.g. MNIST, CIFAR, ImageNet, where samples belong to one class onother.
Subject to uniqueness of arg max, (y =), which is not guaranteed in general, but is assumed for suitable po(=ly), such the softmax case of central interest: if all x have single label y(х) (i.c. p(zlz) = =1 1g=a(s) is "onc-hot" vector), and are finitely constrained [[]æ]]= a 0), then the SCE objective (Equ. B is maximised, and softmax outputs po (ylax) (Bqu. [ increasingly approximate true p(y]a), as class parameters wh are maximally dispersed (l.e. unit vectors th tend to a regular
polytope on the unit sphere) and all representations of a class y align with the class parameter: 2x fw(x)) + yvyy(a) (unique).
5

---

Published in Transactions on Machine Learning Research (12/2023)
Algorithm Variational Classification (VC)
1: Input pe(zly), (0(2)), (x))), (z);); learning rate schedule (nz. ne V ni B
Initialise 0, T,, 10; t
while not converged do
{2i, Vi 1m ~
(sample batch from data distribution p(x,y)]
for {1 m do
6:
2, ~ gr((z/l))) 2 ~ (z(2)))) ps (w) po(yilzs) = Σ, 20(2:10)mn
[c.g. (0(2|xz) = -2 sw()))) ( w-w = f=
7:
end for
9o & 1 D m
Va po (y) (2) + po (=)lul)))
10: Ge < I m
é [log po (3/ 22)) (Ty (zi)]
[e.g. using "reparameterisation trick")
11: IN + T Σ
V n log Pm (3:)
IN < T
12:
i= llg 0 (2:)) + log(1 - -o(TT(())))
13: + 0 + Th ge.
Φ + Φ + n To.
π +- π + t Iu:
1b t- 3 + n gv.
tet+1
14: end while
3.3
Optimising the VC Objective
The VC objective (Eqn. 7 lower bound that can be maximised by gradient methods, e.g. SGD:
the first term can be calculated by sampling (z(rr) (using the "reparameterisation trick" as necessary Kingma & Welling 2014) and computing (y(l)) by Bayes' rule;
the third term is standard multinomial cross-entropy:
the second term, however, is not readily computable since go (zly) is implicit and cannot easily be evaluated, only sampled, as 2~ g~ (z|z) (parameterised by fw) for class samples x x>p(xxy)
Fortunately, we require log ratios log pe(=ly]
(=(=)g) for each class y, which can be approximated by training binary
classifiers to distinguish samples of 9s (zlg) from those of po po(z)y) This so-called density ratio trick underpins learning methods such Noise rastivievee Estimation (Gutmann & Hyvärinen 2010) and contrastive self-supervised learning (e.g. Oord et al. 2018 Chen et al.
2020) and has been used comparably to train
variants of the VAE (Makhzani et al. 2015 Mescheder et al. 2017).
Specifically, we maximise the following auriliory objective w.r.t. parameters vp of a set of binary classifiers:
Laux [. p(N) log + | po (og-
where a is the logistic sigmoid function o
Th = w. + bb and き (wy, bylvey.
It is easy to show that Equ. is optimised if (z)= log (u(=lp) (=lg)) Vy y. Hence, when all binary classifiers are trained, (2) approximates the log ratio for class y required by the VC objective (Equ. 7. Optimising the VC objective might, in principle, also require gradients of the approximated log ratios w.r.t. parameters 0 and Φ. However, the gradient w.r.t. the & found within the log ratio is always zero (Mescheder et al. 2017)) and so the gradient w.r.t. can be computed from Eqn. See Algorithm
[ for a summary.
This approach adversarial since (a) the VC objective is maximised when log ratios give minimal KL divergence, i.e. when (s(=ly) = po(y))) and latents sampled from go(zly) or po(zly) are indistinguishable; whereas (b) the auxiliary objective is maximised if the ratios marimal and the two distributions are fully discriminated. Relating
Generative Adversarial Network (GAN) [Goodfellow et al. 2014ah, the encoder fw acts generator and cach binary classifier as a discriminator. Unlike a GAN, VC requires a discriminator per class that each distinguish generated samples from a learned, rather than static, reference/ peference/noise /oosse distribution pz(z)). However, whereas a GAN discriminator distinguishes between complex distributions in the data domain, VC discriminator compares a Gaussian to an approximate Gaussian in the lower dimensional latent domain, far simpler task. The auxiliary objective does not change the complexity relative to softmax classification and can be parallelised across classes, adding marginal computational overhead per class.

---

Published in Transactions on Machine Learning Research (12/2023)
3.3.1 Optimum of the VC Objective
In $3.2 we showed that the empirical distribution (0(2))) that opitimises the ELBOve need not match the anticipated po(zly). Here, we perform similar analysis to identify gø- (2|x) that maximises the VC objective, which, by construction of the objective, is expected better match the anticipated distribution.
Letting 3 3=1 1 to simplify (see appendix A.2 for general case), the VC objective is maximised w.r.t. (0(2/x) if:
Ep(xl= [log Uo (zly)] - pllll) log [ogpo(vl=))o((ll)) +c,
(10)
for constant This is satisfied if, for each class y,,
go - pa (=lw) Pogte' uy polul="]]
pofula)
(11)
giving a unique solution if each a has single label (see $3.2 see appendix A.2 for proof). This shows that each go (=ly) fits (=(z)) scaled by a ratio of po(y]=) to its weighted average. Hence, where po (w]=) is above average, ( > Pe (zly)), and vice versa. In simple terms, (z(y)) reflects p=(x)y) but is "peakier" (fitting obscrvation in Figure []. We have thus shown empirically (Figure [ and thcorctically that the VC objoctive aligns the empirical and anticipated latent distributions. However, these distributions are not identical and we leave to future work the derivation of an objective that achieves both pa(ylz) =(())> and qs(zl)) (o<2ly)=o(=ll))) (zlv).
3.4 Summary
The latent variable model for classification (Eqn. abstracts typical softmax classifier, giving interpretability to its components:
the encoder (fw) transforms a mixture of analytically unknown class-conditional data distributions p(xly) to a mixture analytically defined latent distributions po(zly);
assuming latent variables follow the anticipated class distributions po (z(y)), the output layer applies Bayes' rule to give po(ylz) (see figure 2 and thus meaningful estimates of label distributions p(ylæ:) (by Eqn. 4
ELBÖvo generalises softmax coosseentooy, treating the input to the softmax layer as a latent variable and identifying the anticipated class-conditionals po(zly) implicitly encoded within the softmax layer. Extending this, the VC objective (fvc) encourages the empirical latent distributions (=(2)y) to fit po(2))). Sofumax cross-entropy loss is recovered from Lvv by setting (i) (z|xr) dz-fu(z); (ii) po (z(y)) to (equal-scale) exponential family distributions, e.g. equivariate Gaussians; and (iii) 8= 0. This is analogous to how a deterministic auto-encoder relates to a VAE. Thus the VC framework elucidates assumptions made implicitly in softmax classification and by generalising this special case, allows these assumptions, e.g. the choice of pe pz(z)),, to be revised on a sast/dita-peecific basis.
4 Related Work
Despite notable differences, the energy-based interpretation of softmax classification of Grathwohl et al. 2019) is perhaps most comparable to our own in taking an abstract view to improve softmax classificatinn. However, their gains, o.g. in calibration and adversarial robustness, come at significant cost to the main aim: classification accuracy. Further, the required MCMC normalisation reportedly slows and destabilises training. In contrast, we use tractable probability distributions and retain the order of complexity. Our approach is also notionally related to Bayesian Neural Networks (BNNs) or related approaches such as MC-dropout (Gal & Ghahramani] 2016), although these are Bayesian with respect to model parameters, rather than latent variables. In principle, these mightt. be combined (e.g. Murphy 2012 as an interesting future direction.
Several previous works adapt the standard ELBO, used to learn model of p(z), to conditional analog for learning p[)[=) (Tang & Salakhutdinov 2013 Sohn et al. 2015). However, such works focus on generative scenarios rather than discriminative classification, e.g. x being a face image and y|æ being the same face in a different pose determined by latent 2; or x being part of an image and y|z its completion given latent content 2. The Gaussian stochastic neural network (GSNN) model (Sohn al. (2155) is closer to our own by conditioning q(2|x, (x, y) only on 2, however the model neither generalises softmax classification nor considers class-level latent priors q(zly) as in variational classification.
7

---

Published in Transactions on Machine Learning Research (12/2023)
Variational classification subsumes number of works that add regularisation term to softmax cross-entropy loss function, which can be interpreted as prior over latent variables in the "MAP" case ($3.2. For
example, several semi-supervised learning models can be interpreted as treating the softmax outputs as latent variables and using a latent prior to guide predictions for unlabelled data (Allen et al. 2020 Closer to variational classification, several works can be interpreted as Ireating softmax impuls as latent variables with regularisation term that encourages prior beliefs, such as determánistic label predictions (i.e. all probability mass on single class), which can be encouraged by imposing a large margin between class-conditional latent
distributions (Liu et al. 2016 Wen et al. 2016 Wan et al. 2018 2022 Scott et al. 2021
Variational classification also relates to works across several learning paradigms in which Gaussian mixture prior is imposed in the latent space, e.g. for representation learning (Xie et al., 2016 Caron et al.) 2018). in auto-encoders (Song et al. 2013 Ghosh et al., 2019) and in variational auto-encoders (Jiang et al. 2016
Yang et al. 2017 Prasad al. 2020 Manduchi et al. 2021
5
Empirical Validation
Our goal to empirically demonstrate that the latent structure induced by the VC objective beneficial relative to the standard softmax classifier. variational classifier can be substituted wherever a sofumax classifier used, by making distributional choices appropriate for the data. In particular, variational classification does not set to address any one drawback of softmax classifier, rather it aims to better reverse the generative process and capture the data distribution, providing multiple benefits. We illustrate the effectiveness of a VC through variety of tasks on familiar datasets from the visual and text domains.
Specifically, we set out to validate the following hypotheses:
H1: The VC objective improves uncertainty estimation, leading to more calibrated model.
H2: The VC objective increases model robustness to changes in the data distribution.
H3: The VC objective enhances resistance to adversarial perturbations.
H4: The VC objective aids learning from fewer samples.
For fair comparison, make minimal changes to adapt standard softmax classifier to variational classifier. As described in $3.4 train with the VC objective (Eqn. 7 under the following assumptions: (5(222) is delta distribution parameterised by a neural network fo X + Z; conditinndi priors po(zly) are multi-variate Gaussians with parameters learned from the data (we use diagonal covariance for simplicity). To provide an ablation across the components of the VC objective, we compare classifiers trained to maximise three objective functions (see
CE: equivalent to standard softmax cross-entrope under the above assumptions and corresponds to the MLE form the VC objective (....1
/ u) 2.9
(L (altllo) logpo(ylz) [o(cla))ooxpo(ll)
Jen
+ log (ogp=())
GM: includes class prions and corresponds the MAP form of the VC objective (33.2.1 ii equivalent to Wan al. (2018) with just the Gaussian Prior.
This is
JaM Jee $f
S p(z.v) v) / (0(pl)) po(2)y)
VC: includes entropy of the empirical latent distributions and corresponds to the Bayesian form of the VC objective ($3.2. ii
f[ p(a,v) v) /. (2(p))) log (=lg)
Jvc Jam
a.y
8

---

Published in Transactions on Machine Learning Research (12/2023)
CIFAR-IO
CIFAR- R-000
TINY- MAA-EMAEEEET
CE
GM
VC
VMF*
CE
GM 9
VC
VMF*
CE
GM
VC
Acc. (%, 1)
WRN
96.2 0. 95.0 02 96.3 oz
80.3 + o.
79.8 + 0.2
80.3 +a
RNET
93.7 o. 93.0 a 01
93.2 o1 94.0 e
73.2 2.
74.2 0.
73.4 a.
69.94 oo
59.7 200
59.3 =
59.3 a1
ECE (%, 4)
WRN
=.
3.5
2.1
11.1
19.6
4.8
RNET
=. 4.1
3.2 0.
5.9
8.7
10.3
5.1
7.9
12.3 8.75
7.4
Table 1: Classification Accuracy and Expected Calibration Error (mean, std.dev. over 5 runs). Accuracy is comparable between VC and CE across encoder architectures and data sets, while calibration of VC notably improves. * from Scout et al. 20211 o our implementation of Wan el al. (2018)
5.1 Accuracy and Calibration
We first compare the classification accuracy and calibration of each model on three standard bench- marks (CIFAR-10, CIFAR-100, and TINY-IMAGENET), across two standard ResNet model architectures
WideRes.Net-28-1 (WRN) and ResNet-50 (RNET)) (He et al. 2016
Zagoruyko & Komodakis 2016):
Calibration is evaluated terms of the Expected Calibration Error (ECE) (see Appendix Cp. Table shows that the VC and GM models achieve comparable accuracy to softmax cross entropy (CE), but that the VC model is consistently, significantly more calibrated (H1). Unlike approaches such as Platt's scaling (Platt et al. 1999) and temperature scaling (Guo et al. 2017 post hoc calibration is performed requiring
additional data or associated hyperparameters tuning.
We also compare MC-Dropout Gal & Ghahramani 2016) for CIFAR-10 and CIFAR-100 on ResNel-50 (p = 0.2, averaging over 10 samples). As seen previonsly (Ovadia et al. 2019 although calibration improves relative to CE (3.3%, L.4,, resp.), the main goal of classification, prediction accuracy, reduces (92.7%, 70.1%).
5.2
Generalization under distribution shift
When used in real-world settings, machine learning models may encounter distribution shift relative to the training data. It can be important to know when a model's output reliable and can be trusted, requiring the model to be calibrated out-of-distribution (OOD) data and know when they do not know. To test performance under distribution shift, we use the robustness benchmarks, CIFAR-10-C, CIFAR-100-C and TINY-IMAGENEI-CC proposed by Hendrycks & Dietterich (2019), which simalate distribution shift by adding various synthetic corruptions of varying intensities to a dataset. We compare the CE model, with and without temperature scaling, to the VC model. Temperature scaling was performed as in Guo et al. (2017 with the temperature luned on an in-distribution validation set.
Both models are found to perform comparably in terms of classification accuracy (Figure according to previous results ($5.1). However, Figure 3 shows that the VC model has consistently lower calibration error as the corruption intensity increases (left to right) (H2). We note that the improvement in calibration between the CE and VC models increases as the complexity of the datasct increascs.
When deployed in the wild, natural distributional shifts may occur in the data due to subtle changes in the data generation process, e.g. a change of camera. We test resilience to natural distributional shifts on two tasks: Natural Language Inference (NI.)) and detecting whether cells are cancerous from microscopic
Model
Model
0.25
5.45
2.5
0.30
3.40
VC
3.25
5.3
0.20
3.20
0:15
9.25
5.3
0.10
0.66
0.6
3.2
3.0
3.0
5.0
Stift
Interaay
Figure3 3: Calibration under distribution shift: (1) CIFAR-1O-C, (m) CIFAR-100-C, (r)
Boxes indicate quartiles, whiskers indicate min/max, across 16 types of synthetic distribution shift.
9

---

Published in Transactions on Machine Learning Research (12/2023)
images. NLI requires verifying if hypothesis logically follows from premise. Models are trained on the
SNLI dataset (Bowman et al. 2015) and tested on the MNLI dataset (Williams et al. 2018) taken from
more diverse sources. Cancer detection uses the CAMELYON17 dataset (Bandi et al. 2018 from the WILDs
datascts (Koh et al. 2021,, where the train and eval sets contain images from different hospitals.
Table 2 shows that the VC model achieves better cali- bration under these natural distributional shifts (H2). The CAMELYON17 (CAM) dataset has relatively small number (1000) of training samples (hence wide error bars are expected), which combines distribution shift with a low data setting (H4) and shows that the VC model achieves higher (average) accuracy in this more challenging real-world setting.
Accuracy (1)
Calibration (L)
CE
VC
CE
VC
NLI CAM
71.2 0.1
71.2
7.3
3.4 + 0.2
79.2 2.8
84.5
8.4 2.
1.8 + 1.3
Table 2: Accuracy and Calibration (ECE) under dis- tributional shift (mean, std. err., 5 runs)
We also test the ability to detect OOD examples. We compute the AUROC when model is trained CIFAR-10 and evaluated on the CIFAR-10 validation set mixed (in twn) with SVHN CIFAR- and CELEBA (Goodfellow et al. 2013 Liu et al. 2015). We compare the VC and CE models using the as a means of identifying OOD samples.
probability of the predicted class arg max, Po (V/ x))
Table 3 shows that the VC model performs comparably to the CE model. We also consider p(z) as a metric to detect OOD samples and achieve comparable results, which is broadly consistent with the findings of (Grathwohl et al., 2019). Although the VC model learns to map the data to more structured latent space and, from the results above, makes more calibrated predictions for OOD data, it does not appear to be better able to distinguish OOD data than a standard softmax classifier (CE) using the metries tested (we note that "OOD" is a loosely defined term).
Model Pon (ylz))
SVHN
C-100
CelebA
0.92
0.88
0.90
Pvc(yl=)
0.93
0.86
0.89
AUROC for OOD detection. Models trained on CIFAR-10, evaluated on in and out-of-distribution samples.
Table 3:
5.3 Adversarial Robustness
cfar
102
We test model robustness to adversarially gen- crated images using the common Fast Gradi- end. Sign Melhod (FGSM) of adversarial alack (Goodfellow al.) 2014b). This "attack" is arbitrarily chosen and VC is not explicitly tai- lored towards it. Perturbations are generated as sigx (V.((x,))) U)), where f(x, v) is the model loss for data sample x and correct class y; and e is the attack magnitude. We compare all models trained on MNIST and CTFAR-10 against FGSM attacks of different magnitudes.
82
Aocursy
69
0.0
6.1
0.2
8.3
0.4
0.5
0.0
0.1
0.2
0.4
Epsilon
Eositon
Figure Prediction accuracy for increasing FGSM adver- sarial attacks (1) MNIST; (r) CIFAR-10
Results in Figure show that the VC model is consistently more (FGSM) adversarially robust relative to the standard CE model, across attack magnitudes on both datasets (H3).
5.4 Low Data Regime
In many real-world settings, datasets may have relatively few data samples and it may be prohibitive or impossible to acquire more, c.g. historic data or rare medical cases. We investigale model performance when data is scarce on the hypothesis that a prior over the latent space enables the model to better generalise from fewer samples. Models are trained on 500 samples from MNIST, 1000 samples from CIFAR-10 and 50 samples from AGNEWs.
CE
GM
VC
MNIST
93.1 1
94.4 = 0.1
94.2 1 0.2
CIFAR-10 AGNEWS
52.7 + a.s
54.2 + 0.6
56.3 # 0.6
56.3 + s.3
61.5+ 2.9
66.3 + 4.6
Table 4: Accuracy in low data regime (mean, std.err., 5 runs)
10

---

Published in Transactions on Machine Learning Research (12/2023)
Results in Table 4 show that introducing the prior (GM) improves performance in low data regime and that the additional entropy term in the VC model maintains or further improves accuracy (H4), particularly on the more complex datasets.
We further probe the relative benefit of the VC model over the CF baseline as training sample
size varies (H4) on 10 MedMNIST classifcation
8 4-
datasets (Yang et al. 2021 collection of real-world medical datasets of varying sizes.
Figure b shows the increase in classification accuracy for the VC model relative to the CE model against number of training samples (log scale). The results show clear trend that the benefit of the additional latent structure im- posed in the VC model increases exponentially the number of training samples decreases. Together with the results in Table this suggests that the VC model offers most significant benefit for small, complex datasets.
10e3
10e6
10e5
Dataset train size
Figure 5: Accuracy increase of VC vs CE on 10 MedMNIST classification datasets of varying training set size. Blue points indicate accuracy on a dataset (mean, std.err. 3 runs). Green line shows a best-fit trend across dataset size.
6 Conclusion
We present Variational Classification (VC), a latent generalisation of standard softmax classification trained under cross-entropy loss, mirroring the relationship between the variational auto-encoder and the deterministic auto-encoder ($3) We show that softmax classification is a special case of VC under specific assumptions that are effectively taken for granted when using a softmax output layer. Moreover we see that latent distributional assumptions, "hard-coded" in the softmax layer and anticipated to be followed for accurate class predictions, neither enforced theoretically objective based on the ELBO to better align the empirical latent distribution to that andicipated. A series of experiments on image and text datasets show that, with marginal computational overhead and without tuning syper-parameters other than for the original classification task, variational classification achieves comparable prediction accuracy to standard softmax classification while significantly improving calibration, adversarial robustness (specifically FGSM), robustness to distribution shift and performance in low data regimes.
satisfied empirically. We propose novel training
In terms of limitations, we intentionally focus on the oulpul layer of classifier, treating the encoder fu as "black-box". This leaves open question of how, and how well, the underlying neural network achieves its role of transforming a mixture of unknown data distributions p(xly) to mixture of specified latent distributions p(zly). We also prove that optimal empirical latent distributions go(2lg) are "peaky" approximations to the anticipoted po pz)yy), leaving open the possibility of further improvement to the VC objective.
The VC framework gives new theoretical insight into the highly familiaa softmax classifier, opening up several interesting future directions. For example, ((2[=) might be modelled by a stochastic distribution, rather than delta distribution, to reflect uncertainty in the latent variables, similarly VAE. VC may also be extended to semi-supervised learning and related to approaches that impose structure in the latent space.
7 Acknowledgements
Carl is gratefully supported by ETH AI Centre Postdoctoral Fellowships and small projects grant from the Haslerstiftung (no. 23072). Mrinmaya acknowledges support from the Swiss National Science Foundation (Project No. 197155) a Responsible Al grant by the Haslerstiftung: and an ETH Grant (ETH-19 21-1).
11

---

Published in Transactions on Machine Learning Research (12/2023)
References
Kemal Adem, Serhat Kiligarslan, and Onur Cömert. Classification and diagnosis of cervical cancer with stacked autoencoder and softmax classification. Expert Systems with Applications, 115:537-564, 2019.
Carl Allen, Ivana Balaževié, and Timothy Hospedales. A probabilistic model for discriminative and neuro- symbolic semi-supervised learning. arXiv preprint arXiv:2006. 05896, 2020.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.
IEEE Transactions on Medical Imaging, 2018.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508 05326, 2015.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In European Conference on Computer Vision, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
Partha Ghosh, Mchdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arxi 1903. 12436, 2019.
Ian J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recog- nition from street view imagery using deep convolutional neural networks. arXio preprint 1r1 2013.
lan J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014a.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint Xrv: 141.. 6572, 20148 4b.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secrotly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pottern Recognition, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. 12261, 2019.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding:
An unsupervised and generative approach to clustering, arXiv preprinl or orXiw:161.005148 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
12

---

Published in Transactions on Machine Learning Research (12/2023)
Marcus Klasson, Cheng Zhang, and Hedvig Kjellström. A hierarchical grocery store image dataset with visual and semantic labels. In IEEE Winler Conference on Applications of Computer Vision, 2019.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 2021.
Alex Krizhevsky, Πya Sutskever, and Geoffrey E Hinton. Imagenct classification with deep convolutional neural networks. In Neural Information Processing Systems, 2012.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In International Conference on Machine Learning, 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision, 2015.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprinl arXiv:1511.0644 2015.
Laura Manduchi, Kieran Chin-Cheong, Holger Michel, Sven Wellmann, and Julia Vogt. Deep conditional gaussian mixture model for constrained clustering. In Neural Information Processing Systems, 2021.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In International Conference on Machine Learning, 2017.
Milad Mirbabaic, Stefan Stieglitz, and Nicholas RJ Frick. Artificial intelligence in discase diagnostics: A critical review and classification on the current state research guiding future direction. Health and Technology, 11(4):693-77) 731, 2021.
Jishnu Mukhoti, Andreas Kirsch, Joost Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainly: A simple baseline. arviv e-prinds, pp. arXiv 2102, 2021.
Kevin P Murphy. Machine learning: probabilistic perspeclive. MIT press, 2012.
Mahdi Pakdaman Naeiui, Gregory Cooper, and Milos Hauskrecht. Oblaining well calibrated probabilities using bayesian binning. In AAAI Conference on Artificial Intelligence, 2015.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 36(11)55477-5811 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1xXi:::807.007/55 2018
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snook. Can you trust your model's uncertainty? evaluating prodictive uncertainty under dataset shift. In Neural Information Processing Systems, 2019.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-744 1999.
Vignesh Prasad, Dipanjan Das, and Brojeshwar Bhowmick. Variational clustering: Leveraging variational autoencoders for image clustering. In International Joint Conference on Neural Networks, 2020.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
Tyler R Scott, Andrew C Gallagher, and Michael C Mozer. von miscs-fished fisher loss: An exploration of embedding geometries for supervised learning. Tn International Conference Computer Vision, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2023)
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint Xiv: 1ee:1773.00000 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Neural Information. Processing Systems, 2015.
Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering. In Iberoamerican Congress on Pattern Recognition, 2013.
Charlie Tang and Russ R Salakhutdinov. Learning stochastic feedforward neural networks. In Newral Information Processing Systems, 2013.
Jacob Tiensuu, Maja Linderholm, Sofia Dreborg, and Fredrik Örn. Detecting exoplanets with machine learning: A comparative study between convolutional neural networks and support vector machines, 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In IEEE Information Theory Workshop, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physsics/000400 2000.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
Weitao Wan, Yuanyi Zhong, Tianpeng Li, and Jiansheng Chen. Rethinking feature distribution for loss functions in image classification. In Conference on Computer Vision and Pattern Recognition, 2018.
Weitao Wan, Jiansheng Chen, Cheng Yu, Tong Wu, Yuanyi Zhong, and Ming-Hsuaa Yang. Shaping deep feature space towards gaussian mixture for visual classification. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European Conference on Computer Vision, 2016.
Adina Williams, Nikila Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embodding for clustering analysis. In International Conference on Machine Learning, 2016.
Bo Yang, Xiao Fu, Nicholas Sidiropoulos, and Mingyi Hong. Towards k-means-friendly spaces: Simultancous deep learning and clustering. International Conference on Machine Learning, 2017.
Jiancheng Yang, Rui Shi, and Bingbing Ni. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In International Symposium on Biomedical Imaging, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ar Xiu preprint arXic:1605.0711 2016.
14

---

Published in Transactions on Machine Learning Research (12/2023)
A Proofs
A.1
Optimising the ELBOvc w.r.t q
Rearranging Equ.
the ELBOve is optimised by
/ E((x,)) f( (o log (o())) (og((
(og
arg max 9o (=)2)
(po)) (as()lo) Σ p(yla) logpo(ulz)
arg max (=(=)))
The integral over is a distribution, the integral is upper bounded by max, Ey p(yla) (r) log (ylz). This maximum is attained if support of go (2(2)) restricted to 2" = arg max, Ey p(yl x) log po (ylz) (which may not be unique).
sum of Ey p(y (zr) log po(z) terms. Since ga(=/a) is probability
A.2 Optimising the VC objective w.r.t. q
Setting 3 = in Equ. to simplify and adding a lagrangian term to constrain (0(2)>) to a probability distribution, we aim to hnd
E Eppooron ga(2/z)) log
arg (a(a)))
(a(s))) f n0 (2(()l)) log po(ele)
logp. (v)) + A(1 (--oootlla)))
Recalling that (z(x))) = Sa go (z(x))(<())) and using calculus of variations, we set the derivative of this functional w.r.t. (2(xx) to zero
Σ ple, vi logpo(ulz) (log (=la)
ro(slg) 1)}
Rearranging and diving through by p(x) gives
Ppll [log (=(v)) (owo +c.
where c -1++ ). Further, if each label y occurs once with each x, due to sampling or otherwise, then this simplifies to
(<(=)"")e" po(v") (2)po=))"")
which holds for all classes E V. Integrating over 2 shows e" = J po(y)2)ppozz1)) to give
(o(=ly) = pofuls)oo(elle) = po(2 Po(a(a)
J. po(ols)poteln)
We note, it is straightforward to include to show
(=(2))) (0(2)v) Eyptele) (/
15

---

Published in Transactions on Machine Learning Research (12/2023)
B Justifying the Latent Prior in Variational Classification
Choosing Gaussian class priors in Variational classification can be interpreted in two ways:
Well-specified generative model: Assume data IE X' is generated from the hierarchical model: - + x, where p(y) is categorical; p(zly) are analytically known distributions, e.g. (2; Ey); the dimensionality of is not large; and ==)(22 for an arbitrary invertible function
+ X (if X' is of higher dimension than 2, assume maps one-lo-one to a manifold X). Accordingly, p(x) is a mixture of unknown distributions. If {pe(zly)}e includes the true distribution p(zly), variational classification effectively aims to invert h and learn the parameters of the true generative model. In practice, the model paramcters and h-1 may only be identifiable up some equivalence, but by reflecting the true latent variables, the learned latent variables should be semantically meaningful.
Miss-specifted model: Assume data is generated as above, but with z having a large, potentially uncountable, dimension with complex dependencies, e.g, details of every blade of grass or strand of hair an image. general, impossible learn all such latent variables with a lower dimensional model. The latent variables of VC might learn a complex function of multiple true latent variables.
The first scenario is ideal since the model might learn disentangled, semantically meaningful features of the data. However, it requires distributions to be well-specifted and a low number of true latent variables. For natural data with many latent variables, the second case seems more plausible but choosing po(zly) to be Gaussian may nevertheless be justifiable by the Central Limit Theorem.
C Calibration Metrics
One way to measure if model is calibrated is to compute the expected difference between the confidence and expected accuracy of model.
EP( P(O = y| p) -p)
(12)
This is known as expected calibration error (ECE) (aacini et al. 2015). Practically, ECE estimated by sorting the predictions by their confidence scores, partitioning the predictions in M equally spaced bins (Bi.. ... BM) and taking the weighted average of the difference between the average accuracy and average confidence of the bins. In our experiments we use 20 equally spaced bins.
Bm |aco Bmn) confl Bm)
ECE- Σ
(13)
0OD Detection
SVNN
SWII CIFAR-10
GFAR-10
-20
-20
-50
-60
-10
Figure 6: t-SNE plots of the feature space for classifier trained on CIFAR-10. (1) Trained using CE. (r) Trained using VC. We posit that similar to CE, VC model is unable to meaningfully represent data from an entirely different distribution.
16

---

Published in Transactions on Machine Learning Research (12/2023)
E Semantics of the latent space
To try to understand the semantics captured in the latent space, we use a pre-trained MNIST model on the Ambiguous MNIST dataset Mukhoti et al. 2021 We interpolate between ambiguous 7's that are mapped close to the Gaussian clusters of classes of "1" and "2" It can be observed that traversing from the mean of the "7" Gaussian to that on the "1" class, the ambiguous 7's begin to look more like "1"s.
2
3
7 7
Figure 7: Interpolating in the latent space: Ambiguous MNIST when mapped on the latent space. (1) VC, (r) CE
F
Classification under Domain Shift
A comparison of accuracy between the VC and CE models under 16 different synthetic domain shifts, We find that VC performs comparably well as CE.
2.8
3.25
5.20
3.10
91
3.65
3M
5.50
3.0
4.2
5.0
2.0
3.0
1.5
2,0
50
Intensitt
Intenaity
Inkensity
Figure 8: Classification accuracy under distributional shift: (left) CIFAR-10-C (middle) CIFAR-100-C (right) TINY-IMAGENEI-C
17


Reviews:
[{"review_id": "sOZhvjsHdy", "reviewer": "Reviewer_zQao", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The paper studies studies the critical problem of improving the reliability of deep neural networks. \n\n* The communication of the paper, its central claims, and methods is clear and concise with high-quality expositional figures.\n\n* Though lacking in some ways, the experimental design is sufficient to corroborate the potential of the method. I find the experiments on generalization under distribution shift particularly compelling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose to enhance softmax classification done via deterministic neural networks with a variational approach to the latent space. By introducing a probabilistic model of the latent space, the authors propose to regularize the softmax classification model in order to align the empirical and expected latent distribution, the resulting objective is termed “Variational Classification.” This approach is aimed specifically at improving model calibration, reducing data complexity, increasing distributional robustness, and increasing adversarial robustness when compared with deterministic neural networks. Though I have evaluated the authors high-level formalization, I was unable to perform a thorough assessment of proofs in the Appendix, and may have missed details therein. Thus, the critical components of my review focuses primarily on two aspects of the paper (1) the related work and (2) the empirical evaluation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* [Critical] It seems straight-forward to me that the hypothetical improvements that this method seeks to achieve are also hypothetical improvements of other probabilistic approaches to machine learning, namely Bayesian neural networks (BNNs). It has been shown in various works that Bayesian neural networks are more resilient to natural data distribution shifts as well as to adversarial perturbations. Moreover their principled uncertainty serves as a key component of active learning approaches which is a tried and true method for reducing sample complexity in supervised learning. Yet, the authors do not discuss any of these works or even that this line of literature exists. I find this a bit peculiar, and would like for the authors to justify this choice. It is clear the method they propose scales much better than faithful approximate posterior inference, however presenting no such discussion feels like a significant omision.\n\n* [Non-Critical] Perhaps, asking for comparison with other probabilistic neural networks is misguided; however, I would also like to see a simple comparison with NoisyAdam or MC Dropout to understand how this method performs in comparison to other probabilistic approaches. I would like to see the empirical comparison across all of the hypotheses listed by the authors given that these are areas in which BNNs purport to give some advantages\n\n* [Non-critical] In the authors evaluation, I find it a bit strange that the GM method performs worse in terms of calibration across almost all datasets. Could the authors discuss why this might be the case? Should their method not improve the model’s calibration?\n\n*[Critical] The experiments on adversarial robustness are far too scant to be useful to any reader. It is also too rudimentary to serve as evidence for the claim of (H3). For example, the authors have provided an epsilon, but no indication of which norm/metric this epsilon is with respect to (I assume l2 given the magnitude of the values) but then again, l_\\infty is a much more common choice. Moreover, the authors do not evaluate with any adaptive attack despite this being a clear standard in the literature. I understand that perhaps FGSM is an initial indicator of improved adversarial robustness but it is hardly practically significant. The authors need to either considerably increase the rigor, detail, and experimental rational of this section of entirely remove it and remove any claim of (H3) because as it stands, it is not supported by theory or by experimental evidence. \n\n\nMinor weaknesses\n\n* Please label each of the subplots of figure 3 so that it is easily digestible", "broader_impact_concerns": "I have no specific broader impact concerns for this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KB5MByvmom", "reviewer": "Reviewer_HL7M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Interesting analysis regarding the anticipated and empirical class-conditional latent distribution\n* Extensive empirical evaluation of the method\n\nWeaknesses:\n* Motivation and certain assumptions were not clear to me (see questions below)\n* It is not shown how the proposed approach compares to the baseline in terms of the number of parameters and training time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method to train classification models. The method is motivated by the ideas from variational inference: treating the pre-softmax layer of the classification neural network as an unobserved random variable. A new VC objective is proposed to ensure that the class-conditional distribution of the aforementioned latent variable does not collapse during training. Empirical evaluation demonstrates that the proposed objective results in better adversarial robustness and improves performance in the low-data regime.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the authors to clarify the following questions:\n* What is the motivation to say that variational posterior is equal to $p_{\\theta}(z|x)$ (section 3.1.1)? This removes the KL term from the ELBO. As a result, we only have MLE objective. How will the analysis of section 3.3 change if this assumption is not made? Do we still need to align anticipated and empirical distribution in this case?\n* $q_{\\phi}$ does not seem to be used for variation distribution from section 3.3 onwards, but rather to distinguish anticipated and empirical class conditional distribution. Is my understanding correct?\n* Title of the figure 1 refers to $p_{\\theta}(z|y)$, is it the same as $q_{\\phi}(z|y)$ from equation 9?\n\nOther comments:\n* In my opinion, the introduction is too long and contains unnecessary details, which belong to section 3.\n* Introduction, paragraph 2: ‘different yet confident predictions elsewhere, which are thus uncertain’. My understanding is that confident and uncertain are two opposite things, how can predictions be both?\n* Algorithm 1 does not contain $\\beta$. \n* Figure 5 requires a more detailed explanation. What does the green line represent? How the locations of the blue points were chosen?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NvY0wogHv7", "reviewer": "Reviewer_kNko", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n[1] The proposed idea is intuitively inspiring, and the authors provide sufficient math in the main paper to clearly illustrate their idea.\n\n[2] The numerical results, except for adversarial attack, look promising.\n\nWeaknesses:\n\nOne issue is that the performance in adversarial robustness seems to have no great improvement. Adversarial attack used in this paper is a white-box attack, i.e., the attacker knows the model parameters. To overcome adversarial attack, some specific properties, e.g., how far the samples are from the decision boundary, should be addressed. Since the proposed method is tailored for clean generalization but not adversarial robustness, the current presentation in this paper is not sufficient to claim that the proposed method improves adversarial robustness. The out-of-distribution robustness is already a good contribution, so I would suggest the authors to try black-box adversarial attack, or either just drop this adversarial robustness discussion.\n\nAnother issue of this paper is the writing. The writing of this paper can be improved. In particular, some senteces are not clear, and some others have no clear connection to others.\n\n[1] Abstract \"Our approach offers a novel probabilistic perspective on the highly familiar softmax classification model, to which it relates similarly to how variational and traditional autoencoders relate\". The sentence is long and hard to capture its meaning. Please consider simplify it.\n\n[2] Abstract \"inherent inconsistency within softmax classification\" is not clear. Elaborate a little bit more on the detail.\n\n[3] Throughout the paper, some times the authors use \"autoencoder\", but some times it becomes \"auto-encoder\". Please make the word consistent.\n\n[4] Page 2, \"it is intractable to learn the parameters of this model by maximising the conditional log likelihood\". What does the word \"parameters\" refer to? Do you mean the neural network weights?\n\n[5] Page 2, \"Instead, a lower bound can be maximised, comparable to the evidence lower bound (ELBO), as used to train a variational auto-encoder (VAE)\". The sentence is a bit unclear. \n\nMy understanding towards this sentence is that, there is a lower bound term used in variational autoencoder, and the authors consider maximizing it. On the other hand, there is another bound called ELBO, and it is used in some other tasks. In such a case, what is the relationship between ELBO, and the one used in common neural network training, as described in the paragraph above this sentence? Is the commonly used method called SCE?\n\n[6] Page 2, \"standard softmax cross-entropy (SCE) objective is such a lower bound for specific choices\". What does \"such\" refer to? In addition, for \"For this correspondence,\" what is the logical relationship between the sentences before and after it?\n\n[7] Page 2, \"that two versions of the class-conditional latent distributions p(z|y) can be described\". Why do we consider two versions? Could you provide one more sentence in this paragraph to quickly mention how to use them? Besides, the writing of whole paragraph needs improvement. The sentence in the middle is short but with four commas, which break the logic of the sentence.\n\n[8] Page 2, \"can differ materially\". How to understand the word \"materially\"? Could you use a more accurate word?\n\n[9] Page 2, \"In particular, the SCE objective can be optimal if empirical class-conditional latent distributions “collapse” to distinct points rather than fit the anticipated distributions.\" My understanding is that if something is \"optimal\", then it will have good properties. But from this sentence, the \"optimality\" is not sth we expect. Do you mean the the empirical loss achieves the minimal value and overfits the training data?\n\n[10] Page 2, \"we take an adversarial approach to implicitly learn the required log probability ratios as an auxiliary task.\" Could you add one sentence here to briefly introduce what is the adversarial approach? The \"adversarial approach\" till now appears twice, but there is no explanation about it now. I would suggest to add a short description like \"similar to generative adversarial network (GAN)\".\n\n[11] Page 2, \"The VC framework ..., which the softmax (or more generally output) layer “flips” by Bayes’ rule\". Check the grammar of \"which\".\n\n[12] Page 5, \"fill out\". Please be more precise on how to understand it.\n\n[13] In the experiment of adversarial robustness, the definition of FGSM is $\\epsilon * sign(L)$. Should it be $\\epsilon*sign(grad(L))$?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a variational classifier to improve the uncertainty estimation and perturbations via a probabilistic approach. The method is similar how variational autoencoder improves the vanilla autoencoder, and numerical results show the advantage of the proposed method over the benchmark algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please consider my comments towards the adversarial robustness, and also revise the writing of this paper. Readers with statistics/probability/GAN background may understand this paper, but it can be potentially hard for other readers to comprehend because the language is not clear.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a strong paper with a new and fresh framework for classification via latent variable models. The basic idea (generalizing standard softmax classifiers to a latent variable model that can be trained with an ELBO-like approach) is simple but interesting, and leads to some nice properties, like the calibration finding the authors show.

This is solid research that is worth accepting.

All reviewers agree that the work is genuinely interesting. The main questions raised (resolved by the authors during the discussion phase) relate to the exact positioning of the work, and the reviewers were convinced.

Beyond this, there were requests to improve the writing, which the authors have done.
Decision: Accept (poster)