NorMatch: Matching Normalizing Flows with Discriminative Classifiers for Semi-Supervised Learning
Authors: Zhongying Deng
Rihuan Ke
Carola-Bibiane Schönlieb
Angelica I Aviles-Rivero
Venue: Accepted by TMLR ()

Abstract: Semi-Supervised Learning (SSL) aims to learn a model using a tiny labeled set and massive amounts of unlabeled data. To better exploit the unlabeled data the latest SSL methods use pseudo-labels predicted from \emph{a single discriminative classifier}. However, the generated pseudo-labels are inevitably linked to inherent confirmation bias and noise which greatly affects the model performance. In this work, we introduce a new framework for SSL named NorMatch. Firstly, we introduce a new uncertainty estimation scheme based on normalizing flows, as an auxiliary classifier,  to enforce highly certain pseudo-labels yielding a boost of the discriminative classifiers. Secondly, we introduce a threshold-free sample weighting strategy to exploit better both high and low confidence pseudo-labels.    Furthermore, we utilize normalizing flows to model, in an unsupervised fashion, the distribution of unlabeled data.  This modelling assumption can further improve the performance of generative classifiers via unlabeled data, and thus, implicitly contributing to training a better discriminative classifier. We demonstrate, through numerical and visual results, that NorMatch achieves state-of-the-art performance on several datasets.

Published in Transactions on Machine Learning Research (02/2024)
NorMatch: Matching Normalizing Flows with Discriminative Classifiers for Semi-Supervised Learning
Zhongying Deng
zd294 @com.ac. uk
Deportment of Applied Mathematics and Theoretionl Physics University of Cambridge
Rihuan Ke
ihhannke@bristol
School Mathemalics Research University of Bristol
Carola-Bibiane Schönlieb
cbs31 @com.ac. ac. uk
Department of Applied Mathematics and Theoretical Physics University of Cambridge
Angelica Aviles-Rivero
ai323Gcom @cw...ommmmmmmmmmmmm ak
Department of Applied Mathematics and Theoretical Physicc University of Cambridge
Reviewed OpenReview: https: openreview. net/ for/formm? oww.aaaaaapppoooooooaaaaaaaaaaaa
Abstract
copi-Supervised Learning (SSL) aims to learn a model using tiny labeled and massive amounts of unlabeled data. To better exploit. the unlabeled data the latest SSL methods pseudo-labels predicted from single discriménative classifier. However, the generated pscudo-labels are inevitably linked to inherent confirmation bias and noise which grcatly affects the model performance. In this work we introduce a new framework for SSL named NorMatch. Firstly, we introduce a new uncertainty estimation scheme based on normaliz- ing flows, as an auxiliary classifier, to enforce highly certain pseudo-labels yielding a boost of the discriminative classifiers. Secondly, we introduce threshold ffree sample weighting strategy to exploit. better both high and low confidence pseudo-labels. Furthermore, utilize normalizing flows to model, in unsupervised fashion, the distribution of unlabeled data. This modelling assumplion can further improve the performance of generative classi- fiers via unlabeled data, and thus, implicitly contributing to training better discriminative classifier. We demonstrate, through numerical and visual results, that NorMatch achieves state-of-the-art performance on several da datasets.
1 Introduction
Deep convolutional neural networks (CNNs) have achieved enormous success in various computer vision
tasks Krizhevsky et al. 2012 Simonyan & Zisserman, 2014 Szegedy et al. 2015 He al. 2016 Long et al,
2015
Chen et al.) 2017 Girshick et al. 2014 Girshick 2015). The key for such outstanding performance
is the large amount of labeled data used in supervised techniques. However, collecting a vast. amount of labeled data is time-consuming and labor-extensive. Semi-supervised Learning (SSL) has been a focus of great interest as it mitigates these drawbacks (Tarvainen Valpola, 20077 Berthelot et al., 2019b a Sohn et al. 2020 Li et al., 2021). SSL works under the assumption of learning with tiny label set and vast amount of unlabeled data, in which the majority of real-world problems unlabeled data is abundant.

---

Published in Transactions on Machine Learning Research (02/2024)
Class
Class
Class A
Class B
Class
Class
Labeled data different classes
Unlabeled data
Mean of different classes
Decision boundary
<) Two classifiers from a) and b) Uncertain region
a) Discriminative classifier only
Normalizing flow classifier
The comparison of a) discriminative classifier, b) normalizing flow classifier (NFC) (Lzmailov et al, 2020), and e) discriminative normalizing flow classifiers in predicting unlabeled data. Here, the data points in all the sub-plots are of the same set of inputs. Our goal is to predict highly certain pseudo-labels for the unlabeled samples (the red dots). Inconsistent predictions from the discriminative classifier and NFC on a sample (e.g., the left red dot) indicate that the pseudo-label is less trustworthy (e.g., the uncertain region in c))- In this case, we will downplay its importance to avoid over-confidence. In contrast, if consistent predictions (e.g,, on the right red dot) are achieved among both classifiers, the pseudo labels have higher certainty. Ideally, if the predictions are consistent under any hypothesises, i.e., any different classifiers, we can fully trust the predicted pseudo-labels.
Figure 1:
The crucial principle behind SSL is how to better handle the unlabeled set. The current SSL techniques (Sohn et al. 2020 Li et al. 2021 Zheng et al. 2022) use predicted classes from discriminative classifier as pseudo- labels, for the unlabeled data, with a threshold to filter out low-confidence predictions. The threshold is essentially used to estimate the uncertainty or confidence of the generated pseudo-labels However, existing threshold-based uncertainty cstimation strategics have some disadvantages. Firstly, manually-so threshold cannot, effectively identify noisy samples on which the discriminative classifier can be over-confident. This issue can further cause noise accumulation and the inherent confirmation bias (Tavainen & Valpola 2017 Arazo et al. 2020) in pseudo-labeling Secondly, the threshold is a sensitive mance, which means that sub-optimal threshold may lead to substantial degradation on some datasets. The intuition is that a high threshold allows for a few pseudo-labels for training while a low threshold introduces high label noise. The optimal threshold depends on the average predicted probability of the discriminative classifier, where the average probability is dependent on the datasets' statisties inchuding the class number and image quality. Finally, thresholding usually discards some low-confidence samples, but they can be hard samples which contribute to better performance.
to the perfor-
In this work, we go around the drawbacks associated with thresholding by using normalizing flows to estimate the uncertainty of pseudo-labels from a discriminative classifier. In particular, Normalizing Flow Classi- ficr (NFC) is used as an auxiliary classifier to estimate the uncertainty of pscudo-labels. The uncortainty estimation is achieved by matching the predictions of Normalizing flow classifier and the discriminative classifier, which we call NorMatch.
NorMatch uses the NFC to prevent a discriminative classifier from being over-confident noisy pscudo- labels; illustrated in Figure I This effect is because pseudo-label having consensus among diverse classifiers is usually of high quality. Diversity is achieved by using two fundamentally different but comple- mentary classifiers the NFC as generative classifier and the Softmax classifier as a discriminative one. NorMatch accepts a pseudo-label if the predicted pseudo-labels of these two classifiers are consistent. Other- wise, NorMatch downplays the importance of such predicted pseudo-label by using the minimum predicted probability of these two classificrs. We call this design Normalizing flow for Consensus-baseel Uncertainty Estimation (NCUE). NCUE is a threshold-free scheme for different datasets. Moreover, our NCUE in Nor- Match leverages wow-confidence samples for model training, which can improve the performance. Overall, our NCUE scheme can effectively tackle the aforementioned three disadvantages of threshold-based uncertainty estimation.
Furthermore, NorMatch also utilizes normalizing flow to model, in an unsupervised fashion, the distribu- tion of unlabeled data. This design is namod Normalizing flow for Unsupervised Modeling (NUM). NUM can contribute to learning a better generative classifier on the unlabeled data, thus further improving the performance of discriminative classifier implicitly. Our contributions are sunmarized follows.
2

---

Published in Transactions on Machine Learning Research (02/2024)
We propose a new SSL method named NorMatch, which utilizes normalizing flows auxiliary generative classifier to estimate pseudo-label uncertainty for the discriminative classifier.
We introduce threshold-free sample weighting scheme to exploit both high- and low-confidence pseudo-labels called NCUE. We further leverage normalizing flows to model the distribution of unlabeled data in an unsupervised manner (NUM).
We demonstrate that our NorMatch achieves better, or comparable, performance than state-of- lhe-art methods several popular SSL datasets including CIFAR-10, CIFAR-100, STI-10 and Mini-ImageNet.
2
Related Work
In this section, we first review the semi-supervised learning mothods, then introduce normalizing flow.
2.1
Semi-Supervised Learning
Semi-Supervised Learning (SSL) methods can be broadly divided into two categories. The first category adopts consistency regularization while the second one builds upon pscudo-labeling. The idea behind Consis- tency regularization is to enforce consistent outputs, for the same unlabeled sample, under different label- preserving perturbations. These perturbations can be RandAugment (Cubuk et al. 2020), Dropout (Srivas- tava et al., 2014) or adversarial transformations (Miyato et al. 2018). With multiple perturbed versions of the same sample, II-Model (Laine & Aila, 2016) minimizes the squared difference between their predictions for consistent output. Mean Teacher (Tarvainen & Valpola 2017) further enforces such consistency bet ween the predictions of model and its exponential moving averaged teacher model. FlowGMM (lzmailov et al. 2020) adopts a normalizing flow model together with Gaussian Mixture Model to enforce probabilistic consistency regularization.
Unlike FlowGMM which uses a single normalizing flow to encode the clustering principle (no discrimina- tive classifier included), our NorMatch uses it as an auxiliary classifier deal with the threshold- uncertainty estimation problem caused by single discriminative classifier. We estimate the uncertainty of pscudo-labels based the consensus of these two classifiers. Based on the uncortainty, we propose fireshold-free sample weighting scheme to assign different weights for pseudo-labels. As result, NorMatch significantly outperforms FlowGMM (see Table
Pscudo-labeling, including self-training, uses the model's predictions as pscudo-labels for the unlabeled data, with the pseudo-labels used for the model training in a supervised fashion. MixMatch (Berthelot et al., 2019b) generates 'soft' pseudo-labels using the averaged prediction of the same image with multiple strong augmentations while ReMixMatch (Berthelot al.] 2019a) uses weakly-augmentter ones obtain pseudo- labels. It further proposes distribution alignment to encourage the distribution of pseudo-labels to match that of ground-truth labels of labeled data. FixMatch (Sohn et. al. 2020 also employs weak augmentation for pseudo-label generation but it obtains the one-hot 'hard' pseudo-labels. Since 'hard' pseudo-labels may contain noise, it further introduces a threshold to filter out low-confidence thus potentially noisy samples. To improve the pseudo-labels strategy of FixMatch, CoMatch (Li et al. 2021 further imposes a smoothness constraint on the pseudo-labels by introducing an extra contrastive learning task. SemCo (Nassar et al. 2021
improves the pseudo-labels by adopting two discriminative classifiers for co-training. Some other methods seek to improve the pseudo-label by modifying the threshold. For example, Dash (Xu et al. 2021] improves FixMatch by proposing adaptive threshold which decreases during training. Adsh (Guo & Li 2022
argues that fixed threshold for all the classes is sub-optimal, so it designs adaptive thresholds for different classes to improve over FixMatch. A few works have explored how to improve pseudo-labelling through the lens of graphs, where different types of Laplacian energies have been used. The CREPE model (Aviles- Rivero et al. 2019) introduced a new energy model basod on the graph 1-Laplacian, which generates highly certain pseudo-labels. LaplaceNet (Sellars et al. 2022) uses quadratic energy along with new multi-sample augmentation scheme.
3

---

Published in Transactions on Machine Learning Research (02/2024)
Likelihood maximization
Normalizing flow classifier
Consensus?
Yes
High weight
Weakly- augmented data
Weak Aug.
CNN
Strongly-
x
Discriminative
classifier
No
Low weight
augmented data
NCUE
Stop gradient
Peudo-labels
CNN
Discriminative
Strong Aug.
Weighted cross-entropy
classifier
The overview of our NorMatch for unlabeled data. The modules with the same colors (i.e.. the CNN and discriminative classifier) share the same set parameters. NorMatch uses the shared CNN backbone extract the features of weakly- and strongly-auugmented versions of the same unlabeled sample. Then the weakly-angmented features input to the Normalizing Flow Classifier (NFC) for Unsupervised Modeling (NUM) by likelihood maximization, and to the discriminative classifier to obtain the pscudo-labels. These fealures are also input to the NFC and the discriminalive classifier to enforce Consensus Uncertainly Estimation (called NCUE). The NCUE generates weights for each sample/pseue httee/psuuoolabble which highlights the consistent predictions and downplays the disagreed ones. The weights together with pseudo-labels are then used to enforce a weighted cross-entropy, which supervises the training for the strongly-augmentte version.
Figure 2:
Our NorMatch also builds on FixMatch, but leverages the consensus of an auxiliary generative classifier and the main discriminative one to improve pseudo-labels. Importantly, NorMatch threshold-fres free,, and simpler yet. more effective than existing methods.
2.2 Normalizing Flow
Normalizing Flows DDin et al. 2014 2016 Kobyzev et al. 2020 composes some invertible and differentiable mapping functions to transform a simple distribution, e.g., standard Gaussian, to match complex one, e.g., the distribution of real data. Such mappings preserve the exact likelihood, which facilitates the probability density estimation for new data. This properly of normalizing flow can be used as a generative classifier, which cannot be achieved by other generative models, such as generative adversarial network (GAN) (Good- fellow et al. 2020) and variational auto-encoder (VAE) (Kingma & Welling 2013). As generative model, normalizing flow can model the marginal distribution of the real data by likelihood maximization. This makes it suitable for unsupervised tasks since no ground-truth labels are needed for model training.
Generative models are widely used to generate data or enforce consistency for SSL. However, scarce works use generative models, especially normalizing flow, as a classifier to help the discriminative one for uncertainty estimation which is the major challenge in pseudo-label-bate flow classifier (NFC) for uncertainty estimation. Remarkably, our NorMatch exploits NFC (Izmailov al. 2020) to weigh each pseudo-label and uses it to model the marginal distribution of unlabeled data, both contributing to better performance for SSL task.
SSL, To this end, we exploit the normalizing
3 Methodology
In this section, we detail our motivation on leveraging the Normalizing Flow Classifier (NFC) as an auxiliary classifier estimate the uncertainty of psoudo-labels predicted from the discriminative classifier. We then present how the NFC-based NorMatch works for unlabeled data. Finally, we illustrate the training and inference process of proposed method.
4

---

Published in Transactions on Machine Learning Research (02/2024)
3.1 Motivation
Our insight is that the consensus among diverse classifiers a pseudo-label can reduce the risk of confir- mation bias. Following the work of (Melville & Mooney 2003), we can define the diversity as the measure of disagreement across different classifiers. Diversity is ensured by using two fundamentally different but complementary classifiers, namely, the NFC as a generative classifier and the Softmax classifier, e.g., a fully connected layer followed by a Softmax activation function, as a discriminative one.
We choose the Normalizing Flow Classifier (NFC) as the auxiliary classifier for the following reasons. (1) Compared to other generative models like GAN or VAE, normalizing flows can evaluate the exact probability density for new test data. This means that given new test data, we can know the probability of the data following the distribution of a specific class. In practice, the class-specific distribution is modeled by the y-th component of the Gaussian Mixture Model (GMM) in (T) (as we implement the NFC as a RealNVP [Papamakarios et al., 2017) followed by a GMM prior (Izmailov et al. 2020 with the RealNVP acting as invertible and diffcrentiable mapping functions). If we further normalize such a class-spocific probability using all the probabilities of all the classes, then the output of the Normalizing Flow can be used to measure the probability of the input data following the distribution of all the different classes, which is illustrated in [ Therefore, normalizing flow can be used as generative classifier while the others cannot. (2) Compared to another fittmax-baseed discriminative classifier as an auxiliary one, the NFC can be used to model the marginal distribution of unlabeled data in an unsupervised way. This can boost the performance (as demonstrated in Table 2 More importantly, NFC, as a generative classifier, is more complementary to the main discriminative classifier than using another discriminative classifier. We then can ensure the diversity of these two classifiers.
Concretely, diversity manifests in the following ways. Firstly, the Normalizing Flow Classifier (NFC) predicts the conditional probability derived from Bayes Theorem while discriminative classifier directly learns the conditional probability distribution p(ylz), where 2 is the feature representation of an image x and is an clement of the label space. Secondly, NFC is an Euclidean distance-based classifier (Izmailov et al., 2020) while the Softmax-based discriminative classifier focuses on cosine distance. The NFC is Euclidean distance- based classifier as predicts the labels based on the following conditional probability:
(2 Ey) x E(I2= pvll Pn(l)=)= ΣR N( Ex)
(1)
where the denominator Ee=1 (2 Ex) is a normalization factor shared by all the class. C' is the class number, and N (Hy, denotes the y-th caas/component in a Gaussian Mixture Model (GMM), param- eterized by the mean Hy and covariance Ny. It shows that the probability is influenced by the Euclidean distance between sample's feature 2 and the mean || - myl12. It worth noting that to facilitate a better understanding, 1 simplisies the NFC by viewing its invertible and differentiable mapping functions f.)(.) identity matrix I, i.e., fx(-)) = I. This simplification makes the Normalizing Flow degrade to Gaussian Mixture Model (GMM), which is casier to understand. But in practice, we use a RoalNVP (Papamakar- ios et al. 2017) as the invertible and differentiable functions f-(,), so the input of 1 i.e., 2, is actually transformed to [x(2) before input to GMM. This further leads to pm (ylz) x E(I]S((z) Mt 11)))))
In contrast to a Normalizing Flow Classifier (NFC), a discriminative classifier makes predictions based on conditional probability, which reads:
pa(vlz) = exp(WT22) 2)
& WT= == | || |||||=|| cos 2),
(2)
DE ex)(
where Wy is the weight for class It is clear that pa())>) is based on the cosine similarity of WT and z. Intuilively, two cosine distance-based discriminative classifiers are less diverse than Euclidean distance- based NFC combined with a cosine iitaance-based discriminative classifier. Furthermore, the Euclidean distance of NFC calculatod between the (z) and the mcan fcature of the y-th class, My- In contrast, the Softmax classifier computes the cosine similarity between the latent feature & and the weight of the y-th component of the classifier W,, as in 2 ie.., is directly used to compute the labels rather than
5

---

Published in Transactions on Machine Learning Research (02/2024)
transformed by any invertible and differentiable functions. Since f2( in the NFC is not equal to in the Softmax classifier and the mean feature of the y-th class (i.e., Myy is different from the weight of the y-th component of the classifier (i.e,, Wy), these two classifiers are considered to be sufficiently diverse.
Lastly, diversity comes from the lens of statistical learning where generative model has a higher asymptotic error than the discriminative one. However, the generative one can reach the asympiotic error much faster. That is, our model enforces these two distinctive performance regimes as complementary this is translated to enforce higher diversity in terms of boundary between classes (discriminative) while also the distribution of individual classes.
As illustrated in Figure | if consistent predictions are achieved among these diverse classifiers, i.e., under different measurements (Euclidean and cosine), the predicted pseudo-labels have higher certainly. Otherwise, the pseudo-label is less reliable, thus its importance should be downplayed. We remark that we downplay low-confidence pscudo-labels rather than simply ignore them as current methods do (Sohn al. 2020 Li We do this because they can be hard samples, which might contribute better performance.
et al. 2021
3.2
NorMatch
The key in our NorMatch is to exploit the discriminative classifier and Normalizing flows for Consensus-based Uncertainty Estimation (NCUE), and apply the Normalizing flow for Unsupervised Modeling (NUM), as de- picted in Figure 2 NCUE estimates the uncertainty for pseudo-labels by emphasizing consistently predicted pseudo-labels and downplaying low-confidence ones that cause disagreement. NUM uses the Normalizing Flow Classifier (NFC) to model the distribution of unlabeled data by likelihood maximization. We detail these two designs next.
3.2.1 Normalizing flow for Consensus-basel Uncertainty Estimation (NCUE)
) where N₂, is the total sample number, are applied with both weak and strong augmentations to obtain two different versions of the same input. For weakly- augmented versions, we use flipping and cropping (st.il denote it as x")). For the strongly-augmentee we have A(x²), being A RandAugment (Cubuk et al.) 2020). These two versions are then input a CNN backbone to extract fcatures for classification. The feature of the weakly-augmented version is fed to the Normalizing Flow Classifier (NFC) and the discriminative one to obtain the probabilities Pv (y(x)) Pa(ul=)) as in 1 and respectively (denoted as Pn, Pa for clarity). We can then obtain the pseudo-label from the discriminative classifier as " = arg max (pa) or 0," = The latter is not one-hot version as the latest methods use it to enforce distribution alignment (Berthelot et al., 2019a Li et al. 2021h. The design choice is evaluated in the experiments.
As shown in Figure the unlabeled data D. = {=:² i=0
version,
With these probabilities Pa; NCUE estimates the uncertainty of a pseudo-label by investigating the consensus of the Normalizing Flow Classifier (NFC) and the discriminative classifier, and then adaptively sets a weight for such pseudo-label. Formally, the NCUE reads:
1, if arg max(pa) = arg max(pa), T(x)) [min(pa. Pn), if arg max(pa) o arg max(()),
(3)
being (xx"") the weight for each unlabeled sample. It means that we accept the pseudo-label if it achieves consensus among these two classifiers. Otherwise, we downplay its importance by min(ps, P.)) as it can be noise. With r(af) sample weight, and of as the pseudo-label, the loss for the unlabeled data, iee, the weighted cross-entropy in Figure 2 is given by:
1 La (fa) B.
Σ
(x))) HOO pa(olA(()))))))))
where Pa(y)A(x?)) 02) is the probability of strongly-augmented version A(x!)) predicted from the discrimina- tive classifier. 0a is the parameters of the CNN backbone and the discriminative classifier. is batch size, i 7 as in (Sohn et al. 2020), and H (y: p) is the cross-entropy.

---

Published in Transactions on Machine Learning Research (02/2024)
3.2.2 Normalizing flow for Unsupervised Modeling (NUM)
NUM models the distribution of the features 2 of unlabeled data a by likelihood maximization estimation. The likelihood for the feature of i-th unlabeled image
pa(z)) = -ma(z"ly= - =c,,
(5)
of the GMM. We then can optimize the parameters of normalizing flow 0n to maximize the joint probability of unlabeled data
where Pm (=fly = c) is obtained by leeding the feature 2 of x. to the e-th
N. Pr (D. On.) III (=)))00.)
(6)
Equivalently, we can achieve the maximization of (6) by minimizing the negative log-likelihood of pn pm (D.,.on). Therefore, we define loss function (0,) for the goal of likelihood maximization in Figure 2 formulated as:
Lu (6,, is
(6.)
(D.l0,).
Remark. We model the probability mass of the latent features p(z), rather than the original input images p(æ), using Normalizing Flow. In NUM, we imput the latent feature 27 to the Normalizing Flow Classifier (NFC) to obtain pm(= c). Then the invertible and differentiable mapping functions (implemented RealNVP (Papamakarios et al. 2017)) in the NFC can be learned to match the complex distribution of = by optimizing 77. Since we use the invertible and differentiable mapping functions, denoted as T : R" - R", to transform a simple GMM p(g) to match a more complex distribution of p(z), the NFC in our method is Normalizing Flow with the p(z) computed by
p(=) = p(g)] det (T)
(z)
PT (z)) det
where y -= (z)
(8)
Ōz
Oz
It is notable that = f where fn is defined in Section 3.1 Furthermore, we remark that the imput of Normalizing Flow not necessarily to the original images x but can also the latent features = if we regard the latent features as a complex distribution. Here, the "complex distribution" is relative concept, which means that the distribution of latent features is usually more complex than GMM. We adopt the latent features as the input of the Normalizing Flow rather than the images I for two reasons: 1) we aim to use the Normalizing Flow as generative classifier, which usually takes semantic features as its input for better performance. Thus, using Normalizing Flow to model latent features containing more semantic information than the images I, can better achieve our goal of improving classification accuracy; 2) The images x are in high dimension (e.g., 5x96x96-27,644 dimension on Mini-ImageNNt dataset) and Normalizing Flow cannot reduce their dimension (otherwise the loss of dimension dimensioon/inoormaati /information can make the Normalizing Flow NOT invertible). In this case, directly modeling p(x) in high-dimensional image space costs too much computational resources which we cannot afford.
3.3 Training and Inference Schemes
For the labeled samples {at ys Ni we adopt cross-cntrops for supervised training. Formally, the Normalizing Flow Classifier (NFC) 0, is trained on labeled set by minimizing
B
(o,) = B
Σ
(y 0w)),
(9)
where Pa (ylaa,, 0,) is the probability distribution of the sample x/. We also define L (ma)) as the supervised loss for the discriminative classifier.
Our total training loss is then formulated as:
L (0a) (0a) + (.0) (Tn))
(10)
7

---

Published in Transactions on Machine Learning Research (02/2024)
where À is hyper-paramette. Note that the gradients of ((,) and (o,))) are only back-propagated to NFC (i.e., 0n) rather than the CNN backbone because we discard the auxiliary Normalizing Flow Classifier (NFC) during inference. In this case, these two loss terms contribute to feature learning by influencing pscudo-labols. Concretely, they influence the learning of NFC, which impacts the weight т of pscudo- labels. T in can adjust the gradient of (Os) to the CNN for feature learning and contribute to better performance. We found that this design worked the best (sce Table 3).
For inference, we only use the discriminative classifier while discard the NFC.
4 Experiments
We conduct exlensive experiments on CIFAR-10, CIFAR-100, STI-10 and Mini-ImageNNl to demonstrate the effectiveness of our NorMatch.
4.1 Experimental Setting
Dataset and Protocols. (1) CIFAR-10 KKrihhvsky et al. 2009 has 10 classes, each with 5,000 images
of size 32x32 for training, and 1,000 images for testing, so there are 60,000 images in total. Following (Sohn et al. 2020 we evaluate our methods in the settings of training with 4, 25, and 400 labels per class, respectively. (2) CIFAR-100 (Krizhevsky et al.] 2009) has the same image size as CIFAR-10, but comprises 100 classes. Each class includes 500 images for training and 100 for testing. We also follow (Sohn) et al.) 2020) to report the results of our models trained on 4, 25, and 100 labels per class, respectively. (3)
STL-10 10 (Coates et al.) 2011) consists of 96x96 images of 10 classes, with 500 training and 800 test images per class. We train our model on 1000 labels, with 100 for each class, following (Sohn et al. 2020 (4)
Mini-ImageNNe is subset of TmageNet (Russakovsky et al., 2015), which includes 84x84 images from 100
classes, with 600 images per class. We adopt the training and testing split from [Iscen et al. 2019 then evaluate NorMatch in the scttings of 40 labels for each class.
Implementation Details. The Softmax classifier is implemented as fully connected layer followed by Softmax activation function. The backbone CNN (Cf. the blue block in Figure is selected as follows. We follow (Shn et al. 2020) to adopt Wide ResNet-28-2 for CIFAR-10 and Wide ResNet-28-8 (Zagoruyko & Komodakis 2016) for CIFAR-100. On STL-10 and Mini- Meai-ImageNet, we use a ResNet-18 (He et al. 2016) as the backbonn CNN, following (Li et al.] 2021 and [Nassar et al.) 2021 respectively. The other training settings for all these datasets are the same (unless otherwise specified). Specifically, we optimize the model using Stochastic Gradient Descend (SGD) with Nesterov momentum (Sutskever et al.] (003)). The initial learning rate is 0.03 and then decreases according a cosine learning decay (Loshchilov The batch size B is 64 and the total training iteration is 20 (1024 epochs with each epoch having 1024 iterations, except on Mini-ImageNet training for 600 epochs). We follow the latest works (Berthelot et al.
Hutter 2016).
2019a; Li et al. 2021 to use distribution alignment to 0," in to Df (except on STL-10 where a one-hot version is used). We use the exponential moving average of model
(4. We do not apply sharpening or one-hot
parameters to report the final performance, as most SSL methods (Berthelot et al. 2019a Sohn et al. 2020
Li eeea al.
2021) do.
For the settings specific to our NorMatch, we set the default value of À in 10 to 1e-6 for all the datasets. The NFC is RealNVP (Papamakarios et al. 2017; (with 6 coupling layers) followed by Gaussian Mixture Model (GMM) prior (Izmailov et al., 2020). As such, On in (10) includes three parts: a) weights of GMM, initialized as for each class, b) the a (init. as 0) and Σ (init. as 1) of GMM, and c) randomly initialized coupling layers of the Normalizing Flow Classifier (NFC). It. is trained with AdamW (Loshchilov & Hutter 2017) optimizer using an initial learning rate 0.001 with a cosine decay.
Our implementation is based on PyTorch Paszke et al. 2019) and our code is available at htttps:/ggitiei
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Ablation study on CIFAR-10 with 40 labels. NCUE and NUM are proposed in Section 3.2.1 and 3.2.2 respectively. The NCUE variant sets the weight to 0 when the NFC's predictions different from the discriminative classifier's, L.e, it simply discards all the low-confidence pseudo-labels. The best result is highlighted in yellow.
Methods
Accuracy
Baseline (FixMatch (Sohn al. 2020j)
87.77
Baseline + NCUE
92.78
Bascline + NCUE variant
91.55
NorMatch (Baseline + NCUE + NUM)
93.41
100
1.0
95
0.8
90
MM
hmm
Acc. high conf
80
Acc. law-canf
NCUE Correct
Acc. Threshold
75
Whi ght low conf
Threshold
Uncert. Discriminative
Uncert. NFC
160
180
200 220
240 260 280 300
Figure 3: The amount of low-uncertainty (high- confidence) pseudo-labels obtained by NCUE (the blue curve) and threshold-based FixMatch (the green curve with the Uhreshold set to 0.95 as in (Sohn al. 2020)), respectively. The red curve denotes the number of correct pscudo- labels measured by using ground-truth labels. The x-axis represents the training epoch while the y-axis denotes the percentage (%) of total samples.
180
200
220
240
260
Figure 4: Visualization of 1) the accuracy of high/low. confidence predictions (the bluc and red curves) and the weight distributions (the dark curve) of our NorMatch; 2) the accuracy of predictions from the threshold-basee based FixMatch, iee., the green curve; 3) the uncertainties of discriminative classifier and NFC (dashed gray and brown curve respectively) of our NorMatch.
4.2 Delving into NorMatch Performance
In this section, we present comprehensive analysis on CIFAR-10 with 40 labels to better understand each module in method. Our bascline model is the vanilla FixMatch with threshold and onc-hot pscudo-labels. For this analysis, only run 300 epochs to save time.
Effectiveness of NCUE. Table [ shows that the Normalizing flow for Consensus--basee Uncertainty Es- timation (NCUE, proposed in Section 3.2.1] significantly improves the FixMatch baseline by 5.01%. Note that FixMatch uses a threshold based uncertainty estimation, so the superiority of NCUE verifies that using the consensus among NFC and the discriminative classifier can be better than using threshold to estimate uncertainty for pseudo-labels.
Furthermore, to investigate whether should simply discard all the low-confidence pseudo-labels, eval- nate NCUE variant which sets the weights of low-confidence samples to 0 (rather than min(pa, Pa) as in 3 This variant (the 3rd row) decreases the performance by 1.23%. The degradation implies that simply ignoring all the low-confidence samples can be sub-optimal, as they can be hard samples and contribute to better performance.
Further analysis on NCUE. To better understand how our NCUE works, we further provide the following visualizations. Figure 3 plots the amount of low-uncertainty (or high-confidence) pseudo-labels. The

---

Published in Transactions on Machine Learning Research (02/2024)
results are obtained by running the proposed training scheme, and then using the generated pseudo-labels to compare NCUE and thresholding. We find that the amount of high-confidence pseudo-labels from NCUE (the blue curve) is very similar to that of the correct pseudo-labels. Thus, our NCUE can adaptively choose a proper amount of high-confidence pscudo-labols for training. In contrast, a fixed threshold of 0.95 (the green curve) ignores loo many samples that have correct pseudo-labels. This comparison explains why our NCUE works better than a fixed threshold, which is widely used FixMatch-based methods (Nassar et al.
2021
Hu et al., 2021).
2) Figure 1 depicts the accuracy of low-uncertainty (or sigh-confidencee pseudo-labels obtained by NCUE (the blue line) and a fixed threshold (the green line denoted as 'Acc. Threshold") respectively. We find that the pseudo-labels' accuracy of these two is similar (~93%), but NCUE has a larger absolute number of correct pseudo-labels as it has a much higher recall rate (i.e., more high-confidence pseudo-labels as in Figure 3 about 92% vs. 85%). That is, for NCUE, 50K training images with 92% high-confidence psoudo-labels, among which ~93% are accurate, totally 50K NOKXX%%xx3==222 accurate pscudo-labels. In contrast, a fixed threshold has only XXKX88%xx%%xx300 accurate pseudo-labels, thus achieving inferior performance.
3) Figure ! also provides the percentage of correct predictions within high- and low-confidence samples (the blue and red curves respectively), as well as the weight distributions of low-confidence samples (the dark line). These curves can show how the consensus re-weighting technique in NCUE works. We observe that the high-confidencee samples have high accuracy, which essential for good performance; the low-confidence samples have low accuracy and small weights. Small weights can alleviate the issue of over-confidence or confirmation bias, and meanwhile, take full use of low-confidence samples for better performance. The advantage of small weights is also verified in Table | where small weight-based NCUE achieves 92.78% outperforming the NCUE variant by 1.23% which simply ignores these low-confidence samples.
4) Finally, Figure 4 visualizes the uncertainty of the discriminative classifier and NFC (dashed gray and brown lines). We measure the uncertainty of these classifiers by using - 1-Pal; with Pall being the average predicted probabilities of a classifier for all the samples. It. can be seen that the discriminative classifier in NorMatch can be more over-confident (lower uncertainty) compared to NFC, thus may lead to confirmation bias. NFC is less over-confident (i.c., higher uncertainty) to alleviate confirmation bias.
Importance of NUM. Tn Table 1 we can also see that the Normalizing flow-based Unsupervised Modeling (NUM, proposed in Section 3.2.2] brings a performance gain over FixMatch + NCUE. With NUM, the normalizing flow is exposed to unlabelled data, in comparison to the case without NUM where the NFC is trained using only the labelled data. In particular, the NFC is based on the calculation of the conditional probability pm (y) hence having the unlabeled data for training potentially enforces better prediction of the labels.
NFC vs. an auxiliary discriminative classifier. We argue that the Normalizing Flow Classifier (NFC) is more diverse and complementary to the main discriminative classifier. Here, we replace the NFC with a discriminative classifier to justify our argument. For fair comparison, the replacement classifier is with similar parameters (also 6 layers, each layer comprising a fully-connected layer followed by ReLU and batch normalization (Loffe & Szegedy 2015)) to NFC. We show their parameters and performance in Table 2 Our NFC is better than using another discriminative classifier by about 1%, demonstrating that NFC is more complementary. Another notable observation is that the NFC is lightweight, with only 0.08M parameters and 0.08M Multiply ACcumulate operations (MACs). This shows its efficiency.
DC + NFC vs. two NFCs. To further support the argument that the discriminative classifier (DC) and the Normalizing Flow Classifier (NFC) are the better options for diversity, we also replace the main discriminative classifier with an NFC. This design choice leads to two NFCs. Note that in this case, the gradient of one of these two NFCs needs to back-propagate to the backbone CNN so that the backbone CNN can be updated. From the last row of Table 2 we observe that two NFCs cause a large performance drop of 12.60% when compared with our default setting (DC NFC). The drop is probably because the diversity of two NFCs is not as large as DC + NFC. While in NorMatch, the diversity of classifiers plays vital role.
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Evaluation on different classifier combinations. Normalizing Flow Classifier (NFC) vs. another Discriminative Classifier (DC). MACs: Multiply ACcumulate operations. The #Param and MACs denote the additional parameters and MACs that the extra classifier introduces.
Methods
#Param
MACs
Accuracy
DC + NFC (Default setting)
0.08M
0.08M
93.41
DC + Another DC
0.09M
0.09M
92.42
NFC NFC
0.08M
0.08M
80.81
Table 3: Evaluation on stopping gradient of the Normalizing Flow Classifier (NFC) to CNN backbone. The denotes the high-confidence samples that have their predicted probability >0.95.
Stop gradient
The
Accuracy
82.31
93.41
X
32.82
29.27
Necessity of stopping gradient of NFC to the backbone CNN. training, the gradients of I. (fn) and (0,) are only back-propagated to NFC (0a) rather than the CNN backbone (please also see the stop gradient symbol in Figure 21. We thus evaluate this design choice in Table 3 We can see that if we allow the gradient to be back-propagated to the CNN backbone and hence play a role in its parameter updates, the training almost fails (with poor accuracy of 29.27%). This is probably because the gradient from the NFC may harm the discriminative feature learning supervised by the main classifier. As result, the features from the backbone CNN can hardly fit these two fundamentally different classifiers simultaneously. This be inferred from the decreased amount of high-confidence samples, e.g.. from 82.31% to 32.82%. The sharp decrease is because the features are not discriminative enough to achieve high confidence, further causing poor performance.
As stated in Section 3.3 during
Pseudo-Labels or No Pseudo-Labels to train NFC on unlabeled data? We further investigate whether the performance can be improved by training the Normalizing Flow Classifier (NFC) with pscudo- labels on unlabeled data. Table | shows that psoudo-label-be based supervised training for NFC docreases the performance (the first two rows). This is probably because the pseudo-labels can contain noise, which makes the NFC less effective. In addition, since both the NFC and the main discriminative classifier are trained with the same set of pscudo-labels, they may suffer from the same set of noise, thus not complementary to each other anymore. As such, noisy pseudo-labels cannot be correctly identified based on the consensus of these two classifiers. This can further lead to confirmation bias.
More NFCs are Better Performance? is natural to ask whether one more Normalizing Flow Classifier (NFC) as an auxiliary classifier can further help. To answer this question, we conduct the experiment by introducing an extra. NFC to our NorMatch, leading to two NFCs with the same architecture. The Normalizing flow for Consensus-basel Uncertainty Estimation (NCUE) is then enforced on these two NFCs and the discriminative classifier in a similar way to equation 3 If and only if these three classifiers predict the same pseudo-label for an unlabeled sample, the weight of such sample is 1; Otherwise, the weight is the minimal probability of these three prodictions. We then show its result in the last row of Table 4 Wo
Table 4: Evaluations (1) using pseudo-labels to train Normalizing Flow Classifier (NFC) unlabeled data, and (2) one more NFC the auxiliary classifier.
Methods
Accuracy
Default setting
93.41
Use pscudo-label to train NFC
92.72
One more NFC auxiliary classifier
92.67
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 6: Classification accuracy (%) on CTFAR-10, CTFAR-100 and STT-10.
Best results are in bold.
CIFAR-10
CIFAR-100
STL-10
Methods
40 labels
250 labels
4000 labels
400 labels
2500 labels
10000 labels
1000 labels
II-Mode
45.7 3..74
58.9910.38
42.75:10.48
62.12:0.11
Mean Teacher
67 -7000
90.81=0.19
46.09+0.57
61.17=0.24
MixMatch
52.16+11.50
88.95=0.88
93.58=0.10
32.39+1.32
60.06+0.37
71.69+0.33
38.02+8.29
ReMixMatch
80.90119.44 64
94.56=10.05
95. 0.12
55.72:12.06
72.57:0.31
7..97:10.6
FlowGMM-cons
80.9
FixMatch
36.19733.37
91.93+0.65
95.74=0.05
51.15+11.75
71.71+0.11
77.40+0.12
65.38+0.42
CoMatch
1..99:t1.39
95 05.099:..33
79. 0..308
SemCo
91.88-0027
96.20+00.88
68.07+00.01
75.55++0.12 12
Dash
86.7843.77
95.44=0.13
95.92=0.06
55.21+0.96
72.82+0.21
78.03+0.14
NorMatch (Ours)
94.70.10.16
95.0610.188
95.8910.12
59.3911..39
73.41100.9
78.55.10.18
81.3810.12
observe a performance drop with one more NFC. This means that using a single NFC can already work well for uncertainty estimation because it sufficiently diverse and complementary to the main discriminative classifier. With one more NFC, i.., two NFCs, only small portion of the psendo-labels consensus among these three classifiers. As result, a large number of samples are discarded even though their pscudo-labels can be truc. Too many samples being discarded can cause a performance drop when compared with using a single NFC.
achieve the
Sensitivity of the model's performance to papper-prramter... The loss weight À for (l,) is the only hyper-parameter in our NorMatch, as in (1)). We then evaluate the scnsitivity of the classification accuracy (%) to À in Figure 5 Note that the likelihood-based loss I., (0,) can be much larger than the cross-entropy- losses in 110.
tune À from very small value, e.g., 1e-7. We can see that À s 1e-6 can improve the performance of FixMatch + NCUE (the abbreviation of "Normalizing flow for Consensus- based Uncertainty Estimation" proposed in Section 3.2.1 i.e., 92.78% obtained by X=0, as is in the 2nd row of Table [ Note that the performance of = 0 is not drawn in the log-scale plot. While a large X (>1e-6) results in the NUM (i.e., "Normalizing flow for Unsupervised Modeling" proposed Section 3.2.2) dominating the training process, which may harm the supervised discriminative feature learning and decrease the performance. Hence, we recommend properly setting À that the NL,((,)) is smaller than the supervised loss this constrains À from being very large.
Evaluation on distribution alignment. We follow the latest works (Berthelot et al. 2019a Li et al. 2021) to apply distribution alignment to D.' (neither sharpened nor one-hot version) in 14. We further evaluate this strategy in Table 5 We observe that the distribution alignment brings 0.3% improvement over the one-hot version (the first two rows). When fully trained for 1024 epochs, our NorMatch with distribution alignment further obtains 94.70%. We thus use it as our final model to compare with the state-of- the-art
methods in Section 4.3
52.83
90
91.1
Table 5: Evaluation distribution alignment (DA).
Methods
Epochs
Accuracy
86
84
84.73
NorMatch w/o DA
300
93.41
82
1E-07
1E-06
1E-05
1E-04
NorMatch w/ DA
300
93.71
NorMatch w/ DA
1024
94.70
Figure 5:
Sensitivity of the model's perfor-
mance to X.
4.3
Comparison with the State of the Art
Table reports the comparison of our NorMatch to the other taat--of-theart methods on CIFAR-10, 10, CIFAR- 100 and STL-100. We observe that NorMatch achieves the best performance on almost all the label splits,
12

---

Published in Transactions on Machine Learning Research (02/2024)
Table 7: Classification accuracy (%) on Mini-ImageNet.
Methods
4000 labels
Mean Teacher Tarvainen & Valpola
2017
27.49
Label Propagation [scen et a..
2019
29.71
PLCB (Arazo et al. 2020
43.51
MixMatch (Berthelot et al. 2019b
50.21
SimPLE (Hu ct al.
49.39
FixMatch (Sohn et al.
2020
40.27
NorMatch (Ours)
48.36
2021 and the state-of-the-art methods such as Mean Teacher (Tarvainen & Valpola, 2017). MixMatch (Berthelot et al., 2019b), and CoMatch (Li et al.
lavorably outperforming the baseline, FixMatch (Li
2021). Below we analyze the results on each dataset in more detail.
Results on CIFAR-10 and CIFAR-100. NorMatch surpasses FlowGMM Izmailov et al. 2020 which uses a single FlowGMM (without the discriminative classifier) to enforce a consistency regularization, by about 15% on CIFAR-10 in the setting of 4000 labels. The better performance demonstrates the elfectiveness of introducing the Normalizing Flow Classifier (NFC) to help estimate the uncertainty of pseudo- labels for the discriminative classifier. In addition, our NorMatch is superior to the latest methods, CoMatch (Li et al. 2021) and SemCo (Nassar al.) 2021), e.g., 1.61% over CoMatch on CIFAR-10 in the 40 labels setting and 5.34% over SemCo on 2500 labels of CIFAR-100. The superiority of NorMatch shows that a fundamentally different but complementary NFC for uncertainty estimation is better than an extra discriminative clas- sifier for co-training (SemCo) or an additional projection head for self- self-training (CoMatch). Compared to Dash (Xu et al. 2021), which employs an adaptive threshold, our NorMatch is threshold-free and with the best performance on the 40 labels CIFAR-10 and on CIFAR-100 for all label counts. This supports our argument that the threshold-free NorMatch simpler yet more effective.
NorMatch does not outperform SemCo or Dash on CIFAR-10 in the setting of 250 or 4000 labels. However, also observe that with these two label counts reach a near fully supervised performance, hence the performance is saturated as more labels are added. Notably, in very ideal setting where all images are labelled for training, our fully supervised baseline obtains an accuracy of 95.44%, which is still lower than that of SemCo or Dash. Nevertheless, NorMatch still obtains a performance comparable to SemCo or Dash in these two label counts.
Results on STI-10. NorMatch putperfoms all the other competitors by at least. 1.58%. Notably, beats the baseline method, FixMatch, by 16%. This significant improvement strongly supports the effectiveness of our NorMatch. Thanks to the NCUE and NUM (sce Section 3.2.1 and 3.2.2), NorMatch also excels CoMatch considerably. Moreover, our NorMatch is much simpler than CoMatch because no threshold needs to be tuned and no graph needs to be constructed for contrastive learning.
Results on Mini- Net We further evaluate NorMatch on the challenging Mfini-mmageNet, and the results are displayed Table 7 Our NorMatch achieves significant improvement over the classical Mean Teacher (Tarvainen Valpola, 2017) and Label Propagation (Iscen al. 2019) mothods. is also clearly better than the FixMatch baseline (Sohn et al., 2020) by 8.09%, owing to the Normalizing Flow Classifier (NFC) for better uncertainty estimation for sreudo-labels The better performance illustrates the scalability of our method on the challenge dataset. Furthermore, NorMatch is simpler than the other state-of-the-art methods, with fewer syper-parameters. especially without the sensitive threshold.
Our NorMatch is on par with PLCB (Arazo et al. 2020 and SimPLE (Hu et al. 2021 but worse than MixMatch (Berthelot et al. 2019b) probably because our bascline method, FixMatch, much worse than MixMatch, even the large improvement (8.09% over the FixMatch bascline) obtained by NorMatch cannot eliminate the huge gap MixMatch.
13

---

Published in Transactions on Machine Learning Research (02/2024)
5 Limitations and Discussions
Though effective, the NFC in our NorMatch inevitably introduces more computational cost, i.c., 0.08M parameters (see Table 2 and 0.08M Multiply ACcumulate operations (MACs). In addition, the training can fail if we do not stop the gradient of NFC to the backbone CNN, as shown in Table g Therefore, we usually need elaborate designs, e.g,, gradient stop strategy, to ensure the effectiveness of the auxiliary NFC. Furthermore, the success of our NorMatch largely relies on the baseline, FixMatch, so it can be inferior to other state-of-the-art methods when FixMatch performs poor, e.g,, on the challenging Mini-ImageNet dataset as shown in Table 7 On the other hand, as a pseudo-labeling-ben based method, our NorMatch can hardly boost the performance for a large gap when the FixMatch can already achieve satisfying results, such 250 and 4000 labels on CIFAR-10 (see Table 6). This is because the noise in pseudo-labels cannot be thoroughly climinated, which may hinder our NorMatch from achieving near-saturated classification accuracy.
Despite the above limitations, our NorMatch improves the performance of baseline FixMatch considerably in most circumstances. Notably, when FixMatch achieves comparable results to the other statc-of-tho--e methods, our NorMatch can oulperform the competitors lavorably owing to the siguificant improvement over FixMatch.
6 Conclusion
In this paper we propose a novel SSL method called NorMatch. NorMatch leverages a normalizing flow classifier (NFC) to help estimate pseudo-label uncertainty for training a discriminative classifier. This is achieved by applying a Normalizing flow for greeting a Consensus-basse based Uncertainty Estimation (NCUE) scheme. NCUE evaluates the consensus of the predictions from NFC and the discriminative classifier, then highlights these consistently predicted pseudo-labels and discounts low-confidence ones that cause disagree- ment. Moreover, NorMatch exploits Normalizing flow for Unsupervised Modeling (NUM), which models the distribution of unlabeled data for better performance. Extensive experiments on CIFAR-10, 10, CIFAR-100, STI.-10, and Mini-ImageNee demonstrate that NorMatch achieves state-of-the-art performance.
Acknowledgements
ZD, ATAR and CBS acknowledge support from the EPSRC grant EP BP/TO03553/1. ATAR acknowledges support from CMIH and CCIMI, University of Cambridge. CBS acknowledges support from the Philip Lev- erhulme Prize, the Royal Society Wolfson Fellowship, the EPSRC advanced career fellowship EP/ PPP/VV2942//// EPSRC grants EP SS/0522045/ and EP TP/T003553/L EP /100445558/1, EP T01799111 the Wellcome Innova- tor Awards 215733/Z /2/19/ and 221633/7/20/ the Europcan Union Horizon 2020 rescarch and innovation programme under the Marie Skodowska-Cure grant agreement No. 777826 NoMADS, the Cantab Capital Institute for the Mathematics of Information and the Alan Turing Institute.
References
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020.
Angelica I Aviles-Rivero, Nicolas Papadakis, Ruoteng Li, Philip Sellars, Samar M Alsaleh, Robby T Tan, and Carola-Bibiane Schönlieb. Energy models for better pseudo-labels: Improving semi-supervised classi- fication with the 1-laplacian graph energy. arXiv preprint arXiv:1906. 08635, 2019.
David Berthelot, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Sumi-super learning with distribution alignment and augmentation anchoring. arXiv preprinl ar Xix::19111.000 2019a.
14

---

Published in Transactions on Machine Learning Research (02/2024)
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmaich: A holistic approach to semi-supervised learning. Advances in neural information processing systems, 32, 2019b.
Liang-Chich Chen, George Papandreou, Tasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848-888,, 2017.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Procoodings, 2011.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-708 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-lineat independent components estimation.
arXiv preprint arXit:1410.8516 2014.
Laurent Dinh, Jascha Sohl- Dickstin and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08803 2016.
Ross Girshick. Fast r-cmn. In Proceedings of the IEEE international conference on computer vision, pp. 1440 1448, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the TEFE conference on computer vision and pattern recognition, pp. 580 587, 2014.
Ian Goodfellow, Jean Pouget-Abadic, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communicatiinss of the ACM, 63(11): 139, 2020.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive thresholding. Tn International Conference on Machine Learning, pp. 8082 8094. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Zijian Hu, Zhengyu Yang. Xucfeng Hu, and Ram Nevatia. Simple: similar pscudo label exploitation for semi- supervised classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 150099-1118,, 2021.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. pmir, 2015.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label propagation for deep semi-supervised learning. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition, pp.
2019.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with PMLR, 2020.
normalizing flows. In International Conference on Machine Learning, pp.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes arXiv preprint ar arXiv:3312.6114, 2013.
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(11):3964-390) 2020.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25:1097-1105, 2012.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv:1610. 02242, 2016.
arXiv preprint
Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: Semi-supervisod learning with contrastive graph regularization. In Proceedings of the IEEE/CVF International Conference Compuler Vision, pp. 9475- 9484, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 3431 3440, 2015.
Tya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprinl arXiv:1608.0300 2016.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Xix preprint 1rXh::111..05101 2017.
Prem Melville and Raymond J Mooney. Constructing diverse classifier ensembles using artificial training examples. In Ijcai, volume 3, pp. 505-5 510, 2003.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regular- ization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-199) 2018.
Islam Nassar, Samitha Herath, Ehsan Abbasnejad, Wray Buntine, and Gholamreza Haffari. All labels are not created equal: Enhancing semi-supervision via label grouping and co-training, In Proceedings of the IEEE/CVE /CVF Conference on Computer Vision and Pattern Recognition, pp. 7241-7250, 2021.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation.
Advances in neural information processing systems, 30, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, ZZeming Lin, Natalia Gimelshein, Luca Autiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Philip Sellars, Angelica Sviles-Rivero, and Garola-Bibiann Schönlicb. Laplacenct: A hybrid graph-cnorgy neural net work for deep semisupervised classification. IEEE Trunsactions on Neural Networks and Learning Systems, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xix: 1556, 2014.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Πya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 2014.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference machine learning, pp- 1139-1147, PMT.R, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference compuler vision and pallern recognilion, pp. 1-9 9, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in newral information processing systems, 30, 2017.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi- supervised learning with dynamic thresholding. In International Conference on Machine Learning, pp. 11525 11536. PMLR, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ar preprint rrXic:1605.0114/ 2016.
Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. Simmatch: Semi-supervised learning with similarity matching. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 14471-1448, 2022.
17


Reviews:
[{"review_id": "rMLXYGq3br", "reviewer": "Reviewer_mhzF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Two techinical parts are presented in this paper: a new psudolabels' purification method when applied to FixMatch and a unsupervised learning loss of NUM. Experiments show that these two skills help to improve the baseline method of FixMatch.\n- The presentation is easy to follow with well organization.\n\n## Weaknesses\n- But my frist question is that how Normalizing flows act in this manner. Normalizing flow is a generative model by transfering random variable to different representation spaces in order to facilitate the computing of probability mass. In my point, the utilizing the Gaussian probability model in the latent space does not mean it is a Normalizing flow model. The name of Normalizing flow classifier and the NUM are not presented properly in my view. Could you explain why they are so called?\n\n- Second, what are the befinits from other pseudolabels' enhancement methods? since there are many works for doing so, I think the authours should review them more compresensively and study different effects among different strategies of pseudolabels' purification?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method called NorMatch for semi-supervised learning based on FixMathch method. Normatch introduces a new uncertainty estimation scheme based on the Normalizing Flow Classifier as called in the paper. This scheme acts as an auxiliary classifier to enforce highly certain pseudo-labels, which improves the pseudo quality of discriminative classifiers. The NorMathch also introduces unsupervised learning by maximizing the log-likelihood of unlabeled data. The experiments show that this could further improve the accuracy in SSL tasks. Overall, the main contributions of this paper is by modeling the Gaussian probibilty model in the latent features space to enchance the uncertainty estimation and representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The Normalizing Flow should be introducted and linked to the method with rigorously formal.\n\n- When there are many methods for pseudolabels' purification, the paper should review them more comprehensively and compare them with  analytical perspectives.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5PmBQKApbT", "reviewer": "Reviewer_GUVx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\t\n1. The Normalizing Flow Classifier (NFC) could use both the labeled and unlabeled data to learn a better decision boundary.\n2. Experiments on four datasets, including the ablation studies and parameter analyses, demonstrate the effectiveness of the proposed method.\n\nWeakness:\n1. Is it possible to try another two combinations of the standard classifier and NFC? Namely, what would happen if we use \"standard classifier + standard classifier\" or \"NFC + NFC\", compared with the current \"standard classifier + NFC\" strategy?\n2. I encourage the authors to discuss the potential limitations of the proposed NFC to give the readers a more comprehensive understanding of this paradigm.\n3. Why the baselines on Mini-ImageNet are less than those on CIFAR and STL? These baselines need to be supplied.\n4. How much additional computational cost would the NFC bring?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new uncertainty estimation scheme using normalizing flows, which aims at improving the pseudo-label quality for semi-supervised learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "please see the weaknesses", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YurEWz5mWo", "reviewer": "Reviewer_Br9c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n- the proposed method sounds reasonable\n\n- the results are better than previous SSL methods\n\n- analysis like an ablation study is conducted to verify each component of the proposed method \n\n\nWeaknesses\n\n- this paper (e.g., the method section and the experiment section) could be further simplified to increase the readability", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new semi-supervised learning method named NorMatch, which leverages an additional classifier through normalizing flow to estimate the uncertainty of pseudo-labels. NorMatch further employs a diversity-promoting term through normalizing flow as a regularization term. Results on several standard semi-supervised learning datasets validate the effectiveness of NorMatch.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Generally, this paper meets the standard of TMLR with a new SSL method. \n\nAs stated before, the authors need to pay more attention to the writing to improve the readability of this paper. (There are so many concepts in the method section and the results are not well displayed in the experiment section)", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the paper “easy to follow” (Reviewer mhzF), the experiments demonstrate the effectiveness of the proposed method (Reviewer GUVx), the results are better than previous SSL methods and ablations are conducted (Reviewer Br9c). During the rebuttal, the authors have made clarification to the paper to address feedback from the reviewers, added more explanations, baselines and a discussion of limitations. In my opinion, the updated draft meets the standard for TMLR acceptance: it tackles an important problem that is of interest to the community, presents insights and a method that works better than previous SSL methods on some datasets, and as far as I can tell, is technically sound and provides convincing evidence for the claims made. 

I encourage the authors to add to the limitations section that some other methods outperform NorMatch on some datasets and perhaps any insights they have on when NorMatch is expected to have an advantage over those methods.
Decision: Accept (poster)