ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring
Authors: David Berthelot
Nicholas Carlini
Ekin D. Cubuk
Alex Kurakin
Kihyuk Sohn
Han Zhang
Colin Raffel
Venue:  (2020)
Keywords: semi-supervised learning

Abstract: We improve the recently-proposed ``MixMatch semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring.
- Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels.
- Augmentation anchoring} feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input.
To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained.

Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between 5 times and 16 times less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach 93.73% accuracy (compared to MixMatch's accuracy of 93.58% with 4000 examples) and a median accuracy of 84.92% with just four labels per class.


ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
REMIXMATCH: SEMI-SUPERVISED LEARNING WITH DISTRIBUTION ALIGNMENT AND AUGMENTATION ANCHORING
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, Colin Raffel Google Research
(cberth, ncarlini, cubuk, kurakin, zhanghan,
Kihyuk Sohn
Google Cloud AI
kihyuks@google.com com
ABSTRACT
We improve the recently-proposee "MixMatch" semi-supervised learning algo- rithm by introducing two new techniques: distribution alignment and augmenta- tion anchoring. Distribution alignment encourages the marginal distribution predictions unlabeled data be close to the marginal distribution of ground- truth labels. Augmentation anchoring feeds multiple strongly augmented versions of input into the model and encourages each output be close the predie- tion for augmend version of the same input. To produce strong aug- mentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMix- Match, significantly more data-efficient than prior work, requiring between 5x and 16x less data to reach the same accuracy. For example, CIFAR- 10 with 250 labeled examples we reach 93%73% accuracy (compared to Mix- Match's accuracy of 93.58% with 4,000 examples) and median accuracy of 84.92% with just four labels per class. We make our code and data open-source at https: htt.bttt...t//..//// com/ wow.gollooeeeeeeeeeeee
remi xmat ch.
INTRODUCTION
Semi-supervised learning (SSL) provides means of leveraging unlabeled data to improve model's performance when only limited labeled data is available. This can enable the use of large, powerful models when labeling data is expensive inconvenient. Research SSL has produced diverse collection of approaches, including consistency regularization (Sajjadi et al. 2016 Laine & Aila 2017) which encourages model to produce the same prediction when the input is perturbed and entropy minimization (Grandvalct & Bengio, 2005) which encourages the model to output high- confidence predictions. The recently proposed "MixMatch" algorithm (Berthelot al., (2019) com- bines these techniques in a unified loss function and achieves strong performance on a variety of image classification benchmarks. In this paper, we propose two improvements which can be readily integrated into MixMatch's 's framework.
First, we introduce "distribution alignment", which encourages the distribution of model's aggre- gated class predictions to match the marginal distribution of ground-truth class labels. This concept was introduced as a "fair" objective by Bridle al. (1992), where a related loss term was shown to arise from the maximization of mutual information between model inputs and outputs. After review- ing this theoretical framework, we show how distribution alignment can be straightforwardly added to MixMatch by modifying the "guessed labels" using running average of model predictions.
Second, we introduce "augmentation anchoring". which replaces the consistency regularization component of MixMatch. For each given unlabeled input, augmentation anchoring first generates weakly augmented version (e.g. using only a flip and crop) and then generates multiple strongly augmented versions. The model's prediction for the weakly-augmented input is treated as the basis

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Ground-truth labels
+ Label guess
Model predictions
Figure I: Distribution alignment. Guessed label distributions are adjusted according to the ratio of the empirical ground-truth class distribution divided by the average model predictions on un-
Figure 2:
Augmentation anchoring. We use the prediction for a weakly augmented image (green, middle) as the target for predictions strong augmentations of the same image (blue).
labeled data.
of the guessed label for of the strongly augmented versions. generate strong augmentations, introduce variant of AutoAugment (Cubuk et al. 2018) based on control theory which we dub "CTAugment". Unlike AutoAugment, CTAugment learns an augmentation policy alongside model training. making it particularly convenient in SSL settings.
We call our improved algorithm "ReMixMatch" and experimentally validate it on suite of stan- dard SSL image benchmarks. ReMixxatch achieves state-of of-the-art accuracy across all labeled data amounts, for example achieving an accuracy of 93.73% with 250 labels on CIFAR-10 compared to the previous state-of-the-art of 88.92% (and compared to 96.09% for fully-supervised classifica- tion with 50,000 labels). We also push the limited-data setting further than ever before, ultimately achieving median of 84.92% accuracy with only 40 labels (just labels per class) CIFAR-I0. To quantify the impact of our proposed improvements, we carry out an extensive ablation study to measure the impact of our improvements MixMatch. Finally, we release all of our models and code to facilitate future work on semi unni-supervised learning.
BACKGROUND
The goal of a semi-supervised learning algorithm is learn from unlabeled data in a way that improves performance on labeled data. Typical ways of achieving this include training against "guessed" labels for unlabeled data or optimizing a heuristically-motivae rely on labels. This section reviews the semi-supervised learning methods relevant to ReMixMatch. with particular focus on the components of the MixMatch algorithm upon which we base our work.
objective that does not
Consistency Regularization Many SSL methods rely on consistency regularization to enforce that the model output remains unchanged when the input is perturbed. First proposed in (Bach- man et al.) 2014).)) (Sajjadi et al., (0166) and (Laine & Aila 2017). this approach was referred to as "Regularization With Stochastic Transformations and Perturbations and the "II-Model" respec- tively. While some work perturbs adversarially (Miyato et al. 2018) or using dropout (Laine & Aila 2017 Tarvainen Valpola 2017). the common perturbation is apply domain-specific data
augmentation (Laine & Aila 2017 Sajjadi et al. 2016 Berthelot et al. [2019] Xie et al. 2019). The
loss function used to measure consistency typically either the mean- seaaned error (Laine & Aila
2017
Tarvainen & Valpola 2017; Sajjadi et al. (016)) or cross-entropy (Miyato et al. 2018 Xie
al.
2019) between the model's output for a perturbed and non- perturbed input.
Entropy Minimization Grandvalet & Bengio (2005) argues that unlabeled data should be used to ensure that classes are well-separated. This can be achieved by encouraging the model's output distribution to have low entropy (i.e., to make "high-confidence" predictions) on unlabeled data. For example, one can explicitly add loss term to minimize the entropy of the model's 's predicted class distribution on unlabeled data (Grandvalet & Bengio 2005 MMyato et al. 2018). Related to this idea "self-trainnin"" methods (McLachlan 1975] Rosenberg et al. (205)) such as Pseudo-Label (Lee) 2013) that use the predicted class on an unlabeled input as a hard target for the same input, which implicitly minimizes the entropy of the prediction.
2

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Standard Regularization Outside of the setting of SSL, it is often useful to regularize models in the over- wver-parameterized regime. This regularization can often be applied both when training on labeled and unlabeled data. For example, standard "weight decay" (Hinton & van Camp (993) where the T,2 norm of parameters is minimized is often used alongside SSL techniques. Similarly, powerful MixUp regularization (Zhang al., 2017) which trains model on linear interpolants of inputs and labels has recently been applied to SSL (Berthelot et al. 2019 Verma et al. 2019).
Other Approaches The three aforementioned categories of SSL techniques does not cover the full literature on semi-supervised learning. For example, there is significant body of research on "transductive" or graph-based supe-super learning techniques which leverage the idea that un- labeled datapoints should be assigned the label of a labeled datapoint if they are sufficiently similar
Gaammerman al. 1998; Joachims, 2003] 1999 Bengio et al. 2006; Liu et al., 2018). Since our work does not involve these (or other) approaches to SSL, we will not discuss them further. A more substantial overview of SSL methods is available (Chapelle et al. 2006).
2.1 MIXMATCH
MixMatch (Berthelot et al. 2019) unifies several of the previously mentioned SSL techniques. The algorithm works by generating "guessed labels" for each unlabeled example, and then using fully- supervised techniques to train on the original labeled data along with the guessed labels for the unlabeled data. This section reviews the necessary details of MixMatch; see (Berthelot et al. 2019) for full definition.
Let X = { Po) :b E (I....,B)) }}} be batch of labeled data and their corresponding one-hot labels representing one of L classes and let 2n be augmented versions of these labeled examples. Similarly, let 2/ = {uz b (1,..., B)} be a batch of unlabeled examples. Finally, Prodel x; be the predicted class distribution produced by the model for input x
MixMatch first produces K weakly augmented versions of each unlabeled datapoint ūs. for k E [1....,,K] Then, it generates "guessed label" qb for each us by computing the average prediction % across the K augmented versions: Tb = K Σx Pmodel 0s.x: 0). The guessed label distribution then sharpened by adjusting its temperature (i.e. raising all probabilities to a power of 1/4 and renormalizing). Finally, pairs of examples (x1, P1) (222, P2) from the combined set of labeled examples and unlabeled examples with label guesses are fed into the MixUp (Zhang et al.] 2017) algorithm to compute examples (z',p') where x' = Ax + (1 A)æz for À Beta(a, a),, and similarly for p'. Given these mixed-up examples, MixMatch performs standard fully-supervised training with minor modifications. A standard cross-entropy loss is used for labeled data, whereas the loss for unlabeled data is computed using a mean square error (i.e. the Brier score (Brier) 1950)) and is weighted with hyperparameter du The terms K (number of augmentations), T (sharpening temperature), a (MixUp Beta parameter), and Au (unlabeled loss weight) are MixMatch's 's hyper- parameters. For augmentation, shifting and flipping used for the CIFAR-10, CIFAR-100, and STL-10 datasets, and shifting alone was used for SVHN.
3 REMIXMATCH
Having introduced MixMatch, we now turn to the two improvements we propose in this paper: Distribution alignment and augmentation anchoring. For clarity, we describe how integrate them into the base MixMatch algorithm; the full algorithm for ReMixMatch is shown in algorithm
3.1 DISTRIBUTION ALIGNMENT
Our first contribution is distribution alignment, which enforces that the aggregate of predictions on unlabeled data matches the distribution of the provided labeled data. This general idea was first introduced over 25 years ago (Bridle et al. [992), but to the best of our knowledge is not used in modern SSL techniques. A schematic of distribution alignment can be seen in fig. [ After reviewing and extending the theory, describe how it can be straightforwardly included in ReMixMatch.
3

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Algorithm ReMixMatch algorithm for producing a collection of processed labeled examples and processed unlabeled examples with label guesses (cf. Berthelot et al. (2019) Algorithm 1.)
1: Input: Batch of labeled examples and their onc-hot labels X' = { (z>. b (1,..., B)}, batch of unlabeled examples 24
{un : b E (I...., B) sharpening temperature T', number of augmentations K. Beta distribution parameter a for MixUp.
2: for b 1 to B do
3:
Strong Augmen((2))))))) // Apply strong data augmentation to 2h
4: . StroongAngment((ua (us); k E (1...., K} 1/ Apply strong data augmentation times to us
5: üg WeakA (u.) // Apply weak data augmentation to us
6: Pmodet(y ūs;; 0) // Compute prediction for weak augmentation of U6
7: q Normalize(qu pl)) ((w) 11 Apply distribution alignment
8: @b Normalize (9) 7 // Apply temperature sharpening to label guess
9: end for
10: X (( pb); b E (L....,B)) B)) 11 Augmented labeled examples and their labels
11: th = ((u,,, (b): bccbeb (l...., BB) 11 First strongly augmented unlabeled example and guessed label
12: ú ((wo,h bE (1...., B), (1,...,K) KY) /1 All strongly augmented unlabeled examples /1 Add weakly augmented unlabeled examples // Combine and shuffle labeled and unlabeled data
13: ú ú U (Lus, 2:); (I,...,B)) B))
14: W Caunll(Concaa(2///
15: X" (M,, Wi)ic ((,...,,&11)) 11 Apply MixUp labeled data and entries from W
4' (MixUp(U,, W WHIRD)) (1,...,, (K1) // Apply MixUp Up to unlabeled data and the rest of W
17: return 24,4'thh
3.1.1 INPUT-OUTPUT MUTUAL INFORMATION
As previously mentioned, the primary goal of an SSL algorithm is to incorporate unlabeled data in a way which improves a model's performance. One way to formalize this intuition, first proposed by Bridle al. (9922). is to maximize the mutual information between the model's 's input and output for unlabeled data. Intuitively, good classifier's prediction should depend as much as possible on formalize this objective
the input. Following the analysis from Bridle al. (1992),
/ʃ ply, x)) log x)y,x) dy dæ p(y)((x)
I(y;,) =
[E, (3/=:; 0))) E,, (x) 0))]
where H(-)) refers the entropy. See Appendix A for a proof. To interpret this result, observe that the second term in eq. (2) is the familiar entropy minimization objective (Grandvalet & Bengio 2005), which simply encourages each individual model output to have low entropy (suggesting high confidence in class label). The first term, however, is not widely used in modern SSL techniques. This term (roughly speaking) encourages that on average, across the entire training set, the model predicts each class with equal frequency. Bridle et al. (1922) refer to this as the model being "fair",
3.1.2 DISTRIBUTION ALIGNMENT IN REMIXMATCH
MixMatch already includes a form of entropy minimization via the "sharpening" operation which makes the guessed labels (synthetic targets) for unlabeled data have lower entropy. We are therefore interested in also incorporating a form of "fairness" in ReMixMatch. However, note that the objec- [Pmodee (y/2;; 8)]) on its own essentially implies that the model should predict each class with equal frequency. This is not necessarily useful objective if the dataset's 's marginal class distri- bution p(y) is not uniform. Furthermore, while it would principle be possible to directly minimize bet-hatch basis, we instead interested in integrating it into MixMatch in way which does not introduce an additional loss term or any sensitive hyperparameters.
tive H(EE ([EE/Ppooeel
this objective
To address these issues, we incorporate form of faimess we call "distribution alignment" which proceeds as follows: over the course of training, we maintain a running average of the model's predictions unlabeled data, which refer to p(y). Given the model's prediction g Pmodee (2/ f)) on an unlabeled example u, we scale q by the ratio p(u)/p(y) and then renormal- ize the result to form valid probability distribution: q Normalize(g ply)/((v) where
4

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Normalize E, We then sharpening and other processing. In practice, compute p(y) the moving average of the model's predictions on unlabeled examples over the last 128 batches. We also estimate the marginal class distribution ply) based on the labeled examples seen during training. Note that better estimate for p(y) could be used if it is known a priori; in this work we do not explore this direction further.
the label guess for u, and proceed usual with
3.2 IMPROVED CONSISTENCY REGULARIZATION
Consistency regularization underlies most SSL methods (Miyato et al. 20188; Tarvainen & Valpola 2017 Berthelot et al. 2019:) Xie et al. 2019). For image classification tasks, consistency is typically enforced between two augmented versions of the same unlabeled image. In order enforce form of consistency regularization, MixMatch generates K (in practice, K = 2) augmentations of each unlabeled example and averages them together to produce "guessed label" for u.
Recent work (Xie et al. 2019) found that applying stronger forms of augmentation can significantly improve the performance of consistency regularization. In particular, for image classification tasks it was shown that using variants of AutoAuument (Cubuk et al. (2018) produced substantial gains. Since MixMatch uses a simple flip-and-crop augmentation strategy, we were interested to see replacing the weak augmentation in MixMatch with AutoAugment would improve performance but found that training would not converge. To circumvent this issue, we propose a new method for consistency regularization in MixMatch called "Augmentation Anchoring". The basic idea is to use the model's 's prediction for a weakly augmented unlabeled image as the guessed label for many strongly augmented versions of the same image.
further logistical concern with using AutoAugment is that it uses reinforcement learning to learn policy which requires many trials of supervised model training. This poses issues in the SSL setting where we often have limited labeled data. To address this, we propose a variant of Au- toAuument called "CTAugment" which adapts itself online using ideas from control theory without requiring any form of reinforcement learning-bast based training. We describe Augmentation Anchoring and CTAugment in the following two subsections.
3.2.1 AUGMENTATION ANCHORING
We hypothesize the reason MixMatch with Autmnntt is unstable is that MixMatch averages the prediction across K augmentations. Stronger augmentation can result in disparate predictions, so their average may not be meaningful target. Instead, given an unlabeled input we first generate an "anchor" by applying weak augmentation to it. Then, we generate K strongly-augmented versions of the same unlabeled input using CTAugment (described below). We use the guessed label (after applying distribution alignment and sharpening) as the target for all of the K versions of the image. This process is visualized in fig. 2
While experimenting with Augmentation Anchoring, we found it enabled us to replace MixMatch's 's unlabeled-dan mean squared error loss with standard cross-entropy loss. This maintained stability while also simplifying the implementation. While MixMatch achieved its best performance at only K 2, we found that augmentation anchoring benefited from a larger value of K 8. We compare different values of K in section to measure the gain achieved from additional augmentations.
3.2.2 CONTROL THEORY AUGMENT
AutoAugment (Cubuk et al.) 2018) a method for learning a data augmentation policy which results in high validation set accuracy. An augmentation policy consists sequence of transformation- parameter magnitude tuples to apply to each image. Critically, the AutoAuument policy is learned with supervision: the magnitudes and sequence of transformations are determined via training many models on a proxy task which e.g. involves the use of 4,000 labels on CIFAR-IO and 1,000 labels on SVHN (Cubuk et al. 2018). This makes applying Automng methodologically problematic for low-label SSL. To remedy this necessity for training policy on labeled data, Rand/uggeet (Cubuk) et al. 2019) uniformly randomly samples transformations, but requires tuning the wyper-parammter for the random sampling on the validation set, which again is methodologically difficult when only very few (e.g., 40 or 250) labeled examples are available.
5

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Thus, in this work, we develop CTAugment, alternative approach to designing peeef-reformanee augmentation strategies. Like RandAugment, CTAugment also uniformly randomly samples trans- formations apply but dynamically infers magnitudes for each transformation during the training process. Since CTAugment does not need to be optimized on a supervised proxy task and has no sensitive hyperparameters,
directly include it in semi-supervised models to experiment with more aggressive data augmentation in remi-supervised learning. Intuitively, for each augmen- tation parameter, CTAugment learns the likelihood that will produce an image which classified as the correct label. Using these likelihoods, CTAAugment then only samples augmentations that fall within the network tolerance. This process is related what is called density matthiing in Fast AutoAugment (Lim et al. 2019), where policies are optimized so that the density of augmented validation images match the density of images from the training set.
First, CTAugment divides each parameter for each transformation into bins of distortion magnitude as is done in AutoAugment (see Appendix for a list of the bin ranges). Let m be the vector of bin weights for some distortion parameter for some transformation. At the beginning of training, all magnitude bins are initialized to have weight set to These weights are used to determine which magnitude bin to apply to given image.
At each training step, for each image two transformations are sampled uniformly at random. To augment images for training, for each parameter of these transformations we produce modified set of bin weights v.. where m, = m, if m; > 0.8 and r. = 0 otherwise, and sample magnitude bins from Categorical (Normalize((r))... To update the weights of the sampled transformations, we first sample a magnitude bin m, for each transformation parameter uniformly at random. The resulting transformations are applied version L. Then, we measure the extent to which the model's prediction matches the label as w
labeled example æ with label p to obtain an augmented
1 2L Σ |mmodel (y) [ 8) pl. The weight for each sampled magnitude bin updated as m, pm, + [I - -pl where p = 0.99 is fixed exponential decay hyperparameter.
3.3 PUTTING IT ALL TOGETHER
ReMixMatch's algorithm for processing a batch of labeled and unlabeled examples is shown in al-
gorithm | The main purpose of this algorithm is to produce the collections X' and 4'. consisting of augmented labeled and unlabeled examples with MixUp applied. The labels and label guesses in X" and U' are fed into standard cross-entropy loss terms against the model's predictions. Algorithm [ also outputs Uh, which consists of single
version of each unlabeled image and its label guesses without MixUp applied. ún is used in two additional loss terms which provide mild boost in performance in addition to improved stability:
Pre-mixup unlabeled loss a separate cross-entropy loss term.
We feed the guessed labels and predictions for example in ii as-is into
Rotation loss Recent result have shown that applying ideas from self-supervised learning to SSL can produce strong performance (Gidaris et al., 2018 Zhai et al. 2019). We integrate this idea by rotating each image u E ú4 as Rotate(u,r) r) where we sample the rotation angle r uniformly from {0, 90, 180, 270} and then ask the model predict the rotation amount as a four-class classification problem.
In total, the ReMixMatch loss is
Σ x,PEX"
H(p, (y):de + du
Σ
H(g. Pmodel ll
(3)
a.geUt
+*a
Σ
Pmode + A
Σ
Pmodel (r) Rotate(u,r):0)
(4)
u.geth
weth
Hyperparameters ReMixMatch introduce two new hyperparameters: the weight on the rotation loss Xy and the weight on the un-augmented example X2A, In practice both are fixed X,- Xi 0.5. ReMixMatch also shares many hyperparameters from MixMatch: the weight for the unlabeled loss des the sharpening temperature T. the MixUp Beta parameter, and the number of augmentations K. All experiments (unless otherwise stated) use 7 0.5, Beta 0.75, and X 1.5. We found

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
using a larger number of augmentations monotonically increases accuracy. and set K = 8 for all experiments (as rumning with K augmentations increases computation by factor of K).
We train our models using Adam (Kingma& & Ba 2015) with fixed learning rate of 0.002 and weight decay (Zhang et al. 2018) with a fixed value of 0.02. We take the final model as an exponential moving average the trained model weights with decay of 0.999.
4 EXPERIMENTS
We now test the efficacy of ReMix.atch on a set standard semi-supervised learning benchmarks. Unless otherwise noted, all of the experiments performed this section use the same codebase and model architecture (a Wide ResNet-28-2 (Zagoruyko & Komodakis 2016) with 1.5 million parameters, as used Oliver et 2018
4.1 REALISTIC SSL SETTING
We follow the Realistic Semi- Sumi-Supervised Learning (Oliver et al. 2018) recommendations for per- forming SSL evaluations. In particular, as mentioned above, this means we use the same model and training algorithm in the same codebase for all experiments. We compare against VAT (Miyato jet al. 2018) and MeanTeacher (Tarvainen Valpola 2017), copying the re-implementations over from the MixMatch codebase (Berthelot et al. 2019).
Fully supervised baseline To begin, we train a fully-supervised baseline to measure the highest accuracy we could hope to obtain with our training pipeline. The experiments we perform use the same model and training algorithm, so these baselines are valid for all discussed SSL techniques. On CIFAR-10, 10, obtain fully-supervised error rate of 4.25% using weak flip crop augmentation, which drops to 3.62% using AutoAugment and 3.91% using CTAugment. Similarly, on SVHN we obtain 2. 2.70% error using weak (flip) augmentation and 2.31% and 2.16% using AutoAugment and CTAugment respectively. While AutoAugment performs slightly better on CIFAR-10 10 and slightly on SVHN compared to CTAugment, it is our intent to design better augmentation strategy; just one that can be used without a pre-training or tuning of hyper-parameters.
CIFAR-10 Our results CIFAR-10 shown in table left. ReMixMatch sets the new state- of-the-art for all numbers of labeled examples. Most importantly, ReMixMatch 16x more data efficient than MixMatch (e.g., at 250 labeled examples ReMcx has identical accuracy com- pared to MixMatch at 4,000).
SVHN Results for SVHN are shown in table right. ReMixMatch reaches state-of-the-art 250 labeled examples, and within the margin of error for state-of-the-art otherwise.
CIFAR-10
SVHN
Method
250 labels
1000 labels
4000 labels
250 labels
1000 labels
4000 labels
VAT
36.03+2..22
18.64+0.40
11.05++0.31
8.41+1001
5.98
4.20=0.15
Mcan Teacher
47.32+4.77
17.32+4.00
10.36+00.25
6.45+22.3
3.75+0010
3.39=0.11
MixMatch
11.08=+0.87
7.75+0.32
6.24
3.78+10.26
3.27+10.11
2.89=0.06
ReMixMatch
6.27+t0.34
5.73::0116
5.1441004
3.10+0.50
2.83+0.30
2.42=0009
UDA, reported*
8.76:0.90
5.874:013
5.29
2.76110.17
2.55:10.09
2.47=0.15
Table I: Results on CIFAR-10 and SVHN. For UDA, due to adaptation difficulties, we report the comparable
results from Xie et al. (2019) which implementation, training procedure, etc. For VAT, Mean Teacher, and MixMatch, we report results using our reimplementation, which makes them directly comparable ReMixMatchi 's scores.
results due to a different network
4.2 STL-10
The STL-10 dataset consists of 5,000 labeled 96 x 96 color images drawn from 10 classes and 100,000 unlabeled images drawn from similar- but not identical distribution. The labeled

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
set is partitioned into ten pre-defined folds of 1,000 images each. For efficiency, only rum analysis on five of these ten folds. We do not perform evaluation here under the Realistic SSL OOlieer eet al. 2018)) setting when comparing to non-Mix.Mathh results. Our results are, however, directly comparable to the Mix Match results. Using the same WRN-37-2 network (23.8 million parameters), reduce the error rate by factor of two compared to MixMatch.
Method
Error Rate
Ablation
Error Rate
Ablation
Error Rate
SWWAE CC-GAN
25.70
22.20
ReMixMatch
5.94
No rotation loss
6.08
MixMatch
10.18 +: 1.46
With K=1
7.32
No pre-mixup loss
6,66
With K=2
6.74
No dist. alignment
7.28
ReMixxatch (K=1)
6.77 + 1.66
With K=4
6.21
L2 unlabeled loss
17.
ReMixMatch (K=4)
6.18 + 1.24
With K=16
5.93
No strong aug.
12.51
MixMatch
11.08
No weak aug.
29.36
Table 2: STL-10 error rate using 1000-label splits. SWWAE and CC- GAN results are from (Zhao et al. 2015 and (Detton et al. 2016).
Table 3: Ablation study. Error rates reported on single 250-Iabel split from CIFAR-10.
4.3 TOWARDS FEW-SHOT LEARNING
We find that ReMixMatch is able to work in extremely low-label settings. By only changing X, from 0.5 to 2 we can train CIFAR-10 with just four labels per class and SVHN with only 40 labels total. On CIFAR-I0 we obtain a median-of-five error rate of 15.08%; on SVHN we reach 3.48% error and on SVHN with the "extra" dataset we reach 2.81% error. Full results are given Appendix B.
4.4 ABLATION STUDY
Because we have made several changes to the existing MixMatch algorithm, here we perform an ablation study and remove one component of Re.ixxatch at a time to understand from which changes produce the largest accuracy gains. Our ablation results are summarized Table 3 We find that removing the pre-mixup unlabeled loss, removing distribution alignment, and lowering K all hurt performance by a small amount. Given that distribution alignment improves performance, we were interested to see whether it also had the intended effect of making marginal distribution of model predictions match the ground-truth marginal class distribution. We measure this directly in appendix D Removing the rotation loss reduces accuracy at 250 labels by only 0.14 percent- age points, but we find that in the 40-label setting rotation loss is necessary to prevent collapse. Changing the cross-entropy loss unlabeled data to L.2 loss used in MixMatch hurts perfor- mance dramatically, as does removing either of the augmentation components. This validates using augmentation anchoring in place of the consistency regularization mechanism of MixMatch.
CONCLUSION
Progress on semi-supervised learning over the past year has upended many of the long-held beliefs about classification, namely, that vast quantities of labeled data is necessary. By introducing aug- mentation anchoring and distribution alignment to MixMatch, we continue this trend: ReMix reduces the quantity of labeled data needed by large factor compared prior work (c.g.. beating MixMatch at 4000 labeled examples with only 250 on CIFAR-10, 10. and closely approaching Mix- Match at 5000 labeled examples with only 1000 on STI.-10). In future work, we are interested in pushing the limited data regime further to close the gap between few-shot learning and SSL. We also note that many real-life scenarios, a dataset begins as unlabeled and is incrementally labeled until satisfactory performance is achieved. Our strong empirical results suggest that it will be possible to achieve gains in this "active learning" setting by using ideas from ReMixMatch. Finally, this paper present results on widely-studied image benchmarks for case of comparison. However, the true power of data-efficient learning will come from applying these techniques to real-world problems where obtaining labeling data is expensive or impractical.
8

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
REFERENCES
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In Advances in Neural Information Processing Systems, 2014.
Yoshua Bengio, Olivier Delallcau, and Nicolas Le Roux. Label Propagation and Quadratic Crite- rion, chapter 11. MIT Press, 2006.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint arXiv: /r05:v:905.02249 2019.
John S. Bridle, Anthony R. Heading, and David C. MacKay. Unsupervised classifiers, mutual information and' phantom targets. In Advances in neural information processing systems, 1992.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthey Weather Review, 78(1):1-3, 1950.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press, 2006.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv: 2018.
Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: An automated, randomized data augmentation strategy. arXiv preprint XXXiv:XXXXXXXXXXX 2019.
Emily Denton, Sam Gross, and Rob Fergus. Semi-supervised learning with context-conditional generative adversarial networks. arXiv preprint wrXiv:1611.064300 2016.
Alexander Gammerman, Volodya Vovk, and Vladimir Vapnik. Learning by transduction. In Pro-
ceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, 1998.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances in Neural Information Processing Systems, 2005.
Geoffrey Hinton and Drew van Camp. Keeping neural networks simple by minimizing the descrip- tion length of the weights. In Proceedings of the 6th Annual ACM Conference on Computational Learning Theory, 1993.
Thorsten Joachims. Transductive inference for text classification using support vector machines. In International Conference on Machine Learning, 1999.
Thorsten Joachims. Transductive learning via spectral graph partitioning. In International Confer- ence on Machine Learning. 2003.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Third Interna- tional Conference on Learning Representations, 2015.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Fifth Interna- fional Conference on Learning Representations, 2017.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICMI. Workshop Challenges in Representation Learning, 2013.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. arXiv preprint arXiv: 0011990000097.. 2019.
Bin Liu, Zhirong Wu, Han Hu, and Stephen Lin. Deep metric transfer for label propagation with limited annotated data. arXiv preprint 1rXiv:1812.08788 2018.

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Geoffrey J. McLachlan. Iterative reclassification procedure for constructing asymptotically opti- mal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 70(350):365-369 1975.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin Dogus Cubuk, and lan Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Pro- cessing Systems, pp. 3235-3246, 2018.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self-training of object detection models. In Proceedings of the Seventh IEEE Workshops Application of Computer Vision, 2005.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep supe-super learning. In Advances Neural Information Processing Systems, 2016.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averrged consis- tency targets improve eemi-supervieed deep learning results. In Advances in neural information processing systems, 2017.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation sistency training for semi-supervised learning. arXiv preprint arXiv: 1902303333 2019.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. arXiv preprint urXiv:1904 12848, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. At: Self-supervised semi- supervised learning, arXiv preprint arXiv:1905.03670 2019.
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv: 022811 2018
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. arXiv preprint arXiv:1710.09412, 2017.
Junbo Zhao, Michael Mathieu, Ross Goroshin, and Yann Lecun. Stacked what- where auto-encoders. XrXiv preprint arXiv: 506.02351, 2015.
10

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
A PROOF OF EQUATION 2
The proof here follows closely Bridle et al. (1992). We begin with the definition
I(y; = // p(u, p(y,z) x)) log
p(y. x)) p(y)p(a)
dy dx
(5)
Rewriting terms we obtain
/ p(a) dx / plyla) log p(ulz)
dy
(6)
p(y)
/ p(x) dz plyll) log p(=)p(p)=) dy
p(yl=)
Then, rewriting both integrals expectations obtain
p(ylr) dy p(ylz) log
Ex
Es [v(]]
p(u /a)) Em (p [x)]]
p(yi (x) log
L Σ Log
Ex
ply [x) p())(())
Ex
(10)
Ex
p(u] log p(y)
E.
(11)
H (modee 0)1) Ex, (P(modd (y)k;; 0))
(12)
B FULL 40 LABEL RESULTS
We now report the full results for running ReMixMatch with just 40 labeled examples. We sort the table by error rate over five different splits (i.e., 40-label subsets) of the training data. High variance is to be expected when choosing so few labeled examples at random.
Dataset
Split (ordered by error rate)
CIFAR-10
10.88
12.65
15.08
16.78
19.49
SVHN
3.43
3.46
3.48
4.06
12.24
+eHNa
2.59
2.71
2.81
3.50
15.14
Table Sorted error rate of ReMixMatch with 40 labeled examples.
11

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
C TRANSFORMATIONS INCLUDED IN CTAUGMENT
Transformation
Description
Parameter
Range
Autocontrast
Maximizes the image contrast by setting the dark- est (lightest) pixel to black (white), and then blends with the original image with blending ra- tio A.
X
[0, 1]
Blur Brightness
Adjusts the brightness of the image. B 0 re- B turns black image, image.
[0, 1]
returns the original
Color
Adjusts the color balance the image like in TV. C returns black & white image, C' returns the original image. Controls the contrast of the image. A C 0 turns gray image, C' = returns the original image.
[0, 1]
Contrast
[0,
Cutout
Sets a random square patch of side-length L (Lximage width) pixels gray.
[0, 0.5]
Equalize
Equalizes the image histogram, and then blends A with the original image with blending ratio Inverts the pixels of the image, and then blends A with the original image with blending ratio Returns the original image.
[0, 1]
Invert
[0, 1]
Identity
Posterize
Reduces each pixel to bits. Takes center crop that is of side-length (Lximage width), and rescales to the original im- age size using method M.
[1, 8]
Rescale
L
[0.5, 1.0]
M
see caption [-45, 45] [0, 1]
Rotate
Rotates the image by 9 degrees.
0
Sharpness
Adjusts the sharpness of the image, where S = returns a blurred image, and original image.
S
returns the
Shear_x
Shears the image along the horizontal axis with rate R.
[-0.3, 0.3]
Shear_y
Shears the image along the vertical axis with rate R.
[-0.3, 0.3]
Smooth
Adjusts the smoothness of the image, where = O returns maximally smooth image, and S returns the original image.
5
[0, 1]
Solarize
Inverts all pixels above threshold value of T. Translates the image horizontally by (Aximage width) pixels.
T
[0, 1]
Translate
[-0.3, 0.3]
Translate.y
Translates the image vertically by (Xximage width) pixels.
A
[-0.3, 0.3]
Table 5: The ranges for all of the listed parameters discretized into 17 equal bins. The only exception is the M parameter of the Rescale transformation, which takes on one of the following six options: anti-alias, bicubic, bilinear, box. hamming, and nearest.
D MEASURING THE EFFECT OF DISTRIBUTION ALIGNMENT
Recall that the goal of distribution alignment is to encourage the marginal distribution the model's predictions p(y) match the true marginal class distribution p(y). To measure whether distribution alignment indeed has this effect, we monitored the KL divergence between p(y) and p(y) over the course of training. We show the KI. divergence for training
CIFAR-10 with 250 labels with
12

---

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
DEL. (p(())
wolundaan roth P wa Muy Without distribution matching
whdhkhrnn www. sp
With distribution matching
0
10
20
30
40
50
Step (millions)
Figure 3: KL divergence between the marginal distribution of model predictions vs. the true marginal distribution of class labels over the course of training with and without distribution alignment. This figure corresponds to a training run on CIFAR-10 with 250 labels.
and without distribution alignment in fig. 3 Indeed, the KL divergence between j(v) and p(y) is significantly smaller throughout training.
E CTAUGMENT PARAMETERS EFFECTS
In this section we compare the effects of varying the CTAugmeet hyper-parameters on CIFARIO with 250 labels, using the standard ReMixMatch settings. The exponential weight decay P does effect the results significantly while depth and threshold have significant effects. The default settings are highlighted in the table. They appear perform well and have been shown be robust across many datasets in our previous experiments.
Depth
Threshold
Error rate
0.80
0.99
23.90
0.80
0.99
6.25
0.80
0.99
6.36
0.50
0.99
10.51
0.80
0.99
6.25
0.90
0,99
10.80
2
0.95
0,99
18.47
0.80
0.9
6.15
0.80
0.99
6.25
0.80
0.999
6.02
Table 6: Effects of hyper-parameters for CTAuument, the bold results are the default settings used for all experiments.
13


Reviews:
[{"review_id": "rJljF0tRKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe authors make three major contributions that improve MixMatch and achieve state-of-the-art in a semi-supervised image classification task. The major contributions include: (1) distribution alignment to calibrate the predicted distribution of unlabeled data; (2) augmentation anchoring to allow more aggressive data augmentation; and (3) CTAugment to train the augmentation policy alongside the semi-supervised model.\nThe authors conduct experiments on SVHN, CIFAR-10 and STL, and show significant improvements over the MixMatch baseline. They also show good results (15.08% error rate) of training with 40 labeled data, in spite of very high variation. In the ablation study, they show the error rate drops as K (number of augmentation) increases. They also conduct ablation studies on the design choices of their method.\n\nDecision\nThe decision for this paper is borderline, tending towards a weak accept. Overall, the paper proposes some simple but interesting ideas, e.g. distribution environments. However, although the proposed method achieves good performance over various (smaller) benchmarks, the method seems ad-hoc and complicated. As pointed out in the weakness section, many design choices are not well motivated, and the effects of those designs are not well studied. The tendency to accept is due to the overall strong results. \n\nStrength\n1. Significant improvement over MixMatch baseline.\n2. The proposed augmentation anchoring and distribution alignment can be easily integrated into existing work.\n3. The proposed CTAugment method lifts the burden of training an RL data augmentation policy. \n\nWeakness\n1. The objective of the update equation of CTAugment’s learned weights seems contradicted with the purpose of how data augmentation is used in the consistency-based SSL method. In other words, the objective of the update equation encourages higher weights for the distortion parameter that leads to lower variation in the predicted distribution. However, the idea of aggressive data augmentation is to generate data that has high variation in the model prediction, and then penalize the variation in the form of consistency loss. The variation induced by aggressive augmentation is the root of the consistency loss that helps regularize the model.\n2. The authors should provide ablation study and analysis of their CTAugment. For example, they should compare with simple random augmentation policy. It is also recommended to show the learned weights of the distortion parameter. Also does larger K value when applied for vanilla MixMatch approach the results in ReMixMatch? \n3. The authors should provide more detail of the setting in the ablation study. For example, the setting of “No strong aug.” and “No weak aug.” are not clear. \n4. The authors hypothesize that “stronger augmentation can result in disparate predictions, so their average may not be a meaningful target.” However, they do not show any analysis to support this hypothesis.\n5. It is recommended to evaluate the method on larger datasets such as CIFAR-100. It is not clear how well these methods scale, and for example using k=8 adds computation which may hinder training scalability. \n\nMinor Comments\n1. For Table 2 and Table 3, it should be “error rate” rather than “accuracy”.\n2. How is the loss weight λr tuned in the 40 labeled setting? How are the hyper-parameters tuned in general?\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1gjMgfaYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two modifications for the MixMatch method [1] and achieves improved accuracy on a range of semi-supervised benchmarks. The first modification enforces the distribution of predicted labels to match the distribution of labeled data. The second modification is adding a learned data augmentation strategy, and adapting the method to work with strong data augmentation. The final method is titled ReMixMatch, and improves significantly over MixMatch, especially in low-data regime. \n\nThe main contribution of the paper is really strong empirical results. The method achieves state of the art results or close to that on multiple benchmarks, with especially large gains in settings with very scarce labeled data, like 40 labels on CIFAR-10. \n\nAnother important contribution is the learned data augmentation strategy, which as far as I understand is novel and overcomes some of the limitations of  existing learned data augmentation techniques. However, the explanation of the strategy wasn’t very clear for me, and the authors didn’t frame it as a major contribution.\n\nThe main drawback of the paper is that it seems to be more engineering-focused, and doesn’t provide much insight into semi-supervised learning. The paper can be summarized as adding two modifications to mix-match, and getting better results. The final method becomes fairly involved. Mix-Match is already an elaborate method, and ReMixMatch additionally introduces learned data augmentation, an additional loss term for matching label distributions between labeled and unlabeled data, consistency-loss, and a self-supervised loss (section 3.3). \n\nFor the reasons above, I think the paper is borderline, but I am currently voting for acceptance based on the strong empirical performance. At the same time, I think the paper can be made stronger and more interesting to read, if the authors added some experiments aimed at understanding the proposed modifications. \n\nOne set of experiments that I think would be interesting is aimed at understanding the distribution-matching part. For example, it would be great if the author could demonstrate that without this loss term the distribution of the predicted classes is wrong in the experiments from Section 4. It would also be interesting to see an experiment where the labeled data has a skewed distribution of classes, but we provide the method with information about the true class distribution, and demonstrating that this information helps predictive performance. \n\nFor the learnable data augmentation it would be great if the authors could provide more insight into the method, how it works, and why is it better than the alternatives. Just analyzing the learned data augmentation in different settings and adding more intuition for what happens would make the paper more insightful and interesting to read. \n\nOn a more minor note, the paper [1] seems to report 4.95 accuracy for MixMatch on CIFAR-10 with 4k labels, while in this paper it’s being reported as 6.24. What is the reason for the difference? Another paper, [2], reports very competitive results on CIFAR-10 for 4k labels. I would recommend discussing these results briefly in the paper. At the same time the empirical performance of ReMixMatch is really impressive, and I don’t think the results in [1] and [2] affect their significance.\n\n[1] MixMatch: A Holistic Approach to Semi-Supervised Learning\nDavid Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, Colin Raffel\n\n[2] There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average\nBen Athiwaratkun, Marc Finzi, Pavel Izmailov, Andrew Gordon Wilson\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1eFU86RYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents ReMixMatch an improved version of MixMatch. The main contributions are the distribution alignment and the augmentation anchoring. Distribution alignment rescales the predictions based on the difference between the model marginals and the ground truth running average estimation. Augmentation anchoring instead of computing the guessed probabilities on unlabelled data as the average probabilities on transformed samples (as in MixMatch), it considers as guessed labels the average probabilities obtained from weak transformations (flip+crop) even when using stronger transformations (Autoaugment like). \n\nThe paper is well written, has interesting experiments and very impressive results. \nHowever, there are some negative points that the authors should clarify:\n- The final method is a mixup of many different techniques, thus, not a strong contribution, but many smaller contributions. \n- As shown in the ablation study, the main contribution on the obtained results seems to be the use of stronger transformations than in MixMatch. This is not so interesting, even though results are impressive. If this is the case, authors should state it more clearly in the paper that a large proportion of the gap in performance between MixMatch and ReMixMatch is the introduction of stronger transformations (Autoaugment style).\n\nOverall the paper is well presented and contributes to further improve the performance on semi-supervised learning. I there fore recommend it for acceptance. However, I would like to see in the paper a more general overview on the fact that strong transformations can further improve semi-supervised methods and ReMixMatch is a way to leverage those transformations.\n\n\nAdditional comments:\n- Instead of using the rescaling trick for distribution alignment, what about enforcing the marginal distribution on the annotated data and the marginal distribution of the model to be similar with KL divergence? Would it be better or worse than the proposed approach?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This works improves the MixMatch semi-supervised algorithm along the two directions of distribution alignment and augmentation anchoring, which together make the approach more data-efficient than prior work.
All reviewers agree that the impressive empirical results in the paper are its main strength, but express concern that the method is overly complicated and hacking together many known pieces, as well as doubt as to the extent of the contribution of the augmentation method itself, with requests for better augmentation controls.
While some of these concerns have not been addressed by authors in their response, the strength of empirical results seems enough to justify an acceptance recommendation.
Decision: Accept (Poster)