Title: TEMPORAL ENSEMBLING FOR SEMI-SUPERVISED LEARNING

Abstract: In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.

1. INTRODUCTION:

It has long been known that an ensemble of multiple neural networks generally yields better predictions than a single network in the ensemble. This effect has also been indirectly exploited when training a single network through dropout (Srivastava et al., 2014) , dropconnect (Wan et al., 2013) , or stochastic depth (Huang et al., 2016) regularization methods, and in swapout networks (Singh et al., 2016) , where training always focuses on a particular subset of the network, and thus the complete network can be seen as an implicit ensemble of such trained sub-networks. We extend this idea by forming ensemble predictions during training, using the outputs of a single network on different training epochs and under different regularization and input augmentation conditions. Our training still operates on a single network, but the predictions made on different epochs correspond to an ensemble prediction of a large number of individual sub-networks because of dropout regularization.

This ensemble prediction can be exploited for semi-supervised learning where only a small portion of training data is labeled. If we compare the ensemble prediction to the current output of the network being trained, the ensemble prediction is likely to be closer to the correct, unknown labels of the unlabeled inputs. Therefore the labels inferred this way can be used as training targets for the unlabeled inputs. Our method relies heavily on dropout regularization and versatile input augmentation. Indeed, without neither, there would be much less reason to place confidence in whatever labels are inferred for the unlabeled training data.

We describe two ways to implement self-ensembling, Π-model and temporal ensembling. Both approaches surpass prior state-of-the-art results in semi-supervised learning by a considerable margin. We furthermore observe that self-ensembling improves the classification accuracy in fully labeled cases as well, and provides tolerance against incorrect labels.

The recently introduced transform/stability loss of Sajjadi et al. (2016b) is based on the same principle as our work, and the Π-model can be seen as a special case of it. The Π-model can also be seen as a simplification of the Γ-model of the ladder network by Rasmus et al. (2015) , a previously presented network architecture for semi-supervised learning. Our temporal ensembling method has connections to the bootstrapping method of Reed et al. (2014) targeted for training with noisy labels. 

z i∈B ← f θ (g(x i∈B )) evaluate network outputs for augmented inputs zi∈B ← f θ (g(x i∈B ))

again, with different dropout and augmentation loss ← -

1 |B| i∈(B∩L) log z i [y i ] supervised loss component + w(t) 1 C|B| i∈B ||z i -zi || 2

unsupervised loss component update θ using, e.g., ADAM update network parameters end for end for return θ

2. SELF-ENSEMBLING DURING TRAINING:

We present two implementations of self-ensembling during training. The first one, Π-model, encourages consistent network output between two realizations of the same input stimulus, under two different dropout conditions. The second method, temporal ensembling, simplifies and extends this by taking into account the network predictions over multiple previous training epochs.

We shall describe our methods in the context of traditional image classification networks. Let the training data consist of total of N inputs, out of which M are labeled. The input stimuli, available for all training data, are denoted x i , where i ∈ {1 . . . N }. Let set L contain the indices of the labeled inputs, |L| = M . For every i ∈ L, we have a known correct label y i ∈ {1 . . . C}, where C is the number of different classes.

2.1. Π-MODEL:

The structure of Π-model is shown in Figure 1 (top), and the pseudocode in Algorithm 1. During training, we evaluate the network for each training input x i twice, resulting in prediction vectors z i and zi . Our loss function consists of two components. The first component is the standard crossentropy loss, evaluated for labeled inputs only. The second component, evaluated for all inputs, penalizes different predictions for the same training input x i by taking the mean square difference between the prediction vectors z i and zi .1 To combine the supervised and unsupervised loss terms, we scale the latter by time-dependent weighting function w(t). By comparing the entire output vectors z i and zi , we effectively ask the "dark knowledge" (Hinton et al., 2015) between the two evaluations to be close, which is a much stronger requirement compared to asking that only the final classification remains the same, which is what happens in traditional training.

It is important to notice that, because of dropout regularization, the network output during training is a stochastic variable. Thus two evaluations of the same input x i under same network weights θ yield different results. In addition, Gaussian noise and augmentations such as random translation are evaluated twice, resulting in additional variation. The combination of these effects explains the difference between the prediction vectors z i and zi . This difference can be seen as an error in classification, given that the original input x i was the same, and thus minimizing it is a reasonable goal.

In our implementation, the unsupervised loss weighting function w(t) ramps up, starting from zero, along a Gaussian curve during the first 80 training epochs. See Appendix A for further details about this and other training parameters. In the beginning the total loss and the learning gradients are thus dominated by the supervised loss component, i.e., the labeled data only. We have found it to be very important that the ramp-up of the unsupervised loss component is slow enough-otherwise, the network gets easily stuck in a degenerate solution where no meaningful classification of the data is obtained.

Our approach is somewhat similar to the Γ-model of the ladder network by Rasmus et al. (2015) , but conceptually simpler. In the Π-model, the comparison is done directly on network outputs, i.e., after softmax activation, and there is no auxiliary mapping between the two branches such as the learned denoising functions in the ladder network architecture. Furthermore, instead of having one "clean" and one "corrupted" branch as in Γ-model, we apply equal augmentation and noise to the inputs for both branches.

As shown in Section 3, the Π-model combined with a good convolutional network architecture provides a significant improvement over prior art in classification accuracy.

2.2. TEMPORAL ENSEMBLING:

Analyzing how the Π-model works, we could equally well split the evaluation of the two branches in two separate phases: first classifying the training set once without updating the weights θ, and then training the network on the same inputs under different augmentations and dropout, using the just obtained predictions as targets for the unsupervised loss component. As the training targets obtained this way are based on a single evaluation of the network, they can be expected to be noisy. Temporal ensembling alleviates this by aggregating the predictions of multiple previous network evaluations into an ensemble prediction. It also lets us evaluate the network only once during training, gaining an approximate 2x speedup over the Π-model.

The structure of our temporal ensembling method is shown in Figure 1 (bottom), and the pseudocode in Algorithm 2. The main difference to the Π-model is that the network and augmentations are evaluated only once per input per epoch, and the target vectors z for the unsupervised loss component are based on prior network evaluations instead of a second evaluation of the network.

After every training epoch, the network outputs z i are accumulated into ensemble outputs Z i by updating Z i ← αZ i + (1 -α)z i , where α is a momentum term that controls how far the ensemble reaches into training history. Because of dropout regularization and stochastic augmentation, Z thus contains a weighted average of the outputs of an ensemble of networks f from previous training epochs, with recent epochs having larger weight than distant epochs. For generating the training targets z, we need to correct for the startup bias in Z by dividing by factor (1 -α t ). A similar bias correction has been used in, e.g., Adam (Kingma & Ba, 2014) and mean-only batch normalization (Salimans & Kingma, 2016) . On the first training epoch, Z and z are zero as no data from previous epochs is available. For this reason, we specify the unsupervised weight ramp-up function w(t) to also be zero on the first training epoch.

Algorithm 2 Temporal ensembling pseudocode. Note that the updates of Z and z could equally well be done inside the minibatch loop; in this pseudocode they occur between epochs for clarity. Require: x i = training stimuli Require: L = set of training input indices with known labels Require: y i = labels for labeled inputs i ∈ L Require: α = ensembling momentum, 0 ≤ α < 1 Require: w(t) = unsupervised weight ramp-up function Require: f θ (x) = stochastic neural network with trainable parameters θ Require: g(x) = stochastic input augmentation function

Z ← 0 [N ×C] initialize ensemble predictions z ← 0 [N ×C] initialize target vectors for t in [1, num epochs] do for each minibatch B do z i∈B ← f θ (g(x i∈B , t)) evaluate network outputs for augmented inputs loss ← -1 |B| i∈(B∩L) log z i [y i ] supervised loss component + w(t) 1 C|B| i∈B ||z i -zi || 2

unsupervised loss component update θ using, e.g., ADAM update network parameters end for

Z ← αZ + (1 -α)z accumulate ensemble predictions z ← Z/(1 -α t )

construct target vectors by bias correction end for return θ

The benefits of temporal ensembling compared to Π-model are twofold. First, the training is faster because the network is evaluated only once per input on each epoch. Second, the training targets z can be expected to be less noisy than with Π-model. As shown in Section 3, we indeed obtain somewhat better results with temporal ensembling than with Π-model in the same number of training epochs. The downside compared to Π-model is the need to store auxiliary data across epochs, and the new hyperparameter α. While the matrix Z can be fairly large when the dataset contains a large number of items and categories, its elements are accessed relatively infrequently. Thus it can be stored, e.g., in a memory mapped file.

An intriguing additional possibility of temporal ensembling is collecting other statistics from the network predictions z i besides the mean. For example, by tracking the second raw moment of the network outputs, we can estimate the variance of each output component z i,j . This makes it possible to reason about the uncertainty of network outputs in a principled way (Gal & Ghahramani, 2016) . Based on this information, we could, e.g., place more weight on more certain predictions vs. uncertain ones in the unsupervised loss term. However, we leave the exploration of these avenues as future work.

3. RESULTS:

Our network structure is given in Table 5 , and the test setup and all training parameters are detailed in Appendix A. We test the Π-model and temporal ensembling in two image classification tasks, CIFAR-10 and SVHN, and report the mean and standard deviation of 10 runs using different random seeds.

Although it is rarely stated explicitly, we believe that our comparison methods do not use input augmentation, i.e., are limited to dropout and other forms of permutation-invariant noise. Therefore we report the error rates without augmentation, unless explicitly stated otherwise. Given that the ability of an algorithm to extract benefit from augmentation is also an important property, we report the classification accuracy using a standard set of augmentations as well. In purely supervised training the de facto standard way of augmenting the CIFAR-10 dataset includes horizontal flips and random translations, while SVHN is limited to random translations. By using these same augmentations we can compare against the best fully supervised results as well. After all, the fully supervised results should indicate the upper bound of obtainable accuracy. 36.02 ± 0.10 Virtual Adversarial (Miyato et al., 2016) 24.63 ADGM (Maaløe et al., 2016) 22.86 SDGM (Maaløe et al., 2016) 16.61 ± 0.24 GAN of Salimans et al. (2016) 18 Enabling the standard set of augmentations further reduces the error rate by 4.2 percentage points to 12.36%. Temporal ensembling is slightly better still at 12.16%, while being twice as fast to train. This small improvement conceals the subtle fact that random horizontal flips need to be done independently for each epoch in temporal ensembling, while Π-model can randomize once per a pair of evaluations, which according to our measurements is ∼0.5 percentage points better than independent flips.

A principled comparison with Sajjadi et al. (2016b) is difficult due to several reasons. They provide results only for a fairly extreme set of augmentations (translations, flipping, rotations, stretching, and shearing) on top of fractional max pooling (Graham, 2014) , which introduces random, local stretching inside the network, and is known to improve classification results substantially. They quote an error rate of only 13.60% for supervised-only training with 4000 labels, while our corresponding baseline is 34.85%. This gap indicates a huge benefit from versatile augmentations and fractional max pooling-in fact, their baseline result is already better than any previous semisupervised results. By enabling semi-supervised learning they achieve a 17% drop in classification error rate (from 13.60% to 11.29%), while we see a much larger relative drop of 65% (from 34.85% to 12.16%).

3.2. SVHN:

The street view house numbers (SVHN) dataset consists of 32 × 32 pixel RGB images of real-world house numbers, and the task is to classify the centermost digit. In SVHN we chose to use only the Table 2 compares our method to the previous state-of-the-art. With the most commonly used 1000 labels we observe an improvement of 2.7 percentage points, from 8.11% to 5.43% without augmentation, and further to 4.42% with standard augmentations.

We also investigated the behavior with 500 labels, where we obtained an error rate less than half of Salimans et al. (2016) without augmentations, with a significantly lower standard deviation as well. When augmentations were enabled, temporal ensembling further reduced the error rate to 5.12%. In this test the difference between Π-model and temporal ensembling was quite significant at 1.5 percentage points.

In SVHN Sajjadi et al. (2016b) provide results without augmentation, with the caveat that they use fractional max pooling, which is a very augmentation-like technique due to the random, local stretching it introduces inside the network. It leads to a superb error rate of 2.28% in supervisedonly training, while our corresponding baseline is 3.05% (or 2.88% with translations). Given that in a separate experiment our network matched the best published result for non-augmented SVHN when extra data is used (1.69% from Lee et al. (2015) ), this gap is quite surprising, and leads us to conclude that fractional max pooling leads to a powerful augmentation of the dataset, well beyond what simple translations can achieve. Our temporal ensembling technique obtains better error rates for both 500 and 1000 labels (5.12% and 4.42%, respectively) compared to the 6.03% reported by Sajjadi et al. for 732 labels.

3.3. CIFAR-100 AND TINY IMAGES:

The CIFAR-100 dataset consists of 32 × 32 pixel RGB images from a hundred classes. We are not aware of previous semi-supervised results in this dataset, and chose 10000 labels for our experiments. Table 3 shows error rates of 43.43% and 38.65% without and with augmentation, respectively. These correspond to 7.8 and 5.9 percentage point improvements compared to supervised learning with labeled inputs only.

We ran two additional tests using unlabeled extra data from Tiny Images dataset (Torralba et al., 2008) : one with randomly selected 500k extra images, most not corresponding to any of the CIFAR-100 categories, and another with a restricted set of 237k images from the categories that correspond to those found in the CIFAR-100 dataset (see appendix A for details). The results are shown in the classification accuracy further. This indicates that in order to train a better classifier by adding extra data as unlabeled inputs, it is enough to have the extra data roughly in the same space as the actual inputs-in our case, natural images. We hypothesize that it may even be possible to use properly crafted synthetic data as unlabeled inputs to obtain improved classifiers.

In order to keep the training times tolerable, we limited the number of unlabeled inputs to 50k per epoch in these tests, i.e., on every epoch we trained using all 50k labeled inputs from CIFAR-100 and 50k additional unlabeled inputs from Tiny Images. The 50k unlabeled inputs were chosen randomly on each epoch from the 500k or 237k extra inputs. In temporal ensembling, after each epoch we updated only the rows of Z that corresponded to inputs used on that epoch.

3.4. SUPERVISED LEARNING:

When all labels are used for traditional supervised training, our network approximately matches the state-of-the-art error rate for a single model in CIFAR-10 with augmentation (Lee et al., 2015; Mishkin & Matas, 2016 ) at 6.05%, and without augmentation (Salimans & Kingma, 2016) at 7.33%. The same is probably true for SVHN as well, but there the best published results rely on extra data that we chose not to use.

Given this premise, it is perhaps somewhat surprising that our methods reduce the error rate also when all labels are used (Tables 1 and 2 ). We believe that this is an indication that the consistency requirement adds a degree of resistance to ambiguous labels that are fairly common in many classification tasks, and that it encourages features to be more invariant to stochastic sampling.

3.5. TOLERANCE TO INCORRECT LABELS:

In a further test we studied the hypothesis that our methods add tolerance to incorrect labels by assigning a random label to a certain percentage of the training set before starting to train. Figure 2 shows the classification error graphs for standard supervised training and temporal ensembling.

Clearly our methods provide considerable resistance to wrong labels, and we believe this is because the unsupervised loss term encourages the mapping function implemented by the network to be flat in the vicinity of all input data points, whereas the supervised loss term enforces the mapping function to have a specific value in the vicinity of the labeled input data points. This means that even the wrongly labeled inputs play a role in shaping the mapping function-the unsupervised loss term smooths the mapping function and thus also the decision boundaries, effectively fusing the inputs into coherent clusters, whereas the excess of correct labels in each class is sufficient for locking the clusters to the right output vectors through the supervised loss term. The difference to classical regularizers is that we induce smoothness only on the manifold of likely inputs instead of over the entire input domain. For further analysis about the importance of the gradient of the mapping function, see Simard et al. (1998) .

4. RELATED WORK:

There is a large body of previous work on semi-supervised learning (Zhu, 2005) . In here we will concentrate on the ones that are most directly connected to our work.

Γ-model is a subset of a ladder network (Rasmus et al., 2015) that introduces lateral connections into an encoder-decoder type network architecture, targeted at semi-supervised learning. In Γ-model, all but the highest lateral connections in the ladder network are removed, and after pruning the unnecessary stages, the remaining network consists of two parallel, identical branches. One of the branches takes the original training inputs, whereas the other branch is given the same input corrupted with noise. The unsupervised loss term is computed as the squared difference between the (pre-activation) output of the clean branch and a denoised (pre-activation) output of the corrupted branch. The denoised estimate is computed from the output of the corrupted branch using a parametric nonlinearity that has 10 auxiliary trainable parameters per unit. Our Π-model differs from the Γ-model in removing the parametric nonlinearity and denoising, having two corrupted paths, and comparing the outputs of the network instead of pre-activation data of the final layer. Sajjadi et al. (2016b) recently introduced a new loss function for semi-supervised learning, so called transform/stability loss, which is founded on the same principle as our work. During training, they run augmentation and network evaluation n times for each minibatch, and then compute an unsupervised loss term as the sum of all pairwise squared distances between the obtained n network outputs. As such, their technique follows the general pseudo-ensemble agreement (PEA) regularization framework of Bachman et al. (2014) . In addition, they employ a mutual exclusivity loss term (Sajjadi et al., 2016a ) that we do not use. Our Π-model can be seen as a special case of the transform/stability loss obtained by setting n = 2. The computational cost of training with transform/stability loss increases linearly as a function of n, whereas the efficiency of our temporal ensembling technique remains constant regardless of how large effective ensemble we obtain via the averaging of previous epochs' predictions.

In bootstrap aggregating, or bagging, multiple networks are trained independently based on subsets of training data (Breiman, 1996) . This results in an ensemble that is more stable and accurate than the individual networks. Our approach can be seen as pulling the predictions from an implicit ensemble that is based on a single network, and the variability is a result of evaluating it under different dropout and augmentation conditions instead of training on different subsets of data. In work parallel to ours, Huang et al. (2017) & Ghahramani, 2002) infer labels for unlabeled training data by comparing the associated inputs to labeled training inputs using a suitable distance metric. Our approach differs from this in two important ways. Firstly, we never compare training inputs against each other, but instead only rely on the unknown labels remaining constant, and secondly, we let the network produce the likely classifications for the unlabeled inputs instead of providing them through an outside process.

In addition to partially labeled data, considerable amount of effort has been put into dealing with densely but inaccurately labeled data. This can be seen as a semi-supervised learning task where part of the training process is to identify the labels that are not to be trusted. For recent work in this area, see, e.g., Sukhbaatar et al. (2014) and Patrini et al. (2016) . In this context of noisy labels, Reed et al. (2014) presented a simple bootstrapping method that trains a classifier with the target composed of a convex combination of the previous epoch output and the known but potentially noisy labels. Our temporal ensembling differs from this by taking into account the evaluations over multiple epochs.

Generative Adversarial Networks (GAN) have been recently used for semi-supervised learning with promising results (Maaløe et al., 2016; Springenberg, 2016; Odena, 2016; Salimans et al., 2016) . It Additive Gaussian noise σ = 0.15 conv1a 128 filters, 3 × 3, pad = 'same', LReLU (α = 0.1) conv1b 128 filters, 3 × 3, pad = 'same', LReLU (α = 0.1) conv1c 128 filters, 3 × 3, pad = 'same', LReLU (α = 0.1) pool1

Maxpool 2 × 2 pixels drop1

Dropout, p = 0.5 conv2a 256 filters, 3 × 3, pad = 'same', LReLU (α = 0.1) conv2b 256 filters, 3 × 3, pad = 'same', LReLU (α = 0.1) conv2c 256 filters, 3 × 3, pad = 'same', LReLU (α = 0.1) pool2

Maxpool 2 × 2 pixels drop2

Dropout, p = 0.5 conv3a 512 filters, 3 × 3, pad = 'valid', LReLU (α = 0.1) conv3b 256 filters, 1 × 1, LReLU (α = 0.1) conv3c 128 filters, 1 × 1, LReLU (α = 0.1) pool3

Global average pool (6 × 6 → 1×1 pixels) dense Fully connected 128 → 10 output Softmax could be an interesting avenue for future work to incorporate a generative component to our solution.

We also envision that our methods could be applied to regression-type learning tasks.

CIFAR-10 Following previous work in fully supervised learning, we pre-processed the images using ZCA and augmented the dataset using horizontal flips and random translations. The translations were drawn from [-2, 2] pixels, and were independently applied to both branches in the Π-model.

SVHN We pre-processed the input images by biasing and scaling each input image to zero mean and unit variance. We used only the 73257 items in the official training set, i.e., did not use the provided 531131 extra items. The training setups were otherwise similar to CIFAR-10 except that horizontal flips were not used.

Implementation:

Our implementation is written in Python using Theano (Theano Development Team, 2016) and Lasagne (Dieleman et al., 2015) , and is available at https://github.com/smlaine2/tempens.

Model convergence As discussed in Section 2.1, a slow ramp-up of the unsupervised cost is very important for getting the models to converge. Furthermore, in our very preliminary tests with 250 labels in SVHN we noticed that optimization tended to explode during the ramp-up period, and we eventually found that using a lower value for Adam β 2 parameter (e.g., 0.99 instead of 0.999) seems to help in this regard.

We do not attempt to guarantee that the occurrence of labeled inputs during training would be somehow stratified; with bad luck there might be several consecutive minibatches without any labeled inputs when the label density is very low. Some previous work has identified this as a weakness, and have solved the issue by shuffling the input sequences in such a way that stratification is guaranteed, e.g. Rasmus et al. (2015) (confirmed from the authors). This kind of stratification might further improve the convergence of our methods as well.

Tiny Images, extra data from restricted categories The restricted extra data in Section 3.3 was extracted from Tiny Images by picking all images with labels corresponding to the 100 categories used in CIFAR-100. As the Tiny Images dataset does not contain CIFAR-100 categories aquarium fish and maple tree, we used images with labels fish and maple instead. The result was a total of 237 203 images that were used as unlabeled extra data. Table 6 shows the composition of this extra data set.

It is worth noting that the CIFAR-100 dataset itself is a subset of Tiny Images, and we did not explicitly prevent overlap between this extra set and CIFAR-100. This led to approximately a third of the CIFAR-100 training and test images being present as unlabeled inputs in the extra set. The other test with 500k extra entries picked randomly out of all 79 million images had a negligible overlap with CIFAR-100. 4 ). Note that the extra input images were supplied as unlabeled data for our networks, and the labels were used only for narrowing down the full set of 79 million images.

:

Squared difference gave slightly but consistently better results than cross-entropy loss in our tests.

Figures and Tables:

Figure 1: Structure of the training pass in our methods. Top: Π-model. Bottom: temporal ensembling. Labels y i are available only for the labeled inputs, and the associated cross-entropy loss component is evaluated only for those.

Figure 2: Percentage of correct SVHN classifications as a function of training epoch when a part of the labels is randomized. With standard supervised training (left) the classification accuracy sufferswhen even a small portion of the labels give disinformation, and the situation worsens quickly as the portion of randomized labels increases to 50% or more. On the other hand, temporal ensembling (right) shows almost perfect resistance to disinformation when half of the labels are random, and retains over ninety percent classification accuracy even when 80% of the labels are random.

CIFAR-10 results with 4000 labels, averages of 10 runs (4 runs for all labels).

Table content: <table><tr><td colspan="2">Error rate (%) with # labels</td></tr><tr><td>4000</td><td>All (50000)</td></tr></table>

SVHN results for 500 and 1000 labels, averages of 10 runs (4 runs for all labels).

Table content: <table><tr><td>Model</td><td colspan="3">Error rate (%) with # labels 500 1000 All (73257)</td></tr><tr><td>Supervised-only</td><td>35.18 ± 5.61</td><td>20.47 ± 2.64</td><td>3.05 ± 0.07</td></tr><tr><td>with augmentation</td><td>31.59 ± 3.60</td><td>19.30 ± 3.89</td><td>2.88 ± 0.03</td></tr><tr><td>DGN (Kingma et al., 2014)</td><td/><td/><td/></tr></table>

CIFAR-100 results with 10000 labels, averages of 10 runs (4 runs for all labels).

Table content: <table><tr><td colspan="2">Error rate (%) with # labels</td></tr><tr><td>10000</td><td>All (50000)</td></tr></table>

CIFAR-100 + Tiny Images results, averages of 10 runs.

Table content: <table><tr><td colspan="2">Error rate (%) with # unlabeled</td></tr><tr><td colspan="2">auxiliary inputs from Tiny Images</td></tr><tr><td>Random 500k</td><td>Restricted 237k</td></tr></table>



Table content: <table/>

store multiple snapshots of the network during training, hopefully corresponding to different local minima, and use them as an explicit ensemble. The general technique of inferring new labels from partially labeled data is often referred to as bootstrapping or self-training, and it was first proposed by Yarowsky (1995) in the context of linguistic analysis. Whitney & Sarkar (2012) analyze Yarowsky's algorithm and propose a novel graph-based label propagation approach. Similarly, label propagation methods (Zhu

Table content: <table/>

The network architecture used in all of our tests.

Table content: <table><tr><td>NAME</td><td>DESCRIPTION</td></tr><tr><td>input</td><td>32 × 32 RGB image</td></tr><tr><td>noise</td><td/></tr></table>

The Tiny Images(Torralba et al., 2008)  labels and image counts used in the CIFAR-100 plus restricted extra data tests (rightmost column of Table

Table content: <table/>

Peer review (iclr2017):
This paper presents a model for semi-supervised learning by encouraging feature invariance to stochastic perturbations of the network and/or inputs. Two models are described: One where an invariance term is applied between different instantiations of the model/input a single training step, and a second where invariance is applied to features for the same input point across training steps via a cumulative exponential averaging of the features. These models evaluated using CIFAR-10 and SVHN, finding decent gains of similar amounts in each case. An additional application is also explored at the end, showing some tolerance to corrupted labels as well. The authors also discuss recent work by Sajjadi &al that is very similar in spirit, which I think helps corroborate the findings here. My largest critique is it would have been nice to see applications on larger datasets as well. CIFAR and SVHN are fairly small test cases, though adequate for demonstration of the idea. For cases of unlabelled data especially, it would be good to see tests with on the order of 1M+ data samples, with 1K-10K labeled, as this is a common case when labels are missing. On a similar note, data augmentations are restricted to only translations and (for CIFAR) horizontal flips. While *standard,* as the paper notes, more augmentations would have been interesting to see --- particularly since the model is designed explicitly to take advantage of random sampling. Some more details might also pop up, such as the one the paper mentions about handling horizontal flips in different ways between the two model variants. Rather than restrict the system to a particular set of augmentations, I think it would be interesting to push it further, and see how its performance behaves over a larger array of augmentations and (even fewer) numbers of labels. Overall, this seems like a simple approach that is getting decent results, though I would have liked to see more and larger experiments to get a better sense for its performance characteristics. Smaller comment: the paper mentions *dark knowledge* a couple times in explaining results, e.g. bottom of p.6. This is OK for a motivation, but in analyzing the results I think it may be possible to have something more concrete. For instance, the consistency term encourages feature invariance to the stochastic sampling more strongly than would a classification loss alone.