Title: SEMI-SUPERVISED LEARNING BY COACHING

Abstract: Recent semi-supervised learning (SSL) methods often have a teacher to train a student in order to propagate labels from labeled data to unlabeled data. We argue that a weakness of these methods is that the teacher does not learn from the student's mistakes during the course of student's learning. To address this weakness, we introduce Coaching, a framework where a teacher generates pseudo labels for unlabeled data, from which a student will learn and the student's performance on labeled data will be used as reward to train the teacher using policy gradient. Our experiments show that Coaching significantly improves over state-of-the-art SSL baselines. For instance, on CIFAR-10, with only 4,000 labeled examples, a WideResNet-28-2 trained by Coaching achieves 96.11% accuracy, which is better than 94.9% achieved by the same architecture trained with 45,000 labeled. On Im-ageNet with 10% labeled examples, Coaching trains a ResNet-50 to 72.94% top-1 accuracy, comfortably outperforming the existing state-of-the-art by more than 4%. Coaching also scales successfully to the high data regime with full ImageNet. Specifically, with additional 9 million unlabeled images from OpenImages, Coaching trains a ResNet-50 to 82.34% top-1 accuracy, setting a new state-of-the-art for the architecture on ImageNet without using extra labeled data. 1

1. INTRODUCTION:

Professional players in competitive sports such as chess, tennis, or swimming often have coaches to help improving their performance. Although coaches typically do not play as well as the players, they observe the players and provide instructions to improve the players' performance. Modern semi-supervised learning (SSL) algorithms do not follow this strategy. They instead have a teacher model that generates pseudo labels for unlabeled data, from which a student model learns by imitation (e.g., Lee (2013) ; Tarvainen & Valpola (2017) ; Laine & Aila (2017)). A weakness of these methods is that the teacher does not adjust itself based on the student's performance and cannot adapt to make the student better over time, unlike professional sport coaches develop their players.

Here, we propose a new semi-supervised learning method, called Coaching as shown in Figure 1 , where the teacher learns throughout the course of student's training. In our method, a teacher generates pseudo labels for unlabeled data, from which the student will learn. The student's performance on labeled data will be used as reward to train the teacher with policy gradient. Experiments show that our method achieves significant improvements over state-of-the-art semisupervised learning baselines and can be up to 10× more data efficient than supervised learning. For instance, with CIFAR-10, only using 4,000 labeled examples, a WideResNet-28-2 can be coached to 96.11% accuracy, outperforming the same model trained with 45,000 labeled examples which achieves 94.9%. Meanwhile, on ImageNet, using ResNet-50 with only 10% labeled examples, our method achieves 72.94% top-1 accuracy, outperforming all existing semi-supervised learning methods with the same amount of labeled data, and approaching the top-1 accuracy of 76.3% of the same ResNet-50 trained with all labels. Coaching also scales to the high data regime. In particular, with all 1.28 million labeled examples from ImageNet, plus 9 million unlabeled and potentially out-of-distribution data from OpenImages (Kuznetsova et al., 2018) , a ResNet-50 can be coached to the accuracy of 82.34%, which is a new state-of-the-art for the architecture without using extra labeled data.

2. METHOD:

Notations. Let T , S respectively be the teacher network and the student network in Coaching, and θ T , θ S be their corresponding parameters. Since we work with both labeled data and unlabeled data, we use (x lab , y lab ) to refer to a pair of an input and its corresponding label, and use x unl to refer to an unlabeled example. In addition, we use (x, y; θ) to denote the cross entropy loss computed on input x by with parameter θ on label y.

As shown in Figure 1 , each training step in Coaching consists of two phases:

Phase 1: The student learns from data pseudo labeled by the teacher. In this phase, the teacher T first performs a forward pass on x unl to compute the class distribution P (•|x unl ; θ T ). From this distribution, the teacher samples a pseudo label ŷunl ∼ P (•|x unl ; θ T ). The pair x unl , ŷunl is then shown to the student S to make an update on its parameters θ S . The update is based on the gradient computed by back-propagating from the cross entropy loss. For instance, if θ S is updated using SGD, then:

EQUATION

where η is the learning rate.

Phase 2: The teacher learns from the student's loss. After the student updates its parameters as in Equation 1, its parameters θ

(t+1) S

is evaluated on a labeled example x lab , y lab using the cross entropy loss. The goal of the teacher in Coaching is to give the pseudo labels ŷunl such that if the student is updated as in Equation 1, then the cross entropy loss (x lab , y lab ; θ (t+1) S

) will be minimized.

Clearly, (x lab , y lab ; θ

(t+1) S ) depends on θ (t+1) S

, which in turn depends on the pseudo label ŷunl that the teacher samples. From the perspective of reinforcement learning, ŷunl can be treated as an onpolicy action of the teacher, which leads to the reward of -(x lab , y lab ; θ (t+1) S

). In this perspective, we propose to train θ T to minimize the value of (x lab , y lab ; θ(t+1)

S:

), where θ(t+1)

is the expected destination that the teacher will guide the student to. This expectation is taken over all possible pseudo labels ŷunl . Formally,

θ * T = arg min θ T R(θ T ) where R(θ T ) = x lab , y lab ; E ŷunl∼P (•|xlab;θ T ) θ (t+1) S (2)

To find θ * T , we differentiate R(θ T ) in Equation 2 with respect to θ T . Here, we present the resulting gradient g

(t):

T , which has the form

EQUATION

The full derivation can be found in Appendix A, but intuitively, the differentiation depends on two tools. The first tool is the is the chain rule, which we leverage to differentiate R(θ T ) with respect to θ T . The second tool is the REINFORCE equation (Williams, 1992) , which we leverage to establish the relationship between E ŷunl θ (t+1) S and θ T .

Coaching combines the two steps above in an SGD step. We summarize the method in Algorithm 1.

Algorithm 1 The Coaching method.

Input :Labeled data xlab, ylab and unlabeled data xunl.

Initialize θ

T and θ 

θ (t+1) S := θ (t) S -η • g (t) S Compute g (t)

S with pseudo labels as in Equation 1 and update θS θ

(t+1) T := θ (t) T -η • h (t) • g (t) T

Compute the gradient g

(t)

T as in Equation 3 and update θT end return θ (N)

Only the student model is used for predictions and evaluations Generalize to an arbitrary batch size. Above, we have only discussed Coaching for a single unlabeled data x unl and a single labeled data x lab , y lab . Now, we describe how to scale Coaching to an arbitrary batch size. Scaling x lab , y lab to a minibatch of labeled example, X lab , Y lab is straightforward, as we can simply replace all computations of the cross entropy (x lab , y lab ; θ (t+1) S

) with the average cross entropy on the minibatch (X lab , Y lab ; θ

(t+1) S:

). To scale a single unlabeled example x unl to a minibatch of unlabeled examples

X unl = {x (1) unl , x (2) unl , ..., x (B)

unl }, we treat each batch of pseudo labels Ŷunl as a compound action sampled from the joint distribution

EQUATION

Since every pseudo label ŷ(i) unl is sampled independently, applying REINFORCE as in Equation 3 simply factors the per-instance cross entropy (x unl , ŷunl ; θ T ) into the batch cross entropy

B i=1 (x (i) unl , ŷ(i) unl ; θ T ).

3. EXPERIMENTS:

Compared to other methods that use both labeled data and unlabeled data, Coaching has three main advantages:

1. The teacher does not only demonstrate its knowledge to the student but also adjusts its teaching strategy in an adaptive manner with the student, throughout the course of the student's learning. 2. The teacher in Coaching can benefit from advanced SSL techniques such as consistency regularization. 3. The student in Coaching never learns directly from labeled data. This does not only prevent overfitting when limited labeled data is available, but also allows us to finetune the trained student in Coaching directly on labeled data to further boost the student's performance.

We perform experiments to verify the strength of Coaching. In Section 3.1, we consider the low data regime with typical benchmarks for SSL methods. After that, in Section 3.2, we consider the high data regime which contains potentially out-of-distribution data. Additional Implementation Details. To improve the stability and accuracy of the method, we apply a few minor enhancements to the teacher:

Model:

1. Use cosine distance instead of dot product. As the dot product h (t) in Equation 3 has a large value range, in order to stabilize training, we compute h (t) using the gradients' cosine distance.

2. Use a baseline for h (t) . To further reduce the variance of h (t) , we maintain a moving average b of h (t) and subtract b from h (t) every time we compute g

(t)

T as in Equation 3. 3. Additional supervised loss for the teacher. We find that adding the supervised loss (x lab , y lab ; θ T )

to the teacher's objective results in a faster learning and better student.

4. Consistently regularize the teacher. In the low data regime, consistency regularization improves the teacher and the student. More details are in Section 3.1.

5.:

Pre-training the teacher. When the number of classes is large, it is beneficial to initialize the teacher with a trained model so that the pseudo labels are better than random at the beginning of the student's learning. If we pre-train the teacher, Point 3 has minimal effect.

6. Finetuning the student. Since the student in Coaching only learns from unlabeled data and pseudo labels generated by the teacher, finetuning a converged student on labeled data often improves the student's performance.

The details mentioned above are mutually orthogonal. Since (1) and ( 2) are crucial to stabilize the Coaching process, they are always used in our experiments. In addition, we apply ( 3) and ( 4) to the low data regime, and apply (5) for the high data regime for computational efficiency and strong performance. We will explain these decisions in the corresponding sections. 2019)). We discuss these baselines more in Section 4. We choose these baselines for three reasons. First, the purely supervised baseline serves to verify our implementation and to demonstrate the overfitting of our models when labeled data is scarce. Second, comparing Coaching with Pseudo-Label confirms the benefits of continuing to train the teacher throughout the course of the student's learning. Finally, we compare against UDA because is the state-of-the-art on the datasets that we consider.

To ensure a fair comparison, we re-implement these baselines in our environment. We follow 

3.2. RESULTS ON HIGH DATA REGIME:

We have seen Coaching achieves strong performance for low data image classification tasks. Another aspect of these tasks is that the unlabeled data also come from the same domain as the labeled data, which is a restricted assumption. In this section, we show that Coaching also excels in the regime where we have a large labeled dataset and an order of magnitude more unlabeled data. In this regime, we also test the performance of our method when the unlabeled set may have out-of-domain images, i.e., the images belong to categories that do not exist in ImageNet. Results. We present our results in Table 2 . As can be seen, Coaching outperforms all relevant SSL baselines. Specifically, for the image size of 224, Coaching outperforms Billion-scale SSL by about 1% top-1 accuracy, even though Billion-scale SSL uses 10 times more unlabeled data. Meanwhile, for the image size of 331, Coaching achieves the top-1 accuracy of 79.86%, comfortably outperforming the top-1 accuracy of 79.04% by UDA. This improvement is particularly significant, since Coaching simply uses all data from OpenImages, while UDA has to select and balance the class distribution of their unlabeled data using a pre-trained teacher. This difference suggests that the teacher in Coaching can give helpful pseudo labels to the student, even on potentially out-of-distribution data.

Datasets:

It is worth mentioning that Coaching also outperforms the strong supervised baselines of DropBlock and FixRes, and is on par with FixRes+CutMix. However, DropBlock and CutMix are both regularization techniques orthogonal to Coaching. Similar to consistency regularization in Section 3.1, these techniques can be incorporated into the teacher in Coaching to improve performance. Insights about Coaching on OpenImages. Figure 2 shows five images taken from OpenImages, along with their OpenImages tags and the top 5 classes predicted by a teacher trained on ImageNet.

From the figure, we can see that there are non-trivial overlapping contents between the OpenImages tags and the ImageNet top classes, such as sunglasses in the first image. We also see that for the images whose contents match stronger with an ImageNet class, such as the first and the third image, the entropy of the teacher's prediction is smaller. As a result, when the teacher samples a pseudo label from these distribution, contents similar to an ImageNet class will receive more consistent labels, while content alien to ImageNet will have higher entropy on their labels. We suspect this is why a teacher trained on ImageNet can teach a student via pseudo labels on OpenImages. 

3.3. ANALYSIS:

Ablation Study of Implementation Details. To understand the contribution of each implementation detail of Coaching, we study their contributions on top of a purely supervised model. We conduct this study on ImageNet-10% and visualize the results in Figure 3 . From the figure we see that RandomAugment and UDA both improve the final accuracy significantly, respectively by 3.13% and 7.19% top-1 accuracy. On top of UDA, Coaching delivers a smaller improvement of 4.32% top-1 accuracy. However, since UDA's accuracy is already high, we believe that the improvement of 4.32% top-1 accuracy is significant. Finally, finetuning only slightly improves over Coaching. However, this extra boost is a unique advantage of Coaching: it is possible for the student in Coaching to finetune on labeled data because the student never directly learns from these labeled data.

Coaching overfits less than Supervised Learning. In our Coaching framework, the student never directly learns from labeled data. This behavior is helps the student to avoid overfitting, especially when labeled data is scarce. In Figure 4 , we visualize the training accuracy of Coaching and Supervised Learning on CIFAR-10 with 4,000 labels and on ImageNet with 10% labels. As shown, the training accuracy of both the teacher and the student of Coaching stay relatively low. Meanwhile, the training accuracy of the supervised model eventually reaches 100% and causes overfitting. 

4. RELATED WORK:

Pseudo-Label. Pseudo-Label (Lee, 2013) is one of the simplest semi-supervised learning algorithms: First, a teacher model is trained on labeled data. Then, the converged teacher model generates pseudo labels for unlabeled data. These unlabeled data and their pseudo labels are combined with the labeled data to train another model, which is called the student model. An inherent weakness of Pseudo-Label is that once the teacher generates an incorrect pseudo label for an unlabeled datum, the student can only naively learn from this wrong label. This phenomenon is called the confirmation bias. Arazo et al. ( 2019) addressed the confirmation bias by generating soft labels from the teacher and by adding noise to these labels. However, this is a manual fix from an outside model designer.

The main difference between Pseudo-Label and Coaching is that in Coaching, the teacher is trained along with the student throughout the course of training. This allows wrong knowledge learned by the teacher to be fixed in an end-to-end manner, leading to stronger performances.

Semi-supervised Learning (SSL). Pseudo-Label belongs to a more general group of algorithms known as Semi-supervised Learning. Unlike Pseudo-Label, typical SSL methods combine both labeled and unlabeled data to train a single model. Hence, the objective function of SSL is typically the sum of a supervised loss and an unsupervised loss. The supervised loss is often the cross-entropy computed on the labeled data. Meanwhile, the unsupervised loss can be a self-supervised loss ( Learning is that in Coaching, the pseudo labels are chosen to improve the student, and hence there is no need for an outer loop. We suspect this is an advantage of our method, since gradients to be very powerful for models to navigate in the parameter space.

5. CONCLUSION:

In this paper, we proposed the Coaching method for semi-supervised learning. Key to Coaching is the idea that the teacher learns from the student's loss and improves itself to generate pseudo labels in a way that helps student's learning the most. The learning process in Coaching consists of two main updates: updating the student based on the pseudo labeled data produced by the teacher and updating the teacher based on the student's performance. Experiments on standard CIFAR-10 and SVHN show that Coaching is much better than supervised learning and consistenly better than other semi-supervised learning methods. Coaching scales well to large problems, and successfully uses out-of-domain data to improve ImageNet classification.

A DERIVATION OF THE TEACHER'S UPDATE RULE:

In this section, we present the detailed derivation of the Teacher's update rule in Equation 3 from Section 2.

Mathematical Notations and Conventions. Since we will work with the chain rule, we use the standard Jacobian notations. 3 Specifically, for a differentiable function f : R m → R n , and for a vector x ∈ R m , we use the notation ∂f ∂x ∈ R n×m to denote the Jacobian matrix of f , whose dimension is n × m. Additionally, when we mention the Jacobian of a function f at multiple points such as x 1 and x 2 , we will use the notations of ∂f . Furthermore, by mathematical conventions, a vector v ∈ R n is treated as a column matrix -that is, a matrix of size n × 1. For this reason, the gradient vector of a multi-variable real-valued function is actually the transpose of of its Jacobian matrix.

Finally, all multiplications in this section are standard matrix multiplications. If an operand is a vector, then as discussed in the previous paragraph, the operand is treated as a column matrix.

Dimension Annotations. Understanding that these notations and conventions might cause confusions, in the derivation below, we annotate the dimensions of the computed quantities to ensure that there is no confusion caused to our readers. To this end, we respectively use |S| and |T | to denote the dimensions of the parameters θ S , θ T . That is, θ S ∈ R |S|×1 and θ T ∈ R |T |×1 .

We now present the derivation. We need to compute:

EQUATION

To simplify our notation, let us define θ(t+1)

EQUATION

Then, by the chain rule, we have

EQUATION

The first factor in Equation 7 can be simply computed via back-propagation. We now focus on the second term. We have

EQUATION

Note that in Equation 8 above, the Jacobian of (x unl , ŷunl ; θ S ), which has dimension 1 × |S|, needs to be transposed to match the dimension of θ (t) S , which, as we discussed above, conventionally has dimension |S| × 1.

Now, since θ (t):

S in Equation 8does not depend on θ T , we can leave it out of subsequent derivations. Also, to simplify notations, let us define the gradient

EQUATION

Then, Equation 8becomes

EQUATION

S (ŷ unl ) has no dependency on on θ T , except for via ŷunl , we can apply the REINFORCE equation (Williams, 1992) to achieve

EQUATION

Here, the last equality in Equation 11 is is due to the definition of the cross entropy loss, which is the negative of the log-prob term in the previous line. Now, we can substitute Equation 11 into Equation 7 to obtain

EQUATION

Finally, if we use Monte Carlo approximation for every term in Equation 12 using the sampled ŷunl , then we have Equation 3 from Section 2. Note that in Section 2, we use the gradient notation, which results in the transposes.

B TRAINING SPEED:

Coaching performs up to 5 forward passes and 3 backward passes. Compared to vanilla backpropagation training, this is more 3 than times more expensive in FLOPs. However, many computations in Coaching are parallelizable. For example, the forward pass of the student and the forward pass for the teacher on unlabeled data (the top half of Figure 1 ), can be run in parallel since they do not depend on each other. Therefore, on computing hardware with sufficient memory, we find Coaching to be between 2 and 2.5 times slower than standard back-propagation training.

transformation. This is to avoid a few degenerating cases. For instance, when Brightness is applied twice on an image, both times with small levels, the image will become almost black.

In our experiments, RandomAugment's performance is not far behind compared to AutoAugment. For example, on full ImageNet with ResNet-50, RandomAugment achives 77.98% top-1 accuracy, which is close to the top-1 accuracy of 77.6% reported by Cubuk et al. (2019).

C.3 HYPER-PARAMETERS:

To tune hyper-parameters, we follow Oliver et al. ( 2018) and allow each method to have 128 trials of hyper-parameters. When we tune, we let each model train for up to 50,000 steps. The optimal hyper-parameters are then used to run experiments that last for much more steps, as we report below.

In our experiments with Coaching, training for more steps typically leads to stronger results. We stop at 1 million steps for CIFAR-10 and SVHN, and at 0.5 million steps for ImageNet simply because otherwise, these experiments will take too long. Meanwhile, in our experiments with purely supervised learning, Pseudo-Labels, and UDA, training for more steps overfits the models, and we have to employ early stopping.

We report the hyper-parameters for our baselines and for Coaching in Section 3.1. For the highresource experiments in Section 3.2, we use the same hyper-parameters, because tuning them is too expensive. Our hyper-parameters can be found in 2019), we do not use a larger batch size for the UDA objective. Instead, we use the same batch size for both the labeled objective and the unlabeled objective. This is to avoid instances where some particularly small batch sizes for the labeled objective cannot be split on our computational hardware.

:

Code will be made available up on the paper's acceptance.

Joint Foto Tree. A proprietary image dataset of Google, which is not available to public

Standard: https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant

Figures and Tables:

Figure 1: Each step of gradient descent in Coaching consists of two steps. Updating the Student (top): The teacher network T samples the labels ŷ of unlabeled data xunl for the student S to learn from. Updating the Teacher (bottom): The teacher updates itself using policy gradient to improve the student's performance on labeled data xlab.

t = 0 to N -1 do Sample an unlabeled example xunl and a labeled example xlab, ylab Sample ŷunl ∼ P (•|xunl; θT )

Figure 2: An illustration of why OpenImages help ImageNet classification. Top: OpenImages tags. Middle: A sample image from OpenImages. Bottom: Top 5 labels for the image predicted by a teacher ResNet-50 trained on ImageNet. Some OpenImages tags overlap significantly with some ImageNet classes, such as wheel and car wheel in the second image. The class predictions also have a higher entropy when the ImageNet classes overlap less with the OpenImages contents (images 2, 4, 5), than when the ImageNet classes overlap more (images 1, 3).

Figure 3: Breakdown of the gains of different components in Coaching. The gain of Coaching over UDA, albeit smaller than the gain of UDA over RandomAugment, is significant as UDA is already very strong.

Oliver et al. (2018)'s train/eval/test splitting, and we use the same amount of resources to tune hyperparameters for our baselines as well as for Coaching. More details are in Appendix C. Data augmentations. In our implementation of UDA and Coaching, we use RandomAugment, which is a randomized augmentation strategy over all the operations in the search space of AutoAugment (Cubuk et al., 2019). We use RandomAugment because it is simple to implement, requires no expensive search, and achieves similar performance compared to UDA with AutoAugment. More details of RandomAugment can be found in Appendix C.2. Image For CIFAR-10 and SVHN, all existing better results use a larger model and more advanced regularization techniques. For instance, Xie et al. (2019) reports 97.3% with UDA(Xie et al., 2019), but their backbone model is PyramidNet, which has 18× more parameters than WideResNet-28-2 and they train with Shake-Drop regularization(Yamada et al., 2018). Similarly, for ImageNet-10%, the only better published result is 73.21% top-1 accuracy, achieved by MOAM-S 4 L (Zhai et al., 2019). This accuracy is only slightly better than Coaching's 72.94%, but uses a 4× wider ResNet-50. We believe that the enhancements in architectures, regularization techniques, and model sizes, can be applied to Coaching to further improve our results.

Table content: <table><tr><td>Methods</td><td>CIFAR-10 (4,000)</td><td>SVHN (1,000)</td><td>ImageNet (10%)</td></tr><tr><td colspan="4">Purely Supervised on full dataset 94.92 ± 0.17 97.41 ± 0.16 76.89/93.27</td></tr><tr><td>Temporal Ensemble</td><td colspan="2">83.63 ± 0.63 92.81 ± 0.27</td><td>-</td></tr><tr><td>Mean Teacher</td><td colspan="2">84.13 ± 0.28 94.35 ± 0.47</td><td>-</td></tr><tr><td>VAT + EntMin</td><td colspan="2">86.87 ± 0.39 94.65 ± 0.19</td><td>-/83.39</td></tr><tr><td>LGA + VAT</td><td colspan="2">87.94 ± 0.19 93.42 ± 0.36</td><td>-</td></tr><tr><td>ICT</td><td colspan="2">92.71 ± 0.02 96.11 ± 0.04</td><td>-</td></tr><tr><td>MixMatch</td><td colspan="2">93.76 ± 0.06 96.73 ± 0.31</td><td>-</td></tr><tr><td>Purely Supervised</td><td colspan="3">82.14 ± 0.25 88.17 ± 0.47 57.75/80.23</td></tr><tr><td>Pseudo Labels</td><td colspan="3">83.79 ± 0.11 89.81 ± 0.41 58.21/82.19</td></tr><tr><td>UDA (our implementation)</td><td colspan="3">94.53 ± 0.18 97.11 ± 0.17 68.07/88.19</td></tr><tr><td>Coaching</td><td colspan="3">95.60 ± 0.19 97.79 ± 0.11 72.39/90.52</td></tr><tr><td>Coaching + Finetune</td><td colspan="3">96.11 ± 0.07 98.01 ± 0.07 72.94/90.80</td></tr></table>

. We experiment with all labeled examples in ImageNet. Additionally, we take unlabeled images from the entire 4 th version of OpenImages dataset(Kuznetsova et al., 2018), which has 9 million natural images. A few samples from OpenImages can be found in Figure2. Unless otherwise specified, for both datasets, we use the image size of 224 × 224.Baselines. Since this regime of high data has not been extensively studied, we are only aware of two relevant, strong baselines. Our first baseline is Billion-scale Semi-supervised Learning (Billion-scale SSL; Yalniz et al. (2019)). Billion-scale SSL uses unlabeled data from the YFCC100M dataset(Thomee et al., 2015), studies several self-training settings, with various model architectures for teachers and students. Here, we restrict our comparison to the settings that use ResNet-50 for both the teacher and the student. Our second baseline is UDA(Xie et al., 2019), for which the authors select unlabeled images algorithmically from the JFT dataset.2   Other than these baselines, we compare Coaching to techniques that enhance supervised learning, such as DropBlock(Ghiasi et  al., 2018), CutMix (Yun et al., 2019), and FixRes (Touvron et al., 2019).Implementation details. We implement Coaching the same as in Section 3.1, except for one part: Instead of directly training and consistently regularizing the teacher, we initialize the teacher using a pre-trained ResNet-50 (pre-trained on full ImageNet). Then, throughout the course of the student's learning, we only train the teacher to minimize the student's cross entropy loss. We do not use additional supervised loss for the teacher because because once the teacher is pre-trained, adding another loss to the teacher has minimal effect. We do not consistently regularize the teacher because Xie et al. (2019) has found that consistency regularization requires in-domain data, while we do not filter our unlabeled images from OpenImages. Image classification accuracy with full ImageNet plus unlabeled images. Results are organized by image size because image size has a strong impact on models' performance.

Table content: <table><tr><td>Methods</td><td>Unlabeled images</td><td colspan="2">Image size</td><td colspan="2">Top-1 Top-5</td></tr><tr><td/><td/><td colspan="2">Train Test</td><td/><td/></tr><tr><td>Supervised</td><td>None</td><td>224</td><td colspan="3">224 76.89 93.27</td></tr><tr><td>DropBlock</td><td>None</td><td>224</td><td colspan="3">224 78.35 94.15</td></tr><tr><td>FixRes + CutMix</td><td>None</td><td>224</td><td>320</td><td>79.8</td><td>94.9</td></tr><tr><td>Coaching</td><td>OpenImages</td><td>224</td><td colspan="3">320 79.80 94.87</td></tr><tr><td>FixRes</td><td>None</td><td>224</td><td>384</td><td>79.1</td><td>94.6</td></tr><tr><td>Coaching</td><td>OpenImages</td><td>224</td><td colspan="2">384 80.10</td><td>95.07</td></tr><tr><td>Billion-scale SSL</td><td>YFCC 100M</td><td>224</td><td>224</td><td>77.6</td><td>-</td></tr><tr><td>Coaching</td><td>OpenImages</td><td>224</td><td colspan="2">224 78.62</td><td>94.26</td></tr><tr><td>UDA</td><td>JFT</td><td>331</td><td colspan="2">331 79.04</td><td>94.45</td></tr><tr><td>Coaching</td><td>OpenImages</td><td>224</td><td colspan="2">331 79.86</td><td>94.92</td></tr><tr><td>Coaching+iterative</td><td>OpenImages</td><td>224</td><td colspan="2">331 82.34</td><td>96.09</td></tr></table>

Comparing to state-of-the-art SSL results.Yalniz et al. (2019) reports the top-1 accuracy of 81.2% for a ResNet-50 student. However, they need to pre-train a much bigger network ResNext-101-32x48 teacher (829 million parameters, 32x larger than ResNet-50) on 1 billion Instagram images with weak labels(Mahajan et al., 2018). Then, they use the pseudo-labels from this teacher to train a ResNet-50 student for 2 billion steps. The fact that they use weakly labeled data from Instagram, much bigger architecture in ResNext-101-32x48 makes their results not directly comparable to ours.Meanwhile, without the need of a much bigger dataset and architecture as used inYalniz et al. (2019), Coaching achieves almost as good top-1 accuracy. To achieve this, we iterate the process of Coaching by turning the student into the teacher after convergence. After 17 iterations, our final student achieves 82.34% top-1 accuracy on ImageNet, outperforming Yalniz et al. (2019)'s 81.2%, even though we do not have the weakly labeled data from Instagram.

Table content: <table/>

Rasmus et al., 2015; Noroozi & Favaro, 2018; Gidaris et al., 2018), or consistency regularization (Laine & Aila, 2017; Tarvainen & Valpola, 2017; Miyato et al., 2018; Berthelot et al., 2019; Xie et al., 2019). Self-supervised losses typically encourage the model to develop a common sense about the images. Meanwhile, consistency regularization enforces that the model is invariant against certain transformations of the data. The main difference between Coaching and SSL methods is that the student in Coaching never learns directly from labeled data. This helps the student in Coaching to avoid overfitting to labeled data, especially when labeled data is limited.Meta Learning. In Meta Learning, there is typically an outer loop that optimizes the performance of a model trained in an inner loop(Finn et  al., 2017; Metz et al., 2019). Meta Learning has been applied to perform self-training and SSL in the low data regime (Agarwal et al., 2019; Ren et al., 2018; Boney & Ilin, 2018; Hsu et al., 2019). A crucial difference between Coaching and Meta

Table content: <table/>

, 5, 6. We note that our settings for UDA is different from originally reported by Xie et al. (2019). In their work, Xie et al. (2019) use a much larger batch size for their UDA objective. In our implementation of UDA, we keep these batch sizes the same. This leads to a much easier implementation of data parallelism in our framework, TensorFlow (Abadi et al., 2016) running on TPU big pods. To compensate for the difference, we train all UDA baselines for much longer than Xie et al. (2019). During the training process, we also mask out the supervised examples with high confidence. Effectively, our UDA model receives roughly the same amount of training with labeled examples and unlabeled examples as the models in Xie et al. (2019). We have also verified that on ImageNet-10% with the augmentation policy from AutoAugment (Cubuk et al., 2019), our UDA implementation achives 68.77% top-1 accuracy, which is similar to 68.66% that Xie et al. (2019) reported.

Table content: <table><tr><td>Hyper-parameter</td><td>CIFAR-10</td><td>SVHN</td><td>ImageNet</td></tr><tr><td>Weight decay</td><td>0.0005</td><td>0.001</td><td>0.0002</td></tr><tr><td>Label smoothing</td><td>0</td><td>0</td><td>0.1</td></tr><tr><td>Batch normalization decay</td><td>0.99</td><td>0.99</td><td>0.99</td></tr><tr><td>Learning rate</td><td>0.4</td><td>0.05</td><td>1.28</td></tr><tr><td>Number of training steps</td><td>50,000</td><td>50,000</td><td>40,000</td></tr><tr><td>Number of warm up steps</td><td>2500</td><td>0</td><td>2000</td></tr><tr><td>Batch size</td><td>1024</td><td>128</td><td>2048</td></tr><tr><td>Dropout rate</td><td>0.4</td><td>0.5</td><td>0.2</td></tr><tr><td>Pseudo label threshold</td><td>0.95</td><td>0.975</td><td>0.7</td></tr></table>

Hyper-parameters for supervised learning and Pseudo-Labels.

Table content: <table><tr><td>Hyper-parameter</td><td>CIFAR-10</td><td>SVHN</td><td>ImageNet</td></tr><tr><td>Weight decay</td><td>0.0005</td><td>0.0005</td><td>0.0002</td></tr><tr><td>Label smoothing</td><td>0</td><td>0</td><td>0.1</td></tr><tr><td>Batch normalization decay</td><td>0.99</td><td>0.99</td><td>0.99</td></tr><tr><td>Learning rate</td><td>0.3</td><td>0.4</td><td>1.28</td></tr><tr><td>Number of training steps</td><td>1,000,000</td><td>1,000,000</td><td>500,000</td></tr><tr><td>Number of warm up steps</td><td>5,000</td><td>5,000</td><td>5,000</td></tr><tr><td>Batch size</td><td>128</td><td>128</td><td>2048</td></tr><tr><td>Dropout rate</td><td>0.5</td><td>0.6</td><td>0.25</td></tr><tr><td>UDA factor</td><td>2.5</td><td>1</td><td>20</td></tr><tr><td>UDA temperature</td><td>0.7</td><td>0.8</td><td>0.7</td></tr></table>

Hyper-parameters for UDA. Unlike originally done by Xie et al. (

Table content: <table/>

Peer review (iclr2020):
This paper studies the teacher-student models in semi-supervised learning. Unlike previous methods in which only the student will learn from the teacher, this paper proposes a method to let the teacher learn from the student by reinforcement learning. Experimental results demonstrate the proposal’s performance. The paper achieves some good empirical results compared to other baselines. However, the proposed method is implemented with many tricks listed on Page 4, and with data augmentation techniques, which may not be used in previous methods. Additionally, the paper is weak in technology. There is no clear explanation of why the proposed method works except a metaphor for sports coaches. I vote for a clear rejection of the paper. First, the paper is weak in experiments. It works hard to achieve a good experimental result, through many tricks listed in the “Additional Implementation Details” in Page 4, and through the data augmentation used in Page 5. However, these tricks to improve the performance may not be used in previous methods, as the paper does not run experiments on baselines under the same setting, but use the results reported in Oliver et al. (2018). Additionally, the paper only uses one number of labeled data for each data set, it makes readers doubt that the proposed method only works under this number of labeled data. The paper fails to clearly state why we need to let the teacher learn from the student. Actually, I doubt if this is necessary. Given the strong learning capacity of neural networks, the proposed method will easily be overfitting. Assume we have a very weak student network at the beginning, then by training in the way proposed in the paper, the teacher network will have all labeled data classified correct, and all unlabeled data classified into the same labels as the student network. I cannot see from the simple proposal why such overfitting can be avoided. The paper is weak in both technology and experiments. It is also poorly written without clearly stating the motivation for this problem. I would vote for a reject for the paper. ------------------------------------------- The rebuttal has cleared some of my concerns. However, it is still not clear why the proposed method work and how does it prevents overfitting. The paper also needs more experimental results to confirm its effectiveness. I will increase my score a little bit, but would not vote for an accept this time. But I believe with further revision, the paper may be worth publishing in the future, if the questions in all reviews can be addressed.