Semi-supervised Learning by Coaching
Authors: Hieu Pham
Quoc V. Le
Venue:  (2020)
Keywords: semi-supervised
teacher
student
label propagation
image classification

Abstract: Recent semi-supervised learning (SSL) methods often have a teacher to train a student in order to propagate labels from labeled data to unlabeled data. We argue that a weakness of these methods is that the teacher does not learn from the student’s mistakes during the course of student’s learning.  To address this weakness, we introduce Coaching, a framework where a teacher generates pseudo labels for unlabeled data, from which a student will learn and the student’s performance on labeled data will be used as reward to train the teacher using policy gradient.

Our experiments show that Coaching significantly improves over state-of-the-art SSL baselines. For instance, on CIFAR-10, with only 4,000 labeled examples, a WideResNet-28-2 trained by Coaching achieves 96.11% accuracy, which is better than 94.9% achieved by the same architecture trained with 45,000 labeled. On ImageNet with 10% labeled examples, Coaching trains a ResNet-50 to 72.94% top-1 accuracy, comfortably outperforming the existing state-of-the-art by more than 4%. Coaching also scales successfully to the high data regime with full ImageNet. Specifically, with additional 9 million unlabeled images from OpenImages, Coaching trains a ResNet-50 to 82.34% top-1 accuracy, setting a new state-of-the-art for the architecture on ImageNet without using extra labeled data.

Under review as a conference paper at ICLR 2020
SEMI-SUPERVISED LEARNING BY COACHING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent semi-supervised learning (SSL) methods often have teacher to train a student in order to propagate labels from labeled data unlabeled data. We argue that a weakness of these methods is that the teacher does not learn from the student's mistakes during the course of student's learning. To address this weakness, we introduce Coaching, a framework where a teacher generates pseudo labels for unlabeled data, from which a student will learn and the student's performance on labeled data will be used as reward to train the teacher using policy gradient.
Our experiments show that Coaching significantly improves over state-of-the-art SSL baselines. For instance, CIFAR- 10, with only 4,000 labeled examples, WideResNee-22822 trained by Coaching achieves 96.11% accuracy, which is better than 94.9% achieved by the same architecture trained with 45,000 labeled. On Im- ageNet with 10% labeled examples, Coaching trains a ResNet-50 to 72.94% top-1 accuracy, comfortably outperforming the existing oftte-of-theart by more than 4%. Coaching also scales successfully to the high data regime with full ImageNet. Specifically, with additional 9 million unlabeled images from Openlmages, Coach- ing trains ResNet-50 to 82.34% top-1 accuracy, setting new state-of-the- for the architecture on ImageNet without using extra labeled data.
INTRODUCTION
Professional players in competitive sports such as chess, tennis, or swimming often have coaches to help improving their performance. Although coaches typically do not play as well as the players, they observe the players and provide instructions to improve the players' performance. Modern semi-supervised learning (SSL.) algorithms do not follow this strategy. They instead have teacher model that generates pseudo labels for unlabeled data, from which a student model learns by imitation (e.g., Lee (2013); Tarvainen & Valpola (2017); Laine Aila (2017)). weakness of these methods is that the teacher does not adjust itself based on the student's performance and cannot adapt make the student better over time, unlike professional sport coaches develop their players.
Here, we propose new semi-supervised learning method, called Coaching as shown in Figure 1, where the teacher learns throughout the course of student's 'ss training. In our method, a teacher generates pseudo labels for unlabeled data, from which the student will learn. The student's 's performance labeled data will be used as reward to train the teacher with policy gradient.
Updating the Student:
Uni labe led Data Ofuit?
Teacher
Labe Okybh
Pseudo
Supervised Learning Backprop
Student [5)
Updating the Teacher:
Labeled
Bata
Student (S]
Compute Cross
Entropy Loss
Cross en entropy Loss (R)
Policy Grodient with as Expected Reward
Teacher (T)
Figure 1: Each step of gradient descent in Coaching consists of two steps. Updating the Student (top): The teacher network T' samples the labels ý of unlabeled data Jami for the student S to learn from. Updating the Teacher (bottom): The teacher updates itself using policy gradient improve the student's performance labeled data Tia-
Code will he made available up the paper's acceptance.

---

Under review as a conference paper at ICLR 2020
Experiments show that our method achieves significant improvements state-of-the-are semi- supervised learning baselines and can be up to 10x more data efficient than supervised learning. For instance, with CIFAR-10 only using 4,000 labeled examples, WideResNet-282 can be coached to 96.1 16% accuracy, outperforming the same model trained with 45,000 labeled examples which achieves 94.9%. Meanwhile, ImageNet, using ResNet-50 with only 10% labeled examples, our method achieves 72.94% top-1 accuracy, outperforming all existing semi-supervised learning methods with the same amount of labeled data, and approaching the top-1 accuracy of 76.3% of the same ResNet-50 trained with all labels. Coaching also scales to the high data regime. In particular, with all 1.28 million labeled examples from ImageNet, plus million unlabeled and potentially out-of-distribun data from OpenImages (Kuznetsova et al., 2018), ResNet-50 can be coached to the accuracy of 82.34%, which a new state-of-the-an for the architecture without using extra labeled data.
2 METHOD
Notations. Let T, S respectively be the teacher network and the student network in Coaching, and or, As be their corresponding parameters. Since we work with both labeled data and unlabeled data, we use (Zlab, Mab)) to refer to a pair of an input and its corresponding label, and use Zunl to refer to an unlabeled example. addition, we use E(r, yı 0) to denote the cross entropy loss computed on input by with parameter 8 on label y.
As shown in Figure 1, each training step in Coaching consists of two phases:
Phase 1: The student learns from data pseudo labeled by the teacher. In this phase, the teacher T first performs a forward pass on Tunl to compute the class distribution P(-|Fuml: 0r). From this distribution, the teacher samples pseudo label Dunl P[-Iarun) 0T). The pair Tanl, Yuni is then shown to the student S to make an update on its parameters 0g. The update is based on the gradient computed by back prooppaaaggg from the cross entropy loss. For instance, if As is updated using SGD, then:
af(Tanl Funt; Os)
(1+1)
0's n
8 17 17.95
00g
08=0/
where 7j the learning rate.
Phase 2: The teacher learns from the student's loss. After the student updates its parameters as Equation 1, its parameters (((+1) is evaluated on a labeled example Waab Mah using the cross entropy loss. The goal of the teacher in Coaching is to give the pseudo labels Junl such that if the
student updated as in Equation then the cross entropy loss (Fiab. ylas; (((+1) will be minimized.
Clearly, (Tia, Mabi 0g o(++)) depends on 0 (t+1) which in turn depends on the pseudo label úuml that the teacher samples. From the perspective of reinforcement learning, thun can be treated as an on-
policy action of the teacher, which leads to the reward of -((rub Vlab; O ((t+1)) In this perspective,
we propose to train 0r to minimize the value of E(Maa,, Ylab: 0((++1),, where Ø((-1) is the expected destination that the teacher will guide the student to. This expectation is taken over all possible pseudo labels Jnnt- Formally,
8 arg min R(Or) where R(Or) =(
= -((aub Ylab: Egg.r (((awee))
To find 07, we differentiate R(Or) in Equation with respect to 0r. Here, we present the resulting
gradient 9T which has the form
(t)
(t) 9T n (o6O)² T. (
@6l Maabi 0s)
(F Dunt 6x)
00g
Ns
00T
Or o
(3)
scalar k(O)
2

---

Under review as a conference paper at ICLR 2020
The full derivation can be found in Appendix A, but intuitively, the differentiation depends on two tools. The first tool is the the chain rule, which we leverage to differentiate R(0r) with respect to 8r. The second tool is the REINFORCE equation (Williams, 1992), which we leverage to establish the relationship between E
g(+++)) and Br.
Coaching combines the two steps above in an SGD step. We summarize the method in Algorithm 1.
Algorithm The Coaching method.
Input :laabeled data Tlab: Mab and unlabeled data Funl-
Initialize 0 and 0 do Sample an unlaheled example Junl and labeled example Tab, Ylab
for t= 0 to
Sample Vanl ~ P(-[Funt; Pr)
0(++))
0
-7.98
Compute gss with pseudo labels as in Equation I and update 0s b Compute the gradient 97 as in Equation and update
of of++1) = 07 17. 10 9T
end
return 0g
Only the student model is used for predictions and evaluations
Generalize to an arbitrary batch size. Above, we have only discussed Coaching for single unlabeled data Tunl and single labeled data Tlab: Vlab- Now, we describe how to scale Coaching to an arbitrary batch size. Scaling Flab, Ylab minibatch of labeled example, Xlab, Ylab is straightforward, as we can simply replace all computations of the cross entropy [T Mari ( with the average cross entropy the minibatch 8 1(Xia,, Yan; 0(++))) To scale single unlabeled example Fan! to (B),
minibatch unlabeled examples Xunl = {I (1) .2(2) Funl }. we treat each batch of pseudo labels Yuni compound action sampled from the joint distribution
B II
P Xunii 0r)
(1) Yun! Demi
(B) ..... Punl Xunti 0r)
(1) ()))) 0r)
Manl F
Since every pseudo label ( is sampled independently, applying REINFORCE as in Equa- tion simply factors the per-instance cross entropy (TEntt Punt: 0r) into the batch cross entropy Σ ( o 0r).
EXPERIMENTS
Compared to other methods that use both labeled data and unlabeled data, Coaching has three main advantages:
1. The teacher does not only demonstrate its knowledge the student but also adjusts its teaching strategy in an adaptive manner with the student, throughout the course of the student's 's learning.
2. The teacher Coaching can benefit from advanced SSL techniques such as consistency regular- ization.
3. The student in Coaching never learns directly from labeled data. This does not only prevent overfitting when limited labeled data is available, but also allows us to finetune the trained student in Coaching directly labeled data to further boost the student's performance.
We perform experiments to verify the strength of Coaching. In Section 3. we consider the low data regime with typical benchmarks for SSL methods. After that, in Section 3.2, we consider the high data regime which contains potentially out-of cittof-diisributtin data.
Model Architectures. In our experiments, our teacher model and our student model always have the same architecture but with different weights. For CIFAR-10 10 and SVHN, 28-2 (Zagoruyko & Komodakis, 2016), which has 1.45 million parameters. For ImageNet, we use a ResNet-50 (He al., 2016), which has 25.5 million parameters. For experiments that train only one
the WideResNet-

---

Under review as a conference paper at ICLR 2020
model, we apply exponential moving average with decay rate of 0.99 on the weights of the model. For experiments that have a teacher model and a student model, we apply this exponential moving average on the weights of the student model only.
Additional Implementation Details. To improve the stability and accuracy of the method, we apply a few minor enhancements to the teacher:
1. Use cosine distance instead of dot product. As the dot product 1 in Equation 3 has a large value range, in order to stabilize training, we compute (()) using the gradients' cosine distance.
2. Use baseline for ((t,, To further reduce the variance of ft()), we maintain moving average b of (1)) and subtract from h every time we compute 9T as in Equation 3.
3. Additional supervised loss for the teacher. We find that adding the supervised loss (Tjnb Vlabi 0r) to the teacher's 's objective results in a faster learning and better student.
4. Consistently regularize the teacher: In the low data regime, consistency regularization improves the teacher and the student. More details are in Section 3.1.
5. Pre-training the teacher. When the number of classes is large, it is beneficial to initialize the teacher with a trained model so that the pseudo labels are better than random at the beginning of the student's s learning. If we pre-train the teacher, Point 3 has minimal effect.
6. Finetuning the student. Since the student in Coaching only learns from unlabeled data and pseudo labels generated by the teacher, finetuning converged student on labeled data often improves the student's 's performance.
The details mentioned above are mutually orthogonal. Since (1) and (2) are crucial to stabilize the Coaching process, they are always used in our experiments. In addition, we apply (3) and (4) to the low data regime, and apply (5) for the high data regime for computational efficiency and strong performance. We will explain these decisions in the corresponding sections.
3.1 RESULTS ON Low DATA REGIME
Datasets. We consider three datasets with reduced numbers of labeled instances: CIFAR- 10 (Krizhevsky, 2009) with 4,000 labeled examples, SVHN (Netzer al., 2011) with 1,000 labeled examples, and ImageNet (Russakovsky et al., 2015) with 128,000 labeled examples, which is approx- imately 10% of the whole ImageNet. All images in these datasets are used as unlabeled examples, which means that even the labeled images be used unlabeled examples. We the image size of 32 x 32 for CIFAR-10 10 and SVHN, and the image size of 224 x 224 for ImageNet. These datasets, label reductions, and image sizes standard for low data image classification.
Baselines. We compare Coaching against 3 baseline training algorithms Purely Supervised, Pseudo- Label (Lee, 2013), and Unsupervised Data Augmentation (UDA; Xie et al. (2019)). We discuss these baselines more in Section 4. We choose these baselines for three reasons. First, the purely supervised baseline serves to verify labeled data is scarce. Second, comparing Coaching with Pseudo-Labe confirms the benefits of continuing to train the teacher throughout the course of the student's 's learning. Finally, compare against UDA because is the state-of-the the- on the datasets that we consider.
implementation and to demonstrate the overfitting of our models when
To ensure fair comparison, re-implement these baselines in environment. We follow Oliver et al. (2018)'s train/eval/test splitting, and we use the same amount of resources to tune hyper- parameters for our baselines as well as for Coaching. More details are in Appendix C.
Additional baselines. addition to the three main baselines discussed above, we also include four other baselines: Temporal Ensemble (Laine & Aila, 2017), Mean Teacher (Tarvainen & Valpola, 2017), VAT (Miyato et al., 2018), LGA (Jackson & Schulman, 2019), ICT (Verma et al., 2019), and MixMatch (Berthelot et al., 2019). We use results reported by Oliver et al. (2018). Since these methods do not share the same controlled environment, the comparison to them is not direct, and should be contextualized suggested by Oliver al. (2018).
4

---

Under review as a conference paper at ICLR 2020
Data augmentations. In our implementation of UDA and Coaching, which is a randomized augmentation strategy over all the operations in the search space of AutoAug- ment (Cubuk et al., 2019). We use RandomAugment because it is simple to implement, requires no expensive search, and achieves similar performance compared to UDA with Augomng.ent. More details of RandomAugment can be found in Appendix C.2.
RandomAugment,
CIFAR-10 (4,000)
SVHN (1,000)
ImageNet (10%)
Methods
Purely Supervised on full dataset
94.92 + 0.17
97.41 0.16
76.89/93.27
Temporal Ensemble
83.63 + 0.63
92.81 + 0.27
Mean Teacher
84.13 + 0.28
94.35 + 0.47
VAT + EntMin
86.87 + 0.39
94.65 + 0.19
-883.99
LGA VAT
87.94 + 0.19
93.42 0.36
ICT
92.71 = 0.02
96.11 0.04
MixMatch
93.76 + 0.06
96.73 1 0.31
Purely Supervised
82.14 + 0.25
88.17 + 0.47
57.75/ 80.23
Pseudo Labels
83.79 1 0.11
89.81 + 0.41
58.21 82.
UDA (our implementation)
94.53 1 0.18
97.11 1 0.17
68.07 18.
Coaching
95.60+4 + 0.19
97.79 + 0.11
72.39/90.52 /90.52
Coaching + Finetune
96.11 + 0.07
98.01 +: 0.07
72.94/90.80
Table 1: Image Classification Accuracy on reduced CIFAR-10, SVHN, and ImageNet. Higher is better. For CIFAR-10 and SVHN, we report mean over runs, while for ImageNet, we report Top- 1/oop- accuracy of a single run. Results in the second block are taken from past papers, while the rest shares the same environment and parammerrtttrr settings. All methods share the same model architecture: 22deResNet-28-22 for CIFAR-10 and S SVHN, and ResNet-50 for ImageNet.
Main results. In Table 1, present main results before and after finetuning the student on labeled data. The results confirm that Coaching significantly outperforms UDA and other strong baselines in semi-supervised learning.
On CIFAR-10 10 and SVHN, compared to the state-of-the-art UDA, Coaching's error rate reduction roughly 30% and 10%. As UDA's accuracy is already relatively high, such error reductions significant. On CIFAR-10, Coaching is also the first approach to exceed supervised learning on the all labels by using merely 4,000 labeled examples. Meanwhile, outperforms UDA by almost 5% in top-1 accuracy, going from 68.07% to 72.94%. Even prior to finetuning on labeled data, Coaching still outperforms UDA and other baselines.
ImageNet- Coaching
Comparing to existing sate-of-theearr methods. To the best of our knowledge, Coaching has achieved new state-of-the-an performances among the same model architectures on three datasets considered in this section.
For CIFAR-10 and SVHN, all existing better results larger model and more advanced regular- ization techniques. For instance, Xie et al. (2019) reports 97.3% with UDA (Xie et al., 2019), but their backbone model PyramidNet, which has 18x more parameters than WideRessNet-8-22 and they train with Shake-Drop regularization (Yamada et al., 2018). Similarly, for Image 10%, the only better published result 73.21% top-1 accuracy, achieved by MOAM-S* (Zhai et al., 2019). This accuracy is only slightly better than Coaching's 72.94%, but uses 4x wider ResNet-50. We believe that the enhancements in architectures, regularization techniques, and model sizes, can be applied to Coaching to further improve our results.
3.2 RESULTS ON HIGH DATA REGIME
We have seen Coaching achieves strong performance for low data image classification tasks. Another aspect of these tasks is that the unlabeled data also come from the same domain the labeled data, which is a restricted assumption. In this section, we show that Coaching also excels in the regime where have large labeled dataset and order of magnitude we also test the performance of our method when the unlabeled set may have out-of- oft-of-domain images, Le, the images belong categories that do not exist in ImageNet.
unlabeled data. In this regime,
5

---

Under review as a conference paper at ICLR 2020
Datasets. We experiment with all labeled examples in ImageNet. Additionally, we take unlabeled images from the entire 4th version of OpenImages dataset (Kuznetsova et al., 2018), which has million natural images. A few samples from OpenImages can be found in Figure 2. Unless otherwise specified, for both datasets, we use the image size of 224 x 224.
Baselines. Since this regime of high data has not been extensively studied, of two relevant, strong baselines. Our first baseline is Billion-scale Semi-supervised Learning (Billion-scale SSL; Yalniz et al. (2019)). Billion-scale SSL uses unlabeled data from the YFCC100M dataset (Thomee et al., 2015), studies several self- self-training settings, with various model architectures for teachers and students. Here, we restrict our comparison to the settings that use ResNet-50 for both the teacher and the student. Our second baseline is UDA (Xie et al., 2019), for which the authors select unlabeled images algorithmically from the JFT dataset.
only aware
Other than these baselines, we compare Coaching to techniques that enhance supervised learning, such as DropBlock (Ghiasi al., 2018), CutMix (Yun al., 2019), and FixRes (Touvron al., 2019).
Implementation details. We implement Coaching the same as in Section 3.1, except for one part: Instead of directly training and consistently regularizing the teacher, we initialize the teacher using a pre-trained ResNet-50 (pre-trained on full ImageNet). Then, throughout the course of the student's learning, we only train the teacher to minimize the student's cross entropy loss. We do not use additional supervised loss for the teacher because because once the teacher is pre-trained, adding another loss the teacher has minimal effect. We do not consistently regularize the teacher because Xie et al. (2019) has found that consistency regularization requires in-domain data, while we do not filter our unlabeled images from OpenImages.
Image size
Methods
Unlabeled images
Top-1
Top-5
Train
Test
Supervised
None None
224
224
76.89
93.27
DropBlock
224
224
78.35
94.15
FixRes + CutMix
None
224
320
79.8
94..9
Coaching
OpenImages
224
320
79.80
94.87
FixRes Coaching
None
224
384
79.1
94.66
OpenImages
221
381
80.10
95.07
Billion-scale SSL
YFCC 100M
224
224
77.6
Coaching
OpenImages
224
221
78.62
94.26
UDA
JFT
331
331
79.04
94,45
Coaching
OpenImages
224
331
79.86
94.92
Coaching+iterative
OpenImages
224
331
82.34
96.09
Table 2:
Image classification accuracy with full ImageNet plus unlabeled images. Results are organized by image size hecause image size has a strong impact on models' performance.
Results. We present our results in Table 2. As can be seen, Coaching outperforms all relevant SSL baselines. Specifically, for the image size of 224, Coaching outperforms Billion-scale SSL by about 1% top-1 accuracy, even though Billion-scale SSL uses 10 times more unlabeled data. Meanwhile, for the image size of 331, Coaching achieves the top-1 accuracy of 79.86% confortably outperforming the top-1 accuracy of 79.04% by UDA. This improvement is particularly significant, since Coaching simply uses all data from Openlmages, while UDA has to select and balance the class distribution of their unlabeled data using pre-trained teacher. This difference suggests that the teacher in Coaching can give helpful pseudo labels to the student, even on potentially dut-o--distribution data.
It is worth mentioning that Coaching also outperforms the strong supervised baselines of DropBlock and FixRes, and is on par with FixRes+CutMix. However, DropBlock and CutMix are both regular- ization techniques orthogonal to Coaching. Similar to consistency regularization in Section 3.1, these techniques can be incorporated into the teacher in Coaching to improve performance.
Joint Foto Tree. A proprictary image dataset of Gongle, which is not available to public

---

Under review as a conference paper at ICLR 2020
Comparing to state-of-the-art SSL results. Yalniz et al. (2019) reports the top-1 accuracy of 81.2% for a ResNet-50 student. However, they need to pre-train a much bigger network ResNext- 101-32x48 teacher (829 million parameters, 32x larger than ResNet-50) on 1 billion Instagram images with weak labels (Mahajan et al., 2018). Then, they use the pseudo-labels from this teacher to train a ResNet-50 student for 2 billion steps. The fact that they use weakly labeled data from Instagram, much bigger architecture in ResNext- 101-32448 makes their results not directly comparable to ours.
Meanwhile, without the need of a much bigger dataset and architecture as used in Yalniz et al. (2019), Coaching achieves almost as good top-1 accuracy. To achieve this, we iterate the process of Coaching by turning the student into the teacher after convergence. After 17 iterations, our final student achieves 32.34% top-1 accuracy on ImageNet, outperforming Yalniz et al. (2019)'s 's 81.2%, even though we do not have the weakly labeled data from Instagram.
Insights about Coaching on Openlmages. Figure shows five images taken from OpenImages, along with their OpenImages tags and the top 5 classes predicted by a teacher trained on ImageNet. From the figure, we can see that there are non-trivial overlapping contents between the OpenImages tags and the ImageNet top classes, such as sunglasses in the first image. We also see that for the images whose contents match stronger with an ImageNet class, such as the first and the third image, the entropy of the teacher's prediction is smaller. As result, when the teacher samples a pseudo label from these distribution, contents similar to an ImageNet class will receive more consistent labels, while content alien to ImageNet will have higher entropy their labels. We suspect this is why teacher trained on ImageNet can teach a student via pseudo labels on Openlmages.
person,
wheel,
person,
langlasses,
van,
tree,
man, human body
shorts
land vehicle
squirrel
man
school bus
beach wagon
fox squirrel
lakeside
paddlewheel
miniskirt
convertible
titi
valley
toyshop
sunglasses
golfcart
squirrel monkey
stone wall
confectionery
neck brace
car wheel
marmoset
black stork
gondola
moving van
grille
macaque
curly-coated retriever balance
An illustration of why OpenImages help ImageNet classification. Top: OpenImages tags. Middle: A sample image from Openlmages. Bottom: Top 5 labels for the image predicted by a teacher ResNet-50 50 trained on ImageNet. Some OpenImages tags overlap significantly with some ImageNet classes, such wheel and car wheel in the second image. The class predictions also have higher entropy when the ImageNet classes overlap less with the OpenImages contents (images 2, 4, 5), than when the ImageNet classes overlap more (images 1, 3).
Figure 2:
3.3 ANALYSIS
Ablation Study of Implementation Details. To understand the contribution of each implementation detail of Coaching, we study their contributions on top of a purely supervised model. We conduct this study on ImageNet-109% 10% and visualize the results in Figure 3. From the figure we see that RandomAugment and UDA both improve the final accuracy significantly, respectively by 3.13% and 7. 7.19% top-1 accuracy. On top of UDA, Coaching delivers smaller improvement of 4.32% top- accuracy. However, since UDA's accuracy already high, we believe that the improvement of 4.32% top-1 accuracy is significant. Finally, finetuning only slightly improves over Coaching. However, this extra boost a unique advantage of Coaching: it possible for the student Coaching to finetune on labeled data because the student never directly learns from these labeled data.
Coaching overfits less than Supervised Learning. In our Coaching framework, the student never directly learns from labeled data. This behavior is helps the student to avoid overfitting, especially when labeled data is scarce. In Figure 4, we visualize the training accuracy of Coaching and CIFAR-10 with 4,000 labels and on ImageNet with 10% labels. As shown,
Supervised Learning the training accuracy of both the teacher and the student of Coaching stay relatively low. Mcanwhile, the training accuracy of the supervised model eventually reaches 100% and causes overfitting.
7

---

Under review as a conference paper at ICLR 2020
Holst on Stucy on 10
1.00
Toi ring decuracy on CRRA 1040000
1.0
Telving Actinhy or imags 11%5
Parly Suceriass
Thaiting berurary
Teadher Staterr
M
#Coeching
Superviodd
1
6:25 Training Trahtin Progress
0:25
0.50 0.75
Figure 3: Breakdown of the gains of different components in Coaching. The gain of Coaching over UDA, albeit smaller than the gain of UDA over Rugdon, is significant as UDA is already very strong.
Thaining Progress
Figure 4: Training accuracy of Coaching and of supervised learning on (IFAR-10-4.00 and ImageNet- 10%. Both the teacher and the student in Coaching have lower training accuracy, effectively avoiding overfitling.
RELATED WORK
Pseudo-Label (Lee, 2013) is one of the simplest semi-supervised learning algo- rithms: First, a teacher model is trained on labeled data. Then, the converged teacher model generates pseudo labels for unlabeled data. These unlabeled data and their pseudo labels are combined with the labeled data to train another model, which is called the student model. An inherent weakness of Pseudo-Label is that once the teacher generates an incorrect pseudo label for an unlabeled datum, the student can only naively learn from this wrong label. This phenomenon is called the confirmation bias. Arazo et al. (2019) addressed the confirmation bias by generating soft labels from the teacher and by adding noise to these labels. However, this is manual fix from outside model designer. The main difference between Pseudo-Label and Coaching is that in Coaching, the teacher is trained along with the student throughout the course of training. This allows wrong knowledge learned by the teacher to be fixed in an and-to-end manner, leading to stronger performances.
Pseudo-Label.
Semi-supervised Learning (SSL). Pseudo-Label belongs to a more general group of algorithms known as Semi-supervised Learning. Unlike Pseudo-Label, typical SSL methods combine both labeled and unlabeled data to train single model. Hence, the objective function of SSL is typically the sum of supervised loss and an unsupervised loss. The supervised loss is often the cross- computed on the labeled data. Mcanwhile, the unsupervised loss can be supeersud loss (Rasmus et al., 2015; Noroozi & Favaro, 2018; Gidaris et al., 2018 8), or consistency regularization (Laine & Aila, 2017: Tarvainen & Valpola, 2017; Miyato et al., 2018; Berthelot et al., 2019; Xie et al., 2019). Self-supervised losses typically encourage the model to develop a common sense about the images. Meanwhile, consistency regularization enforces that the model is invariant against certain transformations of the data. The main difference between Coaching and SSL methods is that the student in Coaching never learns directly from labeled data. This helps the student in Coaching to avoid overfitting to labeled data, especially when labeled data is limited.
Meta Learning. In Meta Learning, there is typically outer loop that optimizes the performance of a model trained in an inner loop (Finn et al., 2017: Metz et al., 2019). Meta Learning has been applied perform self-training and SSL in the low data regime (Agarwal et al., 2019; Ren et al., 2018; Boney & Ilin, 2018; Hsu et al.. 2019). A crucial difference between Coaching and Meta Learning is that in Coaching, the pseudo labels are chosen improve the student, and hence there is no need for an outer loop. We suspect this is an advantage of our method, since gradients to be very powerful for models to navigate in the parameter space.
CONCLUSION
In this paper, we proposed the Coaching method for semi-supervised learning. Key to Coaching is the idea that the teacher learns from the student's loss and improves itself to generate pseudo labels in a way that helps student's learning the most. The learning process in Coaching consists of two main updates: updating the student based on the pseudo labeled data produced by the teacher and updating the teacher based on the student's 's performance. Experiments on standard CIFAR- 10 and SVHN show that Coaching is much better than supervised learning and consistenly better than other semi-supervised learning methods. Coaching scales well to large problems, and successfully uses out-of-domain data to improve ImageNet classification.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Benoit G, Derek. Murrayand Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. Arxiv 1605.08695, 2016. 14
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. International Conference on Machine Learning. 2019. 8
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias deep semi-supervised learning. Arxiv, 1908. 02988, 2019. 8
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi- semi-supervised learning. In Advances in Neural Information Processing Systems, 2019. 4,
Rinu Boney and Alexander Ilin. Semi-supervised few-shot learning with maml. Workshop Track of International Conference on Learning Representations, 2018.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. AutoAugment: Learning augmentation policies from data. In IEEE Conference on Computer Vision and Pattern Recognition, 2019. 5, 13, 14
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning. 2017.
Golnaz Ghiasi, Tsung- Lin, and Quoc V. Le. Dropblock: regularization method for convolutional networks. In Advances in Neural Information Processing Systems, 2018. 6
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In IEEE Conference on Computer Vision and Pattern Recognition, 2018. 8
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Kyle Hsu, Chelsea Finn, and Sergey Levine. Unsupervised learning via meta learning. In International Conference Learning Representations, 2019. 8
Jacob Jackson and John Schulman. Semi-supervised learning by label gradient alignment. Arxiv 1902.02336, 2019,
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. Arxiv, 1811.00982, 2018. 2, 6
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, In International Conference on Learning Representations, 2017. 1, 4,
Dong-Hyun Lec. Pseudo-Label: The simple and efficient semi-supervised learning method for deep neural networks. In International Conference Machine Learning Workshop, 2013. 1, 8
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. Arxiv, 1805.00932 2018. 7
Luke Metz, Niru Maheswaranathan. Brian Cheung, and Jascha Sohl-Dickstein. Meta-learning update rules for unsupervised representation learning. In International Conference on Learning Representations, 2019. 8

---

Under review as a conference paper at ICLR 2020
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. 4, 8
Yuval Netzer, Tao Wang, Alessandro Coates, Adamand Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In IEEE Conference Computer Vision and Pattern Recognition, 2018. 8
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, 2018. 4, 14
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015. 8
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 2015.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve sumi-supereieed deep learning results. Advances in Neural Information Processing Systems, 2017. 1, 8,8
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data multimedia research. Arxiv 1503.01817, 2015. 6
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In Advances in Neural Information Processing Systems, 2019,
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for semi-supervised learning. In International Joint Conference on Artificial Intelligence, 2019. 4
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992. 3, 12
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. Arxiv, 1904. 12848, 2019. 4, 5, 8, 14
I. Zeki Yalniz, Herv'e J'egou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. Arxiv 1905.00546, 2019. 6,7
Yoshihiro Yamada, Masakazu Iwamura, Takuya Akiba, and Koichi Kise. Shakedrop regularization for deep residual learning. Arxiv, 1802.0237, 2018.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy train strong classifiers with localizable features. In International Conference on Computer Vision, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016. 3
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S Self-supervised semi- supervised learning, Arxiv, 1905.03670, 2019. 5
10

---

Under review as a conference paper at ICLR 2020
A DERIVATION OF THE TEACHER'S UPDATE RULE
In this section, we present the detailed derivation of the Teacher's update rule in Equation 3 from Section 2.
Mathematical Notations and Conventions. Since we will work with the chain rule, we use the standard Jacobian notations. Specifically, for a differentiable function IR - IK", and for vector x RRm, we use the notation dr E Raxm to denote the Jacobian malrix of f, whose dimension m.. Additionally, when we mention the Jacobian function f at multiple points such # and æ2, we will the notations of a and 01
0f
Furthermore, by mathematical conventions, vector E R is treated column matrix that matrix of size n x 1. For this reason, the gradient vector of multi-variable real-valued function is actually the transpose of of its Jacobian matrix.
Finally, all multiplications in this section are standard matrix multiplications. If an operand is vector, then discussed in the previous paragraph, the operand is treated as a column matrix.
Dimension Annotations. Understanding that these notations and conventions might cause confu- sions, in the derivation below, we annotate the dimensions of the computed quantities to ensure that there no confusion caused our readers. To this end, we respectively use |S] and [T') denote the dimensions of the parameters 0s, 0r. That is, Os E R and Or E R|T
We now present the derivation. We need to compute:
ôR
θ f (Ilbb: Blab: E P( (-12m000T))
001
001
(5)
1x[T]
To simplify our notation, let us define
it++1)
A
am P((2m))07) [0(+1)
(6)
|S|x1
Then, by the chain rule, have
(Jab.. Mabi En t (
000
r0r
1x|T
0
(Tab Daabi 0
00T
ae (Ilab, Ylab; Os)
00 (++1)
00s
0g
0g=00
70+1
00T
|S|x|T|
1x
The first factor in Equation can be simply computed via back pack-propagation. We now focus on second term. We have
00((+1)
[o(+t2)
R P(. P(-
00
a0r
S|x/TI
(8)
i
8(0) -1
(8l
(ee (Funts Funt 0s) 00g
'
E Em P(.(aai(())
00r
0g 0==0(
Standard: https: wi kipeii wwwp///ggg////oo www.//////bbiiaaaa awwaaannmaaaaaaeeaee
11

---

Under review as a conference paper at ICLR 2020
Note that in Equation 8 above, the Jacobian of (T Punt; 0g), which has dimension 1 |S], needs to be transposed to match the dimension of ( which, as we discussed above, conventionally has dimension |S| 1.
Now, since ~ in Equation does not depend on Or, we can leave out of subsequent derivations. Also, to simplify notations, let us define the gradient
[e (Funt; Dumi: 0s)
(2) 95 (unt
(9)
O0s
9g=
Then, Equation 8 becomes
00 00
0
-1
B ((l
95 (Daml)
(1)
00T
00r
(10)
xs|x1
|
Since ( (9aml) has no dependency on on 0r, except for via Pont we can apply the REINFORCE equation (Williams, 1992) achieve
00
00T
=
17 00r
E (()
Is (Qam))
(t)
|
Olog P (Muml Fuul 0r)
(1)
17 EEga PP (((1sun)0r))
Is (Dunl)
00T
(11)
(S] x1
1x/21
HE (Tunl. Punt: 0r)
(1)
17 Pg-a-P((
Is (Bunl)
00T
|s|x1
1x[T]
Here, the last equality in Equation 11 is is due to the definition of the cross entropy loss, which is the negative of the log-prob term in the previous line.
Now, we can substitute Equation 11 into Equation 7 to obtain
OR
de (Fiab. Vlab: 0g)
00
00T
00g
os
(1))
00T
1X|T]
x|
1x|8|
(12)
17
Ol (Tiab, Mahi 0s)
B Paa P(:
95 (Junt)
(8)
Ol (Iunt, Dunt 0r)
00g
0. 05=88-1
((++))
00T
|s|x1
1x|T]
1x SS
Finally, if we use Monte Carlo approximation for every term in Equation 12 using the sampled junt. then have Equation 3 from Section 2. Note that in Section 2, we the gradient notation, which results in the transposes.
B TRAINING SPEED
Coaching performs up to 5 forward passes and 3 backward passes. Compared to vanilla back- propagation training, this is more 3 than times more expensive in FLOPs. However, many computa- tions in Coaching are parallelizable. For example, the forward pass the student and the forward pass for the teacher unlabeled data (the top half of Figure 1), be in parallel since they do not depend on each other. Therefore, on computing hardware with sufficient memory, we find Coaching to be between 2 and 2.5 times slower than standard back- pcokppropggation training.
12

---

Under review as a conference paper at ICLR 2020
C EXPERIMENTAL DETAILS
C.I DATASET SPLITS
We describe how we select the reduced datasets for the experiments on low data image classification in Section 3.1.
For CIFAR-10, we download the five training data batch files from www cs toronto, edu/ - /rri//oa ci far html. Then, we load all the images into list of 50,000 images, keeping the order as downloaded. The fisrt 5,000 images ares reserved for validation. The next 4,000 images are used as labeled data. For SVHN, we download the data from the mat files on of ufld]. stanford eduu / housenanbers/ /, and follow the same procedure as with CIFAR-10 10. We note that this selection process leads to slight imbalance in the class distribution for both CIFAR-10 10 and SVHN, but the settings are the same for all of our experiments.
For ImageNet, master/ www.srcceeeecceeeeeeeeee cicciinnetttii /on/ incept ion/ data/ download annd_ppeprocees imagenet sh. This results in 1,024 training TFRecord shards of approximately the same size. The order of the images in these shards are deterministic. For ImageNer- 10%, we use the first 102 shards; for ImageNet-20%, we the first 204 shards; and on. The last 20 shards, corresponding to roughly 25,000 images, are reserved for hyper-parameten tuning.
follow the procedure in wotp///a///wwwo/adddddaaa
C.2 RANDOMAUGMENT: A DATA AUGMENTATION POLICY
We develop data augmentation policy that achieves similarly high performance with Auto ment (Cubuk et al., 2019) in a few cases that we consider, but which does not require learning controller to generate policies. We names our policy RandomAugment. Our goal when developing RandomAugmen is not to outperform AutoAuu,ent, which is why we do not conduct extensive experiments with Random Augment. Instead, we simply want to avoid indirectly using labeled data for our experiments, especially for the low data regime experiments in Section 3.1.
Each policy of RagdomAugment consists of two operations that applied sequentially on an image. Each operation applies a uniformly sampled transformation with probability 0.5, and with level uniformly chosen between and 10. For more comprehensive discussion of the probability and the level of transformation, we refer readers the AutoAugment paper (Cubuk et al., 2019).
CIEAR-10 ImageNet
SWHN
AutoContrast
AutoContrast
Brightness
Brightness
Color
Color
Coutrast
Contrast
Equalize
Equalize
Invert
Sharpness
Sharpness
Posterize
Posterize
Sample Pairing
Solarize
Solarize
ShearX
ShearY
SheurX
Translate
ShearY
TranslateX
Translate
Table 3: Transformations that Rugmentugment uniformly samples for our datasets. We refer our readers to Cubuk et al. (2019) for the detailed descriptions of these transformations.
We manually design the set of transformations for each of our datasets: CIFAR-10, ImageNet, and SVHN. The set of transformations for CIFAR-10 and for ImageNet are the same, and are slightly different from the set of transformation for SVHN. This is because the numbers in the SVHN have different requirement for invariant. For instance, numbers should not be invariant against rotations like 6 and 9, and should not be invariant against horizontal translation like 3 and 8. Table 3 presents the transformation for our dataset. In addition to these operations, we only allow RugmemAugment to select the three transformations AutoContrast, Brightness, and Invert in the first augmenting
13

---

Under review as a conference paper at ICLR 2020
transformation. This is to avoid few degenerating cases. For instance, when Brightness is applied twice on an image, both times with small levels, the image will become almost black.
In our experiments, RandomAugment's 's performance is not far behind compared to AutoAugment. For example, on full ImageNet with ResNet-50, RandomAugment achives 77.98% top-1 accuracy, which is close the top-1 accuracy of 77.6% reported by Cubuk et al. (2019).
C.3 HYPER-PARAMETERS
To tune paper--arameters, we follow Oliver et al. (2018) and allow each method to have 128 trials of hyper-parameters. When we tune, we let each model train for up 50,000 steps. The optimal hyper-parameters are then used to run experiments that last for much more steps, as we report below. In our experiments with Coaching, training for more steps typically leads stronger results. We stop at 1 million steps for CIFAR-I0 10 and SVHN, and at 0.5 million steps for ImageNet simply because otherwise, these experiments will take too long. Meanwhile, in our experiments with purely supervised learning, Pseudo-Labels, and UDA, training for more steps overfits the models, and we have employ early stopping.
We report the hyper-panne parameters for our baselines and for Coaching in Section 3.1. For the high- resource experiments in Section 3.2, we use the same hyper-parameters, because tuning them too expensive. Our hyper-parameters can be found in Table 4, 5, 6.
We note that our settings for UDA different from originally reported by Xie et al. (2019). In their work, Xie et al. (2019) use much larger batch size for their UDA objective. In our implementation of UDA, we keep these batch sizes the same. This leads to much easier implementation of data parallelism in our framework, TensorFlow (Abadi et al., 2016) running TPU big pods. To compensate for the difference, we train all UDA baselines for much longer than Xie et al. (2019). During the training process, we also mask out the supervised examples with high confidence. Effectively, our UDA model receives roughly the same amount of training with labeled examples and unlabeled examples as the models in Xie et al. (2019). We have also verified that ImageNet- 10% with the augmentation policy from AutoAugment (Cubuk et al., 2019), our UDA implementation achives 68. 77% top-1 accuracy, which is similar 68.66% that Xie et al. (2019) reported.
Hyper-parameter
CIFAR-10 10
SVHIN
ImageNet
Weight decay
0.0005
0.001
0.0002
Label smoothing
0
0.1
Batch normalization decay
0.99
0.99
0.99
Learning rate
0.4
0.05
1.28
Number training steps
50,000
50,000
40,000
Number of warm up steps
2500
0
2000
Batch size
1024
128
2048
Dropout
0.4
0.5
0.2
Pscudo label thresbold
0.95
0.975
0.7
Table 4: Hyper-parameters for supervised learning and Pseudo-Labelle
Hyper-parameter
CIFAR-10
SVHN
ImageNet
Weight decay
0,0005
0.0005
0.0002
Laeel smoothing
0
0.1
Batch normalization decay
0.99
0.99
0.99
Learning rate
1.28
Number of training sleps
L,000,000
1,000,0000
500,000
Number warm up steps
5,000
5,000
Batch size
128
128
2048
Dropout rate
0.5
0.6
0.25
UDA factor
2.5
20
UDA tomporature
0.7
0.8
0.7
Table S: Hyper-parameters for UDA. Unlike originally done by Xie et al. (2019), we do not use a larger batch size for the UDA objective. Instead, we use the same batch size for both the labeled objective and the unlabeled objective. This is to avoid instances where some particularly small batch sizes for the labeled objective cannot be split on our computational hardware.
14

---

Under review as a conference paper at ICLR 2020
CIFAR-10
SVHN
ImageNet
Weight decay
0.0005
0.0005
0,0002
Label smoothing
0.1
0.1
0.1
Common
normalization decay
0.99
Number of training steps
1,000,000
1,000,000
500,000
Number warm up steps
2,000
2,000
1,000
Learning rate
0.3
0.15
0.8
Student
Batch
128
128
Dropout rale
0.35
0.45
0.1
Leaminng rate
0.125
0.05
0.5
Batch size
128
128
2048
Teacher
Dropout
0.5
0.65
UDA factor
1.0
2.5
16.0
lemperatuure
Table 6: Hyper-parametter for Coaching.
15


Reviews:
[{"review_id": "SJg8ZTnHcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes Coaching for semi-supervised learning. The teacher generates pseudo labels for unlabeled data and the performance of the student on labeled data is used as a reward to train the teacher. The empirical results are very impressive.\n\nOverall, the paper is clear and easy to follow. The idea looks interesting to me. However, I find that there are some weaknesses.\n\nFirst, the method is not well-motivated in the Introduction section.\n\nSecond, the derivation of the teacher's update rule is incorrect. In Eq. (11), $g_S^{(t)}(\\hat y_{unl})$ is a vector, $\\frac{\\partial \\ell(x_{unl}, \\hat y_{unl}; \\theta_T)}{\\partial \\theta_T}$ is also a vector. What do you mean by multiplying two vectors? The left side of Eq. (11) is a matrix while the shape does not match on the right side.  It is incorrect to get Eq. (3) from Eq. (11), especially the transposed $g_S^{(t)}$. Where does the transpose come from?\n\nAnd $g_T^{(t)}$  in Eq.(3) is inconsistent with line 6 in Algorithm 1, where $\\eta h^{(t)}$ is not included in $g_T^{(t)}$.\n\nFor experiments, I wonder what is the performance of pure Coaching without RandomAugment and the consistency loss/UDA in Table 1. I think this is a fair comparison with the baselines like Mean Teacher and VAT. I suggest adding this to the ablation study as well.\n\nAnd I expect more explanations on the \"additional implementation details\". For example, why is it correct to use cosine distance instead of the dot product? In this case, is it a valid gradient? Same for other tricks. We need to understand why it works apart from adding a bunch of tricks together. And I doubt whether the improvement is due to these tricks or the method itself.\n\nI would be willing to increase the score if all the concerns are addressed in the authors' response.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeoS1G2tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a simple but novel coaching method for teacher-student based semi-supervised learning framework. The coaching method consists of a two-stage update: first update Student network according to the pseudo label produced by Teacher network on the unlabeled dataset; second update Teacher network according to the Student network's performance on the labeled dataset. The authors propose a novel policy gradient update for the Teacher network. The authors evaluate the coaching method on several different semi-supervised learning dataset CIFAR-10, SVHN and ImageNet. The comparison with baselines are thorough. I appreciate that the authors explain the design of the experiments and tuning in details.\n\nI vote for acceptance but I still have some questions. I would be willing to increase my score if the authors address my questions in the rebuttal.\n1. The motivation of coaching is not accurate. In the second sentence of Introduction, the authors mention \"Although coaches do not play as well as the players\". However, we are training neural networks. There is no such thing as \"coaches do not play as well as the players\". They are the same parametric neural networks. (The authors also use the same architectures for both teacher and student neural networks.) I would remove this analogy sentence.\n2. The authors mention that their coaching method beats the fully supervised learning on the CIFAR-10. I am not convinced by this result. The author explained that this is due to the less overfitting in the student network. However, besides coaching, there are many other regularization method we can use to avoid overfitting. Using far less labels in training strictly reduces the amount of information we have. There is a simple test the authors can do. We can use the full labelled data set and use sampled results from Teacher network to train a Student network on it. We can perform coaching on this regime. This coaching on full dataset should do better than coaching on partially-labeled dataset (4000 labels).\n3. The state of art for CIFAR-10 is 99% now [1]. It would be nice to see the performance of author's approach applies to the state-of-art network/structure.\n\nReferences:\n[1] Huang, Yanping, et al. \"Gpipe: Efficient training of giant neural networks using pipeline parallelism.\" arXiv preprint arXiv:1811.06965 (2018).\n\n-----\nThe authors' response does not answer the questions about motivation and why the method works, which is also questioned by the two other reviewers. For example, why the authors need to have two networks is unclear; I guess that true labels should be more helpful to guide a student network to learn than a teacher network. Even though the authors seem to have state-of-art semi-supervised learning results, some extra explanations are needed. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJe2x9RTFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the teacher-student models in semi-supervised learning. Unlike previous methods in which only the student will learn from the teacher, this paper proposes a method to let the teacher learn from the student by reinforcement learning. Experimental results demonstrate the proposal’s performance.\n\nThe paper achieves some good empirical results compared to other baselines. However, the proposed method is implemented with many tricks listed on Page 4, and with data augmentation techniques, which may not be used in previous methods. Additionally, the paper is weak in technology. There is no clear explanation of why the proposed method works except a metaphor for sports coaches. I vote for a clear rejection of the paper.\n\nFirst, the paper is weak in experiments. It works hard to achieve a good experimental result, through many tricks listed in the “Additional Implementation Details” in Page 4, and through the data augmentation used in Page 5. However, these tricks to improve the performance may not be used in previous methods, as the paper does not run experiments on baselines under the same setting, but use the results reported in Oliver et al. (2018). Additionally, the paper only uses one number of labeled data for each data set, it makes readers doubt that the proposed method only works under this number of labeled data. \n\nThe paper fails to clearly state why we need to let the teacher learn from the student. Actually, I doubt if this is necessary. Given the strong learning capacity of neural networks, the proposed method will easily be overfitting. Assume we have a very weak student network at the beginning, then by training in the way proposed in the paper, the teacher network will have all labeled data classified correct, and all unlabeled data classified into the same labels as the student network. I cannot see from the simple proposal why such overfitting can be avoided.\n\nThe paper is weak in both technology and experiments. It is also poorly written without clearly stating the motivation for this problem. I would vote for a reject for the paper. \n\n-------------------------------------------\nThe rebuttal has cleared some of my concerns. However, it is still not clear why the proposed method work and how does it prevents overfitting. The paper also needs more experimental results to confirm its effectiveness. I will increase my score a little bit, but would not vote for an accept this time. But I believe with further revision, the paper may be worth publishing in the future, if the questions in all reviews can be addressed.  ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose a new method of semi-supervised learning and provide empirical results. Reviewers found the presentation of the method confusing and poorly motivated. Despite the rebuttal, reviewers still did not find clarity on how or why the method works as well as it does.
Decision: Reject