LEARNING TO IMPUTE: A GENERAL FRAMEWORK FOR SEMI-SUPERVISED LEARNING
Authors: Wei-Hong Li
Chuan-Sheng Foo
Hakan Bilen
Venue:  (2020)
Keywords: Semi-supervised Learning
Meta-Learning
Learning to label

Abstract: Recent semi-supervised learning methods have shown to achieve comparable results to their supervised counterparts while using only a small portion of labels in image classification tasks thanks to their regularization strategies. In this paper, we take a more direct approach for semi-supervised learning and propose learning to impute the labels of unlabeled samples such that a network achieves better generalization when it is trained on these labels. We pose the problem in a learning-to-learn formulation which can easily be incorporated to the state-of-the-art semi-supervised techniques and boost their performance especially when the labels are limited. We demonstrate that our method is applicable to both classification and regression problems including image classification and facial landmark detection tasks.

Under review as a conference paper at ICLR 2020
LEARNING TO IMPUTE: A GENERAL FRAMEWORK FOR SEMI-SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent semi-supervise learning methods have shown to achieve comparable results to their supervised counterparts while using only a small portion of labels in image classification tasks thanks to their regularization strategies. In this paper, we take a more direct approach for semi-supervised learning and propose learning to impute the labels of unlabeled samples such that a network achieves better generalization when it is trained on these labels. We pose the problem in a learning-to- formulation which can easily be incorporated the state-of-the-an semi-supervised techniques and boost their performance especially when the labels are limited. We demonstrate that our method is applicable to both classification and regression problems including image classification and facial landmark detection tasks.
INTRODUCTION
Semi-supervised learning (SSL) (Chapelle et al. 2009) is one of the approaches to learn not only from labeled samples but also unlabeled ones. Under certain assumptions such as presence of smooth prediction functions that map data to labels, of low-dimensional manifolds that the high-dimensional data lies (Chapelle et al. 2009). SSL methods provide way to leverage the information at unlabeled data and lessens the dependency on labels. Recent work (Tarvainen & Valpola 2017, Miyato et al., 2018 Berthelot et al. 2019 Xie et al.) 2019) have shown that semi-supervised learning by using only small portion of labels can achieve competitive results with the supervised counterparts in image classification tasks (i.e. CIFARIO, SVHN). They built on a variation of the well-known iterative bootstrapping method (Yarowsky 1995) where in each iteration a classifier is trained on the current set of labeled data, the learned classifier used to generate label for unlabeled data. However, the generalization performance of this approach is known to suffer from fitting the model on wrongly labeled samples and overfitting into self- gelfraenertted labels (Tarvainen & Valpola 2017). Thus, they mitigate these issues by various regularization strategies.
While there exist several regularization (Srivastava et al. 2014) and augmentation (Zhag et al., 2018) De Vries & Taylor 2017) techniques in image recognition problems which are known to increase the generalization performance of deep networks, specific regularization strategies for semi-supervised classification used to estimate correct labels for the unlabeled data by either encouraging the models to produce confident outputs (Lee) 2013) Grandvalet & Bengio 2005) and/or consistent output distributions when its inputs are perturbed (Tavvainen & Valpola 2017 Miyato et al. [2018] Berthelot et al. 2019).. The assumption here is that if good regularization strategy exists, it can enable the network to recover the correct labels for the unlabeled data and then the method can obtain similar performance with the supervised counterpart when trained on them. Though this ad-hoc paradigm shown be effective, it raises a natural question for more direct approach: Can we instead encourage the network to label the unlabeled data such that the network achieves better generalization performance when trained with them?
In this paper, we propose a new learning-to-leant method for cunissupee learning that can be put in a meta-learning framework to address this question. Our method involves learning an update rule to label unlabeled training samples such that training our model using these predicted labels improves its performance not only on itself but also on a meta-validation set. Crucially, our method is highly generic and can easily be incorporated to the ofate-of-the of-tthe-rt methods (Lee) 2013 Berthelot et al. (201)) and boost their performance, in particular, in the regime where the number of available labels is limited. Moreover, method is not limited to classification problems, show that it can

---

Under review as a conference paper at ICLR 2020
be extended to semi-supervised regression tasks where the output space is continuous and achieves significant performance gains.
2 RELATED WORK
Semi-supervised classification. There is a rich body of literature in SSL CChppelle et al., 2009) for
classification. Most of the recent work (Lee 2013 Tarvainen & Valpola 2017 Miyato al. 2018 Berthelot et al. [019 Xie et al. (019)) builds on the idea of the bootstrapping technique of Yarowsky (1995) that involves iterative optimization of classifier on set of labeled data and refinement of its labels for the unlabeled data. This paradigm is known to overfit on the noisy eeff-genetteed labels and thus suffer from low generalization performance. To alleviate the sensitivity to inaccurate labeling, researchers introduce various regularization strategies. Grandvalet& & Bengio (2005) propose a minimum entropy regularizer that encourages each unlabeled sample to assigned to only one of the classes with high probability. Lee (2013) instead follow a more direct approach, use the predicted label with the maximum probability for each unlabeled sample as true-label which is called "pseudo- label". An orthogonal regularization strategy is to encourage a classifier to be invariant to various stochastic factors and produce consistent predictions for unlabeled data when the noise is added to intermediate representations (Srivastava et al.) 2014) and to input in an adversarial manner (Miyato) et al., 2018). In the same vein, Laine & Aila (2017); Tarvainen & Valpola (2017) introduce the TI-model and Mean-Teacher that regularized to be consistent over previous training iterations by using a temporal ensembling and tcacher/student networks respectively. Recently Berthelot et al. (2019) introduced MixMatch algorithm that further extended the previous work by unifying the idea of the consistency regularization and augmentation strategies (Zhang et al. 2018). While the recent techniques are shown to be effective in several classification benchmarks, the idea of consistency regularization is still implicit in terms of generalization performance. Here we take an orthogonal and more direct approach to this approach and learn to impute the labels of the unlabeled samples that improve the generalization performance of a classifier. We also show that our method can be used with the recent SSL techniques.
Semi-supervised regression. Some of the recent techniques in semi-supervised classification are shown to be applicable to regression problems. Jean et al. (2018) have adapted various existing SSL methods such as label propagation, VAT (Miyato al. 2018 and Mean Teacher (Tarvainen & Valpola, 2017) and studied their performance in regression. The same authors also proposed a Bayesian SSL approach for SSL. regression problems which based on the recent deep kernel learning method (Wilson et al.] (016) based approach for cune-supe learning that aims at minimizing predictive variance of unlabeled data. As in the classification task, these methods are typically ad-hoc and does not aim to generate labels for the unlabeled data that are optimized to improve the generalization in regression.
Meta-learning. Our method is also related to meta-ng (Schmidhuber 1987 Bengio et al., 1992) and inspired from the recent work (Andrychowicz et al. 2016 Finn et 2017) where the goal is typically to quickly Iearn a new task from a small amount of new data. (Anrycchowcc et al. (2016); Finn et al. (2017) propose a learn gradient through gradient approach to train model which has a strong generalization ability to unseen test data and can be easily adapted to new/unseen tasks. [Sung et al. (2018) introduce the relation network to learn an additional metric such that the learned model's feature embedding can be generalized for unseen data and unseen tasks. Ren et al. (2018b) adopt the meta learning to learn the weight of samples to tackle the sample imbalance problem. Lorraine & Duvenaud 2):8); MacKay et al. (2019) employ meta learning for automatically learning the hyper-parametter
of the deep neural networks. Meta-learning has recently been applied to unsupervised learning (Hsu et al. 2019) and SSL for few shot learning (Ren et al.) 2018a)) Ren jt al. (20T8a) adapts the prototypical network to use unlabeled examples when producing prototypes, enabling the prototypical network to exploit those unlabeled data to assist few shot classification. Antoniou & Storkey (2019) propose to learn a label-free loss function, parameterized as a neural network that enables the classifier to leverage the information from validation set and achieves better performance in few-shot learning. (Li et al. 2019) propose a meta-learning technique to initialize self-traaning model and to filter out noisy pseudo labels for semi-supervised few-shot learning. Similarly, our work also builds on the ideas of optimizing for improving the generalization for unseen samples. However, in contrast to the existing meta-learning methods that is proposed for few-shot
2

---

Under review as a conference paper at ICLR 2020
learning problems, we focus semi-supervisse learning in general classification and regression problems where the number of samples are not limited to few.
METHOD
Consider dataset D that consists of [C] labeled samples L = {(x1 y1). (Z2g U2). ..... [c) | 4/4)) (])))
which is further split into training T and meta-validation set V with [T] and [V] samples respectively. We also have a set of unlabeled samples U = (æ" {0" I==..... Further, we let 2y denote a model (a function parameterized for instance, predict labels y from given samples x as y = ()))))
deep neural network) with parameters 0, that is trained to
We are interested in imputing missing labels z of the unlabeled samples that are not only accurate but also improve performance at inference time, when included in our training set for optimizing model parameters 0. A straightforward approach to this problem is to first train model to optimize the following cost function on the training set T:
arg min Σ {(Fo(z))))) y)
where l is task specific loss, such soft-max followed by cross-entropy loss for classification or squared loss for regression tasks.
One can then use the trained model b to impute labels for samples in the unlabeled set U as z = ()" as in Tarvainen & Valpola (2017): Berthelot et al. (0199), where x denotes randomly perturbed input image x" (e.g. random crops). Note that the label imputation procedure can be replaced with other pseudo-label prediction techniques.
expand T by adding the unlabeled samples with their imputed pseudo-labels i.e. A T UM and further train by using Eq. (T] However, as many pseudo-labels z will likely be noisy, there is no guarantee that training on the augmented train set will improve the performance of the model on the meta-valice meta-validattoo set V:
Now
CV Σ ((o)(z)))))
Note that in all our experiments V is randomly sampled from T which ensures that our method trained using any additional data.
Our goal is also to minimize the expected value of the loss in Eq. ( with respect to the model parameters 0 via an algorithm such as stochastic gradient descent (SGD). Here, we go step further and consider how the label imputation affects the ability of SGD optimize the loss in Eq. 2
We pose this as meta- meta-learning problem, which is derived next. We start by considering the loss value of the model & on the augmented training set A::
Σ ((Da(x)) y) + Σ ((D)(x)),
(3)
TET
xe Ell
We first simulate step of SGD using the loss in Eq. (3) to drive the update of the model parameters 0. At step I, SGD updates the parameters for minimizing Eq. (3) follows:
ge+l 0¹
v
Σ ((Fo(z).)) TET
+
Σ [(Fo(x"))
(4)
where Ve is the gradient operator. We then wish to find z that minimizes the meta-validation objective CV (0)+))) in Eq. (2) with g++1, This corresponds to the following bilevel optimization problem
min Σ ΦEV
( (0: v) subject to g++1
8¹ - Vo
Σ TET
((Fo)))))) + Σ 28 El
((Fo(=")
2)
(5) Our goal is to learn predicted pscudo-labels that minimize the meta-validation loss in Eq. 5 To this end, we propose two options, denoted as Option and Option 2. In the former, we treat z as latent (or learnable) parameters and compute the gradients of the velta-aliitiion loss w.r.t to updatez, which is used to update model parameters 8. In the latter, x is considered as the output of network $ and the gradients are thus computed w.r.t the model parameters 0 for updating 0.
3

---

Under review as a conference paper at ICLR 2020
Optimization. be written as:
The gradients of CV w.r.t x for Option 1 and w.r.t for Option 2 are respectively
ac
[00'
(Option 1)
V
(6)
Oz
06++1
Oz
VEV
(Option 2)
OC
ac
dz
@ de (2))) v)
(I (I-NVoVoc1)
00
00
ag
where is identity matrix and Vo v indicates a second order derivative which is supported by
standard deep learning libraries such Pytorch Paszke et al. (2017), Tensorflow et al. (2015).
We then use the gradients in Eq. (6) or Eq. 7 to update the model parameters such that the updated model predict pseudo-labels that minimize the meta-validation loss. In Option
update the pscudo-labels as = 2 - T} dz and the updated pseudo-labels are then used to update the model parameters by minimizing the loss C = Exreel eu {( (a={", 2:). In Option 2, the gradient in BCV
ACV
Eq. 7 is used to update the model parameters as 0++1 0r '7 388
We further depict the optimization details of Option and 2 in Alg. 1 We first estimate the with the current model 0', update the model optimize the loss C and then re-estimate with 0%, This part is so far similar to the self-labeling method of Lee 2013). The loss function C can trivially be replaced by those used in recent work such MeachTeacher (Tarvainen & Valpolal (017) or MixMatch (Berthelot et al. 2019: we applied our method in conjunction with all three loss functions and report results in Section 4 Next we apply the updated model mini-batch of unlabeled samples, compute the loss and simulate a SGD step w.r.t this loss. In Option 1, we initialize = with
the output of (0 and update it by using the gradient from Eq. 66) In Option 2, the meta-update is computed w.r.t 0 and used to update the model parameters by using the gradient from Eq. 7
Algorithm 1: Pseudo-code of our method for two variants.
Input: T, U.
D training set, unlabeled and meta-validation set resp. Required: model function & and its initialization parameters 8 a batch of training data
By {æ,, y1 Jim..... .887 a batch unlabeled data B (x+") Jem..... 1891 and another batch of training data
(treated as meta-validation data during training) BY = (xx, 1: 1n...... learning rate Yas weight À. for = 0.... K- do (........... 1 g (x=)(()))) (xt") 1.. 1189
b est amtimate pseudo labels for unlabeled samples, o. is randomly perturbed input x.
C = EiE f( (x:). M1) - + >Σ 15 (()))
0¹ = 0¹ 1 ddd
update the model and move to the meta update b estimate pseudo labels using the
dor
x = (2.
updated model
C= 1551 E(D (m)),
0++1 0' a
D simulate a SGD step > evaluate on validation data
06
C² (0(tt = Σ 115
ElDbee (2)).
if option I then
& = x 77 de
acv
correct pseudo labels
CU = - E( (x)"),
0++1 = 0'
act!
update the model with corrected pseudo labels update the model directly with meta-gradient
else if option 2 then
0C+1 0' sC
Output: OK 86
end
4 EXPERIMENTS
We evaluate the performance of our method on multiple classification and regression benchmarks, and analyze the results below. Note that the validation sets only for tuning hyperparameters
4

---

Under review as a conference paper at ICLR 2020
GT SL PL
Ours option
Ours option 2
GT
51.
Pl.
Ours option
Ours option 2
Figure 1: Results on the toy binary classification task. The top and bottom rows correspond to the predictions on the test and unlabeled data respectively. Points in green and pink are labeled training samples. Best seen in color.
such learning rates and for early stopping. The meta-validation sets (CV in Eq. (5) are sampled from the training set in all our experiments as in |Rajeswaran et al. (2019): Finn et al. (2017); Vinyals al. (2016); Sung et al. (2018). During training, training set at each iteration: one acts as training and the other one acts as the meta-validatee trained on more data than the baselines. We train the model on the former and optimize the pseudo labels on the latter. T
sample two minchbatches of data from the same
set
to ensure that our method
4.1 CLASSIFICATION RESULTS
Toy datasets. We first validate our method on a synthetic binary classification dataset, two noisy concentric circles" where each circle corresponds and randomly pick 50, 2500 and 1000 for training, testing and validation sets respectively and use the rest of the samples as unlabeled data. Figure [ illustrates the dataset where the labels for both test (top row) and unlabeled data (bottom row) are indicated in blue and orange colors. The labeled training samples for two classes are shown in green and pink in GT.
class. To this end, we generate 10,000 samples
Experiment 1. In Fig. Te depict the predictions of two baselines: supervised learning (SL) which trained only on the laheled training set (i.e. green and pink samples), Pseudo-Labbling (Lee 2013) (PL) which iteratively first trains a network the labeled and unlabeled data and then labla the unlabeled ones, and two variants of our method. For all the methods, we use a shallow small capacity
network containing two fully-connected layers (2 15 15 2), one leaky ReLU activation layer in-between and sigmoid function at the end. We first observe that SI. misclassifies most of the outer circle points and predicts them as inner circle labels. This is due to the limited labeled data and its non-uniform spread over the input space. We see that PL significantly improves over SL by leveraging the unlabeled data. However many of its predictions are still inaccurate, especially in the regions of low label density and high ambiguity. This occurs because the iterative relabeling procedure is sensitive to the initial labeling of the unlabeled samples and thus it gets stuck in a non-optimal minimum. Both the variants of our method largely overcome the bias in the training data by labeling the unlabeled samples in a way that they lead to accurate predictions on the meta-validation set. This shows that the meta-updates are successful to correct the imputed labels based on the signal from the meta- validation set (see Fig. bottom row) and to prevent our method to simply overfit into the pseudo-labels While both variants agree on the most of the samples, they slightly differ in the low-density regions (i.e. the area between the circles).
CIFAR-10 & -100. We also evaluate our method on the CIFAR image classification bench- marks (Krizhevsky et al., 2009) that commonly used for evaluation of both supervised and semi-supervised classification. Both datasets contain 50,000 and 10,000 training and testing samples respectively. In our experiments, we strictly follow the training and testing protocol for semi-
'Our implementation in PyTorch will available https:://annymoaus 4open. sci ocience/r/ 021100958866-440 9 8791ef weleffll6666 c1c610 hhupss//sckkit-lam..og
5

---

Under review as a conference paper at ICLR 2020
supervised learning which is proposed by the previous work (Oliver et al. 2018 Berthelot et al. 2019) where 5000 of training samples are used as validation data, the remaining 45,000 training samples are split into labeled and unlabeled training sets. As in Oliver et al. (2018), randomly pick 77 samples as labeled data and the rest ((2) = 45, 000 |7| samples) are unlabeled data. We report results for multiple training data-regimes [T] 250, 500, 1000, 2000, 4000 CIFAR-10 and [T] = 1000, 2000, 3000, 4000, 5000 on CIFAR-100, Note that we do nor use the validation data in training of the model parameters but only for hyperparameter selection.
Experiment 2. First compare method the SL and PL baselines for various number of training samples and report their performance in Table [ for all the methods (see Appendix A for details). First we observe that PI. improves over SL by leveraging the unlabeled data in all the settings. Both our methods (option-1/2) that builds on PL. achieves substantial performance gains over PL. Interestingly the relative improvements are higher in the more challenging case of when the labeled data is limited. We also observe that option-2, which involves updating the model parameters with meta- eeta-gradient, is a better strategy for these benchmarks.
We use 13-layer conv-net as the classifier
CIFAR-10
CIFAR-100
#labels
250
500
1000
2000
4000
1000
2000
3000
4000
5000
SL
57.95
46.06
37.21
25.32
19.03
82.32
72.08
65.06
61.81
57.90
PI
45.19
36.58
27.19
22.12
17.5
76.30
67.32
60.76
56.53
53.82
PL+option1
43.74
35.76
26.46
21.14
16.81
75.67
66.03
60.36
56.28
53.01
PL+option2
42.51
35.43
26.38
20.74
16.65
75.12
64.37
59.03
56.00
52.79
Table 1: Error percentage of testing set CIFAR-10 and CIFAR-100.
Experiment 3. Here we show that our method can be incorporated to the state-of-the art methods such as Mean Teacher (MT) (Tarvainen & Valpola 2017) and MixMatch (MM) (Berthelot et al.. 2019) that use more sophisticated backbone networks, augmentation and regularization strategies, and also boost their performance. For this experiment, follow the implementation of Berthelot eet al. (2019) we adopt a more competitive backbone, -et-Res use the Adam optimizer along with standard data augmentation techniques (see Appendix A for more details).
CIFARIO
CIFAR100
#labels
250
500)
1000
2000
4000
1000
2000
3000
4000
5000
II model
53.02
41.82
31.53
23.07
17.41
Lseu
49.98
40.55
30.91
21.96
16.21
MixUp
47.43
36.17
25.72
18.14
13.15
VAT
36.03
26.11
18.68
14.40
11.05
MT
38.90
28.45
19.36
12.40
10.02
76.78
64.01
55.30
52.03
48.28
MT option!
34.72
27.25
18.07
12.70
10.36
75.84
62.88
57.22
51.75
48.49
MT + option2
27.72
21.05
14.93
11.91
10.31
75.24
61.45
54.56
50.31
45.40
MM
12.33
10.14
8.68
7.79
6.70
65.45
51.14
44.96
41.51
38,16
MM + option1
11.60
10.24
9.60
8.37
6.59
63.62
50.53
45.67
40.76
38,88
MM + option2 10.92
10.80
9.54
8.37
7.06
61.98
51.73
46.45
41.25
39.35
Table 2: Error percentage of testing set on CIFAR-10 and CIFAR-100 and comparison to the state-of-the-art semi-supervised learning methods.
Table 2 depicts the classification error rate for the several sfte-of--he-an techniques including II model (Laine & Aila 2017). PL (Lee) 2013), VAT (Miyato et al. 2018), MT (Tarvainen & Valpola 2017) and MM (Berthelot al. 2019). Note that some methods not report results on CIFAR- 100. All the results except MT and MM are taken from the original papers. As our methods are built on MM and MT, we show the results of our own implementation which are on par with the published ones. From the table, we see that our method achieves significant improvement over MT baseline, especially in the low label regime, up-to 11 points in case of 250 labels in CIFAR- 10. Again our second variant consistently outperforms the first one when used with the MT in both CIFAR-10 10 and -100, whereas the more competitive MM baseline already produces accurate pseudo-labed in CIFAR-10 10 and the two options perform comparably. The reason is, in option2, obtain the

---

Under review as a conference paper at ICLR 2020
meta gradient the model parameters and update the model directly, whereas in optionl, firstly update the pseudo labels and then train the model on them. Though updating the pseudo labels can improve the performance (e.g. MT + option! vs MT), some of the labels can still be noisy after the update and optimizing the model on the noisy ones may degrade the performance. In contrast, in option2, the meta gradient is applied the network parameters directly, which alleviates the potential pseudo-label noise.
In case of MM, our method is able to boost its performance only in case of few labels (250 for CIFAR-I0 and 1000 for CIFAR-100) and does comparable and slightly worse in case of more labels. We believe that is harder to improve the performance of MM in the CIFAR, as its performance approaches to the supervised counterpart quickly. We leave the evaluation of MM on a more challenging benchmark as future work and below we show results for tasks where MM is not applicable.
4.2 REGRESSION RESULTS
Next we move regression task and use the Annotated Facial Landmarks in the Wild (AFLW) dataset (Koestinger et al. 2011 Zhang et al. 2015) where we aim at predicting 5 landmarks' location of faces in images. The AFLW originally designed for supervised facial landmark detection. We use the official train and test splits, randomly pick 10% of samples of the original training set as the validation set only for hyperparameter tuning and early stopping, and use the rest of this data as labeled and unlabeled data in our experiments. We evaluate the baselines and our method for 1%, 2%, 5%, 10% of training data as labeled data and report the standard Mean Square Error (MSE) normalized by the inter-ocular distance as in Zhang al. (2015).
AFLW.
Experiment 4. For the regression task, we adopt the TCDCN backbone architecture in Zhang et al. (2015) and SGD optimizer, use standard data augmentation. We train all methods for 22500 steps. and set the initial learning rate to 0.03 and reduce it by 0.1 every 750 steps. The momentum and the weight decay are set to 0.9 and 0.0005. Here we use SI., PI. and MT as the baselines and also build our method on both the PL. and MT. Note that MM is not applicable to this task, as mixing up two face images doubles the number of landmarks. Table depicts the results for the baselines and ours in terms of mean error rate.
#labels
1%
2%
5%
10%
100%
SL
16.31
14.32
11.78
10.30
7.83
PL
12.97
11.65
9.70
9.01
Ours: PL + option1
11.86
10.86
9.46
8.93
Ours: PL + option2
12.77
11.47
9.85
8.90
MT
12.87
11.38
9.85
8.93
Ours: MT + option1
11.85
10.74
9.43
8.58
Ours: MT + option2
12.45
11.27
9.69
8.79
Table 3: Mean error percentage of testing set on AFLW for the facial landmark detection.
First we observe that the supervised learning 1% of the labels is very challenging and obtains only 16.31% % which is on par with the performance of simply taking the mean of each facial landmark over all training samples (16.58%). As expected, using the unlabeled face images is beneficial and both PL as well as MT methods significantly improves over SL. Our method achieves consistent improvement over PL and MT for different portions of labels. This strongly suggests that our method is able to refine the estimated landmarks of PL and MT on the unlabeled images and further improve its performance, We also analyze the effect of meta-updates during training our model in Fig. 2 To this end, we first plot the regression loss on the meta-validation batch before and after the ueta-update in the left figure. This clearly shows that updating the pseudo landmark positions in the unlabeled images lead to a better accuracy for the same models in the right plot. It is clear that the meta-updated model does not overfit to the meta-vall validatio set and generalizes better to the test images. We also visualize the effect of meta-updates on the landmarks on the example test images and observe that the meta-updates help them to get closer the ground-truth ones.
the meta-validation samples. Second we show the test loss
7

---

Under review as a conference paper at ICLR 2020
Though both option-1 and 2 improve over the baselines, here option-1 outperforms option-2 on the regression problem which is in contrast to the classification tasks above. This is possibly due to the fact that the output space of the landmarks is continuous and less constrained than the label space for classification which makes the option-2 more prone to overfit to the validation set. A promising direction which is worth investigating in future is to alleviate the overfitting by using regularizer to enforce the structure in the output space.
0.25
5.50
5.25
0.20
Meta loss
0.15
Test loss
4.75
4:30
0.10
4.00
0.05
3.75
25
Epoch
Epoch
Figure 2: Illustration of the meta update's effect on facial landmarks detection. Cyan curves are the loss of the model after the meta update while orange curves the model's loss before the meta update. Best seen in color.
We also illustrate success/failure cases the test images in Fig. and depict the ground-truth, pre- dicted landmarks MT and our method when trained with 1% of the labeled data. The performance difference is visually significant and method outputs bottom row shows the cases of extreme pose variation and occlusion where both MT and ours fail to achieve accurate predictions.
accurate landmarks than MT. The
Figure 3: Success (top row) and failure cases (bottom row) in AFLW for facial landmark detection. Green points with circles, cyan points and orange points are ground-truth landmarks, predictions of our method (option-1) and Mean Teacher. Best seen in color.
CONCLUSION
In this paper we have propose a general semi-supervised learning framework, it learns to impute the labels of the unlabeled data such that the training deep network on these labels improves its generalization ability. Our method can easily be used conjunction with several state-of-the- art semi-supervised methods and extended to multiple classification and regression tasks such as image classification and facial landmark detection. We show that our method achieves significant performance gains competitive basclines in challenging benchmarks, especially when the labeled data scarce. As future work, we plan to extend our method to semi-supervised learning in structured output problems.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogencous systems, 2015.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. (113988-398 2016.
Antreas Antoniou and Amos Storkey. Learning to learn via self-critique. arXiv preprint arXiv: 10295, 2019.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pp. 6-8. Univ. of Texas, 1992.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv: 2019.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. al., eds.; [b0o) reviews]. IEEE Transactions on Neural Networks, 20(3):542-544 2009,
Terrance De Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: net:::0080055555 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, pp. 1126-1135, 2017.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pp. 529-536, 2005.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In Interna- tional Conference on Learning Representations, 2019.
Neal Jean, Sang Michael Xie, and Stefano Ermon. Semi- supervised deep kernel learning: Regression with unlabeled data by minimizing predictive variance. In Advances in Neural Information Processing Systems, pp. 5322-5333, 5333, 2018.
Martin Koestinger, Paul Wohlhart, Peter M Roth, and Horst Bischof. Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization. In International Conference on Computer Vision Workshops. IEEE, 2011.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Fifth Interna- tional Conference on Learning Representations, 2017.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML,, volume 3, pp. 2, 2013.
Xinzhe Li, Qianru Sun, Yaoyao Liu, Shibao Zheng, Tat-Seng Chua, and Bernt Schiele. Learning to self-train for semi-supervised few-shot classification. arXiv preprint arXiv:1906.0 00562, 2019.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. arXiv preprint arXiv: 1802, 2018.

---

Under review as a conference paper at ICLR 2020
Matthew MacKay, Paul Vicol. Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint (rXXh::1903300888, 2019.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. Transactions on Puttern Analysis and Machine Intelligence, 2018.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3246, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances Neural Information Processing Systems, 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint 18X3 1803.0067, 2018a.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference Machine Learning, 2018b.
Jürgen Schmidhuber. Evolutionary Principles in Self-referential Learning: On Learning how to Learn: the Meta-metu-meta...heee 1987.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In International Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consis- tency targets improve semi-supervin deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial Intelligence and Statistics, pp. 370-378, 2016.
Qizhe Xie, Zihang Dai, Eduard Hovy. Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv: 12848, 2019.
David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pp. 189-196, 1995.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Machine Learning, 2018.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Learning deep representation for face alignment with auxiliary attributes. Transactions on Pattern Analysis and Machine Intelligence, 38(5):918-995 9-00, 2015.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
To evaluate our method with existing semi-supervisee learning, we implement our method and other methods Pytorch.
A.1 TOY EXPERIMENTS
Experiment 1. To conduct experiments on the concentric circles dataset, we use small network
containing two fully-connected layers (2 S 4 S, 2) and one leaky relu activation layer. We compare our option I and option 2 to the Pseudo Label (PL) Lee (2013) and supervised learning approach and we train all methods for 20 20 steps (i.e. 20 epochs and 20 steps per epoch). We use Adam the optimizer and the learning rate is 0.03. The maximum of the unsupervised loss weight À for both PL and Ours is set to 1 and we adopt the linear-schedule for increasing the weight on the unsupervised loss. Specifically, the Ả is initialized O and increases to gradually in 20 x 5 steps by linear-schedule as Berthelot et al. (201).. During the meta update in both our option and option2, we estimate the pseudo labels of unlabeled samples by gumbel softmax and compute the Mean Square Error (MSE) between prediction and pseudo labels as the unlabeled loss. 7) is for all experiments.
A.2 CLASSIFICATION EXPERIMENTS CIFAR-IO & -100
For all experiments on both CIFAR-10 and CIFAR-100. we follow the optimizing strategy in Berthelot et al. (2019), i.e. we adopt Adam as the optimizer and fix the learning rate. We evaluate the model using exponential moving average of the learned models' parameters with decay rate of 0.999. In addition, the weight decay is set to 0.02 for all methods. We report the median error of the last 20 epochs for comparisons. For image preprocessing. we standard data augmentation such as standard normalization and flipping, random crop as |Berthelot et al. (2019). During the meta update in both our option! and option2, we predict the pseudo labels of unlabeled images by applying softmax on the augmentation of the original unlabeled image and compute the Mean Square Error (MSE) between prediction on the original image and pscudo labels as the unlabeled loss as in Berthelot et al. (2019).
We use a 13-layer conv-net Tarvainen & Valpola 2017 Laine & Aila (011)) in the experiments where we compare our method to the SL. and PL. on both CIFAR- 10 and CIFAR- 100. The architecture of the network is illustrated in Fig. 4 In addition to this, we apply batch normalization on each convolutional and fully connected layers. We use Leaky Relu with negative slope (a 0.1) as the nonlinear activation function on each convolutional layers. On CIFAR- 10, the batch size and learning rate is set to 50 and 0.003, respectively, as Tarvainen & Valpola (2017) while on CIFAR-100, 100, the batch size is 128. We train each method for 40 x 1000 steps. For unsupervised loss weight X, we use the linear- schedule and it increases gradually to 75 40 x (1000 0.4) steps (for both the PI. and Ours).
Experiment 2.
Conve (ermel Filter: 1200
Gua (Korml: (ilteer 250)
Conv (ormel Filter: 250)
Cony (Kornel: 3 Filter: 513)
Canv (Kornel Fitterr 250)
Onny (ormal Filter: 1200
Leaky Sela (er 0.13
Lucky Bola (e- - 0.13
Gonv (Kornel. Filter:
Leakty sate 0.1)
23
Loaky (c 0.1)
Leaky Rote or 0.33
0.
Max 2:
Dropout 0.5)
Loaky (or 0.1)
Lecky Relu (e 0.1)
Fully Conuettd Layer (128 daas)
Input Image 32x32x33
Max Poot (2xx
Leckyy Sela (e
Drspout
Cony (Kerull
Cany
Figure Illustration of the 13-layer Convolutional Network.
Effect of Meta-validation Batch Size. To study the influence of the meta-validation batch size, we set the batch size of the meta-validation mini-batch to 25, 50, 100 (Note, experiment on CIFAR-10, the batch size of the labeled, unlabeled and meta-validation data are set to 50 as in Tarvainen & Valpola 2017): Laine & Aila (2017) and report the results two variants of our method applied to PL for 250 labels case CIFAR-I1 10 in Fig, More specifically,
optionl using 25, 50, 100 samples as meta-validation data at each training iteration obtain 43.4 %. 43.74
% and 43.72 %, respectively while option2 attains 42.7 % 42.51 % and 42.47 %. These results
11

---

Under review as a conference paper at ICLR 2020
again strongly verify that both option and option2 achieve consistent improvements the PL. In addition, clear that the performance our method is not sensitive to the batch size of the inta--vlidation mini-batch (i.e. less than 0.4 %).
Test error percentage
PL
PL+option1
PL+option2
43.0
42.5
42.0
25
50
100
Epoch
Figure 5: Evaluation of the meta-validation mini-batch's size (250 labels on CIFAR-10).
Experiment 3. We adopt the WideResNeet28-22 as the network for all methods as Berthelot al. (2119). The batch size on CIFAR-10 is set to 32 and it is 128 on CIFAR-100. The learning rate initialized as 0.002 and fixed. We use the same consistency weight and ramp-up schedule (linar-schedule to increase the unsupervised loss weight À as Berthelot et al. (2019).
A.3 REGRESSION EXPERIMENTS ON AFLW
Experiment We adopt the TCDCN proposed in Zhang et al. (2015) as the network and SGD as optimizer. An illustration of the TCDCN's architecture is shown in Fig. 6 We adapt the Pseudo Label (PL), Mean Teacher (MT) and the supervised learning methods as the baselines and our method built PL well MT. To estimate the loss an unlabeled image,
firstly crop image from the original image by moving the cropping window a random number of pixels. We then estimate the location of landmarks on the augmented image and subtract the number of moving pixels, resulting in the pseudo label for the original image. We then apply MSE to the prediction of the original image and the pseudo labels estimate the loss. We use the scheuuec to update the unsupervised loss weight À to in 9000 steps.
AFLW Dataset
Input Image 60x60x3
Ground Truth 5x2
Conv (Kernel: 5, Pilter: 20)
Conv (Kernel: 5, Filter: 48)
Conv (Kernet:3 3, Filter: 80)
MSE
Fully Connected Layer
Fully Connected Layer x10)
Max Pool (2 2)
Max (2 2)
Max Pool (2 2)
Hard lanh
Hard Tanh
Hard Fanh
(720 x256)
3,
Gray Image 60x60x1
Prediction 5x2
Figure 6: Illustration the TCDCN,
12


Reviews:
[{"review_id": "Syeqoi3d9r", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses a meta-learning approach to solve semi-supervised learning. The main idea is to simulate an SGD step on the loss of the meta-validation data and see how the model will perform if the pseudo-labels of unlabelled data are perturbed. Experiments on classification and regression problems show that the proposed method can improve over existing methods. The idea itself is intriguing but the derivation and some design choice are not very well-explained.\n\n(1) The derivation from Eq.(3) to (4) is confusing. Note that in Eq.(3), the prediction \\Phi_\\theta also depends on \\theta in addition to the pseudo-label z. When taking a step of SGD, the second term of Eq.(3) (with unlabelled data) will always be zero if both arguments of the loss (\\Phi_\\theta(x) and z_\\theta(x)) change simultaneously. Eq.(4) somehow only considers the gradient of unsupervised loss, then the gradient would be zero because there is no incentive to deviate from the pseudo-label z. The pseudo-code does not help much. The update from \\hat{\\theta}^{t} to \\hat{\\theta}^{t+1} has the same issue: there is no incentive for \\hat{\\theta}^{t} to deviate because z is exactly produced by it.\n\n(2) For classification problems, it is natural to use cross-entropy loss for the probability vector z. Are there any specific reasons for using Gumbel-softmax? In addition, using L2 loss for probability vectors (as mentioned in Appendix A) is known to be problematic as it may create exponentially many local minima (Auer et al, 1996).\n\n(3) The recent work of Li et al. (2019) also considers iteratively improving pseudo-labels with meta-updates so it should be discussed and compared.\n\n(4) Experiments\n- What are the sizes of the meta-validation sets in the experiments?\n- Error bars in the tables and Fig.2?\n- The MM results in Table 2 are noticeably worse than the original results. For example, with 250 labeled data, MM achieved 11.08% in CIFAR-10 as reported in the original paper. (And 4000 labeled data can achieve 4.95%)\n- It is said that option 2 is consistently better than option 1, which is not true for the MM baseline.\n- 22500 training steps for Experiment 4 seems arbitrary. What are the candidates for the hyper-parameters?\n\nTypos:\n- In the first paragraph of Sec.2, one of the x and one of the y should be bold.\n- Above Eq.(4), x^{U\\in U} should be x^i \\in U\n- The transpose in Eq.(7) is not necessary\n- It is said on page 6 that Fig.2 reports classification loss but the task is a regression problem.\n\nRef\n- Auer, P., Herbster, M. and Warmuth, M.K., 1996. Exponentially many local minima for single neurons. In Advances in neural information processing systems (pp. 316-322).\n- Li, X., Sun, Q., Liu, Y., Zheng, S., Chua, T.S. and Schiele, B., 2019. Learning to Self-Train for Semi-Supervised Few-Shot Classification. In Advances in neural information processing systems.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkxfphJd5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks into problem of semi-supervised learning and in order to be mindful of generalization on the unlabeled data, they add a term to the loss function which includes loss on imputed labels.\nI have 3 main concerns with the paper \n1. The authors mention that  the meta-validation set is a random subset of train set. and they check the final performance on meta-validation set. This does not seem a right way to measure performance of the model as meta-validation set is already used in training. The set of labeled points should be partitioned into train and meta-validation set.\n\n2. The derivation of the updates given the added term to the loss. In option 1, the authors mention they use Eqn. 8 to update z, while Eqn 8. has the reverse information. \n\n3.In option 2, z = \\sigmoid(\\Phi_\\theta) and reducing the loss on z, l( \\sigmoid(\\Phi_\\theta) ,  \\Phi_\\theta), does not look very meaningful. trying to get  \\Phi_\\theta close to its sigmoid means getting it close to zero. but we do not know what is the label for unlabeled data, so why getting the label close to zero?\nAlso the authors mention that second order derivatives will come to play without any explanation. I suggest spending more effort on explaining the problem formulation as that's the core of the paper.\n\nMore comments:\n* As mentioned above the problem formulation is not clean and there are unjustified choice there. Moreover, the experiment results are mostly declared without any justification (for example, the proposed method does not always lead to improvement and not all cases are explained. The authors only note that the method works well in low data regime). \n\n* In the first experiment PL is compared to two cases of the proposed algorithm whereas in other experiments PL is compared to combining PL with versions of the proposed method. Is there a reason for this?\n\n* The models used as baseline are only explained briefly in the last page of the paper, while being used multiple time in the experiment section. This is not good writing practice.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkgLDSmstB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a semi-supervised approach to impute the labels of unlabeled samples such that a network achieves better generalization when it is trained on these labels. The proposed strategy can be easily used to improve the state-of-the-art semi-supervised methods. It mainly uses a validation data set to evaluate the updating rules of the unlabeled samples with pseudo-labels. The proposed method is applicable to both classification and regression problems including image classification and facial landmark detection tasks, which has shown in the experiments. But the following should be improved in the following aspects: \n[1] In the proposed method, the model parameters are updated both on the unlabeled samples and validation data set. The experimental results show that such a strategy is effective to improve the performance of the state-of-the-art method. But why the strategy is effective should be further analyzed.\n[2] How the validation data can improve the generalization ability of the model should be given with theoretical analysis. Whether the size of the validation data has a great influence?\n[3] Some experimental settings are not clear. In the experiments, how many unlabeled data is labeled with pseudo-labels. For different size of the unlabeled data, how many samples should be used in the validation data to evaluate the model with pseudo labeled samples.\n[4] How to divide the training data and the validation data? Whether the validation data need much more that the training data? How about the results only with all the labeled samples, which can further improve the confidence of the proposed method.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is insufficient support to recommend accepting this paper.  The reviewers unanimously criticize the quality of the exposition, noting that many key elements in the main development and experimental set up are not clear.  The significance of the contribution could be made stronger with some form of theoretical analysis.  The current paper lacks depth and insufficient justification for the proposed approach.  The submitted comments should be able to help the authors improve the paper.
Decision: Reject