Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper uses a meta-learning approach to solve semi-supervised learning. The main idea is to simulate an SGD step on the loss of the meta-validation data and see how the model will perform if the pseudo-labels of unlabelled data are perturbed. Experiments on classification and regression problems show that the proposed method can improve over existing methods. The idea itself is intriguing but the derivation and some design choice are not very well-explained.

(1) The derivation from Eq.(3) to (4) is confusing. Note that in Eq.(3), the prediction \Phi_\theta also depends on \theta in addition to the pseudo-label z. When taking a step of SGD, the second term of Eq.(3) (with unlabelled data) will always be zero if both arguments of the loss (\Phi_\theta(x) and z_\theta(x)) change simultaneously. Eq.(4) somehow only considers the gradient of unsupervised loss, then the gradient would be zero because there is no incentive to deviate from the pseudo-label z. The pseudo-code does not help much. The update from \hat{\theta}^{t} to \hat{\theta}^{t+1} has the same issue: there is no incentive for \hat{\theta}^{t} to deviate because z is exactly produced by it.

(2) For classification problems, it is natural to use cross-entropy loss for the probability vector z. Are there any specific reasons for using Gumbel-softmax? In addition, using L2 loss for probability vectors (as mentioned in Appendix A) is known to be problematic as it may create exponentially many local minima (Auer et al, 1996).

(3) The recent work of Li et al. (2019) also considers iteratively improving pseudo-labels with meta-updates so it should be discussed and compared.

(4) Experiments
- What are the sizes of the meta-validation sets in the experiments?
- Error bars in the tables and Fig.2?
- The MM results in Table 2 are noticeably worse than the original results. For example, with 250 labeled data, MM achieved 11.08% in CIFAR-10 as reported in the original paper. (And 4000 labeled data can achieve 4.95%)
- It is said that option 2 is consistently better than option 1, which is not true for the MM baseline.
- 22500 training steps for Experiment 4 seems arbitrary. What are the candidates for the hyper-parameters?

Typos:
- In the first paragraph of Sec.2, one of the x and one of the y should be bold.
- Above Eq.(4), x^{U\in U} should be x^i \in U
- The transpose in Eq.(7) is not necessary
- It is said on page 6 that Fig.2 reports classification loss but the task is a regression problem.

Ref
- Auer, P., Herbster, M. and Warmuth, M.K., 1996. Exponentially many local minima for single neurons. In Advances in neural information processing systems (pp. 316-322).
- Li, X., Sun, Q., Liu, Y., Zheng, S., Chua, T.S. and Schiele, B., 2019. Learning to Self-Train for Semi-Supervised Few-Shot Classification. In Advances in neural information processing systems.
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: This paper looks into problem of semi-supervised learning and in order to be mindful of generalization on the unlabeled data, they add a term to the loss function which includes loss on imputed labels.
I have 3 main concerns with the paper 
1. The authors mention that  the meta-validation set is a random subset of train set. and they check the final performance on meta-validation set. This does not seem a right way to measure performance of the model as meta-validation set is already used in training. The set of labeled points should be partitioned into train and meta-validation set.

2. The derivation of the updates given the added term to the loss. In option 1, the authors mention they use Eqn. 8 to update z, while Eqn 8. has the reverse information. 

3.In option 2, z = \sigmoid(\Phi_\theta) and reducing the loss on z, l( \sigmoid(\Phi_\theta) ,  \Phi_\theta), does not look very meaningful. trying to get  \Phi_\theta close to its sigmoid means getting it close to zero. but we do not know what is the label for unlabeled data, so why getting the label close to zero?
Also the authors mention that second order derivatives will come to play without any explanation. I suggest spending more effort on explaining the problem formulation as that's the core of the paper.

More comments:
* As mentioned above the problem formulation is not clean and there are unjustified choice there. Moreover, the experiment results are mostly declared without any justification (for example, the proposed method does not always lead to improvement and not all cases are explained. The authors only note that the method works well in low data regime). 

* In the first experiment PL is compared to two cases of the proposed algorithm whereas in other experiments PL is compared to combining PL with versions of the proposed method. Is there a reason for this?

* The models used as baseline are only explained briefly in the last page of the paper, while being used multiple time in the experiment section. This is not good writing practice.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a semi-supervised approach to impute the labels of unlabeled samples such that a network achieves better generalization when it is trained on these labels. The proposed strategy can be easily used to improve the state-of-the-art semi-supervised methods. It mainly uses a validation data set to evaluate the updating rules of the unlabeled samples with pseudo-labels. The proposed method is applicable to both classification and regression problems including image classification and facial landmark detection tasks, which has shown in the experiments. But the following should be improved in the following aspects: 
[1] In the proposed method, the model parameters are updated both on the unlabeled samples and validation data set. The experimental results show that such a strategy is effective to improve the performance of the state-of-the-art method. But why the strategy is effective should be further analyzed.
[2] How the validation data can improve the generalization ability of the model should be given with theoretical analysis. Whether the size of the validation data has a great influence?
[3] Some experimental settings are not clear. In the experiments, how many unlabeled data is labeled with pseudo-labels. For different size of the unlabeled data, how many samples should be used in the validation data to evaluate the model with pseudo labeled samples.
[4] How to divide the training data and the validation data? Whether the validation data need much more that the training data? How about the results only with all the labeled samples, which can further improve the confidence of the proposed method.

review_title: Official Blind Review #3

Meta-review: There is insufficient support to recommend accepting this paper.  The reviewers unanimously criticize the quality of the exposition, noting that many key elements in the main development and experimental set up are not clear.  The significance of the contribution could be made stronger with some form of theoretical analysis.  The current paper lacks depth and insufficient justification for the proposed approach.  The submitted comments should be able to help the authors improve the paper.
Decision: Reject