Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to use the sparse network to mitigate the task-overfitting. The algorithm is based on an iterative training over the pruning phase and retraining phase. A theoretical analysis of the sparsity effects for the Reptile-based first-order meta-learning is provided which indicates if the sparsity level k satisfies k^2 << parameter dimension p, the sparsity can help with the generalization. The paper studies the Reptile + DSD pruning and Reptile + IHT pruning on two standard few-shot classification benchmark.

The paper is in general well-written. My major comments/questions are the following;

1. As the author pointed in the experiment section, there is a mis-match between the experiments and the theory. The output of Algorithm 1 is not sparse though during the iterative training the initialization of the retraining phase is sparse. But the theorem says the generalization is achieved because of sparsity, which requires k^2 << p. The ablation study seems even put more doubts on what the theorem suggests, which basically shows sparsity alone harms the general performance.

2. Is there a typo in Eq (1)? Should the outside loss be evaluated on the query set rather than the support set? If so, does this typo influences the prove of Theorem 1 based on Shalev-Shwartz et.al. 2009?

3. In Figure 1, first of all, what experiments does this figure corresponds to? In Figure 1 (b), the gap between training and testing for both pruning methods are quite large which seems doesn’t solve the overfitting very much? The test traces are intertwined, so it is not clear that the test accuracy get really improved. Using a consistent color for the Figure 1 (a) and Figure 1 (b) can make it much easier to read.

4. The paper needs to discuss about the computational-complexity. It seems each iteration in Algorithm 1 involves meta-training a sparse network and a dense network. And the algorithm needs the number of iteration t > 1. Is there any difficulty in scaling?

5. In the experiments section, for Omniglot almost all results are overlapping with confidence interval. Maybe it should not mark some numbers with bold font. The results in Mini-imagenet show the improvement by proposed methods. Does the effectiveness related to the total number of image classes in dataset?

Some other comments:

1. As the author mentioned, there are two types of overfitting: the meta-level overfitting and task-level overfitting. Why the proposed methods deal with meta-level overfitting rather than task-level overfitting?

2. How does the random mask generated in Algorithm 1?

3. In experiments, can the training accuracy be also provided? 

In general, this paper study an interesting problem in meta-learning and the paper is written in a clear way. The major problems are a mis-match between the theorem and the methods and the experimental results are not very strong. I will give a borderline rating.

############

Thanks for the authors' feedback and I have read them. I am still not convinced about the proposed method about the effectiveness and the trade-off in computation. I hold the concerns about the rebuttal "this typo can be easily fixed with almost no impact on the technical proofs of the theoretical results".  Therefore I maintain the current rating as borderline leaning to rejection.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: 
In this paper, the authors propose a new method to alleviate the effect of overfitting in the meta-learning scenario. The method is based on network pruning. Empirical results demonstrate the effectiveness of the proposed method.

Pros:
+ The problem is very important in the meta-learning field. The model is intuitive and seems useful. In addition, the generalization bound further gives enough insights for the model.

Cons:
- The proposed method is simple and lacks technical contributions. Adding sparse regularizers is a common practice to alleviate over-fitting in the machine learning field. In addition, the retraining process increases the time complexity of the proposed model (i.e., we need to train three times to get the powerful model). 

- In the experiment parts, it will be more interesting if the authors can do the experiments without pre-training. Since in traditional meta-learning settings (e.g., Reptile and MAML), pre-training process does not be introduced. Thus, it might be more convincing if the authors can training the mask and initialization together.


Post rebuttal:

I have read other reviewers and the authors' responses. I still think the contribution is not enough to be accepted. I do not change my mind.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 

============= Post Rebuttal Assessment =============

I appreciate the authors thorough response: I believe the writing has improved considerably and the presentation is more convincing.

All in all, I think it's a good paper but not completely ready for publication based on the following final assessment.

The theoretical contribution of the paper is not entirely relevant to the proposed method as also mentioned by other reviewers. The empirical aspect of the work is incremental; by combining two prior works. In this situation, I would happily suggest acceptance if 1) the experimental setup is very thorough and 2) the results are consistent and conclusive 3) the improvements are significant.

Regarding these, the paper's most interesting result is 5-way 5-shot classification with IHT/DSD + Reptile. I think that makes the paper borderline since the results are not state of the art and the experiments are not thoroughly done, while on the other hand, it does show clear improvement for this case. For such a borderline scenario, and since I have to choose between weak reject and weak accept, I would lean towards weak reject as I believe the experimental setup can be significantly improved by following some of the items below for the next version:

- A thorough ablation study on all the components of IHT and DSD

- A study of the effect of the hyperparameters of IHT/DSD

- Additional datasets/architectures to show consistent improvement for various cases.

- ResNet results are not reliable in the current form. I understand that it would need a sizable computational buget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.

To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.

dget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.

To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.


============= Summary =============
 
The paper addresses the issue of overfitting the meta-parameters of optimization-based meta learning techniques. It uses an iterative training-pruning-retraining setup to learn the meta-parameters. The pruning step in each iteration is to limit the capacity of the meta-learner and thereby alleviate overfitting. Two pruning methods called DSD and IHT are employed on top of a first-order meta learning technique called Reptile. The combination is tested on 4 few-shot classification scenarios of two datasets; omniglot and miniImageNet. Results suggests improved accuracy on top of the Reptile meta-learning algorithm on miniImageNet.
 
 
============= Strengths and Weaknesses =============

+ overfitting the meta learner due to the small number of samples (shots) per task and the large number of meta-parameters and/or base-learner parameters is an important problem in meta learning which is the focus of this work.
+ the results suggest improvements over the Reptile baseline on miniImageNet.
 
 
General concerns:
- abstract: “the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks.” several methods such as CAVIA and MetaOptNet (among many others) address this issue by limiting the capacity of the learner.
- what is the formal definition of meta-generalization and meta-overfitting? This definition will be helpful for understanding the paper’s arguments, for instance, why “reducing meta-overfitting” and “improving meta-generalization” are two different matters (as suggested in the last part of the abstract).
- Figure 1.b: the green bars (proposed method) don’t seem to improve the generalization (testing accuracy). I can only speculate that the figure is to demonstrate that (in the rightmost plot) the difference between train and test accuracy for green bars is less than it is for the red bars. However, one should note that it seems to be due to the training accuracy being lower, which can be achieved, in the extreme case, by a random classifier (zero gap). So, it’s not necessarily useful when testing accuracy is not improved.
- eq (1) and eq (2): the left-most loss L should be on D^{query}
- page 4: “In view of the “lottery ticket hypothesis” (Frankle & Carbin, 2018), the model in equation 2 can be interpreted as a first-order meta-learner for estimating a subnetwork“ -> eq (2) in the current form still requires a 2nd-order derivative and I cannot see how (Frankle & Carbin, 2018) helps make it first-order optimization.
- the usefulness of the provided theory is in question since the final method does not enforce the sparsity in practice. That is, the output of the final meta-parameters are *dense*.
- page 7 mentions “top k_l entries”: what does “top” mean here? k_l dimensions of \theta with the highest absolute value, maybe?
- why do the two separate steps of only training the pruned subnetwork using Reptile and then retrain the whole network (with the updated subnetwork) again using Reptile? One can instead only train Reptile on the whole \theta^{(i)}_M (with L0 norm constrained using the M projection) in a single Reptile step. Doing the former should be justified over the latter since its twice as expensive. Also it should be shown empirically (as an ablation study) that the former works better. 
- reference list: arXiv versions of many papers are cited, it’s good to cite the peer-reviewed version of papers when applicable. For instance, IHT pruning method does not seem to be peer-reviewed which is a caveat for the current work.
- 10 pages seem excessive for the content of the paper. For instance, the experiments section can be shortened extensively and the theories can be put into the appendix.
 
 
Experimental concerns:
- Has the standard miniImageNet split been used? This split is especially important to be respected since CAVIA’s accuracy is taken from the original paper. 
- The reported number for CAVIA in table 2 is not the best number the original paper achieve. The best number is 51.82 ± 0.65% for one-shot, and 65.85 ± 0.55% for 5-shot. 
- There are quite a few new hyperparameters (3 iteration numbers for pretraining, pruning/finetuning, and retraining, and then the sparsity level k_l). It’s important to mention how the hyper-parameters are chosen, especially since they are different for different setups. 
- there seems to be no meta validation set for Omniglot.
- For a fair comparison, the network size of the baseline as well as the learning rate should be searched with the same budget as the hyperparameter search done for the proposed method.  
- ResNet experiment is especially concerning since 1) there is an even higher-level of hyperparameter tuning for ResNet: first conv layer is not pruned, different pruning rates for different residual blocks are used. 2) the baseline is only tried with one setting for the capacity. It is evident in Table 2 that there is a sweet spot for the capacity of the ConvNet baseline, there is no reason this does not apply to ResNet.
- Ablation study:  “For the purpose of fair comparison, only the retraining phase is removed and other settings are the same as proposed in Section 5.1”: this is not fair. I believe a fair comparison would be to repeat the hyperparameter search for the ablation studies with the same budget as the full version of the proposed method.
- Table 2 only compares with CAVIA while many other meta learning methods could be relevant here such as MetOptNet which also limits the number of base learner’s learnable parameters thereby helping with overfitting. Also, CAVIA has results on ResNet useful for the last experiment.
 
 
 
============= Final Decision =============

While the paper addresses an important problem and reports improvements, there are many concerns with it including the writing, method, and experimental setup.  My “weak reject” rating, however, is mainly based on the experimental concerns especially regarding the hyperparameters and the baselines.
 
 
============= Minor Points =============

- code is not provided. I think it’s in general very helpful to release the codes for reproducibility and replicability of the experiments, especially in the case of an incremental study.
- what meta learning task has been used for Figure 1? 
- There are different ways of including batchnorm in a meta learning setup. For instance, Reptile used two different strategies. How is batchnorm implemented for this paper’s experiments? Particularly, indicate if you are using transductive or non-transductive version of batchnorm at test time.
 
The text as well as the math notations require polishing on various occasions including the following (non-exhaustive) list:
- abstract: “sparsity constrained” -> sparsity-constrained
- abstract: “can not only” -> rephrase so that it’s not read as “cannot only”
- abstract and intro: “ease meta-overfitting” can be interpreted as facilitating overfitting -> maybe better to say “alleviate meta-overfitting”
- intro: “memorize the experience from previous meta-tasks for future meta-task learning with very few samples” → memorize the experience from previous tasks for a future task with very few samples
- intro: “As can be observed from Figure 1(a) that” -> It can be observed from Figure 1(a) that
- intro: “sparsity benefits considerably to the” → sparsity benefits considerably the
- intro: “along with two specifications to two widely used networking pruning methods”: rephrase
- related works: “most closely” -> closest
- page 4: “ultra goal -> maybe ultimate goal?
- page 4: \theta is not defined
- page 4: eq (2): i -> m or m -> i
- page 4: the definition of an unordered set (S) coming as a sample of the product of m \Tau spaces is not precise. Better to say each T_i \in \Tau.
- page 4: J_l is not defined,
- sec 3.3: the minimization are done over \theta, it’s better to put that under “min” and the conditions should be clearly indicated by “with or s.t. or similar”
- page 7: zero-one -> binary
- page 7: “ as the restriction of \theta^{(t)} over the mask M^{(t)}” -> rephrase
- page 7: “it deems to reduce” -> rephrase
- page 8: “this is consist” -> consistent
 
- it’s also good to settle on a single term for each concept to make it an easier read, for instance:
- “meta-training tasks” and “training tasks” have been used interchangeably. I think “training tasks” and “mata training dataset” are better choices since a “meta-training” task can refer to the whole task of meta learning.
- Meta-overfitting and meta-level overfitting
- meta-learner, meta-estimator, model
- meta-level training accuracy, meta-training accuracy
- meta training, meta learning
- base learner, learner, model → I think it’s better to emphasize on learner being “base learner”, avoid model as it is not clear what it refers to.
 
 
 
============= Points of improvements =============
The paper would significantly improve if 1) the text is polished and the equations revised. 2) hyperparameter optimization is done carefully and thoroughly described for both the proposed method as well as the baselines



review_title: Official Blind Review #1

Meta-review: This paper proposes a regularization scheme for reducing meta-overfitting. After the rebuttal period, the reviewers all still had concerns about the significance of the paper's contributions and the thoroughness of the empirical study. As such, this paper isn't ready for publication at ICLR. See the reviewer's comments for detailed feedback on how to improve the paper. 
Decision: Reject