Meta-Learning with Network Pruning for Overfitting Reduction

Abstract: Meta-Learning has achieved great success in few-shot learning. However, the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks. This means that we cannot improve the meta-generalization performance by merely deepening or widening the networks. To remedy such a deficiency of meta-overfitting, we propose in this paper a sparsity constrained meta-learning approach to learn from meta-training tasks a subnetwork from which first-order optimization methods can quickly converge towards the optimal network in meta-testing tasks. Our theoretical analysis shows the benefit of sparsity for improving the generalization gap of the learned meta-initialization network. We have implemented our approach on top of the widely applied Reptile algorithm assembled with varying network pruning routines including Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different over-parameterized deep networks demonstrate that our method can not only effectively ease meta-overfitting but also in many cases improve the meta-generalization performance when applied to few-shot classification tasks.

Reviews and author responses:


In this paper, the authors propose a new method to alleviate the effect of overfitting in the meta-learning scenario. The method is based on network pruning. Empirical results demonstrate the effectiveness of the proposed method.

Pros:
+ The problem is very important in the meta-learning field. The model is intuitive and seems useful. In addition, the generalization bound further gives enough insights for the model.

Cons:
- The proposed method is simple and lacks technical contributions. Adding sparse regularizers is a common practice to alleviate over-fitting in the machine learning field. In addition, the retraining process increases the time complexity of the proposed model (i.e., we need to train three times to get the powerful model). 

- In the experiment parts, it will be more interesting if the authors can do the experiments without pre-training. Since in traditional meta-learning settings (e.g., Reptile and MAML), pre-training process does not be introduced. Thus, it might be more convincing if the authors can training the mask and initialization together.


Post rebuttal:

I have read other reviewers and the authors' responses. I still think the contribution is not enough to be accepted. I do not change my mind.

The paper proposes to use the sparse network to mitigate the task-overfitting. The algorithm is based on an iterative training over the pruning phase and retraining phase. A theoretical analysis of the sparsity effects for the Reptile-based first-order meta-learning is provided which indicates if the sparsity level k satisfies k^2 << parameter dimension p, the sparsity can help with the generalization. The paper studies the Reptile + DSD pruning and Reptile + IHT pruning on two standard few-shot classification benchmark.

The paper is in general well-written. My major comments/questions are the following;

1. As the author pointed in the experiment section, there is a mis-match between the experiments and the theory. The output of Algorithm 1 is not sparse though during the iterative training the initialization of the retraining phase is sparse. But the theorem says the generalization is achieved because of sparsity, which requires k^2 << p. The ablation study seems even put more doubts on what the theorem suggests, which basically shows sparsity alone harms the general performance.

2. Is there a typo in Eq (1)? Should the outside loss be evaluated on the query set rather than the support set? If so, does this typo influences the prove of Theorem 1 based on Shalev-Shwartz et.al. 2009?

3. In Figure 1, first of all, what experiments does this figure corresponds to? In Figure 1 (b), the gap between training and testing for both pruning methods are quite large which seems doesn’t solve the overfitting very much? The test traces are intertwined, so it is not clear that the test accuracy get really improved. Using a consistent color for the Figure 1 (a) and Figure 1 (b) can make it much easier to read.

4. The paper needs to discuss about the computational-complexity. It seems each iteration in Algorithm 1 involves meta-training a sparse network and a dense network. And the algorithm needs the number of iteration t > 1. Is there any difficulty in scaling?

5. In the experiments section, for Omniglot almost all results are overlapping with confidence interval. Maybe it should not mark some numbers with bold font. The results in Mini-imagenet show the improvement by proposed methods. Does the effectiveness related to the total number of image classes in dataset?

Some other comments:

1. As the author mentioned, there are two types of overfitting: the meta-level overfitting and task-level overfitting. Why the proposed methods deal with meta-level overfitting rather than task-level overfitting?

2. How does the random mask generated in Algorithm 1?

3. In experiments, can the training accuracy be also provided? 

In general, this paper study an interesting problem in meta-learning and the paper is written in a clear way. The major problems are a mis-match between the theorem and the methods and the experimental results are not very strong. I will give a borderline rating.

############

Thanks for the authors' feedback and I have read them. I am still not convinced about the proposed method about the effectiveness and the trade-off in computation. I hold the concerns about the rebuttal "this typo can be easily fixed with almost no impact on the technical proofs of the theoretical results".  Therefore I maintain the current rating as borderline leaning to rejection.




============= Post Rebuttal Assessment =============

I appreciate the authors thorough response: I believe the writing has improved considerably and the presentation is more convincing.

All in all, I think it's a good paper but not completely ready for publication based on the following final assessment.

The theoretical contribution of the paper is not entirely relevant to the proposed method as also mentioned by other reviewers. The empirical aspect of the work is incremental; by combining two prior works. In this situation, I would happily suggest acceptance if 1) the experimental setup is very thorough and 2) the results are consistent and conclusive 3) the improvements are significant.

Regarding these, the paper's most interesting result is 5-way 5-shot classification with IHT/DSD + Reptile. I think that makes the paper borderline since the results are not state of the art and the experiments are not thoroughly done, while on the other hand, it does show clear improvement for this case. For such a borderline scenario, and since I have to choose between weak reject and weak accept, I would lean towards weak reject as I believe the experimental setup can be significantly improved by following some of the items below for the next version:

- A thorough ablation study on all the components of IHT and DSD

- A study of the effect of the hyperparameters of IHT/DSD

- Additional datasets/architectures to show consistent improvement for various cases.

- ResNet results are not reliable in the current form. I understand that it would need a sizable computational buget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.

To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.

dget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.

To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.


============= Summary =============
 
The paper addresses the issue of overfitting the meta-parameters of optimization-based meta learning techniques. It uses an iterative training-pruning-retraining setup to learn the meta-parameters. The pruning step in each iteration is to limit the capacity of the meta-learner and thereby alleviate overfitting. Two pruning methods called DSD and IHT are employed on top of a first-order meta learning technique called Reptile. The combination is tested on 4 few-shot classification scenarios of two datasets; omniglot and miniImageNet. Results suggests improved accuracy on top of the Reptile meta-learning algorithm on miniImageNet.
 
 
============= Strengths and Weaknesses =============

+ overfitting the meta learner due to the small number of samples (shots) per task and the large number of meta-parameters and/or base-learner parameters is an important problem in meta learning which is the focus of this work.
+ the results suggest improvements over the Reptile baseline on miniImageNet.
 
 
General concerns:
- abstract: “the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks.” several methods such as CAVIA and MetaOptNet (among many others) address this issue by limiting the capacity of the learner.
- what is the formal definition of meta-generalization and meta-overfitting? This definition will be helpful for understanding the paper’s arguments, for instance, why “reducing meta-overfitting” and “improving meta-generalization” are two different matters (as suggested in the last part of the abstract).
- Figure 1.b: the green bars (proposed method) don’t seem to improve the generalization (testing accuracy). I can only speculate that the figure is to demonstrate that (in the rightmost plot) the difference between train and test accuracy for green bars is less than it is for the red bars. However, one should note that it seems to be due to the training accuracy being lower, which can be achieved, in the extreme case, by a random classifier (zero gap). So, it’s not necessarily useful when testing accuracy is not improved.
- eq (1) and eq (2): the left-most loss L should be on D^{query}
- page 4: “In view of the “lottery ticket hypothesis” (Frankle & Carbin, 2018), the model in equation 2 can be interpreted as a first-order meta-learner for estimating a subnetwork“ -> eq (2) in the current form still requires a 2nd-order derivative and I cannot see how (Frankle & Carbin, 2018) helps make it first-order optimization.
- the usefulness of the provided theory is in question since the final method does not enforce the sparsity in practice. That is, the output of the final meta-parameters are *dense*.
- page 7 mentions “top k_l entries”: what does “top” mean here? k_l dimensions of \theta with the highest absolute value, maybe?
- why do the two separate steps of only training the pruned subnetwork using Reptile and then retrain the whole network (with the updated subnetwork) again using Reptile? One can instead only train Reptile on the whole \theta^{(i)}_M (with L0 norm constrained using the M projection) in a single Reptile step. Doing the former should be justified over the latter since its twice as expensive. Also it should be shown empirically (as an ablation study) that the former works better. 
- reference list: arXiv versions of many papers are cited, it’s good to cite the peer-reviewed version of papers when applicable. For instance, IHT pruning method does not seem to be peer-reviewed which is a caveat for the current work.
- 10 pages seem excessive for the content of the paper. For instance, the experiments section can be shortened extensively and the theories can be put into the appendix.
 
 
Experimental concerns:
- Has the standard miniImageNet split been used? This split is especially important to be respected since CAVIA’s accuracy is taken from the original paper. 
- The reported number for CAVIA in table 2 is not the best number the original paper achieve. The best number is 51.82 ± 0.65% for one-shot, and 65.85 ± 0.55% for 5-shot. 
- There are quite a few new hyperparameters (3 iteration numbers for pretraining, pruning/finetuning, and retraining, and then the sparsity level k_l). It’s important to mention how the hyper-parameters are chosen, especially since they are different for different setups. 
- there seems to be no meta validation set for Omniglot.
- For a fair comparison, the network size of the baseline as well as the learning rate should be searched with the same budget as the hyperparameter search done for the proposed method.  
- ResNet experiment is especially concerning since 1) there is an even higher-level of hyperparameter tuning for ResNet: first conv layer is not pruned, different pruning rates for different residual blocks are used. 2) the baseline is only tried with one setting for the capacity. It is evident in Table 2 that there is a sweet spot for the capacity of the ConvNet baseline, there is no reason this does not apply to ResNet.
- Ablation study:  “For the purpose of fair comparison, only the retraining phase is removed and other settings are the same as proposed in Section 5.1”: this is not fair. I believe a fair comparison would be to repeat the hyperparameter search for the ablation studies with the same budget as the full version of the proposed method.
- Table 2 only compares with CAVIA while many other meta learning methods could be relevant here such as MetOptNet which also limits the number of base learner’s learnable parameters thereby helping with overfitting. Also, CAVIA has results on ResNet useful for the last experiment.
 
 
 
============= Final Decision =============

While the paper addresses an important problem and reports improvements, there are many concerns with it including the writing, method, and experimental setup.  My “weak reject” rating, however, is mainly based on the experimental concerns especially regarding the hyperparameters and the baselines.
 
 
============= Minor Points =============

- code is not provided. I think it’s in general very helpful to release the codes for reproducibility and replicability of the experiments, especially in the case of an incremental study.
- what meta learning task has been used for Figure 1? 
- There are different ways of including batchnorm in a meta learning setup. For instance, Reptile used two different strategies. How is batchnorm implemented for this paper’s experiments? Particularly, indicate if you are using transductive or non-transductive version of batchnorm at test time.
 
The text as well as the math notations require polishing on various occasions including the following (non-exhaustive) list:
- abstract: “sparsity constrained” -> sparsity-constrained
- abstract: “can not only” -> rephrase so that it’s not read as “cannot only”
- abstract and intro: “ease meta-overfitting” can be interpreted as facilitating overfitting -> maybe better to say “alleviate meta-overfitting”
- intro: “memorize the experience from previous meta-tasks for future meta-task learning with very few samples” → memorize the experience from previous tasks for a future task with very few samples
- intro: “As can be observed from Figure 1(a) that” -> It can be observed from Figure 1(a) that
- intro: “sparsity benefits considerably to the” → sparsity benefits considerably the
- intro: “along with two specifications to two widely used networking pruning methods”: rephrase
- related works: “most closely” -> closest
- page 4: “ultra goal -> maybe ultimate goal?
- page 4: \theta is not defined
- page 4: eq (2): i -> m or m -> i
- page 4: the definition of an unordered set (S) coming as a sample of the product of m \Tau spaces is not precise. Better to say each T_i \in \Tau.
- page 4: J_l is not defined,
- sec 3.3: the minimization are done over \theta, it’s better to put that under “min” and the conditions should be clearly indicated by “with or s.t. or similar”
- page 7: zero-one -> binary
- page 7: “ as the restriction of \theta^{(t)} over the mask M^{(t)}” -> rephrase
- page 7: “it deems to reduce” -> rephrase
- page 8: “this is consist” -> consistent
 
- it’s also good to settle on a single term for each concept to make it an easier read, for instance:
- “meta-training tasks” and “training tasks” have been used interchangeably. I think “training tasks” and “mata training dataset” are better choices since a “meta-training” task can refer to the whole task of meta learning.
- Meta-overfitting and meta-level overfitting
- meta-learner, meta-estimator, model
- meta-level training accuracy, meta-training accuracy
- meta training, meta learning
- base learner, learner, model → I think it’s better to emphasize on learner being “base learner”, avoid model as it is not clear what it refers to.
 
 
 
============= Points of improvements =============
The paper would significantly improve if 1) the text is polished and the equations revised. 2) hyperparameter optimization is done carefully and thoroughly described for both the proposed method as well as the baselines




We thank all reviewers for their constructive comments. Per your suggestions, we have carefully edited our submission. We hope that the given concerns have been addressed satisfactorily in the revised manuscript. Below we provide a summary of changes.
1.	At the end of Section 4.1.2, we add a short discussion on the generalization performance of the dense output from the retraining phase. 
2.	More details about the setting of hyper-parameters are provided in the experiment section.
3.	We have included more results of CAVIA on MiniImageNet for a more complete comparison.
4.	Other changes are minor clarifications that are detailed in our responses to the reviewers.
5.	The code is released along with the revised paper at https://drive.google.com/open?id=1VOY1sCA1j5G1LE2AbDrPoZM-1ZwwVOHA and the corresponding datasets can be available here at https://drive.google.com/open?id=17Cftpney_up0u5SCPb5IuskFplrcAuri


Thank you for the insightful review. We hope the main concerns can be addressed by the following clarification.

Response to general concerns:

1.	We would like to highlight that in contrast to CAVIA and MetaOptNet which handle overfitting by limiting the capacity of the inner-task learner, our method explores another direction of limiting the capacity of the inter-task meta-learner to improve generalization performance across tasks. 

2.	The meta-generalization performance is measured by the population risk of meta-learner, while meta-overfitting is quantified by the gap between the population and empirical risks. We will update the paper to clarify these concepts.  

3.	We address your concern about the curves in Figure 1(b). The main purpose of this figure is to demonstrate the power of our method for overfitting reduction. Concerning the loss in training accuracy of our method, we remark that such a trade-off between training accuracy loss and overfitting reduction is usually the case in sparse learning. In order to remedy this issue, we further propose to use a re-training phase to improve the overall training and testing accuracy. 

4.	Yes, the left-most loss in Equation 2 should be evaluated on the query set. This typo can be easily fixed with almost no impact on the technical proofs of our theoretical results.

5.	Sorry for the confusion. We will remove the misleading term of “first-order” in the revised paper.

6.	We would like to address your concern about the usefulness of our sparse generalization theory. First, the ablation study results in Figure 3 affirmatively confirmed our theoretical prediction by showing that the sparse meta-initialization network did reduce the gap between training and testing accuracy. Second, to further justify the improved generalization accuracy with dense re-training, we comment that since the obtained sparse network generalizes well, it is expected to serve as a good initialization for future re-training via SGD. Then roughly speaking, according to the SGD stability theory in [Hardt et al., 2016] the output dense network will also generalize well (with high probability) if the re-training phase converges fast enough. We will update the paper to have this point clarified. 

[Hardt et al., 2016] Hardt M, Recht B, Singer Y. Train faster, generalize better: Stability of stochastic gradient descent, ICML, 2016: 1225-1234.

7.	Yes, the parameters are ranked according to their absolute values.

8.	The reason to fine tune the subnetwork during the iteration is because according to our numerical experience with IHT network pruning, sufficient steps of subnetwork fine tuning tend to substantially improve the stability and convergence behavior of the method. 

9.	We appreciate your suggestions about reducing paper length and citing peer-reviewed references. We'll do our best to address them in the revised paper.

Response to experimental concerns:

1.	Yes, the standard MiniImageNet split was used in our work to ensure fairness of comparison.

2.	We note the best accuracy results by CAVIA (51.82±0.65% for one-shot, and 65.85±0.55% for 5-shot) on MiniImageNet were reported in the 512-channel case. These results, however, are still inferior to our results obtained even with 64 channels (51.91±0.45%for one-shot, and 67.23±0.65% for 5-shot). We will update Table 2 to have the best available results for CAVIA included for a more complete comparison. 

3.	Due to the limited computational resource, we cannot afford tuning the optimal hyper-parameters via grid search. Alternatively, we have tried a small number of hyper-parameter configurations based on our numerical experience and chose the one with the optimal validation performance. 

4.	In our experiment on Omniglot, we followed the implementation of Reptile to split the 1623 character classes into 1200 training classes (including 100 validation classes for MAML and zero validation class for Reptile) and 423 test classes. 

5.	We would like to clarify that in Table 2, we do have listed a set of results under varying network channels for our method and the baselines as well. We expect the performances might be further improved with more extensive hyper-parameter tuning under additional computation budge.  

6.	Since the primal goal of this paper is to demonstrate the benefit of network pruning in reducing the inter-task overfitting, we focus ourselves on the variant of our method based on Reptile and CNN. We did not extensively tune our model with respect to other backbone network architectures, although the related results in Table 2 have already shown some promise of our method implemented with ResNet-18. 

7.	Per your this suggestion, we have additionally applied the identical rule of hyper-parameter selection to the ablation study. The numerical results indicated that the current choice is still optimal among the considered parameter configurations. 

8.	Regarding the comparison with other relevant methods, we remark that since our method was implemented using Reptile as the backbone meta-learner, we focus on the comparison to Reptile to show the benefit of meta-learning with network pruning. Please keep in mind that our principle of limiting the inter-task network capacity is orthogonal to that of the inner-task overfitting reduction methods such as CAVIA and MetaOptNet. We believe it is possible to develop more sophisticated techniques to achieve the best of both worlds in future study. We will add more discussions on the comparison to these state-of-the-art results in the revised paper. 

Response to minor concerns:

We will fix all the minor issues in the revised paper along with which the code will be released.

Thank you for the insightful review. We hope the main concerns can be addressed by the following clarification. 

1.	The degree of novelty of our method is higher than you suggest. We contribute several new insights on generalization theory and algorithm at the intersection of optimization-based meta-learning and non-convex sparse learning which to our knowledge has not been studied in prior work. Particularly, the sparse generalization theory established in our paper is applicable to non-convex deep models and thus contrasts itself from the existing vast body of results which are mostly about parameter estimation and support recovery consistency for convex problems. 
Concerning the increased computational complexity for achieving the practically improved generalization performance, we believe such a trade-off is reasonable and common in statistical learning with sparsity.

2.	Since we use deep neural nets as the base learner, the pre-training phase should be beneficial for generating a good initialization for further processing with network pruning. Our offline numerical results did suggest that sufficient pre-training always leads to better generalization performance than simply doing meta-training and pruning from scratch.

Thank you for the insightful review. We hope the main concerns can be addressed by the following clarification. 

1.	We would like to clarify that the ablation study actually well supports our generalization theory.

- The results presented in Theorem 1&2 basically show that the meta-generalization gap bounds have polynomial dependence on the sparsity level rather than the size of the meta-initialization network. The ablation study results in Figure 3 affirmatively confirm this theoretical prediction by showing that the sparse meta-initialization network did reduce the gap between training and testing accuracy. Therefore, network pruning is beneficial, both in theory and practice, for making the meta-learner less overfitting. 

- To further justify the improved generalization accuracy with re-training, we comment that since the obtained sparse meta-initialization network generalizes well, it is expected to serve as a good initialization for future re-training via SGD. Then roughly speaking, according to the SGD stability theory in [Hardt et al., 2016] the output dense network will also generalize well (in high probability) if the re-training phase converges fast enough. We will update the paper by explicitly remarking this point in the related discussion. 

[Hardt et al., 2016] Hardt M, Recht B, Singer Y. Train faster, generalize better: Stability of stochastic gradient descent, ICML, 2016: 1225-1234.

2. Yes, the outside risk should be evaluated on the query set. This typo can be easily fixed with almost no impact on the technical proofs of the theoretical results.

3. Sorry for the confusion. The curves in Figure 1 correspond to the experiments on MiniImageNet. Although still being relatively large, the generalization gap of the sparse network is obviously reduced in comparison to the dense network. Per your suggestion, we will update the figure layout for better visualization. 

4. Concerning computational complexity, the generalization performance gain of our method is achieved at the price of increased computational cost mainly due to the additional (iterative) network pruning and re-training steps. We believe such a trade-off is reasonable and common in iterative hard thresholding methods for statistical learning with sparsity. 

5. We agree that the total number of classes should play a role in the performance gap between the considered datasets. We also think that another important factor is the regularity of image. Different from the natural scene images in MiniImageNet, the character images in Omniglot have relatively simpler foreground and background, and thus are easier to be classified.

6. Regarding the motivation to deal with inter-task overfitting rather than the inner-task overfitting, since the optimization-based meta-learning is designed to learn fast from small amount of data in future tasks with the help of meta-learner, we conjecture that the former would have higher impact on the overall generalization performance than the latter. It is absolutely an interesting future work to jointly handle the inter-task and inner-task overfitting, e.g., through simultaneously limiting the capacity of the context parameters for inner-task update as suggested by CAVIA and MetaOptNet. 

7. The binary mask is generalized based on a hard thresholding operation to preserve the desired portion (say 50%) of parameters with top absolute values in each layer. 

8. The training accuracy results on the considered data sets are available for check at: https://drive.google.com/open?id=1OtaajWwga_0dTWx8CM8oObT3S7DJiG_f


Meta-review: This paper proposes a regularization scheme for reducing meta-overfitting. After the rebuttal period, the reviewers all still had concerns about the significance of the paper's contributions and the thoroughness of the empirical study. As such, this paper isn't ready for publication at ICLR. See the reviewer's comments for detailed feedback on how to improve the paper. 

Decision: reject