Title: META-LEARNING WITH NETWORK PRUNING FOR OVERFITTING REDUCTION

Abstract: Meta-learning has achieved great success in few-shot learning. However, the existing meta-learning models have been evidenced to overfit on training tasks when using over-parameterized convolutional neural networks. This means that the generalization performance of the meta-learner cannot be improved by merely applying more complex networks. To remedy this deficiency, we propose a sparsity-constrained meta-learning approach to learn from training tasks a subnetwork from which first-order optimization methods can quickly converge towards the optimal network in testing tasks. Our theoretical analysis shows the benefit of sparsity for reducing the generalization gap of the estimated meta-learner. We have implemented our approach on top of Reptile assembled with varying network pruning routines including Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different over-parameterized deep networks demonstrate that our method not only effectively alleviates meta-overfitting but also in many cases improves the overall generalization performance when applied to few-shot classification tasks.

1. INTRODUCTION:

The ability of adapting to a new task with several trials is essential for artificial agents. The goal of few-shot learning (Santoro et al., 2016) is to build a model which is able to get the knack of a new task with limited training samples. Meta-learning (Schmidhuber, 1987; Bengio et al., 1990; Thrun & Pratt, 2012) provides a principled way to cast few-shot learning as the problem of learning-to-learn, which typically trains a hypothesis or learning algorithm to memorize the experience from previous tasks for a future task learning with very few samples. The practical importance of meta-learning has been witnessed in many vision and online/reinforcement learning applications including image classification (Ravi & Larochelle, 2016; Li et al., 2017) , multi-arm bandit (Sung et al., 2017) and 2D navigation (Finn et al., 2017) .

Among others, one particularly simple yet successful meta-learning paradigm is first-order optimization based meta-learning which aims to train hypotheses that can quickly adapt to unseen tasks by performing one or a few steps of (stochastic) gradient descent (Ravi & Larochelle, 2016; Finn et al., 2017) . Reasons for the recent increasing attention to this class of gradient-optimization based methods include their outstanding efficiency and scalability exhibited in practice (Nichol et al., 2018) .

Challenge and motivation.:

A challenge in the existing meta-learning approaches is their tendency to overfit (Mishra et al., 2018; Yoon et al., 2018) . When training an over-parameterized meta-learner such as very deep and/or wide convolutional neural networks (CNN) which are powerful for representation learning, there are two sources of potential overfitting at play: the inter-task overfitting of meta-learner (or meta-overfitting) to the training tasks and the inner-task overfitting of task-specific learner to the task training data. There have been recent efforts put to deal with inner-task overfitting (Lee et al., 2019; Zintgraf et al., 2019) . The study on the inter-task meta-overfitting, however, still remains under explored. Since in principle the optimization-based meta-learning is designed to learn fast from small amount of data in new tasks, we expect the meta-overfitting to play a more important role in influencing the overall generalization performance of the trained meta-learner. For an example, it can be observed from the curves in Figure 1 (a) for MiniImageNet that although a 4layer CNN with 128 channels achieves significantly higher training accuracy than the same network architecture with 32 channels, the two networks have quite comparable testing accuracy. It is thus crucial to develop robust methods that can well handle meta-overfitting especially when the number of training tasks is substantially smaller than the number of meta-parameters.

Sparsity model is a promising tool for high-dimensional machine learning with guaranteed statistical efficiency and robustness to overfitting (Maurer & Pontil, 2012; Yuan et al., 2018; Abramovich & Grinshtein, 2019) . It has been theoretically and numerically justified by Arora et al. (2018) that sparsity benefits considerably the generalization performance of deep neural networks. In the regime of compact deep learning, the so called network pruning technique has been widely studied and evidenced to work favorably in generating sparse subnetworks without compromising generalization performance (Frankle & Carbin, 2018; Jin et al., 2016; Han et al., 2016) . Inspired by these remarkable success of sparsity models, it is reasonable to conjecture that sparsity would also be of help for enhancing the robustness of optimization based meta-learning to meta-overfitting.

Our contribution. In this paper, we present a novel sparse optimization based approach for metalearning with over-parameterized neural networks. The problem is formulated as learning a sparse meta-initialization network from training tasks such that in a new task the learned subnetwork can quickly converge to the optimal solution via gradient descent. The core idea is to reduce metaoverfitting by controlling the counts of the non-zero parameters in the meta-learner. Theoretically, we have established a set of generalization gap bounds for the proposed sparse meta-learner showing the benefit of sparsity in improving its generalization performance. Practically, we have implemented our approach in a joint algorithmic framework of Reptile (Nichol et al., 2018) with network pruning, along with two specifications to network pruning methods, which are Dense-Sparse-Dense (DSD) (Han et al., 2016) and Iterative Hard Thresholding (IHT) (Jin et al., 2016) , respectively. The actual performance of our approach has been extensively evaluated on few-shot classification tasks with over-parameterized deep/wide CNNs including ResNet-18 (He et al., 2016) . Numerical results demonstrate that our method can effectively alleviate overfitting and achieve similar or even superior generalization performance to the conventional dense models. As illusrated in Figure 1 (b), using 128-channel network on 5-way 1-shot tasks, our DSD and IHT based Reptile algorithms are effective in closing the gap between the training accuracy and testing accuracy.

2. RELATED WORK:

Optimization-based meta-learning. The family of optimization-based meta-learning approaches usually learns a good hypothesis which can fast adapt to unseen tasks (Ravi & Larochelle, 2016; Finn et al., 2017; Nichol et al., 2018; Khodak et al., 2019) . Compared to the metric (Koch et al., 2015; Snell et al., 2017) and memory (Weston et al., 2014; Santoro et al., 2016) based meta learning algorithms, optimization based meta-learning algorithms are gaining increasing attention due to their simplicity, versatility and effectiveness. As a recent leading framework for optimizationbased meta-learning, MAML (Finn et al., 2017) is designed to estimate a meta-initialization network which can be well fine-tuned in an unseen task via only one or few steps of minibatch gradient descent. Although simple in principle, MAML requires computing Hessian-vector product for back-propagation, which could be computationally expensive when the model is big. The first-order MAML (FOMAML) is therefore proposed to improve the computational efficiency by simply ignoring the second-order derivatives in MAML. Reptile Nichol et al. (2018) is another approximated first-order algorithm which works favorably since it maximizes the inner product between gradients from the same task yet different minibatches, leading to improved model generalization. In (Lee et al., 2019) , the meta-learner is treated as a feature embedding module of which the output is used as input to train a multi-class kernel support vector machine as base learner. To deal with overfitting, the CAVIA method (Zintgraf et al., 2019) decomposes the meta-parameters into the so called context parameters and shared parameters. The context parameters are updated for task adaption with limited capacity while the shared parameters are meta-trained for generalization across tasks.

Network pruning. Early network weight pruning algorithms date back to Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi et al., 1993) . A dense-to-sparse algorithm was developed by Han et al. (2015) to first remove near-zero weights and then fine tune the preserved weights. As a serial work of dense-to-sparse, the dense-sparse-dense (DSD) method (Han et al., 2016) was proposed to re-initialize the pruned parameters as zero and retrain the entire network after the dense-to-sparse pruning phase. The iterative hard thresholding (IHT) method (Jin et al., 2016) shares a similar spirit with DSD to conduct multiple rounds of iteration between pruning and retraining. Srinivas & Babu (2015) proposed a data-free method to prune the neurons in a trained network. In (Louizos et al., 2017) , an L 0 -norm regularized risk minimization framework was proposed to learn sparse networks during training. More recently, Frankle & Carbin (2018) introduced and studied the "lottery ticket hypothesis" which assumes that once a network is initialized, there should exist an optimal subnetwork, which can be learned by pruning, that performs as well as the original net or even superior.

Despite the remarkable success achieved by both meta-learning and network pruning, it still remains largely open to investigate the impact of network pruning on alleviating the meta-overfitting of optimization based meta-learning, which is of primal interest to our study in this paper.

3. META-LEARNING WITH SPARSITY:

In this section, we propose a sparse optimization based meta-learning approach for few-shot learning with over-parameterized neural networks. Our hope is to alleviate the overfitting of meta-learner by reducing its parameter counts without compromising accuracy.

3.1. PROBLEM SETTING:

We consider the N -way K-shot problem as defined in (Vinyals et al., 2016) . Tasks are sampled from a specific distribution p(T ) and will be divided into meta training set S tr , meta validation set S val , and meta testing set S test . Classes in different datasets are disjoint (i.e., the class in S tr will not appear in S test ). During training, each task is made up of support set D supp and query set D query . Both D supp and D query are sampled from the same classes of S tr . D supp is used to for training while D query is used for evaluation. For a N -way K-shot classification task, we sampled N classes from dataset, and then K samples are sampled from each of these classes to form D supp , namely

D supp = {(x k c , y k c ), k = 1, 2, ..., K; c = 1, 2, ..., N }.

For example, for a 5-way 2-shot task, we sample 2 data-label pairs from each of 5 classes, thus, such a task has 10 samples. Usually, several other samples of the same classes will be sampled to compose D query . For example, D query is used in Reptile (Nichol et al., 2018) in evaluation steps. We use the loss function (ŷ, y) to measure the discrepancy between the predicted label ŷ and the true label y.

Notation. For an integer n, we denote [n] as the abbreviation of the index set {1, ..., n}. We use to denote the element-wise product operator. We say a function g :

R p → R is G-Lipschitz continuous if |g(θ) -g(θ )| ≤ G θ -θ 2 , and g is H-smooth if it obeys ∇g(θ) -∇g(θ ) 2 ≤ H θ -θ 2 .

For any sparsity level 1 ≤ s ≤ p, we say a function g is restricted µ s -strongly convex if there exists

µ s > 0 such that g(θ) -g(θ ) -∇g(θ ), θ -θ ≥ µs 2 θ -θ 2 , ∀ θ -θ 0 ≤ s.

Our ultimate goal is to learn a good initialization of parameters for a convolutional neural network f θ : X → Y, where θ is the model parameters set, from a set of training tasks such that the learned initialization network generalizes well to future unseen tasks. Inspired by the recent remarkable success of MAML (Finn et al., 2017) and the strong generalization capability of sparse deep learning models (Frankle & Carbin, 2018; Arora et al., 2018) , we propose to learn from previous task experience a sparse subnetwork started from which the future task-specific networks can be efficiently learned using first-order optimization methods. To this end, we introduce the following layer-wise sparsity constrained stochastic first-order meta-learning formulation:

EQUATION

where

L D supp T (θ) = 1 N K (x k c ,y k c )∈D supp T (f θ (x k c ), y k c

) is the empirical risk for task T , η is a stepsize constant, ||θ l || 0 denotes the number of non-zero entries in the parameters of l-th layer θ l , k l controls the sparsity level of the l-th layer, and L is the total number of layers in the network.

In general, the mathematical formulation of task distribution p(T ) is unknown but we usually have access to a set of i.i.d. training tasks S = {T i } M i=1 sampled from p(T ). Thus the following empirical version of the population form in equation 1 is alternatively considered for training:

EQUATION

To compare with MAML, our model shares an identical objective function but with the layer-wise sparsity constraints θ l 0 ≤ k l imposed for the purpose of enhancing learnability of the overparameterized meta-initialization network. In view of the "lottery ticket hypothesis" (Frankle & Carbin, 2018) , the model in equation 2 can be interpreted as a first-order meta-learner for estimating a subnetwork, or a "winning ticket", for future task learning. Given the strong statistical efficiency and generalization guarantees of sparsity models (Yuan et al., 2018; Arora et al., 2018) , such a subnetwork is expected to achieve advantageous generalization performance over the dense initialization networks learned by vanilla MAML.

3.3. GENERALIZATION PERFORMANCE ANALYSIS:

We now analyze the generalization performance of the sparsity constrained meta-learning model in equation 2. Let p be the total number of parameters in the over-parameterized network and Θ ⊆ R p be the domain of interest for θ. Let k = L l=1 k l be the total desired sparsity level of the subnetwork. Our first result is a uniform convergence bound for the k-sparse neural networks. Theorem 1. Assume that the domain of interest Θ is bounded by R and the loss function (f θ (x), y) is G-Lipschitz continuous and H-smooth with respect to θ. Then for any δ ∈ (0, 1), with probability at least 1 -δ over the random draw of S, the generalization gap is uniformly upper bounded as

sup θ l 0 ≤k l ,l∈[L] |R(θ) -R S (θ)| ≤ O GR(1 + ηH) k log(M ) log(k/δ) + k 2 log(M ) log(ep/k) M .

We remark that the dominant term O k 2 log(M ) log(ep/k)/M in the above uniform convergence bound is paid for not knowing the sparsity pattern of an arbitrary k-sparse subnetwork. Comparing to the O p log(M ) log(p)/M uniform bound in Lemma 1 (see Appendix A.1) for dense networks, the uniform bound established in Theorem 1 is stronger when p k 2 , which shows the benefit of sparsity constraint imposed on the meta-initialization network. Although quite general, the uniform generalization bound established in the above theorem is still relatively loose due to the k 2 factor appeared in the bound. We next show how to improve upon such a uniform bound via further exploring the optimality of the estimated subnetwork.

Let J = {J ⊆ [p] : |J l | = k l } be the set of index set with layer-wise cardinality k l , l ∈ [L],

and let J * := {J : J ∈ J | min supp(θ)=J R(θ) ≤ min supp(θ )∈J R(θ )}. Our analysis also relies on the quantity ∆ := min supp(θ )∈J \J * R(θ ) -min supp(θ)∈J * R(θ) which measures the gap between the smallest k-sparse meta-loss value and the second smallest k-sparse meta-loss value. The following theorem shows the improved generalization gap bound for the sparse meta-learner θ S := arg min supp(θ)∈J R S (θ).

Theorem 2. Assume that the domain of interest is bounded by R and the loss function (f θ (x), y) is G-Lipschitz continuous and H-smooth with respect to θ. Assume that M is sufficiently large such that ∆ = Ω GR(1 + ηH) k log(M ) log(k/δ)+k 2 log(M ) log(ep/k) M . Then for any δ ∈ (0, 1) the generalization gap of θ S is upper bounded by

|R(θ S ) -R S (θ S )| ≤ O GR(1 + ηH) k log(M ) log(k|J * |/δ) M .

Remark 1. As is expected that the larger the gap value ∆ is, the easier the condition 1)). Specially, the following corollary shows that if further assuming restricted strong convexity and certain signal strength condition on the meta-loss function R, then J * will be a singleton and thus the bound in Theorem 2 holds with |J * | = 1. We denote θ min the smallest (in magnitude) nonzero element of θ. Corollary 3. Assume that R(θ) is µ 2k -strongly convex. Suppose that there exists a k-sparse vector

∆ = Ω k 2 log(M ) log(ep/k)/

θ such that θl 0 ≤ k l , ∀l ∈ [L] and θmin > 2 √ 2k ∇R( θ) ∞ /µ 2k .

Assume that the conditions in Theorem 2 hold. Then for any δ ∈ (0, 1) the generalization gap of θ S is upper bounded by

|R(θ S ) -R S (θ S )| ≤ O k log(M ) log(k/δ)/M .

Remark 2. Particularly, if there exists a layer-wise sparse stationary point θ such that ∇R( θ) = 0, then the signal strength condition θmin > 2 √ 2k ∇R( θ) ∞ /µ 2k can be naturally satisfied.

4. ALGORITHM:

We have implemented the proposed sparse meta-learning model 2 based on Reptile (Nichol et al., 2018) which is a scalable method for solving the meta-learning model of the form 2 but without layer-wise sparsity constraint. In order to handle the sparsity constraint, we follow the principles behind the widely applied dense-sparse-dense (DSD) (Han et al., 2016) and iterative hard thresholding (IHT) (Jin et al., 2016) network pruning algorithms to alternate the Reptile iteration between pruning insignificant weights in each layer and retraining the pruned network.

4.1. MAIN ALGORITHM: REPTILE WITH ITERATIVE NETWORK PRUNING:

The algorithm of our network-pruning-based Reptile method is outlined in Algorithm 1. The learning procedure contains a pre-training phase followed by an iterative procedure of network pruning and retraining. We want to emphasize that since our ultimate goal is not to do network compression, but to reduce meta-overfitting via controlling the sparsity level of the meta-initialization network, the final output of our algorithm is typically dense after the retraining phase which has been evidenced in practice to be effective for improving the prediction accuracy. In the following subsections, we describe the key components of our algorithm in details.

4.1.1. MODEL PRETRAINING:

For model pre-training, we run a few number of Reptile iteration rounds to generate a relatively good initialization. In each loop of the Reptile iteration, we first sample a mini-batch of meta-tasks {T i } s i=1 from the task distribution p(T ). Then for each task T i , we compute the adapted parameters via (stochastic) gradient descent as θTi = θ (0) -η∇ θ L D supp T i (θ (0) ), where θTi denotes the taskspecific parameters learned from each task T i , θ (0) is the current initialization of model parameters, η is the inner-task learning rate, and D supp Ti denotes the support set of task T i . When all the taskspecific parameters are updated, the initialization parameters will be updated according to θ (0) = Algorithm 1: Reptile with Iterative Network Pruning Input : inner loop learning rate η, outer loop learning rate β, layer-wise sparsity level {k l } L l=1 , mini-batch batch size s for meta training. Output: θ (t) . Initialization Randomly initialize θ (0) . / * Pre-training with Reptile * / while the termination condition is not met do Sample a mini-batch tasks {T i } s i=1 of size s; For each task T i , compute the task-specific adapted parameters using gradient descent:

θTi = θ (0) -η∇ θ L D supp T i (θ (0) );

Update the parameters: 0) . end for t = 1, 2, ... do / * Pruning phase * / Generate a network zero-one mask M (t) whose non-zero entries at each layer l are those top k l entries in θ t) ; / * Subnetwork fine-tune with Reptile * / while the termination condition is not met do Sample a mini-batch tasks {T i } s i=1 of size s; For each task T i , compute the adapted parameters using gradient descent:

θ (0) = θ (0) + β 1 s s i=1 θTi -θ (

(t) l ; Compute θ (t) M = θ (t) M (

θTi = θ (t) M -η∇ θ L D supp T i (θ (t) M );

Update the parameters:

θ (t) = θ (t) M + β 1 s s i=1 θTi -θ (t) M ; Compute θ (t) M = θ (t) M (t) ; end / * Retraining phase * / Set θ (t) ← θ (t) M ;

while the termination condition is not met do Sample a mini-batch tasks {T i } s i=1 of size s; For each task T i , compute the adapted parameters using gradient descent:

θTi = θ (t) -η∇ θ L D supp T i (θ (t) );

Update the parameters: 0) with learning rate β. Here we follow Reptile to use 1 s s i=1 θTi -θ (0) as an approximation to the negative meta-gradient, which has been evidenced to be effective for scaling up the MAML-type first-order meta-learning models (Nichol et al., 2018) .

θ (t) = θ (t) + β 1 s s i=1 θTi -θ (t) . end end θ (0) +β 1 s s i=1 θTi -θ (

4.1.2. ITERATIVE NETWORK PRUNING AND RETRAINING:

After model pre-training, we proceed to the main loop of our Algorithm 1 that carries out iterative network pruning and retraining.

Pruning phase. In this phase, we first greedily truncate out of the model a portion of near-zero parameters which are unlikely to contribute significantly to the model performance. To do so, we generate a network binary mask M (t) whose non-zero entries at each layer l are those top k l (in magnitude) entries in θ (t) l , and compute θ (t) M = θ (t) M (t) as the sparsity restriction of θ (t) . Then we fine-tune the subnetwork over the mask M (t) by applying Reptile restrictively to this subnetwork with initialization θ (t) M . Our numerical experience suggests that sufficient steps of subnetwork finetuning tends to substantially improve the stability and convergence behavior of the method.

The fine-tuned subnetwork θ (t) M at the end of the pruning phase is expected to reduce the chance of overfitting to noisy data. However, it is also believed that such subnetwork will reduce the capacity of the network, which could in turn lead to potentially biased learning with higher training loss. To remedy this issue, inspired by the retraining trick introduced in (Han et al., 2016) for network pruning, we propose to restore the pruned weights that would be beneficial for enhancing the model representation power to improve the overall generalization performance in average.

Retraining phase. In this phase, the layer-wise sparsity constraints are relaxed and the pruned parameters are re-activated for fine-tuning. The retraining procedure is almost identical to the pretraining phase, but with the main difference that the former is initialized with the subnetwork generated by the pruning phase while the latter uses random initialization. Such a retraining operation restores the representation capacity of the pruned parameters, which tends to lead to improved generalization performance in practice. For theoretical justification, roughly speaking, since the sparse meta-initialization network obtained in the pruning phase generalizes well in light of Theorem 1 and Theorem 2, it is expected to serve as a good initialization for future retraining via gradient descent. Then according to the uniform stability theory of gradient descent methods in (Hardt et al., 2016) the output dense network will also generalize well if the retraining phase converges fast.

4.2. TWO SPECIAL IMPLEMENTATIONS:

Reptile with DSD pruning. The DSD method is an effective network pruning approach for preventing the learned model from capturing noise during the training (Han et al., 2016) . By implementing the main loop with t = 1, the proposed Algorithm 1 reduces to a DSD-based Reptile method for first-order meta-learning.

Reptile with IHT pruning. The IHT method (Jin et al., 2016) is another representative network pruning approach which shares a similar dense-sparse-dense spirit with DSD. Different from the one-shot weight pruning and network training by DSD, IHT is designed to perform multiple rounds of iteration between pruning and retraining, and hence is expected to have better chance to find an optimal sparse subnetwork than DSD does. By implementing the main loop of Algorithm 1 with t > 1, we actually obtain a variant of Reptile with IHT-type network pruning.

5. EXPERIMENTS:

This section is devoted to illustrating the empirical performance of our proposed algorithms when evaluated on two benchmark datasets Omniglot (Lake et al., 2011) and MiniImageNet (Vinyals et al., 2016) . The baseline algorithms considered for comparison include Reptile and CAVIA (Zintgraf et al., 2019) . The model used throughout the experiment contains 4 sequential modules. Each module has a convolutional layer with 3×3 kernel, followed by a batch normalization and a ReLU activation. Additionally for the experiments on MiniImageNet, a 2 × 2 max-pooling pooling is used on the batch normalization layer output while for Omniglot a stride of 2 is used in convolution. The above network structure design is consistent with those considered for Reptile in (Nichol et al., 2018) . We test with varying channel number {32, 64, 128, 256} in each convolution layer to show the robustness of our algorithms to meta-overfitting.

There are several additional hyper-parameters in our approach including the number of iterations for pre-training, pruning and retraining phases. These hyper-parameters are selected from a small number of candidate configurations based on our numerical experience with optimal validation performance.

5.1. FEW-SHOT CLASSIFICATION ON OMNIGLOT:

Dataset. The Omniglot dataset has 1623 characters from 50 alphabets. Each character contains 20 instances drawn by different individuals. The size of each image is 28×28. We randomly select 1200 characters for meta training and the rest are used for meta testing. Following (Santoro et al., 2016) , we also adopt a data augmentation strategy based on image rotation to enhance performance.

Experimental Settings. In DSD-based Reptile training, we set the iteration numbers of pretraining, pruning/fine-tuning and retraining as 3 × 10 4 , 5 × 10 4 and 2 × 10 4 , respectively. For IHT-based Reptile model training, we first pre-train the model. Then we iterate between the sparse model fine-tuning (with 1.5×10 4 iterations) and dense-model retraining (with 5×10 3 iterations) for t = 4 rounds. The setting of other model training related parameters is identical to those in Nichol et al. (2018) . For this group of experiments, the inner-task learning rate is 0.001 and the number of inner-task gradient descent steps is 5. The inter-task learning rate is initialized as 1.0 and the value decays during the entire learning process. 1 show that when using fewer channels (e.g. 32 and 64), both DSD and IHT based Reptile algorithms achieve comparable performance to baselines. When the channel size increases to 128 and 256, slightly improved performance can be observed. This is consistent with our analysis that overfiting is more likely to happen when channel number is relatively large and weight pruning helps to reduce overfitting, which then leads to accuracy improvement with retraining operation.

5.2. FEW-SHOT CLASSIFICATION ON MINIIMAGENET:

Dataset. The MiniImageNet dataset consists of 100 classes from the ImageNet dataset (Krizhevsky et al., 2012) and each class contains 600 images of size 84 × 84×3. There are 64 classes are used for training, 12 classes for validation and 24 classes for testing.

Experimental Settings. For DSD-based Reptile, with 32 channels, we set the iteration numbers for the pre-traning, pruning and retraining phases respectively as 3 × 10 4 , 5 × 10 4 and 2 × 10 4 , while with 64, 128, 256 channels, the corresponding number is 3 × 10 4 , 6 × 10 4 and 10 4 respectively. For IHT-based Reptile, the setting of iteration numbers is the same as those in Section 5.1. Besides, we also evaluate our method on ResNet-18 (He et al., 2016) with DSD-based Reptile. For ResNet-18, we set different pruning rates for residual blocks with different channels. The first convolutional layer is not pruned. For residual block with 64 channels, 128 channels, 256 channels, 512 channels and fully-connected layer, the corresponding pruning rates are 40%, 60%, 60%, 70%, 70% and the corresponding iteration numbers are 3 × 10 4 , 6 × 10 4 , 10 4 iterations, respectively. For this group of experiments, the inner-task learning rate is 0.001 and the number of inner-task iteration loops is 8. The initial inter-task learning rate is 1.0 which will decay during the learning process.

Results. The classification accuracies of DSD-based Reptile approach, IHT-based Reptile approach and baselines on MiniImageNet dataset are presented in Table 2 . More results are provided in Table 5 in Appendix B. From these results, we can observe that our methods consistently outperform the considered baselines. The following observations can be highlighted:

• In the 32-channel setting in which the model is less prone to overfit, when applying DSD-based Reptile with 40% pruning rate, the accuracy gain is 0.5% on 5-way 1-shot tasks and 1.5% on 5-way 5-shot tasks. In the 64-channel setting, our IHT-based Reptile approach respectively improves about 1.5% and 1.95% over the baselines on 5-way 1-shot tasks and 5-way 5-shot tasks. In most cases, the gap between training accuracy and testing accuracy reduces as the pruning rate increases, which confirms that both our DSD and IHT-based Reptile approaches can considerably ease meta-overfitting.

In the setting of 128-channel, the accuracy of DSD-based Reptile on 5-way 1-shot tasks is nearly 3% higher than the baseline while on 5-way 5-shot tasks the gain is about 4.47%.

• As expected that ResNet-18 exhibits severe overfitting and DSD-based Reptile can improve ∼ 8% upon baseline in this case.

• In the 128-channel setting, the accuracies of our approaches are all higher than those of CAVIA, and the highest gain of accuracy is over 3%. 2 that the accuracy of our algorithms tends to increase as the channel size increases while the baselines behave oppositely.

Although in 256-channel case the performance of IHTbased approach drops compared with the 128-channel setting, it still achieves ∼ 1.2% accuracy gain over the baseline on 5-way 1-shot tasks and ∼ 3.32% on 5way 5-shot tasks. These results clearly confirm the robustness of our algorithms to the metaoverfitting suffered from the overparameterization of CNNs.

Figure 2 plots the evolving curves of training accuracy and testing accuracy under varying pruning rates from 0 to 40% for DSD and IHT based Reptile. From these curves we can clearly observe that in most cases, the gap between training accuracy and testing accuracy reduces when the pruning rate increases, which confirms our theoretical predictions in Theorem 1 and Theorem 2 that sparsity matters in alleviating meta-overfitting.

5.3. ABLATION STUDY:

In order to gain better understanding of the impact of dense retraining on the task-specific testing performance, we further carry out an ablation study. For the purpose of fair comparison, only the retraining phase is removed and other settings are the same as proposed in Section 5.1 and 5.2. This Table 3 : Results of ablation study in the 5-way setting. The "±" shows 95% confidence intervals.

Methods:

Retraining 5-way 1-shot 5-way 5-shot Reptile baseline( 64 The results of ablation study are listed in Table 3 . It can be clearly seen from this group of results that the retraining phase plays an important role in the accuracy performance of our method. Under the same pruning rate, without the retraining phase, the accuracy of both DSD-based and IHT-based Reptile approach drops dramatically. For an instance, in the 64-channel case with 40% pruning rate, the variant of IHT-based Reptile without retraining phase suffers from a ∼ 11% drop in accuracy compared with the baseline. On the other side, as shown in Figure 3 that sparsity structure of the network does help to reduce the gap between training accuracy and testing accuracy even with the retraining phase removed. This confirms the benefit of sparsity for generalization gap reduction as revealed by Theorem 1 and 2. Therefore, the network pruning phase makes the model robust to overfitting but in the meanwhile tend to suffer from the deteriorated training loss. The retraining phase helps to restore the capacity of the model to further improve the overall generalization performance.

6. CONCLUSION:

In this paper, we proposed a sparsity-constrained meta-learning approach for overfitting reduction when using over-parameterized neural networks. We have theoretically proved that the generalization gap bounds of the sparse meta-learner have polynomial dependence on the sparsity level rather than the number of parameters. Our approach has been implemented in a scalable meta-learning framework of Reptile with the sparsity level of parameters maintained by network pruning routines including dense-sparse-dense and iterative hard thresholding. Extensive experimental results on benchmark few-shot classification datasets confirm our theoretical predictions and demonstrate the power of network pruning and retraining for improving the generalization performance of gradientoptimization-based meta-learning approaches.

A TECHNICAL PROOFS:

A.1 PROOF OF THEOREM 1

We need the following lemma which guarantees the uniform convergence of R S (θ) towards R(θ) for all θ when the loss function is Lipschitz continuous and smooth, and the optimization is limited on a bounded domain. The lemma is essentially an implication of the uniform convergence bound in (Shalev-Shwartz et al., 2009, Theorem 5) to our problem regime without imposing sparsity constraint.

Lemma 1. Assume that the domain of interest Θ ⊆ R p is bounded by R and the loss function (f θ (x), y) is G-Lipschitz continuous and H-smooth with respect to θ. Then for any δ ∈ (0, 1), the following bound holds with probability at least 1 -δ over the random draw of sample set S for all θ ∈ Θ,

|R(θ) -R S (θ)| ≤ O GR(1 + ηH) p log(M ) log(p/δ) M . Proof. Let us denote f (θ) := L D supp T i θ -η∇ θ L D supp T i (θ) . Since (f θ (x), y) is G-Lipschitz con- tinuous with respect to θ, we can show that |f (θ) -f (θ )| ≤G θ -η∇ θ L D supp T i (θ) -θ + η∇ θ L D supp T i (θ ) ≤G θ -θ + η ∇ θ L D supp T i (θ) -∇ θ L D supp T i (θ ) ≤G(1 + ηH) θ -θ , which indicates that f is G(1 + ηH)-Lipschitz continuous.

The desired bound then follows readily from the uniform convergence bound in (Shalev-Shwartz et al., 2009, Theorem 5) .

Based on this lemma, we can readily prove the main result in the theorem.

Proof of Theorem 1. For any fixed supporting set J ∈ J , by applying Lemma 1 we obtain that the following uniform convergence bound holds for all w with supp(θ) ⊆ J with probability at least 1 -δ over S:

|R(θ) -R S (θ)| ≤ O GR(1 + ηH) k log(M ) log(k/δ) M .

Since by constraint the parameter vector θ is always k-sparse, we thus have supp(θ) ∈ J . Then by union probability we get that with probability at least 1 -δ, the following bound holds for all w with θ 0 ≤ k:

|R(θ) -R S (θ)| ≤ O GR(1 + ηH) k log(M ) log(k/δ) + k log(M ) log(|J |) M .

It remains to bound the cardinality |J |. From (Rigollet, 2015, Lemma 2.7) we know

|J | = p k ≤ ep k

k , which then implies the desired generalization gap bound. This completes the proof.

A.2 PROOF OF THEOREM 2:

Proof of Theorem 2. Let us now consider the following set of events associated with S:

EQUATION

E : S ∈ T M : |R(θ S ) -R S (θ S )| ≤ O GR(1 + ηH) k log(M ) log(k|J * |/δ) M .

We first show that E 1 ⊆ E 2 . Indeed, if the event E 1 occurs, then from the optimality of θ S we can see that for any θ with supp(θ)

∈ J R(θ S ) < R S (θ S ) + ∆ 2 ≤ R S (θ) + ∆ 2 ≤ R(θ) + ∆.

Particularly, for any θ with supp(θ) ∈ J * , the above bound leads to R(θ S ) -R(θ) < ∆ which according to the definition of ∆ immediately implies supp(θ S ) ∈ J * . Therefore, we must have E 1 ⊆ E 2 . By invoking Theorem 1 we know that the following holds with probability at least 1 -δ/2:

sup supp(θ)∈J |R(θ) -R S (θ)| ≤ O GR(1 + ηH) k log(M ) log(k/δ) + k 2 log(M ) log(ep/k) M < ∆ 2 ,

which implies that under the given conditions on M ,

P (E 2 ) ≥ 1 -δ/2. Next we verify that E ∩ E 2 ⊇ E 3 ∩ E 2 .

Indeed, for any S ∈ E 3 ∩ E 2 , it follows immediately that S ∈ E under the condition on the sample size M , and thus S ∈ E ∩ E 2 . By invoking Lemma 1 with union probability over J * we obtain that

P (E 3 ) ≥ 1 -δ/2. Then, we can derive P(E) ≥ P(E ∩ E 2 ) ≥ P(E 3 ∩ E 2 ) ≥ 1 -P(E 3 ) -P(E 2 ) ≥ 1 -δ.

This proves the desired generalization gap bound.

A.3 PROOF OF COROLLARY 3:

The following lemma, which shows that the support of θ * is unique under the provided conditions, is key to the proof the theorem. Lemma 2. Assume that R(θ) is µ 2k -strongly convex. Suppose that there exists a k-sparse vector θ such that θl 0 ≤ k l , ∀l ∈ [L] and θmin > 2 √ 2k ∇R( θ) ∞ µ 2k .

Then J = supp( θ) is the only element in J * .

Proof. Let us consider any fixed J ∈ J * and denote θ * = arg min supp(θ)=J R(θ). Then from the definition of J * we have R(θ * ) ≤ R( θ). Since R(θ) is µ 2k -strongly convex,

R(θ * ) ≥R( θ) + ∇R( θ), θ * -θ + µ 2k 2 θ * -θ 2 ≥R( θ) -∇ J∪ J R( θ) θ * -θ + µ 2k 2 θ * -θ 2 ≥R(θ * ) -∇ J∪ J R( θ) θ * -θ + µ 2k 2 θ * -θ 2 ,

where the second inequality follows from Cauchy-Schwarz inequality. The above inequality then implies

θ * -θ ≤ 2 ∇ J∪ J R( θ) µ 2k ≤ 2 √ 2k ∇R( θ) ∞ µ 2k .

Assume that J = J. Then the above bound leads to

θmin ≤ θ * -θ ≤ 2 √ 2k ∇R( θ) ∞ µ 2k ,

which contradicts the assumption on θ. Therefore, it must be true that J = J for any J ∈ J * .

Proof of Corollary 3. The bound follows readily by combining Lemma 2 and Theorem 2.

rate which drops about 0.65%, and the best one is the case of 64-channel CNNs with 30% pruning rate that improves about 0.1%. On 5-way 5-shot task settings, most cases outperform the baseline.

As shown in the table, even in 32-channel case, when the pruning rate is 20%, the performance of IHT-based Reptile imporves about 0.26%. For 128-channel and 256-channel cases, our method consistently achieves better performance than the baselines. The most important is that with channels added, the baselines keep increasing, but our method can still improve the performance, which can manifest the impressive advantage of our method. Table ?? presents the hyper-parameters used in all the Omniglot experiments.

B.2 RESULTS ON MINIIMAGENET DATASET:

Here we report the detailed results of experiments on MiniImageNet dataset. We conduct our experiments with ResNet-18 and CNNs with 32, 64, 128, 256 channels based on the settings detailed in 5.2.

Table 5 : Few Shot Classification results on MiniImageNet dataset for 4-layer convolutional network with different channels on 5 way setting. The "±" shows 95% confidence intervals over tasks. The evaluation baselines are run by us.

From the table, we can obviously discover that our method achieves remarkable performance. From the perspective of channels, for each case, the performance of our method improves obviously and most results outperform the baselines. Meanwhile, our method reveals more impressive performance with more channels since that while the accuracy of baseline decreases with channels added in the network, the improvement becomes more prominent. For example, with DSD-based Reptile approach on 5-way 1-shot task settings, for 32-channel case, the accuracy of the best performance increases by about 0.5%, but for 64-channel, 128-channel case, the best improvement respectively reaches 2.31% and 4.4%. For IHT-based Reptie settings, we can notice such phenomenon as well. From the perspective of the pruning rate, we notice that for networks with different number of channels, when under the same condition of pruning rate, networks with more channels will achieve better performance. Although the overfitting problem on ResNet-18 is not addressed, it can be observed that our method eases such phenomenon obviously. Our method also outperforms CAVIA (Zintgraf et al., 2019) , which can increase the network size without overfitting. These results demonstrate that by introducing dense-sparse-dense pruning strategy to meta-learning, the pruning phase can help to avoid learning the useless and thus ease the overfitting, and the retraining phase which preserves the capacity of the model assist to improve the performance of the generalization.

:



Figures and Tables:

Figure 1: (a) The meta-overfitting phenomenon on MiniImageNet (5-way 1-shot) tasks when increasing the number of channels from 32 to 128 in a 4-layer CNN. (b) The proposed DSD and IHT based Reptile algorithms are effective in closing the gap between training accuracy and testing accuracy. Bested viewed in color.

Figure 2: Demonstration of overfitting reduction with network pruning: (a) DSD-based Reptile in the setting of 64-channel and 5-way tasks. (b) IHT-based Reptile in the setting of 128-channel and 5-way tasks.In most cases, the gap between training accuracy and testing accuracy reduces as the pruning rate increases, which confirms that both our DSD and IHT-based Reptile approaches can considerably ease meta-overfitting.

Figure 3: Ablation study: Overfitting reduction by DSD and IHT based Reptile without retraining.

M can be fulfilled with respect to the meta-sample size M . Theorem 2 mainly conveys the message that the generalization gap of θ S is upper bounded by O k log(M ) log(k|J * |)/M with high probability. Since |J * | is expected to be much smaller than |J | = O (p/k) k , this bound would be substantially tighter than the corresponding uniform convergence bound established in Theorem 1 when |J

Table content: <table/>

Results on Omniglot with different number of channels and pruning rates. The "±" indicates 95% confidence intervals.

Table content: <table><tr><td>Results. We present the result-</td><td/><td/></tr><tr><td>s of 5-way classification accura-</td><td/><td/></tr><tr><td>cy comparison in Table 1. More</td><td/><td/></tr><tr><td>results are available in Table 4 in</td><td/><td/></tr><tr><td>Appendix B. The results of Rep-tile with varying channel num-ber are treated as baseline result-s. The results in Table</td><td>Methods Reptile baseline (32) Reptile baseline (64) Reptile baseline (128)</td><td>5-way 1-shot 5-way 5-shot 96.63±0.17% 99.31±0.07% 97.68±0.10% 99.48±0.06% 97.99±0.11% 99.60±0.13%</td></tr><tr><td/><td>Reptile baseline (256)</td><td>98.05±0.13% 99.65±0.06%</td></tr><tr><td/><td colspan="2">DSD+Reptile (32, 10%) 96.42±0.17% 99.38±0.07%</td></tr><tr><td/><td colspan="2">DSD+Reptile (64, 10%) 97.64±0.02% 99.50±0.05%</td></tr><tr><td/><td colspan="2">DSD+Reptile (128, 10%) 98.04±0.10% 99.61±0.10%</td></tr><tr><td/><td colspan="2">DSD+Reptile (256, 10%) 98.12±0.12% 99.68±0.05%</td></tr><tr><td/><td>IHT+Reptile (32, 20%)</td><td>96.54±0.17% 99.57±0.06%</td></tr><tr><td/><td>IHT+Reptile (64, 30%)</td><td>97.77±0.15% 99.52±0.06%</td></tr><tr><td/><td colspan="2">IHT+Reptile (128, 20%) 98.22±0.12% 99.64±0.05%</td></tr><tr><td/><td colspan="2">IHT+Reptile (256, 10%) 98.16±0.12% 99.66±0.05%</td></tr></table>

Results on MiniImageNet with different number of channels and pruning rates.

Table content: <table><tr><td>It is also worth noting from Ta-</td><td/><td/></tr><tr><td>ble</td><td/><td/></tr><tr><td>Methods</td><td colspan="2">5-way 1-shot 5-way 5-shot</td></tr><tr><td>Reptile baseline (32)</td><td colspan="2">50.30±0.40% 64.27±0.44%</td></tr><tr><td>Reptile baseline (64)</td><td colspan="2">51.08±0.44% 65.46±0.43%</td></tr><tr><td>Reptile baseline (128)</td><td colspan="2">49.96±0.45% 64.40±0.43%</td></tr><tr><td>Reptile baseline (256)</td><td colspan="2">48.60±0.44% 63.24±0.43%</td></tr><tr><td>CAVIA baseline (32)</td><td colspan="2">47.24±0.65% 59.05±0.54%</td></tr><tr><td>CAVIA baseline (128)</td><td colspan="2">49.84±0.68% 64.63±0.54%</td></tr><tr><td>CAVIA baseline (512)</td><td colspan="2">51.82±0.65% 65.85±0.55%</td></tr><tr><td colspan="3">DSD+Reptile (32, 40%) 50.83±0.45% 65.24±0.44%</td></tr><tr><td colspan="3">DSD+Reptile (64, 30%) 51.91±0.45% 67.23±0.43%</td></tr><tr><td colspan="3">DSD+Reptile (128, 50%) 52.08±0.45% 68.87±0.42%</td></tr><tr><td colspan="3">DSD+Reptile (256, 60%) 53.00±0.45% 68.04±0.42%</td></tr><tr><td>IHT+Reptile(32, 20%)</td><td colspan="2">50.26±0.47% 63.63±0.45%</td></tr><tr><td>IHT+Reptile(64, 40%)</td><td colspan="2">52.59±0.45% 67.41±0.43%</td></tr><tr><td colspan="3">IHT+Reptile(128, 40%) 52.73±0.45% 68.69±0.42%</td></tr><tr><td colspan="3">IHT+Reptile(256, 60%) 49.85±0.44% 66.56±0.42%</td></tr><tr><td>ResNet-18(baseline)</td><td>39.22±0.45%</td><td>-</td></tr><tr><td>DSD+ResNet-18</td><td>47.23±0.47%</td><td>-</td></tr></table>

Peer review (iclr2020):
============= Post Rebuttal Assessment ============= I appreciate the authors thorough response: I believe the writing has improved considerably and the presentation is more convincing. All in all, I think it*s a good paper but not completely ready for publication based on the following final assessment. The theoretical contribution of the paper is not entirely relevant to the proposed method as also mentioned by other reviewers. The empirical aspect of the work is incremental; by combining two prior works. In this situation, I would happily suggest acceptance if 1) the experimental setup is very thorough and 2) the results are consistent and conclusive 3) the improvements are significant. Regarding these, the paper*s most interesting result is 5-way 5-shot classification with IHT/DSD + Reptile. I think that makes the paper borderline since the results are not state of the art and the experiments are not thoroughly done, while on the other hand, it does show clear improvement for this case. For such a borderline scenario, and since I have to choose between weak reject and weak accept, I would lean towards weak reject as I believe the experimental setup can be significantly improved by following some of the items below for the next version: - A thorough ablation study on all the components of IHT and DSD - A study of the effect of the hyperparameters of IHT/DSD - Additional datasets/architectures to show consistent improvement for various cases. - ResNet results are not reliable in the current form. I understand that it would need a sizable computational buget to perform a proper comparison. However, that means either the results should be taken away or done appropriately. To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication. dget to perform a proper comparison. However, that means either the results should be taken away or done appropriately. To reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication. ============= Summary ============= The paper addresses the issue of overfitting the meta-parameters of optimization-based meta learning techniques. It uses an iterative training-pruning-retraining setup to learn the meta-parameters. The pruning step in each iteration is to limit the capacity of the meta-learner and thereby alleviate overfitting. Two pruning methods called DSD and IHT are employed on top of a first-order meta learning technique called Reptile. The combination is tested on 4 few-shot classification scenarios of two datasets; omniglot and miniImageNet. Results suggests improved accuracy on top of the Reptile meta-learning algorithm on miniImageNet. ============= Strengths and Weaknesses ============= + overfitting the meta learner due to the small number of samples (shots) per task and the large number of meta-parameters and/or base-learner parameters is an important problem in meta learning which is the focus of this work. + the results suggest improvements over the Reptile baseline on miniImageNet. General concerns: - abstract: “the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks.” several methods such as CAVIA and MetaOptNet (among many others) address this issue by limiting the capacity of the learner. - what is the formal definition of meta-generalization and meta-overfitting? This definition will be helpful for understanding the paper’s arguments, for instance, why “reducing meta-overfitting” and “improving meta-generalization” are two different matters (as suggested in the last part of the abstract). - Figure 1.b: the green bars (proposed method) don’t seem to improve the generalization (testing accuracy). I can only speculate that the figure is to demonstrate that (in the rightmost plot) the difference between train and test accuracy for green bars is less than it is for the red bars. However, one should note that it seems to be due to the training accuracy being lower, which can be achieved, in the extreme case, by a random classifier (zero gap). So, it’s not necessarily useful when testing accuracy is not improved. - eq (1) and eq (2): the left-most loss L should be on D^{query} - page 4: “In view of the “lottery ticket hypothesis” (Frankle & Carbin, 2018), the model in equation 2 can be interpreted as a first-order meta-learner for estimating a subnetwork“ -> eq (2) in the current form still requires a 2nd-order derivative and I cannot see how (Frankle & Carbin, 2018) helps make it first-order optimization. - the usefulness of the provided theory is in question since the final method does not enforce the sparsity in practice. That is, the output of the final meta-parameters are *dense*. - page 7 mentions “top k_l entries”: what does “top” mean here? k_l dimensions of 	heta with the highest absolute value, maybe? - why do the two separate steps of only training the pruned subnetwork using Reptile and then retrain the whole network (with the updated subnetwork) again using Reptile? One can instead only train Reptile on the whole 	heta^{(i)}_M (with L0 norm constrained using the M projection) in a single Reptile step. Doing the former should be justified over the latter since its twice as expensive. Also it should be shown empirically (as an ablation study) that the former works better. - reference list: arXiv versions of many papers are cited, it’s good to cite the peer-reviewed version of papers when applicable. For instance, IHT pruning method does not seem to be peer-reviewed which is a caveat for the current work. - 10 pages seem excessive for the content of the paper. For instance, the experiments section can be shortened extensively and the theories can be put into the appendix. Experimental concerns: - Has the standard miniImageNet split been used? This split is especially important to be respected since CAVIA’s accuracy is taken from the original paper. - The reported number for CAVIA in table 2 is not the best number the original paper achieve. The best number is 51.82 ± 0.65% for one-shot, and 65.85 ± 0.55% for 5-shot. - There are quite a few new hyperparameters (3 iteration numbers for pretraining, pruning/finetuning, and retraining, and then the sparsity level k_l). It’s important to mention how the hyper-parameters are chosen, especially since they are different for different setups. - there seems to be no meta validation set for Omniglot. - For a fair comparison, the network size of the baseline as well as the learning rate should be searched with the same budget as the hyperparameter search done for the proposed method. - ResNet experiment is especially concerning since 1) there is an even higher-level of hyperparameter tuning for ResNet: first conv layer is not pruned, different pruning rates for different residual blocks are used. 2) the baseline is only tried with one setting for the capacity. It is evident in Table 2 that there is a sweet spot for the capacity of the ConvNet baseline, there is no reason this does not apply to ResNet. - Ablation study: “For the purpose of fair comparison, only the retraining phase is removed and other settings are the same as proposed in Section 5.1”: this is not fair. I believe a fair comparison would be to repeat the hyperparameter search for the ablation studies with the same budget as the full version of the proposed method. - Table 2 only compares with CAVIA while many other meta learning methods could be relevant here such as MetOptNet which also limits the number of base learner’s learnable parameters thereby helping with overfitting. Also, CAVIA has results on ResNet useful for the last experiment. ============= Final Decision ============= While the paper addresses an important problem and reports improvements, there are many concerns with it including the writing, method, and experimental setup. My “weak reject” rating, however, is mainly based on the experimental concerns especially regarding the hyperparameters and the baselines. ============= Minor Points ============= - code is not provided. I think it’s in general very helpful to release the codes for reproducibility and replicability of the experiments, especially in the case of an incremental study. - what meta learning task has been used for Figure 1? - There are different ways of including batchnorm in a meta learning setup. For instance, Reptile used two different strategies. How is batchnorm implemented for this paper’s experiments? Particularly, indicate if you are using transductive or non-transductive version of batchnorm at test time. The text as well as the math notations require polishing on various occasions including the following (non-exhaustive) list: - abstract: “sparsity constrained” -> sparsity-constrained - abstract: “can not only” -> rephrase so that it’s not read as “cannot only” - abstract and intro: “ease meta-overfitting” can be interpreted as facilitating overfitting -> maybe better to say “alleviate meta-overfitting” - intro: “memorize the experience from previous meta-tasks for future meta-task learning with very few samples” ? memorize the experience from previous tasks for a future task with very few samples - intro: “As can be observed from Figure 1(a) that” -> It can be observed from Figure 1(a) that - intro: “sparsity benefits considerably to the” ? sparsity benefits considerably the - intro: “along with two specifications to two widely used networking pruning methods”: rephrase - related works: “most closely” -> closest - page 4: “ultra goal -> maybe ultimate goal? - page 4: 	heta is not defined - page 4: eq (2): i -> m or m -> i - page 4: the definition of an unordered set (S) coming as a sample of the product of m Tau spaces is not precise. Better to say each T_i in Tau. - page 4: J_l is not defined, - sec 3.3: the minimization are done over 	heta, it’s better to put that under “min” and the conditions should be clearly indicated by “with or s.t. or similar” - page 7: zero-one -> binary - page 7: “ as the restriction of 	heta^{(t)} over the mask M^{(t)}” -> rephrase - page 7: “it deems to reduce” -> rephrase - page 8: “this is consist” -> consistent - it’s also good to settle on a single term for each concept to make it an easier read, for instance: - “meta-training tasks” and “training tasks” have been used interchangeably. I think “training tasks” and “mata training dataset” are better choices since a “meta-training” task can refer to the whole task of meta learning. - Meta-overfitting and meta-level overfitting - meta-learner, meta-estimator, model - meta-level training accuracy, meta-training accuracy - meta training, meta learning - base learner, learner, model ? I think it’s better to emphasize on learner being “base learner”, avoid model as it is not clear what it refers to. ============= Points of improvements ============= The paper would significantly improve if 1) the text is polished and the equations revised. 2) hyperparameter optimization is done carefully and thoroughly described for both the proposed method as well as the baselines