Meta-Learning with Network Pruning for Overfitting Reduction
Authors: Hongduan Tian
Bo Liu
Xiao-Tong Yuan
Qingshan Liu
Venue:  (2020)
Keywords: Meta-Learning
Few-shot Learning
Network Pruning
Generalization Analysis

Abstract: Meta-Learning has achieved great success in few-shot learning. However, the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks. This means that we cannot improve the meta-generalization performance by merely deepening or widening the networks. To remedy such a deficiency of meta-overfitting, we propose in this paper a sparsity constrained meta-learning approach to learn from meta-training tasks a subnetwork from which first-order optimization methods can quickly converge towards the optimal network in meta-testing tasks. Our theoretical analysis shows the benefit of sparsity for improving the generalization gap of the learned meta-initialization network. We have implemented our approach on top of the widely applied Reptile algorithm assembled with varying network pruning routines including Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different over-parameterized deep networks demonstrate that our method can not only effectively ease meta-overfitting but also in many cases improve the meta-generalization performance when applied to few-shot classification tasks.

Under review as a conference paper at ICLR 2020
LEARNING WITH NETWORK PRUNING FOR OVERFITTING REDUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning has achieved great success in few-shot learning. However, the existing meta-learning models have been evidenced to overfit on training tasks when using over parameterized convolutional neural networks. This means that the generalization performance of the meta-learner cannot be improved by mere- ly applying more complex networks. To remedy this deficiency, we propose a sparsity-constraiee meta-learning approach to learn from training tasks a subnet- work from which first-order optimization methods can quickly converge towards the optimal network in testing tasks. Our theoretical analysis shows the benefit of sparsity for reducing the generalization gap of the estimated meta-learner. We have implemented our approach on top of Reptile assembled with varying net- work pruning routines including Dense-Sparse-Demn Dense (DSD) and Iterative Hard Thresholding (IHT). Extensive experimental results on benchmark datasets with different wee-parameterized deep networks demonstrate that our method not only effectively alleviates meta-overfitting but also in many cases improves the overall generalization performance when applied few-shot classification tasks.
INTRODUCTION
The ability of adapting to a new task with several trials is essential for artificial agents. The goal of few-shot learning (Santoro et al.) 2016 is to build a model which is able to get the knack of new task with limited training samples. Meta-learning (Schmidhuber 1987] Bengio et al.) [990] Thrun & Pratt Pratt provides a principled way to cast few-shot learning as the problem of Sturning-to-leune which typically trains a hypothesis or learning algorithm to memorize the experience from previous tasks for future task learning with very few samples. The practical importance of meta-learning has been witnessed in many vision and online/reinforcemmet learning applications including image classification (Ravi & Larochelle, 2016: Li et al. 2017), multi-arm bandit (Sung et al., 2017 and 2D navigation (Finn et al. 2017).
Among others, particularly simple yet successful meta-learning paradigm is first-order optimiza- tion based meta-learning which aims to train hypotheses that can quickly adapt to unseen tasks by performing one few steps of (stochastic) gradient descent (Ravi & Larochelle| (2066] Finn al.) 2017]. Reasons for the recent increasing attention to this class of graadien-optimmzzaton based meth- ods include their outstanding efficiency and scalability exhibited in practice (Nichol al. 2018
Challenge and motivation. A challenge in the existing meta-learning approaches is their tendency overfit (Mishra et al. 2018 Yoon et al. 2018). When training an werr-parameterized meta-learner such as very deep and/or wide convolutional neural networks (CNN) which are powerful for repre- sentation learning. there are two sources of potential overfitting at play: the task overfitting of meta-lcarner (or meta-on vetaa-overiitting) to the training tasks and the inner-task overfitting of task-specific learner the task training data. There have been recent efforts put to deal with inner-task overfit- ting (Lee et al.) 2019 Zintgraf et al.) 2019). The study on the inter-task meta-overfitting, however, still remains under explored. Since in principle the optimization- based meta-learning is designed to learn fast from small amount of data in new tasks, expect the meta-overfittngg to play more important role in influencing the overall generalization performance of the trained meta- meta-learner. For an example, it can be observed from the curves in Figure [[a) for MinilmageNet that although a 4- layer CNN with 128 channels achieves significantly higher traniing accuracy than the same network architecture with 32 channels, the two networks have quite comparable testing accuracy. It is thus

---

Under review as a conference paper at ICLR 2020
Restile with charness Taptilo with charees
Broeline a5D-a Repole
Baselins INT HTT.aaeeo Bestie
0.7
C.7-
0.6
Jocurscy
04
charpwic, trin
c4
baseline 19 choriee,
0.4
tacaline 120 channer, tunaline 228 charmett EIT pruse 50% thain Pii pruse 30%, test
031
120chernels, trah
chermels,
03-
ESO prunt 46% train
128 cturtee,, tee)
02
C2
USD pruss 40h, test
20000
40000
60000
50000
100000
200003
40000
SC000
89200
300002
20,00
43000 90500 50000
100000
(a) Meta-overfitting
(b) Meta-overfitting reduction by DSD and IHT based Reptile
Figure 1:
(a) The meta-overfitting phenomenon on MinilmageNet (5-way I-shot) tasks when in- creasing the number of channels from 32 to 128 in a 4-layer CNN. (b) The proposed DSD and IHT based Reptile algorithms are effective in closing the gap between training accuracy and testing accuracy. Bested viewed in color.
crucial to develop robust methods that can well handle meta-overfitiing especially when the number of training tasks is substantially smaller than the number of meta-parann meta-paraaeters.
Sparsity model is a promising tool for high-dimensional machine learning with guaranteed statistical efficiency and robustness overfitting (Maurer & Pontil 2012 Yuan al.. 2018 Abramovich & Grinshtein 2019). It has been theoretically and numerically justified by Arora et al. (2018) that s- parsity benefits considerably the generalization performance of deep neural networks. In the regime of compact deep learning, the so called network pruning technique has been widely studied and ev- idenced to work favorably in generating sparse subnetworks without compromising generalization performance (Frankle & Carbin 2018 Jin et al. 2016) Han et al., 2016). Inspired by these remark- able success of sparsity models, it is reasonable to conjecture that sparsity would also be of help for enhancing the robustness of optimization based meta-learning to meta-overfitting.
Our contribution. this paper, we present a novel sparse optimization based approach for meta- learning with per-parrameteriee neural networks. The problem is formulated as learning sparse iota-initialliztiin network from training tasks such that in a new task the learned subnetwork can quickly converge to the optimal solution via gradient descent. The core idea is to reduce meta- overfitting by controlling the counts of the non-zero parameters in the meta-learner. Theoretically, of generalization gap bounds for the proposed sparse meta-learner showing
we have established the benefit of sparsity in improving its generalization performance. Practically, we have implement- ed our approach in joint algorithmic framework of Reptile (Nichol al. 2018) with network pruning, along with two specifications to network pruning methods, which are Dense-Sparse- (DSD) (Han et al., 2016) and Iterative Hard Thresholding (IHT) Q al., 2016). respectively. The actual performance of our approach has been extensively evaluated on few-shot classification tasks with ever-parameterizer deep/wide CNNs including ResNet-18 18 (He et al.) 2016). Numerical re- sults demonstrate that our method can effectively alleviate overfitting and achieve similar or even superior generalization performance to the conventional dense models. As illusrated in Figure [(b] using 128-channel network on 5-way 1-shot tasks, our DSD and IHT based Reptile algorithms are effective in closing the gap between the training accuracy and testing accuracy.
2 RELATED WORK
Optimization- based meta-learning. The family of optimization- based meta- meta-learning approaches usually Iearns a good hypothesis which can fast adapt to unseen tasks (Ravi & Larochelle [2166] Compared to the metric (Koch et al., 2015 Snell et al. 2017) and memory (Weston et al. 2014 Santoro et al.) 2016) based meta learn- ing algorithms, optimization based meta-learning algorithms are gaining increasing attention due to their simplicity, versatility and effectiveness. As a recent leading framework for optimization- based meta-learning. MAML (Finn et al.) 2017) is designed to estimate a meta- meta-initializztion net- work which can be well fine-tuned in unseen task via only or few steps of minibatch gradi- ent descent. Although simple in principle, MAML requires computing dessian-vectur product for back- pack-propagation, which could be computationally expensive when the model is big. The first-order
Finn et al. 2017: Nichol et al., 2018, Khodak et al. 2019).
2

---

Under review as a conference paper at ICLR 2020
MAML (FOMAML) is therefore proposed to improve the computational efficiency by simply ig- noring the second-order derivatives in MAML. Reptile Nichoo et al. (2018) is another approximated first-order algorithm which works favorably since it maximizes the inner product between gradients from the same task yet different minibatches, leading to improved model generalization. In (Lee) et al.) 2019), the meta-learner is treated
feature emhedding module of which the output is used
as imput to train a multi-class kernel support vector machine as base learner. To deal with overfitting, the CAVIA method (Zintgraf et al. 2019) decomposes the meta-parameters into the so called con- text parameters and shared parameters. The context parameters are updated for task adaption with limited capacity while the shared parameters are meta-traing for generalization across tasks.
Network pruning, Early network weight pruning algorithms date back to Optimal Brain Dam- age (LeCun al., 1990) and Optimal Brain Surgeon (Hassibi et al., 1993). A dense-to-spare algo- rithm was developed by Han et al. (2015) to first remove near-zero weights and then fine tune the preserved weights. As serial work of conseeto-ppare, the dense-sparse-dense (DSD) method (Han) eet al. 2016) was proposed to re-initialize the pruned parameters zero and retrain the entire net- work after the dense-to-sparse pruning phase. The iterative hard thresholding (IHT) method (Jin) et al. 2016) shares similar spirit with DSD to conduct multiple rounds of iteration between prun- ing and retraining. (Srinivas & Babu (2013) proposed a data-free method to prune the neurons in a trained network. In (Louizos et al. (017), an La-norm regularized risk minimization framework was proposed to learn sparse networks during training. More recently, Frankle & Carbin (2018) intro- duced and studied the "lottery ticket hypothesis" which assumes that once network initialized, there should exist an optimal subnetwork, which can be learned by pruning, that performs as well as the original net or even superior.
Despite the remarkable success achieved by both meta-learning and network pruning, it still remains largely open to investigate the impact of network pruning on alleviating the meta-overfitting of optimization based meta-learning which is of primal interest to our study in this paper.
3 META-LEARNING WITH SPARSITY
In this section, we propose sparse optimization based meta-learning approach for few-shot learning with werrpaaamettrizze neural networks. Our hope is to alleviate the overfitting of meta-learner by reducing its parameter counts without compromising accuracy.
3.1 PROBLEM SETTING
We consider the N-way K-shot problem as defined in (Vinyals al. 2016). Tasks are sampled from specific distribution p(T) and will be divided into meta training sel Str meta validation sel Stal and meta lesting set Strst Classes in different datasets are disjoint (i.e., the class in Str will not appear in Stest). During training, each task is made up of support set "DuPP and query set Druery Both Deupp and Dquery are sampled from the same classes of S Douup is used to for training while Dowery used for evaluation. For a N N-way K-shot classification task, we sampled N classes from dataset, and then K samples are sampled from each of these classes to form Dzupp, namely Daupp = {==",""), ),), = 1,2,... .... K;c = 1, ....NN For example, for 5-way 2-shot task, we sample 2 data-label pairs from each of 5 classes, thus, such task has 10 samples. Usually, several other samples of the same classes will be sampled to compose Douery, For example, Devery is used in Reptile (Nichol et al. 2018) evaluation steps. We use the loss function E(j. y) to measure the discrepancy between the predicted label and the true label y.
Notation. For an integer n, we denote [n] as the abbreviation of the index set {l...., n). We use O to denote the element- wise product operator. We say a function g RP + IR is G-Lipschitz continuous [g(0)= - (00) < GI 0'|12, and g is H-smooth if obeys [g(0)= For any sparsity level I p. we say function g is restricted n. ua-strongly convex if there exists
Vg(0')]l2 < 0| - 0' I|2
pss > 0 such that g() g(0') (Vg()') (00)) 8¹) 2 da |0 8'112 V|I@ llo < s.
3.2 META-LEARNING WITH SPARSITY
Our ultimate goal is to learn a good initialization of parameters for a convolutional neural network fo X + V, where 0 is the model parameters set, from a set of training tasks such that the learned
3

---

Under review as a conference paper at ICLR 2020
initialization network generalizes well to future unseen tasks. Inspired by the recent remarkable suc- cess of MAML (Finn et al. (017) and the strong generalization capability of sparse deep learning models (Frankle & Carbin 2018 Arora et al., 2018), we propose to learn from previous task expe- rience a sparse subnetwork started from which the future task-specific networks can he efficiently learned using first-order optimization methods. To this end, we introduce the following layer-wise sparsity constrained stochastic first-order meta-learning formulation:
min R(0) = ETT (Eoquary (0 (0-n/Vooye Vo-nVooL (0)] s.t. |10/1lo - k1 I [Z],
(1)
8
1/ Elet at JED ((fo(z))) v²)) is the empirical risk for task T, 7) is step- size constant, ||0:|o denotes the number of non-zero entries in the parameters of I-th layer Or, kn controls the sparsity level of the (-th layer, and L is the total number of layers in the network.
where LD (0) NX
In general, the mathematical formulation of task distribution p(T) is unknown but we usually have access of i.i.d. training tasks S {T sampled from p(T). Thus the following empirical version of the population form in equation I is alternatively considered for training:
M
min Rs(0) = M 1=1
Dpquery
(0 (0- VoVo (0)) s.t. |10rllo S kG, LE [L].
(2)
To compare with MAML, our model shares an identical objective function but with the layer-wise sparsity constraints ||0:|lo < & imposed for the purpose of enhancing learnability of the over- parameterized ietaainitialization network. In view of the "lottery ticket hypothesis" (Frankle & Carbin 2018). the model equation[]caa be interpreted a subnetwork, or a "winning ticker", for future task learning. Given the strong statistical efficien- сy and generalization guarantees of sparsity models (Yuan et al. 2018 Arora al. 2018). such a subnetwork is expected to achieve advantageous generalization performance over the dense initial- ization networks learned by vanilla MAML.
first-order meta-learner for estimating
3.3 GENERALIZATION PERFORMANCE ANALYSIS
We analyze the generalization performance of the sparsity constrained meta-leaming model in equation Let p be the total number of parameters in the over wve-parrametrriee network and θ C Rº be the domain of interest for 9. Let k = Σl=1 ky be the total desired sparsity level of the subnetwork. Our first result is uniform convergence bound for the k-sparse neural networks.
Theorem 1. Assume that the domain of interest θ is bounded by R and the loss function É(Jo (a), y) is G-Lipschitz continuous and H smooth with respect to 0. Then for any E (0, 1), with probability least ó over the random draw of S, the generalization gap is uniformly upper bounded
k log((M log(k/0) + k2 log(M) log(ep/k)
sup - ((0))Rs((0)) s (
| SE(LL
M
V log(MM) (V log(ep/k)/M in the above uniform conver- gence bound is paid for not knowing the sparsity pattern of an arbitrary k-sparse subnetwork. Com- paring to the O VVplog (M) log(p)/M) uniform bound Lemma (see Appendix A.I for dense networks, the uniform bound established Theorem I is stronger when p >>>>>>>>>>>/> A.,, which shows the benefit of sparsity constraint imposed on the meta-inttiaizztion network. Although quite general, the uniform generalization bound established the above theorem is still relatively loose due to the 2 factor appeared in the bound. We next show how to improve upon such uniform bound via further exploring the optimality of the estimated subnetwork.
We remark that the dominant term O
Let J = (J
[p] [J]] kx) be the set of index set with layer-wise cardinality k, E [T]]. and let [J:
mimpi R(0) I minam((p)) R(8")- Our analysis also re- R(A') min R(0) which measures the gap
lies on the quantity A == min between the smallest k-sparse meta-loss value and the second smallest k-sparse meta-loss value. The following theorem shows the improved generalization gap bound for the sparse meta-learner 0g arg min s Ro)
4

---

Under review as a conference paper at ICLR 2020
Theorem 2. Assume that the domain of interest is bounded by R. and the loss function E(fo (x). y) is G-Lipschitz contimuous and H -smooth with respect to 8. Assume that M is sufficiently large such
that A Ω GR(( + +nH) &log(M M. log(k/0 18)+ k2 log( M) log(sp/k)
Then for any 8 (0, the
21
generalization gap of 0s is upper bounded by
(R(0s) - Rs(0s)) = so (CR1
k logg(M log (g((k) (6)) M
+
Remark 1. As is expected that the larger the gap value is, the easier the condition 12 log(( M) log(ep/k)/M) can be fulfilled with respect to the meta-sample size M.
Ω
Theorem mainly conveys the message that the generalization gap of fg is upper bounded by O V k log(M)) Iog(A) with high probability. Since |J is expected to be much smaller than | O ((//k)*), ((+))))) this bound would be substantially tighter than the corresponding uniform convergence bound established in Theorem | when | < |J] (e.g.. | = O(1). Specially, the following corollary shows that further assuming restricted strong convexity and certain signal strength condition on the meta-loss function R. then J will be singleton and thus the bound in Theorem holds with I = 1. We denote @min the smallest (in magnitude) nonzero element of A.
Corollary 3. Assume that R(0) is pax-strongly convex. Suppose that there exists a k-sparse vector O such that |0::oo S k1, E [L] and 0min > 2//22[[ M2k. Assume that the conditions in Theorem hold. Then for any 8 E (0,1) the generalization gup of Og is upper bounded by (R(0s) - (0s())) S (olll=((VRR) (VElog(M) (ogk((//))///
Remark 2. Particularly, there exists layer-wise sparse stationary point @ such that VR(0)) 0. then the signal strength condition 8min 2 ( P2x can be naturally satisfied.
ALGORITHM
We have implemented the proposed sparse meta-learning model 2 based on Reptile (Nichol al., 2018) which is a scalable method for solving the meta-lcarning model of the form 2 but without layer-wise sparsity constraint.
order to handle the sparsity constraint, we follow the principles behind the widely applied Lense-sparss-dente
(DSD) (Han et al. (2166) and iterative hard thresh- olding (IHT) (Jin et al., 2016) network pruning algorithms to alternate the Reptile iteration between pruning insignificant weights in each layer and retraining the pruned network.
4.1 MAIN ALGORITHM: REPTILE WITH ITERATIVE NETWORK PRUNING
The algorithm of our network- teenk--runinnn-aan based Reptile method is outlined in Algorithm The learn- ing procedure contains pre-training phase followed by an iterative procedure of network pruning and retraining. We want to emphasize that since our ultimate goal is not to do network compression, but reduce meta-overfitting via controlling the sparsity level of the mnta-initialization network, the final output of our algorithm is typically dense after the retraining phase which has been evidenced in practice to be effective for improving the prediction accuracy. the following subsections, we describe the key components of our algorithm in details.
4.1.1 MODEL PRETRAINING
For model pre-training, we run few number of Reptile iteration rounds to generate relatively good initialization. In each loop of the Reptile iteration, we first sample a mini-batch of meta-tasks [T from the task distribution p(T). Then for each task T,, we compute the adapted parameters
via (stochastic) gradient descent as 0r. g()) 1) V Dop (O())) where Ør, denotes the task-
specific parameters learned from each task T, (00) is the current initialization of model parameters, y is the inner-task learning rate, and D:. denotes the support set of task T,. When all the task-
specific parameters updated, the initialization parameters will be updated according to ((0)
5

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Reptile with Iterative Network Pruning
Input inner loop learning rate outer loop learning rate 9, layer-wise sparsity level (ki)f=r> mini-batch batch size for meta training.
Output: ((0)
Initialization Randomly initialize 6(0) Pre-training with Reptile
while the termination condition is not met do
Sample mini-batch tasks {Ti of size s;
For each task T. compute the task-specific adapted parameters using gradient descent:
pr. = - - Ve (0(0)
Update the parameters: g(0) = 010) + B ( Ei 0r. - 0(0))
end
for 1 1. do
/* Pruning phase
Generate network zero-one mask M whose non-zero entries at each layer I are those top ky
entries in 8(1;
Compute 0 = g(t) 3 M();
/+ Subnetwork fine-tune with Reptile while the termination condition is not met do
Sample mini-batch tasks {T of size s;
For each task T., compute the adapted parameters using gradient descent:
θr. - NoVoD (0
Update the parameters: ((0) = 000 + 8 (+ (+ 0g: 00/
Compute 0/t = g(t) @ M
end
Retraining phase
Set ((0) 4- 0.)
while the termination condition is not met do
Sample mini-batch tasks {Ti) of size s;
For each task T., compute the adapted parameters using gradient descent:
Ôr. 0()) -1) V/VoDoee (0(1));
Update the parameters: ((t) = p(() + 3 (+Σi= 0r. -(())
end end
0 3 ( (!E:z 0r. o(0) with learning rate B. Here we follow Reptile to use ΣE 07, 0(0)
as an approximation to the negative meta-gradinn graddend, which has been evidenced to be effective for scaling up the MAML-type first-order meta-learning models (Nichol et al. 2018).
4.1.2 ITERATIVE NETWORK PRUNING AND RETRAINING
After model pre-training, we proceed to the main loop of our Algorithm I that carries out iterative network pruning and retraining.
Pruning phase. In this phase, we first greedily truncate out of the model a portion of near-zero parameters which are unlikely to contribute significantly the model performance. To do so, we generate network binary mask M(( whose non-zero entries each layer are those top kr (in

---

Under review as a conference paper at ICLR 2020
magnitude) entries in 0f and compute 00 we fine-tune the subnetwork over the mask (1) by applying Reptile restrictively to this subnetwork with initialization 0/ Our numerical experience suggests that sufficient steps of subnetwork fine- tuning tends substantially improve the stability and convergence behavior of the method.
000
M( the sparsity restriction of 0(0, Then
The fine-tuned subnetwork 8/0 at the end of the pruning phase is expected to reduce the chance of overfitting to noisy data. However, it is also believed that such subnetwork will reduce the capacity of the network, which could in turn lead to potentially biased learning with higher training loss. To remedy this issue, inspired by the retraining trick introduced in (Han al.) 2016) for network pruning, we propose to restore the pruned weights that would be beneficial for enhancing the model representation power improve the overall generalization performance in average.
Retraining phase. In this phase, the layer-wise sparsity constraints are relaxed and the pruned parameters are re-activated for fine-tuning. The retraining procedure almost identical to the pre- training phase, but with the main difference that the former is initialized with the subnetwork gen- erated by the pruning phase while the latter uses random initialization. Such a retraining operation restores the representation capacity of the pruned parameters, which tends to lead to improved gen- eralization performance in practice. For theoretical justification, roughly speaking, since the sparse meta-initialiiztiin network obtained in the pruning phase generalizes well in light of Theorem and Theorem 2 it is expected to serve as a good initialization for future retraining via gradient descent. Then according to the uniform stability theory of gradient descent methods in (Hardt et al., 2016) the output dense network will also generalize well if the retraining phase converges fast.
4.2 TWo SPECIAL IMPLEMENTATIONS
Reptile with DSD pruning. The DSD method is an effective network pruning approach for prevent- ing the learned model from capturing noise during the training (Han et al. 2016). By implementing the main loop with / = 1, the proposed Algorithm [ reduces to a DSD-based Reptile method for first-order meta-learning.
Reptile with IHT pruning, The IHT method (Jin et al, 2016) is another representative network pruning approach which shares similar dense-sparse-dense spirit with DSD. Different from the one-shot weight pruning and network training by DSD, IHT is designed to perform multiple rounds of iteration between pruning and retraining, and hence is expected to have better chance to find an optimal sparse subnetwork than DSD does. By implementing the main loop of Algorithm | with I <>> 1, we actually obtain a variant of Reptile with IHT-type network pruning.
EXPERIMENTS
This section is devoted to illustrating the empirical performance of our proposed algorithms when evaluated on benchmark datasets Omniglot (Lake al. (011)) and MinilmageNet (Vinyals al. 2016). The baseline algorithms considered for comparison include Reptile and CAVIA (Zintgraf 2019 The model used throughout the experiment contains sequential modules. Each module has a convolutional layer with 3x3 kernel, followed by batch normalization and a ReLU activation. Additionally for the experiments on MinilmageNet, 2 2 max pooling pooling used ehperimeemnn on the batch normalization layer output while for Omniglot stride of 2 is used in convolution. The above network structure design is consistent with those considered for Reptile in (Nichol al.) 2018). We test with varying channel number {32, 64, 128, 256} in each convolution layer show the robustness of our algorithms to meta-overfitting.
There are several additional syper-parameters in our approach including the number of iterations for pre-training, pruning and retraining phases. These syper-parameters are selected from a smal- I number of candidate configurations based numerical experience with optimal validation performance.
5.1 FEW-SHOT CLASSIFICATION ON OMNIGLOT
Dataset. The Omniglot dataset has 1623 characters from 50 alphabets. Each character contains 20 instances drawn by different individuals. The size of each image is 28. We randomly select
7

---

Under review as a conference paper at ICLR 2020
1200 characters for meta training and the rest are used for meta testing. Following Santorr al., 2016). we also adopt a data augmentation strategy based on image rotation to enhance performance.
Experimental Settings. In DSD-based Reptile training, we set the iteration numbers of pre- training. pruning/fine-tume tenniingngttggggggggeg and retraining as 3 x 10*, 5 x 10 and x 10*, respectively. For IHT-based Reptile model training. model fine-tuning (with 1.5 x 10s iterations) and dense-mode retraining (with 5 x 10 iterations) for 1 4 rounds. The setting of other model training related parameters is identical those in Nichol et al. 2018). For this group of experiments, the inner-task learning rate is 0.001 and the number of inner-task gradient descent steps The inter-task learning rate is initialized as 1.0 and the value decays during the entire learning process.
first pre-train the model. Then iterate between the sparse
Results. We present the result- s of 5-way classification accura- сy comparison in Table T More results are available in Table 4 in Appendix B The results of Rep- tile with varying channel num- ber are treated as baseline result-
Table I: Results on Omniglot with different number of channels and pruning rates. The "=" indicates 95% confidence intervals.
Methods
5-way 1-shot
5-way 5-shot
Reptile baseline (32)
9..63=0.17%
99.31=0.07%
Reptile baseline (64)
97.68-2.110%
99.48=0.06%
The results in Table [ show
Reptile baseline (128)
97.99=0.11%
99.60=0.13%
that when using fewer channel- (e.g. 32 and 64), both DSD and IHT based Reptile algorithms achieve comparable performance to baselines. When the channel size increases 128 and 256, lightly improved performance can be observed. This is consisten- t with our analysis that overfit- ing is more likely to happen when channel number is relatively large and weight pruning helps to re- duce overfitting, which then leads to accuracy improvement with retraining operation.
Reptile baseline (256)
98.05=00.3%
99.65+0.06%
DSD+Reptile (32, 10%)
96.42-10.17%
99.38%
DSD+Repule (64, 10%)
97.64=0.02%
99.50=0.05%
DSD+Reptile (128, 10%)
98.04=0.10%
99.61+0.10%
DSD+Reptile (256, 10%)
0.112
99.68 0.05%
HeT+Reptile (32, 20%)
96.4=10.77%
99.57%0.06%
ReT+Ree (64, 30%)
0..77=0.15%
99.52+00.06%
Repteee (128, 20%)
98.22=#0122%
0..54%
ReTtRee (256, 10%)
98.16-10.22%
99.66-0.05%
5.2 FEW-SHOT CLASSIFICATION
MINIIMAGENET
Dataset. The MinilmageNet dataset consists of 100 classes from the ImageNet dataset (Krizhevsky et al., 2012) and each class contains 600 images of size 84 84x3. There are 64 classes are used for training, 12 classes for validation and 24 classes for testing.
Experimental Settings. For DSD-based Reptile, with 32 channels, we set the iteration numbers for the pre-traning, pruning and retraining phases respectively as 3 x 10 5 x 10 and 2 10², while with 64, 128, 256 channels, the corresponding number is x 101, 6 x 10a and 10a respectively. For IHT-based Reptile, the setting of iteration numbers is the same those in Section 5. Besides, also evaluate our method on ResNet-18 18 (He et al. 2016) with DSD-based Reptile. For ResNet- 18, we set different pruning rates for residual blocks with different channels. The first convolutional layer is not pruned. For residual block with 64 channels, 128 channels, 256 channels, 512 channels and fully- connected layer, the corresponding pruning rates are 40%. 60%. 60%, 70%. 70% and the corresponding iteration numbers are x 10a, 6 x 10-, 10 iterations, respectively. For this group of experiments, the inner-task learning rate is 0.001 and the number of inner-task iteration loops is 8. The initial inter-task learning rate is 1.0 which will decay during the learning process.
Results. The classification accuracies of DSD-based Reptile approach, IHT-based Reptile ap- proach and baselines on MinilmageNet dataset are presented in Table More results are provided in Table Appendix B From these results, we can observe that our methods consistently outperform the considered baselines. The following observations can be highlighted:
In the 32-channel setting in which the model is less prone to overfit, when applying DSD-based Reptile with 40% pruning rate, the accuracy gain is 0.5% on 5-way 1-shot tasks and 1.5% on 5-way 5-shot tasks. In the 64-channel setting, our IHT-based Reptile approach respectively im- proves about 1.5% and 1.95% over the baselines 5-way 1 1-shot tasks and 5-way 5-shot tasks.
8

---

Under review as a conference paper at ICLR 2020
Accussy 65
Accuricy S
Accurscy
20
80
-3
80
70
65
50
20
40
20
40
20
-0
60
20
40
60
(a) DSD-based Reptile the 64-channel setting
(b) [HT-based Reptile in the 128-channel setting
Figure 2: Demonstration of overfitting reduction with network pruning: (a) DSD-based Reptile in the setting of 64-channel and 5-way tasks. (b) IHT-based Reptile in the setting of 128-channel and 5-way tasks. In most cases, the gap between training accuracy and testing accuracy reduces as the pruning rate increases, which confirms that both our DSD and IHT-based Reptile approaches can considerably ease meta-overfitting.
In the setting of 128-channel, the accuracy of DSD-based Reptile on 5-way 1-shot tasks nearly 3% higher than the baseline while on 5-way 5-shot tasks the gain is about 4.47%.
As expected that ResNet-18 exhibits severe overfitting and DSD-based Reptile can improve ~ 8% upon baseline this case.
In the 128-channel setting, the accuracies our approaches are all higher than those of CAVIA, and the highest gain of accuracy is over 3%.
It is also worth noting from Ta- ble 2 that the accuracy our algorithms tends to increase the channel size increases while the baselines behave opposite- ly. Although in 256-channel case the performance of IHT- based approach drops compared with the 128-channel setting, it still achieves ~ 1.2% accuracy gain over the baseline on 5-way 1-shot tasks and 3.32% on 5- way 5-shot tasks. These result- s clearly confirm the robustness of our algorithms to the meta- overfitting suffered from the over- parameterization of CNNs.
Table 2: Results MinilmageNet with different number of channels and pruning rates.
Methods
5-way 1-shot
5-way 5-shot
Reptile baseline (32)
50.30=10.00%
64.27+0.44%
Reptile baseline (64)
51.08=00.44%
65.46-00.33%
Reptile baseline (128)
49.96-10.45%
64.40:10.43%
Reptile baseline (256)
48.60.10.44%
63.2410.43%
CAVIA baseline (32)
47.24=0.65%
59.05=:0.54%
CAVIA baseline (128)
49.84=0.68%
64.63+0.54%
CAVIA baseline (512)
51.82++.65%
6..85+
DSD+Reptile (32, 40%)
50.8310.45%
65.2410.44%
DSD+Reptile (64, 30%)
51.91=0045%
67.23+0.43%
DSD+Reptile (128, 50%)
52.08-=0.55%
68.87+0.42%
DSD+Reptile (256, 60%)
53.00-0.45%
68.0440.42%
IHT+Reptile(32, 20%)
50.26-0.47%
63.63+0.45%
Figure plots the evolving curves of training accuracy and testing accuracy under varying pruning
IHT+Reptile(44, 40%)
52.5910.45%
67.4110.43%
IHT+Reptilec128, 40%)
0.45%
0.42%
rates from to 40% for DSD and IHT=Reptile(256, 60%)
49.85+0.444
66.56+0.42%
ResNet-
39.52%
IHT based Reptile. From these curves we can clearly observe that in most cases, the gap between training accuracy and testing ac- curacy reduces when the pruning rate increases, which confirms our theoretical predictions in The- orem [ and Theorem 2 that sparsity matters in alleviating meta-overfitiing
DSD+ResNet-18
47.23-0.47%
5.3 ABLATION STUDY
In order to gain better understanding of the impact of dense retraining the task-specific testing performance, we further carry out an ablation study. For the purpose of fair comparison, only the retraining phase is removed and other settings are the same proposed in Section 5.1 and This

---

Under review as a conference paper at ICLR 2020
Table 3: Results of ablation study in the 5-way setting. The "4" shows 95% confidence intervals.
Methods
Retraining
5-way I-shot
5-way 5-shof
Reptile baseline(64)
51.08=0.44%% 49.96=00.55%
63.46+00.33%
Reptile baseline(128)
64.40=00.33%
DSD+Reptile(64, 40%)
X
43.92%10.3%%
60.09:1045%
DSD+Repüle(128, 60%)
47.06::0.4%%
55.07=:0.44%
IHT+Reptile644, 40%)
X
40.03++0.41%
60.59+00.55%
IHT+Reptilec1228, 60%)
52.71+0045%
DSD+Reeptile664, 40%)
51.96%
66.6410.43%
DSD+Reptilc(126 60%)
52.2710.45%
68.4410..2%%
IHT+Reptile664, 40%)
5..59+0.45%
67.41+0.43%
IHT+Reptile(128, 60%)
52.95+0.45%
68.04+0.42%
DSD based Reptiee without retraining
retraining epoch
0.8
0.7-
0.7-
Accarey
Aeraney:
0.6
0.4-
125 anneels,
128 channels, train channels, test
128 arannels, test
0.3-
129 of annelss, rate 60% trein
0.3
128 channels, rate 60% train
129 channels, 60% test
128 cheimeess, rate 60%,
20000
40000
60000
80000
100000
20000
40000
50000
B0000
10000C
terations
iterations
(a) Ablation study on DSD-based Reptile
(b) Ablation study IHT-based Reptile
Figure 3: Ablation study: Overfitting reduction by DSD and IHT based Reptile without retraining.
study is conducted on MinilmageNet dataset for both DSD-based and IHT-based Reptile approaches.
The results of ablation study are listed in Table 3 It can be clearly seen from this group of results that the retraining phase plays an important role in the accuracy performance of our method. Under the same pruning rate, without the retraining phase, the accuracy of both DSD-based and IHT-based Reptile approach drops dramatically, For an instance, in the 64-channel case with 40% pruning rate, the variant of IHT-based Reptile without retraining phase suffers from
11% drop in accuracy compared with the baseline. On the other side, as shown in Figure S that sparsity structure of the network does help reduce the gap between training accuracy and testing accuracy even with the retraining phase removed. This confirms the benefit of sparsity for generalization gap reduction as revealed by Theorem [nd Therefore, the network pruning phase makes the model robust over- fitting but in the meanwhile tend to suffer from the deteriorated training loss. The retraining phase helps restore the capacity of the model to further improve the overall generalization performance.
6 CONCLUSION
In this paper, we proposed a when using over-parameterized neural networks. We have theoretically proved that the generaliza- tion gap bounds of the sparse meta-learner have polynomial dependence on the sparsity level rather than the number of parameters. Our approach has been implemented in a scalable meta-learning framework of Reptile with the sparsity level of parameters maintained by network pruning routines and iterative hard thresholding. Extensive experimental results on
meta-learning approach for overfitting reduction
including dense- cense-sparse-dente benchmark few-shot classification datasets confirm theoretical predictions and demonstrate the power network pruning and retraining for improving the generalization performance of gradient- optimization-ban based meta-learning approaches.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Felix Abramovich and Vadim Grinshtein. High-dimensional classification by sparse logistic regres- sion. IEEE Transactions on Information Theory, 65(5):3088-3079, 2019.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Zhang. Stronger generalization bounds for deep nets via compression approach. In International Conference Machine Learning, pp- 263, 2018.
Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In IJCN, 1990.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1111 1135. JMLR. org, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vaj- da, Manohar Paluri, John Tran, al. Dsd: Dense-sparse-dense training for deep neural networks. arXiv preprint arXiv: 1607.4438,, 2016.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pp. 1225-1234, 2016.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning, In IEEE international conference neural networks, pp. 293-299. IEEE, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Xiaojie Jin, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.05442 2016.
Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. Provable guarantees for gradient- based meta-learning, arXiv preprint arXiv:1902.10644 2019.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances neural 1990.
information processing systems, pp.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Z. Li, Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. 2017.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1 regularization. arXiv preprint arXiv: 172.00/322 2017.

---

Under review as a conference paper at ICLR 2020
Andreas Maurer and Massimiliano Pontil. Structured sparsity and generalization. Journal of Ma- chine Learning Research, (3(Mar)6671-69) 690, 2012.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive
meta-learner. In International Conference on Learning Representations, 2018. URL https:
//openreview wwt/forummiddaaa...eee A
DmUzWAW
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: (XX::9803.00999 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR, 2016.
Philippe Rigollet. 18. s997: High dimensional statistics. Lecture Notes, Cambridge, MA, USA: MIT Open-Course Ware, 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with augmented neural networks. In International conference on machine learn- ing, pp. 1842-1850, 2016.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The meta-meta-... hook.) Diploma thesis, Institut Informatik, Tech. Univ. Munich, 1:2, 1987.
Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex opti- mization. In COLT, 2009.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Suraj Srinivas and R Venkatesh Babu. Data-free parameter pruning for deep neural networks. arXiv preprint arXiv: 1507.06149 2015.
F. Sung, L. Zhang, T. Xiang, T. Hospedales, and Y. Yang. Learning to learn: Meta-critic networks for sample efficient learning. arXiv preprint arXiv: (okiv:/706. (ot:://060092,, 2017.
S. Thrun and L. Pratt. Learning to learn. Springer Science & Business Media, 2012.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pp. 3630-3638, 2016.
Weston, S. Chopra, and A. Bordes. Memory networks. arXiv preprint ar..v: 2014.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances Neural Information Processing Systems, pp. 7332-7342, 2018.
Xiao-Tong Yuan, Ping Li, and Tong Zhang. Gradient hard thresholding pursuit. Journal of Machine Learning Research, 18:1-43 2018.
Luisa Zintgraf. Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning, In International Conference Machine Learning, pp. 7693-7702, 2019.
12

---

Under review as a conference paper at ICLR 2020
A TECHNICAL PROOFS
A. PROOF OF THEOREM 1
We need the following lemma which guarantees the uniform convergence of Rs(()) towards R(0) for all 0 when the loss function is Lipschitz continuous and smooth, and the optimization is lim- ited on bounded domain. The lemma is essentially an implication the uniform convergence bound in (Shalev-Shwartz et al. 2009. Theorem 5) to our problem regime without imposing sparsity constraint.
Lemma Assume that the domain of interest C R is bounded by and the loss function ((fo(w)) v) is G-Lipschitz continuous and H H-smooth with respect to O. Then for any 8 E (0, 1), the over the random draw of sample set for all
following bound holds with probability at least 0 E O.
plog log(p/6)
R((0) Ro(0))
(GR(1 + (
M
Proof. Let denote f(0) LD ( (0))))))))))) Since ((fo(a), v) is G-Lipschitz con- tinuous with respect to 0, we can show that
(f(0) - f(0")) SG 17 VoCD (0) 8' + 17 NVVoD Do
<G [10 0 + V VoLDany (0) VoLDguor (f'))
S + HH) 0
which indicates that / is G(1 + y - continuous. The desired bound then follows readily from the uniform convergence bound in (Shalev-Shwartz al. 2009 Theorem 5).
L
Based this lemma,
readily prove the main result in the theorem.
For any fixed supporting set J J. by applying Lemma we obtain that the following uniform convergence bound holds for all w with supp(0) C J with probability at least 1 over S:
Proof of Theorem 7
k klog( (M log(k/0)
(R(0) Rs(()) SO (RRL+ nt)
M
Since by constraint the parameter vector 0 is always k-sparse, we thus have (0) E J. Then by union probability we get that with probability least 6- 6, the following bound holds for all w with |l@||o < k:
A logg M) log(k/8) + A log( M) log( [T)
log((T)
(R(0) Rs) SO GR( + nH)
M
remains to bound the cardinality [J]. From (Rigollet 2015
Lemma 2.7) we know |J]| = (8) <
(=2)* k which then implies the desired generalization gap bound. This completes the proof.
A.2 PROOF OF THEOREM 2
Proof of Theorem
Let now consider the following of events associated with
A 3 RR(0) -s(())
:: SETM
E (S M supp(Os) e Jo).
sup Supp())E.T
llog(MM log (18)))
Eg S SETM:
sup "app(")
R((0) - Rs(0))
GR( + nH)
M
13

---

Under review as a conference paper at ICLR 2020
and
llog( M) log( k/T (1])
so o &: SETM: (Os)))
(GR(1+n7)) (
M
We first show that El S Eg. Indeed, if the event El occurs, then from the optimality of Os we can see that for any with supp(0) E
R(0s) Rs(Os) A Rs(
A S R(0) + A. 2
Particularly, for any with supp(!) J the above bound leads R(0s)-R(0)) A which according to the definition of A immediately implies supp(0s) E J Therefore, we must have & S Eg. By invoking Theorem we know that the following holds with probability least 5/2:
k loog(M) log(k/6) + A2 log (Mg(M) log(ep/k)
A
sup (R(0) Rs(0)) o (υpp(б)ee
GR([+ 7
M
2
which implies that under the given conditions M,
P ((2)) 2 8/2.
Next we verify that & & 2 &g n &g. Indeed, for any E Eg n Eg, it follows immediately that & under the condition on the sample size M, and thus S & Eg. By invoking Lemma with union probability over J we obtain that
P ( 2 1- 1-/2.
Then, we can derive
(PEE) 2 (P)EE (E 2 1 2- 1P(E)) 2 1
This proves the desired generalization gap bound.
A.3 PROOF OF COROLLARY
The following lemma, which shows that the support of 0* is unique under the provided conditions, key to the proof the theorem.
Lemma 2. Assume that R(0) is pax-strongly convex. Suppose that there exists a k-sparse vector such that |00:lo Pello I k2, VI E [L] and
2v 2
0min
12k
Then J = supp(() is the only element in J
Proof. Let consider any fixed J E J" and denote 0* arg upp()) R(0). Then from the definition of J we have R(0") I R(0). Since R()) is jux-strongly convex,
R(0") (() (V) 0- 0) +
P2k |0 - 0||
2
2R(0) | fu. (0) 0| + 122
|00 0|2
2
2R(0") - Ju. R() 01
U288 | - 0112
where the second inequality follows from implies
inequality. The above inequality then
2 2// VR
|10- Đ1
P2s Assume that # J. Then the above bound leads to
12k
imin < 110 - - 2 VR(O)
U2b
which contradicts the assumption on Ø. Therefore, it must be true that / = J for any J E J
Proof of Corollary 3 The bound follows readily by combining Lemma 2 and Theorem
14

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL EXPERIMENTAL RESULTS
This appendix contains full experimental results for Omniglot, MinilmageNet datasets, the consoli- dated version of the tables found in the main text is included well.
B.1 RESULTS ON OMNIGLOT DATASET
Table Few Shot Classification results on Omniglot dataset for 4-layer convolutional network with shows
different channels 5-way 1-shot and 5-way 5-shot tasks. The "+" shows 95% confidence intervals over tasks. The evaluation baselines are run by us.
Methods
5-way 1-shot
5-way 5-shot
Reptile bascline(32)
96.63-+0.17%
99.31=0.07%
Reptile baseline(64)
97.68=00.0%%
99.48=0.06%
Reptile baseline( 128)
97.99:10.11%
99.60=0.13%%
Reptile baseline(256)
98.05-+0.13%
99.65=0.06%
DSD+ReptileC32, 10%)
96.42+0.17%
99.38=0.07%%
DSD+Reptilc(322 20%)
95.98.1..1%%
99.33=0.07%%
DSD+Reptilc(32, 30%)
96.2210.17%
99.23
DSD+Reptile(32, 40%)
96.53+0.17%
99.37=0.07%
IHT+Reptile(32, 10%)
96.65:1..1%%
99.49-0.06%
IHT+Reptilc(32,222 20%)
96.5440.17%
99.57=0.06%
IHT+Reptile(32,3300 30%)
96.45+0.17%
99.52+0.06%
IHT+Reptiile32, 40%)
96.21+0.18%
0..0%%
DSD+Reptilc(64,109 10%)
97.64.1..2%%
99.50
DSD+Rcptilc644.0%% 20%)
97.60:10.07%
99.49=0.04%%
30%%
97.47=0.05%
99.49=0.05%
DSD+Reptile(44400%)
97.43+1.001%
99.45=0.03%
IHT+Reptilec64, 10%)
97.63++0.14%
99.49
2HT-Reptiler(4422%) 20%)
97.60++0.13%
+.06%
30%%
97.77++0.15%
99.52+0.06%
IHT-Rcptilc(64,4000 40%)
97.5110.1%
99.48-0.07%
DSD4Repthle(128, 10%)
98.04.00110%
10.61
DSD+Reptile(128, 20%)
97.99+0.100%
99.62-1..12%
DSD+Reptile((188.00 30%)
97.9640.12%%
99.63=0.12%%
DSD+Reptile(128,1) 40%)
97.9910.11%%
99.61=10.10%
IHT=Reptilec128, 10%)
18.12.10.2%%
99.63+0.066%
IHT+Roptile(118, 20%)
98.22.10.12%
99.64-0.05%
IHT+Reptile(128, 30%)
98.01+0.113%
99.65
IHT+Roptilcc128,, 40%)
98.06.10.12%
99.63-0.069%
DSD+*Keprile(256, 10%)
98.12.0.12%
99.68 0.05%
DSD+Repptle((56, 20%)
98.02-+0.13%
99.66=0.05%
DSD+Reptilc(256, 30%)
97.96:10.13%
99.67-0.05%
DSD+Reptile(256, 40%)
97.99
99.63-0.06%
IHT=Reptile(256. 10%)
98.16:10.12%
99.6610.05%
IHT+Reptiee(256, 20%)
98.08
99.69
Reptilepil) 30%)
98.05-+0.13%
99.64+0.05%
IHT+Reptile(256, 40%)
97.90++0.13%
99.05%
We performed our methods on CNNs with 32, 64, 128, 256 channels respectively as mentioned in Section [5] The baselines and all the results of Omniglot dataset are reported in Figure 4 For each case, both DSD-based Reptile approach and IHT-based Reptile approach are evaluated on various pruning rates. The settings the same proposed in Section 5.1
For 32-channel case and 64-channel cases, which less prone to be overfitting, both DSD-based Reptile approach and IHT-based Reptile approach can perform well. On 5-way 1-shot task settings, can be observed that the accuracy doesn't it drop too much and can achieve similar performance baselines. The accuracy of the worst result is the of 32-channel CNNs with 20% pruning
15

---

Under review as a conference paper at ICLR 2020
rate which drops about 0.65%, and the best one is the of 64-channel CNNs with 30% pruning rate that improves about 0.1%. On 5-way 5-shot task settings, most cases outperform the baseline. As shown in the table, IHT--based Reptile imporves about 0.26%. For 128-channel and 256-channel cases, our method con- sistently achieves better performance than the baselines. The most important is that with channels added, the baselines keep increasing, but our method can still improve the performance, which can manifest the impressive advantage of our method. Table ?? presents the parrmeeprsmeters used in all the Omniglot experiments.
in 32-channel case, when the pruning rate is 20%, the performance of
B.2 RESULTS ON MINIIMAGENET DATASET
Here report the detailed results of experiments MinilmageNet dataset. We conduct our exper- iments with ResNet-18 18 and CNNs with 32, 64, 128, 256 channels based on the settings detailed in
Table 5: Few Shot Classification results MinilmageNet dataset for 4-layer convolutional network with different channels on 5 way setting. The "4" shows 95% confidence intervals over tasks. The evaluation baselines are run by us.
Methods
5-way 1-shot
5-way 5-shot
Reptile baseline(32)
50.30::0.40%
64.27=0.44%
Reptile baseline(64)
51.08:10.44%
65.46:10.43%
Reptile baseline( (128)
49.96::0.45%
64.40=0.43%
Reptile baseline(256)
48.60:+0.44%
63.24=0043%
CAVIA baseline (32)
47.24.10.65%
59.05=00544%
CAVIA baseline (128)
4..84+10688%
64.63+40.54%
CAVIA baseline (512)
51.82++0.65%
65.85+0.55%
DSD+Reptilc(32, 10%)
50.65
65.29-00.4%%
DSD+Reptiic(32, 20%)
49.9410043%
64.65-0043%
DSD+Reptile(32, 30%)
50.18+00.3%
65.78-0.41%
DSD+Reptile(32, 40%)
50.83:10.45%
65.24=0..4%
IHT+Reptile(32,, 10%)
50.45:10.45%
63.9110.46%
IHT+Reptile(32, 20%)
50.26:10.47%
63.63:10.45%
IHT+Reptile(32, 30%)
50.21+10.44%
65.05=0.45%
IHT+Reptilc(32, 40%)
49.7410.46%
64.15=0045%
DSD+Reppilc(64, 10%)
50.98.
65.3120.43%
DSD+Reppilec04,2009 20%)
51.91+0.45%
67.21+0.43%
DSD+Reppille(4.30%%
51.91+00.5%
6..23
DSD+Rcpiilc(64.40% 40%)
51.96.10.45%
66.64
IHT+Reptilee64, 10%)
52.23:10.55%
66.08%0..3%%
HHT*Reptile 20%)
52.13+00.66%
66.78=0.43%
IHT+Reptile(64,300 30%)
51.98:40.45%
66.1440.43%
IHT=Reptilec64, 40%)
1
67.41
DSD+*Kepiile1128,33 30%)
50.95%
00.43%
DSD+Reppile(118. 40%)
52.15+0.45%
68. 69.14%0043%
DSD+Reptile(12,,500 50%)
52.08.10.45%
68.87.10.22%
DSD+Reptile(128.G0) 60%)
52.27-10.45%
68.44-0..2%%
IHT=Reptile(128, 30%)
51.64410.5%
67.05=0.43%
IHT+Reptile(128.ce 40%)
52.73:+0.45%
68.69-0.422%
IHT=Reptiile118, 50%)
52.76.L0.45%
67.63-10.43%
IHT=#Reprile118.. 60%)
1..45%
DSD=RReprile(556, 60%)
53.00 12.05.0.457 0.45%
68.04 0.42%
IHT+Roptüle(256. 60%)
49.85.10.44%
66.56_0.42%
ResNet-I8(basllne)
39.22.10.45%
D8D+ResNet-18
47.23-10.47%
16

---

Under review as a conference paper at ICLR 2020
From the table, obviously discover that our method achieves remarkable performance. From the perspective of channels, for each case, the performance our method improves obviously and most results outperform the baselines. Mcanwhile, our method reveals more impressive perfor- mance with more channels since that while the accuracy of baseline decreases with channels added prominent. For example, with DSD-based Reptile
in the network, the improvement becomes approach on 5-way 1-shot task settings, for 32-chamnel case, the accuracy of the best performance increases by about 0.5%, but for 64-channel, 128-channel case, the best improvement respective- ly reaches 2.31% and 4.4%. For IHT--bsed Reptie settings, we can notice such phenomenon as well. From the perspective of the pruning rate, we notice that for networks with different number of channels, when under the same condition of pruning rate, networks with more channels will achieve better performance. Although the overfitting problem on ResNet-18 not addressed, it can ob- served that our method eases such phenomenon obviously. Our method also outperforms CAVIA Zintgraf et al. 2019). which can increase the network size without overfitting. These results demon- strate that by introducing dense- seseessppaeeeeessss pruning strategy meta-learning, the pruning phase can help to avoid learning the useless and thus ease the overfitting, and the retraining phase which preserves the capacity of the model assist to improve the performance of the generalization.
17


Reviews:
[{"review_id": "Syg1bRkLYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n============= Post Rebuttal Assessment =============\n\nI appreciate the authors thorough response: I believe the writing has improved considerably and the presentation is more convincing.\n\nAll in all, I think it's a good paper but not completely ready for publication based on the following final assessment.\n\nThe theoretical contribution of the paper is not entirely relevant to the proposed method as also mentioned by other reviewers. The empirical aspect of the work is incremental; by combining two prior works. In this situation, I would happily suggest acceptance if 1) the experimental setup is very thorough and 2) the results are consistent and conclusive 3) the improvements are significant.\n\nRegarding these, the paper's most interesting result is 5-way 5-shot classification with IHT/DSD + Reptile. I think that makes the paper borderline since the results are not state of the art and the experiments are not thoroughly done, while on the other hand, it does show clear improvement for this case. For such a borderline scenario, and since I have to choose between weak reject and weak accept, I would lean towards weak reject as I believe the experimental setup can be significantly improved by following some of the items below for the next version:\n\n- A thorough ablation study on all the components of IHT and DSD\n\n- A study of the effect of the hyperparameters of IHT/DSD\n\n- Additional datasets/architectures to show consistent improvement for various cases.\n\n- ResNet results are not reliable in the current form. I understand that it would need a sizable computational buget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.\n\nTo reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.\n\ndget to perform a proper comparison. However, that means either the results should be taken away or done appropriately.\n\nTo reiterate, the reason I am asking for the above experiments is that the paper comes with incremental novelty as the main point, in my opinion. Thus it has to be backed up by extensive experiments to be confidently considered for publication.\n\n\n============= Summary =============\n \nThe paper addresses the issue of overfitting the meta-parameters of optimization-based meta learning techniques. It uses an iterative training-pruning-retraining setup to learn the meta-parameters. The pruning step in each iteration is to limit the capacity of the meta-learner and thereby alleviate overfitting. Two pruning methods called DSD and IHT are employed on top of a first-order meta learning technique called Reptile. The combination is tested on 4 few-shot classification scenarios of two datasets; omniglot and miniImageNet. Results suggests improved accuracy on top of the Reptile meta-learning algorithm on miniImageNet.\n \n \n============= Strengths and Weaknesses =============\n\n+ overfitting the meta learner due to the small number of samples (shots) per task and the large number of meta-parameters and/or base-learner parameters is an important problem in meta learning which is the focus of this work.\n+ the results suggest improvements over the Reptile baseline on miniImageNet.\n \n \nGeneral concerns:\n- abstract: “the existing meta-learning models have been evidenced to overfit on meta-training tasks when using deeper and wider convolutional neural networks.” several methods such as CAVIA and MetaOptNet (among many others) address this issue by limiting the capacity of the learner.\n- what is the formal definition of meta-generalization and meta-overfitting? This definition will be helpful for understanding the paper’s arguments, for instance, why “reducing meta-overfitting” and “improving meta-generalization” are two different matters (as suggested in the last part of the abstract).\n- Figure 1.b: the green bars (proposed method) don’t seem to improve the generalization (testing accuracy). I can only speculate that the figure is to demonstrate that (in the rightmost plot) the difference between train and test accuracy for green bars is less than it is for the red bars. However, one should note that it seems to be due to the training accuracy being lower, which can be achieved, in the extreme case, by a random classifier (zero gap). So, it’s not necessarily useful when testing accuracy is not improved.\n- eq (1) and eq (2): the left-most loss L should be on D^{query}\n- page 4: “In view of the “lottery ticket hypothesis” (Frankle & Carbin, 2018), the model in equation 2 can be interpreted as a first-order meta-learner for estimating a subnetwork“ -> eq (2) in the current form still requires a 2nd-order derivative and I cannot see how (Frankle & Carbin, 2018) helps make it first-order optimization.\n- the usefulness of the provided theory is in question since the final method does not enforce the sparsity in practice. That is, the output of the final meta-parameters are *dense*.\n- page 7 mentions “top k_l entries”: what does “top” mean here? k_l dimensions of \\theta with the highest absolute value, maybe?\n- why do the two separate steps of only training the pruned subnetwork using Reptile and then retrain the whole network (with the updated subnetwork) again using Reptile? One can instead only train Reptile on the whole \\theta^{(i)}_M (with L0 norm constrained using the M projection) in a single Reptile step. Doing the former should be justified over the latter since its twice as expensive. Also it should be shown empirically (as an ablation study) that the former works better. \n- reference list: arXiv versions of many papers are cited, it’s good to cite the peer-reviewed version of papers when applicable. For instance, IHT pruning method does not seem to be peer-reviewed which is a caveat for the current work.\n- 10 pages seem excessive for the content of the paper. For instance, the experiments section can be shortened extensively and the theories can be put into the appendix.\n \n \nExperimental concerns:\n- Has the standard miniImageNet split been used? This split is especially important to be respected since CAVIA’s accuracy is taken from the original paper. \n- The reported number for CAVIA in table 2 is not the best number the original paper achieve. The best number is 51.82 ± 0.65% for one-shot, and 65.85 ± 0.55% for 5-shot. \n- There are quite a few new hyperparameters (3 iteration numbers for pretraining, pruning/finetuning, and retraining, and then the sparsity level k_l). It’s important to mention how the hyper-parameters are chosen, especially since they are different for different setups. \n- there seems to be no meta validation set for Omniglot.\n- For a fair comparison, the network size of the baseline as well as the learning rate should be searched with the same budget as the hyperparameter search done for the proposed method.  \n- ResNet experiment is especially concerning since 1) there is an even higher-level of hyperparameter tuning for ResNet: first conv layer is not pruned, different pruning rates for different residual blocks are used. 2) the baseline is only tried with one setting for the capacity. It is evident in Table 2 that there is a sweet spot for the capacity of the ConvNet baseline, there is no reason this does not apply to ResNet.\n- Ablation study:  “For the purpose of fair comparison, only the retraining phase is removed and other settings are the same as proposed in Section 5.1”: this is not fair. I believe a fair comparison would be to repeat the hyperparameter search for the ablation studies with the same budget as the full version of the proposed method.\n- Table 2 only compares with CAVIA while many other meta learning methods could be relevant here such as MetOptNet which also limits the number of base learner’s learnable parameters thereby helping with overfitting. Also, CAVIA has results on ResNet useful for the last experiment.\n \n \n \n============= Final Decision =============\n\nWhile the paper addresses an important problem and reports improvements, there are many concerns with it including the writing, method, and experimental setup.  My “weak reject” rating, however, is mainly based on the experimental concerns especially regarding the hyperparameters and the baselines.\n \n \n============= Minor Points =============\n\n- code is not provided. I think it’s in general very helpful to release the codes for reproducibility and replicability of the experiments, especially in the case of an incremental study.\n- what meta learning task has been used for Figure 1? \n- There are different ways of including batchnorm in a meta learning setup. For instance, Reptile used two different strategies. How is batchnorm implemented for this paper’s experiments? Particularly, indicate if you are using transductive or non-transductive version of batchnorm at test time.\n \nThe text as well as the math notations require polishing on various occasions including the following (non-exhaustive) list:\n- abstract: “sparsity constrained” -> sparsity-constrained\n- abstract: “can not only” -> rephrase so that it’s not read as “cannot only”\n- abstract and intro: “ease meta-overfitting” can be interpreted as facilitating overfitting -> maybe better to say “alleviate meta-overfitting”\n- intro: “memorize the experience from previous meta-tasks for future meta-task learning with very few samples” → memorize the experience from previous tasks for a future task with very few samples\n- intro: “As can be observed from Figure 1(a) that” -> It can be observed from Figure 1(a) that\n- intro: “sparsity benefits considerably to the” → sparsity benefits considerably the\n- intro: “along with two specifications to two widely used networking pruning methods”: rephrase\n- related works: “most closely” -> closest\n- page 4: “ultra goal -> maybe ultimate goal?\n- page 4: \\theta is not defined\n- page 4: eq (2): i -> m or m -> i\n- page 4: the definition of an unordered set (S) coming as a sample of the product of m \\Tau spaces is not precise. Better to say each T_i \\in \\Tau.\n- page 4: J_l is not defined,\n- sec 3.3: the minimization are done over \\theta, it’s better to put that under “min” and the conditions should be clearly indicated by “with or s.t. or similar”\n- page 7: zero-one -> binary\n- page 7: “ as the restriction of \\theta^{(t)} over the mask M^{(t)}” -> rephrase\n- page 7: “it deems to reduce” -> rephrase\n- page 8: “this is consist” -> consistent\n \n- it’s also good to settle on a single term for each concept to make it an easier read, for instance:\n- “meta-training tasks” and “training tasks” have been used interchangeably. I think “training tasks” and “mata training dataset” are better choices since a “meta-training” task can refer to the whole task of meta learning.\n- Meta-overfitting and meta-level overfitting\n- meta-learner, meta-estimator, model\n- meta-level training accuracy, meta-training accuracy\n- meta training, meta learning\n- base learner, learner, model → I think it’s better to emphasize on learner being “base learner”, avoid model as it is not clear what it refers to.\n \n \n \n============= Points of improvements =============\nThe paper would significantly improve if 1) the text is polished and the equations revised. 2) hyperparameter optimization is done carefully and thoroughly described for both the proposed method as well as the baselines\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxXjkpqKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the authors propose a new method to alleviate the effect of overfitting in the meta-learning scenario. The method is based on network pruning. Empirical results demonstrate the effectiveness of the proposed method.\n\nPros:\n+ The problem is very important in the meta-learning field. The model is intuitive and seems useful. In addition, the generalization bound further gives enough insights for the model.\n\nCons:\n- The proposed method is simple and lacks technical contributions. Adding sparse regularizers is a common practice to alleviate over-fitting in the machine learning field. In addition, the retraining process increases the time complexity of the proposed model (i.e., we need to train three times to get the powerful model). \n\n- In the experiment parts, it will be more interesting if the authors can do the experiments without pre-training. Since in traditional meta-learning settings (e.g., Reptile and MAML), pre-training process does not be introduced. Thus, it might be more convincing if the authors can training the mask and initialization together.\n\n\nPost rebuttal:\n\nI have read other reviewers and the authors' responses. I still think the contribution is not enough to be accepted. I do not change my mind.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgZbj0g5r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use the sparse network to mitigate the task-overfitting. The algorithm is based on an iterative training over the pruning phase and retraining phase. A theoretical analysis of the sparsity effects for the Reptile-based first-order meta-learning is provided which indicates if the sparsity level k satisfies k^2 << parameter dimension p, the sparsity can help with the generalization. The paper studies the Reptile + DSD pruning and Reptile + IHT pruning on two standard few-shot classification benchmark.\n\nThe paper is in general well-written. My major comments/questions are the following;\n\n1. As the author pointed in the experiment section, there is a mis-match between the experiments and the theory. The output of Algorithm 1 is not sparse though during the iterative training the initialization of the retraining phase is sparse. But the theorem says the generalization is achieved because of sparsity, which requires k^2 << p. The ablation study seems even put more doubts on what the theorem suggests, which basically shows sparsity alone harms the general performance.\n\n2. Is there a typo in Eq (1)? Should the outside loss be evaluated on the query set rather than the support set? If so, does this typo influences the prove of Theorem 1 based on Shalev-Shwartz et.al. 2009?\n\n3. In Figure 1, first of all, what experiments does this figure corresponds to? In Figure 1 (b), the gap between training and testing for both pruning methods are quite large which seems doesn’t solve the overfitting very much? The test traces are intertwined, so it is not clear that the test accuracy get really improved. Using a consistent color for the Figure 1 (a) and Figure 1 (b) can make it much easier to read.\n\n4. The paper needs to discuss about the computational-complexity. It seems each iteration in Algorithm 1 involves meta-training a sparse network and a dense network. And the algorithm needs the number of iteration t > 1. Is there any difficulty in scaling?\n\n5. In the experiments section, for Omniglot almost all results are overlapping with confidence interval. Maybe it should not mark some numbers with bold font. The results in Mini-imagenet show the improvement by proposed methods. Does the effectiveness related to the total number of image classes in dataset?\n\nSome other comments:\n\n1. As the author mentioned, there are two types of overfitting: the meta-level overfitting and task-level overfitting. Why the proposed methods deal with meta-level overfitting rather than task-level overfitting?\n\n2. How does the random mask generated in Algorithm 1?\n\n3. In experiments, can the training accuracy be also provided? \n\nIn general, this paper study an interesting problem in meta-learning and the paper is written in a clear way. The major problems are a mis-match between the theorem and the methods and the experimental results are not very strong. I will give a borderline rating.\n\n############\n\nThanks for the authors' feedback and I have read them. I am still not convinced about the proposed method about the effectiveness and the trade-off in computation. I hold the concerns about the rebuttal \"this typo can be easily fixed with almost no impact on the technical proofs of the theoretical results\".  Therefore I maintain the current rating as borderline leaning to rejection.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a regularization scheme for reducing meta-overfitting. After the rebuttal period, the reviewers all still had concerns about the significance of the paper's contributions and the thoroughness of the empirical study. As such, this paper isn't ready for publication at ICLR. See the reviewer's comments for detailed feedback on how to improve the paper. 
Decision: Reject