Title: Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
Authors: Hesham Mostafa
Xin Wang
Venue: ICLR 2019
Keywords: sparse
reparameterization
overparameterization
convolutional neural network
training
compression
pruning

Abstract: Modern deep neural networks are highly overparameterized, and often of huge sizes.  A number of post-training model compression techniques, such as distillation, pruning and quantization, can reduce the size of network parameters by a substantial fraction with little loss in performance.  However, training a small network of the post-compression size de novo typically fails to reach the same level of accuracy achieved by compression of a large network, leading to a widely-held belief that gross overparameterization is essential to effective learning.  In this work, we argue that this is not necessarily true.   We describe a dynamic sparse reparameterization technique that closed the performance gap between a model compressed through iterative pruning and a model of the post-compression size trained de novo.  We applied our method to training deep residual networks and showed that it outperformed existing reparameterization techniques, yielding the best accuracy for a given parameter budget for training.  Compared to existing dynamic reparameterization methods that reallocate non-zero parameters during training, our approach achieved better performance at lower computational cost.  Our method is not only of practical value for training under stringent memory constraints, but also potentially informative to theoretical understanding of generalization properties of overparameterized deep neural networks.  



TL;DR: We describe a dynamic sparse reparameterization technique that allow training of a small sparse network to generalize on par with, or better than, a full-sized dense model compressed to the same size.  

Review discussion:
The paper provides a dynamic sparse reparameterization method allowing small networks to be trained at a comparable accuracy as pruned network with (initially) large parameter spaces. Improper initialization along with a fewer number of parameters requires a large parameter model, to begin with (Frankle and Carbin, 2018).  The proposed method which is basically a global pooling followed by a tensorwise growth allocates free parameters using an efficient weight re-distribution scheme, uses an approximate thresholding method and provides automatic parameter re-allocation to achieve its goals efficiently. The authors empirically demonstrate their results on MNIST, CIFAR-10, and Imagenet and show that dynamic sparse provides higher accuracy than compressed sparse (and other) networks.
The paper is addressing an important problem where instead of training and pruning, directly training smaller networks is considered. In that respect, the paper does provide some useful tricks to reparameterize and pick the top filters to prune. I especially enjoyed reading the discussion section.
However, the hyperbole in claims such as "first dynamic reparameterization method for training convolutional network" makes it hard to judge the paper favorably given previous methods that have already proposed dynamic reparameterization and explored. This language is consistent throughout the paper and the paper needs a revision that positions this paper appropriately before it is accepted.
The proposed technique provides limited but useful contributions over existing work as in SET and DeepR.
However, an empirical comparison against them in your evaluation section can make the paper stronger especially if you claim your methods are superior.
How does your training times compare with the other methods? Re-training times are a big drawback of pruning methods and showing those numbers will be useful.
Thank you for the review.   We have substantially revised the manuscript with the following major changes:
1. Contributions in Introduction (also the rest of the manuscript) completely rewritten to state novelty accurately
2. Inclusion of results of additional performance benchmark against existing methods, DeepR and SET
3. Inclusion of results of computational cost benchmarked against existing methods, DeepR and SET
4. Revised Experimental Results section and two additional appendices that further expanded the scope of comparison to structured compression methods
We hope the improved manuscript is worthy of publication now.
Our response to your specific comments:
1. We have rewritten the entire manuscript to state our novelty accurately.  We now make only two claims in contributions, which are carefully limited to the exact scope of this investigation.  We specifically eliminated claims of "first".
2. We have strengthened the manuscript by including results of full comparisons to SET and DeepR (the new Figure 2, Table 2) to support our main claim that ours outperformed these methods.
3. We have included a new Table 3, containing the computational overhead of our parameter reallocation in comparison to those of DeepR and SET.
Indeed, re-training times are a drawback of pruning methods. We now state clearly that our method actually runs for a fewer number of epochs than pruning-based methods--our method only trained for the same number of epochs as the original training of the large dense model in pruning methods minus the additional retraining (see the revised Appendix A).
* Note: Due to the high computational requirements of DeepR, the results for DeepR on resnet50 were not available in time for the revision submission. We include the results (top-1, top-5 accuracy) below. The accuracy of DeepR lags behind our method and behind SET.
| Method                                    |     Sparsity = 0.9     |     Sparsity = 0.8     |
| Mocanu et al. 2018 (SET)      |         70.4, 90.1        |         72.6, 91.2        |
| Bellec et al. 2017 (DeepR)      |         70.2, 90.0        |         71.7, 90.6        |
| Ours                                         |         71.6, 90.5        |         73.3, 92.4        |
Thanks for the comprehensive revision as well as providing new experiments and comparisons. Your revision adequately addresses the concerns I raised in my original review.
This paper presents a method for training neural networks where an efficient sparse/compressed representation is enforced throughout the training process, as opposed to starting with are large model and pruning down to a smaller size.  For this purpose a dynamic sparse reparameterization heuristic is proposed and validated using data from MNIST, CIFAR-10, and ImageNet.
My concerns with this work in its present form are two-fold.
First, from a novelty standpoint, the proposed pipeline can largely be viewed as introducing a couple heuristic modifications to the SET procedure from reference (Mocanu, et al., 2018), e.g., substituting an approximate threshold instead of sorting for removing weights, changing how new weights are redistributed, etc.  The considerable similarity was pointed out by anonymous commenters and I believe somewhat understated by the submission.
Regardless, even if practically effective, these changes seem more like reasonable engineering decisions to improve the speed/performance rather than research contributions that provide any real insights.
Moreover, there is no attendant analysis regarding convergence and/or stability of what is otherwise a sequence of iterates untethered to a specific energy function being minimized.
Of course all of this could potentially be overcome with a compelling series of experiments demonstrating the unequivocal utility of the proposed modifications.
But it is here that unfortunately the paper falls well short.
Despite its close kinship with SET, there are surprisingly no comparisons presented whatsoever.
Likewise only a single footnote mentions comparative results with DeepR (Bellec et al., 2017), which represents another related dynamic reparameterization method.  In a follow up response to anonymous public comments, some new tests using CIFAR-10 data are presented, but to me, proper evaluation requires full experimental details/settings and another round of review.
Moreover, the improvement over SET in these new results, e.g., from a 93.42 to 93.68 accuracy rate at 0.9 sparsity level, seems quite modest.  Note that the proposed pipeline has a wide range of tuning hyperparameters (occupying a nearly page-sized Table 3 in the Appendix), and depending on these settings relative to SET, one could easily envision this sort of minor difference evaporating completely.
But again, this is why I strongly believe that another round of review with detailed comparisons to SET and DeepR is needed.
Beyond this, the paper repeatedly mentions significant improvement over "start-of-the-art sparse compression methods." But this claim is completely unsupported, because all the tables and figures only report results from a single existing compression baseline, namely, the pruning method from (Zhu and Gupta, 2017) which is ultimately based on (Han et al., 2015).
But just in the last year alone there have been countless compression papers published in the top ML and CV conferences, and it is by no means established that the pruning heuristic from (Zhu and Gupta, 2017) is state-of-the-art.
Note also that reported results can be quite deceiving on the surface, because unless the network structure, data augmentation, and other experimental design details are exactly the same, specific numbers cannot be directly transferred across papers.
Additionally, numerous published results involve pruning at the activation level rather than specific weights.  This definitively sacrifices the overall compression rate/model size to achieve structured pruning that is more naturally advantageous to implementation in practical hardware (e.g., reducing FLOPs, run-time memory, etc.).  One quick example is Luo et al., "ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression," ICCV 2017, but there are many many others.
And as a final critique of the empirical section, why not report the full computational cost of training the proposed model relative to others?  For an engineered algorithmic proposal emphasizing training efficiency, this seems like an essential component.
In aggregate then, my feeling is that while the proposed pipeline may eventually prove to be practically useful, presently this paper does not contain a sufficient aggregation of novel research contribution and empirical validation.
Other comments:
- In Table 2, what is the baseline accuracy with no pruning?
- Can this method be easily extended to prune entire filters/activations?
4. On benchmark against post-training compression baselines:  Thank you for raising this important point.  We agree that the line of work from Han et al. 2015 to Zhu & Gupta 2017 is only a subset of all compression methods.
Even though Zhu & Gupta 2017 is the strongest sparse compression baseline known to us, we now state clearly that we close the performance gap to the iterative pruning method of Zhu and Gupta 2017, instead of saying "compression methods" in general.
5. On structured compression method such as ThiNet:  In the previous version of the manuscript, we did not benchmark against structured compression method such as ThiNet because they (a) produce dense instead of sparse models, and (b) significantly underperformed non-structured compression, such as Zhu & Gupta 2017, despite their efficiency on GPUs.  In the revision, we made the following changes to address this issue: (a) we included comparisons against two representative structured pruning methods in Table 2; (b) we included a new Appendix C to compare and contrast a wide range of methods, painting a broad picture of relevant existing methods to show where our method stands; (c) we did additional experiments to impose group structure on sparsity using our method, and show degraded results (the new Appendix D); (d) we specifically discussed the issue of structured versus non-structured sparsification, and its implications for optimal computing hardware architecture (last two paragraphs of the Discussion section).
6. On the difficulty of comparing results across papers:  In the revision we included our own experiments of DeepR and SET, carefully controlled for comparison to ours.  For comparison with ThiNet (Luo et al. 2017) and SSS (Huang & Wang 2017), we adapted the results from the original papers (See the new Table 2).  To ameliorate the potential minor differences in experimental protocols, we also report the relative difference from the full dense model performance reported in that same paper (square brackets in the new Table 2)--comparison of methods can now be based on how much accuracy degradation from a controlled baseline a method introduces, rather than on absolute accuracy figures.
7. On computational cost:  We now include quantifications of computational overhead of our method, DeepR and SET, in the last paragraph of the Experimental Results section and in Table 3.
8. On other comments:  (a) We included the full dense baseline in the new Table 2 (rightmost column).  (b) We included a new Appendix D to present extra experiments where we applied our methods to group pruning of 3x3 kernels.  We show that this led to a significant but minor degradation in performance.  We also discussed the pros and cons of structured vs. non-structured sparsification in Discussion and Appendix C as stated above.
* Note: The current PDF of the manuscript has blanked DeepR entry in Table 3.  Due to the high computational requirements of this experiment, it is still running.  We will fill in the numbers as soon as they are available.
Thank you for the review.   We have substantially revised the manuscript with the following major changes:
1. Contributions in Introduction (also the rest of the manuscript) completely rewritten to state novelty accurately
2. Inclusion of results of additional performance benchmark against existing methods, DeepR and SET
3. Inclusion of results of computational cost benchmarked against existing methods, DeepR and SET
4. Revised Experimental Results section and two additional appendices that further expanded the scope of comparison to structured compression methods
We hope the improved manuscript is worthy of publication now.
Our response to your specific comments:
1. On benchmarking against existing methods:  We fully agree with you on the weakness of the manuscript due to lack of quantitative comparisons to prior work.  We believe this revision adequately rectified it.
Specifically, we performed additional WRN-28-2 on CIFAR10 and Resnet-50 on Imagenet experiments for DeepR and SET.  The results are presented in the new Figure 2, Table 2 and Table 3.  We now show with concrete evidence that our method outperformed both DeepR and SET.
2. On hyperparameter tuning:  For hyperparameters of training the models, we did not tune them and simply used the same hyperparameter settings in the original papers where these models were described.  For hyperparameters of reparameterization by DeepR and SET, because the original papers did not attempt experiments at the same scale, we did a hyperparameter sweep for DeepR and reported the best result; for SET, in order to make a fair comparison, we used the exact same hyperparameters for both SET and ours.  These facts were not clearly stated in the previous version, but now clearly stated together with the list of hyperparameters (Table 4) in the revision.
3. On our method's similarity to SET:  Our method is inspired by SET, based on the similar sparse reparameterization mechanism, but with an adaptive threshold and a heuristic for automatic parameter reallocation across layers. These differences might seem incremental, but we believe our work made a substantial stride forward from what was reported in Mocanu et al. 2018, for the following reasons (as now stated in the revised manuscript). (a) Our method did produce significantly better generalizing sparse models than SET, and fully closed the performance gap toward compression by iterative pruning, of which SET in some cases still fell short (we now provide quantitative results).
(b) Automatic parameter reallocation without manual configuration of sparsity per layer makes sparse training much more scalable: the burden of hyperparameter tuning is constant instead of scales with network depth. (c) Finally, Mocanu et al. 2018 demonstrated their method on multi-layer perceptrons on MNIST. We believe scaling up to deep convolutional networks such as Resnet-50 and to large datasets such as Imagenet is not just a trivial increment.
Although the experimental section has been expanded a bit and the overall paper has been upgraded, I still believe that the novelty of this work is limited.  As mentioned in my original review, the proposed pipeline represents reasonably engineered modifications of the existing SET pipeline, but there are no significant insights beyond this.
Moreover, the added heuristics often seem to be marginally important.
For example, if we look at Figure 2(a) in the revision, the proposed algorithm has almost the same performance as SET. For these and other reasons below, I continue to believe then that this work is perhaps below the bar for ICLR, while acknowledging the effort it takes to engineer and test this type of model.
Other lingering concerns with this work:
- With regard the hyperparameter tuning, the rebuttal comments state that "we did not tune them and simply used the same hyperparameter settings in the original papers where these models were described."  This seems to reinforce the notion that this is not a fundamentally different model, but one intimately related to existing papers.
- The revised version claims in the introduction that the proposed method is more efficient than existing dynamic sparse reparameterization training techniques.
But this seems like an overstatement, because if we look at Table 3, the proposed method requires almost exactly the same computational complexity as the existing SET method (the difference is only in the forth significant digit).
- The rebuttal states that "(Zhu & Gupta, 2017) is the strongest sparse compression baseline known to us."  But as I mentioned previously, there have been numerous DNN compression methods introduced at multiple major ML and CV conference over the last year, and it is essential to check these proceedings to become aware of the latest developments.  This is especially true for an empirical paper of this type without any analytical contribution.
Also, the revision continues to imply without justification that (Zhu & Gupta, 2017) represents the state-of-the-art (see bullet point 4 on page 5).
- According to the rebuttal, some DeepR experiments are still running and therefore could not be included in the revision.
But this is not an excusable omission, because as an obvious, direct competitor to the proposed method, these results should have been present in the original submission.
- As mentioned in my original review, the proposed algorithmic steps are not minimizing any particular energy function per se, and yet there is no discussion of convergence or stability.  This was not addressed in the rebuttal.
- With regard to "numerous DNN compression methods introduced at multiple major ML and CV conference over the last year", we did a comprehensive survey of compression papers, which are listed in Appendix C, together with their properties. We still see that the best compression performance is achieved by Zhu and Gupta, 2017. With adequate due diligence we honestly claim that `tensorflow. contrib. model_pruning` (Zhu and Gupta, 2017) is a "state-of-the-art" sparse compression method "known to us" at the time of this paper being written. We would appreciate it if you could provide us with a specific stronger baseline, demonstrably stronger than Zhu & Gupta 2017, and we are happy to benchmark our method against it and call it "state-of-the-art".
Absent such a method in the literature that you could refer to us, the claim that we did not compare ours to the best sparse compression method in existence is a rather weak one, and unfair. Note that most recent pruning techniques deals with structured or filter-wise sparsity that does not directly compare with our method and significantly underperformed non-structured pruning (see Appendix C, D, see Liu et al. 2018).
- We believe there is a misunderstanding regarding your response on hyperparameter tuning in writing "This seems to reinforce the notion that this is not a fundamentally different model, but one intimately related to existing papers. "  We referred to "hyperparameters for training", such as learning rate, momentum, L1/L2 decay, etc. , these were taken the same as those presented in the original paper (e. g. He et al. 2015) where the model (e. g. Resnet-50) was presented. These do not include hyperparameters for sparse reparameterization (which is the subject of this paper and we did tune those).
We do not believe this is related to the novelty of our method at all--our paper does not produce "fundamentally different model"s, but trains sparse versions of existing, successful models, such as Resnet. If anything, using the exact hyperparameters for training as in the original papers only strengthens our method--our method is robust enough that we do not need to re-tune hyperparamters for training it and it just worked well with the original network's hyperparameters.
- With regard to DeepR, we did demonstrate a significant accuracy/performance advantage over DeepR for the case of WRN-28-2 on CIFAR10.  Experiment on Imagenet is much more challenging due to the high computational cost of DeepR, but we agree that it should be (and will be) part of the paper (note that the DeepR paper never attempted experiments at this scale).  With concrete evidence, our results on CIFAR10, together with the significantly faster training times of our method are strong indicators that our method is superior to DeepR in both accuracy and speed.  We agree with you that Table 3 shows ours and SET both introduce negligible computational overhead, but our method has other advantages over SET (e.g. producing better accuracy, able to train at high sparsity levels), besides the slight reduction in computational cost.
Thank you for your comments.  We believe the argument on the lack of novelty lacks factual support.  Our point-by-point response:
- "Moreover, the added heuristics often seem to be marginally important... Figure 2(a) ...almost the same performance as SET" This argument disregards key evidence for the opposite.
First, our algorithm did lead to significantly better accuracy than SET on all the benchmarks we tested, no matter how small the differences are in certain cases.  The differences are in fact more substantial in other cases.
For instance, the improvement on Imagenet is more prominent (Table 2).
Second, the improvement is especially more apparent at high sparsity levels where SET failed catastrophically while our method does not (see Figure 6 in appendix E).
Finally, our automatic parameter allocation heuristic that discovered the number of parameters to allocate to each layer is entirely novel.
- Even if our method lacked any novelty or performance improvement as compared to SET (which it does not as our revised manuscript shows with quantitative evidence), it would still be a significant stride forward from Mocanu et al. 2018 just to apply their exact same method to show its usefulness in training deep sparse convolution nets such as Resnet-50 in action on large datasets like Imagenet.
In other words, Mocanu et al. 2018 did not even show if SET is at all applicable to convolution layers, let alone deep Resnets.
- We agree with you that a theoretical guarantee of convergence and stability is not presented, and the heuristic could not be cast into an optimization procedure of an objective function.
But we do not think this is an adequate reason to dismiss an empirical paper.  Convergence guarantees are hard to come by in deep networks where there are no convergence guarantees of even basic stochastic gradient descent in most real-world cases.  We practiced empirical rigor in the paper: we exhaustively described our method and experiments in detail, empirically validated our method's performance and stability, and publicized all source code for all experiments, a standard of which even the papers you cited in your review (e.g. Mocanu et al. 2018) fell shy (note that there is no theoretical guarantee of SET's stability in that paper either).  Rigorous empirical findings is beneficial to the field because the dissemination of knowledge on "what works" would eventually lead to theory on "why it works".
Due to the high computational requirements of DeepR, the results for DeepR on resnet50 were not available in time for the revision submission. We include the results (top-1, top-5 accuracy) below. The accuracy of DeepR lags behind our method and behind SET.
| Method                                    |     Sparsity = 0.9     |     Sparsity = 0.8     |
| Mocanu et al. 2018 (SET)      |         70.4, 90.1        |         72.6, 91.2        |
| Bellec et al. 2017 (DeepR)      |         70.2, 90.0        |         71.7, 90.6        |
| Ours                                         |         71.6, 90.5        |         73.3, 92.4        |
Thank you again for your useful comments, which were common concerns raised by all 3 reviewers, including (1) lack of comparison to prior work, and (2) inaccurate claims of contributions.
In the revision submitted together with a point-by-point rebuttal to your review on Nov. 23, we have fully addressed these concerns, and Reviewer #3 found our revision satisfactory.
As it has been weeks since our previous response, we wonder whether you would share the same view on the revised manuscript's suitability for publication, or if there were still lingering concerns, we would appreciate it if you could reply to our most recent point-by-point rebuttal with specifics so that we could address them.
Thank you very much!
Weaknesses:
1-The authors claim that: "Compared to other dynamic reparameterization methods that reallocate non-zero parameters during training, our approach broke free from a few key limitations and achieved much better performance at lower computational cost." => However, there is no quantitative experiments related to other dynamic reparameterization methods. There should be at least sparsity-accuracy comparison to claim achieving better performance. I expect authors compare their work at-least with with DEEP R, and NeST even if it is clear for them that they produce better results.
2-The second and fourth contributions are inconsistent: In the second one, authors claimed that they are the first who designed the dynamic reparameterization method. In the fourth contribution, they claimed they outperformed existing dynamic sparse reparameterization.
Moreover, it seems DEEP R also is a  dynamic reparameterization method because DEEP R authors claimed: "DEEP R automatically rewires the network during supervised training so that connections are there where they are most needed for the task, while its total number is all the time strictly bounded."
3- The authors claimed their proposed method has much lower computational costs, however, there is no running time or scalability comparison.
Suggestions:
1-Authors need to motivate the applications of their work.
For instance, are they able to run their proposed method on mobile devices?
2-For Figure 2 (c,d) you need to specify what each color is.
3-In general, if you claim that your method is more accurate or more scalable you need to provide quantitative experiments. Claiming is not enough.
4-It is better to define all parameters definition before you jump into the proposed section.
Otherwise, it makes paper hard to follow.
For instance, you didn't define the R_l directly (It is just in the Algorithm 1).
Thank you for the review.  We have substantially revised the manuscript with the following major changes:
1. Contributions in Introduction (also the rest of the manuscript) completely rewritten to state novelty accurately
2. Inclusion of results of additional performance benchmark against existing methods, DeepR and SET
3. Inclusion of results of computational cost benchmarked against existing methods, DeepR and SET
4. Revised Experimental Results section and two additional appendices that further expanded the scope of comparison to structured compression methods
We hope the improved manuscript is worthy of publication now.
Our response to your comments on weaknesses:
1. The revised manuscript now includes direct quantitative comparisons to all direct sparse training techniques with a strict parameter budget (i.e. DeepR and SET), for the deep residual net experiments (on CIFAR10 and Imagenet, see Figure 2 and Table 2).  We did not include NeST because NeST does not impose a strict parameter budget during training--it grows a small network to a large one and then prunes it down.  Our claim here is that our method yielded the best accuracy given a strictly fixed parameter budget throughout training so only DeepR and SET are relevant to this claim.  We further explain in full detail the relationships between our method and numerous others in a new Appendix C.
2. We apologize for the confusion.  The claim was indeed worded incorrectly.  The correct claim is that we are the first to apply sparse dynamic reparameterization to training of large CNNs (such as Resnets) on large datasets, because previous methods of the same kind were demonstrated only on small networks.  We have completely rewritten the contributions with this claim removed.
3. Per your suggestion, we added a last paragraph to the Experimental Results section and included a new Table 3 with numbers to support our claim on efficiency.  Our scalability argument is supported by the fact that our method discovers layer-wise sparsity automatically during training without the need to predefine sparsity per layer by manual configuration as required by other methods, so that the cost of hyperparameter tuning is constant instead of scaling with network depth.
Our response to your suggestions:
1. Per your suggestion, we revised the Introduction section and included the follwing sentence: "... a dynamic sparse reparameterization technique able to train sparse models de novo without the need to compress a large model, a desirable feature for training on memory- and power-constrained devices."  Furthermore, a related, more nuanced point on hardware-efficiency was discussed in the last two paragraphs of the Discussion section.
2. We rectified the unnecessary use of color and made these panels grayscale with specific text labels.
3. We have included new results (see the new Figure 2, Table 2 and Table 3) and revised the text to provide concrete support of the claims.
4. We now defined these parameters in the text (in the revised Methods section) in addition to in Algorithm 1.
* Note: Due to the high computational requirements of DeepR, the results for DeepR on resnet50 were not available in time for the revision submission. We include the results (top-1, top-5 accuracy) below. The accuracy of DeepR lags behind our method and behind SET.
| Method                                    |     Sparsity = 0.9     |     Sparsity = 0.8     |
| Mocanu et al. 2018 (SET)      |         70.4, 90.1        |         72.6, 91.2        |
| Bellec et al. 2017 (DeepR)      |         70.2, 90.0        |         71.7, 90.6        |
| Ours                                         |         71.6, 90.5        |         73.3, 92.4        |
Thank you again for your useful comments, which were common concerns raised by all 3 reviewers, including (1) lack of comparison to prior work, and (2) inaccurate claims of contributions.
In the revision submitted together with a point-by-point rebuttal to your review on Nov. 23, we have fully addressed these concerns, and Reviewer #3 found our revision satisfactory.
As it has been weeks since our previous response, we wonder whether you would share the same view on the revised manuscript's suitability for publication, or if there were still lingering concerns, we would appreciate it if you could reply to our most recent point-by-point rebuttal with specifics so that we could address them.
Thank you very much!

Meta-review:
role: AC
data: metareview: 
The authors presents a technique for training neural networks, through dynamic sparse reparameterization. The work builds on previous work notably SET (Mocanu et al., 18), but the authors propose to use an adaptive threshold for and a heuristic for determining how to reparameterize weights across layers. 
The reviewers raised a number of concerns on the original manuscript, most notably 1) that the work lacked comparisons against existing dynamic reparameterization schemes, 2) an analysis of the computational complexity of the proposed method relative to other works, and that 3) the work is an incremental improvement over SET.
In the revised version, the authors revised the paper to address the various concerns raised by the reviewers. To address weakness 1) the authors ran experiments comparing the proposed approach to SET and DeepR, and demonstrated that the proposed method performs at least as well, or is better than either approach. While the new draft is in the ACs view a significant improvement over the initial version, the reviewers still had concerns about the fact that the work appears to be incremental relative to SET, and that the differences in performance between the two models were not very large (although the author’s note that the differences are statistically significant). The reviewers were not entirely unanimous in their decision, which meant that the scores that this work received placed it at the borderline for acceptance. As such, the AC ultimately decide to recommend rejection, though the authors are encouraged to resubmit the revised version of the paper to a future venue.


Decision: Reject