Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Proposes a reparameterization of neural network models, where each weight is raised to the power of alpha, where alpha > 1, but keeps its original sign.
The proposed reparameterization and its implications on the optimization process are briefly discussed mathematically, with the general intuition being that "the rich get richer", i.e., low-magnitude weights tend to stay close to zero, while high-magnitude ones tend to become more significant in the resulting model.

Thus, the reparameterization implicitly acts as a regularizer that prefers models with few high-magnitude weights. The resulting model is more suitable for post-pruning schemes that create sparse models.

Finally, the reparameterization is empirically shown to enhance a Continual Learning scheme that uses iterative pruning (PackNet [Arun and Lazebnik 2017]), improving results on various continual learning baselines.
main_review: * Generally, the general idea is interesting and extends previous ideas in the field of pruning and sparse models.   
* The paper is mostly well written and ideas are presented clearly (the plots are great).  
* Empirically, there are decent results in various settings (pruning for sparse models; and implications in an iterative pruning model for continual learning).
* I have some reservations, mostly regarding some unclear paragraphs or sentences. 
* I am ambivalent about the rating, **unsure between 6 and 7**. I believe the author response can help in this regard.

---

### A. Originality and significance


1. **Originality.** I believe the proposed reparameterization is original *enough*.   
Similar reparameterizations were previously studied in the sparse context, e.g. the authors here adequately cite [Vaskevicius et al. 2019] that reparameterize a linear regression model as $u\circ u- v\circ v$, which at a certain initialization scale limit converges to a minimum-L1 solution (inducing sparsity, see also [Woodworth et al. 2020]).  
However, the reparameterization in the paper reviewed here is more *general and flexible* (the exponent can be any $\alpha>1$ and moreover preserves the signs of the exponentiated weights, thus there is no need to split the parameters into $u$ and $v$).

2. **Significance.** The results are interesting to me and seem important. The reparameterization is rather elegant, can evidently yields sparser models. I believe it is immediately (and easily) applicable to general sparse schemes and more specifically pruning schemes. The authors show (Table 1) how it improves several pruning schemes and advances SOTA algorithms on sparse ResNet-50 for ImageNet.  
The observations on the continual learning setting are interesting but limited, since the paper focuses on a setting where the task identities are known during inference, and it seems that some of the conclusions only apply to models like PackNet that are based on iterative *pruning*, and do not apply to the wide continual learning scope.

---

### B. Quality and clarity
Most of the paper is clearly written and overall the paper has a good flow. I also believe the experimental results are strong enough.  
However, some of the theoretical analysis/intuitions are not really clear to me, and there are a few paragraphs that would benefit from rephrasing. Moreover, I was left with some questions and remarks I would like to specify.

1. I could not understand what exactly do the authors claim in lines 121-124. Please clarify.
2. I could not understand what is the difference between plain backpropagation and what the authors refer to in lines 136-145. This paragraph was extremely unclear to me. Could the authors please clarify? (I believe this paragraph needs to be improved).
3. The description of Algorithm 1 in the beginning of Section 4 is not self contained (some parts cannot be understood without reading the PackNet paper). For instance, it is not clear enough that this setting assumes the task identities are known during inference (thus it was unclear to me while reading why do you store *all* forward masks and not only the current one and the backward one). I also believe it should be explained that the space complexity requirement is valid for both train and inference. Moreover, I could not understand the exact origin of the sparsity range in $S$ in the algorithm. How is it determined?
4. Should the line 12 in Algorithm 1 be split into two lines? This is confusing.
5. Figure 2c: which dataset and model is presented here? CIFAR or ImageNet? Not stated.
6. In line 208: should it really say $s_n\cdot P$? If so, why $s_n$?
7. In line 212-213: the authors state that the algorithm "overcomes" the assumption of known task number. Is it actually true? Shouldn't one at least know the order of $T$ to set the sparsity ranges?
8. In Section 6.3 (continual learning experiments). Is the comparison to EWC (for instance) fair? 
Since EWC doesn't assume known task-identities during inference like EPN does.
9. Do any of the experiments use explicit regularization (weight decay)? Could not find the answer in the paper or Appendix.   
I guess the answer is no, but I want to make sure. This should also be explicitly written in the paper.
10. It can be interesting to understand or discuss the layer-wise effect of Powerpropagation. For instance, perhaps earlier layers are more dense while later layers are more sparse, etc.

---

### C. Minor remarks
Following are some minor remarks by line numbers. Please consider fixing them.
* 52: occupy => occupies
* 100 (and more): I don't believe the $\circ$ in the exponent was defined.
* Figure 1: should the caption say $\alpha=3.0$ instead of $4.0$?
* 188: Percent sign seems unnecessary
* Algorithm 1, line 12: Should the index in the superscript be $i$?
* 221, 225, 231: Spare => Sparse
* 223: Throughout what?
* 263: **D**atasets should be lowercase.
* 284: shows => show
* 289: constitues => constitute
* 300: significant => signficantly
* 303: Do you actually mean the *gradient* flattens? Or that the plot flattens?
* Generally: Instead of writing "details in the Appendix", consider writing the exact section of the appendix.

* Table 4: Isn't PackNet's forgetting better than EPN? A negative forgetting is often called "backward transfer" and is a good property of an algorithm.
limitations_and_societal_impact: I believe the authors addressed these issues adequately.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method to parameterize weights such that the resulting model is inherently sparse. This is achieved by dynamically updating the parameters proportional to their magnitude i.e., the higher magnitude weights receive larger updates while the low magnitude weights remain unaffected. The idea is novel and useful for training sparse networks geared towards applications in resource-constrained environments. Additionally, as elaborated in the paper, it is also useful in the cases of continual learning where a single model is used to learn multiple tasks. The claims are supported by experimental evaluation.
main_review: Strengths

- The idea of transforming the parameters with an exponent is a simple yet effective technique.
- The paper is well written with necessary mathematical derivations. Experiments adequately substantiate the proposed method and claims.
- The perspective of using sparsity in learning better models in continual learning is interesting and is very useful.

Weaknesses

- Section 3 last paragraph: The authors mention the concern that powerpropagation can lead to identical masks at initialization and at convergence which is intuitive as the small weights are not modified frequently. But it is also mentioned that the empirical results indicate that this is not the case. Can you please provide a possible justification?
- Can you please discuss how the exponentiation might impact the efficiency of the network in resource-constrained devices as exponentiation can be an expensive operation. This can increase the training complexity multiple folds.
- Section 6.3: Please mention the model architecture used for training the permuted MNIST dataset. The results in this section are impressive, however, it is limited to only one dataset in each of standard image classification and reinforcement learning settings. Moreover, permuted MNIST is synthetically generated. Hence, including another continual learning benchmark like CIFAR100 split into 10 or 20 tasks will add more value to the paper.

Overall a good paper. Minor improvements will help add more value to the paper.
limitations_and_societal_impact: Limitations are discussed. No adverse societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is focused on weight reparameterization, proposing a versatile method that leads to inherently sparse models.  The authors theoretically analyze this extension and backup the effectiveness of their proposed method with encouraging empirical results.
main_review: The basic idea of altering the gradient descent behavior by an approximated factor, exhibiting a "rich get richer" dynamic, is well-worthed and original.  Several experiments are conducted to demonstrate the performance of the proposed model. Promising results in terms of accuracy have been achieved in combining the proposed method -- Powerpropagation, with interactive pruning methods and sparse-to-sparse algorithms with a slight increase in FLOPs. Still, it is not clear what is the impact of alpha. Can you provide a more systematic analysis? 

I like the diversity and the choice of the type of experiments performed and the special attention accorded to the catastrophic forgetting phenomenon in continual learning. However, the bridge between sparse training networks and continual learning appears possible in previous works (e.g. [1]). Please can you comment on the expected impact of SpaceNet on Figure 4?  Perhaps this shall be reflected in some nuances added to the discussion (rows 342-344).

[1] G. Sokar et al. "SpaceNet: Make Free Space For Continual Learning", https://arxiv.org/abs/2007.07617, 2020
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This is another paper where the goal and idea of the paper are almost summarized well in the title and the abstract. The only missing part is the parameterization itself which is very simple yet effective. A weight, w, is reparameterized using another underlying variable, v, as w = v|v|^($\alpha$ - 1), essentially reparameterizing the weight (real number) using an exponentiation operation as the name suggests. 

This decoupling as the paper discusses in L106-115 leads to sparse solutions, due to the gradient scaling of the underlying variable resulting in the rich getting richer due to the exponentiation involved. The paper uses this observation, coupled with some toy understanding on MNIST gave the motivation to use this reparameterization to naturally induce sparser solutions.

The paper focuses on coupling this with sparsity-inducing methods and continual learning methods that use sparsity to encode multiple sub-networks and shows that combining this reparameterization with the SOTA techniques further improves accuracy with minimal overhead. 
main_review: Let me start with a general comment before going into more details.

The paper is very engaging to read but sometimes becomes dense because of the things and ideas moving around. I liked it very much once I understood what the authors were trying to do, but long bodies of text couple with notations at places made it harder at times. It is a simple idea and the explanation could be made simpler.

Especially Section 2, it very strong but at times tries to explain everything at a stretch creating a chance to miss things. It would be great to have a simple running example with a gradient update equation would have been great like. Instead of update being w - $\eta\Delta$w the update now becomes something like w = sign(v)(v - $\eta\Delta$v)^$(\alpha)$ = sign(v)(v - $\eta\Delta$w$\alpha$|v|^$(\alpha-1)$)^$(\alpha)$ would have made things clearer. Even now there are two cases when v > 1 and v < 1 due to the nature of exponentiation, making gradient scaling while monotonic, not trivial to understand. Most weights are < 1 so the dynamics of a weight with $\alpha=2$ and v = 0.1 vs v = 0.2 do differ when looked at from w's perspective. 

I strongly hope either author comment on this or fix this in the next revisions making it easier to follow. 

I think that is my only major concern in the whole paper. Now coming to the strengths. 

The authors identified extremely appropriate applications and showed statistically significant improvement when using powerpropagation instead of the normal weights. Both the sub-network-based CL and sparse network learning benefit from this simple yet powerful idea. 

The experiments are very thorough and the authors should be appreciated for their efforts. 
limitations_and_societal_impact: Limitations are decently discussed and the authors seem to be aware of the potential of the method.
review_title: null

Meta-review: This paper suggests a simple re-parameterization, motivated by recent theoretical results, which for improves SOTA results in the areas of pruning and continual learning. All reviewers seemed to like the idea, the writing and the results. The rebuttal addressed remaining concerns, and all reviewers voted for acceptance.

Decision: Accept (Poster)