Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper describes a method based on Fisher-information to estimate the importance of different network parameters. Once the top-k parameters in terms of importance are identified, the authors only update these parameters during training. The authors demonstrated the usefulness of their approach in transfer learning experiments (fine-tune only the task-important parameters), distributed/federated learning experiments (to reduce the communication volume through communicating only the sparse updates), and model checkpointing (by updating only a subset of parameters each epoch, and storing the deltas instead of the full parameter tensors at each epoch). 
main_review: The paper is clearly written. I find the idea simple and practically relevant. The experimental results are convincing. Novelty is rather limited, however, as estimating parameter importance using Fisher information (the squared norm of the parameter gradient) has been explored before (see [1] for example). 

Minor comments:
- L118 : Missing reference
- For clarity, I recommend the author use the term ‘mask sparsity’ instead of ‘sparsity’. The latter term is typically used to refer to weight sparsity (i.e, the percentage of weights that are zero), and refers to a body of literature (sparse training, pruning, etc..) that is different from this work.

Overall, I think the paper is borderline as it does not offer any new insights or novel methodologies. However, the practical application of the simple idea presented (pre-compute an update mask based on Fisher information to only update the important parameters) could prove generally useful.

[1]Kirkpatrick, James, et al. "Overcoming catastrophic forgetting in neural networks." Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work introduces FISH - a method of training (or fine-tining) a neural network by updating only a small fraction of its parameters. FISH uses an approximation of the Fisher Information of each of the network’s parameters to determine the top $k$ ones to update. The method is evaluated via transfer learning and distributed learning tasks using GLUE and CIFAR-10 datasets. The experimental results show that FISH has competitive or better performance in comparison to existing methods by updating just a small fraction of a networks’ parameters at a time. Further, FISH can reduce the memory usage when saving checkpoints.
main_review: ### Strengths
- The idea of using an approximation to the Fisher Information of each parameter for selecting the parameters that have the largest impact on the output is well motivated.
- The proposed model with FISH mask outperforms other benchmarks while having a lower level of sparsity, including a random mask (and with the exception of Bit-Fit in Table 1 which is harder to compare to since its sparsity is much lower).
- To the best of my knowledge, the paper provides a comprehensive literature review and background. 
The paper is organized and written clearly.

### Room for improvement and questions
1. What is the time/complexity of computing the (approximate) Fisher Information? The text mentions that it is computationally efficient but does not elaborate further. 
2. The work would benefit from reporting error bars/running experiments with several random seeds. 
3. Have the authors explored how the frequency of mask computation affects the models’ performance (other than using a fixed mask throughout training and updating it at every epoch)?
4. Are the FISH models in Fig 2 trained with a fixed mask or with a mask which is updated at every epoch?
5. Is it feasible to use the FISH mask in the case of unsupervised learning? 

### Nitpicks
- Line 92: “over N samples”
- Line 94: $F_{\theta}$ should be $\hat{F}_{\theta}$ 
- Line 118: missing reference
- Line 177: “of which” should be removed
- Line 213: “server. Results”


limitations_and_societal_impact: Not explicitly. The limitations could include computational overhead when computing the Fisher information for some particular datasets for which its approximation is not sufficient. Potential negative impact on society can occur if the underlying model that is trained using FISH is biased or has unethical applications.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose a scheme to select and update only part of the weights of a neural network during training. The selection of the weights to update is done using an approximation of the Fisher information, and the mask is computed by keeping only the weights associated with the larger fisher information values. The authors then propose an extensive empirical evaluation of their method for transfer learning, distributed training and efficient checkpointing. Finally, the authors perform an ablation study to show the effect of their design choices.
main_review: Review Summary
--------------

As a disclaimer, I am not an expert for neural networks training and in particular, I might have missed part of the newest literature.
The contribution of the paper, while not very surprising, is well explained and evaluated. Some improvement could be made with the limitation (**L`1**) and the discussion of the results (**M1**) but I feel that this paper is worth accepting to NeurIPS.

Major comments and questions
----------------------------

The paper is clear and the main idea is well explained. The numerical experiments seems convincing, in particular for the transfer learning ones. I have a few comments and clarifications I would like to ask:

- **M1:** In the results comments for distributed learning and efficient checkpointing, while the performance drop by 0.5 to 1%, the authors state that the performance are comparable. To make this statement clearer, I think repeating the experiment and adding standard deviation would improve the results. Else, I would state that there is a small performance loss here.

- **M2:** I feel like this experiments are a bit unfair for the proposed method, in particular with distributed training and small checkpoint experiments. Indeed, the idea of training only part of the weights for transfer learning makes sense but for a randomly initialized network, training for a few tens of epochs before switching to training only the most important weights (as decided with the FISH mask) would make sens. It seems that the authors implemented such feature with the `warmup_epochs` parameter but it is unused in the script they provided. Could the authors comments on this?

- **M3:** A nice addition to Figure.2 (left) would be to add the performance of the random mask for different sparsity levels too. This way, one would see the value of selecting the coordinates with the proposed methodology.


Minor comments, nitpicks and typos
----------------------------------

- l.118: Broken hyperlink.

- References: It seems that this line of work is somewhat linked to block coordinate descent algorithms and active set ideas in classical convex optimization. The core idea in this two line of work is to only update part of the parameters at once. In particular, the idea of using the Hessian information -- which is linked to the fisher matrix -- to select and split the coordinate between workers in distributed optimization has been proposed in the context of sparse optimization [A]. I feel that highlighting the connection between the two fields would probably be beneficial to draw attention to this line of work from both side.

Extra references
----------------

[A]: Scherrer, C., Tewari, A., Halappanavar, M. & Haglin, D. J. [Feature Clustering for Accelerating Parallel Coordinate Descent](https://proceedings.neurips.cc/paper/2012/hash/4e732ced3463d06de0ca9a15b6153677-Abstract.html). in Advances in Neural Information Processing Systems (NeurIPS) 28–36 (2012).
limitations_and_societal_impact: 
- **L1:** One major limitation I see for the proposed method is that it does not seem to apply to regression. The full paper seems to have been written with classification task in mind, where the logits are amendable to the score function and all experiments are performed for classification tasks. Could the authors comments on the applicability of their method to the regression setting? Indeed, in this case, the score function $p(y | x$) is not directly accessible. If the method cannot be applied in this setting, this limitation should be made clear in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper attempts to discover a fixed sparse subnetwork before the standard training. Only training the masked subnetwork can sufficiently match or even exceed the performance of the dense counterpart. The performance achieved by the proposed method is promising.
main_review: Originality: The novelty of this paper is limited. 

(1) The pruning technique that is used in this paper, Fisher approximations, is already widely used in the pruning literature [1][2]. It is not surprising to see that fisher pruning performs well before training. On the other hand, there is recently an increased number of works on pruning before training, e.g., SNIP [3], GrsSP[4], Synflow [5]. This paper is likely to belong to this field. However, none of these papers is cited and compared properly in the submission.  Fairly, directly applying fisher pruning to the pruning before training field is still valuable if it can outperform or match the above-mentioned methods with higher sparsity. I would highly encourage the authors to compare with them and report the results. Either positive or negative results are valuable for the community.

(2) As the proposed methods are doing sparse training (dense-to-sparse training), an important class of sparse training methods is ignored, that is, dynamic sparse training (DST). Instead of keeping the mask fixed,  DST allows the mask to dynamically change during training, improving the expressibility of the model gradually. Thus, a comparison with the SOTA DST methods such as RigL [6], ITOP [7] is suggested.

Quality:

The paper is easy to understand. However, these are some big mistakes that demonstrate that the authors have limited knowledge of the research field of pruning. As far as I know, the common definition of sparsity refers to the ratio of the pruned/masked parameters to the total number of parameters. The usage of sparsity in this paper is exactly the opposite to it, e.g., line 228. Moreover, there are some typos here and there, e.g., line 118; the color in figure 1 is difficult to discriminate for readers.

Although I like the fisher pruning idea, the carefulness in the current version degrades the quality of the paper significantly. 


Reference
[1] Theis, Lucas, et al. "Faster gaze prediction with dense networks and fisher pruning." arXiv preprint arXiv:1801.05787 (2018).
[2] Liu, Liyang, et al. "Group Fisher Pruning for Practical Network Compression." International Conference on Machine Learning. PMLR, 2021.
[3] Lee, Namhoon, Thalaiyasingam Ajanthan, and Philip HS Torr. "Snip: Single-shot network pruning based on connection sensitivity." arXiv preprint arXiv:1810.02340 (2018).
[4] Wang, Chaoqi, Guodong Zhang, and Roger Grosse. "Picking winning tickets before training by preserving gradient flow." arXiv preprint arXiv:2002.07376 (2020).
[5] Tanaka, Hidenori, et al. "Pruning neural networks without any data by iteratively conserving synaptic flow." arXiv preprint arXiv:2006.05467 (2020).
[6] Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International Conference on Machine Learning. PMLR, 2020.
[7] Liu, Shiwei, et al. "Do we actually need dense over-parameterization? in-time over-parameterization in sparse training." International Conference on Machine Learning. PMLR, 2021.
limitations_and_societal_impact: There are some limitations in the paper that needs to be addressed in the rebuttal. I would consider increasing my score if they are addresses.
review_title: null

Meta-review: The paper studies an approach to pre-selecting a subset of parameters of a neural network to update during training. The idea is to look at (an empirical approximation to) the Fisher information of each parameter, and choose the parameters with the k-largest Fisher information values. Only these parameters are updated during training. The paper evaluates this idea for transfer learning (updating only a subset of the parameters for new tasks), sparse training from scratch, distributed training, and efficient checkpointing (where we save a sequence of sparse updates to the model). The proposed method achieves performance similar to dense training, with fewer updated parameters (between 0.5% and 2% depending on the task), and some performance improvements compared to several previous sparse mask generation techniques. 

The reviewers all appreciated the paper’s simplicity, clarity and reasonably thorough experimental validation. Questions raised during the review included the novelty of the work (given that the Fisher information has been widely used as a measure of parameter importance, including for neural network pruning), and the relationship to pruning / sparsification methods. The authors’ response points out that although the use of Fisher information is not a novelty of this paper, the setting is different: this paper selects a subset of parameters to update, rather than selecting a subset of parameters to retain, which makes it applicable to transfer learning in dense models. 

After several rounds of discussion, the reviewers converged to a recommendation to accept. While there are some limitations to the novelty of the submission, the proposed approach is practical, clearly described and well-substantiated. 
Decision: Accept (Poster)