Training Neural Networks with Fixed Sparse Masks
Authors: Yi-Lin Sung
Varun Nair
Colin Raffel
Venue: NeurIPS 2021 Poster (2021)
Keywords: Fisher information
sparse updates
parameter-efficient transfer learning
distributed training
efficient checkpointing

Abstract: During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration. Recent work has shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements. In this paper, we show that it is possible to induce a fixed sparse mask on the model’s parameters that selects a subset to update over many iterations. Our method constructs the mask out of the $k$ parameters with the largest Fisher information as a simple approximation as to which parameters are most important for the task at hand. In experiments on parameter-efficient transfer learning and distributed training, we show that our approach matches or exceeds the performance of other methods for training with sparse updates while being more efficient in terms of memory usage and communication costs. We release our code publicly to promote further applications of our approach.

Training Neural Networks with Fixed Sparse Masks
Yi-Lin Sung" UNC Chapel Hill ylsungåcs .nc. edu
Varun Nair" Duke University vn40@duke. .ddu
Colin Raffel UNC Chapel Hill craffel cerffe@@maal..cmm com
Abstract
During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration. Recent work has shown that it possible update only small subset of the model's parameters during training, which can alleviate storage and communication requirements. In this paper, we show that it is possible to induce fixed sparse mask on the model's parameters that selects a subset to update over many iterations. Our method constructs the mask out simple approximation
the k parameters with the largest Fisher information as to which parameters are most important for the task at hand. In experiments on parameter-efficient transfer learning and distributed training, we show that our approach matches or exceeds the performance of other methods for training with sparse updates while being more efficient in terms of memory usage and communication costs. We release our code publicly to promote further applications of our approach
Introduction
Stochastic gradient descent (SGD) is a vital component of the modern pipeline for training deep neural networks. Along with the minimization of loss function by gradually updating a model's parameters. SGD minimizes the loss over small random subset of the dataset at each training iteration, which allows training over large datasets. In practice, minimizing a large neural network's 's training loss using SGD often results in making SGD an invaluable tool.
algorithm, gradient descent allows for the efficient
models that generalize well to new data 3 17 27,,
While effective, standard SGD requires that all model parameters are updated at every iteration of training. As a result, communicating changes to the model requires communicating the updated value of every parameter. Since modern neural networks often have millions billions of parameters 40) this communication can become excessively expensive. A concrete example of the negative impacts of these costs arises in the setting of transfer learning. In transfer learning, a model's parameters are initialized from an existing pre-trained model before being fine-tuned (i.c. trained) on task of interest. Pre-trained models can be fine-tuned huge number of times for example, the Hugging Face model repository has thousands of fine-tuned variants of the BERT model [12] Each of these fine-tuned variants requires a unique copy of the model's 's parameters, each of which takes up around 500MB of disk space. Relatedly, in distributed training [II and federated learning 35), workers compute updates for centralized model parallel on different subsets data. After certain number of updates, the workers each communicate the newly-computed parameter values back to the centralized model. The communication step can cause a significant amount overhead (particularly when the model is large) since the workers must communicate the updated values of all parameters when using standard SGD.
"Equal contribution.
"Code for our work can be found alttppp//ggttaat.cco/a/auaaattas https://bugginnffcce co/mode13
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Top
Figure I: Diagram comparing our proposed method to standard SGD. In traditional mradient-based training (left), all of a model's parameters are updated every iteration. We propose FISH Mask. a method for precomputing sparse subset of parameters to update over many subsequent training iterations. To construct the FISH Mask, we find the k parameters with the largest Fisher information (right, top). Then, we train the model with traditional gradient descent, but only update those parameters chosen by the mask (right, bottom).
These issues could be mitigated if it was possible to only update few parameters during training while still maintaining performance close to that of training all parameters. This has led to various work on parameter-efficien training neural networks. For example, Adapters [19 []] introduce additional parameters into pre-trained model in the form of small task-specific modules that are fine-tuned while the rest of the model's parameters are kept fixed. Diff Pruning [[6] and BitFit 66] demonstrate that it is possible to fine-tune model while only updating small subset of the existing parameters. In distributed and federated learning settings, Aji and Heafield [2] and Konečný et al. (3 have shown that it is possible for each worker only update sparse subset of model's parameters, thereby reducing communication costs.
Existing methods for training with sparse updates typically work in one of three ways: they either add parameters to the model (Adapters), choose hand-defined and heuristically-motisee of parameters (BitFit), or allow the subset of parameters to change over the course of training (Diff Pruning and methods in distributed and federated training). In this paper, we argue for pre-computin sparse subset of existing parameters to update and keeping the subset fixed over many iterations of training. This approach yields various benefits: First, by updating a subset existing parameters instead of adding parameters (as is done in Adapters), we avoid any increase in the total size of the model. Second, by avoiding hand-defining the mask, we can ensure that our procedure model- agnostic. Third, by pre-computing mask, avoid the computational and memory overhead that apparent when updating the mask over the course of training. also allows workers in the distributed training setup to operate strictly complementary subsets of parameters. Finally, keeping the mask fixed over many iterations, we can ensure that only a specific fixed number of parameters are updated. We are not aware of any existing techniques that satisfy these desiderata.
subset
Motivated by these benefits, we introduce new method for pre-computing fixed sparse masks. Our approach first estimates the importance of each parameter using empirical approximation of its Fisher information. Then, we construct the mask by choosing the k parameters with the largest Fisher information. The resulting mask, which we deem "FISH (Fisher-nnuueed Induced Sparse uncHingig mask", can be re-used for many subsequent training iterations. We demonstrate the effectiveness of using a FISH Mask in wide variety of settings, including parameter-efficient transfer learning, distributed training with long delays across workers, and reducing checkpoint size. Broadly speaking, FISH Mask training can dramatically reduce storage and communication requirements, while sacrificing minimal performance compared to standard gradient descent and outperforming relevant prior methods for training with sparse updates.
2 The Fisher-Induced Sparse uncHanging (FISH) Mask
The main contribution of this paper is a method for pre-computing sparse subset of model's parameters to update over many subsequent training iterations. To construct such a subset, we use an approximation of each parameter's Fisher information as a signal of how important the parameter for a given task. We refer to the resulting mask (i.e. binary array indicating which parameters are

---

included in the subset) as FISH (Fisher-Induced Sparse uncHanging) mask. In this section, we provide the necessary background and detail the steps necessary for computing FISH Mask. This process is diagrammed in fig.
2.1 Fisher Information
Our goal is to select the subset of parameters that are (in some sense) the most important update. One way to measure a parameter's importance is to consider how much changing the parameter will change the model's output. We denote pe (ylar) ) as the output distribution over y produced by a model with parameter vector IR given input x. One way to measure how much a change in parameters would change model's prediction would be compute DKL (po (y (x:) po+ (+++(yla))) where E IR (1 a small perturbation. It can be shown 34 37
that as 5 + 0, to second order,
E. (pe (y/ar) || pe+s(y|z)) = F66 O(
(1)
where Fo x|00 the Fisher information matrix 14 defined as
F9 = Ex>p(x) (y-- Va log log logg
Given this relation, it can be seen that the Fisher information matrix is closely connected to how much each parameter affects the model's predictions. Indeed, this has led the Fisher information matrix to be widely used in modern machine learning, e.g. as a measure of parameter importance [21], as preconditioner in gradient descent 4 37 (4), way measure the amount of "information" in each parameter of a neural network (1. or as a way to decide which parameters to prune when
performing model compression 43
10 47
When applied to large neural networks, the [0] x 10 size of the Fisher information matrix makes intractable to compute. Prior work therefore frequently approximates Fe diagonal matrix, or equivalently, as vector in IR (l), Separately, when training machine learning models we seldom have the ability to draw samples x p(z); instead, we are given finite training set of such samples. Furthermore, it is rarely necessary to compute the expectation over in eq. (2) over the full training set; instead, it can often be well-approximated over N samples .1,..., UN- These constraints result in the following common approximation:
N Σ
E (V loggoo(u)z)))
where Fg R! This approximation also has an intuitive interpretation: given entry in Fg relates to the average of the square gradient of the model's 's output with respect to given parameter. If given parameter heavily affects the model's 's output, then its corresponding entry in F will be large, so we can reasonably treat Fg as approximation of the importance of each parameter.
Note that both eq. 2 and eq. B3) include an expectation over y ~ po(y)z). When the number of classes is small, this expectation can be computed exactly. For tasks with many possible classes, it is common to approximate the expectation with few samples from po lr) supervised learning settings, we have access to the ground-truth label ys for each sample 2; in our training set. This leads to the possibility of replacing E, Ey-os(ulzz)) (V log (o(y)rr))) (y)x;)) in eq. (3) with (V& log Pe )2 Performing this approximation is referred to the "empirical Fisher". It has been shown that using the empirical Fisher can lead to degenerate behavior when used as preconditioner in an optimizer 34 26 Since our use of the Fisher information is largely based on heuristically-motivaee
notion
of parameter importance, we experimented with both the empirical and standard (eq. (3)) Fisher approximations and found that they produced similar performance. Furthermore, the Empirical Fisher faster to compute than the standard Fisher as long as more than one sample is used to approximate the expectation Eyy-pely)):)- We discuss this further in section 4.4.1
2.2 Computing Fixed Sparse Masks
Recall that our goal to select a subset of parameters (or, equivalently, a sparse mask over parameters) to update over many iterations of training while keeping the remainder of the parameters fixed. Having established the Fisher information as useful tool for estimating the importance of given parameter, we therefore first compute the approximate Fisher information (as described in the previous section)
3

---

for all of a model's parameters. Then, to construct the FISH Mask, we simply choose the k parameters with the largest Fisher information, where l is set according to the desired mask sparsity level Specifically, FISH Mask comprises the parameters {0, Fo. 2 (Fa)(F))x) Computing the FISH Mask is cheap because Fg can be computed efficiently using backpropagation, and (as we will show in section 4.4.2) we obtain a reliable mask for relatively small values of N. Further, the fact that we re-use the mask for many iterations prevents us from having to compute Fg frequently, As we will show in section [ we find that this simple procedure is sufficient to produce a mask that can be reused for many iterations (over 100,000 iterations in some cases) in a wide variety of settings without sacrificing substantial performance compared to standard gradient- based training.
Note that in some applications of transfer learning, new linear classifier layer must be added to the model to make it applicable to the downstream task. Since the FISH Mask depends on po (y)zz and is computed before training begins, this means that must compute the FISH Mask using the wandomly-initializzn
classifier before any training has begun. We find that computing the Fisher classifier layer still provides good signal of parameter importance. When applying FISH Mask in transfer learning settings where a new classifier layer is added, we always include the parameters of the classifier in the mask.
information through the randomly-nnitializze
3 Related Work
Our approach bears similarity and takes inspiration from existing approaches for parameter-efficient transfer learning and distributed training of machine learning models. In this section, we outline related methods, some of which we will compare to directly in section 4 We also briefly describe how our work is related to and differs from work in network pruning.
3.1 Parameter-Effficiet Transfer Learning
Transfer learning [36]. where a model is initialized from pre-trained checkpoint before being fine- tuned on a related downstream task, can dramatically improve performance and speed up convergence
on the downstream task [12] 8 40 Standard practice is to update all of the model's parameters
during fine-tuning, though in some cases reasonable performance can be attained by only fine-tuning
the output layer of the model 20 38
Training only the output layer has the benefit that adapting given pre-trained model to a downstream task only requires adding a relatively small number of new parameters, but typically results in worse performance compared to training the full model [8 24)
Various methods have been proposed that endeavor to match the performance of fine-tuning the full model while only updating or adding a small amount of parameters. Adapters [19 41 5 are small
subnetworks that are added between a pre-trained neural networks layers. Various works 19 have shown that, when appropriately designed, updating only the parameters in the adapters and the output layer can approach the performance of fine-tuning all parameters. For example, Houlsby et al. [19 add on average 3.6% more parameters to adapt pre-trained BERT model [12 to tasks in the GLUE benchmark [48) Concurrent work by Mahabadi et al.
3 32
33 improves the efficiency of Adapters by generating the weights of task- adapters via hypernetwork. A second concurrent approach by Mahabadi et al. B2] introduces COMPACTER, which utilizes matrix decomposition and low-rank parameterization for the adapters' weights COMPACTER is shown to achieve the same performance as standard fine-tuning of the T5 models 40 while only adding 0.047% as many parameters as the original model. Finally, very recent work has shown that it is possible to train language models to perform a task by only optimizing the parameters of a "prompt" that is injected into the input of the model's 's layers 229 28). This can yield extremely parametter-efficiett results (as low as 0.01 % task-specific parameters 28 but this class of methods is only applicable to next-step-prediition language models. The main drawback of all Adapter-style methods is that they increase the parameter count and computational cost of the model. This makes them inapplicable to the distributed training and efficient checkpointing settings consider in this paper. We therefore only compare directly to other methods that do not add any parameters.
More closely related to our approach are methods for choosing small subset of the model's existing parameters to update. In an extreme case, Zhao et al. [50) find a sparse mask to multiply against
"To avoid confusion, we use "mask sparsity" rather than "sparsity", as in some works of literature (e.g. network pruning) the latter term has opposite meaning that denotes the percentage of weights being zero

---

pre-trained parameters (which are not otherwise updated). That is, instead of fine-tuning the models, a binary mask is learned that marks which parameters should be zeroed out. The resulting performance degrades heavily when the mask is made very sparse, suggesting that it is likely beneficial to update parameters. More recently, Guo al. [16] propose "Diff Pruning", where a sparse binary mask is found over the course of training that denotes which parameters should be updated or fixed at the value from the pre-trained model. Mask sparsity in the binary mask is enforced through a smooth approximation of the Lg norm introduced by Louizos et al. (31]. Guo et al. [16] also show improved performance by imposing a structure on the mask according to which parameters correspond to a particular weight matrix or bias vector. Ultimately, Diff Pruning is shown to both be more parameter- efficient and outperform Adapters when applied to fine-tuning BERT on the GLUE benchmark. However, using Diff Pruming requires significantly more memory during training in order to store and update the mask. Another recent result by Ben-Zaken et al. [6 demonstrated that simply updating the bias parameters in BERT a simple and strong baseline, it is not universally applicable- - for example, the pre-trained T5 model 40) does not have any bias vectors. In section we show that using FISH Mask outperforms all of these approaches in parameter-effiiien fine-tuning of BERT on GLUE
attain competitive performance with Diff Pruning. While this provides
3.2 Distributed Training
As models and datasets grow, it becomes inefficient or impossible to train a model on a single machine. This has motivated the need for distributed training strategies where computation for training a model is shared across many machines (called workers) 11] A major consideration in distributed training communication costs, since workers need regularly communicate parameter updates with one another. To minimize communication costs, workers can compute multiple updates on their copy of the model before communicating their changes, but this gives rise to the "stale gradient" problem where workers are operating on an out-of-date copy of the model. The standard and straightforward approach to dealing with stale gradients
simply apply updates in the "wrong" order, which can be effective in practice [T] 42). An orthogonal approach to reducing communication costs is to have workers only update small subset of the model's parameters 2 L3 44 45).
For example, Aji and Heafield [2] simply have workers communicate only those updates corresponding to the gradients with top-k largest magnitude at each step. This bears a similar motivation to FISH Mask, but results in "mask" that changes at every iteration and therefore requires workers communicate after each update. In contrast, pre-computing a FISH Mask allows workers to perform multiple iterations before communicating their updates, thereby further reducing communication costs.
An extreme variant of distributed training is federated learning 22 35 In federated learning, asynchronous workers perform many updates on private data before communicating the changes back centralized model. The training involves one server and multiple clients, and the server model's gradient is the combination of the workers' gradients. As with any form of asynchronous training, communication costs and stale gradients are significant issues. McMahan al. [35] demonstrated that averaging the updates computed by individual workers is an effective approach to dealing with stale gradients and Konečný et al. (33 investigated techniques for significantly reducing communication costs. Our method is complementary to the techniques for reducing communication proposed by Konečný et al. (23
3.3 Network Pruning
Past work in network pruning has also explored techniques for sparsifying neural networks (i.e. zeroing out many parameters for compression purposes) while sacrificing minimal performance. Most relevant to our work, Theis et al. [46] propose utilizing the Fisher to prune and decrease the overall number of parameters for gaze prediction, and Liu et al. [30] also use the Fisher to discover groups of parameters to prune from common backbone architectures. Critically, these works differ from our work in that we do not train neural networks with sparse weights. Instead, we focus on using the Fisher to inform the selection of a fixed sparse subset of weights in a non-sparse network to update over the course of training.
5

---

4 Experiments
We evaluate the efficacy of the FISH Mask in three settings: parameter-efficient transfer learning. distributed training, and training with efficient checkpointing. For parameter-efficient transfer learning, we demonstrate that our approach matches the performance of standard gradient-based training on the GLUE benchmark [48] while updating only 0.5% of the model's parameters per task. For distributed training, we evaluate FISH Mask training for both transfer learning on GLUE and training from scratch on CIFAR-10 [25]. In both settings, we find that we can dramatically reduce communication without sacrificing significant performance, though from-scratch training on CIFAR-10 requires a higher mask sparsity level than fine-tuning on GLUE. Finally, we demonstrate a novel application of training with sparse updates: Minimizing the size of checkpoints over training. We show that using FISH Mask while training on CIFAR-10 with mask sparsity level of 10% can shrink checkpoint size on disk by a factor of 5 while sacrificing only a small amount of accuracy. Throughout our experiments, we report results with varying mask sparsity to get sense of the savings induced by the FISH Mask. For ease of comparison, we report mask sparsity in terms of the total percentage of parameters that are updated. This percentage can be converted to a value of k used for the top-k operation when constructing the mask simply by multiplying it against the total number of parameters in the model.
We also include ablation studies to measure the impact of the number of samples used to estimate the Fisher information as well as the choice of true or empirical Fisher. All experiments for GLUE are run with the LARGE variant of BERT, which contains 16 attention heads, 24 layers, and 330 million parameters in total [2]. and most experiments are run on RTX 3090 GPU. For experiments on CIFAR-10, 10, we use a ResNet-34 [8] with various optimizations for fast convergence We report the average performance across 5 seeds for all experiments.
4.1 Crrameter-Efficieee Transfer Learning
In parameter-efficient transfer learning. the goal is to fine-tune a pre-trained model while updating as few parameters as possible. We focus on fine-tuning BERT on the GLUE benchmark [48]. which is the primary setting used for evaluation prior work. For all experiments, we fine-tune for epochs and perform a seeper-parameer search across learning rate E {1 x 10-4, x 10-5, 1 x 10-5} and batch size E {8, 16) for each GLUE task. We find the learning rate of 5 x 10 5 and batch size of 16 to be effective for most tasks, with the exception of batch size = 8 used for RTE. Additional pyper-parametesd
such as choice of optimizer, sequence length, and others, follow from the default configuration for BERT presented in the Hugging Face library (49. Test set results are reported by submitting to the GLUE benehmark using the final model checkpoint following a hyper-parameter search on validation results, unless otherwise noted.
Baselines We compare GLUE task performance of the FISH Mask to several other baselines and transfer learning. In Dense Fine-tuning, we fine-tune all
methods focused on parameters of pre-trained model, as is typical in standard transfer learning. In the Random Mask baseline, we randomly select and fix k parameters to update at the start of training. To compare to prior work, we reproduce Bit-Fit (6) which only the bias parameters of the BERT model are updated across training. Our reproduction follows the original paper and performs a hyper-parameter search with learning rates in the [1 x 10-3, 1 x 10-") range. We also reproduce results from Diff' Pruning [16.. which updates the sparse mask over the course of training. Our reproduction of Diff' Pruning at 0.5% mask sparsity follows the paper's s code-basd and training settings, and reports the GLUE test set results using the best validation checkpoint. Due to restrictions on the number of permissible submissions to the GLUE test server, we are only able to report results with a mask sparsity of 0.5% for those methods where we control the mask sparsity level. We therefore include additional validation set results for varying mask sparsity levels when using a FISH Mask.
Results Our results on parameter-efficien transfer learning with the FISH Mask can be seen in table FISH Mask training results in effectively the same performance (82.6%) as standard "dense" fine-tuning (82.5%), despite updating just 0.5% of BERTLARGE parameters. The Random Mask baseline achieves a significantly lower average GLUE score, which demonstrates the value of using
5 'Our implementation is based on htt.t////ie/////ghohoocoooo/oaaa

---

Mothod
Updated Params/Task
QNLI
SST-2
MNLI,
MNLI
CoLA
MRPC
STS-B
RTE
QQP
AVG
Dense Fine luninn
100%
93.4
94,9
87.0
$6.1
61.0
86.6
86.5
70.9
80.5
82.5
Random Mask
0.50%
89.8
93.4
83.7
84.0
43.2
71.8
87.7
61.3
77.2
76.8
Bit-Fit
0.08%
94.5
85.0
60.3
86.3
85.0
78.5
81.2
DiIT Pruning [T6
0.50%
91.9
93.8
86.0
85.5
61.0
86.2
85.6
67.5
80.1
81.5
FISH Mask
0.08%
94.0
85.3
56.4
86.2
85.7
70.2
79.3
81.3
FISH Mask
0.50%
93.1
94,7
86.5
85.9
61.6
87.1
86.5
71.2
80.2
82.6
Table GLUE lest server evaluation results with BERT MRPC and QQP are reported as an average of Pearson and Spearman
average of Fl-score and accuracy, and STS-B is reported correlation. Accuracy is reported for all other tasks. All results are reproduced experimentally. Training with the FISH surpasses (82.6) other methods and equals dense fine-tuning performance (82.5) whilst updating only 0.5% of model parameters.
95
95
Random Mask
90-
RSHH Mask
90-
GUUE valdation soone
85
Dense fine trninuning
GLUE validationn score
85
Dense fine tuning
75
70-
70-
65
65
60
80
0.02
0.1
0.5
2.5
32
256
1024
16384
Mask Sporsiy
FISH Mask samples
Figure 2: (Left) GLUE validation performance randomly selected mask and the FISH Mask at of 85%, training
varying levels of mask sparsity. Compared to the densely fine-tuned baseline with the FISH Mask is competitive at 0.5% mask sparsity. (Right) GLUE validation performance varying levels of dataset samples used to compute the FISH Mask. Few samples needed to effectively compute the FISH Mask and obtain good performance. Results in both (Left) and (Right) averaged over seeds.
the Fisher information in selecting parameters to update. Finally, FISH Mask training competitive with all other parameter-efficient transfer learning approaches; the next-best score of 81.5% is achieved by Diff Pruning [16]. As mentioned in section we do not include direct comparison to Adapter-based methods since they add parameters the model, though we note that our method is able to match BERTLARGE 's performance using a significantly lower mask sparsity level (0.5% vs. 3.6%) than the method proposed by Houlsby et al. [99..
Figure[ishows shows the change in performance on the GLUE validation set as we change the mask sparsity level for both random mask and FISH Mask. We find that the FISH Mask consistently outperforms the random mask baseline and is still strong even at a lower mask sparsity level of 0.1%. These results demonstrate that the FISH Mask can be useful tool in mitigating the storage costs of saving many fine-tuned models since it only requires that the updated parameters and their respective indices saved.
4.2 Distributed Training
Now, we turn to using FISH Mask to reduce communication costs in distributed training settings. We consider the setting where distributed workers compute many subsequent updates a local copy of the model before transmitting their changes back to a central server. In our experiments, we assume all workers sample data i.i.d. from the same dataset and that all workers compute the same number of updates between each communication step, though our results could carry over to settings where workers different datasets and varying amounts of updates (e.g. in Federated Learning (35)).
Let denote the parameter vector stored on the central server and 8, represent update computed by worker i. After each uptate/communicatioon step, the server must transmit the updated 0 back to all workers. When standard gradient-bast training is used, all parameters are updated, so |& = [0]. there M workers, the communication costs for normal training then M [8, M 0 for worker-

---

to-server communication and | for server-to- communication, for a total communication cost of (.
Using a sparse mask will effectively reduce the size of each 6; to k. Without loss of generality, we assume that |o,] is the same across all workers. Furthermore, assuming updates are aggregated on the central server by summing them together, the server can either communicate the full updated parameter vector back to all workers, or the server can communicate all other workers' updates to a given worker. These two options amount to server-to-worker communication costs of either M|0] or (M - 1)]|6:| ~ M In general, we expect M be small and [8] to be large, so we typically have | < M) Therefore, we can achieve significant communication savings by using sparse updates. For simplicity, we set M 2 in all of our experiments; this makes the savings in communication equal to the mask sparsity level. To apply FISH in distributed training, the central server computes Fg and single FISH Mask which is shared across all workers.
Baselines For distributed training experiments, we compare FISH Mask training to three baselines: First, in standard training, we tune all the parameters in single machine (i.e. standard, non- distributed training). This provides an upper bound on performance for distributed training techniques. Second, in densely-updated distributed training, workers use standard gradient descent compute updates over all parameters, and all of the worker's updates are added together on the centralized server. Third, in random mask distributed training, we randomly select subset of parameters to update for workers. For fair comparison, we keep the overall training batches the same for all methods, so the training iterations of worker are half of that of the manle-mmchine baseline.
Experimental setting In preliminary experiments the GLUE benchmark (described in ap- pendix[A). we found that densely-updated distributed training saw no real degradation in performance even for long communication delays. This suggests that stale gradients could be less of an issue in transfer learning settings. We therefore instead focused on from-scratch training of a ResNet-34 on CIFAR-10. We train the model for 100 total epochs, with 50 epochs performed by each of the two workers. Notably, we found that the performance of sparse update methods was poor for from- scratch training unless we performed epochs of standard training as "warmup" before beginning distributed training. Beyond this change, all the models and hyper-parameter follow those mentioned in section 4.1 We searched for the best initial learning rate in (0.4, 0.2, 0.08, 0.04, 0.02). We measure performance using a varying number of parameter updates between each worker-server communication step. We report performance in terms of the accuracy achieved under certain communication budget, where the communication cost is measured in terms of the equivalent number of full model parameter updates. For example, a method that updates 10% of the model's parameters and performs 5 communication steps over the course of training has the same cost as method that communicates all of a model's parameters once (since we must transmit both the updated parameters and their locations; more details are in section
Results Results from training ResNet-34 on CIFAR- 10 are shown in fig. As in section 4.1 using a FISH Mask works better than using random mask for all communication costs. Furthermore, we generally find that using FISH Mask with sparsity level of 10% attains a better commnica- tion/performance trade-off than densely-updated distributed training. For example, FISH Mask training attains comparable performance to standard training when only communicating two copies of the model, whereas densely-updated training performs significantly worse at this communication amount. Notably, performance was relatively poor when only updating 2% of the model's I's parameters with FISH Mask, suggesting there is a lower bound under which it is difficult to attain reasonable results. As whole, our results show significant promise for dramatically reducing computational costs in distributed training settings.
4.3 Efficient Checkpointing
Over the course of training machine learning model, is common to save intermediate checkpoint files that store the model's parameter values. These checkpoints can be useful for restarting training from a given iteration rather than starting from scratch in the event that the training job crashes or is otherwise stopped. They are also commonly used for post-hoc analysis, for example for evaluating a model's performance on new datasets or metrics over the course of training. Since checkpoints store full copy of the model's parameters, they can take up significant amount of space on disk.
8

---

Standard training
94
Accuracy (56
90
FISH mask, 2% sparsity
FISH mask, 10% sparsity
Random mask, 10% spansity
86
Dense updates
10
10a
10a
10a
Communication cost per worker
Figure 3: CIFAR-10 validation set accuracy achieved by a ResNet-34 through distributed training at different communication costs. X-axis refers to the total number of model communications required for a single worker. Standard imon-distributed training achieves an accuracy of 93.9%.
Furthermore, depending on the checkpointing frequency, hundreds of checkpoints are often written to disk over the course of training run. The development cycle of machine learning model can result hundreds of different model variants being trained. Combining these factors with the on-disk space needed to store the parameters of modern models (around GB for BERT results in potentially massive storage costs.
Training with sparse updates can significantly reduce these storage costs. Specifically, if only a small subset of the parameters are updated between checkpoint saves, then the checkpoint only needs to store the updated parameter values and indices denoting the position of the updated parameters' values. Assuming that the storage costs for a parameter value and index is the same (e.g. using 32-bit float for parameter values and a 32-bit integer for indices), using sparse mask will reduce storage cost when the mask sparsity level is less than 50%. Note that this setting allows the "mask" to change over the course of training, and is therefore a relaxation of the setting in section 4.1 where the requirement is that the same subset of parameters is updated over the entire fine-tuning run. follows from our results in section 4.1 that FISH Mask training could be readily applied to reducing checkpoint size in parameter-efficient transfer learning. However, we found that the strict requirement of fixing the mask over entire from-scratch training on CIFAR-10 10 resulted in a significant degradation in performance. This is in line with past work demonstrating the difficulty of identifying fixed sparse subnetworks to train before training begins [15). We therefore focus on trom-scratch training on CIFAR- and allow the mask to change every time a checkpoint is written, which does not increase storage requirements over using the same fixed mask from the start of training.
Overall, we use the same experimen- tal setup as in section[422 4.2 We measure performance when updating the mask every epoch (which is common choice practice), every 2 epochs, every 4 epochs, and leaving the mask fixed over the course of training. We performed new search over learning rates in (0.4, 0.2, 0.08, 0.04, 0.02). We compare to baselines of standard training (which to serve as an upper bound the performance of using a FISH Mask) and using random mask.
Table 2: CIFAR- 10 validation set accuracy when using the FISH Mask and the Random Mask reduce checkpoint sizes. "Epoch" refers to allowing the mask to change each epoch, and the number is how many epochs update masks. Standard training achieves an accuracy of 93.9 (=..1)%
Mask sparsity level
0.5%
2%
10%
Random Mask (1 Epoch)
74.8p.6
84.40.2
90.0p.2
FISH Mask (1 Epoch)
90.5p.3
93.0g.3
93.9o.1
FISH Mask (2 Epochs)
90.3g.5
92.5a.1
93.7o.2
FISH Mask (4 Epochs)
89.4g.6
92.
93.40.2
FISH Mask (Fixed)
78.5g.7
90.6p.2
93.0p.2
we show the results of FISH Mask training when keeping the mask fixed over the course of training or updating at each epoch. We find that updating the FISH Mask every epoch can match the performance of normal training (93,9% accuracy) at mask sparsity level of 10%,
Results In table 2

---

which would reduce storage requirements by a factor of 5. At lower mask sparsity levels, we see some degradation in performance. We find that accuracy tends to decrease as we decrease the frequency of updating the mask, but this effect is relatively small. As mentioned earlier, we also found that using fixed mask significantly degraded performance, though only by a few percent at a mask sparsity level of 10%. This suggests that the FISH Mask could also be useful for identifying sparse subnetworks to train before training begins, as conjectured by the Lottery Ticket Hypothesis [[5]. We leave the exploration of this possibility for future work. Lastly, the FISH Mask's performance is unanimously better than the Random Mask across the three mask sparsity levels.
4.4 Ablations
Having established the effectiveness of training with FISH mask, we now ablate a few design choices to help demonstrate the robustness of our approach. We perform all ablation experiments in the parameter-efficient transfer learning setup described in section 4.1
4.4.1 True Fisher vs. Empirical Fisher
In section 2 we note that past work has approximated the Fisher information matrix (eq. 33) using either the expectation over po (y(zx) ("true Fisher") or ground-truth labels ("empirical Fisher"). While past work has shown that using the empirical Fisher can be detrimental in optimization settings 34 26]. we mainly use the Fisher information as signal of parameter importance. The empirical Fisher also has the benefit that it avoids marginalizing over or sampling from po(u]z)) and only requires computing the gradient for single value of y. When comparing the performance of using the true or empirical Fisher to compute 0.5%-s FISH Mask for parameter-efficien transfer learning, we observe that both methods achieve near-identical performance with an average validation-set GLUE score of 82.5 in both cases. Since computing the empirical Fisher can more computationally efficient, we used the empirical Fisher for all experiments.
4.4.2 Sample Ablation
We also ablate the number of samples, N, used to compute the FISH Mask to study if more samples are beneficial. The results for parameter-efficient transfer learning on GLUE are shown in fig. right. At sample count of O, the FISH Mask is equivalent to the Random Mask baseline presented in section [4.] in which parameters to update are selected at random instcad of informed by the Fisher information. We observe that FISH Mask performance is surprisingly stable across many values of samples, with just 32 samples needed to achieve the performance. These suggest that using the approximate Fisher information is data-efficient approach of computing parameter importance, and we therefore ran all experiments with sample count N 1024, except in distributed training we use N 256 for efficiency.
the GLUE validation set
5 Conclusion
In this work, we proposed FISH Mask training as novel method for pre-computing fixed sparse masks of model's parameters update over many subsequent iterations. The FISH Mask estimates the importance of each of a model's parameters by first approximating the Fisher information of each parameter and then selecting the k parameters with the largest Fisher information include in the mask. We demonstrate the usefulness of FISH Mask training in several settings, including parameter-efficient transfer learning, distributed training, and redueing storage requirements of model checkpoints. In future work, we hope to explore methods for improving the performance of FISH Mask training at lower mask sparsity levels, possibly by considering other measures of parameter importance. We also hope to further demonstrate the efficacy of FISH Mask in real-world settings where the benefits of sparse parameter updating are even more pronounced, such as in Federated Learning- The integration of FISH Masks across tasks and sharing amongst practitioners could also be a useful line of inquiry, as recent frameworks such as AdapterHub (39 have enabled for Adapter modules [19].
10

---

Acknowledgments and Disclosure of Funding
We thank Yoon Kim, Michael Matena, and Demi Guo for helpful discussions.
References
[1] Alessandro Achille, Giovanni Paolini, and Stefano Soatto. Where is the information in a deep neural network? arXiv preprint arXiv:/905. 2019.
[2] Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint 170.000004 2017.
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overpa- rameterized neural networks, going beyond two layers. CoRR, abs/1811.04915 2018. URL 04918
http::/arxxvvorrg/eae
[4] SI Amari. Neural leaming in structured parameter spaces-natural riemannian gradient. Advances in neural information processing systems, pages 127-133, 1997.
[5] Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint 190X.v:1999.080 08478, 2019.
[6] Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg, BitFit: Simple parameter-efficient fine-tuning for transformer- based masked language models. 2021.
[7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerberttVosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020. URL luttps://arrie / 14165
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020. URL https
//arxiv.e orgoogg////// abs/ /000..2222.0000 05709
[9] Ting Chen, Simon Korblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi- sumi-sspreiied learners. CoRR, abs/2000.11002,, 2020. URL, htt.ttp:://axxi..rrggg/
10029
[10] Elliot J. Crowley, Jack Turner, Amos Storkey, and Michael O' O'Boyle. Pruning neural networks: it time to nip it in the bud? 2018
[1]] Dean, G. Corrado, Rajat Monga, Kai Chen, M. Devin, Quoe V. Le, Mark Z. Mao, Marc Aurelio Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. NIPS, 2012.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171 4171-4186, Minneapolis. Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N19-1429 URL. htt.tttttssswww.../. aclweb. wwweee..rrgg/ee org/ wwwgg//nthlogyy./ooo NN1 N123
[13] Nikoli Dryden, Tim Moon, Sam Ade Jacobs, and Brian Van Essen. Communication quantization for data-parallel training of deep neural networks. In 2nd Workshop Machine Learning in HPC Environments (MLHPC), 2016.
[14] Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 222(594-604):399936 1922.

---

[15] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint vrXiv:1803.03633. 2018.
[16] Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efffiien transfer learning with diff pruning. CoRR, abs/2012.07463, 2020. URL UtL.ttpp:///rr....rrr//bb////
07463
[17] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning. 2016.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs//s/12 1512.03385, 2015. URL http:// /tttt:://rxii...o/ org/ awtaaaaab111 03385
[19] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for NLP. CoRR,
2019. URLpttt:://rxit org/
00751
[20] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifica- tion. CoRR, abs/ 0tts/1801.00 801.06146 2018. URL htkKlatt:///axx..ooo/aaa//////// httpr///ab///11111111111
[21] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins. Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trrabska-Barwinska. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13), 2017.
[22] Jakub Konečný, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimiza- tion: Distributed machine learning for on-device intelligence. arXiv preprint arXiv: 1610.02527, 2016.
[23] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprini arXiv: 1610.0549, 2016.
[24] Simon Korbblith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2019.
[25] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[26] Frederik Kunstner, Lukas Balles, and Philipp Hennig. Limitations of the empirical fisher approximation for natural gradient descent. arXiv preprint arXiv: /905. 12558, 2019.
[27] Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klbert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-48. Springer, 2012.
(28) Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 /04. 00..1044..699 08691, 2021.
[29] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini 2rXiv:2101.000
2021.
[30] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang. Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7021-7032, PMLR, 18-24 Jul 2021. httthttss:///roeeeete
mlr. Satsss/v339/lii220 html
[31] Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through /_0 regularization. arXiv preprini 17Xiv:17 2017.
[32] Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021.
[33] Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson.
Parameter-efffiiien multi-task fine-tuning for transformers via shared hypernetworks, 2021.
12

---

[34] James Martens. New insights and perspectives on the natural gradient method. arXiv preprint arXiv:1412.1193, 2014.
[35] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera
y Arcas. Communication-efficien learning of deep networks from decentralized data, 2017.
[36] Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions knowledge and data engineering, 22(10):134--1359 2009.
[37] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv: 1301.313. 2013.
[38] Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint 1931v:7903.05987. 2019.
[39] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: framework for adapting transformers. CoRR, abs/2007.07779, 2020. URLhttss:://arxi wwtp:/iiiwrg//aaaa http//bs/200... 077779
[40] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21, 2020.
[41] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. arXiv preprint arXiv: 1ee.i10050004400 2017.
[42] Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Shawe-Taylor, R. Zemel, Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances in Neural Information Processing Sys- tems, volume 24, Curran Associates, Inc., 2011.
cc/ .pdf
[43] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems, 2020.
[44] Nikko Strom. Scalable distributed dnn training using commodity gpu cloud computing. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
[45] Zeyi Tao and Qun Li. esgd: Communication efficient distributed deep learning on the edge. In {USENIX} Workshop on Hot Topics in Edge Computing (HotEdge 18), 2018.
[46] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. CoRR, abs/1
0tss/8801.5555 3557777777 2018. URL http://arxiv
htr/abb/188011 05787
[47] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: 101.07780 2018.
[48] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benehmark and analysis platform for natural language understanding. CoRR, abs/1804.07461, 2018. http:/// wwtp:/o///aaaa///eeeee 07461
[49] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et Huggingface's 's transform- ers: State-of-the-arr natural language processing. arXiv preprint arXiv: 191000377 2019.
[50] Mengjie Zhao, Tao Lin, Martin Jaggi, and Hinrich Schütze. Masking as efficient alternative to finetuning for pretrained language models. CoRR, abs/2004. 12406, 2020. URL https
7/arxiv. org/ org/abs//ae abs/2004 2004. 12406
13


Reviews:
[{"review_id": "zi7jCGIALOJ", "reviewer": "Reviewer_Z6Qg", "summary": "This paper attempts to discover a fixed sparse subnetwork before the standard training. Only training the masked subnetwork can sufficiently match or even exceed the performance of the dense counterpart. The performance achieved by the proposed method is promising.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The novelty of this paper is limited. \n\n(1) The pruning technique that is used in this paper, Fisher approximations, is already widely used in the pruning literature [1][2]. It is not surprising to see that fisher pruning performs well before training. On the other hand, there is recently an increased number of works on pruning before training, e.g., SNIP [3], GrsSP[4], Synflow [5]. This paper is likely to belong to this field. However, none of these papers is cited and compared properly in the submission.  Fairly, directly applying fisher pruning to the pruning before training field is still valuable if it can outperform or match the above-mentioned methods with higher sparsity. I would highly encourage the authors to compare with them and report the results. Either positive or negative results are valuable for the community.\n\n(2) As the proposed methods are doing sparse training (dense-to-sparse training), an important class of sparse training methods is ignored, that is, dynamic sparse training (DST). Instead of keeping the mask fixed,  DST allows the mask to dynamically change during training, improving the expressibility of the model gradually. Thus, a comparison with the SOTA DST methods such as RigL [6], ITOP [7] is suggested.\n\nQuality:\n\nThe paper is easy to understand. However, these are some big mistakes that demonstrate that the authors have limited knowledge of the research field of pruning. As far as I know, the common definition of sparsity refers to the ratio of the pruned/masked parameters to the total number of parameters. The usage of sparsity in this paper is exactly the opposite to it, e.g., line 228. Moreover, there are some typos here and there, e.g., line 118; the color in figure 1 is difficult to discriminate for readers.\n\nAlthough I like the fisher pruning idea, the carefulness in the current version degrades the quality of the paper significantly. \n\n\nReference\n[1] Theis, Lucas, et al. \"Faster gaze prediction with dense networks and fisher pruning.\" arXiv preprint arXiv:1801.05787 (2018).\n[2] Liu, Liyang, et al. \"Group Fisher Pruning for Practical Network Compression.\" International Conference on Machine Learning. PMLR, 2021.\n[3] Lee, Namhoon, Thalaiyasingam Ajanthan, and Philip HS Torr. \"Snip: Single-shot network pruning based on connection sensitivity.\" arXiv preprint arXiv:1810.02340 (2018).\n[4] Wang, Chaoqi, Guodong Zhang, and Roger Grosse. \"Picking winning tickets before training by preserving gradient flow.\" arXiv preprint arXiv:2002.07376 (2020).\n[5] Tanaka, Hidenori, et al. \"Pruning neural networks without any data by iteratively conserving synaptic flow.\" arXiv preprint arXiv:2006.05467 (2020).\n[6] Evci, Utku, et al. \"Rigging the lottery: Making all tickets winners.\" International Conference on Machine Learning. PMLR, 2020.\n[7] Liu, Shiwei, et al. \"Do we actually need dense over-parameterization? in-time over-parameterization in sparse training.\" International Conference on Machine Learning. PMLR, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are some limitations in the paper that needs to be addressed in the rebuttal. I would consider increasing my score if they are addresses.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "uaYtcRlPsiR", "reviewer": "Reviewer_citk", "summary": "In this paper, the authors propose a scheme to select and update only part of the weights of a neural network during training. The selection of the weights to update is done using an approximation of the Fisher information, and the mask is computed by keeping only the weights associated with the larger fisher information values. The authors then propose an extensive empirical evaluation of their method for transfer learning, distributed training and efficient checkpointing. Finally, the authors perform an ablation study to show the effect of their design choices.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Review Summary\n--------------\n\nAs a disclaimer, I am not an expert for neural networks training and in particular, I might have missed part of the newest literature.\nThe contribution of the paper, while not very surprising, is well explained and evaluated. Some improvement could be made with the limitation (**L`1**) and the discussion of the results (**M1**) but I feel that this paper is worth accepting to NeurIPS.\n\nMajor comments and questions\n----------------------------\n\nThe paper is clear and the main idea is well explained. The numerical experiments seems convincing, in particular for the transfer learning ones. I have a few comments and clarifications I would like to ask:\n\n- **M1:** In the results comments for distributed learning and efficient checkpointing, while the performance drop by 0.5 to 1%, the authors state that the performance are comparable. To make this statement clearer, I think repeating the experiment and adding standard deviation would improve the results. Else, I would state that there is a small performance loss here.\n\n- **M2:** I feel like this experiments are a bit unfair for the proposed method, in particular with distributed training and small checkpoint experiments. Indeed, the idea of training only part of the weights for transfer learning makes sense but for a randomly initialized network, training for a few tens of epochs before switching to training only the most important weights (as decided with the FISH mask) would make sens. It seems that the authors implemented such feature with the `warmup_epochs` parameter but it is unused in the script they provided. Could the authors comments on this?\n\n- **M3:** A nice addition to Figure.2 (left) would be to add the performance of the random mask for different sparsity levels too. This way, one would see the value of selecting the coordinates with the proposed methodology.\n\n\nMinor comments, nitpicks and typos\n----------------------------------\n\n- l.118: Broken hyperlink.\n\n- References: It seems that this line of work is somewhat linked to block coordinate descent algorithms and active set ideas in classical convex optimization. The core idea in this two line of work is to only update part of the parameters at once. In particular, the idea of using the Hessian information -- which is linked to the fisher matrix -- to select and split the coordinate between workers in distributed optimization has been proposed in the context of sparse optimization [A]. I feel that highlighting the connection between the two fields would probably be beneficial to draw attention to this line of work from both side.\n\nExtra references\n----------------\n\n[A]: Scherrer, C., Tewari, A., Halappanavar, M. & Haglin, D. J. [Feature Clustering for Accelerating Parallel Coordinate Descent](https://proceedings.neurips.cc/paper/2012/hash/4e732ced3463d06de0ca9a15b6153677-Abstract.html). in Advances in Neural Information Processing Systems (NeurIPS) 28–36 (2012).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n- **L1:** One major limitation I see for the proposed method is that it does not seem to apply to regression. The full paper seems to have been written with classification task in mind, where the logits are amendable to the score function and all experiments are performed for classification tasks. Could the authors comments on the applicability of their method to the regression setting? Indeed, in this case, the score function $p(y | x$) is not directly accessible. If the method cannot be applied in this setting, this limitation should be made clear in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6hours"}, "extra_text": {}}, {"review_id": "5Jg-GS14TcN", "reviewer": "Reviewer_bsUF", "summary": "The paper describes a method based on Fisher-information to estimate the importance of different network parameters. Once the top-k parameters in terms of importance are identified, the authors only update these parameters during training. The authors demonstrated the usefulness of their approach in transfer learning experiments (fine-tune only the task-important parameters), distributed/federated learning experiments (to reduce the communication volume through communicating only the sparse updates), and model checkpointing (by updating only a subset of parameters each epoch, and storing the deltas instead of the full parameter tensors at each epoch). ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written. I find the idea simple and practically relevant. The experimental results are convincing. Novelty is rather limited, however, as estimating parameter importance using Fisher information (the squared norm of the parameter gradient) has been explored before (see [1] for example). \n\nMinor comments:\n- L118 : Missing reference\n- For clarity, I recommend the author use the term ‘mask sparsity’ instead of ‘sparsity’. The latter term is typically used to refer to weight sparsity (i.e, the percentage of weights that are zero), and refers to a body of literature (sparse training, pruning, etc..) that is different from this work.\n\nOverall, I think the paper is borderline as it does not offer any new insights or novel methodologies. However, the practical application of the simple idea presented (pre-compute an update mask based on Fisher information to only update the important parameters) could prove generally useful.\n\n[1]Kirkpatrick, James, et al. \"Overcoming catastrophic forgetting in neural networks.\" Proceedings of the national academy of sciences 114.13 (2017): 3521-3526.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2esTEHzojKS", "reviewer": "Reviewer_wqY5", "summary": "This work introduces FISH - a method of training (or fine-tining) a neural network by updating only a small fraction of its parameters. FISH uses an approximation of the Fisher Information of each of the network’s parameters to determine the top $k$ ones to update. The method is evaluated via transfer learning and distributed learning tasks using GLUE and CIFAR-10 datasets. The experimental results show that FISH has competitive or better performance in comparison to existing methods by updating just a small fraction of a networks’ parameters at a time. Further, FISH can reduce the memory usage when saving checkpoints.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The idea of using an approximation to the Fisher Information of each parameter for selecting the parameters that have the largest impact on the output is well motivated.\n- The proposed model with FISH mask outperforms other benchmarks while having a lower level of sparsity, including a random mask (and with the exception of Bit-Fit in Table 1 which is harder to compare to since its sparsity is much lower).\n- To the best of my knowledge, the paper provides a comprehensive literature review and background. \nThe paper is organized and written clearly.\n\n### Room for improvement and questions\n1. What is the time/complexity of computing the (approximate) Fisher Information? The text mentions that it is computationally efficient but does not elaborate further. \n2. The work would benefit from reporting error bars/running experiments with several random seeds. \n3. Have the authors explored how the frequency of mask computation affects the models’ performance (other than using a fixed mask throughout training and updating it at every epoch)?\n4. Are the FISH models in Fig 2 trained with a fixed mask or with a mask which is updated at every epoch?\n5. Is it feasible to use the FISH mask in the case of unsupervised learning? \n\n### Nitpicks\n- Line 92: “over N samples”\n- Line 94: $F_{\\theta}$ should be $\\hat{F}_{\\theta}$ \n- Line 118: missing reference\n- Line 177: “of which” should be removed\n- Line 213: “server. Results”\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not explicitly. The limitations could include computational overhead when computing the Fisher information for some particular datasets for which its approximation is not sufficient. Potential negative impact on society can occur if the underlying model that is trained using FISH is biased or has unethical applications.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies an approach to pre-selecting a subset of parameters of a neural network to update during training. The idea is to look at (an empirical approximation to) the Fisher information of each parameter, and choose the parameters with the k-largest Fisher information values. Only these parameters are updated during training. The paper evaluates this idea for transfer learning (updating only a subset of the parameters for new tasks), sparse training from scratch, distributed training, and efficient checkpointing (where we save a sequence of sparse updates to the model). The proposed method achieves performance similar to dense training, with fewer updated parameters (between 0.5% and 2% depending on the task), and some performance improvements compared to several previous sparse mask generation techniques. 

The reviewers all appreciated the paper’s simplicity, clarity and reasonably thorough experimental validation. Questions raised during the review included the novelty of the work (given that the Fisher information has been widely used as a measure of parameter importance, including for neural network pruning), and the relationship to pruning / sparsification methods. The authors’ response points out that although the use of Fisher information is not a novelty of this paper, the setting is different: this paper selects a subset of parameters to update, rather than selecting a subset of parameters to retain, which makes it applicable to transfer learning in dense models. 

After several rounds of discussion, the reviewers converged to a recommendation to accept. While there are some limitations to the novelty of the submission, the proposed approach is practical, clearly described and well-substantiated. 
Decision: Accept (Poster)