In defense of parameter sharing for model-compression
Authors: Aditya Desai
Anshumali Shrivastava
Venue: ICLR 2024 poster (2024)
Keywords: parameter sharing
model compression
pruning
Primary area: general machine learning (i.e., none of the above)
TL;DR: in terms of memory-accuracy tradeoff parameter sharing is a better compression technique than pruning for model compression at start

Abstract: When considering a model architecture, there are several ways to reduce its memory footprint. Historically, popular approaches included selecting smaller architectures and creating sparse networks through pruning. More recently, randomized parameter-sharing (RPS) methods have gained traction for model compression at
start of training. In this paper, we comprehensively assess the trade-off between
memory and accuracy across RPS, pruning techniques, and building smaller models. Our findings demonstrate that RPS, which is both data and model-agnostic, consistently outperforms smaller models and all moderately informed pruning strategies, such as MAG, SNIP, SYNFLOW, and GRASP, across the entire compression range. This advantage becomes particularly pronounced in higher compression scenarios. Notably, even when compared to highly informed pruning techniques like Lottery Ticket Rewinding (LTR), RPS exhibits superior performance in high compression settings. This points out inherent capacity advantage that RPS enjoys over sparse models. Theoretically, we establish RPS as a superior
technique in terms of memory-efficient representation when compared to pruning
for linear models. This paper argues in favor of paradigm shift towards RPS based
models. During our rigorous evaluation of RPS, we identified issues in the state-
of-the-art RPS technique ROAST, specifically regarding stability (ROAST’s sensitivity to initialization hyperparameters, often leading to divergence) and Pareto-continuity (ROAST’s inability to recover the accuracy of the original model at zero
compression). We provably address both of these issues. We refer to the modified
RPS, which incorporates our improvements, as STABLE-RPS

Published as a conference paper at ICLR 2024
IN DEFENSE OF PARAMETER SHARING FOR MODEL- COMPRESSION
Aditya Desai
Anshumali Shrivastava
Department of Computer Science Rice University
Department of Computer Science
Rice University & ThirdAI Corp.
Houston, TX 77054
Houston, TX 77054
apd100css rice.ice...... edu
anshumalien csc.. rice.
ABSTRACT
When considering a model architecture, there are several ways to reduce its mem- ory footprint. Historically, popular approaches included selecting smaller archi- tectures and creating sparse networks through pruning. More recently, randomized parameter-sharing (RPS) methods have gained traction for model compression at start of training. In this paper, comprehensively assess the trade-off between memory and accuracy across RPS, pruning techniques, and building smaller mod- els. Our findings demonstrate that RPS, which is both data and model-agnostic, consistently outperforms smaller models and all moderately informed pruning strategies, such MAG, SNIP, SYNFLOW, and GRASP, across the entire com- pression range. This advantage becomes particularly pronounced in higher com- pression scenarios. Notably, even when compared to highly informed pruning techniques like Lottery Ticket Rewinding (LTR), RPS exhibits superior perfor- mance high compression settings. This points out inherent capacity advantage that RPS enjoys over sparse models. Theoretically, we establish RPS as a superior technique terms of memory- efficient representation when compared to pruning for linear models. This paper argues in favor of paradigm shift towards RPS based models. During our rigorous evaluation of RPS, we identified issues in the state- of-the-art RPS technique ROAST, specifically regarding stability (ROAST's sensi- tivity to initialization hyperparameters, often leading to divergence) and Pareto- continuity (ROAST's inability to recover the accuracy of the original model at zero compression). We provably address both of these issues. We refer to the modified RPS, which incorporates our improvements, as STABLE-RPS.
INTRODUCTION
An essential question in model design is that given a memory budget for parameters, what is the best architecture one can create for a given task. The solution would require us to consider the model's capacity,
and generalization capability. Limited understanding of these aspects makes the problem hard. Traditionally, model architectures are designed with task- domain expertise, often augmented with neural architectural search (Pham al. (018)). However, recently, it was shown by Lottery ticket hypothesis (Frankle & Carbin] (LTH) and follow-up works (Frankle et al. 2020b ab that given model architecture, the model architecture itself can be sparsified to obtain lighter models. If done intelligently, this sparsification can often maintain the accuracy to certain levels of sparsity. This has drawn interest to another exciting question given memory budget and an architecture. what is the best model we can derive from the architecture. While pruning was traditionally applied as post-processing operation (Mozer & Smolensky 1988) LeCun et al.) 1990 Hassibi & Stork) 1992 Han et al., 2015). LTT and follow-up works show that global pruning is an excellent technique to reduce the memory requirement of the model at the start of training. Since then, many works to reduce the memory footprint of the model at the beginning of training have focused on pruning as a critical technique of the algorithm (Lee et al. 2018 Tanaka et al.) 2020; Zhang et al. 2022 Evci et al. 2020; Alizadeh al. 2022). In this paper, raise the question "Is pruning the optimal method to reduce model parameters?" where optimality is measured in terms of model-memory and quality trade-off.

---

Published as a conference paper at ICLR 2024
We suspect that pruning parameters to reduce its memory footprint is a harsh operation affecting the model capacity adversely. A simple illustration highlights this issue. Consider pruning a n x embedding table. If we compress more than d' using pruning, we start getting degenerate zero emheddings. The same problem is present in pruning models with other components such as linear or convolutions. As we prune more and more parameters, we start dropping nodes and filters, which would affect model capacity. In the case of embedding tables, recent results from parameter sharing based compression(Chen
al. 2015 Desai et al., 2023; 2022) seems promising where 100GB sized embeddings were reduced to 10MB without loss of quality (Desai & Shrivastava (022). In this paper, we evaluate parameter sharing as an alternative to reduce the memory footprint of the model. In our embedding table example, randomized parameter sharing (RPS) is guaranteed to give non-degenerate embeddings no matter how small the memory and unique embeddings with high probability. In the paper, we theoretically show that parameter sharing has better memory capacity (thus accuracy) tradeoff in arguably simple setting of linear models. We also rigorously evaluate memory-accurace tradcoff's of RPS, which is model, initialization, and data agnostic, against various uninformed, moderately informed, and highly informed pruning strategies.
We find that RPS not only outperforms RAND but also consistently outperforms moderately in- formed pruning methods such as MAG, SNIP(Lee et al.] 2018), GRASP(Zhang et al.] (022) and SYNFLOOW (Tanaka et al., 2020), especially in high compression regimes (it is competitive in low- compression region to best out of all pruning methods). While RPS is inferior to the highly informed and computationally expensive pruning technique of Lottery Ticket Rewinding (LTR) (Frankle al.) 2020a) in low compression region, it surprisingly is significantly better than LTR in high compres- sion region. This highlights RPS's inherent capacity advantage compared to pruning. Due this capacity disadvantage, even highly informed pruning (like LTR) is worse than vanilla uninformed parameter--harinng method. While we restrict our attention to RPS, which is the model, initializa- tion, and data-agnostic method, our results encourage research in informed models to achieve even better memorr-aaccuaacy tradeoff.
In our endeavor to rigorously evaluate RPS, we use the SOTA RPS scheme of ROAST (Desai et al., 2023) and find two significant issues in ROAST (1) stability: The ROAST with its -emoo-y sharing introduces new hyper-paramese This standard deviation determines the different multiplicative factors used for each component. We find it essential to choose standard deviation that keeps the multiplicative factors in check. Any extreme choice, such as a low standard deviation, will blow up multiplicative factors, causing diver- gence, and high standard deviation will suppress multiplicative factors, causing slow learning. We provide a gradient scaling scheme that provably improves the stability of the training and removes its sensitivity to initialization standard deviation. (2) Pareto-continuity ROAST cannot recover the accuracy of the entire model when compression is set to 1x X (no compression). This is a desirable property for any compression method, and pruning achieves this naturally. We propose a different mapping function that maintains the cache-efficient memory coalescing, obtains the optimal number of collisions, and also has the Pareto-continuity property when compression is set to 1x, there is no collision; thus, the model with RPS is equivalent to the entire model. ROAST, along with im- provements in gradient updates and mapping function, is collectively referred to as STABLE- RPS in the paper. We want to stress that without the mapping scheme used in STABLE-RPS. the theoretical results of the superiority of parameter sharing over pruning would not have been possible.
of initialization standard-deviation of the ROAST array.
We make the following contributions,
Identify and elucidate the shortcomings of the existing state-of-the-art parameter-sharing method ROAST about issues related to stability and Pareto-continuity. Introduce a set of enhancements that provably and demonstrably rectify the stability and Pareto- collectively referred to STABLE-RPS. We conduct a comprehensive evaluation of STABLE-RRP in comparison to building small models and pruning at initialization methods such as (1) uninformed pruning. (2) Moderately informed pruning, and (3) highly informed pruning. Our findings establish that uninformed parameter shar- moderately informed pruning strategies at all compression factors and highly informed pruning at higher compression.
continuity issues of ROAST. These enhancements
ing is already competitive/sppetio
We provide theoretical insights to explain why compressed models based on parameter sharing exhibit superior capacity compared to pruned models.
2

---

Published as a conference paper at ICLR 2024
Embedeing lable
Linear
Embedding Jable
ROAST chunks
Flat global leyoul of ROAST chunks
RONST chunks
Random Mapping
1st fold
2nd fold
3rd foid
Random Mapping
IM
ROAST Mapping
STABLE-RPS Mapoing
Figure 1: ROAST and STABLE-RPS mapping functions. While ROAST maps chunks independently. STABLE-RPS conceptually flattens the chunks and builds it into single parameter array, divides it. into partitions of size m, and independently hashes each partition.
BACKGROUND
There are various methods for model memory footprint reduction, such designing smaller models
(landola et al.) 2016 Howard et al. 2017: Prabhu et al. 2018), low-rank component models (Jder-
berg et al.] 2014; Novikov et al.) 2015). parameter sharing based components (Chen et 2015
Desai et al. 2022 2023 202 I Desai & Shrivastava 2022) and sparse network (Spring & Shrivas-
tava 2017] 7 Chen al.] 2020; Mocanu et al. 20188] Bellec et al. 2017). In this work, we focus on
the problem of deriving a model architecture from given full architecture, explicitly focusing pruning and parameter sharing.
2.1 PARAMETER SHARING BASED METHODS
Randomized shanmettr--sarinngeee
methods (Chen et al., 2015] Desai et al. 2022 2023) disen-
tangle the parameters from the model architecture. They maintain an array of parameters, say M, and the weights needed inside the model are derived from this array using two hash functions, / and g. Using hash functions h and g with appropriate domain, derive the weight in the model as, (1) While choice of hash functions h have evolved over past few years. For we use the signed hash function. i.e., g maps each weight to {=1}. As is clear, the size of |.M] can be controlled independently, and thus, we can build models of any size. In the forward pass, the weights are derived the fly and used for computation, whereas in the backward pass, the parameter array M updated. The choice of hash functions is critical for good systemic performance. For instance, hash function ht that independently maps each element will lead many cache misses, causing system slow-down. ROAST(Desai et al. (202)) uses ROBE(Desai et al.) 2022) hashing modified to the general neural network to minimize cache misses. Also, it uses a single parameter array across all model components. We discuss the working of ROAST here.
w M [h(i(ww)) g(id(w))
Consider linear layer inside a model. Let the linear layer be identified by module id i. ROAST divides the weight matrix inside the linear layer into chunks of size By x BN where Bx x BN are the tile sizes used by the underlying matrix multiplication kernels. Let t(w) be the tile to which w belongs. Any given tile inside the matrix can be identified by two integers (zxt), (()). Also, let o(w) denote the row-major offset of weight w the tile l(w). The hash function h is defined as,
hii, w) = h (i, (t(w),, (t((w)) + o(w)
(2)
where h' N + [(m - Bx)] where [k] [O.... k-1). The mapping function essentially hashes the tile ID created with (i, (t)))))) ((((t((w) (w). and then all the weights of the tile can be found y((())))((())))))))))))))))))(
in contiguous chunk at this location. ROAST uses universal hash family for h i.e.
h(x, y. 2) ((Ar By + C2 + D)%P)
(3)
where large prime and A, B, C are randomly drawn integers from [1, P) and D is randomly drawn from [0, P.. The mapping is illustrated in the figure (let).
2.2 PRUNING METHODS
While there are various kinds of pruning techniques - structured and unstructured, terms of accuracy-memory tradeoff, global pruning has seen the best results (Frankle et al.] 2020b). The general recipe for (global) pruning is assigning a score to each parameter and then pruning the pa- rameters with the lowest scores. Pruning has traditionally been applied a post-training model compression technique. However, the Lottery ticket hypothesis (LTH) used iterative magnitude pruning (IMP) to find sparse sub-networks of randomly initialized models that achieve high
3

---

Published as a conference paper at ICLR 2024
Table 1: Sensitivity of ROAST to init-stdev. (compression the sensitivity to init-stdev
original compressed
STABLE-RPS eliminates
Compression
Accuracy
Norm of Solution
init-stdev
0.001
0.01
0.1
0.001
0.01
0.1
10
1.33
ROAST
10.0
86.9
91.5
90.4
79.4
man
639
64
27
40
STABLE-RPS
91.4
91.2
91.3
91.2
91.4
56
56
56
10x
ROAST
10.0
81.9
87.2
86.4
77.3
nan
1.1k
110
29
40
STABLE-RPS
87.3
87.3
87.4
87.0
87.4
53
53
53
53
100x
ROAST
10.0
69.6
75.5
74.2
66.8
nan
2.5k
248
41
43
STABLE-RPS
75.2
75.8
75.2
74.9
75.1
48
47
48
48
48
x000
ROAST
10.0
10.0
53.1
56.0
52.1
nan
pan
549
116
41
STABLE-RPS
52.8
52.3
53.6
54.8
52.1
45
45
45
46
45
CIFAR- 100
(VGG-11) 11)
ROAST
1.0
67.2
67.5
63.6
19.8
8k
659
73
67
75
1.33x
STABLE-RPS
68.1
68.1
68.9
68.2
68.6
115
115
115
115
109
ROAST
38.7
1.0
66.0
62.8
19.9
8k
57
106
68
75
10x
STABLE-RPS
65.8
66.5
66.1
66.0
66.3
117
117
117
117
116
ROAST
1.0
54.5
57.1
54.4
20.2
14k
1741
192
70
75
100x
STABLE-RIS
57.5
57.4
57.3
57.4
58.2
94
94
93
94
93
1000
ROAST
1.0
17.6
25.4
24.7
17.1
nan
5595
675
122
72
STABLE-RPS
23.5
23.4
23.7
23.5
23.6
83
85
84
84
85
accuracy. Since then, pruning for training-cost reduction has been an active area of research. We have seen progress in LTH-based methods such as Lottery Ticket Rewinding (LTR), which, instead of re-initializing the weights to the initial weights, rewinds the weights to a later point in training (Frankle al., 2020a). Also, variety of single-shot pruning strategies have evolved. Most pruning strategies differ in the scoring mechanism. Some standard baselines are RAND: randomly assigns the score and MAG: assigns a score based on the absolute value of the parameter. Single-shot Net- work Pruning based on Connection Sensitivity (SNIP) (Lee et al. 2018) uses B.
o 0| for scoring 700
the parameters whereas Gradient Signal Preservation(GRASTI (Zhang et al. 2022) uses H ac 0. A random sample of training data is used for this scoring. Synapuc flow pruning
(NNLOW)(Tanal)) et al. 2020) uses scoring mechanism purely based on the weights and thus does not use any data. There have been other methods such as Alizzdeh et al. y: EEvi et al.] (2020) and more. In this paper, we use MAG, SNIP, SYNFLOW and GRASP representative of moderately informed pruning and LTR as representative of highly informed pruning.
3 STABILITY OF STABLE-RPS
This section discusses the stability issues with ROAST and how to fix them. We start with empirical evidence of the sensitivity of ROAST with the hyperparameter of the standard deviation of initializa- tion used for ROAST array (henceforth called init-stdev). The Accuracy columns in Table [ show that when we change the init-stdev from 1e-3 to 10, the convergence of ROAST suffers at the two extremes. When the init-stdev is very low, the optimization diverges; when the init-stdev is very high, the optimization slows down. This is consistently observed across different sparsity levels and datasets. When we look at the norms of the solutions obtained during optimization, that with lower init-stdev, the norms explode for most cases, indicating divergence, and with higher init-stdev, the norms are lower, meaning slow learning. As we can see with STABLE-RE learning scheme, which we will describe in later part of the section, we can almost eliminate the sensitivity to the init-stdev and obtain the best results at all levels of init-stdev. Thus STABLE-RPS reduces a hyper-parameter which turns out to be very sensitive towards the convergence of the RPS.
observe
Why is global memory sharing in RPS unstable?: ROAST performs global memory sharing, which means that parameters from the ROAST array are shared across different components. ROAST uses multiplicative scaling factors say Xs) to maintain the relative scales of different components. The Às are dependent deviation of 0.1 and the ROAST array is initialized with init-stdev of 0.01, then the multiplicative factor À for this component is 0.1/0.01 = 10. These scale factors which have to be used due to global memory sharing are the root cause of this extreme sensitivity to init-stdev.
the init-sdev. For instance, if component needs an initialization standard
This can be explained considering the effective change in value of parameters in gradient de- (CDn step. Consider a parameter x shared across & different weights with scaling factor A1y... dg- Then, have the recovered weights 3; = Apr. The effective update of x; in one
4

---

Published as a conference paper at ICLR 2024
(6) GD step can be written follows where 2;
recovered weight at time L and L is the loss function.
UC dx
Oc
(x:)(t+))) (1) Xig(i)
Xig(i) Ex9
(4)
dzy
These Às can have varied values. For instance, the RESNET20 model has scales varying from 5 for init-sdev of 0.0588. This implies that some parameters will see approximately 25x larger effective updates than what they would see without parameter sharing. With larger effective gradients, it is unsurprising that the learning rates that worked for the full model will not work for the compressed model. Intuitively, with larger values of λs, we will be forced to use smaller learning rates. We characterize this "stability region" of learning rates, i.e., the range of learning rate v) that can be used with guaranteed convergence of gradient descent algorithm, in the next theorem.
Theorem 1. (Stability of RPS with global memory sharing) Consider function F, which f Lipschitz smooth, with a parameters. Under RPS setup with RPS array of size m, hashing function and scaling factors À1, X2.... An. The stability range of gradient descent for learning rate goes
from (0, 2, for full model training to (maxy 1 X
(0,
for compressed model
The proof is presented in appendix C Thus, the possible range of learning rates that can be used for compressed models shrinks with increasing As (which depends the choice of init-stdev) and increasing compression.
STABLE-RPS gradient scaling scheme:
We propose the following gradient scaling mechanism to remove the sensitivity to init-stdev and improve convergence. Before the update step, the gradients scaled down using scaler r Rm So the update step becomes,
Venl Po VL
(5)
where o is element-wise multiplication. Ver is used as the gradient in the subsequent optimizer steps. For instance in gradient descent the weights æ are updated as (+(+1) = ,.(t) VenC. We propose two gradient scalers. One directly follows from the above theorem The other is based on maintaining effective update sizes.
We define the theory driven scaler follows,
[b
6
EiMOn= X
The gradient of parameter j from the RPS array is scaled with a factor of l2 norm of all component scaling factors used for recovery from this location j. With this new scaling, the modified stability range of RPS with this scheme essentially remains unchanged.
Theorem (Stability of RPS with gradient scaling) Consider a function F with n parameters, which is Lipschitz smooth with constant Lg. Under RPS setup with RPS array of size m, hashing function h, and scaling factors '1 ...... The stability range of gradient descent for learning rate is maintained at (0, L,) under compression when using the gradient scaler [b]]
The gradient scaler effectively kills the effect of multiplicative factors. However, one can see in the proof presented in that it is pessimistic scaling. We find better way scale below.
Another way of computing scaler is to maintain effective change in value of parameters after one ... ,,,y.. using
gradient descent step. Under the same notation of parameter being shared across æ
scalers \1 ... the effective change observed in parameter v; after the update is given in equation We want to scale the change with y such that the effective change of all the recovered weights similar to those with and without parameter sharing.
ac De:
ac
2 weighted average
full
[PPS]
dz.
A
i==1
simple average
We propose
and thus, the effective update scaler for each parameter in RPS is,
i= E Vje (0,...m ...
r
(8)
5

---

Published as a conference paper at ICLR 2024
CIFAR-10 (RESNET20)
CPARTORESGTT)
62-
-0.6
AcCLEIICO chop
STALLE-RS HASHEDMET moople
-14
ROAST messery
-0.
25.01
#hriiguadl #hoomppeseeee]
Spansity
ariginal
Figure 2: Pareto-continuity of STABLE-RPS, ROAST and HashedNet mapping (all mappings used with effective update scaler described in section 3p. STABLE- RRP shows Pareto-coninnitty and improvement over other mappings.
While both scaling mechanisms work well in stabilizing the convergence and make it robust to init- stdev, we find that an effective average-based gradient scaler works better in practice. Throughout the paper, we use this scaling method including Table
4 PARETO-CONTINUTE
OF STABLE-RPS
This section discusses the mappings used RPS and their shortcomings concerning Pareto- continuity. Early papers RPS used element-wise mapping where each weight is independently mapped into the weight repository. Later this idea was extended to mapping chunks to improve cache-efficiency. All these mappings lack one fundamental property that all compression techniques should have, namely pareto-continune should revert one-to-one mapping, causing no-collisions and thus recover exact full model (mod- ulo permuted weight layout in memory). Pruning-based compression trivially has this property. The way randomized hashing is used these methods, there will be some collisions even at full memory. A related issue with these mappings is that they do not have optimal load factor. The load factor is the maximum number of collisions under a hash function. If hash function maps from [n] [m]
as memory budget tends to full model memory, the mapping
where [n] {O...., 1). then optimal load factor is [n/m]] However, for these mappings, the load factor is usually much higher than optimal. A mapping that has optimal load factor will have Pareto-continuity property well.
STABLE-RPS mapping: want to build mapping that (1) has optimal load factor ( and hence, Pareto-continuity) and (2) maintains the cache-efficiency of ROAST. An illustration of STABLE-RPS mapping is shown the figure [ Consider ROAST mapping. It divides the model into different- shaped chunks. We flatten out the chunks into single array of weights. The location of each weight, say i, in this array is called its global index (()). We perform what we call RANDOMFOLD mapping on the global index.
RANDOMFOLD hash function h2 [n]] + [m]] where [n] = fO. n 1} which uses a random hash function is defined as,
hx) = (u((æ/m)) + (x%m)))%m
The RANDOMFOLD mapping divides the range [n]] into partitions of size m and then applies a random hash function each partition number. The mapping of partition, then, is circular and wraps around the m (see figure
T.. The STABLE-RPS mapping is then defined as
STABLE-RPS( = h (G)))
(10)
We summarize the properties of STABLE- RPS in the theorem below,
Theorem The STABLE-RPP hash function has (1) optimal load property and, thus, Pareto- continuity (2) Under the assumption that ROAST chunks are smaller than memory budget, the number of cache-line fetches under STABLE-RPS for each ROAST chunk C(STABLE-RPS) S C(ROAST) + where C denotes number of cache-line fetches for ROAST chunk.
The proof is presented in appendix D The STABLE-RPS has optimal load factor, which leads to pareto-continune It pays slight cost regarding cache efficiency due to RANDOMFOLD potentially separating ROAST chunks into different partitions.

---

Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-100
IINGEEA
RESNET-20
20
1.00 (DaraTor))
(AEATORM)
RAND
GRASP
VGG-11
SNUP STAGLE E-R7S
40
LTS
SMALL VOUUE
3,501 (ooppresaidee
Seriginal
Pecespoowed
Figure 3: Memory-Accuracy tradcoff of STABLE-RIP RPS against different pruning methods and small models for training. This figure generated using approximately 1300 experiments. All of this tradeoff- will be made public so that researchers can use it for plotting baselines.
In our empirical evaluation, we try the abovementioned three mappings element-win wise mapping, ROAST mapping, and STABLE-RPS mapping. We use all these mappings with gradient scaling im- provements of STABLE-RPS for stable training. The results are shown in figure As we can see, clement-wise mapping and ROAST mapping do not achieve full model accuracy at full model mem- ory. This is overcome by STABLE RPS mapping. Also, STABLE-RF mapping outperforms the other two mappings higher compression rates well due its optimal load property, which minimizes the maximum collisions per bucket.
PARAMETER SHARING VS. OTHER OPTIONS
5.1 EMPIRICAL STUDY
We first empirically demonstrate the advantage of STABLE-RPS parameter sharing method on vision tasks. We extensively evaluate
tradeoffs on various model reduction techniques. The results presented in this section summarize over 1300+ experiments performed on V100 and Quadro RTX 8000 GPUs. We reestablish some known results on pruning, stress test existing results on higher and unexplored orders of compression, and add more methods to the mix. We compare the following methods.
STABLE-RUE R (uninformed): Model, data, and initialization agnostic parameter sharing method. RAND (uninformed): Model, data, and initialization agnostic pruning technique. In terms of information, this is equal STABLE-RPS.
MAG, SNIP, SYNFLOW, GRASP (Moderately informed). These techniques use data, initialization of model, both. As pointed by Tanaka al. (2020), most of these methods except for GRASP) benefit from iterative pruning instead of one-shot pruning. We use 100 pruning iterations for all of them except GRASP, for which we perform one-shot pruning.
Lottery ticket Rewinding(LTR) (Highly informed) We rewind the weights to 500 iterations for all the experiments in line with the observations made by Frankle et al. (2020a). While there has been much research on finding winning tickets early, we use LTR as representative method for the memory-accuracy tradeoff that achievable using these methods.
SMALL-MODEL (uninformed): To reduce the model size while using standard modules, we reduce the hidden dimensions of the model while keeping the depth same.
In Figure on the x-axis we have compression factor original
compressed A factor 10 means only 10% of parameters remain. We show error bars, which are + standard deviations of accuracy. Most experimental settings (except LTR) are run on 3 or more seeds. The exact experimental details (learning rate schedules, ppper-parameterseett. for pruning, STABLE-RPS and LTR are provided in appendix E We make the following observations,

---

Published as a conference paper at ICLR 2024
STABLE-RPS vs. Others
In a comparison of uninformed choices, STABLE-RPP outperforms RAND at all levels of compres- sion. This shows that under similar information, the capacity of STABLE-RIP RPS is strictly better than RAND pruning. We will revisit this comparison theoretically towards the end of the section. When compared with moderately informed pruning techniques, STA.LuueeSsSTALLLERR.... which is unin- formed, is always superior in higher compression regions and competitive or better in the lower- compression areas. A highly informed pruning method such LTR is also worse in high com- pression regimes when compared to an uninformed STABLE-RPS, The fact that even the most informed pruning cannot beat uninformed parameter sharing high compression highlights the severity of capacity disadtttadsad pruning as a paradigm faces in comparison with STABLE-RPS, SMALL-MODEL, performance is significantly inferior STABLE-RPS
Moderately informed Pruning and LTR
There is no one clear winner in moderately informed pruning methods. This does not contradict results presented by Tanaka et al. (2020) since we implement iterative versions of pruners, which improve over their baseline single-shot performance.
We evaluate I.TR at higher compression, which was not done before. We find that LTR experi- ences layer collapse, making it, at times, worse than moderately informed pruning methods. This confirms the discussion by Tanaka et al. (2020), which shows that LTR balances the weight magni- tudes with expensive training, but this balancing is imperfect. implies that LTR can be improved in high compression regions potentially using ideas from Tanaka et al. (2020).
SMALL-MODEL vs. Others.
SMALL-MODEL generally gives worse tradcoff than most moderately informed pruning and LTR. However, its tradeoff curve is better than RAND. This implies that a bit informed sparse networks are more powerful than constructing SMALL-MODEL with standard modules.
The difference in the capacity of small model and STABLE-RPS significantly large. Also, in this case, STABLE BEAB--SPPSs not data/initialization informed (unlike pruning). shows that there is potential for us to discover new standard modules based on parameter sharing that can be better than standard modules such as Linear and Convolutions.
5.2 PRUNING VS. STABLE- RPS: THEORETICAL VIEW
We now analyze the effectiveness of RAND pruning and STABLE-RPS in a theoretical setting. We first analyze the dimensionality reduction problem (preserving norms and inner products under data compression). In data compression, pruning implies dropping vector components, and STABLE-RPS means sketching vector components under the specific mapping function. We later show that the quality of learning linear models can be reduced to norm preservation under data compression. One important thing to note here is that we get these theoretical results only for STABLE- EPPS mapping, and these results would not have been possible if we were analyzing other previously used mappings.
Data compression: Consider two vectors x, y E R". Let the parameter budget m = n/k. For the sake of simplicity, assume that & is integer. Let â, ÿ be the compressed and y. The estimation of inner products is,
a Σ x10;1(1)
(é, 9) prune = k
(11)
5=1 1 Σ x x - i) [ (,9(1)((h()) =
(Ø. 1) STABLE RPS
(12)
where 1 (i) is an indicator if 7th component is sampled under pruning and 1 (M(j) = i) an indicator if the hash mapping h maps j to and g()) is the sign hash function. The following theorem shows the theoretical advantage STABLE-RPS, for preserving norms. We have a detailed analysis of inner products and norms in appendix A
Theorem 4, Given vectors & E RD under pruning and STABLE--RPS with memory budget m = n//k for k E N, the estimates for both methods unhiased. The variances are as follows,
V somple () prume) =("
Σ
(m - n)
Σ
2
(13)
m(n 1)
8

---

Published as a conference paper at ICLR 2024
(TTABEE)
2 n m
Σ 2223
(14)
Let x, and ys be i.i.d drawn from distribution with N(u 0, o.). Let this data distribution be D. Then we have,
Ep sample ((², (rume) 2 Ep(Vn((i, (0(BBB))))SBBBSS)) with equality being achieved if and only if all a equal.
(15)
Interprefation: clear that the variance of approximation methods depends on what x we are considering. So, we want to understand for what portion of data is pruning or STABLFF-RPS superior. Hence, we take the expectation over data, In the case of a perfectly balanced distribution (which im- plies a radially symmetric distribution), the expectations of data points favor pruning and STABLE REBB.- However, in practice, distribution is hardly symmet- generally power-law; in this situation, STABLE-RPS is strictly better in terms of portions of data-space that it approximates better. The actual advantage will depend on the scale of power-law with higher power-law favoring STABLE-RPS.
equal. So exactly the same proportions
ric. In fact, distributions
Model Compression: Now let us look at pruning and STABLE-RPS in the context of model param- eters for linear models. Consider the learning of linear model for y given signal vector 21 ... Fn- Let the correlation of output variable with each I; be p:- Let the vector = (p1, p2..--nn) We can then analyze the best residual obtained (i.e., learned model residual) once the pruning sample of weights or mapping of STABLE RPS has been decided. The following result reduces the residual obtained under model compression data compression on the vector p.
Theorem 5.
The optimal (least) residual for linear regression problem for target with standard deviation By and input signals (I1, 22, .., 22,,., In) each of which has standard deviation Tg with corre- Coo(y,a,) can be written as,
lations Ps 0.0g
ReScomp (0) o (1-
(16)
where p is the vector of all p.s and (p. is the estimate of norm of p vector under data compression (pruning or STABLE-RPS) and ReScomp is residual under compression.
The residual obtained by the compressed model (both pruning and STABLE-RPS) depends how well the norm of the vector p is preserved under data compression. As have seen in the previous theorem, if the correlations are not balanced (which usually the case in actual data), then STABLE RRREE-SPP is better than pruning average.
Interpretation:
DISCUSSION AND CONCLUSION
Pruning has been one of the most successful tools for model compression post-training the start of training. This paper brings to light the capacity issue with sparse models obtained via pruning, which limits the success of even the most intelligent pruning schemes. On the other hand, parameter sharing is much more powerful. Uninformed parameter sharing is already better than the best pruning scheme high compression and competitive or better than moderately informed pruning at all compression. This paper argues in favor of a paradigm shift toward parameter-sharing based compression. Many directions need exploring fully realize the power parameter sharing. We briefly discuss them here. This paper talks about parameter sharing from the start of training and shows the benefits of the parameter-sharing paradigm. To widely realize the benefits of parameter sharing, we need to be able to develop a post-training parameter-sharing scheme. Another issue with parameter sharing in its current form is that requires the same computation as the original architecture. To make it practical, we must develop implementations leveraging parameter sharing to reduce computational load. Additionally, an interesting question whether we can leverage the power of parameter sharing design some fundamental layer of neural network which is low memory, low computation but high capacity. Finally, the most immediate progress in this research direction can be made if we devise informed parameter sharing strategies.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
This work was supported by National Science Foundation SHF-2211815, BIGDATA-1881177, ONR DURIP Grant, and grants from Adobe, Intel, Total, and VMware
REFERENCES
Monica Agrawal, Marinka Zitnik, and Jure Leskovec. Large-scale analysis of disease pathways in the human interactome. In PACIFIC SYMPOSIUM on BIOCOMPUTING 2018: Proceedings of the Pacific Symposium, pp. 111-122. World Scientific, 2018
Milad Alizadeh, Shyam A Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initializa- tion using meta-gradients. arXiv preprint arXivv2202.08133. 2022.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv:1711.051366 2017.
Beidi Chen, Tharun Medini, James Farwell, Charlie Tai, Anshumali Shrivastava, et al. Slide: In defense of smart algorithms over hardware acceleration for large-scale deep learning systems. Proceedings of Machine Learning and Systems, 2:291-306, 2020.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pp. 2285-2294, PMLR, 2015.
Aditya Desai and Anshumali Shrivastava. The trade-offs of model size in large recom- mendation models 100gh to 10mb criteo-th dlrm model. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 33961-33972. Curran Associates, Inc., 2022. URL ht httpss: http:://ppooeedddnn..eer..pppppe wwtperrrrppsccppppp.eppepep cc fitp:eerrrileeseppe///// File/ 91 2at.a221128822531111355555550000
pdf
Aditya Desai, Yanzhou Pan, Kuangyuan Sun, Li Chou, and Anshumali Shrivastava. Semantically constrained memory allocation (sema) for embedding in efficient recommendation systems. arXiv preprint 11..i:::103.0612 2021.
Aditya Desai, Li Chou, and Anshumali Shrivastava. Random offset block embedding (robe) for compressed embedding tables in deep learning recommendation systems. In D. Marculescu, Y. Chi, and C. Wu (eds.), Proceedings of Machine Learning and Systems, volume 4, pp. 762- 778. 2022. URL
wwwpaiinnssmlll.l....mm mwsys wtt.::/ssssssss.ss.....e.e.e.mee
files/paper/2022/
Paper...e.. .pdf
Aditya Desai, Keren Zhou, and Anshumali Shrivastava. Hardware-aware compression with random operation access specific tile (roast) hashing. In International Conference on Machine Learning, pp. 7732-7749. PMLR, 2023.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803.03600333333 2018
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 3259-3269. PMLR, 13-18 Jul 2020a. URL https: //proceedings.co .ml.. www.:.pressvllll htwr119//rraaee22/0 Frankle20a.....m html
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576 2020b.
10

---

Published as a conference paper at ICLR 2024
Huifeng Guo, Ruiming Tang. Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfin: factorization- machine based neural network for etr prediction. arXiv preprint arXiv:1703.0422 2017.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
2017.
Weihua Hu, Matthias Fey, Marinka Zitikk Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint 0rXiv:2005.006683 2020.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters andj 0.5 mb model size. arXiv preprint 102.:7660, 2016.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprini arXiv: 1xXiv:1405.3866 2014.
Yann LeCun, B Boser, John S Denker, Donnie Henderson, RE Howard, Wayne E Hubbard,
LD Jackel, and DS Touretzky. Advances in neural information processing systems. San Fran- cisco, CA, USA: Morgan Kaufmmann Publishers Inc, pp. 396-404, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity, arXiv preprint arXiv:1810002340, 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):2338,, 2018.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from network via relevance assessment. Advances in neural information processing systems, 1, 1988.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architec- ture search via parameter sharing. pp. 4092-4101, 2018. URL https::/arxiv..nrt org/ /
18C2.03268
Ameya Prabhu, Girish Varma, and Anoop Namboodiri. Deep expander networks: Efficient deep networks from graph theory. In Proceedings of the European Conference Computer Vision (ECCV), pp. 20-35, 2018.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, pp- 1161-1170, 2019.
Ryan Spring and Anshumali Shrivastava. Scalable and sustainable deep learning via randomized hashing. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 445 454, 2017.
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information pro- cessing systems, 33:6377-6389, 2020.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Ben- gio, al. Graph attention networks. stat, 1050(20):0--8555) 2017.
11

---

Published as a conference paper at ICLR 2024
Minjia Zhang, Wenhan Wang, and Yuxiong He. Grasp: Optimizing graph-based nearest neighbor search with subgraph sampling and pruning. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1395 1405, 2022.
12

---

Published as a conference paper at ICLR 2024
A THEORY: DIMENSIONALITY REDUCTION ANALYSIS ON AND NORMS
Consider two vectors x and y in R space. There are two methods to perform dimensoinality reduction
1. Subsample the vector ( making it sparse pruning) The transformation is
Sa
(17)
where S is X sampling matrix composed of only zeros and ones. Also, each row has exactly one element non-zero. Each column has a maximum of one non-zero.
2. Projection ( (randomly combining the different elements ) In this case, we perform the following transformation
RPx
(18)
where P is random permutation matrix and R is a m n matrix. R is the random STABLE- RPS matrix.
Let us look at the inner product preservation both cases.
A.1 PRUNING
the estimator of inner products
(2. = 7
2 1
(19)
1=1
where 11) implies that i is selected.
E( E((,,)) 0))
m m n
W:Yi (x, v)
(20)
2
n²
1
((t, 3))
Σ (1);
(21)
m²
n²
1,0)2,,,1(11166)
(22)
m²
h==1
n²
n
Σ
1(1) +
Σ
,,y,,,,,
(23)
m²
1=1
E((ē, 0)) =
n m
Σ
2
+
n² m(m -
Σ
I.birayi
(24)
m2
m² n(n
i=1
n2
?? 2
+
n mm
(25)
(n
Thus, the variance,
v((,,0)
((m
1 -1
>
??
+
(m - 1) -1
Σ
(26)
(n
13

---

Published as a conference paper at ICLR 2024
( (--()))))) (m--)) n1)
V((a ((- o.
Σ
(27)
A.2 PARAMETER SHARING
the estimator of inner products. Let h be the final position mapping resulting from P and R. The signs from R are represented by the function g
( 1)
Σ (=(())(())))))
Σ M50()1((N() =i)
(28)
where 1 is indicator. The equation can be simplified
771
(0,
Σ
(,0_6))9((01(6))==))))) =1)
(29)
which can be further simplified as,
(2, 1) = Σ x,,0 A())
(30)
1
E E((, (x.
(31)
00 Σ 2.wo=raa((es00sccss((((1111 -bb))(b((() h(d))
(32)
a.b,c,d
Only 4 cases give the non-zero terms in expectations
a b = c= d
b c = d
d
d + =
E((,
Σ 2i
(33)
+
(34)
(1))
(35)
+
Σ
0,y),,,, = A()))
(36)
Thus variance,
v((2,0)) Σ
E(1(0) R()))
(37)
iFj
ran E ((((()) M()))
(38)
Pr(h(1) M(j) Pr(i and in different chunk) Pr( i and from different chunks collide )
(39)
14

---

Published as a conference paper at ICLR 2024
Assming that divides
1 m 1 1
Pr(h(() = h(j)
(40)
71
m
(((200)) m
1 n -
Σ
23y} +
Σ
(41)
A.3 NORMS
For norms (precisely square of norm) we can just use x = y in the previous analysis
-1-
(m (m- am--1 n1) m(n
Σ
V prune ((2,
+
(42)
i=1
2 n Edjeo
VSPABLE-
(43)
m1
A.4 COMPARISON
As is clear from the two expressions, which method of compression better depends on the x and y hand. So in order get sense of which method better on "average", let take some interesting distributions from which these points are drawn.
A.5 AVERAGE CASE OVER DIFFERENT DISTRIBUTIONS (INNER PRODUCTS)
Let us assume that each of these x and y are independently drawn from distribution defined below
D; ~ ~~ (00, of)
(44)
In real scenarios each input may have different strengths or importance ( as we will see in the next section thus, we assume them be distinct and of course, can always get the scenario of equal strengths by setting all equal to single a. Let us look inner products first,
(("==
Ep V prune ( (k,
Σ o
(45)
m
Ep STABLE-
Σojof
(46)
m n 1
($)
We know that,
Σ i
a,jz = Σ 2a,a; S
>
= (n - 1) Σ a3
(47)
{<}
:zj
Thus,
(D SWABLE-RPS((EE... (rs((( 5 n m
(n (n-1)
Σ 0.0 = ED prune (2,
(48)
-
Thus parameter sharing is strictly better than pruning and equality happens only when data is uni- formly distributed. all standard deviations are same). Whenever there is a slight imbalance in signal strengths STABLE-RPS is superior pruning.
15

---

Published as a conference paper at ICLR 2024
A.6 AVERAGE CASE OVER DIFFERENT DISTRIBUTIONS (NORMS)
Under same distribution,
Ep(V SSTABLE-RPS((AA
2 n m
o
(49)
n 1
Ep(V (V prune (2.2)))
n
1
Σ 3
12
m(n - 1)
(m (m-n)) m)
Σ
n2
o2
(50)
m
im1
( ( n2
1)
3
1
Σ oंo0
+
Σ
o 2
m
(m
1)
ifj
ifJ;ú.j=1
(51)
((3 3 (3mm1777 m )) + m))
((-)))))
n)
Σ oं
(52)
(( m
(( n
Σ ドデ
o,
(53)
(V SPABLE-RRS
(54)
Thus,
Ep( (V "ruuee (eruue(((.))) ())) 2 ED( EeoV SPSBLE-
(55)
Thus parameter sharing is strictly better than pruning and equality happens only when data is uni- formly distributed. all standard deviations are same). Whenever there is a slight imbalance in signal strengths STABLE-RU RPS is superior pruning on average.
B THEORY: ANALYSIS OF LINEAR MODELS UNDER PRUNING AND STABLE-RPS
Consider the following features x 22, ... such that, Var(æ;)= = V*x)) = o Vi
and
Cou(E;, =
(56)
Pruning.
Res(ū) = Varly - Σ (()))))
(57)
(es( = V(v) + Σ ((1 2 Σ Ca1(C))aa)
(58)
Minimizing the residual, we get,
C(1,2;)
1(1)a;
(59)
02
Cly x,)2 1()
Res(i)=V(6 Σ
(60)
02
Res(ū) = oz 1- Σ
1())
(61)
0fo?
16

---

Published as a conference paper at ICLR 2024
Let p(t) Co Gy0e
Res(ÿ) og - Σ ((0)) 11_2n(89(1))
(62)
- k Σ (()))((1)) Res(i) - y
(63)
Res(s) og (1
1 priae
(p, P)
(64)
1:
where p is the vector of all correlation coefficients where is the estimation from data compression of
STABLE-RPS
m
Res(ÿ) Var y
E- Σ oj
Σ (1,
(65)
5=1
)) - Σ a,C(y, Σ
Res( - = V V(r) +
agv Σ 1(1.)))zz)
1(i. (lxx))
(66)
Using the same analysis,
æj = Cly, 1, ((. j)g.z.)
(67)
V(Z. ((,,))n)) j)sr.))
ko
Res(O)= = Vyy + "'afkot - Σ a agkozaa
(68)
= Σ (azko²)
(69)
Res(i) oz 1- Σ
(70)
ko
Res(ū) o3 ! ((11E)
Cly,
(71)
O20y
Res(ÿ) =z 1 1- Σ Σ 1 (i. (i..)).p(()
(72)
Res(i) = =o 11
o ( 1..
1
(73)
So the residual is minimized based on how large is the second term.
B. COMPARISON
It is clear from the above two analysis that essential question is about poom-prresevation of the vector of ((2), p(3),..., p(n) under sampling and STABLE-RPS sketching. The residual in case of compression be obtained by just setting ( = 1 for all and hence k
17

---

Published as a conference paper at ICLR 2024
Res( = o (1 (p.
(74)
ReScompl (1) Res
{p.p) - (0,
(75)
Res(ÿ)
(p.
B.2 SUMMARY
We now analyse the effectiveness of RAND pruning and STABLE-RPS theoretical setting. We first analyse the advantage dimensionality reduction problem (preserving norms and inner products under data compression). In context of data compression, pruning implies dropping vector compo- nents and STABLE-RPS implies sketching of vector components under the specific mapping function. We later show that quality of learning of linear models can be reduced to norm preservation under data compression. One important thing note here that we get these theoretical results only for STABLE-RPS mapping and these results would not have been possible if we were analysing ROAST.
Data Compression Consider two vectors 2, y R Let the parameter budget be m n/k. For the sake of simplicity we assume that & is an integer. Let æł, be the compressed x and y. The estimation of inner products is,
(i, 11 prume = k Σ TEy (()
(76)
a
(é, STABLE-RPS
Σ
Σ (3(())(((())) i)
Σ
y (n -
(77)
i) is an indicator if the hash mapping i maps j to and 9()) is the sign hash function. The following theorem shows the theoretical advantage of STABLE-RPS. In order to make equations simpler, assume that the parameters are first permuted before applying the STABLE RPS mapping.
where ((i) is an indicator if 8th component is sampled and 1(h())
Theorem 6. Given vectors I, It and under pruning and STABLE-RPSwith memory budget m = n2/k l for ls E N, the estimates for both methods are unbiased. The variances are as follows,
sample (ft, Q) prune) = ((m -
-
+
(m n) m(n - 1)
Σ ip
(78)
1 m
Va SSSABLEERRS)
2.93
Σ a.y.T,y1
(79)
m n
As can be seen, depending on exact vectors, either method can be better than each other. As both these methods are data agnostic, let us take look at an average case over some data distribution. Let D4 and y be i.i.d drawn from any distribution with μ 0, a o4. Let this data distribution be D. Then we have,
Ep (V sample ((,, 1 pmmee) Z (V,n(( 0) STABLEE-RS) with equality being achieved if and only if all ais equal.
(80)
We can similarly analyse preservation norms and the results are summarized below, Theorem 7. Given vectors E R and under pruning and STABLE-RPSwith memory budget m n/k k for ls E N, the estimates for both methods are unbiased. The variances are as follows,
x + (
(
(m-n)
(m (m)
V sample ((2, 2)) prune) =
-
m(n (mmn-1)
Σ
(81)
1=1
18

---

Published as a conference paper at ICLR 2024
(
2 n m
Σ 22
(82)
m -
1#1
Let x; and yi be i.i.d drawn from distribution with N(p 0, σ.). Let this data distribution be D. Then we have,
ED V sample ((2, 9 prune)
(83)
with equality being achieved if and only if all ass are equal.
Interpretation: As pointed out in theorems, the variance really depends on what æ (or x, y) are we considering. So, we want to understand for what portion of points is pruning or STABLE-RPS superior. Hence, we take the expectation over data. In case of perfectly balanced distribution (which implies radial symmetric distribution), the expectations are equal. So exactly same proportions of data points favor pruning STABLE-RPS. However, in practice distribution is hardly symmetric. In fact distributions are generally power-law, In this situation STABLE-RPS is strictly better in terms of portions of data-space that approximates better. The actual advantage will depend on scale of power-law with higher power-law favoring STABLE-RPS
Model Compression Now let us take a look at pruning and STABLE-RPSin context of model parameters for linear models. Consider the learning of a linear model for y given signal vec- tor 21, Z1...., Fn. Let the correlation of output variable y with each z; be P:. Let the vector = (21.P2,--PPr) We can then analyse the best residual obtained (i.e. learned model residual) once the pruning sample of weights mapping of STABLE-RPS has been decided. The following result reduces the residual to data compression on the vector p.
Theorem 8. The optimal (least) residual for linear regression problem for target y with standard deviation By and input signals (I1, 22, each of which has standard deviation a, with corre- Cov(y.a.)
lations Pt 0,00
can be written as,
Rescomp (g) (0) Res(ū) -p.p) ((.
(84)
Res(ū)
1- (p.P)
where p is the vector of all pi and (p, p) is the estimate of norm of vector data compression (pruning or STABLE-RPS)
Interpretation: As can be seen from the above theorem, the residual obtained by the compressed model (both pruning and STABLF-RPS) depends on how well the norm of the vector is preserved under compression. As we have seen the previous theorem, if the correlations are not balanced (which is usually the case in real data), then STABLE-RIE RPS is better than pruning on average.
C CONVERGENCE OF ROAST AND STABLE-RPS
We consider a simple case of Lipschitz continuous function with gradient descent and look at the stability region for the learning rate.
The update is
8 = -
(85)
Using the lipschitz smoothness, we have
+ 1 [1l x|12
Fly)
(86)
F(041)) S F(0.) (VF())), VF( n2 I IIFF(0))1
(87)
F(0(41) & F(0.) +IV + 2 +
(88)
19

---

Published as a conference paper at ICLR 2024
F(0++1) F(())) - m/ VF(0)) - -
(89)
For stability, 7) < 2/T.
ROAST
Under parameter sharing we consider the following function A is a diagonal matrix with
Aa g()),
G( F(# ARt)
(90)
[VG())) VG(b))
ARO1) VF(02 - AR(b)) S ((AR))((VE() = AHRU) - VF = AR
((AR)),)|(Re) AR
(ARR))T, 1/211
Thus, - =
0² (ARR) = max(eigen-value)(RR AR) = max Σ x21(M(1) = =)
(91)
Thus the stability range is (0, Immx,, En ((())())
Now let us analyse the gradient scaling algorithm. Again, Consider update of the gradient descent algorithm, we go from th to 1/--1 in compressed space which corresponds to 0, ARt, Where A and R are defined above. Also, the gradient scaler is written as diagonal m x m matrix I' (this is a different notation than used in main text where was a vector in Rm .)
STABLE-RPS
We have the update equation.
Di = t'e 1 (IIVG(())
(92)
Thus,
But1 = 1 17 (ARTR AT) VF((.)
(93)
As F is a L-smooth function, we have,
5 (FF(0.). -B2) + 1 110141 - -ell2
(94)
F(0(41) < F(0)-))F(())) (ARTR AT) VF(O)+ L VI(OO) (ARTRTAT) (ARTRTAT) VF(0.)
2
(95)
T F(0++)) & F(O.) - -(VF()) (AARTRATT) 2 (ARTRTAT) (ARTRTAT) VF(6.)
(96)
Consider the matrix,
M (ARRRTAA) nL (ARCRTAT)" (ARERTAT)
(97)
The matrix (ARER AT) i diagonal matrix with (ARTRTAT); = X7rne) Thus matrix (ATTRTAT)) (ARTRTAT) AT) is also diagonal matrix with ((ARTRTAT) AARRRTAAT+ (ARTRTAT))
X Thus diagonal element of the matrix above equation is
nL X2 XPrm())
M = X7mM()
(98)
2
20

---

Published as a conference paper at ICLR 2024
If 7/ Eaneoey X and if L, then,
14, M4 > 0
(99)
as all A > 0 Since M is diagonal matrix, M is a positive definite and thus F(0++1) < F(,)) and stable. Note that the y chosen here is pessimistic (assumes that other X can be arbitrarily close to 0 and thus performs aggressive scaling. In practice we find that effective update scaler performs well.
D PARETO CONTINUITY THEORY
The load factor of each bucket is essentially the number of partitions the original flattened out vector divided into since each partition adds exactly one element each bucket. Thus the load for each bucket is
((t)
S
(100)
Which is optimal. When n = m, the number of collisions is and thus RPS model and full model equivalent (except for memory layout of the weights).
The cache fetches can increase if a roast chunk is separated while partitioning. The maximum number of separations is first it is separated while partitioning and then each of the chunk goes through wrap around (which breaks it into two) thus 4. This means maximum additional of 3 cache line fetches that are potentially partially filled.
E EXPERIMENTAL DETAILS
Learning Rate Schedules We choose the learning rate schedule given by Tanaka et al. (2020) for the milestone steps. We find the the learning rate given in Tanaka et al. (2020) is not the best learning rate even if we use there code base as starting point of our implementation. So we run the baseline full models with varying learning rate from {0.001, 0.01, 0.1, 1, 10} with drop-rate of 0.1 and schedules that are mentioned in table We find that 0.1 works better for all the full model trainings.
We use the same learning rate schedules and other common hyperparmeters for STABLE E-RPS and pruning methods.
Table 2: hyperparameters for both models RESNET20 and VGG11
Hyperparameter
CIFAR-10
CIFAR-100
TINY-IMAEENEI
Base Learning Rate
0.1
0.1
0.1
Milestones
80,120
60,120
30,60,80
Learning rafe drop
0.1
0.1
0.1
batch size
128
128
128
total epochs
160
160
100
STABLE-RPS hyperparameters The standard deviation for the RPS array is set to 0.01 for RESNET20 model and 0.05 for VGGI1 model. The choice is made following the rule of thumb that scale factors should be in check. However, with gradient scaling, is required be very strict about these parameters
Pruning iterations We use 100 iterations for snip and synflow which has been shown to improve its performance. For mag and rand, the number of iterations don't 't really matter. For GrasP, tried 100 pruning iterations but that turns out to be bad for the pruning. thus choose to do only iteration (single shot) for grasp is originally done.
21

---

Published as a conference paper at ICLR 2024
F THEOREM VALIDATION.
F.1 VALIDATION OF THEOREM 1 AND THEOREM 2
The theorem and theorem talk about the theoretical stability region of the SGD algorithm for L- smooth functions under parameter sharing. We validate the theorem with the following experiment.
Consider parameter sharing setup for the function f(0) (0, 0). We know that L for this function We run simulations with |0] 128 (recovered full parameter) and m = | 32 (actual parameter repository). The X's for recovery are drawn from uniform distribution [0, Amar) we
vary Amaz from 0.5 to 2.0. We vary learning rate increments of and see what the max learning rate where we get convergence.
empincal scaled (theorem 2)
1.4
theory-scaled (theorem 2)
empirical unscaled theory unscaled (theorem 1)
1.2
Max Learning rate
1.0
10
*X* xK
*
0.8
(-1,(()
Figure 4: Reduction in stability region due to parameter sharing. (validation of theorem and theorem 2)
The plot shows our empirically computed values of learning rate and theoretically computed values suggested by theorems 1,2. As we can see, the two values are very close to each other validating the stablity region analysis for both unscaled and scaled parameter sharing.
F.2 VALIDATION OF THEOREM 4
Theorem 4 talks about the inner product preservation under data compression and says that the expected (over data) variance of compression for STABLE-RPS is strictly lower than that for prun- ing with equality obtained when the data is balanced all dimensions have distribution with same Handard-deviationn In this experiment we consider the following distribution. Given a power-law- coefficient say c, the distribution X, Y is defined as
X ~ NO0
if c = 0, then all of them have standard deviation and is the balanced case. The plot shows the expected (over data) variance (over compression) inner product preservation under the two methods. In this plot the compressed dimension compression. The plot validates the fact that indeed variance of STABLE-RPI is superior with c and for = 0, it is the same.
32, full dimension is 128 so it is 4x
22

---

Published as a conference paper at ICLR 2024
pruning stable-rps
25
2ero
20
Eo(V((z v))
5-
0.00
0.25
0.30
0.75
1.00
1.25
1.50
1.75
power-law.coeflcleet
Figure 5: The data v, y is drawn from the following distribution. Given power-la--coeffcient e, the distribution of x. y X is defined as [[i] NO (validation of theorem 4)
emprical(k=2)
theoreticallk=z)
emprical(k=4)
200
theoreticallk-4)
emprical(k=g)
Residual Value
theoreticalkk-8)
emprical(k-16)
theoreticalkk=16)
0.0
0.2
0.4
0.6
0.8
power-law.cofffcceet
Figure 6: Data is created follows, y Dim1 OGT; + Oi i x NO (.,0,.22
is the power-law coefficient. This plot is created for pruning. k is the compression rate k n/m. (validation of theorem 5)
F.3 VALIDATION OF THEOREM 5
The theorem represents residual of linear regression in terms of data-compression of correlation vectors. We validate the expression,
Res(ū) 03 -1-1))
(101)
The data is created follows, the data has dimension 256. The all dimensions are sampled from (0 1). The target is created by following,
Σ
ar,; +e
(102)
where e ~ A/ 0.2) and a; = 1- where is the power law coefficient. We plot residual under compression for different compression rates k m where is the size of compressed model. As
23

---

Published as a conference paper at ICLR 2024
shown under different power-law of coefficients (and hence correlations), that our theoretical prediction matches empirical observation.
G EFFECT OF PARETO MAPPING ON LATENCY
The pareto mapping does not follow strict memory alignment of ROAST. However, the theorem states that the effect cache-miss is not much. We validate this in the following experiment. We consider a model with single linear layer of the size specified in the table. We measure the forward and backward time in ms for this model. This shows that while there go from ROAST to STABLE-RPS, but the impact is minimal. Also, it is much better than HashedNet mapping in which cache- inefh--ieffcciency is severe in large matrices.
impact on latency when
FWD BWD (ms) of single linear layer under difffrent mappings
Model
mairix size:
128x128
256X256
512x512
1024<1024
2048
4096x409%
10Z40K10240
20480K204880
Pytorch
parameter
5.53
4MB
0.82
0.93
0.92
0.91
1.15
2.62
10.05
38.81
32MB
19.53
75.41
HNet
128MI5
64MB
2.24
1.56
2.29
1.55
2.32
1,58
2.55
2.83
3.74
9.21
7.31
47.13
38.23
147.58
183.79
256MB
3.55
3.53
3.74
3.97
5.47
11.34
56.16
215.47
4MI5
1.07
0.93
0.92
0.92
1.08
1.79
6.99
26.45
32MB
1.23
1.22
1.24
1.41
2.17
7.83
30.36
ROAST
128MB
64MB
2.19
2.15
2.23
2.21
2.41
1.78
2.59
3.27
9.40
31.42
30.01
255MB
3.61
3.52
3.53
3.59
3,89
4.78
11.19
34.65
312NIB
7.90
14.31
37.77
4MB
0.92
0.93
0.92
0.92
1.12
2.27
7.86
29.92
STABLE-RPS
128MB
6AMB
2.16
1.56
2.24
1.57
2.20
1.56
2.20
1.59
2.57
1.82
3.71
3.08
10.40
9.79
34,91
35.58
256MB
12.09
312MB
6.72
7.02
15.16
39.93
H MORE DOMAINS
We ran experiments on PPI dataset (Agrawal et al. 2018 and OGB-ARXIV dataset (Hu al. 2020 with Graph Attention Network (Velickovic et al. 2017) Model. We see similar observations. We the implementations provided by the DGL library. Low rank is known to be worse than global mask pruning. We add it here to show comparison.
Proteln-protehh
Interaciion
mag
graup
antoww
STAELERSS
23
1.4M (compressible)
3K (bissmnorm)
bweakk
Acoursa
04-
neg grass Syeflow
(oompresaibble)
3.5k (blas/norm)
Figure 7: Comparison on graph domain. Low-rank methods cannot obtain more compression than shown in the graph.
Another domain we experiment with is the deep learning recommendation models on click through task. We experiment with Criteo-Kaggle dataset with two models DEEPFM (Guo et al.) 2017) and AUTOINT (Song et al.) 2019). As most the parameters in these models come from the embedding tables, only compress those. The effect of pruning particularly bad in embedding
24

---

Published as a conference paper at ICLR 2024
CEEPFM
AUTOINT
ROC-AUG
ROCIAUC
0.76
574M parameters
0.76
574M
STABLE (PS]pep
STABLE RP5 topoch)
STABLE S'ABLE--PPS)Dppp Deppe))
STABLE 41% Jupoch)
STARLF-BPS
mpech)
374
endi2eppoch
2epochh
0.72
Compression
Compresston
Figure 8: Comparison of pruming RPS on Deep learning recommendation models. Full Model achieves best validation AUC epoch. The dataset has 40M samples
table compression. We show results of STABLE RPS, RAND and SYNFLOW which for embedding layer is same as MAG pruning. We also show results at 1, 2 and 3 epochs as they improve with more epochs.
25


Reviews:
[{"review_id": "7t4MXGoVyu", "reviewer": "Reviewer_YEB1", "summary": "Stable-RPS is an extension of ROAST, which is a method for sharing parameters using hashing. Parameters are mapped from a shared parameter array into each layer, along with a sign and scaling factor. The paper addresses the stability of ROAST and ensuring that the original performance of the model is maintained at 1x compression (Pareto-continuity). Modifications:\n\n1. A gradient scaling function depending on the scaling factors applied when ROAST maps parameters to layers from the global store\n2. A better hash function that ensures Pareto-continuity\n\nThe authors then demonstrate that this method of parameter sharing is competitive with contemporary pruning methods.\nThe authors also include various theoretical results to explain this improved performance.", "questions": "Why is there no discussion of the main limitation of random parameter sharing methods versus pruning methods: that they do not reduce the numpy of floating point operations required, while pruning methods do? If pruning methods and RPS perform similarly up to compression factors of 100x then pruning has a significant edge in saving FLOPs. When would RPS really be competitive?\n\nAre there any experimental results on networks other than ResNet-20 or VGG-11? For the same parameter budget there are now many network architectures that perform much better. In other words, where would MobileNet or EfficientNet be placed on Figure 3? Unfortunately, as the compression ratio increases the network quickly enters regions where the accuracy is not worthwhile.\n\nExperimental results are demonstrated on CIFAR-10, CIFAR-100 and Tiny-ImageNet. Would it be possible to explore this method on a contemporary large scale model? What would a scaling law for random parameter sharing look like?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper explores missing potential for random parameter sharing in deep neural networks. The early results, such as [Chen et al 2015][hashing], demonstrated competitive performance at the time but the method has received less attention since. It would be valuable for the field to have a comprehensive exploration of the potential of this direction of research.\n\nThe improvements to ROAST presented are well motivated and address clear shortcomings of an existing method. They are a good contribution to the field.\n\nThe theory investigating random parameter sharing presented in 5 theorems is a useful insight into how random parameter sharing works and will be useful to future work in this area. It seems likely that future research may focus on other alternative hashing methods and will benefit by performing similar analysis.\n\n[hashing]: https://arxiv.org/abs/1504.04788", "weaknesses": "Figure 1 fails to explain how ROAST works or how STABLE-RPS relates to ROAST. I don't know what a ROAST array is or what a ROAST++ array is. I don't understand why the resulting Linear array has exactly the same elements (same colors) in both cases.\n\nEquations for ROBE-Z and ROAST are insufficient. I have no way of replicating either method from these descriptions. It looks like integer and modulo division are being used but outside of pseudocode these should be defined with explanation of what they're doing in the equation.\n\nThere are 5 theorems stating results, but these results are not checked by experiment. Experimental verification would free the reader from checking the derivation or trusting that it is correct.\n\nParameter counts for the networks in Figure 3 would be useful, to understand how many parameters the network has at different compression levels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3GThaugz7", "reviewer": "Reviewer_Fae8", "summary": "The authors provide theoretical analysis, algorithmic refinement, and empirical experimentation of randomized parameter sharing (RPS) for model compression, in contrast with the prominent paradigm of parameter pruning for model compression. Guided by their theoretical analysis, the authors’ refined algorithm STABLE-RPS resolves some prior issues with existing RPS methodologies—convergence stability and ability to recover full accuracy at no compression. Further, their large scale empirical analysis reveals that STABLE-RPS can outperform nearly all pruning methods in terms of accuracy-compression tradeoff except for a leading method, lottery ticket rewinding, where STABLE-RPS only outperforms at high compression rates. Research on RPS is motivated by the objective of exploring and improving alternative paradigms for model compression that may be able to deliver better accuracy-compression tradeoff.", "questions": "1. Is STABLE-RPS compatible with parameter quantization (to achieve additional compression)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "• Thorough and useful theoretical analysis of existing RPS methods which ultimately inform design of improved RPS alrogithm\n\n• Design and implementation of STABLE-RPS algorithm which resolves two significant issues with existing RPS techniques—convergence stability and Pareto-continuity\n\n• Extensive empirical analysis of STABLE-RPS on three datasets and two architectures compared to seven existing pruning or model compression strategies which demonstrate that STABLE-RPS can outperform nearly all pruning methods in terms of accuracy-compression tradeoff", "weaknesses": "• STABLE-RPS cannot outperform a leading pruning method, lottery ticket rewinding, in low to medium compression regime", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EtoyBFmOz8", "reviewer": "Reviewer_L2s3", "summary": "The paper argues towards Randomized Parameter Sharing based models. The authors identified issues and provided solutions in the RPS technique ROAST, regarding stability (ROAST’s sensitivity to initialization hyperparameters, leading to divergence) and Pareto-continuity (ROAST’s inability to recover the accuracy of the original model at zero compression). The authors addressed this by proposing STABLE-RPS. The authors evaluated the method against many SOTA pruning methods and provided a theoretical grounding to their work.", "questions": "Diversity of datasets: Can more experiments be done with different types of datasets and model architectures evaluated to assess the proposed method’s performance? \n\nSome discussion around what are there any specific scenarios where the proposed method might not perform well?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Identification and Resolution of Stability, Pareto-Continuity Issues: The authors have identified key issues with existing techniques and proposed STABLE-RPS is an innovative method to address these issues \n\nRigorous Theoretical Foundation: The authors have also established a strong mathematical foundation to analyze the compression methods. This rigorous approach provides clear insights into how these methods affect vector inner products and under which conditions they perform optimally.\n\nQuality: The work done in the submission is of good quality with clear motivation and clarity. It could be a significant contribution if more empirical evidence is shown by authors.", "weaknesses": "Limited Experimental Validation: The paper could benefit more extensive experimental validation to complement the theoretical analysis. The authors only provided experimental evidence on small datasets. Given, that authors claim RPS is the way to go forward it would be good if they can follow up with more experiments\n\nLack of discussion around additional computation overhead in proposed STABLE-RPS method compared to other methods like ROAST. Some datapoints on what is the end to end training speedups that one can expect with this method will also help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "opB7kZnZPK", "reviewer": "Reviewer_sVzo", "summary": "This paper looks into the problem of enhancing the quality of compression methods that use parameter sharing. It argues that existing parameter sharing methods, such as ROAST, can have instability issues during initialization and suggests a solution to reduce the instability called STABLE-RPS. The main idea of STABLE-RPS is to reshape the blocks into a single array, split it into several chunks, and apply a hashing function to determine which chunks to share parameters. The paper claims that this approach can improve the stability and pareto-continuity. Experiments on ResNet20/VGG on CIFAR10/100/TinyImagenet demonstrate that the proposed method outperforms several existing pruning methods.", "questions": "1. How does STABLE-RPS compare with neural architecture search based methods? Can RPS still outperform search methods in different compression settings?\n\n2. How does the proposed method affect latency? It seems that flattening and chunking blocks would disrupt the original matrix multiplication operations, which could slow down the model execution significantly. The paper should provide latency results for the proposed method and other methods.\n\n3. Why does Theorem 3.2 involve cache line fetches? Cache line fetches are related to execution speed, not memory efficiency. The paper should explain how cache line fetches affect the memory consumption of the proposed method.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "While parameter sharing has been heavily studied in the past, the paper made an interesting observation of the shortcomings of existing random parameter-sharing methods in terms of stability and Pareto-continuity. Furthermore, the paper proposes techniques to improve the stability and pareto-continuity of parameter sharing methods.", "weaknesses": "1. The writing of the paper lacks clarity and needs some major improvements. For example, the problem statement is vague --- “Is pruning the correct method to reduce model parameters?”. However, the authors do not define what are the “correct methods” for model compression. Are methods that exhibit better stability and Pareto-continuity considered as correct? Are there other correctness conditions?  The paper also uses unconvincing examples to support its arguments. For instance, the authors try to explain pruning adversely affects model capacity, and use an example \"Consider pruning a n × d embedding table. If we prune beyond d×, we start getting degenerate zero embeddings.\" But what does \"pruning beyond dx\" even mean, and why is not it obvious that pruning reduces model capacity? \n\n2. The paper is narrowly focused on comparing with pruning-based methods, which are only a subset of the model compression techniques. The paper does not justify why pruning methods are the most relevant baselines for the proposed method, which belongs to the parameter sharing paradigm. The paper also ignores other important compression methods, such as quantization, distillation, and low rank factorization.\n\n3. The evaluation setups are weak. Evaluation is done on tiny datasets such as CIFAR10/100 and Tiny-ImageNet and very old architectures such as ResNet-20 and VGG-11 (10-year old), which raise questions on how well the observation from this work can be generalized to larger datasets, such as ImageNet, and more recent architectures, such as vision transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)