Hardware-aware compression with Random Operation Access Specific Tile (ROAST) hashing
Authors: Aditya Desai
Keren Zhou
Anshumali Shrivastava
Venue: Submitted to ICLR 2023 (2023)
Keywords: model compression
hardware aware

Abstract: Advancements in deep learning are often associated with increasing model sizes.
Training and deploying large models require sophisticated hardware and incur
significantly higher costs. Thus, model compression is a widely explored approach
to solving the problem. However, SOTA techniques fall short in one or more
desirable aspects of compression - for instance, pruning does not reduce memory
for training, quantization can only provide up to $32\times$ compression, HashedNet
is cache-inefficient, etc. This paper proposes a model-agnostic, cache-friendly,
and hardware-aware model compression approach: Random Operation Access
Specific Tile (ROAST) hashing. ROAST collapses the parameters by clubbing them
through a lightweight mapping. While clubbing these parameters, ROAST utilizes
cache hierarchies by aligning the memory access pattern with the parameter access
pattern. ROAST is up to $\sim 25 \times$ faster to train and $\sim 50 \times$ faster to infer than the
popular parameter sharing method HashedNet. Additionally, ROAST introduces
global weight sharing, which is empirically and theoretically superior to local
weight sharing in HashedNet, and can be of independent interest. With ROAST, we
can efficiently train and deploy the model using a much smaller memory footprint
($\sim 10 \times - 100 \times$ lesser) in text and image classification tasks

Under review as a conference paper at ICLR 2023
HARDWARE-AWARE COMPRESSION WITH RANDOM OP- ERATION ACCESS SPECIFIC TILE (ROAST) HASHING
Anonymous authors
Paper under double-blind review
ABSTRACT
Advancements deep learning are often associated with increasing model sizes. Training and deploying large models require sophisticated hardware and incur significantly higher costs. Thus, model compression is a widely explored approach to solving the problem. However, SOTA techniques fall short in one more desirable aspects of compression for instance, pruning does not reduce memory for training, quantization can only provide up 32x compression, HashedNet is cache-inefficient, etc. This paper proposes a model-agnostic, cache-friendly, and hardware-aware model compression approach: Random Operation Access Specific Tile (ROAST) hashing. ROAST collapses the parameters by clubbing them through lightweight mapping. While clubbing these parameters, ROAST utilizes cache hierarchies by aligning the memory access pattern with the parameter access pattern. ROAST is up to ~25 faster to train and ~50x faster to infer than the popular parameter sharing method HashedNet. Additionally, ROAST introduces global weight sharing, which is empirically and theoretically superior to local weight sharing in HashedNet, and can be of independent interest. With ROAST, we can efficiently train and deploy the model using much smaller memory footprint (~ 10 100x lesser) in text and image classification tasks.
INTRODUCTION
Models across different domains, including Natural Language Processing (NLP), Computer Vision (CV), and Information Retrieval (IR), are exploding in size. State-of-the-ar (SOTA) results in these domains are being obtained at disproportionate increase in model sizes, questioning the sustainability of deep learning (Thompson et al., 202T). For instance, SOTA architectures for vision include VGG (Simonyan & Zisserman (214) (1 150M params, 0.6GB) and ViT (Dosovitskiy et al. 2020) (up to 304M params, 1.2GB). Additionally, SOTA NLP models range from BERT (Devlin et al. 2018 (340M params, 1.36GB) to GShard (Lepikhin et al., 2020) (600B params, 2.4TB). Similarly, industrial-scan recommendation models such as DLRM (Naumov et al. 2019 Mudigere et al. 2021 can have up to 10s of trillions of parameters (50TB).
Large models, such as the above, come with various challenges. They need high-end distributed hardware for training and deployment, incurring higher costs. Additionally, the required model- parallel setup has higher inference and vrainin--iteration latency for these models. Model compression a research direction that aims to resolve these issues by reducing the memory footprint of the model. Compression of the order of 100x can eliminate the need for model-parallel setup for many SOTA models like (ad(Rdd al. 2019), GshardgLepikhin et al. 2020), DLRM (Naumov al. 2019) which now can fit on single GPU. Furthermore, compressing large models to small sizes come with immediate latency benefits. For example. Desai et al.) (2022) showed that by compressing the DLRM model 1 x x and using 1 GPU instead of 8 GPUs, we could get 3 faster inference at lower cost. Also, in the case of CPU inference, smaller model is efficient. For example, (Diamos et al. 2016) showed that if a single RNN layer can fit in registers, it leads to 146x faster inference.
Thus, the ML community has heavily invested in model compression. variety of model compression paradigms now exist in literature like pruning (Han et al. 2016b), quantisation (Han et al. 2b) knowledge distillation (Buciluà et al.) 2006). parameter- shariigg (Chen et al. 2015] Desai et al. 2022), and low rank decomposition (Hrinchuk al. 2020 Yin et 2021). Table [ compares these approaches on three considerations (1) if the model memory is reduced for training. (2) if the memory size can be controlled independently of the model, and (3) if the approach considers the underlying

---

Under review as a conference paper at ICLR 2023
Table 1:
Various compression techniques on three aspects (1) Memory reduction during training (
apart from inference) (2) arbitrary control over memory (3) Hardware awareness / cache-effiiency
Some versions of pruning that are tuned to the underlying hardware and are cache-efficient
Training memory
Arbitrary control on memory
Cache efficient
reduction
Pruning
No
No
No*
Low-rank decomposition
Yes
No
Yes
Low-precision
Yes
No
Yes
Quantization aware training (QAT)
No
No
N.A
Parameter sharing HashedNet
Yes
Yes
No
Knowledge Distillation
No
No
N.A
ROAST (ours)
Yes
Yes
Yes
memory hierarchies and is cache-efficient. We want the techniques fare positively in these three aspects. However, techniques like pruning, QAT, and knowledge distillation require us to use the memory of the original model while training and only reduce inference time memory. Additionally, there are limits to compression obtained by quantization and pruning depending on which component we are compressing. For example, we cannot prune embedding table (N d) more than d'x we do not want any embedding vector to have all zeros. HashedNet provides memory reduction during training and arbitrary control over memory. However, the look-ups in HashedNet are randomly and independently distributed across the total memory. This makes HashedNet cache-inefficiem:
This paper presents Random Operation Access Specific Tile (ROAST) hashing, a rraaeeer--haring approach that provides cache-efficiency and arbitrary control over memory during training as well as inference. ROAST does not change the model's functional form and can applied to all computational modules of model, such as MLP layers, attention blocks, convolution layers, and embedding tables. ROAST is hardware aware: it proposes a tile-based hashing scheme tuned to the memory access pattern of the algorithmic implementation of the operation being performed. ROAST uses this hash function to recover blocks of the model from a single array of parameters ROAST array. ROAST is superior to Hashed due to two factors (1) Unlike HashedNet, ROAST proposes global weight-sharing where parameters are shared across the different computational modules. As shall see, global weight-sharing is empirically and theoretically superior to local weight-sharing and might be of independent interest. (2) ROAST uses block-based hashing, which is theoretically superior to count-sketch hashing used in HashedNet. (Desai et al. 2022)
We show that with ROAST. we can train a BERT-2-2 (2 layers, 2 attention heads) model on the largest available text-classffication datasets (amazon-pollrrty, yelp-polarity) using 100x lesser memory without loss of quality. In cases where the model is overly parameterized, like using BERT-12-11 12-12 in the text classification task above, we can still obtain similar compression of x00x. Thus it is good alternative to neural architecture search. The results extend to CV datasets as well. Specifically, we can train a ResNet-9 model with 10x lesser memory for the CIFARIO 10 dataset. Importantly, we show that ROAST, due to its hardware-aware nature, is significantly faster than HashedNet: ROAST is up to ~ 25x faster to train and ~ 50x faster to infer than HashedNet for large matrix multiplications. Our current implementation of ROAST matrix multiplication is about 1.34x slower than full matrix multiplication in pytorch. This is a testament to how optimized CUBLAS libraries are. We believe, with enough investigation, we can make ROAST-MM comparably efficient to pytorch-MM as well.
One of the goals of model compression, apart from reducing memory usage, is to reduce computational workload for deployment. ROAST, currently, is not devised to decrease computation; it only decreases the memory footprint of a model. Reducing computation with a small memory is left for future work. However, it should be noted that reducing the memory footprint can significantly reduce computation latency and power consumption. As shown in (Han et al., 20 2016a), accessing memory from RAM is 6400x costlier than 32bit INT ADD and 128x more expensive than on-chip SRAM access in terms of energy consumption. Additionally, RAM access generally is ~100 slower than a floating-polt operation. So, this model compression with ROAST, in our opinion, is an important step for efficient training and inference.
Limitations of ROAST:
2 RELATED WORK
This section briefly reviews the rich history of model compression paradigms. Model compression can be generally classified into two categories: (1) Compressing a learned model and (2) Learning a compressed model. ROAST lies in the second category.
2

---

Under review as a conference paper at ICLR 2023
Compressing learned models: 1) Pruning: Pruning (Zhu & Gupta 2017) is technique to remove parts of a large model, including weights, blocks, and layers, to make the model lighter. Pruning can be performed as one-time operation gradually interspersed with training. 2) Quantization: Quantization can involve reducing the precision of the parameters of a model. Mixed precision models sometimes used where different precision is used with different weights. KMeans quantization is another type of quantization, where models' weights are clustered using KMeans, and each cluster's centroid is used for all cluster weights. Model compression, in this case, is achieved by reducing the number of distinct weights. 3) Knowledge distillation: Knowledge distillation (Buciluá et al. 2006) is widely applied in model compression with focus on distilled architectures. Knowledge distillation involves training a teacher model (large original model); then, a student model is trained using the logits of the teacher model. Empirically, the student model trained under this paradigm generalizes better than the student model trained standalone. Many variations exist on this basic idea of knowledge distillation.
While these techniques have successfully reduced memory for inference, one of the drawbacks of this line of compression is that the memory usage while training the model is not reduced. ROAST, however, provides solution that reduces the model's memory during training and inference.
Learning compressed models 1) Low-rank decomposition: In this method, matrices in the model decomposed into a product of two low-rank matrices, thus saving memory per matrix. A gener- alization of low-rank decomposition to tensors is called tensor-train decomposition 2) Parameter sharing: Parameter sharing approaches such HashedNet (Chen et al. (015) are generally used for matrix compression. These approaches randomly share weights among different parameters, reducing the model's memory usage.
This line of research provides model reduction even during training. However, Low-rank decompo- sition does not offer arbitrary control over memory footprint, and HashedNets are inefficient due to heavy aache-trashing caused by non-local lookups. Conversely, ROAST is model-agnostic parameter-sharing approach that can arbitrarily reduce the model size without affecting the functional form while keeping the model recovery efficient.
BACKGROUND
HashedNet: Compressing MIP matrices Previous work (Chen et al., 2015) introduced a weight sharing method to compress weight matrices of MLP models. They map each matrix parameter to a shared parameter array using a random hash function xxhash (Collet, 2016). In the forward pass, this mapping is used to recover a weight matrix and perform matrix multiplication for each MLP layer. In the backward pass, the gradients of each weight matrix are mapped to the shared compressed array and aggregated using the sum operation. It should also be noted that each MLP layer uses an independent array of parameters. One of the main concerns with HashedNet is that memory accesses the compressed array are non-coalesced. Thus, fetching a compressed matrix via HashedNet requires significantly more memory read transactions than fetching an uncompressed matrix for which memory accesses can coalesce. Our evaluation shows that uncoalesced memory accesses lead to high latency, especially for large matrices.
Random Block Offset Embedding Array (ROBE) for embedding compression In ROBE DDesai et al. 2022). the embedding table is generated using an array of parameters. The embedding of token obtained by drawing chunks of the embedding from the ROBE array. The locations of the chunks are decided randomly via light-weight universal hash functions. Authors of ROBE showed that ROBE hashing theoretically superior to feature hashing used in HashedNet. Also, the use of chunks causes memory accesses to coalesce, making embedding lookup efficient.
ROAST proposes a component agnostic, global parameter sharing approach that tunes the hashing function to match memory accesses of algorithmic implementation of operation over available hardware, thus giving a superior parameter sharing scheme.
4 RANDOM OPERATION ACCESS SPECIFIC TILE (ROAST) HASHING
Let M be the compressed memory from which parameters will be used, f be the model the function that we want to run using M, and W be the recovered weights used in f. f can be considered as a composition of operations (O,(X, W. By operation, we mean the smaller functions that, when composed together, give us the model f. Here X, is the input to the operation, and W; is the weights (i.e., learnable parameters) that O2 uses. Generally, W,s are distinct and do not share parameters.
3

---

Under review as a conference paper at ICLR 2023
NSP
Moss
2D ROAST tiles in MM(x, K) operation
BERT
1D ROAST tiles in L(t,E) operation
E(U
Masked
ef Serbinca
Cesddng
Poyers Eleee 20117
example: Attention matrix K
A
example: embedding
000
Parameter Memory (M)
[00)
Figure I: Generic model compression with operation-specific blocking for BERT as an example (left) Shows how 2D tiles are mapped to M in case of MM operation. (right) Shows how ID tiles are mapped M in case of operation. À is the module-specific GMS scaling factor
Random Operation Access Specific Tile (ROAST) hashing is way to perform efficient model- agnostic parameter sharing-base compression. The following distinct aspects of ROAST set it apart from previous parameter sharing-based methods. (1) ROAST is generic technique applicable to all computational modules. (2) ROAST proposes to tune its mapping from W; to MM in a way that coalesces memory accesses according to how memory is accessed during the operation. This makes ROAST efficient and up to 45x faster than competing approaches like HashedNet. (3) ROAST proposes Global Memory Sharing (GMS) opposed to Local Memory Sharing (LMS) used in HashedNet. We show GMS to be theoretically and empirically superior to LMS in Section 5 and o
4.1 ROAST OPERATIONS IN DEEP LEARNING
Any model f can be considered composition of smaller functions {O,(X, W.)]. There multiple ways to perform this decomposition depending upon what we consider a valid (or small enough) operation. In ROAST, consider three types of operations: (1) L(1, W), lookup that accesses M and recovers 1th element of W, say w. By element, we mean some particular part of W that is identifiable by integer. An example with embedding tables is given in figure (2) MM(X, W), matrix multiplication that multiplies X with W and returns the result, and (3) N(X). various operations that only act on the input but do not interact with M. In ROAST, in order to limit the memory usage, we make sure that L is used only on a small w and MM is performed without recovering the entire matrix. We find that most deep learning models, if not all, can be written as a composition of operations N, MM and L, where L is only applied on small parameters. Let us discuss how ROAST implements L and MM operations the following paragraphs.
Lookup (L(1, W)) We recover parameter weight of any shape in row-major format. Thus, we can consider w = W (1) to ID vector without loss of generality. ROAST recovers w from M in blocked fashion. Consider w to be composed of chunks of size Z. Each chunk e located in M using a universal hash function h1 and is recovered from the location hy (c) in M. Let C(i) give the chunk number of index i and (i)) give the offset of in this chunk.
w [i]] (M[[ + O()] h : N- + {0. .... |M - Z}
(1)
The recovered W has X as a scaling factor discussed section 4.2 The hash function hashes to range [O.... |M - }} to avoid overflows while reading the memory. For example, Figure [ (right) illustrates the embedding lookup using L with chunk size of 2. ROAST uses to implement computational modules such embeddings. bias vectors, and so on. We generalize the embedding lookup kernel from ROBE DDeaa et al. 2022) to implement our L kernel.
Matrix multiplication (MM(X, W.) 2D matrix multiplication is one of the most widely used operations in deep learning. We implement our ROAST-MM kernel with parameter sharing performed in way that the algorithm for matrix multiplication accesses coalesced pieces of M. An efficient
4

---

Under review as a conference paper at ICLR 2023
implementation of matrix multiplication on GPU follows block multiplication algorithm to the on-chip shared memory efficiently. While computing C A x B. A, B and C are divided in tiles of size Zo x Z Z1 Z2 and Zo Ze respectively. Thus, we divide our 2D weight matrix into tiles of size 71 x Z-2- The tile, (I, where x and are the coordinates of the tile, is located in M in row-major format via universal hash function h2(xr.y) Let Ci (f, j) and C2( give the r-coordinate and y-coordinate of the tile to which i, j belongs, Similarly, let O (i,j) and O2(i, j) give the z-offfett and y-offset of location (i on the tile. Then, we use the following mapping for ROAST-MM,
W = AMlla(CC((a..... C2(i, + Z201(i,)) O2(i,))
h2 : (0.---/1 |M 71 [2]
Again, is the scaling factor discussed section 4.2 The hash function hashes to a range {O...., [M] Z, Zz] } to avoid overflows while reading the chunk. Figure S (left) illustrates ROAST- MM with chunk size 2 x 2. The above mapping used whenever 2D tile is accessed in the matrix multiplication algorithm. The pseudo code for ROAST-MN is shown in algorithm [] We talk about implementation of this kernel and its evaluation in the later part of the paper. ROAST uses ROAST-MM kernel to implement computational modules such MLP layers, attention blocks, etc. Each module invoking ROAST kernels uses independent hash functions.
Algorithm 1 ROASS-MM(I x H x O)
Require: X E R M,, X, h N + {O.... [M] Z, Ensure: output = MM((X,M((()) M[h;;:))
value- +- TILEE Zo. Zz)
Allocate 2D tile of size Zo Zz accumulate results
for i E {0, l, 1/2n] - - do
for j E 1, [0/22] 1] do
value[:;:] 0
for k E (0,1, [H/Z]] 1} do
value + value + MM i + Zo. k: + M(h(k : ++ Zi.jij+ Z2)))
D Access to the weight tile passes through the hash function
end for outputli [i: : + Zo.j i [g] < À * value
end for end for
Apart from scaling each recovered parameter with module-specife X, we can also multiply it with another independent hash function Nk + {+1} (k=1 or k=2).
4.2 GLOBAL MEMORY SHARING (GMS)
HashedNet uses local memory sharing (LMS), which states that each layer will have independent compressed memory. In contrast, ROAST proposes global memory sharing (GMS), wherein we share memory across modules. However, modules cannot directly the parameters stored in M as each module's weights requires initialization and optimization at different scales. For instance, in the Xavier's initialization (Glorot & Bengio, 2010), weights are initialized with distribution Uniform(-1/44
1/VD) where n is size of the input to the module. In GMS, we must ensure that each module gets weights at the required scale. To achieve this, we first initialize the entire ROAST parameter array with values from the distribution Uniform(-11 (1//C, 1/C) for some constant C. Then, for each module, we scale the weights retrieved from the ROAST array by factor of A C=C/nn
One can understand the benefit of GMS over LMS in terms of the number of distinct functions in if that can be expressed using fixed M. Consider family of functions with parameters. GMS can potentially express |M| functions across different random mappings. In LMS, let separate parameters be of sizes R1, 722, and each of them is mapped into memories MI, Mr. Thus, n = Σ, n, and M = Σ, [M,l. Then LMS can only express |M1]n [Ma]"a.... "ea....| MM. 1* different functions. Thus expressivity of LMS is strictly less than that of GMS and can be orders of magnitude less depending on exact values of n, and [M,]- We also show that GMS is superior to LMS in terms of dimensionality reduction (feature hashing) in Section 5
5

---

Under review as a conference paper at ICLR 2023
Local Memory Sharing
Global Memory Sharing
Figure 2: Local memory sharing each module compresses its parameters separately. In Global memory sharing, all parameters from accross the modules share the same memory
4.3 FORWARD AND BACKWARD PASSES
Recall that ROAST, operations are of three types L, MM and N. The forward pass proceeds by applying each operation in sequence. If an operation is of type N, we directly apply its function on the input. For L and MM operations, outputs are computed according to the procedure described in Section 4.1
The gradient of the loss w.r.t a weight wy in M is the X-scaled aggregation of gradients of loss w.r.t all the parameters that map this weight. For simplicity of notation, consider 0 as the complete parameter, X(j) as the scaling factor we use for the module that 8 belongs to, and h be the mapping from 0 to M. See Appendix B.1 for more details.
V n. Sww)
Σ x() (()
j.b())m
4.4 IMPLEMENTATION OF ROAST-MM
The nigh-performance community has heavily investigated the fast implementation of the General Matrix Multiplication (GEMM) kernel, fundamental operation in many computational workloads, including deep learning. Optimized implementations of GEMM kernels are available in vendor libraries such cuBLAS (NVIDIA Corporation| 2022a) and CUTLASS (NVIDIA Corporation 2022b). Unfortunately, these implementations do not support custom tile loading operations, which the key of ROAST-MM. To implement ROAST-MM to a reasonable level of efficiency, we used Triton (Tillet et al.] (019): an intermediate language for tiled neural network computations. Triton abstracts out the shared memory management make it helpful in customizing tiled operations with high efficiency.
our implementation of ROAST-MM, the optimal size of coalesced tiles is a parameter that depends on the shape of the weight matrix. Therefore, different tile sizes can lead to different parallelism, occupancy, and shared memory efficiency, resulting in different execution times. We autotune this parameter to obtain the best performance for particular matrix shapes. We propose two strategies for autotuning each ROAST-MM layer- (1) Optimize the inference workload by autotuning the forward kernel and sharing the tile size with the backward kernels. (2) Optimize the training workload by autotuning the forward and backward kernels together. Extensive evaluation of this kernel is presented in appendix C.2
FEATURE HASHING QUALITY: GLOBAL MEMORY SHARING ADVANTAGE OVER LOCAL MEMORY SHARING
We can consider model compression dimensionality reduction of parameter vector (a one dimensional vector of all parameters model) of size into a vector of size |M m. Quality of inner- mmer-prrduct preservation is used as metric to measure the quality of dimensionality reduction. In terms of dimensionality reduction, ROAST uses ROBE hashing, which shows that chunk based hashing is theoretically better than hashing individual elements. In this section, we compare ROAST's GMS proposal against HashedNet's LMS using a chunck size of one. Consider two parameter vectors x, y E Rt we are interested in how the inner product of parameter vectors are preserved under hashing. Let x = [21. ..., Fg] and y [y1, y2. y/3] be composed of vectors of sizes N1, n22 ...R;; where [] denotes concatentation. In LMS, let each piece map memory of size f.m where E.J- = 1. The estimated inner product with GMS is
(.. = Σ
Σ 1(b()))))())z Σ 1(b(i)=))(())i)

---

Under review as a conference paper at ICLR 2023
Table Experimental settings: The datasets and models used in experiments.
Domain
Task
Dataset
#Samples
Model
Model size
NLP
coxtcclassifiaation
amazon-polarij
3.6M/0.44 4M
BERT-2-2
37.43M
NLP
text-classification
yelp-polarity
560K/38K
BERT-2-2
37.43M
CV
image-llassification
cifar10
50K/10K
ResNet
6.5M
The estimated inner product with LMS be written
fim
nt
TM
k
(1, V)
[1(1)=))900011 Σ
1((00)=)9(()m(
(1) Mi W.(.mm)
J.
(4)
Theorem Let F. y R and be composed of vectors x: [11,, 322 .... ...c and y Then the inner product estimation of global and local weight sharing are unbiased.
[y/, 1/2 ykl
E((x,, y) G.m) = (x, B((x, v) 1. (x, v)
(5)
The variance for inner product estimation can be written as,
vo((z.v))> = Σ f.
Σ
+ (on, ai) (2 8b)
(6)
V VL((z,)))
Σ V
where
Vi = Σ 0
1 fim
Earbra
where (01 O2..... an.) and yt (bi, b2.... Dmn)
(8)
where V. is local memory sharing variance and V is global memory sharing variance.
The two terms in VG can be understood as follows: The first term is the local variance with individual terms reduced by a factor of f:. This because each piece of the vector being distributed in memory that is 1 1/fix larger. However, in GMS, there is possibility of more collisions across pieces. This leads the second term in Vg. Note that, for a given æ, and finite value for m, Vo is always bounded. At the same time, V, is unbounded due to 0 < fi < 1 in the denominator. So the number of pieces increases or particular fi grows smaller, V,, increases. While we cannot prove that Ve is strietly less than WL, the data. Practically, each piece of the parameter vector is a computational block like a matrix for multiplication embedding table lookup. These blocks are initialized at scale proportional the square root of their size. So the norms of these vectors are similar. Let us assume the norm of each piece be /a. Also, let assume that over random data distributions over x and y. all the inner products to be 3 in expectation. Then,
Intuition:
investigate the equation under some assumptions
NG
22 (a² 2 + 82)
V v 1 m
(a²
B²)
1 f f2
1 fk
1
1 (a²
Vo
E.)
m
(9)
Thus, Vz greater than Ve, and it can be much greater depending on the exact values of f.. The proof of the theorem and other details are presented in Appendix B.2
EXPERIMENTAL EVALUATION
Setup: In this section, we evaluate the ROAST compression approach on two types of tasks. The
details of the tasks, datasets and models used are mentioned table 2 For image-ckassificttin
tasks, we choose the cifar- 10 dataset and the leader for the DawnBenchmark (Coleman et al. 2017) a ResNet-9 mode for cifar-10. 10. The target accuracy for this benchmark is 94% and hence we perform hyper-parameter tuning to get test accuracy of > 94%. We stop the tuning once

---

Under review as a conference paper at ICLR 2023
Table 3: Text classification task. (above) ROAST shows up to 100x compression without loss of quality on BERT-2-2 model. (below) Even in case of overparameterizze model of BERT-12-12, ROAST able to maintain the quality similar compression.
Text-classification Acc
Model
size
amazon-polarity
yelp-polarity
BERT-2-2
37.4M
93.4
90.8
BERT-1-1
30.3M
92.01
90.2
ROAST-1Ox (BERT-2-22
3.7M
94.6
91.1
ROAST-100x (BERT-2-2)
393K
93.8
91.0
BERT-12-12 (BERT-base)
108M
93.5
90.8
ROAST- I0x (BERT-
10.IM
94.64
90.9
ROAST- 100x (BERT-base)
1.1M
93.9
90.8
Table 4: Image classification task: (a see that ResNet-9 model can be trained in 10x smaller memory. (below). Pruning gives 100x post-training compression but requires complete memory for training. We can prune ROAST-10x model, which uses 10x lesser memory, further 10x to give 100x post-training model
Image-classffication Acc (target: 94%
Model
Size
cifar-10
ResNet-9
6.5M
94.2
ROAST-5x
1.2M
94.58
ROAST-
650K
94.15
PRUNE- 10x
650K (6.5M)
95.59
PRUNE- 100x
65K (6.5M)
94.8
PRUNE- 1000x
6.5K (6.5M)
93.34
ROAST-10x- PRUNE- 10x
65K (650K)
94.06
reach this accuracy and hence the results for CIFAR-10 10 should be compared w.r.t whether it crosses 94.0%. For NL.P tasks, we use two largest available text-classification datasets on huggingface (HuggingFace, 2022). For the model, use BERT-x-y (x:number of layers, y:number of attention heads) architecture with classification head. On both NLP datasets, using models larger than BERT-2- 2 lead to similar test accuracy and hence we choose BERT-2-2 as the base model. The other hyper parameters for NLP tasks are { batch 64 for amazon--olarity and 32 for yelp-polarity learning rate 2e-5, AdamW optimizer, Linear scheduler)
Roast for compression As we can see in tables 3 and 4 with ROAST, we can achieve similar quality of model in much smaller space. Specifically, for text-classification, we see that train and deploy the BERT-2-2 model in 100x lesser space. Similarly, we can train and deploy ResNet model in 10x lesser space for same target test accuracy. Thus, ROAST an effective method for training and deploying models on memory-constrainee systems.
Managing excess parameters It is clear from table that BERT-base architecture highly over parameterized for the tasks under consideration. However, even in this case, ROAST can be used to control the memory footprint while maintaining the functional form of the larger model.
Pruning and ROAST We perform unstructured perative-magnitude pruming (Han et al. 2016b) on ResNet model and find that pruning gives upto 100x compression. However note that pruning requires us to train the model using memory required to store the original model. However, compression with ROAST means using lesser memory even for training. Additionally, pruning can used in conjunction with ROAST to obtain smaller models using smaller memory. In table we see that we can prune 90% of weights in 10x compressed ROAST array and still achieve the same quality.
Local vs. Global memory sharing In the figure 3 show that the quality of the model while using global memory sharing is, indeed, better than local memory sharing. This supports our theoretical observation about these memory sharing schemes.
Efficiency of ROAST operators as compared to HashedNet Table shows the inference perfor- mance of simple model using ROAST-MM for matrix multiplication compressed memory. Our model linearly transforms the input vector and computes its norm. We optimized the ROAST-MM kernel for this experiment using the inference-optimal strategy. We make the following observations
8

---

Under review as a conference paper at ICLR 2023
Table Inference times of different square weight matrices using an input batch 512. For ROAST, the tile parameters of each matrix multiplication are autotuned. The measurements were taken using
TF32 on NVIDIA A100 GPU (48GB). We used PyTorch's matmul function (MM) for the full
uncompressed matrix multiplication. bbad good
Inference time (ms)
Weight matrix dimensions (Dim Dim)
Model
M size
512
1024
2048
4096
8096
10240
20480
Average
Full size
1MB
4MB
16MB
64MB
128MB
420MB
1.6GB
PY
0.10
0.11
0.12
0.22
0.69
1.18
3.91
0.91
4MB
0.31
0.34
0.6.3
2.02
6.20
9.67
35.22
7.77
32MB
0.31
0.41
0.86
3.64
13.66
22.11
92.40
19.06
64MB
0.31
0.46
1.09
6.47
31.21
42.45
178.07
37.15
HashedNet
128MB
0.31
0.60
1.62
9.10
34.62
56.03
229.31
47.37
256MB
0.32
0.62
1.82
10.25
38.28
62.67
256.22
52.88
512MB
0.33
0.68
2.05
10.59
40.55
65.74
272.23
56.03
4MB
0.28
0.30
0.27
0.48
0.99
1.36
4.83
1.22
32MB
0.28
0.29
0.27
0.44
1.01
1.38
4.88
1.22
64MB
0.28
0.29
0.27
0.44
1.00
1.40
4.93
1.23
ROAST
128MB
0.30
0.27
0.27
0.45
1.01
1.39
4.91
1.23
256MB
0.30
0.27
0.27
0.44
1.01
1.40
4.90
1.23
512MB
0.30
0.30
0.27
0.45
1.02
1.39
4.95
1.24
0.50
3.00
Accuracy
Accuracy
0.00
ROAST 100x Local
ROAST 100X
ROAST-100X-Glood
ROAST-100X-GII Globall
0.00
ROAST-10X-Global
ROAST-10X-Global
LOAsT-10x-Local
ROAST-10X-
50000
100000
150000
200320
200050
20000 48000 10000
10000
soccce 120000 100000 tw0033
Iterations
Iterations
(a) GMS LMS (oolaztt-oolartt)
(b) GMS LMS (yelp-polarity)
Figure 3: Effect of local and global memory sharing with compression of BERT-12-12 model for text-classffication tasks. yelp, rolling mean of measurements is taken to reduce noise in plots
from Table 7 (1) ROAST-MM outperforms HashedNet kernel consistently across the different multi- plication workloads. On an average over different workloads, ROAST-MM is up to 45x faster than HashedNet. (2) ROAST-MM is 1.34x slower than PyTorch-MM. This is expected Pytorch-MM uses extremely optimized libraries for matrix multiplication and ROAST-MA implementation is still interesting to note that ROAST-MM's performance better in terms of scaling efficiency than Py with the increase in workload. As the workload increases 1600 (from 512x512 to 20480x20480), PyTorch-MM takes 39 time, HashedNet takes 106x time whereas ROAST-MM only takes around 16x time. We present more detailed measurements across different optimizers for training-optimat strategy in the appending C.2
comparatively under-optimized.
CONCLUSION
Traditionally model compression has focused on memory reduction during inference. However, model memory during training also an important consideration. While some the existing methods such as HashedNet and Low-rank factorisation provide model reduction during training, these methods either do not provide cache-efficient model recovery or have implicit cap on memory reduction. ROAST overcomes these obstacles and provides cache-efficient, arbitrary control over the memory footprint of model during training and inference. ROAST, essentially provides a practical parameter sharing method. ROAST is theoretically better than HashedNet in terms of dimensionality reduction due to block based hashing and global memory sharing. We empirically validate the efficiency advantage of ROAST over HashedNet and that achieve high compression with ROAST.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil, Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pp. 2285-2294. PMLR, 2015.
Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. Dawnbench: An end-to-end deep learning bench- mark and competition. Training, 100(101):102, 2017.
Yann Collet. xxhash: Extremely fast hash algorithm, xxHash [Accessed May 15, 2022].
Aditya Desai, Li Chou, and Anshumali Shrivastava. Random offset block embedding (robe) for compressed embedding tables in deep learning recommendation systems. Proceedings of Machine Learning and Systems, 4:762-778, 2022.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Greg Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen, Jesse Engel, Awni Hannun, and Sanjeev Satheesh. Persistent mnss: Stashing recurrent weights on-chip. In International Conference on Machine Learning, pp. 2024-2033. PMLR, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010. /929, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: Efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-254, 2016a.
Song Han, Huizi Mao, and William J. Dally, Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. arXiv: Computer Vision and Puttern Recognition, 2016b. 20166.
Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova, and Ivan Oseledets. Tensorized embedding layers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4860, 2020.
HuggingFace. HuggingFace Transformer, 2022. https://gttchbb.ccm//uuggg
Eransformers [Accessed May 14, 2022].
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, et al. fiih--eeffrmmacc, distributed training of large-scale deep learning recommendation models. arXiv preprint 1XXi:2114400158.. 2021.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhul- gakov, Andrey Mallevich, Ili Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang
10

---

Under review as a conference paper at ICLR 2023
Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. arXiv:1906.0009) 2019.
NVIDIA Corporation. NVIDIA cuBLAS, 2022a.
https://@evelleer..vvdd....00.
cublas [Accessed May 14, 2022].
NVIDIA Corporation. NVIDIA CUTLASS, 2022b.
cut lass [Accessed May 14, 2022].
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel Manso. Deep learning's diminishing returns: The cost of improvement is becoming unsustainable. IEEE Spectrum, 58( 10): 50-53, 2021.
Philippe Tillet, H. T. Kung, and David Cox. Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations, pp. 10-19. Association for Computing Machinery, New York, NY, USA, 2019. ISBN 9781450367196. URL http::://o...rgg1....1
3328973
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th Annual International New York. NY, USA, 2009. Associ- ation for Computing Machinery, ISBN 978 9711605581161. doi: 10. 10.1145/1533774.5555555
Conference on Machine Learning, ICML '09, pp. 1113-1122 1120,
URL
https: :top:://do1: wtwp:io.rg11..0 1ee.....111111331111 1553516
Chunxing Yin, Bilge Acun, Carole-Jean Wu, and Xing Liu. Tt-rec: Tensor train compression for
deep learning recommendation models. Proceedings of Machine Learning and Systems, 3, 2021.
Michael Zhu and Suyog Gupta. To prune, not prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01876 2017.

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL DATA FOR REVIEWERS PARTS OF WHICH WILL GO IN MAIN PAPER IN FINAL VERSION
A.1 EXTENDED TABLE WITH EPOCH INFORMATION AND MORE BASELINES
Table 6: Table 3 extended version
Text-dassiliation
Epochs reach the yelp-polarity
Epochs reach
Model
size
amazon-polarity
Comment
BFRT-2-2
37.4M
93.4
5.6
90.8
5.4
BERT-I-1
30.3M
92.01
7,02
90.2
2.8
ROAST-IOX GMS (BERT-2-2)
3.7M
94.6
7.3
90.8
2.8
ROAST-IOUX- (BERT-2-2)
393K
90.8
PRUNE- 10x (BERT-2-2)
3.74M
93.5
9.02
89.65
full-9-1 schedule
PRUNE (BER
374K
91.36
9.8
89
9.8
9.ll schedule
PRUNE 10x(HERT-22)
1.4M
93.24
8.94
89,8
sull-T-9-schelllle
PRUNE-TOXX(BERT-21
370K
90.73
9.15
87.7
9.82
full-1-9-schedule
BERT-12-12
108M
BERT-12-12- 100-100--MM- LMS
10.1M
93.49
4.84
90.9
4.69
BERT-12-12 12- 1000- LMS
101M
92.9
4.87
90.7
9:03
Iest-lnnsoooation convergence for specific target accuracy
Model
size
amazon--plarte
yelp-polarity
Comment
larget
epochs
farget
epochs
12
TORM
93.4
90.8
4.7
RERT-12-12-106 12--00-M
10.1M
93.4
3.77
90.9
4.69
BERT-12-12- GMS
10.1M
93.4
1.97
90.9
3.4
BERT-2- 12- MS
10.MM
92.9
4.78
90.7
9.09
BERI-12-12-1000
10.1M
92.9
3.00
90.7
3.74
We add lot of information and new results to the table. Specifically,
We add the GMS and LMS results to the table separately. So that readers can get idea of each of the method on the task.
We add unstructured pruning (best pruning quality wise) resutls for NLP tasks as well. The pruning results obtained in the following manner. With the full-9-1 schedule, start from the fully trained model, perform iterative pruning during next epochs and then tune the final pruned model for more epoch. Whereas in the full-1-9 schedule,
again start from the fully trained model, perform pruning in the next epoch and then tune the model further for 9 epochs. We note the best achieved accuracy with the final model structure and the epoch at which this accuracy is reached.
For each result, we note the number of epoch when the best accuracy was reached.
We append an additional small table which notes the number epochs required to reach a target accuracy compare the convergence of different models.
We make the following observations.
GMS reaches better accuracy than LMS for the same amount of compression for both the datasets, Additionally, GMS reaches the same target accuracy faster than the LMS.
The ROAST approach is more effective than pruning approaches in NLP tasks of text- classification for architectures like BERT.
It is interesting that GMS-10x converges faster than original model on both datasets. We leave investigating this future work.
A.2 GMS vs LMS FOR YELP
As can be seen from the two plots in figure4 it is clear the GMS performs superior to LMS both the compression settings.
12

---

Under review as a conference paper at ICLR 2023
0.43
0.9
Accuracy
Accuracy
0.89
ORIG ADAST-100X-Globa
ORIG ROAST-10X-Globa Global
0.5
0.07
ROAST-10DX-Local
ROAST-1OX-Local
o5aco
340000 1600c0
1200C0
140020
Iterations
Iterations
(a) GMS vs. LMS x00x Yelp
(b) GMS LMS 10x Yelp
Figure 4: Separate plots for 10x and 100x ROAST for Yelp dataset for better visibility. Also, rolling mean of measurements was used to reduce noise in the plots
B
THEORY
ROAST is a generalized model compression which performs operation specific system-friendly lookup and global memory sharing. This raises some interesting theoretical questions
B.1 BACKWARD PASS FOR MODEL SHARING WEIGHTS ACROSS DIFFERENT COMPONENTS
A general function sharing weight, say across different components can be written as f(r, (())) The interpretation that was used in g(.) and then again used ahead in f. (In case MIP, we can think of x being used in multiple layers)
Let fgi. 92) where both 91 and 92 are functions of æ.
@f(,,99) Ōr
@f(..99)
0gi
(g(g))))
0g2
(10)
Og1
Öx
0ge
Fx
91 = and 92 = g(x)
0f(g1,92)
0f(x,g(1)) dr
Df(v.g(x))
0g(x) ly=e
(11)
dx
0g(z)
dr
0f()),92)
0 (f(r,
Of g()))
(12)
Or
Ox
Ox
Renaming,
a ((r,g(()) 0f(2,(()) -=
af(2,
(13)
Ōr
d2
Qy
y=,,==
Thus, we can essentially consider each place where x appears as new variables and then gradient w.r.t is just summation of partial derivatives of the function w.r.t these new variables. Thus, is easy to implement this in the backward pass. In order to make sure that the memory utilization in backward pass is not of the order of the recovered model size, we not use the auto-differentiaatin of fensorhlow/yytrch.. We implement our own backward pass and it can be found in the code.
B.2 GLOBAL FEATURE HASIIING LOCAL FEATURE IIASIING.
We can consider model compression techniques as dimensionality reduction of the parameter vector (a one dimensional vector of all parameters in a model) size n into a vector size |M] = m. Quality of imner- product preservation is used metric to measure the quality of dimensionality reduction. In terms of dimensionality reduction, ROAST uses ROBE hashing Desai et al. (2022). which showed that chunk based hashing is theoretically better than hashing individual elements. In
13

---

Under review as a conference paper at ICLR 2023
this section, analyse GMS proposal of ROAST against LMS of HashedNet. For the purpose of this comparison we assume a chunk size of 1. Consider two parameter vectors 2, y E R". We are interested in how inner product between these parameter vectors preserved under hashing. Let x [ (2..... and y = (/- be composed of k pieces of sizes n,, f2, .p... LMS, let each piece be mapped into memory of size fam where Σ; f+
The estimators of inner product in the GMS case can be written as,
(1, v) G.
1(h(0)=))(00=0)) [((()))))(())])))
(14)
J=1 ==1
The estimate of inner product with LMS can be written as, frm
c V7
F.
I0(0))))(()))() I(M()=))((())))))
Σ
(a wil. G.fem)
1=1
(15)
Note that
(x,, 3) L.m.J
(1t (.(f.))))
(16)
The GMS estimator is the standard feature hashing estimator and the LMS is essentially sum of GMS estimators for each of the piece. as Elg(t)] = O, it is easy to check by linearity of expectations that Expectation The suffix refers local hashing and G refers global hashing.
Eg (((x,))).m)) (ax,))
(17)
E, = (((x, v) L.m.F) (x, 8)
(18)
Let us now look at the variance. Let us follow the following notation,
Va V(la, (1,,, GMS variance of entire vectors
Vz (( v) L.m.. F) LMS variance of entire vectors
Vi = V((II, (..m)) variance of each piece
we can write Vi as follows. The following equation is easy to derive and it can be found the lemma of Weinberger et al. (2009)
V =
+
Eabbrabby)) where dt = {a), (2...an.) and yr = (bi, b2. bm,)
(19)
As, each the piece is independently hashed in LSM, we can see
V1
Vi
(20)
1=1
Let us now look at VG. Again, using lemma from Weinberger et (2099)
1
Ve
(21)
m
i
i
The expression can be split into terms that belong same pieces and those across pieces
&
Ve = Σ Σ
Σ
1=1
Σ
(a?4²)
+
Σ
,,y;,,,))
m
[1=1 12=1.11412 iEpiece 1l Epieces 12
iEpiece-ll Ep
14

---

Under review as a conference paper at ICLR 2023
VG fiVi
E
au 1 (w1 (1r2 Mv2)
(22)
m
11=1 12=121122
Observation 1: In Vz. we can see that there terms with T which makes unbounded. It makes sense if number of pieces increase lot a lot of compressions will not work for example if number of peices > IM). Also, it will affect Vz, a lot when some ft is very small which can often be the case. For example, generally embedding tables in DLRM model are much larger than that of matrix multiplciation modules (MLP). which can make f = 0.001 for MLP components.
Observation 2: Practically we can assume each piece, no matter the size of the vector, be of same norm. The reason lies in initialization. According to Xavier's ' initialization the weights of a particular node are initialized with norm So for now lets assume more practical case of all norms being equal to vo. Also, in order to make the comparisons we need to consider some average case over the data. So let us assume that under independent randomized data assumption, the expected value of all inner products are B. With this in expectation over randomized data, we have
kel 1) (a² + B²)
Va Σ fiV
(23)
m
Now note that, 1 fi
Σ )abbaadb)) where Ty (a1, a2...am) and yı (b1, bz..
(24)
m
(dropping the subscript "I" below)
Vi2 =
1
(11=12/l11 + (2,8))) xi(v?)
(25)
Vi 11 1 f m
(3)
B3) Σ x 5
(26)
Note that for each negative term, there are n, positive terms. To simplify we disregard this term in the equation above. This is an approximation which is practical and only made to get sense of V. and Ve relation.
k(k-1) (a² + 82)
VL - Vo Σ
Vi Σ
liV
m
V1 - VG Σ 1
1
k(k - 1) (a? + 82)
1)((a² +
fi
m
V1 - VG Σ 1
(
1
- 1)((a² + +82)
k(k - 1) (a² + B2)
ft
m
k(k - 1)
((a2 + 3² k(k - 1) (a² + 32)
V, - Vo 2
V. VG 2
Note that we ignored a term which reduces the Vz bit, Let the error be
Vz VG The above equation shows for the best case, Vo might be slightly more than Vy, However for general case where harmonic mean is much worse than arithmetic mean, Vz will be much larger depending on exact s
(27)
15

---

Under review as a conference paper at ICLR 2023
Table 7: Inference times of different square weight matrices using an input batch 512. For ROAST, the tile parameters of each matrix multiplication are autotuned. The measurements were taken using
TF32 on NVIDIA A100 GPU (48GB). We used PyTorch's matmul function (MM) for the full
uncompressed matrix multiplication. bbad good
Inference time (ms)
Weight matrix dimensions (Dim Dim)
Model
M size
512
1024
2048
4096
8096
10240
20480
Average
Full size
1MB
4MB
16MB
64MB
128MB
420MB
1.6GB
PY
0.10
0.11
0.12
0.22
0.69
1.18
3.91
0.91
4MB
0.31
0.34
0.6.3
2.02
6.20
9.67
35.22
7.77
32MB
0.31
0.41
0.86
3.64
13.66
22.11
92.40
19.06
64MB
0.31
0.46
1.09
6.47
31.21
42.45
178.07
37.15
HashedNet
128MB
0.31
0.60
1.62
9.10
34.62
56.03
229.31
47.37
256MB
0.32
0.62
1.82
10.25
38.28
62.67
256.22
52.88
512MB
0.33
0.68
2.05
10.59
40.55
65.74
272.23
56.03
4MB
0.28
0.30
0.27
0.48
0.99
1.36
4.83
1.22
32MB
0.28
0.29
0.27
0.44
1.01
1.38
4.88
1.22
64MB
0.28
0.29
0.27
0.44
1.00
1.40
4.93
1.23
ROAST
128MB
0.30
0.27
0.27
0.45
1.01
1.39
4.91
1.23
256MB
0.30
0.27
0.27
0.44
1.01
1.40
4.90
512MB
0.30
0.30
0.27
0.45
1.02
1.39
4.95
1.24
C ROAST-MM LATENCY MEASUREMENTS
C.I INFERENCE OPTIMIZATION
C.2 TRAINING OPTIMIZATION
See tables
(or))rd(ms)
(optimized for forward backward) dim (Matrix dimension = dim dim)
Memory (mb)
512
1024
2048
4096
8096
10240
20480
Average
Full
0.16
0.12
0.12
0.24
0.66
0.91
3.03
0.75
(uncompressed)
0.37
0.35
0.65
2.04
6.23
9.62
35.64
7.84
32
0.39
0.42
0.90
3.67
13.73
22.06
92.83
19.14
64
0.33
0.47
1.11
6.45
25.78
42.51
178.20
36.41
HashedNet
128
0.28
0.56
1.61
9.07
34.21
56.07
229.34
47.31
256
0.20
0.54
1.72
9.95
38.17
62.47
258.11
53.02
512
0.14
0.50
1.88
10.37
40.40
65.43
272.19
55.84
0.30
0.31
0.31
0.50
1.43
2.01
7.54
1.77
32
0.30
0.33
0.35
0.55
1.44
2.09
7.59
1.81
64
0.29
0.31
0.33
0.56
1.45
2.08
7.80
1.83
ROAST
128
0.25
0.27
0.28
0.54
1.41
2.09
7.84
1.81
256
0.16
0.18
0.19
0.46
1.33
2.02
7.82
1.74
512
0.21
0.06
0.13
0.41
1.29
1.97
4.98
1.29
Table 8: Inference (forward pass time) for different shapes of square weight matrix with input batch of 512. The pile-parameters of multiplication are optimized for each function over "forward backward" pass The measurements are taken with tf32 on A100 (48GB)
D VARIANCE IN QUALITY OVER DIFFERENT RUNS
The figure shows three of ROASTed BERT and BERT models
16

---

Under review as a conference paper at ICLR 2023
backward(ms)
(optimized for forward backward) dim (Matrix dimension dim dim)
Memory (mb)
512
1024
2048
4096
8096
10240
20480
Average
Full
0.35
0.22
0.24
0.48
1.35
2.01
7.65
1.76
(uncompressed)
0.65
0.53
0.95
2.60
8.51
13.21
56.59
11.86
32
0.68
0.69
1.80
6,36
24.13
38.95
160.54
33.31
0.74
1.06
2.81
10.78
41.35
67.02
271.86
56.52
HashedNet
128
0.91
1.34
3.40
12.41
51.00
81.25
337.31
69.66
256
1.29
1.84
4.02
14.57
58.03
91.18
376.83
78.25
512
2.08
2.62
4.90
16.24
62.45
98.46
391.46
82.60
0.54
0.54
0.60
1.20
2.54
3.72
13.99
3.30
32
0.57
0.61
0.69
1.06
2.71
4.04
15.07
3.54
64
0.64
0.73
0.77
1.17
2.82
4.18
15.50
3.69
ROAST
128
0.79
0.81
0.89
1.38
3.17
4.73
18.30
4.30
256
1.19
1.17
1.27
1.77
3.56
5.17
18.33
4.64
512
2.11
1.92
2.12
2.53
4.33
5.98
22.71
5.96
Table 9: Backward pass for different shapes of square weight matrix with input batch of 512. The sile-parameters of multiplication are optimized for each function over "forward + backward" pass The measurements are taken with tf32 on A100 (48GB)
0.30
0.00
Accuracy
Accuracy
0:90
9.09
or ginal-seed] ginal-seed2
ROAST-100X-Seedl
ROAST-100X-seed2
geadd
ROAST-100X-seed3
lterations
Iterations
(a) yelp original
(b) yelp ROAST100x
0.50
Accuracy
Accuracy
0.88
or riginal-seed1 riginal-seed22 or seedd
ROAST-100X-sEed1
ROAST-100X-Seed2
0725
R0AS1-100X-seeds
6300
2:00 10000
1900
Iterations
iterations
(c) ag news original
(d) ag_news ROAST100x
Figure 5: Three runs of original and ROAST-100x runs
17

---

Under review as a conference paper at ICLR 2023
update weights (optim.step()(ms)) (optimized for forward backward)
dim (Matrix dimension dim dim)
optim
Model Full
msize
512
1024
2048
4096
8096
10240
20480
Average
adagrad
0.14
0.11
0.15
0.60
2.16
3.41
13.45
2.86
0.14
0.11
0.11
0.11
0.11
0.12
0.54
0.18
32
0.35
0.33
0.33
0.33
0.33
0.34
0.36
0.34
64
0.61
0.61
0.61
0.61
0.61
0.62
0.61
0.61
HashedNet
128
1.15
1.14
1.14
1.14
1.15
1.19
1.18
1.15
256
2.22
2.21
2.21
2.21
2.22
2.26
3.87
2.46
512
4.36
4.36
4.35
4.35
4.37
4.40
4.47
4.38
4
0.11
0.11
0.11
0.11
0.12
0.11
0.11
0.11
0.33
0.34
0.34
0.33
0.33
0.33
0.33
0.33
64
0.60
0.61
0.61
0.60
0.61
0.61
0.61
0.61
ROAST
128
1.14
1.14
1.14
1.14
1.14
1.14
1.14
1.14
256
2.21
2.21
2.21
2.21
2.21
2.21
2.21
2.21
512
4.38
4.35
4.36
4.35
4.35
4.36
4.35
4.36
Full
0.15
0.15
0.23
1.06
3.89
6.18
24.47
5.16
4
0.15
0.23
0.16
0.16
0.16
0.16
0.16
0.17
0.57
0.57
0.57
0.57
0.57
0.57
0.59
0.57
64
1.06
1.06
1.06
1.06
1.06
1.06
1.16
1.08
HashedNet
128
2.03
2.05
2.04
2.04
2.05
2.04
2.23
2.07
256
3.98
3.99
3.98
3.99
4.00
4.00
4.22
4.02
adam
512
7.89
7.89
7.89
7.89
7.91
7.90
8.13
7.93
4
0.15
0.23
0.15
0.16
0.16
0.15
0.16
0.17
0.57
0.57
0.57
0.57
0.57
0.57
0.57
0.57
64
1.07
1.06
1.06
1.06
1.06
1.07
1.06
1.06
ROAST
128
2.05
2.03
2.04
2.04
2.03
2.04
2.04
2.04
256
4.01
3.98
3.99
3.99
3.99
3.99
3.99
3.99
512
7.89
7.89
7.89
7.89
7.89
7.89
7.89
7.89
Full
0.08
0.07
0.08
0.20
0.62
0.97
3.92
0.85
4
0.08
0.07
0.08
0.07
0.07
0.08
0.08
0.08
0.12
0.12
0.12
0.12
0.12
0.12
0.17
0.13
64
0.19
0.20
0.20
0.20
0.20
0.21
0.31
0.22
HashedNet
128
0.35
0.34
0.34
0.35
0.35
0.37
0.48
0.37
256
0.64
0.64
0.64
0.64
0.65
0.67
0.83
0.67
sgd
512
1.23
1.23
1.23
1.23
1.25
1.24
1.25
1.24
4
0.07
0.07
0.07
0.08
0.07
0.07
0.23
0.10
32
0.12
0.12
0.13
0.12
0.12
0.12
0.12
0.12
64
0.22
0.19
0.20
0.19
0.19
0.20
0.29
0.21
ROAST
128
0.34
0.35
0.34
0.34
0.34
0.35
0.40
0.35
256
0.64
0.65
0.64
0.64
0.64
0.65
0.64
0.64
512
1.27
1.23
1.23
1.23
1.28
1.23
1.62
1.30
Table 10: Weight update operation (optimizer.step() for different shapes of square weight matrix with input batch of 512. The tile- allamaraeesers of multiplication are optimized for each function over "forward + backward" pass The measurements are taken with tf32 on A100 (48GB)
18

---

Under review as a conference paper at ICLR 2023
total fwd + bkwd + optimize (ms) (optimized for forward + backward)
dim (Matrix dimension dim dim)
optim
Model Full
msize
512
1024
2048
4096
8096
10240
20480
Average
adagrad
0.65
0.46
0.51
1.32
4.17
6.33
24.13
5.37
1.16
0.99
1.71
4.74
14.86
22.95
92.78
19.88
32
1.43
1.44
3.03
10.37
38.19
61.35
253.72
52.79
64
1.68
2.14
4.53
17.83
67.74
110.15
450.66
93.53
HashedNet
128
2.34
3.04
6.15
22.62
86.36
138.51
567.83
118.12
256
3.71
4.59
7.95
26.73
98.42
155.92
638.80
133.73
512
6.58
7.47
11.13
30.96
107.21
168.30
668. 12
142.83
4
0.95
0.95
1.02
1.81
4.09
5.84
21.64
5.19
1.21
1.27
1.38
1.94
4.49
6.46
23.00
5.68
64
1.54
1.64
1.70
2.34
4.87
6.86
23.90
6.12
ROAST
128
2.18
2.22
2.31
3.06
5.72
7.97
27.28
7.25
256
3.57
3.56
3.67
4.43
7.10
9,40
28.35
8.58
512
6.70
6.32
6.62
7.29
9.97
12.31
32.04
11.61
Full
0.50
0.48
0.60
1.78
5.89
9.11
35.01
7.62
4
1.00
1.56
1.76
4.81
14.94
23.07
86.76
19.13
1.43
1.78
3.29
10.60
38.45
61.64
253.20
52.91
64
2.03
2.63
4.97
18.35
68.28
110.63
450.86
93.96
HashedNet
128
3.18
4.27
7.02
23.54
87.47
139.30
568.72
119.07
256
5.45
6.30
9.71
28.66
100.19
157.55
633.80
134.52
adam
512
10.08
10.94
14.64
34.56
110.71
171.67
672.24
146.41
4
1.00
1.27
1.05
1,86
4.06
5,89
21.71
5.26
1.45
1.56
1.52
2.21
4.72
6.69
23.28
5.92
64
2.13
2.02
2.18
2.80
5.34
7.39
24.35
6.60
ROAST
128
3.26
3.11
3.23
3.95
6.62
8.85
28.22
8.18
256
5.82
5.33
5,45
6.21
8.97
11.15
30.19
10.45
512
9.82
9.87
10.14
10,90
13.52
15.82
35.59
15.09
Full
0.44
0.43
0.46
0.90
2.62
3.90
14.68
3.35
4
1.25
0.95
1.70
4.72
14.76
22.96
86.70
19.01
0.99
1.23
2.86
10.17
38.10
61.16
252.99
52.50
64
1.16
1.84
4.11
17.51
67.28
109.78
450.34
93.15
HashedNet
128
1.59
2.24
5.28
21.84
85.46
137.54
566.88
117.26
256
2.21
3.00
6.35
25.19
96.91
1.5443
630.75
131.26
sgd
512
3.42
4.28
8.06
27.91
104.03
164.94
665.29
139.70
4
0.92
0.92
0.98
1.79
3.94
5.82
22.39
5.25
32
0.95
1.00
1.17
1.75
4.28
6.25
22.77
5.45
64
1.62
1.15
1.26
1,92
4.45
6,45
24.01
5.84
ROAST
128
1.38
1.44
1.52
2.26
4.90
7.25
27.18
6.56
256
2.04
2.10
2.14
2.85
5.53
7.91
26.98
7.08
512
3.56
3.20
3.36
4.18
7.10
9.17
31.20
8.82
Table 11: Total training step time for different shapes of square weight matrix with input batch of 512.
The tile-parameters of multiplication are optimized for each function over "forward + backward" pass The measurements are taken with tf32 on A100 (48GB)
19


Reviews:
[{"review_id": "mNaOvlgRfa", "reviewer": "Reviewer_MkTt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces ROAST (Random Operation Access Specific Tile) hashing, a model-agnostic, hardware-aware model compression framework. ROAST essentially provides a global parameter sharing method to give arbitrary control to the user over the memory footprint of model during both training and inference. Evaluation with both BERT and ResNet-9 demonstrates the feasibility of 100x memory footprint reduction without accuracy degradation.\n", "summary_of_the_review": "As the NN scales, there are growing concerns for efficient resource usage for both training and inference--in particular for memory capacity and bandwidth. That said, this paper tackles a timely problem to mitigate this memory bottelenck. On the positive side, the proposed hardware-friendly paramter sharing scheme looks promising and presents impressive results for both BERT-2-2 and ResNet-9. Also, the authors address the memory capacity problem not only for inference (which most existing works tackle) but also for training.\n\nHowever, I still have several concerns about this work, at least in its current form, as follows:\n\n* ROAST is evaluated on relatively simple classification tasks with a small number of classes. It would have made this work much stronger if the authors evaluate more complex tasks such as ImageNet, SQuAD, and so on.  At the end of the day, the proposed framework would be most useful for large models that can handle complex tasks, but evaluation is currently falling short. I am wondering how ROAST would perform on those complex tasks.\n\n* The baseline used for evaluation is HashedNet, which was introduced in 2015 and hence seems outdated. Is there any stronger, more SOTA baselines to compare against?  How would ROAST compare against them?\n\n* ROAST-MM introduces a scaling factor $\\lambda$, which is dependent upon \"some constant $C$\". It is not clear to me how to set this parameter. Also, how sensitive is the performance of ROAST to the setting of this parameter? How much effort is needed to set this parameter properly? What factor does affect the setting of this parameter? Model? Hardware configuration (like cache size)? Or, both?  Please elaborate.\n", "strength_and_weaknesses": "**Strengths**\n\n* 100$\\times$ reduction of memory usage with no accuracy drop is impressive.\n\n* ROAST addresses the memory usage in both training and serving.\n\n* It is interesting to see only three ROAST operations are sufficient for running DL models.\n\n**Weaknesses**\n\n* ROAST has been evaluated on relatively simple tasks with a small number of classes and it is not clear how it is applicable to larger, more complex tasks without causing accuracy drop.\n\n* The baseline used for evaluation is HashedNet, which seems somewhat outdated.\n\n* In ROAST-MM there is a model-specific parameter $\\lambda$, which is determined by \"some constant $C$\", and it is not clear how sensitive the performance is to the setting of this parameter.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "auJzT9wfku", "reviewer": "Reviewer_sRtG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a model compression approach: Random Operation Access Specific Tile (ROAST) hashing. The authors consider three operations for block-based hashing to reduce memory usage, and they also introduce a global memory-sharing method to improve model accuracy. Experiments on BERT and ResNet show that the proposed ROAST exceeds HashedNet and achieves high compression when training models.", "summary_of_the_review": "While the paper proposes an interesting idea, the experiments are insufficient, and some details of the proposed technique are missing. ", "strength_and_weaknesses": "## Strength\n1. The proposed ROAST achieves similar quality in almost 100x less space for compression model training. ROAST performs well in many research areas including text classification and image classification. The experimental results show that the proposed global memory sharing method performs better than previous local memory-sharing. \n\n## Weakness\n1. Lack of comparison of time costs between GMS and LMS. Will this global memory share cause access conflict?\n\n2. For Local vs. Global memory sharing in Figure 3(b), the GMS shows more significant accuracy degradation than LMS. How do you know this is not relevant to the GMS method but to other factors? \n\n3. Lack of experiments on different models and datasets. Many text-classification tasks are introduced in the introduction but only experiment on BERT-2-2 and BERT-base. For the image classification task, only ResNet-9 on CIFAR-10 is used.\n\n4. Need more details about the implementation of ROAST operations. It is hard to follow.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MxFpMUOPFK", "reviewer": "Reviewer_4EgW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Weight sharing is one of the effective way to compress the model. This paper propose a new hash method to lower the memory usage for neural network models. Compare with the previous hash method, which use the local memory sharing, this paper propose a new idea call global memory sharing, which is aimed to lower the memory usage. \n\n\n\n", "summary_of_the_review": "Overall, the paper's idea is good in the area of weight sharing. The problem of this paper is the lack of a comprehensive comparison with state-of-the-art. At least it should be compared with more weight sharing methods. \n\nWhen compared with model compression method like pruning, the author only list the model size. It is better to also compare the energy consumption and latency. ", "strength_and_weaknesses": "Strength: Overall, this paper is well organised. Experiment results has well proved the efficiency of the method\n\nWeakness: 1.\tWeight sharing can lower the memory usage but can not reduce the number of operations.  As authors claimed, the RAM access is around 100X slower than the computation. This is true. However, from the hardware perspective, a efficient way to lower the memory access delay is cache. The real memory access delay can be much shorter, and can be almost the same as the computation time. Besides, as the computation of neural network is regular, we can use pipelined design to further eliminate the negative impact of memory access. Besides, with the cache design, we can substantially reduce the energy consumption of memory access. I understand this is the common problem of memory sharing, not the problem of this paper only. However, it would be better to do a more comprehensive comparison with other model compression methods not only on model size but also on latency and energy consumption. \n\n2.\tThe comparison is not enough. In terms of weight sharing, please do a more comprehensive comparison with other state-of-the-art weight sharing methods. \n\n3.\tDoes table 5 compared under the same accuracy? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "28I27Zu9Vx", "reviewer": "Reviewer_rkTz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper presents ROAST, a hashing-based technique for reducing the memory usage of ML models for both training and inference. ROAST improves on the previously existing hashing techniques for this purpose (ROBE/HashedNet) by using block-based (cache/HW – friendly) memory accesses. It also proposes the policy of globally sharing weights (across layers) as opposed to prior art (HashedNet) which shared weights per layer only. This in turn allows ROAST to achieve much higher compression ratios, which translate to much larger performance gains with much lower memory footprint over the state-of-the-art techniques. \n\nThe authors provide theoretical and empirical proof showing that their global memory sharing approach is superior to local memory sharing used in prior art. Lastly, authors demonstrate strong accuracy results using ROAST for a range of Text classification and Image classification benchmarks. \n", "summary_of_the_review": "The results show in Figure 3b) violate the statement proved showing GMS should achieve higher performance than LMS. Without a good justification for this miss, it is hard to confirm that the GMS technique has been effectively scaled to all DNN layers. Further, it is important to understand if the GMS approach brings with it any addition to training complexity (e.g., increased hyperparameter tuning). If the authors can add data for these requests, it will help consolidate the presented results. ", "strength_and_weaknesses": "Section 2 and 3 provide a good overview of the targeted problem space and the immediate baselines (ROBE/HashedNet) which are closest to the proposed ideas. The diagrams provided in the paper are helpful in explaining the proposed ideas and authors also present detailed proofs that help justify the superiority of GMS approach over LMS. Authors show large speed up compared to the previous hashing approach, HashedNet. \n\nIn Section 6, authors explain the experimental setup designed to verify the efficacy of ROAST. While the authors explain that they perform some hyperparameter tuning to achieve a given target accuracy it fails to communicate how using GMS approach in ROAST impacts the trainability or ease of training of their models. E.g., Adding the epochs of training incurred to achieve the target accuracy for the different baselines in Table 3 and 4, might help explain that. The intuition is this, since the weights are shared across all layers the cumulative gradient update application might make it harder to get the same level of accuracy with fewer unique weights (compared to LMS).\n\nIn Table 4, authors demonstrate that ROAST can be combined with other model compression techniques e.g., PRUNING to achieve further model compression. While this is lucrative, structured pruning that takes advantage of the block-wise nature of ROAST memory access could achieve even better accuracy with low training effort. Consider trying that as well. \n\nTable 5 shows the inference runtime for a range of memory size and model size values on a GPU with 48MB cache. While the ROAST inference time dominates over HashedNet baseline, especially for large model sizes (high compression ratio). The comparison with PyTorch based matrix multiplication (MM) is also interesting. While PyTorch with its efficient matrix multiply implementation achieves fast inference time for lower model sizes, as the model size grows the benefit of PyTorch over ROAST reduces. Hinting potentially that ROAST based memory optimization could further help improve PyTorch MM implementation for hashed DNNs. \nThe results in Figure 3b are somewhat contradictory to that result that GMS is superior to LMS for a range of models. Considering that the Local ROAST approaches (10x and 100x) end on a slightly higher accuracy number than corresponding GMS approaches. Even seeing accuracy values that match closely for GMS and LMS approaches is surprising. Kindly explain why the yelp-polarity model defies the expected trend as seen in Figure 3a. \n\nI appreciate the authors clearly stating the limitations of the proposed work (focus on reducing memory access during training/inference only, not compute cost)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject