The trade-offs of model size in large recommendation models : 100GB to 10MB Criteo-tb DLRM model
Authors: Aditya Desai
Anshumali Shrivastava
Venue: NeurIPS 2022 Accept (2022)
Keywords: embedding compression
DLRM
criteo TB

Abstract: Embedding tables dominate industrial-scale recommendation model sizes, using up to terabytes of memory. A popular and the largest publicly available machine learning MLPerf benchmark on recommendation data is a Deep Learning Recommendation Model (DLRM) trained on a terabyte of click-through data. It contains 100GB of embedding memory (25+Billion parameters). DLRMs, due to their sheer size and the associated volume of data, face difficulty in training, deploying for inference, and memory bottlenecks due to large embedding tables. This paper analyzes and extensively evaluates a generic parameter-sharing setup (PSS) for compressing DLRM models. We show theoretical upper bounds on the learnable memory requirements for achieving approximations to the embedding table. Our bounds indicate exponentially fewer parameters suffice for a good approximation. To this end, we demonstrate a PSS DLRM reaching 10000$\times$ compression on criteo-tb without losing quality. Such a compression, however, comes with a caveat. It requires 4.5 $\times$ more iterations to achieve the same saturation quality. The paper argues that this tradeoff needs more investigation as it might be significantly favorable. Leveraging the small size of the compressed model, we show a 4.3$\times$ improvement in training latency leading to similar overall training times. Thus, in the tradeoff between the system advantage of a small DLRM model vs. slower convergence, we show that scales are tipped towards having a smaller DLRM model, leading to the same quality, faster inference, easier deployment, and similar training times.

The trade-offs of model size in large recommendation models 100GB to 10MB Criteo-th DLRM model
Aditya Desai
Anshumali Shrivastava Department of Computer Science Rice University/Thirda.. Corp.
Department of Computer Science
Rice University Houston, Tx 77005
Houston, Tx 77005 anshumali@rae edu
apd10@rice.o
Abstract
Embedding tables dominate industrial-scale recommendation model sizes, using up to terabytes memory. A popular and the largest publicly available machine learning MLPerf benchmark on recommendation data is Deep Learning Recom- mendation Model (DLRM) trained on a terabyte of click thcc-hh data. contains 100GB of embedding memory (25+Billion parameters). DLRMs, due their sheer size and the associated volume of data, face difficulty in training, deploying for inference, and memory bottlenecks due to large embedding tables. This paper analyzes and extensively evaluates generic parameter-sharing setup (PSS) for compressing DLRM models. We show theoretical upper bounds on the learnable memory requirements for achieving approximations to the embedding table. Our bounds indicate exponentially fewer parameters suffice for good approxima- tion. To this end, we demonstrate PSS DLRM reaching 10000x compression on criteo-tb without losing quality. Such a compression, however, comes with a caveat. It requires 4.5 x more iterations to achieve the same saturation quality. The paper argues that this tradeoff needs more investigation favorable. Leveraging the small size of the compressed model, we show a 4.3x improvement in training latency leading to similar overall training times. Thus, in the tradeoff between the system advantage of small DLRM model vs. slower convergence, we show that scales are tipped towards having a smaller DLRM model, leading to the same quality, faster inference, easier deployment, and similar training times.
might be significantly
1 Introduction
Recently, recommendation systems have emerged as one of the largest workloads machine learning [1] Recommendation systems form the backbone of good experience online platforms such as e-commerce and web search, where there is a flood of information. Thus, considerable effort goes into building recommendation systems. Deep learning recommendation models give a state-of-the-art performance. However, recommendation models suffer from a critical challenge sparse features with millions of categorical values 2 B These ofate-of-the-art
g methods learn dense representation of the categorical values in parameter structure called embedding table.
Most parameters in recommendation models come from embedding tables. For example, in the popular Criteo-th ML.err benchmark model, the embedding tables are around 100GB, whereas other parameters only amount 10MB. Industrial-scale recommendation models one of the largest models built, and the size of the embedding table can go as large as hundreds of terabytes. For example, research article from Facebook discusses training model of size 50TB over 128 GPUs (3] The scale of these models leads to some unfavorable effects slower inference time, slower training time per iteration, and significant engineering challenges in training/deployment. In light of these issues, many works have investigated learning of compressed representation of embedding tables
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

using various principles (1) compositional embeddings 19] (2) exploiting power-law in the observed frequencies of tokens [0 1 [3 (3) low-rank decomposition [16], and parameter-sharing
methods 17 18] Parameter-sharing methods will be the focus of our paper.
We discuss the tradeoffs of parameter-sharing- bbasdd compressed recommendation models in terms of quality, inference, and training times. The general idea in machine learning has primarily shifted to larger models and more data. Larger models lead to better capacity, faster convergence, and better generalization. However, the community is realizing that the current route to DL is unsustainable 19]. We must proceed cautiously on this path of building larger models. The large-scale nature of DLRM comes from blown-up embedding parameters a seemingly inadvertent effect of naive usage of embedding tables. While using large embedding tables gives faster convergence in terms of iterations, the inference is slow and cumbersome. On the other hand, while compressed embedding tables are guaranteed to provide faster inference, it must be seen if they will maintain quality and suffer from slow training times due to slower convergence. In this paper, we make two contributions. First, we theoretically analyze the existence of low-memory PSS for approximating the embedding tables. We find that exponentially fewer parameters are required for good approximations. Secondly, we leverage the system advantage of small models and make to combat their slow convergence. We are able to achieve similar training times for compressed recommendation models. In our experiments, we obtain high compression for embedding tables. Additionally, the compressed DLRM models seemingly have no downside in quality and training time while improving inference latency, requiring simple hardware, and reducing costs. We summarize results below.
implementation choices
We analyze two types of approximations to the embedding table n x d. Firstly, we look at a macro- (c PSS approximation which encapsulates the effect of the entire embedding table inside the model. Here, we evaluate the preservation of inner products of the form (Ex, Ey). x, e Rd under compression. We see that with a memory of O(d(d+ + log (1 2), the embedding table can be approximated up to relative error with probability (1 5). Secondly, we look at micro- ò., p) PSS approximation which looks at how individual embeddings are approximated under compression. Here, we evaluate the preservation of computations of the form (F( .x) E R We see that with a memory of (d)g((())) lllg(1/)))))) (1/5))==) we can approximate these computations for important (high norm) embeddings up to relative approximation with probability (1 (1-d) 8). This result motivates us to evaluate the extents of compression that can be achieved for embedding tables. We can obtain 10000x compression (10MB embedding tables) the criteo-tb dataset to achieve the same target quality an order of magnitude higher compression than previously reported. This small model improves inference time, eliminates engineering challenges, and makes the models easy to deploy on low-resource devices. However, as expected, this 10 MB sized embedding model requires 4.5 epochs to train.
Consistent with a general observation in machine learning that the number of iterations required to converge decreases with an increase in parameters, the compressed recommendation models suffer from slow convergence. We observe that the higher the compression, the slower the convergence. For instance, a 1000x compressed DLRM model requires 1.9 epochs to converge, while a 10000x compressed model requires 4.5 epochs to converge. This is worrisome as training times are one of the significant aspects of the model. However, highly compressed model completely changes the storage of model parameters (c.g., multiple nodes to a single node, RAM to GPU, etc.). Hence, observe a steep decrease in the latency of embedding lookup/gradient update operations. An illustration of this tradeoff is presented in figure We investigate how this improvement in latency compensates for increased training iterations. Unfortunately, existing parameter-sharing methods do not demonstrate much training time per iteration improvement due to various reasons such as high algorithmic complexity, poor cache efficiency, and sub-optimal implementation strategies not cognizant of the underlying hardware. We evaluate various implementation choices and demonstrate that the advantage of faster training time per iteration almost compensates for the slower convergence in compressed models. For instance, for 1000x compressed model, we can lower the training latency by 1. 1.7x, making the overall training time only 1.1x original model. Similarly, we can improve the training latency 4.3x for a 10000x compressed model making the overall training time 1.04x
2 Parameter-shared embedding tables
In this section, we analyze the parameter-shared embedding tables. We start with a few definitions.

---

Cost of overall training
Training time por iteration
No Dwcsss paramotors
Cost per time
5
(Se]1
Overall training time
M-SE
Iteration to commengenca
GPuggood
Figure I: (left)This not-to-scale illustration shows tradeoffs of having larger parameters ( and hence the memory footprint of the model ). Each line has scale. As we move towards the right, memory increases, and need to have memory farther away from computational hardware. This increases the latency and, thus, inference time and training time per iteration. The energy cost also increases due to larger active hardware. We observe that the convergence becomes faster the number of parameters increases. The solid lines show the overall training time and costs, showing that more parameters are not always better. (right) An Illustrating the JLT-based setup of PSS, the computation involving E is performed in the sketched space using much lesser memory of learnable parameters M. The sketching is achieved using sketching matrix S
Definition 1 (Parameter shared setup). Under parameter shared setup (PSS) for embedding table of size d, we have set of weights M (usually [M] nd) and recovery function M RIMI (O.... 13 Rd. The embedding of a token i {O..., n....n111 1} recovered as M i).
M can be represented with any memory layout, for instance, 2D or 1D array. We denote the size of .M as |M| and memory to store M [M]. M shares the weights in M generate the entire embedding table. Note the inherent trade-off between [M] and [M]. We can make M large in order to get a very simple M (for example, M=E and M M, () (i)) or we can keep M very small the cost of complicated M (for example M [0,1] and M combines these bits to get bit representation of E). An effective PSS for embedding table E Raxd would have (lM) + IM) < nd. Also, for efficient training and inference, directly training M as a part of end-to-end training. However, the initial part of the discussion will focus approximating ground truth embedding table E.
want M(M, i) computation to be cheap. Training PSS involves
One of the key questions we want answer is how small a PSS we can create for ground truth embedding table E. That is, how small can ([M) (Ml) be. For an arbitrary E, it is impossible to accurately encode in sublinear memory. So, next, we define (c,8) approximate PSS. While creating an alternative representation of embedding table E, let see what we would like to achieve. First, we would like to preserve all the pairwise inner products between two embeddings of tokens from the embedding table. Additionally, we also want to minimally affect the subsequent computations performed on the embeddings retrieved from the embedding table. In recommendation models, the embeddings retrieved are passed through neural network layers, which will perform inner products on embeddings. Based on this, we analyze two types of (e, 8SPS.
Definition (Macro (e, (PSS). In cases where Mæ, x Rd is well defined and can be interpreted as compressed version of Ex, a PSS (M,M) is Macro (eSS)-PSS for an embedding table E if the following holds with probability (1 - -),
Vz,ye Rdd
[MMx, - (Ex, Eyyl e||z||a|lvl2
The Macro-PSS captures the effect of the entire embedding table along with the model. Consider the set of all vectors Ex,r Rdd Under our PSS compression, we want the geometry of this space of vectors be approximately preserved. Hence, in this definition, we preserve the inner products between all the vectors of the form Ex, E Rd.
3

---

Definition (Micro (c, (. PSPSS)) A PSS (M.M) is a Micro (c, P)-PSS for an embedding table if the following holds with probability (1-6).
1}; Vx E Rd ViE fo..., n- n-),, s.t where p()), , the importance factor for embedding is defined by.
p(i) > i)) (BTT))) s <c| ||r||
|
p(i)
o(E)
where a (E) is the maximum singular value of the embedding matrix.
The Micro-PSS captures the effect of compression on each embedding on the model individually. Note that it is impossible to guarantee good recovery for all the embeddings from the embedding table. Thus, we define Micro-PS5 with an additional parameter p. which measures the importance threshold of embeddings. As we will see, we can guarantee good recovery in logarithmic memory for embeddings with importance (as defined in the above definition) higher than p.
Interestingly, the Micro (c, ò, p) -PSS for embedding table E also preserves the pairwise inner products between the embeddings of different tokens, as is mentioned in the following result.
Theorem 4. Ler (M, be Micro (c. PPPSSS for embedding table E, we have,
Vi.j E (0,,..,,-11 s.t p(1), p
(2 +
Proof. Presented the appendix
Micro-PSS caters to the practical embedding tables in which few embeddings are much more important than a long tail of non-important embeddings. our formulation, we characterize this importance by the norm of the embeddings. It has found different manifestations in literature. For instance, mixed-dimensions tensor-tras train recommendation have higher rank for important tokens. In existing methods, have to pre-identify the important tokens, which is usually done by looking at the frequencies of their occurrence in data. On the contrary, in PSS, this happens naturally, and do not have to identify these tokens beforehand.
embeddings [TO] 12 13 14 15) have longer embeddings and
2.1 (e. (e,0)-PSS via Dohnson-Lindenstraass Transforms (JLT)
This section is dedicated to analyzing the existence of (e, 6SSS for embedding tables leveraging JLT matrix-basee based sketching. We will use the following definition of JLT from (0)
Definition 5. A randomly generated matrix S E Rl is a J (e, (, if with probability at least (1 - (1, f), for any J-element subset V R the following holds,
Vv1, U/ E V (Su1,, (v1, v2)1 elleillz [eall
JLT matrices directly give us a bound on the number of parameters required for Macro (E,O)-PSS. It is stated in the following theorem,
Theorem 6. Let the embedding table be E Ruxdd Consider a matrix S which is JLT 8,9")) Then M and M defined by
M=SE ((M)))
a Macro(e, 8)-SSS.
Proof. The property of S that we are interested in is called 12-subspace embeddings 20) We leverage the result from 20 that JLT 5, 9d) a 12-subspace embedding matrix. More details are in the appendix
Using independent standard normal variables to instantiate 2 we can construct a JLT(ee, 6,9 9 matrix S of size & n where k Q((d log( (e 2)) Thus the model size |M] |SE] is kd ((d(d+ + log(1 (/))))-2. Thus, with JLT matrix sketching, we can obtain a (acro-(e,6)-PS5 in space |M] Odd 22). This might be explanation to why we can achieve high compression

---

in embedding tables. We will discuss the storage cost and execution of M shortly. However, we want note here that we can relax the restrictions on the distribution of S so that the storage cost of [M] (1) and the cost of applying M is (kdd
The Macro-PSS does not discuss the individual embedding vector quality; hence, we analyze the Micro-PSS for embedding tables in the following theorem.
Theorem 7. Let the embedding table be E Rand Consider matrix S which is JLT 6, 6, +n)
Then M and M defined by
M-SE (MM)))((ee))(M))
a Micro (e, ò, P) where E, E R is a one-hot encoding of integer i (i.e ei[i] = 1 and rest all elements of e; are 0).
Proof. The complete proof is presented in the appendix. The intuition is that we want to maintain the computations of the form e: Ex, which can be Ex e Rº. So the inner products between vectors of sets A = {eo..... Ex-1] and column space of E need to be preserved. Once we observe this, we apply JLT to the set of discrete points and the 1/2 - net of column space of E, which contains less than 9d points, and get our result.
inner product between e R and
Using S constructed by standard normal variables [20) we can construct J JL 6, 9 n)) matrix S of size x where k (lloog(9* + n) + log( (1/5)) 2 2). Thus the model size [M] [SL] is kd Mdd((lo((" n) + log (1/()) p ?)). Thus, we can obtain Micro-(e, 6, p)-PSS in space Ommax(d (d, log(n)) (epepppp)) ")). Next, we will discuss the storage and execution cost of M..
Storage of JLT matrices and relaxations. In using PSS in end-to-end training. we would learn the memory M while keeping the mapping M constant. Thus, unlike in sketching for linear algebra (20). we care about the costs of (I) storage of learnable parameters [M] (2) Cost of storing mapping [M] and (3) cost of computing M(M, (M, It is clear that using standard normal JLT is not feasible due to storage of matrix S (= km) will be more expensive than storing E itself! There are lot of sparse (C, 5, f) matrices which will reduce the storage cost of S 22 23 23 showed that the matrix needs to have minimum column sparsity of (ogc""logf (1/c) and hence the cost of storing S while using JLT matrices is lower bounded by M(ne llogo///// (og(())) which can still be considerably high. However, the issue with storage costs of S can be alleviated in practice by relaxing the complete independence condition on entries of S. For example, the JLT matrix (which requires the same bound on k as given in theorem 7 suggested by (2I), which selects each entry be 0 with probability 2/3, +1 with probability 1/6 independently can be generated on-the-fly using universal hashing. (24) analyze why simpler hash functions can work well with data having enough entropy. The benefit of using universal hashing is that mapping M can now be stored in (1)) memory. Thus using the 21 sparse sketching with relaxed independence, we can store JLT((,,, f) matrix in 0(1) memory. Thus, total memory for Macro-PSS would be [M]+ + [M]
Od) whereas, total
2). ). In both cases, the cost memory for Micro-PSS would be [M|+ + [M] = OOd( ((((max( (d. log(n)) (n)))) 2 2p-2)
of applying mapping M to recover an embedding is (kd))
2.2 Training end-to-end (c, PSS for embedding table E
The above discussion directly gives us an algorithm to compute a (c, 8) PSS from a trained embedding table E. We just need to compute M = SE and while retrieving embedding compute M(M, i) (Se;) M. We can also directly train an (c,6) PSS in an end-to-end manner. The idea is to train the compressed M directly. Thus, we have a matrix of learnable weights M Rloxd Let us now look at how the forward and backward pass of embedding retrieval mapping M looks like
(irward(i) = (Se;) M
The forward function takes an integer and returns Riid array which embedding of i.
bic A) (Se;)
The backward pass takes in all the arguments of the forward pass along with the A e R Rixd which are gradients of loss with respect to the output of the forward pass. We can back-propagate further to W using the above formulation. The result of the backward pass is a k x d matrix.
If we use a sparse S such with sparse JLT, we can achieve sparse gradients of W. That is, only few W's gradient entries are non-zero. As we will see in section 3 whether to propagate sparse or dense gradients is vital implementation choice. We were able to achieve better training per iteration by exploiting this choice.
5

---

Table State of the art in embedding compression on criteo datasets. * are not PSS) )
Method
Dataset
Compression
Quality
Hashing Trick
Crieto Kaggle
4x
worse
QR Trick
Criteo Kaggle
4x
similar/slighdy worse
MD Embedding*
Criteo Kaggle
16x
better/similar
TT- Te-Rec*
Criteo kaggle/Criteo TB
112x
117 x
better/simiar
ROBE
Criteo Kaggle/Criteo TB
1000x
better/similar
2.3 Practical PSS and existing SOTA methods
Existing parameter-sharie based embedding compression methods can also seen as PSS with varied distributions over S. We state few of them below.
Hashing Trick In this method, entire embeddings for token i is drawn from a randomly hashed location. In terms of PSS, we can define the mapping function MA over 2D matrix Mn e Rixd where n.
Mr (M, 0 M.)A(1).
where h is hash function. QR dccomposition[L] [9] In this method, embedding for token is drawn in chunks from separate memory vectors. In terms of PSS, can define the mapping function M, over, say I, pieces of 2D memory M1, M2,
..M e Rexd/! as follows
My([M1, M2, MM3... (d/t) (j+ (d/1)) MM000)MM[[hpp M,
(1)
In words, the jth chunk of 5th embedding is recovered using the chunk at the location h1(1) in memory M,
In HashedNets, authors proposed mapping model weights randomly into parameter array. ROBE-Z extended this idea by hashing chunks of embedding vector instead of individual elements. In terms of PSS, we can define the mapping function M,
HashedNet [17) and ROBE-Z [8]
over ID memory M, as
MI Z: (j+1) Z] M,(A(i,)) A(i,)) +
That is, the 5th chunk tth embedding is recovered using the chunk at the location (ii,j) from Mp. Here, h N {0. MM is hash function. If we set Z then get the mapping function for HashedNet.
We can summarize the state-of-the- embedding compression in the table Our Micro-PSS theory suggests that we should be able approximate the embedding table in memory the order of (log(n)). Thus, it is important to question if 1000x compression is the best compression we can achieve on dataset like CritcoTB where we have large (~400GB sized) embedding tables. As we shall see, one of the significant roadblocks in aiming to achieve higher compression is training time. The higher the compression, the more iterations are needed to train the model to a certain quality. In the next section, discuss how overcome this bottle-neck preventing training of highly compressed PSS.
3 Road to 10000x compression with PSS
T shows the quality of embeddings for the criteo-kaggle dataset over varying compression rates across different popular recommendation models. We can see that the state-of-the-an compression of the criteo-kagg! kaggle dataset is 1000x. It is a consequence of PSS theory that larger embedding tables (prevalent in industrial scale recommendation models), such as those for critco-tb, should obtain more compression. We aim to study higher compression for large embedding tables and improve over existing SOTA (1000x).
A significant roadblock in building highly compressed embedding tables is the slow convergence. Table(3]shone that the number of iterations required converge increases with higher compression across different deep-learnig recommendation models. Specifically, for 1000x compression, it takes up to 4x iterations, and for models with 10000x compression, most models not converge in 15 epochs. If model does not converge, is hard to judge if the convergence is too slow or if its capacity consumed. In order to be able to train for larger iterations, we should able to train the highly compressed models faster. Indeed, we expect some system benefits with highly compressed models,

---

Table 2: These experiments are run on single GPU for simple embedding lookup and loss is taken to be sum of all retrieved elements with batch size of 10240 and embedding dimension=128 higher latency (worse) lower latency (better)
(a) lorward pass
(d) forward puss
4M
8M
16M
32M
64M
128M
256M
4M
8M
16M
32M
64M
128M
256M
10x
0.40
0.36
0.37
0.37
0.37
0.37
0.68
0.75
0.77
0.50
0.92
0.88
0.88
10'x
0.31
0.37
0.54
0.54
0.54
0.65
10'x
0.27
0.29
0.30
0.31
0.30
0.32
0.31
16
0.31
0.31
0.31
0.24
0.37
0.37
0.36
10*x
0.27
0.27
0.27
0.26
0.28
0.29
0.30
32
0.31
0.31
0.31
0.23
0.37
0.37
0.36
bac kward pass (sparse Talse)
(c)
backward
pass (sparse
false)
10x
2.55
4.81
9.15
17.81
35.10
69.73
chunk sizze
1.29
1.56
2.02
2.96
4.69
8.15
15.17
10"x
2.13
7.42
14.34
0.80
1.07
1.54
2.40
4.13
7.70
14.61
10
0.35
0.37
0.41
0.54
0.73
1.08
1.78
16
0.58
0.82
1.27
2.12
3.85
7,43
14.34
10'x
0.35
0.34
0.35
0.36
0.40
0.47
32
0.57
0.80
1.27
2.12
3.85
7.42
14.34
(c)
backward pass (sparse
(1) backward
passs
truc)
10x
2.67
2.53
2.52
2.51
2.51
2.52
2,54
3.17
3.41
3.56
3.68
3.94
3.98
4.00
10
2.48
2.58
2.68
2.64
2.66
2.65
2.52
2.61
2.64
2.74
2.73
2.91
2.77
2.74
10'x
2.56
2.48
2.58
2.56
2.63
2.65
2.53
16
2.50
2.66
2.54
2.53
2.52
2.51
2.53
10'x
2.52
2.53
2.45
2.55
2.60
2.54
2.62
32
2.48
2.61
2.62
2.50
2.64
2.60
2.50
as shown in figure 1 We find that existing methods cannot train highly compressed embedding tables because of the implementation choices. We propose the following implementation choices.
Hashing chunks is better than hashing elements: As suggested in 18] we find that hashing chunks instead of individual elements indeed helpful in reducing the latency of PSS systems. As shown in table dee), both forward and backward passes are affected lot by chunk sizes. It is also interesting note that the effect of chunk sizes diminishes after size 16. Hence, we can choose a chunk size of 32 for hashing.
Dense gradients suitable for highly compressed PSS Generally, in embedding tables, im- plementations use sparse gradients. Dense gradients are particularly wasteful in embedding tables, where in each iteration, only a few weights are involved in the computation. However, the same does not apply to highly compressed PSS. In the case of PSS, the algorithm for sparse gradient computation is more involved than that for dense gradients. The algorithms for sparse and dense gradient computation are specified in algorithm [ Also, the scatter operation is much faster when the memory size of embeddings smaller. Hence, dense gradients work well with higher compressions. From table bb.c). we can see that at higher compression rates, the backward pass is up to 8x faster with dense gradients as compared to sparse gradients.
Algorithm 1 Backward Pass for PSS
Require: Go e RBxd I BBxd sparse (True, False), M: compressed memory.
D Go is the gradient w.r. output, and I is the mapping matrix showing the locations from which the output was accessed
Ensure: Gú is gradient w.r.t input.
if sparse then
unique, invIdx
True)
g Tensor - (umique.shape.)
g. g.scatter_adee (nddixx f..flatten)
G sparse sparseTTensor (eensor(ooaatoon
(location = unique, values g, size = M.size)
else if dense then
9 Tensor Tensor(MM.szee) (Msize)
d.scatter_add(( I.flatten(), (latte))) end if
Forward/Backwant kernel optimizations Latency for PSS, and CUDA kernels in general, is very sensitive to the usage of shared memory, occupancy, and communication between CPU and GPUs. We also optimize our PSS code to minimize the data movement costs, implement custom kernels to fuse operations to improve shared memory usage, and optimize CUDA grid sizes to obtain the best performance.
With the implementation choices mentioned above, we are ready to train a 10000x compressed PSS for critco-tb. We implement our PSS using ROBE-style hashing.
7

---

Table 3: Criteo-kaggle: Quality and convergence on of 5 popular models vs, compression. The standard deviation of all AUC results is within 0.0009. (left) Quality of models for criteo-kaggle dataset is largely maintained till 1000x compression after which either the convergence is too slow / capacity of model drops (max 15 epochs) (right) Epochs needed to converge to target AUC set to
minimum of value achieved by original model. (max 15 epochs)
Quality of models (15 epochs max)
Epochs to reach target AUC
ORIG
10x
10"x
10"x
10'x
Tagee AUC
ORIG
10x
10'x
10'x
10%
00'x
DLRM
0.8031
0.8032
0.8029
0,8048
0.8001
0.8029
1.00
1.52
1.78
2.78
DCN
0.7973
0.7982
0.7978
0.7991
0.7967
0.7973
1.00
1.00
1.87
1.93
AUTOINT
0.7968
0.7972
0.7968
0.7987
0.7957
0.7968
1.00
1.00
2,00
2.50
14.93
DEEPFM
0.7957
0.7961
0.7953
0.7951
0.7943
0.7951
1.00
1.00
1.00
3.93
XDEAPFM
0.8007
0.8016
0.7998
0.89
0.795
0.7989
1.62
1.50
1.74
3.93
FIBINET
0.8016
0.8021
0.8011
0.8011
0.7963
0.8011
1.00
0.93
1.00
2.99
4 Experimental results on Criteo datasets
We perform experiments criteo-kaggle and criteo-th datasets in order confirm the following hypothesis,
1. Theory in section 2 dictates that important embeddings in table (E e R=xd)) can be represented in memory logarithmic in n. High compression should be possible in these datasets. Specifically, higher compression should be possible in larger embedding tables.
2. The system advantage of smaller PSS should compensate for the convergence advantage of the original model.
Criteo- -aggle and criteo-th datasets have 13 integer and 26 categorical features. criteo- kaggle data was collected over seven days, whereas the criteo-th dataset was collected over 23 days. criteo-tb is one of the largest recommendation datasets in the public domains with around 800 million token values making the embedding tables of size around 400GB. (with d == 128). criteo-kaggle is smaller and has 2 GB-sized embedding table. Note that industrial-scale models are much larger than the DLRM model we talk about here. For example, Facebook recently published model sized 50TB (3]. One can extrapolate the benefits of PSS to industrial-scall models based on this case study.
Datasets:
Models: Facebook MLPerf 2 model, available under Apache-2.0 license, for the criteo-tb dataset, achieves the target AUC (0.8025) with the embedding memory of around 100GB. This model uses maximum cap of 40M indices per embedding table, leading to a total of 204M embeddings. This model cannot be trained on single GPU (like V100) and is trained using multiple GPUs (4 or 8). For the criteo-tb dataset, we use the DLRM MLPerf model. For criteo-kaggl dataset, we use an array of saae-off-hhe-ar models DLRM2) DCN AUTOINT(5), DEEPFM (6) XDEEPFM and FIBINET. We use our PSS implementation described in 3 as a compressed model.
Quality of model vs. Excess parameters: Tables b)b and b) show the results for the two datasets across different values of compression. In table 3(b), we can see that across different models, the quality of the model is maintained until 1000x compression. As criteo-tb embedding tables are much larger than the criteo-kaggle dataset, according to the section 2 we should see larger values of compression. Indeed, we obtaín a 10000x compression without loss of quality of the model for criteo-th. This level of compression is unprecedented in embedding compression literature. These experiments validate our first hypothesis and provide new state-of of-te-of-thearr the-art embedding compression. In the rest of the section, we evaluate how the system advantage of PSS compares against the original embeddings in various aspects.
Inference time In figure 2a we compare the inference time of test data (89M samples) in the criteo-th dataset with a batch size of 16384 with PSS on a single Quadro RTX-8000 GPU and the original model on 8 Quadro RTX-8000 GPUs. The total time for inference for the original model is around 203 seconds, whereas PSS is around 97 seconds. In the original model trained on 8GPUs, embedding lookup is model-parale parallel, whereas the rest of the computation bbttmmp, top-mlp, and interactions) is data-para aatallar There is a steep improvement of over 3x in embedding time lookup as we move from a distributed GPU setup for embedding tables to PSS on single GPU, The extra computation time in PSS is much smaller than all-to-all communication costs in the original embedding table lookup. We include the time required for data distribution (initial) in bot and data-gather (final) in top-mlp timings. This communication cost high, and we can see that we are better off performing the entire computation on a single GPU.
8

---

Table 4: Criteo-th: : Quality and convergence on criteo-tb with DLRM vs. compression. The standard deviation of AUC results is within 0.0009. (1) 10*x compression also reaches the target AUC. For 10xx convergence is too slow/mode capacity reached. (max 15 epochs) J (2) Epochs required to reach 0.8025 AUC for DLRM model on criteo-th dataset. Relative time computes the ratio of A to original model
ORIG
10*x
10'x
10'x
10"x
Target reached
Yes
Yes
Yes
Yes
No
Epochs to reach target (A)
1.00
1.94
1.9
4.55
Time 1000 iteration (B)
50.6
31.04
29.6
11.6
Relative Time (C = A B)
1.19
1.11
1.04
Model training time per iteration vs excess parameters Figure 2h records the time-taken for 1000 iterations of training. While using embedding tables, can choose to back-propagate sparse or dense embedding gradients. The general idea is to use sparse gradients when very few gradients non-zeros. In the original model, prohibitive w.r.t to computation and communication. In PSS, however, we compare both modes for gradient back-propagation. We see that dense gradients perform exceptionally well at higher rates of compression. The performance of sparse gradients is constant across different compression rates as the workload is similar. Generally, it seems good to use dense gradients for PSS when the effective memory size (i.e.. the final memory of compressed embedding tables is small. It is noteworthy that the training time per iteration reduces significantly with higher compression. For example, with 1000x compression, the training time is 1.7x lesser than the time taken by the original model, whereas with 10000x compression, the time per iteration is 4.37 x lesser.
only sparse gradients using dense gradients is
Model convergence and overall time vs. excess parameters We observe a consistent trend in convergence: as the number of excess parameters in the models reduces, the convergence becomes slower. This can be seen across models and datasets as shown in tables 3(a) and 4a.. As an example, table foolesssows that with 10000x compression, we require 4.55 epochs as compared the original model's 's one epoch. This might seem unfavorable at first. However, as seen in the previous section, there significant gain in training time per iteration. As seen in table 4 this gain largely compensates for the disadvantage in terms of convergence, making the overall training times similar. For example, while we need 4.55 epochs with 10000x compressed PSS, the training time per iteration goes down by factor of 4.37. Hence, the overall training time is only 1.04x original time.
Engineering challenges and costs excess parameters: The excess parameters in recommen- dation model primarily appear due to the construction of embedding tables. The industrial-scale embedding tables can go large hundreds of terabytes. With the increase in the model's size, the model needs to be distributed across different nodes and GPUs. The complexities of efficiently running distributed model training include many considerations, such as non- non-uniform memory allocation and communication costs. The increasing literature detailing the engincering solutions to training such models is evidence of the fact that training such models require engincering ingenuity to solve the challenges involved [3] 25 26, 27 28 29]. For example, in [3], the authors detail their solution to train a model of size 50TB on a new distributed system. Similarly, in 25),, authors talk about their optimizations of recommendation models on CPU clusters. In this paper, we argue that the large-scale nature of embedding- based recommendation models appears due to embedding tables where a complete x d table with blown up n does not add value to model capacity while adding significant engineering challenges. Distributed systems also imply significant energy costs. PSS can avoid these downsides by fitting entire learnable parameters on a single GPU/node.
Discussion on the model-training speed with different methods. Most of the other compression techniques, such as pruning, low-precision quantized embeddings, TT-REC, and MD embeddings, are limited in the amount of compression they offer. For instance, methods like pruning, TT-REC, and MD embeddings cannot offer a compression more than dx for an embedding table E : n x d. Low- now-precision embeddings cannot offer compression of more than 32x. Thus, for industrial scale models where is very large, it is unlikely that these compression techniques will be able to eliminate model -arallel training. Parameter sharing methods such as hashing trick and QR embeddings, while they can give arbitrary compression in theory. have failed to maintain quality beyond 4x compression and thus cannot eliminate model-parallel training. In the case of criteo-th embedding compression in literature, TT-REC gives the compression of 117x. Paraphrasing from TT-REC [16], they are able to

---

Inference time profile
Training time profile
PHOSSSnoe)
SHDrsparse
FriDispars)
173
10.24
19/21 15.07
10:10
17 50
55.55
20.04
1X0e
24.0
25
(a) [critco-tb] DLRM inference time taken for over 89M samples with batch size of 16384. Communi- cation causes embedding lookup for original model much slower than that for PSS models when entire model is located on single GPU
124
DIEGIAL 0IG Uai
130.
1:002 nonua
(b) DLRM training time for 1000 iterations of training with mini-batch size of 2048 and SGD optimizer
train a 37.4x compressed models in 1.11x time. While, with our PSS using ROBE-Z embeddings, are able to train 10000x compressed model with 1.04x time.
5 Limitations and future work
This paper shows that we can compress the embedding tables for CTR prediction models as much 10000x. However, CTR is a specific recommendation problem with the binary output. would be interesting to see if these compression rates hold for other recommendation-rellteee such ranking retrieval. The theory presented in the paper is general, and hence, it is promising for extensions of this work to other recommendation problems or otherwise. However, we should note that the theory only evaluates the effect of embedding tables on the first layer (dot product with vector z e R of the model and does not talk about its impact on the entire recommendation model. At this stage, is difficult, if impossible, to analyze the effect of embedding compression over the whole recommendation model. We propose to extend compressed embeddings to other recommendation problems and theories for compressed embeddings in the future.
scenarios,
Conclusion
Today's prevalent idea in deep learning(DI.) is overparameterized models with stochastic iterative learning. While this paradigm has led to success in DI., the sustainability of this route to success is grim question [19.. We evaluate the value of embedding -based excess parameters in DLRM models. We conclude that compressed DLRM models are better for fast inference and are easy to train and deploy as expected. More importantly, they can also be trained in the same overall time despite having slower convergence rate. The critical observation is that compressed models enjoy system advantage they can exploit, which reduces time per iteration. The paper also highlights that the correct way to compare the quality of models of highly different sizes is to run the models for equal time rather than equal iterations.
7
Acknowledgements
This work was supported by National Science Foundation SHF-2211815, BIGDATA-1838177 ONR DURIP Grant, and grants from Adobe, Intel, Total, and VMware. We also thank Ben Coleman for giving helpful writing comments on the paper.
References
[1] Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Mark Hempstead, Bill Jia, The architectural implications of facebook's 's dnn- based personalized recommendation. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pages 488-501. IEEF, 2020.
[2] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sun- daraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanic Pereira, Xianjie Chen, Wenlin Chen,
10

---

Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. arXiv:1906.000% 2019.
[3] Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, et al. High-performance distributed training of large-scale deep learning recommendation models, arXiv preprimt arXiv:2104.05/56 2021.
[4] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD, 2017.
[5] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1161-1170, 2019.
[6] Huifeng Guo, Ruiming Tang, Yumming Ye, Zhenguo Li, and Xiuqiang He. Deepfmm: dactorizationmmachee based neural network for ctr prediction. arXiv preprint arXiv:1703.042477 2017.
[7] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang. Fibinet: combining feature importance and bilinear feature interaction for throuhh rate prediction. In Proceedings of the 13th ACM Conference Recommender Systems, pages 169-177, 2019.
[8] Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1754-1763, 2018.
[9] Hao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, and Jiyan Yang. Compositional embeddings using complementary partitions for memory-efficien recommendation systems. arXiv:1909.02101 2019.
[10] Antonio Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, and James Zou. Mixed dimension embeddings with application to memory-efficient recommendation systems. arXiv:1909.1 1880, 1810, 2019.
[11] Manas R Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang, Jay K Adams, Pranav Khaitan, Jiahui Liu, and Quoe V Le. Neural input search for large scale recommendation models. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 2387-2397, 2020.
[12] Haochen Liu, Xiangyu Zhao, Chong Wang, Xiaobing Liu, and Jiliang Tang. Automated embedding size search in deep recommender systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2307- 2316, 2020.
[13] Siyi Liu, Chen Gao, Yihong Chen, Depeng Jin, and Yong Li. Learnable embedding sizes for recommender systems. arXiv preprint arXiv:2101.07577, 2021.
[14] Weiyu Cheng, Yanyan Shen, and Linpeng Huang. Differentiable neural input search for recommender systems. arXiv preprint arXiv:2006. 04466, 2020.
[15] Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Jiliang Tang. Autoemb: Automated embedding dimensionality search in streaming recommendations, arXiv preprint arXiv:2002. 1255, 2020.
[16] Chunxing Yin, Bilge Acun, Carole-Jean Wu, and Xing Liu. Tt-rec: Tensor train compression for deep learning recommendation models. Proceedings of Machine Learning and Systems, 2021.
[17] Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International conference on machine learning, pages 2285-2294 PMLR, 2015.

---

[18] Aditya Desai, Li Chou, and Anshumali Shrivastava. Random offset block embedding array (robe) for criteotb benchmark miperf drrm model: 1000 compression and 2.7x faster inference. arXiv preprini arXiv: 1000091 2021.
[19] Neil C Thommson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. Deep learning's diminishing returns: The cost of improvement is becoming unsustainable. IEEE Spectrum, 58(10):50-55, 2021.
[20] David P Woodruff. Sketching tool for numerical linear algebra. arXiv preprint arXiv:1411.435) 2014.
[21] Dimitris Achlioptas. Database-friendll random projections: Johnson-lindenstraus with binary coins. Journal of computer and System Sciences, 66(4))671-687, 2003.
[22] Anirban Dasgupta, Ravi Kumar, and Tamás Sarlós. sparse johnson: Lindenstrauss transform.
In Proceedings of the forty-second ACM symposium on Theory of computing, pages 341-350, 2010.
[23] Daniel M Kane and Jelani Nelson. Sparser johnson- lindenstrauus transforms. Journal of the ACM (JACM), 61(1):1-23 2014.
[24] Michael Mitzenmacher and Salil P Vadhan. Why simple hash functions work: exploiting the entropy in a data stream. In SODA, volume 8, pages 746-755. Citeseer, 2008.
[25] Dhiraj Kalamkar, Evangelos Georganas, Sudarshan Srinivasan, Jianping Chen, Mikhail Shiryaev, and Alexander Heinecke. Optimizing deep learning recommender systems training on cpu cluster architectures. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15. IEEE, 2020.
[26] Biye Jiang, Chao Deng, Huimin Yi, Zelin Hu, Guorui Zhou, Yang Zheng, Sui Huang, Xinyang Guo, Dongyue Wang, Yue Song, et al. Xdl: industrial deep learning framework for high- dimensional sparse data. Proceedings of the 1st International Workshop on Deep Learning Practice for High DithhniDDnns Sparse Data, pages 1-9, 2019.
[27] Jongsoo Park, Maxim Naumov, Protonu Basu, Summer Deng, Aravind Kalaiah, Daya Khudia, James Law, Parth Malani, Andrey Malevich, Satish Nadathur, et al. Deep learning inference in facebook data centers: Characterization, performance optimizations and hardware implications. arXiv preprini 1eXXiv:8//1.099 2018
[28] Maxim Naumov, John Kim, Dheevatsa Mudigere, Srinivas Sridharan, Xiaodong Wang, Whitney Zhao, Serhat Yilmaz, Changkyu Kim, Hector Yuen, Mustafa Ozdal, et al. Deep learning training in facebook data centers: Design of scale-up and scale-out systems. arXiv preprint arXiv:2003.095/8 2020.
[29] Jie Amy Yang. Jongsoo Park, Srinivas Sridharan, and Ping Tak Peter Tang. Training deep learning recommendation model with quantized collective communications. 2020.
12


Reviews:
[{"review_id": "yeYFv101Xb-", "reviewer": "Reviewer_HDZP", "summary": "This paper focuses on the problem of compressing large embedding tables in recommender systems. The authors provide a theoretical analysis of the memory requirements and approximation error of compressing embedding tables under the defined parameter sharing setup. They also show the increase of compression rate accelerates the training of each iteration while leading to more iterations until model convergence. The conclusion is that compressed models can be trained in the same overall time and are faster for inference compared with uncompressed ones.", "questions": "Q1. Please see W1, what do you think of it?\n\nQ2. Please see W2, what are the existing implementation choices for PSS methods?\n\nQ3. Please see W3, can you summarize the main contribution of the paper?", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1. This paper is mostly well-organized and clearly presented. Figures are helpful, e.g. Figure 1 does help to clarify the tradeoffs of training larger embedding tables.\n\nS2. The authors conduct experiments on real-world datasets. The results are sufficient to show the compensation of faster training speed for slower convergence of compressed models.\n\nS3. The conclusions about compressed models are intuitive and are beneficial to the application of compressed models in real systems.\n\nW1. The theoretical analysis of the memory requirements and approximation error is based on the defined parameter sharing setup. It is not clear whether parameter-sharing methods have become state-of-the-art methods of embedding compression. Thus the impact of the conclusions may be limited.\n\nW2. Section 3 introduces several implementation choices for PSS, but the originality is not clear since the choices used by existing PSS methods are not mentioned.\n\nW3. Due to the above two points, the contribution of this paper is not quite substantial and should be better summarized.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "xkHWBJshUsH", "reviewer": "Reviewer_XHj9", "summary": "The paper studies how random projections on embedding matrices could lead to heavy compressions of practical deep models. They picked a well-associable Criteo-TB dataset with binary loss functions and discovered that most SOTA baselines such as DLRM could be compressed by 10k times without observable performance degradation. The paper also argue that previous failures on random projection could be due to a lack of sufficient training epochs. Some theoretical background was introduced as well as a few other papers on related implementation details.", "questions": "Sections 2.1-2.3 on theoretical analysis exist without a purpose. Many things should be discussed. For example:\n* Line 107. Norm of recovered embedding from PSS. Why does it matter to have l2-loss in addition to inner-product loss? Do you expect the embedding vectors to be unit-normed?\n* Line 123. The appearance of singular value is not motivated. Would scale of the embedding vectors matter? What actual values do you observe in trained models?\n* In general, what actual knowledge could a reader gain besides saying that \"random projection in general is a good idea\"?\n\nI did not go into the details of Section 2.5. According to the authors, the most successful model runs without sparse hashes, so I don't know how I should appreciate Sparse-JL. Also, what does QR decomposition do here?", "limitations": "The paper did not discuss limitations beyond the presented variants of a single dataset. I would suspect that random projection may not generally work well in retrieval-type of problems, such as search and masked language modeling.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Originality: While Johnson Lindenstrauss Transform tells us that random projections from sparse vectors could just work, showing that the simple idea actually works is a good piece of knowledge to have.\n* Originality: The paper is based on well-known datasets and SOTA methods. The results appear associable.\n\nWeaknesses:\n* Significance: In my experience, random projections do not tend to work well in retrieval problems with cross-entropy loss. This paper is only limited to binary prediction losses in some variants of the same dataset. The authors may want to discuss limitations of their work, esp. with respect to retrieval problems such as BPR loss in recommender systems or XE loss in masked-language modeling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "j1J_RXiGXRv", "reviewer": "Reviewer_foGx", "summary": "This paper proposed a new compression method, Parameter Shared Setup (PSS), and its training method to compress the large-scale recommendation embedding table. The proposed new compression can achieve higher compression rate on larger dataset with almost no decrease in quality and training speed. It can achieve even 10000 × compression on criteo-tb DLRM model, which is much better than previous method. It provides both theoretical and experimental proof.\n\n", "questions": "As mentioned above, more speed experiment comparing with other baselines should be included.", "limitations": "The compression should consider how to increase the number of data point without retraining. That is also valuable especially for recommendation", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength.\n1. The proposed method can compress the embedding table in memory of the order of O(logn). Larger dataset will benifit more from logn. Criteo-tb DLRM model can be compressed 10000 times.\n2. Previous methods show that higher compression rate will suffer from logner training time. But in this paper, the larger number of epochs needed for convergence can be compensated by the faster iteration, which keeps the total training time comparable with others.\n3. This paper provide both theoretical and experimental explanation for the performance.\n\nWeakness.\nThe speed experiment needs more comparison  between different compression methods. Now the author only consider the comparison among different compression rate by their own method. More comparison is required to show the speed performance. The dense gradients should be effective on ROBE, if ROBE also requires training and can compress it 1000 times. Now the speed experiment only supports the longer epochs and faster single integration but can not show the advantage of the new methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept