Learnable Embedding sizes for Recommender Systems
Authors: Siyi Liu
Chen Gao
Yihong Chen
Depeng Jin
Yong Li
Venue: ICLR 2021 Poster (2021)
Keywords: Recommender Systems
Deep Learning
Embedding Size

Abstract: The embedding-based representation learning is commonly used in deep learning recommendation models to map the raw sparse features to dense vectors. The traditional embedding manner that assigns a uniform size to all features has two issues. First, the numerous features inevitably lead to a gigantic embedding table that causes a high memory usage cost. Second, it is likely to cause the over-fitting problem for those features that do not require too large representation capacity. Existing works that try to address the problem always cause a significant drop in recommendation performance or suffers from the limitation of unaffordable training time cost. In this paper, we proposed a novel approach, named PEP (short for Plug-in Embedding Pruning), to reduce the size of the embedding table while avoiding the drop of recommendation accuracy. PEP prunes embedding parameter where the pruning threshold(s) can be adaptively learned from data. Therefore we can automatically obtain a mixed-dimension embedding-scheme by pruning redundant parameters for each feature. PEP is a general framework that can plug in various base recommendation models. Extensive experiments demonstrate it can efficiently cut down embedding parameters and boost the base model's performance. Specifically, it achieves strong recommendation performance while reducing 97-99% parameters. As for the computation cost, PEP only brings an additional 20-30% time cost compare with base models. 

Published as a conference paper at ICLR 2021
LEARNABLE EMBEDDING SIZES FOR RECOMMENDER SYSTEMS
Siyi Liu', Chen Gao?, Yihong Chen', Depeng Jin', Yong Li
UUiiersitty of Electronic Science and Technology of China, Chengdu, China
Beijing National Research Center for Information Science and Technology. Department of Electromic Engineering, Tsinghua University, Beijing 100084, China University College London, London, United Kingdom
ssui. Pirii.lul122 2 yma il. com, gel 60ma i 1 t iiguua.ggu.... edu. cn, yihong-chen@out.oom com, {jindp, 1i 1iyong07 singhuua. edu.
ABSTRACT
The embedding -bsedd representation learning is commonly used in deep learning recommendation models map the sparse features dense vectors. The traditional embedding manner that assigns a uniform size to all features has two issues. First, the numerous features inevitably lead to gigantic embedding table that causes high memory usage cost. Second, it is likely to cause the over- over-fitting problem for those features that do not require Existing works that try to address the problem always cause significant drop in recommendation performance or suffer from the limitation of unaffordable train- ing time cost. In this paper, we propose novel approach, named PEF[] (short for Plug-in Embedding Pruning), to reduce the size of the embedding table while avoiding the drop of recommendation accuracy. PEP prunes embedding parame- ter where the pruning (hreshold(s) can be adaptively learned from data. Therefore automatically obtain a mixed-dimension redundant parameters for each feature. PEP a general framework that can plug in various base recommendation models. Extensive experiments demonstrate it can efficiently cut down embedding parameters and boost the base model's 's per- formance. Specifically, achieves strong recommendation performance while reducing 97-99% parameters. As for the computation cost, PEP only brings an additional 20-30 20-30% time cost compared with base models.
large representation capacity.
sabeedding-shhemee by pruning
INTRODUCTION
The success of deep learning-basee recommendation models (Zhang et al. 2019) demonstrates their advantage in learning feature representations, especially for the most widely-used categorical fea- tures. These models utilize the embedding technique to map these sparse categorical features into real-valued dense vectors extract users' preferences and items' characteristies. The learned vec- tors are then fed into prediction models, such as the inner product in FM (Rendle) (0100), self- attention networks in AutoInt (Song al. 2019), obtain the prediction results. The embedding table could contain large number of parameters and cost huge amounts of memory since there are always a large number of raw features. Therefore. the embedding table takes the most storage cost.
A good case in point is the YouTube Recommendation Systems (Covington et al. 2016). It demands tens of millions of parameters for embeddings of the YouTube video IDs. Considering the increas- ing demand for instant recommendations in today's 's service providers, the scale of embedding tables becomes the efficiency bottleneck of deep learning recommendation models. On the other hand, features with uniform embedding size may hard to handle the heterogeneity among different fea- tures. For example, some features are more sparse, and assigning too large embedding sizes likely
"Chen Gao the Corresponding Author. The work performed when Siyi Liu is an intern Tsinghua University. Codes are available htttp://githhb..... htt/:///////oooooo///o//////////
81 zes wes--ffr--ess -or--

---

Published as a conference paper at ICLR 2021
to result in over-fitting issues. Consequently, recommendation models tend to be sub-optimal when embedding sizes are uniform for all features.
The existing works towards this problem can be divided into two categories. Some works Zhang ctt al, 2020, Shi et al. 2020; Kang et al. 2020) proposed that some closely-related features can
share parts of embeddings, reducing the whole cost. Some other works (Joglekar et al.) 2020 Zhao
eet al. 2020b a Cheng et al. 2020) proposed to assign embeddings with flexible sizes to different
features relying on human-designed rules (Ginart et al. 2019) or neural architecture search (Joglekar et al. 2020 Zhao et al. 2020b a Cheng et al. 2020). Despite a reduced embedding size table, these methods still cannot perform well on the two most concerned aspects, recommendation performance and computation cost. Specifically, these methods either obtain poor recommendation performance or spend a lot of time and efforts getting proper embedding sizes.
In this paper, to address the limitations of existing works, we proposed a simple yet effective pruning-based framework, named Plug-in Embedding Pruning (PEP), which can plug in various embedding- based recommendation models. Our method adopts direct manner-pruning those un- necessary embedding parameters in one shot-to reduce parameter number.
Specifically, we introduce the learnable threshold(s) that can be jointly trained with embedding pa- rameters via gradient descent. Note that the threshold utilized to determine the importance of each parameter automatically. Then the elements in the embedding vector that are smaller than the threshold will be pruned. Then the whole embedding table is pruned to make sure each feature has suitable embedding size. That is, the embedding sizes flexible. After getting the pruned em- bedding table, we retrain the recommendation model with the inspiration of the Lottery Ticket Hy- pothesis (LTH) (Frankle & Carbin| 2018), which demonstrates that subnetwork can reach higher accuracy compared with the original network. Based on flexible embedding sizes and the LTH, our PEP can cuts down embedding parameters while maintaining and even boosting the model's recommendation performance. Finally, while there is always trade-off between recommendation performance and parameter number, our PEP can obtain multiple pruned embedding tables by run- ning only once. In other words, our PEP can generate several memory-efficiee once-for-all, which can well handle the various demands for performance or memory-effciieng in real-world applications. We conduct extensive experiments on three public benchmark datasets: Criteo, Avazu, and MovieLens- IM. The results demonstrate that our PEP can not only achieve the best performance compared with state-of-the-as baselines but also reduces 97% 99% parameter usage. Further studies show that our PEP is quite computationally-effiiien requiring a few addi- tional time for embedding-size learning. Furthermore, visualization and interpretability analysis on learned embedding confirm that our PEP can capture features' intrinsic properties, which provides insights for future researches.
embedding matrices
2 RELATED WORK
Existing works try to reduce the embedding table size of recommendation models from two perspec- tives, embedding parameter sharing and embedding size selection.
2.1 EMBEDDING PARAMETER SHARING
The core idea of these methods is to make different features re-use embeddings via parameter shar- ing. KKng et al. 2020) proposed MGQE that retrieves embedding fragments from a small size of shared centroid embeddings and then generates final embedding by concatenating those frag- ments. IZhang et al. (2020) used the double-hash trick to make low-frequency features share a small embedding-tail while reducing the likelihood of a hash collision. (Shi al.] (2020) tried to yield unique embedding vector for each feature category from small embedding table by combining multiple smaller embedding (called embedding fragments). The combination is usually through concatenation, add, element wise multiplication among embedding fragments.
However, those methods suffer from two limitations. First, engineers are required to carefully design the parameter-sharing ratio to balance accuracy and memory costs. Second, these rough embedding- sharing strategies cannot find the redundant parts in the embedding tables, and thus it always causes drop in recommendation performance.
2

---

Published as a conference paper at ICLR 2021
Comparison of our PEP and existing works (AutoInt is a base recommendation model and others are embedding-parann embeeding--rraattee-rruiiiiiii parameter--dduttionn methods.) Parameter Number
Table 1:
Method
Performance
Computation Cost
Autoint (Song et al.
2019)
MDE (Gnaat et al. 2019)
NIS (Joglekar ct al. 2020
X
X
DartsEmb (hao al. 2020b
X
DNIS (Cheng al.
2020)
X
Our PEP
In this work, our method automatically chooses suitable embedding usages by learning from data. Therefore, engineers can be free from massive efforts for designing sharing strategy, and the model performance can be boosted via removing redundant parameters and alleviating the over-fitting is- sue.
2.2 EMEBDDING SIZE SELECTION
The embedding saabedding-sharing methods assign uniform embedding sizes to every feature, which may still fail to deal with the heterogeneity among different features. Recently, several methods proposed new paradigm of mixed-dimension embedding table. Specifically, different from assigning all fea- tures with uniformed embedding size, different features can have different embedding sizes. MDE (Ginart et al., 2019) proposed a human-definee rule that the embedding size of a feature is pro- portional its popularity. However, this rule-based method is too rough and cannot handle those important features with low frequency. Additionally, there are plenty of requiring a lot of truning efforts. Some other works (Joglekar et al. 2020; Zhaoo et al.] [0200b a Cheng et al. 2020) assigned adaptive embedding sizes to different features, relying on the advances Neural Architecture Search (NAS) (Elsken et al. 2019), significant research direction of Au- tomated Machine Learning (AutoML) (Hutter et al. 2019). NIS (Joglekar et al. 2020 used a reinforcement learning-basse algorithm to search embedding size from a candidate set predefined by human experts. A controller is adopted to generate the probability distribution of size for spe- cific feature embeddings. This was further extended by DartsEmb (Zhao et al., 2222bb by replacing the reinforcement learning searching algorithm with differenniablee search (Liu et al.) 2018). Au- toDim (Zhao et al. 2020a) allocated different embedding sizes for different feature fields, rather than individual features, in same way DartsEmb. DNIS (Cheng et al. 2020 made the candi- date embedding size to be continuous without predefined candidate dimensions. However, all these NAS bas-da methods require extremely high computation costs in the searching procedure. Even for methods that adopt differential architecture search algorithms, the searching cost is still not afford- able. Moreover, these methods also require great effort in designing proper search spaces.
in MDE
Different from these works, our pruning-based method can be trained quite efficiently and does not require any human efforts in determining the embedding--Se candidates.
3 PROBLEM FORMULATION
Feature-based recommender system is commonly used in today's information services. In general, deep learning recommendation models take various raw features, including users' profiles and items' attributes, input and predict the probability that user like item. Specifically, models take the combination of user's profiles and item's 's attributes, denoted by x, as its' input vector, where x is the concatenation of all fields that could defined follows:
X
X1; X2; XM)
(1)
where M denotes the number of total feature fields, and x; is the feature representation (one-hot vector in usual) of the i-th field. Then for x;, the embedding based recommendation models generate corresponding embedding vector vi via following formulation:
Vi Vixis
(2)
II is also known as click-through prediction.
3

---

Published as a conference paper at ICLR 2021
d
1
02
04
04
V1
0.3
02
-00
04
Interaction unction
033
033
4.4
0.1
02
Pruning
a3
04
Q2
y
33
001
0.001.22
V3
((5) 0.15
VM 01
0.
03
VM
02
05
433
V
SV, Figure 1: The basic idea of PEP.
where V1 E Rm4xd is an embedding matrix of i-th field, n, denotes the number features in the i-th field, and d denotes the size of embedding vectors. The model's embedding matrices V for all fields of features can be formulated as follows,
= (V1 V2, V2....,
(3)
The prediction score could calculated with V and model's other parameters (mainly refer to the parameters in prediction model) & as follows,
x V.
(4)
where is the predicted probability and o represent the prediction model, such as FM (Rendle 2010) or AutoInt (Song et al.) 2019). As for model training, to learn the models parameters, the optimizer minimizes the training loss follows,
min L(V, D),
(5)
where D -= (xxx)) y) represents the data fed into the model, denotes the input feature, y denotes the ground truth label, and L is the loss function. The Logloss is the most widely-used loss function in recommendation tasks (Rendle 2010 Guo al. 2017 Song al. 2019) and calculated as follows,
(us log (0,) (1 vs) log (1 --))),
where [D] is the total number of training samples and regularization terms are omitted for simplifi- cation.
METHODOLOGY
4.1
LEARNABLE EMBEDDING SIZES THROUGH PRUNING
As mentioned above, a feasible solution for memory-effiiintt embedding learning is to automati- cally assign different embedding sizes d; for different features embeddings v,. which our goal. However, to learn d, directly is infeasible due its discreteness and extremely-larg optimization space. To address it, we propose novel idea that enforce column-wise sparsity V. which equiva- lently shrinks the embedding size. For example, as it shown Figure the first value in embedding vi is pruned and set to zero, leading to a dj = dj 1 embedding size in effect, Furthermore, some unimportant feature embeddings, like V3. dropped by set all values to zereFl Thus our method can significantly cut down embedding parameters. Note that the technique of sparse matrix storage help to significantly save memory usage (Virtanen al. 2020).
In such a way, we recast the problem of ambedding-size selection into learning column-wise sparsity for the embedding matrix V. To achieve that, we design sparsity constraint V as follows,
min L, s.t. || k
where ||. llo denotes the Lo-norm, i.e. the number of non-zeros and k is the parameter budget, which is, the constraint on the total number of embedding parameters.
*Our PEP benefit from such kind of reduction, as demonstrated in Section 5.1 5.3 and 5.4
4

---

Published as a conference paper at ICLR 2021
However, direct optimization of Equation is NP-hard due to the non-convexity of the Lo-norm constraint. To solve this problem. the convex relaxation of Lo-norm. called L norm, has been studied for a long time (Taheri & Vorobyov| 2011 Beck & Teboulle 2009 Jain et al. 2014). For example, the Projected Gradient Descent (PGD) Jain et al.) 2014) in particular has been proposed project parameters to L, ball to make the gradient computable in almost closed form. Note that the L1 ball projection is also known as Soft Thresholding (Kusupati et al. 2020). Nevertheless, such methods are still faced with two major issues. First, the process of projecting the optimization values onto L1 ball requires too much computation cost, especially when the recommendation model has millions of parameters. Second, the parameter budget l requires human experts to manually set at a global level. Considering that features have various importance for recommendation, such operation is obviously sub-optimal. To tackle those two challenges, inspired by Soft Threshold Reparameter- ization (Kusupati et al. 2020), we directly optimize the projection of V and adaptively pruning the V via learnable threshold(s) which can be updated by gradient descent. The re- parammttrrizaiion of V can be formulated follows,
V V= s) = sign(() (e)RLL -s())),
(8)
where V E RNxx denotes the re-parameterized embedding matrix, and g serves pruning threshold value, of which sigmoid function a simple yet effective solution We set the initial value of trainable parameter E R (called Sinit) make sure that the threshold(s) g start close to zero. The sign(-) function converts positive input value to and negative input value to -1, and zero input will keep unchanged.
As S(V, 8) is applied to each element of V, and thus the optimization problem in Equation 5 could be redefined follows,
min ((S(V, O,
(9)
Then the trainable pruning parameter could be jointly optimized with parameters of the recommen- dation models o, through the standard back-propagation Specifically, the gradient descent update equation for V at t-th step is formulated as follows.
((+1) (t) no VsV, ( (s(V() s), + VvS(V.s),
(10)
where The is f-th step learning rate and @ denotes the Hadamard product. To solve the non- differentiablilty of 8.),, we use sub-gradient to reformat the update equation as follows,
v(++) + v(O - - VSV. ( .)) D) @ 1 (((((),)) +))+
(11)
where 1 denotes the indicator function. Then, as long as we choose a continuous function g in S.)), then the loss function & (S(V(), s), D) would be continuous for 8. Moreover, the sub-gradient of L with respect can be used of gradient descent on s as well.
Thanks to the automatic differentiation framework like TensorFlow (Abadi et al., 2016) and PyTocc (aaszke et al.) 2019), we are free from above complex gradient computation process. Our PEP code can be found in Figure 7 of Appendix A.2 As we can see, it is quite simple to incorporate with existing recommendation models, and there is no need for us to manually design the back- propagation process.
4.2 RETRAIN WITH LOTTERY TICKET HYPOTHESIS
After pruning the embedding matrix V to the target parameter budget P, we could create a binary pruning mask m E 1) that determines which parameter should remain or drop. Then we retrain the base model with a pruned embedding table. The Lottery Ticket Hypothesis (Frankle & Carbin 2018) illustrates that a sub-network in a randomly-initialzzed dense network can match the original network, when trained in isolation in the same number of iterations. This sub-network is called the winning ticket. Hence, instead of randomly re-initializing the weight, we retrain the base model while re- re-initializing the weights back to their original (but masked now) weights @ Vo- This initiation strategy can make the training process faster and stable, keeping the performance consistent, which is shown in appendixAA66
Mooe details about how to choose suitable g(s) are provided in Appendix A.I
5

---

Published as a conference paper at ICLR 2021
MovieLens 1M
MavicLens 1M
MovieLens 1M
0.86
0.855
0.850
0.85
AUC
0.845
UE
MGQE
MSQE
0.84
MGQE
0.83
0.840
MDE
DartsEmb
DarisCmb
DartsEmb
0.82
PEP
0.835
PEP
0.83
PEP
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Parameter (x10=)
Parameter (x10*)
Parameter (x10P)
(a) FM
(b) DeepFM
(c) AutoInt
Figure 2: AUC-# Parameter curve on MovieL IM with three base models.
4.3 PRUNING WITH FINER GRANULARITY
Threshold parameter in Equation 8 is set to scalar that values of every dimension will have the same threshold value. We name this version as global wise pruning. However, different dimensions in the embedding vector v; may have various importance, and different fields of features may also have highly various importance. Thus, values in the embedding matrix require different sparsity budgets, and pruning with global threshold may not be optimal. To better handle the heterogeneity among different features/dimensionss in V, we design following different threshold tactic with dif- ferent granularities.
(1) Dimension Wise: The threshold parameter s as vector E Rd. Each value in an embedding will be pruned individually. (2) Feature Wise: The threshold parameter is defined as a vector s E RVV. Pruning on each features' embedding could be done in separate ways. (3)
Wise: this variant combines the above genre of threshold to obtain the finest granularity pruning. Specifically, thresholds are set as a matrix E R Nxd.
5 EXPERIMENTS
Dataset. We use three benchmark datasets: MovieLens- 1M, Criteo, and Avazu, in our experiments.
Metric. We adopt AUC (Area Under the ROC Curve) and Logloss to measure the performance of models.
Baselines and Base Recommendation Models. We compared our PEP with traditional UE (short for Uniform Embedding). We also compare with the recent advances in flexible embedding sizes: MGQE (Kang et al. 2020), MDE (Ginart et al. 2019). and DartsEmb (Zhao et al. 2020b 5 We deploy PEP and all baseline methods to three representative Teature-based recommendation models: FM (Rendle 2010),, DeepFM (Guo et al. 2017), and AutoInt (Song et al. 2019), compare their performance
5.1
RECOMMENDATION ACCURACY AND PARAMETER NUMBER
We present the curve of recommendation performance and parameter number in Figure 2 and 4 including our method and state-of-the-an baseline methods. Since there is a trade-off between rec- ommendation performance and parameter number, the curves are made of points that have different sparsity d
Our method reduces the number parameters significantly. Our PEP achieves the highest reduce-ratio of parameter number in all experiments, especially in relatively large datasets (Critco and Avazu). Specifically, in Critco and Avazu datasets, our PEP-O can reduce 99.90% param- eter usage compared with the best baseline (from the 106 level to the 10 level, which is very signiffccntt.). Embedding matrix with such low parameter usage means that only hundreds of em- beddings are non-zero. By setting less-important features' embedding to zero, our PEP can break the limitation in existing methods that minimum embedding size is one rather than zero. We con- duct more analysis the MovieLens dataset in Section 5.3 and 5.4 to help us understand why our method can achieve such an effective parameter decreasing.
'We do not compare with NIS (Joglekar et al. 2020) since it has not released codes and its reinforcement-
learning based search is really slow.
"More details of implementation and above information could be found in Appendix A.4
We report five points of method, marked from 0 4.

---

Published as a conference paper at ICLR 2021
Criteo
0.8000
0.804
0.802
0.7975
0.8000
0.7950
0.7925
UE
UE
0.738
0,7900
MGE
MIDD
-
VIDE
MA0DE
0.796
MGDE
0.7875
DartsEmo
DortsBmb
0.794
PEP
PEP
0.792
10
16f
109
10
Parameter
(a) FM
(b) DeepFM
(c) AutoInt
Figure 3: AUC-# Parameter curve on Criteo with three base models.
0.766
Avazu
Avazu
Avazu
0.7675
0.768
0.7650
0.762
0.7625
0.766
0.760
0.758
0.7575
- DE
VIDE
VISDE
0.764
MGQE
UE
0.756
DarbEmb
DurtEmb
0.762
PEP
10a
16"'
10c
0.7525
10
10
107
Parameter
Parameter
(a) FM
(b) DeepFM
(c) AutoInt
Figure 4: AUC-# Parameter curve Avazu with three base models.
Our method achieves strong recommendation performance. Our method consistently outper- forms the uniform embedding based model and achieves better accuracy than other methods in most cases. Specifically, for the FM model on the Criteo dataset, the relative performance im- provement of PEP over UE is 0.59% and over Dmr is 0.24% terms of AUC. Please note that the improvement of AUC or Logloss at such level is still considerable for feature-based rec- ommendation tasks (Cheng et al., 2016 Guo et al. 2017), especially considering that we have reduced a lot of parameters. A similar improvement on other datasets and other recommendation models. It is worth noting that our method could keep strong AUC performance under extreme sparsity-regime. For example, when the number parameters is only in the 103 level (a really small one), the recommendation performance still remarkably outperforms the Linear Regression model (more details can be found in Appendix A.5).
also be observed from the experiments
To summarize it, with the effectiveness of recommendation accuracy and parameter-size reduction. the PEP forms a frontier curve encompassing all the baselines at all the levels of parameters. This verifies the superiority that our method can handle different parameter-size budgets well.
5.2 EFFICIENCY ANALYSIS OF OUR METHOD
As is shown in Section 5.1 learning suitable parameter budget can yield a higher-accuracy model while reducing the model's parameter number. Nevertheless, it will induce additional time find apposite sizes for different features. In this section, study the computational cost and compare the runtime of each training epoch between PEP and DartsEmb on the Criteo dataset. We implement both models with the same batch size and test them on the same platform.
The training time of each epoch on three different models given in Table We can observe that our PEP's additional computation-cost is only 20% to 30%, which is acceptable compared with the base model. DartsEmb, however, requires nearly double computation time to search good embedding size in its bi-level optimization process. Furthermore, DartsEmb needs to search multiple times to fit different memory budgets, since each one requires a complete re-running. Different from DartsEmb, application scenarios, only a single running. As a result, our PEP's s time cost on embedding size search can be further reduced in real-world systems.
PEP can obtain several embedding schemes, which can be applied in different

---

Published as a conference paper at ICLR 2021
Table 2: Runtime of each training epoch
Criteo between base model, DartsEmb, and our PEP.
Runtime (Second)
FM
DeepPM
AutoInt
Avg. time increase
Base Model
1,039
1,222
1,642
0
DartsEmb
2,239
2,285
3.154
98.02%
PEP
1,341
1,525
1,963
24.47%
without 261256
with
2.0
incease Eousl Decrewse
2o00 ce 0.222 033 0012 0.23
0.002
0.022
0033
0.002
toreet gendar
0.55
0.
0.922
GUSE 0.061
C.
Sicipatioo
Cip code
0:33 0.300
0:003
ThE
2.12 BL44 0.226 EUB
100 06 0.953
0.022 0.61 OAL 6017
0 0.013 0.001
elsarel prews
5.00
0 000 2,00 6.03 0.006
0.012 0.00 0033
0.001 0.001
(a) VVT on original embedding
(b) VVT on sparse embedding
(c) Variation of matrix values be- tween original and sparse embod- ding
Figure 5: Interpretable analysis on Moviel 1M dataset.
Probability of Features
10-2
1.0
1.0
10-1
0.8
10-4
Average Sparsic
0.8
10-5
0.6
Epoch
0.6
Sparsis
10-4
0.4
10
[52, 139)
75
0.2
10-a
0.2
[139, 378)
100
0.0
10-9
0.0
254
548
1104
6188
10a
10
10t
10
10
10
Epoch
Feature Frequency
(a) PDF of feature frequencies of (b) Sparsity trajectory generated (c) Sparsity heatmap generated by MovieLens- IM dataset
by PEP on FM
PEP on FM
Figure 6: Correlation between Sparsity and Frequency.
5.3 INTERPRETABLE ANALYSIS PRUNED EMBEDDINGS
The feature-basse based recommendation models usually apply the embedding technique to capture two or high order feature interactions. But how does our method work on features interactions? Does our method improve model performance by reducing noisy feature interactions? In this section, we conduct an interpretable analysis by visualizing the feature interaction matrix, calculated by VVT. Each value the matrix is the normalized average of the absolute value of those two field features' dot product result, of which the higher indicates those two fields have stronger correlation.
Figure (a) and (b) illustrate the interaction matrix without and with pruning respectively, and 5 (c) shows the variation of matrix values. We can see that our PEP can reduce the parameter number between unimportant field interaction while keeping the significance those meaningful field features' interactions. By denoising those less important feature interactions, the PEP can reduce embedding parameters while maintaining or improving accuracy.
5.4 CORRELATION BETWEEN SPARSITY AND FREQUENCY
As is shown in Figure 6 (a), feature frequencies among different features are highly diversified.
Thus, using embeddings with uniform size may not handle their heterogeneity, and this property play an important role in embedding size selection. Hence, some recent works (Zhao et al. 20206 Ginart et al. 2019: Cheng al., 2020 Kang al. 2020; Zhang et al., 2020 Joglekar et al. 2020 explicitly utilize the feature frequencies. Different from them, our PEP shrinks the parameter in an end-to-end automatic way, thus circumvents the complex human manipulation. Nevertheless, the
8

---

Published as a conference paper at ICLR 2021
frequency of features is one of the factors that determines whether feature is important not. Thus, we study whether our method can detect the influence of frequencies and whether the learned embedding sizes relevant to the frequency.
We first analyze the sparsity trajectory trajeityyrraecctorry during training, which is shown in (b) where differ- colors indicate different groups of features divided according to their popularity. For each group, we first calculate each feature's s sparsity, then compute the average on all features. Shades in pic- tures represent the variance within group. We can observe that PEP tends to assign high-frequenc, features larger sizes to make sure there is enough representation capacity. For low-frequency fea- tures, the trends are on the contrary. These results are accord to the postulation that sighffreuencc features deserve more embedding parameters while a few parameters are enough for low- low-frequency feature embeddings.
Then we probe the relationship between the sparsity of pruned embedding and frequencies of each feature. From Figure (c), we can observe that the general relationship is concord with the above analysis. However, as we can see, some low-frequency features are assigned rich parameters, and some features with larger popularity are assigned small embedding size. This illustrates that simply allocating more parameters to high-frequency features, as most previous works do, can not handle the complex connection between features and their popularities. Our method performs pruning based on data, which can reflect the feature intrinsic proprieties, and thus can cut down parameters in more elegant and efficient way.
6 CONCLUSION
In this paper, we approach the common problem of fixed-size embedding table in today's feature- based recommender systems. We propose a general plug-in framework to learn the suitable em- bedding sizes for different features adaptively. The proposed PEP method is efficient can be easily applied to various recommendation models. Experiments on three fate-of-thhe-rt recommendation models and three benchmark datasets verify that PEP can achieve strong recommendation perfor- mance while significantly reducing the parameter number and can be trained efficiently.
ACKNOWLEDGEMENTS
This work was supported part by The National Key Research and Development Program of China under grant 2020AA the National Natural Science Foundation of China under U1936217, 61971267, 61972223, 61941117, 136603
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, and Xiaoqiang Zhang. Tensorflow: A system for large-scale machine learning. 2016.
Amir Beck and Mare Teboulle. A fast iterative triinkagetthresholdingg algorithm for linear inverse problems. Siam Journal on Imaging Sciences, 2(1):183-200 2009.
Heng Tze Cheng, Levent Koe, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, and Mustafa Ispir. Wide & deep learning for recom- mender systems. 2016.
Weiyu Cheng, Yanyan Shen, and Linpeng Huang. Differentiable neural input search for recom- mender systems. arXiv preprint arXiv:2006. 04466. 2020.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey. J. Mach. Learn Res., 20(55):1-21, 2019.
We define the sparsity of embedding by the ratio of the number of non-zero values to its original emhedding size.

---

Published as a conference paper at ICLR 2021
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Antonio Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, and James Zou. Mixed dimen- sion embeddings with application memory-efficient recommendation systems. arXiv preprint 0rXivv1909.118100 2019.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfim: factorization- machine based neural network for cur prediction. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 1725-1731, 2017.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, sys- tems, challenges. Springer Nature, 2019.
Prateek Jain, Ambuj Tewari, and Purushottam Kar. On iterative hard thresholding methods for high- dimensional m-estimation. 2014.
Manas R Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang, Jay K Adams, Pranav Khaitan, Jiahui Liu, and Quoe V Le. Neural input search for large scale recommendation models. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2397,,397, 2020.
Bang-Cheng Kang, Derek Zhiyuan Cheng, Ting Chen, Xinyang Yi, Dong Lin, Lichan Hong, and Ed H Chi. Learning multi-granulas quantized embeddings for large-vocals categorical features in recommender systems. In Companion Proceedings of the Web Conference 2020, pp. 562-566, 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference Learning Representations, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, pp- 8026-8037, 2019.
Steffen Rendle. Factorization machines. In 2010 IEEE International Conference Data Mining, pp. 905O1000. IEEE, 2010.
Hao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, and Jiyan Yang. Compositional em- beddings using complementary partitions for memory- recommendation systems. Pro- ceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 165-175, 2020.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1161-1170, 2019.
Omid Taheri and Sergiy A. Vorobyov. Sparse channel estimation with Ip-norm and reweighted 11-norm penalized least mean squares. In IEEE International Conference Acoustics, 2011.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, and Paul Van Mulbregt. Au- thor correction: Scipy 1.0: fundamental algorithms for scientific computing in python. Nature Methods, 17(Suppl. 1):1-12, 2020.
Caojin Zhang, Yicun Liu, Yuanpu Xie, Sofia Ira Ktena, Alykhan Tejani, Akshay Gupta, Pranay Ku- mar Myana, Deepak Dilipkumar, Suvadip Paul, Ikuhiro Ihara, et al. Model size reduction using frequency based double hashing for recommender systems. In Fourteenth ACM Conference on Recommender Systems, pp. 521-526, 2020.
10

---

Published as a conference paper at ICLR 2021
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys (CSUR), 52(1):1-38, 2019.
Xiangyu Zhao, Haochen Liu, Hui Liu, Jiliang Tang, Weiwei Guo, Jun Shi, Sida Wang, Huiji Gao, and Bo Long. Memory-efficient embedding for recommendations. arXiv preprint arXiv:2006. 14827, 2020a.
Xiangyu Zhao, Chong Wang, Ming Chen, Xudong Zheng, Xiaobing Liu, and Jiliang Tang. Au- toemb: Automated embedding dimensionality search streaming recommendations. arXiv preprint arXiv:2002. 2020b.

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 DESCRIPTION OF (s))
Following Kusupati et al. 2020 a proper threshold function g(s) should have following three properties:
1.
g(8) > 0, lim g(s) = 0, and lim g(8) = xc.
2.
3G E R++ 30 >0<g(> GY&ER.
3.
g' (mii which reduce the updating speed at the initial pruning.
A.2 PYTORCH CODE PEP
We present the main codes of PEP here since it is really casy-to-use and can plug in various embedding-basse recommendation models.
import torch
import torch.nn
import torch.nn. funcctional as
class TEP(nn.Module);
replace original embedding module (torch.m Embedding) def iniit___ (self, opt):
super(PEP, self)...intt__
self, wwweshold_ddecc typeettpp_eepe opt["threshold_yyer
define the granularity of
self.latent_dis * opt[ "artl"latettdiee
# define the initial embedding size total numbers features initial value of
self.feature_num
init opte'threshold._
self.g torchhsignoid self.5 hetffinnntttrrssooddeeoooooo
define the threshold function
initialize the embedding matrix self.v nn. nn. www.inixiiieeuumee wntpruunniinmeeeeeeeeeer self.sparse_ self.v.data
self, self.latent_dim)
def init threshold(self, init): iiiiiialize threshold with different granularities
if tels.threshold, typpettpp___p
5 = nn. parameetttrriii 'dimension':
elif twreshold_dtte type 'feature':
5 nn (init Norch.ones([selt) wtweessfffffttreeemm.. featuree_um, 1]))
elif self. www.fhhheehoddcccommm type 'feature dimension':
Pinit
self. (eff.laaeet_ddn))
else:
Paraantttteii torch.ones(1)) set global by default
return
def soft threshold(self, s): the reparameterizzation function s(v, D)
return torch.sign(v) * torch.eelutoorch.aa((sssssf...
def forward(self, x):
self.sparse_v self.soft threshold(self.v, s self.s) # the pruned sparse embedding matrix
F.embedding(x, self.sparse_v) rretrieving feature' embeddings by given feature return
Figure 7: Py PoTchc code of PEP.
A.3 WHOLE PROCESS OF PEP
We summarizes the pruning and retrain process by Algorithm
12

---

Published as a conference paper at ICLR 2021
Algorithm Our PEP
Input:
Initial embedding Vi(), base model ø, and target parameter P.
Output: Well trained sparsity embedding V.
1: while do not reach P do
2: Pruning V through Equation 9
3: end while
4: Obtain binary pruning mask m = 1()(')).
5: Reset the remaining embedding parameter initial values.
6: while do not coverage do
7: Minimize the training loss C(V(") @ m, D) with SGD.
8: end while
Table 3: Statistics of three utilized benchmark datasets.
Dataset
# Samples
# Fields
Features
MovieLens-IM
739, 015
3, 864
Critco
44, 617
39
086, 810
Avazu
40, 400 000
22
645, 394
A.4 EXPERIMENTAL SETUP
A.4.1 DATASETS
We experiment with three public benchmark datasets: Moviel.ens-1M MM, Criteo, and Avazu. Table summarizes the statisties of datasets.
MovieLens-IMM It widely-used benchmark dataset and contains timestamped user-movie ratings ranging from 1 to 5. Following AutoInt (Song et al., 2019), we treat samples with rating 2 as negative samples and samples with rating as positive samples. Other samples will be treat as neutral samples and removed.
Crited This is benchmark dataset for feature- recommendation task, which contains 26 categorical feature fields and 13 numerical feature fields. It has about 45 million users' clicking records on displayed ads.
Avazu Avazu dataset contains 11 days' user clicking behaviors which released for the Kaggle challenge, There are 22 categorical feature fields in the dataset, and parts of the fields are anonymous.
Preprocessing Following the general preprocessing steps (Guo al. 2017 Song al. 2019). for numerical feature fields in Critco, we employ the log transformation of log (x) >> 2 proposed by the winner of Criteo Competition [2] to normalize the numerical features. Besides, we consider features of which the frequency is less than ten as unknown and treat them as single feature "unknown" for Criteo and Avazu datasets. For each dataset, all the samples are randomly divided into training, validation, and testing set based on the proportion of 80%, 10%, and 10%.
A.4.2 PERFORMANCE MEASURES
We evaluate the performance of PEP with the following two metrics:
AUC. The area under the Receiver Operating Characteristic ROC curve (AUC) means the probability to rank randomly chosen positive sample higher than randomly chosen negative sample. A model with higher AUC indicates the better performance of the model.
p
13

---

Published as a conference paper at ICLR 2021
Logloss. As loss function widely used in the feature-bused recommendation, Logloss test data can straight way evaluate the model's performance. The lower the model's Logloss, the better the model's performance.
A.4.3 BASELINES
We compared our proposed method with the following state-of-the-art methods:
UE (short for Uniform Embedding). The uniform-embedding manner is commonly accepted in existing recommender systems, of which all features have uniform embedding sizes.
MGQE (Kang et al. 2020). This method retrieves embedding fragments from small size of shared centroid embeddings, and then generates final embedding by concatenating those frag- ments. MGQE learns embeddings with different capacities for different items. This method is the most strongest baseline among emnbadding--pramettr--ee parameter-sharing methods.
MDE (short for Mixed Dimension Embedding (Ginart et al., 2019)). This method based on human-crafted rule, and the embedding size of specific feature is proportional its popularity. Higher-frequene features will be assigned larger embedding sizes. This is the state-of-the-ar human-rule-basee method.
DartsEmb (Zhao et al., 2020b). This is the oftte-of-the-ai neural architecture search-based based method which allows features to automatically search for the embedding sizes in given space.
A.4.4 IMPLEMENTATION DETAILS
2019) and DeepFM (Guo et al. 2017), we employ Adam optimizer with the learning rate 0.001 to optimize model parameters both the pruning and re-training stage. For g(s), we apply g(s)
Following Autolnt (Song et al.
1 Ite- in all experiments and initialize the to -15, -150 and -50 in MovieLens- 1M, Criteo and Avazu datasets respectively. Moreover, the granularity of PEP set as Dimension-win wise for PEP-2, PEP-3, and PEP-4 on Criteo and Avazu datasets. And others are set as Feature Dimension- wise. The base embedding dimension d is set to 64 for all the models before pruning. We deploy our method and other baseline methods to three state-of-the-an models: FM (Rendle 2010), DeepFM (Guo et al. 2017), and AutoInt (Song et al.] (019), to compare their performance, Besides, in the retrain stage, we exploit the carly-stopping technique according to the loss of validation dataset during training. We use PyTorch (Paszke et al. (201)) to implement our method and train it with mini-batch size 1024 on a single 12G-Memory NVIDIA TITAN V GPU.
Implementation of Baseline For Uniform Embedding, test the embedding size varying from [8, 16, 32, 64], for the MovieLens- -MM dataset. For Criteo and Avazu dataset, we vary the embedding size from [4, 8, 16] because performance starts to drop when > 16.
For other baseline methods, we first turn the hyper-parameters to make models have the highest recommendation performance or highest parameter reduction rate. Then tune those methods that can balance those two aspects. We provide the experimental details of our implementation for these baseline methods as below, following the settings of the original papers. For the grid search space of MDE, we search the baseline dimension d from [4, 8, 16, 32]. the number of blocks K from [8, 16). and from [0.1, 0.2, 0.3). For MGQE, we search the baseline dimension d from [8, 16, 32), the number of subspace D from [4, 8, 16], and the number of centroids K from [64, 128, 256, 512). For DartsEmb, we choose three different candidate embedding spaces to meet the different memory budgets: (1,,,8), (2, 4, 16) and (4, 8,32
A.S COMPARISON BETWEEN PEP-O AND LINEAR REGRESSION
The Linear Regression (LR) model is embedding free model that only makes predictions based on the linear combination raw features. Thence, it is worth comparing our method on the extremely- sparse level (PEP-0) with LR.
Table 4 shows that our PEP-O significantly outperforms the LR in all cases. This result verity that PEP-0 does not depend on the LR part in FM and DeepFM to remain strong recommendation performance. Therefore, even at an satremely-sparss level, our PEP still has high application value in the real-world scenarios.
14

---

Published as a conference paper at ICLR 2021
Table 4: Performance comparison between PEP-O and Linear Regression.
MovieLens-IM
Criteo
Avazu
Methods
AUC
# Param
AUC
Param
Param
LR
0.7717
0
0.7881
0
0.7499
0
PEP-0 (FM)
0.8368
6,541
0.7941
1.067
0.7598
1,479
PEP-0 (DeepFM)
0,8491
8,604
0.7986
1.227
0.7622
2,215
PEP-O (AutoInt)
0.8530
9,281
0.7922
3,116
0.7607
2,805
PEP-O (AutoInt+I.R)
0.7980
1,117
0.7620
2,225
It is worth noting that the AutoInt model does not contain the LR component, so the PEP-0 in AutoInt on Criteo and Avazu dataset lead to a large performance drop. We try to include LR in PEP-O in Autolnt and test the performance outperforms the Autolnt without LR; It can be explained that LR helps our PFP-O acquire a more stable performance.
As we can see, the accuracy on Criteo and Avazu
A.6 THE LOTTERY TICKET HYPOTHESIS
rely on the Lottery Ticket Hypothesis reinitialize the pruned embeddings table (called winning ticket) into their original initial values. Here we conduct experiments verify the effectiveness of this operation in our PEP. We compare our method with its variation that uses random re-initialization for retraining to examine the influence of initializa- tion. We also compare the standard PEP with the original base recommendation model verify the influence of embedding pruning. To evaluate the importance of retraining, we further test the performance of PEP with the pruning stage only. We choose FM as the base recommendation model and use the same settings as the above experiments.
In the retraining stage in Section 4.2
We present the results in Figure 8 and 9 We can observe that the winning ticket with original initialization parameters can make the training procedure faster and obtain higher recommendation accuracy compared with random re-initialization. This demonstrates the effectiveness of our design of retraining. Moreover, the randomly reinitialize winning ticket still outperforms the unpruned model. By reducing the less-important features' embedding parameters, model performance could over-parametere parametered embeddings. This can be explained that it is likely to get over- fitted for those over wwer-parameterized embeddings when embedding sizes are uniform.
benefit from denoising those over-
Moreover, it is clear that the performance of PEP without retraining gets little bit downgrade, but still outperforms the original models. And the margin between without retrain and the original model is larger than the margin between with and without retraining. These results demonstrate that the PEP chiefly benefits from the suitable embedding size selection. We conjecture the benefit of retraining: during the search stage, less-important elements in embedding matrices are pruned gradually until the training procedure reaches a convergence. However, in earlier training epochs when these elements have not been pruned, they may have negative effects on the gradient updates for those important elements. This may make the learning of those important elements suboptimal. Thus, a retraining step can eliminate such effects and improve performance.
A.7 PRUNING WITH FINER GRANULARITY
In this section, we analyze the four different thresholds with different granularity mentioned in Sec- tion 4.3 The experiments are conducted the MovieLens-IN dataset with base model FM. Figure 10 (a) and (b) demonstrates the varying of embedding parameters and test AUC evolving with train- ing epoch. As we can see, the Disture-Dimension granularity can reduce much more embedding parameters than others. Mcanwhile, it achieves the highest performance at the retrain stage com- pared with other granularities. With the minimum granularity, the Dimeesin wise pruning can effectively determine the importance of embedding values. Besides, the Dimension-we wise prun-
We omit the results of AutoInt with IRR on the Moviel.eng IM dataset because there is performance drop for the AutoInt model compared with other models.
15

---

Published as a conference paper at ICLR 2021
Criteo
Criteo
praning
0.800
0.460
Without ut retrain
Wanning Eckettrandom reintt
0.488
Winning tickettorginal
0.796
0.455
0.794
0.792
Without pruning
Winning sikettrandiom
Winning ticketiorggnal init)
1
2
3
9
Epoch
(a)
Logloss
(b)
AUC
Figure 8: Logloss and AUC as training proceeds on Critco dataset (choosing FM as the base model).
Avazu
Avazu
Without pruniigg
Without retrain
twink)
Winning ticketloriginal
0.3895
0.3090
0.763
0.3885
Without pruning
0.762
Without retrain
0.3880
Winning ticket anddom reinit)
0.3875
0.761
Winniing tickettoriginaalii
10
11
12
10
12
Epoch
Epoch
(a)
Logloss
(b)
AUC
Figure 9: Logloss and AUC as training proceeds on Avazu dataset (choosing FM as the base model).
ing can achieve comparable AUC with fewer training epochs. Hence we adopt this granularity on PEP-2, PEP-3, and PEP-4 in large datasets time spent training.
A.8 ABOUT LEARNABLE g(s)
Pruning threshold(s) gss) can be learned from training data to reduce parameter usage in the em- bedding matrix. However, why can our PEP learn suitable g(s) with training data? We deduce that the increase of in g(s) can decrease the training loss. In other words, our PEP tries to update in the optimization process to achieve lower training loss.
In Figure I1
plot the FM's training curves with/without PEP MovieLens-IIe 1M and Critco datasets to confirm our assumption. Our PEP can achieve much lower training loss when pruning. Besides, it verifies that PEP could learn embedding sizes in stable form.
The stability shown in Figure II can be explained that our PEP obtains a relatively stable embedding parameter number at later stage of pruming (e.g., when epoch is larger than 30 in MovieLens dataset) as shown Figure TI And embedding parameters are well-trained. Thus, the training loss curve looks relatively stable. Note that the figure shows sequence of changing thresholds. The point when we get the embedding table for some sparsity level is not a converged point for this exact level, which instead requires retraining with fixed threshold.
16

---

Published as a conference paper at ICLR 2021
MovieLens-1M
MovieLens-1M
250000
Global
0.86
Dimension
0.85
Parameters
200000
Feeture Dimension
150000
0.84
100000
0.83
Global
50000
0.82
Feature
Reature-Dimennsinn
120
100
160
Epoch
Epoch
(a) Numbers of embedding parameters evolving (b) Test AUC evolving with training epoch with training epoch increase
crease at retrain stage
Figure 10: Influence of different granularity on Moviel 1M dataset (Choose FM as base model)
MovieLens 1M
Criteo
1.4
Without PEP
Without PEP
0.525
1.2
Training loss
1.0
0.8
ming
0.425
0.6
0.4
0.375
0.2
0.350
6000
8000
10000
6000 Gradient Scops
10000
Gradient Stops
(a) MovieLens-IM
(b) Criteo
Figure 11: Training loss of FM with/without PEP
17


Reviews:
[{"review_id": "VJzrepetvbS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors apply the technique of 'learning' the sparsity structure [1] in the context of Neural Recommender Systems. \n \nIt's an interesting and very practical application of the technique in the context of RecSys. In designing large-scale RecSys models, where the number of distinct items and users can go in the range of millions, storing and retrieving can become a challenge. A parameter compression technique can indeed prove to be very effective in such situations. \n\nSome strong points of the paper:\n\n1) Provides a practical application of the technique described in the very recent ICML paper [1]. \n2) Paper is well-motivated and generally well-written. \n3) Results indicate a huge improvement in terms of the number of parameters, without sacrificing the performance in the downstream tasks. \n\nSome questions for the authors:\n1) Can you please recheck equation 8? In the original paper, the equation is defined for each element of the matrix, where the mod. operator makes sense, you this case, what is the significance of the mod. the operator on the entire matrix? \n\nOverall, given a very practical solution to the exploding users and items situation in a RecSys setting, I recommend an accept for the paper. \n\n\n\n\n\n\nRef:\n[1] Soft Threshold Weight Reparameterization for Learnable Sparsity, Kusupati et al, ICML'20.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HABnOAh2a76", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes PEP (Plug-in Embedding Pruning) to reduce the size of embedding table while incurring insignificant drop in accuracy. The related work is well summarized into Embedding Parameter Sharing and Embedding Size Selection methods and the motivation for the current approach is well explained. The paper draws inspiration from Lottery Ticket Hypothesis. The problem formulation of Embedding pruning is done in a crisp way avoiding additional hyper parameter tuning that can be found in other methods. Similar to LTH, the paper shows that the initiation strategy can make the training process faster and stable. The results show an impressive 97-99% parameter pruning via PEP. As for the computation cost, PEP results show an additional 20-30% time cost compare with base models.\n\nOne of the main pros of this techniques is the learnable threshold(s) for pruning that can automatically jointly train with other parameters via gradient descent. As L0 based sparsity constraint on loss function is non-differentiable, the paper uses a neat re-parameterization technique to directly optimize the projection of sparsity matrix. The paper utilized the automatic differentiation framework to avoid complex back propagation and gradient calculation process. The main code of PEP is easy to read and understand. It’s only a few lines of code and well commented. \n\nThe results are well summarized and analyzed. Several baseline models (FM, DeepFM and AutoInt) were chosen that represented feature based models which was a fair strategy to follow. The one additional result that should have been added was running time comparison with MGQE, MDE, and DartsEmb pruning algorithms. The run time comparison of PEP was only shown with DartsEmb where the paper claims only 20-30% additional computation cost.\n\nPost Pruning analysis on MovieLens Dataset was interesting and show that PEP assigns high- frequent features larger sizes and fewer parameters are enough for low-frequent feature embeddings. A missing portion which when added can make the paper wholesome is analysis of PEP across models for the same dataset. The results show that PEP performed best compared to other methods for FM model whereas for AutoInt model, AUC of PEP method was lower than MGDE for instance. Any hypothesis, intuition of why that would happen will add value to the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ClYwupJ4FD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel approach to reduce size of the embedding table while not to drop in accuracy and computational optimization. Fixed-size embedding table has two problems, high memory usage cost and overfitting problem for those features that do not require too large representation. This paper recast the problem of embedding-size selection into learning column-wise sparsity, constraint K (eq(7)) and then convert S(V,s) problem (eq(8)). Paper used three benchmark datasets and some classical methods to verify effect.\n\nPros.\n1.proposed a novel idea to learn embedding size  instead of fixed-size.\n2.Experiments baseline with classical datasets and models show promising results on recommendation ranking problem.\n3.The method has generalization ability across different recommendation models.\n\nCons\n1.In METHODOLOGY part,  paper didn’t explain how to convert constraint by K in eq(7) to S(V,s) in eq(8)  and prove them.\n2.Symbols in paper are confused such as V and v , and  N has different meanings in METHODOLOGY part.\n3.The method first trains a sparse embedding matrix and then do re-training. During the sparse embedding learning phase, the memory size can not be reduced to a sparse format leading to significant memory cost, this is not feasible for very large scale sparse features for CTR task.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7XRRhNznmN-", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the embedding compression problem for recommendation via pruning. The paper is written very clearly with very strong experiments (multiple datasets/recsys models/recent baselines), showing ~99% parameter reduction while maintaining the same or even better performance.\n\nI have a few questions:\n\n- It's a bit surprising to me this method works. Like NIS or AutoEmb, they choose to use AutoML or validation data to select a suitable dim, as merely relying on a training set would lead to zero compression (using all parameters will have the lowest training loss). The key eq 8. actually doesn't encourage the model for sparsity, right? It's not clear to me why the model will increase `s`  during training, as using all parameters should decrease the training loss faster. Any explanation?\n\n- Ablation study: it'd be good to see the effect of the sparsity constraint. That is to say, we may train a model without any modification, prune small values, and then re-train the model, basically following the Lottery Ticket paper. Via showing the performance of this baseline, we'd know the effect of the eq 8. It's not clear to me the better performance is from eq8 or the lottery ticket.\n\n- What's the performance without re-training the model? Is it necessary to retrain the model for a good performance? What's the possible reason behind this?\n\n- It'd be better to mention that one potential drawback of this method against the baselines is the sparse embedding table may not be very easily used in practice. If we store the full dense embedding matrix, we won't save the space. If we use sparse representation, it'd be slower for matrix operations. In contrast, baselines like NIS, are free of such an issue, as they have the same, smaller dimension for all feature values.\n\nIn summary, the results seem to be very promising, which is the most important reason for me to give a high rating. However, it's not clear to me why such a simple method works (doesn't rely on validation set, automatically prunes itself with training set only) very well, and a few ablation study experiments might be added. It's likely that I'd change my score based on the author's reply, e.g. I'd be happy to raise the score if the authors provide a clear explanation/intuition for the idea.\n\n[updated after discussion]\nThank the authors for their efforts to add ablation study and make the manuscript more clear in presentation, it greatly resolves my questions and thus I raised the score. Overall the paper is in a good shape now. However I do want to point out a couple of things: 1) the performance of UE in base model FM/DeepFM/AutoInt seems to be a bit weired, as in previous CTR paper, deep models should outperform FM significantly. It's not clear to me is this due to different experimental settings or training schemes. 2) if the target is better performance (instead of compression), there is no clever way to choose s, other than mannual picking for each dataset/model.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received overall positive scores. All the reviewers agree that the approach presented in the paper is simple yet effective and the results are very impressive with >95% parameter reduction while maintaining the accuracy. The authors promptly revised the paper based on initial reviews. Therefore, I recommend accept and hope the authors incorporate the additional comments from Reviewer3 after the discussion. 
Decision: Accept (Poster)