UMEC: Unified model and embedding compression for efficient recommendation systems
Authors: Jiayi Shen
Haotao Wang
Shupeng Gui
Jianchao Tan
Zhangyang Wang
Ji Liu
Venue: ICLR 2021 Poster (2021)
Keywords: recommendation system
model compression
ADMM
resource constrained

Abstract: The recommendation system (RS) plays an important role in the content recommendation and retrieval scenarios. The core part of the system is the Ranking neural network, which is usually a bottleneck of whole system performance during online inference.  In this work, we propose a unified model and embedding compression (UMEC) framework to hammer an efficient neural network-based recommendation system.  Our framework jointly learns input feature selection and neural network compression together, and solve them as an end-to-end resource-constrained optimization problem using ADMM.  Our method outperforms other baselines in terms of neural network Flops, sparse embedding feature size and the number of sparse embedding features.  We evaluate our method on the public benchmark of DLRM, trained over the Kaggle Criteo dataset. The codes can be found at https://github.com/VITA-Group/UMEC.

Published as a conference paper at ICLR 2021
UMEC: UNIFIED MODEL AND EMBEDDING COMPRES- SION FOR EFFICIENT RECOMMENDATION SYSTEMS
Jiayi Shen', Shupeng Gui"", Haotao W Jianchao Tan', Texas A&M University, University of Rochester. UUniversity of Texas at Austin, "Kwai Inc. as jsiyyaa-677ee amu edu, sgui2eur... rwchester....................... edu, (atwang, at lasswang @utexas,ecoomm (jianchactan, jiliu) 8kuai shou. com.
Zhangyang Wang', Ji Liu"
ABSTRACT
The recommendation system (RS) plays important role in the content recom- mendation and retrieval scenarios. The core part the system is the ranking neural network, which is usually bottleneck of whole system performance during online inference. Hammering an efficient neural network-based recommendation system involves entangled challenges of compressing both the network parameters and the feature embedding inputs. We propose a unified model and embedding compression (UMEC) framework to jointly learn input feature selection and neural network compression together, which is formulated as resource- resource-constraipee constrained opti- mization problem and solved using the alternating direction method of multipliers (ADMM) algorithm. Experimental results on public benchmarks show that our UMEC framework notably outperforms other non-integrated baseline methods. The codes can be found at https: : // //g//gtuuui com/
coom/IIII VITA- Group/ /MME.
INTRODUCTION
As the core component recommendation system (RS), recommendation models (RM) based on ranking neural networks are widely adopted in general content recommendation and retrieval applications. In general, an effective recommendation model consists of two components: a feature embedding sub-model and prediction sub-model, illustrated in Figure 1. Usually, an RM adopts neural networks to serve two sub-models. Formally, we denote an RM as f(: W), where W is the learnable parameters of f. For the inference, the model ff takes the input feature data to predict the confidence of the content, serving the recommendation applications. Specifically, we further define the embedding and prediction sub-models as (:; We) and fp(:: Wp) respectively, where We and Wp ). The embedding feature, v == [el:; W.), is the input of fp with the input data Hence, we can express the RM as f(; W) & (f W.); W,). Given a ranking training loss ((-) (i.e., binary cross entropy (BCE) loss), the learning goal of the ranking model can be written as
are their own learnable parameters and W = (We. e. W,
min W
Σ {(((:; W), (#.N)ED
where y is the ground-truth label, and D the training dataset.
Nowadays, extremely large-scale data have been poured into the recommendation system to predict user behavior many applications. the online inference procedure, the heaviest computation component is the layer-wise product between the hidden output vectors and the model parameters W. for a neural network-basee RM. A slimmed neural network structure would save a great amount of power consumption during the inference. Hence, the main idea of an RM compression is to slim the structural complexity of ((W) and reduce the dimension of hidden output vectors.
To obtain an efficient ranking model (for example, MIP based) for RS, may apply existing model compression methods to MLPs directly. For example, Li et al. (2016) removes the entire filters in the network together with their connecting feature maps in terms of magnitudes, which can also
Equual Contribution.

---

Published as a conference paper at ICLR 2021
be applied to MLP structures to remove a specific neuron as well its connections. Molchanov et al. (2019) approximates the importance of a neuron (filter) to the final loss by using the first and second-order Taylor expansions, which can also be applied to pruning the neurons without hassle. There are also some compression methods focusing on dimension reduction of embedding feature vectors. Ginart et al. (2019) proposes mixed dimension embedding scheme by designing non-uniform embedding dimensions scaling with popularity of features. Joglekar et al. (2020) uses Neuron Input Search (NIS) learn the embedding dimensions for the sparse categorical features.
However, the performance of these compression methods highly depends on the hyper- eeper-parameter tun- ing and the background knowledge of the specific recommendation model. For example, an embedding dimension compression method may require the user search for the best dimension with multiple train- ing and search rounds.
Inpul
Sparse Inpul.
Embeskding model
Firheekiig Feature
Prediction Sub-model f
Concatemaaiio
We would like to solve the RM compression prob- lem with a unified resource-constrained optimization framework. only relies on the resource consump- tion of the RM f(W) to compress both the MLP and the embedding dimensions, without any multi- stage heuristics nor expensive hyper-parameterss tun- ing. Our novel unified model and embedding com- pression (UMEC) framework directly satisfies both the requirement of resource consumption of the rank- ing neural network model and the prediction accuracy goal of the ranking model, with end-to-en aod-to-end gradient-based training. We reformulate the optimization of training loss associated with hard con- straints into minimax optimization problem solved by the alternating direction method of multipliers (ADMM) method (Boyd et al., 2011).
Figure 1: An example of the neural network- based recommendation system: the deep learning recommendation model (DLRM), proposed by Naumov et al. (2019).
To summarize our contributions, we present the merits of UMEC as the following:
To the best of our knowledge, UMEC is the first unified optimization framework for the recommendation system scenario. Unlike those existing works that treat the selection of input feature and compression of the model as two individual problems, UMEC jointly learns both together via unifying both original prediction learning goal and the model compression related hard constraints.
We reformulate the joint input feature selection and model compression task as constrained optimization problem. We convert resource constraints and To sparsity constraints into soft optimization energy terms and solve the whole optimization using ADMM methods.
Extensive experiments performed over large-scale public benchmarks show that our method largely outperforms previous state-of-the-art input feature selection methods and model compression methods, endorsing the benefits of the proposed end-to-end optimization.
2 RELATED WORK
Recommendation Models With the recent development of deep learning techniques, plenty of works have proposed learnin-bbased recommendation systems grapple with personalized recom- mendation tasks. Hidasi et al. (2015) applies recurrent neural networks (RNN) on long session data to model the whole session and achieve better accuracy than the traditional matrix factorization approaches. Covington et al. (2016) proposes a high-level recommendation algorithm for YouTube videos by specifying candidate generation model followed by a separate ranking model, which demonstrates a significant performance boost thanks to neural networks. Wu et al. (2019) proposes a recommendation algorithm based on the graph neural network to better capture the complex transitions among items, achieving more accurate item embedding. Their method outperforms traditional sequential approaches. Naumov et al. (2019) exploits categorical features and designs a parallelism scheme for the embedding tables to alleviate the limited memory problem. Huang et al. (2020) proposes a multi-attention based model to mitigate the low efficiency problem in the group recommendation scenarios.
2

---

Published as a conference paper at ICLR 2021
Input Feature Compression for Recommendation Models The numerous input features for practical recommendation scenarios necessitate the selection of useful ones to save memory resources and facilitate computational efficiency during inference. In factorization machinex(FM)-base basedd and context-aware recommendation systems, for example, Mao et al. (2017) conducts input feature selection in automatic manner by feature ranking and feature sub-sampling. while managing to improve the prediction quality. Ginart et al. (2019) designs a mixed dimension embedding layers scheme and adjusts the dimension of particular embedding feature according to the frequency of that item. Joglekar et al. (2020) performs Neural Input Search (NIS) method to search for the embedding sizes for the sparse categorical features, and designs the multi-size embedding framework to leverage the model capacity more efficiently. Song et al. (2020) proposes a discovering framework to automatically find the interaction architecture of the prediction model.
Model Compression Model compression aims to reduce the complexity of Deep Neural Network (DNN) models to achieve inference efficiency and resource saving. Generally, model compression techniques can be categorized into pruning (Han al., 2015; Wen et al., 2016; Molchanov et al., 2019), quantization (Hubara et al., 2017: Wu et al., 2018: 8; Ajanthan et al., 2019) and distillation (Hinton et al., 2015; Polino et al., 2018: Tung & Mori, 2019). Han al. (2015) proposes a magnitude-baset and element-wise pruning technique, which does not guarantee the reduction computation efficiency, energy and memory costs. Structured pruning can solve this issue by removing whole certain types of structures from the network, such as filter pruning and channel pruning methods (Li et al., 2016; He al., 2017; Molchanov al., 2019). Recent works (Frankle & Carbin, 2019; Chen al., 2020b;a; 2021c;b;a; Ma et al., 2021) reveal that with appropriate compression techniques, the identified sparse subnetworks are capable of training in isolation to match the dense model performance. In the meanwhile, researchers also develop great interest in investigating hardware- aware compression, which utilizes practical resource requirements(e..g g. energy, latency, Flops) guide the compression (He et al., 2018; Yang et al., 2019; Liu et al., 2019).
3 UNIFIED MODEL AND EMBEDDING COMPRESSION ON RS
For an arbitrary NN-based recommendation model f(() an RS, the terminology of "compression" refers reduce the power consumption with respect to all the computation operations for the inference. The learning parameters of f(W) can be expressed as W = (w(Op 12 [Н]], where L. is the number of layers in the S(W). For an arbitrary layer I, the layer's weight is denoted (!)) Rd.xddd The recommendation inference on the layer is executed as
x (++))
W (" (1) + b()
where (1) Rd is the input of I-th layer, and (()) is the corresponding learning bias term. Since the RM consists of two components: feature embedding sub-model and neural network prediction sub-model, the "compression" can be interpreted as shrinking the dimension of embedding feature and reducing the number of parameters in W and b.
Perspective of model parameters W and b It is to reduce the input output neurons for the specific layer I. For example, the i-th neuron of x (!+1) and j-th neuron of (!) are chosen to remove, then the weight matrix (l)) will be reshaped to (dout - 1) x (d., - 1).
Perspective of feature embedding vectors The reduction of d,, would result in the elimination of the input features. For example, (1))) is defined as the concatenation of a series of embedding vectors, (11) = [e1; e2; ... enl, where [:::;]] represents the concatenation operator, and n denotes the number of embedding feature vectors in the RS. Without loss of generality, for the first embedding
vector e₁ IRd the zero-out of first dj columns in (I) will result in the elimination of the whole embedding vectors for e1. On the other hand, the compression on fe (We) would reduce the feature embedding vectors' dimensions.
3.1 A COMPRESSION EXAMPLE: GROUP LASSO FOR WEIGHT PRUNING
We first consider a simple example with an RS model consisting of f. (W.) and (W,,). For this RS, to reduce the computation consumption, we present straightforward model compression method
3

---

Published as a conference paper at ICLR 2021
with Group Lasso. The method considers both the feature embedding selection and network slimming. We only need to apply the sparse regularization terms to the weight matrices. The optimization problem can be expressed as
min +
Σ
w off
+ 1=2
N
W
where ((W) is the original objective function. The As are of these regularization terms. g. denotes the feature groups for the input layer. || ll1 indicates the L1 norm. The optimization problem (1) restricts the group-wise sparsity for the input layer and the neuron-wise sparsity (or group size = 1) for the rest model layers while training with the original objective function f(W). The optimization problem can be solved with the projected gradient descent method. However, the optimization problem (I) has its own limitation. The hyper-parameter
to control the importance
only have the latent impact on the model size and we could not directly affect the model size. It is not friendly when the users would like to restrict the model size to a specific number. On the other hand, the hyper-pe parameters also make the optimization problem hard to tune, requiring multiple arguments searching rounds. To avoid these shortages, we are supposed to have new approaches to handling the compression of RS.
3.2 RESOURCE- DONSTRRINND MODEL COMPRESSION
In this section, we propose a novel unified model and embedding compression method on RS and treat it as a constrained optimization problem. We discuss how to solve the problem using gradient-based optimization algorithm.
Computation Resource Function for RS Model To avoid the hyper-parameter tuning in the optimization problem (1), we define the resource function to measure the resource consumption for specific RS model structure, which can be used guide the compression of structure. For example, considering the computation Flops, the resource of an MLP can be defined as
L
RFlops Σ
2 1xd a
x d
d
where d0 and d indicate the number of input and output neurons. For pruned layer I, the input and output dimension are restricted by number pruned neurons, annotated as ( and g(++)) Therefore, we can define a resource consumption function in terms of from each layer
2 (d - -s(0) (ahl) - s((+1) + +(a. - s ,(41))
RFlops (s)
For simplicity, the resource function Retops only takes the s as the number of neurons. In a more general case, can be substituted with the product between the number of pruned groups and the size of each pruned group.
RS Model Compression can formulate the following optimization problem,
With the definition of the resource function,
min ((W), s.t. RFlops (W) I Reodget
where Roudget denotes the upper bound of the overall model computation Flops set by users. Since we consider the group-wise sparsity for each layer, we utilize the set of variables = {s(1) s(2) s(L)) control the model's Flops. The (0 defines the lower bound of the I-th layer's
4

---

Published as a conference paper at ICLR 2021
zero group number for the pruned model. Therefore, reformulate the optimization problem (2)
min 8(W)
(3a)
Wie
s.t. Retops (8) S Roudget
(3b)
Σ1
W ! =0) 2 v1=,1
(3c)
Σn
(
$0
(3d)
where I(-) an indicator function with the output {0. 1}. When the condition satisfied, the denotes the
indicator function results in value of 1, otherwise, it would be 0. Σ; (
zero group number for the /-th layer. The resource constraint (3b) bounds the resource consumption of the recommendation model. The inequalities in both (3c) and (3d) regularize the Flops consumption for each layer.
3.3 OPTIMIZATION
To address the optimization problem (3), we adopt the alternating direction method of multipliers (ADMM) for the reformulation. In details, the optimization problem contains non-continuous indicator function in constraint (3c, 3d), and non-convex constraint (3b), which makes the problem difficult solve. Therefore, we first reformulate the inequality constraints soft regularizations and introduce minimax optimization with dual variables. Then we tackle the non-differentiable objective function using self-defined numerical differentiation. last, summarize all the optimization details into a gradient-bass -baeed optimization formulation.
Minimax Optimization Reformulation Let || |WWalle.2 be the bottom-(s, 2) "norm", which de- notes the Frobenius-norm of the sub-matrix (group) composed of bottom-s groups, by sorting from top to bottom in terms of norm of each group. The equivalent formulation of the sparsity constraint (3c, 3d) can be
W w. 1l. + Σ 1 1 12: 2
The above equivalence (4) is proved by Tono et al. (2017), where the LHS is called DC (difference of convex functions) representation of the Cy-constraint. With the transformation, we can reformulate the problem in Eq. (3), into this minimax optimization problem:
min max (WW) (1) W.a 9.220
( aa
1012 (()
+ (Riiops (s)
Reudga
2072
resource loss
sparsity loss: S(y.8.W)
where y and are dual variables. The operator [ indicates the ceiling function. In the problem (5), the importance of regularization terms are tuned automatically based on the minimax optimization.
The sparsity loss S(y, 8, W) = ()) ())
1,0 and the resource Σ'= |
loss 2 (Rioos (s) Reudee) are introduced to substitute the original constraints (3b, 3c, 3d). Therefore, the optimal solution to the new problem (5) can serve as the optimal for the original problem (3). To solve it, we present the details of update rules for all learnable variables in appendix A.
We wrap up all the update rules together to be unified optimization algorithm, as described in Algorithm 1. In this algorithm, we only show an example that the input layer is column-wise grouped. For more general case, the column-wise group sparsity can be applied to an arbitrary layer in the RS model.
Prune and Finetune Solving the minimax optimization problem (5) will make some channel weights extremely close but not exactly equal to zeros. In order extract the slimmed network, for
5

---

Published as a conference paper at ICLR 2021
each W(),, discard the g0 groups with smallest L2 norms || ..ge (extremely close to after we have solved problem (5). Then we finetune the explicitly pruned network to further improve the performance.
Algorithm 1: Gradient- algorithm to solve problem (5) for UMEC.
Input: Resource budget Rougge. learning rates fh,, 1/2 fJs, 74, number of total iterations 7. Result: DNN pruned weights W
Initialize L 1, W
// random a pre-trained dense model
for t do
W prox,, S(v' way (we we (W*)):
// Proximal-SCD
8¹ 112
(V.(y'' wet+) Va (Noupa (5¹) Rantga) // Gradient (STE)
Descent
y (1) =00 +
(())
[aaset
// Gradient Ascent
y (0)+ -000 + n18
VV= 21.,.....
c41 2t + n/ (Retoga (5') Reudgat)
// Gradient Ascent
W = W
4 EXPERIMENTS
To demonstrate the effectiveness of UMEC, real-world recommendation dataset. And we would like to answer the following questions: Whether our unified optimization framework that jointly considers input feature selection and prediction model compression has superiority over previous state-of oft-----ahtaae the-hht-tttt art RS compression methods which solve the problem from only one aspect? If yes, why?
experiments standard public RM the
4.1 EXPERIMENTAL. SETUP
Dataset and Model We conduct all our experiments on the state-of-the-art ads CTR prediction model named DLRM (Naumov al., 2019), as illustrated in Figure 1. Following Naumov al. (2019), we use the Criteo Al Labs Ad Kaggle' and Terabyte² datasets for our experiments. The Criteo AI Labs Ad Kaggle dataset contains approximately days. The Terabyte dataset consists of approximately 4.4 billion click log samples collected over 24 days and we perform uniform sampling with 12.5% sampling rate from the raw data following the sampling scheme in Naumov al. (2019). Both datasets contain samples that have 13 continuous and 26 categorical input features. Following the official setting, for both datasets we split the data of the last day into validation and testing sets, and use data from the rest days the training set.
million click log samples collected over seven
Baseline methods The latency of ads CTR prediction model mainly comes from two sources: feature embedding and prediction model inference. Feature embedding layers map categorical input values into vector space. Both the number of input categorical values and the output feature space dimension can affect the latency of feature embedding stage. The inference latency of the predication model mainly depends on the prediction model size. Following the above analyses, methods to accelerate ads CTR prediction models mainly fall into three categories:
Prediction model compression: compressing the prediction model with traditional model compression methods, such as Li et al. (2016); Molchanov et (2019); Yang et al. (2019).
Input feature selection: reducing the number of categorical input features by filtering out non-essential ones.
"https://ww.kkgeee http://wwwggle..ooccccceeeeeeeee wttpe//cccccccciiceeeeeeeeoccect tes-rittcdiisss aaal hhttps://aabs. criteo. wtt.e...o..1///////too
i logs/

---

Published as a conference paper at ICLR 2021
Embedding dimension reduction: shrinking the dimension of embedding layers' output space. Two recent works (Ginart et al., 2019; Joglekar et al., 2020) fall into this category.
Our unified optimization framework jointly considers input feature selection and model compression. We will further show that our method can be generalized to considering input feature dimension- reduction as well. we compare our method with baselines from all three categories to show the superiority of our method over those baselines tackling the RS compression problem from only one aspect.
General Training and Evaluation Details We use the following settings for experiments on the Critco Ad Kaggle dataset: we use SGD optimizer with learning rate 0.1 to optimize the BCE loss (E(W) in method); set training batch size to 128, initial feature embedding dimension to 16, and use the three-layer MLP prediction model with hidden dimensions 512 and 256.
For our method, we choose Adam optimizer (Kingma & Ba, 2014) for and SGD optimizer for W, y, and z. We set learning rates 7),, 72, f/8, 7/4 in Algorithm to be [0.1, 0.05, 0.1, 2.0] respectively, and T be 306,969 which equals to epoch. In our experiments, we initialize the model weights from a dense model that has been pre-trained for epoch using ((W) to facilitate training. After the training process in Algorithm 1, we zero out the redundant neurons (as described in the last paragraph in Section 3) and extract the sub-network as the pruned model. We then perform fine-tuning for epochs. For a fair comparison, we also fine-tune 2 epochs for all baseline methods. Following Naumov et al. (2019), the best accuracy on the validation set is reported.
For experiments on the Criteo Ad Terabyte dataset, the training settings generally follow those mentioned above with several exceptions: batch size equal to 4096, equal to 157,655, the hidden dimensions of the three-layer MLP prediction model equal to 256 and 128.
Evaluation Metrics We evaluate the performance with the following two metries: Accuracy and Compression Ratio (CR). Accuracy: CTR prediction accuracy on the validation set. Compression ratio (CR): the ratio between the Flops that has been deducted and original model Flops. More formally, CR = 1 - (yWW)) (W)). where py(+) calculates the Flops of (sub-)model. W and Wo are the pruned and original dense prediction models, respectively. The larger CR, the more compact model we will obtain.
4.2 PREDICTION MODEL COMPRESSION FOR RS MODEL
We design and conduct extensive experiments to examine compression performance among our method and several state-of-the-art model compression methods. The baseline methods include handcrafted structures, one-shot magnitude pruning (MP) (Li et al., 2016), and two recent state-of- the-art methods, namely Taylor pruning (TP) (Molchanov al., 2019) and energy congttccontr model compression (ECC) (Yang et al., 2019).
In Figure 2 and 3, demonstrate the best accuracy on validation set under different compression ratios (CRs) on the Criteo Al Labs Ad Kaggle and Terabyte datasets. Each curve shows the CR- Accuracy trade-off of one compression method. The doted magenta line represents the accuracy achieved by the original dense DLRM model.
Several observations can be drawn from this comparison figure. First, method continuously outperforms all other methods with a margin of at least 0.01% %. Second, only our method can successfully compress the original dense model the ideal budget while still manages to maintain the performance without any degradation within a certain CR scope.
These results show that our joint model compression and input selection method can achieve much better balance of efficiency vs accuracy than traditional model compression methods.
4.3 INPUT FEATURE SELECTION FOR RS MODEL
We evaluate the effectiveness of our proposed method on input feature selection task. The input feature selection task can be easily realized using our proposed framework, by keeping the resource constraint for the features groups in input layer (i.e., Eq. (3c) and ignoring constraint for weights of hidden layers(i.e., Eq. (3d)). We use Group Lasso self-designed baseline. The loss function
7

---

Published as a conference paper at ICLR 2021
78.475
78.550
78,900
78.875
Durs
Ours
PCC
PCC
78.825
78
TP
TP
Dorse Model
Denue Model
0.0
0.7
0.9
0.5
0.6
0.7
0.5
1.0
Compresaion Rotlo (CR)
Compression Rotio (CR)
Figure 2: Results on RS prediction model com- pression by different methods on the Criteo Al Labs Ad Kaggle dataset.
Figure Results on RS prediction model com- pression by different methods on the Criteo AI Labs Ad Terabyte dataset.
of Group Lasso is shown in Eq. (1). Since we are only considering input feature selection in this sub-section, the sparsity regularization on hidden layers (e.g., the second term in S W)) is ignored. We tune hyper-parameter A among 0.0005, 0.001, 0.005, 0.01 for the Criteo AI Labs Kaggle dataset, and 0.001, 0.003, 0.005 for the Terabyte dataset, in order to get different compression ratios. After Group Lasso training. all input features whose L2 norms are below a threshold will be removed, and the Flops for the new model are calculated accordingly. We empirically find that setting th to 0.01 on the Kaggle dataset and 0.001 on the Terabyte dataset can achieve good performance. For both method and Group Lasso, train and fine-tune for epoch and 2 epochs, respectively. In Figure and 5, the performance on the holdout validation set at the end of the compressing
79.00
81.075
81.050
78.95
81.025
81.000
78.05
00.975
00.950
Durs Group LASSO
80.025
Group LASSO
Dence Mocel
Dence Model
00 900
0.125 Compression
0.15
Compression
Figure 4: Results RS input feature selection by different methods on the Criteo Al Labs Ad Kaggle dataset.
Figure Results on RS input feature selection by different methods on the Criteo Al Labs Ad Terabyte dataset.
process is reported. We demonstrate the effectiveness of our method in input feature selection task, by showing its superior performance under all the compress ratios. Another observation is that the pruned model of both methods will fail to recover the performance the original dense model too 0.21 compression ratio on the Kaggle dataset,
many input features are removed. For example, which corresponds to removing out of the total 27 input features, both methods have a considerable drop of accuracy. This indicates that reducing the number of input features will cause considerable harm to the model's performance.
4.4 EMBEDDING DIMENSION REDUCTION FOR RS MODEL
We show that our method can also be applied to embedding dimension reduction. In this new scenario, W in Eq. (3) is embedding layer weights, instead of previous prediction model weights. Eq. (3d) is removed (i.e., no longer to prune the input dimension). The rest parts the objective function remain the same.
8

---

Published as a conference paper at ICLR 2021
Feature Input Layer
Hidden Layer
Hidden Layer 2
250
2.5
80
200
150
1.5
1.0
40
20
10 Training teration
Training Iteration
Training Iteration
Figure 6: The number of pruned features/neurons for each layer during the training process using UMEC with Roddee = 0.5 x (Wo) on the Criteo AI Labs Ad Kaggle dataset, where py (Wg) denotes the Flops of the original dense model. For the input layer, the mentioned dense model has 27 features in total. For the two hidden layers, has 512 and 256 neurons respectively. The convergency of the binary cross entropy (BCE) loss during training is shown in Figure 7 in the appendix.
We compare our method with the offte-of-the of-tte-art embedding dimension reduction method named mixed dimension embedding layers (MDEL) (Ginart et al., 2019). Following Ginart et al. (2019), we use size of the embedding layers to help define compression ratio. More specifically, in this setting, CR = - Ps (WW) (WW/p- /ps (Wo). where (.).) calculates the number of parameters a (sub-)model. W and Wo the pruned and original dense embedding sub-models, respectively. To evaluate performance change after pruning, we define Aace = Acew - Acew, where Accy and Acew, are CTR prediction accuracy of pruned and original dense model, respectively.
MDEL only provides results on the Criteo AI Labs Ad Kaggle dataset, conduct the comparison this single dataset. Exper- iment results are shown in Table 1. As we can see, both methods have slight improvement in accuracy around 50% CR, while the accuracy improvement of our method is larger than that of MDEL.
Table 1: Results on embed- ding dimension reduction.
Method
CR
Aace
MDEL
0.5
0.10%
0.5
0.24%
Also, our method can still achieve better accuracy improvements than MDEL with even 10% larger CR. These results show that our ADMM based compression algorithm is better than MDEL.
Ours
0.6
0.16%
4.5 SPARSITY ANALYSES OF UMEC
Below we present how the sparsity of the prediction model evolves during the pruning phase in Section 4.2. As recall, the aforementioned prediction model consists of one input layer and two hidden layers, with 27 input features, 512 and 256 neurons respectively. We take the Criteo Al Labs Ad Kaggle dataset as an example. As shown in Figure 6, when Rpuiger 0.5 x py (WV). the pruned input features/neurons converge to 0, 102 and 250 respectively. During training, the last hidden layer gets the most percentage of neurons removed and the input layer gets the least percentage. We can come to several conclusions from this result. First, such phenomenon aligns with the observation in Section 4.3 and serves as piece of evidence that we need enough input features to adequately represent the raw input information and contribute to the downstream classification task. Second, to perform the relatively easier task, i.e., binary classification, the requirement of neurons from the last hidden layer falls far below the original design. Thus considerable resource consumption can be saved here. Third, it demonstrates that our resource-constrainee unified optimization plays a smart role in globally finding the optimal resource allocations across all layers.
5 CONCLUSION
The goal of this paper is tackle with the new problem that jointly compressing input features and neural network structures in the recommendation models. We propose UMEC framework by integrating these two objectives into unified constrained optimization problem, solved by the ADMM method. We conduct extensive experiments and demonstrate the effectiveness of our proposed UMEC method by observing its superior performance than other state-of-the-ar baseline methods.

---

Published as a conference paper at ICLR 2021
REFERENCES
Thalaiyasingam Ajanthan, Puneet K Dokania, Richard Hartle,, and Philip HS Torr. Proximal mean- field for neural network quantization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4871 14880, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
Stephen Boyd, Neal Parikh, and Eric Chu. Distributed optimization and statistical learning via the alternating direction method of multiplierss Now Publishers Inc, 2011.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv preprint arXiv:2012.06900 2020a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv, abs/2007.12223, 2020b.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks, 2021a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference Learning Representations, 2021b. URL https: https/////pp.....//////eeeee/ wwwperrwwwwwwweeeeeeeeeeeeeeeeee wttpr/iwwwwee.ffur/eeeeee ftrmmm/rrmmmm//////// wtt.u//iiiiIISxmmmmmmmm
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery tickets too. In International Conference on Learning Representations, 2021c. URL URRhhttts:://peeeeeeee net/ net/forum:2 Eorum wew.rrumidd-AMMhMhhh..e 1AMMho_9jERR
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2019. URL httpt::ttt//errreeee /opprreveeeee/ee/// ew. net. rumm -13RF
Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan- distiller: Searching to compress generative adversarial networks. arXiv preprint 088988006.08198. 2020.
Antonio Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, and James Zou. Mixed dimen- sion embeddings with application memory-efficient recommendation systems. arXiv preprint arXiv: 11810, 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.001. 2015.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
Yihui He, Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European Conference Computer Vision (ECCV), pp. 784-800, 2018
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06923 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv 2015.
preprint
Zhenhua Huang, Xin Xu, Honghao Zhu, and MengChu Zhou. An efficient group recommendation model with multiattention. based neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020.
10

---

Published as a conference paper at ICLR 2021
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Ylniv,,v, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869-689) 2017.
Manas R Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang. Jay K Adams, Pranav Khaitan, Jiahui Liu, and Quoe V Le. Neural imput search for large scale recommendation models. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2387-2397, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXi::1608.087: 08710, 2016.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3296-3305, 2019.
Haoyu Ma. Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.03255 2021.
Xueyu Mao, Saayan Mitra, and Viswanathan Swaminathan. Feature selection for fm-based context- aware recommendation systems. In 2017 IEEE International Symposium Multimedia (ISM), pp. 252-255. IRFE, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1124--1272, 2019.
Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sun- daraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krish- namoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. CoRR, abs//900..0099, 2019. URL https: / rxiv. org/ abs / 906. 00091.
Atsushi Nitanda. Stochastic proximal gradient descent with acceleration technigues. In Advances in Neural Information Processing Systems, pp. 1574-1582. 2014.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint 1rXivv1802..5600 2018
Qingquan Song, Dehua Cheng. Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. Towards automated neural interaction discovery for click-through rate prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 945-955, 2020.
Katsuya Tono, Akiko Takeda, and Jun-ya Gotoh. Efficient de algorithm for constrained sparse optimization. arXiv preprint
2017.
Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1365-1374, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, pp. 2074-2082, 2016.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. In International Conference Machine Learning, pp. 5363-5372. PMLR, 2018.
11

---

Published as a conference paper at ICLR 2021
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. Session-bies based recommendation with graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33. pp. 346-353 2019.
Haichuan Yang. Yuhao Zhu, and Ji Liu. Ecc: Watform-indeeeeee energy coryy-oonttaaieed deep neural network compression via bilinear regression model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11206-11215, 2019.
A APPENDIX
A.1
UPDATE RULE FOR ALL LEARNABLE VARIABLES
To solve the optimization problem (5), we utilize the ADMM strategy to solve a series of sub-problems as follows.
Update W The optimization problem on W can solved with proximal-SGI. (Nitanda, 2014). The definition of the proximal operator is as follows:
Prox,Sty...w) = arg minw 2 l|w - wll + ns (y. s, W)
where W is the direct stochastic gradient descent update of W. The sub nub-problem above has the closed-form solution for W = (W(1), (2)), W())+.
W
if W. >= || [s]
W
W,
otherwise,
2ny
where W; denotes the i-th sub-matrix, and W is the weight for arbitrary layer.
Update dual variables z and y
For the sub-optimization problem, we have
max ( w 0.020
Σ I=2
+ + (RFlppss)
Raudge))
1,2
and we can adopt the gradient ascent method as the update rule.
Update 8 The optimization on 8 relies on both sparsity and resource loss. However, both of them are non-differentiable due to the ceiling function [s]. Straight-througl estimator (STE) (Bengio et al., 2013) is an effective approach for optimization through the non-differentiable functions. The main idea is to adopt the back- prckppropagttion
simple proxy to be the derivative of the non-differentiable formulation. Thus, be applied as what happened in the differentiable case. For W 1,,2
can apply the numerical differentiation || 1,2 respect to s:
[W as the proxy derivative of W | with
W 2 Os where 2 is the column-group wise L2 format of W, Dim( is the total number of column groups of W. and W We is the j-th least column group in W
W least-min(Dimm W), 1)>
For general resource consumption function (e.g., Flops), the non-differentiable part of R the ceiling function [s], for which
common STE (Bengio et al., 2013): 0[s] 1.
ds
0 0s
1.
Now wrap up all the update rules together be unified optimization algorithm 1. In this algorithm, we only show an example for the input layer to be column-wise grouped. For a more general case, the column wise sparsity can be applied to arbitrary layer in the RS model.
A.2 CASCADED PIPELINE AS A BASELINE
12

---

Published as a conference paper at ICLR 2021
Training Loss
We provide experimental results under cascaded pipeline using UMEC, where the compression processes for the input feature and the prediction model are carried out sequentially. Two scenarios are demonstrated: a) Con- ducting input feature selection first, then prediction model compression, of which the results are shown in Figure 8; b) Conducting embedding dimension reduction first, then pre- diction model compression, of which the results are shown in Figure We set CR values of first stage in cascaded pipeline as 0.116, 0.1 16, 0.302, 0.302, 0.302 respectively for a), and 0.1, 0.1, 0.1 for b). Then we try our best to set the final CR values of the whole cascaded pipeline to be similar to the CR values used in our joint framework respectively for fair comparisons in both figures. As example, we conduct the experimental comparisons on the Criteo AI Labs Ad Kaggle dataset.
0.48
0.47
0.46
50000 100000 150000 200000 250000 300000 Training Iteration
Figure 7: The convergency of the bi- nary cross entropy (BCE) loss during training using UMEC with Roudge 0.5 x (yW))) where py(W)) denotes the Flops of the original dense model.
Each curve shows the CR-Accuracy trade-off one com- pression method. Comparing the results of the cascaded pipeline with the joint framework, Figure 8 and Figuire 9 both support the superiority of the joint framework.
79.04
78.9
70.8
79.03
78.7
79.00
78.6
70.99
78.5
jontt
Cascaded
78.98
0.4
0.5
Compression
0.6
0.7
0.8 (CR)
0.9
1.0
0.55
0.60
Compression Ratio (CR)
0.65
0.70
0.75
0.50
0.05
Figure 8: Results on the pipeline with a cas- cade of input feature selection and prediction model compression, and comparison with the corresponding jointly optimized framework.
Figure 9: Results on the pipeline with a cas- cade of embedding dimension reduction and prediction model compression, and compari- with the corresponding jointly optimized framework.
Table 2: Real-device energy consumption and latency of different methods. Best values among all compression methods are shown in bold.
Method
Acc (%)
Energy (10-3)
Latency (ms)
Ours
78.971
8,81
0.123
ECC
78.939
9.05
0.132
Handcrafted
78.963
9.32
0.140
MP
78.956
9.16
0.136
TP
78.958
9.23
0.137
Dense Model
78,978
9.49
0. 144
A.3 ENERGY CONSUMPTION AND INFERENCE LATENCY OF THE NETWORK
We evaluate the energy cost and latency of all models on a GTX 2080 GPU. Following Yang et al. (2019), we use the nvidia smi utility² monitor the energy consumption. We follow the settings in Fu et al. (2020) to measure inference latency on the aforementioned real-device. All experiments are implemented with PyTorch. As shown Table 2, our method achieves the best accuracy with the least energy consumption and latency among all compared compression methods.
Phtpps: / devel oper. down' oad, nvi di com hwtppppwwwwecccoo/////ee/////e/e smi 367. 38 .d.f
13


Reviews:
[{"review_id": "EJZ-SbrNJq", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework of a unified recommendation system, which balances the compression degree of the model and the accuracy of the model by compressing the embedding layer model, while optimizing feature selection and neural network compression. This paper proposes a resource-constrained method to simultaneously optimize model compression and ensure model accuracy, and use ADMM's solver to optimize constrained models. The experiment verifies the effectiveness of the method by comparing the accuracy of different model compression ratios.\n\nThe paper’s key strengths:\n1.\tThis article proposes a constraint optimization method to optimize model learning and model compression at the same time. This method is quite original. And the author proposed computational flops in the article, which defines resource consumption by the number of neuros input and output from the network.\n2.\tAiming at the problem of formulated constraint optimization, this paper proposes the use of ADMM to optimize the strategy. And in the appendix of the article, it provides a lot of details about optimization, which is easy to understand.\n\n\nThe paper’s key weaknesses:\n1.\tThe biggest problem of this article is that although it is proposed to optimize feature selection and model selection at the same time to achieve an effect of simultaneously ensuring model accuracy and model compression. But in fact, feature selection itself is performing sparse learning, and the resulting model will become sparse, which has a similar effect to the function of model compression.\n\n2.\tIn the experimental part, the author compares the accuracy of different methods under different compression ratios to verify the effectiveness of the method. For the experiment in Figure 2, the author showed the effect of compression ratio from 0.5 to 1.0. But I questioned why the author did not show the effect of different methods from 0 to 0.5, because the compression ratio is higher when CR is above 0.5, and there will be more ACC at this time. So I think it might be more meaningful to show the method comparison of CR between 0 and 0.5.\n\n3.\tFor formula 2, R_budget is a hyperparameter, which represents the parameter of model computation. However, there is no theoretical guarantee and experimental verification of model accuracy and model convergence under different R_budgets. I hope the author will analyze the discussion of a scenario under different R_budget settings.\n\n4.\tThe experimental scenario in this article is relatively single, and only a set of data is selected for experimental verification. I hope the author can provide more scenarios to verify the effectiveness of the method.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6klxucqLfY0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the compression of recommendation models (RMs). That is new and relatively less studied in the model compression field, but of great practical value. The main unique challenge of RM compression lies in the entanglement of compressing both the network parameters and the feature embedding inputs, and the latter often accounts for more of the computational bottleneck. \n\nTo this end, the authors proposed the UMEC framework, by integrating these two sub-tasks into one unified constrained optimization problem, solved by ADMM. Specifically, they develop a resource-constrained optimization that directly sets the target resource consumption and eases the practical usage. The authors conduct extensive experiments and demonstrate the effectiveness of UMEC by observing its superior performance than other state-of-the-art baseline methods. The paper is very well written, and the notations and technical details are clearly presented. \n\nQuestion 1: My major concern is that, although the authors reported many baseline comparisons and ablation studies, all experiments are on only one dataset (i.e., Criteo AI Labs Ad Kaggle), and one task (CTR prediction). It is unclear whether the proposed method can be generally useful or can be scaled up to industry-level large systems. \n\nQuestion 2: in Section 4.4 the authors compared with Ginart et al. (2019) for input dimension reduction, while they mentioned another prior work Joglekar et al. (2020) using AutoML to search for feature dimensions. Is it possible to also compare with the later one?\n\nQuestion 3: in Eqn (1), why only enforcing structured sparsity for the input layer? Wouldn’t it be more natural if also extended to the remaining layers?", "extra_scores": {}, "extra_text": {}}, {"review_id": "3PWLhSmoBli", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recommendation models (RM) based on ranking neural networks are nowadays widely adopted in real-world recommendation and retrieval applications. Studying the new problem of RM compression can save energy/latency in those practical systems, and hence has strong application promise.\n\nA typical RM consists of two components: a feature embedding sub-model and a prediction sub-model, both implemented by neural networks. RM compression therefore has to jointly consider the two parts, which makes it different from current CNN/RNN compressions where no input-level compression is needed in general.\n\nTo resolve this unique challenge, this paper proposed UMEC, the first unified optimization framework for the recommendation system scenarios. Unlike existing prior arts that treat feature selection and model compression as two separate problems, UMEC jointly learns both of them together, via unifying the original prediction learning goal and the model compression related hard constraints. The optimization is done by first relaxation then ADMM. \n\nOverall this paper is technically sound. The writing is in general good and easy to follow. I have the following questions and suggestions:\n\nThe motivation mentions the real latency or power consumption, but the approach uses only FLOPs as the constraint as well as for the evaluation. Is it possible for the authors to report real resource usage such as latency (ms) or energy (J)?\n\nSeveral acronyms appear before formally defined, such as ADMM (in abstract), BCE (first paragraph), etc. There are also typos like:\n-\ta RM -> an RM\n-\thyper-parameters tuning -> hyper-parameter tuning\n-\tadvertising dataset -> an advertising dataset\n-\twhile manages to -> while managing to\n-\tand adjust the dimension -> should be “adjusts”\n\nOne last nitpick: ADMM was only mentioned in the abstract and the conclusion parts, but not in the method. While I understand that their minimax optimization is essentially ADMM, this may confuse some readers. I suggest the authors to revise their optimization terms for better consistency. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-tlawgpMDv5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1.\tOverview:\n\nThe paper proposes a new unified optimization framework to solve the RM compression problem. It jointly compresses the prediction network and the feature embedding, and automatically optimizes their complexity budgets under the total resource constraint. In this way, it avoids treating selection of input feature and compression of model as two individual problems, and requires no expensive hyper-parameters tuning for either part. The novel unified model and embedding compression (UMEC) method can directly satisfy both the requirement of resource consumption of the ranking neural network model and the prediction accuracy goal of the ranking model, with end-to-end gradient-based training. \n\n2.\tMethod:\n\nThe overall idea is straightforward, sensible and solidly executed. To ease the hard resource constraint, they first reformulate the inequality constraints as soft regularizations and introduce minimax optimization with dual variables, using the DC (difference of convex functions) representation trick (4). In this way, the importance of regularization terms is automatically tuned by two dual variables in minimax. I find this handling very clever compared to directly tackling a regularized objective (1) with hard-to-tune hyperparameters. \n\n3.\tExperiments:\n\nThe authors compared with three groups of baselines: model compression, feature selection and embedding reduction. However, I think those are still not enough for showing the proposed joint optimization is indeed superior, by only comparing to each one aspect. The more natural baseline could be to compare with their sequentially cascaded pipeline, yet not end-to-end optimized. For example, what if first running an input feature selection or dimension reduction, then compressing the prediction model (using SOTA methods from each category)\n\nThe authors observed that during training, last hidden layer getting the most percentage of neurons removed and the input layer getting the least percentage. I can understand that for binary classification, the last layer can perhaps be less parameterized. But does your observation go against the practice or validity of input feature selection or dimension reduction, i.e. are you suggesting they’re not as useful for compressing RM?\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Existing works mostly focus on model compression for the classification task. This paper aims for an efficient recommendation system that can well balance the model compression and model accuracy, which therefore brings in new challenges and opportunities. The authors propose to unify the model compression and feature embedding compression and develop an effective and reasonable solution.  The concerns raised by the reviewers have been well fixed and all reviewers agree on the paper's contribution.  The paper is therefore recommended for acceptance. 
Decision: Accept (Poster)