APG: Adaptive Parameter Generation Network for Click-Through Rate Prediction
Authors: Bencheng Yan
Pengjie Wang
Kai Zhang
Feng Li
Hongbo Deng
Jian Xu
Bo Zheng
Venue: NeurIPS 2022 Accept (2022)

Abstract: In many web applications, deep learning-based CTR prediction models (deep CTR models for short) are widely adopted. 
Traditional deep CTR models learn patterns in a static manner, i.e., the network parameters are the same across all the instances. 
However, such a manner can hardly characterize each of the instances which may have different underlying distributions. 
It actually limits the representation power of deep CTR models, leading to sub-optimal results. 
In this paper, we propose an efficient, effective, and universal module, named as Adaptive Parameter Generation network (APG), which can dynamically generate parameters for deep CTR models on-the-fly based on different instances. 
Extensive experimental evaluation results show that APG can be applied to a variety of deep CTR models and significantly improve their performance. 
Meanwhile, APG can reduce the time cost by 38.7\% and memory usage by 96.6\% compared to a regular deep CTR model.
We have deployed APG in the industrial sponsored search system and achieved 3\% CTR gain and 1\% RPM gain respectively.

APG: Adaptive Parameter Generation Network for Click-Through Rate Prediction
Bencheng Yan", Pengjie Wang Kai Zhang, Feng Li, Hongbo Deng, Jian Xu, Bo Zheng Alibaba Group
China
{bencheng. ybc, adam 1f, ,eaam1ffchblll dhw.17114444 48, xiyu.x] xj, bozheng Balibaba-aababba-aaaaaaaaaaaaaaa wwwiiibaabinncco. inc. com
yeng,e pengj wpj victorlanger zk,
Abstract
In many web applications, deep learning-based CTR prediction models (deep CTR models for short) are widely adopted. Traditional deep CTR models learn patterns in static manner, i.e., the network parameters the same across all the instances. However, such manner can hardly characterize each of the instances which may have different underlying distributions. It actually limits the representation power of deep CTR models, leading to sub-optimal results, In this paper, we propose an efficient, effective, and universal module, named Adaptive Parameter Generation network (APG), which can dynamically generate parameters for deep CTR models on-the-fly fly based on different instances. Extensive experimental evaluation results show that APG can be applied to a variety of deep CTR models and significantly improve their performance. Meanwhile, APG can reduce the time cost by 38. 7% and memory usage by 96.6% compared to regular deep CTR model. We have deployed APG the industrial sponsored search system and achieved 3% CTR gain and 1% RPM gain respectively.
Introduction
Recently, deep CTR models have achieved great success in various web applications such rec- ommender systems, web search, and online ad- Formally, a regular deep CTR model can be expressed as y/ = Fe (x;) where Zg Y/4 are the input features and the pre- dicted CTR of the instance i respectively. è is the parameter, and F is usually implemented as neural network.
0.6
Cod Usere
Cold User
vertising 4 g 31 [.
ccm
608
CTR
Ago Level
Figure 1: An example of feature distribution from different users (i.e., active vs. cold users). Left: the CTR distributions are varied from different user groups and a custom pattern should be considered. Right: A common pattern is welcomed to model the similar age distributions from different groups.
Improving the performance of deep CTR models has been a very hot topic in the research and industrial areas. Existing works be broadly divided into two categories: (1) Focusing on
3;, more and more elaborated information (e.g..
user behavior features [37 |2),, multimodal information
knowledge graph B6 30
ett) is
introduced enrich feature space (i.e., 2;); (2) Focusing on J, advanced architectures (including
feature interaction modeling 4 9 31, automated architecture search [4] 26 and so on) are designed
improve the model performance.
These authors contributed equally to this work and are co-first authors.
It Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

However, few works focus on the improvement of the model parameters θ especially for the weight matrix W NxM used in hidden layers of deep CTR models P It is another orthogonal aspect for the performance improvement. Actually, most of the existing works simply adopt a static manner, i.e. all the instances share the same parameters W. We argue that such a manner is sub-optimal for pattern learning and limits the representation power of deep CTR models. On the one hand, although the common patterns among instances can be captured by the shared parameters W. it is not friendly to custom pattern modeling. Specifically, taking the industrial sponsored search system as an example, the feature distribution can be varied from different users (e.g., active vs. cold users), different categories (e.g., clothing vs. medicine), and on (see Figure (a) an example). Simply applying the same parameters across all the instances can hardly capture the characteristic of each instance from different distributions. On the other hand, the learned common pattern may not be suitable for each of the instances. For example, the shared parameters tend to be dominated by sehhfrequency features and may give misleading decision for the long-tailed instances. This leads us to the following question: Do we really need the same and shared parameters for all instances?
Ideally, besides modeling the common pattern, the parameters should be more adaptive and can be dynamically changed for different instances to capture custom patterns at the same time. Then, the representation power (or model capacity) can be enhanced by the dynamically changed parameter space. To achieve this goal. we design a new paradigm for CTR prediction. The key insight is to propose Adaptive Parameter Generation network (APG) to dynamically generate parameters depending on different instances. Firstly, we propose a basic version (Section [.T] of APG which can be expressed as y/ (Z((2)(ZZ)) where G refers neural network (c.g., MLP) and generates the adaptive parameters Wi E IR XM by the input-aware condition z; E IR However, the basic model suffers from two problems: (1) inefficient in time and memory. Directly generating the weight matrix W, of deep CTR model needs O(NMD) cost in computation and memory storage, which D times the cost in regular deep CTR model and is costly especially for a web-scale application where N, M are usually set as a large value (e.g.. ~=M=/,,00). The empirical results (Section 4.4) also show the basic model needs an extra 111x x training time and 31 x memory usage. (2) sub-effective in pattern learning. The parameter generation process is totally dependent on the condition 2,. which may only capture the custom patterns and ignore the common patterns which contribute to understand users' behaviors (see Figure (b)), leading to sub-effective pattern learning.
Then, we extend the above basic model to an efficient and effective version of APG: (1) For the efficiency, motivated by the low-rank methods [6 M which show that the weight matrix resides on a low intrinsic dimension, we parameterize the target weight matrix W; as the production of three low-rank matrices U.S.V, V where U, I. NxK Vi E K S, IR K and K & mén (M, N). In addition, the decomposed feed-forwarding is proposed to avoid the heavy computation of the weight matrix W; reconstruction. Then we further take the center matrix S, as the specific parameters which are dynamically generated to capture the custom patterns and the rest two matrices U, V as the shared parameters which are randomly initialized and shared across instances to capture common patterns. In this way, the complexity of generating the specific parameters can be easily controlled by setting a small K. As a result, APG achieves and O(KKD+NK MK) memory complexity compared to that both have O(NM) in regular deep CTR model. We empirically find APG can speed up the training time by 38.7% and reduce memory usage by 96.6% relative to a regular deep CTR model (Section 4.4 (2) For the effectiveness, apart from the natural effectiveness of APG in custom pattern learning, to model the common pattern, the shared weights U and V considered in the adaptive parameter generation. Then we further extend U and V to an over parameterization version which enriches the model capacity without any additional memory and time cost during inference. Besides, we also find there exists inherent similarity between different generated S, which implicitly model the common information (Section 4.6).
time complexity
In summary, the contributions of this paper are presented as follows: (1) We propose a new learning paradigm in deep CTR models where the model parameters input-aware and dynamically gen- erated to boost the representation power. It is orthogonal to many prior methods and is a universal module that can be easily applied in most existing deep CTR models. (2) We present APG which generates the adaptive parameters in an efficient and effective way, and theoretically analyze the
For simplicity, in this paper, we mainly discuss the weight matrix W. Our method can also be easily applied to the parameters of other modules (e.g., transformer, attention network, etc) in deep CTR models since most of them can be regarded as variety of MLP with a set of weight matrices [3
2

---

computation and memory complexity. (3) Extensive experimental evaluation results demonstrate that the proposed method is universal module and can improve the performance of most of the existing deep CTR models. we also provide a systematic evaluation of APG in terms of training time and memory consumption. Finally, we have developed APG in the industrial sponsored search system and achieved 3% CTR gain and 1 % RPM gain respectively.
2 Related Work
Deep CTR Models. A traditional CTR prediction method usually adopts a deep neural network to capture the complex relations between users and items. Most of them focus on (1) introducing abundant useful informations 37
36 30 to improve the model understanding, (2) de- signing advanced architecture 4 4 20 to achieve better performance. All of them adopt
22
S I
static parameters manner which limits the model capacity of these methods, leading to sub-optimal performance
Coarse-grained Parameter Allocating. There are some research areas that bring a coarse-grained parameter allocating strategy that may be related to our goals, including multi-domain learning [5 and multi-task learning [1% 21. Both them maintain and allocate different parameters to different domains tasks manually. However, such coarse-grained parameter allocating can hardly be extended to a fine-graines (e.g., user, item, or instances sensitive) manner. not only costs too much memory to maintain and store a large number of parameters when considering the fine-grained modeling but also lacks flexibility and generalization since the parameter allocation is manually pre-defined. We also conduct experiments compare this kind of methods (Section 4.5
Dynamic Deep Neural Networks. Our method is related to recent works of dynamic neural networks used in computer vision and natural language processing, in which the model parameters [ 32
7 1CC 20 20 and architecture
[7 can be dynamically changed. In our paper, we take the first step to bring the idea about dynamic networks into deep CTR models and develop it in real applications where the efficiency in computation and memory is extremely needed and the common and custom pattern learning are also required.
3 Method
In this paper, we denote scalars, vectors and tensors with lower-case (or upper-case), bold lower-case, and bold upper-case letters, e.g., ni (or N), æ. X, respectively.
3.1 Basic Model
In this section, we introduce the basic version of APG. Generally speaking, the basic idea of APG is to dynamically generate parameters W, by different condition 2,, i.c., W; G(z)) where G refers to the adaptive parameter generation network. Then the generated parameters are applied to the deep CTR models, i.e., Y/ FG( (æ;). Next, present (1) how design the condition 2; for different instance and (2) how to implement G.
3.1.1 Condition Design
In this paper, we propose three kinds of strategies (including group-wise, mix-wise, and self-wise) to design different kinds of z;.
Group-wise. The group-wise strategy tries to generate different parameters depending on different instance groups The purpose that, sometimes, the instances can be divided into different groups 2 and instances in the same group may have similar patterns. Thus is the representation identifying different groups. An example can be found in Appendix A.
Mix-wise. To further enrich the expression power and flexibility of the adaptive parameters, mix-wise strategy is designed take multiple conditions into consideration. Specifically, given condition embeddings {z] E Rdj E {0, 1,.,, k - 1}} of the instance i, we propose two aggregation policies to consider different conditions at the same time: (1) Inpul Aggregution. This policy firstly aggregates different condition embeddings and then feeds the aggregated embeddings into APG to obtain the mix-wis wise based parameters. The aggregation functions can be (but are not limited to):
3

---

K
X X<CmmatXX,
W
S
S. NxX
U
U
S,
V
V
Pukk
Kxo
Kar
Pod
Noci
Nodi
(a) Regular
Basic Model
(c) Low-rank
(d) Adding shared parameters
Adding over parameterization.
perameterization.
Specific parameter
parameter
Figure The comparison of different versions of APG.
Concatenation, Mcan, and Attention. (2) Output Aggregation This policy firstly feeds different 2 into different APG and obtains the corresponding parameters W respectively. Then the aggregation is applied to these adaptive parameters W.' Similarly, the aggregation function includes Concatenation, Mean, and Attention. Examples can be found in Appendix A.2
Self-wise. The above two strategies need additional prior knowledge to generate parameters. Self- wise strategy tries to use a simple and easily obtained knowledge (i.e., self-knowledge) to guide parameters generation, i.e., the network parameters are generated by their own input. For example, for the (-th hidden layer of deep CTR model, we can set z; = Z; For I-th hidden layer, 2; h h where h is the input of the 1-th hidden layer.
3.1.2 Parameters Generation
After obtaining the conditions, we adopt multilayer perceptror generate parameters depending on these conditions (see Figure 2 (b)), i...
W, reshape(MLP(x..)
where W. E RRNM 2; E TRD, and the operation reshape refers to reshaping the vectors produced by the MLP into matrix form. Then deep CTR model with APG can be expressed
1/1 a (W.::)
where the activation function. Here we take deep CTR model with MLP layer an example, and other deep CTR models can also be easily extended since most of them can be regarded as variety MLP with of weight matrices 35
3.2 Effective and Efficient Adaptive Parameter Generation Network
As introduced in Section 1 the above basic model has two problems: (1) inefficient in time and memory and (2) sub-effective in pattern learning. To address the above problems, we propose some extensions including low-rank parameterization, decomposed freedforwarding, parameter sharing, and over parameterization to parameterize the weight matrix in an efficient and effective way.
APG
U
U"
y"
Instance
DeepCTR
Low-rank parameterization. Inspired by the recent suc- cess of the low-rank methods [16] [ which have demon- strated that strong performance can be achieved by op- timizing a task in low-rank subspace, that the adaptive parameters also have a low "intrinsic rank". To this end, we propose parameterize the weight matrix W; NNM as low-rank matrix, which is the product of three sub-matrices
Model
Specthc
Figure 3: The framework of APG. For simplicity, the process of decomposed fol-forwwddnggggg is omitted in this figure.
hypothesize
U, E N S, k V, K and the rank K < min(N, M) (see Figure (c)). Formally, the weight generation process can be expressed as:
U,, S,, V; reshupe(MLP(z..)
(3)
In this paper, take MLP example, and other implementations of G can also be considered.

---

U
Deecmpooed Feed fonerriniiggg
X;
32
(6) Training Phase
(b) Inference Phase
Data
Data Row
Bhared parameter
Spedile
Model parmmnttr Now
Specific panamiter
Figure 4: An example of the decomposed feed- Figure 5: An example of over parameterization.
forwarding.
Intuitively, we set small value of K to control the time and memory cost. Meanwhile, ignoring the substantial storage and computation cost, K can also set to a higher value to enlarge the adaptive parameter space (see detailed discussion in Appendix E.. Overall, through the low-rank parameterization, we can significantly reduce dimensionality K' & min (N, M,, to enable more compact model. As result, the computation complexity of the specific parameter generation process can be reduced to D[NNK+MK*KKK (see Table The memory cost is also reduced to O(NN++MK+XKID (see Table
the parameter space, i.e.,
Decomposed Feed-forwarding. After low-rank parameterization, the Eq can be written as y: = o (W, = o(U, Vi)a)) = (VU,)S((V
(4)
Here instead of directly reconstructing the weight matrix W, by the sub-matrix production, we design decomposed feed-fore for-rdiing and apply to each sub-matrix sequentially (see Figure 4. Such de- sign helps us avoid the heavy computation of the sub-matrix production which costs O(NKK+NMK Actually, this design benefits from the low-rank parameterization which naturally supports the decom- posed feed-forwadding. Since the computation complexity in Eq is OKNK+KK+MK)), the total time cost of deep CTR model with APG per layer is reduced to O((NK+
1)) (see Table M
Parameter sharing. In this section, we present our common pattern modeling strategy. Thanks to decomposing the weight matrix into U, St, and V., it allows be more flexible. Consequently, we divide these three matrices into (1) specific parameters that capture custom patterns from different instances; (2) shared parameters that are shared across instances characterize the common patterns. Specifically, we define S, as the specific parameters since the matrix scale is totally controlled by K and is more efficient for the generation process. U and V regarded the shared parameters (see
Figure 2 (d)). Then we can rewrite Eq S and 4 as:
S (reshap((MLP(z))
(5)
y/1 (VV))(
(6)
Furthermore, such a design also contributes the efficiency. Since the size of the generated specific parameter is reduced to K x K. the computation complexity of the specific parameter generation can be further reduced to O(KKD) Eq [ and the total time cost of deep CTR model with APG is reduced to O<KKD+NK+MK+++ to O(KKD+NK +MK) where NK and MK refers to the storage of the shared parameters U and V respectively (see Table
(see Table Meanwhile the memory cost is reduced
Over Parameterization Compared with the shared weight matrix W E NxM regular deep CTR model, U and V in APG can be hardly scaled to large matrices due to the efficiency constraint with K < min(N, .M., leading possible performance drop. To address this problem and further enlarge the model capacity, we follow the idea about the over parameterization [ 5 to enrich the number of the shared parameters (see Figure (e)). Specifically, we replace shared parameters in Eq 6 with two large matrices, i.e.,
U U-U'U, == V-V'v"
where U' E NRNPP E BRPKK E KK E RPPM and P>>> K. Although U (or V) is exactly equal to U (or V'V") at the mathematical perspective, replacing V with V'v" can contribute into two folds: (1) Since P > K, more shared parameters are introduced to enlarge the model capacity [ : (2) The form as the multiple matrix production can result in an implicit regularization and thus enhance generalization [I
No Additional Inference Latency and Memory Cost. During the training phase, we can set P to large value to enlarge the model representation power. Remarkably, when P. > max( N, M), the shared
5

---

Table 1: The computation and memory complexity per layer of different versions of APG. SPG refers to the time or memory cost in the process of specifie parameter generation. R-Wi is the time cost to reconstruct W, FF refers to the time cost in the feed beed-foowarding process of regular deep CTR model. SPS refers to the memory cost to store the shared parameters. The total computation cost is the sum of SPG, R-Wi, and FF. The total memory cost is the sum of SPG and SPS. Since over parameterization dose not introduce any additional time cost, it is not depicted here.
Versions
Compatation complesity per
Memory complectit
SPG
RWWi
IF
Total Cess
SPG
SPS
Toxal Cost
We.
(UNM)
OVNA
CYNM)
Win,
CHNMD)
CONN)
CINMD)
CIMMED
DYNK+MK-KKUID
OUNM)
OHNNEMKK NWW.
DYINK
OVIIK-ME-KKKD)
DY/NK (MK RKUD)
MK RAYD iD
DYAN MK KKUD)
MR KK(I))
DVERDI
< XR)
ONKEDANK KK)
OVERDH
CYARD (MK)
parameter space can be larger than that in regular deep CTR model. During the inference phase, we explicitly pre-compute and store V, U, and use these two matrices for inference (see Figure 5) Critically, this guarantees that we can introduce abundantly shared parameters without any additional latency and memory cost during the inference phase.
3.3 Complexity
In this section, we detailedly analyze the proposed model complexity including the memory and computation complexity during the inference phase. For analysis, the parameters generation network is implemented as a single perceptron layer, and the per layer costs in a regular deep CTR model and an adaptive deep CTR model are compared. Summarization can be found in Table 11
Memory Complexity. For a regular deep CTR model, the memory cost is O(NM) per layer, i.e., storing the shared weight matrix. For APG, the memory cost comes from two parts: (1) The memory cost of generating S in Eq S is O(KKD); (2) The shared parameters U, V cost O(NN + M)K) during the inference phase. Then the total memory complexity of APG is O(KKD+NK+MK) per layer.
Computation Complexity. A regular deep CTR model needs O(NM) per layer for the feed-forward computation. APG needs O(KKD) in Eq 5 to calculate the specific parameters. Meanwhile, the ferdrdiwngdiin computation of deep CTR model O(NK+KK+MK) by the decomposed feed- forwarding in Eq 8. In total, the computation complexity is O/KKD+NK<KK+MK)
In summary, APG has O(KK in memory cost and O/KKD+NK*KK+*NM in compu- tation cost. Since K < minn N, M) and D is usually set smaller than N, M, the memory and computation cost of APG can even be much smaller than that (i.e., O(NM) in regular deep CTR model. The experimental results in Section 4.4 also show the efficiency of APG.
4 Experiments
4.1 Experimental Settings
The detailed settings including datasets, baselines, and training details are presented in Appendix B
Datasets. Four real-world datasets are used including Amazon, MovieLens, IAAC, and IndusData. The first three are public datasets and the last is an industrial dataset.
Baselines. Here, compare our method with two kinds of methods (1) Existing CTR predic- tion methods include (DL PNN 24), FIBINET(1]), DIFM DeepFM DCNBI], and AutoInt 27;; (2) Coarse- grained parameter allocating methods. multi-task learning: MMoE 19 and multi-domain learning: Star
25
Training Details. See Appendix B.3 for the detailed introduction.
4.2 Performance Evaluation with Existing Deep CTR Models
Results on public datasets. We first apply APG on CTR prediction tasks on public datasets. APG universal module and can be applied to most of the existing deep CTR models. Hence, to evaluate the performance of APG, we apply APG to various existing deep CTR models, and report the results of the original model (denoted as Base) and the model with APG. Here, AUC (%) [ score is reported We can find that with the help of APG, all of
as the metric The results are shown in Table 2
Note 0.1 % absolute AUC gain regarded as significant for the CTR task
R 28

---

Table 2:
The AUC (%) results of Click-Throught Rate (CTR) prediction different datasets. Note Base refers to the original results of the corresponding methods and Base+APG refers to the results with the help of APG. Ave is the average results across all cases. A refers the improvement of APG compared to Base.
Data
Method
WDL
PNN
FIBINET
DIFM
DeepFM
DCN
AutoInt
Ave
Base
79.21
79.5
79.78
79.84
79.3
79.29
79.36
79,46
MovicLens
Basc+APG
79.73
79.67
79.82
79.94
79.60
79.62
79.64
79.70
A
+0.39
+0.17
+0.04
+0.10
+0.30
+0.33
+0.28
+0.24
Base
69.15
69.16
68.88
69.17
69.1
68,98
68.96
69.06
Amazon
Base+APG
69.43
69.37
69.19
69.23
69.43
69.42
69.38
+0.22
+0.21
+0.31
+0.06
+0.33
+0.44
+0.42
0.28
Base
65.17
65.3
65.15
65.76
65. 64
64,78
64,99
65.26
IAAC
APG
65.94
65.87
66.15
66.42
66.17
66.39
66.21
66.15
A
+0.77
+0.57
+1.0
+0.66
+0.53
+1.61
+1.22
+0,91
Table 3: The settings of different APG versions.
Table 4: The AUC results of the evaluation of different versions of APG. refers the difference relative to Base.
Version
Annotation
Base
Wx.
WDL
g as the baseline and the hackhone
vl
Wie,
Basic model (Section 3.I
LAAC
AweABC)
Ave(A)
Base
79.21
69.15
65.17
71.18
v2
Low Lowkrank paramelerization
79.51
09.33
66.52
71.45
+0.27
Decomposed lood-forwarding
v2
79.49
69.24
65.61
71.45
+0.27
((8.(Va)))
Parameter sharing
v4 79.61
69.36
65.78
71.58
+0.40
Over parameterization
71.70
+0.52
the methods achieve a significant improvement on all datasets. For example, the gains of DCN is 0.33% ~ 1.61% (other methods also can obtain similar improvement). It demonstrates that (1) giving adaptive parameters for models can enrich the parameter space and learn more useful patterns for different instances; (2) the proposed APG is universal framework that can boost the performance of many other methods. Such nice property encourages APG to be applied various scenarios and various methods.
Results on industrial application. We also develop APG in the industrial sponsored search system, and achieve 0.2% AUC gain on the industrial dataset, 3% CTR gain and 1% RPM (Revenue Per Mile) gain during online A/B test. Detailed analysis are presented in Appendix
4.3 Effectiveness Evaluation
In this section, we implement various versions of APG (see Table 3 and conduct experiments to detailedly evaluate the effectiveness of APG. The AUC results are reported in Table Note since the design of decomposed fee-rforwrdiing does not influence the AUC performance of APG, we do not compare version v3 in this section.
The impact of the basic model. Compared with the base, v1 achieves significant improvements on AUC results overall datasets. It demonstrates the effectiveness introduce specific parameters to give a custom understanding of different instances.
The impact of the low-rank parameterization. Since the parameter matrix resides low intrinsic dimension П. encourages a pewffofformence drop when adopting a low-rank based method to reduce the computation and memory cost. Considering the effectiveness, vl and v2 do not provide much performance difference. Both of them achieve high performance and perform much better than Base. More importantly, by low-rank parameterization, we can generate adaptive parameters in more efficient way (see Section 4.4 for details).
The impact of the parameter sharing. In APG, we introduce the shared parameters to characterize common patterns. Comparing the performance versions v2, and v4 can further improve the performance by introducing the shared parameters, demonstrating the effectiveness of APG in common pattern modeling. Furthermore, sharing parameter also contributes to the efficiency, due to fewer specific parameters generated (see Section 4.4).

---

Table 5: The inference time and memory cost for different versions of APG. A is the relative difference with respect to Base.
[393,64,32116]
[393,128,64,2]]
(393,256,128,64]
[393,512,256,128]
[393,1024,512,256
Method
Cost
A
Cost
A
Cost
A
Cost
A
Cost
A
Time
Base
4.52
5.27
5.89
7.12
13.04
v1
16.52
265.5%
50.31
854.6%
126.27
2043.8%
420.33
5803.5%
1462.78
11117.6%
v2
13.83
206.0%
39.53
682.8%
90.43
1435.3%
363.07
4999.3%
1065.35
8069.9%
v3
6.31
39.6%
5.42
7.3%
5.71
-3.1%
6.31
-1 11.4%
11.23
-13.9%
v4
4.49
-0.7%
4.78
-5.3%
5.46
-7.3%
5.89
-17.3%
7.99
-38.7%
Memory (M)
Base
0.50
0.89
1.93
4.61
12.91
vl
11.21
2160.1%
24.20
2613.0%
56.32
2818.1%
144.84
3041.9%
419.11
3146.4%
v2 0.74
50.0%
0.91
1.7%
1.21
-37.3%
1.98
-57.0%
3.22
-75.1%
v4
0.27
-45.0%
0.29
-68.0%
0.31
-84.1%
0.35
-92.4%
0.44
-96.6%
The impact of the over parameterization. Comparing the performance with or without over parameterization (i.e., v4 vs. v5), it shows that v5 performs better than v4 in all cases, which indicates adding more shared parameters can enrich the model capacity and lead to better performance.
In addition, we also give detailed analysis about the impact of the condition design in Appendix D and and the impact of the pyper-parameters in Appendix D Furthermore, the influence to different frequency instances are also discussed in Appendix F
4.4 Efficiency Evaluation
In this section, we evaluate the time and memory efficiency of our proposed method. To this end. we train different versions (see Table 3) of APG on the dataset IAAC and analyze the influence of each extension introduced Section 2.2 Since over parameterization does not introduce any cost, it is not considered here. The decomposed feed-forwarding does not bring extra memory cost, it is not discussed the memory usage. For all versions, we set K 4 and the backbone (also the base) is WDL with 3 hidden layers. We gradually increase the number of hidden units from [64,32,16] to 1022,,12,2566 with a fixed inputt shape=393 to evaluate the memory and time cost in different model scales. In Table we report the inference time per epoch, memory usage of each version.
For the basic model (v1), although it achieves high performance (see Table 4h, it is time expensive (265.5% ~ 11 11..6% relative to Base) and memory costly (2160. % ~ 31 3146.6% relative to Base). Such an inefficient model can hardly be accepted in web-scale applications.
When introducing the low-rank parameterization (see v2 in Table compared to the basic model, the inefficiency problem in v2 is addressed across all cases. Moreover, compared with Base, vl can reduce memory usage (e.g., -75.1% 55.1% in the large scale model). Note when we set N, M to a small value (c.g.. [64,32,16] or [123,,,.32], the memory cost. theoretically in O((NK+MK)D) sensitive with K and D, leading to a little increasement. For the time efficiency, the contribution of the low-rank parameterization can be summarized as follows: (1) Although v2 still needs a high time requirement due to the weight matrix W. reconstruction, compared with v1, the overall time cost of v2 is decreased. (2) The low-rank parameterization naturally contributes to the decomposed feed- feed-forwarding which plays a key role in efficient learning.
more
For version v3 which adopts decomposed feed-forwarding, it is free from the high computation of reconstructing the weight matrix W, and achieves great improvement by -13.9% in time cost when considering large scale model. In addition, since v3 does not introduce any extra memory cost, has the same memory usage as v2. Note in the small model, the computation is far from the GPU bottleneck and time cost insensitive with GFlops, leading to less improvement.
"Since in this paper we mainly focus on the improvement of the hidden layers, we only count the memory cost of these hidden layers and the other parts (e.g., embedding layers) not included here.
8

---

Table 6: The comparison with coarse-grained parameter allocating methods. Mem refers to the memory cost (M) of the hidden layers.
Moviel.cm cens
Amazon
IAAC
Setting
AUC
Mcm
Setting
AUC
Mem
Setting
AUC
Mem
Base
79.21
1.29
69.15
1.51
65.17
1.3
Base+MMoE/s
movie
79.31
123.42
item
69.17
315.93
item
65.22
332,97
Basc+MMoEgg
movie gender
79.28
32.91
item cutegorry
69.14
5.78
item brand
65.20
70.53
Baso+Starya
movie
79.28
3784.23
item catlgooy
69.20
11034.02
item
65.36
13753.71
Base+Stur.
movie gender
79.25
311.56
69.20
42.84
item brand
65.28
28666.88
movie
79.58
0.24
69.35
0.26
item
65.76
0.38
Basc+APGic
movie gender
79.46
0.24
item catleoory
69.30
0.26
item brand
65.59
0.38
When introducing the sharing parameters, v4 only needs to generate a small weight matrix, where time and memory cost (i.e., O(KKDJ) in the generation process are free from the scales of model. Thus, the memory requirement of v4 is best among all cases, reducing memory by -45% to -96.6% relative to Base. v4 also speeds up inference time substantially, by -0.7% to -38.7% relative to Base. It also supports the theoretical complexity analysis in Section 3.3
Overall, such nice properties in terms of high performance, efficient memory usage, and low time requirement of our proposed model are welcomed for web-scale applications.
4.5 Comparison with coarse-grained parameter allocating methods
In this section, we compare the performance with coarse-gras gaaseegrained parameter allocating methods (CGPMs), including Star and MMoE. Specifically, we conduct two settings (see Table O i.c., coarse- grained (cg) and fine-grained (fg). for each method. For the former, methods adopt coarse-grained parameter modeling strategies, e.g., developing different parameters for different movie genders in MovieLens. For the latter, methods adopt fine- fine-grained parameter modeling strategies, e.g., developing different parameters for different movies in MovieLens. The results are presented in Table 3 We can find, compared with Star and MMoE, APG shows superiority in efficiency and effectiveness. The reasons are that (1) the specific parameters of APG are dynamically generated on-the-fly, without any need to store these specific parameters, and the architecture of APG is also designed efficiently to further save memory. For Star and MMoE, they have to store these specific parameters due to the manually allocating strategies used CGPMs. Even worse, when comes to a fine-grained setting, more parameters are maintained. (2) APG has better generalization than Star and MMoE, leading to better performance. Actually, APG adopts parameter generation manner which provides a potential opportunity for generalization, and it has the ability imply the inherent similarity between different specific parameters (see Section 4.6 for additional experiments). Especially for the fine-grained setting where there may be a lack of enough instances train the specifie parameters, the inherent connections among different specific parameters are helpful for parameter learning.
4.6 Visualization
In this section, we visualize the generated specific pa- rameters. Specifically, we take the group-wise strategy on Amazon as an example. The item category is set the con- dition, and there is a total of 10 different categories. Then we plot the generated specific parameters (i.e., S.) into 2-D space by PCA
MI5
Caf ait PIOE
33 The visualization is presented in Figure 6 Each point refers to the specific parameters gen- erated for one specific category. Interestingly, the observed groupings (i.e, in the same dashed circle) correspond to similar categories. This shows that the learned specific parameters by APG are meaningful and can capture the relations among different specific parameters implicitly.
34
Figure 6: The visualization of the gener- ated specific parameters.

---

Conclusion
In this paper, we propose an efficient, effective, and universal module to adaptively generate parame- ters for different instances. In this way, the model can carefully characterize the patterns for different instances by adopting different parameters. Experimental results show that with the help of APG, all of the existing deep CTR models can make great inprovements, which also encourages wide application for APG. Furthermore, the effectiveness and efficiency of APG are also analyzed in detail. Currently, APG requires users to set some hyper-parameters. e.g., condition strategies, K, P. and etc. In the future, we will attempt to automatically implement APG with different settings for different situations.
References
[1] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
[2] Liangliang Cao, Jiebo Luo, Andrew Gallagher, Xin Jin, Jiawei Han, and Thomas Huang. Aworldwide tourism recommendation system based on geotaggedweb photos. In ICASSP, pages 2274-2277. IEEE, 2010.
[3] Xu Chen, Hanxiong Chen, Hongteng Xu, Yongfeng Zhang, Yixin Cao, Zheng Qin, and Hongyuan Zha. Personalized fashion recommendation with visual explanations based on multimodal attention network: Towards visually explainable recommendation. In SIGIR, pages 765-774, 2019.
[4] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
[5] Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful enn via asymmetric convolution blocks. In ICCV, pages 1911-1920, 2019.
[6] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. Diverse branch block: Building a convolution as an inception-like unit. In ICCV, pages 10886-10895, 2021.
[7] Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, and Baoquan Chen. Decouple learning for parameterized image operators. In ECCV, pages 442-458, 2018.
[8] Tom Fawcett. An introduction to roe analysis. Puttern recognition letters, 27(8):861-874, 2006.
[9] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfim: factorization-machine based neural network for ctr prediction. arXiv preprint 173.v:1703.04247, 2017.
[10] David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. CoRR, abs//609..00 (00.009990000 2016.
[11] Li He, Hongxu Chen, Dingxian Wang, Shoaib Jameel, Philip Yu, and Guandong Xu. Click- through rate prediction with multi-modal hypergraphs. In CIKM, pages 690-699, 2021.
[12] Tongwen Huang, Zhiqi Zhang, and Junlin Zhang, Fibinet: combining feature importance and bilinear feature interaction for click-through. rate prediction. In RecSys, pages 169-177, 2019.
[13] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. NIPS, 29, 2016.
[14] Manas R Joglekar, Cong Li, Mei Chen, Taibai Xu. Xiaoming Wang, Jay K Adams, Pranav Khaitan, Jiahui Liu, and Quoe V Le. Neural input search for large scale recommendation models. In KDD, pages 2397, 2020.
[15] Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. Melu: Meta-learned user preference estimator for cold-start recommendation. In KDD, pages 1073-1082, 2019.
10

---

[16] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint AXXv:1804008833 2018.
[17] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, pages 2736- 2744, 2017.
[18] Wantong Lu, Yantao Yu, Yongzhe Chang, Zhen Wang, Chenhui Li, and Bo Yuan. dual input-aware factorization machine for ctr prediction. In IJCAI, pages 3139-3145, 2020.
[19] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of mixture-of-experts. In KDD, pages 1930- 1939, 2018.
[20] Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint (exxxx:206.044400 2021.
[21] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hehert. Cross-stitch networks for multi-task learning. In CVPR, pages 3994-4003, 2016.
[22] Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang. Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. In CIKM, pages 2685-2692, 2020.
[23] Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contex- tual parameter generation for universal neural machine translation. In EMNLP, pages 425-435, 2018.
[24] Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction, In ICDM, pages 1 1149-11 1 1154. IEEE, 2016.
[25] Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, and Xiaoqiang Zhu. One model to serve all: Star topology adaptive recommender for multi-domain ctr prediction. CIKM 2021, 2021.
[26] Qingquan Song, Dehua Cheng, Hanning Zhou, Jiyan Yang, Yuandong Tian, and Xia Hu. Towards automated neural interaction discovery for click-through rate prediction. In KDD, pages 945-955,200 2020.
[27] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, pages 161--1170, 2019.
[28) Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, pages 1 1611-1170, 2019.
[29] Yi Tay, Zhe Zhao, Dara Bahri, Don Metzler, and Da-Cheng Juan. Hypergrid transformers: Towards single model for multiple tasks. 2021.
[30] Hongwei Wang, Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. Knowledge graph convolu- tional networks for recommender systems. In WWW, pages 3307-3313, 2019.
[31] Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In ADKDD' 17, pages 2017
[32] Ze Wang, Zichen Miao, Jun Hu, and Qiang Qiu. Adaptive convolutions with per-pixel dynamic filter atom. In ICCV, pages 12302-12311 2021.
[33] Svante Wold, Kim Eshensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.

---

[34] Zhihang Yuan, Bingzhe Wu, Guangyu Sun, Zheng Liang, Shiwan Zhao, and Weichen Bi S2dnas: Transforming static enn model for dynamic inference neural architecture search. In ECCV, pages 175-192. Springer, 2020.
[35] Weinan Zhang, Jiarui Qin, Wei Guo. Ruiming Tang, and Xiuqiang He. Deep learning for click-through rate estimation. IJCAI 2021 (Survey Track), 2021.
[36] Jun Zhao, Zhou Zhou, Ziyu Guan, Wei Zhao, Wei Ning, Guang Qiu, and Xiaofei He. Intentge: scalable graph convolution framework fusing heterogeneous information for recommendation. In KDD, pages 2347-2357, 2019.
[37] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through prediction. In KDD, pages 1059-1068. ACM, 2018.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [No] Not
applicable.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] The theories in this paper can be deduced in a straightforward manner.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [No] All data, instructions and syper-paramese
explicitly written in the main paper and/or in the Suppl (see Appendix B The code is currently proprietary.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and Appendix B
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] But we average the results with different random seeds after running experiments multiple times
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] We provide the URLs of the public
datasets
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "gWqCpdDspSP", "reviewer": "Reviewer_gHsX", "summary": "This paper aims to improve an important application -- CTR prediction.\n\nThe authors claim that a static way of parameterization is suboptimal, where the weight of CTR network is shared by all the instances.\nInstead, they prefer a dynamic way of parameterization, where a personalized weight is generated for each instance.\n\nSpecifically, they apply a hypernetwork to generate the weight for each instance. Moreover, they apply techniques like low-rank parameterization, Decomposed Feed-forwarding, Parameter sharing and Over Parameterization to improve the efficiency and effectiveness of their method.\n\nThey present the experimental results on both public datasets and online real-world search system.", "questions": "May I ask if authors could read more related papers of hypernetworks and put this work into that context and re-evaluate the novelty?\n\nIs this one the first paper that introduce hypernetworks into CTR prediction?\n\nThe current version seems to be a better fit to some application-toward conferences like SIGIR, KDD, considering this paper neither provide deep and theoretical analysis about why same and shared parameters for all instances is suboptimal nor present original idea of hypernetworks.\n\nIt is no doubt that this paper is a good application paper that apply hypernetworks for CTR prediction.", "limitations": "This paper has no potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The paper is well-organized. The motivations for using low-rank parameterization, Decomposed Feed-forwarding, Parameter sharing and Over Parameterization are presented in a very clear way. The figures are very clear and beautiful.\n\n(2) The experiments are comprehensive, including public datasets and online real-world search engine.\n\nWeakness:\n\n(1) The novelty is limited. The core idea of this paper is using a shared hypernetwork to generate weight for each instance or group, which was originally proposed in \"Hypernetworks, David Ha, Andrew Dai, and Quoc V Le. 2016.\" and has been widely applied like in \"Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, ACL 2021\" The techniques like low-rank parameterization and Parameter sharing are also not new, which could be found in \"HyperGrid Transformers: Towards A Single Model for Multiple Tasks, ICLR 2021.\"\n\n(2) Related work is missing. Hypernetwork should be an important piece of the related work in the paper but authors seem to completely unaware of the existing of this piece.\n\n(3) The improvement is limited on public datasets. In Table 2, it seems to me that the improvement of APG is not that large. For example, the average gain of APG is only 0.24 in terms of AUC on MovieLens dataset. In particular, the SOTA CTR architecture DIFM has only been improved 0.1, 0.06 and 0.66 on the three datasets, without significant test, it is really hard to be convinced that this method can substantially improve the CTR performance.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_v42aLEn2S", "reviewer": "Reviewer_QUZk", "summary": "This paper studies the problem of enabling input-aware model parameters which are dynamically generated in order to boost representation power of deep CTR (click-through rate) prediction models. A novel and general method APG (Adaptive Parameter Generation) is proposed, being able to work together with a variety of existing CTR prediction models. By means of employing several techniques such as low-rank parameterization, decomposed feed-forwarding, parameter sharing and over parameterization, APG generates adaptive parameters in an efficient and effective way. Computational complexity is analyzed carefully; empirical results demonstrate APG's efficiency and effectiveness. Notably, APG is claimed to be deployed in an industrial sponsored search system, achieving performance gain in online A/B testing.", "questions": "In Line 283 “Since over parameterization does not introduce any cost, it is not considered here”: Why? It seems that over parameterization would introduce more computational cost during training.\n\n\n", "limitations": "The writing has to be polished carefully. Several typos are listed as below:\n\n* Line 87: analysis → analyze\n* Line 106: is lack → lacks\n* Line 107: kinds → kind\n* Line 118: General → Generally\n* Line 215: detailed analyze → analyze in detail\n* Line 242: Since APG → APG\n* Line 282: analysis → analyze\n* Line 349: detailed analyzed → analyzed in detail", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* **Originality**: Dynamic neural networks are widely studied in computer vision and natural language processing, in which model parameters and architectures could be input-aware and thus dynamically generated. It is claimed that this paper is the first to bring the idea about dynamic neural networks into deep CTR prediction models.\n* **Clarity**: The paper is well organized and clearly presented. However, typos are widespread throughout the text.\n* **Significance**: Computational complexity of the proposed APG is carefully analyzed, and the efficiency is also validated empirically. A wide variety of baselines are employed to demonstrate the generality of APG. It is noteworthy that APG is claimed to be deployed in an industrial sponsored search system, achieving performance gain in online A/B testing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "PxO8VzNqYr", "reviewer": "Reviewer_Zx1w", "summary": "This paper proposes to learn input-aware parameters in deep CTR models to boost its representation power. A clear method iteration trace is presented with clear design and solid analysis. The final version archives both efficiency gain and accuracy improvement. Most impressively, the method is tested in industrial production system. ", "questions": "The novelty of such fine-grained input-aware parameter allocation methods. The related work section states that this is almost first work to apply such heuristic, right?\n\nDo we have benchmark on the effectiveness of various condition designs?\n\nThe gain from v4 to v5 is huge, why an over parameterization trick so powerful? If parameter sharing is beneficial, base model should not be such poor?", "limitations": "The method iteration are surprisingly positive which may imply some deep insights. Let's say  \n- base -> v1 parameter personalization improves performance \n- v1 -> v2 low rank parametrization doesn't change performance too much.\n- v2 -> v4 appropriate parameter sharing improves performance \n- v4 -> v5 over-parameterization (more sharing parameters) largely improves performance\n\nIt seems model favors heavy parameter sharing plus certain parameter personalization. More effort is needed to study the root factor and a mode direct approach based on it.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe paper is well written and quite readable. Methods' iteration records are plausible.\n \nWeaknesses\nCurrent baseline on parameter allocation is too weak. This is not necessarily a weakness since advanced baseline to learn fine-grained parameters for model parameters may not be existed. I am not an expert of related literatures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8YuC0fIwpl3", "reviewer": "Reviewer_2in4", "summary": "The authors of the paper address the problem of click-through rate (CTR) prediction customized to different instances. They propose an adaptive parameter generation (APG) method for generating weight matrices in instance-adapted CTR systems. Generating the entire weight matrix directly is very time and memory consuming. To reduce time and memory complexity, they propose to decompose the weight matrix into a product of three vectors, namely S, U, and V. Only S is dynamically generated to capture different instances of custom patterns, while U and V are shared instances that capture common patterns. Results from three public datasets and real search systems are shown.", "questions": "1. In a group-wise conditional design, is there any extra effort required to divide instances into different groups?\n2. S is used to capture custom patterns because of its low rank. Have you tried using U and V for this purpose? Did they achieve similar results?\n3. V5, over parameterization, appears to be a separate technology. How it connects to APG.\n4. What is the APG version reported in Table 2?\n5. The training time and memory complexity of v5 is not provided in Table 5. Is it similar to the basic version? What is the trade-off between the efficiency and effectiveness of v5?", "limitations": "1. Can provide a comparison between using generated and fixed S in the current v1-v5 version.\n2. Can provide comparisons with existing methods of automatically generating weight matrices (e.g., in the AutoML domain).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. By decomposing the matrix and then decomposing the feedforward, the time and memory complexity of the CTR model with APG is greatly reduced.\n2. Improve scalability to large inputs through over-parameterization.\n3. The paper is well written.\n\nWeakness: \n1. Does not provide a comparison of between generating the S vector and using fixed S vector on the current v1-v5 versions. \n2. It does not provide a comparison with existing methods that automatically generate weight matrices (e.g., in the AutoML field). \n3. A small typo on line 210.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept