Dynamic Parameterized Network for CTR Prediction
Authors: Jian Zhu
Congcong Liu
Pei Wang
Xiwei Zhao
Guangpeng Chen
Jin Jun Sheng
Changping Peng
Zhangang Lin
Jingping Shao
Venue: ICLR 2022 Submitted (2022)
Keywords: Recommendation System
Feature modeling
User Behavior modeling
Dynamic Network

Abstract: Learning to capture feature relations effectively and efficiently is essential in click-through rate (CTR) prediction of modern recommendation systems. Most existing CTR prediction methods model such relations either through tedious manually-designed low-order interactions or through inflexible and inefficient high-order interactions, which both require extra DNN modules for implicit interaction modeling. In this paper, we proposed a novel plug-in operation, Dynamic Parameterized Operation (DPO), to learn both explicit and implicit interaction instance-wisely. We showed that the introduction of DPO into DNN modules and Attention modules can respectively benefit two main tasks in CTR prediction, enhancing the adaptiveness of feature-based modeling and improving user behavior modeling with the instance-wise locality. Our Dynamic Parameterized Networks significantly outperforms state-of-the-art methods in the offline experiments on the public dataset and real-world production dataset, together with an online A/B test. Furthermore, the proposed Dynamic Parameterized Networks has been deployed in the ranking system of one of the world's largest e-commerce companies, serving the main traffic of hundreds of millions of active users.

Under review as a conference paper at ICLR 2022
DYNAMIC PARAMETERIZED NETWORK FOR CTR PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning capture feature relations effectively and efficiently is essential in click- through rate (CTR) prediction of modern recommendation systems. Most existing CTR prediction methods model such relations either through tedious manually- designed low-order interactions or through inflexible and inefficient high-order interactions, which both require extra DNN modules for implicit interaction model- ing. In this paper, we proposed a novel plug-in operation, Dynamic Parameterized Operation (DPO), to learn both explicit and implicit interaction instance- wisel.. We showed that the introduction of DPO into DNN modules and Attention modules can respectively benefit two main tasks in CTR prediction, enhancing the adap- tiveness of feature-based modeling and improving user behavior modeling with the instance-wiss locality. Our Dynamic Parameterized Networks significantly out- totte-of-thhaare art methods in the offline experiments and real-world production dataset, together with an online A/B test. Furthermore, the proposed Dynamic Parameterized Networks has been deployed in the ranking system of one of the world's largest e-commerce companies, serving the main traffic of hundreds of millions of active users.
performs state-of-the
the public dataset
INTRODUCTION
Click-through rate (CTR) predictied, which aims to estimate the probability of a user clicking an item, of great importance in recommendation systems and online advertising systems (Cheng et al. 2016)] Guo et al.] 2017 Rendle 2010 Zhou et al.) 2018b). Effective feature modeling and user behavior modeling are two critical parts of CTR prediction.
Deep neural networks (DNNs) have achieved tremendous success variety of CTR prediction methods for feature modeling (Cheng et al. 2016; Guo et al. 2017: Wang et al. 2017). Under the hood, its core component is linear transtormation followed by nonlinear function, which models weighted interaction between the flattened inputs and contexts by fixed kernels, regardless of the intrinsic decoupling relations from specific contexts (Rendle et al. 2020). This property makes DNN learn interaction in an implicit manner, while limiting its ability is often captured by feature crossing component (Rendle) 2010 Song et al., 2019). Most existing solutions exploit a combinatorial framework (feature crossing component DNN component) to leverage both implicit and explicit feature interactions, which is suboptimal and inefficient (Cheng ct al. 2016 Wang et al. 2017). For instance, wide & deep combines a linear module in the wide part for explicit low-order interaction and a DNN module learn high-order feature interactions. Follow-up works such as Deep & Cross Network (DCN) follows a similar manner by replacing the wide part with more sophistic networks, however, posits restriction to input size which inflexible.
to model explicit relation, which
Above-mentioned methods pay little attention to user behavior modeling. Recently, attention-basee methods like DIN and DIEN have attracted many interests that attempt to capture user preferences based on users' historical behaviors (Zhou al., 2018b] 2019 Feng et al. 2019). With regard to the interaction of characteristics, the use of attention mechanisms in these methods can be treated as an explicit modelling of the interaction of characteristics while neglecting the modelling of implicit interactions of characteristics.
The methods mentioned above either model implicit and explicit feature interactions isolated or adopt a suboptimal way to combine them, which can be inefficient. In this work, we aim to address these problems by introducing a small MLP layer that dynamically generates kernels conditioned

---

Under review as a conference paper at ICLR 2022
by the current instance to capture both implicit and explicit feature interactions. The core idea is to first generate context weights and biases from the context stream, and then aggregate them with the input stream adaptively. We formulate generic function and implement it with an efficient dynamic parameterized operation (DPO). The first weight generator projects contextual features into nigh-dimensional representation, which models implicit conditional bias. The second feature aggregator aims to fuse input features and projected contextual representation in a multiplicative way, e.g., matrix multiplication and convolution, maintaining both low- and high-order information.
For caature-based modeling, we introduce feature-based DPO where the weight-generate operation dynamically produces instance-wis filters conditioned on the embedded context. The feature- aggregate function then applies instance-wiss -wine filters to the flattened input by matrix multiplication, allowing to learn multiplicative features. In particular, we further propose a new class of DPO, called field-based DPO, which is not only instance-specifi but also field-specific. In that case, the filters vary from field to field and from instance instance, allowing more complex interactions along the field dimension.
For user behavior modeling, we introduce sequence-based DPO that consists of two variants: behavior- behavior dynamic operation and query- berrybbb dynamic operation. A representative method of dynamic convolution (Chen et al. 2020 Yang et al., 2019) shares the convolution kernel, which is generated by the global average of the inputs. Similarly, (Wu et al.] 2019) proposed DyConv, lightweight fine-grained convolution that depends only on time-step, remnfrringg the encoder- based language modeling framework. However, our methods incorporate both local and global information as they jointly use locality-aware methods (e.g., convolution or separable convolution) followed by global average pooling layer to produce instance-wise weights. The query-behavior dynamic operation is specialized designed for the decoder-based framework in CTR prediction, aiming to capture target-behavior dependency.
To our best knowledge, this is the first attempt to extend the business of dynamic neural networks to CTR prediction with extensive experiments on two fundamental scenarios. The comprehensive study against existing solutions validates the superiority of our proposed method. Moreover, we demonstrate that incorporating DPO into the real-world ranking system is beneficial.
Our contribution can be summarized as followed:
We propose a generic formulation for capturing multiplicative interaction via weight- generate and aggrreee function, termed DPO.
For feature- based modeling, we propose two variants, named field-based and feature- DPO, offering a unifying view of implicit and explicit feature interaction. Decomposing these operations, we find they implicitly inherit low- and second-order representation.
For user behavior modeling, we propose two sequence-based variants: and query-behaving DPO. The first one computes locally perceptual dynamic filters and the second one learns target-behavine dependency in a multiplicative manner. We demonstrate that such operations can benefit the self-attention layers by higher computational efficiency through modeling locality as inductive bias.
The proposed dynamic parameterized networks outperform state-of-the-art methods by a significant margin on both public and real-world production datasets. We also give prehensive study about the relationship of our proposed methods to previous Factorization Machine (Rendle (2100) and CrossLayer (Wang et al. 2017). We further demonstrate the effectiveness and superiority of our method with an online A/B test in real-world applications by incorporating it into the fine-rank stage of the real-world ads system.
2
METHOD
We first review the mainstream approaches of feature-based and user behavior (sequence-based) modeling under the situation of CTR prediction[] After that, we introduce DPO and provide several specific instantiations designed for traditional feature- based and sequence-based modeling.
RRllted work is in Appendix due space limitation.
2

---

Under review as a conference paper at ICLR 2022
2.1 PRELIMINARY
Traditional CTR prediction methods mainly predict a probability of user click an item, which serves fundamental evaluation criterion for computing advertising systems. Typically, in a given scenario (the contexts), users click on certain items (item profiles) based on their own needs (query) and pautorferences (user profiles). Consequently, a model considers four fields of features, i.c., query, user profile, item profile, and contexts to predict: CTR F(query, user profile, item profile, contexts)
(1)
where item and user profiles contain up to tens of fine the-grained static attributes depending on the specific circumstances.
Sequence-based CTR prediction involves user behaviors additionally:
(2) where the models can learn from the behaviors that have occurred under certain contexts and query in the past to make judgments on the current items. As mentioned in KFAtt (Liu et al. 2020), the behavior module can be formulated as: 0,- ser Behavior (g, kI:7, U1.T), where k1.7 and V are given historical clicked items and corresponding query words. The most used strategy is to adopt the the self-attention mechanism (Vaswani et al., 2017). which naturally learns multiplicative interaction between query and the historical behavior.
CTR F(query, user behavior, user profile, item profile, contexts)
2.2 FORMULATION
Namely, multiplicative interaction (Jayakumar al. 2020) has been proposed to fuse two different sources of information with the goal of approximating function [aarget( z) E IRS where æ and are the input and context respectively. Similarly, give generie formulation of DPO in CTR prediction task as:
y, C(z) Σ f( ((m))((2)))))
Here is the index of position (in the field, sequence). whose response is calculated with the generated output of & over all existing positions. x is the input embedding, while # denotes any specified context. The generate function g aims compute dynamic weights and bias followed as one of the inputs of the pairwise aggregate function f, which learns the interactive features reflecting the relationship between æ; and zj. The output is finally normalized by factor C(z).
MLP and convolution typically process input and context features in an additive way with fixed weights. While in Eqn. (3) using instance-wiss wise generated weights and bias from contexts z, the additive nature is transformed to multiplicative. DPO is also different from bilinear layer (Lin et al. 2015 He & Chua 2017) for Eqn. computte representation based on the generated weights over all positions, whereas bilinear layer aggregates information over all positions between æ and z, leading to large memory consumption. Furthermore, our generated dynamic weights can maintain more local information, which complements the global counterpart, e.g., self-attention. DPO is flexible block and can easily work together with MLP and self-attention layers.
2.3 FEATURE-BASED DPO
Given E I.th and Ra inputs and context, due the lack of position information, the generic formulation degrades as y = f(æ; g(z; 0)). For simplicity, we consider f in the form of a linear transformation: f(x; z) Wzz)a, where W(z) is an instance-wise wwo-dimensional matrix generated by function g. Now, We discuss the choice of function g. Following the hypernetworks (Ha et al. 2016), natural choice of is fully-connected layer form dynamic weights and bias:
y W z + 6)
W
++b)
x W TT x BT
(4)
DyWeights
DyBias
explicit
implicit
where 6, E (Ruxme R Re Raxe RR). However, the size of W has quadratic space complexity, unsuitable for deployment in real-world application. Here, we consider a low-rank method in practice, e.g., a two-layer MLP:
g(=) W (WF 2 b1) b2
(5)
3

---

Under review as a conference paper at ICLR 2022
Dynamic Weights
Salf
Dymamic Weights Dysanic Bias
Dynamic Biss
Fontents
Weiglt Ganerite Layers
Conten
Commotoo Layen.
Fickd-nise Fully cotll-oonnettee
Fally connect layer
Figure I:
Illustration of feature-based and field-based dynamic parameterized operation.
where (i,,b)) E (RexI,R)) and (W2, b2) E (Te(x(me+)) Bm is non-linear function. Then, we can decompose the output into explicit dynamic weights and bias. The right inductive bias depends on how we select context and g. We denote the complexity of f is ( less than Om + nc) of plain MLP layer, while g scales up to O0lmc In). To reduce the complexity, we set as small number and use
mechanism (Vaswani al. 2017).
Relation to Cross Network: cross layer (Wang et al. (017) take the feature interaction formulation 1 To 3 b 2;, where z; ww is scalar. We prove CrossLayer is the simplest formulation of DPO. Let's s take 2ip as 2, æ; as x and only use 1-layer MIP as weight-generate function, whose hidden states are I, (i.e. 2 = æo, W E R" W = 0, B = 1), Thus, we get scalar output of g as the same as the multiplicative term of CrossLayer. In this way, DPO aims imitate multiplicative operation.
2.4 FIELD-BASED BASED DYNAMIC PARAMETERIZED OPERATION
Given E R2xx and R Xx as inputs and context, where ( and le represent the field numbers respectively, our goal is modeling the interaction between æ; and zj over all field positions. A simple idea is to treat field- based operation as multiple feature-based operations followed by summation over all output. Thus, Eqg. (3) can be expressed y; fan;; ciz) Evy (z(z;; 0)), which means all fields share the same instance-wise weights.
However, the field-based operation interacts between all fields, which sometimes introduces unnec- essary feature coupling (i.e., multiplicative interaction of brand ID and time, etc.). The empirical evidence finds over-coupling brings more noise and then results in underfitting, albeit their capacity of learning high-order features. considerable method is to use Self-Field dynamic operation without heavily hand-crafted feature engineering, formulated as: y, fm;;; gi (a;;; 0)) by removing cross-field interactions. Apart from Summation- and Self-based methods, a more attentive solution can be used to aggregate the dynamic attributes: y; = f(æ;; Ev, h(zz; wy (z); O)), where h is an attention layer. Beyond taking position into consideration, we can interact the whole context with inputs without explicit summation instead of concatenation. formulated as: y. f ga ([21, #2..., zy21; 0)). where [t, is a concatenation operation. These four methods learn pairwise field- held-hased interaction from coarse to fine to model high-order representation, while the feature-based method combines both low- and high-order information over all fields. The complex weight-generate function can be designed for the right inductive bias, but we do not specifically consider such method for online serving and leave it to future work.
Relation to FM: Here, we slightly modify the origin FM implementation (Rendle 2010 as: y ( atf ay by removing the LR term, that takes interaction among all field positions into consideration. Given inputs and the context as I; and {xy, Vjj * i}. the function f is simply matrix multiplication and is the identity function, then Eqn. (3) can be decomposed to: yi - Evjy; a æj. Thus, FM can viewed as the self-excluded version field-based dynamic operation, where the context is other field features different to the input features.
En Exjst
2.5 SEQUENCE-BASED DYNAMIC PARAMETERIZEI OPERATION
User behavior modeling focus on learning from their historical actions to predict whether the users click the current items. As a comparison, transformer- based solutions (Liu et al. (2200) Zhou et al.) 2018a) explored the encoder-decoder framework to learn long-range dependencies both source-to-
4

---

Under review as a conference paper at ICLR 2022
Dynamic Weights
Dysamic kemclk
Dynamii Bas
Dysamic Buss
Local aware Weight Gannnic Layar
Lood
Ganant Layes
Convolation
Faury
Fully-comeert layer
Figure 2: Illustration of homogeneous behavior and heterogeneous query-behavior dynamic parame- terized operation.
souree and source-to-ture target,, where the encoder exploits multi-head self-attention extract session interest and the decoder aggregates the query-specific interest. Following the encoder-decoder framework, we consider two variants, i.e., homogeneous Snhavio--Behvvin and heterogeneous dynamic operation (homo- and hetero- DPO). We show their multiplicative attributes on Apppendix D
Homogeneous Schavior-Behavion DPO aims to capture feature interaction at different time-steps of behavior. Given Z IR as inputs and context, where represents the behavior length. For user behavior modeling, our goal is to model the long- and short-term feature interaction. As mentioned above, a long-term function aims to learn non-local interaction between all positions while short-term ones only care about the local information. Thus, natural way is adopt global-aware neight-generate function and local-aware feature-aggregation f. Different to Section 2.3) and Section 2.4] we adopt convolution with learned weights. For simplicity, we consider function f in the form of ID-convolution with kernel size k. while feature-based and field-based methods only MLP.
which is widely used for modeling local sequence information
14
y, = f(xsi C(z) E9(z;;0)
1
Σ
Σ
(6)
C(x)
1-1-
5=
(l = W WW
1,1 Σ
æ b1.1) b2.1
Eqn. (6))shww the function I can act as a convolution operation without bias term which models local neighborhood by dynamic weight, where æ; is the extracted behavior in position i. Eqn. (7) gives a instantiation
generate function g. Firstly, we aggregate all sequence information and project them into select operator E R by W1,1 and b1.1. where W1,1 Raad b l Id and is activation function. Secondly, and b2, E RTc To use dynamic depthwise-convole multiplicative interaction correspond to global aggregation features. To further strengthen locality, we can adopt mocal-aware function to capture short-term information of context æ (e.g. convolution, separable convolution etc.).
explicit aggregate expert weight, where Wa Rdx (ne) we can set = 1. Eqn. (7) captures the
Heterogeneous Query- -ehavi DPO aggregates all sequential behaviors as context targeting to interaction with query. Give IR" and Z TR we take function linear transformation as mentioned in [Section 2.3] Eqn. (3) learns interaction between query and behavior over all length followed by summation, and the simplest formulation can be derived as:
y f(x; C(z)
Σ
0(zz; g( C(z)
Σ 2,;0)T
Vjer
VJEt
Similar to feature-based and field-based methods, query-behavior dynamic operation can easily learn rich multiplicative interaction and conditional inductive bias. The weight-generate function aims to learn the weight representation W, E mm Typically, we can exploit a specific aggregation function, such Eqn. (7) Compared to self-attention in decoder (Liu et al.) (020). DPO focuses attention on instance-weight based on context, while self-attention takes bipartite attention matrix to aggregate value units. Thus, conjecture they orthogonal and complementary solutions.
5

---

Under review as a conference paper at ICLR 2022
Table Comparison with different algorithms feature- bastrr datasets over 5-runs results. Se-
Datasets
Movielens tag
Avazu
Criteo
Base Model
Auc
Logloss
Auc
Logloss
Auc
Logloss
FM (Rendle
2010)
0.9388
0.2797
0.7497
0.3740
0.7933
0.4574
AFM (Xaao al 2017
0.9414
0.2804
0.7454
0.3766
0.7953
0,4554
HOFM
Blondel el
2016
0.9410
0.3088
0.7516
0.3756
0.7960
0,4551
NFM
He & Chua
2017
0,9355
0.2955
0.7531
0.3761
0.7968
0,4537
PNN
Qu et al.
60
0.9469
0,2792
0.7526
0.3737
0.8026
0.4509
CIN
Lian et al.
2018
0.9494
0,2600
0.7533
0.3756
0,8042
0,4472
AFN
Cheng et al. 2020)
0.9477
0.2753
0.7512
0.3731
0.8061
0,4458
CrossNet Wang et al.
2017)
0,9323
0.2929
0.7498
0,3756
0.7915
0,4585
CrossMix Waan et al.
2020
0.9379
0,2934
0.7526
0.3738
0.8019
0.4490
DNN
0,9521
0.2576
0.7533
0.3745
0.8028
0.4483
Feature-based DPN
0.9535
0.2538
0.7556
0.3733
0,8097
0.4425
Field-based DPN
0.9507
0,2561
0.7536
0.3750
0.8049
0.4467
Table Ablation study on MovieL.ens-lag dataset over S-runs results. show Auc and logloss.
(a) Instantiations of tright-generate functions: I feature-
(b) Layers and Context: we compare the results by replacing fully- connected layer with I and festure-bused dynamic operation of the 2-layer DNN baseline. Also, compure dynamic results of different context.
bused dynamic operation of different is added into first layer of 2ayer (LL((300-000) Std of metrics Ie.3.
Anc
Logloss
Params
Models, Eqn. (5)
Context
Auc
Logloss
Params
Base (3-layer MLP)
0,9471
0.2656
192k
Base layer MLP)
None
0.9521
0.2576
101k
Base (2-layer P)
0.9521
0,2576
I0Ik
Base (2-layer MLP, 400)
0,9514
0.2568
175k
f:l
Zo
0.2563
129k
fs2
0.9522
0.2582
372k
Hepee
0.9524
0,2715
371k
fel + fc2
(So. z0)
0.9532
0.2562
400k
Eyn. (5)
1=4, oraigmoidd
0,9522
0.2756
12%k
I:l + I:2
(itg
0.9535
0.2538
400k
Fgn. (3]
1m4, grssoftmx
0.9527
0,2563
129k
fel + F:2
(#hp- Mig
0.9530
0.2549
401k
E40.
2 d'esoftmax
0.9515
0.2622
10lk
(it,, sby)
0.2545
0,9503
0,2609
108k
[:1 + R:2
(2), =()
0.9544
0.2566
400k
Wo + Po(=)Q
0.9520
0.2566
fel fc2
(#m #m)
0.9530
0.2575
400k
(c) Instantiations of field-based dynamic parameterized network: we compare diffcrent aggregation function for MoK. Eqn. (5)
shown TTible Zajan two-layer lield-based DPN, Also, compare the parameters and time cost with leature-bused method and
implicit interaction models. We implement all models on 12 cores Intel(R) Xeon(R) CPU ES 2683 1002.10GI1 with TensorFlow op.
Model DNN
context
Logloss
CPU Second/Eipoch
None
0.9521 1
3e
101k
44.7
Field-DNN (implicit), 100-100
None
0.9445.1
0.26691146-3
1lk
76.7
Larger Field-DNN, 200-200
None
0.9473=2o-4
0,2663:b44-3
44k
[75.4
Feature-based
(#t, Re))
0.9544:2:0-4
0.2566:1-4--3
61.4
Field based DPN (Summmtioo)
(ato: Big
0.9451
0.2720:1:ee-3
46k.
44.2
Field-hased DPN (Self implicit)
Mig
1e-3
0.2607:12e:3
57k
86.4
Field-bused DPN (Summation implicit)
(#tp- Mig
0.9495:1e-3
1.2595:11e-3
57k
90.0
Field-based DPN (Attcation implicit)
(20. Mig
1.9489=1o-3
0,2620-4:2-3
57k
93.7
Field-based DPN (Concul implicit)
Mig
0.9507=1e-3
0.2561+:2e3
87k
95.5
EXPERIMENTS
We perform comprehensive experiments on feature modeling and user behavior modeling of public and real-world production CTR prediction datasets.
3.1 EXPERIMENTS ON FEATURE MODELING
Setting, We evaluate with MovieLens-tag, Criteo, Avazu with following questions:
How does DPN perform (effectiveness and efficiency) compared with other base models?
How do different contexts and Weight-generate functions influence the performance?

---

Under review as a conference paper at ICLR 2022
We use AUC and Logloss as metries for public datasets. For all experiments, evaluate the effec- tiveness of baseline models with the same training setting in AFN (Cheng et al. 2020) implemented by TensorFlow (Abadi et al., 2016). We adopt Adam (Kingma & Bal (2144) as optimizer with best searched learning rate of a batch size 4096 for all models. We fix the embedding ranks as 10 across all datasets and use same deep neural network (e.g., 3 layers MIP of 400-400-400) with Batch Normalization and ReLU (loffe& & Szegedy, [2015] Nair & Hinton| (2010) if without specifically noted. The details of our proposed methods are described in Appendix B
Comparison with state-of-the-art results. [Table I shows the results from AFN (Cheng et al. 2020) and our reimplemented results in the same setting. We note all these methods are single model without DNN components. First, our feature-based DPN consistently achieves better performance than other explicit interaction methods and also the implicit DNN baseline, which confirms the dynamic aspects contributes to implicit feature interaction. Additionally, when the train dataset and features get larger, the overwhelming margin get larger (e.g. 0. 0.13% on MovieLens-tagg 0.23% on Avazu, 0.69% on Criteo), showing promising potential ability for applied in real-world production. Secondly, our field-based DPN perform better than the other explicit interaction module. We note field-based methods models the relation over different attributes (i.e., UserID, MovielD etc.) where low- and high-order information are captured in a totally different way. Specially, field-based DPN obtain additive module in parallel with multiplicative one while other high-order interaction methods follow an opposite stacked framework to learn the multiplicative features (Qu et al. 2016 Cheng
al. 2020 He & Chua 2017).
Effectiveness of different instantiations of weight-generate function. Table 2a compares different types of feature-based dynamic operation added to the DNN baseline (right after the embedding layer for replacing the fully-connected layer). After we search the best DNN baseline model, we replace a dynamic operation with the first fully- fully-connected layer. We list the results of different weight- weight-generate functions, where not all methods perform better than the baseline. We implement the idea (Ha et al. 2016) as HyperDense which slightly improve the baseline while add a big chunk of computation resulting for optimization difficulty. When we adopt our proposed simple and effective method as shown in Eqn. (5) gate mechanism can be exploited for better performance, which means mixture of kernels have better generality. Furthermore, we explore some approaches to reduce the complexity of g, such as Multi-Heal mechanism. Matrix decomposition. However, they instead downgrade the performance even cannot compete on par with the baseline. We provide more ablation study on the Appendix E
hypernetworks-Be
shows the results of deeper dynamic parameterized network with different context. We separately replace DPO with first, second, and all fully connected layers in 2-layer MLP. Table shows that more feature- based DPO in general lead to better results regardless of context. We argue multiple feature- based operations can learn rich and high-order dynamic interaction by imitating MLP. High-order message can be processed with non-linear function layer by layer, which is hard to be found useful via multiplicative models.
Multi- Feature-based DPN with different contexts. Table 2b
In Table 2b. we also study the effectiveness of different context. We set the flatted inputs æo as the inputs of DPN and evaluate the performance of different contexts such as To and 2g where wo and 2g is the flattened outputs of inputs and context embeddings respectively. We found they share similar results for most experiments while getting the best performance when we set the context as 2t (i.e. use the tag information of context embeddings as context inputs). Under careful selection of hyperparameters, this best result mainly originates from the expert knowledge of MovieLens dataset and recommendation system. Meanwhile, it reveals a nice property of our methods: the intrinsic decoupling attribute can be more separably modeled. Interestingly, results of the infrequent user on MovieLens datasets, as shown in TTalle 3) We believe the dynamic interaction can warm up the infrequent user embeddings, demonstrating the potential of our methods for the cold-start problem.
find our methods improve
Effectiveness and Efficiency of Field- based DPNs. We design a family of field- based dynamic networks aiming to capture atomic relationships among fields' features. Table 2c presents results of different aggre- gate function. We find that all dynamic oper- ations with different context aggregate func-
Table 3: Results of infrequent user moviclens datascts where occur times of a user is less than 20.
Model
context
Ane
Logloss
DNN (implicit)
None
1.9386=t1o4
0.2956 le-2
Feature-based DPN
(Ze. #2)
0.9411-80-4
0.2911=t40-3
Feature-bused
[Zm. zm.)
0.9399.15e-4
0.29
Feature hased DPN
(2., z.,)
0.9408.1.5
0,3000.1.5
7

---

sDPN Under review as a conference paper at ICLR 2022
Table 4:
Adapt sequence-based DPO (sDPO) on Transformer. We evaluate the effectiveness of combination of multi-head self-attention mechanism and sDPO
Table 5: Comparison with state-of oh-thh-arttt art on Amazon dataset for user behavior modeling. We record the mean AUC over 5 runs. We mainly compare our methods with a well-known attention mechanism.
Encoder
Decoder
Model
All
New
inFreq
MSA
Homo
MSA
Heter
AUC
DIN Zhou ct al.
2018b
0.8292
0.8029
0.7937
DIEN Zhoo ct al.
2019
0.8675
0.8457
0.8375
X
X
0.8718
Trans Vaswani
al.
2017
0.8718
0.8522
0.8438
V
/
V
0.8755
KFAtt
Liu
2020
0.8789
0.8578
0.8496
EXX
X
0.8728
DIN Heter
0,8836
0.8554
0.8583
V
0.8809
DIEN Heter
0,8693
0.8476
0.8414
X
V
0.8731
Trans + Heter
0,8809
0.8608
0.8526
X
0.8775
Trans + Homo
0.8728
0.8530
0.8440
X
V
0.8849
0.8849
0.8615
0.8590
tions perform better than the static compo- nent, even only correspond to themselves.
We may hypothesize that additional contexts can benefit the field features after having been processed for imitating linear transformation, containing multiplicative interaction between inputs and contexts. However, we find the time-consuming worrying in the CPU machine when the dimensions of outputs are relatively large, making it venturesome to be applied on real-world production.
3.2 EXPERIMENTS ON USER BEHAVIOR MODELING
Experiment setting, We evaluate sequence-based DPN (sDPN) on Amazon Electronics Datasets. We only adopt AUC as metrics with the same training setting KFAtt (Liu al. 2020) implemented by TensorFlow. The task is to predict whether user will write a review for the target item after reviewing historical items. We refer the readers KFAtt (Liu et 2020) for more details.
Comparison with state-of-the-art. From Table 5 we can see that sequence-based DPN achieves the best performance than all state-of-the- thettss on total situations, including the strong baseline KFAtt. When incorporating heterogeneous DPO into the attention mechanism, we find both DIN (Zhou et al.) 20185) and DIEN (Zhou al. 2019) perform better than the origin baseline where the armed DIN outperform all the base models on Amazon datasets by a large margin, which shows that heterogeneous DPO can effectively learn complementary representation which can benefit the attention mechanism.
Adaptation to Self-Attention. Table presents us how sDPO performs when incorporated with self attention mechanism. For homogeneous DPO, find it performs slightly better than MHSA counterparts. When we use session- wise representation for user behavior modeling, self-attention can capture local information over handeraft scope of time designed by experts while narrow neighbor interaction of convolution may not contribute to learning users' attention due to the short session. For heterogeneous DPO, we find it can effectively facilitate the decoder counterpart no matter which encoder we adopt. Overall, the sequence-basee based DPN(sDPN) achieve best results than other combination, which shows the effectiveness of our propose homogeneous and heterogeneous DPO.
3.3 EXPERIMENTS ON REAL-WORLD PRODUCTION DATASET AND ONLINE A/B TESTING
We conduct all feature-based, Production dataset. In the offline experiments, we observe the progressive improvement from modern rank models consisting of advanced user behavior and multi-modal features model in advertising system. Table shows our DyMLP and DyJoint get significant advancement compared to origin implementation. For feature interaction modeling, DyMLP shows better performance than the commonly used DNN component while only add a little extra cost. Despite we don't 't explore the effectiveness of ensemble DPN models in a public dataset, Table 6 6/presents even one lightweight field-based DPO that can benefit the generality. To reduce the complexity, we use multi-head mechanism in feature-based DPN which may influence the effectivvness.
and sequence- requence-based based experiments the Real-world
8

---

Under review as a conference paper at ICLR 2022
Table 6: Results Real-world Production dataset. We show the details how incorporate DPNs into
online ads system in Appendix G.I] For feature modeling. we name DPNs as DyMLP. For both feature and user behavior modeling together with online model, we name DPNs as DyJoint.
Module
MLP
Feature
Field
KFAu
Homo
Heter
Auc(+gain)
Throughput (batch/s)
Base
X
X
X
0.7523
101
X
X
X
X
0.7530(10.07)
101
DyMLP
X
J
X
X
X
0.7550(10.27)
88
Online
X
X
X
0.7598
55
X
X
X
O. 0.7609(10.11)
50
DyJoint
X
X
0.7618(10.20)
48
X
X
0.7624(10.26)
48
X
0.7633 (70.35)
46
For sequence-based DPO, we conduct more ab- lation studies in Appendix G.3] Our homo- geneous DPO can act as specific form like dynamic convolution. Incorporating it with session-basee self-attention encoder, we can in-
Table 7: Results of Online A/B testing.
Model
CTRgain
eCPMgain
TP99 latency
Online
0%
0%
24ms
DyJoint
11.0%
11.2%
29ms
ject inductive bias learned from local neighbor-
hood information into global long-term dependencies based on Transformer-like models (Vaswani al. 2017 Feng et al. 2019). Beyond that, heterogeneous DPO can learn conditional multi- plicative interaction which models the user interest on given items, showing greater power than the homogeneous component. Combined with all techniques, we get the best results by large margin to the online model.
In the online A/B test, Table shows DyJoint contributes 1.0% CTR gain against the online compo- nent, which demonstrates the superiority over the highly optimized base model on our ad system. However, DyJoint leads to larger online latency compared to the base model due to increment of model parameters and memory access.
4 CONCLUSION
In this paper, we describe a new class of neural networks that captures both explicit and implicit inter- action via dynamic parameterized operation. Our proposed block can be easily inserted into existing CTR predicting architectures for fusing features from different modalities. Our experiments show that it overwhelms the existing Ceature-crossing-bale tasks. Furthermore, confirm its representation effectiveness in the real-world production dataset. For the theoretical understanding, we decouple dynamic operation for comprehensive study with high-order feature-crossing methods and self-attention. Overall, we open new where current mainstream solutions are dominated by self-attention mechanisms and MLP in CTR prediction.
and attention-
models on two fundamental

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Geoffrey Irving. Michael Isard, et al. Tensorflow: system for large-scale machine learning. In 12th {USENIX} symposium operating systems design and implementation (fOSDI) 16), pp. 265-283 2016.
Mathieu Blondel, Akinori Fujino, Naonori Ueda, and Masakazu Ishihata. Higher-order factorization machines. In NIPS'16 Proceedings of the 30th International Conference Neural Information Processing Systems, volume 29, pp. 3359-3367, 2016.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic Attention convolution kernels. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1030--1033, 2020.
convolution:
Heng-Tze Tze Cheng. Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye. Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop deep learning for recommender systems,, pp. 7-10, 2016.
Weiyu Cheng, Yanyan Shen, and Linpeng Huang. Adaptive factorization network: Learning adaptive- order feature interactions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3609-3616, 2020.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198. 2016.
Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. Deep session interest network for click-through rate prediction. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pp. 2301-23000 2019.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization- machine based neural network for etr prediction. arXiv preprint 1rXiv:1703.04:: 2017.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv:1609.09111 2016.
Yizeng Han, Gao Huang, Shiji Song, Le Yang. Honghui Wang. and Yulin Wang. Dynamic neural networks: A survey, arXiv preprint arXiv:2 arXiv:2102.04990 2021.
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5(4):1-19, 2015.
Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 355-364, 2017.
Jie Hu, Li Shen, and Gang Sun. soueeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132 7141,, 2018.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2018.
Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padman- abhan, Giuseppe Ottaviano, and Linjun Yang. Embedding-baid based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2553-2561, 2020.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
10

---

Under review as a conference paper at ICLR 2022
Siddhant M. Jayakumar, Jacob Menick, Wojeiech M. Czarnecki, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Tch, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In ICLR 2020 Eighth International Conference on Learning Representations, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.660 6980, 2014.
Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen. Involution: Inverting the inherence of convolution for visual recognition. arXiv preprint arXiv:2103.06255, 2021.
Yunsheng Li, Yinpeng Chen, Xiyang Dai, mengchen liu, Dongdong Chen, Ye Yu, Lu Yuan, Zicheng Liu, Mei Chen, and Nuno Vasconcelos. Revisiting dynamic convolution via matrix decomposition. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. xdeepfm:: Combining explicit and implicit feature interactions for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 7754-17634 2018.
Tsung- Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear enn models for fine-grained visual recognition. In Proceedings of the IEEE international conference on computer vision, pp. 1449-14577 2015.
Hu Liu, Jing Lu, Xiwei Zhao, Sulong Xu, Hao Peng, Yutong Liu, Zehua Zhang, Jian Li, Junsheng Jin, Yongjun Bao, et al. Kalman filtering attention for user behavior modeling in ctr prediction. arXiv preprint arXiv:2010.00983 2020.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. Product-based neural networks for user response prediction. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp.
2016.
Steffen Rendle, Factorization machines. 2010 IEEE International Conference on Data Mining, pp. 995-1000. IEEE, 2010.
Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. Neural collaborative filtering vs. matrix factorization revisited. In Fourteenth ACM Conference Recommender Systems, pp. 240-248, 2020.
Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pp. 285-295, 2001.
Ying Shan, T Ryan Hoens, Jian Jiao, Haijing Wang. Dong Yu, and JC Mao. Deep crossing: Web-scale modeling without manually crafted combinatorial features. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 255 255, 2016.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expeere layer. arXiv preprint arXiv: 006770000006.... 2017.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1161-1170, 2019.
11

---

Under review as a conference paper at ICLR 2022
Yang Song. Ali Mamdouh Elkahky, and Xiaodong He. Multi-rate deep learning for temporal recommendation. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 909-912, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958 2014.
Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In International Conference on Machine Learning, pp. 6166-6175. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: (70.00377,,,7 2017.
Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD' 17, pp. 12, 2017.
Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. Den-m: Improved deep & cross network for feature cross learning in web-scale learning to rank systems. arXiv preprint arXiv:2008. 13535, 2020.
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference Learning Representations, 2019.
Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. Attentional factorization machines: learning the weight of feature interactions via attention networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 3119-3125, 2017.
Brandon Yang, Gabriel Bender, Quoc V. Le, and Jiquan Ngiam. Condconv: Conditionally parameter- ized convolutions for efficient inference. In Advances in Neural Information Processing Systems, volume 32, pp. 1307-1318, 2019.
Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. A dynamic recurrent model for next basket recommendation. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 229-732,00 2016.
Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. Atrank: An iirention-based user behavior modeling framework for recommendation. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 32, 2018a.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1059-1068, 2018b.
Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. Deep interest evolution network for tlick-through rate prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5941-5948, 2019.
12

---

Under review as a conference paper at ICLR 2022
Table Statistics of datasets feature modeling.
Datasets
instance
fields
features
Criteo
45302405
39
2086936
Avazu
40428967
22
1544250
Moviclens-tag
2006859
3
90445
Movielens- 1M
739012
3529
Real-world Production
12 Billion
96
N/A
Explicit layer
Implici layer
Dytumic
Embadding layer
Embesking layer
Embedding layer
(a) stacked paradigm
(b) parallel paradigm
(c) paradigm
Figure 3; Illustration of stacked, parallel and ours paradigm for traditional CTR prediction.
A RELATED WORK
Feature Crossing as high-order interaction has been widely explored in CTR prediction and recommendation system by capture cross feature which can be defined as I x! Factorization Machine (Rendle 2010) was firstly proposed calculate similarity by using inner product between two different features (e.g. item feature and user feature) borrowed from the Dollaborative-fiterii based model (Sarwar al., 2001). With the rise of deep neural network, recent works focus on combining feature crossing with Embedding+ MLP paradigm for better performance (i.e. stacked and parallel paradigm) as shown in Fig. (3a) and Fig. (3b). Late fusion is mostly used parallel paradigm. Wide and deep (Cheng al. 20166) propose wide component to form order-2 features by cross cross-produs product transformation between sparse imputs and deep layer for order-1 features. DCN (Wang et al. 2017) propose network. DeepFM (Guo et al. (217)) adopts a parallel structure to fuse the FM and DNN outputs. To facilitating the high-order representation, XDeepFM (Lian al., 2018)) are proposed generate vector-wise high-order feature, achieving further advancement by stacking multiple interaction layers. Contrast polynomial networks, AutoIntSong al. 2019) exploit multi-head self-attention layer to automatically learn the high-order feature interaction of all atom features(e.g item id, user id, brand id etc.). Ensembled with DNN. AutoInt achieve better result. Constrast parallel ensemble methods, stacked paradigm uses explicit layer to extract cross features followed by an implicit layer to process them, DNN. NeuFM (He & Chua 2017) uses Bi-Interaction layer with stacked deep neural network for bridging generalized linear models and deep lcarning. AFN (Cheng et al.) 2020) models cross features of adaptive orders followed by 3-layer MLP. Despite the inherit strong generality incorporated with DNN, stack paradigm often resorts further ensemble with DNN for better results.
parallel structure composed by bit-wise cross-layer and deep neural
However, due to either the low capacity high latency of so-called interaction layer, those methods typically failed in real-world rank system. Recent application of polynomial networks on CTR prediction can be special case of multiplicative interaction formulate scalar-wise, element-
13

---

Under review as a conference paper at ICLR 2022
wise feature-wist multiplier fusion between projected embeddings. We prove that methods can degrade as the bit-wise cross-layer (Wang et al. 2017) and pairwise interaction layer (Rendle 2010).
User behavior modeling aims to extract users' dynamic and evolving interest over items. Earlier works (Covington et al., 2016; Song et al., 2016 Yu et al. 2016)) use a
manner to aggregate total behaviors, failed to learn the interaction between target and past behaviors. Recent works (Zhou et al.) 2019 2018b a Liu et al. 2020) achieve massive improvement by adopting attention mechanism for modeling long-range representation between user historical behaviors and other target features. Orthogonally, our proposed methods can be formulated as multiplicative interaction between two different streams.
Dynamic Neural Networks enjoy preferable properties than static ones, whose computation graph and forward parameters are fixed with limited representation power (Han et al., (02T). Instance-wise dynamic neural networks typically have dynamic structure (Tanno et al. 2019)) dynamic weights (Yang et al., 2019; Ha et al., [016) [Cenn al. 2020p, or dynamic layers (Huang et al.) (018) in inference time. Jayakumar (Jayakumar et al. (0020) provide theoretical comprehension about why dynamic weights are beneficial: HyperNetworks (Ha et al. 2016) can be thought of generalised multiplicative interaction in the non-factorised sense, whose filters are generated by conditional inputs.
B IMPLEMENTATION DETAILS OF DPN
B.I DATASETS AND EXPERIMENTAL PROTOCOLS
Public dataset for feature-interaction modeling, We conduct experiments on three public real- world datasets in Section 3.1 as shown in Table Movieleens-tage]] has been prepared into (user, movie, tag) format for personalized recommendation. Totally, there are 2006859 tuples, constituting of categorical fields: UserID, MovielD and tag. Avazifis click-through rate dataset including 22 feature fields of user features and advertisement attributes. There are about 40 million instances in total. Crited is a benchmark dataset for displayed ads CTR prediction, containing 26 categorical fields and 13 numeric fields. It has about 45 million user-click records on displayed ads. We split the dataset in 7:2:1 for training, validataion and testing respectively.
Public dataset for user behavior modeling We conduct sequence modeling Amazon book dataset (McAuley et al. 2015) to learn users' interest. We use the 5-core Electronics subset, including 1689188 instances with 192403 users and 63001 goods from 801 categories. The task is to predict the intent of writing users' review for a target item regard to the historical reviews. In practice, we use the last review of each user as test split while others as train split. The negative instances are randomly sampled from not-reviewed items.
Real-world Production dataset a daily traffic log generated from the search advertising system. Typically, we sample negatively a first 32-days click-through logs for training data containing 2.4 billion samples and non- non-negatively the next-day click-through logs for test data containing about I million samples. For the nser-behavior data setting, we follow the KFAtt (Liu et al.) 2020) to choose user clicks/queries in previous 70 days as behavior v/k: The other features consist of the query, user and ad profile, ad image and contexts, built for modeling feature interaction.
Metries: AUC is almost the default offline evaluation metrics in advertising system due to the direct relation to online performance. Thus, we use AUC evaluation metric for evaluation on both public dataset and real-world production dataset. For fair comparison, we additionally add Logloss as another metric when evaluated public dataset. We denote that 0.001-level improvement of AUC on CTR prediction task is significant, which has been a common sense in previous works (Cheng
et al. 2016, Guo et al. 2017: Lian al. 2018 Wang et al. 2020 Wang et al. 2017).
hhtpp://wwwwkagge httpp://wwwwkaalllccoc//cttodood
14

---

Under review as a conference paper at ICLR 2022
B.2 EMBEDDING LAYER
For all ctr models, the categorical features and dense features (UserlD,, ItemID, User Age etc) are hashed sparse one-hot features the inputs of embedding layers. We project i-th one-hot feature from sparse high-dimensional space to a lower one via Zemb, = W,e, where e; E {0, 1 and W e Rex=; v and e means vocab and embedding size respectively. Thus, for feature-based modeling, we take the output of the concatenation of all the embedding vectors: Zo [Temb.u, emb,, ... emmb, as our named features. For field-based modeling, we instead take the embedding vectors of every sparse ID as fields aiming to learning the interaction of pairwise features (e.g. [UserID, ItemID], [UserID, UserAgel).
B.3 FEATURE-BASED AND FIELD-BASED MODELING
As mention in Eqn. (3) all of our methods induce extra information features 2 for boosting gener- alization. However, we don't use extra fields as show in Table which represents the effectiveness comes from the combination of fully-connected layer and Ieature-crossing module. For feature-basee and field-based DPN, we stack several feature- and field- DPO layer followed by classifier layer to build them. The implicit interaction module of bease DPN is fully-connected layer with non-linear function, while the implicit part of field-based DPN has two parts: field-interaction module W and linear transformation module W,, where W, E Rexn and Wi R Thus the static outputs can be calculate by W X, Wi, where Xo E R=xey 1, We name it as Field-DNN.
B.4 USER BEHAVIIR MODELING ON AMAZON DATASET
To evaluate the effectiveness of adapting sDPO on various methods, we adpot different strategy to combine sDPO with DIN, DIEN and Transformer. For DIN and DIEN, we produce the heterogeneous DPO outputs followed by element-wise summation with origin attention mechanism outputs. For Transformer, sum the homogeneous and heterogencous DPO outputs and self attention outputs in encoder and decoder respectively. Except for the increment benefit over other methods, we test the effectiveness only using DPO, i.e. sequence- based Dynamic Parameterized Network (sDPN). The sDPN simply replaces the self attention counterparts on Transformer framerwork by DPO layers.
B.5 USER BEHAVIOR MODELING FOR REAL-WORLD PRODUCTION DATASET
We follow the setting of KFAtt (Liu et al. 2020 along with the feature-based methods and field-based methods, evaluated only on real-world production dataset. There, 96 multi-modal features are first embedded into 16-dimension vectors and then fed for field-based and feature dynamic operation. The origin implement make use of 4-layer MLP with dimension 1024, 512. 256, 1. When there is 30 minutes" time interval between adjacent behaviors, we conduct session segmentation. For each instance, we use at most 10 sessions and 25 behaviors per session. The learnt hidden user interest, 69 E 1264 is concatenated to the output of Ist FC layer together with a 150-dimensional visual feature vector. Our homogeneous behavior-behavior operation are used to fuse the session-based self-attention encoder output in additive way with kernel size 3. Our heterogeneous query-behavior operation are used to fuse the predicted interest produced by cross-attention. All the weight generate function are a SE-layer (Hu et al. 2018) whose intermediate down ratio is grid searched from [0. 125, 0.25, 0.5).
15

---

Under review as a conference paper at ICLR 2022
C
DETAILS OF THE RELATION TO FM
As shown in (Rendle 2010 FM can be formulated as:
v(œ) = b
W +
1/. Uj > 24JH
i=1
j=i+1
= linear (x; 0
11,3 x
Σ )ydz))
1=1
Vj!=x
linear 0)
Σ
xz)))
Vyl=i
Where means the embedding table, mean the categorical features. Thus can decompose the output y as summation of each field-wise interaction features. Thus the function f and function can be represented for matrix multiplication and identity function. The self-excluded version means use the Exjt=i Zy as context features which exclude the input feature Z;.
D ANALYSIS MULTIPLICATIIE ATTRIBUTE OF SDPO
D.1 ANALYSIS OF HOMOGENEOUS DPO
y: = f Eg(z):0)
((z) V1
gl (2,i 0) + Σ )1((2;;0))
C(x)
1=|- ビ=
(10)
W 2 W Σ 2j + bb,)) + (2..)))aaa xi+t + gb(
j=0
Wi, (.(4))) + (.)
As shown in Eqn. (10) we show the homogeneous DPO actually learns both multiplicative and additive relation among local neighbor and each behavior features followed by summation aggregate function when we don't it use non-linear function. To further reduce complexity, common way is to learn the feature interaction between global embedding aggregated firstly and local neighbor, formulated as:
3:
Σ -=|-
(W (WT. Σ xy + b1.)) + b2,)) xist
(11)
j=U
where can be sigmoid and softmax function.
16

---

Under review as a conference paper at ICLR 2022
D.2 ANALYSIS OF HETEROGENEOUS DPO
y = f(;; ((z)
Σ 0(z
Vjet
9u C(z)
23z 0) x + gb!
1
Σ 2)
C(z)
YJEC
YJE!
= -WW (WT
Σ
2j + b1) + b2) x gb(2)
(12)
j=0
-W2 2) (W2 + b22) gb
W C(z)
Σ 2z) Wi + W (6(
query-whoo whole
query-ear
As shown in Eqn. (12) We decompose the simplest formulation of Eqn. (8) where W1 E Riden W2 Ra=(m=e) b R be Rm x E R E R". The equation omits the reshape operation. We can see that Heterogeneous DPO captures multiplicative and additive between each behavior and query different to the self attention mechanism. However, Eqn. (12) be enhanced by non-linear function, Empirically, better way is to learn the relation of the embedding representing for the whole behavior sequence and query features, which further reduce the computations and also have strong compatibility with complex weight-generation function such as SE-layer (Hu et al. 2018).
E ADDITIONAL EXPERIMENTS ON MOVIELENS-TAG
E.
ABLATION STUDY
We perform ablations variants of feature-based and field-based DPN. Unless specified otherwise, all experimental results in this sections report MovieLens AUC and logloss by training a DPN architecture that replace weighted fully-connected layer.
Varing depth and width for feature-based and field-based DPN Table 9 presents the impact of different depth and width on performance. Our experiments indicate that Feature-hased DPN can achieve better performance with similar time consuming over a wide range hyperparameters, which demonstrate the robustness of the method. However, the atomic field-based DPN still lack the capacity to compete on par with MLP baseline. Furthermore, we find the atomic fields' interaction is hard to inert into feature-based DPO due to the extra fields dimension.
presents the performance of matrix decomposition and multi-head mechanism on feature-basse DPN. Despite their success when applied in other domains (Vaswani al. 2017 Li al. 2021 Wu et al. (019), we find these methods dont't work traditonal CTR prediction. We conjecture the multi-head mechanism hurts the expressivity of fully-connected layer due the separable interaction among channels. For matrix decomposition, we aim to generate instance-wise kernels by Po(2 Po. where P E Raxrr Φ(z) R. Q E Rrxm, We set r = 32 in Table 10 Thus, we may solve the low-rank completion problem better performance. We conjecture simply utilization of dynamic parameterized fully-connected layer by matrix decomposition leads to optimization difficulty. We leave it into future study.
MD, MH Plain Table 10
F ADDITIONAL EXPERIMENTS ON MOVIELENS- 1 1M
Besides experiments on three public datasets, We extraly conduct feature-based and field-based ablation studies on widely-used IoV.eLens-IM.
MovieLens-1MM is a relative small dataset (Harper & Konstan 2013). contains users' rating on movies with 7 features, e.g. user features and movie features and rating. We follow the experiment
17

---

Under review as a conference paper at ICLR 2022
Table 9: Comparison of different depth and width over 5-runs results on MovieLens-tagg
Model
Ane
Logloss
Auo(infreg)
Logloss(infreg)
Params
CPU Time/spoch
DNN, 100-100
0.9498=62-4
0.2607=:30-3
0.9358=5c-4
0.2986=:3--
13k
35s
DNN, 100-100-100
0.9468 4-4-4
1e
0.94
0.2983:1:2-44
24k
37s
Fe DPN, MoK, 100-100
0.9516:137-4
0:4581114e-3
1.9389-18:44
0.2951 le.2
53k
39%
DNN, 200-200
9-9-
0.2575-:22-3
1.9370::1:3
0.2945:::4-3
47k
38s
DNN, 200-200-200
0.9476=55-4
0.2633++2o-3
0.9336=1e-3
0.2962=4:0-3
88K
41s
Fe DPN, MoK, 200-200
0.9533:1---4
0.2527:1:8-4
0.9412-=9e-4
0.2897::22-3
187k
48%
DNN, 64-64 MoK, 64-64
0.9487=22o-4
0.2619=1o-3
0.9341=1o4
0,298443e-3
6k
35s
0.950241e-3
0.25934:3e-3
0.9370::2--3
0.2959::5e3
25k
37s
Fi DPN, Sum, MoK, 32-32
0.9414:17e
3.
0,9268 le
3.3164133.3
6k
40s
Fi DPN, Sum, MoK, 64-64
1.
0,2777433-3
0.9280 1e-3
0.3119-16-3
20k
40s
Fi DPN, Sum, MoK, 128-128
1.9453=1o-3
0,2723+26-3
0.9309=
0,3103#30-3
73k
51s
Table 10: Variants of Feature-based DPN over 5-runs results on MovieLennstagg
Model
Head
Auc
Logloss
Auc(infreg)
Logloss(infros)
Params
CPU Timelepoch
DNN, 200-200 Fe DPN, MoK, 100-100
0.9507++90-4
0.2575+22-3
1.9370=1oo3
0.2945:44o-3
47k
38s
1-1
0.95161:3e-4
0.2-81
8c-4
0.2951 1e-2
53k
39,
Fe DPN, McK, 100-100
2.1
0,9498 1e4e 4
0.2618:12e.3
0.9379.1 3e:3
0.2913.1
44k
43s
DPN, MoK. 100-100
1-2
0.95131:56-4
0.2581-22--3
5.-
0.2963-:55-3
23k
40s
Fe DPN, MoK, 100-200 200
1-2
0.9511
0.2591
0.9385.16e 4
0.2958
34k
42s
DPN,
100-100
4.9479.99-4
0.2657::2e-3
0.9353-1 1e.3
0.2974-144-3
14k
39s
Fe DPN, MoK, 200-200
2-2
1.2719=t
1.9380=#te3
1.2994=1o-2
48k
43s
Fe DPN, MD, 100-100
1-1
1.946711ee3
0.2708:1:e-3
0.9330:1
0.3007::.5.33
20k
41s
Fe DPN, MD. 200-200
1-1
0.94941:5e-4
0.26281::e-3
0.9362:166-4
0.2975:::4-44
30k
46s
al. 2019), which randomly split 80% of dataset into train data, 10% into valid data and the rest 10% into test data. And we set the ratings below 1s and 2s as Os, above 4s and 5s 1s. All rating 3s should be removed.
setting of AutoInt (Song et
Training For ablation study on MovieLens-IM, our baseline models follow the settings in AutoInt (Song et al. (019)) with random initialized weights. implemented by Tensorflow (Abadi et al.) 2016) with the tricks (Srivastava al., [2014] Nair & Hinton 2010) but don't 't use BN (loffe & Szegedy 2015) due to the poor performance. The feature-basse
model take the same setting as DeepCrossing (Shan et al. 2016) introduced in AutoInt with 4-yeyrr MIP of size 100. The function g is 2-layer MLP. The field-based models use three dynamic layer and set hidden states as 64 the same as their public code (c.g. replace self-attentive layer with our field-based dynamic operation). After that, we use grid search to select hyperparameters, (i.e. dropout rate, learning rate). For fair comparison, all methods set embedding dimension as 16 and use Adam (Kingma & Ba 2014) as default optimizer with 31 = 0.9, 32 0.999. batch size is 1024. However, only replace feature- operation on the final two layers before classifier, due to the memory limitation and insert 1-layer field-based operation firstly.
Comparison with state-of-the-art Table shows the results from AutoInt (Song al. 2019). Due to the random data splits, we rerun their experiments for comparison. We note the 0.00 -level fluctuation on evaluation metric compared to the cited' ones is acceptable when trained and evaluated on randomly split setting. Nevertheless, our basture-based method surpasses the other existing high-order based methods (Wang al., 2017 Lian et al. 2018 He & Chua 2017 by good margin, even the proposed field- based operation by ourselves. However, what surprise us is the heavily tuned DNN model still excels high-order parts in contrast to the results reported in the AutoInt. We infer that MLP- b-bse DNN model can be good universal approximate function to attain a suitable local minimum with good hyperparameters. In this way, there is no doubt feature-based solution is better than others, because of its capacity for learning both implicit and explicit feature interaction. As far as we know, our proposed method is state-of of-the--rt solution in the same setting, even without the DNN component.
FeatureRec
18

---

Under review as a conference paper at ICLR 2022
Table 11: Hyper-parametens Dynamic Parameterized Network in MovieLens- IM.
Slyper-paaamedd
feature-based
field-based
fields
MovielD, zipcode, Age, Occupation, Gender, Timestamp
embedding size
hidden size
64
64
2- 2-layer MLP
2-layer MLP
sigmoid
sigmoid
residual
False
True
LN
False
False
non-linear
Relu
Relu
initializer
glorot
glorot
context
21-1
#1-1
Layers
Heads
Learning rate
3e-4
le-3
Adam
le-8
le-8
Adam 31
0.9
0.9
Adam 82
0,999
0.999
Batch size
1024
1024
Params
86k
30k
Table 12: Comparison with diffcrent algorithms over 10-runs results. Std 1t-3= 4 means below while means above. Throughput means training time of one epoch. We also cite the results from AutoInt (Song et al. 2019).
Model
Auc
Logloss
Params
MI.P(our)
0.8475
0.3785
53k
DeepCrossing
0.8448
0.3814
N/A
AutoInt(ours)
0.8448
0,3812
39k
NFM
0.8357
0,3883
N/A
CrossNet
0,7968
0.4266
N/A
CIN
0.8286
0.4108
N/A
Feature-based
0,8522
0.3726
86k
Field-based
0.8452
0.3806
30k
Comparison with different heads in feature-based operation Table 13a compares different head numbers of 4 dynamic parameterized layers with context genre. We find good head number can effectively improve the performance, while larger lead to bad results. We derive this large gap mainly due to the reason our bascline model use relative smaller embedding size and hidden state with increasing head numbers. When head numbers equal to 4, the size total parameters is about 11k, which seriously downgrades the performance. However, properly head number trades off the extent of implicit and explicit high-order interaction, inducing larger variance of selection on dynamic weights/bias.
Comparison with different non-linear function in feature-based operation Table 13b shows the performance of different non-linear function in Eqn. (5) Here, we = 4 and head numbers equal to 2, for simplicity. We find even using plant low-rank approximation without non- non-lineat function, the result still better than high-order part reported in Table This gives evidence that non-restrict multiplicative interaction among hidden states can be a complementary component for further improving the capacity of MLP. When using non-linear function sigmoid and softmax, we found the performance moves significant step about -ee0l- Constrained non-linear function controls the outputs selection layer between (0, 1), which can be seen as the attention over kernels (e.g. mixture of expert layers). We note that the best non-linear function depends the dataset respectively. Such methods for kernels aggregation are friendly for optimization, allowing us to learn diversity of weights while consume nearly same time as plain fully-connected layer.
19

---

Under review as a conference paper at ICLR 2022
Table 13: Ablation study of feature-based dynamic operation
(a) Comparison with different head num-
(b) Comparison with nou-lincar function.
(c) Comparison with ditferent context. 1=
ber 1m4, a is sigmoid, context is genre.
1m4, hm2, context is genre.
Am2, d softmax.
head
Auc
Logloss
Auc
Logloss
context
Auc
Logloss
0.8483
0.3781 0.3738
identity sigmoid
0.8494
0.3781
genre
0.8511
0.3741
0.8504
0.8504
0.3738
gender
0.8457
0,3782
4
0.8136
0.4126
softmax
0.8506
0.3745
genre, gender
0.8452
0,3800
all
0.8522
0.3726
Table 14: Ablation study of field-based dynamic operation
(a) Comparison with different aggre- gation methods.
(b) Comparison with different combination methods field-based
and crature-based operation.
Agg
Auc
Logloss
Methods
structure
Auc
Logloss
Summation
0.8372
0.3868
Self + Feature
stacked
0.8471
0.3787
Self
0.7917
0.4264
Self (One layer) Feature
stacked
0.8520
0.3738
Attention
0.8452
0.3806
Self + Feature
parallel
0,8532
0.3725
Concat
0.8443
0,3808
Weighted Sum Feature
stacked
0.8527
0.3746
Weighted Sum Feature
parallel
0,8536
0.3729
Comparison with different context feature-based operation Table compares the different context of feature-bu based dynamic operation. We can that different combination of fields as context will lead to variant results, where the worst only improve slightly than AutoInt, but less than DNN model. Interestingly,
select separately the user feature gender and movie feature genre as context. The gender typically consist two classes of male and female. Thus, the multiplicative interaction can be understood how the gender influences whether user rates movie. Contrast human intuition on gender, the results reveal that gender is not main reason for different rating on movie, e.g. explicit modeling gender interaction with other features leads unnecessary bias. However, when modeling with genre of movies, we get large improvement on performance compared to the genders'. We conjecture genre plays important role on rating of users. Moreover, when combined with all field features, we can achieve best results, representing the given features can be classified effectively in high dimension.
Comparison with different aggregation function in field-based operation The feature-based methods give empirical evidence that how to select head numbers, non-linear function and context. For simplicity, we choose the same best setting in ablation study of field-base dynamic operation. Table 14a compares four methods of how to aggregate differnet field information as mentioned in Section 2.4 When only using the field to correspond itself, we find it perform worst due to the scarcity of field-isse feature interaction compared to the simple summation version. When using weighted-sum layer and concatenation-MLP layer, we find such methods improve almost 0.005-level compared to summation-basee solution. The main difference using learned linear combination of all field features, which implicit models low-order feature interaction. Thus, linear transformation overall fields is beneficial for modeling high-order representation. However, we notice the DNN result overwhelm the fieldwise operations' meaning the multiplicative interaction brought by high-order component is not the main course in CTR prediction.
conclude implicit
Does field-wise component contribute to feature-wise component? Despite the low capacity of field-wise dynamic operation, we explore the contribution when combining feld-wise component with feature-wise's Table 14b) shows there are obvious benefit when we use both operation at same time no matter how we stack them, while facilitating the performance when composed of parallel structure. The stack structure directly exploit the field-based features as the inputs of feature- based layers. Once the fundamental representation has low diversity and expressivity, subsequent fancy modules can't 't perform their proper ability, e.g. three layers stacked Self-based operation dominates the effectiivnesss. However, we find the aggregation methods weighted summation effectively boost the performance sightly no matter which structure used. Thus, confirm
20

---

Under review as a conference paper at ICLR 2022
Table 15: Experiment Results Real world Production dataset
DyMLP
SelfAtt
KFAtt
Encoder (k 3)
Decoder
Auc
Sep + SE Homo
Sep + SE Heter
0.7633
Sep + Homo, k
Sep + Heter
0.7626
SE+ + Homo
SE+ + Heter
0.7622
ConvID
Sep + SE Heter
0.7624
Conv ID, k= 5
Sep + SE Heter
0.7624
X
Sep + SE Homo
Sep + SE Heter
0.7617
X
Self + MI.Ps Homo, k=1
Heter
0.7604
when combined those two methods in a suitable way, field-wise component can contribute to the feature-wise component for further advancement of performance.
G ADDITIONAL EXPERIMENTS ON REAL--WORLD PRODUCTION DATASET FOR USER BEHAVIOR MODELING
G.1 INCORPORATING DYNAMIC OPERATION INTO ONLINE RANKING SYSTEM
In previous section, we mainly discuss the dynamic mechanism among different inputs and context. However,
industry system scales Dehavior-baseed and reature-based modeling containing many techniques to precisely extract user interest and maintain low latency at the same time. We now introduce the whole behavior modeling module and feature modeling module deployed in our CTR prediction system, which consists of three parts: MLP-based deep neural network, transformer- based (Vaswani et al. (2017) encoder aims to learn long-range dependency of session based behaviors, and KFAtt-based (Liu et al. 2020) decoder to predict user interest along with total user actions. We term the encoder-decoder part as Dy DyTrass, the MLP part as DyMLP and the whole module as DyJoint, followed by:
Replace MLP with feature-based dynamic operation
Insert field-based dynamic operation before MLP
Combine Sehavior-Behaviie dynamic operation with Self-Attention
Combine Query- Behavier dynamic operation with KFAtt
G.2 IMPLEMENT DETAILS
Encoder: Within Session Interest Extractor KFAtt (Liu al., 2020) mainly adopt the multi muati-head self-attention used in Transformer (Vaswani et al. (017)). While this seff-attention is only conduct in ression-based behavior for efficiency. Typically, we divide the behavior sequence into Sessions according to their occurring time. The self-attention can be formulated as:
MultiHead((,, K,, Vs) Concat head, Wo
(13)
head, Attention(K, K,W,K v.wY) softmax(K, WK, KT / Vdn..
(14)
Incorporated our proposed methods, the module can be formulated as:
M MultiHeadd Ke, V.) + DynamicOO((Katt.tluu g..))
(15)
Decoder: Query-specific Interest Aggregator KFAtt acts as the decoder to aggregate interest from all sessions for query-specific interest prediction. Incorporated with query-behavior decoder, it can be formulated as:
2 = Concat(head, head, Wo + DynamicOp(query.dl (())
(16)
21

---

Under review as a conference paper at ICLR 2022
G.3 ABLATION STUDY
Effect of varying dynamic kernels in encoder In session-based behavior modeling, we investigate the kernel size of homogeneous behavior operation which influences the temporal receptive fields. The larger kernel size means the larger locality information. [Table shows when we use larger dynamic kernel size, there exists marginally performance downgrading. The main reason is the session-based behaviors are not longer than 10, i.c. the locality has been learned by self-attention (Vaswani et 2017
Effect of convolution generation function Table 15 innesstigates the benefits of convolution- function. The separated convolution with k=3 shows further advancement in user behavior modeling. In this experiment, the weight-generatl function and aggregation function both captures neighborhood representation, in order to learn short-term behaviors to short-term behaviors relationships, besides the diieeniionbbehvior association.
Effect of dynamic compared to static Table 15 also shows dynamic convolution with plain static convolution. As we can see, the dynamic operation generalize better than static one. But the locality brought by convolution still works in self-attention mechanism. When we decrease the kernel size as 1 without locality information and global context, we find it hardly improve the performance. We conjecture that such method can act plain MoE over time-steps, demonstrating interaction among neighbor behaviors facilitates global connected features while features based on behaviors themselves only highlight the diagonal of attention matrix.
H UNDERSTANDING OF DYNAMIC PARAMETERIZED OPERATION
Our dynamic parameterized operation is method to fuse two different stream by pretending ordinary "matrix multiplication" " or "Convolution" with contextual kernels. In atom scene, we outspread the context representation from II" Th As mentioned in section 2.3 when in the simplest situation, the dot production can be used to calculate similarity (Fuantyy(Huaggggg et al. 2020
Rendle al., 2020). Thus, calulate times similarity if set sigmoid as non-linear function learn robust features on the hypothesis that single calculation is not precise, i.c. <<z²z) + (=2 Wx), also can be seen bilinear operation(Lin al. 2015) Hence, the next dynamic layer aims to integrate refined similarity into another context, yielding high-order similarity. Without the dynamic bias term, DPO can be reduced to deep bilinear model. Beyond its capacity for channel interaction, DPO shows instance-wise interaction between channels and time-steps, i.e. convolve extracted time- steps with produced kernels by channels. Although fancy weight-generate function not discussed in our work, we have shown it can introduce strong inductive bias, e.g. attention mechanism and local interaction.
Relation to Dynamic Convolution: We denote behavior- operation coincides with the ideas
of Dynamic Convolution(Han et al. 202TT] Chen et al. 2020, Yang et al. 2019 Li et al. 202T) while
differ on the motivation. DPO be degraded to DyConv Wu et al.) 2019), if set g to combine boftnax-normallnn experts only depend on time-steps for depth-wise convolving.
Relation to Mixture of Experts: We denote it can act another homogeneous implementation of
MoF(Shazzer et al. 2017 Jacobs et al. 1991) where the weight-generate function can be armed with attention mechanism. Despite the similar complexity, our formulation is more favorable to explain the multiplicative interaction between two different features with an inner aggregation approach. However, Moe resorts to weighted aggregate the outputs of highly abstract expert towers.
Relation to Multiplicative Interaction: Eqn. shows dynamic parameterized operation can be
decompose as explicit feature interaction term and implicit feature interaction term. When W is 3D Tensor, the explicit term can be simply seen as bilinear fusion operation (Lin et al. 2015). which captures channel interaction. The implicit term means low-order feature interaction mechanism.
22


Reviews:
[{"review_id": "zWgrfthWzIe", "reviewer": "Reviewer_pP6H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1) This paper is organised well and clearly written. 2) The idea of behavior modeling is novel to me. 3) Detailed theoretical analysis.\n\nCons:\n1) Even though I think the proposed method is technically sound, the results cannot convince me. For the experimental results shown in Table 1, some baselines run worse than their normal performances, and the result of Dynamic Parameterized Network seems not good enough, e.g. compared with the results in [1]. \n2) Eq.5, how to demostrate the rationality of using the low-rank strategy.\n3) few typos, e.g., section 2.5, actions.to\n\n[1] Field-wise Learning for Multi-field Categorical Data, NeurIPS 2020.", "summary_of_the_paper": "The paper proposed a module DPO for CTR prediction to enhence the explicit and implicit information. The authors claimed that they provide the first attempt to extend the dynamic neural networks to CTR prediction, and experiments show that DPN (Dynamic Parameterized Network) significantly outperforms other state-of-the-art methods. ", "summary_of_the_review": "The idea is somewhat novel, but the result is not good enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kFcdvrjuCoO", "reviewer": "Reviewer_AzFU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The algorithm is deployed in a real-world production system.\n- Many variants are proposed and empirically studied.\n\nCons:\n- Not enough novelty. The idea is almost the same as [1] and [2]. Both [1] and [2] conducts feature crossing by using one feature to generate the parameters of a neural network that takes another feature as input. Note that [1] is a recent work that has been deployed in a large-scale real-world e-commerce system as well and is a well-known work among some industry practitioners, especially in China.\n- More details about the experimental setup may be needed to assess if the setup is fair. For example, please consider reporting the total number of parameters of each baseline, since sometimes performance can be increased by simply increasing the model's capacity. It seems possible that the so-called state-of-art baselines are not well-tuned.\n- The reported +1% improvement in the online A/B test could be meaningless without details about the production systems. For example, +1% improvement in an early-stage business with a weak baseline is not as impressive as +1% improvement in a well-developed business with a strong baseline.\n- The writing can be improved.\n\n[1] CAN: https://arxiv.org/abs/2011.05625\n\n[2] A Meta-Learning Perspective on Cold-Start Recommendations for Items. NeurIPS 2017.", "summary_of_the_paper": "This submission is on modeling feature interactions for CTR prediction. It proposes a framework that follows meta-learning. Specifically, to model the interaction between feature F1 and feature F2, it uses a meta neural-network g(F1) that takes F1 and produces the parameter for another neural network f(F2) that takes F2, i.e., the outcome of feature crossing is f(F2) where f's parameter is g(F1). It outperforms the baselines and is deployed online.", "summary_of_the_review": "The idea is almost the same as the existing works, especially [1].", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UxR3LCb5nXs", "reviewer": "Reviewer_2XAa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. A good way to model and utilize feature/context interactions in order to better predict CTR.\n\n2. Promising offline/online results and detailed discussion of offline results.\n\n3. Well written paper. Clear illustration of the methodology as well as discussions about the relations with existing methods\n\nWeaknesses:\n1. Table 7 is way too simple, and so are related descriptions for the experiments. Are there more metrics to demonstrate the superiority of the new method? Is it affecting different segments of queries/users evenly? More details would be helpful to make the A/B test results stronger. Also, it seems CPC also increased. Any intuitions?\n\n2. Why TP99 jumped a lot? Is it purely from the network structure change? How about TP50/90/99.9?\n\n3. Page 4 \"Relation to FM\" would be more interesting to have details. The current high-level discussion is confusing. What is \"self-excluded version\"?\n\n4. In section 2.1 Preliminary, why is it helpful to separate traditional/sequence-base CTR prediction? \n\n5. To distinguish from \"a random paper which tweaks the neural network structure for better performance\", I would suggest more intuitions or discussions on why the added interaction component helps with the CTR prediction accuracy.\n", "summary_of_the_paper": "The paper proposed a new method to handle feature/context interactions within the eCTR prediction neural networks. Details are provided in different scenarios. Experiments are conducted for offline and real-world experiments with promising results.", "summary_of_the_review": "It's a methodology paper working on an important problem (CTR prediction). The proposed method is not extremely innovative but the idea is natural and makes sense, so are the results positive and promising. The novelty is a subjective matter but personally I see enough contributions to clarify and materialize some of the vague commonsense in the industry. One minor concern is about its relevance to \"ICLR\" which focuses on representation learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Although scores are somewhat mixed, even ignoring the most negative review the overall score would still be somewhat below the acceptance threshold.

The authors and reviewers had a robust discussion, mostly about the novelty, experimental setting, and the significance of the results. Although the discussion ultimately did not reach a consensus, I think there are valid points on both sides. E.g. I somewhat disagree with the reviewer that the paper is too application-focused for ICLR, though several other points remain valid. The overall message that the experiments seem not totally convincing was highlighted by multiple reviewers.
Decision: Reject