Year: 2021

Abstract:
Learning to capture feature relations effectively and efficiently is essential in clickthrough rate (CTR) prediction of modern recommendation systems. Most existing CTR prediction methods model such relations either through tedious manuallydesigned low-order interactions or through inflexible and inefficient high-order interactions, which both require extra DNN modules for implicit interaction modeling. In this paper, we proposed a novel plug-in operation, Dynamic Parameterized Operation (DPO), to learn both explicit and implicit interaction instance-wisely. We showed that the introduction of DPO into DNN modules and Attention modules can respectively benefit two main tasks in CTR prediction, enhancing the adaptiveness of feature-based modeling and improving user behavior modeling with the instance-wise locality. Our Dynamic Parameterized Networks significantly outperforms state-of-the-art methods in the offline experiments on the public dataset and real-world production dataset, together with an online A/B test. Furthermore, the proposed Dynamic Parameterized Networks has been deployed in the ranking system of one of the world’s largest e-commerce companies, serving the main traffic of hundreds of millions of active users.

heading: 1 INTRODUCTION
text: Click-through rate (CTR) prediction, which aims to estimate the probability of a user clicking an item, is of great importance in recommendation systems and online advertising systems (Cheng et al., 2016; Guo et al., 2017; Rendle, 2010; Zhou et al., 2018b). Effective feature modeling and user behavior modeling are two critical parts of CTR prediction.
Deep neural networks (DNNs) have achieved tremendous success on a variety of CTR prediction methods for feature modeling (Cheng et al., 2016; Guo et al., 2017; Wang et al., 2017). Under the hood, its core component is a linear transformation followed by a nonlinear function, which models weighted interaction between the flattened inputs and contexts by fixed kernels, regardless of the intrinsic decoupling relations from specific contexts (Rendle et al., 2020). This property makes DNN learn interaction in an implicit manner, while limiting its ability to model explicit relation, which is often captured by feature crossing component (Rendle, 2010; Song et al., 2019). Most existing solutions exploit a combinatorial framework (feature crossing component + DNN component) to leverage both implicit and explicit feature interactions, which is suboptimal and inefficient (Cheng et al., 2016; Wang et al., 2017). For instance, wide & deep combines a linear module in the wide part for explicit low-order interaction and a DNN module to learn high-order feature interactions. Follow-up works such as Deep & Cross Network (DCN) follows a similar manner by replacing the wide part with more sophistic networks, however, posits restriction to input size which is inflexible.
Above-mentioned methods pay little attention to user behavior modeling. Recently, attention-based methods like DIN and DIEN have attracted many interests that attempt to capture user preferences based on users’ historical behaviors (Zhou et al., 2018b; 2019; Feng et al., 2019). With regard to the interaction of characteristics, the use of attention mechanisms in these methods can be treated as an explicit modelling of the interaction of characteristics while neglecting the modelling of implicit interactions of characteristics.
The methods mentioned above either model implicit and explicit feature interactions isolated or adopt a suboptimal way to combine them, which can be inefficient. In this work, we aim to address these problems by introducing a small MLP layer that dynamically generates kernels conditioned
by the current instance to capture both implicit and explicit feature interactions. The core idea is to first generate context weights and biases from the context stream, and then aggregate them with the input stream adaptively. We formulate a generic function and implement it with an efficient dynamic parameterized operation (DPO). The first weight generator projects contextual features into high-dimensional representation, which models implicit conditional bias. The second feature aggregator aims to fuse input features and projected contextual representation in a multiplicative way, e.g., matrix multiplication and convolution, maintaining both low- and high-order information.
For feature-based modeling, we introduce feature-based DPO where the weight-generate operation dynamically produces instance-wise filters conditioned on the embedded context. The featureaggregate function then applies instance-wise filters to the flattened input by matrix multiplication, allowing to learn multiplicative features. In particular, we further propose a new class of DPO, called field-based DPO, which is not only instance-specific but also field-specific. In that case, the filters vary from field to field and from instance to instance, allowing more complex interactions along the field dimension.
For user behavior modeling, we introduce sequence-based DPO that consists of two variants: behaviorbehavior dynamic operation and query-behavior dynamic operation. A representative method of dynamic convolution (Chen et al., 2020; Yang et al., 2019) shares the convolution kernel, which is generated by the global average of the inputs. Similarly, (Wu et al., 2019) proposed DyConv, a lightweight fine-grained convolution that depends only on time-step, reinforcing the encoder-based language modeling framework. However, our methods incorporate both local and global information as they jointly use locality-aware methods (e.g., convolution or separable convolution) followed by a global average pooling layer to produce instance-wise weights. The query-behavior dynamic operation is specialized designed for the decoder-based framework in CTR prediction, aiming to capture target-behavior dependency.
To our best knowledge, this is the first attempt to extend the business of dynamic neural networks to CTR prediction with extensive experiments on two fundamental scenarios. The comprehensive study against existing solutions validates the superiority of our proposed method. Moreover, we demonstrate that incorporating DPO into the real-world ranking system is beneficial.
Our contribution can be summarized as followed:
• We propose a generic formulation for capturing multiplicative interaction via weightgenerate and feature-aggregate function, termed DPO.
• For feature-based modeling, we propose two variants, named field-based and feature-based DPO, offering a unifying view of implicit and explicit feature interaction. Decomposing these operations, we find they implicitly inherit low- and second-order representation.
• For user behavior modeling, we propose two sequence-based variants: behavior-behavior and query-behavior DPO. The first one computes locally perceptual dynamic filters and the second one learns target-behavior dependency in a multiplicative manner. We demonstrate that such operations can benefit the self-attention layers by higher computational efficiency through modeling locality as inductive bias.
• The proposed dynamic parameterized networks outperform state-of-the-art methods by a significant margin on both public and real-world production datasets. We also give a comprehensive study about the relationship of our proposed methods to previous Factorization Machine (Rendle, 2010) and CrossLayer (Wang et al., 2017). We further demonstrate the effectiveness and superiority of our method with an online A/B test in real-world applications by incorporating it into the fine-rank stage of the real-world ads system.

heading: 2 METHOD
text: We first review the mainstream approaches of feature-based and user behavior (sequence-based) modeling under the situation of CTR prediction1. After that, we introduce DPO and provide several specific instantiations designed for traditional feature-based and sequence-based modeling.
1Related work is in Appendix A due to space limitation.

heading: 2.1 PRELIMINARY
text: Traditional CTR prediction methods mainly predict a probability of a user click an item, which serves as a fundamental evaluation criterion for computing advertising systems. Typically, in a given scenario (the contexts), users click on certain items (item profiles) based on their own needs (query) and pautorferences (user profiles). Consequently, a model considers four fields of features, i.e., query, user profile, item profile, and contexts to predict:
CTR = F(query, user profile, item profile, contexts) (1) where item and user profiles contain up to tens of fine-grained static attributes depending on the specific circumstances.
Sequence-based CTR prediction involves user behaviors additionally: CTR = F(query, user behavior, user profile, item profile, contexts) (2)
where the models can learn from the behaviors that have occurred under certain contexts and query in the past to make judgments on the current items. As mentioned in KFAtt (Liu et al., 2020), the behavior module can be formulated as: v̂q = UserBehavior(q, k1:T , v1:T ), where k1:T and v1:T are given T historical clicked items and corresponding query words. The most used strategy is to adopt the the self-attention mechanism (Vaswani et al., 2017), which naturally learns multiplicative interaction between query and the historical behavior.

heading: 2.2 FORMULATION
text: Namely, multiplicative interaction (Jayakumar et al., 2020) has been proposed to fuse two different sources of information with the goal of approximating function ftarget(x, z) ∈ Rc, where x and z are the input and context respectively. Similarly, we give a generic formulation of DPO in CTR prediction task as:
yi = 1
C(z) ∑ ∀j f(xi; gi(zj ;θ)) (3)
Here i is the index of a position (in the field, or sequence), whose response is calculated with the generated output of z over all existing positions. x is the input embedding, while z denotes any specified context. The generate function g aims to compute dynamic weights and bias followed as one of the inputs of the pairwise aggregate function f , which learns the interactive features reflecting the relationship between xi and zj . The output is finally normalized by a factor C(z).
MLP and convolution typically process input and context features in an additive way with fixed weights. While in Eqn. (3), using instance-wise generated weights and bias from contexts z, the additive nature is transformed to multiplicative. DPO is also different from bilinear layer (Lin et al., 2015; He & Chua, 2017) for Eqn. (3) computes representation based on the generated weights over all positions, whereas bilinear layer aggregates information over all positions between x and z, leading to large memory consumption. Furthermore, our generated dynamic weights can maintain more local information, which complements the global counterpart, e.g., self-attention. DPO is a flexible block and can easily work together with MLP and self-attention layers.

heading: 2.3 FEATURE-BASED DPO
text: Given x ∈ Rm and z ∈ Rn as inputs and context, due to the lack of position information, the generic formulation degrades as y = f(x; g(z;θ)). For simplicity, we consider f in the form of a linear transformation: f(x; z) =W (z)x, where W (z) is an instance-wise two-dimensional matrix generated by function g. Now, We discuss the choice of function g. Following the hypernetworks (Ha et al., 2016), a natural choice of g is a fully-connected layer to form dynamic weights and bias:
y = (Ŵ Tz + b̂)T︸ ︷︷ ︸ DyWeights x+ (Ẇ Tz + ḃ)︸ ︷︷ ︸ DyBias = zTŴx︸ ︷︷ ︸ explicit + Ẇ Tz + B̂Tx+ ḃ︸ ︷︷ ︸ implicit
(4)
where (Ŵ , b̂, Ẇ , ḃ) ∈ (Rn×mc,Rmc,Rn×c,Rc). However, the size of Ŵ has quadratic space complexity, unsuitable for deployment in real-world application. Here, we consider a low-rank method in practice, e.g., a two-layer MLP:
g(z) =W T2 σ(W T 1 z + b1) + b2 (5)
where (W1, b1) ∈ (Rn×l,Rl) and (W2, b2) ∈ (Rl×(mc+c),Rmc+c), σ is a non-linear function. Then, we can decompose the output into explicit dynamic weights and bias. The right inductive bias depends on how we select context z and g. We denote the complexity of f is O(mc) less than O(mc+ nc) of plain MLP layer, while g scales up to O(lmc+ ln). To reduce the complexity, we set l as a small number and use a multi-head mechanism (Vaswani et al., 2017).
Relation to Cross Network: A cross layer (Wang et al., 2017) take the feature interaction formulation as xi+1 = x0 · xiwi + bi+1 + xi, where xiwi is scalar. We prove CrossLayer is the simplest formulation of DPO. Let’s take x0 as z, xi as x and only use 1-layer MLP as weight-generate function, whose hidden states are 1, (i.e. z = x0, Ŵ ∈ Rn, Ẇ = 0, B̂ = 1 ). Thus, we get a scalar output of g as the same as the multiplicative term of CrossLayer. In this way, DPO aims to imitate multiplicative operation.

heading: 2.4 FIELD-BASED DYNAMIC PARAMETERIZED OPERATION
text: GivenX ∈ Rt1×m andZ ∈ Rt2×n as inputs and context, where t1 and t2 represent the field numbers respectively, our goal is modeling the interaction between xi and zj over all field positions. A simple idea is to treat field-based operation as multiple feature-based operations followed by summation over all output. Thus, Eqn. (3) can be expressed as yi = f(xi; 1C(z) ∑ ∀j gi(zj ;θ)), which means all fields share the same instance-wise weights.
However, the field-based operation interacts between all fields, which sometimes introduces unnecessary feature coupling (i.e., multiplicative interaction of brand ID and time, etc.). The empirical evidence finds over-coupling brings more noise and then results in underfitting, albeit their capacity of learning high-order features. A considerable method is to use Self-Field dynamic operation without heavily hand-crafted feature engineering, formulated as: yi = f(xi; gi(xi;θ)) by removing cross-field interactions. Apart from Summation-based and Self-based methods, a more attentive solution can be used to aggregate the dynamic attributes: yi = f(xi; ∑ ∀j h(zj ;wj)gi(zj ;θ)), where h is an attention layer. Beyond taking position into consideration, we can interact the whole context with inputs without explicit summation instead of concatenation, formulated as: yi = f(xi; gi([z1, z2..., zf2];θ)), where [·, ·] is a concatenation operation. These four methods learn pairwise field-based interaction from coarse to fine to model high-order representation, while the feature-based method combines both low- and high-order information over all fields. The complex weight-generate function can be designed for the right inductive bias, but we do not specifically consider such a method for online serving and leave it to future work. Relation to FM: Here, we slightly modify the origin FM implementation (Rendle, 2010) as: y =∑ ∀i ∑ ∀j>i x T i xj by removing the LR term, that takes interaction among all field positions into consideration. Given inputs and the context as xi and {xj ,∀j 6= i}, the function f is simply matrix multiplication and g is the identity function, then Eqn. (3) can be decomposed to: yi =
1 t1−1 ∑ ∀j 6=i x T i xj . Thus, FM can be viewed as the self-excluded version of field-based dynamic operation, where the context is other field features different to the input features.

heading: 2.5 SEQUENCE-BASED DYNAMIC PARAMETERIZED OPERATION
text: User behavior modeling focus on learning from their historical actions to predict whether the users click the current items. As a comparison, transformer-based solutions (Liu et al., 2020; Zhou et al., 2018a) explored the encoder-decoder framework to learn long-range dependencies both source-to-
source and source-to-target, where the encoder exploits multi-head self-attention to extract session interest and the decoder aggregates the query-specific interest. Following the encoder-decoder framework, we consider two variants, i.e., homogeneous Behavior-Behavior and heterogeneous Query-Behavior dynamic operation (homo- and hetero- DPO). We show their multiplicative attributes on Appendix D.
Homogeneous Behavior-Behavior DPO aims to capture feature interaction at different time-steps of behavior. Given X = Z ∈ Rt×n as inputs and context, where t represents the behavior length. For user behavior modeling, our goal is to model the long- and short-term feature interaction. As mentioned above, a long-term function aims to learn non-local interaction between all positions while short-term ones only care about the local information. Thus, a natural way is to adopt global-aware weight-generate function g and local-aware feature-aggregation f . Different to Section 2.3 and Section 2.4, we adopt convolution as f which is widely used for modeling local sequence information with learned weights. For simplicity, we consider function f in the form of a 1D-convolution with kernel size k, while feature-based and field-based methods only use MLP.
yi = f(xi; 1
C(z) ∑ ∀j g(zj ; θ)) = 1 C(x) b k2 c∑ l=b− k2 c t∑ j=0 gl(xj ; θ)xi+l (6)
t∑ j=0 gl(xj ; θ) =W T 2,lσ(W T 1,l t∑ j=0 xj + b1,l) + b2,l (7)
Eqn. (6) shows the function f can act as a convolution operation without bias term which models local neighborhood by dynamic weight, where xi−l is the extracted behavior in position i. Eqn. (7) gives a instantiation of weight-generate function g. Firstly, we aggregate all sequence information and project them into a select operator s ∈ Rd byW1,l and b1,l, whereW1,l ∈ Rn∗d, b1,l ∈ Rd and σ is activation function. Secondly, we use s to explicit aggregate expert weight, whereW2,l ∈ Rd×(nc) and b2,l ∈ Rnc. To use dynamic depthwise-convolution, we can set c = 1. Eqn. (7) captures the multiplicative interaction correspond to global aggregation features. To further strengthen locality, we can adopt local-aware function to capture short-term information of context x (e.g. convolution, separable convolution etc.).
Heterogeneous Query-Behavior DPO aggregates all sequential behaviors as context targeting to interaction with query. Give x ∈ Rm and Z ∈ Rt∗n, we take function f as linear transformation as mentioned in Section 2.3. Eqn. (3) learns interaction between query and behavior over all length followed by summation, and the simplest formulation can be derived as:
y = f(x; 1
C(z) ∑ ∀j∈t g(zj ;θ)) = g( 1 C(z) ∑ ∀j∈t zj ;θ) Tx (8)
Similar to feature-based and field-based methods, query-behavior dynamic operation can easily learn rich multiplicative interaction and conditional inductive bias. The weight-generate function g aims to learn the weight representation Wg ∈ Rm×c. Typically, we can exploit a specific aggregation function, such as Eqn. (7). Compared to self-attention in decoder (Liu et al., 2020), DPO focuses attention on instance-weights based on context, while self-attention takes bipartite attention matrix to aggregate value units. Thus, we conjecture they are two orthogonal and complementary solutions.
Table 1: Comparison with different algorithms of feature-based datasets over 5-runs results. Std≈1e-3.
Datasets Movielens-tag Avazu Criteo
Base Model Auc Logloss Auc Logloss Auc Logloss
FM (Rendle, 2010) 0.9388 0.2797 0.7497 0.3740 0.7933 0.4574 AFM (Xiao et al., 2017) 0.9414 0.2804 0.7454 0.3766 0.7953 0.4554 HOFM (Blondel et al., 2016) 0.9410 0.3088 0.7516 0.3756 0.7960 0.4551 NFM (He & Chua, 2017) 0.9355 0.2955 0.7531 0.3761 0.7968 0.4537 PNN (Qu et al., 2016) 0.9469 0.2792 0.7526 0.3737 0.8026 0.4509 CIN (Lian et al., 2018) 0.9494 0.2600 0.7533 0.3756 0.8042 0.4472 AFN (Cheng et al., 2020) 0.9477 0.2753 0.7512 0.3731 0.8061 0.4458 CrossNet (Wang et al., 2017) 0.9323 0.2929 0.7498 0.3756 0.7915 0.4585 CrossMix (Wang et al., 2020) 0.9379 0.2934 0.7526 0.3738 0.8019 0.4490
DNN 0.9521 0.2576 0.7533 0.3745 0.8028 0.4483 Feature-based DPN 0.9535 0.2538 0.7556 0.3733 0.8097 0.4425 Field-based DPN 0.9507 0.2561 0.7536 0.3750 0.8049 0.4467
Table 2: Ablation study on MovieLens-tag dataset over 5-runs results. We show Auc and logloss.
(a) Instantiations of weight-generate functions: 1 featurebased dynamic operation of different g is added into first layer of a 2-layer MLP(300-300). Std of metrics≈ 1e-3.
g Auc Logloss Params
Base (3-layer MLP) 0.9471 0.2656 192k Base (2-layer MLP) 0.9521 0.2576 101k Base (2-layer MLP, 400) 0.9514 0.2568 175k
HyperDense 0.9524 0.2715 371k Eqn. (5), l=4, σ=sigmoid 0.9522 0.2756 129k Eqn. (5), l=4, σ=softmax 0.9527 0.2563 129k Eqn. (5), h=2, σ=softmax 0.9515 0.2622 101k Pφ(z)Q 0.9503 0.2609 108k W0 + Pφ(z)Q 0.9520 0.2566 117k
(b) Layers and Context: we compare the results by replacing fullyconnected layer with 1 and 2 feature-based dynamic operation of the 2-layer DNN baseline. Also, we compare dynamic results of different context.
Models, Eqn. (5) Context Auc Logloss Params
Base (2-layer MLP) None 0.9521 0.2576 101k
fc1 z0 0.9527 0.2563 129k fc2 z0 0.9522 0.2582 372k fc1 + fc2 (z0, z0) 0.9532 0.2562 400k fc1 + fc2 (x0, x0) 0.9535 0.2538 400k fc1 + fc2 (x0, yl0 ) 0.9530 0.2549 401k fc1 + fc2 (xt, xt) 0.9533 0.2545 400k fc1 + fc2 (zt, zt) 0.9544 0.2566 400k fc1 + fc2 (zm, zm) 0.9530 0.2575 400k
(c) Instantiations of field-based dynamic parameterized network: we compare different aggregation function for MoK, Eqn. (5) as shown in Table 2a in a two-layer field-based DPN. Also, we compare the parameters and time cost with feature-based method and implicit interaction models. We implement all models on 12 cores Intel(R) Xeon(R) CPU E5-2683-v4@2.10GHz with TensorFlow op.
Model context Auc Logloss Params CPU Second/Epoch
DNN None 0.9521±5e-4 0.2576±2e-3 101k 44.7 Field-DNN (implicit), 100-100 None 0.9445±1e-3 0.2669±4e-3 11k 76.7 Larger Field-DNN, 200-200 None 0.9473±8e-4 0.2663±4e-3 44k 175.4 Feature-based DPN (zt, zt) 0.9544±6e-4 0.2566±4e-3 400k 61.4
Field-based DPN (Summation) (x0, yl0 ) 0.9451±1e-3 0.2720±7e-3 46k 44.2 Field-based DPN (Self + implicit) (x0, yl0 ) 0.9488±1e-3 0.2607±2e-3 57k 86.4 Field-based DPN (Summation + implicit) (x0, yl0 ) 0.9495±1e-3 0.2595±1e-3 57k 90.0 Field-based DPN (Attention + implicit) (x0, yl0 ) 0.9489±1e-3 0.2620±2e-3 57k 93.7 Field-based DPN (Concat + implicit) (x0, yl0 ) 0.9507±1e-3 0.2561±2e-3 87k 95.5

heading: 3 EXPERIMENTS
text: We perform comprehensive experiments on feature modeling and user behavior modeling of public and real-world production CTR prediction datasets.

heading: 3.1 EXPERIMENTS ON FEATURE MODELING
text: Setting. We evaluate with MovieLens-tag, Criteo, Avazu with following questions:
• How does DPN perform (effectiveness and efficiency) compared with other base models?
• How do different contexts and weight-generate functions influence the performance?
We use AUC and Logloss as metrics for public datasets. For all experiments, we evaluate the effectiveness of baseline models with the same training setting in AFN (Cheng et al., 2020) implemented by TensorFlow (Abadi et al., 2016). We adopt Adam (Kingma & Ba, 2014) as optimizer with best searched learning rate of a batch size 4096 for all models. We fix the embedding ranks as 10 across all datasets and use same deep neural network (e.g., 3 layers MLP of 400-400-400) with Batch Normalization and ReLU (Ioffe & Szegedy, 2015; Nair & Hinton, 2010) if without specifically noted. The details of our proposed methods are described in Appendix B.
Comparison with state-of-the-art results. Table 1 shows the results from AFN (Cheng et al., 2020) and our reimplemented results in the same setting. We note all these methods are single model without DNN components. First, our feature-based DPN consistently achieves better performance than other explicit interaction methods and also the implicit DNN baseline, which confirms the dynamic aspects contributes to implicit feature interaction. Additionally, when the train dataset and features get larger, the overwhelming margin get larger (e.g. 0.13% on MovieLens-tag, 0.23% on Avazu, 0.69% on Criteo), showing promising potential ability for applied in real-world production. Secondly, our field-based DPN perform better than the other explicit interaction module. We note field-based methods models the relation over different attributes (i.e., UserID, MovieID etc.) where low- and high-order information are captured in a totally different way. Specially, field-based DPN obtain additive module in parallel with multiplicative one while other high-order interaction methods follow an opposite stacked framework to learn the multiplicative features (Qu et al., 2016; Cheng et al., 2020; He & Chua, 2017).
Effectiveness of different instantiations of weight-generate function. Table 2a compares different types of a feature-based dynamic operation added to the DNN baseline (right after the embedding layer for replacing the fully-connected layer). After we search the best DNN baseline model, we replace a dynamic operation with the first fully-connected layer. We list the results of different weight-generate functions, where not all methods perform better than the baseline. We implement the hypernetworks-based idea (Ha et al., 2016) as HyperDense which slightly improve the baseline while add a big chunk of computation resulting for optimization difficulty. When we adopt our proposed simple and effective method as shown in Eqn. (5), gate mechanism can be exploited for better performance, which means mixture of kernels have better generality. Furthermore, we explore some approaches to reduce the complexity of g, such as Multi-Head mechanism, Matrix decomposition. However, they instead downgrade the performance even cannot compete on par with the baseline. We provide more ablation study on the Appendix E.
Multi-Layer Feature-based DPN with different contexts. Table 2b shows the results of deeper dynamic parameterized network with different context. We separately replace DPO with first, second, and all fully connected layers in 2-layer MLP. Table 2b shows that more feature-based DPO in general lead to better results regardless of context. We argue multiple feature-based operations can learn rich and high-order dynamic interaction by imitating MLP. High-order message can be processed with non-linear function layer by layer, which is hard to be found useful via multiplicative models.
In Table 2b, we also study the effectiveness of different context. We set the flatted inputs x0 as the inputs of DPN and evaluate the performance of different contexts such as x0 and z0 where x0 and z0 is the flattened outputs of inputs and context embeddings respectively. We found they share similar results for most experiments while getting the best performance when we set the context as zt (i.e. use the tag information of context embeddings as context inputs). Under careful selection of hyperparameters, this best result mainly originates from the expert knowledge of MovieLens dataset and recommendation system. Meanwhile, it reveals a nice property of our methods: the intrinsic decoupling attribute can be more separably modeled. Interestingly, we find our methods improve results of the infrequent user on MovieLens datasets, as shown in Table 3. We believe the dynamic interaction can warm up the infrequent user embeddings, demonstrating the potential of our methods for the cold-start problem.
Effectiveness and Efficiency of Fieldbased DPNs. We design a family of fieldbased dynamic networks aiming to capture atomic relationships among fields’ features. Table 2c presents results of different aggregate function. We find that all dynamic operations with different context aggregate func-
Table 4: Adapt sequence-based DPO (sDPO) on Transformer. We evaluate the effectiveness of combination of multi-head self-attention mechanism and sDPO.
Encoder Decoder
MSA Homo MSA Heter AUC
! % ! % 0.8718 ! ! ! ! 0.8755 ! ! ! % 0.8728 ! % ! ! 0.8809 ! % % ! 0.8731 % ! ! % 0.8775 % ! % ! 0.8849
Table 5: Comparison with state-of-the-art on Amazon dataset for user behavior modeling. We record the mean AUC over 5 runs. We mainly compare our methods with a well-known attention mechanism.
Model All New inFreq
DIN (Zhou et al., 2018b) 0.8292 0.8029 0.7937 DIEN (Zhou et al., 2019) 0.8675 0.8457 0.8375 Trans (Vaswani et al., 2017) 0.8718 0.8522 0.8438 KFAtt (Liu et al., 2020) 0.8789 0.8578 0.8496
DIN + Heter 0.8836 0.8554 0.8583 DIEN + Heter 0.8693 0.8476 0.8414 Trans + Heter 0.8809 0.8608 0.8526 Trans + Homo 0.8728 0.8530 0.8440
sDPN 0.8849 0.8615 0.8590
tions perform better than the static component, even only correspond to themselves. We may hypothesize that additional contexts can benefit the field features after having been processed for imitating linear transformation, containing multiplicative interaction between inputs and contexts. However, we find the time-consuming is worrying in the CPU machine when the dimensions of outputs are relatively large, making it venturesome to be applied on real-world production.

heading: 3.2 EXPERIMENTS ON USER BEHAVIOR MODELING
text: Experiment setting. We evaluate sequence-based DPN (sDPN) on Amazon Electronics Datasets. We only adopt AUC as metrics with the same training setting in KFAtt (Liu et al., 2020) implemented by TensorFlow. The task is to predict whether a user will write a review for the target item after reviewing historical items. We refer the readers to KFAtt (Liu et al., 2020) for more details.
Comparison with state-of-the-art. From Table 5 we can see that sequence-based DPN achieves the best performance than all state-of-the-arts on total situations, including the strong baseline KFAtt. When incorporating heterogeneous DPO into the attention mechanism, we find both DIN (Zhou et al., 2018b) and DIEN (Zhou et al., 2019) perform better than the origin baseline where the armed DIN outperform all the base models on Amazon datasets by a large margin, which shows that heterogeneous DPO can effectively learn complementary representation which can benefit the attention mechanism.
Adaptation to Self-Attention. Table 4 presents us how sDPO performs when incorporated with self attention mechanism. For homogeneous DPO, we find it performs slightly better than MHSA counterparts. When we use session-wise representation for user behavior modeling, self-attention can capture local information over handcraft scope of time designed by experts while narrow neighbor interaction of convolution may not contribute to learning users’ attention due to the short session. For heterogeneous DPO, we find it can effectively facilitate the decoder counterpart no matter which encoder we adopt. Overall, the sequence-based DPN(sDPN) achieve best results than other combination, which shows the effectiveness of our propose homogeneous and heterogeneous DPO.

heading: 3.3 EXPERIMENTS ON REAL-WORLD PRODUCTION DATASET AND ONLINE A/B TESTING
text: We conduct all feature-based, field-based, and sequence-based experiments on the Real-world Production dataset. In the offline experiments, we observe the progressive improvement from modern rank models consisting of advanced user behavior and multi-modal features model in our advertising system. Table 6 shows our DyMLP and DyJoint get significant advancement compared to origin implementation. For feature interaction modeling, DyMLP shows better performance than the commonly used DNN component while only add a little extra cost. Despite we don’t explore the effectiveness of ensemble DPN models in a public dataset, Table 6 presents even one lightweight field-based DPO that can benefit the generality. To reduce the complexity, we use a multi-head mechanism in feature-based DPN which may influence the effectiveness.
For sequence-based DPO, we conduct more ablation studies in Appendix G.3. Our homogeneous DPO can act as a specific form like dynamic convolution. Incorporating it with a session-based self-attention encoder, we can inject inductive bias learned from local neighbor-
hood information into global long-term dependencies based on Transformer-like models (Vaswani et al., 2017; Feng et al., 2019). Beyond that, heterogeneous DPO can learn more conditional multiplicative interaction which models the user interest on given items, showing greater power than the homogeneous component. Combined with all techniques, we get the best results by a large margin to the online model.
In the online A/B test, Table 7 shows DyJoint contributes 1.0% CTR gain against the online component, which demonstrates the superiority over the highly optimized base model on our ad system. However, DyJoint leads to larger online latency compared to the base model due to the increment of model parameters and memory access.

heading: 4 CONCLUSION
text: In this paper, we describe a new class of neural networks that captures both explicit and implicit interaction via dynamic parameterized operation. Our proposed block can be easily inserted into existing CTR predicting architectures for fusing features from different modalities. Our experiments show that it overwhelms the existing feature-crossing-based and attention-based models on two fundamental tasks. Furthermore, we confirm its representation effectiveness in the real-world production dataset. For the theoretical understanding, we decouple dynamic operation for comprehensive study with high-order feature-crossing methods and self-attention. Overall, we open a new era where current mainstream solutions are dominated by self-attention mechanisms and MLP in CTR prediction.

heading: A RELATED WORK
text: Feature Crossing as high-order interaction has been widely explored in CTR prediction and recommendation system by capture a cross feature which can be defined as ∏n i=0 x ai i . Factorization Machine (Rendle, 2010) was firstly proposed to calculate similarity by using inner product between two different features (e.g, item feature and user feature) borrowed from the collaborative-filtering based model (Sarwar et al., 2001). With the rise of deep neural network, recent works focus on combining feature crossing with Embedding+MLP paradigm for better performance (i.e. stacked and parallel paradigm) as shown in Fig. (3a) and Fig. (3b). Late fusion is mostly used in parallel paradigm. Wide and deep (Cheng et al., 2016) propose wide component to form order-2 features by cross-product transformation between sparse inputs and a deep layer for order-1 features. DCN (Wang et al., 2017) propose to use parallel structure composed by bit-wise cross-layer and deep neural network. DeepFM (Guo et al., 2017) adopts a parallel structure to fuse the FM and DNN outputs. To facilitating the high-order representation, xDeepFM (Lian et al., 2018) are proposed to generate vector-wise high-order feature, achieving further advancement by stacking multiple interaction layers. Contrast to polynomial networks, AutoInt(Song et al., 2019) exploit a multi-head self-attention layer to automatically learn the high-order feature interaction of all atom features(e.g. item id, user id, brand id etc.). Ensembled with DNN, AutoInt achieve better result. Constrast to parallel ensemble methods, stacked paradigm uses an explicit layer to extract cross features followed by an implicit layer to process them, i.e. DNN. NeuFM (He & Chua, 2017) uses Bi-Interaction layer with stacked deep neural network for bridging generalized linear models and deep learning. AFN (Cheng et al., 2020) models cross features of adaptive orders followed by 3-layer MLP. Despite the inherit strong generality incorporated with DNN, stack paradigm often resorts to further ensemble with DNN for better results.
However, due to either the low capacity or high latency of so-called interaction layer, those methods typically failed in real-world rank system. Recent application of polynomial networks on CTR prediction can be seen as a special case of multiplicative interaction to formulate scalar-wise, element-
wise or feature-wise multiplier fusion between projected embeddings. We prove that our methods can degrade as the bit-wise cross-layer (Wang et al., 2017) and pairwise interaction layer (Rendle, 2010).
User behavior modeling aims to extract users’ dynamic and evolving interest over items. Earlier works (Covington et al., 2016; Song et al., 2016; Yu et al., 2016) use a target-independent manner to aggregate total behaviors, failed to learn the interaction between target and past behaviors. Recent works (Zhou et al., 2019; 2018b;a; Liu et al., 2020) achieve massive improvement by adopting attention mechanism for modeling long-range representation between user historical behaviors and other target features. Orthogonally, our proposed methods can be formulated as multiplicative interaction between two different streams.
Dynamic Neural Networks enjoy preferable properties than static ones, whose computation graph and forward parameters are fixed with limited representation power (Han et al., 2021). Instance-wise dynamic neural networks typically have dynamic structure (Tanno et al., 2019), dynamic weights (Yang et al., 2019; Ha et al., 2016; Chen et al., 2020), or dynamic layers (Huang et al., 2018) in inference time. Jayakumar (Jayakumar et al., 2020) provide theoretical comprehension about why dynamic weights are beneficial: HyperNetworks (Ha et al., 2016) can be thought of generalised multiplicative interaction in the non-factorised sense, whose filters are generated by conditional inputs.
B IMPLEMENTATION DETAILS OF DPN
B.1 DATASETS AND EXPERIMENTAL PROTOCOLS
Public dataset for feature-interaction modeling. We conduct experiments on three public realworld datasets in Section 3.1 as shown in Table 8. MovieLens-tags2 has been prepared into (user, movie, tag) format for personalized recommendation. Totally, there are 2006859 tuples, constituting of 3 categorical fields: UserID, MovieID and tag. Avazu3 is click-through rate dataset including 22 feature fields of user features and advertisement attributes. There are about 40 million instances in total. Criteo4 is a benchmark dataset for displayed ads CTR prediction, containing 26 categorical fields and 13 numeric fields. It has about 45 million user-click records on displayed ads. We split the dataset in 7:2:1 for training, validataion and testing respectively.
Public dataset for user behavior modeling We conduct sequence modeling on Amazon book dataset (McAuley et al., 2015) to learn users’ interest. We use the 5-core Electronics subset, including 1689188 instances with 192403 users and 63001 goods from 801 categories. The task is to predict the intent of writing users’ review for a target item regard to the historical reviews. In practice, we use the last review of each user as test split while others as train split. The negative instances are randomly sampled from not-reviewed items.
Real-world Production dataset is a daily traffic log generated from the search advertising system. Typically, we sample negatively a first 32-days click-through logs for training data containing 2.4 billion samples and non-negatively the next-day click-through logs for test data containing about 1 million samples. For the user-behavior data setting, we follow the KFAtt (Liu et al., 2020) to choose user clicks/queries in previous 70 days as behavior v/k. The other features consist of the query, user and ad profile, ad image and contexts, built for modeling feature interaction.
Metrics: AUC is almost the default offline evaluation metrics in advertising system due to the direct relation to online performance. Thus, we use AUC as a evaluation metric for evaluation on both public dataset and real-world production dataset. For fair comparison, we additionally add Logloss as another metric when evaluated on public dataset. We denote that 0.001-level improvement of AUC on CTR prediction task is significant, which has been a common sense in previous works (Cheng et al., 2016; Guo et al., 2017; Lian et al., 2018; Wang et al., 2020; Wang et al., 2017).
2https://grouplens.org/datasets/movielens/ 3https://www.kaggle.com/c/avazu-ctr-prediction 4https://www.kaggle.com/c/criteo-display-ad-challenge
B.2 EMBEDDING LAYER
For all ctr models, the categorical features and dense features (UserID, ItemID, User Age etc) are hashed to sparse one-hot features as the inputs of embedding layers. We project i-th one-hot feature from sparse high-dimensional space to a lower one via xemb,i =Wiei, where ei ∈ {0, 1}v andW ∈ Re×v; v and e means vocab and embedding size respectively. Thus, for feature-based modeling, we take the output of the concatenation of all the embedding vectors: x0 = [xemb,0, xemb,1, ..., xemb,n] as our named features. For field-based modeling, we instead take the embedding vectors of every sparse ID as fields aiming to learning the interaction of pairwise features (e.g. [UserID, ItemID], [UserID, UserAge]).
B.3 FEATURE-BASED AND FIELD-BASED MODELING
As mention in Eqn. (3), all of our methods induce an extra information features z for boosting generalization. However, we don’t use extra fields as show in Table 8 which represents the effectiveness comes from the combination of fully-connected layer and feature-crossing module. For feature-based and field-based DPN, we stack several feature- and field-based DPO layer followed by classifier layer to build them. The implicit interaction module of feature-based DPN is fully-connected layer with non-linear function, while the implicit part of field-based DPN has two parts: field-interaction moduleWf and linear transformation moduleWl, whereWf ∈ Rn×n andWl ∈ Rel−1×el . Thus the static outputs can be calculate byWfX0Wl, whereX0 ∈ Rn×el−1 . We name it as Field-DNN.
B.4 USER BEHAVIOR MODELING ON AMAZON DATASET
To evaluate the effectiveness of adapting sDPO on various methods, we adpot different strategy to combine sDPO with DIN, DIEN and Transformer. For DIN and DIEN, we produce the heterogeneous DPO outputs followed by element-wise summation with origin attention mechanism outputs. For Transformer, we sum the homogeneous and heterogeneous DPO outputs and self attention outputs in encoder and decoder respectively. Except for the increment benefit over other methods, we test the effectiveness only using DPO, i.e. sequence-based Dynamic Parameterized Network (sDPN). The sDPN simply replaces the self attention counterparts on Transformer framerwork by DPO layers.
B.5 USER BEHAVIOR MODELING FOR REAL-WORLD PRODUCTION DATASET
We follow the setting of KFAtt (Liu et al., 2020) along with the feature-based methods and field-based methods, evaluated only on real-world production dataset. There, 96 multi-modal features are first embedded into 16-dimension vectors and then fed for field-based and feature dynamic operation. The origin implement make use of 4-layer MLP with dimension 1024, 512, 256, 1. When there is a 30 minutes’ time interval between adjacent behaviors, we conduct a session segmentation. For each instance, we use at most 10 sessions and 25 behaviors per session. The learnt hidden user interest, v̂q ∈ R64 is concatenated to the output of 1st FC layer together with a 150-dimensional visual feature vector. Our homogeneous behavior-behavior operation are used to fuse the session-based self-attention encoder output in additive way with kernel size = 3. Our heterogeneous query-behavior operation are used to fuse the predicted interest produced by cross-attention. All the weight generate function are a SE-layer (Hu et al., 2018) whose intermediate down ratio is grid searched from [0.125, 0.25, 0.5].

heading: C DETAILS OF THE RELATION TO FM
text: As shown in (Rendle, 2010), FM can be formulated as:
y(x) = b+ n∑ i=1 wixi + n∑ i=1 n∑ j=i+1 < vi,vj > xixj
= linear(x;θ) + 1
n− 1 n∑ i=1 (vixi ⊗ ∑ ∀j!=i vjxj)
= linear(x;θ) + 1
n− 1 n∑ i=1 f(xi; g( ∑ ∀j!=i xj))
(9)
Where v means the embedding table, x mean the categorical features. Thus we can decompose the output y as summation of each field-wise interaction features. Thus the function f and function g can be represented for matrix multiplication and identity function. The self-excluded version means we use the ∑ ∀j!=i xj as context features which exclude the input feature xi.

heading: D ANALYSIS MULTIPLICATIVE ATTRIBUTE OF SDPO
text: D.1 ANALYSIS OF HOMOGENEOUS DPO
yi = f(xi; 1
C(z) ∑ ∀j g(zj ; θ))
= 1
C(x) b k2 c∑ l=b− k2 c t∑ j=0 gl,w(xj ; θ)xi+l + t∑ j=0 gl,b(xj ; θ)
= b k2 c∑ l=b− k2 c (W T2,l(W T 1,l t∑ j=0 xj + b1,l) + b2,l) Txi+l + gb(x̂)
= b k2 c∑ l=b− k2 c ( t∑ j=0 xTjW1,lW2,lxi+l +W T s,lxi+l) + gb(x̂)
(10)
As shown in Eqn. (10), we show the homogeneous DPO actually learns both multiplicative and additive relation among local neighbor and each behavior features followed by summation aggregate function when we don’t use non-linear function. To further reduce complexity, a common way is to learn the feature interaction between global embedding aggregated firstly and local neighbor, formulated as:
yi = b k2 c∑ l=b− k2 c (W T2,lσ(W T 1,l t∑ j=0 xj + b1,l) + b2,l) Txi+l (11)
where σ can be sigmoid and softmax function.
D.2 ANALYSIS OF HETEROGENEOUS DPO
y = f(x; 1
C(z) ∑ ∀j∈t g(zj ;θ))
= gw( 1
C(z) ∑ ∀j∈t zj ;θ) Tx+ gb( 1 C(z) ∑ ∀j∈t zj ;θ)
= (W T2 (W T 1 t∑ j=0 zj + b1) + b2) Tx+ gb(ẑ)
= (W T2 (W T 1 ẑ)) Tx+ (W T2 b1 + b2) Tx+ gb(ẑ)
= ẑTW1W2x︸ ︷︷ ︸ query−whole +W Ts x = 1 C(z) t∑ j=0 (zTj W1W2x︸ ︷︷ ︸ query−each ) +W Ts x+ gb(ẑ)
(12)
As shown in Eqn. (12), We decompose the simplest formulation of Eqn. (8) whereW1 ∈ Rd∗n,W2 ∈ Rn∗(m∗c), b1 ∈ Rn, b2 ∈ Rmc,x ∈ Rm, z ∈ Rn. The equation omits the reshape operation. We can see that Heterogeneous DPO captures multiplicative and additive between each behavior and query different to the self attention mechanism. However, Eqn. (12) can’t be enhanced by non-linear function, Empirically, a better way is to learn the relation of the embedding representing for the whole behavior sequence and query features, which further reduce the computations and also have strong compatibility with complex weight-generation function g such as SE-layer (Hu et al., 2018).

heading: E ADDITIONAL EXPERIMENTS ON MOVIELENS-TAG
text: E.1 ABLATION STUDY
We perform ablations on variants of feature-based and field-based DPN. Unless specified otherwise, all experimental results in this sections report MovieLens AUC and logloss by training a DPN architecture that replace weighted fully-connected layer.
Varing depth and width for feature-based and field-based DPN Table 9 presents the impact of different depth and width on performance. Our experiments indicate that Feature-based DPN can achieve better performance with similar time consuming over a wide range hyperparameters, which demonstrate the robustness of the method. However, the atomic field-based DPN still lack the capacity to compete on par with MLP baseline. Furthermore, we find the atomic fields’ interaction is hard to inert into feature-based DPO due to the extra fields dimension.
MD, MH vs Plain Table 10 presents the performance of matrix decomposition and multi-head mechanism on feature-based DPN. Despite their success when applied in other domains (Vaswani et al., 2017; Li et al., 2021; Wu et al., 2019), we find these methods dont’t work in traditonal CTR prediction. We conjecture the multi-head mechanism hurts the expressivity of fully-connected layer due to the separable interaction among channels. For matrix decomposition, we aim to generate instance-wise kernels by Pφ(z)Q, where P ∈ Rn×r, φ(z) ∈ Rr×r,Q ∈ Rr×m. We set r = 32 in Table 10. Thus, we may solve the low-rank completion problem for better performance. We conjecture simply utilization of dynamic parameterized fully-connected layer by matrix decomposition leads to optimization difficulty. We leave it into future study.

heading: F ADDITIONAL EXPERIMENTS ON MOVIELENS-1M
text: Besides experiments on three public datasets, We extraly conduct feature-based and field-based ablation studies on widely-used MovieLens-1M.
MovieLens-1M is a relative small dataset (Harper & Konstan, 2015), contains users’ rating on movies with 7 features, e.g. user features and movie features and rating. We follow the experiment
setting of AutoInt (Song et al., 2019), which randomly split 80% of dataset into train data, 10% into valid data and the rest 10% into test data. And we set the ratings below 1s and 2s as 0s, above 4s and 5s as 1s. All rating 3s should be removed.
Training For ablation study on MovieLens-1M, our baseline models follow the settings in AutoInt (Song et al., 2019) with random initialized weights, implemented by Tensorflow (Abadi et al., 2016) with the tricks (Srivastava et al., 2014; Nair & Hinton, 2010) but we don’t use BN (Ioffe & Szegedy, 2015) due to the poor performance. The feature-based model take the same setting as DeepCrossing (Shan et al., 2016) introduced in AutoInt with 4-layer MLP of size 100. The function g is 2-layer MLP. The field-based models use three dynamic layer and set hidden states as 64 the same as their public code5, (e.g. replace self-attentive layer with our field-based dynamic operation). After that, we use grid search to select hyperparameters, (i.e. dropout rate, learning rate). For fair comparison, all methods set embedding dimension as 16 and use Adam (Kingma & Ba, 2014) as default optimizer with β1 = 0.9, β2 = 0.999 , batch size is 1024. However, we only replace feature-based operation on the final two layers before classifier, due to the memory limitation and insert 1-layer field-based operation firstly.
Comparison with state-of-the-art Table 12 shows the results from AutoInt (Song et al., 2019). Due to the random data splits, we rerun their experiments for comparison. We note the 0.001-level fluctuation on evaluation metric compared to the cited’ ones is acceptable when trained and evaluated on randomly split setting. Nevertheless, our feature-based method surpasses the other existing high-order based methods (Wang et al., 2017; Lian et al., 2018; He & Chua, 2017) by a good margin, even the proposed field-based operation by ourselves. However, what surprise us is the heavily tuned DNN model still excels high-order parts in contrast to the results reported in the AutoInt. We infer that MLP-based DNN model can be a good universal approximate function to attain a suitable local minimum with good hyperparameters. In this way, there is no doubt feature-based solution is better than others, because of its capacity for learning both implicit and explicit feature interaction. As far as we know, our proposed method is state-of-the-art solution in the same setting, even without the DNN component.
5https://github.com/DeepGraphLearning/RecommenderSystems/tree/master/ featureRec
Comparison with different heads in feature-based operation Table 13a compares different head numbers of 4 dynamic parameterized layers with context z as genre. We find a good head number can effectively improve the performance, while larger lead to bad results. We derive this large gap is mainly due to the reason our baseline model use relative smaller embedding size and hidden state with increasing head numbers. When head numbers equal to 4, the size of total parameters is about 11k, which seriously downgrades the performance. However, a properly head number trades off the extent of implicit and explicit high-order interaction, inducing larger variance of selection on dynamic weights/bias.
Comparison with different non-linear function in feature-based operation Table 13b shows the performance of different non-linear function in Eqn. (5). Here, we set l = 4 and head numbers equal to 2, for simplicity. We find even using plant low-rank approximation without non-linear function, the result still better than high-order part reported in Table 1. This gives evidence that non-restrict multiplicative interaction among hidden states can be a complementary component for further improving the capacity of MLP. When using non-linear function sigmoid and softmax, we found the performance moves a significant step about 0.001-level. Constrained non-linear function controls the outputs of selection layer between (0, 1), which can be seen as the attention over kernels (e.g. mixture of expert layers). We note that the best non-linear function depends on the dataset respectively. Such methods for kernels aggregation are friendly for optimization, allowing us to learn diversity of weights while consume nearly same time as plain fully-connected layer.
Comparison with different context in feature-based operation Table 13c compares the different context of feature-based dynamic operation. We can see that different combination of fields as context z will lead to variant results, where the worst only improve slightly than AutoInt, but less than DNN model. Interestingly, we select separately the user feature gender and movie feature genre as context. The gender typically consist two classes of male and female. Thus, the multiplicative interaction can be understood as how the gender influences whether a user rates a movie. Contrast to human intuition on gender, the results reveal that gender is not main reason for different rating on movie, e.g. explicit modeling gender interaction with other features leads to unnecessary bias. However, when modeling with genre of movies, we get large improvement on performance compared to the genders’. We conjecture the genre plays important role on rating of users. Moreover, when combined with all field features, we can achieve best results, representing the given features can be classified effectively in high dimension.
Comparison with different aggregation function in field-based operation The feature-based methods give empirical evidence that how to select head numbers, non-linear function and context. For simplicity, we choose the same best setting in ablation study of field-base dynamic operation. Table 14a compares four methods of how to aggregate differnet field information as mentioned in Section 2.4. When only using the field to correspond itself, we find it perform worst due to the scarcity of field-wise feature interaction compared to the simple summation version. When using a weighted-sum layer and concatenation-MLP layer, we find such methods improve almost 0.005-level compared to summation-based solution. The main difference is using a learned linear combination of all field features, which implicit models low-order feature interaction. Thus, we can conclude implicit linear transformation overall fields is beneficial for modeling high-order representation. However, we notice the DNN result overwhelm the field-wise operations’, meaning the multiplicative interaction brought by high-order component is not the main course in CTR prediction.
Does field-wise component contribute to feature-wise component? Despite the low capacity of field-wise dynamic operation, we explore the contribution when combining field-wise component with feature-wise’s. Table 14b shows there are no obvious benefit when we use both operation at same time no matter how we stack them, while facilitating the performance when composed of parallel structure. The stack structure directly exploit the field-based features as the inputs of featurebased layers. Once the fundamental representation has low diversity and expressivity, subsequent fancy modules can’t perform their proper ability, e.g. three layers stacked Self-based operation dominates the effectiveness. However, we find the aggregation methods of weighted summation can effectively boost the performance sightly no matter which structure we used. Thus, we confirm
when combined those two methods in a suitable way, field-wise component can contribute to the feature-wise component for further advancement of performance.

heading: G ADDITIONAL EXPERIMENTS ON REAL-WORLD PRODUCTION DATASET FOR USER BEHAVIOR MODELING
text: G.1 INCORPORATING DYNAMIC OPERATION INTO ONLINE RANKING SYSTEM
In previous section, we mainly discuss the dynamic mechanism among different inputs and context. However, real-world industry system scales behavior-based and feature-based modeling containing many techniques to precisely extract user interest and maintain low latency at the same time. We now introduce the whole behavior modeling module and feature modeling module deployed in our CTR prediction system, which consists of three parts: a MLP-based deep neural network, a transformerbased (Vaswani et al., 2017) encoder aims to learn long-range dependency of session-based behaviors, and a KFAtt-based (Liu et al., 2020) decoder to predict user interest along with total user actions. We term the encoder-decoder part as DyTrans, the MLP part as DyMLP and the whole module as DyJoint, followed by:
• Replace MLP with feature-based dynamic operation • Insert field-based dynamic operation before MLP • Combine Behavior-Behavior dynamic operation with Self-Attention • Combine Query-Behavior dynamic operation with KFAtt
G.2 IMPLEMENT DETAILS
Encoder: Within Session Interest Extractor KFAtt (Liu et al., 2020) mainly adopt the multi-head self-attention used in Transformer (Vaswani et al., 2017). While this self-attention is only conduct in session-based behavior for efficiency. Typically, we divide the behavior sequence into Sessions according to their occurring time. The self-attention can be formulated as:
MultiHead(Ks,Ks, Vs) = Concat(head1, . . . , headh)WO (13)
headi = Attention(KsW Q i ,KsW K i , VsW V i ) = softmax(KsW Q i W K i T KTs / √ dk)VsW
V i (14)
Incorporated our proposed methods, the module can be formulated as:
MultiHead(Ks,Ks, Vs) +DynamicOp(Ks,k, gs(Ks)) (15)
Decoder: Query-specific Interest Aggregator KFAtt acts as the decoder to aggregate interest from all sessions for query-specific interest prediction. Incorporated with query-behavior decoder, it can be formulated as:
vq = Concat(head1, . . . , headh)WO +DynamicOp(query, g(K)) (16)
G.3 ABLATION STUDY
Effect of varying dynamic kernels in encoder In session-based behavior modeling, we investigate the kernel size of homogeneous behavior operation which influences the temporal receptive fields. The larger kernel size means the larger locality information. Table 15 shows when we use larger dynamic kernel size, there exists marginally performance downgrading. The main reason is the session-based behaviors are not longer than 10, i.e. the locality has been learned by self-attention (Vaswani et al., 2017).
Effect of convolution generation function Table 15 investigates the benefits of convolution-based weight-generate function. The separated convolution with k=3 shows further advancement in user behavior modeling. In this experiment, the weight-generate function and aggregation function both captures neighborhood representation, in order to learn short-term behaviors to short-term behaviors relationships, besides the dimension-behavior association.
Effect of dynamic compared to static Table 15 also shows dynamic convolution with plain static convolution. As we can see, the dynamic operation generalize better than static one. But the locality brought by convolution still works in self-attention mechanism. When we decrease the kernel size as 1 without locality information and global context, we find it hardly improve the performance. We conjecture that such method can act as plain MoE over time-steps, demonstrating interaction among neighbor behaviors facilitates global connected features while features based on behaviors themselves only highlight the diagonal of attention matrix.

heading: H UNDERSTANDING OF DYNAMIC PARAMETERIZED OPERATION
text: Our dynamic parameterized operation is one method to fuse two different stream by pretending ordinary "matrix multiplication" or "Convolution" with contextual kernels. In atom scene, we outspread the context representation from Rn to Rm∗c. As mentioned in section 2.3, when c = 1 in the simplest situation, the dot production can be used to calculate similarity(Huang et al., 2020; Rendle et al., 2020). Thus, we calulate c times similarity if we set sigmoid as non-linear function to learn robust features on the hypothesis that single calculation is not precise, i.e. σ(ztx)→ σ(ztWx), also can be seen as bilinear operation(Lin et al., 2015) . Hence, the next dynamic layer aims to integrate refined similarity into another context, yielding high-order similarity. Without the dynamic bias term, DPO can be reduced to deep bilinear model. Beyond its capacity for channel interaction, DPO shows instance-wise interaction between channels and time-steps, i.e. convolve extracted timesteps with produced kernels by channels. Although fancy weight-generate function is not discussed in our work, we have shown it can introduce strong inductive bias, e.g. attention mechanism and local interaction.
Relation to Dynamic Convolution: We denote behavior-behavior operation coincides with the ideas of Dynamic Convolution(Han et al., 2021; Chen et al., 2020; Yang et al., 2019; Li et al., 2021) while differ on the motivation. DPO can be degraded to DyConv(Wu et al., 2019), if we set g to combine softmax-normalized experts only depend on time-steps for depth-wise convolving.
Relation to Mixture of Experts: We denote it can act as another homogeneous implementation of MoE(Shazeer et al., 2017; Jacobs et al., 1991) where the weight-generate function can be armed with attention mechanism. Despite the similar complexity, our formulation is more favorable to explain the multiplicative interaction between two different features with an inner aggregation approach. However, Moe resorts to weighted aggregate the outputs of highly abstract expert towers.
Relation to Multiplicative Interaction: Eqn.(4) shows dynamic parameterized operation can be decompose as explicit feature interaction term and implicit feature interaction term. When Ŵ is a 3D Tensor, the explicit term can be simply seen as bilinear fusion operation (Lin et al., 2015), which captures channel interaction. The implicit term means a low-order feature interaction mechanism.

Review summaries:
This submission is on modeling feature interactions for CTR prediction. It proposes a framework that follows meta-learning. Specifically, to model the interaction between feature F1 and feature F2, it uses a meta neural-network g(F1) that takes F1 and produces the parameter for another neural network f(F2) that takes F2, i.e., the outcome of feature crossing is f(F2) where f's parameter is g(F1). It outperforms the baselines and is deployed online.

The paper proposed a new method to handle feature/context interactions within the eCTR prediction neural networks. Details are provided in different scenarios. Experiments are conducted for offline and real-world experiments with promising results.