Dynamic Parameterized Network for CTR Prediction

1 INTRODUCTION . Click-through rate ( CTR ) prediction , which aims to estimate the probability of a user clicking an item , is of great importance in recommendation systems and online advertising systems ( Cheng et al. , 2016 ; Guo et al. , 2017 ; Rendle , 2010 ; Zhou et al. , 2018b ) . Effective feature modeling and user behavior modeling are two critical parts of CTR prediction . Deep neural networks ( DNNs ) have achieved tremendous success on a variety of CTR prediction methods for feature modeling ( Cheng et al. , 2016 ; Guo et al. , 2017 ; Wang et al. , 2017 ) . Under the hood , its core component is a linear transformation followed by a nonlinear function , which models weighted interaction between the flattened inputs and contexts by fixed kernels , regardless of the intrinsic decoupling relations from specific contexts ( Rendle et al. , 2020 ) . This property makes DNN learn interaction in an implicit manner , while limiting its ability to model explicit relation , which is often captured by feature crossing component ( Rendle , 2010 ; Song et al. , 2019 ) . Most existing solutions exploit a combinatorial framework ( feature crossing component + DNN component ) to leverage both implicit and explicit feature interactions , which is suboptimal and inefficient ( Cheng et al. , 2016 ; Wang et al. , 2017 ) . For instance , wide & deep combines a linear module in the wide part for explicit low-order interaction and a DNN module to learn high-order feature interactions . Follow-up works such as Deep & Cross Network ( DCN ) follows a similar manner by replacing the wide part with more sophistic networks , however , posits restriction to input size which is inflexible . Above-mentioned methods pay little attention to user behavior modeling . Recently , attention-based methods like DIN and DIEN have attracted many interests that attempt to capture user preferences based on users ’ historical behaviors ( Zhou et al. , 2018b ; 2019 ; Feng et al. , 2019 ) . With regard to the interaction of characteristics , the use of attention mechanisms in these methods can be treated as an explicit modelling of the interaction of characteristics while neglecting the modelling of implicit interactions of characteristics . The methods mentioned above either model implicit and explicit feature interactions isolated or adopt a suboptimal way to combine them , which can be inefficient . In this work , we aim to address these problems by introducing a small MLP layer that dynamically generates kernels conditioned by the current instance to capture both implicit and explicit feature interactions . The core idea is to first generate context weights and biases from the context stream , and then aggregate them with the input stream adaptively . We formulate a generic function and implement it with an efficient dynamic parameterized operation ( DPO ) . The first weight generator projects contextual features into high-dimensional representation , which models implicit conditional bias . The second feature aggregator aims to fuse input features and projected contextual representation in a multiplicative way , e.g. , matrix multiplication and convolution , maintaining both low- and high-order information . For feature-based modeling , we introduce feature-based DPO where the weight-generate operation dynamically produces instance-wise filters conditioned on the embedded context . The featureaggregate function then applies instance-wise filters to the flattened input by matrix multiplication , allowing to learn multiplicative features . In particular , we further propose a new class of DPO , called field-based DPO , which is not only instance-specific but also field-specific . In that case , the filters vary from field to field and from instance to instance , allowing more complex interactions along the field dimension . For user behavior modeling , we introduce sequence-based DPO that consists of two variants : behaviorbehavior dynamic operation and query-behavior dynamic operation . A representative method of dynamic convolution ( Chen et al. , 2020 ; Yang et al. , 2019 ) shares the convolution kernel , which is generated by the global average of the inputs . Similarly , ( Wu et al. , 2019 ) proposed DyConv , a lightweight fine-grained convolution that depends only on time-step , reinforcing the encoder-based language modeling framework . However , our methods incorporate both local and global information as they jointly use locality-aware methods ( e.g. , convolution or separable convolution ) followed by a global average pooling layer to produce instance-wise weights . The query-behavior dynamic operation is specialized designed for the decoder-based framework in CTR prediction , aiming to capture target-behavior dependency . To our best knowledge , this is the first attempt to extend the business of dynamic neural networks to CTR prediction with extensive experiments on two fundamental scenarios . The comprehensive study against existing solutions validates the superiority of our proposed method . Moreover , we demonstrate that incorporating DPO into the real-world ranking system is beneficial . Our contribution can be summarized as followed : • We propose a generic formulation for capturing multiplicative interaction via weightgenerate and feature-aggregate function , termed DPO . • For feature-based modeling , we propose two variants , named field-based and feature-based DPO , offering a unifying view of implicit and explicit feature interaction . Decomposing these operations , we find they implicitly inherit low- and second-order representation . • For user behavior modeling , we propose two sequence-based variants : behavior-behavior and query-behavior DPO . The first one computes locally perceptual dynamic filters and the second one learns target-behavior dependency in a multiplicative manner . We demonstrate that such operations can benefit the self-attention layers by higher computational efficiency through modeling locality as inductive bias . • The proposed dynamic parameterized networks outperform state-of-the-art methods by a significant margin on both public and real-world production datasets . We also give a comprehensive study about the relationship of our proposed methods to previous Factorization Machine ( Rendle , 2010 ) and CrossLayer ( Wang et al. , 2017 ) . We further demonstrate the effectiveness and superiority of our method with an online A/B test in real-world applications by incorporating it into the fine-rank stage of the real-world ads system . 2 METHOD . We first review the mainstream approaches of feature-based and user behavior ( sequence-based ) modeling under the situation of CTR prediction1 . After that , we introduce DPO and provide several specific instantiations designed for traditional feature-based and sequence-based modeling . 1Related work is in Appendix A due to space limitation . 2.1 PRELIMINARY . Traditional CTR prediction methods mainly predict a probability of a user click an item , which serves as a fundamental evaluation criterion for computing advertising systems . Typically , in a given scenario ( the contexts ) , users click on certain items ( item profiles ) based on their own needs ( query ) and pautorferences ( user profiles ) . Consequently , a model considers four fields of features , i.e. , query , user profile , item profile , and contexts to predict : CTR = F ( query , user profile , item profile , contexts ) ( 1 ) where item and user profiles contain up to tens of fine-grained static attributes depending on the specific circumstances . Sequence-based CTR prediction involves user behaviors additionally : CTR = F ( query , user behavior , user profile , item profile , contexts ) ( 2 ) where the models can learn from the behaviors that have occurred under certain contexts and query in the past to make judgments on the current items . As mentioned in KFAtt ( Liu et al. , 2020 ) , the behavior module can be formulated as : v̂q = UserBehavior ( q , k1 : T , v1 : T ) , where k1 : T and v1 : T are given T historical clicked items and corresponding query words . The most used strategy is to adopt the the self-attention mechanism ( Vaswani et al. , 2017 ) , which naturally learns multiplicative interaction between query and the historical behavior . 2.2 FORMULATION . Namely , multiplicative interaction ( Jayakumar et al. , 2020 ) has been proposed to fuse two different sources of information with the goal of approximating function ftarget ( x , z ) ∈ Rc , where x and z are the input and context respectively . Similarly , we give a generic formulation of DPO in CTR prediction task as : yi = 1 C ( z ) ∑ ∀j f ( xi ; gi ( zj ; θ ) ) ( 3 ) Here i is the index of a position ( in the field , or sequence ) , whose response is calculated with the generated output of z over all existing positions . x is the input embedding , while z denotes any specified context . The generate function g aims to compute dynamic weights and bias followed as one of the inputs of the pairwise aggregate function f , which learns the interactive features reflecting the relationship between xi and zj . The output is finally normalized by a factor C ( z ) . MLP and convolution typically process input and context features in an additive way with fixed weights . While in Eqn . ( 3 ) , using instance-wise generated weights and bias from contexts z , the additive nature is transformed to multiplicative . DPO is also different from bilinear layer ( Lin et al. , 2015 ; He & Chua , 2017 ) for Eqn . ( 3 ) computes representation based on the generated weights over all positions , whereas bilinear layer aggregates information over all positions between x and z , leading to large memory consumption . Furthermore , our generated dynamic weights can maintain more local information , which complements the global counterpart , e.g. , self-attention . DPO is a flexible block and can easily work together with MLP and self-attention layers . 2.3 FEATURE-BASED DPO . Given x ∈ Rm and z ∈ Rn as inputs and context , due to the lack of position information , the generic formulation degrades as y = f ( x ; g ( z ; θ ) ) . For simplicity , we consider f in the form of a linear transformation : f ( x ; z ) =W ( z ) x , where W ( z ) is an instance-wise two-dimensional matrix generated by function g. Now , We discuss the choice of function g. Following the hypernetworks ( Ha et al. , 2016 ) , a natural choice of g is a fully-connected layer to form dynamic weights and bias : y = ( Ŵ Tz + b̂ ) T︸ ︷︷ ︸ DyWeights x+ ( Ẇ Tz + ḃ ) ︸ ︷︷ ︸ DyBias = zTŴx︸ ︷︷ ︸ explicit + Ẇ Tz + B̂Tx+ ḃ︸ ︷︷ ︸ implicit ( 4 ) where ( Ŵ , b̂ , Ẇ , ḃ ) ∈ ( Rn×mc , Rmc , Rn×c , Rc ) . However , the size of Ŵ has quadratic space complexity , unsuitable for deployment in real-world application . Here , we consider a low-rank method in practice , e.g. , a two-layer MLP : g ( z ) =W T2 σ ( W T 1 z + b1 ) + b2 ( 5 ) where ( W1 , b1 ) ∈ ( Rn×l , Rl ) and ( W2 , b2 ) ∈ ( Rl× ( mc+c ) , Rmc+c ) , σ is a non-linear function . Then , we can decompose the output into explicit dynamic weights and bias . The right inductive bias depends on how we select context z and g. We denote the complexity of f is O ( mc ) less than O ( mc+ nc ) of plain MLP layer , while g scales up to O ( lmc+ ln ) . To reduce the complexity , we set l as a small number and use a multi-head mechanism ( Vaswani et al. , 2017 ) . Relation to Cross Network : A cross layer ( Wang et al. , 2017 ) take the feature interaction formulation as xi+1 = x0 · xiwi + bi+1 + xi , where xiwi is scalar . We prove CrossLayer is the simplest formulation of DPO . Let ’ s take x0 as z , xi as x and only use 1-layer MLP as weight-generate function , whose hidden states are 1 , ( i.e . z = x0 , Ŵ ∈ Rn , Ẇ = 0 , B̂ = 1 ) . Thus , we get a scalar output of g as the same as the multiplicative term of CrossLayer . In this way , DPO aims to imitate multiplicative operation .

Summary: The paper proposed a new method to handle feature/context interactions within the eCTR prediction neural networks. Details are provided in different scenarios. Experiments are conducted for offline and real-world experiments with promising results.