Dynamic Parameterized Network for CTR Prediction

1 INTRODUCTION . Click-through rate ( CTR ) prediction , which aims to estimate the probability of a user clicking an item , is of great importance in recommendation systems and online advertising systems ( Cheng et al. , 2016 ; Guo et al. , 2017 ; Rendle , 2010 ; Zhou et al. , 2018b ) . Effective feature modeling and user behavior modeling are two critical parts of CTR prediction . Deep neural networks ( DNNs ) have achieved tremendous success on a variety of CTR prediction methods for feature modeling ( Cheng et al. , 2016 ; Guo et al. , 2017 ; Wang et al. , 2017 ) . Under the hood , its core component is a linear transformation followed by a nonlinear function , which models weighted interaction between the flattened inputs and contexts by fixed kernels , regardless of the intrinsic decoupling relations from specific contexts ( Rendle et al. , 2020 ) . This property makes DNN learn interaction in an implicit manner , while limiting its ability to model explicit relation , which is often captured by feature crossing component ( Rendle , 2010 ; Song et al. , 2019 ) . Most existing solutions exploit a combinatorial framework ( feature crossing component + DNN component ) to leverage both implicit and explicit feature interactions , which is suboptimal and inefficient ( Cheng et al. , 2016 ; Wang et al. , 2017 ) . For instance , wide & deep combines a linear module in the wide part for explicit low-order interaction and a DNN module to learn high-order feature interactions . Follow-up works such as Deep & Cross Network ( DCN ) follows a similar manner by replacing the wide part with more sophistic networks , however , posits restriction to input size which is inflexible . Above-mentioned methods pay little attention to user behavior modeling . Recently , attention-based methods like DIN and DIEN have attracted many interests that attempt to capture user preferences based on users ’ historical behaviors ( Zhou et al. , 2018b ; 2019 ; Feng et al. , 2019 ) . With regard to the interaction of characteristics , the use of attention mechanisms in these methods can be treated as an explicit modelling of the interaction of characteristics while neglecting the modelling of implicit interactions of characteristics . The methods mentioned above either model implicit and explicit feature interactions isolated or adopt a suboptimal way to combine them , which can be inefficient . In this work , we aim to address these problems by introducing a small MLP layer that dynamically generates kernels conditioned by the current instance to capture both implicit and explicit feature interactions . The core idea is to first generate context weights and biases from the context stream , and then aggregate them with the input stream adaptively . We formulate a generic function and implement it with an efficient dynamic parameterized operation ( DPO ) . The first weight generator projects contextual features into high-dimensional representation , which models implicit conditional bias . The second feature aggregator aims to fuse input features and projected contextual representation in a multiplicative way , e.g. , matrix multiplication and convolution , maintaining both low- and high-order information . For feature-based modeling , we introduce feature-based DPO where the weight-generate operation dynamically produces instance-wise filters conditioned on the embedded context . The featureaggregate function then applies instance-wise filters to the flattened input by matrix multiplication , allowing to learn multiplicative features . In particular , we further propose a new class of DPO , called field-based DPO , which is not only instance-specific but also field-specific . In that case , the filters vary from field to field and from instance to instance , allowing more complex interactions along the field dimension . For user behavior modeling , we introduce sequence-based DPO that consists of two variants : behaviorbehavior dynamic operation and query-behavior dynamic operation . A representative method of dynamic convolution ( Chen et al. , 2020 ; Yang et al. , 2019 ) shares the convolution kernel , which is generated by the global average of the inputs . Similarly , ( Wu et al. , 2019 ) proposed DyConv , a lightweight fine-grained convolution that depends only on time-step , reinforcing the encoder-based language modeling framework . However , our methods incorporate both local and global information as they jointly use locality-aware methods ( e.g. , convolution or separable convolution ) followed by a global average pooling layer to produce instance-wise weights . The query-behavior dynamic operation is specialized designed for the decoder-based framework in CTR prediction , aiming to capture target-behavior dependency . To our best knowledge , this is the first attempt to extend the business of dynamic neural networks to CTR prediction with extensive experiments on two fundamental scenarios . The comprehensive study against existing solutions validates the superiority of our proposed method . Moreover , we demonstrate that incorporating DPO into the real-world ranking system is beneficial . Our contribution can be summarized as followed : • We propose a generic formulation for capturing multiplicative interaction via weightgenerate and feature-aggregate function , termed DPO . • For feature-based modeling , we propose two variants , named field-based and feature-based DPO , offering a unifying view of implicit and explicit feature interaction . Decomposing these operations , we find they implicitly inherit low- and second-order representation . • For user behavior modeling , we propose two sequence-based variants : behavior-behavior and query-behavior DPO . The first one computes locally perceptual dynamic filters and the second one learns target-behavior dependency in a multiplicative manner . We demonstrate that such operations can benefit the self-attention layers by higher computational efficiency through modeling locality as inductive bias . • The proposed dynamic parameterized networks outperform state-of-the-art methods by a significant margin on both public and real-world production datasets . We also give a comprehensive study about the relationship of our proposed methods to previous Factorization Machine ( Rendle , 2010 ) and CrossLayer ( Wang et al. , 2017 ) . We further demonstrate the effectiveness and superiority of our method with an online A/B test in real-world applications by incorporating it into the fine-rank stage of the real-world ads system . 2 METHOD . We first review the mainstream approaches of feature-based and user behavior ( sequence-based ) modeling under the situation of CTR prediction1 . After that , we introduce DPO and provide several specific instantiations designed for traditional feature-based and sequence-based modeling . 1Related work is in Appendix A due to space limitation . 2.1 PRELIMINARY . Traditional CTR prediction methods mainly predict a probability of a user click an item , which serves as a fundamental evaluation criterion for computing advertising systems . Typically , in a given scenario ( the contexts ) , users click on certain items ( item profiles ) based on their own needs ( query ) and pautorferences ( user profiles ) . Consequently , a model considers four fields of features , i.e. , query , user profile , item profile , and contexts to predict : CTR = F ( query , user profile , item profile , contexts ) ( 1 ) where item and user profiles contain up to tens of fine-grained static attributes depending on the specific circumstances . Sequence-based CTR prediction involves user behaviors additionally : CTR = F ( query , user behavior , user profile , item profile , contexts ) ( 2 ) where the models can learn from the behaviors that have occurred under certain contexts and query in the past to make judgments on the current items . As mentioned in KFAtt ( Liu et al. , 2020 ) , the behavior module can be formulated as : v̂q = UserBehavior ( q , k1 : T , v1 : T ) , where k1 : T and v1 : T are given T historical clicked items and corresponding query words . The most used strategy is to adopt the the self-attention mechanism ( Vaswani et al. , 2017 ) , which naturally learns multiplicative interaction between query and the historical behavior . 2.2 FORMULATION . Namely , multiplicative interaction ( Jayakumar et al. , 2020 ) has been proposed to fuse two different sources of information with the goal of approximating function ftarget ( x , z ) ∈ Rc , where x and z are the input and context respectively . Similarly , we give a generic formulation of DPO in CTR prediction task as : yi = 1 C ( z ) ∑ ∀j f ( xi ; gi ( zj ; θ ) ) ( 3 ) Here i is the index of a position ( in the field , or sequence ) , whose response is calculated with the generated output of z over all existing positions . x is the input embedding , while z denotes any specified context . The generate function g aims to compute dynamic weights and bias followed as one of the inputs of the pairwise aggregate function f , which learns the interactive features reflecting the relationship between xi and zj . The output is finally normalized by a factor C ( z ) . MLP and convolution typically process input and context features in an additive way with fixed weights . While in Eqn . ( 3 ) , using instance-wise generated weights and bias from contexts z , the additive nature is transformed to multiplicative . DPO is also different from bilinear layer ( Lin et al. , 2015 ; He & Chua , 2017 ) for Eqn . ( 3 ) computes representation based on the generated weights over all positions , whereas bilinear layer aggregates information over all positions between x and z , leading to large memory consumption . Furthermore , our generated dynamic weights can maintain more local information , which complements the global counterpart , e.g. , self-attention . DPO is a flexible block and can easily work together with MLP and self-attention layers . 2.3 FEATURE-BASED DPO . Given x ∈ Rm and z ∈ Rn as inputs and context , due to the lack of position information , the generic formulation degrades as y = f ( x ; g ( z ; θ ) ) . For simplicity , we consider f in the form of a linear transformation : f ( x ; z ) =W ( z ) x , where W ( z ) is an instance-wise two-dimensional matrix generated by function g. Now , We discuss the choice of function g. Following the hypernetworks ( Ha et al. , 2016 ) , a natural choice of g is a fully-connected layer to form dynamic weights and bias : y = ( Ŵ Tz + b̂ ) T︸ ︷︷ ︸ DyWeights x+ ( Ẇ Tz + ḃ ) ︸ ︷︷ ︸ DyBias = zTŴx︸ ︷︷ ︸ explicit + Ẇ Tz + B̂Tx+ ḃ︸ ︷︷ ︸ implicit ( 4 ) where ( Ŵ , b̂ , Ẇ , ḃ ) ∈ ( Rn×mc , Rmc , Rn×c , Rc ) . However , the size of Ŵ has quadratic space complexity , unsuitable for deployment in real-world application . Here , we consider a low-rank method in practice , e.g. , a two-layer MLP : g ( z ) =W T2 σ ( W T 1 z + b1 ) + b2 ( 5 ) where ( W1 , b1 ) ∈ ( Rn×l , Rl ) and ( W2 , b2 ) ∈ ( Rl× ( mc+c ) , Rmc+c ) , σ is a non-linear function . Then , we can decompose the output into explicit dynamic weights and bias . The right inductive bias depends on how we select context z and g. We denote the complexity of f is O ( mc ) less than O ( mc+ nc ) of plain MLP layer , while g scales up to O ( lmc+ ln ) . To reduce the complexity , we set l as a small number and use a multi-head mechanism ( Vaswani et al. , 2017 ) . Relation to Cross Network : A cross layer ( Wang et al. , 2017 ) take the feature interaction formulation as xi+1 = x0 · xiwi + bi+1 + xi , where xiwi is scalar . We prove CrossLayer is the simplest formulation of DPO . Let ’ s take x0 as z , xi as x and only use 1-layer MLP as weight-generate function , whose hidden states are 1 , ( i.e . z = x0 , Ŵ ∈ Rn , Ẇ = 0 , B̂ = 1 ) . Thus , we get a scalar output of g as the same as the multiplicative term of CrossLayer . In this way , DPO aims to imitate multiplicative operation .

Summary: This submission is on modeling feature interactions for CTR prediction. It proposes a framework that follows meta-learning. Specifically, to model the interaction between feature F1 and feature F2, it uses a meta neural-network g(F1) that takes F1 and produces the parameter for another neural network f(F2) that takes F2, i.e., the outcome of feature crossing is f(F2) where f's parameter is g(F1). It outperforms the baselines and is deployed online.