From Intervention to Domain Transportation: A Novel Perspective to Optimize Recommendation

1 INTRODUCTION . For information retrieval ( IR ) systems , the users ’ willingness to interact is often intervened by the content we show . Recommendations not only impact the potential response of the users , but also change the nature of the data collected for training machine learning models ( Bottou et al. , 2013 ) . The interventional nature of recommendation , as well as the fact that many questions in IR are counterfactual , e.g. , `` what the response would have been if we recommended something else '' , makes it natural to rephrase recommendation in the context of causal inference or missing data problems ( Rubin , 2005 ; Little et al. , 2002 ) . Among the two disciplines , technical tools of importance weighting ( IW ) and domain adaptation ( DA ) have been commonly applied . Recently , there has been widespread interest in adapting those tools for learning the best recommendation ( Schnabel et al. , 2016 ; Bonner and Vasile , 2018 ) , but we find a critical discussion missing from most existing literature in this direction . In many scientific disciplines , the effectiveness of IW and DA relies heavily on overlapping between source and target domain ( Austin and Stuart , 2015 ; David et al. , 2010 ) . The underlying reason also happens to be the motivation of exploration-exploitation techniques including bandits and reinforcement learning ( Auer et al. , 2002 ; Sutton and Barto , 1998 ) : in the less-explored regions where few observations are collected , the confidence of selecting any hypothesis is discounted . Unfortunately , modern IR systems are often cautious about exposing users to less relevant content for the sake of immediate satisfaction and revenue , even with the help of bandits or RL ( Bonner and Vasile , 2018 ) . In particular , many real-world recommenders are deterministic , e.g. , an item is shown with probability either zero or one , which further limits the exploration and thus the coverage of collected data . In Section 3 , we rigorously prove for both IW- and DA-based learning the hardness or impossibility results caused by insufficient overlap . Intuitively , when a less-explored instance is passed to a candidate hypothesis , even if DA or IW can improve the accuracy of the prediction , they inevitably increase the uncertainty of that prediction as a consequence of the bias-variance tradeoff . To a certain point , the increased uncertainty from having too many instances in the weakly overlapped regions will fundamentally limit what any learning procedure can achieve . Further , if the instance lies in the non-overlapped region , we can barely exceed regular extrapolation1 ( Xu et al. , 2021b ) . We resolve the fundamental limitation of insufficient overlapping in IR with the key insight that the interventional impact of recommendation is not exclusive to observed data , but also interferes with the target domain of interest — the instances that will be affected by the recommendations we are about to make . For instance , the placement of a recommendation on the webpage proposed to the user directly influences the occurrence of clicks . Therefore , we alternatively consider optimizing recommendation as searching for an intervention that best transports the patterns it learns from the source domain to its intervention domain . The procedure is best described by transport instead of transfer because it will interfere with how the learnt patterns are carried to the target domain . This change of view describes a complex but realistic working mechanism that involves learning , intervention , and transportation , as depicted in Figure 1 . On the one hand , we wish to learn patterns that can be transported to the target domain of interest . On the other hand , the patterns we identify lead to the intervention ( in the form of recommendation ) that generates the target domain . Following this working mechanism , we impose constraints on the domain transportation between the ( reweighted ) source domain and the future intervention domain . We point out that reweighting is essential for learning from interventional feedback data ( Rosenbaum and Rubin , 1983 ) , especially for the working mechanism we identify here because the relationship between the source and target domain is more involved . Towards this end , we propose a novel solution via transportation-constraint risk minimization . We rigorously investigate for our objective : • the consistency , generalization , and excessive risk bounds for the theoretical insights and guarantees ; • equivalence to a two-player minimax game efficiently solved by gradient descent-ascent algorithm ; • convergence to the standard IW-based solution for the ideal fully-overlapped bandit feedback ; We demonstrate the empirical performance of our method via comprehensive experiments via simulation , real-data analysis , as well as online experimentation and testing in a real-world IR system . 2 RELATED WORK . Bottou et al . ( 2013 ) first reveals the importance of characterizing the working mechanism of IR systems . However , they study specifically the advertising systems where the underlying causal structures are assumed given . Our work significantly extends their discussion by moving beyond causal mechanisms . Issues caused by ignoring the interventional nature of recommendation have been pointed out for numerous scenarios ( Chen et al. , 2020 ) . Steck ( 2010 ) and Liang et al . ( 2016 ) address optimizing recommendation in the context of missing data and causal inference , and the majority of recent follow-up work investigate particularly inverse propensity weighting ( IPW ) and learning domain invariant representation methods ( Chen et al. , 2020 ; Saito , 2020 ; Yang et al. , 2018 ; Joachims et al. , 2017 ; Bonner and Vasile , 2018 ; Agarwal et al. , 2019 ; Xu et al. , 2020 ) . As mentioned in Section 1 , they largely ignore the overlapping issue even when the recommendation is deterministic . Our theoretical results on the interplay between overlapping and learning performance are novel and add to the current venue of revealing the tradeoff for IW- and DA-based learning ( Byrd and Lipton , 2019 ; Johansson et al. , 2019 ; Ben-David and Urner , 2012 ) . See Appendix D for further discussions . Existing solutions for insufficient overlapping focus primarily on reducing or calibrating the variance , e.g. , as discussed in Swaminathan and Joachims ( 2015b ) ; Wang et al . ( 2020 ) ; Johansson et al . 1The presumption for this argument is that the underlying causal mechanism is unknown , which applies to the settings we discuss in the paper . ( 2020 ) . They do not account for the unique working mechanism of IR where the domain discrepancy can be actively controlled . We devise the domain-transportation constraint using the Wasserstein distance originated from the optimal transport theory ( Courty et al. , 2016 ; Redko et al. , 2017 ) . The idea of converting a Wasserstein-constraint objective to two-player minimax game is similar to that of the generative adversarial network ( Gulrajani et al. , 2017 ) , however , we do not make any generative assumption on the observed data . Finally , learning from bandit feedback and the associated counterfactual risk minimization can be viewed as a special case of our framework ( Swaminathan and Joachims , 2015a ) . We will use it as a test bed to examine the guarantees of the proposed approach . 3 PRELIMINARIES . We use upper-case letters to denote random variables and measures , and bold-font letters for vectors and matrices . We denote by ∥ · ∥2 , ∥ · ∥∞ and ∥ · ∥L the ℓ2 , ℓ∞ norm and the Lipschitz constant of a function . For clarity , we borrow the item recommendation setting to illustrate the IR system we discuss next . The set of all users and items are given by U and I . To be concise , we denote the features ( or pre-trained embeddings ) for users and items by xu for u ∈ U and xi for i ∈ I . We study primarily the practical implicit-feedback setting with Yui ∈ { 0 , 1 } , e.g. , users express preferences implicitly via clicks . Let D be the population of all possible ( xu , xi ) or ( u , i ) depending on the context . The embeddings can also be free parameters under the representation mapping f from X or U ∪ I to Rd . We let P be any base probability measure supported on the subsets of D , and let Pn be the empirical versions according to the collected feedback of Dn where n is the sample size . We assume the weight of a user-item is positive and bounded by a constant M . Equipped with either individual weights wui ∈ [ 0 , M ] or weighting function w : U × I ( or X × X ) → [ 0 , M ] , the reweighted measures are given by such as : Pw ( u , i ) ∝ w ( u , i ) · P ( u , i ) . The candidate recommendation algorithm f ∈ F outputs a score or probability for each user-item pair . Top-k recommender often ranks the output and then renders the top-k items to the user . We denote this procedure by the function : reco ( u , i ; f ) ∈ { 0 , 1 } . Define the new measure onD induced by deploying f on D as : Pf ( u , i ) ∝ reco ( u , i ; f ) · P ( u , i ) . If we choose the base measure P as uniform , it is clear that Pf is a uniform measure on the ( u , i ) pairs that will be exposed : { ( u , i ) ∈ D : reco ( u , i ; f ) = 1 } . Finally , given a loss function ℓ for f , we define the corresponding risk under P : EPR ( f ) = E ( u , i ) ∼P ℓ ( f ( u , i ) , yui ) , the weighted risk under Pw : EPwR ( f ) = E ( u , i ) ∼P ℓ ( f ( u , i ) , yui ) · w ( u , i ) . Our setting covers a wide range of recommendation problems from the classical collaborative filtering to sequential recommendation ( when user embedding is a function of the item sequence ) , as well as those using bandits for exploration where f ( u , i ) becomes the exploration probability , and the base measure is given by the inverse propensities . We defer the detail of that setting to Appendix A.1 . Consequence of Insufficient Overlapping . We now rigorously justify our previous intuition that insufficient overlapping causes uncertainty that fundamentally limits what IW- and DA-based learning can achieve . While the results we show for IW are universal , i.e. , it applies to any hypothesis class , we discuss particularly for DA the skip-gram negative sampling ( SGNS ) algorithm ( Mikolov et al. , 2013 ) . The reason is that SGNS is a widespread technique for learning user or item embedding in IR systems ( Xu et al. , 2021a ) , and its more straightforward formulation will allow us to present concise theoretical results to the readers . As a gentle introduction , IW-based learning uses the ratio Q ( u , i ) /P ( u , i ) – where P and Q represent the source and target domain – to reweigh the loss of each instance in order to correct the distribution shift . On the other hand , DA-based learning assumes there exists some hypothesis that makes the optimal joint risk of inff∈F [ EQR ( f ) + EPR ( f ) ] small . For IW , we show the hardness of learning , i.e. , the best target risk inff EQR ( f ) can be uncontrolled for any F due to insufficient overlapping . For DA , we show the impossibility of learning , i.e. , there does not exist a solution of SGNS that can make the optimal joint risk small when P and Q are not aligned . Proposition 1 . Let P and Q be the training and target distribution supported on SP , SQ ⊆ D , and w ( u , i ) = Q ( u , i ) P ( u , i ) , for ( u , i ) ∈ SP ∩ SQ . Dn consists of training instances sampled i.i.d from P . Hardness of learning with IW : if SP = SQ , for any f ∈ F , with probability at least 1− δ , it holds : EQR ( f ) ≲ EPn , wR ( f ) + M ( log 1δ + logN∞ ( 1 n , F ) n + √ Md1 ( P∥Q ) ( log 1δ + logN∞ ( 1 n , F ) n , and P ( ∣∣EQR ( f ) − EPn , wR ( f ) ∣∣ ≳ √ ( d1 ( P∥Q ) − 1 ) /n ) > 0 under mild condition , where M : = max { w ( u , i ) } , N∞ ( 1n , F ) : = N ( 1n , F , ℓ 2n ∞ ) is the 1 n -covering number for F in ∥ · ∥∞ based on 2n i.i.d samples from P , and d1 ( P∥Q ) = ∫ SQ ( dP/dQ ) dP is a divergence measure . Further , if SP ̸= SQ , for any prior and posterior distributions π and τ on F , for risks associated with the 0-1 loss , it holds with probability at least 1− δ : Ef∼τ , QR ( f ) ≲ Ef∼τ , Pn , wR ( f ) + dKL ( π∥τ ) + log d1 ( P∥Q ) δ n + Ef∼τ , SQ\SPR ( f ) +O ( Dn ( τ ) ) , where dKL is the Kullback-Leibler divergence , d1 ( P∥Q ) is the same as above but defined only on the overlapped part , and Dn ( τ ) consists of disagreement terms detailed in Appendix A.2 . Impossibility of learning for DA with SGNS : let R ( f ) be the associated risk for the SGNS algorithm , and Cu , i ( P ) , Cu , i ( Q ) be the co-occurrence statistics under P , Q , as mentioned in Xu et al . ( 2021a ) . Even if SP = SQ , it holds that : inf f [ EPR ( f ) + EQR ( f ) ] ≥ 1 |SP | ∑ ( u , i ) ∈SP dKL ( Cu , i ( P ) ∥Cu , i ( Q ) ) + c , where c takes some positive value and equals to inff EPR ( f ) + inff EQR ( f ) when P = Q a.s. We defer all the proofs to Appendix A.2 . For IW , we reveal the generalization bounds for both the frequentist setting ( the first set of bounds ) and the PAC-Bayesian setting ( the second bound ) . While they emphasize conceptually on different learning guarantees ( Germain et al. , 2009 ) , and while the latter can handle the existence of non-overlap ( SP ̸= SQ ) , we find them both showing insufficient overlapping ( characterized by the discrepancy terms d1 and dKL ) can cause arbitrarily large gaps leading to poor generalizations to the target domain . For DA with SGNS , our lower bound reveals the impact of insufficient overlapping indirectly via a notion of co-occurrence statistics defined in Xu et al . ( 2021a ) . In particular , if the source and target domains are well-aligned in terms of the co-occurrence statistics , the increased divergence term can make it impossible to find a hypothesis that simultaneously performs well in both domains . Although this result holds only for SGNS , it nevertheless suggests the limitations of DA caused by insufficient overlapping in IR .

Summary: The authors propose a novel way to learn new recommendation deterministic policies from feedback collected under different recommendation policies. The authors start by pointing out that the majority of current approaches, either Counterfactual (Inverse Propensity Weighting) either Domain Adaptation-based,  do not work well when the support of the current and future recommendation policies do not fully overlap (one of the main causes being that the current recommendation is deterministic and therefore not exploring all actions). To this end, the authors propose a novel way of re-weighting and regularizing empirical risk that encourages the search for policies that have better reward (lower risk) and are still overlapping in evidence with the current policy. To escape the support issue that is present in the existing approaches, the authors switch the regularization parameter to an IPM metric (WASSERSTEIN) that can handle non-overlapping supports.  The authors provide both theoretical and empirical evidence that the resulting algorithm outperforms existing SOTA methods.