TotalRecall: A Bidirectional Candidates Generation Framework for Large Scale Recommender \& Advertising Systems

Recommender ( RS ) and Advertising/Marketing Systems ( AS ) play the key roles in E-commerce companies like Amazon and Alibaba . RS needs to generate thousands of item candidates for each user ( u2i ) , while AS needs to identify thousands or even millions of high-potential users for given items so that the merchant can advertise these items efficiently with limited budget ( i2u ) . This paper proposes an elegant bidirectional candidates generation framework that can serve both purposes all together . Specifically , our framework finds its practical applications on generating millions of users for a large number of items , and the marketing scenarios of small and medium enterprises ( SMEs ) with cloud-service . Besides , our framework is also superior in these aspects : i ) . We theoretically show that our modeling objective is equivalent to the modeling objectives of the SOTA algorithms if measured by the metrics of u2i and i2u . ii ) . Our framework can easily incorporate many DNN-architectures , and achieve SOTA results with the same architecture . iii ) . We empirically show that our framework can diversify the generated candidates , and ensure fast convergence . 1 INTRODUCTION . In the Internet era , E-commerce companies usually act as a platform to connect both users and merchants . They can collect huge amount of data of users ’ behavior on merchants ’ items , and utilize the data to serve them better , for example , to build better search/recommender/advertising systems . When users browse the E-commerce websites/apps , they usually want to buy some desired items , or find something interesting , and that ’ s what the search engine and recommender system ( RS ) are built for . Besides , the merchant usually wants to mine the potential users of its items , and then starts a campaign to prompt those users to buy the items . The advertising system has the functionality to help the merchant , and we may call it the potential users mining system ( PUMS ) . In an industrial large-scale RS , there are usually two stages , the candidates generation ( CG ) stage and the ranking stage ( Covington et al. , 2016 ) . CG plays a very import role as it sets the upper bound of the recommender . A single CG method usually lacks diversity , so the industrial CG system usually consists of different subsystems to generate candidate items , for example , i ) . hot items , ii ) . items similar to users ( Covington et al. , 2016 ; Li et al. , 2019 ) , iii ) . items similar to clicked items ( Linden et al. , 2003 ) , iv ) . items clicked by similar users ( Wang et al. , 2006 ) . The PUMS has been studied and applied in marketing community for many years ( Bult & Wansbeek , 1995 ; Kim et al. , 2005 ) , and recently it is also introduced to social network to attract potential customers of companies ( Pennacchiotti & Popescu , 2011 ; Pang et al. , 2013 ) . Similarly in E-commerce companies , it can be used to discover potential customers for merchants , brands and etc . Although it is widely used , people usually treat it as a simple binary classification problem , i.e. , to predict whether a given user is the potential user or not . Models like LR , SVM , XGBoost and etc . ( Hosmer Jr et al. , 2013 ; Chang & Lin , 2011 ; Chen & Guestrin , 2016 ) are usually employed . In this paper , we unify the two systems in our TotalRecall ( TR ) framework , and apply it in both large-scale PUMS and marketing scenarios of SMEs with the cloud-service . Our main contributions are listed below : • Bidirectional Candidates Generation : TR can be trained once and then used to generate candidate items ( u2i ) and users ( i2u ) all together without any additional operations . Theoretically we show that it is due to the modeling of joint probability of u and i . • Achieving SOTA results in both RS and PUMS : We compare to two different algorithms , i.e. , Matrix Factorization ( MF ) with Bernoulli distribution and sequential modeling and MF with Multinomial distribution , and achieve comparable or better results than previous SOTA . • Equivalence of the algorithms based on Bernoulli and Multinomial distributions : To the best of our knowledge , we are the first to show that modeling the user-item interaction with Bernoulli and Multinomial distributions are equivalent . • Fast Convergence , and Diversified Candidates : TR converges much faster compared to MF models , up to 16× times . Our experiments show that TR can increase the diversity of the generated candidates . 2 RELATED WORK . 2.1 RECOMMENDER SYSTEM ( RS ) . RS has been studied for many years , and in the early years , Collaborative Filtering ( CF ) and its variants ( Su & Khoshgoftaar , 2009 ) are widely adopted . Later its descendant MF ( Funk ) is proposed to solve the problem more elegantly with higher accuracy , and Probabilistic Matrix Factorization ( PMF ) ( Mnih & Salakhutdinov , 2008 ) builds a solid theoretic foundation for MF based on probability theory . Although MF performs very well in many datasets ( Rendle et al. , 2020 ) , it is usually not adopted in industry . The modern industrial RS can involve up to billions of users and hundreds of millions of items , and consist of multiple stages , e.g. , CG and ranking stages ( Covington et al. , 2016 ) . Various deep neural networks are adopted to tackle the complicated system , for example , in the CG stage , the problem is formulated as the multi-class classification , and users ’ and items ’ representations are detached with the two-tower architecture to balance the efficiency and accuracy in online serving ( Covington et al. , 2016 ; Li et al. , 2019 ; Cen et al. , 2020 ) . In the ranking stage , the problem is formulated as the binary classification , and various models has been developed : wide & deep model ( Cheng et al. , 2016 ) captures explicit and implicit feature crossing , Deep Interest Network ( Zhou et al. , 2018 ) and Deep User Perception Network ( Ni et al. , 2018 ) utilize attention mechanism ( Bahdanau et al . ( 2014 ) ) to capture users ’ short-term interests . 2.2 POTENTIAL USERS MINING SYSTEM ( PUMS ) . PUMS mines the potential users for given items . The ‘ item ’ could be anything that users can interact with , for example , an insurance product ( Kim & Street , 2004 ) , a company/business ( Pennacchiotti & Popescu , 2011 ; Pang et al. , 2013 ; Lo et al. , 2016 ) , a specific information ( e.g. , tweet ) in social medias ( Tang et al. , 2015 ; Gui et al. , 2019 ) and even another user ( Guy , 2018 ) . These problems are usually treated as binary classification and solved with tools like SVM , LR and etc . for each individual item independently ( Hosmer Jr et al. , 2013 ; Chang & Lin , 2011 ; Chen & Guestrin , 2016 ) . In the E-commerce company like Amazon , the ‘ item ’ could be a product , a brand , a product category , a merchant and etc , and the number of the ‘ items ’ range from thousands to hundreds of millions . It will be impractical to model each item individually , so we usually model the items all together using binary classification similar to Rendle et al . ( 2020 ) , which will be stated in detail later in Sec . 5 . 2.3 SAMPLED-SOFTMAX AND INFONCE LOSS . In the CG stage of RS , sampled-softmax ( SSM ) loss ( Jean et al. , 2015 ) is widely used with different background distributions , such as log-uniform distribution ( Powers , 1998 ) , empirical marginal distribution ( Yi et al. , 2019 ) and mixed distribution ( Yang et al. , 2020 ) . In contrastive learning , InfoNCE loss is proposed in Oord et al . ( 2018 ) to learn representations through self-supervision . Later it is widely adopted in CV ( Chen et al . ( 2020 ) ) and NLP ( Gao et al . ( 2021 ) ) and performs very well . InfoNCE and SSM losses are both trying to solve multi-class classification of very large vocabulary . They both use negative sampling and can be written in the same analytical form : l = 1 |D| ∑ ( x , y ) ∈D − log exp ( fϑ ( x , y ) − log pn ( y|x ) ) exp ( fϑ ( x , y ) − log pn ( y|x ) ) + ∑ y′∈Yx exp ( fϑ ( x , y ′ ) − log pn ( y′|x ) ) , ( 1 ) where ( x , y ) ∈ D is the training data , and usually x is the input features and y is the class , fϑ ( x , y ) is the output of the neural network parametrized by ϑ , and Yx contains hundreds or thousands of negative samples from background distribution pn ( y|x ) . The main differences are that the pn ( y|x ) in SSM can be arbitrarily chosen , while in InfoNCE it is the empirical marginal distribution p̂data ( y ) . Recently , InfoNCE loss is exploited in RS to optimize p ( y|x ) p ( y ) instead of p ( y|x ) for the debiasing purpose , and it can be easily implemented with in-batch negative sampling and features of y can be fully utilized ( Zhou et al. , 2021 ) . 3 METHOD . 3.1 PROBLEM DEFINITION . In E-commerce companies , when a user u clicked an item i at time t , a record ( u , i , t ) was logged as the implicit feedback . Given the raw logs { ( u , i , t ) } , there are usually two ways to process the data : one is to directly create a user-item interaction matrix Sui with the value su , i = 1 if the ( u , i ) pair appears in the log ( Su & Khoshgoftaar ( 2009 ) ) ; the other is to formulate the problem as next-item prediction , i.e. , from the raw log we create a dataset D = { ( xu , t , yu , t ) : t ∈ { 1 , 2 , ... , Tu } |u ∈ { 1 , 2 , ... , N } } , where xu , t = { yu,1 : ( t−1 ) } represents u ’ s clicks prior to t-th click yu , t and Tu is the number of clicks of u ( Covington et al . ( 2016 ) ) . We define the click sequence xu , t as the pseudo-user , so we can create a pseudo-user-item interaction matrix from D , and all possible click sequences form the pseudo-user set . Therefore , they both deal with the 2D interaction matrix Sui , irrespective of whether the u is a real user or pseudo-user or not . In the remaining of the paper , we will not distinguish between user and pseudo-user , and only focus on solving the Sui . Sui consists of all users as the row and all items as the column , and its entries are either 1 or unknown . Let ’ s assume all users form the set U = { u1 , u2 , ... , uM } , and all items form the set I = { i1 , i2 , ... , iK } . Given a user u ∈ U , we will generate candidate items from the item pool I ( RS , u2i ) , while given an item i ∈ I , we shall find out the potential users out of all users U ( PUMS , i2u ) . The candidates generation of RS and PUMS are of the opposite direction , and traditionally they are solved separately . But essentially they are both trying to estimate some kind of probabilities of the unknown entries in Sui , so we propose to model them jointly .

Summary: This paper jointly considers the problems of recommender systems (RS) and advertising systems (AS) as it is a coin’s two sides. The key is to model the joint probability of (user, item) instead of conditional probabilities (user | item) and (item | user) independently. The proposed model adapts an existing large-vocabulary loss function to the optimization objective. Experiments on real-world datasets are conducted to compare with both Matrix Factorization and Sequential Modeling methods.