Zero-Shot Recommender Systems

Performance of recommender systems ( RecSys ) relies heavily on the amount of training data available . This poses a chicken-and-egg problem for early-stage products , whose amount of data , in turn , relies on the performance of their RecSys . In this paper , we explore the possibility of zero-shot learning in RecSys , to enable generalization from an old dataset to an entirely new dataset . We develop an algorithm , dubbed ZEro-Shot Recommenders ( ZESREC ) , that is trained on an old dataset and generalize to a new one where there are neither overlapping users nor overlapping items , a setting that contrasts typical cross-domain RecSys that has either overlapping users or items . Different from previous methods that use categorical item indices ( i.e. , item ID ) , ZESREC uses items ’ generic features , such as natural-language descriptions , product images , and videos , as their continuous indices , and therefore naturally generalizes to any unseen items . In terms of users , ZESREC builds upon recent advances on sequential RecSys to represent users using their interactions with items , thereby generalizing to unseen users as well . We study three pairs of real-world RecSys datasets and demonstrate that ZESREC can successfully enable recommendations in such a zero-shot setting , opening up new opportunities for resolving the chicken-and-egg problem for data-scarce startups or early-stage products . 1 INTRODUCTION . Many large scale e-commerce platforms ( such as Etsy , Overstock , etc . ) and online content platforms ( such as Spotify , Overstock , Disney+ , Netflix , etc ) have such a large inventory of items that showcasing all of them in front of their users is simply not practical . In particular , in the online content category of businesses , it is often seen that users of their service do not have a crisp intent in mind unlike in the retail shopping experience where the users often have a clear intent of purchasing something . The need for personalized recommendations therefore arises from the fact that not only it is impractical to show all the items in the catalogue but often times users of such services need help discovering the next best thing — be it the new and exciting movie or be it a new music album or even a piece of merchandise that they may want to consider for future buying if not immediately . Modern personalized recommendation models of users and items have often relied on the idea of extrapolating preferences from similar users . Different machine learning models define the notion of similarity differently . Classical bi-linear Matrix Factorization ( MF ) approaches model users and items via their identifiers and represent them as vectors in the latent space [ 13 ; 28 ] . Modern deep-learningbased recommender systems [ 34 ; 12 ; 25 ] , which are also used for predicting top-k items given an item , learn the user-to-item propensities from large amounts of training data containing many ( user , item ) tuples , optionally with available item content information ( e.g. , product descriptions ) and user metadata . As machine learning models , the performance of RecSys relies heavily on the amount of training data available . This might be feasible for large e-commerce or content delivery websites such as Overstock and Netflix , but poses a serious chicken-and-egg problem for small startups , whose amount of data , in turn , relies on the performance of their RecSys . On the other hand , zero-shot learning promises some degree of generalization from an old dataset to an entirely new dataset . In this paper , we explore the possibility of zero-shot learning in RecSys . We develop an algorithm , dubbed ZEro-Shot Recommenders ( ZESREC ) , that is trained on an old dataset and generalize to a new one where there are neither overlapping users nor overlapping items , a setting that contrasts typical cross-domain RecSys that has either overlapping users or items [ 40 ; 38 ; 3 ; 16 ] . Naturally , generalization of RecSys to unseen users and unseen items becomes the two major challenges for developing zero-shot RecSys . For the first challenge on unseen users , we build on a rich body of literature on sequential recommendation models [ 12 ; 27 ; 25 ; 14 ; 17 ; 21 ] . These models are built with sequential structure to encode temporal ordering of items in user ’ s item interaction history . Representing users by the items they have consumed in the past allows the model to extrapolate the preference learning to even novel users who the model did not see during training , as long as the items these unseen users have interacted with have been seen during training . However , such deep learning models encode item via its categorical item index , i.e. , the item ID , and therefore fall short in predicting a likely relevant but brand-new item not previously seen during training . This brings us to the second challenge of developing zero-shot recommender systems , i.e. , dealing with unseen items . To address this challenge , ZESREC goes beyond traditional categorical item indices and uses items ’ generic features such as natural-language descriptions , product images , and videos as their continuous indices , thereby naturally generalizing to any unseen items . Take natural-language ( NL ) descriptions as an example . One can think of NL descriptions as a system of universal identifiers that indexes items from arbitrary domains . Therefore as long as one model is trained on a dataset with NL descriptions , it can generalize to a completely different dataset with a similar NL vocabulary . In ZESREC we build on state-of-the-art pretrained NL embedding models such as BERT [ 8 ] to extract NL embeddings from raw NL descriptions , leading to an item ID system in the continuous space that is generalizable across arbitrary domains . For instance , in e-commerce platform , one could use items ’ description text ; and similarly in the online content platforms , one could use movie synopsis or music track descriptions to represent an item . Combining the merits of sequential RecSys and the idea of universal continuous ID space , our ZESREC successfully enables recommendation in an extreme cold-start setting , i.e. , the zero-shot setting where all users and items in the target domain are unseen during training . Essentially ZESREC tries to learn transferable user behavioral patterns in a universal continuous embedding space . For example , in the source domain , ZESREC can learn that if users purchase snacks or drinks ( e.g. , ‘ Vita Coconut Water ’ with a lemonade flavor ) that they like , they may purchase similar snacks or drinks with different flavors ( e.g. , ‘ Vita Coconut Water ’ with a pineapple flavor ) , as shown in the case study . Later in the target domain , if one user purchase ‘ V8 Splash ’ with a tropical flavor , ZESREC can recommend ‘ V8 Splash ’ with a berry flavor to the user ( see Fig . 4 of Sec . 4.6 for details ) . Such generalization is possible due to the use of the NL descriptions as universal identifiers , based on which ZESREC could easily identify similar products of the same brand with different flavors . To summarize our contributions : • We identify the problem of zero-shot recommender systems and propose ZESREC as the first hierarchical Bayesian model for addressing this problem . • We introduce the notion of universal continuous identifiers that makes recommendation in a zero-shot setting possible . • We provide empirical results which show that ZESREC can successfully recommend items in the zero-shot setting . • We conduct case studies demonstrating that ZESREC can learn interpretable user behavioral patterns that can generalize across datasets . 2 RELATED WORK . Deep Learning for RecSys . Deep learning has been prevalent in modern recommender systems [ 29 ; 33 ; 34 ; 19 ; 4 ; 10 ; 32 ] due to its scalability and superior performance . As a pioneer work , [ 29 ] uses restricted Boltzmann machine ( RBM ) to perform collaborative filtering in recommender systems , however the system is a single-layer RBM . Later , [ 34 ] and [ 19 ] build upon Bayesian deep learning to develop hierarchical Bayesian models that tightly integrate content information and user-item rating information , thereby significantly improving recommendation performance . After that , there are also various proposed sequential ( or session-based ) recommender systems [ 12 ; 27 ; 2 ; 18 ; 22 ; 39 ; 14 ; 31 ; 41 ; 25 ] , GRU4Rec [ 12 ] was first proposed to use gated recurrent units ( GRU ) [ 6 ] for recommender systems . Since then , follow-up works such as hierarchical GRU [ 27 ] , temporal convolutional networks ( TCN ) [ 2 ] , and hierarchical RNN ( HRNN ) [ 25 ] have achieved improvement in terms of accuracy by utilizing cross-session information [ 27 ] , causal convolutions [ 2 ] , as well as meta data and control signals [ 25 ] . Another line of work focusing on building self-attention based sequential models such as SASRec [ 14 ] , BERT4Rec [ 31 ] , and S3Rec [ 41 ] . In this paper we build on such sequential RecSys and note that our ZESREC is model agnostic , that is , it is compatible with any sequential RecSys . Cross-Domain and Cold-Start RecSys . There is a rich literature on cross-domain RecSys focusing on training a recommender system in the source domain and deploying it in the target domain where there exist either common users or items [ 40 ; 38 ; 3 ; 16 ] . These works are also related to the problem of recommendation for cold-start users and items , i.e. , users and items with few interactions ( or ratings ) available during training [ 11 ; 20 ; 42 ; 23 ] . There are also works [ 24 ; 9 ] handling cold start on both user and item with meta-learning , however they can not generalize across domains . In summary , prior systems are either ( 1 ) not sufficient to address our zero-shot setting where there are neither common users nor common items in the target domain or ( 2 ) unable to learn user behavior patterns that are transferable across datasets/domains . Therefore , they are not applicable to our problem of zero-shot recommendations . 3 ZERO-SHOT RECOMMENDER SYSTEMS . In this section we introduce our ZESREC which is compatible with any sequential model . Without loss of generality , here we focus on NL descriptions as a possible instantiation of universal identifiers , but note that our method is general enough to use as identifiers other content information such as items ’ images and videos . We leave exploration for other potential modalities to future work . Notation . We focus on the setting of zero-shot recommendation where there are neither overlapping users nor overlapping items between a source domain and a target domain . We assume a set Vs of Js items and a set Us of Is users in the source domain , as well as a set Vt of Jt items and a set Ut of It users in the target domain . We let I = Is + It and J = Js + Jt ; we use j ∈ Vs ∪ Vt to index items and i ∈ Us ∪ Ut to index users . The zero-shot setting dictates that Vs ∩ Vt = ∅ and that Us ∩ Ut = ∅ . We denote the collection of all users ’ interactions as a 3D tensor ( with necessary zero-padding ) R ∈ RI×Nmax×J , where Nmax is the maximum number of interactions among all users . We use the subscript ‘ ∗ ’ to represent the collection of all elements in a certain dimension . Specifically , each user i has a sequence of Ni interactions ( e.g. , purchase history ) with various items denoted as Ri∗∗ = [ Rit∗ ] Nit=1 , where Rit∗ ∈ { 0 , 1 } J is one-hot vector denoting the t-th item user i interacted with . The same user i has different user embeddings at different time t , reflecting dynamics in user interests ; here we denote as uit ∈ RD the latent user vector when user i interacts with the t-th item in her history , and we use U = [ uit ] I , Nmax i=1 , t=1 ∈ RI×Nmax×D ( with necessary zero-padding ) to denote the collection of user latent vectors . We denote as vj ∈ RD the item latent vector and V = [ vj ] J j=1RJ×D as the collection . For simplicity and without loss of generality , in this paper we focus on using NL descriptions ( i.e. , a sequence of words ) to describe items . For item j we denote its NL description as xj , and the number of words as Mj ; similar to V , we let X = [ xj ] Jj=1 . With slight notation overload on t , we denote as R ( s ) and R ( t ) the sub-tensor of R that corresponds to the source and target domains , respectively . Similarly , we also have U ( s ) , U ( t ) , V ( s ) , V ( t ) , X ( s ) , and X ( t ) . Problem Setup . A model is trained using all users ’ interaction sequences from the source domain , i.e. , { Ri∗∗ } i∈Us , and then deployed to recommend items for any user ι ∈ Ut in the target domain , given user ι ’ s previous history Rι∗∗ , which can be empty . In practice we append a dummy item at the beginning of each user session , so that during inference we could conduct recommend even for users without any history by ingesting the dummy item as context to infer the user latent vector . In our zero-shot setting , the model is not allowed to fine-tune or retrain on any data from the target domain . Definition of Zero-shot Learning in RecSys . Our zero-shot setting includes three unique properties : ( 1 ) cold users , ( 2 ) cold items , and ( 3 ) domain gap . It is fundamentally different from previous content-based cold start as the latter setting usually satisfies either ( 1 ) or ( 2 ) , but not often both .

Summary: A great recommender system relies on great training set. However, at the beginning, there is no such data availability. This paper tries to solve the zero-shot recommendation problem where there is no user or item overlaps. The two challenges are generalize to unseen users and to unseen items. For unseens users, sequential recommendation represents users as a sequence of their interacted items. As long as the items are seen before, the users can be represented. As for items, the unique ids are not useful. However, the attributes such as natural language description can be universal. This paper proposed an approach based on hierarchical Bayesian model. The item universal embedding is using pre-trained BERT network with a single layer neural network. Extensive experiments are carried out to demonstrate the effectiveness of the proposed approach.