Year: 2021

Abstract:
Performance of recommender systems (RecSys) relies heavily on the amount of training data available. This poses a chicken-and-egg problem for early-stage products, whose amount of data, in turn, relies on the performance of their RecSys. In this paper, we explore the possibility of zero-shot learning in RecSys, to enable generalization from an old dataset to an entirely new dataset. We develop an algorithm, dubbed ZEro-Shot Recommenders (ZESREC), that is trained on an old dataset and generalize to a new one where there are neither overlapping users nor overlapping items, a setting that contrasts typical cross-domain RecSys that has either overlapping users or items. Different from previous methods that use categorical item indices (i.e., item ID), ZESREC uses items’ generic features, such as natural-language descriptions, product images, and videos, as their continuous indices, and therefore naturally generalizes to any unseen items. In terms of users, ZESREC builds upon recent advances on sequential RecSys to represent users using their interactions with items, thereby generalizing to unseen users as well. We study three pairs of real-world RecSys datasets and demonstrate that ZESREC can successfully enable recommendations in such a zero-shot setting, opening up new opportunities for resolving the chicken-and-egg problem for data-scarce startups or early-stage products.

text: Performance of recommender systems (RecSys) relies heavily on the amount of training data available. This poses a chicken-and-egg problem for early-stage products, whose amount of data, in turn, relies on the performance of their RecSys. In this paper, we explore the possibility of zero-shot learning in RecSys, to enable generalization from an old dataset to an entirely new dataset. We develop an algorithm, dubbed ZEro-Shot Recommenders (ZESREC), that is trained on an old dataset and generalize to a new one where there are neither overlapping users nor overlapping items, a setting that contrasts typical cross-domain RecSys that has either overlapping users or items. Different from previous methods that use categorical item indices (i.e., item ID), ZESREC uses items’ generic features, such as natural-language descriptions, product images, and videos, as their continuous indices, and therefore naturally generalizes to any unseen items. In terms of users, ZESREC builds upon recent advances on sequential RecSys to represent users using their interactions with items, thereby generalizing to unseen users as well. We study three pairs of real-world RecSys datasets and demonstrate that ZESREC can successfully enable recommendations in such a zero-shot setting, opening up new opportunities for resolving the chicken-and-egg problem for data-scarce startups or early-stage products.

heading: 1 INTRODUCTION
text: Many large scale e-commerce platforms (such as Etsy, Overstock, etc.) and online content platforms (such as Spotify, Overstock, Disney+, Netflix, etc) have such a large inventory of items that showcasing all of them in front of their users is simply not practical. In particular, in the online content category of businesses, it is often seen that users of their service do not have a crisp intent in mind unlike in the retail shopping experience where the users often have a clear intent of purchasing something. The need for personalized recommendations therefore arises from the fact that not only it is impractical to show all the items in the catalogue but often times users of such services need help discovering the next best thing — be it the new and exciting movie or be it a new music album or even a piece of merchandise that they may want to consider for future buying if not immediately.
Modern personalized recommendation models of users and items have often relied on the idea of extrapolating preferences from similar users. Different machine learning models define the notion of similarity differently. Classical bi-linear Matrix Factorization (MF) approaches model users and items via their identifiers and represent them as vectors in the latent space [13; 28]. Modern deep-learningbased recommender systems [34; 12; 25], which are also used for predicting top-k items given an item, learn the user-to-item propensities from large amounts of training data containing many (user, item) tuples, optionally with available item content information (e.g., product descriptions) and user metadata.
As machine learning models, the performance of RecSys relies heavily on the amount of training data available. This might be feasible for large e-commerce or content delivery websites such as Overstock and Netflix, but poses a serious chicken-and-egg problem for small startups, whose amount of data, in turn, relies on the performance of their RecSys. On the other hand, zero-shot learning promises some degree of generalization from an old dataset to an entirely new dataset. In this paper, we explore the possibility of zero-shot learning in RecSys. We develop an algorithm, dubbed ZEro-Shot Recommenders (ZESREC), that is trained on an old dataset and generalize to a new one where there are neither overlapping users nor overlapping items, a setting that contrasts typical cross-domain
RecSys that has either overlapping users or items [40; 38; 3; 16]. Naturally, generalization of RecSys to unseen users and unseen items becomes the two major challenges for developing zero-shot RecSys.
For the first challenge on unseen users, we build on a rich body of literature on sequential recommendation models [12; 27; 25; 14; 17; 21]. These models are built with sequential structure to encode temporal ordering of items in user’s item interaction history. Representing users by the items they have consumed in the past allows the model to extrapolate the preference learning to even novel users who the model did not see during training, as long as the items these unseen users have interacted with have been seen during training. However, such deep learning models encode item via its categorical item index, i.e., the item ID, and therefore fall short in predicting a likely relevant but brand-new item not previously seen during training.
This brings us to the second challenge of developing zero-shot recommender systems, i.e., dealing with unseen items. To address this challenge, ZESREC goes beyond traditional categorical item indices and uses items’ generic features such as natural-language descriptions, product images, and videos as their continuous indices, thereby naturally generalizing to any unseen items. Take natural-language (NL) descriptions as an example. One can think of NL descriptions as a system of universal identifiers that indexes items from arbitrary domains. Therefore as long as one model is trained on a dataset with NL descriptions, it can generalize to a completely different dataset with a similar NL vocabulary. In ZESREC we build on state-of-the-art pretrained NL embedding models such as BERT [8] to extract NL embeddings from raw NL descriptions, leading to an item ID system in the continuous space that is generalizable across arbitrary domains. For instance, in e-commerce platform, one could use items’ description text; and similarly in the online content platforms, one could use movie synopsis or music track descriptions to represent an item.
Combining the merits of sequential RecSys and the idea of universal continuous ID space, our ZESREC successfully enables recommendation in an extreme cold-start setting, i.e., the zero-shot setting where all users and items in the target domain are unseen during training. Essentially ZESREC tries to learn transferable user behavioral patterns in a universal continuous embedding space. For example, in the source domain, ZESREC can learn that if users purchase snacks or drinks (e.g., ‘Vita Coconut Water’ with a lemonade flavor) that they like, they may purchase similar snacks or drinks with different flavors (e.g., ‘Vita Coconut Water’ with a pineapple flavor), as shown in the case study. Later in the target domain, if one user purchase ‘V8 Splash’ with a tropical flavor, ZESREC can recommend ‘V8 Splash’ with a berry flavor to the user (see Fig. 4 of Sec. 4.6 for details). Such generalization is possible due to the use of the NL descriptions as universal identifiers, based on which ZESREC could easily identify similar products of the same brand with different flavors. To summarize our contributions:
• We identify the problem of zero-shot recommender systems and propose ZESREC as the first hierarchical Bayesian model for addressing this problem. • We introduce the notion of universal continuous identifiers that makes recommendation in a zero-shot setting possible. • We provide empirical results which show that ZESREC can successfully recommend items in the zero-shot setting. • We conduct case studies demonstrating that ZESREC can learn interpretable user behavioral patterns that can generalize across datasets.

heading: 2 RELATED WORK
text: Deep Learning for RecSys. Deep learning has been prevalent in modern recommender systems [29; 33; 34; 19; 4; 10; 32] due to its scalability and superior performance. As a pioneer work, [29] uses restricted Boltzmann machine (RBM) to perform collaborative filtering in recommender systems, however the system is a single-layer RBM. Later, [34] and [19] build upon Bayesian deep learning to develop hierarchical Bayesian models that tightly integrate content information and user-item rating information, thereby significantly improving recommendation performance. After that, there are also various proposed sequential (or session-based) recommender systems [12; 27; 2; 18; 22; 39; 14; 31; 41; 25], GRU4Rec [12] was first proposed to use gated recurrent units (GRU) [6] for recommender systems. Since then, follow-up works such as hierarchical GRU [27], temporal convolutional networks (TCN) [2], and hierarchical RNN (HRNN) [25] have achieved improvement in terms of accuracy by utilizing cross-session information [27], causal convolutions [2], as well as meta data and control
signals [25]. Another line of work focusing on building self-attention based sequential models such as SASRec [14], BERT4Rec [31], and S3Rec [41]. In this paper we build on such sequential RecSys and note that our ZESREC is model agnostic, that is, it is compatible with any sequential RecSys.
Cross-Domain and Cold-Start RecSys. There is a rich literature on cross-domain RecSys focusing on training a recommender system in the source domain and deploying it in the target domain where there exist either common users or items [40; 38; 3; 16]. These works are also related to the problem of recommendation for cold-start users and items, i.e., users and items with few interactions (or ratings) available during training [11; 20; 42; 23]. There are also works [24; 9] handling cold start on both user and item with meta-learning, however they cannot generalize across domains. In summary, prior systems are either (1) not sufficient to address our zero-shot setting where there are neither common users nor common items in the target domain or (2) unable to learn user behavior patterns that are transferable across datasets/domains. Therefore, they are not applicable to our problem of zero-shot recommendations.

heading: 3 ZERO-SHOT RECOMMENDER SYSTEMS
text: In this section we introduce our ZESREC which is compatible with any sequential model. Without loss of generality, here we focus on NL descriptions as a possible instantiation of universal identifiers, but note that our method is general enough to use as identifiers other content information such as items’ images and videos. We leave exploration for other potential modalities to future work.
Notation. We focus on the setting of zero-shot recommendation where there are neither overlapping users nor overlapping items between a source domain and a target domain. We assume a set Vs of Js items and a set Us of Is users in the source domain, as well as a set Vt of Jt items and a set Ut of It users in the target domain. We let I = Is + It and J = Js + Jt; we use j ∈ Vs ∪ Vt to index items and i ∈ Us ∪ Ut to index users. The zero-shot setting dictates that Vs ∩ Vt = ∅ and that Us ∩ Ut = ∅. We denote the collection of all users’ interactions as a 3D tensor (with necessary zero-padding) R ∈ RI×Nmax×J , where Nmax is the maximum number of interactions among all users. We use the subscript ‘∗’ to represent the collection of all elements in a certain dimension. Specifically, each user i has a sequence of Ni interactions (e.g., purchase history) with various items denoted as Ri∗∗ = [Rit∗]Nit=1, where Rit∗ ∈ {0, 1}J is one-hot vector denoting the t-th item user i interacted with. The same user i has different user embeddings at different time t, reflecting dynamics in user interests; here we denote as uit ∈ RD the latent user vector when user i interacts with the t-th item in her history, and we use U = [uit] I,Nmax i=1,t=1 ∈ RI×Nmax×D (with necessary zero-padding) to denote the collection of user latent vectors. We denote as vj ∈ RD the item latent vector and V = [vj ] J j=1RJ×D as the collection. For simplicity and without loss of generality, in this paper we focus on using NL descriptions (i.e., a sequence of words) to describe items. For item j we denote its NL description as xj , and the number of words as Mj ; similar to V, we let X = [xj ]Jj=1. With slight notation overload on t, we denote as R(s) and R(t) the sub-tensor of R that corresponds to the source and target domains, respectively. Similarly, we also have U(s), U(t), V(s), V(t), X(s), and X(t).
Problem Setup. A model is trained using all users’ interaction sequences from the source domain, i.e., {Ri∗∗}i∈Us , and then deployed to recommend items for any user ι ∈ Ut in the target domain, given user ι’s previous history Rι∗∗, which can be empty. In practice we append a dummy item at the beginning of each user session, so that during inference we could conduct recommend even for users without any history by ingesting the dummy item as context to infer the user latent vector. In our zero-shot setting, the model is not allowed to fine-tune or retrain on any data from the target domain.
Definition of Zero-shot Learning in RecSys. Our zero-shot setting includes three unique properties: (1) cold users, (2) cold items, and (3) domain gap. It is fundamentally different from previous content-based cold start as the latter setting usually satisfies either (1) or (2), but not often both.

heading: 3.1 FROM CATEGORICAL DOMAIN-SPECIFIC ITEM ID TO CONTINUOUS UNIVERSAL ITEM ID
text: Most models in recommender systems learn item embeddings through interactions. These embeddings are indexed by categorical domain-specific item ID, which is transductive and cannot be generalized to unseen items.
In this paper, we propose to use item generic content information such as NL descriptions and image to produce item embeddings, which can be used as continuous universal item ID. Since such content information is domain agnostic, the model trained on top of it can be transferable from one domain to another, therefore making zero-shot recommender systems feasible. Based on the universal item embeddings, one can then build sequential models to obtain user embeddings by aggregating embeddings of items in the user history.
Here we introduce the notion of universal embedding networks (UEN) that use continuous universal embeddings to index items (and therefore users) rather than categorical ID that is not transferable across domains. We call the UEN generating item and user universal embeddings item UEN and user UEN, respectively.

heading: 3.2 MODEL OVERVIEW
text: We propose a hierarchical Bayesian model with a probabilistic encoder-decoder architecture. The encoder ingests items from user history to yield the user embedding, while decoder computes recommendation scores based on similarity between user embeddings and item embeddings.
Generative Process. The generative process of ZESREC (in the source domain) is as follows:
1. For each item j: • Compute the item universal embedding: mj = fe(xj). • Draw a latent item offset vector j ∼ N ( 0, λ−1v ID ) . • Obtain the item latent vector: vj = j +mj . 2. For each user i:
• For each time step t: – Compute the user universal embedding: nit = fseq([viτ ] t−1 τ=1). – Draw a latent user offset vector ξit ∼ N ( 0, λ−1u ID ) .
– Obtain the latent user vector: uit = ξit + nit. – Compute recommendation score Sitj for each user-interaction-item tuple (i, k, j), Sitj = fsoftmax(u > itvj) and draw the t-th item for user i: Rit∗ ∼ Cat([Sitj ]Jj=1).
Here fsoftmax(·) is the softmax function: fsoftmax(u > itvj) = exp(u > itvj)/ ∑ j exp(u > itvj). Cat(·) is a categorical distribution. fe(·) is item UEN (see Sec. 3.3), fseq(·) is user UEN (see Sec. 3.4). iτ in viτ indexes the τ -th item that user i interacts with. λu and λv are hyperparameters. The latent item offset j = vj −mj provides the final latent item vector vj with the flexibility to slightly deviate from the contentbased item universal embedding mj . Similarly, the latent user offset ξit = uit − nit provides the final latent user vector uit with the flexibility to slightly deviate from the user universal embedding nit. Intuitively, j and ξit provide domain-specific information on top of the
domain-agnostic information from mj and nit. In the target domain we will remove j from vj , and ξit from uit, which can be seen as an attempt to remove the bias learned from the source domain.
Training. The MAP estimation in the source domain can be decomposed as following: P (U(s),V(s)|R(s),X(s), λ−1u , λ−1v ) ∝ P (R(s)|U(s),V(s)) · P (U(s)|V(s), λ−1u ) · P (V(s)|X(s), λ−1v ).
Maximizing the posterior probability is equivalent to minimizing the joint Negative Log-Likelihood (NLL) of U(s) and V(s) given R(s), X(s), λ−1u , and λ −1 v :
L = Is∑ i=1 Ni∑ t=1 − log(fsoftmax(u>itvit )) + λu 2 Is∑ i=1 Ni∑ t=1 ||uit − fseq({viτ } t−1 τ=1)|| 2 2 + λv 2 Js∑ i=1 ||vj − fe(xj)||22, (1)
where nit = fseq({viτ }t−1τ=1) and mj = fe(xj). See the Appendix for a full Bayesian treatment of ZESREC using inference networks and generative networks [15].
Inference and Recommendation in the Target Domain. Once the model is trained using sourcedomain data, it can recommend unseen items j ∈ Vt (where Vt ∩ Vs = ∅) for any unseen user i ∈ Ut (where Ut ∩ Us = ∅) from the target domain based on the approximate MAP inference below:
p(R(t)|X(t)) = ∫ p(R(t)|U(t),V(t),X(t))p(U(t),V(t)|X(t))dU(t)dV(t)
≈ ∫ p(R(t)|U(t),V(t),X(t))δ
U (t) MAP
(U(t))δ V
(t) MAP
(V(t))dU(t)dV(t),
where δ(·) denotes a Dirac delta distribution. U(t)MAP and V (t) MAP are the MAP estimate of U (t) and V(t) given X(t), which we approximate as:
(U (t) MAP,V (t) MAP) ≈ argmax U(t),V(t) p(U(t),V(t)|X(t)) =
( fseq(fe(X (t))), fe(X (t)) ) . (2)
The reason for the approximation is that ZESREC has no access to interactions R(t) in the target domain, making the cross-entropy loss in the Eqn. 4 disappear. The user and item latent matrices U
(t) MAP,V (t) MAP in the target domain enable us to perform zero-shot recommendation by computing
recommendation scores based on inner products and recommend item argmaxj fsoftmax(u > itvj).
As long as the unseen items’ NL descriptions are available, ZESREC could obtain both the unseen users’ latent vectors and the unseen items’ latent vectors based on the item universal embedding network. This is in contrast to previous methods that rely on catogorical domain-specific item ID, which is not transferable as unseen items have completely different ID from items in the training set. Note that ZESREC is general enough to adapt to other data modalities such as images and videos.
3.3 ITEM UNIVERSAL EMBEDDING NETWORK: mj = fe(xj)
The purpose of the item universal embedding network, denoted as fe(·), is to extract item embeddings that are universal across domains. The network consists of a pretrained BERT network [8], denoted as fBERT, followed by a single-layer neural network, denoted by fNN(·). Formally we have
mj = fe(xj) = fNN(fBERT(xj)), (3)
where mj is the universal embedding for item j and xj is the NL description for item j. We use the embedding for the ‘CLS’ token from the last layer of BERT as the output of fBERT(xi). This UEN is jointly trained with the sequential model using the objective function in Eqn. 4. Note that fNN(·) is necessary as we need to adapt the pre-trained BERT for recommendation tasks.
3.4 USER UNIVERSAL EMBEDDING NETWORK: nit = fseq([viτ ] t−1 τ=1)
The user UEN fseq(·) is built on top of the item UEN in Eqn. 3. Specifically, given user i’s interaction sequence until time t, Ri∗∗ = [Riτ∗]t−1τ=1, we first replace it the corresponding NL descriptions l (x) i = [xiτ ] t−1 τ=1, and then fed it into item UEN in Eqn. 3 to obtain l (m) i = [miτ ] t−1 τ=1 = [fe(xiτ )] t−1 τ=1 where item it is user i’s t-th interaction. Each vector in the sequence, mj , is the universal embedding for item j. During training, we obtain item latent vector vj based on mj (see Eqn. 4), while during inference vj = mj (see Eqn. 2). We can then treat each vj as the input at each time step of the sequential model, which gives us the final user universal embedding nit = fseq([viτ ] t−1 τ=1). Note that this user UEN is used during both training (Eqn. 4) and inference (Eqn. 2).

heading: 4 EXPERIMENTS
text: In this section, we evaluate our ZESREC against various in-domain and zero-shot baselines on three source-target dataset pairs, with the major goals of addressing the following questions:
Q1 How accurate (effective) is ZESREC compared to the baselines? Q2 If one allows training models using target-domain data, how long does it take for non
zero-shot models to outperform zero-shot recommenders? Q3 Does ZESREC yield meaningful recommendations for users with similar behavioral patterns
in the source domain and target domain?

heading: 4.1 DATASETS
text: We use three different real-world dataset pairs, one from Amazon [26] and two from MIND [37]:
• Amazon [26]: A publicly available dataset collection which contains a group of datasets in different categories with abundant item metadata such as item description, product images, etc. In our experiments, we consider two datasets: (1) ‘Prime Pantry’, which contains 300K interactions, 10K items, and 76K users, and (2) ‘Grocery and Gourmet Food’, which contains 2.3M interactions, 213K items, and 739K users. • MIND [37]: A large-scale news recommendation dataset collected from the user click logs of Microsoft News. It includes 4-week user history and 5th-week interactions. In our experiments, we simulate zero-shot learning by leave-one-out splitting on subcategories under the largest category in MIND: ‘sports’. We consider two pairs: (1) ‘Others’ to ’NFL’, where ‘Others’ contains all the other subcategories except ‘NFL’. ‘Others’ contains 169K interactions, 8K items, and 57K users, while ‘NFL’ contains 1M interactions, 11K items, and 203K users. (2) ‘Others’ to ‘NCAA’, where ‘Others’ includes all the subcategories except ‘NCAA’. ‘Others’ contains 797K interacions, 17K items, and 206K users, while ‘NCAA’ contains 238K interactions, 31K items, and 56K users.
We adopted a rigorous experimental setup for zero-shot learning to ensure (1) no overlapping users and items and (2) no temporal leakage. See the Appendix for details. Datasets in Amazon pair are divided into training (80%) and test (20%) sets. For datasets in the two MIND pairs, we follow the official splitting [37] and use the four-week user history as the training set and interactions in the fifth week as the test set. Due to strong recency bias in the news recommendation [37], for MIND we run experiments under two settings to measure temporal invariance of the model: (1) evaluate on interactions in the first day of the week (1st-day setting), and (2) evaluate on interactions of the whole week, i.e., the full test set (whole-week setting). For all datasets, We further split 10% of the training set by user as validation set.

heading: 4.2 BASELINES AND ZESREC VARIANTS
text: To demonstrate the effectiveness of our model, we compare ZESREC against two groups of baselines: in-domain methods and zero-shot methods.
In-Domain Methods. We compare variants of our model ZESREC against a variety of state-of-theart session-based recommendation models including GRU4Rec [12], TCN [2], and HRNN [25]. We also consider their extensions, HRNN-Meta, GRU4Rec-Meta, and TCN-Meta, which use items’ NL description embeddings to replace item ID hidden embeddings. Besides aforementioned sophisticated models, we introduce POP which is a simple baseline conducting recommendation only based on item global popularity. It can be a strong baseline in certain domains. All the above 7 methods are trained directly on target-domain data and therefore are considered ‘oracle’ methods.
Zero-Shot Methods. Since no previous work has been done on this thread, we consider two intuitive zero-shot models (1) EmbeddingKNN: a K-nearest-neighbors algorithm based on inner product between the user embedding and item embedding. The item embedding is the BERT embedding from NL description, while the user embedding is an average over embeddings of the user’s interacted items, and (2) Random: recommending items by random selection from the whole item catalogue without replacement.
ZESREC Variants. We evaluate three variants of our ZESREC, including ZESREC-G, ZESRECT, and ZESREC-H which use GRU4Rec, TCN and HRNN as base models, respectively.

heading: 4.3 IMPLEMENTATION DETAILS
text: We adopted pre-trained google/bert_uncased_L-12_H-768_A-12 BERT model from Huggingface [36] to process item description and generate item embedding. The dimension of BERT embedding is 768. For the model architecture, we use BERT embedding as input to a single-layer neural netwrok (i.e., fNN(·)). The output dimension for the NN is set to D (see the Appendix for more details).

heading: 4.4 EXPERIMENT SETUP AND EVALUATION METRICS
text: We conducted three sets of experiments to answer questions proposed earlier in this section.
Zero-Shot Experiments. We trained in-domain baselines on target domain training set, while our ZESREC is trained on source domain. All models are tested on the testing set of the target domain for an apples to apples comparison.
Incremental Training Experiments. To measure how long it takes for non-zero-shot models to outperform zero-shot recommenders, we conducted incremental training experiments on in-domain base models GRU4Rec, TCN, HRNN as well as GRU4Rec-Meta, TCN-Meta, HRNN-Meta. Note that the variants of our ZESREC are NOT retrained on target domain. It is also inevitable that non-zero-shot models eventually outperform ZESREC because ZESREC does not have access to target-domain data. Due to space constraint, we only reported results for incremental training experiments on Amazon pair and MIND NCAA pair under the 1st-day setting. Importantly, under the whole-week setting for the MIND dataset, the ZESRec variants already significantly outperform in-domain competitors which are trained on the full target training set; therefore it is meaningless to compare incremental training results under this setting, as in-domain competitors will never beat ZESRec variants. For all the source-target dataset pairs, we group the interactions by user and sort interactions within each user based on interaction timestamps. We randomly select users until we get enough interactions and build three datasets containing 2.5K, 5K, and 10K interactions, respectively.
Case Studies. To gain more insight what ZESREC learns, we perform several case studies. Specifically, we randomly select users from the test set of the target domain Amazon Prime Pantry (where we evaluate ZESREC) and only keep users for whom ZESREC correctly predicts the 6-th items in the sequence given the first 5 items as context, as we want to focus on sequences where ZESREC works. We use these users as queries to find users with similar behavioral patterns from the source domain Amazon Grocery and Gourmet Food (where we train ZESREC) based on user embeddings from ZESREC. User embeddings are generated based on the first 5 items of the sequence.
Simulated Online Scenarios. In the experiments, ZESRec only accesses target domain data during inference to simulate online scenarios, where new businesses just open and the customers are using
service in real-time. This real-time online access setting is substantially different from batch access ahead of time as it prevents us from training a recommender in the target domain before serving.
Evaluation Protocol. For evaluation, we adopted Recall (R@20) and the ranking metric Normalized Discounted Cumulative Gain (NDCG) [30] (N@20). We removed all the repetitive interactions (e.g., user A clicked item B two times in a row) to only focus on evaluating the model’s capability of capturing the transition between user history to the next item.

heading: 4.5 EXPERIMENTAL RESULTS
text: Zero-Shot Experiments. The experimental results on three dataset pairs are in Table 1. Overall, our ZESREC outperforms zero-shot baselines Embedding-KNN and Random by a large margin in most cases; it can also achieve performance comparable to in-domain baselines. On the Amazon pair, ZESREC beats POP and achieves comparable performance with HRNN-Meta and HRNNInteractions, suggesting the existence of shared recommendation patterns in two domains. For the two MIND pairs, we made the following observations: (1) Under the whole-week setting, ZESREC consistently outperforms HRNN-Meta and HRNN-Interactions by a large margin. Comparing the first-day results and the whole-week results, it is obvious that the in-domain models overfit the history and fail to generalize well on latest user-item interactions. This reflects strong recency bias in news recommendation. In contrast, our zero-shot learning naturally comes with an inductive bias to only model the transition from user history to the next item. (2) Under the first-day setting, ZESREC still achieves reasonable performance comparing with in-domain models on both pairs. (3) Surprisingly in the MIND NFL pair, the ZESRec variants perform well under the whole-week setting even when the source domain contains much fewer interactions than the target domain (169K VS 1M).
Incremental Training Experiments. Incremental training results are in Fig. 2. Overall, almost all the in-domain baselines are not able to outperform ZESREC by retraining on at most 10K interactions in target domain, which shows the vast importance of conducting zero-shot learning in recommender domain. For new business operating an early-stage RecSys, it’s hard to train a good RecSys with limited interactions. This is a chicken-and-egg problem, as training good RecSys requires sufficient interactions, while in turn, collecting sufficient interactions requires a satisfactory RecSys to attract users. Therefore the first 10K interactions are crucial to get the RecSys started.

heading: 4.6 CASE STUDIES
text: To gain more insight, we randomly select pairs of user purchase histories from the source and target domains in the Amazon pair as case studies according to the procedure in Sec. 4.4. The goal is
to demonstrate our ZSR could learn relevant dynamics of users’ purchase history from the source domain and successfully recommend unseen products to an unseen user in the target domain.
Fig. 3 shows the purchase history of a user in the source domain (top), ‘Amazon Grocery and Gourmet Food’, and the purchase history of an unseen user in the target domain (bottom), ‘Amazon Prime Pantry’, where all items are unseen during training. The user in the source domain bought ‘Tension Tamer Tea’, which is a type of herbal tea, after buying some sugary snacks (KitKat) and other tea. Such a pattern is captured by ZESREC, which then recommended ‘Lemon Zinger Herbal Tea’ to an unseen user after she bought some sugary snacks (‘Energy Bars from Clif Bar’) and some black tea. This case study demonstrates ZESREC can learn the user behavioral pattern that ‘users who bought sugary snacks and tea tend to buy caffeine-free herbal tea later’. More interestingly, another case study in Fig. 4 demonstrates that ZESREC can learn the user behavioral pattern that ‘if users bought snacks or drinks that they like, they may later purchase similar snacks or drinks with different flavors’. Specifically, in the source domain, the user purchased ‘Vita Coconut Water’ with four different flavors; such a pattern is captured by ZESREC. Later in the target domain, an unseen user purchase ‘V8 Splash’ with a tropical flavor, ZESREC then successfully recommends ‘V8 Splash’ with a berry flavor to the user.

heading: 5 CONCLUSION
text: In this paper, we identify the problem of zero-shot recommender systems where a model is trained in the source domain and deployed in a target domain where all the users and items are unseen during training. We propose ZESREC as the first general framework for addressing this problem. We introduce the notion of universal continuous identifiers leveraging the fact that item ID can be grounded in natural-language descriptions. We provide empirical results, both quantitatively and qualitatively, to demonstrate the effectiveness of our proposed ZESREC and verify that ZESREC can successfully learn user behavioral patterns that generalize across datasets (domains). The main limitation of this work is that commonality between domains is crucial for ZESRec, as is the case for most other works in transfer learning. Future work includes quantification of such commonality and exploring other modalities, e.g., images and videos, as alternative universal identifiers. It would also be interesting to investigate the interpretability provided by the pretrained BERT model and to incorporate additional auxiliary information to further improve the zero-shot performance.

heading: A APPENDIX
text: A.1 DATA USAGE AND PRIVACY DISCUSSIONS
Data Usage: For real-world scenarios, new business owners who hope to do zero-shot recommendation need to check the dataset policy before usage. On the other hand, if multinational corporations hope to establish a branch in a new region they could use their own data from other existing regions.
Privacy: To protect the privacy of user data, we encourage people who want to adopt our methods to train the model using source-domain data with differential privacy embedded. Some related references include [1; 5; 35].
A.2 BAYESIAN TREATMENT WITH INFERENCE NETWORKS
Training. Using Jensen’s inequality we have the following evidence lower bound (ELBO):
log p(R|X, λu, λv) ≥ Eq[log p(R,U,V|X, λu, λv)]− Eq[log q(U,V|X)],
where the expectation is over q(U,V|X). We have
L = Is∑ i=1 Ni∑ t=1 − log(fsoftmax(u>itvit)) + λu 2 Is∑ i=1 Ni∑ t=1 ||uit − fseq({viτ } t−1 τ=1)|| 2 2 + λv 2 Js∑ j=1 ||vj − fe(xj)||22,
(4)
For the variational distribution q(U,V|X) we have the following factorization:
q(U,V|X) = q(V|X)q(U|V,X) = q(V|X)q(U|V),
where U = [ui]Ii=1, V = [vj ] J j=1, and X = [xj ] J j=1
q(vj |X) = q(vj |xj) = N ( vj ; fe,µ(xj), fe,σ2(xj) ) ,
q(ui|V,X) = N ( ui; fseq,µ({vjk} ni k=1), fseq,σ2({vjk} ni k=1) )
Overall,
log p(R|X, λu, λv) ≥ Eq[log p(R,U,V|X, λu, λv)]− Eq[log q(U,V|X)]
= Eq[ Is∑ i=1 Ni∑ k=1 log(fsoftmax(u T ikvik))]
+ Eq[log p(V|X)] + Eq[log p(U|V)]− Eq[log q(U,V|X)] + C
= Eq[ Is∑ i=1 Ni∑ k=1 log(fsoftmax(u T ikvik))]− λv 2 ∑ j ‖fe,σ2(xj)‖1
+ Eq[log p(U|V)]− Eq[log q(U,V|X)] + C, where the expectation is over q(U,V|X). Below we discuss several important terms including −Eq[log q(U,V|X)] and Eq(U,V|X)[log p(U|V)] in the ELBO.
The term −Eq[log q(U,V|X)] is the entropy of q(U,V|X). −Eq(U,V|X)[log q(U,V|X)] = −Eq(V|X)Eq(U|V)[log q(V|X) + log q(U|V)]
= −Eq(V|X)Eq(U|V)[log q(V|X)]− Eq(V|X)Eq(U|V)[log q(U|V)] = −Eq(V|X)[log q(V|X)]− Eq(V|X) [ Eq(U|V)[log q(U|V)] ] = 1
2 ∑ j [1> log fe,σ2 (xj)] + 1 2 ∑ i Eq(V|X) [ Ni∑ t=1 1> log fseq,σ2 ({viτ } t−1 τ=1) ] + C
≈ 1
2 ∑ j [1> log fe,σ2 (xj)] + 1 2Nv ∑ i ∑ V [ Ni∑ t=1 1> log fseq,σ2 ({viτ } t−1 τ=1) ] + C,
where V is sampled for Nv times to get a Monte Carlo estimate of Eq(V|X)[log fseq,σ2(viτ }t−1τ=1)]. In practice, it is found that one sample is usually sufficient due to the use of SGD-based optimization process.
Similarly the term Eq(U,V|X)[log p(U|V)] = Eq(V|X) [ Eq(U|V)[log p(U|V)] ] = −λu
2 Eq(V|X) [ Is∑ i=1 Ni∑ t=1 ‖fseq,σ2({viτ }t=1τ=1)‖1 ] + C
= − λu 2Nv ∑ V Is∑ i=1 Ni∑ t=1 ‖fseq,σ2({viτ }t−1τ=1)‖1 + C,
where V is sampled for Nv times from q(V|X) to get a Monte Carlo estimate of Eq(V|X)[‖fseq,σ2(vik} ni k=1)‖22]. In practice, it is found that one sample is usually sufficient due to the use of SGD-based optimization process.
Inference. Inference can be done via Monte Carlo estimates of p(R|X, λu, λv). Specifically,
p(R|X, λu, λv) = Eq(p(R|U,V)) ≈ 1
NvNu ∑ V(n) ∑ U(n) p(R|U(n),V(n)),
where V(n) ∼ q(V|X) and U(n) ∼ q(U|V(n),X). One could also use MAP inference to trade accuracy for speed.
p(R|X) = ∫ p(R|U,V,X)p(U,V|X)dUdV ≈ ∫ p(R|U,V,X)δUMAP(U)δVMAP(V)dUdV,
where δ(·) denotes a Dirac delta distribution. UMAP and VMAP are the MAP estimate of U and V given X:
(UMAP,VMAP) ≈ argmax U,V
q(U,V|X) = ( fseq(fe,µ(X)), fe,µ(X) ) .
A.3 IMPLEMENTATION DETAILS
We use pre-trained google/bert_uncased_L-12_H-768_A-12 BERT model from Huggingface [36] to process item description and generate item embedding. The dimension of BERT embedding is 768. We use BERT embedding as input to a single-layer neural netwrok (NN) and the output dimension for the NN is set to D, which equals to the hidden dimension of the sequential model.
For ZESREC variants we use the default optimal setting: we set the hidden dimension D as 300, the dropout rate as 0.2, and the number of training epochs as 20. We use Adagrad [7] as the optimizer with a learning rate of 0.1, and train ZESREC variants in the source domain with early stopping based on validation loss. We set the hyperparameter λv as a relatively large value 100 to restrain the variance of the item offset vector j .
For base models (HRNN, TCN, GRU4Rec) and corresponding base-meta models (HRNN-Meta, TCN-Meta, GRU4Rec-Meta), we set the dropout rate as 0.2 and the number of training epochs as 20; we choose Adagrad [7] as the optimizer. We train base models and base-meta models in the target domain with early stopping and perform hyperparameter tuning, both are based on the validation loss. We tried the hidden dimension D in {128, 300} and the learning rate η in {0.01, 0.1, 1}, and choose to use the configurations {D : 128, η : 1} for base models and {D : 128, η : 0.1} for base-meta models.
For all datasets, we treat the rating as implicit feedback (interactions between user and item). Since we are considering session-based recommendation and using sequential model, we filter out users with only 1 interaction as the sequential model need to ingest at least one item from user history as context to perform next-step prediction.
No Temporal Leakage. We adopted a rigorous experimental setup for zero-shot learning: Firstly, we makes sure that there are no overlapping users and items between the source domain and the target domain. Secondly, we temporally split the two domains, meaning that all the training interactions in the source domain must be
happened before all the testing interactions in the target domain.
All experiments were ran on a GPU machine with Nvidia Tesla V100 16G memory GPU.
A.4 MODEL ARCHITECTURE
SEQ
SEQ
SEQ… User UniversalEmbedding
1-Layer NN
Pretrained BERT Model
X
1-Layer NN
Pretrained BERT Model
…
0.36 0.29
…
0.09 0.02
Prediction Score
Item Universal Embedding
Pretrained BERT Model
1-Layer NN
Item Universal Embedding
Pretrained BERT Model
1-Layer NN
Item Universal Embedding
Item Universal Embedding Item Universal Embedding … …
Latent Item Offset Vector+ Latent Item Offset Vector+
Latent Item Offset Vector+
Latent Item Offset Vector … Latent Item Offset Vector
+
+ Latent User Offset Vector
Figure 6: Model Architecture
The graphical model for ZESREC is shown in Fig. 5. In the MAP estimation version of ZESREC, we set λu → ∞ to remove latent user offset vector, thereby preventing ZESREC from over-parameterization. Fig. 6 shows a simplified deterministic model architecture from the neural network point of view. Below we elaborate on the process in terms of two stages: training in source domain and inference in target domain.
Training in Source Domain. During training, on the encoder side, we generate BERT embeddings from items’ NL descriptions as item universal embeddings and then add the learnable item offset vectors to them, which yield the final item embeddings (item latent vectors). The sequential model will aggregate item embeddings of items in user history to generate user embeddings. On the decoder side, we compute item latent vectors the same way we do on the encoder side. Here we share item latent vectors on both encoder and decoder sides to reduce number of parameters. Empirically we find this prevent overfitting and improve performance.
Inference in Target Domain. In this phase, we will remove the item offset vectors on both the encoder and decoder sides. We use item universal embeddings directly instead as the final item latent vectors since we have no access to interactions in the target domain, and therefore the model is unable to estimate the item offset vectors.

Review summaries:
This paper studies "zero shot recommendation" where source and target domain have no overlap in terms of user and items. The paper proposes to use item content features, such as leveraging BERT on descriptions, instead of IDs. Experiments are conducted on two offline datasets.

A great recommender system relies on great training set. However, at the beginning, there is no such data availability. This paper tries to solve the zero-shot recommendation problem where there is no user or item overlaps. The two challenges are generalize to unseen users and to unseen items. For unseens users, sequential recommendation represents users as a sequence of their interacted items. As long as the items are seen before, the users can be represented. As for items, the unique ids are not useful. However, the attributes such as natural language description can be universal. This paper proposed an approach based on hierarchical Bayesian model. The item universal embedding is using pre-trained BERT network with a single layer neural network. Extensive experiments are carried out to demonstrate the effectiveness of the proposed approach.