Everyone's Preference Changes Differently: Weighted Multi-Interest Retrieval Model
Authors: Hui Shi
Yupeng Gu
Yitong Zhou
Bo Zhao
Sicun Gao
Jishen Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: recommendation system
sequential models
temporal dynamics
user behavior modeling
multi-interest representation

Abstract: User embeddings (vectorized representations of a user) are essential in recommendation systems. Numerous approaches have been proposed to construct a representation for the user in order to find similar items for retrieval tasks, and they have been proven effective in industrial recommendation systems. Recently people have discovered the power of using multiple embeddings to represent a user, with the hope that each embedding represents the user's interest in a certain topic. With multi-interest representation, it's important to model the user's preference over the different topics and how the preference change with time. However, existing approaches either fail to estimate the user's affinity to each interest or unreasonably assume every interest of every user fades with an equal rate with time, thus hurting the performance of candidate retrieval. In this paper, we propose the Multi-Interest Preference (MIP) model, an approach that not only produces multi-interest for users by using the user's sequential engagement more effectively but also automatically learns a set of weights to represent the preference over each embedding so that the candidates can be retrieved from each interest proportionally. Extensive experiments have been done on various industrial-scale datasets to demonstrate the effectiveness of our approach. 

Under review as a conference paper at ICLR 2023
EVERYONE'S PREFERENCE CHANGES DIFFERENTLY: WEIGHTED MULTI-INTEREST RETRIEVAL MODEL
Anonymous authors
Paper under double-blind review
ABSTRACT
User embeddings (vectorized representations of user) are essential in recom- mendation systems. Numerous approaches have been proposed to construct a representation for the user in order to find similar items for retrieval tasks, and they have been proven effecttive industrial recommendation systems. Recently people have discovered the power of using multiple embeddings to represent user, with the hope that each embedding represents the user's interest in certain topic. With multi-interest representation, it's important to model the user's s preference over the different topics and how the preference change with time. However, existing approaches either fail to estimate the user's affinity to each interest or unreasonably assume every interest of every user fades with an equal rate with time, thus hurting the performance of candidate retrieval. In this paper, we propose the Multi-Interee Preference (MIP) model, an approach that not only produces multi-interest for users by using the user's sequential engagement cally learns set of weights to represent the preference over each embedding so that the candidates can be retrieved from each interest proportionally. Extensive experiments have been done on various industrial-seat -ccale datasets to demonstrate the effectiveness of our approach.
effectively but also automati-
INTRODUCTION
Today, the recommendation system is widely used in online platforms to help users discover relevant items and deliver a positive user experience. the industrial recommendation systems, there are usually billions of entries in the item catalog, which make it impossible to calculate the similarity between a user and every item. The common approach is, illustrated in Figure [ retrieving only hundreds or thousands of candidate items based on their similarity the embedding approximate level (e.g. inverted indexes, location sensitive hashing) without consuming too much computational power, and then sending the retrieved candidates to the more nuanced ranking models. Thus, finding effective user embedding is fundamental to the recommendation quality.
The user representations learned from the neural networks are proven work well large-scale online platforms, such as Google (Cheng et al. (016). YouTube (Covington et al. 2016), and Alibaba (Wang et al.) 2018). Mostly, the user embeddings are learned by aggregating the item embeddings from the user engagement history, via sequential models (?Hidasi et al.) 2015 Quadrana et al. 2017] Kang McAuley 2018 You al. 2019). These works usually rely on the sequential model, e.g. recurrent neural network (RNN) model or an attention mechanism, to produce single embedding that summarizes the user's one or more interests from recent and former actions.
Recently researchers (Epasto & Perozzi 2019: Weston et al., 2013 Pal et al. 2020 Li et al. 2005
have discovered the importance of having multiple embeddings for an individual, especially in the retrieval phase, with the hope that they can capture user's 's multiple interests. The intuition is quite clear: if multiple interests could be similar and can be decoded to all the true interests of the user, directly using the single collapsed embedding to retrieve the closest items might result items that the user is not quite interested in, as illustrated in Figure I
user are collapsed into a single embedding, though this embedding
'The code available h atttp:://anonymous.oom 4oper

---

Under review as a conference paper at ICLR 2023
Recommended
items
Ranking
Candidates (bundreds of thousands)
+
+
User interested categorics
Embedding model
User embedding
Retrieval
Other categories in corpus
Single user representation
User behavioral
large-scute Item
Categories retrieved
data
catalogg (- (-billions)
+
Figure Mis-representtaion with single user embedding in the retrieve-then-rank framework.
Though, conventional sequential models like RNN or the Transformer network do not naturally produce multiple sequence-level embeddings as desired in the multi-interest user representation. Existing solutions fall into two directions: 1) split-by-cluster approaches first cluster the items in the user engagement history by category labels (Li et al. 2019) or item embedding vectors (Pal et al. 2020) and then compute a representation embedding per cluster; 2) split-by-attention models adopt transformer-like architecture with two modifications. The query vectors in the attention are learnable vectors instead of the projections from the input and the results of each attention head are directly taken as multiple embeddings (Zhuang al. 2020; Cen al. 2020). The limitation of the two approaches is obvious: the split-by-cluster method works best with dense the item feature (Xue et al., 2005); and split-by-attention models bias towards the popular categories owing to its shared query vector among all the users and are inflexible to adjust the number of interests, which is fixed in the training phase as the number of attention heads.
Moreover, the existing multi- intti-t works ignore one important aspect: the weights for each embedding. In the retrieval stage, given the limited number of items to return, retrieving items from each embedding uniformly will cause a recall problem when the user clearly indicates high affinity towards one or two categories. Some existing approaches, e.g. PinnerSage (Pal et al.) 2020). use exponentially decayed weights to assign higher score interests that have more frequent and recent engagements. However, the methods still assume that in the same period, regardless of the interest is enduring ephemeral, the level of interest decays equally for any user. Furthermore, these works also assume the number embeddings to be fixed across all users. Not only is this hyperparameter costly to find, but also the assumption that all users have the same number of interests is questionable. Some dormant users can be well represented using one or two vectors, while others might have far more diverse set of niche interests that requires tens of embeddings to represent.
In this paper, we propose Multi-Interest Preference (MIP) model that learns user embeddings on multiple interest dimensions with personalized and context-aware interest weights. The MIP model is consist of a clustering-enhancee feed-forward network to predict the weights for each embedding from the interest embedding as well as the temporal patterns of the interest. The clustering enhanced attention overcomes the aforementioned shortcomings from two aspects: the query, key, and value vectors are projected from the user's engaged items, thus the output of the attention is personalized and minimized the bias toward globally popular categories; moreover, the clustering module can be applied before or after the multi-head attention, releasing the assumption that item features are pre-computed or the item-category labels are available. The main contribution of this paper and the experimental evidence can be summarized follows:
multi-head attention module to compute multiple interests and
custering-enhanced
We propose multi-interest user representation model that minimizes the bias towards popular categories and is applicable no matter if the item embeddings are pre-computed. MIP successful in various industry-scale datasets (Section 4.1 4.2); Appendix revells the bias in global query vector and the error from fixed number of clusters in the split-by- attention approaches, comparing to MIP
In addition to the multi-facet vector representations of a user, MIP will assign weights to each embedding, which is automatically customized for each user interest, and improve the recall of candidate generation by retrieving more candidates from the most representative embedding (Section
Although if the cluster algorithms require, MIP still asks for a number of clusters during the training phase, the number of clusters in MIP in the inference phase can be trivially increased
2

---

Under review as a conference paper at ICLR 2023
decreased without re-training of the model. And the experimental results (Appendix A.3 show that re-configuring the number of clusters has an insignificant impact on the retrieval performance, thus allowing the system trade off the storage and computation cost for better performance. Thus, MIP does not require prior knowledge of the numher of interests of users during the model training phase.
2 RELATED WORK
This work relates to two important aspects of existing recommendation systems: sequential models and the multi-interest framework.
Sequential models. A basic consensus in the recommendation system is that user embeddings should be inferred from the user's historical behavior, and thus the sequential models have been the heart of recommendation models. A typical and classical sequential model is the Markov Chain (Rendle) al., 2010 He & McAuley 2016). While Markov Chain captures short-term patterns of engagement sequence well, it fails to make the recommendation that requires memorizing long sequences, With stronger representation power on long sequences, Recurrent neural networks (RNNs), have been adopted for learning user embedding from arbitrarily long sequences, e.g. GRU4Rec (Hidasi et al.) 2015) and etc. (Xu et al., 2019 Devooght & Bersini 2017). Besides the standard RNN models, specialized recurrent units are proposed to meet the special need of incorporating certain information, e.g. user demographic information (Donkers et al. 2017). global context (Xia et al. (217). interest drifts with time (Chen et al., 2019)) and interaction session (Hidasi et al. (015). Recently, the success of the Transformer network (Vaswani et al.] 2017) has brought evolution to sequential modeling tasks and has been soon adapted to the recommendation models, e-g. ComiRec (Cen al., 2020),
BERT4Rec (Sun et al. 2019). TiSASRec CLi al. 2020), SASRec (Kang & MccAuley 2018). MIND (L 2019). Tex2e (Zhuang et al. 2020 and also our MIP.
Multi-interest user representation. Representing users by multiple embeddings greatly improved the recommendation quality, but not every existing recommendation model can easily extend to multi-interee framework. Classical collaborative filtering and matrix factorization methods do not naturally produce multiple user embeddings, and so do RNNs and attention based models. To discover multiple interests from user engagement history. heuristic methods (Jiang et al., 2020: Yue Xiang 2012) and unsupervised learning methods like clustering (Pal al. 2020 Wandabwa et al., 2020) and community mining (Wang) 2007,; Yu, 2008) have been adopted. Besides, researchers have made efforts to modify the existing neural networks to produce multiple results, for instance, the capsule network (Li et al.] 2019 (Sabour al. 2017 Cen al.) (020) and multi-head attention models (Li et al. 2020 Cen et al.] 2020] Zhou et al.) 2018). However, they require an estimation of the number of interests of users as hyperparameter and do not learn the weight of interests. Therefore, unlike MIP, they produce an equal number of clusters for every user and treat each interest with uniform importance.
Relation to previous works. The motivation of MIP is to acquire weighted multiple user embeddings with standard self-attention but without explicit item-category labels. ComiRec and PinTTxt22 use global-query attention to produce multiple embeddings, which introduces bias toward frequent items or popular categories and the phenomenon is shown in Appendix I Furthermore, they also predefine number of interests that is uniform for all the users. Tisas and BERT4Rec adopt self-attention but can not learn multiple embeddings. MIND relies on the category labels to produce multiple embeddings from self-attention and capsule networks, while the category labels are some- times unavailable or vague in other applications, e.g. YouTube and Pinterest. PinnerSage produces multiple embeddings without category labels, but requires pre-computed item embeddings. Lastly, MIP learns the weights of the multiple embeddings. The comparison are summarized in Table I
METHODOLOGY
In this section, we formulate the recommendation problem and the neural architecture to model the multiple user interests with preference weights in detail.
3

---

Under review as a conference paper at ICLR 2023
User
Sequential
Additional Input
Preference
Name
Embedding
Model
Weight
GRU4Rec
single
RNN
interaction session
N/A
TiSASRec
single
time-aware self- self-attention
timestamps
N/A
BERT4Rec
single
self-attention
N/A
MIND
multiple
label-aware self-attention
category labels
X
ComiRec
multiple
global-query attention
X
PinText2
multiple
shared global-query attention
X
PinnerSage
multiple
N/A
heuristic
MIP
multiple
time-aware SA
timestamps
learned
Table 1: Comparison of MIP existing recommendation models.
3.1 PROBLEM STATEMENT
Let denote the collection of items (with the ID u or the feature vector p) in the data store, and the N represents the of users. For each of user U, observe the historical sequence of items that the user has engaged with, S = (Pr, pi......PPPpp ) or (ft, Ut ..., "
and the timestamps
of the engagement T = (ff, ...,..... Iii,,)) The objective is to learn a set of user embedding x RR (i= 1,... k) and their weights wN. Since the user representation is learned only from the history of that user, hereinafter, omit the superscript for in both the input and output sides for simplicity. Notations as summarized in Table
We distinguish two cases of the I: Dense feature p. In industrial recommendation systems, the learning of item features is often decoupled from the recommendation model. For instance, the item feature may contain semantic information extracted from natural language models and visionary features from computer vision models. Given those feature vectors, the recommendation object mainly focuses on learning the user embeddings from the history of user behaviors. 2) Sparse feature v. On the contrary, in many public datasets, the items have no attainable feature except item IDs (encoded as one-hot vectors), and the recommendation model must learn the dense representation of items through only the user-item interactions. The proposed model is capable of handling both cases.
3.2 MULTI-INTEREST USER REPRESENTATION
Item Encoding, We distinguish two conditions of an item. In most practical systems, every item has a pre-computed feature vector (denote as P;) defined in the space where the distance can capture the similarity between the items. In other cases where the recommendation system knows nothing but unique ids (denoted by an one-hot vector vÿ) about the item, the system needs to learn the dense item feature through embedding layer:
Pj W emmb B3
(I)
Additionally, how the engaged items reflect the user's interest may be implied by the sequential order and the time when they interact. Thus, we concatenate the item features with both its positional and
Notations
Description
Notations
Description
I,U
Item set and user set
Vector concatenation operator Cluster assignment, E R'. Attention head superscript
I
Abbreviation of l,, length of S The item embedding dimension
h
An item embedding, p E Rd Short form of the py¹ and 1
Wh bh
Query projection weights and bias Key projection weights and bias Projected key/query vector size
P., la
Wh b dmodet
M
Attention mask matrix
S
Abbreviation of S", Enggageeent history of user u
2,, Z
User embedding vector(s), z; E Rd, E Bd Indicator function, if cond is true, otherwise Maximum number of embedding vectors per user
[cond]
Table 2: Notations
4

---

Under review as a conference paper at ICLR 2023
Multi-interest User Embedding (Sec.3.2)
Clustering (Sec.3.6)
Cluster Weight (Sec.3.3)
Engagement Prediction (Sec.3.4)
[ ... vyl
[C 4y IP
Max
prediction
Ttem Encoding
Multiply
[e -. e,]
[G C]
Clustering
Mask
Linear
Multi- Head Attention
[ ...
Φ.]
$
FFN
dot product
inveen item
Select by index
sparse feature process
Ui LmMlMal
dense feature process
[z1 4
[w ... Wx]
common process
trainable modules
non-trainable operators
Figure 2: An overview of MIP architecture. The input to the model is the user engagement history containing item embeddings ([v1...v or (pl)..pi)) and their timestamps (ft1...t)). The multi-interest user embedding module produces l embeddings, where k is decided by the clustering method or as a hyperparameter. With clustering and multi-interest representation, the cluster weight module will then estimate the cluster weights for each cluster. Finally, the multi-interest embeddings with corresponding weights are combined to predict the user's interest in an unseen item p. The processes with dense and sparse features are shown with different arrows.
temporal information to produce the action encoding:
ej Pp; (4:;:
and the positional (p) and temporal (r) encodings are given by Vaswani et al. (2017
Ta (f+) sin (f//(Tmex) )3//m)) (2)+1(t)) cos (ta/ (tt/(Trnng (Tma//) )
Pay sin(i/ (pman)) (2/mm)) P25+1 1 (i) cos(i possi/(pmuns (pmia)) 35/mm) )pmas))5//
The
are set as Tmae 10 my = 1, and my = 1. The unit of timestamps is day. In the Section A.2 other forms of encodings are compared, and the Equation 3 Thas weak advantage.
User Representation Candidates To find only a few vectors to represent the entire user-engaged items, the model has consider the similarity within the engaged items. The process is achieved through the seff-atteniinn layer:
: (WW e; + bb)TT (W"e, + Vt))//dmmodel: at, = sof softmaz; (st:s)
(4)
We constrain the attention scores with a mask matrix M. In the sparse feature case, let M., 1 for any i, while in the dense feature case, M indicate the intra-cluster relationship. i.e. M,,; (c;
where the C denotes the cluster-ID. Thus, M requires the attention model to only aggregate items within the same cluster. The generation of M in the dense feature case detailed in Section 3.6 With M and attention scores each attention head aggregates the sequences as:
dig
(5)
To process the aggregated vector from all attention heads, the dropout layer and feed-forward network (FFN) are applied, and compute the output vectors as
Φb FFN (o):
(6)
where j = 1, The FFN() has fully-connected layers with hyperbolie tangent activation function after the first layer, i.e. FFN W (tanh(W'z b')) + b.
Now, each Φ summarizes the item features pj. its adjacency with other items, and neighbor item features. The remaining question is which vectors from the } are the most representative.
Cluster Representations So far, the multi-head attention module has produced I output vectors $1,..., Φi. and each Φ; uses as the (unprojected) query. To find representative vectors, the model utilizes the clustering results. Denote the last item in each cluster as Pu..... Pug we take the &.. that uses the last item as query, to represent the corresponding clusters. Then let the user embeddings be 2; Φ and let the user representation be E k then
2 .....

---

Under review as a conference paper at ICLR 2023
3.3 CLUSTER WEIGHT MODULE
Besides the multi-interest assumption, it's also likely that the user favors each interest unequally. As mentioned in the earlier section of this paper, ranking these interests greatly benefit the candidate generation task given its limited budget. In general, a higher weight should be assigned an interest cluster if the user engages more with items that belong it. In order utilize both the contextual cluster information and the user's engagement sequence in that cluster, we build two-layer feed-forward network on top of the cluster representation 2, and the temporal encoding of items T that belong to the cluster j. We will mask those items that belong to other clusters as zero to make the input dimension consistent. The cluster weight can be written as
Wi FFN((N TTi ml)
(8)
Concretely, the (FN() consists of two fully-connected layers with sigmoid function in between, e.g. (xN = W (sigmoid( +8)) + b. Physically, the clusters learned from Section 3.2 are topics that the user likes, the preference weights of the clusters should be positive. Therefore, the output of the second layer are normalized to the range of [0, +oo] by softplus function
3.4 USER- ITEM ENGAGEMENT PREDICTION
Intuitively, user will engage with an item as long as the item match one of his/her interests (not all). In other words, it's important that the item embedding is very close to of the user embeddings, rather than being as close as possible to all of them. Therefore, the user-item affinity should depend on the maximum of item embedding and user embedding on each interest dimension. Furthermore, user's affinity to each interest cluster should be embodied in the likelihood as well: a higher weight should be assigned an interest cluster when the user engages more with items that belong to Therefore, we propose the likelihood that a user will engage with an item as follow:
y = max Linearrzz p)} max(w, (w> (Wo(zy p) + b.))
(9)
where Z [ x is the aforementioned user multi interest embedding (on & interest dimen- sions), and p is the item embedding.
3.5 LOSS FUNCTION
Given the set of items with a positive label (I+) and negative label (I_), the negative log-likelihoee (NLL) loss of our model can be written as:
f= h= Σ
log(v")
Σ P.
log(1 yz v²))/ (
(10)
uEH P.EZZ
wEll
3.6 CLUSTERING
We have referred to a clustering step in producing attention mask (M in Eq. 5). finding cluster representation through last item in each cluster (μ4 in Eq. 7 and learning the cluster weight (C in Eq. 8). Generically, the clustering algorithm takes the set of vectors and produces the clusters T,, T.,,... ...,. Ly which is a partition of S. From the partition, the following items are computed: cluster assignment C. list of the same length as S that C, if p; E Ly, i 1, I.... and r 1..., k; attention mask M, a symmetric binary matrix that M,, = [ec= and last item index μ that μrr denotes the index of the latest item if cluster r.
The clustering is applied differently with the dense and sparse features. With dense features, the exogenous item embeddings (p,) are clustered, and the cluster results lead to non-constant M mask, while with the sparse feature, the clustering is applied to the user representation candidates (Eq. 6). When unspecified, the model uses the Ward Jr (1963)'s clustering method

---

Under review as a conference paper at ICLR 2023
P@20
@@20
AUC
NLL
Amazon
MovieLens
Taobao
PinnerSage
74.02
29.61
81,45
103.3
Items
425,582
15,243
823,971
Interactions
51M
20M
100M
TiSASRec
84.96
47.8
Training
57,165
127,212
343,171
ComiRec
86.35
34.54
87.50
40.7
Test
5,000
5,000
10,000
MIP
88.20
35.28
89.26
37.7
Validation
5,000
5,000
10,000
Table 3: Performance on the Pinterest dataset.
Table 4: Dataset statistics.
EXPERIMENTS
We conduct an exhaustive analysis to demonstrate the effectiveness of MIP on the data from Pinterest, one of the largest online content discovery platforms, and on public datasets.
4.1 WITH SPARSE FEATURE
We first evaluate MIP on learning from collaborative filtering datasets, where the item features are absent and will be learned from the user-item interactions.
Three public datasets are used: Amazon-book (hereinafter, Amazon). Taobac and MovieLeng We adopted a 10-core setting previous works (Li et al. 2020, Wang et al. 2019 and filtered out rare items that appear less than 10 times the whole dataset, and the inactive users who interact with less than 100 items. We split each user's engagement history to non-overlapping sequences of length 100, and use the first 50 items to learn the user embedding(s) and the last 50 items as positive samples to rank. For each sequence, another 50 negative samples are uniformly randomly selected from the items that the user does not interact with. The numbers of sequences in the datasets are listed in Table
Dataset:
Models configurations: We compare the open-sourced baseline models with MIP. The configurations follow the principles: 1) all the item and user embedding vectors have the same size (d = 32); 2) the models adopt multi-head attention module, the number of attention heads is the same (H = 8);
3) the bascline models should have comparable or more parameters than MIP. We let the hidden size in GRU4Rrec be 128, the key and query projected dimension (dmodet in Eq. is labeled in place with the results, and if the model contains a position masse FFN (E. the FFN is two-layered mully-connected with hidden size of 32. The BERT4Rec model is originally proposed to predict the item directly as a classification task, which is unrealistic in the retrieval setting, so we take its last BERT output the user embedding compute the similarity between the NLL loss. We disabled the passall-parallel mini-batch in GRU4Rec since the session information absent. We also replace the text encoder in the PinText2 with an item embedding layer since the input in our experiments is items instead of sequences of words.
and item, and train with
The models are evaluated in the retrieval scenario, where the recommendation system needs to recommend a batch of items to the user. The following metrics are used: area under the ROC curve (AUC), negative log- likelihood (NLL), precision top-K K (P@ K), and recall top-K (R @ (K. The metrics are reported in 10-2
Metries:
Training: All the models are trained for 100 epochs on Tesla T4 GPU with an early stop strategy that ceases the training when validation AUC does not improve for 20 epochs. The MIP is trained with two-phase. In the first phase, the parameters in the cluster weight module are frozen, and the weights for any cluster are set 1. After the model converges in the first phase, the parameter freeze removed, and then all parameters are trained until converge. Appendix AA.4comppree and analyzed this training strategy of MIP.
yy= - log exp((2x). We set ==
generat edtee///aaaa /tt./kklee. www.arn......................... cluster. Aggl https::/scckktt-leeeeee http://ttteeaaa..//taatbbbbebbbe h/wouulleeeeelleeeooeooo e
We adopted the scikit-lcarn implementation.
htt.stttrinnmnmmllll.... with n n_cluster=5, and other
default arguments. In Appendix A.3 other clustering methods are compared.
'ntps:/jjmncaulcy.esdd htt.:/yyunessdduuutaataaaaoooooo
7

---

Under review as a conference paper at ICLR 2023
Config
Amazon
Taobao
MovieLens
Model
Params.
layers
mmadel
AUC
R@
AUC
R@
AUC
R@ 50
GRU4Rec
66338
68.62
63.5
81.55
74.48
96.13
90.31
BERT4Rec
50242
64
68.11
63.15
81.47
74.52
95.95
90.11
BERT4Rec
55426
72.10
66.52
81.46
74.47
96.02
90.24
PinTTex22
69634
256
55.83
54.13
71.58
66.88
88.27
81.68
TiSASRec
67586
72.11
66.67
81.46
74.43
96.02
90.16
ComiRec
67586
256
71.72
67.36
70.92
63.61
96.25
90.65
MIP
49347
32
80.47
78.85
88.49
88.43
93.04
93.34
Table
Performance on public datasets. Params excludes the parameters in the embedding matrix.
The performance is summarized in Table 5 MIP has stronger performance on Amazon and Taobao datasets and is trivially worse than GRU4Rec and ComiRec in AUC on MovieLens. Intuitively, the results might be because the shopping scenario fits more to our multi-interest assumption, users purchase only items from their interested categories, like type of sports or habits, while the majority of people would watch all types of movies, and if they like the movie largely depends on the movie quality, instead of movie category. In addition, since models have very close performance, MIP still competitive approach in applications that do not support the strong multi-interest assumption.
Results:
4.2 WITH DENSE FEATURE
The dataset contains user engagement history collected from Pinterest, an image-sharing and social media service that allows users share and discover visual content (images and videos). The interactions between user and an item (also referred to as a pin) are categorized into impression (pin is shown to the user), clickthrough (user clicks the pin), re-pin (user saves the pin into their board collection), and hide (user manually hides the pin). In total, there are 38 million interactions from 510 thousand users during three weeks period of time, Each pin is represented as 156-dimension feature extracted by the PinSage model (Ying et al. 2018).
Dataset:
Active users could have very long engagement sequences in a day. Let the engagement sequence of a user be {ji.j2. ... In }. then for each training sample, we use the first engagements to predict the / future engagements. We also enforce a one-day gap between these two segments, because nearby user engagements are usually very consistent with each other (falling into the same category) and thus make the prediction task easy. / is set to 50, which is the same as the setting in Cen et al. (2020). We treat click-through and re-pin as the positive label and hide (which is less often), and impression (without click or re-pin) as the negative label. Since these negative data have also been recommended to the at some point, they are likely to be still relevant the user, and thus correlate with the positive data. In order to alleviate the bias, we introduce the random negative data where pins are sampled from the whole of pins. The entire negative dataset will consist of 50% observed negative data (impression and hide), and 50% random negative data.
Baselines and Model Configuration: We compare the multi-interest models PinnerSage and ComiRec, and the singlc-embedding model Tisas with the same setting as in Section 4.1
Results and Analysis. As shown Table 3 MIP outperforms all the state-of-the-at models that published very recently. The detailed comparison of the model and explanation of the improvement are follows:
PinnerSage shares the clustering algorithm with MIP, but differs in 1) the cluster is rep- resented by the medoid, 2) and the cluster weight uses the heuristic model. Instead, MIP learns the cluster representations and weights from data end-to-end from data, which aligns with our intuition.
TiSASRes has a similar attention model structure MIP, except only using the last output of the attention model as the user embedding. The comparison confirms the necessity of multi-interest representation, as in ComiRec and MIP.
Compared to ComiRec, MIP interestingly shows that self-attention has stronger represen- tation power than attention with global-quee -uery. In Appendix [..], we use synthetic data to illustrate the internal difference between the two types of models.
8

---

Under review as a conference paper at ICLR 2023
(xxp Decay)
(IquE Weighs)
10.0
5.0
MIP
2.5
Recall
7.5
AUC
-1
2.5
-2.5
2
MIP/Equal Weight)
-5.0
MIP
Amazon
Taobao MovicLens
Amazon
Taobuo MovieLens
P@20 R@20
AUC NLL
(1) On public dataset.
(2) On Pinterest dataset.
Figure 3: Cluster weight variants performance (difference to the best baseline models)
4.3
ABLATION STUDY
This section focus on the claim that the user's preference should be dynamically learned from the temporal pattern by demonstrating the effectivvness of the learned personalized cluster weights module of MIP. We refer the reader to Appendix for a comprehensive ablation study on attention mechanism (A.1 positional and temporal encoding (.2)) re-configuration and comparison of the clustering method post-training AA.))
Configurations. To validate the assumption that the preference trends (weights of multiple interests) change from user to user, we compare the MIP with two variants that disable the cluster weight module. The first one (referred as MIP (Equal Weight)) constantly assigns as cluster weight, thus leading to equally weighted uses the heuristic formula, an exponential decay weights given by wy = Σc, petpp-A(tnow PPl et al., 2020). We let the know be the last user engagement time 4, since in practice it's unrealistic to update the weights real-time according to the current timestamp. According to Pal et al. (2020), we also set À = 0.01, which balances well between emphasizing recently engaged items and accentuating frequently engaged categories. We let the number of clusters A 5.
representation. Another one (referred as MIP (Exp Decay))
The variants on the cluster weights are evaluated on Pinterest and the public dataset.
Results and Analysis The performances are shown in Figure 3 In most experiments, the AUC and recall can benefit from learned cluster weights (our proposal). From Fig. 3 the equal weights and exponential decay have an advantage on AUC and recall respectively, but learned weights are overall superior to other options. Note that the exponential decay rate is global to any user and any cluster, therefore, the results demonstrated that personalized and context-aware cluster weight estimation is more reasonable solution to the recommendation.
5 CONCLUSIONS
In this paper, we study the problem of multi-interest user embedding for recommendation systems. We follow the recent findings on representing users with multiple embeddings, which has been proven helpful over the single user representation. However, we find that in industrial recommendation systems, is important to have set of weights for these multiple embeddings for more efficient candidate generation process due to its budget on the number of items returned. More specifically, we define the likelihood of an engagement based on the closest user embedding to the item embedding and update the weight for the corresponding cluster. We also illustrate the different numbers of interests (embeddings) that users could have, which is fundamentally different from the assumption of similar works. In addition, attention mechanism is applied in the model architecture, and have done extensive studies on different model design choices. Finally, case studies on multiple real-world datasets have demonstrated advantage over state-of-the-art approaches.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang. and Tang. Controllable inttis-nttersst framework for recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2942-2951, 2020.
Xu Chen, Yongfeng Zhang, and Zheng Qin. Dynamic explainable recommendation based on neural attentive models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp- 53-60, 2019.
Heng-Tze Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrddo, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pp. 7-10, 2016.
Paul Covington, Jay Adkan, and Emre Sargin. Deep neural networks youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Robin Devooght and Hugues Bersini. Long and short-term recommendations with recurrent neural networks. In Proceedings of the 25th Conference on User Modeling, Adoptation and Personalization, pp. 13-21 21, 2017.
Tim Donkers, Benedikt Loepp, and Jürgen Ziegler. Sequential user-hased recurrent neural network recom- mendations. In Proceedings of the Eleventh ACM Conference Recommender Syxtems, pp. 152-160,, 2017.
Alessandro Epasto and Bryan Perozzi. Is a single embedding enough? learning node representations that capture multiple social contexts. In The World Wide Web Conference, pp. 394 404, 2019.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. density- based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-22222111212222 1996.
Ruining He and Julian McAuley. Fusing similarity models with markov chains for sparse sequential recom- mendation. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 191-200. IEEE, 2016.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session- basiio recommendations with recurrent neural networks. arXiv preprint 1rX.v:151 2015.
Hao Jiang, Wenjie Wang. Yinwei Wei, Zan Gao, Yinglong Wang, and Liqiang Nie. What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 3487-3495, 2020.
Wang-Chen Cheng Kang and Julian McAuley. Self-attentive sequential recommendation. In 2018 IEEE International Conference on Data Mining (ICDM), pp- 197-206. IEEE, 2018.
Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang. Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. Multi-interest network with dynamic routing for recommendation at tmall. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp- 2615-2623, 2019.
Jiacheng Li. Yujie Wang, and Julian McAuley. Time interval aware self-attention for sequential recommendation. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 322-330, 2020.
Yu Li, Liu Lu, and Li Xuefeng. A hybrid collaborative filtering method for multiple-interest and multiple-content recommendation in e-commerce. Expert systems with applications, 28(1):67-77 77, 2005.
Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, and Jure Leskovee, Pinnersage: Multi-modal user embedding framework for recommendations at pinterest. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 2311-2320, 2020.
Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. Personalizing session- based recommendations with hierarchical recurrent neural networks. In proceedings of the Eleventh ACM Conference on Recommender Systems, pp. 130-137, 2017.
Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. Factorizing personalized markov chains for best-basket recommendation. In Proceedings of the 19th international conference on World wide web, pp. 120- 2010.
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. Dynamic routing between capsules. arXiv preprint arXiv: 170.0092,, 2017.
10

---

Under review as a conference paper at ICLR 2023
Hui Shi, Yang Zhang, Hao Wu, Shiyu Chang, Kaizhi Qian, Mark Hasegawa-Johnson. and Jishen Zhao. Contin- uous enn for nonuniform time series. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3550-3354, IEEE, 2021.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888 905, 2000.
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert-ree: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 1441-1450, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:/706.03700 2017.
Herman Wandabwa, M Asif Nacem, Farhaan Mirza, Russel Pears, and Andy Nguyen. Multi-interest user profiling in short text microblogs. In International Conference on Design Science Research in Information Systems and Technology, pp. 154-168. Springer, 2020.
Fang Wang. Multi-interest communities and community based recommendation. 2007.
Jizhe Wang. Pipei Huang, Huan Zhao, Zhiho Zhang, Binqiang Zhao, and Dik Lum Lee. Billion-scale com- modity embedding for e-commerce recommendation in alibaba. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 839-848, 2018.
Xiang Wang, Xiangnan He, Meng Wang. Fuli Feng, and Tat-Seng Chua, Neural graph collaborative filtering. In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pp. 165-174, 2019.
Joc H Ward Jr. Hicrarchical grouping to optimize an objective function. Journal of the American statistical association, 58(301):2362244, 1963.
Jason Weston, Ron Weiss, and Hector Yee. Nonlinear latent factorization by embedding multiple user interests. In Proceedings of the 7th ACM conference on Recommender systems, pp. 65 68, 2013.
Bin Xia, Yun Li. Qianmu Li, and Tao Li. Attention-bass recurrent neural network for location recommendation. In 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE). pp. IEEE, 2017.
Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Jiajie Xu, Victor S Sheng S. Sheng, Zhiming Cui, Xiaofang Zhou, and Hui Xiong. Recurrent convolutional neural network for sequential recommendation. In The World Wide Web Conference, pp. 109883344,22 2019.
Gui-Rong Xue, Chenxi Lin, Qiang Yang, WenSi Xi, Hua-Jun Zeng, Yong Yu, and Zheng Chen. Scalable collaborative filtering using cluster-based smoothing. Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pp. 1 114-121, 2005.
Rex Ying, Ruining Hc, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovee. Graph convolutional neural networks for web-scale recommender systems. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974-983, 2018
Jiaxuan You, Yichen Wang, Aditya Pal, Pong Eksombatchai, Chuck Rosenburg, and Jure Leskovec. Hicrarchical temporal convolutional networks for dynamic recommender systems. The world wide web conference, pp- 2236-222 2246, 2019.
Li Yu. Using ontology to enhance collaborative recommendation based on community. 2008 The Ninth International Conference on Web-Age Information Management, pp. 45-49. IEEF, 2008.
Wu Yue and Chen Xiang. A multi-interest model of recommendation system based customer life cycle. In 2012 Fifth International Conference on Intelligent Computation Technology and Automation, pp. 22 25. IEEE, 2012.
Tian Zhang, Raghu Ramakrishnan, and Miron Livny. Birch: an efficient data clustering method for very large databases. ACM sigmod record, 25(2):033-114, 1996.
Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengehao Zhao, Xiusi Chen, and Jun Gao. Atrank: An attention-based user behavior modeling framework for recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Jinfeng Zhuang, Jennifer Zhao, Anant Subramanian, Srinivas, Yun Lin, Balaji Krishnapuram, and Roelof van Zwol. Pintext 2: Attentive bag of annotations embedding. In Proceedings of DLP--DD 2020, 2020.
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A. VARIANTS OF ATTENTION MECHANISM
To interpret the performance improvement of our models against other attention models that have been applied in the recommendation system. We further construct synthetic dataset and visualize the internal attentions and the user representations aggregated from different attentions.
Synthetic dataset. Without loss of generality, we assume there's ' G global clusters in the corpus, representing different global categories, each of which a d-dimensional Gaussian distribution. Each user is interested in up to & (< G) categories, referred as user clusters. We generate the oracle user interest model by sampling no more than & clusters from G global clusters following a multinomial distribution. Then each of the item in the user engagement history is sampled by two steps: uniformly sample one cluster from user clusters, then sample from the d-dimensional Gaussian distribution. Note that in the synthetic model, the -lluster affinity is measured in Euclidean distance, while in the recommendation model, the affinity is decided by cosine distance. To eliminate this discrepancy, we force the Gaussian distributions center on the unit sphere, so that the rankings by cosine distance and Euclidean distance are consistent.
We use 21) dataset (d- 2) for visualization purpose and another high-dimensional dataset for quantitative evaluation. For 2D dataset, we set a relatively small G = 8 and A: = 4 in order to have a clear boundary between clusters. Since there are only 162 distinct subsets[Wwith with G = k we use 100 of them for training, 31 for validation and the remaining 31 for testing. For high-dimensional dataset, we set G 1024 and & 8, and let d 16,32, 64, 128. We generate 10000 users for training. 1000 for validation and another 1000 for testing.
Attention models. We focus on comparing our attention model (i.e. self-attention), the attention model utilized in ComiRec (i.e. Non-shared query), and the one used in Pin'Text2 (i.e. Shared query). The comparison of the attentions are T
For simplicity, we remove the temporal and positional encoding from the computation of attentions, skip the Ward clustering step from MIP, and directly represent user as Equation 7 Also, the dropout layer is removed in order to eliminate randomness in visualization.
Metrics. We visualize the intermediate results and user representations learned from the 2D dataset for qualitative evaluation. For high-dimensional data, we evaluate the performance by AUC and normalized discounted cumulative gain (nDCG).
Qualitative results and interpretations. Figure 4 shows the learned user representations given the engagement history. There are three observations. 1) When H = 1, global query attention fails to capture all the user interests, while the self attention model is free from the limitation. 2) Viewing from the third row, self-attention model is more accurate in learning cluster representations than global query models. The later is systematically bias due to the global query shown in global query models Figure (6) 3) All the models learn super-clusters, depending on the bias in the dataset. For the example show in Figure 4 the two adjacent clusters on the top side of the unit cirele is often represented to be a super-cluster.
We also visualize the internal attention scores and self attention models (Figure 5. Some attention heads show highly similar attention patterns because their queries are close to each other, which
Global query
Model
Self-attention
Non-shared
Shared (W"p,, + b")
Key vector k
Query vector
q W'p, + by
q shared between sequences q shared between att. heads Dot-product
Yes
No
No AT
Yes
No
kh
e/
q
k
c/ q.t ksh
Table 6: Variations of multi-head attention.
Number of ways to select more than 4 clusters from pool of clusters: 162
(9)
+
12

---

Under review as a conference paper at ICLR 2023
non-shared query
shared query
self attention
ose
ground cruth
(b) !|-
H
(d) H
Figure 5: Learned attention scores in self attention model. Darker color represents higher values. (a) the indicator function [c,, (b-d) az,- The input sequence is re-ordered for better visualization.
uloavaling latt
1a0 onlcolage
projoted uee caballing
Figure 4: Learned user representations.
H-1
11
H18
1.00
cutt query
0.95
d-16
d-155
d-12 H-4
d-32 H-3
dm64 H-8
d 28
H
H--
H--3
1.0
AUCC
0.9
0.8
shared query
shared query
self atsention
chared
d-
d-16
d=12
d=22
d-54
12
13
H-3
Figure 7: Performance comparison high di- mensional synthetic dataset. denotes the fea- ture dimension and H is the number of attention heads.
Figure 6: Learned global interests in global query models. The query vector is reversely projected and normalized.
can be verified from Figure Figure compares the ground truth attention model with the learned attention. The learned attention shows clear boundaries between clusters in the heatmap. Note that the ground truth ignores the adjacency of clusters but the self attention model considers the similarity between clusters, so Figure a) is block diagonal while Figure b(dd) has dark blocks off the diagonal.
Quantitative results. Previous results show intuitive comparison between global query models and the self attention model, and the quantitative results further confirm the consistency of performance gain of self attention. Experiments are repeated on dataset for feature dimension
16, 32, 64, 128 and number of attention heads H = 4. 8. Figure 7 shows that the MIP model constantly and significantly outperforms global query models. As illustrated in the 2D dataset, the performance gain benefits from the personalized user representation, rather than matching to the globally popular clusters. Another observation from the result that for global query models,
4 under-performs Hl models, as the number of attention head decides the number of global clusters the model can learn; however for self attention model, H = 4 performs even better than H = 8. The explanation is that self-atted attentitn model does not require growing number of attention heads with respect to the number global clusters, and H could be already enough for capturing user interest but easier than H 8 train.
13

---

Under review as a conference paper at ICLR 2023
Dataset
Configuration
AUC
Recall
item embedding
76.34
73.75
Configuration
AUC
NLL
+ positional
76.19
74.28
Amazon
item embedding
0.8923
0.377
+temporal
78.59
76.94
+ positional
0.8846
0.386
+both
79.31
78.22
+ temporal (one-hot)
0.8850
0.388
item embedding
82.06
81.75
+ temporal (two-hot)
0.8846
0.385
+positional
86.54
86.22
Taobao
+ temporal (sinusoid)
0.8921
0.377
+temporal
86.72
86.56
both (one-hot)
0.8861
0.382
+both 86.59
85.83
+ both (two-hot)
0.8852
0.387
item embedding
95.26
94.45
both (sinusoid)
0.8926
0.377
+positional
95.01
94.31
MovieLens
+tempoal 94.96
94.19
Table 8: Influence of temporal and positional encoding attention on the performance MIP
+ both
94.61
94.12
Table 7: Ablation study the positional and temporal encoding on public datasets. (in 10
A.2 POSITIONAL AND TEMPORAL ENCODING
In Eq. 2 the sequential (positional) and temporal information are encoded and included in the self-attention module to produce the multi-interest representations. The motivation is that given items from the same category, the recent ones might better represent the user's current interest than the obsolete items. We verify 1) if the incorporation of positional and temporal encoding is critical to the performance: 2) how the encoding (Eq. 3 method affects the performance.
Configuration. The MIP are configured on the two set of choices: the Eq. 2 can be configured alternatively:
item embedding only: Ej = Pi-
positional: e; = [P p(j)), where p(j) is given by Eq,
temporal: Ej [py; T ((,)], where 7 (l;) ) is given by Eq.
positional and temporal: Eq. 2 and Eq.
and there several other choices of temporal encodings other than the sinusoidal form in the Eq.
2018); Create exponential buckets [0, 6). [b, 62) 62)........ [6+ oo) with base b, and encode the timestamp
One-hot (hoo et al.
one-hot vector, i.e. = tookup(bucketes Two-hot (Shi et al. 2021); Similarly create exponential boundaries {0. b. b², bn-1, }0), and encode the timestamp b' St b-1
logu(t) i and T.-1 i + logs (t), where
Dataset and Results. The options include positional and temporal information evaluated all the dataset (Tab. 7). and the encodings methods are compared exhaustively on Pinterest dataset
(Tab. 8)
Analysis. Two conclusions can be made from Tab. 7 and Tab. g 1) including both temporal and positional information safe option, which has the best performance Amazon and Pinterest and marginally (< 0.01) worse performance on Taobao and MovieLens; and 2) the model is insensitive encoding methods.
A.3 CLUSTERING OPTIONS
The Ward's algorithm is applied to MIP considering its success in PinnerSage(Pal et al., 2022). it's beneficial to explore the selection of clustering algorithm and the number of clusters the collaborative filtering dataset. To illustrate the impact, we evaluate MIP with a wide range of clustering algorithms.
Model Configuration and training: MIP models are configured with an attention module that takes both positional and temporal encoding. For unweighted MIP. clustering method is applied to the encoded user engagement history (2.) (computed from Eq. in the training stage. For weighted MIP, Ward's algorithm is applied {2.) and the number of clusters is set 5. To keep the MIP
14

---

Under review as a conference paper at ICLR 2023
Clustering Method
Inference Clusters
Unweighted MIP
Weighted MIP
Amazon
Taobao
Movielens
Amazon
Taobao
MovieLens
None
73.11
82.09
95.97
71.56
80.58
95.53
79.31
86.49
94.61
Ward
71.99
80.99
95.72
80.47
87.85
95.25
10
72.16
81.20
95.78
80.84
88.42
95.25
71.58
80.62
95.53
79.26
86.18
94.86
K-Means
71.95
81.03
95.71
80.66
88.02
95.17
10
72.14
81.22
95.77
80.62
88.61
95.10
72.28
80.72
95.34
78.99
85.84
94.46
Spectral
72.37
81.08
95.73
80.79
87.61
94.81
10
72.64
81.26
95.78
81.19
88.40
95.07
71.98
80.63
95.52
79.39
86.29
94.61
BIRCH
72.03
81.02
95.71
80.65
88.03
95.25
72.44
81.21
95.78
80.91
88.53
95.25
DBSCAN
71.98
80.63
95.52
70.05
75.58
89.63
Table 9: Comparison of clustering options in AUC (in 10-2). clusters is independent of training, i.e. changing the number inference clusters does not require the re-train of the model.
Note that the number of inference
fully differentiable, the cluster embedding is the encoding of the last item in each cluster, instead of the medoid.
Inference:
The choice of the clustering in the inference phase is independent of its configuration during the training. We explore the inference options the pre-trained models. Different types of clustering methods are compared:
Ward: hierarchical clustering method that minimizes the sum of squared distances within all clusters.
K-Means: iterative method also minimized the sum of in-cluster summed squared distances.
(SSc & Malik 2000): performs clustering on the projection of the normalized
Laplacian computed from the affinity matrix.
BIRCH(Zhang et al. 1996): another hierarchical method that clusters the points by building the Clustering Feature Tree.
DBSCAN((ster et al. 1996): a density-based clustering method that does not require specifying the number of clusters.
The number clusters set to 8, and when required. Note that during training, number of clusters is fixed to 5, however, after training, MIP can produce other numbers of embeddings per user, which gives the system huge flexibility to trade-off between storage/computation cost and recommendation performance.
Result and analysis: has marginal impact the performance. While PinnerSage reported that Ward's algorithm outperforms the K-Means, their result does not conflict with our observation here. Recall that for PinnerSage and experiment on the Pinterest dataset, the clustering method is applied to the exogenous item embeddings, thus the clustering methods can be influenced by the non-flat geometry and outliers. However, with the collaborative filtering dataset, the clustering method is applied the encodings produced by multi-hean head self-attention layers which average the embedding of the items and all other items (Eq. 4.. The encodings after the multi-head self-attention should be smoothly distributed, and as a result, any clustering methods work almost equally well on that. 2) Selecting the number clusters is non-trive con-trivial trade-off. The motivation to decrease the number of clusters the storage and computation cost which grow linearly as the number of clusters increases. For unweighted MIP, though the non clusterrin (each item is cluster) settings have the best AUC, decreasing the number of user embedding from 50 (non-clus (non-clustering) to 10 is still acceptable. For weighted MIP, since it's impossible to learn the clustering weights without applying a clustering method, the trade-off can be more complicated: besides the storage concern, when the number of clusters increases the
There are two observations from Table 1) the different clustering algorithm
15

---

Under review as a conference paper at ICLR 2023
MIP Model
Amazon
Taobao
MovieLens
Epoch
AUC
R@50
Epoch
AUC
R@50
Epoch
AUC
R@50
equal weight
73.11
66.67
4
82.09
74.86
95.97
90.06
two-phase
6
80.47
78.85
4
88.49
88.43
93.04
93.34
direct-train
18
57.61
57.47
28
80.33
80.23
14
92.43
93.68
Table 10: Comparison training strategy and the performance weighted MIP. The columns Epoch shows the training epochs when the best validation AUC is achieved.
Latency/ Recall
BERT4Rec
BERT4Rec
MIP
MIP
GRU4Rec
PinTex12
TiSASRec
ComiRec
(1==1 I)
(1= 2)
(total)
(clustering)
Train
218.57
925.36
1058.34
555.79
452.94
479.59
998.62
5.86
(std)
(0.81)
(108.96)
(733.33)
(76,96)
(67.07)
(67.50)
(56.31)
(0.76)
Inference
1.15
38.34
57.53
14.46
14.54
14.61
40.05
5.93
(std)
(0.20)
(56.60)
(56.23)
(54,99)
(56.34)
(55.70)
(27.08)
(0.72)
500
63,50
63.15
66.52
54.13
66.67
67.36
78.85
Table 11: Latency and performance comparison of the models. Training and inference latencies are measured in ms, and brackets show the standard deviations.
average information to learn the weights of each cluster decreases and consequently may hurt the overall performance; on the other hand, 10-cluster settings are better than the 5-cluster settings for all the dataset.
A.4 TRAINING WEIGHTED MIP WITH SPARSE FEATURE DATASET
when training MIP on a collaborative filtering dataset, we first train the equally weighted MIP until converging, then lift the parameter freezing to train the cluster weight module together. We detailed the reason and explanation of the strategy here.
As mentioned in Section 4.1
Configurationss The model is trained twice with the same configuration in Section 4.1 The first training adopts two-phase strategy. The first phase model is denoted as MIP (Equal Weight), and the final model is denoted as MIP (two-phase). In the second round, the model is directly trained from random initialization and is denoted as MIP (direct-train).
Results: Table TO shows the performance and number of training epochs achieve the best perfor- mance. The difference between the two-phase and direct-train is significant. Two-phase training not only shortens the training process but also achieves remarkably better results.
Analysis: It's unsurprising that two-phase can accelerate the training. but we need to understand why random-initialized MIP under-performs equally weighted MIP, while the MIP is generalization to equally weighted MIP. The clue is lying in the experiment on the Pinterest dataset. On the Pinterest dataset, the weighted model randomly initialized and outperforms the equally weighted one. The difference between the two experiments is two-fold: 1) the Pinterest dataset contains dense item features thus no item embedding matrix needs to be learned; 2) the clustering algorithm is applied to the item embedding in the Pinterest dataset but to the output of multi-head self-attention (MHA) in the collaborative filter dataset. As a result, in the Pinterest experiments, the clustering assignment is reasonably reliable throughout the training, since the item embeddings are always meaningful. On the contrary. on the collaborative filtering dataset, with the random initialization, the item encodings after MHA much noisier at the beginning of the training, and thus the cluster assignment can be fairly random. Then the input and output of the cluster weight module may be meaningless, which causes hardship in learning the weighted MIP.
A.5 MODEL LATENCY COMPARISON
Seeing the performance gain, another prominent question will be what is the time cost of the performance increase. In this section, we profiled the model latency on a desktop computer with a 12-core Intel i7-8700k CPU, and single Nvidia GeForee RTX 2080 Ti GPU. The neural network training and inference is on the GPU with vanilla PyTorch framework (version I. 1.12) without any further optimization on the computation. The clustering algorithm in MIP is performed by CPU with
16

---

Under review as a conference paper at ICLR 2023
NLL
Triplet (a: =0.2)
Triplet =0.5)
Triplet (a =0.8)
AUC
R@50
AUC
R@50
AUC
R@50
AUC
R@50
92.32
92.97
88,46
88.20
90.33
90.08
90.60
90.61
Table 12: MIP Performance on MovieLens with difference loss functions.
Python's 's scikit-learn package. We set the batch size to 1 and the dataset to Amazon, then measure and summarize the training and inference latency in Table
There few observations from Table 11 First, comparing the neural network inference latency, the clustering step time cost trivial. PinTeex2,, ComiRec, and TiSASRec has similar training and inference latency, while the performance of them are worse than MIP. BERT4Rec has similar latency as MIP since our sequential model architecture are similar, while the BERT4Rec has worse performance. GRU4Rec has the least inference time. Notice that the standard deviations of the inference latency of PinText2,, TiSASRec, and ComiRec are large. It indicates, though on average the three models are faster inference, MIP inference latency is less possible to be very large while other latency might be several times longer than average.
Conclusively, MIP, as well as other baselines compared, can satisfy the latency requirement when applying online, even without further optimization on the computation and serving. MIP has higher time cost compared to some of the baselines, but the performance increase is also appealing.
A.6 LOSS FUNCTION
Beside the NLL loss function in Equation 10 triplet loss is also widely used in contrastive learning and recommendation system. Let the anchor vector to be p", the positive vector is pt and the negative p the triplet loss is given by:
La Σ (sim(p", "emmpppp"")-- sim(p", NEM
p -pm)_simpp", + a)
(11)
In the single user embedding models, the p" is the single user embedding, and pt , positive and negative vectors. While in our multiple embedding assumption, to train with the triplet loss, have modify the form. Similarly to Equation we introduce the maximum operator in the similarity measure. Define the triplet loss for multiple embedding {z, 1kk as:
La Σ (maz (ofj.p*)) ) max (sim(zj 11 + +a) - y + +a) (12) PEL
PEN
where is hyperparameter of the nositive-negative margin. We let the MIP to train Movielens dataset to investigate the impact to the performance of loss function choice. From the observation in Appendix A.4 the MovieLens dataset can be learnt well without two phase training, so we directly train from scratch.
The results Table 12 illustraass that the triplet loss marginally performs worse than the NLL loss we used. A possible reason might be the migration from single-anchor to muli-anchor, but the further investigation is left as future work.
17


Reviews:
[{"review_id": "n--40L8xLi", "reviewer": "Reviewer_TbF3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-interest user embedding retrieval model MIP for recommendation. The idea is to apply clustering to each user's historical-interacted item embedding sequence. Each cluster is then multiplied with a weight to discriminate their importance. \n\nAs they explicitly use clustering methods on item embeddings, the cluster assignment of each item is known. This can be used to derive the importance weights of clusters, i.e., a higher weight should be assigned to an interest cluster if the user engages more with items that belong to it.\n\nThey conduct extensive experiments on three public datasets to conclude the superior performance of MIP in contrast to other retrieval models.", "summary_of_the_review": "Due to the above concerns (mostly due to the efficiency problem in W1), I would vote for rejection. But I would like to increase my score if the above questions could be addressed in the author response.\n\n---\n\nAFTER THE AUTHOR RESPONSE:\n\nThe authors' response relieved my concern about efficiency.\nThe proposed method is at least feasible on the online system although it brings more time cost. Thus I'd like to raise my score to above the acceptance threshold.", "strength_and_weaknesses": "S1. The motivation for assigning weights to different user interests is strong and interesting, and the proposed method of deriving importance weight according to user's engagement with each cluster is novel and intuitive.\n\nS2. They conduct extensive experiments on three public datasets. Overall, the improvements of the proposed method are significant and promising.\n\n---\n\nW1. A major concern is the efficiency of MIP. For each instance, they perform clustering on the user behavior sequence, which seems to be very time-consuming during both model training and inference. The time cost of  MIP should be illustrated to show whether it could be applied to real-world recommendation systems.\n\nW2. I also have several questions about the details of MIP. Most of them are about the motivation of designs in the method.\n\nQ1. The paper discriminates two situations of item features: dense features and sparse features. Why is this needed? In real-world systems, it is common that items are related to both sparse and dense features. How can you tackle this problem?\n\nQ2. What is the benefit of performing self-attention on the user behavior sequence before clustering? What if removing the self-attention part?\n\nQ3. Why use the last item in each cluster as the cluster representation? \n\nQ4. Why do you need to input cluster representation along with engagement vector to predict cluster weight in Equation (8)?\n\nQ5. Is the method guaranteed to reach convergence considering it has an inserted clustering module?\n\n---\n\nMinor: What does the number mean in the training/test/validation rows of Table 4?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DeKLhnE5EP", "reviewer": "Reviewer_ufEd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the multi-interest for user embeddings in recommender retrievers. The authors consider the different weights of interests as well as time-varying interests and integrate a multi-head attention module and a cluster strategy with their weights. The proposed MIP model is then validated using publicly available datasets.\n", "summary_of_the_review": "This work focuses on an important and interesting problem. However, the general solved task is not clear where the unusual objective function and metrics are adopted. There are some misleading notions and statements which makes it very confusing. The experiments are not convincing where the pre-processing is not appropriate and important baselines are missing. \n", "strength_and_weaknesses": "Strengths\n1. This work focuses on the encoding of user embeddings, and it is an interesting and important in recommender retrievers. \n2. Incorporating time information into the multi-interest task is intriguing because user preferences change over time.\n3. The authors run several experiments on various public datasets to demonstrate the effectiveness of the proposed method.\n\nWeakness\n+ The authors point that this work focuses on the retrieval in recommender systems. The models, however, are optimized using the binary cross entropy loss and tested using the metric AUC, NLL, which are rarely used in retrievers.\n+ In general, the sparse and dense features can be combined with the item embedding to allow the item embeddings to be treated as a whole in models. The authors create various operations and run separate experiments on these two types of features. This means that the proposed method cannot be used in cases where there are both sparse and dense features.\n+ The authors claim that \"propose a multi-interest user representation model that minimizes the bias towards popular categories\". However, \"a higher weight is assigned to an interest cluster if the user engages more with items that belong to it\". How do the authors deal with the popularity bias?\n+ In Eq 9, each user chooses one interest, which is inconsistent with the goal of multi-interest.\n+ The experiments are insufficiently convincing.\n    + The adopted metrics AUC and NLL are more appropriate for ranking models, rather than retrievers.\n    + The authors filter users with less than 100 items and thus there are only a significant limited number of users are observed. Experiments should be conducted on larger datasets. The data statistics after filtering should be more detailed. Furthermore, the authors do not provide the details about the test data.\n    + Several important baseline methods in multi-interest task are missing, such as MIND [Multi-Interest Network with Dynamic Routing for Recommendation at Tmall].\n    + In appendix A.1, the experiments are conducted on a synthetic dataset which is not convincing. \n+ The notions, such as k in 3.1 and [;] in equations, are not well defined.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7qCSEa7ppQ_", "reviewer": "Reviewer_QxyP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a multi-interest retrieval model, the objective of which is to increase the performance of the retrieval stage in a standard two-stage recommendation system. Extensive experiments have been done on various large-scale datasets to show the effectiveness of the proposed approach.", "summary_of_the_review": "The authors propose an effective recommendation algorithm that produces multi-interest for users and learns a set of weights to represent the preference over each embedding so that the candidates can be retrieved from each interest proportionally. Overall, the paper is readable and the contributions are significant and somewhat new.", "strength_and_weaknesses": "Strength \n+ The introduction and its motivation are clear.\n+ The performance seems impressive.\n+ The ablation tests are reasonable.\n\nWeakness\n- The primary issue of this work regards the presentation. \n   1) Table 2 misses many notations (e.g., v, \\phi, M, etc.)\n   2) The notation of the concatenation should be [;]?\n   3) Are d and d_model the same?\n   4) Why not provide the math equation(s) of FFN() and Linear()?\n   5) p is learned from v if it is a cold start data? Are they mutually exclusive inputs?\n   6) Does the order of selected \\phi matter?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Ey-5zw8K0", "reviewer": "Reviewer_C5FP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors work on the problem of generating multiple interest embedding to represent user interest across multiple topics via sequence modeling through user’s sequential actions. The main contribution compared to the previous papers is that the authors learns a set of weights to represent the preference over each cluster-level embedding so that the candidates can be retrieved from each interest proportionally. Experiment is conducted on an offline measure using both public dataset and private large scale dataset(Pinterest Data Set).\n", "summary_of_the_review": "The paper studies a very novel and practical problem with a simple and clean solution. The paper is well written and easy to follow. The paper has the potential impact to inspire a lot of people working on the field. I recommend this paper to get accepted.", "strength_and_weaknesses": "[Advantage]\n1. The paper works on the user embedding generation problem which is very practical as it is widely used as the major recommendation retrieving mechanism in most internet companies. \n\nIt is known that a single embedding may not be representative enough to capture a user's multi-dimensional interest. It makes sense to use clustering based methods to generate multiple versions of embedding to increase general embedding representation power.\n\nThe research problem is practical and can benefit others.\n\n2. The main contribution of learning cluster weight makes sense and can significantly help to improve the retrieving efficiency given the candidate quota is usually limited during the retrieval stage.  The proposed method is straightforward and technically sound.\n\n3. The experiment section is comprehensive and the overall paper is well written and easy to follow.\n\n\n[Concerns and Questions]\n\n1.With the intra-cluster Mask M being applied to user representation learning, does it mean inter-cluster information is ignored during the embedding training?\n\nFor instance, the user's sequence for TV and Movie may be clustered into 2 different clusters, but the actions across these 2 clusters can still be beneficial to each other.\n\n2. Since this work considers cluster based multi-interest embedding learning. It is important to analyze how the number of clusters impact the performance. To be specific, it needs to provide analysis on the special case when only using 1 cluster(i.e. does not consider multi-interest factors) vs considering multiple clusters.\n\n3. Is online LE conducted in Pinterest production? How is the performance? The result will be more convincing with production LE/Launch.\n\n4. Nit: In table 2, notation for vector concatenation operator should be “[;]” instead of “[:]”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject