Learning Multi-Faceted Prototypical User Interests
Authors: Nhu-Thuat Tran
Hady W. Lauw
Venue: ICLR 2024 poster (2024)
Keywords: multi-faceted representation
user interests
item characteristics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: We seek to uncover the latent interest units from behavioral data to better learn user preferences under the VAE framework. Existing practices tend to ignore the multiple facets of item characteristics, which may not capture it at appropriate granularity. Moreover, current studies equate the granularity of item space to that of user interests, which we postulate is not ideal as user interests would likely map to a small subset of item space. In addition, the compositionality of user interests has received inadequate attention, preventing the modeling of interactions between explanatory factors driving a user's decision.
To resolve this, we propose to align user interests with multi-faceted item characteristics. First, we involve prototype-based representation learning to discover item characteristics along multiple facets. Second, we compose user interests from uncovered item characteristics via binding mechanism, separating the granularity of user preferences from that of item space. Third, we design a dedicated bi-directional binding block, aiding the derivation of compositional user interests.
On real-world datasets, the experimental results demonstrate the strong performance of our proposed method compared to a series of baselines.

Published as a conference paper at ICLR 2024
LEARNING MULTI-FACETED PROTOTYPICAL USER INTERESTS
Thu-Thuat Tran
Hady W. Lauw
Singapore Management University
Singapore Management University Singapore
Singapore
nttran. 2020 112200phi 08phdcs. smu smu.edu. sg
hadyw wwwee smu. edu.sg
ABSTRACT
We seek uncover the latent interest units from behavioral data better learn user preferences under the VAE framework. Existing practices tend to ignore the multi- ple facets item characteristics, which may not capture at appropriate granularity. Moreover, current studies equate the granularity of item space to that of user inter- ests, which we postulate not ideal as user interests would likely map small subset of item space. In addition, the compositionality of user interests has received inadequate attention, preventing the modeling of interactions between explanatory factors driving a user's decision. To resolve this, propose to align user interests with multi-faceted item characteristics. First, we involve prototype-based represen- tation learning to discover item characteristics along multiple facets. Second, we compose user interests from uncovered item characteristics via binding mechanism, separating the granularity of user preferences from that of item space. Third, we design a dedicated bi-directional binding block, aiding the derivation of compo- sitional user interests. On real-workh datasets, the experimental results demonstrate the strong performance of our proposed method compared to series of baselines.
INTRODUCTION
Preference learning lies at the heart of Collaborative Filtering (CF). A common thread among many CF models (Seehin et al. (2015): Liang et al. 2018); Wang et al. (2019a) He et al. (2020) is rendering a single vector for preference representation, ignoring the fact that user preferences are complex and diverse. Thus, discovering hidden factors behind user preferences could provide insights into what governs consumption patterns of users and thereby boosting the recommendation performance.
Nonetheless, preference learning is considerably challenging because of its unstructured nature. In natural language, one can consider a word or a token as a modular unit. However, in CF, we first need to obtain such 'units", i.e., by deriving multiple vectors, each is unit, of user's preferences. Yet it is quite elusive what structure and level of granularity these preference units should appropriately be, and more importantly, how obtain them in unsupervised setting based only on observed behavior data.
One means to derive a preference 'unit' is by grouping related items into a cluster that represents a meaningful interest, then aggregating item representations produce interest representation and finally, jointly learning item grouping and recommendation under Variational AutoEncoder (VAE) framework Ma et al. (2019) Tran & Lauw (2022): Wang et al. (202a).. Marrying item grouping-basee interest derivation with VAË inherits the best of both worlds. For one, multiple item groups increase representation capacity of VAE to capture multiple intentions of users behind consumption behaviors Ma et al. (2019). For another, interest derivation process inherits VAE's strengths, including non-linear probabilistic modeling, multinomial likelihood as a proxy of ranking loss, and ieformationnthhoreee regularization term, which are shown to boost the recommendation accuracy Liang et al. (2018).
Despite showing improvement to a degree, existing VAE-based recommendation models involving item grouping for interest derivation has several shortcomings. First, these studies ignore that items may be grouped by arbitrary characteristics, owing to their many facets. For example, as depicted in Figure shoes can be grouped by multiple facets, e.g., brand. color top height. Thus, more fine mee-grained structure that focuses on each facet would better reflect item space structure. Second, the assumption that the number of interests per equals the number of item groups is sub-optimal as

---

Published as a conference paper at ICLR 2024
space grouped single choracteristics, egg. top height
b] Group items by brand (Adidas Nike)
c) Group items by color /white vs. green VL pink)
Group htems by top height (high low)
Figure I:
Discovering item space structure under multiple facets. Figure (a), shoe space is combi- nation of facets, which requires many 12 clusters. Figures (b), (c), (d), each divides shoe space by a single facet. The total number of clusters is 7, which is slightly more than half of that in Figure (a).
user only interacts with small subset of items among the whole item space. Third, the compositionality of user interests has received less attention, which may not capture the complexity of user's interest.
Towards addressing these shortcomings, we introduce FACETVAE, which stands for FACETed Variational AutoEncoder, distinguishing itself by three key innovations. Firstly, to fully reveal to better align with user interests, we discover multiple groups
the structure of item space underlying item space along multiple dimensions via prototype-basee representation. For example, three dimensions to group items in Figure a brand, color top height. Secondly, under each dimension, we aggregate representations of user-adopted items belonging to a specific group, e.g., shoes assigned to Nike group under band dimension. The output an array of low-level user interests towards multiple item characteristics. These low-level interests are ingredients to construct compositional (high-level) user interests. By this design, we separate the number of user interests from that of item space granularity. Thirdly, we introduce novel bi-directional binding block to better derive compositional user interests, which includes prototype competition explain low-level interests and low-level interests competition to attend high-level interests.
Contributions. The primary contributions of this paper are three-fold. First, we propose to discover item space structure under multiple-facet lens to better derive compositional user interests from uncovered item characteristics. Second, we introduce FACETVAE to improve VAE-bbased preference learning. Our novelties include identifying multi-faceted item structure, binding compositional user interests and bi-directional user interest binding. Third, we extensively conduct experiments on real-world datasets to demonstrate the kate-offthh--a recommendation accuracy of PACETVAE. In addition, we provide a qualitative analysis to ease the understanding of FACETVAE's inner working.
RELATED WORK
Multi-interest user modeling. The most popular method is item grouping, which has been explored for Collaborative Filtering (CF) al. (2019), CF with side information Tran Lauw (0022);
Wang et al. (2023a) and sequential recommendation Li et al. (2019); Cen et al. (2020) x Xiao et al. (2020) Tan et al. (222a)) a Zhang et al. 2022); Wang & Shen 2022) Wang et al. (202aa) The second method relies on representation learning on graph, L.e., DGCF Wang et al. (2020) divides user (item)
representation into K factors then uses routing mechanism aggregate information from neighbors to obtain multiple interests of user (item), DCCF Ren et al. (2023) leverages intent prototypes to aggregate global context information at a graph embedding layer. The third method projects user embedding vector into multiple spaces, each captures (2013): Tan et al. (b02 Bao et al. (2022). Our work falls into item grouping- based approach yet is more generalized by
aspect of their preferences Weston et al.)
disentangling and binding compositional user interests.
Disentangled representation learning. The idea is to discover the factors of variation underlying data Bengio al. 2013). This principle has been applied in MF et al. (2019); Waan al. (2020);
Ren et al. (2023). sequential recommendation Ma et al. (2020): Zheng et al. (2021 | 2022), side
information recommendation Zhang et al. (2020): Tran & Lauw (2022); Wang et al. 2023a
2

---

Published as a conference paper at ICLR 2024
citation recommendation Wang al. (2022c). bundle recommendation Zhao et al. (2022). Our work is close to MacridVAEEMMs et al. 2011). DGCF Wang et al. (2020) and DCCF DCCFFRen et al. (2023). The pro- posed FACET VAE and Macrid VAE have in common micro-discnte soccro-disnttagllmmeee
which does not appear in DGCF and DCCF, FACET generalizes Vac by disentangling item space under multi- faceted lens. Furthermore, FACETVAE binds low-level user's s interests into compositional ones, which is another novelty, FACETVAE also relates to disentangling representation' 's dimensions Higgins et al.) (2017); Kim & Mnih (0188; Chen et al. (2018); Locatello et al. (2019). Not only do we disentangle single vector representation, but we also discover multiple clusters of item space under multiple facets.
Multiple clusters discovery. Prior arts on discovering multiple clusters under data Jain al. (2008); Zhao et al. (2017): Wang et al. (2018: 2019b); Yao et al. (2019) optimize clustering objective, merely aiming at grouping data points and therefore, they are widely different from ours. For one, our clustering process is guided by recommendation objective. For another, prior works assume each data point is from a concept while we assume that each item interacts with different concepts (prototypes of facets) to derive representation for a data point (a user).
Binding problem in neural network. The idea to group relevant low-level features into semantically meaningful high-level ones Greff et al. (220)), which has some commonality with ours. related problem object-centric learning, aiming to derive representations for multiple objects in
the input image Locatello al. (2020); Singh et al. (2022) ; Chang et al. (2022); Singh et al. (2023);
Jia et al.) (023). Our motivation differs from those of these mentioned works as we aim to derive user's interests from their adoptions for collaborative filtering task.
PRELIMINARIES
Our problem follows the typical settings of collaborative filtering, including M users and N items. Let yf" = 1 be an observed interaction between user u and item i and yf" = 0 means no interaction has been recorded between the two. Let {i y/" = 1} be the set of interacted items of u. The target is to predict the probability that a user a will interact with an item based on u's past interactions.
Problem Formulation. There exist complex patterns driving user's item adoption behaviors. Thus, uncovering these hidden explanatory factors would not only enhance interpretability of user preferences but also provides pathway to improve recommendation task. As such, we seek for set of vectors z" = {z) E TR
d representing K interests of user and K is pre-defined number.
=k=1
Multi-interest modeling under VAE framework. We briefly describe representative work MacridVAE Ma et al. (2019) for illustration. MacridVAE involves three main steps, item grouping, user interest aggregating, and decoding and learning. First, set of K prototypes E kkxd is employed to group N items T E NxN into K groups, generating assignment matrix C E IR N Prototypes are randomly initialized and learned in data-driven manner. Second, given C and context matrix E E N Ndd" MacridVAE aggregates a uuse''s adopted items belonging to specific cluster to produce set of vectors (h)) Then an interest vector is sampled from Gaussian distribution with parameters estimated via neural network 42, i.e., (aY b ((hh)) Vk- 1,2, ..., K.
C <1
(a bb) = s Ener Cie R3
N ||
Idiag(ao exp( b)))
Viieso (Ca)²
is Gumbel-San Sumbel-SSfftax// Jang et al. (2017) )): Maddison al. 2017 to approximate one-hot vector, i.e.. if item belongs to cluster k then Cj R I and Cij & 0 v 7 k. is temperature hyper-parameter to obtain more skewed distribution. do is hyper-parameter with value around 0.1. Thirdly, decoder predicts score r(z)) then normalize to obtain probability of user-item interaction.
-T,T) T
EK Car(zz)
r r(z)) eap(
plu! (x,, C)
IN DK C(rr(Z))
Tdoc
Assignment matrix C used to weight the prediction, i.c., if item é probably belongs to th cluster, the predicted score of item i by 4th interest will be given the corresponding weight. The final prediction score is summed over predictions of K user interests. Finally, MacridVAE's learning objective includes two terms: cross-entropy loss to reconstruct observed user-item interactions and Kullback- Leibler divergence regularize variational distribution of user interests with prior distribution. We skip bias vector when calculating hü ease understanding.
3

---

Published as a conference paper at ICLR 2024
Lawelovel Intereat Represetatiioe
1
Unar's ratiigi
Group by locet 1
Aoroup
Pradicted eser's rolleg
Irroup
Facel
Internet Binding Block
Reprsseetations
Grropp 3
Group by focer 2
Group 2-11
Dax
High-koyel Interest Reproserzatine
Group tecet
Facet 3
Divergence Regu Regularization
Group 3-1 (
weightdd
Group
weig and moann
3-3
Bax
Reconstructio Objective
Figure 2: Architecture of FACETVAE. Input includes item IDs with shaded circles are adopted items and dashed ones are not adopted items. FACETVAL groups items into multiple clusters under multiple facets to derive low-level interests (dashed items not considered), then composes them into high- level one by binding block. KI. divergence regularization is to disentangle representation dimensions.
Limitations. Existing VAE methods that seek to discover user interests via prototype-based item grouping have three main shortcomings. First, multiple facets underlying item space are inadequately uncovered. As in Figure I it requires to uncover three facets (color, brand, top height) as well as facet-wise item groups, e.g., Nike vs. Adidas under brand facet, to fully model item space. Second, the assumption that the granularity of item space equals to that of user interests, causes a dilemma. On the one hand, large number of item groups required to model item space structure yet is exaggerated to model user interests as a user often only consumes small subset of item space. On the other hand, while a small number of item groups is reasonable for modeling user interests, insufficient capture item space structure, which might be multi-faceter as in Figure [ Third, the complexity of user interests has received inadequate attention. As multiple factors may influence a user's decision, the compositionality of user interests is crucial when deriving interest representations.
4 MULTI-FACETED PROTOTYPICAL USER INTERESTS LEARNING
Figure iillustrates FACETVAB. a solution for the mentioned shortcomings. The key innovations i) discovering item space structure along multiple facets; ii) binding compositional user interests from discovered item characteristics; iii) composing user interests via a bi-directional binding block.
4.1 MULTI-FACETED ITEM SPACE STRUCTURE DISCOVERING
We aim at capturing the granularity of item space to better infer user preferences, e.g., uncovering three facets (color, brand, top height) and their corresponding granularity, c.g., Adidas vs. Nike shoes under brand facet in Figure These facets assumed to be latent and to be uncovered unsupervisedly.
To realize, we infer a set of F matrices C {C assuming facets underlying item space. Fach C E RRNx represents clustering of N items to J clusters under facet f. Here, we assume item groups under each facet. item groups under each facet differs. Moreover, due to the inaccessibility to the prior information about facets behind data, we have assumed uniform facet distribution. This naturally holds as facets underlying item space simultaneously exists. Multi-faceted item grouping brings two salient benefits.
straightforward to apply our method when the number of
Efficiently modeling item space. As illustrated in Figure (a), one can use single facets to group items. However, it requires the exaggerated number of groups, which scales exponentially with the number of facets and the number of clusters per facet, i.c., J/ x J2 x x JF assuming J, groups under fth facet. Contrarily, multi-faceted item grouping only requires J1 + J2 + ... + Jr groups, a linear function of Jy-
4

---

Published as a conference paper at ICLR 2024
Composition of multiple facets. As being driven by many factors, user interests might include multi- ple item characteristics, c.g., white low top Nike shoes or green high top Adidas shoes. Multi-faceted item grouping enables composing such complicated interests from discovered items characteristies.
Concretely, we involve P E RFxxxd as the prototype collection of FF facets, Py E / be the prototypes under f facet. Prototypes are expected to convey a specific characteristic of item space. For example, under brand facet, there two prototypes Adidas and Nike. We assume these prototypes are latent. They are updated in a data-driven manner and then used to aggregate items having the same characteristic. In real-world scenarios where item category knowledge is available, one could embed semantic information into these prototypes by classifying them into the corresponding category labels. We perform facet-wise item grouping by estimating the assignment score of item to cluster under f as
Cys = ((( Sye2 Spol)s Syy = P// [T2
(P/5l12)
Following Ma al. 2019 Φ is Gumbel-Softmaat Maddison et al. (2017): Jang al. (2017) to approximate one-hot vector of the cluster distribution Cfi E R2 of item under facet J. szúj is based on cosine similarity between item embedding vector T; E Rd and prototype P3₁ E R and -1- L2 norm. Using cosine similarity prevents all items are associated to cluster with highest magnitude [|pri|le T is the temperature to concentrate the weight to the most similar cluster. By calculating Equation [or F facets and N items, we obtain assignment score matrix C E R FxNxJ
The output of Equation satisfies Σ/=1 Cạş 1 and Cfij 2 0. This naturally creates a competition between J clusters, i.e., they compete to other J clusters for attending to an item i. For example, under color facet in Figure [T] if green shoes are tied to cluster J, i.e., the assignment score of green shoes to cluster J is high, which results in lower assignment scores of green shoes to other clusters. Therefore, the nature of this competition enables grouping related items into meaningful clusters.
4.2 BINDING COMPOSITIONAL USER INTEREST REPRESENTATIONS
This section presents the derivation of user interests from the uncovered item space structure.
Low-level user interest representation. Intuitively, if a user adopts an item belonging to a specific group, is likely that they are interested in item characteristic captured by that group. motivating us to derive user interest representation based on item group clues. The output is called low-level as it is supposed capture single item characteristic, e.g., Nike shoes under brand facet.
21 Let fj E RRd" be low-level interest of user towards characteristic j under facet f. hý, is derived from user u's interacted items given cluster distribution C,, bias hene E RRd" activation function 7o
hi = 70 Σ Crj E./VZ (nC) with Σ (Crr))2
E, E Rd Ra is the context vector used to derive interests. For simplicity, a default setting dene d.
H
(h))) f= E F xxdx captures low-level user's interests underlying F facets.
F.J
High-level user interest representation. A user's decision driven by multiple factors, e-g., brand and/or color when they buy a new pair of shoes. Thus, modeling the composition of factors having influence on a user's decision is essential. We regard H as the ingredients to compose high-level user's interests. Formally, we employ set of K prototypes denoted by Q E kxd Q will retrieve low-level user interests from H and then compose them into high-level user interests.
Nevertheless, this is non-trivial task for a couple of reasons. For one, it demands dedicated mechanism that wisely binds low-level user interests. For example, low top and high top shoes should be assigned to different hegh-level interests as a pair of shoes do not simultaneously contain these two characteristics. For another, high-level interests are required to be distinct to capture the diversity of user preferences and alleviate the negative effect of noisy and redundant interests.
Bi-directional binding block. We present bi di-directional binding mechanism as a solution. Not only does it bind high-level user interests over low-level ones, but also enables competition between
5

---

Published as a conference paper at ICLR 2024
low-level interests to attend high-level counterparts. Given Q and H", our binding block works as |so Atmaa(ssmmm (simmmm) by w ((Qx))/TO) softmar(sim(un w ((Q))//T)
A 2 1,2..... K
1,2,...
L b ...
is a linear projection. sim(-.) is cosine similarity. Tu, syper-parameters.. is activation func- tion. Each facet has the same weight 1/VF x J, modeling prior belief of uniform facet distribution.
Afjx k is the binding score of low-level interest j under facet f to high-level interest k, consisting the softmax over K prototypes and the softmax over J clusters. Firstly, as output of softmax over K pro- totypes are non-negative and sums to I, this requires K prototypes to compete attend low-level inter- ests from H Thus, it encourages high-level interests to be different from others to capture the diver- sity of user's 's preferences. Secondly, softmax over J clusters under each facet f creates another petition between / clusters, which constraints each prototype to mainly bind to one item characteristic. This constraint enables wisely binding for more interpretable interests, e.g.. low top and high top shoes are tied to different high-level interests as is unnatural to combine these two features into one interest.
After obtaining high-level interest representation of u, calculate item-interest score matrix A
En Ctts
B ik
softmar( 1.2...,K
fjk
N.K B (B i E BRN
(4)
VF
Until now have C showing the relations between items and clusters and A capturing the relations between clusters and user u's interests. Therefore, B" in Equation
4 describes the relations between items and user u's interests, which is used in prediction step in Equation
4.3 MODEL LEARNING
Micro disentanglement, We follow the common practice in VAE literature Higgins et al. (2017); Ma et al. (2019) to derive micro-disentaagllmmen. i.e., disentangling dimensions of an interest gdd"
representation. Firstly, Vi in Equation 3 is processed by encoder g" R + R2d to estimate the parameters of variational distribution
f E Rd and o E Rd as following
1
(a b (v
1 / //|la|1l2:
of ou exp(
b
(5)
ao's value is around 0.1 Ma et al. (0119) The final representation of 4 interest of user u, i.c., zy E Rd, is sampled from Gaussian distribution with estimated parameters, i.c., z ~ N(p (d)ag(or)){) Yk: = 1,2,. 2..,, K. A regularization term based on Kullback- Leibler (KL) divergence is added to match the estimated variational distribution with prior distribution, ie.. (xz( C)llp(z")) In which, O[z C) I q(z))x", C) IT,,, N(u (diag(a))) is variational distribution and p(z") = NO0 (a is factorized prior distribution to achieve micro diicro-disentanglemen. Dxz(gllp) - when q and matches.
2 1= decoder predicts the probability that user v interacts with item as Decoder. Given {z
p(y)) ΣK
Dk=1
(z))
with r(z exp(simm T.)/Tacc)
(6)
E 2==1 (2
T; item i's vector. sém(.,)) cosine similarity. Tden temperature hyper- hyper-parameter, p(y)") is normalized over N items. B from Equation Tis used to weight the prediction of an item, i.e., higher weight is given to an item provided its higher level of similarity with current interest.
Learning objective. FACETVAE minimizes an objective summing over batch of user Buser
N
C= Σ [Cy -Ch
Σ
v"in(p(p")) + 3DxI(((z")'' C)llp(z")))
(7)
NEBuser
NE RESueer 1=1
For each user, LYecon is to reconstruct the observed interactions of user u via cross-entropy loss p(v.") follows categorical distribution. Cheg is regularization term described in Section 4.3
hyper-paramete 3 is introduced to control the influence of regularization objective versus The recommendation objective, similar to Multi-VAE Linnnng et al. (2088) and MacridVAF Ma al. (2019).

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
Datasets. We consider three real-world datasets: i) Moviel.ens-IM (MI.-IM (6,035 users, 3,126 movies, 574,376 ratings); ii) CiteULike- (5.551 users, 16.945 ariticles, 204,929 interactions); iii) Yelp 4 (29,111 users, 22,121 11 businesses, 1,052,627 reviews). We regard movies/ariclessbuuinesss as items and ratings/reviews interactions. The implementation can be found in the link https://gtthub.com htt.///ww..coeeeeeeeeeeea//eeeee
Competitors. We compare FACETVAE against closely related baselines, ie.. disentangled represen- (2119). DGCF Wang et al. (2020), DCCF Ren et al. (2023); multi-vector
tation Va al.
user representation DPCML Bao al. (2022); VAH-based Rec (hhnninn [Shenbnn et al. (2020). We also include recently developed Collaborative Filtering models, i.e., contrastive learning NCL Lin et al.)
(2022), SimGCL Yu et al. (2022), representation theory- DirectAU Wang et al. (2022b), graph- based UltraGCN Mao et al. (2021b), cosine-contrastive loss-based SimpleX Mao et al. (2021a).
Hyper-parameter settings. To ensure fair comparison, we tune the hyper-parametee of baselines fol- lowing original papers. Dimension d' = 64 and the number of user interests K = 4, where applicable, are fixed for all models. The details of paree-rrreetter tuning is presented in supplementary materials.
Metrics We report full ranking evaluation Zhao et al. (2020) of Recall and Normalized Discounted Cumulative Gain (NDCG) Tamm et al. (2021) as recommendation metrics. Both metrics are truncated at top 20 and top 50. Recall and NDCG are abbreviated as R and N in tables, respectively.
5.1 RECOMMENDATION PERFORMANCE COMPARISON
Table 1: Recommendation performance comparison. Boldfaced numbers are the highest while the runners-up are underlined. denotes statistical significance between the boldfaced and the underlined on a paired t-test with p-value < 0.01. R and N stand for Recall and NDCG, respectively.
CiteULike-a
Yelp
ML-1M
Model
R@00
R@:50
N@20
N@000
R@20
R@:00
N@20
500
R@20
R@50
N@:0
N@
SimpleX
0.2607
0,3857
0.1534
0.1865
0.1163
0.2116
0,0611
0.0857
0,2706
0.4262
0.1988
0.2486
NCL
0.2378
0.3654
0.1353
0.1688
0.1551
0.2624
0.0851
0.1130
0.2766
0.4409
0.2026
0.2557
DirectAU
0.2534
0.3842
0.1412
0.1751
0.1677
0.2763
0.0994
0.1274
0.2695
0.4209
0.1947
0.2428
SimGCL
0.2444
0.3706
0.1375
0.1703
0.1634
0.2711
0.0929
0,1207
0,2870
0,4537
0.2082
0.2623
UltraGCN
0.2575
0,3839
0.1497
0.1835
0,1114
0.2065
0.0576
0.0820
0.2836
0.4467
0,2079
0,2603
DGCF
0.2066
0.3348
0.1165
0.1499
0.1337
0.2375
0.0713
0.0981
0.2598
0.4228 0.1890
0,2414
DOCF
0.2210
0,3536
0.1245
0.1589
0.1537
0,2573
0.0881
0.1150
0,2694
0.4360 0.1990
0,2523
DPCML
0.2498
0.3770
0.1429
0.1759
0.1125
0.2094
0.0576
0.0824
0.2754
0.4319
0,2050
0.2553
RecVAE
0.2398
0.3481
0.1435
0.1725
0.1137
0.2104
0.0596
0.0844
0.2952
0,4607
0,2146
0.2678
MacridVAE
0.2744
0.3974
0.1632
0.1958
0.1794
0,2835
0.1134
0,1405
0.2925
0,4553
0,2138
0.2669
FACETVAE
0.2837
0.4100
0.1695 0.2029
0.1859
0.2901
0.1192
0.1463'
0.2968
0.4586 0.2178 0.2707
Table reports the recommendation accuracy of FACETVAE and baselines. The results show that FACETVAL VAE achieves significantly higher accuracy than those of baselines CiteULike-a and Yelp. On ML,IM, FACETVAF. demonstrates better performance than ReC, the best baseline, w.r.t. out of metries. Notably, FACETVAE only requires roughly 1/3 of RecVAE's number of parameters 1~1.22 vs. ~3.6M) to get the reported results, demonstrating the efficiency of FACETVAE.
There are two key takeaways. First, VAE-based models are top performing on all chosen datasets, demonstrating the strength of VAE framework that FACETVAE inherits. For example, on CiteULike-a and Yelp, MacridVAE achieves much better performance than both ing models (DGCF, DPCML, DCCF) and single representation models (SimpleX. DirectAU, SimGCL, inter alia). On ML.IM,, despite modeling user interest as a single vector, Rec VeE outperforms DGCF, DCCF and DPCML. which disentangle multiple factors of user interests. Second, prototype- hased rep- resentation learning plays the key role in modeling multiple user interests under VAE framework. This is evidenced by the markedly performance gap between MacridVAE (using protoyyp-bbase represen- tation) and Rec VAE (without prototype- prosstype-based based representation) on CiteULike-a and Yelp. FACET VAE
sasrunngledmmuuiiittteees model-
htwp://p/ww.oooollaa//////oooooo http:///aanghao.nn/DD...mm hhtps://wwww.yep.coo/aaeee
7

---

Published as a conference paper at ICLR 2024
Figure 3: Item space structures produced by FACETVAE on ML-IM. Each plot shows multiple clusters of facet. Items are colored according to arg max, Cty in Equation It can be seen that green and purple groups of the first facet (the first plot) are grouped into one cluster (genoa lemon) in the second facet (the second plot). Contrarily, the brown group of the first facet is divided into black and blue groups in the second facet. In the third facet, these two are clustered into orange group.
generalizes MacridVAE via disentangling prototype-baser based representation under multi-faceted lens, achieving higher accuracy than both MacridVAE and RecVAE on three datasets.
5.2
MODEL STUDIES
Efficiency Analysis. FACETVAE and the closest baseline MacridVAE are bounded by the computa- tional cost of grouping N items into clusters. While it is O(K N) complexity in MacridVAE given K clusters, that of FACETVAE is O(FJN + FJK). In which OF (JF))) N) is the complexity of grouping items under F' facets each has J clusters and O OFFJK) is the complexity of binding block. Despite requiring higher computational demand, i.e., F is larger than K, FACETVAE's complexity is still a linear function of number of items N as binding block's s complexity does not depend on N. More importantly, PACETVAE achieves significantly higher accuracy than MacridVAE thanks to multi-faceted item grouping. To verify, we report the running time of FACETVAE and MacridVAE on Yelp dataset, On the other datasets, the running time of two models are roughly the same. For Macrid K is 16 as it produces the best results. For FACETVAE, K is 8 achieving higher Recall and comparable NDCG compared to MacridVAE (as presented in supplementary materials). Then, the training time (second/epoch) and inference time (second) of FACETVAE are 10.068s and 2.31 Is, respectively. Those of MacridVAE are 9.736s and 1.963s, respectively. Clearly, FACETVAE only requires slightly higher running time than MacridVAE yet achieves better overall performance.
Table 2: Multi-faceted item space disentangling. Given F J 12 prototypes, the total number of item characteristics is J Boldfaced number is the best while runner-up is underlined per column.
Setting
ML-1M
CiteULike-a
Yelp
F
R@20
R@50 N@20
N@50
R@20
R@50 N@20 N@50
R@:20
R@50
N@20
N@50
1
0.2936
0.4577 0.2110
0.2646
0.2824
0.4069 0.1692 0.2022
0.1804
0.2857
0.1146
0.1419
2
0.2951
0.4579
0.2131
0.2663
0.2819
0.4077
0.1688
0.2021
0.1848
0.2912 0.1180
0.1456
3
0.2968
0.4575 0.2168 0.2693
0.2820
0.4076 0.1690 0.2023
0.1856
0,2912 0.1189
0.1463
4
0.2964
0,4570 0.2173 0.2697
0,2806
0.4086 0.1688 0.2026
0,10
0.2885 0.1200
0.1470
6
2
0.2934
0.4546 0,2152 0.2676
0.2778 0.4040 0.1661 0.97
0820
0.2831 0.1191
0.1455
Multi-faceted item space disentangling. We fix the total number of prototypes used to group items is 12 then we vary F and J satisfying F x J = 12. When F 1, it reduces to faneeefacettd item grouping. Table 2 pressents the results. The key takeaways are first, grouping item space under multiple facets, i.e., > 1, results in overall higher recommendation accuracy, demonstrating its ability to discover fine- fine-grained structure of item space. Second, setting F and J of which J is large, e.g., F = 3, J = 4 or F' = 4, = 3, generally results in better performance. These results are consistent with our hypothesis in Section 4. that the number of item characteristics FACETVAE can discover is up to JF. The more item characteristics are discovered, the better modeling user interest is.
Item space disentangling visualization. We visualize the item groups produced by FACETVAE in Figure 3 to qualitatively examine whether FACETVAE can discover multi-facete item space structure. We use t-SNE van der Maaten & Hinton (008 to visualize item representations
8

---

Published as a conference paper at ICLR 2024
2D space. Evidently, the neighbors of items, i.e., those with the same color, across facets vary, demonstrating that FACETVAE is capable of disentangling multi-faceted item space.
Table 3: Analysis of low-level and high-level user's interests. F=J=3 for CiteULike-a and ML-IM while F = = 4 for Yclp. Boldfaced number is the highest in each column. (*) refers to Equation
User Interest
Binding block
Number user interests
ML-IM
CiteULiike-
Yelp
setting
R@20
N@
R@ 20 N@
R@20
N@
low lovel
removed
F x J
0.2826 0.2091
0.2710 0.1628
0.1835
0.1242
softmax over K prototypes only (*) softmax over J clusters only (*)
4
0.2962 0.2148
0,2685
0.1590
0.1818
0.1159
4
0.2908
0.2133
0.2820
0.1692
0.1794
0.1249
high level
bi-directional
4
0.2968
0.2178
0,2837
0.1695
0.1859
0.1192
bi-directional
8
0.2948 0.2171
0,2837 0.1696
0.1846 0.1214
Low-level vs. high-level user interests. Table 3 presenns results when using low-level and high-level user interests (Section 4.2) for recommendation. Clearly, leveraging high-level user interests results in higher accuracy as they are more expressive than low-level ones. For instance, using 4 high-level user interests leads to much better accuracy than using 9 low-level user interests and ML-IM. On Yelp, we observe the same trend yet it depends on the setting of binding block.
ULike-a
Analysis of binding block. We also study binding block via reported results in Table Firstly, bi-directional binding obviously achieves larger performance than uni uni-directional counterpart, i.e., per- forming softmax over K prototypes J clusters only. Secondly, bi-directional binding's 's effect is data- dependent. While softmax over K prototypes has stronger influence on MI. than softmax over clusters, we observe the opposite trend on CiteULike-a. On Yelp. this effect is metric-dependent.
Interpretability of user's interests. We study the interpretability of user's 's interests produced by FACETVAE. After training, we retrieve three items with highest score predicted by each interest of a user (see Equation 6 in Table 4 These examples suggest that FACET VAE has the potential to discover the multiple interpretable interests of users. However, we note that user's interests are derived in an unsupervised manner, which may result in one interest with many items dominates the less popular ones or ambiguous interests.
Due to limited space, we present more experimental results to understand FACET VAE, including analysis of number of user interests K, values of F and J versus recommendation performance and the influence of micro-disentaagllemet recommendation accuracy, in supplementary materials.
6 CONCLUSION
We introduce FACETVAE to resolve shortcomings of VAE-based disentangled recommendation models, including inadequately item space discovering, same level of granularity between user interests and item space assumption, which causes a dilemma and improperly user interest complexity handling. FACETVAE is characterized by three main innovations 1) disentangling item space under multi-faceted manner, 2) binding compositional user interests from low-level ones discovered from item space and 3) effectively binding user interests via bi-directional binding block. Future work extending FACETVAE includes improving the efficiency of multi-faceted item grouping and discovering the number of facets and the number of clusters per facet in a data-driven manner.
Table 4: Top three items (movies) with highest score predicted by multiple interests of a ML-IM. We present movie's title and tags (inside parentheses). Three interests have their own semantics, i.c., Star Wars series, comedy movies, and drama movies, respectively.
Interest
Interest 2
Interest
1. Star Wars: Episode V The Empire Strikes Back
1. High Fidelity (Comedy)
American Beauty
(Action Adventure Drama Sci-Fi War)
(Comedy | Drama)
2. Star Wars: Episode V1 Return of the Jedi
2. Wonder Boys
2. Braveheart
(Action Adventure Drama Sci-Fi War)
(Comedy Drama)
(Action Dramal War)
3. Star Wars: Episode IV A New Hope
3. American Beauty
3. The Shawshank Redemption
(Action Adventure Drama Sci-Fi War)
(Comedy Drama)
(Drama)

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research/project is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: MISG2-RP-2021-022),
REFERENCES
Shilong Bao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. The minority matters: A Hiiversty-proomoote collaborative metric learning algorithm. In NeurlPS, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35: 15:1798-1825 2013.
Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. Controllable multi-interest framework for recommendation. In KDD, pp. 2942-2951, 2020.
Michael Chang, Thomas Griffiths, and Sergey Levine. Object representations as fixed points:
Training iterative refinement algorithms with implicit differentiation. In NeurlPS, 2022.
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders, 2018. http::t:: htl.apppss/bpeeeeeeeeeeee///rrrr
BJdMRoCIF
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. CoRR, abs/2012.05208, 2020. URL /RL./tttss///xxxvoorr//////
05208
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation. In SIGIR, pp. 339-648. 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Prateek Jain, Raghu Meka, and Inderjit S. Dhillon. Simultaneous unsupervised learning of disparate clusterings. In Proceedings of the 2008 SIAM International Conference on Data Mining (SDM), pp. 858 869, 2008.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterization with gumbel-softmax. In 5th International Conference Learning Representations, 2017.
Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization.
In The Eleventh International Conference on Learning Representations, 2023. URL https
/openreview net wwwp:eiwwwwwee//truuune...e
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. Proceedings of the 35th International
Conference Machine Learning, pp. 2654 2663, 2018.
Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. Multi-interest network with dynamic routing for recommendation at tmall. In CIKM, pp. 2615-2623, 2019.
Dawen Liang, Rahul G. Krishnan, Matthew Hoffman, and Tony Jebara. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference, pp. 689-698, 2018.
Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. Improving graph collaborative filtering with neighborhood-emit enriched contrastive learning. Proceedings of the ACM Web Conference 2022, pp. 2329, 2022.
10

---

Published as a conference paper at ICLR 2024
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentan- gled representations. In Proceedings of the 36th International Conference Machine Learning, pp. 4114-4124, 2019.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In NeurlPS, volume 33, pp. 15525-1158,, 2020.
Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu. Learning disentangled representations for recommendation. In Advances in Neural Information Processing Systems, volume 32, 2019.
Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu. Disentangled self- supervision in sequential recommenders. In Proceedings the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 483-491, 2020.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables. In 5th International Conference Learning Representations, 2017.
Kelong Mao, Jieming Zhu, Jinpeng Wang. Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiuqiang He. Simplex: A simple and strong baseline for collaborative filtering. In CIKM, pp. 1243-1252, a021a.
Kelong Mao, Jieming Zhu, Xi Xiao, Biao Lu, Zhaowei Wang, and Xiugiang He. Ultragen: Ultra simplification of graph convolutional networks for recommendation. In CIKM, pp. 1253-1262, 2021b.
Xubin Ren, Lianghao Xia, Jiashu Zhao, Dawei Yin, and Chao Huang. Disentangled contrastive collaborative filtering. SIGIR, pp. 1137-1146, 2023.
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th International Conference on World Wide Web, WWW '15 Companion, pp. 111-112, 2015.
Ily Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I. Nikolenko. Reevae: A new variational autoencoder for top-n recommendations with implicit feedback. In WSDM, pp. 528-536, 2020.
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-c learns to compose. In International Conference on Learning Representations, 2022.
Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. In The Eleventh International Conference Learning Representations, 2023.
Yan-Martin Tamm, Rinchin Damdinov, and Alexey Vasilev. Quality metrics in recommender systems: Do we calculate metrics consistently? In Fifteenth ACM Conference on Recommender Systems, pp. 708-713, 2021.
Qiaoyu Tan, Jianwei Zhang, Jiangehao Yao, Ninghao Liu, Jingren Zhou, Hongxia Yang, and Xia Hu. Sparse-interest network for sequential recommendation. In WSDM, pp- 598-606, 2021a.
Y. Tan, C. Yang, X. Wei, Y. Ma, and X. Zheng. Multi-facet recommender networks with spherical optimization. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 1524-1535, 2021b.
Nhu-Thuat Tran and Hady W. Lauw. Aligning dual disentangled user representations from ratings and textual content. In KDD, pp. 1798-1806, 2022.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (116):2779260555 2008.
Chenyang Wang, Zhefan Wang. Yankai Liu, Yang Ge. Weizhi Ma, Min Zhang, Yiqun Liu, Junlan Feng. Chao Deng, and Shaoping Ma. Target interest distillation for multi interest recommendation. In CIKM, pp. 2007-2016. 2022a.
11

---

Published as a conference paper at ICLR 2024
Chenyang Wang, Yuanqing Yu, Weizhi Ma, Min Zhang, Chong Chen, Yiqun Liu, and Shaoping Ma. Towards representation alignment and uniformity in collaborative filtering. In KDD, pp. 1816-1825, 2022b.
Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. Neural graph collaborative filtering. In SIGIR, pp. 165-174, 2019a.
Xiang Wang. Hongye Jin, An Zhang, Xiangnan He, Tong Xu, and Tat-Seng Chua. Disentangled graph collaborative filtering. In SIGIR. pp. 1001-1010, 2020.
Xin Wang, Hong Chen, Yuwei Zhou, Jianxin Ma, and Wenwu Zhu. Disentangled representation learning for recommendation. IEEE TPAMI, 45(1):408-424, 2023a.
Xin Wang, Zirui Pan, Yuwei Zhou, Hong Chen, Chendi Ge, and Wenwu Zhu. Curriculum disentangled representation learning across multiple environments for social recommendation, In ICML., 2023b.
Xing Wang, Guoxian Yu, Carlotta Domeniconi. Jun Wang. Zhiwen Yu, and Zili Zhang. Multiple co-clusterings, In 2018 IEEE ICDM, pp. 1308-1313, 2018.
Xing Wang. Jun Wang, Carlotta Domeniconi, Guoxian Yu, Guoqiang Xiao, and Maozu Guo. Multiple independent subspace clusterings. AAAI, 33(00)):3333-3333333 2019b.
Yifan Wang, Yiping Song, Shuai Li, Chaoran Cheng, Wei Ju, Ming Zhang, and Sheng Wang. Dis- encite: Graph-based disentangled representation learning for context-specific citation generation. In AAAI, volume 36, pp. 11449-14458 2022c.
Zhikai Wang and Yanyan Shen. lime-aware iulti-interres capsule network for sequential recommen- dation. In SDM, pp. 558-566, 2022.
Jason Weston, Ron J. Weiss, and Hector Yee. Nonlinear latent factorization by embedding multiple user interests. In ACM RecSys, pp. 65-68, 2013.
Zhibo Xiao, Luwei Yang, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. Deep multi-interest network for click-through rate prediction. In CIKM, pp. 2265-2268, 2020.
Shixin Yao, Guoxian Yu, Jun Wang, Carlotta Domeniconi, and Xiangliang Zhang. Multi-view multiple clustering. In IJCAI, pp. 4121-4127, 2019.
Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoe Viet Hung Nguyen. Are graph augmentations necessary? simple graph contrastive learning for recommendation. In SIGIR, pp. 1294-1303, 2022.
Shengyu Zhang. Lingxiao Yang, Dong Yao, Yujic Lu, Fuli Feng. Zhou Zhao, Tat-seng Chua, and Fei Wu. Re4: Learning to re-contrast, re-attend, re-construct for multi-interest recommendation. In The Web Conference, pp. 2216-2226, 2022.
Yin Zhang, Ziwei Zhu, Yun He, and James Caverlee, Content-collaborrtiive disentanglement repre- sentation learning for enhanced recommendation. In ACM RecSys, pp. 43-52, 2020.
Handong Zhao, Zhengming Ding, and Yun Fu. Multi-view clustering via deep matrix factorization. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), 2017.
Sen Zhao, Wei Wei, Ding Zou, and Xianling Mao. Multi-view intent disentangle graph networks for bundle recommendation. AAAI, 36:4379-4487, 2022.
Wayne Xin Zhao, Junhua Chen, Pengfei Wang, Qi Gu, and Ji-Rong Wen. Revisiting alternative exper- imental settings for evaluating top-n item recommendation algorithms. In CIKM, pp. 2329-2332, 2020.
Yu Zheng, Chen Gao, Xiang Li, Xiangnan He, Yong Li, and Depeng Jin. Disentangling user interest and conformity for recommendation with causal embedding. In Proceedings of the Web Conference 2021, pp. 2980-2991, 2021.
Yu Zheng, Chen Gao, Jianxin Chang. Yanan Niu, Yang Song. Depeng Jin, and Yong Li. Disentangling long and short-term interests for recommendation, In Proceedings of the ACM Web Conference 2022, pp. 2256-2267, 2022.
12

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTARY MATERIALS
A.1 DATASETS
We conduct experiments on three real-world datasets. MovieLens-IM (ML-1M (6,035 users. 3, 126 items, 574,376 interactions) contains user's 's ratings for movies. A review that a user wrote for item considered an interaction between the two. CiteULLkee=$( (5,551 users, 16,945 items, 204,929 interactions) contains academic articles and users' log history. An interaction between user and an item (article) is user saving an article to their collection. Yelp (21p](9.11)) items, 1,052,627 interactions) contains reviews (interactions) written by users for businesses (items). Due to huge volume, we only consider interactions from 2016 onwards. We pre-process ML.-MM following Ma et al. (2019) and Yelp following Lin al. (2022) and keep CiteULike-a dataset as the original. We construct training, validation and test sets by randomly dividing users' interactions with ratio 8:1:1, respectively. All cold-starts users and items in validation and test sets are discarded.
(29.111 users, 22, 121
A.2 IMPLEMENTATION DETAILS
Algorithm presents detailed training procedure of FACETVAE.
The code, data and related materials can be found in the link https://github.com P PreferredhT/EacetAA
A.3 EXPERIMENTS
A.3.1 DETAILS ON COMPARATIVE METHODS
MacridVAE Ma et al. (2019) disentangles macro and micro levels of user's interests based on
3-VAE.
DGCF Wang et 2020) iteratively refines multiple factor representations of users and items
to disentangle their multiple interests from interaction graph.
DPCML Bao et al.
(2022) improves Collaborative Metric Learning by using multiple
representations for
Shenbin et al. (2000) improves VAE-based CF model by presenting novel composite
prior distribution, new method set /3 in (- 3-VAE and an alternative training algorithm.
DirectAU Wang al. 2022b) improves representation learning in CF by optimizing uniformity
and alignment.
NCL. et al. 20022) enhances graph-based CF by incorporating structural and semantic neighbors via contrastive learning.
SimpleX Mao et al (2021a) improves collaborative filtering by designing novel cosine contrastive
loss and incorporating large negative sampling.
SimGCL Yu et al.
(2022) proposes novel contrastive learning approach for CF by augmenting random noise to representations and regulating their uniformity.
UltraGCN Mao al. (2021b) approximates message passing limit of message passing layer and leverage item-item relation information to improves graph-based CF
DCCF Ren et al. (2023) leverages an adaptive approach for self-supervised augmentation to disentangles intents behind user-item interactions.
A.3.2 EYPER-PARAMETER SETTINGS
For FACETVAE, tune the hyper-parameters in the same range MacridVAE. Dropout rate is 0.5. do {0.05, 0.075, 0.1). 3 min(33, T with 8g {0.01, 0.05, 0.1, 0.2, 0.5, 1} and update the number of model's parameters updates, T E {0.1k, 0.5k, 1k, 5k, 10k, 20k}. 5 e [0.1, 0.2], Tdec {0.1, 0.15, 0.2), To (0.1, 0.2, 0.5, 1, 5, 8}. y is tanh while to is Le..)))LU(033) for
http:://groonlleeonnng/assstttto org/danases/mooiilenns/
"http///wanghao.in/CCC wttp.ngghhhh.innnn..mmmm https:/wwww.ecpp.co/daa.s
13

---

Published as a conference paper at ICLR 2024
Algorithm 1: Training procedure of FACETVAE
Input:
User interacted items x" {i ya" 1}
Model parameters θ
item matrix in decoder E RR-xd context matrix E Nxx"" and bias vector he
clustering prototype representations Fxxxx and binding prototype representations Q E RR Kxd
parameters of lincar projection layer w IRd + Rdt and neural network g Be 7 2d
Number of facets number of clusters per facet J, number of interests K
Output:
Updated +
Function Sulti-faceted item grouping (T, P) f= I to F do
for N do
CJa Gumbel (/1 with 8/6 =
(T.) (T.)" (P///(-///T/ll [|srslla) Vj 1,2,
return C RR
Function Low-level interest aggregation(s", C
for / to F do
for 1 to do
h;; 7o Eicoo Crty bene)
with == EiE (Csa)) // To linear
V2
activation function
return H = = J, H BBDXPxJXd" // batch
10
sizc
Function Binding block(HH* (h 1 و-
12
for k= to do for f = to F do
13
for j = to J do
A |ssottmaxx (sim(b)) ((Q)))/T)) softmaxj(sim(I),, ((Q.)))1))) // sim(:s) is cesine similarid
V2 E JP Σ ) //
non-linear activation function Vi 1,2,..,,
B softmaxx En Cra 777
17
VPXJ
return V = fritke E R Bxd
B" BxNXK // B batch size
18
19
Function Encoder(V"
(
29
for k = to K do
21
(al by) 9 (v²)
22
e* = / |2
of eap( c bb))
N( [diag (
return k E Bxdd
25
26
Function Decoder ( B") for k= to K do
27
r(z%) eapppsmmmm (rmpsim((Z)... T sim(.,)) is cosine similarity
ply?) KK R r())) E r(g)
29
Vi 1,2..., N
30
return (p(y"))
31 C Multe faceted item grouping (T,
32
Low level interest aggregation(x" C)
33 4 B" + Binding block(H")
34 R + Encoder(V)
35 (p(y)) ( <- Decoder (22"))(((()))))))))
36 Calculate loss Euegueer DiN
-
of (nl(((()))) (1
37 Update 2 to minimize L
14

---

Published as a conference paper at ICLR 2024
0.285
0.296
0.186
0.270
0.292
0.182
0.255
0.258
0.179
8 101216
6 10 12
16
4 6 101116
K
K
K
0.170
0.217
0.126
0.160
0.214
0.119
0.210
0.112
0.150
4 6 12
2 4 10
2 8 101216
K
K
K
(a) Results CiteULike-a
(b) Results on ML- IM
(c) Results on Yelp
Figure 4: Performance of FACETVAE (red line) w.r.t number of interests K. We include results of MacridVAE (blue line) for contrasting. Small circles are Recall @ 20 while triangles are NDCG @ 20.
ML-1M and tanh for other datasets. de = d = 64 for CiteULike-a and Yelp and d = 300 for ML-IM. F on NVIDIA RTX 2080 Ti GPU machine ten times with different random seeds. For FACETVAE, we set maximal number of epochs is 200 and stop training after 15 epochs without improving Recall@221 on validation set. Averaged results over ten runs test set are reported.
for CiteULike-a and ML.1IM and F
for Yelp. All models are trained
A.3.3 ADDITIONAL RESULTS
Number of user's interests K. We analyze performance of FACETVAD and MacridVAE w.r.t. various numbers of user's interests K in Figure 4 There three key takeaways. First, FACETVAE achieves higher recommendation accuracy than MacridVAE across various values of K w.r.t. most of the metrics, except NDCG@20 20 with K 16. These evidences ascertain that multi-faceted item space disentangling adopted in FACETVAL can discover more useful item characteristics for modeling user interests. Second, FACET VAE is robust to the number of user interests, which is evidenced in Figure (a) and (b), i.e., FACETVAE's performance does not get hurted when increasing the number of user interests. Contrarily, Macrid assumes the number of user's 's interests equals to the number of item groups, which causes representation dilemma, i.e., small number of item groups might be sufficient to capture user's interests yet insufficient to model item space structure, while large number of item groups might be sufficient to describe item space but might cause redundant and noisy user interests. This dilemma negatively affects model performance, which is shown in Figure 4 (a) and (b) CiteULike- and ML.11M, M,, respectively. Third, FACETVAL E demonstrates its efficiency when dealing with large dataset. It is clear in Figure (c) that FACET VAE with K = 4 achieves higher Recall @ 20 than MacridVAE with K 10, K 12 K 16 on the largest dataset Yelp. A similar trend is also observed on NDCG @ 20 when setting K = 6 or K = 8 for MacridVAE and K = 4 for FACETVAB It is note that setting a large value of K prohibitively expensive for huge datasets when predicting interactions between users and million of items as the complexity scales with K.
Analysis of F and J. We investigate model performance w.r.t. various values of the number of facets F and the number of item groups under each facet J in Figure 5 The key observations are first, the values of F and J are data-dependent, e.g., on CiteUl and ML, setting F and achieves highest results while on Yclp Recall favors F = 3 and J = 4 or J 5 and NDCG prefers F = 5 and J 4. Second, small datasets prefer small values of F and while large dataset favors large values of these two. This observation is expected as large data is usually more complicated in terms of item space structure. Last but not least, large values of F and J might result in redundant and noisy discovered item characteristics, which negatively affecting model performance.
Miiroo-dsentaanlleeen vs. Recommendation Accuracy. As described in the main text, 3 controls the level of disentanglement between dimensions of representation vector. The value of 3 follows
15

---

Published as a conference paper at ICLR 2024
82000
02779
3.290
0,208
12008
2005
0.093
2
02800
0LITON
12499
2280
2280
2.222
0.307
02220
0.231
0.7023
02451
3300
CUNG2
2,80
5200
(a) Recall 20 on CiteULike-a
(b) Recall 20 on ML-IM
(c) Recall 2: Yelp
B12041
1 Paacete 179
0.106)
0.1000
0.107O
0.007
320
0.1013
630
E2007
02209
5.UM
0.1683
0.106
0.160
0.164
0.0007
0,1000
0.1083
0.107
0.7144
8:2100
02337
3220
03399
0.100
0.1171
011181
Number of clasters por Racet Q
Number of chasters per facet 01
Mamber of chasters per facet UI
dd NDCG@D22 20 on CiteULike-a
(c) NDCG@20 on ML-IM
(I) NDCG@20 on Yelp
Figure 5: VAEESs performance w.r.t. F and J.
0.285
0.2976
0.1860
0.280
0.2970
0.1855
0.2964
0.275
0.1850
0.1k
0.5K
5k
10k
20k
0.1k
0.5%
20k
0. 0.1k
0.5k
Sk
10k
20k
T
T
T
0.170
0.2180
0.1196
0.165
0.2177
0.1192
0.2175
0.1188
0.160
0.1k
0.5k
5k
10k
20k
0.1k
0.5k
5k
10k
20k
0.1k
0.5
5k
10k
20k
(a) Results on CiteULike-a
(b) Results on ML-IM
(c) Results on Yelp
Figure 6: Performance w.r.t. number of annealing steps T. The higher T is, the lower level of disentangiement is. Circle symbols are Recall@20 while triangle symbols are NDCG@20.
an annealing procedure, i.e., min update) as Ma et al. (019)). 33 the maximal value of ,, updale is the number of model's parameters update by gradient descent and T is the total of annealing steps. We fix 3o as default choice and vary the values T. Clearly, increasing T results in lower level of disentanglement. We present model performance w.r.t. T in Figure 66 There are two key takeaways. For one, we observe that on CiteULike-a, increasing to lower disentanglement level leads to improvement in both Recall and NDCG. Contrarily, ML-1M and Yelp not sensitive to micro- disentanglement. For another, there is a trade-off between Recall and NDCG w.r.t. disentanglement level on these datasets. In general, three chosen datasets prefer low level of disentanglement, T > 1k, to achieve good performance w.r.t. Recall and NDCG. In conclusion, these evidences imply that the strong performance of FACETVAE mainly comes from the binding compositional user interests, verifying the effectiveness of our approach in this paper.
item disentangling and
16


Reviews:
[{"review_id": "BtDUUlJcMS", "reviewer": "Reviewer_Ruv4", "summary": "This work propose to learn multi-faceted prototypical user interests based on VAE framework. The proposed method is technically sound, although it can be viewed as a natural extension based on MacridVAE, which limits the technical contribution somehow. The paper is well written and easy to follow, related work is well discussed and the extensive experiments and ablation studies verify the model performance.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well motivated to solve the multi-facet user interest modeling problem based on VAE framework. The proposed method is well motivated and presented.\n2. This paper is well written and easy to follow, especially the related work and limitation discussion in preliminaries.\n3. Extensive experiments are conducted, with state-of-the-art methods as baselines. The results demonstrate the improvement.", "weaknesses": "1. In real world recommender systems, different product categories will have different facets, in total, it can reach up to thousands of facets in the recommendation space, especially when we want to do whole website recommendation. It's unclear how this method can handle and scale to large scale product facets.\n2. This work can be viewed as an incremental upon MacridVAE, which limits the technical novelty contribution.\n3. The performance on large scale (millions of users and items) datasets is unclear.\n4. How to leverage product knowledge/other knowledge source to automatically choose #facets should be discussed to make it applicable in real world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRTwYakCsX", "reviewer": "Reviewer_BBhv", "summary": "This paper presented a new approach to uncover latent user interests for recommender system. Specifically, it proposed FACETVAE which tries to align user interests with multi-facted item characteristics and prototype-based user representation. It tries to separate granularity of user preferences from that of item space. WIth a dedicated bi-directional binding block, it demonstrates better recommendation performance compared to other baseline VAE approaches.", "questions": "1. For Table 1, what is the number of clusters and the number of parameters for the baseline MacridVAE? \n\n2. Is it possible to report the performance with respect to the number of parameters, and computation time across the baselines and the proposed approach? That seems to be able to give readers a better understanding of accuracy and efficiency.\n\n3. In Table 2, why is the number of prototypes fixed to 12 instead of other numbers? Can the best F and J have different trends with different number of prototypes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well organized and written.\n2. The paper is well motivated.\n3. The paper proposed a mathematically solid approach and demonstrated its clear strength over baselines.", "weaknesses": "The proposed approach has worse complexity than baseline MacridVAE. For real-world recommendation problems with huge spaces of users and items, the approach seems less applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HdSH5WWbmv", "reviewer": "Reviewer_JT1U", "summary": "This paper proposes FACETVAE to align user interests with multi-faceted item characteristics. The paper puts forward low-level and high-level user interests and introduces a bi-directional binding block to learn compositional user interests. Rich experimental results demonstrate the  good performance of FACETVAE.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well motivated and the idea of learning multi-faceted prototypes is interesting.\n\n- The paper conducts comprehensive experiments that encompass various aspects such as prediction performance, visualizations, parameter studies, and case studies.\n\n- The authors have made their code and dataset publicly available.", "weaknesses": "1. The proposed method is somehow incremental and is a combination of existing techniques. This work is based on an existing work that learns prototypical representation of items and users that seperate items into different groups. This work further incorporates disentangled representation learning to explore item representations of various aspects of items, which has been also considered in previous works[1]. \nThe main difference appears to be  the combination of these various aspects.\n2. Figure 2 is confusing because it takes explicit user-item interaction in an rating form. However, the paper is designed based on implicit user-item interaction. This misleads the reader on the output form and training strategy of the proposed method. \n3. Would it be better to include item attribute information in prototype learning as auxiliary information? Without low-level attributes as input, the so-called low-level user interest about a facet might also be a composite feature of some facets. This could result in the difference in meaning between the two levels of interests not being clear.\n4.  More qualitative analysis and case studies about on how learned latent representations corresponds to different aspects of characteristics may help to better evaluate the interpretability of the model. For example, more details on how the \"latent interest\" corresponds to explicit interests 1, 2, and 3 in Table 4 would be appreciated.\n\n[1] Learning Disentangled Representations for Recommendation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)