TotalRecall: A Bidirectional Candidates Generation Framework for Large Scale Recommender \& Advertising Systems
Authors: Qifang Zhao
Yu Jiang
Yuqing Liu
Meng Du
Qinghui Sun
Chao Xu
huan xu
Zhongyao Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: Recommender System
Advertising System
Collaborative filtering
Matrix Factorization
Contrastive Learning
Candidates Generation
Embeddings

Abstract: Recommender (RS) and Advertising/Marketing Systems (AS) play the key roles in E-commerce companies like Amazaon and  Alibaba. RS needs to generate thousands of item candidates for each user ($u2i$), while AS needs to identify thousands or even millions of high-potential users for given items so that the merchant can advertise these items efficiently with limited budget ($i2u$). This paper proposes an elegant bidirectional candidates generation framework that can serve both purposes all together. Besides, our framework is also superior in these aspects: $i).$ Our framework can easily incorporate many DNN-architectures of RS ($u2i$), and increase the HitRate and Recall by a large margin. $ii).$ We archive much better results in $i2u$ candidates generation compare to strong baselines. $iii).$ We empirically show that our framework can diversify the generated candidates, and ensure fast convergence to better results.

Under review as a conference paper at ICLR 2022
TOTALRECALL: A BIDIRECTIONAL CANDIDATES GENERATION FRAMEWORK FOR LARGE SCALE REC- OMMENDER & ADVERTISING SYSTEMS
Anonymous authors Paper under double-bling review
ABSTRACT
Recommender (RS) and Advertising/Marketnng Systems (AS) play the key roles in E-commerce companies like Amazon and Alibaba. RS needs to generate thou- sands of item candidates for each user (u2i), while AS needs to identify thousands millions of high-potential users for given items that the merchant advertise these items efficiently with limited budget (i2u). This paper proposes elegant bidirectional candidates generation framework that serve both pur- poses all together. Specifically, our framework finds its practical applications on generating millions of users for large number of items, and the marketing sce- narios of small and medium enterprises (SMEs) with cloud-service. Besides, our framework is also superior in these aspects: i). We theoretically show that modeling objective is equivalent to the modeling objectives of the SOTA algo- rithms if measured by the metries of u2i and i2u. ii). Our framework can easily incorporate many aNhharchit,,,ures, and achieve SOTA results with the same architecture. áii). We empirically show that our framework can diversify the gen- erated candidates, and ensure fast convergence.
INTRODUCTION
In the Internet era, E-commerce companies usually act platform to connect both users and merchants. They can collect huge amount of data users' behavior on merchants' items, and utilize the data serve them better, for example, to build better seachllccomeenderadierrtisisi systems.
When users browse the E-commerce websites/apps, they usually want to buy some desired items, or find something interesting, and that's what the search engine and recommender system (RS) are built for. Besides, the merchant usually wants to mine the potential users of its items, and then starts a campaign to prompt those users to buy the items. The advertising system has the functionality to help the merchant, and we may call it the potential users mining system (PUMS).
In an industrial large-scale RS, there are usually two stages, the candidates generation (CG) stage and the ranking stage (Covington al. 2016). CG plays very import role of the recommender. A single CG method usually lacks diversity, so the industrial CG system usually consists of different subsystems to generate candidate items, example, i). hot items, it). items similar to users (Covington et al. 2016 Li et al. 2019 mii) items similar to clicked items (Linden)
the upper bound
et al. 2003 iv). items clicked by similar users Wang et al. 2006).
The PUMS has been studied and applied in marketing community for many years (Bult & Wansbeek 1995 Kim et al. 2005), and recently it is also introduced to social network to attract potential cus- tomers of companies (Pennacchiotti & Popescu 2011 Pang et al. 2013). Similarly in E-commerce companies, it can be used to discover potential customers for merchants, brands and etc. Although it is widely used, people usually treat it simple binary classification problem, i.c., to predict whether a given user is the potential user or not. Models like LR, SVM, XGBoost and etc. (Hos- Chen & Guestrin 2016) are usually employed.
mer Jr al. 2013 Chang & Lin 2011

---

Under review as a conference paper at ICLR 2022
In this paper, unify the two systems in TetalRecall (TR) framework, and apply it in both large-scale PUMS and marketing scenarios of SMEs with the cloud-service Our main contributions are listed below:
Bidirectional Candidates Generation: TR can be trained once and then used to gener- ate candidate items (u2i) and users (12u) all together without any additional operations. Theoretically we show that it is due to the modeling of joint probability of u and i.
Achieving SOTA results both RS and PUMS: We compare to two different algorithms, i.e., Matrix Factorization (MF) with Bernoulli distribution and sequential modeling and MF with Multinomial distribution, and achieve comparable or better results than previous SOTA.
Equivalence of the algorithms based on Bernoulli and Multinomial distributions: To the best of our knowledge, we are the first show that modeling the user-item interaction with Bernoulli and Multinomial distributions are equivalent.
Fast Convergence, and Diversified Candidates: TR converges much faster compared to MF models, up to 16x times. Our experiments show that TR can increase the diversity of the generated candidates.
2 RELATED WORK
2.1 RECOMMENDER SYSTEM (RS)
RS has been studied for many years, and in the early years, Collaborative Filtering (CF) and its vari- ants (Su Khoshgoftaar 2009) are widely adopted. Later its descendant MF (Funk) proposed to solve the problem more elegantly with higher accuracy, and Probabilistic Matrix Factorization (PMF) (Mnih & Salakhutdinov 2008) builds solid theoretic foundation for MF based on proba- bility theory.
Although MF performs very well in many datasets (Rendle al., 2020), it is usually not adopted in industry. The modern industrial RS can involve up to billions of users and hundreds of millions of items, and consist of multiple stages, e.g., CG and ranking stages (Covington al. 2016).
Various deep neural networks are adopted to tackle the complicated system, for example, in the CG stage, the problem is formulated as the multi-clas classification, and users' and items' representa- tions are detached with the two-tower architecture to balance the efficiency and accuracy in online serving [Covington et al. 2016 et al., 2019) Cen et al., 2020).
the binary classification, and various models has been developed: wide & deep model (Cheng et al., (016)) captures explicit and implicit feature crossing, Deep Interest Network (Zhou et al.] (018)) and Deep User Perception Network (Ni et al. 2018) utilize attention mechanism (Bahdanau et al. 2014)) to capture users' short-term interests.
In the ranking stage, the problem is formulated
2.2 POTENTIAL USERS MINING SYSTEM (PUMS)
PUMS mines the potential users for given items. The 'item' could be anything that users can interact with, for example, an insurance product (Kim & Street 2004), company/business (Pennacchiotti Popescu 2011 Pang al.] 2013, Lo al.] (016). a specific information (e.g., tweet) social medias (Tag al. 2015; Gui et al. 2019) and even another user (Guy) 2018). These problems are usually treated as binary classification and solved with tools like SVM, LR and etc. for each individual item independently (Hosmer Jr et al. 2013 Chang & Lin 2011 Chen & Guestrin 2016).
In the E-commerce company like Amazon, the 'item' could be a product, a brand, a product category, merchant and etc, and the number of the 'items' range from thousands to hundreds of millions. It will be impractical to model each item individually. so we usually model the items all together using binary classification similar to Rendle et al. (2020), which will be stated in detail later in Sec.
2

---

Under review as a conference paper at ICLR 2022
2.3 SAMPLED-SOFTMAN AND INFONCE LOSS
In the CG stage of RS, sampled-sofiie softtmx (SSM) loss (Jean et al. 2015) is widely used with differ-
ent background distributions, such log-uniform distribution PPowers
1998), empirical marginal
distribution (ii et al. 2019) and mixed distribution (Yang et al. 2020).
In contrastive learning, InfoNCE loss is proposed in Oord et al. (2018) to learn representations
through self-supervisiin. Later it is widely adopted in CV (Chen et al. 2020 and NLP (Gao et al. and performs very well.
202T
InfoNCE and SSM losses are both trying to solve multi-class classification of very large vocabulary. They both use negative sampling and can be written in the same analytical form:
exp(Sa(x,1) v) log exp(fo(a, y) (vl=)) + +E exp(fo log (v'lz))
Poggnnlle (n(())
Σ - llog
1=
(s..))ED
where (z,,y) E "Di is the training data, and usually æ is the input features and is the class, So (ar, y) the output of the neural network parametrized by 1, and Y, contains hundreds or thousands of negative samples from background distribution P. (y(z). The main differences that the pn (y)xz)
in SSM can be arbitrarily chosen, while in InfoNCE is the empirical marginal distribution Rdata(y).
ply) instead of p(y ar) for the debiasing purpose, and it can be easily implemented with in-batch negative sampling and features of be fully utilized (Zhou et al. 2021)
Recently, InfoNCE loss is exploited in RS to optimize
p(ylz)
3 METHOD
3.1 PROBLEM DEFINITION
In E-commerce companies, when user clicked an item at time t, record (u, i, was logged as the implicit feedback.
Given the raw logs {(u, i, }. there are usually two ways to process the data: one is to directly create a user-item interaction matrix Sus with the value su.i 1 if the (u, i) pair appears in the log (Su & Khoshgoftaar (2009)); the other is to formulate the problem as next-item prediction, i.c., from the raw log we create a dataset 'D = {(Iu,,, Vu.z) : {1,2, ..., Tw lu E (1,2, ..., N}), where Fu.t ( 1) represents u's clicks prior to 1-th click Va.t and T, is the number of clicks of u (Covington et al. (2016)). We define the click sequence Zu,z as the pseudo-user, so we can create pseudo-user-item interaction matrix from D, and all possible click sequences form the pseudo-user
set.
Therefore, they both deal with the 2D interaction matrix Sus. irrespective of whether the is a real user or pseudo-user or not. the remaining of the paper, we will not distinguish between user and pseudo-user, and only focus solving the Sui-
Sui consists of all users as the row and all items as the column, and its entries are either or unknown. Let's assume all users form the set U {ui, ut..... u.r }. and all items form the set I == (i1, i2,,,,,.., ix.,, J. Given a user u E U. we will generate candidate items from the item pool II (RS, w2i), while given an item
shall find out the potential users out of all users U (PUMS, i2u).
The candidates generation of RS and PUMS are of the opposite direction, and traditionally they are solved separately. But essentially they are both trying estimate some kind of probabilities of the unknown entries in S.i. so we propose to model them jointly.
3.2 RECOMMENDER SYSTEM
In RS, there are usually two types of modeling objectives when dealing with Suis and we briefly introduce them in the following sections (See App. A.T for more details).
3

---

Under review as a conference paper at ICLR 2022
3.2.1
GAUSSIAN AND BERNOULLI DISTRIBUTION
The u-i interaction score Su.á of S.. is treated the random variable, and assumed be i..d. following a certain distribution, for example, Gaussian (Mnih & Salakhutdinov (2008)) or Bernoulli (Johnson (2014); He et al. (2017); Rendle 20200) distributions.
3.2.2 MULTINOMIAL DISTRIBUTION
Here the modeling objective is the random variable defined as the interacting item of u. For example, given the total number of interactions N,, = Eier 8u,, of u, 8u assumed to be sampled from a multinomial distribution Mult( Pa, where pa, is K-diimnniional vector whose elements Although not explicitly stated in many papers
are the probability and sum to (Liang et al. 2018).
(Covington et al. (2016] Yi at.. actually assumed.
2019
2019
Cen et 2020), the multinomial distribution
The modeling objectives in Sec. 3.2. and 3.2.2 are quite different, as the former Su.i and the latter is 8.- In MF, we usually model 8u,i, while in the sequential modeling of the CG stage in industry, we model 8.. Later in Sec. we will show that modeling 8, and su,i can be theoretically equivalent, and it is verified by the experiments in Sec. 4.1 and However, modeling s, can boost the converging speed a lot.
3.3 POTENTIAL USERS MINING SYSTEM
The PUMS seems to be the twin system of RS, and similarly we could also define two modeling objectives Sa,; and 8,, where 8; denotes the user who interacts with the item Su,i and 8, are assumed to follow Bernoulli and Multinomial distributions respectively. In Sec. 5 we will compare the two methods.
3.4 RECAL:: UNIFICATION OF RS AND PUMS
Motivated by Sec. 3.2 and 3.3 propose to model the two systems jointly. Specifically, the opti- mization objectives 8, and 8; in Sec. 3.2.2 and 3.3 are formulated in one framework and optimized together. Later show that by properly choosing the model architecture and loss function, treating u and i equally and modeling the joint probability.
1 (uli)
$9 (u, = ||u||2/|@@1e
exp ( (u, 1)) puu,
logp (u, lo u)) pg
u IRC
EE IR
RS: u2i
log pu, 1) log (u ())(()+ (
PUMS: i2u
Figure Model architecture of RotalRecall. Our framework will make to dot product (o i) converge to the logarithm of the joint probability p(u, i), and thus can be directly applied in RS to emulate the conditional probability p(ilu), and also PUMS to replace p(uli).
3.4.1 MODEL. ARCHITECTURE
We choose the classical two-tower architecture (Huang et al. 2013 Covington et al. 2016 Rendle al. 2020) for reasons: one is that users and items can be treated equivalently, and the other is
4

---

Under review as a conference paper at ICLR 2022
that no feature crossing occurs before the final logits (See Fig. D. So users' and items' embeddings can be inferred separately, and then approximate nearest neighbor (ANN) (Liu et al. 2004 search algorithm can be applied in online serving.
The output of two towers are d-dimensional vectors u = (u) E R& and i = go R., where the model parameter. The dot product (ulz), [)), the function of it is used as the sufficient statistics of the probability distributions defined in Sec. Sec. [333 (Check [A.. for details). We find that 12-normalizing and i and then rescaling the dot product by the temperature will lead to better and robust results:
(u
④o (u,
(u,, 2) 7
3.4.2 LOsS FUNCTION
We propose the Bi-InfoNCE loss:
exp(do(u,) i) log (())
1= |S WEU,iEI,S..
Σ
log
exp(oo(u, - (()) + Ever, exp(oo(u, - log (('))
exp(oe(u,) i) log ((u))
log exp(ou(u, log ((o)((u) Ewvev (xp(ool"' log ((u'))
(3)
where I, C. Il and Ug C ' contain hundreds or thousands of in-batch negative samples, and ((i) and ((u) are empirical marginal distribution calculated using the training data Sui. can be shown that optimizing this loss will make do (u, i) converge to log Pdata (u, i) (See App. A.3 for details.).
we only optimize the first part of Eq. Φ i) will converge to log Pdata (i/e))))))) (2), and optimizing only the second part will let (v i) converge to log Pdata (uli). We refer to the two cases as Uni- InfoNCE losses.
3.5 EQUIVALENCE BETWEEN MODELING WITH BERNOULLI AND MULTINOMIAL DISTRIBUTIONS
discussed the previous sections, we can use both Bernoulli and Multinomial distributions to model the same user-item interaction matrix Suis and usually their results are comparable in both RS and PUMS (See Sec. 4.1 5).
The apparent connection between the two algorithms motivates us explore their relations, and we prove that they are theoretically equivalent on modeling the user-item interaction matrix. Specifi- cally, different negative sampling methods of Bernouli distribution correspond to different objec- tives modeled with Multinomial distributions.
For example, in MF with Bernoulli distribution (MFs). could have 5 different negative sampling methods, i.e., for a given positive (u, i) pair, we sample the negative pairs these ways: 1). 'MF, [0:8][] we randomly sample is from the item set and construct negative pairs with the given w; 2). "MF; [8:0]', we sample 8 us from the user set U, and construct 8 negative pairs with the given ;; 3). 'MF, [4:4]', we sample 4 us and 4 is. and construct negative pairs with the given and respectively; 4). similar to 3), but the negative samples are doubled; 5). 'MF, [8]', we sample 8 (u, i) pairs uniformly from the M
pairs with i Il and u e U as the negative pairs.
We prove that 1) is equivalent to modeling p(ilu), 2) to p(uli), 3) and 4) to p(u)ptt and 5) to p(u, i). The details can be found in App. A.4
p(u.))
3.6 MOTIVATION AND PRACTICAL APPLICATIONS OF TR
TR developed to solve two practical problems: one is the large-scale potential users mining, and the other is the marketing in SMEs.
Thii sampling method is actually used in MF, in RS by Rendle al. 2020
5

---

Under review as a conference paper at ICLR 2022
Table 1: Baselines from Dacrema al. (202) and Rendle al. (2200) and results. The best
results are highlighted in bold, the second best result is underlined.
Movielens
Pinterest
Method HR 10
Result from
NDCG@ 10
HR @
NDCG@ 10
Popularity
0.4533
0,2542
0.2740
0.1409
Dacrema al.
(222
SLIM
0.7162
0,4468
0.8679
0,5601
Dacrema et al.
2021
iALS
0.7109
0.4382
0.8761
0.5590
Dacrema al.
2021
MLP+GMF
0.7093
0,4349
0.8777
0.5576
Dacrema ct al.
2021
MF, by Rendle
0.7294
0.4523
0.8895
0.5794
Rendle et
2020
MF, (ours)
0.726
0,4505
0.8925
0.5838
TR (ours) 0.7281
0.4541
0.8936
0.5866
For example, we have billion users and 10,000 items, and we need to mine the top million users for each of the items. This problem can be easily solved with TR: we infer the users' and items' embeddings with TR, and then apply the ANN search to find the top 10 items for each user. Now have 10 billion (u, i) pairs and their similarity scores, and just pick the top million for each item based on the same scores. This method also indirectly ensures that each person will receive at most 10 advertisement messages.
This can only be achieved if we model the joint probability p(u, i). We also list some alternative methods solve the problem in App. A.5] but they either not good in terms of accuracy or not practical in terms of computation and storage costs, efficiency, risk and maintainability. TR has been deployed in production environment serve this purpose.
In the marketing scenario SMEs, two strategies are widely adopted: i). SMES pick some specific groups of users, and then choose the top products for each user and send the personalized campaign messages. This can be regarded as the RS. it). SMEs have some new or special products and want find the target users. This is the PUMS.
The two marketing requirements can be completed either with two separate models or with the unified model TR. Due the limited computation and storage budgets, and the shortage of talents, TR is the best choice for them. Actually, TR is already implemented in one of our cloud-service products for SMEs.
EXPERIMENTS: RS
As stated in Sec. 3.2 types of modeling objective well studied in literature, and usually they use different datasets and baselines. So we compare our results with them separately using different datasets. The details of the datasets and processing methods can be found in App. A.6
4.1 MF WITH BERNOULLI DISTRIBUTION
We compare TR to MF, in Rendle al. (2020). as described in Sec. 3.2.1 Our implementation of MF, is bit different from Rendle et al. (2020): we implement it using tensorflow (Abadi et al., 2016) with Adam optimizer (Kingma Ba 2014).
We tried 1.2 regularization but failed to obtain the results reported by Rendle et al. (020). We also tried dropout (Srivastava et al.) 2014), but was unstable and need be re-tuned for differ- ent dimensions d. Finally regularization, so we use it rather than the one defined by Eq. (2) in Rendle et al. (2020).
find that the (e(u, i) defined in Eq. 2 can actually act as kind of
We use Movielens-Im Im and Pinterest datasets, and the evaluation metrics NDCG@ 10. MF, and TR have the same two-tower architecture and same number of parameters, and the key differences ative sampling, ie.. MF, [0:8], and optimize the binary cross-entropy loss, and TR uses implicit in-batch negative sampling and Bi-InfoNNCE loss.
HiteRate @ 10 and
the training data and the optimizing objectives: MF, uses explicit neg-

---

Under review as a conference paper at ICLR 2022
Table 2: For Amazon books data, 0.1, batch-size 128. For Taobao data, = 0.067, batch-size 2048. ComiRec-SA and ComiRec-DR are two multi-interest models proposed Cen al. (2020). The best results are highlighted in bold, the second best result is underlined.
Amazon Books
Taobao Metrics @ 50
Metrics @5
Recall
Hit Rate
Recall
Hit Rate
Popular
2,400
5.226
0.735
9.309
YouTube [SSM w/o 12-norm]
7.312
15.894
6.172
39.108
ComiRec-SA
8.467
17.202
9.462
51.064
ComiRec-DR
8.106
17.583
9.818
52.418
Youtube [SSM with 12-norm]
9.37 (+28%)
19.25 (+21%)
6.9 (+12%)
42.87 (+10%)
TR- Youtubee [Uni-InfoNCE u2i]
10.34 (+41%)
21.15 (+33%)
7.15 (+16%)
43.79 (+12%)
TR- Youtube [Bi-InfoNCE]
10.12 (+38%)
20.554 (+29%)
7.14 (+16%)
43.57 (+11%)
TR-Youtee Youtube [next-7-prediction]
11.571 (+58%)
23.221 (+46%)
The results are shown in Tab. In general, MF, and TR are comparable, and this is consistent with our theory in Sec. 3.5 because they are both modeling p(ilu) or p(u. i) ultimately. Besides, the diversity of TR is also larger than MF, as shown in Fig. 3 (Check App. A.6.Ifor more details.)
When measuring the training efficiency, TR converges to the optima up 16x times faster than MF, (See Fig. 3 in Appendix). We attribute it to two reasons: i). For MF, [0:8] used here, 9 processed samples will consist of only 1 positive samples on average. While Bi-InfoNCE (or Uni-InfONCE / SSM), 9 processed samples are all positive samples. it). For MF,, each sample is either positive or negative, and the binary-cross-entroy loss allows it to compare with itself only. In TR, Bi-InfoNCE (or Uni-InfoNCe SSM) loss will compare each sample with all the other samples in the batch.
Bearing in mind that MF, and InE can theoretically converge to the same optimum, the qualitative explanation can provide some intuitive understandings of the fast convergence. The fast converging speed makes it more practical to use TR in industrial applications with huge amount of data.
4.2 SEQUENTIAL MODELING WITH MULTINOMIAL DISTRIBUTION
We compare TR to the sequential modeling with Multinomial distribution, as described in Sec. 3.2.2 This modeling method is usually used in the CG stage of industrial large-scale RS, because it can deal with huge amount of data very efficiently, and yield good results.
For simplicity, we use the popular Youtube DNN (Covington et al. 2016 implemented by Cen al. (2020) as the backbone architecture of TR. Other complicated architectures can be easily incorporated into TR framework.
We use the same datasets as Cen et al. 2020 for a fair comparison, i.c., Amazon books data and Taobao click data in Tianchi competition The data is split into train, validation and test data. The training data is usually processed as next-item prediction as in Sec. [.1] and we extend into next-n-item prediction, i.e., instead of predicting only the next item, predict all the next items.
The results are shown Tab. -
More results are shown in Fig. and 5 in Appendix.
Our experiments show that proper regularization using 12-normalization and the scaling parameter can greatly improve the results, as confines do(u,, i) inside [-1/7, 1/7]. Better training data due to next-n--rediction can also improve the results, because it alters the distribution of the training data so that it gets close to the test data (See Fig. Gl Appendix).
Besides, have three observations. i). Comparable results: TR's result is comparable to the SOTA results from SSM and Uin-InfoNCE, This shows that the modeling objectives p(u, i) and
"http:/fjmcaulley.oo wwtp///aaaaaaaaaaoonnnneeeeeeeee html hupps:/tianchi.i alwaaa. htwp::/wwwwmmmaaeeeaaaaaaaaaaaaa

---

Under review as a conference paper at ICLR 2022
p(ilu) perform similarly if measured by the metrics for u2i candidate generation. ii.. Improved diversity: We try SSM with different background distributions, and compare with Uni- Ini-InfoNCE and Bi-InfoNCE, and find that Bi-InfoNCE gives better overall diversity measured by the related metrics (See Fig. 4 in Appendix). We don't have a rigorous explanation and would like to leave it for the future study. iii). Faster convergence: Compared to SSM or Uni-InfoNCE without 12- normalization, adding 12-normalization or using Bi-InfoNCE will make the models learn faster (See Fig. Appendix). This because 12-normalization will restrict the vector space that u and could explore, and Bi-InfoNCE utilizes the data more efficiently compared to Uni- Ini-Nn i.e., the utilization rate is doubled.
EXPERIMENTS: PUMS
In E-commerce, there are usually two practical scenarios of i2u candidates generation: one is to mine the potential users for single item, and the other is to find users for group of similar items. Because the marketing staff or some merchants regularly pick a group of similar items, and set the specific campaigns to advertise them.
Previously, the problem is usually formulated as the binary classification for each individual case, but it is not generally applicable because the number of items or groups could be quite large, and is impractical to build so many models.
One solution is merge all the binary classification models into one, just like MF, in Sec. 3.2.1 and 3.3 The other is our TR framework.
5.1 DATASET AND EVALUATION METRICS
We use Movielens- Im because it also used MF, RS, and can be used to mine the potential users of both the single item and the group of similar items.
Movielens- I m records 6040 users and 3706 movies. The data split into training, validation and test data, and we use the validation data to select the proper model configurations. The final results are reported on the test data.
We use precision @ 10 and recall @ 10 to evaluate the accuracy of different algorithms, i.e., mine the top 10 users for a single item or a group of items, and then evaluate the precision and recall. We count the distinct users of the overall results measure the diversity. (See App. A.7 for details about the dataset and metrics.)
5.2 POTENTIAL USERS OF SINGLE ITEM
We first select movies which have been watched by at least 50 users, and results in 2514 movies. For each of these movies, we randomly pick 1% of its users to form the valid and test datasets, with half of the movies as the validation data and the other half the test data (See App. A.7). The remaining data is used to train the model, and the validation data is for selecting the best hyper- parameters, and the results are reported on the test data.
The results are shown in Tab 3 "TR [Bi-InfoNCE]] achieves the best results in terms of the accuracy and diversity. High diversity implies that the advertisements can reach more users, even the non- active users. thus alleviate the advertising fatigue.
Besides. TR uses the same training data and model configurations for both RS and PUMS, and achieves comparable in both cases. The learned u/i representations be applied the two sys- tems without any additional modifications. treat and equally, but the outcomes depend highly on the negative sampling methods (See Tab. 3 This is consistent with our theoretical analysis in Sec. 3.5
contrast, MF, 's modeling objective (Eq. 4 seems to
Last but not least, TR learns 16x times faster than MF, (See Fig. 7 in Appendix). The reason is the same as in Sec. 4.1
Our study of mining potential users of group of items be found in App. A.7.2
8

---

Under review as a conference paper at ICLR 2022
Table 3: Overall performance mining the potential users of a single item. The % is omitted for precision and recall. The hyper-parameters of MF, are batch-size 4096 and 0.125, and for TR they are batch-size = 512 and r = 0.125. We use d = 96 for all models for a fair comparison. The diversity values of models with very low accuracy omitted.
model
precision 10
recall 10
distinct users
MF. [0:8]
1,47
5.61
MF, [8:0]
4.44
12.82
2046
MF. [4:4]
3.12
9.28
MF, [8:8]
3.02
8.64
TR [Uni-InfoNCE u2i]
1.19
4.51
TR [Uni-InfoNCE 22u]
4.35
12.78
2064
TR [Bi-InfONCE]
4.58
13.82
2202
CONCLUSION
In this paper, CF and the sequential modeling (i.e., next-item prediction) of RS, and PUMS are abstracted into unified u-i interaction matrix Suis and thus are treated equivalently deriving some probabilities for the unknown entries from the known entries of Sus. We also show their connections and differences based on probability theory.
We prove that modeling p(su,f 0/1 u, with different sampling methods can be equivalent to modeling p(ilu), p(uli). p(u, i) and p(u.))
p(a)p())
We propose the TR framework solve u2i and i2u CG all together. In TR, and i are treated equally, and the joint probability p(u, is actually derived, instead of the conditional probability p(ilu) or p(uli). Therefore, the learned representations of u and in TR can be used in both w2i and i2u CG.
Our extensive experiments show that TR can achieve comparable results each domain measured by the metrics for either u2i or i2u candidates generation. TR learns 16x times faster than MF, Besides, TR generates diversified candidates.
In the future, plan to study the vector space of TR in depth, and try unify different objects and users into one space. for example, products, merchants, brands, categories products. such a space, everything is linked, and candidates generation be applied any pair of objects.
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. OSDI, volume 16, pp. 265-283, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint
2014.
Jan Roelf Bult and Tom Wansbeek. Optimal selection for direct mail. Marketing Science, 14(4): 378-394 1995.
Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang. and Jie Tang. Controllable multi- interest framework for recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 2942-2951, 2020.
Chih-Chung Chang and Chih-Jen Lin. Libsvm: library for support vector machines. ACM trans- actions on intelligent systems and technology (TIST), 2(3):1-27, 2011.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, ACM, 2016.

---

Under review as a conference paper at ICLR 2022
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for rec- ommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, pp. 7-10. ACM, 2016.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems, pp. 191-198, 2016.
Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. A troubling analysis of reproducibility and progress in recommender systems research. ACM Transactions on Information Systems (TOIS), 39(2):1-49, 2021.
Simon Funk. Netflix update: Try this home.
URL http:://afteer.or///ssoo
/ttpaa///.......... html
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint wxXi::210.0082/1 2021.
Xue Geng, Hanwang Zhang, Jingwen Bian, and Tat-Seng Chua. Learning image and user features for recommendation in social networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4274-4282, 2015.
Tao Gui, Peng Liu, Zhang, Liang Zhu, Minlong Peng, Yunhua Zhou, and Xuanjing Huang. Mention recommendation in twitter with cooperative multi-agent reinforcement learning, In Pro- ceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 535-535--44 0455544,77 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Ido Guy. People recommendation social media. In Social information access, pp. 570-623. Springer, 2018
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (fiis), 5(4):1-19, 2015.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural col- laborative filtering. In Proceedings of the 26th international conference world wide web, pp. 173-182, 2017.
David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. Applied logistic regression, volume 398. John Wiley Sons, 2013.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333-2338, 2013.
Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing (Volume I: Long Papers), pp. 1-10, 2015.
Christopher C Johnson. Logistic matrix factorization for implicit feedback data. Advances in Neural Information Processing Systems, 27(78):1-9, 2014.
YongSeog Kim and W Nick Street. An intelligent system for customer targeting: a data mining approach. Decision Support Systems, 37 2004.
10

---

Under review as a conference paper at ICLR 2022
YongSeog Kim, W Nick Street, Gary Russell, and Filippo Menezer. Customer targeting: A neural network approach guided by genetic algorithms. Management Science, 51(2):264 276, 2005.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiiv:442.000 6980, 2014.
Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. Multi-interess network with dynamic routing for recommen- dation at tmall. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 2615-2623, 2019.
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. Variational autoencoders for collaborative filterin.. In Proceedings of the 2018 world wide web conference, pp. 689-698, 2018.
Greg Linden, Brent Smith, and Jeremy York. Amazon. com recommendations: Item-to-item collab- orative filtering. IEEE Internet computing, 7(1):66-80, 2003.
Ting Liu, Andrew W Moore, Alexander G Gray, and Ke Yang. An investigation of practical approx- imate nearest neighbor algorithms. In NIPS, volume 12. pp. 2004. Citeseer, 2004.
Siaw Ling Lo, Raymond Chiong, and David Cornforth. Ranking of high-value social audiences on twitter. Decision Support Systems, 85:34 48,4448,22 2016.
Andriy Mnih and Russ R Salakhutdinov. Probabilistic matrix factorization. In Advances in neural information processing systems, pp. 1257-1264, 2008
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-laheled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 188-197, 2019.
Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo Si. Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks. arXiv preprint arXiv: 10727, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 1807.0880,,33748, 2018.
Guansong Pang, Shengyi Jiang, and Dongyi Chen. A simple integration of social relationship and text data for identifying potential customers in microblogging. In International Conference on Advanced Data Mining and Applications, pp. 397-409. Springer, 2013.
Marco Pennacchiotti and Ana-Maria Popescu. Democrats, republicans and starbucks afficionados: user classification in twitter. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 430-438, 2011.
David MW Powers. Applications and explanations zipf's law. New methods in language processing and computational natural language learning, 1998.
Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. Neural collaborative filtering matrix factorization revisited. In Fourteenth ACM Conference on Recommender Systems, pp- 240-248, 2020.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 3859 38699,38699188 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958 2014.
Xiaoyuan Su and Taghi M Khoshgoftaar. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009, 2009.
11

---

Under review as a conference paper at ICLR 2022
Liyang Tang, Zhiwei Ni, Hui Xiong, and Hengshu Zhu. Locating targets through mention in twitter. World Wide Web, 18(4): 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Jun Wang. Arjen P De Vries, and Marcel JT Reinders. Unifying based and item based col- laborative filtering approaches by similarity fusion. In Proceedings of the 29th annual interna- tional ACM SIGIR conference on Research and development in information reirieval, pp. 501- 508, 2006.
Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong. Yang Li, Simon Xiaoming Wang, Taibai Xu, and Ed H Chi. Mixed negative sampling for learning two-tower neural networks in recom- mendations. In Companion Proceedings of the Web Conference 2020, pp- 441 441-447, 2020.
Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz. Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. Sampling- mendations. In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 269-277, 2019.
cumpling-bias-coreettee bias-corrected neural modeling for large corpus item recom-
Chang Zhou, Jianxin Ma, Jianwei Zhang. Jingren Zhou, and Hongxia Yang. Contrastive learning for debiased candidate generation in large-scale recommender systems. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3985-3995, 2021.
Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma. Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- ACM, 2018.
Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1079-1088, 2018.
A APPENDIX
A.1
DISTRIBUTIONS
In Sec. 3.1 we define the interaction matrix Sui and it has the known entries with value 1 and unknown entries. Here we define a corresponding matrix P of the same shape, and its entries are the probabilities estimated by a certain algorithm.
In Sec. 3.2.1 the corresponding probability matrix PMr is defined as follows:
Pu.i == p(S., 1/u,
where EE p(su,su, 1.
Here we can assume su,s follows Gaussian distribution: Sa, N(o(u, 0?), or Bernouli distri- bution: 8a,i ~ (( i)), where af-) is the sigmoid function.
In Sec. 3.2.2 for given user u. S. Mult(Na: Pu). where p.. (Pul:Pu2..... Pak) is K- dimensional vector. The probability matrix Puzi is defined with the same Pai = p(ilu). Pui is modeled as
exp (u,
Pus Ejeres exp 6o(u.
(S)
In Sec. 3.3 of PUMS, we can define the specific PME and Pezu in a similar manner. For example, we use 8; to denote the user who will interact with the item and 8; Mult( N, Pc). where P: = (pri, Ppii,Pei.... P.,. and it modeled as
pars) Pars)) is M-dimensional vector. P... is defined with the same Puc p(uli),
exp expoolu,
Pui EoSU exp exp (0
(6)
12

---

Under review as a conference paper at ICLR 2022
A.2 LOSS FUNCTIONS
In this paper, we focus on Bernoulli and Multinomial distributions, because they usually lead to better results than Gaussian distribution (He al. (2017); Rendle al. (2020)) Following the maximum likelihood estimation (MLE) principle, the loss for Bernoulii distribution is:
=
Σ
Su,i log a ( i)) + - -u,;) log(1 - (bo 8))),
(7)
[D]
(M.))E
where Su,i E {0, 1}, and 8u,; 1 are the positive entries in Sui. and Su,i = 0 are negative samples randomly sampled from the entries Suis and the training dataset consist of all the positive and randomly sampled negative samples.
The loss for Multinomial distribution is softmax- ciimaaxxccros--ntrrpp loss (Liang et al. 2018):
log exp 00 (0000o i) Ever exp (u
Σ
1 = Su,i NEU.
(8)
where [Su,il is the sum of the entries in Sui.
In practice, the vocabulary I can be very large so that calculating the partition function Ever exp (u, i') becomes impractical. SSM and InfoNCE loss can be used to solve this problem:
exp(oo i) - log pr. u))
1 = SS vEt,
Σ
- -log exp(oo(u, P. (ilu)) + Ever, exp(do (u, 2)
log Pm
(7' (V'lu)
u))
(9)
where Il C II contains hundreds or thousands of negative samples, and pa (ill) is a predefined background distribution. It has been shown that optimizing Eq. 9 will make Φo(u, i) converge to log (data(ilu) (ilu) (Jean et al. 2015 Oord et al. 2018).
A.3 BI-INFONCE LOSS
Here we prove that Φo (u, i) in Eq. converges log Pdata Ataatau,,7) i).
In Eq in the optimum of the first part, following the proof by Jean et al. (2015); Oord et al. (2018).
have exp(do(u.i)) Pdatalilu) 9:
assume:
d:
()) log Paata (ilu) + f(u) log Pdata (u, i) - log Pdata (u) + f(u)
for a given and any i. Similarly, in the optimum of the second part, (u, log Pasta log (i) g(i) for a given and any u.
Thus, for any u and i, this will always hold: log Paata (u) f(u) log Pdata (1) + g(),, so must be some constant. Then we have (u, = log Pdata (u, i) C, where C is some constant that is independent of and i. And when @o(u,, a) converges to log Pdata i), both parts in Eq. can reach their optima.
A.4 EQUIVALENCE BETWEEN BERNOULLI AND MULTINOMIAL DISTRIBUTIONS ON MODELING THE USER-ITEM INTERACTION MATRIX
Here show that by properly choosing the training data distribution, (o(u. i) in Eq. 2 is actually approaching the conditional distribution p(ilu) or p(uli), or joint distribution p(u, i). or p(u.i)
p(u)p())
Our derivation is inspired by Noise Contrastive Estimation (Gutmann & Hyvärinen) 2010
In the modeling with Bernoulli distribution, we have two sets of samples, X and Y. X is the set of positive samples and Y is the set of negative samples randomly sampled based on certain distribu- tion (.).
Assume X {I1, B2...., } contains L samples, and Y = {311 32..... yr contains T samples,
and Z = X XUY- {21, 22, LL+T} contains all the L T samples. Here (u, i), Y/ = (u. i) and x; = iu, where u E U and E I. We assign each sample zj a binary class Cj;: C, = 1 if comes from X, and Cj- 0 if 2, is from W.
13

---

Under review as a conference paper at ICLR 2022
We assume the joint probability of positive samples in X is parameterized by @ is Pmodel (2)
Pmndel (2; 0). So we have the conditional probabilities:
pl=|C 1; - Pmodell2;0)
= - Pn)
(10)
The posterior probabilities are:
Pmnddol (z:0)P(C =
p(C 1|z;0)
(11)
Pmodel (axodel(2;000
1+ exp( +xpp(-G((z;))
where
((2;0) log Pmooeel 2: P(P) =
(12)
Pr =0)==)
and we also have
p(C 00 1 exp(G(zz;0)
So the log- likelihood is:
LTT
((0) = Σ ==1
log P(C, = 1|z;; + - C₁) log P(C₁ = 0|z]; 0).
Through optimizing the binary classification using samples in Z and the corresponding binary classes C. we are actually recovering the MF with Bernoulli distribution defined in Sec. 3.2.1 The dot product dh (u, will converge G(z; 0) in Eq. 12
Φ0(u,)) i) log Pmodel (z: P(C= Pr (=)=
(13)
Different negative sampling strategies will lead to very different optimal (o(u,)), i). For example, if we randomly sample items for each positive (u, pair to form negative samples with the then we have (=) )) = Pn (u, i) = Priata (u) I/K, where Paata(u) (u) is the empirical marginal probability
of the positive samples. Substitute Pr (2) into Eq. 13 we have
Φo(u, i) log Pmodel (u, i,,;;00 0)
C z log Pmodel (i + C.
Pdata (u)
Similarly, if randomly sample n users for each positive (u, pair to form negative samples with the i, then da (u, i) will converge to the conditional probability log (wli;; O).
For each positive (u, i) pair, if randomly sample items form negative samples with the u, and also n users to form negative samples with the i, then Φ i) will converge to the point-wise mutual information log (u:)dal
(m.dea(u,;:0)
And if the negative samples are randomly sampled uniformly from all users and items globally, then (e(u,i) i) will converge to the joint probability log Pmodel (1, é; 0).
Therefore, show that MF with Bernoulli distribution and Multinomial distribution can be equiv- alent if we choose the proper negative sampling methods.
A.5 MOTIVATION AND PRACTICAL APPLICATIONS OF TR
For the example of mining potential users given billion users and 10,000 items, we list all the alternative methods that we can think of:
1. For each item, we create the training sample, and then train a binary classification model, and finally predict for the I billion users and pick the top million. We need to repeat 10,000 times.
2. By emulating the RS, we model p(uli) and obtain the embeddings of users and items, and then apply the ANN search (e.g.. Faiss) to find the top million users for each item. This method has serious defect: ANN has high accuracy if we retrieve the top hundreds or thousands of candidates, but its accuracy and efficiency drops dramatically if we extend the range to millions. So, this method is not practical.
14

---

Under review as a conference paper at ICLR 2022
3. Also, model p(u(t) and obtain the embeddings of users and items, and then do brute force search: we calculate the similarity score of all the users and items, which is the Cartesian product of 1 billion users and 10,000 items, ie., 10 trillion, and then pick the top 1 million users for each item. It's very inefficient and consume huge number of resources.
4. We build two models for p(ilu) and p(ul)) respectively. Then we use the embeddings of p(ilu) and apply the ANN search to find the top 10 items for each user. Now we have 10 billion u-i pairs, and we use the embeddings from p(uli) () to calculate the similarity score of the u-i pairs, and finally pick the top million for each item.
TR has just one less model than the last alternative and one less step, but in practice, is problematic to maintain two correlated models, especially when they very large. Two models usually imply that the calculation is doubled, the storage doubled, and the risk and instability are doubled.
To conclude, TR is the best choice that can balance the accuracy, efficiency, risk and maintainability in practice.
A.6 EXPERIMENT RESULTS: RS
A.6.1
MF WITH BERNOULLI DISTRIBUTION
Movielens
0.49
0.445
0.715
0.61
0.71
0.705
Me-MF by Rendle
01455
by
0
Mil by Rendle
0455
by us
041
0.685
16
32
64
12B
255
1201
Ensoddiing dimension
Embedding dimension
Pinterwal
Pinterest
0.585
0.89
0.58
NDCGe10
0.575
0.885
0.882
- MF by Rendle
0.585
MI
-I-Mif by
0.56
TR
0.878
0.355
16
32
128
256
16
12A
250
Embedding dimersion
Embedding dimension
Figure 2: Comparison of results of MF given by Bernoulli distribution and Multinomial distribution. 'MF by Rendle' from Rendle et al. (2020). 'MF by us' is our implementation. In general these results quite close and outperform Neural Collaborative Filtering by large margin (Rendle al. 22020))
We compare MF in Rendle et al. (020) using the same datasets: binarized Moviclens- Im (Harper & Konstan 2015) and dataset from Pinterest (Geng et al. 2015).
Train and Evaluation Configurations
For each user, the last item is held out as the test set, and the second last is held out the validation set, and the remaining as the training set. We use the validation set and training set to select the best hyper-parameters, and then re-train the model using both training and validation sets.
15

---

Under review as a conference paper at ICLR 2022
During evaluation, each user has his own candidates pool, and it consists of the withheld item (i.e., the ground and 99 sampled negative items. These methods will score the 100 items and rank them by the score.
Movie-lens
Pinterest
0.73
0.735
0.001
0.72
0.003
44
0.805
0.71
0.005
0.885
0.835
0.001
0.69
612
1004
0995
8152
16384
37386
65536 131007355166
data throughout (110000)
dbta throughppt (*10000)
Movie
Pinterest
0.45
0.545
2.445
4-4 de44
0.58
N5
2.435
NDCG@-00
0.335
2/455
TR
MF
2/135
512
1024
2043
4796
5192
16384 32760 65336 131072362344
0.525
8352
16384
32768
$5596
130072
202344
$24288
dotathroughhut (010000)
throughutt (*10000)
Movie-let lons
Pinterest
2300
5650
MF
2850
MF
5600
2750
5700
2550
9600
2800
9690
512
1024
2048
4096
5192
16384 32766 65536 330072262144
8192
16384
32268
65596
133072
262144
S24298
shrouhrouhhput (110000)
Figure 3: Comparison of the converging speed and diversity of MF modeled with Bernoulli distri- bution (MF) and Multinomial distribution (TR). The z-axis is the data throughput, i.e., the number of records consumed by the model, and the y-axis is the metric measured on the test data. For Movie-lens, TR is about 16x times faster, and for Pinterest it is about 10x times.
The hyper-parameterss and batch-size chosen using the validation data. For Movielens- Im, use т 0.125, batch-size = 4096 for MF,, and 7 = 0.111, batch-size = 512 for TR. For Pinterest, = 0.143, batch-size 8192 for MFs, and 0.167, batch-size 8192 for TR. We have E {16, 32, 64, 96, 128, 196} for a comprehensive comparison.
Evaluation Metrics
Two evaluation metrics employed measure the performance of top-N ranked items:
HitRate @ N
Σ
8(12,N n I. > 0),
(14)
DEU
where Σ,,N denotes the top-N ranked items for u, I, is the set of testing items, and 6(-) is the indicator function.
16

---

Under review as a conference paper at ICLR 2022
NDCG@ N
Σ
(15)
U
log(1 + ra)
WEU
where r, is the rank of the withheld item in the top-N items for u.
Results
The performance of different methods is shown in Fig. 2 In general, MF with Bernoulli distribution and Multinomial distribution are comparable, which is consistent with our theory in Sec. 3.5
The comparison of the converging speeds shown in Fig. We can see that modeling with Multinomial distribution is much better than Bernoulli distribution.
A.6.2 SEQUENTIAL MODELING WITH MULTINOMIAL DISTRIBUTION
We compare to the models implemented ComiRec (Cen et al. 2020). ComiRec provides the codes and two preprocessed datasets: Amazon books data (Ni et al., 2019) and Taobao click data (Zuu et al. 2018). So we use the same datasets and evaluation metrics for a fair comparison.
Metries and Comparing Methods
We use HitRate @ N and Recall N to measure the accuracy. HitRate @ N is defined in Eq. 14 and Recall @ N is defined below:
I Iu,N nIw
Recall @ N
(16)
UU
In
REU
Metrics 'Distinct items N counts the distinct items users' top-N recommended list, and 'Distinct cates @ N' counts the distinct categories of these items. They measure the diversity of the recommendation results.
The comparing methods include the following:
MostPopular recommends the most popular items to all users.
YouTube DNN (Covington et al. 2016) is a simple and very efficient neural network model adopted in the CG stage of the large-scale industrial RS.
ComiRec-SA & ComiRec-DR (Cen et al, (020) use self-attention (Vaswani et al. 2017)
or dynamic routing (Sabour al. 201 7) to explicitly incorporate users' diverse interests in the model's architecture to increase accuracy and diversity.
Results
The results evaluated by the above metrics can be found in Fig. 4 Recall comparable, and this is consistent with our theory in Sec. 3.5 The diversity of TR with Bi-InfoNCE is much better than other methods.
We can see that their HitRate and
The performance wext-n-predictton are shown in Fig. 5 marginal distributions of train and test data in Fig, 6 and show that the performance gain in Amazon data may due the reduced discrepancy between the train and test data.
We investigate the KL Divergence the
In Taobao data, the discrepancy gets larger when we increase and in experiments the accuracy does not increase.
A.7 EXPERIMENT RESULTS: PUMS
There are different ways of creating the validation and test datasets, such randomly picking 10 users for each chosen item or each group of items.
Here choose to holdout 1% users for the practical reason: usually mine the potential users based on the historical behaviors and then deliver advertisement to them in the future, and the im- plicit assumption is that the distribution of the future close the past, i.e., popular items in the
17

---

Under review as a conference paper at ICLR 2022
0.23
2.19
HR.800
2.27
0.55
tomp
utlnce temp
Ba- Nemp
3.33
-B- bi
snd,
0.99
50m
900
1000
1500
2000
(0000))
throushhtp (110000)
Amazon
Amazon
300000
1400
180000
Disonec tterre 8930
$50000
3300
120000
2000
300000
temp
A2000
60000
800
1000
500
1000
2500
2000
throuphout (000))
detuthrouuhput (*10000)
Taobao
Tadbao
0.35
HR.
b Lnoe, Jemp
Recol
0.06
2.75
unt.nce tomp
0.04
uni nirrce_ton pomp
SanT, Icmp
0.2
by
blunce
8.15
ust
0.00
10000
20000
50000
15000
25000
35000
data throughpin (*10000)
dhtathroughput
Taoboo
380000
320000
9600
8500
330000
300000
- Lnoo_Jomp
5200
250000
temp
240000
4900
5000
10000
15000
20000
25000
50000
55000
thruuhrouph ('10000)
throughht (*10000)
Figure 4: Comparison accuracy and diversity metrics of different methods on Amazon books and Taobao data. HitRate @ 50 and Recall@50 measure the accuracy, and Distinct items @ 50 and Distinct cates @ measure the diversity. 'x *<tempp' means the users and items representations u and are 12-normalized, and then their dot product is rescaled with temperature way, the model converges much faster to higher accuracy. As for diversity, Bi-InfoNCE surpass Uni-InfoNCE and SSM by large margin.
0.067. By this
18

---

Under review as a conference paper at ICLR 2022
024
0.12
0.1
0.2
0.11
0.305
HRB30
0.21
-B-nor_2
nsc_
018
0.365
nec_10
037
0.00
600
data throughuut (*10000)
data datathroughout (*10000]
Figure 5: Results of Next-n-prediction for Amazon data. The improvement of accuracy is substantial when n <= 5.
KL Divergence
0.9
0.85
Tacbao
0.8
0.6
0.55
0.5
Figure KL Divergence between two marginal distributions of items: Pies (1) of test data and (d) of next-n training data. Although they are not the full distributions of u and i, to a certain extent they can reveal the discrepancy between the distributions of the training data and test data. In Amazon data, if we increase the discrepancy becomes smaller, and the accuracy also increases (See Fig. 5). In Taobao data, the discrepancy gets larger when we increase n, and experiments the accuracy does not increase.
19

---

Under review as a conference paper at ICLR 2022
Table 4: Statistics of the split Movielens datasets in PUMS experiments.
records
# movies
users
train (w/o negative samples)
990296
3076
6040
valid
5125
1257
2698
4788
1257
2547
past will be bought by more users in the future, and vice versa. So we holdout validation and test datasets in accordance with the overall distribution emulate the practical scenario.
A.7.1 POTENTIAL USERS OF A SINGLE ITEM
We create the training, validation and test data described in Sec. 5.2 The statistics of the resulting datasets shown Tab. 4
The Metrics
Our evaluation metrics of accuracy Precision@) N and Recall @ N:
KúN nu4
Precision@ N
Σ
test
&E
N
where Üs, denotes the top-N ranked users for and Ui is the set of users for the testing item i.
1
ÜNN nud
Recall @ N
Thest sEln
[244]
We count the distinct users of all the test items measure the diversity.
Movie lons
Morie-lnns
0.697
0.145
0.135
0.11
TR
MF
0.05
MI
6.95
512
3224
2043
A0D6
5155
1E300 32700 55538 300072 202344
6.1
0225
1152
30004 327G8 05036 135072 202344
data throuthroug/put (*10000)
data throughruhpuu (710000)
Figure 7: Comparison of the converging speed between MF and TR. The æ-axis in log scale, We can see that TR can obtain better results much faster.
Results of Converging Speed
The comparison of the converging speed can be found in Fig. 7 Similar to the results Fig. 3 of
RS, TR is much faster than MF with Bernoulli distribution.
A.7.2 POTENTIAL USERS OF A GROUP OF ITEMS
For given genre, define the popularity of its movies in this way: first rank the movies based on the number of users who have watched them, and then we pick the top/middle/bottom 10% movies different groups. So obtained 54 groups from 18 genres.
In practice, movies of different genres and different popularity will behave very differently, e.g., pre- cision and recall of popular Action movies and unpopular Western movies will differ lot. Movie-
20

---

Under review as a conference paper at ICLR 2022
Table 5: Overall performance of mining the potential users of a group of items. The % omitted for precision and recall. The
of MF, batch-size 4096 and T 0.125, and for TR batch-size = 512 and т = 0.111. We use 96 for all models.
model
precision 10
recall@ 10
distinct users
MFA [0:8] mean
5,93
1.52
MF. [8:0] sum
5.93
2.58
MF, [8:0] max
14.63
4.23
MF, [8:0] mean
17.04
4.57
246
MF, [4:4] mean
11.11
2.63
MF, [8:8] mean
12.78
3.05
TR [Uni-InfoNCE u:2i] mean
4.63
1.38
TR (Uni-InfoNCE i2u] mean
16.85
7.25
262
TR [Bi-InfoNCE] Sum
12.41
6.79
TR [Bi-InfoNCE) max
16.48
7.01
TR [Bi-InfoNCE]
mean
20.56
7.58
258
lens have 18 genres, and we divide the movies of the same genre into 3 categories based on their popularity,
split the chosen movies into 54 groups.
Given a group and its affined items, we define its potential users this way: if a user interacts with one more items in the group (excluding the items he has interacted previously), then he will be the potential users of the group. For example, given the group C { i2, 13} and the user u,, U1 has interacted with i in the historical data, then cannot pick him the potential user; if in the future data, ú1 interact with i i3, then he will be regarded as the potential user of G.
Different groups have different number of movies, c.g., the group 'Drama-top10% contains 93 movies, while 'Documentary top 10% has only 3 movies, and their precision and recall will also differ a lot. We report the averaged precision and recall over all groups Tab. 5
For small datasets like movie-lens- Im, the model generates the potential users of group this way: for each item the group, we first calculate de (u. of Eq. 2 and then the score PIR (u, i) exp(oo(u, i)) for TR and (MF(U, i) = (ón(u, i) ") for MF,, with respect the candidate users. Thus each candidate user will have several scores, and then we aggregate the scores to obtain match score between the user and the group, and finally pick the top-k users. The aggregation methods could be max, mean or sum
In Tab. 5 'MF, [0:8]' and "TR [Uni-InfoNCE u2i]' are specifically for u2i CG of RS, they give the worst results here. Models tailored for i2u ("MF, [8:0]' and "TR Unni-InfNNCE i2u]') and the general model for bidirectional CG ("TR [Bi-InfoNCE') perform much better.
The aggregation methods also affect the accuracy, and mean > max > sum. If a user has scores with many items of the group, the could be high even if individual scores are low, and this also implies that the user has few interactions with the items of the group in the historical data. So would not be suitable that sum ranks this user high. In contrast, mean and max will not have this problem, and mean considers more scores than just the max one and gives best results.
+a()) is the sigmoid function.
21


Reviews:
[{"review_id": "pT_8--PF8AW", "reviewer": "Reviewer_vmLg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "++ It is a good try to consider RS and AS together.\n\nHowever, there are major flaws in the current forms of the paper.\n\n-- The motivation is not strong. Why we should consider RS and AS jointly; what are the benefits and what are the challenges. Motivation should be stated more clearly.\n\n-- The technical contribution is not enough. Comparing the objective of the proposed model (see Eq. (3)), it is a direct expansion of the existing loss functions (see Eq. (1)) from uni-loss to bi-loss (user & item). \n\nSmaller ones:\n\n-- The organization and writing can be improved. The mathematical notations are not correct in some places (both in the body text and in the Appendix).\n\n-- For evaluating advertising systems, the MovieLens dataset is not a suitable one and the CTR (click-through rate) metric should be reported. \n\n-- many typos:\nIn Abstract: Amazaon -> Amazon, generation compare to strong baselines -> generation compared to strong baselines.\nIn Contributions: with higher accuracy compare to other methods -> with higher accuracy compared to other methods.\nLines before Eq. (2): find that l2-normalize u and i and then rescale the dot product -> find that l2-normalizing u and i and then rescaling the dot product.\n", "summary_of_the_paper": "This paper jointly considers the problems of recommender systems (RS) and advertising systems (AS) as it is a coin’s two sides. The key is to model the joint probability of (user, item) instead of conditional probabilities (user | item) and (item | user) independently. The proposed model adapts an existing large-vocabulary loss function to the optimization objective. Experiments on real-world datasets are conducted to compare with both Matrix Factorization and Sequential Modeling methods.", "summary_of_the_review": "Lacking motivation, contribution weak, evaluation not solid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oybBV8sYd86", "reviewer": "Reviewer_qfRw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of this work are listed as follows:\n\n1.\tThe proposed Bi-InfoNCE loss is easy to deploy.\n\n2.\tTR has improvements on both accuracy and diversity.\n\nHowever, this work has the following weaknesses:\n\n1.\tIt seems that the main contribution of this work is the proposed Bi-InfoNCE loss. However, similar contrastive learning (Uni-InfoNCE) method [1] has been proposed and verified in candidate generation module. It is straightforward that both users and items could be used as negative samples if we attempt to jointly provide results for recommender system (which provides items for a user) and advertising system (which provides users for an item). Therefore, the technical novelty is limited.\n\n2.\tRelated recommendation models [1] should be compared.\n\n3.\tIn PUMS, the authors only conduct evaluations on MovieLens-1M, which is not a real widely-used advertising dataset. Moreover, the baseline MF is also not sufficient and competitive enough in advertising. I suggest that the authors should conduct PUMS evaluations on real advertising datasets with SOTA advertising models.\n\n4.\tThe universality of TR is not verified. The authors should deploy TR on different real-world recommendation and advertising models for comprehensive comparisons.\n\n5.\tIn Table 2, why not use the original next-item-prediction task in evaluation?\n\n6.\tIn Table 3, will more negative users help to improve the performance of MF (e.g., MF_b[16:0]).\n\n7.\tIn Eq. (3), The Bi-InfoNCE loss further considers negative users compared to the Uni-InfoNCE loss used in [1]. However, does the improvements of Bi-InfoNCE mainly derive from more negative samples? Since larger batch sizes may improve the performance given in [2]. The authors can select two negative user sets in Eq. (3), or just double the batch size of the Uni-InfoNCE loss for more comparisons.\n\n\nReferences:\n\n[1] Zhou C, Ma J, Zhang J, et al. Contrastive learning for debiased candidate generation in large-scale recommender systems[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021: 3985-3995.\n\n[2] Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[C]//International conference on machine learning. PMLR, 2020: 1597-1607.\n", "summary_of_the_paper": "This work proposes a bidirectional candidate generation framework for both recommender system (which provides items for a user) and advertising system (which provides users for an item). Specifically, the authors conduct a Bi-InfoNCE loss on the classical two-tower architecture to jointly learn both user and item representations. The Bi-InfoNCE loss considers each (u, i) pair as a positive instance, and selects both users and items as negative instances in InfoNCE. In experiments, the authors find that the proposed TR model outperforms MF in both recommender system and advertising system datasets.", "summary_of_the_review": "The technical novelty is limited. Moreover, the improvements and universality of TR are not perfectly verified.\n\nIn conclusion, I will vote for Reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SJZdvqpqI_", "reviewer": "Reviewer_eQ3i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-organized and clearly-written. The main idea is clear and easy to understand.\n2. The proposed Bi-InfoNCE loss helps to model the joint distribution for bidirectional candidates generation.\n3. The authors conduct experiments on several public datasets.\n\nWeakness:\n1. The idea of introducing InfoNCE loss to recommender system is not novel, which has been used in existing works. The main contribution of this paper is the design of Bi-InfoNCE loss to optimize the joint distribution $p_{data}(u,i)$. However, it is not demonstrated why modeling $p_{data}(u,i)$ would be better than separately modeling $p_{data}(u|i)$ and $p_{data}(i|u)$ for u2i or i2u candidate generation. From the results in Table 2, Uni-InfoNCE u2i performs even better than Bi-InfoNCE, which further weakens the motivation of the Bi-InfoNCE loss.\n2. Besides, in Table 2, ComiRec methods perform significantly better than the proposed methods for Taobao dataset, but no discussion is provided. In this way, it is not approporiate to say ''TR achieves much better results by a large margin''.\n3. The comparison results to MF in Figure 2 and Table 1 are not promising. TR performs better for Pinterest but MF achieves better HR for Movielens. Results at other truncated top values are not provided. The overall improvement over MF with point-wise loss is not significant.", "summary_of_the_paper": "This paper proposes a method for both user2item and item2user candidates generation in recommender systems. The main idea is to employ infoNCE loss for optimizing the two tower model. To achieve bidirectional candidates generation, the authors propose to do negative sampling in batch from both user side and item side, and use the Bi-InfoNCE loss for optimization.", "summary_of_the_review": "The advantage of the proposed Bi-InfoNCE loss is neither clear demonstrated nor well supported by the experimental results. Thus I tend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7MhefJ0uOsi", "reviewer": "Reviewer_knEH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nThe overall architecture seems to be novel in the literature to my knowledge. This paper has tried to evaluate the proposed method from various aspects through several experiments.\n\nCons:\n\nOne of the claims will be that jointly addressing recommendations and advertising is effective in e-commerce services. However, it seems this paper did not sufficiently motivate the claim. Although Sec. 1 and Sec. 2.2 cite several related studies, those studies lack a discussion of situations in which both recommendation and advertising are performed in one e-commerce platform. I think it would help the reader's understanding if this paper explains how such platforms do recommendations and advertising and how the proposed joint approach is helpful for them in more detail. Also, if any previous studies perform both recommendation and advertising simultaneously, I recommend citing them.\n\nWhile the proposed method will contain some promising ideas, it seems that its technical novelty is somewhat weak. For example, SimCLR (Chen et al., 2020) has considered the idea of using the l2 normalization and temperature parameter of a similar score function. The Bi-InfoNCE loss is an appealing idea but does not seem to be completely novel. Similar ideas have already been proposed in different contexts [1,2,3]. Since those papers have recently been made available, I perceive them as contemporaneous work of this paper. However, considering them, I feel the derivation of the Bi-InfoNCE loss is reasonable but fairly straightforward, and the uniqueness and technical contribution is somewhat weak.\n\nMoreover, this paper did not explain well how the proposed method is effective in Sec. 3. For example, this paper calls the proposed method 'TotalRecall.' I interpreted the proposed method takes care of the improvement of recall, but Sec. 3 did not provide any implications. Also, this paper claims that the proposed method was effective in terms of convergence speed and diversity in the experiments. However, Sec. 3 did not clearly explain what part of the proposed method induces such effectiveness. I feel that the contribution of the proposed method is somewhat weak if it just happens to perform well in the experiments. For example, in Sec. 3.2.2, this paper claims modeling s_u can 'boost the converging speed a lot.' I recommended presenting the reason more quantitatively in Sec. 3 rather than just showing experimental results.\n\nWhile the experiments tried to evaluate the proposed method from various perspectives, the results do not always strongly support the effectiveness of the proposed method. For example, the performance of the Uni-InfoNCE outperformed the one of the Bi-InfoNCE in Table 2. In Fig. 4, the 'bi_nce_temp' method does not significantly improve the results compared to the 'ssm_temp' method for HR and recall. Also, there are many parts where noteworthy experimental results are missing. In Sec. 4.1, the experiments did not show the results for Uni-InfoNCE. There are two variants of Uni-InfoNCE (u2i and i2u), but the results of i2u are not shown in Table 2. Although many missing results may not work out so well, I recommend presenting more comprehensive and consistent results for all experiments.\n\nThe clarity of the paper appears to be somewhat poor. This paper has so many unclear parts about the proposed method and experiments. Please see the comments below.\n\n- The notation 'N' is used many times with different meanings in the whole paper. I recommend consistently using it throughout the manuscript.\n- In the experiments, how many negative samples were used for all the methods in each experiment? I interpreted this paper used 100 negative samples in Sec. 4 according to the description of A.4.1. Did the TR methods also use 100 samples in Sec. 5? If so, it means that the TR methods used much larger samples than MF_b methods. Why?\n- In the experiments, this paper refers too much to the figures in the appendix to explain the main experimental results. The appendix would be a material to support the main part, and I think it would be better to put the main results in the appendix much more sparingly.\n- In the experiments, why was the effect of different sizes of negative samples only evaluated in Sec. 5? It would be more consistent to do the same assessment in Sec. 4.\n- In the experiments, please ensure what the bold and underline highlighting means for all tables. Table 4 appears to have duplicate underlines.\n- Sec. 1: this paper mentions 'Theoretically we show that the superiority is due to the modeling of joint probability of u and i.' However, I did not find any theoretical claims to show the superiority of the joint distribution.\n- Sec. 2.3: I recommend specifying x and y of Eq. (1).\n- Sec. 3.1: what exactly is y_{u,t}? Is this an indicator variable of a specific item that was clicked on by u at time t?\n- Sec. 3.2.2 and A.1: I did not know what the variable s_u indicates. According to the text, s_u is drawn from the multinomial distribution Mult(N_u, p_u), which means that s_u is a K-dimensional count vector where s_{u, i} (i=1, ..., K) denotes the i-th element of s_u and corresponds to the number of interactions between u and i. However, Eq. (5) refers to s_u as an indicator of item i. Also, if S_{u, i} is a binary matrix, s_u will result in a binary vector. As in (Liang et al., 2018), the observation of s_u may happen to be binary, but this paper fixes the variable to be binary in the problem definition of Sec. 3.1.\n- Sec. 3.4.1: I did not understand why this paper chose a classical two-tower architecture. It seems that the presented two reasons do not exclude other possible architectures.\n- Sec. 3.4.1: 'in online serving .' -> 'in online serving.'\n- Sec. 3.4.1: the notation of f_{\\theta}(u) is confusing since it quite overlaps the notation of f_{\\theta}(u, i) in Eq. (1). Also, f_{\\theta}, g_{\\theta}, and \\phi_{\\theta} are also confusing since they share the same parameter \\theta. I recommend using different variables or specifying the detail of \\theta.\n- Sec. 3.4.1: \\phi_{\\theta} is used for both unnormalized and normalized score functions, which are confusing a little. For both functions, it is not clear what \\theta is.\n- Sec. 3.4.2: the definition of |S_{u, i}| seems unclear in Eq. (3). S_{u, i} is defined as a matrix, and the notation of |S_{u, i}| is not specified.\n- Sec. 4.1: 'NCF' of the caption of Figure 2 is not defined in this paper.\n- Sec. 4.1 and Sec. 5.2: this paper claims the TR methods converge faster than compared methods from the results of Fig. 3 and Fig. 7, but their x-axis denotes the number of processed data samples. I recommend showing the results with actual runtime on the x-axis.\n- Sec. 4.1: it would be better to explain what distinct items are before introducing Fig. 3. The term 'distinct items' is first explained in A.4.2.\n- Sec. 4.2: I didn't understand how to make the data for next-n-item prediction, which should be explained in Sec. 3.1. Also, this paper should ensure that using such data leads to fair comparison for all methods.\n- Sec. 4.2: why did not the experiment report the NDCG as in (Cen et al., 2020)?\n- Sec. 4.2: the difference of four (TR-)Youtube methods in Table 2 seems to be not explained.\n- Sec. 4.2: why did not Table 2 report the result of 'TR-Youtube [next-7-prediction]' for the Taobao dataset?\n- Sec. 4.2: how does each method in Fig. 4 correspond to the method in Table 2? Also, I did not fully understand the meaning of '_temp' in the caption of Fig. 4.\n- Sec. 4.2: what is the coherent vector? How is it induced by Eq. (3) and how does it improve diversity?\n- Sec. 5: in the first paragraph, please provide more specific examples that motivate mining for a group of similar items in real E-commerce services.\n- Sec. 5: in the first paragraph, this paper mentions 'Previously, ... so many models.' However, in the experiment of Sec. 5, only 54 item groups were used. I think binary classification approaches are also applicable for item groups.\n- Sec. 5.1: this paper mentions that 'We use Movielens-1m because it is also used in MFb of RS.' However, the Pinterest dataset was also used in the experiment of Sec. 4. Why was only the Movielens-1m dataset used?\n- Sec. 5.2: this paper mentions 'TR uses the same training data and model configurations for both RS and PUMS.' However, the procedure for creating the datasets appears to be different in Sec. 4 and Sec. 5. I didn't know if this paper used the same TR models for both experiments.\n- Sec. 5.2 and Sec. 5.3: I recommend showing all the scores of the distinct users even if the accuracies are very low.\n- Sec. 5.3: 'For small dataset' -> 'For small datasets'?\n- Sec. 5.3: I recommend showing all the results of aggregation methods: several results with max and sum are missing in Table 4.\n- A.1: did s_{u, i} follow the Gauss distribution even if it is a binary random variable?\n- A.2: S -> S_{u,i}?\n- A.2: what is exactly N in Eq. (8)? Intuitively, I think Eq. (8) should be divided by M * K.\n- A.3: I did not understand the claim of this section. I recommend providing more formal proofs to show the convergence of the score function.\n- A.4: I think this section is not well-structured. Although the titles of subsections indicate the considered methods, they explain the evaluation metrics.\n\n\n[1] Nguyen et al., Contrastive Learning for Natural Language-Based Vehicle Retrieval, CVPR 2021 Workshop.\n\n[2] Bai et al., Connecting Language and Vision for Natural Language-Based Vehicle Retrieval, CVPR 2021 Workshop.\n\n[3] Lee et al., Compressive Visual Representations, arXiv:2109.12909.\n\n==========\n\nI appreciate that the authors have responded to my concerns one by one. While some of the responses are convincing and have been reflected in the manuscript, it appears that not all the responses have been considered in the revision yet. I would like to see the completed version. In the following, I have two suggestions for the revised version.\n\nThe revision has added Sec. 3.6 to explain additional motivation. However, I wonder if the contents should be presented before explaining the proposal method, such as Sec. 1 or the first half of Sec. 3. Discussing motivation at the end of Sec. 3 makes the presentation feel like it is not progressing seamlessly. Also, it seems the response claims that the motivation is to save computational resources. Since it becomes a key claim of this paper, it will be better to show the results of computational efficiency in the experiments compared to the approach with separate models. Please note that the efficiency is not trivial: the resources for separate models could be smaller than the proposed joint approach for achieving enough performance in practice. Also, if the proposed method is running on an actual platform, I think the paper would be more convincing if this paper show the results of deploying it. If a method makes profits, showing those results is often seen as an influential contribution to the studies about online platforms (e.g., [4]).\n\nThe response often mentioned that the reason why some of the experimental results are missing is that those results were not so good and readers may be overwhelmed. I don't believe that not showing the results is an appropriate manner to report experiments. If this paper does not show the results, readers cannot know how good or bad the results are. Even if the results are not so good, reporting all the results will provide valuable information for subsequent studies. In the first place, readers will not be overwhelmed even if this paper shows the missing results. Bolding and underlining should be used for the readers who only want to know a summary of the results.\n\n[4] McMahan et al., Ad Click Prediction: a View from the Trenches, KDD 2013.\n", "summary_of_the_paper": "This paper proposes to address candidate generations of users and items for recommender and advertising systems simultaneously by a single proposed model. The proposed model provides two ideas for existing two-tower recommendation models: it introduces a normalization of the score function and a bidirectional version of the InfoNCE loss. This paper evaluated the proposed method through several experiments.\n", "summary_of_the_review": "While this paper addresses an interesting problem, the significance is not sufficiently discussed in the paper. The novelty and effectiveness of the proposed method will be quite weak. The clarity needs to be improved.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are in agreement that the paper is below the acceptance threshold. Main concerns focus around novelty, experiments, and justification of the paper's main claims.
Decision: Reject