Year: 2019

Abstract:
Cold-start and efficiency issues of the Top-k recommendation are critical to largescale recommender systems. Previous hybrid recommendation methods are effective to deal with the cold-start issues by extracting real latent factors of cold-start items(users) from side information, but they still suffer low efficiency in online recommendation caused by the expensive similarity search in real latent space. This paper presents a collaborative generated hashing (CGH) to improve the efficiency by denoting users and items as binary codes, which applies to various settings: cold-start users, cold-start items and warm-start ones. Specifically, CGH is designed to learn hash functions of users and items through the Minimum Description Length (MDL) principle; thus, it can deal with various recommendation settings. In addition, CGH initiates a new marketing strategy through mining potential users by a generative step. To reconstruct effective users, the MDL principle is used to learn compact and informative binary codes from the content data. Extensive experiments on two public datasets show the advantages for recommendations in various settings over competing baselines and analyze the feasibility of the application in marketing.

heading: 1 INTRODUCTION
text: With the explosion of e-commerce, most customers are accustomed to receiving a variety of recommendations, such as movies, books, news, or hotels they might be interested in. Traditional recommender systems just recommended items that are similar to what they liked or rated in the previous. Recommendations help users find their desirable items, and also creates new revenue opportunities for vendors, such as Amazon, Taobao, eBay, etc. Among them, one of the most popular recommendation methods, collaborative filtering is dependent on a large amount of user-item interactive information to provide an accurate recommendation. However, most of new e-commerce vendors do not have enough interactive data, which leads to low recommendation accuracy, i.e., cold-start issues.
Previous studies on cold-start issues generally modeled as a combination of collaborative filtering and content filtering, known as hybrid recommender systems. Specifically, they learned real latent factors by incorporating the side information into the interactive data. Such as Collaborative Deep Learning (CDL) (Wang et al., 2015), Visual Bayesian Personalized Ranking (VBPR) (He & McAuley, 2016), Collaborative Topic modeling for Recommedation (CTR) (Wang & Blei, 2011), and the DropoutNet for addressing cold start (DropoutNet)(Volkovs et al., 2017), ABCPRec for Bridging Consumer and Producer Roles for User-Generated Content Recommendation (ABCPRec)(Tsukuda et al., 2019). All of the above hybrid recommender systems were modeled in real latent space, which leads to low efficiency for the online recommendation with the increasing scale of datasets.
Recent studies show the promising of hashing based methods to tackle the efficiency challenge by representing users and items with binary codes (Zhang et al., 2014; Zhou & Zha, 2012; Zhang et al., 2016; Liu et al., 2019), because the preference score can be represented by the Hamming distance calculated via XOR operation efficient (Wang et al., 2014). However, the existing hashing based recommendations are learning-based frameworks, which leads to NP-hard problems of optimizing
discrete objectives. Thus many scholars learned binary codes by some approximate techniques, such as the two-stage hashing learning method utilized in Preference Preserving Hashing(PPH) (Zhang et al., 2014) and the Iterative Quantization(ITQ) (Zhou & Zha, 2012). To reduce information loss, two learning-based hashing frameworks: bit-wise learning and block-wise learning were respectively proposed in hashing based recommendation frameworks (Zhang et al., 2016; Wang et al., 2019; Zhang et al., 2018; Zheng et al.).
However, due to the requirement of binary outputs for learning-based hashing frameworks, the training procedure is expensive for large-scale recommendation, which motivates us to propose a generative approach to learn hash functions. In this paper, we propose the collaborative generated hashing(CGH) to learn hash functions of users and items from content data with the principle of Minimum Description Length (MDL) (Dai et al., 2017).
In marketing area, mining potential customers is crucial to the e-commerce. CGH provides a strategy to discover potential users by the generative step. To reconstruct effective users, uncorrelated and balanced limits are imposed to learn compact and informative binary codes with the principle of the MDL. Especially, discovering potential customers is vital to the success of adding new items for a recommendation platform (Papies et al., 2017). Specifically, for a new item, we can generate a new potential user by the generative step (detailed in Section 2.1), and then search the nearest potential users in the user set. By recommending a new product to the potential users who might be interested in but didn’t plan to buy, further e-commerce strategies can be developed to attract those potential users.
We organize the paper as follows: Section 2 introduce the main techniques of CGH. We first introduce the framework of CGH and compare it with the closely related competing baselines: CDL (Wang et al., 2015) and DropoutNet (Volkovs et al., 2017); we then formulate the generative step in Section 2.1 and the inference step in Section 2.2, respectively; we finally summarize the training objective and introduce the optimization in Section 2.3. Particularly, we demonstrate the process of mining potential users for the marketing application in Section 2.1. Section 3 presents the experimental results for marketing analysis and recommendation accuracy in various settings. Section 4 concludes the paper.
The main contributions of this paper are summarized as follows:
(1) We propose the Collaborative Generated Hashing (CGH) with the principle of MDL to learn compact but informative hash codes, which applies to various settings for recommendation.
(2) We provides a marketing strategy by discovering potential users by the generative step of CGH, which can be applied to boost the e-commence development.
(3) We evaluate the effectiveness of the proposed CGH compared with the state-of-the-art baselines, and demonstrate its robustness and convergence properties on the public datasets.

heading: 2 COLLABORATIVE GENERATED HASHING
text: The framework of the proposed CGH is shown in Fig. 1(c), where U , V and R are respectively observed user content, item content and rating matrix. B and D are binary codes of users and items, respectively. CGH consists of the generative step marked as dashed lines and the inference step denoted by solid lines. Once training is finished, we fix the model and make forward passes to obtain binary codes B and D through the inference step, and then conduct recommendation. For the marketing application, we create a new user via the generative step.
In comparison of CGH with the closely related baseline CDL (Wang et al., 2015), the proposed CGH aims to learn binary codes instead of real latent vectors P and Q due to the advantage of hashing for online recommendation; plus the CGH optimizes an objective with the principle of MDL, while CDL optimized the joint objective of rating loss and item content reconstruction error. In comparison of CGH with DropoutNet (Volkovs et al., 2017), CGH can be used as a marketing strategy by discovering potential users; plus CGH learns hash functions by stacked denoising autoendoer, while DropoutNet obtained real latent factors by the standard neural network.
In the following we start by first formulating the generative process and demonstrating the application in marketing area; we then formulate the inference step; we finally summarize the training objective and the optimization method.

heading: 2.1 MINING POTENTIAL USERS
text: . Give a sparse rating matrix R and item content data V ∈ Rdv , where dv is the dimension of the content vector, and V is stacked by the bag-of-words vectors of item content in the item set V . most previous studies were focus on modeling deterministic frameworks to learn representations of items for item recommendation, such as CDL, CTR, DropoutNet, et.al. In this paper, we discover a new strategy from a perspective of marketing for item recommendation – mining potential users.
We demonstrate the process of mining potential users by an item through the generative step in Fig. 2. After the inference step, the binary code of item j is available. By maximizing the similarity function δ(bi,dj) (detailed in Section 2.1), the optimal binary code bp is obtained. Then we generate the new user up through the generative step. Finally we find out potential users from the user set by some nearest neighborhood algorithms, such as KNN. As a marketing strategy, it can discover potential users for both warm-start items and cold-start items. Thus, from a perspective of marketing, it can be regarded as another kind of item recommendation.
The generation process (also referred as decoding process) is denoted by dashed lines in Fig. 1 (c). Fix binary codes bi and dj of the user i and the item j, the bag-of-words vector ui of the user i (vj of the item j) is generated via p(θu). The ratings rij is generated by bi and dj . We use a simple Gaussian distribution to model the generation of ui and vj given bi and dj like Stochastic Generative Hashing (SGH) (Dai et al., 2017):
p(ui|bi) = N (Cubi, λu-1I ), p(vj |dj) = N (Cvdj , λv -1I ),
(1)
where Cuk = [cuk]rk=1, cuk ∈ Rdu is the codebook (Dai et al., 2017) with r codewords, which is similar for Cv , and du is the dimension of the bag-of-the-words vector of users. The prior is modeled as the multivariate Bernoulli distribution on hash codes: p(bi) ∼ B(ρu), and p(dj) ∼ B(ρv), thus the prior probability are as follows:
p(bi) = ∏r
k=1 ρbikui (1− ρui) 1−bik , p(dj) = ∏r
k=1 ρ djk vj (1− ρvj)
1−djk , (2)
We formulate the rating with the similarity between binary codes of users and items like the most successful recommender systems, matrix factorization (Koren et al., 2009). Then the rating is thus drawn from the normal distribution centered at the similarity value,
p(rij |bi,dj) ∼ N (δ(bi,dj), C−1ij ), (3)
where δ(bi,dj) = 1 − 1rHamdis(bi,dj) denotes the similarity between binary codes bi and dj . Hamdis(bi,dj) represents Hamming distance between the two binary vectors, which has been widely applied in hashing-based recommender system (Wang & Blei, 2011; Lian et al., 2017; Zhang et al., 2018). Cij is the precision parameter that serves as confidence for rij similar to that in CTR (Wang & Blei, 2011) (Cij = a if rij = 1 and Cij = b otherwise) due to the fact that rij = 0 means the user i is either not interested in item j or not aware of it.
With the generative model constructed, the joint probability of both observed ratings, content vectors and binary codes is given by
p (R,U ,V ,B,D) = ∏ i,j p(rij |bi,dj)p(ui|bi)p(vj |dj)p(bi)p(dj) (4)

heading: 2.2 CONSTRAINTS ON BINARY LATENT VARIABLES
text: The inference process (also referred as encoding process) shown in the Fig. 1 (c) with dashed lines, the binary latent variables bi (dj) depends on the content vector ui (vj) and the ratingR (shadowed in Fig 1). Inspired by the recent work on generative hashing (Dai et al., 2017) and DropoutNet (Volkovs et al., 2017), we use a multivariate Bernoulli distribution to model the inference process of bi and dj with linear parametrization, i.e.,
q(bi|ũi) = B(σ(TuT ũi)) q(dj |ṽj) = B(σ(TvT ṽj)),
(5)
where ũi = [ui,pi], ṽj = [vj , qj ]. pi and qj are the results of r-dimension matrix factorization (Koren et al., 2009) of R, i.e., rij ≈ pTi qj . Tu = [tuk]rk=1, tuk ∈ Rdu+r, Tv = [tvk]rk=1, tvk ∈ Rdv+r are the transformation matrices of linear parametrization. From SGH (Dai et al., 2017), the MAP solution of the eq. (5) is readily given by
bi = argmax bi
q(bi|ui) = sign(T Tu ui) + 1
2 ,
dj = argmax dj
q(dj |vj) = sign(T Tv vj) + 1
2 .
(6)
With the linear projection followed by a sign function, we can easily get hash codes of users and items. However, hashing with a simple sign function suffers from large information loss according to (Zhang et al., 2016), which motivates us to add constraints on parameters in the inference step.
To derive compact and informative hash codes for users and items, we add balanced and uncorrelated constraints in the inference step. The balanced constraint is proposed to maximize the entropy of each binary bit (Zhou & Zha, 2012), and the uncorrelated constraint makes each bit is independent of others. Then we can obtain compact and informative hash codes by the following constraints,
Balanced constraint: ∑ k bik = 0, ∑ k djk = 0
Uncorrelated constraint: bibTi = Ir,djd T j = Ir
(7)
From the eq. (6), bi and dj are only dependent on parameters Tu and Tv , respectively, thus we add constraints on Tu and Tv directly. So eq. (7) is equivalent to the following constraints,
Balanced constraint: T Tu 1r = 0, T Tv 1r = 0, Uncorrelated constraint: T Tu Tu = Idu+r, T Tv Tv = Idv+r.
(8)
By imposing the above constraints in the training step, compact and informative hash codes can be obtained through the inference process. Next we summarize the training objective and its optimization.

heading: 2.3 TRAINING OF CGH
text: Since our goal is to reconstruct users, items and ratings by using the least information of binary codes, we train the CGH with the MDL principle, which finds the best parameters that maximally compress the training data and meanwhile keep the information carried, thus CGH aims to minimize the expected amount of informations related to q:
L(q) =Eq[log p(R,U ,V ,B,D)− log q(B,D)] =Eq[log p(R|B,D) + log p(U |B) + log p(V |D)−KL(q(B|Ũ)||p(B))− KL(q(D|Ṽ )||p(D))]
(9)
Maximizing the posterior probability is equivalent to maximizing L(q) by only considering the variational distribution of q(B,D), the objective becomes
LMAP (Θ,Φ) =− ∑ i,j Cij 2 (rij − δ(bi,dj))2 − λu 2 ∑ i (ui − Cvbi)2 − λv 2 ∑ j (vj − Cvdj)2
− KL(qφu ||pθu)− KL(qφv ||pθv )−∇(Θ,Φ) (10)
where Θ = {θu,θv}, Φ = {φu,φv}, ∇(Θ,Φ) is the regularizer term with parameters Θ and Φ. By training the objective in eq. (10), we obtain binary codes, but some bits probably be correlated. To minimize the reconstruction error, SGH had to set up the code length as long as r = 200. Our goal in this paper is to obtain compact and informative hash codes, thus we impose the balance and independent constraints on hash codes by eq. (8). Maximizing the eq. (10) is transformed to minimizing the following constrained objective function of the proposed Collaborative Generative Hashing (CGH),
LCGH(Θ,Φ) = ∑ i,j Cij 2 (rij − δ(bi,dj))2 + λu 2 ∑ i (ui − Cubi)2 + λv 2 ∑ j (vj − Cvdj)2+
KL(qφu ||pθu) + KL(qφv ||pθv ) + αu ∥∥T Tu 1r∥∥22 + αv ∥∥T Tv 1r∥∥22 +
βu ∥∥T Tu Tu − Idu+r∥∥22 + βv ∥∥T Tv Tv − Idv+r∥∥22 +∇(Θ,Φ).
(11)
The objective of CGH in eq. (11) is a discrete optimization problem, which is difficult to optimize straightforwardly, so in the training stage, the tanh function is utilized to replace the sign function in the inference step, and then the continuous outputs are used as a relaxation of hash codes.
With the relaxation, we train all components jointly with back-propagation. After training, we fix them and make forward passes to map the concatenate vectors in Ũ and Ṽ to binary codes B and D, respectively. The recommendation in various settings is then conducted using B and D by the similarity score estimated as before δ(bi,dj) = 1− 1rHamdis(bi,dj). The training settings are dependent on the recommendation settings, i.e, warm-start, cold-start item, and cold-start user. LCGH(Θ,Φ) aims to minimize the rating loss and two content reconstruction errors with regularizers. (a.) For the warm-start recommendation, ratings for all users and items are available, then the above objective is trivially optimized by setting the content weights to 0 and learning hashing function with the observed ratingsR. (b.) For the cold-start item recommendation, ratings for some items are missing, then the objective is optimized by setting the user content weight to 0 and learning parameters with the observed ratings R and item content V . (c.) The training setting for the cold-start user recommendation is similar to the cold-start item recommendation.

heading: 3 EXPERIMENTS
text: We validate the proposed CGH on two public dataset: CiteUlike1 and RecSys 2017 Challenge dataset2 from the following two aspects:
(1) Marketing analysis. To validate the effectiveness of CGH in marketing area, we fist defined a metric to evaluate the accuracy of mining potential users; we then test the performance for warm-start item and cold-start item, respectively.
(2) Recommendation performance. We test the performance of CGH for recommendation in various settings including: warm-start, cold-start item, and cold-start user in terms of Accurcy@k (Yin et al., 2014).
In the following, we first introduce the experimental settings, followed by the experimental results analysis from the above aspects.

heading: 3.1 EXPERIMENTAL SETTINGS
text: To evaluate the power of finding out potential users and the accuracy of recommendation in different settings. (1) For the CiteUlike dataset, it contains 5,551 users, 16,980 articles, 204,986 observed user-article binary interaction pairs, and articles abstract content. Similar to (Wang & Blei, 2011), we extract bag-of-the-words item vector with dimension dv = 8000 by ranking the TF-IDF values. (2) For the RecSys 2017 Challenge dataset, it is the only publicly available datasets that contains both user and item content data enabling both cold-start item and cold-start user recommendation. It contains 300M user-item interactions from 1.5M users to 1.3M items and content data collected from the career oriented social network XING (Europern analog of LinkedIn). Like (Volkovs et al., 2017), we evaluate all methods on binary rating data, item content with dimension of du = 831 and user content with the dimension of dv = 2738. user features and 2738 item features forming the dimensions of user and item content .
We randomly split the binary interaction (rating)R into three disjoint parts: warm start ratingsRw, cold-start user ratingsRu, and cold-start item ratingsRv , andRw is furtherly split into the training dataset Rwt and the testing dataset Rwe. Correspondingly, the user and item content datasets are split into three disjoint parts. The randomly selection is carried out 5 times independently, and we report the experimental results as the average values.

heading: 3.2 EVALUATION METRIC
text: The ultimate goal of recommendation is to find out the top-k items that users may be interested in. Accuracy@k was widely adopted by many previous ranking based recommender systems (Koren, 2008; Chen et al., 2009). Thus we adopt the ranking-based evaluation metric Accuracy@k to evaluate the quality of the recommended item ranking list.
Metric for Marketing Application. For a new application of the recommender system, there haven’t yet a metric to evaluate the marketing performance. Thus, in this paper, we define an evaluation metric similar to the ranking-based metric Accuracy@k used for the warm-start and cold-start recommendation in this paper.
From Fig. 2, we discover the k nearest potential users for an item j. The basic idea of the metric is to test whether the user that really interested in an item appears in the k potential users list. For each positive rating (rij = 1) in the testing dataset Dtest: (1) we randomly choose 1000 negative users (users k with rkj = 0) and find k potential users in the 1001 user set; (2) we check if the positive user i (with positive rating rij = 1) appears in the k potential users list. If the answer is ’yes’ we have a ’hit’ and have a ’miss’ otherwise.
The metric also denoted by Accuracy@k is formulated as:
Accuracy@k = #hit@k
|Dtest| , (12)
1http://www.citeulike.org/faq/data.adp 2http://www.recsyschallenge.com/2017/
where |Dtest| is the size of the test set, and #hit@k denotes the number of hits in the test set.

heading: 3.3 ACCURACY FOR MINING POTENTIAL USERS
text: The experiments evaluate the performance of the marketing application in mining potential users for warm-start items on the test dataset Rwe and cold-start items on Rv . Specifically, we first train the model on the training dataset Rwt and the corresponding user and item content data. When the training is completed, we fix parameters and obtain hash codes bi and dj by making forward passes. Then we generate k potential users for items in the test dataset by the procedure demonstrated in Fig. 2, and evaluate the quality of the potential users list by Accuracy@k defined in Section 3.2. The
marketing analysis for warm start item and cold-start item are reported in Fig. 3 (Left.), which shows the accuracy values varies with the numbers of potential users. It indicates the accuracy increases with the number of potential users for both cold-start and warm start settings. It’s reasonable because mining more potential users will have greater accuracy value defined in Section 3.2. Especially, the proposed CGH is effective for cold-start item, which indicates further e-commerce strategies can be developed for new items to attract those potential users. Besides, from the perspective of marketing, warm-start recommendation and cold-start recommendation has less gap than traditional recommendation.
Robust Testing. We evaluate the performance varies with the number of users who really interested in the target item in test set. The experimental results shown in Fig. 3 (Center.) indicates the accuracy grows steadily with the size of test set, which reveals the CGH for marketing application is robust. Thus, it is practical to be used in the sparse and cold-start settings.
Convergence of CGH. Fig. 3 (Right.) demonstrates the convergence of the proposed CGH, which reveals the reconstruction errors of ratings, users content, items content and the total error with the number of samples seen by CGH are converged, which furtherly validate the correction and effectiveness of the proposed CGH.

heading: 3.4 ACCURACY FOR RECOMMENDATION
text: Accuracy for warm-start Recommendation. Fig. 4 (Left.) shows the accuracy comparison of warm-start recommendation on CiteUlike dataset. In which collaborative generated embedding (CGE) denotes the real version of CGH. The figure shows the proposed CGH (CGE) has a comparable performance with other hybrid recommender systems. The proposed CGH is hashing-based recommendation, hence binary vectors apply to recommendation which has the advantage in online recommendation as introduced in Section 1; while the baselines are real-valued recommendations which conducts recommendation on real latent space. Due to real latent vectors intuitively carried more information than hash codes. Thus it is acceptable to have small gaps between the real-valued hybrid recommendation and the hashing-based recommendation. In addition, there is still small gap of the real version CGE in comparison with DropoutNet, because the reconstruction error is consid-
ered in CGH(CGE), while DropoutNet didn’t consider it. However, the reconstruction is significant in the generative step of CGH, which makes it feasible to mining effective potential users, thus CGH(CGE) has the advantage in marketing application.
Accuracy for cold-start item recommendation. This experiment studies the accuracy comparison between competing hybrid recommender systems and CGH under the same cold-start item setting. We test the performance on the test dataset Rv introduced in Section 3.1. Specifically, in Rv each item (cold-start item) has less than 5 positive ratings. Then we select users with at least one positive rating as test users. For each test user, we first choose his/her ratings related to cold-start items as the test set, and the remaining ratings as the training set. Our goal is to test whether the marked-off cold-start items can be accurately recommended to the right user.
The experimental results for cold-start item recommendation are shown in Fig. 4 (Center.). We conclude that CGH has a comparable performance with competing baselines and achieves better performance than CTR. The results evaluated by another metric MRR (detailed in Appendix.A) are similar.
Accuracy for cold-start user recommendation. We also test the performance on the cold-start user setting on the test dataset Ru introduced in Section 3.1. Specifically, in Ru, each user (cold-start user) has less than 5 positive ratings. Then we select items with at least one positive rating as test items. For each test item, we first choose ratings related to cold-start users as the test set, and the remaining ratings as the training set. Our goal is to test whether the test item can be can be accurately recommended to marked-off user.
Due to the fact that only Dropout can be applied to cold-start user recommendation, so we only compare the performance of CGH with Dropout. The experimental results for cold-start user recommendation shown in Fig. 4 (Right.) indicates our proposed CGH has similar performance with DropoutNet. Besides, CGH has the advantage of the application in marketing area.

heading: 4 CONCLUSION
text: In this paper, a generated recommendation framework called collaborative generated hashing (CGH) is proposed to address the cold-start and efficiency issues for recommendation. The two main contributions are put forward in this paper: (1) we develop a collaborative generated hashing framework with the principle of Minimum Description Length together(MDL) with uncorrelated and balanced constraints on the inference process to derive compact and informative hash codes, which is significant for the accuracy of recommendation and marketing; (2) we propose a marketing strategy by the proposed CGH, specifically, we design a framework to discover the k potential users by the generate step; (3) we evaluate the proposed scheme on two the public datasets, the experimental results show the effectiveness of the proposed CGH for both warm-start and cold-start recommendation.

heading: A. MRR RESULTS ON FOR RECOMMENDATION
text: We evaluate the accuracy in terms of the MRR (Yin et al., 2014) metric shown in Table 1 for warmstart recommendation. Our proposed CGH performs almost as well as the best result of the realvalued competing baselines. Table 1 summarizes MRR results for the four algorithms, the best result is marked as ‘?q and the second best is marked as ‘oq . We find that the performance of CGH is very close to the best result, that is consistent with the outcome of Accuracy@k reported in Fig. 4.

Review summaries:
This paper introduces a collaborative generated hashing (CGH) method to learn hash funcations of users and items from content data. The approach first provides a strategy to discover potential users by the generative step and inference through adding balanced and uncorrelated constraints. The experiments demonstrates some effectiveness on improving accuracy for both warm-start and cold-start recommendations.

The work considers the problem of efficient user and item recommendations in the warm- and cold-start settings. It aims at improving computational efficiency of the best candidate selection in these settings by utilizing binary codes representation. The transformation from an actual to a binary code representation is learned in a hybrid manner using both collaborative and content information. In order to keep such representations compact yet expressive enough, the authors impose a set of constraints that ensure balanced and uncorrelated transformations. Once binary codes are learned, the inference can be made by virtue of efficient Hamming distance computations. Moreover, the search for candidate entities can be performed via the generative step that projects binary codes onto actual feature space, where kNN-based techniques can be further utilized.