Relational Stacked Denoising Autoencoder for Tag Recommendation

Tag recommendation has become one of the most important ways of organizing and indexing online resources like articles, movies, and music. Since tagging information is usually very sparse, effective learning of the content representation for these resources is crucial to accurate tag recommendation. Recently, models proposed for tag recommendation, such as collaborative topic regression and its variants, have demonstrated promising accuracy. However, a limitation of these models is that, by using topic models like latent Dirichlet allocation as the key component, the learned representation may not be compact and effective enough. Moreover, since relational data exist as an auxiliary data source in many applications, it is desirable to incorporate such data into tag recommendation models. In this paper, we start with a deep learning model called stacked denoising autoencoder (SDAE) in an attempt to learn more effective content representation. We propose a probabilistic formulation for SDAE and then extend it to a relational SDAE (RSDAE) model. RSDAE jointly performs deep representation learning and relational learning in a principled way under a probabilistic framework. Experiments conducted on three real datasets show that both learning more effective representation and learning from relational data are beneficial steps to take to advance the state of the art.  Due to the abundance of online resources like articles, movies, and music, tagging systems (Yu et al. 2014) have become increasingly important for organizing and indexing them. For example, CiteULike1 uses tags to help categorize millions of articles online and Flickr2 allows users to use tags to organize their photos. However, it is often not easy to compose a set of words appropriate for the resources. Besides, the large variety in phrasing styles of the users can potentially make the tagging information inconsistent and idiosyncratic. With such technical challenges, research in tag recommendation (TR) (Gupta et al. 2010; Wang et al. 2012) has gained in popularity over the past few years. An accurate tag recommendation system not only can save the pain of users searching for candidate tags on Copyright c  Tag recommendation methods can roughly be categorized into three classes (Wang et al. 2012): content-based methods, co-occurrence based methods, and hybrid methods. Content-based methods (Chen et al. 2008; 2010; Shen and Fan 2010) utilize only the content information (e.g., abstracts of articles, image pixels, and music content) for tag recommendation. Co-occurrence based methods (Garg and Weber 2008; Weinberger, Slaney, and van Zwol 2008; Rendle and Schmidt-Thieme 2010) are similar to collaborative filtering (CF) methods (Li and Yeung 2011). The co-occurrence of tags among items, usually represented as an tag-item matrix, is used for tagging. The third class of methods (Wu et al. 2009; Wang and Blei 2011; Yang, Zhang, and Wang 2013; Zhao et al. 2013; Bao, Fang, and Zhang 2014; Chen et al. 2014), also the most popular and effective ones, consists of hybrid methods. They make use of both tagging (co-occurrence) information (the tag-item matrix) and item content information for recommendation. In hybrid methods, learning of item representations (also called item latent factors in some models) is crucial for the recommendation accuracy especially when the tag-item matrix is extremely sparse. Recently, models like collaborative topic regression (CTR) (Wang and Blei 2011) and its variants (Purushotham, Liu, and Kuo 2012; Wang, Chen, and Li 2013) have been proposed and adapted for tag recommendation to achieve promising performance. These models use latent Dirichlet allocation (LDA) (Blei, Ng, and Jordan 2003) as the key component for learning item representations and use probabilistic matrix factorization (PMF) (Salakhutdinov and Mnih 2007) to process the co-occurrence matrix (tag-item matrix). However, when using LDA, the resulting item representations tend to be quite sparse. Consequently, more dimensions may be needed for the representations to be effective. Unfortunately PMF with the low-rank assumption usually works with quite a small number of latent dimensions, which is not in line with the nature of LDA (or CTR). On the other hand, deep learning models like stacked denoising autoencoder (SDAE) (Vincent et al. 2010) and convolutional neural networks (Krizhevsky, Sutskever, and Hinton 2012) recently show great potential for learning effective and compact representations and deliver state-of-the-art performance in computer vision (Wang and Yeung 2013) and natural language processing (Salakhutdinov and Hinton 2009; Kalchbrenner, Grefenstette, and Blunsom 2014) applications. Intuitively, the effectiveness and compactness of deep learning models like SDAE seem to fit PMF perfectly and can potentially lead to significant boost of recommendation performance. Besides, since relational data exist as an auxiliary data source in many applications (e.g., natural language processing, computational biology), it is desirable to incorporate such data into tag recommendation models. For example, when recommending tags for articles in CiteULike, the citation relations between articles (Vu et al. 2011; Wang and Li 2013) may provide very useful information. However, incorporating relational information into deep neural network (DNN) models like SDAE is non-trivial since with the relational data, the samples are no longer i.i.d., which is the assumption underlying DNN models. In this paper, we propose novel methods to address the above challenges. The main contributions of this paper are summarized as follows: • We adapt SDAE and use it in conjunction with PMF (or a simplified version of CTR) to significantly boost the recommendation performance. • To satisfy the need for relational deep learning, we develop a probabilistic formulation for SDAE and, by extending this probabilistic SDAE, we propose a probabilistic relational model called relational SDAE (RSDAE) to integrate deep representation learning and relational learning in a principled way. Besides, RSDAE can be naturally extended to handle multi-relational data (with more details provided in the supplementary material). • Extensive experiments on datasets from different domains show that our models outperform the state of the art. Problem Statement and Notation Assume we have a set of items (articles or movies) Xc to be tagged, with Xc,j∗ ∈ R denoting the content (attributes) of item j. In the case of tagging articles (papers) in CiteULike, the items are papers, and the content information can be the bag-of-words representation of paper abstracts. Assume we have a set of I tags {t1, t2, · · · , tI} as candidates to be recommended to tag each item. Then a tag-item matrix R can be used to represent the tagging information for all the items. Each matrix entry Rij is a binary variable, where Rij = 1 means that tag ti is associated with item j and Rij = 0 otherwise. Tag recommendation is to predict the missing values in R∗j = [R1j ,R2j , · · · ,RIj ] (i.e., recommend tags to items). Besides, we use IK to denote a K-dimensional identity matrix and S = [s1, s2, · · · , sJ ] to denote the relational latent matrix with sj representing the relational properties of item j. Note that although we focus on tag recommendation for articles and movies in this paper, our proposed models are flexible enough to be used for other applications such as image and video tagging. From the perspective of SDAE, the J-by-B matrix Xc represents the clean input to the SDAE and the noise-corrupted matrix of the same size is denoted by X0. Besides, we denote the output of layer l of the SDAE, a J-by-Kl matrix, by Xl. Row j of Xl is denoted by Xl,j∗, Wl and bl are the weight matrix and bias vector of layer l, Wl,∗n denotes column n of Wl, and L is the number of layers. As a shorthand, we refer to the collection of all layers of weight matrices and biases as W. Note that in our models an L/2-layer SDAE corresponds to an L-layer network. Probabilistic Stacked Denoising Autoencoders In this section we will first have a brief review on SDAE and then give a probabilistic formulation of generalized SDAE, which will be a building block of our relational stacked denoising autoencoder (RSDAE) model. Stacked Denoising Autoencoders SDAE (Vincent et al. 2010) is essentially a feedforward neural network for learning representations of the input data by learning to predict the clean input itself in the output. Normally the hidden layer in the middle is constrained to be a narrow bottleneck and the input layer X0 is a noise-corrupted version of the clean input data. Learning of an SDAE involves solving the following regularized optimization problem: min {Wl},{bl} ‖Xc −XL‖F + λ ∑