A guide to Collaborative Topic Modeling recommender systems | by Mario Damiano Russo | Towards Data Science

Recommender Systems are a broad class of machine learning models with the aim of forecasting the unobserved rating that a user u would give to an item i.
In this guide, we will discuss Collaborative Topic Modeling/Regression (CTM/CTR) as introduced by Wang and Blei (2011) [3], a recommender system for text-based items with enhanced accuracy and out-of-matrix prediction capabilities. We will also provide a Python3 implementation for those who are not concerned with the mathematical details behind the model.
CTM builds upon two well-known models, namely Probabilistic Matrix Factorization (PMF) [1] and Latent Dirichlet Allocation (LDA) [2], therefore preliminary knowledge of said models is required. For those unfamiliar with them, there are plenty of very good guides available online, among which:
Latent Dirichlet Allocation, by Thushan Gangedara
Probabilistic Matrix Factorization, by Benjamin Draves
Probabilistic Matrix Factorization (PMF) is a very simple yet powerful approach, that allows us to infer item and user latent features when our only available data is a (U X I) sparse ratings matrix.
Unfortunately, the simplicity of such approach is also its demise, since PMF is unable to generalize the recommendations for new, completely unrated items: since no ratings are observed for item j, the model cannot derive its latent vector of qualities. In technical jargon, we say that PMF is unable to do out-of-matrix prediction.This issue is particularly troublesome in contexts where unrated items do not get recommended since they have no ratings, and they keep having no ratings because they do not get recommended. An example of such scenario is the sector of scientific publications, where new papers from lesser-known authors, despite being qualitatively good, are unable to reach potentially interested readers.
The core idea behind CTM is that we can represent the latent qualities vector of document i, Qi, as:
Where θi is the (K X 1) vector of topic proportions for item i obtained from traditional LDA estimates, and εi is a (K X 1) offset vector that adjusts topic proportions. The rationale for the offset vector is that it adjusts the topic proportions by incorporating informations from the observed ratings. For example: suppose you have two scientific papers, and both of them are 50% about “Machine Learning” and 50% about “Biology”. Now, it might be that according to the observed ratings the first paper appeals more to Machine Learning researchers, whereas the second one appeals more to Biology researchers. Using the offset vector εi we are able to calibrate the topic proportions to reflect how much the different topics appeal to different individuals.
Just like PMF and LDA, on which it is built upon, CTM also has its own generative process:
Where σ2_P and σ2_Q represent the variance we impose a priori on the distribution of the elements of the vectors in P and Q. Similarly, σ2 represents the variance we impose a priori on the distribution of the ratings. For those unfamiliar with bayesian statistics, these values can be simplistically viewed as regularizing hyperparameters for our model.
By representing the generative process using plate notation, we can clearly see that CTM is nothing more than a stacked version of an LDA model on top of a PMF one, where the vector of topic proportions θi is used as the mean of the generative distribution of Qi.
In a Recommender System setting, we are interested in obtaining estimates of all the latent variables that contribute to determining the rating rui. Therefore, we will be interested in θi, Qi, and Pu. Since the full posterior P(θ, Q, P) is analytically intractable, we resort to estimation via Maximum Likelihood, where the likelihood of our data is defined as:
Like in most MLE scenarios, it is convenient to work with the log-likelihood:
It is convienient to derive each one of the four components individually.
First part:
Second part:
Thirth part:
Imposing α=1 for convenience, the above distribution becomes:
Fourth part:
Our log-likelihood will be:
The strategy initially pursued in the paper was to use Coordinate Ascent, iteratively alternating between the optimization of [P, Q] and θ. But as the same authors pointed out, using estimates for θ obtained via standard LDA estimation and then optimizing only for [P, Q] gives comparable results and saves a lot of time during the training. We will therefore assume we already got our θ estimates from vanilla LDA, and we will optimize [P, Q] via gradient ascent. It is easy to derive the gradient of our log-likelihood with respect to [P, Q] as:
The formula above is the main result of this section, and we will implement it in the following one in Python3.
In this section, we will create a recommender system for Steam games. We will use two datasets:
Steam 200k, containing information about per-game playtime for over 200k user-game interactions;
Steam games complete dataset, containing various information for over 40k different Steam games, including title and description
NOTE: The codes reported here are not exhaustive and should be intended as a reference. You can access the full, functional Notebook with all the codes HERE.
First of all, we start by opening the Steam games complete dataset, keeping only the name and descriptions of each game. We also drop the games for which we have no textual description.
games = pd.read_csv("steam_games.csv")# keep only the columns of interestgames = games.loc[:, ["name", "game_description"]]# Drop NaN game descriptions and namesgames = games[~games.game_description.isna()]games = games[~games.name.isna()]# Drop the introductory " About This Game " textgames.game_description = games.game_description.apply(lambda x: x.replace(" About This Game ", ""))# drop single-space descriptionsgames = games[games.game_description != " "]games.head(5)
Then, we open the Steam 200k dataset. The rating variable we will try to forecast is the total playtime in hours, which will be scaled to be between 0 and 1.
ratings = pd.read_csv("steam-200k.csv", header = None)# drop last empty columnratings.drop(4, axis = 1, inplace=True)# rename columnsratings.columns = ["UserID", "Title", "Action", "Value"]# keep only "play" variablesratings = ratings[ratings.Action != "purchase"]# and drop the "Action" column, as now it is all "play"sratings.drop("Action", axis = 1, inplace = True)# to ease computations, constrain PlayTime to be between 0 and 1pt = ratings.Valuept_scaled = (pt - pt.min()) / (pt.max() - pt.min())ratings.Value = pt_scaledratings.head(3)
At this point, we need to find the games for which we have both a sparse vector of observed ratings and the description. We lowercase all titles and remove special characters [!,.-”?: ] in both datasets to increase compatibility between titles. We also drop all games with non-unique titles to avoid mismatching. We end up with 1982 titles for which we have both rating and description.
We now turn our head to creating the (U X I) ratings matrix by combining the two datasets we preprocessed.
R = pd.pivot_table(data=ratings, values = ["Value"], index=["UserID"], columns=["Title"])# remove the level on top of game names called "Value"R.columns = R.columns.droplevel()# remove leftover columns name from pivot operationR.columns.name = ""# lastly, fill in the NaNs with 0'sR.fillna(0, inplace=True)R.head(3)
From this matrix, we hold out the vector of ratings for “need for speed undercover”, the 1105th game in our dataset. We will use this to test the validity of our out-of-matrix predictive power.
Our final ratings matrix will be of shape 10058 users by 1981 games, with only 0.23% of the ratings being observed.
Before training our CTM model, we need to extract the topics and their proportions in each game description by training an LDA model. The first thing we do is to lemmatize game descriptions to reduce variance in the vocabulary and improve LDA estimates.
nlp = spacy.load("en")# lemmatize game descriptionsgames["lemmas"] = [[[token.lemma_ if token.lemma_ != "-PRON-" else token.text.lower() for token in sentence if token.pos_ in {"NOUN", "VERB", "ADJ", "ADV", "X"}] for sentence in nlp(speech).sents] for speech in games.game_description]
We then train our LDA model to find K=15 topics across all of our game descriptions, and determine in which percentage each topic appears in each description.
## Train LDA model ##ldacorpus = [dictionary.doc2bow(text) for text in instances]tfidfmodel = TfidfModel(ldacorpus)model_corpus = tfidfmodel[ldacorpus]num_topics = 15num_passes = 30chunk_size = len(model_corpus) * num_passes/200model = LdaMulticore(num_topics=num_topics,                    corpus=model_corpus,                    id2word=dictionary,                    workers=multiprocessing.cpu_count()-1,                    chunksize=chunk_size,                    passes=num_passes,                    alpha=0.1)## #### obtain the matrix of topic proportions per document ##all_topics = model.get_document_topics(model_corpus, per_word_topics=True, minimum_probability=0.0)corpus_topics = []for doc_topics, word_topics, phi_values in all_topics:    corpus_topics.append([topic[1] for topic in doc_topics])    corpus_topics = np.array(corpus_topics)theta = corpus_topics.copy().T## #### remove the heldout game from the theta matrix ##thet = pd.DataFrame(theta)heldout_topics = thet.iloc[:, heldout_idx]thet.drop(heldout_idx, axis = 1, inplace=True)theta = thet.values## ##
Where theta is the (K X I) matrix that tells us the proportions of each one of the K=15 topics appears in each one of the I games.
Once we have our ratings matrix and our matrix of per-item topic proportions, we can build our CTM model.
We start by splitting our ratings matrix in X_train and X_val.
# train - test splitdef train_test_split(ratings, percs = [0.8, 0.2]):        validation = np.zeros(ratings.shape)    train = ratings.copy()        for user in np.arange(ratings.shape[0]):        val_ratings = np.random.choice(ratings[user,:].nonzero()[0],                                        size = round(len(ratings[user,:].nonzero()[0]) * percs[1]),                                        replace=False                                        )        train[user, val_ratings] = 0        validation[user, val_ratings] = ratings[user, val_ratings]        return train, validationX_train, X_val = train_test_split(R.values)
We also define a MSE function for our predicted ratings.
from sklearn.metrics import mean_squared_errordef mse(prediction, ground_truth):    prediction = prediction[ground_truth.nonzero()].flatten()    ground_truth = ground_truth[ground_truth.nonzero()].flatten()    return mean_squared_error(prediction, ground_truth)
Lastly, we build our CTM model:
from tqdm import trangeimport sysclass CTR():    """    Collaborative Topic Regression Model as developed by Wang and Blei (2012).    Leverages topic proportions obtained from LDA model to improve predictions    and allow for out-of-matrix predictions.        Parameters:        - sigma2: expected variance of ratings                   (variance of the ratings Normal prior)        - sigma2_P: expected variance of the elements of the                    preference vector        - sigma2_Q: expected variance of the elements of the                    quality vector    """    def __init__(self, epochs=200, learning_rate=0.001, sigma2=10, sigma2_P=10, sigma2_Q=10):        self.epochs = epochs        self.learning_rate = learning_rate        self.sigma2 = sigma2        self.sigma2_P = sigma2_P        self.sigma2_Q = sigma2_Q            def fit(self, theta, X_train, X_val):        """        Fit a CTR model.                Parameters:            - theta: (K X I) matrix of topic proportions obtained via LDA.            - X_train: (U X I) ratings matrix to train the model on.            - X_test: (U X I) ratings matrix to validate the model on.        """                K = theta.shape[0]        U, I = X_train.shape                #initialize P and Q matrices.        # P is initialized randomly        self.P = np.random.randint(0, 10) * np.random.rand(K, U)        # Q is initialized to be equal to theta        self.Q = theta.copy()                self.train_error = []        self.val_error = []                # obtain the pairs of (u, i) indices for which we observe a rating        users, items = X_train.nonzero()                        # begin training        for iteration in trange(self.epochs, file=sys.stdout, desc='CTR'):            for u, i in zip(users, items):                error = X_train[u, i] - np.dot(self.P[:, u].T, self.Q[:, i])# we are MAXIMIZING the likelihood via gradient ascent                self.P[:, u] += self.learning_rate * (-self.P[:, u]/self.sigma2_P + (self.P[:, u] * error)/self.sigma2)                self.Q[:, i] += self.learning_rate * (-(self.Q[:, i] - theta[:, i])/self.sigma2_Q + (self.Q[:, i] * error)/self.sigma2)self.train_error.append(mse(np.dot(self.P.T, self.Q), X_train))            self.val_error.append(mse(np.dot(self.P.T, self.Q), X_val))                def predict_ratings(self):        """        Returns the matrix of predicted ratings.        """        return np.dot(self.P.T, self.Q)                def predict_out_of_matrix(self, topics):        """        Returns the (U X 1) vector of predicted ratings         for an unrated item, using the item's topic proportions.                Parameters:            - topics: (K X 1) array of topic proportions                      for the unrated item.        """        return np.dot(self.P.T, topics)
And train the model:
ctr = ctr = CTR(sigma2_P=5, sigma2_Q=5, sigma2=1)ctr.fit(theta, X_train, X_val)
At the end of the training, our ctr object will have learned the latent matrices P and Q and will be able to predict the missing values in the ratings matrix via their dot product.
Below is the MSE performance throughout the training for 100 epochs, with a 0.126 MSE on the validation set recorded during the last training epoch.
The performance on the single held-out observation had a MSE of 0.35. Of course, this is not as high as the one recorded for in-matrix prediction, but it is quite impressive if we consider we are recommending an out-of-matrix item.
We are satisfied with our results, but how well does Collaborative Topic Regression compare to traditional Probabilistic Matrix Factorization in this scenario? Quite well apparently. The vanilla factorization approach reaches a 0.32 MSE versus the 0.126 achieved by CTM.
The latent topic structure that characterizes CTM adds a further layer of interpretability to make inference on. First of all, the added benefit of CTM over traditional PMF is that we are able to label the learned latent dimensions, seeing how prominent each topic truly is in each item. Note: these are not the topic proportions learned from LDA, but the topic relevances that are learned by the CTM model!
We can also study the distribution of user preferences across the K latent dimensions via a simple horizontal sum of the P matrix:
The same can be done for items instead of users, to see how are the genres distributed across our items:
In this article, we presented Collaborative Topic Modeling as explained in Wang and Blei (2011), an improved matrix factorization recommender system for text-based items. The model performs generally better than vanilla matrix factorization methods, and presents additional benefits such as out-of-matrix prediciton and latent dimensions interpretability.
In general, the highest challenge of calibrating a CTM model is finding the optimal combination of hyperparameters that gives the best performance on our data. In fact, we need to tune:
alpha, from the LDA model.
number of topics K
learning rate
sigma2, sigma2_P, sigma2_Q, hyperparameters on the prior distributions of R, P, and Q respectively.
Secondly, there can be a lot of variability in the performance of the model across separate trainings due to the stochastic nature of the model, both in the LDA phase and in the CTM training phase.
Despite not being a guaranteed improvement over PMF, CTM is indeed a method that should be considered when dealing with the task of recommending text-based documents, thanks to its generally higher performance and the out-of-matrix predictions capabilities.
[1] Mnih, Andriy, and Russ R. Salakhutdinov. “Probabilistic matrix factorization.” Advances in neural information processing systems. 2008.
[2] Blei, David M., Andrew Y. Ng, and Michael I. Jordan. “Latent dirichlet allocation.” Journal of machine Learning research 3.Jan (2003): 993–1022.
[3] Wang, Chong, and David M. Blei. “Collaborative topic modeling for recommending scientific articles.” Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. 2011.