Venue: NAACL 2018

# Deep Dirichlet Multinomial Regression

Adrian Benton Mark Dredze

Center for Language and Speech Processing

Johns Hopkins University

Baltimore, MD 21218, USA

{adrian,mdredze}@cs.jhu.edu

###### Abstract

Dirichlet Multinomial Regression (_DMR_) and other supervised topic models can incorporate arbitrary document-level features to inform topic priors. However, their ability to model corpora are limited by the representation and selection of these features - a choice the topic modeleer must make. Instead, we seek models that can learn the feature representations upon which to condition topic selection. We present _deep Dirichlet Multinomial Regression_ (_d DMR_), a generative topic model that simultaneously learns document feature representations and topics. We evaluate _d DMR_ on three datasets: New York Times articles with fine-grained tags, Amazon product reviews with product images, and Reddit posts with subreddit identity. _d DMR_ learns representations that outperform _DMR_ and _LDA_ according to heldout perplexity and are more effective at downstream predictive tasks as the number of topics grows. Additionally, human subjects judge _d DMR_ topics as being more representative of associated document features. Finally, we find that supervision leads to faster convergence as compared to an _LDA_ baseline and that _d DMR_'s model fit is less sensitive to training parameters than _DMR_.

## 1 Introduction

Fifteen years of research on topic models, starting from Latent Dirichlet Allocation (_LDA_) Blei et al. (2003), have led to a variety of models for numerous data settings. These models identify sets (distributions) of related words that reflect semantic topics in a large corpus of text data. Topic models are now routinely used in the social sciences and humanities to analyze text collections Schmidt (2012).

Document collections are often accompanied by metadata and annotations, such as a book's author, an article's topic descriptor tags, images associated with a product review, or structured patient information associated with clinical records. These document-level annotations can provide additional supervision for guiding topic model learning. Additional information can be integrated into topic models using either _downstream_ or _upstream_ models. Downstream models, such as supervised _LDA_Mcauliffe and Blei (2008), assume that these additional document features are generated from each document's topic distribution. These models are most helpful when you desire topics that are predictive of the output, such as models for predicting the sentiment of product reviews. Upstream models, such as Dirichlet Multinomial Regression (_DMR_), condition each document's topic distribution on document features, such as author RosenZvi et al. (2004), social network McCallum et al. (2007), or document labels Ramage et al. (2009). Previous work has demonstrated that upstream models tend to outperform downstream models in terms of model fit, as well as extracting topics that are useful in prediction of related tasks Benton et al. (2016).

_DMR_ is an upstream topic model with a particularly attractive method for incorporating arbitrary document features. Rather than defining specific random variables in the graphical model for each new document feature, _DMR_ treats the document annotations as features in a log-linear model. The log-linear model parameterizes the Dirichlet prior for the document's topic distribution, making the Dirichlet's hyperparameter (typically \(\alpha\)) document-specific. By making no assumptions on model structure of new random variables, _DMR_ is flexible to incorporating different types of features.

Despite this flexibility, _DMR_ models are typically restricted to a small number of document features. Several reasons account for this restriction: 1) Many text corpora only have a small number of document-level features; 2) Model hyperparameters become less interpretable as the dimensionality grows; and 3) _DMR_ is liable to overfit the hyperparameters when the dimensionality of document features is high. In practice, applications of _DMR_ are limited to settings with a small number of features, or where the analyst selects a few meaningful features

by hand.

A solution to this restriction is to learn low-dimensional representations of document features. Neural networks have shown wide-spread success at learning generalizable representations, often obviating the need for hand designed features (Collobert and Weston, 2008). A prime example is word embedding features in natural language processing, which supplant traditional lexical features (Brown et al., 1992; Mikolov et al., 2013; Pennington et al., 2014). Jointly learning networks that construct feature representations along with the parameters of a standard NLP model has become a common approach. For example, (Yu et al., 2015) used a tensor decomposition to jointly learn features from both word embeddings and traditional NLP features, along with the parameters of a relation extraction model. Additionally, neural networks can handle a variety of data types, including text, images and general metadata features. This makes them appropriate for addressing dimensionality reduction in _DMR_.

We propose **deep** Dirichlet Multinomial Regression (_dDMR_), a model that extends _DMR_ by introducing a deep neural network that learns a transformation of the input metadata into features used to form the Dirichlet hyperparameter. Whereas _DMR_ parameterizes the document-topic priors as a log-linear function of document features, _dDMR_ jointly learns a feature representation for each document along with a log-linear function that best captures the distribution over topics. Since the function mapping document features to topic prior is a neural network, we can jointly optimize the topic model and the neural network parameters by gradient ascent and back-propagation. We show that _dDMR_ can use network architectures to better fit text corpora with high-dimensional document features as compared to other supervised topic models. The topics learned by _dDMR_ are judged as being more representative of document features by human subjects. We also find that _dDMR_ tends to converge in many fewer iterations than _LDA_, and also does not suffer from tuning difficulties that _DMR_ encounters when applied to high-dimensional document features.

## 2 Model

Our model builds on the generative model of _DMR_: an LDA-style topic model that replaces the hyperparameter (vector) of the topic distribution Dirichlet prior with a hyperparameter that is output from a log-linear model given the document features. Our model deep DMR (_dDMR_) replaces this log-linear model with an arbitrary function \(f\) that maps a real-valued vector of dimension \(F\) to a representation of dimension \(K\). For simplicity we make no assumptions on the choice of this function, only that it can be optimized to minimize a cost on its output by gradient ascent. In practice, we define this function as a neural network, where the architecture of this network is informed by the type of document features, e.g. a convolutional neural network for images. We use neural networks since they are expressive, generalize well to unseen data, and can be jointly trained using straightforward gradient ascent with back-propagation.

The generative story for _dDMR_ is as follows:

1. Representation function \(f\in\mathbb{R}^{F}\rightarrow\mathbb{R}^{K}\)
2. Topic-word prior parameters: \(\omega^{bias}\in\mathbb{R}^{V}\)
3. For each document \(m\) with features \(\alpha_{m}\in\mathbb{R}^{F}\), generate document prior: 1. [label=()]
2. \(\widetilde{\theta}_{m}=exp(f(\alpha_{m}))\)
3. \(\theta_{m}\sim Dirichlet(\widetilde{\theta}_{m})\)
4. For each topic \(k\), generate word distribution: 1. \(\widetilde{\phi}_{k}=exp(\omega^{bias})\)
5. \(\phi_{k}\sim Dirichlet(\widetilde{\phi}_{k})\)
6. For each token \((m,n)\), generate data: 1. Topic (unobserved): \(z_{m,n}\sim\theta_{m}\) 2. Word (observed): \(w_{m,n}\sim\phi_{z_{m,n}}\)

where \(V\) is the vocabulary size and \(K\) are the number of topics. In practice, the document features need not be restricted to fixed-length feature vectors, e.g. \(f\) may be an RNN that maps from a sequence of characters to a fixed length vector in \(\mathbb{R}^{k}\). _DMR_ is a special case of _dDMR_ with the choice of a linear function for \(f\). Figure 1 displays the graphical model diagram for _dDMR_.

### Inference and Parameter Estimation

We infer the random variables of the topic model using collapsed Gibbs sampling, and estimate the model parameters using gradient ascent with backpropagation. We use alternating optimization: one

Figure 1: The graphical model for _dDMR_. \(f\) is shown as a feedforward fully-connected network, and the document features are given by the image (a cat carrier).

 

iteration of collapsed Gibbs sampling (sample topics for each word) and then an update of the parameters of \(f\) by gradient ascent to maximize the log-likelihood of the tokens and topic assignments. Given the parameters, the sampling step remains unchanged from _LDA_Griffiths and Steyvers (2004). The network parameters are estimated via back-propagation through the network for a fixed sample. Eq. 1 shows the gradient of the data log-likelihood, \(\mathscr{L}\), with respect to \(\widetilde{\theta}_{m,k}=exp(f(\alpha_{m})_{k})\), the prior weight of topic \(k\) for document \(m\). \(\psi\) is the digamma function (derivative of the log-gamma function), \(n_{m}\) is the number of tokens in document \(m\), and \(n_{m,k}\) is the count of how many tokens topic \(k\) was assigned to in document \(m\).

\[\begin{split}\frac{\delta\mathscr{L}}{\delta\widetilde{\theta}_{ m,k}}=\psi&(\sum_{k=1}^{K}\widetilde{\theta}_{m,k})-\psi(\sum_{ k=1}^{K}\widetilde{\theta}_{m,k}+n_{m})\\ &+\psi(\widetilde{\theta}_{m,k}+n_{m,k})-\psi(\widetilde{\theta }_{m,k})\end{split} \tag{1}\]

## 3 Data

We explore the flexibility of our model by considering three different datasets that include different types of metadata associated with each document. For each dataset, we describe the documents and metadata.

New York TimesThe New York Times Annotated Corpus Sandhaus (2008) contains articles with extensive metadata used for indexing by the newspaper. For supervision, we used the "descriptor" tags associated with each article assigned by archivists. These tags reflect the topic of an article, as well as organizations or people mentioned in the article. We selected all articles published in 1998, and kept those tags that were associated with at least 3 articles in that year - 2424 unique tags. 20 of the 200 most frequent tags were held out from training for validation purposes: { _"education and schools"_, _"law and legislation"_, _"advertising"_, _"budgets and budgeting"_, _"freedom and human rights"_, _"telephones and telecommunications"_, _"bombs and explosives"_, _"sexual harassment"_, _"reform and reorganization"_, _"tecachers and school employees"_, _"tests and testing"_, _"fatures and options trading"_, _"boxings"_, _"firearms"_, _"company reports"_, _"embargoes and economic sanctions"_, _"hospitals"_, _"states (us)"_, _"bridge (card game)"_, and _"auctions"_. Articles contained a mean of 2.1 tags, with 738 articles not containing any of these tags. Tags were represented using a one-hot encoding.

Articles were tokenized by non-alphanumeric characters and numerals were replaced by a special token. Words occurring in more than 40% of documents were removed, and only the 15,000 most frequent types were retained. There were a total of 89,397 articles with an average length of 158 tokens per article.

Amazon ReviewsThe Amazon product reviews corpusMcAuley and Yang (2016) contains reviews of products as well as images of the product. We sampled 100,000 Amazon product reviews: 20,000 reviews sampled uniformly from the _Musical Instruments_, _Patio, Lawn, & Garden_, _Grocery & Gourmet Food_, _Automotive_, and _Pet Supplies_ product categories. We hypothesize that knowing information about the product's appearance will indicate which words appear in the review, especially for product images occurring in these categories. 66 of the reviews we sampled contained only highly infrequent tokens, and were therefore removed from our data, leaving 99,934 product reviews. Articles were pre-processed identically to the New York Times data.

We include images as supervision by using the 4096-dimensional second fully-connected layer of the Caffe convolutional neural network reference model, trained to predict ImageNet object categories1. Using these features as supervision to _dDMR_ is similar to fine-tuning a pre-trained CNN to predict a new set of labels. Since the Caffe reference model is already trained on a large corpus of images, we chose to fine-tune only the final layers so as to learn a transformation of the already learned representation.

Footnote 1: Features used directly from [http://jmcauley.ucsd.edu/data/amazon/](http://jmcauley.ucsd.edu/data/amazon/)

RedditWe selected a sample of Reddit posts made in January 2016. A standard stop list was used to remove frequent function words and we restricted the vocabulary to the 30,000 most frequent types. We restricted posts made to subreddits, collections of topically-related threads, with at least ten comments in this month (26,830 subreddits), and made by users with at least five comments across these subreddits (total of 1,351,283 million users). We then sampled 10,000 users uniformly at random and used all their comments as a corpus, for a total of 389,234 comments over 7,866 subreddits (token length mean: 16.3, median: 9)2.

Footnote 2: The sampled comment IDs can be found here: [https://github.com/abenton/deep-dmr/blob/master/resources/reddit_comment_ids.txt](https://github.com/abenton/deep-dmr/blob/master/resources/reddit_comment_ids.txt)

This corpus differs from the others in two ways. First, Reddit documents are very short, which is problematic for topic models that rely on detecting correlations in token use. Second, the Reddit metadata that may be useful for topic modeling is necessarily high-dimensional (e.g. subreddit identity, a proxy for topical content). _DMR_ may have trouble exploiting high-dimensional supervision.

## 4 Experiments

Model EstimationWe used the same procedure for training topic models on each dataset. Hyperparameter gradient updates were performed after 

a burnin period of 100 Gibbs sampling iterations. Hyperparameters were updated with the adaptive learning rate algorithm Adadelta Zeiler (2012), with a tuned base learning rate and fixed \(\rho=0.95\lx@note{footnote}{We found this adaptive learning rate algorithm improved model fit in many fewer iterations than gradient descent with tuned step size and decay rate for all models.}\). All models were trained for a maximum of 15,000 epochs, with early stopping if heldout perplexity showed no improvements after 200 epochs (evaluated once every 20 epochs). Hyperparameters were fit on every other token in the corpus, and (held-out) log-likelihood/perplexity was calculated on the remaining tokens.

For the architecture of the _dDMR_ model we used single-hidden-layer multi-layer perceptrons (MLPs), with rectified linear unit (ReLU) activations on the hidden layer, and linear activation on the output layer. We sampled three architectures for each dataset, by drawing layer widths independently at random from \([10,500]\), and also included two architectures with \((50,10)\) and \((100,50)\), _(hidden, output)_ layers 4 . We compare the performance of _dDMR_ to _DMR_ trained on the same feature set as well as _LDA_.

Footnote 4: We included these two very narrow architectures to ensure that some architecture learned a small feature representation, generalizing better when features are very noisy or only provide a weak signal for topic modeling. We restricted ourselves to only train _dDMR_ models with single-hidden-layer MLPs in the priors for simplicity and to avoid model fishing.

For the New York Times dataset, we also compare _dDMR_ to _DMR_ trained on features after applying principal components analysis (PCA) to reduce the dimensionality of descriptor feature supervision, sweeping over PCA projection width in \(\left\{10,50,100,250,500,1000\right\}\). Comparing performance of _dDMR_ to PCA-reduced _DMR_ tests two modeling choices. First, it tests the hypothesis that explicitly learning a representation for document annotations to maximize data likelihood produces a "better-fit" topic model than learning this annotation representation in unsupervised fashion - a two-step process. It also lets us determine if a linear dimensionality reduction technique is sufficient to learning a good feature representation for topic modeling, as opposed to learning a non-linear transformation of the document supervision. Note that we cannot apply PCA to reduce the dimensionality for subreddit id in Reddit since it is a one-hot feature.

Documents in each dataset were partitioned into ten equally-sized folds. Model training parameters of L1 and L2 regularization penalties on feature weights for _DMR_ and _dDMR_ and the base learning rate for each model class were tuned to minimize heldout perplexity on the first fold. These were tuned _independently for each model_, with number of topics fixed to 10, and _dDMR_ architecture fixed to narrow layer widths (50, 10). Model selection was based on the macro-averaged performance on the next eight folds, and we report performance on the remaining fold. We selected models separately for each evaluation metric. For _dDMR_, model selection amounts to selecting the document prior architecture, and for _DMR_ with PCA-reduced feature supervision, model selection involved selecting the PCA projection width.

EvaluationEach model was evaluated according to heldout perplexity, topic coherence by normalized pointwise mutual information (NPMI) Lau et al. (2014), and a dataset-specific predictive task.

Heldout perplexity was computed by only aggregating document-topic and topic-word counts from every other token in the corpus, and evaluating perplexity on the remaining heldout tokens. This corresponds to the "document completion" evaluation method as described in Wallach et al. (2009), where instead of holding out the words in the second half of a document, every other word is held out.

NPMI Lau et al. (2014) computes a an automatic measure of topic quality, the sum of pointwise mutual information between pairs of \(m\) most likely words normalized by the negative log of each pair jointly occurring within a document (Eq. 2). We calculated this topic quality metric on the top 20 most probable words in each topic, and averaged over the most coherent 1, 5, 10, and over all learned topics. However, models were selected to only maximize average NPMI over all topics.

\[\texttt{NPMI}=\sum_{i=1}^{m}\sum_{j=i+1}^{m}\frac{\log\frac{P(w_{i},w_{j}))}{ P(w_{i})P(w_{j})}}{-\log P(w_{i},w_{j})}\quad(2)\]

For prediction tasks, we used the sampled topic distribution associated with a document, averaged over the last 100 iterations, as features to predict a document-level label. For New York Times articles we predicted 10 of the 200 most frequent descriptor tags restricting to articles with exactly one of these descriptors. For Amazon, we predicted the product category a document belonged to (one of five), and for Reddit we predicted a heldout set of document subreddit IDs. In the case of Reddit, these heldout subreddits were 10 out of the 100 most prevalent in our data, and were held out similar to the New York Times evaluation. SVM models were fit on inferred topic distribution features and were then evaluated according to accuracy, F1-score, and area under the ROC curve. The SVM slack parameter was tuned by 4-fold cross-validation on 60% of the documents, and evaluated on the remaining 40%.

We also collected human topic judgments using Amazon Mechanical Turk Callison-Burch and 

Dredze, 2010). Each subject was presented with a human-readable version of the features used for supervision. For New York Times articles we showed the descriptor tags, for Amazon the product image, and for Reddit the name, title, and public description of the subreddit. We showed the top twenty words for the most probable topic sampled for the document with those features, as learned by two different models. One topic was learned by _dDMR_ and the other was either learned by _LDA_ or _DMR_. The topics presented were from the 200-topic model architecture that maximized NPMI on development folds. Annotators were asked "_to choose which word list best describes a document_..." with the displayed features. The topic learned by _dDMR_ was shuffled to lie on either the right or left for each Human Intelligence Task (HIT). We obtained judgments on 1,000 documents for each dataset and each model evaluation pair - 6,000 documents in all. This task can be difficult for many of the features, which may be unclear (e.g. descriptor tags without context) or difficult to interpret (e.g. images of automotive parts). We excluded the document text since we did not want subjects to evaluate topic quality based on token overlap with the actual document.

## 5 Results

Model Fitting_ddmr_ achieves lower perplexity than _LDA_ or _DMR_ for most combinations of number of topics and dataset (Table 1). It is striking that _DMR_ achieves higher perplexity than _LDA_ in many of these conditions. This is particularly true for the Amazon dataset, where _DMR_ consistently lags behind _LDA. Supervision alone does not improve topic model fit if it is too high-dimensional for learning_. Perplexity is higher on the Reddit data for all models due to both a larger vocabulary size and shorter documents.

It is also worth noting that finding a low-dimensional linear projection of the supervision features with PCA does not improve model fit as well as _dDMR_. _dDMR_ benefits both from joint learning to maximize corpus log-likelihood and possibly by the flexibility of learning non-linear projection (through the hidden layer ReLU activations).

Another striking result is the difference in speed of convergence between the supervised models and _LDA_ (Figure 2). Even supervision that provides a weak signal for topic modeling, such as Amazon product image features, can speed convergence over _LDA_. In certain cases (Figure 2 left), training _dDMR_ for 1,000 iterations results in a lower perplexity model than _LDA_ trained for over 10,000 iterations.

In terms of actual run time, parallelization of model training differs between the supervised model and _LDA_. Gradient updates necessary for learning the representation can be trivially distributed across multiple cores using optimized linear algebra libraries (e.g. BLAS), mitigating the additional cost incurred by hyperparameter updates in supervised models. In contrast, the Gibbs sampling iterations can also be parallelized, but not as easily, ultimately making resampling topics the most expensive step in model training. Because of this, the potential difference in runtime for a single iteration between _dDMR_ and _LDA_ is small, with the former converging in far fewer iterations. In our experiments, per iteration time taken by _DMR_ or _dDMR_ was at most twice as long as _LDA_ across all experiments.

_dDMR_ performance is also insensitive to training parameters relative to _DMR_. While _DMR_ requires heavy L1 and L2 regularization and a very small step size to achieve low heldout perplexity, _dDMR_ is relatively insensitive to the penalty on regularization and benefits from a higher base learning rate (Figure 3). We found that dDMR _is easier to tune than_ DMR, requiring less exploration of the training parameters. This is also corroborated by higher variance in perplexity achieved by _DMR_ across different cross-validation folds (Table 1).

Topic QualityResults for the automatic topic quality evaluation, NPMI, are mixed across datasets. In many cases, _LDA_ and _DMR_ score highly according to NPMI, despite achieving higher heldout perplexity than _dDMR_ (Table 2). This may not be surprising as previous work has found that perplexity does not correlate well with human judgments of topic coherence (Lau et al., 2014).

However, in the human evaluation, subjects find that _dDMR_-learned topics are more representative of document annotations than _DMR_ (Table 3). While subjects only statistically significantly favored _dDMR_ models over _LDA_ on the Reddit data, they favored _dDMR_ topics over _LDA_ across all datasets, and significantly preferred _dDMR_ top

\begin{table}
\begin{tabular}{c c|l||c|c|c c} \hline \multicolumn{7}{l}{Z} & Model & NYP & Amazon & Reddit \\ \multirow{3}{*}{10} & _LDA_ & **3492** & (8) & 2496 & (7) & 3811 & (15) \\  & _DMR_ & **3385** & (6) & 2475 & (9) & 3753 & (10) \\  & _dDMRR_ & **3417** & (8) & 2476 & (8) & **3624** & (13) \\ \cline{2-7}  & _LDA_ & 3081 & (6) & 2275 & (7) & 3659 & (19) \\  & _DMRR_ & 3081 & (4) & 2556 & (48) & 3650 & (8) \\  & _dDMRR_ & **3082** & (8) & 2222 & (7) & **3581** & (16) \\ \cline{2-7}  & _LDA_ & 2760 & (8) & 2469 & (9) & 3605 & (17) \\ \cline{2-7}  & _DMR_ & 2797 & (34) & 2407 & (20) & 3640 & (40) \\  & _dDMRR_ & **2657** & (8) & **2197** & (13) & **3597** & (17) \\ \hline \multirow{3}{*}{100} & _LDA_ & 2618 & (8) & 2246 & (10) & 3676 & (19) \\  & _DARR_ & 2917 & (27) & 2410 & (75) & 3532 & (30) \\  & _DARR_ & 2644 & (32) & **2215** & (6) & **3642** & (18) \\ \cline{2-7}  & _LDA_ & 2513 & (8) & 2217 & (7) & 3653 & (19) \\ \cline{2-7}  & _DARR_ & 2630 & (13) & 2480 & (65) & 3909 & (15) \\ \cline{2-7}  & _dDMRR_ & **2525** & (14) & **2214** & (12) & **3587** & (11) \\ \hline \end{tabular}
\end{table}
Table 1: Test fold heldout perplexity for each dataset and model for number of topics \(Z\). Standard error of mean heldout perplexity over all cross-validation folds in parentheses.

 



\begin{table}
\begin{tabular}{c l||r r|r r|r r|r r|r r r}  & & \multicolumn{3}{c}{New York Times} & \multicolumn{4}{c}{Amazon} & \multicolumn{5}{c}{Reddit} \\ \hline \(Z\) & Model & 1 & 5 & 10 & Overall & \multicolumn{2}{c}{...} & \multicolumn{2}{c}{...} \\ \cline{6-17}  & _LDA_ & 52 & 49 & 43 & 43 & 25 & 23 & 20 & 20 & **125** & **82** & **56** & **56** \\
10 & _DMR_ & 53 & 50 & 42 & 42 & **58** & **43** & **31** & **31** & 43 & 35 & 30 & 30 \\  & _DMR-PCA_ & **63** & **53** & **45** & **45** & **45** & **45** & **45** & **45** & **45** & **45** & **45** \\  & _dDMR_ & **57** & **51** & **44** & **44** & 24 & 21 & 19 & 19 & 109 & 62 & 46 & 46 \\ \hline \multirow{3}{*}{20} & _LDA_ & 62 & 59 & 54 & 45 & 27 & 25 & 23 & 20 & **121** & **87** & **59** & **42** \\  & _DMR_ & 63 & 60 & 56 & 45 & 66 & 56 & **53** & **43** & 81 & 49 & 41 & 34 \\  & _DMR-PCA_ & **76** & **61** & **57** & **47** & **97** & **61** & **53** & 40 & 109 & 66 & 49 & 38 \\ \hline \multirow{3}{*}{50} & _LDA_ & 80 & 66 & 62 & 44 & 30 & 27 & 25 & 20 & **135** & **96** & **64** & 34 \\  & _DMR_ & 80 & **67** & **63** & **46** & **136** & **81** & **73** & **58** & 51 & 46 & 41 & 33 \\  & _DMR-PCA_ & **82** & **67** & 63 & 45 & 71 & 65 & 62 & 44 & 121 & 74 & 54 & **36** \\  & _dDMR_ & 76 & 65 & 61 & 45 & 71 & 65 & 62 & 44 & 121 & 74 & 54 & **36** \\ \hline \multirow{3}{*}{100} & _LDA_ & 77 & 71 & 66 & 40 & **45** & **38** & 30 & 20 & 135 & 74 & 54 & 31 \\  & _DMR_ & **80** & **74** & 70 & **45** & **147** & **83** & **75** & **59** & 111 & 67 & 50 & **34** \\  & _DMR-PCA_ & **79** & **69** & **75** & **45** & & & & & & & & & \\  & _dDMR_ & 77 & 73 & 68 & 44 & 68 & 67 & 66 & 55 & **135** & **78** & **55** & 31 \\ \hline \multirow{3}{*}{200} & _LDA_ & 78 & 74 & 70 & 36 & 60 & 39 & 34 & 18 & **135** & **100** & **67** & **29** \\  & _DMR_ & 91 & 76 & 80 & 42 & 69 & 67 & 67 & 61 & 132 & 84 & 59 & **32** \\  & _DMR-PCA_ & **94** & **76** & **81** & 42 & & & & & & & & & \\  & _dDMR_ & 78 & 70 & 66 & **45** & **85** & **73** & **69** & 39 & **135** & 87 & 61 & 30 \\ \hline \end{tabular}
\end{table}
Table 2: Top-1, 5, 10, and overall topic NPMI across all datasets. Models that maximized overall NPMI across dev folds were chosen and the best-performing model is in bold.

Figure 3: Heldout perplexity on the Amazon data tuning fold for _DMR_ (orange) and _dDMR_ (purple) with a (50, 10) layer architecture as a function of training parameters: L1, L2 feature weight regularization, and base learning rate. All models were trained for a fixed 5,000 iterations, with horizontal jitter added to each point.

Figure 2: Heldout perplexity as a function of iteration for lowest-perplexity models with \(Z=100\). The vertical dashed line indicates when models are burned in and hyperparameter optimization begins.



ics over _DMR_ on two of the three datasets. This is contrary to themodel rankings according to NPMI, which suggest that _DMR_ topics are often higher quality when it comes to human interpretability.

We also qualitatively explored the product image representations _DMR_ and _dMDR_ learned on the Amazon data. To do so, we computed and normalized the prior document distribution for a sample of documents for lowest perplexity _DMR_ and _dDMR_\(Z=200\) topic models: \(p(k|m)=\frac{\widetilde{g}_{n}}{\sum_{k=1}^{K}\widehat{g}_{m,x}}\), the prior probability of sampling topic \(k\), conditioned on the features for document \(m\). We then marginalize over topics to yield the conditional probability of a word \(w\) given document \(m\): \(p(w|m)=\sum_{k=1}^{Z}p(w|k)p(k|m)\). Table 4 contains a sample of these probable words given document supervision. We find that _dDMR_ identifies words likely to appear in a review of the product pictured. However, some images lead _dDMR_ down a garden path. For example, a bottle of "Turtle Food" should not be associated with words for human consumables like "coffee" and "chocolate", despite the container resembling some of these products. However, the image-specific document priors _DMR_ learned are not as sensitive to the actual product image as those learned by _dDMR_. The prior conditional probabilities \(p(w|m)\) for "Turtle Food", "Slushy Magic Cup", and "Rawhide Dog Bones" product images are all ranked identically by _DMR_.

Predictive PerformanceFinally, we consider the utility of the learned topic distributions for downstream prediction tasks, a common use of topic models. Although token perplexity is a standard measure of topic model fit, it has no direct relationship with how topic models are typically used: to identify consistent themes or reduce the dimensionality of a document corpus. We found that features based on topic distributions from _dDMR_ outperform _LDA_ and _DMR_ on the Amazon and Reddit data when the number of topics fit is large, although they fail to outperform _DMR_ on New York Times (Table 5). Heldout perplexity is strongly correlated with predictive performance, with a Pearson correlation coefficient, \(\rho=0.898\) between F1-score and heldout perplexity on the Amazon data. This strong correlation is likely due to the tight relationship between words used in product reviews and product category: a model that assigns high likelihood to a words in a product review corpus should also be informative of the product categories. Prior work showed that upstream supervised topic models, such as _DMR_, learn topic distributions that are effective at downstream prediction tasks Benton et al. (2016). We find that topic distributions learned by _dDMR_ improve over _DMR_ in certain cases, particularly as the number of topics increases.

## 6 Related Work

With the widespread adoption of neural networks, others have sought to combine topic and neural models. One line of work replaces generative, _LDA_-based, topic models with discriminatively-trained models based on neural networks Cao et al. (2015) model \(\theta\) and \(\phi\) using neural networks with softmax output layers and learn network parameters that maximize data likelihood. They also learn n-gram embeddings to identify topics whose elements are not restricted to unigrams. Chen et al. (2015) similarly expresses the (smoothed) supervised LDA Mcauliffe and Blei (2008) generative model as a neural network, and give an algorithm to discriminatively train it. Wan et al. (2012) take a similar approach to _dDMR_ where they use a neural network to extract image representations that maximize the probability of SIFT descriptors extracted from the image. However, this model is used for image classification, not for exploring a corpus of documents as is typical of topic models. These models are computationally attractive in that they avoid approximating the posterior distribution of topic assignments given tokens by dropping the assumption that \(\theta\) and \(\phi\) are drawn from Dirichlet priors. Model fitting is performed by back-propagation of a max-margin cost. In contrast, we use neural networks to learn feature representations for documents, not as a replacement for the _LDA_ generative story. This is similar to variants of SPRITE Paul and Dredze (2015), where many document-level factors are combined to generate a document-topic prior. In contrast to several of these models, the core of our topic model remains unchanged, meaning that _dDMR_ is agnostic to many other extensions of _LDA_.

There has been extensive work in modeling both textual and visual topics. Models such as Corr-LDA Blei and Jordan (2003) suppose that a text document and associated image features are generated by a shared latent topic. This property is shared by other topic models over images, such as STMTwitterLDA Cai et al. (2015) and Zhang et al. (2015). While these models try to model images, we instead use images in the Amazon data to better estimate topic distributions.

Our experiment on using images to model Ama

\begin{table}
\begin{tabular}{l|c|c} \hline  & _LDA_ & _DMR_ \\ \hline New York Times & 51.1\% & 51.9\% \\ Amazon & 51.9\% & 61.4\%* \\ Reddit & 55.5\%* & 57.6\%\% \\ \hline \end{tabular}
\end{table}
Table 3: % HITs where humans preferred _dDMR_ topics as more representative of document supervision than the competing model. \(*\) denotes statistical significance according to a one-tailed binomial test at the \(p=0.05\) level.

 

zon product reviews resembles work on image caption generation, yet the similarity is superficial. The relationship between an image and its caption is relatively tight (Fang et al., 2015) - objects in the image will likely be referenced in the caption. For Amazon product reviews, visual features of the product, like color, may be explicitly mentioned in the review, but then again, they may not. Also, the aim of topic models is to extract common themes of co-occurring words, and how those themes are distributed across each document. The similarity between our work and captioning lies only in the fact that we extract image features from a CNN trained as an object recognizer to inform document-topic distributions.

## 7 Conclusion

We present deep Dirichlet Multinomial Regression, a supervised topic model which both learns a representation of document-level features and how to use that representation for informing a topic distribution. We demonstrate the flexibility of our model on three corpora with different types of metadata: topic descriptor tags, images, and subreddit IDs. _dDMR_ is better able to fit text corpora with high-dimensional supervision compared to _LDA_ or _DMR_. Furthermore, we find that document supervision greatly reduces the number of Gibbs sampling iterations for a topic model to converge, and that the _dDMR_ prior architecture makes it more robust to training parameters than _DMR_. We also find that the topic distributions learned by _dDMR_ are more predictive of external

\begin{table}
\begin{tabular}{|c|p{86.9pt}|p{34.1pt}|p{142.3pt}|} \hline
**Image** & **Item** & _dDMR_ Probable Words & _DMR_ Probable Words \\ \hline  & **Guitar Foot Rest** & **gifl easy cover well fit mover** fits **job gas hose light heavy easily stand back nice works use enough pressure** & fit easy well works car light would **scar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar sound quality work guitar guitar sound quality work guitar guitar sound quality work guitar audio text 

document labels such as known topic tags or product category as the number of topics grows and that _dDMR_ topics are judged as more representative of the document metadata by human subjects. Source code for training _dDMR_ can be found at [http://www.github.com/abenton/deep-dmr](http://www.github.com/abenton/deep-dmr).

## References

* Benton et al. (2016) Adrian Benton, Michael J Paul, Braden Hancock, and Mark Dredze. 2016. Collective supervision of topic models for predicting surveys with social media. In _Proceedings of the AAAI Conference on Artificial Intelligence_. pages 2892-2898.
* Blei and Jordan (2003) David M Blei and Michael I Jordan. 2003. Modeling annotated data. In _Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval_. ACM, pages 127-134.
* Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. _Journal of Machine Learning Research_ 3(Jan):993-1022.
* Brown et al. (1992) Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. _Computational linguistics_ 18(4):467-479.
* Cai et al. (2015) Hongyun Cai, Yang Yang, Xuefei Li, and Zi Huang. 2015. What are popular: exploring twitter features for event detection, tracking and visualization. In _Proceedings of the 23rd ACM International Conference on Multimedia_. ACM, pages 89-98.
* Callison-Burch and Dredze (2010) Chris Callison-Burch and Mark Dredze. 2010. Creating speech and language data with amazon's mechanical turk. In _NAACL-HLT Workshop on Creating Speech and Language Data With Mechanical Turk_. pages 1-12.
* Cao et al. (2015) Ziqiang Cao, Sujian Li, Yang Liu, Wenjie Li, and Heng Ji. 2015. A novel neural topic model and its supervised extension. In _Proceedings of the AAAI conference on Artificial Intelligence_. pages 2210-2216.
* Chen et al. (2015) Jianshu Chen, Ji He, Yelong Shen, Lin Xiao, Xiaodong He, Jianfeng Gao, Xinying Song, and Li Deng. 2015. End-to-end learning of LDA by mirror-descent back propagation over a deep architecture. In _Advances in Neural Information Processing Systems_. pages 1765-1773.
* Collobert and Weston (2008) Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In _Proceedings of the 25th International Conference on Machine Learning_. ACM, pages 160-167.
* Fang et al. (2015) Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollar, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. pages 1473-1482.
* Griffiths and Steyvers (2004) Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. _Proceedings of the National Academy of Sciences_ 101(suppl 1):5228-5235.
* Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_. pages 530-539.

\begin{table}
\begin{tabular}{l l||c c|c c c|c c c} \hline  & & \multicolumn{3}{c}{New York Times} & \multicolumn{3}{c}{Amazon} & \multicolumn{3}{c}{Reddit} \\ \cline{2-10} \(Z\) & Model & F1 & Accuracy & AUC & & & & & \\ \hline \hline \multirow{4}{*}{10} & _LDA_ & 0.208 & **0.380** & 0.767 & **0.662** & **0.667** & **0.891** & 0.130 & 0.276 & 0.565 \\  & _DMR_ & 0.236 & 0.367 & 0.781 & 0.311 & 0.407 & 0.619 & 0.092 & 0.229 & **0.597** \\  & _DMR-PCA_ & **0.280** & 0.347 & 0.758 & & & & & \\  & _dDMR_ & 0.154 & 0.347 & **0.790** & 0.608 & 0.656 & 0.864 & **0.170** & **0.300** & 0.596 \\ \hline \multirow{4}{*}{20} & _LDA_ & 0.315 & 0.463 & 0.784 & 0.657 & 0.659 & 0.887 & **0.121** & 0.258 & **0.579** \\  & _DMR_ & 0.319 & 0.477 & 0.805 & 0.294 & 0.405 & 0.647 & 0.057 & 0.245 & 0.520 \\  & _DMR-PCA_ & 0.343 & **0.540** & **0.831** & & & & & \\  & _dDMR_ & **0.424** & 0.523 & 0.797 & **0.706** & **0.711** & **0.911** & 0.071 & **0.274** & 0.566 \\ \hline \multirow{4}{*}{50} & _LDA_ & 0.455 & 0.613 & 0.849 & 0.630 & 0.634 & 0.870 & 0.131 & 0.199 & 0.542 \\  & _DMR_ & 0.478 & 0.650 & 0.877 & 0.396 & 0.499 & 0.619 & **0.145** & 0.261 & **0.580** \\  & _DMR-PCA_ & 0.505 & **0.667** & **0.887** & & & & & \\  & _dDMR_ & **0.507** & 0.657 & 0.856 & **0.716** & **0.726** & **0.916** & 0.118 & **0.272** & 0.551 \\ \hline \multirow{4}{*}{100} & _LDA_ & 0.531 & 0.657 & 0.874 & 0.646 & 0.649 & 0.874 & 0.148 & 0.201 & 0.538 \\  & _DMR_ & 0.552 & 0.683 & 0.898 & 0.392 & 0.463 & 0.688 & 0.107 & 0.233 & 0.512 \\ \cline{1-1}  & _DMR-PCA_ & **0.602** & **0.687** & **0.917** & & & & & \\ \cline{1-1}  & _dDMR_ & 0.514 & 0.653 & 0.893 & **0.650** & **0.660** & **0.893** & **0.172** & **0.316** & **0.614** \\ \hline \multirow{4}{*}{200} & _LDA_ & 0.566 & 0.683 & 0.903 & 0.646 & 0.651 & 0.882 & 0.111 & 0.227 & 0.517 \\ \cline{1-1}  & _DMR_ & 0.576 & 0.670 & **0.917** & & & & & \\ \cline{1-1}  & _DMR-PCA_ & **0.648** & **0.762** & 0.915 & & & & & \\ \cline{1-1}  & _dDMR_ & 0.605 & 0.730 & 0.903 & **0.716** & **0.721** & **0.909** & **0.198** & **0.323** & **0.580** \\ \hline \end{tabular}
\end{table}
Table 5: Top F-score, accuracy, and AUC on prediction tasks for all datasets.

 

Julian McAuley and Alex Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. In _Proceedings of the 25th International Conference on World Wide Web_. International World Wide Web Conferences Steering Committee, pages 625-635.
* Mcauliffe and Blei (2008) Jon D Mcauliffe and David M Blei. 2008. Supervised topic models. In _Advances in Neural Information Processing Systems_. pages 121-128.
* McCallum et al. (2007) Andrew McCallum, Xuerui Wang, and Andres Corrada-Emmanuel. 2007. Topic and role discovery in social networks with experiments on enron and academic email. _Journal of Artificial Intelligence Research_ .
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in Neural Information Processing Systems_. pages 3111-3119.
* Paul and Dredze (2015) Michael J Paul and Mark Dredze. 2015. Sprite: Generalizing topic models with structured priors. _Transactions of the Association for Computational Linguistics_ 3:43-57.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods on Natural Language Processing_. volume 14, pages 1532-1543.
* Ramage et al. (2009) Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D Manning. 2009. Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 248-256.
* Rosen-Zvi et al. (2004) Michal Rosen-Zvi, Thomas Griffiths, Mark Steyvers, and Padhraic Smyth. 2004. The author-topic model for authors and documents. In _Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence_. AUAI Press, pages 487-494.
* Sandhaus (2008) Evan Sandhaus. 2008. The new york times annotated corpus. _Linguistic Data Consortium, Philadelphia_ 6(12):e26752.
* Schmidt (2012) Benjamin M Schmidt. 2012. Words alone: Distantling topic models in the humanities. _Journal of Digital Humanities_ 2(1):49-65.
* Wallach et al. (2009) Hanna M Wallach, Iain Murray, Ruslan Salakhutdinov, and David Mimno. 2009. Evaluation methods for topic models. In _Proceedings of the 26th Annual International Conference on Machine Learning_. ACM, pages 1105-1112.
* Wan et al. (2012) Li Wan, Leo Zhu, and Rob Fergus. 2012. A hybrid neural network-latent topic model. In _Proceedings of the 15th International Conference on Artificial Intelligence and Statistics_. volume 12, pages 1287-1294.
* Yu et al. (2015) Mo Yu, Matthew R Gormley, and Mark Dredze. 2015. Combining word embeddings and feature embeddings for fine-grained relation extraction. In _Proceedings of the 14th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. pages 1374-1379.
* Zeiler (2012) Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. _arXiv preprint arXiv:1212.5701_ .
* Zhang et al. (2015) Hao Zhang, Gunhee Kim, and Eric P Xing. 2015. Dynamic topic modeling for monitoring market competition from online text and image data. In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, pages 1425-1434.