Introduction
Word embedding models, which learn to encode dictionary words with vector space representations, have been shown to be valuable for a variety of natural language processing (NLP) tasks such as statistical machine translation BIBREF2 , part-of-speech tagging, chunking, and named entity recogition BIBREF3 , as they provide a more nuanced representation of words than a simple indicator vector into a dictionary. These models follow a long line of research in data-driven semantic representations of text, including latent semantic analysis BIBREF4 and its probabilistic extensions BIBREF5 , BIBREF6 . In particular, topic models BIBREF7 have found broad applications in computational social science BIBREF8 , BIBREF9 and the digital humanities BIBREF10 , where interpretable representations reveal meaningful insights. Despite widespread success at NLP tasks, word embeddings have not yet supplanted topic models as the method of choice in computational social science applications. I speculate that this is due to two primary factors: 1) a perceived reliance on big data, and 2) a lack of interpretability. In this work, I develop new models to address both of these limitations.
Word embeddings have risen in popularity for NLP applications due to the success of models designed specifically for the big data setting. In particular, BIBREF0 , BIBREF1 showed that very simple word embedding models with high-dimensional representations can scale up to massive datasets, allowing them to outperform more sophisticated neural network language models which can process fewer documents. In this work, I offer a somewhat contrarian perspective to the currently prevailing trend of big data optimism, as exemplified by the work of BIBREF0 , BIBREF1 , BIBREF3 , and others, who argue that massive datasets are sufficient to allow language models to automatically resolve many challenging NLP tasks. Note that “big” datasets are not always available, particularly in computational social science NLP applications, where the data of interest are often not obtained from large scale sources such as the internet and social media, but from sources such as press releases BIBREF11 , academic journals BIBREF10 , books BIBREF12 , and transcripts of recorded speech BIBREF13 , BIBREF14 , BIBREF15 . A standard practice in the literature is to train word embedding models on a generic large corpus such as Wikipedia, and use the embeddings for NLP tasks on the target dataset, cf. BIBREF3 , BIBREF0 , BIBREF16 , BIBREF17 . However, as we shall see here, this standard practice might not always be effective, as the size of a dataset does not correspond to its degree of relevance for a particular analysis. Even very large corpora have idiosyncrasies that can make their embeddings invalid for other domains. For instance, suppose we would like to use word embeddings to analyze scientific articles on machine learning. In Table TABREF1 , I report the most similar words to the word “learning” based on word embedding models trained on two corpora. For embeddings trained on articles from the NIPS conference, the most similar words are related to machine learning, as desired, while for embeddings trained on the massive, generic Google News corpus, the most similar words relate to learning and teaching in the classroom. Evidently, domain-specific data can be important.
Even more concerningly, BIBREF18 show that word embeddings can encode implicit sexist assumptions. This suggests that when trained on large generic corpora they could also encode the hegemonic worldview, which is inappropriate for studying, e.g., black female hip-hop artists' lyrics, or poetry by Syrian refugees, and could potentially lead to systematic bias against minorities, women, and people of color in NLP applications with real-world consequences, such as automatic essay grading and college admissions. In order to proactively combat these kinds of biases in large generic datasets, and to address computational social science tasks, there is a need for effective word embeddings for small datasets, so that the most relevant datasets can be used for training, even when they are small. To make word embeddings a viable alternative to topic models for applications in the social sciences, we further desire that the embeddings are semantically meaningful to human analysts.
In this paper, I introduce an interpretable word embedding model, and an associated topic model, which are designed to work well when trained on a small to medium-sized corpus of interest. The primary insight is to use a data-efficient parameter sharing scheme via mixed membership modeling, with inspiration from topic models. Mixed membership models provide a flexible yet efficient latent representation, in which entities are associated with shared, global representations, but to uniquely varying degrees. I identify the skip-gram word2vec model of BIBREF0 , BIBREF1 as corresponding to a certain naive Bayes topic model, which leads to mixed membership extensions, allowing the use of fewer vectors than words. I show that this leads to better modeling performance without big data, as measured by predictive performance (when the context is leveraged for prediction), as well as to interpretable latent representations that are highly valuable for computational social science applications. The interpretability of the representations arises from defining embeddings for words (and hence, documents) in terms of embeddings for topics. My experiments also shed light on the relative merits of training embeddings on generic big data corpora versus domain-specific data.
Background
In this section, I provide the necessary background on word embeddings, as well as on topic models and mixed membership models. Traditional language models aim to predict words given the contexts that they are found in, thereby forming a joint probabilistic model for sequences of words in a language. BIBREF19 developed improved language models by using distributed representations BIBREF20 , in which words are represented by neural network synapse weights, or equivalently, vector space embeddings.
Later authors have noted that these word embeddings are useful for semantic representations of words, independently of whether a full joint probabilistic language model is learned, and that alternative training schemes can be beneficial for learning the embeddings. In particular, BIBREF0 , BIBREF1 proposed the skip-gram model, which inverts the language model prediction task and aims to predict the context given an input word. The skip-gram model is a log-bilinear discriminative probabilistic classifier parameterized by “input” word embedding vectors INLINEFORM0 for the input words INLINEFORM1 , and “output” word embedding vectors INLINEFORM2 for context words INLINEFORM3 , as shown in Table TABREF2 , top-left.
Topic models such as latent Dirichlet allocation (LDA) BIBREF7 are another class of probabilistic language models that have been used for semantic representation BIBREF6 . A straightforward way to model text corpora is via unsupervised multinomial naive Bayes, in which a latent cluster assignment for each document selects a multinomial distribution over words, referred to as a topic, with which the documents' words are assumed to be generated. LDA topic models improve over naive Bayes by using a mixed membership model, in which the assumption that all words in a document INLINEFORM0 belong to the same topic is relaxed, and replaced with a distribution over topics INLINEFORM1 . In the model's assumed generative process, for each word INLINEFORM2 in document INLINEFORM3 , a topic assignment INLINEFORM4 is drawn via INLINEFORM5 , then the word is drawn from the chosen topic INLINEFORM6 . The mixed membership formalism provides a useful compromise between model flexibility and statistical efficiency: the INLINEFORM7 topics INLINEFORM8 are shared across all documents, thereby sharing statistical strength, but each document is free to use the topics to its own unique degree. Bayesian inference further aids data efficiency, as uncertainty over INLINEFORM9 can be managed for shorter documents. Some recent papers have aimed to combine topic models and word embeddings BIBREF21 , BIBREF22 , but they do not aim to address the small data problem for computational social science, which I focus on here. I provide a more detailed discussion of related work in the supplementary.
The Mixed Membership Skip-Gram
To design an interpretable word embedding model for small corpora, we identify novel connections between word embeddings and topic models, and adapt advances from topic modeling. Following the distributional hypothesis BIBREF23 , the skip-gram's word embeddings parameterize discrete probability distributions over words INLINEFORM0 which tend to co-occur, and tend to be semantically coherent – a property leveraged by the Gaussian LDA model of BIBREF21 . This suggests that these discrete distributions can be reinterpreted as topics INLINEFORM1 . We thus reinterpret the skip-gram as a parameterization of a certain supervised naive Bayes topic model (Table TABREF2 , top-right). In this topic model, input words INLINEFORM2 are fully observed “cluster assignments,” and the words in INLINEFORM3 's contexts are a “document.” The skip-gram differs from this supervised topic model only in the parameterization of the “topics” via word vectors which encode the distributions with a log-bilinear model. Note that although the skip-gram is discriminative, in the sense that it does not jointly model the input words INLINEFORM4 , we are here equivalently interpreting it as encoding a “conditionally generative” process for the context given the words, in order to develop probabilistic models that extend the skip-gram.
As in LDA, this model can be improved by replacing the naive Bayes assumption with a mixed membership assumption. By applying the mixed membership representation to this topic model version of the skip-gram, we obtain the model in the bottom-right of Table TABREF2 . After once again parameterizing this model with word embeddings, we obtain our final model, the mixed membership skip-gram (MMSG) (Table TABREF2 , bottom-left). In the model, each input word has a distribution over topics INLINEFORM0 . Each topic has a vector-space embedding INLINEFORM1 and each output word has a vector INLINEFORM2 (a parameter, not an embedding for INLINEFORM3 ). A topic INLINEFORM4 is drawn for each context, and the words in the context are drawn from the log-bilinear model using INLINEFORM5 : DISPLAYFORM0
We can expect that the resulting mixed membership word embeddings are beneficial in the small-to-medium data regime for the following reasons:
Of course, the model also requires some new parameters to be learned, namely the mixed membership proportions INLINEFORM0 . Based on topic modeling, I hypothesized that with care, these added parameters need not adversely affect performance in the small-medium data regime, for two reasons: 1) we can use a Bayesian approach to effectively manage uncertainty in them, and to marginalize them out, which prevents them being a bottleneck during training; and 2) at test time, using the posterior for INLINEFORM1 given the context, instead of the “prior” INLINEFORM2 , mitigates the impact of uncertainty in INLINEFORM3 due to limited training data: DISPLAYFORM0
To obtain a vector for a word type INLINEFORM0 , we can use the prior mean, INLINEFORM1 . For a word token INLINEFORM2 , we can leverage its context via the posterior mean, INLINEFORM3 . These embeddings are convex combinations of topic vectors (see Figure FIGREF23 for an example). With fewer vectors than words, some model capacity is lost, but the flexibility of the mixed membership representation allows the model to compensate. When the number of shared vectors equals the number of words, the mixed membership skip-gram is strictly more representationally powerful than the skip-gram. With more vectors than words, we can expect that the increased representational power would be beneficial in the big data regime. As this is not my goal, I leave this for future work.
Experimental Results
The goals of our experiments were to study the relative merits of big data and domain-specific small data, to validate the proposed methods, and to study their applicability for computational social science research.
Quantitative Experiments
I first measured the effectiveness of the embeddings at the skip-gram's training task, predicting context words INLINEFORM0 given input words INLINEFORM1 . This task measures the methods' performance for predictive language modeling. I used four datasets of sociopolitical, scientific, and literary interest: the corpus of NIPS articles from 1987 – 1999 ( INLINEFORM2 million), the U.S. presidential state of the Union addresses from 1790 – 2015 ( INLINEFORM3 ), the complete works of Shakespeare ( INLINEFORM4 ; this version did not contain the Sonnets), and the writings of black scholar and activist W.E.B. Du Bois, as digitized by Project Gutenberg ( INLINEFORM5 ). For each dataset, I held out 10,000 INLINEFORM6 pairs uniformly at random, where INLINEFORM7 , and aimed to predict INLINEFORM8 given INLINEFORM9 (and optionally, INLINEFORM10 ). Since there are a large number of classes, I treat this as a ranking problem, and report the mean reciprocal rank. The experiments were repeated and averaged over 5 train/test splits.
The results are shown in Table TABREF25 . I compared to a word frequency baseline, the skip-gram (SG), and Tomas Mikolov/Google's vectors trained on Google News, INLINEFORM0 billion, via CBOW. Simulated annealing was performed for 1,000 iterations, NCE was performed for 1 million minibatches of size 128, and 128-dimensional embeddings were used (300 for Google). I used INLINEFORM1 for NIPS, INLINEFORM2 for state of the Union, and INLINEFORM3 for the two smaller datasets. Methods were able to leverage the remainder of the context, either by adding the context's vectors, or via the posterior (Equation EQREF22 ), which helped for all methods except the naive skip-gram. We can identify several noteworthy findings. First, the generic big data vectors (Google+context) were outperformed by the skip-gram on 3 out of 4 datasets (and by the skip-gram topic model on the other), by a large margin, indicating that domain-specific embeddings are often important. Second, the mixed membership models, using posterior inference, beat or matched their naive Bayes counterparts, for both the word embedding models and the topic models. As hypothesized, posterior inference on INLINEFORM4 at test time was important for good performance. Finally, the topic models beat their corresponding word embedding models at prediction. I therefore recommend the use of our MMSG topic model variant for predictive language modeling in the small data regime.
I tested the performance of the representations as features for document categorization and regression tasks. The results are given in Table TABREF26 . For document categorization, I used three standard benchmark datasets: 20 Newsgroups (19,997 newsgroup posts), Reuters-150 newswire articles (15,500 articles and 150 classes), and Ohsumed medical abstracts on 23 cardiovascular diseases (20,000 articles). I held out 4,000 test documents for 20 Newsgroups, and used the standard train/test splits from the literature in the other corpora (e.g. for Ohsumed, 50% of documents were assigned to training and to test sets). I obtained document embeddings for the MMSG, in the same latent space as the topic embeddings, by summing the posterior mean vectors INLINEFORM0 for each token. Vector addition was similarly used to construct document vectors for the other embedding models. All vectors were normalized to unit length. I also considered a tf-idf baseline. Logistic regression models were trained on the features extracted on the training set for each method.
Across the three datasets, several clear trends emerged (Table TABREF26 ). First, the generic Google vectors were consistently and substantially outperformed in classification performance by the skipgram (SG) and MMSG vectors, highlighting the importance of corpus-specific embeddings. Second, despite the MMSG's superior performance at language modeling on small datasets, the SG features outperformed the MMSG's at the document categorization task. By encoding vectors at the topic level instead of the word level, the MMSG loses word level resolution in the embeddings, which turned out to be valuable for these particular classification tasks. We are not, however, restricted to use only one type of embedding to construct features for classification. Interestingly, when the SG and MMSG features were concatenated (SG+MMSG), this improved classification performance over these vectors individually. This suggests that the topic-level MMSG vectors and word-level SG vectors encode complementary information, and both are beneficial for performance. Finally, further concatenating the generic Google vectors' features (SG+MMSG+Google) improved performance again, despite the fact that these vectors performed poorly on their own. It should be noted that tf-idf, which is notoriously effective for document categorization, outperformed the embedding methods on these datasets.
I also analyzed the regression task of predicting the year of a state of the Union address based on its text information. I used lasso-regularized linear regression models, evaluated via a leave-one-out cross-validation experimental setup. Root-mean-square error (RMSE) results are reported in Table TABREF26 (bottom). Unlike for the other tasks, the Google big data vectors were the best individual features in this case, outperforming the domain-specific SG and MMSG embeddings individually. On the other hand, SG+MMSG+Google performed the best overall, showing that domain-specific embeddings can improve performance even when big data embeddings are successful. The tf-idf baseline was beaten by all of the embedding models on this task.
Computational Social Science Case Studies: State of the Union and NIPS
I also performed several case studies. I obtained document embeddings, in the same latent space as the topic embeddings, by summing the posterior mean vectors INLINEFORM0 for each token, and visualized them in two dimensions using INLINEFORM1 -SNE BIBREF24 (all vectors were normalized to unit length). The state of the Union addresses (Figure FIGREF27 ) are embedded almost linearly by year, with a major jump around the New Deal (1930s), and are well separated by party at any given time period. The embedded topics (gray) allow us to interpret the space. The George W. Bush addresses are embedded near a “war on terror” topic (“weapons, war...”), and the Barack Obama addresses are embedded near a “stimulus” topic (“people, work...”).
On the NIPS corpus, for the input word “Bayesian” (Table ), the naive Bayes and skip-gram models learned a topic with words that refer to Bayesian networks, probabilistic models, and neural networks. The mixed membership models are able to separate this into more coherent and specific topics including Bayesian modeling, Bayesian training of neural networks (for which Sir David MacKay was a strong proponent, and Andreas Weigend wrote an influential early paper), and Monte Carlo methods. By performing the additive composition of word vectors, which we obtain by finding the prior mean vector for each word type INLINEFORM0 , INLINEFORM1 (and then normalizing), we obtain relevant topics INLINEFORM2 as nearest neighbors (Figure FIGREF28 ). Similarly, we find that the additive composition of topic and word vectors works correctly: INLINEFORM3 , and INLINEFORM4 .
The INLINEFORM0 -SNE visualization of NIPS documents (Figure FIGREF28 ) shows some temporal clustering patterns (blue documents are more recent, red documents are older, and gray points are topics). I provide a more detailed case study on NIPS in the supplementary material.
Conclusion
I have proposed a model-based method for training interpretable corpus-specific word embeddings for computational social science, using mixed membership representations, Metropolis-Hastings-Walker sampling, and NCE. Experimental results for prediction, supervised learning, and case studies on state of the Union addresses and NIPS articles, indicate that high-quality embeddings and topics can be obtained using the method. The results highlight the fact that big data is not always best, as domain-specific data can be very valuable, even when it is small. I plan to use this approach for substantive social science applications, and to address algorithmic bias and fairness issues.
Acknowledgements
I thank Eric Nalisnick and Padhraic Smyth for many helpful discussions.
Supplementary Material
]
Related Work
In this supplementary document, we discuss related work in the literature and its relation to our proposed methods, provide a case study on NIPS articles, and derive the collapsed Gibbs sampling update for the MMSGTM, which we leverage when training the MMSG.
Topic Modeling and Word Embeddings
The Gaussian LDA model of BIBREF21 improves the performance of topic modeling by leveraging the semantic information encoded in word embeddings. Gaussian LDA modifies the generative process of LDA such that each topic is assumed to generate the vectors via its own Gaussian distribution. Similarly to our MMSG model, in Gaussian LDA each topic is encoded with a vector, in this case the mean of the Gaussian. It takes pre-trained word embeddings as input, rather than learning the embeddings from data within the same model, and does not aim to perform word embedding.
The topical word embedding (TWE) models of BIBREF22 reverse this, as they take LDA topic assignments of words as input, and aim to use them to improve the resultant word embeddings. The authors propose three variants, each of which modifies the skip-gram training objective to use LDA topic assignments together with words. In the best performing variant, called TWE-1, a standard skip-gram word embedding model is trained independently with another skip-gram variant, which tries to predict context words given the input word's topic assignment. The skip-gram embedding and the topic embeddings are concatenated to form the final embedding.
At test time, a distribution over topics for the word given the context, INLINEFORM0 is estimated according to the topic counts over the other context words. Using this as a prior, a posterior over topics given both the input word and the context is calculated, and similarities between pairs of words (with their contexts) are averaged over this posterior, in a procedure inspired by those used by BIBREF43 , BIBREF36 . The primary similarity to our MMSG approach is the use of a training algorithm involving the prediction of context words, given a topic. Our method does this as part of an overall model-based inference procedure, and we learn mixed membership proportions INLINEFORM1 rather than using empirical counts as the prior over topics for a word token. In accordance with the skip-gram's prediction model, we are thus able to model the context words in the data likelihood term when computing the posterior probability of the topic assignment. TWE-1 requires that topic assignments are available at test time. It provides a mechanism to predict contextual similarity, but not to predict held-out context words, so we are unable to compare to it in our experiments.
Other neurally-inspired topic models include replicated softmax BIBREF34 , and its successor, DocNADE BIBREF37 . Replicated softmax extends the restricted Boltzmann machine to handle multinomial counts for document modeling. DocNADE builds on the ideas of replicated softmax, but uses the NADE architecture, where observations (i.e. words) are modeled sequentially given the previous observations.
Multi-Prototype Embedding Models
Multi-prototype embeddings models are another relevant line of work. These models address lexical ambiguity by assigning multiple vectors to each word type, each corresponding to a different meaning of that word. BIBREF43 propose to cluster the occurrences of each word type, based on features extracted from its context. Embeddings are then learned for each cluster. BIBREF36 apply a similar approach, but they use initial single-prototype word embeddings to provide the features used for clustering. These clustering methods have some resemblance to our topic model pre-clustering step, although their clustering is applied within instances of a given word type, rather than globally across all word types, as in our methods. This results in models with more vectors than words, while we aim to find fewer vectors than words, to reduce the model's complexity for small datasets. Rather than employing an off-the-shelf clustering algorithm and then applying an unrelated embedding model to its output, our approach aims to perform model-based clustering within an overall joint model of topic/cluster assignments and word vectors.
Perhaps the most similar model to ours in the literature is the probabilistic multi-prototype embedding model of BIBREF45 , who treat the prototype assignment of a word as a latent variable, assumed drawn from a mixture over prototypes for each word. The embeddings are then trained using EM. Our MMSG model can be understood as the mixed membership version of this model, in which the prototypes (vectors) are shared across all word types, and each word type has its own mixed membership proportions across the shared prototypes. While a similar EM algorithm can be applied to the MMSG, the E-step is much more expensive, as we typically desire many more shared vectors (often in the thousands) than we would prototypes per a single word type (Tian et al. use ten in their experiments). We use the Metropolis-Hastings-Walker algorithm with the topic model reparameterization of our model in order to address this by efficiently pre-solving the E-step.
Mixed Membership Modeling
Mixed membership modeling is a flexible alternative to traditional clustering, in which each data point is assigned to a single cluster. Instead, mixed membership models posit that individual entities are associated with multiple underlying clusters, to differing degrees, as encoded by a mixed membership vector that sums to one across the clusters BIBREF28 , BIBREF26 . These mixed membership proportions are generally used to model lower-level grouped data, such as the words inside a document. Each lower-level data point inside a group is assumed to be assigned to one of the shared, global clusters according to the group-level membership proportions. Thus, a mixed membership model consists of a mixture model for each group, which share common mixture component parameters, but with differing mixture proportions.
This formalism has lead to probabilistic models for a variety of applications, including medical diagnosis BIBREF39 , population genetics BIBREF42 , survey analysis BIBREF29 , computer vision BIBREF27 , BIBREF30 , text documents BIBREF35 , BIBREF7 , and social network analysis BIBREF25 . Nonparametric Bayesian extensions, in which the number of underlying clusters is learned from data via Bayesian inference, have also been proposed BIBREF44 . In this work, dictionary words are assigned a mixed membership distribution over a set of shared latent vector space embeddings. Each instantiation of a dictionary word (an “input” word) is assigned to one of the shared embeddings based on its dictionary word's membership vector. The words in its context (“output” words) are assumed to be drawn based on the chosen embedding.
Case Study on NIPS
In Figure FIGREF33 , we show a zoomed in INLINEFORM0 -SNE visualization of NIPS document embeddings. We can see regions of the space corresponding to learning algorithms (bottom), data space and latent space (center), training neural networks (top), and nearest neighbors (bottom-left). We also visualized the authors' embeddings via INLINEFORM1 -SNE (Figure FIGREF34 ). We find regions of latent space for reinforcement learning authors (left: “state, action,...,” Singh, Barto,Sutton), probabilistic methods (right: “mixture, model,” “monte, carlo,” Bishop, Williams, Barber, Opper, Jordan, Ghahramani, Tresp, Smyth), and evaluation (top-right: “results, performance, experiments,...”).
Derivation of the Collapsed Gibbs Update
Let INLINEFORM0 be the number of output words in the INLINEFORM1 th context, let INLINEFORM2 be those output words, and let INLINEFORM3 be the input words other that INLINEFORM4 (similarly, topic assignments INLINEFORM5 and output words INLINEFORM6 ). Then the collapsed Gibbs update samples from the conditional distribution INLINEFORM7
We recognize the first integral as the mean of a Dirichlet distribution which we obtain via conjugacy: INLINEFORM0
The above can also be understood as the probability of the next ball drawn from a multivariate Polya urn model, also known as the Dirichlet-compound multinomial distribution, arising from the posterior predictive distribution of a discrete likelihood with a Dirichlet prior. We will need the full form of such a distribution to analyze the second integral. Once again leveraging conjugacy, we have: INLINEFORM0
INLINEFORM0
where INLINEFORM0 is the number of times that output word INLINEFORM1 occurs in the INLINEFORM2 th context, since the final integral is over the full support of a Dirichlet distribution, which integrates to one. Eliminating terms that aren't affected by the INLINEFORM3 assignment, the above is INLINEFORM4
where we have used the fact that INLINEFORM0 for any INLINEFORM1 , and integer INLINEFORM2 . We can interpret this as the probability of drawing the context words under the multivariate Polya urn model, in which the number of “colored balls” (word counts plus prior counts) is increased by one each time a certain color (word) is selected. In other words, in each step, corresponding to the selection of each context word, we draw a ball from the urn, then put it back, along with another ball of the same color. The INLINEFORM3 and INLINEFORM4 terms reflect that the counts have been changed by adding these extra balls into the urn in each step. The second to last equation shows that this process is exchangeable: it does not matter which order the balls were drawn in when determining the probability of the sequence. Multiplying this with the term from the first integral, calculated earlier, gives us the final form of the update equation, INLINEFORM5

Question: What supervised learning tasks are attempted with these representations?
Answer: document categorization, regression tasks

Question: What is MRR?
Answer: mean reciprocal rank

Question: Which techniques for word embeddings and topic models are used?
Answer:  skip-gram, LDA

Question: Why is big data not appropriate for this task?
Answer: Training embeddings from small-corpora can increase the performance of some tasks

Question: What is an example of a computational social science NLP task?
Answer: Visualization of State of the union addresses