Toward Interpretable Topic Discovery via Anchored Correlation Explanation

Abstract: Many predictive tasks, such as diagnosing a patient based on their medical chart, are ultimately defined by the decisions of human experts. Unfortunately, encoding experts' knowledge is often time consuming and expensive. We propose a simple way to use fuzzy and informal knowledge from experts to guide discovery of interpretable latent topics in text. The underlying intuition of our approach is that latent factors should be informative about both correlations in the data and a set of relevance variables specified by an expert. Mathematically, this approach is a combination of the information bottleneck and Total Correlation Explanation (CorEx). We give a preliminary evaluation of Anchored CorEx, showing that it produces more coherent and interpretable topics on two distinct corpora.

A clinician can look at a patient's electronic health record (EHR) and not only decide whether the patient has diabetes but also produce a succinct summary of the clinical evidence. Replicating this feat with computational tools has been the focus of much research in clinical informatics. There are major initiatives underway to codify clinical knowledge into formal representations, most often as deterministic rules that can be applied in a semi-automated fashion BIBREF0 . However, representing the intuitive judgments of human experts can be challenging, particularly when the formal system does not match the expert's knowledge. For example, many deterministic disease classifiers used in clinical informatics rely heavily upon administrative codes not available at time of diagnosis. Further, developing and testing such systems is time- and labor-intensive.
We propose instead a lightweight information theoretic framework for codifying informal human knowledge and then use it to extract interpretable latent topics from text corpora. For example, to discover patients with diabetes in a set of clinical notes, a doctor can begin by specifying disease-specific anchor terms BIBREF1 , BIBREF2 , such as “diabetes” or “insulin.” Our framework then uses these to help discover both latent topics associated with diabetes and records in which diabetes-related topics occur. The user can then add (or remove) additional anchor terms (e.g., “metformin”) to improve the quality of the learned (diabetes) topics.
In this workshop paper, we introduce a simple approach to anchored information theoretic topic modeling using a novel combination of Correlation Explanation (CorEx) BIBREF3 and the information bottleneck BIBREF4 . This flexible framework enables the user to leverage domain knowledge to guide exploration of a collection of documents and to impose semantics onto latent factors learned by CorEx. We present preliminary experimental results on two text corpora (including a corpus of clinical notes), showing that anchors can be used to discover topics that are more specific and relevant. What is more, we demonstrate the potential for this framework to perform weakly supervised learning in settings where labeling documents is prohibitively expensive BIBREF5 , BIBREF6 .
With respect to interpretable machine learning, our contributions are twofold. First, our framework provides a way for human users to share domain knowledge with a statistical learning algorithm that is both convenient for the human user and easily digestible by the machine. Second, our experimental results confirm that the introduction of simple anchor words can improve the coherence and human interpretability of topics discovered from data. Both are essential to successful and interactive collaboration between machine learning and human users.
Anchored Correlation Explanation can be understood as a combination of Total Correlation Explanation (CorEx) BIBREF3 , BIBREF7 and the multivariate information bottleneck BIBREF4 , BIBREF8 . We search for a set of probabilistic functions of the inputs INLINEFORM0 for INLINEFORM1 that optimize the following information theoretic objective: INLINEFORM2 
The first term is the CorEx objective INLINEFORM0 , which aims to construct latent variables INLINEFORM1 that best explain multivariate dependencies in the data INLINEFORM2 . Here the data consist of INLINEFORM3 -dimensional binary vectors INLINEFORM4 . Total correlation, or multivariate mutual information BIBREF9 , is specified as INLINEFORM5 where INLINEFORM6 is the KL divergence. Maximizing INLINEFORM7 over latent factors INLINEFORM8 amounts to minimizing INLINEFORM9 , which measures how much dependence in INLINEFORM10 is explained by INLINEFORM11 . At the global optimum, INLINEFORM12 is zero and the observations are independent conditioned on the latent factors. Several papers have explored CorEx for unsupervised hierarchical topic modeling BIBREF3 , BIBREF10 , BIBREF11 .
The second term involves the mutual information between pairs of latent factors INLINEFORM0 ) and anchor variables INLINEFORM1 specified in the set INLINEFORM2 . This is inspired by the information bottleneck BIBREF4 , BIBREF8 , a supervised information-theoretic approach to discovering latent factors. The bottleneck objective INLINEFORM3 constructs latent factors INLINEFORM4 that trade off compression of INLINEFORM5 against preserving information about relevance variables INLINEFORM6 .
Anchored CorEx preserves information about anchors while also explaining as much multivariate dependence between observations in INLINEFORM0 as possible. This framework is flexible: we can attach multiple anchors to one factor or one anchor to multiple factors. We have found empirically that INLINEFORM1 works well and does not need to be tuned.
Anchors allow us to both seed CorEx and impose semantics on latent factors: when analyzing medical documents, for example, we can anchor a diabetes latent factor to the word “diabetes.” The INLINEFORM0 objective then discovers other words associated with “diabetes” and includes them in this topic.
While there is not space here for a full description of the optimization, it is similar in principle to the approaches in BIBREF3 , BIBREF7 . Two points are worth noting: first, the TC objective is replaced by a lower bound to make optimization feasible BIBREF7 . Second, we impose a sparse connection constraint (each word appears in only one topic) to speed up computation. Open source code implementing CorEx is available on github BIBREF12 .
There is a large body of work on integrating domain knowledge into topic models and other unsupervised latent variable models, often in the form of constraints BIBREF13 , prior distributions BIBREF14 , and token labels BIBREF15 . Like Anchored CorEx, seeded latent dirichlet allocation (SeededLDA) allows the specification of word-topic relationships BIBREF16 . However, SeededLDA assumes a more complex latent structure, in which each topic is a mixture of two distributions, one unseeded and one seeded.
 BIBREF1 first proposed anchors in the context of topic modeling: words that are high precision indicators of underlying topics. In contrast to our approach, anchors are typically selected automatically, constrained to appear in only one topic, and used primarily to aid optimization BIBREF17 . In our information theoretic framework, anchors are specified manually and more loosely defined as words having high mutual information with one or more latent factors. The effects of anchors on the interpretability of traditional topic models are often mixed BIBREF18 , but our experiments suggest that our approach yields more coherent topics.
In health informatics, “anchor” features chosen based on domain knowledge have been used to guide statistical learning BIBREF2 . In BIBREF6 , anchors are used as a source of distant supervision BIBREF19 , BIBREF20 for classifiers in the absence of ground truth labels. While Anchored CorEx can be used for discriminative tasks, it is essentially unsupervised. Recent work by BIBREF21 is perhaps most similar in spirit to ours: they exploit predefined anchors to help learn and impose semantics on a discrete latent factor model with a directed acyclic graph structure. We utilize an information theoretic approach that makes no generative modeling assumptions.
To demonstrate the utility of Anchored CorEx, we run experiments on two document collections: 20 Newsgroups and the i2b2 2008 Obesity Challenge BIBREF22 data set. Both corpora provide ground truth labels for latent classes that may be thought of as topics.
The 20 Newsgroups data set is suitable for a straightforward evaluation of anchored topic models. The latent classes represent mutually exclusive categories, and each document is known to originate from a single category. We find that the correlation structure among the latent classes is less complex than in the Obesity Challenge data. Further, each category tends to exhibit some specialized vocabulary not used extensively in other categories (thus satisfying the anchor assumption from BIBREF1 ).
To prepare the data, we removed headers, footers, and quotes and reduced the vocabulary to the most frequent 20,000 words. Each document was represented as a binary bag-of-words vector. In all experiemnts, we used the standard training/test split. All CorEx models used three layers of 40, 3, and 1 factors. fig:big shows an example hierarchical topic model extracted by Anchored CorEx.
The Obesity Challenge 2008 data set includes 1237 deidentified clinical discharge summaries from the Partners HealthCare Research Patient Data Repository. All summaries have been labeled by clinical experts with obesity and 15 other conditions commonly comorbid with obesity, ranging from Coronary Artery Disease (663 positives) to Depression (247) to Hypertriglyceridemia (62).
We preprocessed each document with a standard biomedical text pipeline that extracts common medical terms and phrases (grouping neighboring words where appropriate) and detecting negation (“not” is prepended to negated terms) BIBREF23 , BIBREF24 . We converted each document to a binary bag-of-words with a vocabulary of 4114 (possibly negated) medical phrases. We used the 60/40 training/test split from the competition.
We are primarily interested in the ability of Anchored CorEx to extract latent topics that are unambiguously associated with the 16 known conditions. We train a series of CorEx models with 32 latent topics in the first layer, each using a different anchor strategy. tab:obesity:topics shows the Obesity and Obstructive Sleep Apnea (OSA) topics for three iterations of Anchored CorEx with the ten most important terms (highest weighted connections to the latent factor) listed for each topic. Unsupervised CorEx (first row) does not discover any topics obviously related to obesity or OSA, so we choose the topics to which the terms obesity and obstructive sleep apnea are assigned. No unambiguous Obesity or OSA topics emerge even as the number of latent factors is decreased or increased.
In the second iteration (second row), we add the common name of each of the 16 diseases as an anchor to one factor (16 total). Adding obesity as an anchor produces a clear Obesity topic, including several medications known to cause weight gain (e.g., acebutolol, klonopin). The anchored OSA topic, however, is quite poor and in fact resembles the rather generic topic to which obstructive sleep apnea is assigned by Unsupervised CorEx. It includes many spurious or non-specific terms like drug.
This is likely due to the fact that obesity is a major risk factor of OSA, and so OSA symptoms are highly correlated with obesity and its other symptoms. Thus, the total correlation objective will attempt to group obesity and OSA-related terms together under a single latent factor. The sparse connection constraint mentioned in sec:methods prevents them from being connected to multiple factors. Indeed, sleep apnea appears in the obesity topic, suggesting the two topics are competing to explain OSA terms.
In the third iteration, we correct this by adding sleep apnea as a second anchor to the OSA topic, and the resulting topic is clearly associated with OSA, including terms related to respiratory problems and medications used to treat (or believed to increase risk for) OSA. There is no noticeable reduction in quality in the Obesity topic.
In a series of follow-up experiments, we investigate the suitability of using anchored CorEx to perform weakly supervised classification. We interpret each anchored latent factor as a classifier for an associated class label and then compute test set F1 (using a threshold of 0.5) and area under the curve (AUC) scores (Obesity Challenge only).
tab:class compares the classification performance of Unsupervised and Anchored CorEx on the soc.religion.christianity category from 20 Newsgroups for different choices of anchors. For both types of CorEx, the topic containing the corresponding terms is used as the classifier, but for Anchored CorEx those terms are also used as anchors when estimating the latent factor. Unsupervised CorEx does a reasonable job of discovering a coherent religion topic that already contains the terms God, Christian, and Jesus. However, using the terms Jesus and Christian as anchors yields a topic that better predicts the actual soc.religion.christianity category.
tab:obesity:class shows the Macro-AUC and F1 scores (averaged across all diseases) on the Obesity Challenge data for the final anchored CorEx model and a Naive Bayes (NB) baseline, in which we train a separate classifier for each disease. Surprisingly, Anchored CorEx outperforms Naive Bayes (NB) by a large margin. Of course, Anchored CorEx is not a replacement for supervised learning: NB beats Anchored CorEx on 20 Newsgroups and does not represent a “strong” baseline for Obesity 2008 (teams scored above 0.7 in Macro-F1 during the competition). It is nonetheless remarkable that Anchored CorEx performs as well as it does given that it is fundamentally unsupervised.
We have introduced a simple information theoretic approach to topic modeling that can leverage domain knowledge specified informally as anchors. Our framework uses a novel combination of CorEx and the information bottleneck. Preliminary results suggest it can extract more precise, interpretable topics through a lightweight interactive process. We next plan to perform further empirical evaluations and to extend the algorithm to handle complex latent structures present in health care data.
This work was partially supported by DARPA award HR0011-15-C-0115. David Kale was supported by the Alfred E. Mann Innovation in Engineering Doctoral Fellowship.

Figure and table captions:
Figure 1. A hierarchical topic model learned by CorEx. Anchored latent factors are labeled in red with anchor words marked with a “*”.
Table 1. Evolution of Obesity and Obstructive Sleep Apnea (OSA) topics as anchors are added. Colors and font weight indicate anchors, spurious terms, and intruder terms from other known topics. Multiword and negated terms are the result of the preprocessing pipeline.
Table 3. Classification performance on Obesity 2008.
Table 2. F1 scores on soc.religion.christianity.

Questions about this paper:
How do they incorporate expert knowledge into their topic model?

Answers:
They use an anchored information theoretic topic modeling using Correlation Explanation and  information bottleneck.