Adaptation of Topic Model to New Domains Using Recursive Bayes

In real-world applications, the topic model using latent Dirichlet allocation (LDA) should be adaptive to meet the unknown domain knowledge. Compensating such domain mismatch assures a robust document representation at different time stamps. This study presents a recursive Bayes algorithm to implement an adaptive topic model (ATM) in which LDA is evolved to new domains from adaptation data epoch by epoch without waiting for long batch data. By properly characterizing LDA parameters using the conjugate priors, the reproducible prior/posterior distributions are derived for efficient implementation of ATM. In the experiments, ATM continuously captured the topic evolution and consistently improved the document modeling and categorization from new domain data. 1 Introduct ion Latent Dirichlet allocation (LDA) [2] is popular as a mixture topic model for document representation. LDA characterizes the hierarchy of words, topics and documents and generalizes the probabilistic latent semantic analysis based topic model [6] by merging the Dirichlet priors for the mixture topics. LDA is deteriorated when the topics and domains are changed in new documents. A possible solution to this circumstance is to retrain LDA using new documents. Such batch training is time consuming and is not generalized well in ill-posed conditions. A flexible LDA should be adaptive at different time stamps instead of performing batch training. This study presents a sequential model to continuously adapt LDA to meet the changing domains from newly-collected documents. Previously, a dynamic topic model (DTM) [3] was proposed to capture the evolution of topics in a single large document collection. A Gaussian noise model was adopted to chain the topic parameters in a state space model. Also, a continuous-time DTM [8] was developed to improve time resolution by a Brownian motion model, and was adopted for dating documents. These DTMs used the distribution of previous topics to evolve new topics in a single document collection. No Bayesian model adaptation was performed. Considering the limited amount of new documents, which are observed incrementally, the information systems are continuously adapted to new topics and domains. Adapting topic model at different epochs is helpful for enhancing system robustness. The online learning algorithms were exploited for speech recognition [4] and information retrieval [5]. The reproducible Dirichlet densities were formulated to build a recursive Bayes (RB) algorithm. In addition, an online LDA [1] was proposed to incrementally capture the thematic patterns and identify the dynamics of topics in text streams. This paper presents a novel adaptive topic model (ATM) where the incremental adaptation is performed to trace the latent topics over time and simultaneously update the LDA parameters and hyperparameters at different learning epochs. Adopting the conjugate prior using Dirichlet density activates an RB learning, which is employed to fulfill a rapid implementation of ATM. 2 Adaptive topic model 2 . 1 Mode l construc t ion An adaptive topic model is presented to meet new domains at different learning epochs. At the t-th epoch, the documents ) (t w are collected for model adaptation. Figure 1(a) displays the graphical model of ATM. The hyperparameters } , { η ν φ = are used to express the variations of model parameters }. , , { β λ s m = In an RB learning procedure [4][5], the parameters } , , { t t t t s β λ m = at epoch t are recursively estimated by maximizing a posteriori distribution, which is equivalent to the product of a likelihood function of current data ) (t w and a prior density given the hyperparameters } , { 1 1 1 − − − = t t t η ν φ updated at epoch . 1 − t The incremental learning is depicted in Figure 1(b). The data generation is described as follows: 1. For each epoch t with adaptation data ) (t w : Choose the word distributions associated with K topics z by )}. ( Dir ~ { 1 − t k t k η β Choose the mean of the Dirichlet parameters α of topic mixture θ by ). ( Dir ~ 1 − t t ν m 2. For each document ) (t d w and each of N words : ) (t dn w The generation process is the same as that of LDA [2]. The newest domain knowledge can be traced through an incremental learning procedure. The overfitting problem is avoided and the model mismatch is resolved at different time stamps.