Question: What model do they propose in the paper?
Evidence: Statistic topic models such as Latent Dirichlet Allocation (LDA) and its variants BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 have been proven to be effective in modeling textual documents. In these models, a word token in a document is assumed to be generated by a hidden mixture model, where the hidden variables are the topic indexes for each word and the topic assignments for words are related to document level topic weights. Due to the effectiveness and efficiency in modeling the document generation process, topic models are widely adopted in quite a lot of real world tasks such as sentiment classification BIBREF5 , social network analysis BIBREF6 , BIBREF5 , and recommendation systems BIBREF7 .
Most topic models take the bag-of-words assumption, in which every document is treated as an unordered set of words and the word tokens in such a document are sampled independently with each other. The bag-of-words assumption brings computational convenience, however, it sacrifices the characterization of sequential properties of words in a document and the topic coherence between words belonging to the same language segment (e.g., sentence). As a result, people have observed many negative examples. Just list one for illustration BIBREF8 : the department chair couches offers and the chair department offers couches have very different topics, although they have exactly the same bag of words.
There have been some works trying to solve the aforementioned problems, although still insufficiently. For example, several sentence level topic models BIBREF9 , BIBREF10 , BIBREF11 tackle the topic coherence problem by assuming all the words in a sentence to share the same topic (i.e., every sentence has only one topic). In addition, they model the sequential information by assuming the transition between sentence topics to be Markovian. However, words within the same sentence are still exchangeable in these models, and thus the bag-of-words assumption still holds within a sentence. For another example, in BIBREF12 , the embedding based neural language model BIBREF13 , BIBREF14 , BIBREF15 and topic model are integrated. They assume the generation of a given word in a sentence to depend on its local context (including its preceding words within a fixed window) as well as the topics of the sentence and document it lies in. However, using a fixed window of preceding words, instead of the whole word stream within a sentence, could only introduce limited sequential dependency. Furthermore, there is no explicit coherence constraints on the word topics and sentence topics, since every word can have its own topics in their model.
We propose Sentence Level Recurrent Topic Model (SLRTM) to tackle the limitations of the aforementioned works. In the new model, we assume the words in the same sentence to share the same topic in order to guarantee topic coherence, and we assume the generation of a word to rely on the whole history in the same sentence in order to fully characterize the sequential dependency. Specifically, for a particular word INLINEFORM0 within a sentence INLINEFORM1 , we assume its generation depends on two factors: the first is the whole set of its historical words in the sentence and the second is the sentence topic, which we regard as a pseudo word and has its own distributed representations. We use Recurrent Neural Network (RNN) BIBREF16 , such as Long Short Term Memory (LSTM) BIBREF17 or Gated Recurrent Unit (GRU) network BIBREF18 , to model such a long term dependency.
With the proposed SLRTM, we can not only model the document generation process more accurately, but also construct new natural sentences that are coherent with a given topic (we call it topic2sentence, similar to image2sentece BIBREF19 ). Topic2sentence has its huge potential for many real world tasks. For example, it can serve as the basis of personalized short text conversation system BIBREF20 , BIBREF21 , in which once we detect that the user is interested in certain topics, we can let these topics speak for themselves using SLRTM to improve the user satisfactory.
We have conducted experiments to compare SLRTM with several strong topic model baselines on two tasks: generative model evaluation (i.e. test set perplexity) and document classification. The results on several benchmark datasets quantitatively demonstrate SLRTM's advantages in modeling documents. We further provide some qualitative results on topic2sentence, the generated sentences for different topics clearly demonstrate the power of SLRTM in topic-sensitive short text conversations.
We propose Sentence Level Recurrent Topic Model (SLRTM) to tackle the limitations of the aforementioned works. In the new model, we assume the words in the same sentence to share the same topic in order to guarantee topic coherence, and we assume the generation of a word to rely on the whole history in the same sentence in order to fully characterize the sequential dependency.
Answer: Sentence Level Recurrent Topic Model (SLRTM).