apache/spark — pull request #476 by yinxusen (2014-04-22 05:26:32+00:00)
Title: JIRA issue: [SPARK-1405](https://issues.apache.org/jira/browse/SPARK-1405) Gibbs sampling based Latent Dirichlet Allocation (LDA) for MLlib

(This PR is based on a joint work done with @liancheng four months ago.)

## Overview

LDA is a classical topic model in machine learning, that provides the ability to extract topics from corpus. Gibbs sampling (GS for short) is a common way to optimize LDA model.

The LDA model consists of four matrices, two 1-dim matrices:

* Document counts
* Topic counts

plus two 2-dim matrices:

* Document-Topic counts
* Topic-Term counts

## Implementation details

* An accumulator is used to aggregate all updated values and applies them on the old model computed in the last iteration.

* [Chalk](https://github.com/scalanlp/chalk) is used for term segmentation. Though it is easy to rewrite it with Lucene analyzers, I think MLlib should not take the burden to maintain an implementation of tokenizer.

* `SparkContext.wholeTextFiles()` is convenient for offline experimentation, while `SparkContext.textFile()` is better for online applications.

* Document dictionary and term dictionary are broadcasted to translate document names and terms into `Int` IDs.

* Topic assignment matrix from the last iteration is cached for the current iteration, and then unpersisted to release memory.

* LDA suffers similar stack overflow problem of MLlib ALS ([SPARK-1006](https://spark-project.atlassian.net/browse/SPARK-1006)). To workaround this issue, we checkpoint every a few iterations.