meta-toolkit/meta — pull request #40 by skystrife (2014-04-28 17:27:13+00:00)
Title: Add sequence labeling library component

This PR adds both a sequence labeling library component (sequences, trellises, analyzers for sequence data, etc) and an implementation of a linear chain conditional random field for supporting sequence labeling tasks such as POS tagging and chunking.

Evaluation work has been done so far only on POS tagging, but the current CRF model (when trained with the default POS tagging analyzer) achieves 97% token accuracy on the traditional testing section of the Penn Treebank's WSJ corpus.

No alternative models are given yet, but the groundwork is there now for writing HMMs, MeMMs, etc.

Once merged, we can make the `ngram_pos_analyzer` use our model for assigning the POS tags to the sentences instead of relying on pre-generated data.

(I don't yet know about licensing issues with releasing our pre-trained model under MIT/NCSA, so I'm not including it for now.)