proycon/pynlpl — pull request #6 by amitbeka (2014-06-01 11:38:27+00:00)
Title: lm: add support for trie ngrams storage

ARPALanguageModel now supports two modes: simple stores the ngrams
probabilities and backoffs in regular Python dict(), while 'trie' mode
stores the values in marisa_trie.RecordTrie struct, which is more space
efficient (~35% reduction). For using the trie mode, need to install
marisa-trie package.

Signed-off-by: Amit Beka