eastein/chatlight — Issue #6: training tools
Author: eastein
Created: 2011-11-08 05:48:57+00:00
State: open

Optional argument to chatlight: dictionary file, ordered by counted occurences in a specific corpus. In JSON format like so:

[
["hello", 34],
["world", 56]
]

Where the "tuples" are a word and an incidence count in the corpus of text that's considered representative. Zero-incidence words that are considered valid but did not appear in the corpus should be indicated by a record with an incidence of 0, not be dropped.

Given this file and another path to write state to (an absolute filename whose directory already exists to read/write JSON to), record incidence in chat of non-categorized words.

Another utility should exist that takes the same two files and lists non-categorized words experienced in chat, ordered in some hybrid (TBD) of rarity in corpus and commonality in chat.