Adding MIT license
Makefile includes wordcounts
new tokenizer script
new metadata parser for RMP data
moving tokenizer to core bookworm module
Clarifying error messages
database now loads wordcounts from huge pickle-created tables instead of a book at a time
wordcounts are now derived from big pickle blobs
adding new function to encode a a pickled set of wordcounts and write out in tabular form for mysql ingest
returning support for file-based format
script to enable conversion of old file types
fix load behavior
documentation
Merge branch 'dev' of ../Presidio
Merge branch 'noLittleFiles' of ../Presidio
Makefile has clean entry
changed all target
Merge branch 'noLittleFiles' of ../Presidio
missing dollar sign
Fixing unigram count writeout in tokenBatches