make all now produces a bookworm
Makefile includes wordcounts
new tokenizer script
new metadata parser for RMP data
moving tokenizer to core bookworm module
Clarifying error messages
database now loads wordcounts from huge pickle-created tables instead of a book at a time
wordcounts are now derived from big pickle blobs
adding new function to encode a a pickled set of wordcounts and write out in tabular form for mysql ingest
returning support for file-based format
script to enable conversion of old file types
fix load behavior
documentation
Merge branch 'dev' of ../Presidio
Merge branch 'noLittleFiles' of ../Presidio
Makefile has clean entry
changed all target
Merge branch 'noLittleFiles' of ../Presidio
missing dollar sign
Fixing unigram count writeout in tokenBatches