move threshold to TF-IDF reduce step instead of csv writing. makes writing go faster.
make number of documents in corpus configurable in the job script
fix massive issue with csv generation (not outputting the correct idf valuesgit status), and revert back to threshold in csv generation because we want real tf-idf values for all documents