uwsampa/nltk-hadoop — pull request #2 by zbsimon (2014-11-24 10:22:41+00:00)
Title: cleaned up commit 327f5de for pull request

* mapred_tfidf.py:
* switched argument parsing to argparse.
* if MapReduce will overwrite directories, users are now asked whether they want to continue.
* added '--force' flag to automatically force directory overwrites.
* better error handling.

* map_reduce_utils:
* 'numeric' words such as '14th' are filtered out.

* mappers/reducers:
* added documentation for each script, removed unneeded conditionals, added helper methods for print formatting.
* word_count_red no longer has to re-read each file to see how long it is.
* cos_sim_map now ensures that two documents are sent to the same reducer regardless of which order they arrive in.
* removed old comments that didn't apply any more.