aaronj1335/cs388-final-project — pull request #8 by calvinmm (2014-04-25 19:26:00+00:00)
Title: Preprocessor Implemented

The preprocessor has been implemented.
It utilizes the `multiprocessing` library to hopefully speed up performance when dealing with the huge corpus. The CLI works as follows:

`Usage: python preprocess.py [input_dir] [output_dir] [num_files]`

The first argument is the input directory containing the `.pos` files (it will search recursively within this). The second argument is the output directory, where the temporary files and the final merged files have been created. The last argument is for the merging, it specifies how many merged files will be created.

Here is an example of it running:
```
$ python util/preprocess.py data/penn-treebank-wsj data/convert 4
$ wc -l data/convert/*.pos
277169 data/convert/section_0.pos
276935 data/convert/section_1.pos
277313 data/convert/section_2.pos
275948 data/convert/section_3.pos
1107365 total
```

#5