added preprocessmapred
prep for non-mr preprocess
#56: added MR and non-MR version of preprocess. for MR version, need to find way to group related lines together. non-MR version reads line by line. TODO: add support for reading from ftp source.
#56: Added sftp support w/ JSch; preprocess step 1 now works correctly (extraction of fields)
updated PreProcess sans MR a bit
revised non-mr PreProcess, now writes 'productId\tuserId score'
#56: added class to remove all unique occurrences of a product (by scanning first token of each line, which is the productId). we assume all products are grouped together (which appears to be true; would be relatively trivial to write a small script to check)
misc updates to comments
#62: Added hashing functionality to convert the first x tokens for each line in a file to a integer. The hashing function is as follows: . The class can be found at hadoop08.hash.HashUnhash, and it takes five parameters: , , , , and .
"#62: Added hashing functionality to convert the first x tokens for each line in a file to a integer. The hashing function is as follows: 'tokens[i].hashCode() + (Character.toString(tokens[i].charAt(0)).hashCode())'. The class can be found at hadoop08.hash.HashUnhash, and it takes five parameters: 'inputPath', 'outputPath', '[hash | unhash]', 'HashMapPath', and 'numTokens'."
Merge branch 'calvin.56.rec-preprocess' of https://github.com/gekonwi/brandeis.big_data into calvin.56.rec-preprocess
#62: Added hashing functionality to convert the first x tokens for each line in a file to a integer. The hashing function is as follows: 'tokens[i].hashCode() + (Character.toString(tokens[i].charAt(0)).hashCode())'. The class can be found at hadoop08.hash.HashUnhash, and it takes five parameters: 'inputPath', 'outputPath', '[hash | unhash]', 'HashMapPath', and 'numTokens'.
added a function to swap first 2 tokens in a comma separated file with 3 tokens per line
misc updates
remove noise
Merge pull request #90 from gekonwi/calvin.56.rec-preprocess

pa4 preprocessing code