lintool/Ivory — pull request #24 by ferhanture (2012-03-01 18:21:08+00:00)
Title: Toy data for Hooka/CLIRUtils tests, preprocess driver for plain text coll., ...

-- Prepared a toy dataset for de-en bilingual prob.s, and corresponding tests using these data files in CLIRUtilsTest. I check for the correct probability of two de-en word pairs...

-- Preprocess driver was tested on a plain text collection that Doug sent me last year for deduping. The algo had correctly identified the duplicate pairs, so it seems to be working fine.