knowitall/cluewebextractor — pull request #1 by xdavidjung (2013-02-05 23:06:05+00:00)
Title: Remove long lines, non-English warc entries. 

Forgot to commit these changes last week.

Updates to remove very long lines and more sophisticated language detection so we can remove non-English entries.

Had to make the CLI a bit more complicated (now requires a directory needed by the lang-detect module) but not by much, since I decided to add the profiles directory to the repo itself. It's pretty small.