bixo/bixo — pull request #51 by pferrel (2012-06-01 17:06:45+00:00)
Title: created a NotSoSimpleCrawlTool

This is the version of the SimpleCralwTool I use. It has the option to overwrite a previous crawl, outputs mahout sequence files in a directory parallel to "parse", called "mahout". It uses boilerpipe optionally and implements a nutch style regex filter file.

All changes are optional and I think are a proper superset of SimpleCrawlTool so given the same options as SimpleCrawlTool it should produce the same results.

I am using the crawl tool often and so will update with fixes but I also have some utility classes in the project that will be of little interest to bixo. Sorry, I'll move them out eventually. I think you can ignore them, there should be no cross dependencies.