Universal Crawler for web or local file system crawlering. The module enables you to extend the functionality of any location-related searches in really easy way -  by overriding the appropriate default functions in the subclass. For example you can change default web filters or add some statistics. In "thread-version" branch you can also find multi-threaded version. In "icm" branch you cand find extended module for collecting statistics from logs of Apache Hadoop jobs' tasks. Let me know if you will use my code, have some suggestions or just find this helpfull :)