wgit (rubygems): Wgit was primarily designed to crawl static HTML websites to index and search their content - providing the basis of any search engine; but Wgit is suitable for many application domains including: URL parsing, data mining and statistical analysis. 
Keywords: analytics|data-mining|mongodb|ruby|search-engine|web-crawler|web-scraper