Web crawler with very flexible crawling options.  Can either use standalone or can be used with resque to perform clustered crawls.