GitHub repository: stewartmckee/cobweb
Primary language: JavaScript
Description: Web crawler with very flexible crawling options.  Can either use standalone or can be used with resque to perform clustered crawls.