github.com/dineshsprabu/concurrent-web-crawler (go): Concurrent Web Crawler is a light weight configurable crawler with concurrecy enabled. Configrations like 'MaxConcurrencyLimit' will be helpful for controlled concurrency with limited number of subroutines, 'CrawlDelay' can be helpful to slow down the crawling rate on subroutine level. 'StoragePath' can be useful for specifying stroage location where pages could be saved.  Note: Please be careful while crawling websites and do read the robots.txt before you crawl a website.
Keywords: concurrent-web-crawler|crawler|go-crawler|spider|web-crawler