GitHub repository: rbk/python-crawler
Primary language: Python
Description: Spider / web crawler. 1 million links in 2 hours.