Scalable, extensible web-crawler written in python