Scalable and customizable web crawler