A large-scale distributed web crawler optimized for Amazon cloud services