A web crawler that is distributed in the cloud.