Learning to crawl