Crawler updated to autocompute crawling schedule