A flexible frontier for web crawlers