Get processed and discovered url lists for crawl job