scale to 2 workers if necessary, better error handling on 404, fix for last crawled not properly storing