Recursive crawl and a little bit of error handling related to 404s