Fixed crawler to not get hung up on 404s