Minor changes to the crawler (robustness, etc)