Crawler cleanup

See individual commits for specific changes.