Add ability to stop the crawl (from gnapse)

https://github.com/gnapse/anemone/commit/268271ca3d9e7c56951ab4cc1c80ddfd1ec937fc
Add ability to see whether crawl was stopped early