Now fully convinced that crawler is behaving properly! Have left all my
debug stuff in for now so that we can convince ourselves again in the
morning. 
Huzzah!
Final thing I'm really worried about it whether we're reading and
writing to file properly at the beginning and end of crawl.