include urlsWritten in checkpoint, and be backward compatible with old checkpoints
avoid deadlock in case of BeansException at crawl startup
avoid infinite loop spitting out stack traces on BeansException on startup