I feel like this should be tracked by the queue so that the state could be distributed between crawl nodes, but I'll have a think about it. :)