The crawler is completely separated from the parser, the robots is closing in on completion, and we can reap unused threads now!