heritrix output processing, nearly done