extract solr now splits up the huge requests into smaller more manageable ieces, also keeps track of the URL calls made inside metrics.txt