Indexer changes to reduce memory footprint and improve performance

These include:

  * Store batches of documents on the filesystem before sending to Solr

  * Don't send both 'fullrecord' and 'json'.  Use a Solr copyField to derive
    one from the other.

  * Convert data cleanup functions to use in-place string functions.

  * Use multiple indexing threads
Fix up for 'did_something' check in indexer
Remove unused hash variable
Merge pull request #589 from hudmol/indexer-performance

Indexer changes to reduce memory footprint and improve performance