traject-project/traject — pull request #62 by jrochkind (2014-11-24 18:12:03+00:00)
Title: add batching to SolrJsonWriter

I am not too interested in MRI performance compared to JRuby -- because of the GIL, you are never going to get JRuby-levels of total throughput on MRI.

I am instead interested in if we get get SolrJsonWriter to be as performant as SolrJWriter on JRuby -- then it could be the default on both MRI and JRuby, and we could even get rid of the solrj.jar dependency which would be nice.

So I added batching to SolrJsonWriter. I used stdlib ruby Queue so it would run on MRI too.

It is hard to tell how much it helped, because I get SUCH varying results when running on against a real Solr. I think maybe it improved things, but still not to SolrJ levels. But I'm not entirely sure it improved performance at all.

@billdueber , if you wanted to do any, even messy, perf comparison of SolrJWriter vs SolrJsonWriter on Jruby, it would be helpful to see if you get what i was getting.