basho/yokozuna — pull request #358 by coderoshi (2014-04-09 17:39:45+00:00)
Title: Soften JVM overload failures

After a week of replicating #330, I've concluded that there is little that we can do about an overloaded Solr JVM. What we can do is reduce the flood of failure messages by making some simple adjustments in key configuration files:

### solrconfig.xml
Suggested by http://wiki.apache.org/solr/SolrPerformanceProblems, slow or overly frequent commits can overload the cache warmer. Since all other `autowarmCount`s were set to 0, I followed suit with the `fieldValueCache`.

Since all cache auto warmers were set to zero, I also bumped up with default `maxWarmingSearchers` (suitable for read-only slaves, as mentioned in the config comments) to 4. I found that on high loads of large objects, I'd get an [exceeded limit of maxWarmingSearchers=X](http://wiki.apache.org/solr/FAQ#What_does_.22exceeded_limit_of_maxWarmingSearchers.3DX.22_mean.3F) message, which this resolved.

### default_schema.xml
For objects with many fields, I set the `catch-all field` to false by default, as per #316. This helped tremendously in the reduction of incidental field indexing.

### jetty.xml
I bumped up the `maxIdleTime` from 50 seconds to 60, since that's the default timeout for many of our PB clients. It's also important that this value matches the ibrowse client timeout, explained below.

The `acceptors` I kept at their default, which is 2. Solr recommends that you have 2 acceptors per CPU, so this is something that we should document. A better long term option is to make this a riak.conf, or possibly even automatically set acceptor count by interrogating the system, but I'm unsure this is something we want to tackle for beta.

### yz_solr.erl
Issue #320 was rife with `org.eclipse.jetty.io.EofException`. This is because ibrowse default times out at 30 seconds. Since jetty defaulted to 50 seconds, it was Solr complaining about the unexpected closed socket. This at least stops that complaint.

### ibrowse.conf
I made no changes to ibrowse.conf. Though we could increase the number of pooled and opened sockets, similar to this `{dest, "localhost", 10014, 100, 100000, []}.` it doesn't resolve the underlying problem of a slow and overloaded JVM. It might be a advanced setting in isolated cases, but in all of my tests increasing the connection pool just prolonged the inevitable crash from a JVM that wasn't indexing large objects fast enough.

### riak.conf
Followed many JVM tuning suggestions for larger heaps (http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning), from tweaking pages sizes, settings per heap space, ratios, etc.

```
-d64 -Xms8g -Xmx32g -XX:+UseStringCache -XX:+UseCompressedOops -XX:NewRatio=3 \
-XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m \
-XX:CMSFullGCsBeforeCompaction=1 -XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=70 -XX:CMSTriggerPermRatio=80 \
-XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled \
-XX:+ParallelRefProcEnabled -XX:+UseLargePages -XX:+AggressiveOpts
```

I found some values that allowed me to support slightly larger objects sizes on my test cluster, for my particular test set (large JSON objects, 1-2M with 40-10k indexes of various types, updated 8-25 client threads), but expanding those options to be yokozuna default seems too pedantic.

I analyzed opened sockets in my tests, and when the JVM takes a long time, the connection remains active until the process is complete. If the data coming in is too large and fast, with insufficient JVM resources to manage the data being indexed in a timely manner, you'll see ports in use. The only solution I could see to this problem is ensuring that the JVM runs fast enough that the connections aren't used for more than a few seconds, which depends entirely on Solr being fast.

## Other change

There's also a minor change in here. While making these _yz_solr changes, Dmitri found an error where his key was named for the state of the Oregon, causing the delete query to fail on a reserved word (_yz_rk:OR), so I wrapped it in quotes.

## Documentation

Issue #330 gives us a glimpse of some problems that an ill-fitting JVM can cause. We're going to need to find and share plenty of documentation about how/where a user can got o investigate their resource requirements, and troubleshoot problems: basho/basho_docs#1012.