solr: Pull in lucene intervals change

This contains the core changes needed for intervals, stripping off any maven build additions
originally at https://github.com/bloomberg/lucene-solr/commit/423ef0a20ef652eccf49148eb30e5a3f201df626
Further fixes to the upstream flaxsearch/lucene-solr-intervals have been folded in as well, apart
from fixes needed for migration from 4.8.1 to 4.10.3
solr: DebugComponent debug.track output tweaks (SimpleOrderedMap instead of NamedList, shardAddress instead of shard, numeric values as such instead of string)

Summary:
For our use only, not for JIRA.

before:

{"responseHeader":{"status":0,"QTime":11},"response":{"numFound":0,"start":0,"maxScore":0.0,"docs":[]},"debug":{"track":{"rid":"...","EXECUTE_QUERY":{"http://url1/|http://url2/":{"QTime":"1","ElapsedTime":"8","RequestPurpose":"GET_TOP_IDS","NumFound":"0","Response":"..."},"http://url3/|http://url4/":{"QTime":"2","ElapsedTime":"8","RequestPurpose":"GET_TOP_IDS","NumFound":"0","Response":"..."}}}}}

after:

{"responseHeader":{"status":0,"QTime":11},"response":{"numFound":0,"start":0,"maxScore":0.0,"docs":[]},"debug":{"track":{"rid":"...","EXECUTE_QUERY":[{"shardAddress":"http://url3","QTime":1,"ElapsedTime":9,"RequestPurpose":"GET_TOP_IDS","NumFound":0,"Response":"..."},{"shardAddress":"http://url2","QTime":0,"ElapsedTime":9,"RequestPurpose":"GET_TOP_IDS","NumFound":0,"Response":"..."}]}}}

Test Plan: -

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D79578
SOLR-5734 follow-on: make RTimer use System.nanoTime instead of System.currentTimeMillis

Summary: -

Test Plan: -

Reviewers: raiyenga

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D81500
solr: timing-debug to include lucene-search-time (for both ungrouped and grouped searches)

Summary:
Example input/output:

curl "http://localhost:8787/solr/collection1/select?rows=0&wt=json&debug=timing&q=language_id:1"

{"responseHeader":{"status":0,"QTime":69},"response":{"numFound":26,"start":0,"maxScore":0.9354614,"docs":[]},"debug":{"timing":{"time":48.0,"prepare":{"time":0.0,"query":{"time":0.0},"stats":{"time":0.0},"debug":{"time":0.0}},"process":{"time":48.0,"query":{"time":48.0,"LuceneSearchTime":10.0},"stats":{"time":0.0},"debug":{"time":0.0}}}}}

Test Plan: -

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D81484
solr: add distributedProcess, take, handle, finish sub-stage timers.

Summary:
example output

before:
`{"responseHeader":{"status":0,"QTime":58},"response":{"numFound":627,"start":0,"maxScore":1.0,"docs":[{"suid":"N5Q1JKBE5TS0"}]},`
`"stageTiming":{"time":57.0,`
`"stage1000":{"time":0.0},`
`"stage2000":{"time":47.0},`
`"stage3000":{"time":10.0}}}`

after:
`{"responseHeader":{"status":0,"QTime":62},"response":{"numFound":1275,"start":0,"maxScore":1.0,"docs":[{"suid":"KO0YMS19KNB5"}]},`
`"stageTiming":{"time":60.0,`
`"stage1000":{"time":0.0,"distributedProcess":{"time":0.0},"finish":{"time":0.0}},`
`"stage2000":{"time":51.0,"distributedProcess":{"time":0.0},"take":{"time":46.0},"handle":{"time":0.0},"finish":{"time":0.0}},`
`"stage3000":{"time":9.0,"distributedProcess":{"time":0.0},"take":{"time":9.0},"handle":{"time":0.0},"finish":{"time":0.0}}}}`

Test Plan: -

Reviewers: raiyenga, mjustice, dcollins

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D88065
solr: debug=timing to include metrics surrounding elapsedTime

Summary:
example output

before:
`{"responseHeader":{"status":0,"QTime":64},"response":{"numFound":627,"start":0,"maxScore":1.0,"docs":[{"suid":"N5Q1JKBE5TS0"}]},"debug":{`
`"timing":{"time":33.0,"prepare":{"time":4.0,"query":{"time":3.0},"stats":{"time":0.0},"debug":{"time":0.0}},"process":{"time":29.0,"query":{"time":29.0,"LuceneSearchTime":0.0},"stats":{"time":0.0},"debug":{"time":0.0}}},"submitWaitingTimeSum":3}}`

after:
`{"responseHeader":{"status":0,"QTime":30},"response":{"numFound":627,"start":0,"maxScore":1.0,"docs":[{"suid":"N5PN873M0C8W"}]},"debug":{`
`"distribTiming":{`
`"EXECUTE_QUERY":{"submitWaiting":{"count":4,"sum":3,"min":0,"max":0},"elapsed":{"count":4,"sum":60,"min":15,"max":15},"takeWaiting":{"count":4,"sum":0,"min":0,"max":0}},`
`"GET_FIELDS":{"submitWaiting":{"count":1,"sum":0,"min":0,"max":0},"elapsed":{"count":1,"sum":9,"min":9,"max":9},"takeWaiting":{"count":1,"sum":0,"min":0,"max":0}}},`
`"timing":{"time":12.0,`
`"prepare":{"time":3.0,"query":{"time":3.0},"stats":{"time":0.0},"debug":{"time":0.0}},`
`"process":{"time":9.0,"query":{"time":8.0,"LuceneSearchTime":0.0},"stats":{"time":1.0},"debug":{"time":0.0}}}}}`

Test Plan: -

Reviewers: raiyenga, mjustice, dcollins

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D88641
solr: distribTiming to include debug timing breakdown

Summary:
This is so that min/max/count/sum stats for LuceneSearchTime per-stage are available.

Example output:

  ...
{"responseHeader":{"status":0,"QTime":1541},
"grouped":{ ... },
"debug":{"distribTiming":{

  "TOP_GROUPS":{
"submitWaiting":{"count":4,"sum":2,"min":0,"max":0},
"elapsed":{"count":4,"sum":5353,"min":1265,"max":1388},
"takeWaiting":{"count":4,"sum":0,"min":0,"max":0},
"breakdown":{
  "timing":{"time":{"count":4,"sum":5239,"min":1222,"max":1373},
"prepare":{"time":{"count":4,"sum":20,"min":1,"max":7},"query":{"time":{"count":4,"sum":20,"min":1,"max":7}},"stats":{"time":{"count":4,"sum":0,"min":0,"max":0}},"debug":{"time":{"count":4,"sum":0,"min":0,"max":0}}},
"process":{"time":{"count":4,"sum":5218,"min":1215,"max":1372},"query":{"time":{"count":4,"sum":5218,"min":1215,"max":1372},"LuceneSearchTime":{"count":4,"sum":5128,"min":1195,"max":1350}},"stats":{"time":{"count":4,"sum":0,"min":0,"max":0}},"debug":{"time":{"count":4,"sum":0,"min":0,"max":0}}}}}},

  "EXECUTE_QUERY":{
"submitWaiting":{"count":2,"sum":0,"min":0,"max":0},
"elapsed":{"count":2,"sum":128,"min":56,"max":72},
"takeWaiting":{"count":2,"sum":0,"min":0,"max":0},
"breakdown":{
  "timing":{"time":{"count":2,"sum":103,"min":43,"max":60},
"prepare":{"time":{"count":2,"sum":1,"min":0,"max":1},"query":{"time":{"count":2,"sum":1,"min":0,"max":1}},"stats":{"time":{"count":2,"sum":0,"min":0,"max":0}},"debug":{"time":{"count":2,"sum":0,"min":0,"max":0}}},
"process":{"time":{"count":2,"sum":102,"min":43,"max":59},"query":{"time":{"count":2,"sum":102,"min":43,"max":59},"LuceneSearchTime":{"count":2,"sum":12,"min":6,"max":6}},"stats":{"time":{"count":2,"sum":0,"min":0,"max":0}},"debug":{"time":{"count":2,"sum":0,"min":0,"max":0}}}}}},

  "GET_FIELDS":{
"submitWaiting":{"count":1,"sum":0,"min":0,"max":0},
"elapsed":{"count":1,"sum":41,"min":41,"max":41},
"takeWaiting":{"count":1,"sum":0,"min":0,"max":0},
"breakdown":{
  "timing":{"time":{"count":1,"sum":19,"min":19,"max":19},
"prepare":{"time":{"count":1,"sum":1,"min":1,"max":1},"query":{"time":{"count":1,"sum":1,"min":1,"max":1}},"stats":{"time":{"count":1,"sum":0,"min":0,"max":0}},"debug":{"time":{"count":1,"sum":0,"min":0,"max":0}}},
"process":{"time":{"count":1,"sum":18,"min":18,"max":18},"query":{"time":{"count":1,"sum":18,"min":18,"max":18}},"stats":{"time":{"count":1,"sum":0,"min":0,"max":0}},"debug":{"time":{"count":1,"sum":0,"min":0,"max":0}}}}}}},

  "timing":{"time":5361.0,
"prepare":{"time":22.0,"query":{"time":22.0},"stats":{"time":0.0},"debug":{"time":0.0}},
"process":{"time":5338.0,"query":{"time":5338.0,"LuceneSearchTime":5140.0},"stats":{"time":0.0},"debug":{"time":0.0}}}}}

Test Plan: -

Reviewers: raiyenga

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D95618
solr: distribTiming to include debug timing breakdown (shards.tolerant=true behaviour)

Summary: D95618 added the distribTiming info, for that addition choosing same solution as SOLR-6175 patch.

Test Plan: -

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D97162
solr: distribTiming timing metrics to optionally include raw values

Summary:
D95618 added distribTiming breakdown itself.

sample output before:

  "elapsed":{"count":4,"sum":599,"min":141,"max":161}

sample output after:

  "elapsed":{"rawValues":[
{"key":"http://nsrchdevnj2:15255/solr/collection1_shard4_replica1/","val":147},
{"key":"http://nsrchdevnj1:15255/solr/collection1_shard3_replica1/","val":144},
{"key":"http://nsrchdevnj2:15250/solr/collection1_shard2_replica1/","val":163},
{"key":"http://nsrchdevnj1:15250/solr/collection1_shard1_replica1","val":213}],
"count":4,"sum":667,"min":144,"max":213}

Test Plan: -

Reviewers: raiyenga

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D99774
Differential Revision: https://all.phab.dev.bloomberg.com/D101206
SOLR-5730: EarlyTerminatingSortingCollector support for chronologically sorted searches

Summary:
 - Merging bbsolr-4.6.1-news EarlyTerminatingSortingCollector support in
 - Changes from Sorter to Sort in various places (bbsolr-4.8.0-news includes LUCENE-5493 Sorter/Sort rename)
 - Tweak SegmentInfo.toString (used for the 'Registered new searcher' trace)
 - Handle removal of mergeSortKey from schema.xml more robustly
   (don't fail requests which continue to send segmentTerminateEarly=true parameter).
 - Adds support for group=true searches, both distributed (CommandHandler.java) and non-distributed (Grouping.java) code paths.

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D78324
Differential Revision: https://all.phab.dev.bloomberg.com/D80718
Differential Revision: https://all.phab.dev.bloomberg.com/D81196
Differential Revision: https://all.phab.dev.bloomberg.com/D80724
lucene: make genLookaheadCountLimit configurable, add genLookbackCountLimit parameter

Summary: DRQS 53627004

Test Plan: -

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D91140
Differential Revision: https://all.phab.dev.bloomberg.com/D91812
solr: make Lucene's genLook(ahead|back)CountLimit configurable

Summary: DRQS 53627004, D91140 had the correspondingly required lucene change.

Test Plan: -

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D91776
TREQ 2799103: solr: Start RTimer for SearchHandler from right when the request starts

Summary: Snapshot start time as soon as we get control from Jetty, to get a more accurate measure.

Test Plan: Tested on Dev

Reviewers: cpoersch, dcollins, mjustice

Reviewed By: cpoersch

Differential Revision: https://all.phab.dev.bloomberg.com/D97800
solr: add admin/collections?action=DELETEREPLICA&core=... support, make collection=... and shard=... parameters optional

Summary: -

Test Plan: -

Reviewers: raiyenga, dcollins, mjustice

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D99184
Return TermQuery in case TermsQuery resolve down to one term

Summary: Generally TermsQuery will end up as ORed BooleanQuery of TermQueries. It would be efficient to result in a TermQuery if there is only one valid term after analyser chain is applied. This would also avoid the necessity to use TermQuery especially if the caller is not sure whether the analyser would break the terms further down, thus fixing the issue of 'TermQuery already set'.

Test Plan: Unit tested. Unit test files are on the way.

Reviewers: cpoersch, dcollins

Reviewed By: cpoersch

CC: mjustice, raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D105430
Differential Revision: https://all.phab.dev.bloomberg.com/D106308
Differential Revision: https://all.phab.dev.bloomberg.com/D106654
solr: admin/collections?action=DELETEREPLICA&deleteInstanceDir=(false|true) support

Summary: If the new parameter is not supplied existing behaviour will prevail, if deleteInstanceDir=true is supplied the instance directory of the replica will be deleted (via the admin/cores?action=UNLOAD request that delete-replica initiates).

Test Plan: Not yet tested, our use case is during primary build cloning where delete-replica (as opposed to core-unload) should reclaim diskspace.

Reviewers: raiyenga

Reviewed By: raiyenga

CC: dcollins, mjustice, ksrivisa

Differential Revision: https://all.phab.dev.bloomberg.com/D112844
solr: Log leadership election joins and watches

Summary: Makes it easier to debug leadership transfer issues

Test Plan: Staged rollout

Reviewers: cpoersch

Reviewed By: cpoersch

Differential Revision: https://all.phab.dev.bloomberg.com/D113760
solr: select?replicaAffinity=(node|host) support

Summary:
select?nodeAffinity=(false|true) support added by D107516 is retained but deprecated.

Illustration: `4-hosts-x-2-ports=8-instances 8-shards 2-replica system`

  http://host1:port1/solr/collection1_shard1_replicaA/
  http://host1:port1/solr/collection1_shard3_replicaA/

  http://host1:port2/solr/collection1_shard5_replicaA/
  http://host1:port2/solr/collection1_shard7_replicaA/

  http://host2:port1/solr/collection1_shard2_replicaA/
  http://host2:port1/solr/collection1_shard4_replicaA/

  http://host2:port2/solr/collection1_shard6_replicaA/
  http://host2:port2/solr/collection1_shard8_replicaA/

  http://host3:port1/solr/collection1_shard1_replicaB/
  http://host3:port1/solr/collection1_shard3_replicaB/

  http://host3:port2/solr/collection1_shard5_replicaB/
  http://host3:port2/solr/collection1_shard7_replicaB/

  http://host4:port1/solr/collection1_shard2_replicaB/
  http://host4:port1/solr/collection1_shard4_replicaB/

  http://host4:port2/solr/collection1_shard6_replicaB/
  http://host4:port2/solr/collection1_shard8_replicaB/

`.../select` plain will route sub-requests to a random selection of solr cores and so could potentially use all 8 JVM instances

  http://host1:port1/solr/collection1_shard1_replicaA/
  http://host4:port1/solr/collection1_shard2_replicaB/
  http://host3:port1/solr/collection1_shard3_replicaB/
  http://host2:port1/solr/collection1_shard4_replicaA/
  http://host1:port2/solr/collection1_shard5_replicaA/
  http://host4:port2/solr/collection1_shard6_replicaB/
  http://host3:port2/solr/collection1_shard7_replicaB/
  http://host2:port2/solr/collection1_shard8_replicaA/

`.../select?replicaAffinity=node` will route sub-requests to a random selection of solr cores whilst maintaining node affinity i.e. sub-requests that can go to the same solr instance will go to the same solr instance e.g.

  http://host1:port1/solr/collection1_shard1_replicaA/
  http://host4:port1/solr/collection1_shard2_replicaB/
  http://host1:port1/solr/collection1_shard3_replicaA/
  http://host4:port1/solr/collection1_shard4_replicaB/
  http://host3:port2/solr/collection1_shard5_replicaB/
  http://host2:port2/solr/collection1_shard6_replicaA/
  http://host3:port2/solr/collection1_shard7_replicaB/
  http://host2:port2/solr/collection1_shard8_replicaA/

`.../select?replicaAffinity=host` will route sub-requests to a random selection of solr cores whilst maintaining host affinity i.e. sub-requests that can go to the same host machine will go to the same host machine e.g.

  http://host1:port1/solr/collection1_shard1_replicaA/
  http://host2:port1/solr/collection1_shard2_replicaA/
  http://host1:port1/solr/collection1_shard3_replicaA/
  http://host2:port1/solr/collection1_shard4_replicaA/
  http://host1:port2/solr/collection1_shard5_replicaA/
  http://host2:port2/solr/collection1_shard6_replicaA/
  http://host1:port2/solr/collection1_shard7_replicaA/
  http://host2:port2/solr/collection1_shard8_replicaA/

`.../select?replicaAffinity=host&replicaAffinity=node` will route sub-requests to a random selection of solr cores whilst maintaining first host affinity and secondly node affinity (the latter clearly only applies if multiple JVMs on a given machine contain the same shard).

Test Plan: not yet tested

Reviewers: raiyenga, dcollins, mjustice, ksrivisa

Reviewed By: raiyenga

Differential Revision: https://all.phab.dev.bloomberg.com/D112412
Differential Revision: https://all.phab.dev.bloomberg.com/D116114
Differential Revision: https://all.phab.dev.bloomberg.com/D153072
solr: SOLR-6359: Customize number of logs and records to keep with UpdateLog

Summary:
See https://issues.apache.org/jira/browse/SOLR-6359
Cherry-picking from Github and putting in review for inclusion locally,
as it hasn't been picked up upstream as yet.

Test Plan: Unit tests pass, staged rollout

Reviewers: cpoersch, mjustice, dcollins, ksrivisa

Reviewed By: dcollins

Differential Revision: https://all.phab.dev.bloomberg.com/D113536
Added GenericTextQuery for keyword searches. Terms Query/Filter made efficient for single term and fault tolerant if the query resulted in empty term.

Summary:
1. GenericTextQuery can be used for keyword phrases with single term or phrases. This will result in
  a. MatchAllDocs Query for zero terms
  b. Term Query for single term and
  c. PhraseQuery for multiple terms.
2. Term Filter/Query is used for single term searches which will fail if the term value does not resolve to just one term.
3. Terms Query/Filter accepts any number of terms
and will result in
  a. MatchAllDocs Query/Filter if zero terms
  b. Term Query/Filter if single term and
  c. Boolean Query or TermsFilter if multiple terms found.

Test Plan: to be again unit tested

Reviewers: cpoersch, dcollins

Reviewed By: dcollins

CC: dcollins, raiyenga, mjustice

Differential Revision: https://all.phab.dev.bloomberg.com/D106424