Issue 1:
Hello,

I've got an application which uses match_phrase queries to look for, and highlight, exact matches within file texts. It worked fantastically on ElasticSearch version 2.0.0 and 2.0.1, but breaks on 2.1.0.

The query appears to run correctly but the highlighter highlights ALL instances of terms in the query instead of just those that appear within matched phrases. For example, if the query contains the word "is", then every instance of the word "is" - throughout the entire document - is highlighted.

Example:

As an example, I've prepared a series of commands in the Sense chrome plugin to replicate the problem:

POST /test_index
{
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings" : {
"file" : {
"properties" : {
"text" : { "type" : "string"}
}
}
}
}

POST /test_index/file
{
"text": "The quick brown fox jumped over the other, lazier, fox."
}

POST /test_index/file
{
"text": "Doc Brown is one quick fox."
}

POST /test_index/file/_search
{
"query": {
"match_phrase": {
"text": "quick brown fox"
}
},
"highlight": {
"fields": {
"text": {
"number_of_fragments": 0
}
}
}
}

Version 2.0.1 results

Under version 2.0.1, the search call returns the following hits array:

"hits": [
{
"_index": "test_index",
"_type": "file",
"_id": "AVF-PCoeysoVw9xNBYa3",
"_score": 0.55737644,
"_source":{
"text": "The quick brown fox jumped over the other, lazier, fox."
},
"highlight": {
"text": [
"The <em>quick</em> <em>brown</em> <em>fox</em> jumped over the other, lazier, fox."
]
}
}
]

This is the expected and desired output.

Version 2.1.0 results

Under version 2.1.0, the search call returns the following hits array:

"hits": [
{
"_index": "test_index",
"_type": "file",
"_id": "AVF-MdXsEcdhEnh7dUcO",
"_score": 0.55737644,
"_source": {
"text": "The quick brown fox jumped over the other, lazier, fox."
},
"highlight": {
"text": [
"The <em>quick</em> <em>brown</em> <em>fox</em> jumped over the other, lazier, <em>fox</em>."
]
}
}
]

This is the same match, and even has the same score, but the word "fox" is incorrectly highlighted when it occurs without the rest of the phrase.

Tested solutions

In attempt to rectify this problem, I attempted two things:

First, on the theory that perhaps it had switched to the postings highlighter rather than the plain highlighter, I attempted to force the use of the plain version with the "type": "plain" flag. Unfortunately, this had no visible effect.

Second, duplicated the original query within a "highlight_query" block. This too, unfortunately, had no visible effect.

Theory

Currently, my theory is that this problem is related to the upgrade to lucene-5.3.0. (#13239)

Specifically, I wonder if it has to do with how NearSpansOrdered are processed. (https://issues.apache.org/jira/browse/LUCENE-6537)

That said, this is purely conjecture. It's quite possible - or even likely - that the problem lies elsewhere.

I'm hoping that an elasticsearch expert might have a better idea.

Thank you.

Issue 2:
Hello,

After upgrading our Production cluster to Version 1.3.5 , the cluster is constantly in red state with :

{

"cluster_name" : "totango_prod_hist",

"status" : "red",

"timed_out" : false,

"number_of_nodes" : 26,

"number_of_data_nodes" : 20,

"active_primary_shards" : 100,

"active_shards" : 200,

"relocating_shards" : 0,

"initializing_shards" : 2,

"unassigned_shards" : 2

}

When i went over the logs i found this exception:

2014-11-19 16:21:01,253][WARN ][index.engine.internal ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]

[2014-11-19 16:21:01,401][WARN ][cluster.action.shard ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@739f95fa]]]

[2014-11-19 16:23:52,294][WARN ][index.engine.internal ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]

[2014-11-19 16:23:52,569][WARN ][cluster.action.shard ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@503839bb]]]

[2014-11-19 16:26:42,678][WARN ][index.engine.internal ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]

[2014-11-19 16:26:42,934][WARN ][cluster.action.shard ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@3674e0a0]]]

[2014-11-19 16:29:32,599][WARN ][index.engine.internal ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]

[2014-11-19 16:29:32,855][WARN ][cluster.action.shard ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@5a27fe79]]]

[2014-11-19 16:32:23,194][WARN ][index.engine.internal ] [elasticsearch-prod-hist08] [2014_11][5] failed engine [refresh failed]

[2014-11-19 16:32:23,442][WARN ][cluster.action.shard ] [elasticsearch-prod-hist08] [2014_11][5] sending failed shard for [2014_11][5], node[fiY35nZBRLOZzQ_AUBOcQA], [P], s[INITIALIZING], indexUUID [Ms7CtS7kRg6pth9YXgPSEg], reason [engine failure, message [refresh failed][IllegalStateException[parentFilter must return FixedBitSet; got org.apache.lucene.search.BitsFilteredDocIdSet@39c4890a]]]

Please advise ?

Thanks,

Costya.