Issue 1:
Summary

There seems to be an unexplained inconsistency between the results of _percolate and _search when using the same document and query set.

Description

I have an index defined with a custom analyzer set as my default and a few string fields. The analyzer tokenizes on commas, and nothing else. Curling the mapping returns the following output:

{
"news_documents_20140729": {
"mappings": {
"0000000022-nid": {
"_all": {
"enabled": false
},
"_timestamp": {
"enabled": true,
"store": true
},
"_ttl": {
"enabled": true
},
"dynamic": "false",
"analyzer": "csv_lowercase_commas",
"properties": {
...
"subjects": {
"type": "string"
},
"product": {
"type": "string",
"null_value": "DJGM"
},
...

The analyzer referenced above is defined as follows:

"csv_lowercase_commas" : {
"type" : "custom",
"filter" : [ "lowercase" ],
"tokenizer" : "commas"
}

And the commas tokenizer is defined as follows:

"commas" : {
"pattern" : ",+",
"type" : "pattern"
}

I've indexed a very simple document to the above index, as the shown type..

{ "subjects":"N/DJMT", "product":"DJGM"}

I search for the above document using the following three queries, and the document is returned each time, as expected.

Match All Query

{"query": { "match_all":{}}}

Term Query

{
"query": {
"filtered": {
"filter": {
"and": [
{
"term": {
"product": "djgm"
}
},
{
"in": {
"subjects": [
"n/djmt"
]
}
}
]
}
}
}
}

Phrase Query

{
"query": {
"filtered": {
"filter": {
"term": {
"product": "djgm"
}
},
"query": {
"match_phrase" : {
"subjects" : "n/djmt"
}
}
}
}
}

Here's the problem

I add each of the above queries to the .percolator type in the above described index, and then percolate {"doc":{ "subjects":"N/DJMT", "product":"DJGM"}} -- i.e., the exact same document content I indexed and searched on previously. However, the result contains only the Match All Query and the Phrase Query.

To be more concise...

Searching for the document using the above queries returns the document independent of which query is used to perform the search. Percolating the same document, with the above three queries indexed in the same index's .percolator type, returns only the first two queries.

The phrase query is inexplicably missing from the percolate results.

More Information

This was verified on a 1.1.0 instance, within a clean index. It was originally discovered in a production index through a slightly more complicated, albeit technically synonymous (by my assessment), example.

Sorry, I don't have access to a 1.3 cluster at the moment, so I haven't verified it there.

Also verified on a 1.3.4 test cluster I just spooled up for testing.

A quick grep through the issue tracker using percolate is:open label:bug didn't show anything similar.

(Naive) Partially-substantiated Insights

Running the same experiment, except removing the '/' in the string 'n/djmt' in both all of the queries and the document, produces the expected, correct results. In other words, _percolate returns all three queries, and _search returns the document in response to all three queries. (Did this on a 1.3.4 cluster with the same mappings as described above.)

The combination of (1) the punctuation being a variable in the problem and (2) the fact that analysis is applied to phrase queries and not to term queries, has me believing the issue has to do with analysis, specifically it not being applied properly when percolating.

Lastly, if I'm missing something trivial, then I apologize in advance for wasting time. It's very likely I'm ignorant of some crucial fact here.

Issue 2:
Recently, we moved from version 0.19.11 to last stable 0.90.0 and found one very strange behavior that looks like an issue.

If on the item root level we have specified our custom index_analyzer, search_analyzer (or just analyzer), then index_analyzer works well, but not the search_analyzer. Also, fif we update existing mapping with explicitly specifying search_analyzer on the field level, then it still doesn't seem to work and ES uses standard one.

To reproduce:

Create a new index and define our custom analyzer de_stem:

curl -XPUT 'http://localhost:9200/issue/' -d '{"index": {"number_of_shards": 1,"analysis": {"filter": {"de_snowball": {"type": "snowball","language": "German"}},"analyzer": {"de_stem": {"type": "custom","tokenizer": "standard","filter": ["lowercase", "de_snowball"]}}}}},"number_of_replicas": 0}}'

Put mapping with specified index_analyzer and search_analyzer :

curl -XPUT 'http://localhost:9200/issue/item/_mapping' -d '{"item": {"index_analyzer" : "de_stem","search_analyzer" : "de_stem","properties": {"content": {"dynamic": false,"properties": {"body": {"type": "string"}}}}}}}'

Try search_analyzer for the field content.body with Analyze API

curl -XGET 'localhost:9200/issue/_analyze?pretty=true&field=content.body' -d 'Apple'

Actual result:

{
"tokens" : [ {
"token" : "apple",
"start_offset" : 0,
"end_offset" : 5,
"type" : "<ALPHANUM>",
"position" : 1
} ]
}

Expected result:

{
"tokens" : [ {
"token" : "appl",
"start_offset" : 0,
"end_offset" : 5,
"type" : "<ALPHANUM>",
"position" : 1
} ]
}

The right (expected) result still possible to get, but with explicitly specified search_analyzer:

curl -XGET 'localhost:9200/issue/_analyze?pretty=true&field=content.body&analyzer=de_stem' -d 'Apple'

Index Analyzer is set well

As we see above, search_analyzer seems wasn't set, but index_analyzer works well.

Let's index a document:

curl -PUT 'http://localhost:9200/issue/item/1' -d '{"content" : {"body": "10 Things We Hate About Apple"}}'

If index_analyzer was set well to de_stem the word Apple should be indexed as appl, but not apple (as standard analyzer does).

Let's search for appl first:

curl -XGET 'http://localhost:9200/issue/_search?search_type=count&pretty=true' -d '{"query":{"query_string":{"fields":["content.body"],"query":"appl"}}}'

It works! We get back 1 result:

"hits" : {
"total" : 1,
"max_score" : 0.0,
"hits" : [ ]
}

For the word apple, as expected, it doesn't work since search_analyzer is standard, but index_analyzer is de_stem (so, actual search term will stay apple, but indexed is appl):

curl -XGET 'http://localhost:9200/issue/_search?search_type=count&pretty=true' -d '{"query":{"query_string":{"fields":["content.body"],"query":"apple"}}}'

"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}

Specifying search analyzer with Put Mapping API doesn't help

Ok, i try to update mapping and specify explicitly search analyzer for content.body field on the existing index we created above:

curl -PUT 'http://localhost:9200/issue/item/_mapping' -d '{"item": {"properties": {"content": {"dynamic": false,"properties": {"body": {"type": "string", "search_analyzer": "de_stem"}}}}}}'

Response is ok, but the all problems described above stay the same. So, it seems the search_analyzer for the field content.body is still standard.