elasticsearch/elasticsearch — Issue #1401: Synonym filter not working correctly for multi-term synonyms.
Author: lukas-vlcek
Created: 2011-10-17 13:39:13+00:00
State: open
Labels: bug

I was trying to get my head around new multi term synonyms (LUCENE-3233) but it does not seem to work correctly based on my expectations. It seems that both synonym forms **[a b,c]** and **[a b => c]** suffer from specific issues. Both cases are discussed in detail below.

# Type of synonym rule: [a b => c]
Let's start with script that defines one simple multi-term synonym: "application server => eap". So whenever `application server` multi-term is found it is replaced with `eap`:

cat /config/analysis/synonym.txt
application server => eap

Now, let's use the following repro script:

curl -X DELETE 'localhost:9200/test/'
curl -X PUT 'localhost:9200/test/' -d '
{
"settings" : {
"index" : {
"number_of_shards" : 1,
"number_of_replicas" : 0,
"analysis" : {
"analyzer" : {
"ta" : {
"type" : "custom",
"tokenizer" : "standard",
"filter" : [ "standard", "word_delimiter", "lowercase", "synonym", "stop", "kstem" ]
}
},
"filter" : {
"synonym" : {
"type" : "synonym",
"synonyms_path" : "analysis/synonym.txt"
}
}
}
}
}
}'

curl -X PUT 'localhost:9200/test/test/_mapping' -d '
{
"test" : {
"properties" : {
"plain" : { "type" : "string", "store" : "yes", "term_vector" : "no", "analyzer" : "ta" },
"fvh" : { "type" : "string", "store" : "yes", "term_vector" : "with_positions_offsets", "analyzer" : "ta" }
},
"_all" : { "analyzer" : "ta" }
}
}'

curl -X GET 'localhost:9200/_cluster/health?wait_for_status=green&timeout=5s'

curl -X POST 'localhost:9200/test/test' -d '{"plain":"Hello EAP World!","fvh":"Hello EAP World!"}'
curl -X POST 'localhost:9200/test/test' -d '{"plain":"This is one really nice application server.","fvh":"This is one really nice application server."}'
curl -X POST 'localhost:9200/test/test' -d '{"plain":"Tomcat server is down.","fvh":"Tomcat server is down."}'
curl -X POST 'localhost:9200/test/test' -d '{"plain":"Application of drugs.","fvh":"Application of drugs."}'
curl -X POST 'localhost:9200/test/test' -d '{"plain":"This is ApplicationServer and EAP world.","fvh":"This is ApplicationServer and EAP world."}'

curl -X POST 'localhost:9200/test/_refresh'

echo "======================="
echo "application server"
curl 'localhost:9200/test/_search?pretty=1' -d '
{
"query":{"query_string":{"query":"application server"}},
"fields" : [ "plain","fvh"],
"highlight" : { "fields" : { "plain" : {}, "fvh" : { "number_of_fragments":0 }}}
}'

echo "======================="
echo "eap"
curl 'localhost:9200/test/_search?pretty=1' -d '
{
"query":{"query_string":{"query":"eap"}},
"fields" : ["plain","fvh"],
"highlight" : { "fields" : { "plain" : {}, "fvh" : { "number_of_fragments":0 }}}
}'

echo "======================="
echo "server"
curl 'localhost:9200/test/_search?pretty=1' -d '
{
"query":{"query_string":{"query":"server"}},
"fields" : ["plain","fvh"],
"highlight" : { "fields" : { "plain" : {}, "fvh" : { "number_of_fragments":0 }}}
}'

## Problematic Queries

### Query: application server
No documents containing `application server` match. Instead I get only documents matching individual terms `application` or `server`. Analyzer itself works correctly, see:
curl 'localhost:9200/test/_analyze?pretty=1&text=application+server&analyzer=ta'

Can the problem be in query parsing then?

### Query: eap
I am getting incorrect highlighting for two documents.

Original text: `This is ApplicationServer and EAP world.`
Highlighted text: `This is ApplicationServer and EAP world.`

Original text: `This is one really nice application server.`
Highlighted text: `This is one really nice application server.`

**Problem:** In both cases the word `server` should have been highlighted as well.

# Type of synonym rule: [a b, c]

Let's take the same script except modify definition of `synonym` analysis.

....
"filter" : {
"synonym" : {
"type" : "synonym",
"synonyms" : [ "application server, eap" ]
}
}
....
Or change content of the `/conf/analysis/synonym.txt` file accordingly.

Run the same repro script.

## Problematic Queries

### Query: application server

Original text: `This is ApplicationServer and EAP world.`
Highlighted text: `This is ApplicationServer and EAP world.`

Original text: `Hello EAP World!`
Highlighted text: `Hello EAP World!`

In both cases the word `world` should not be highlighted.

Also both documents containing only individual terms from the multi-term synonym match as well (`application` or 'server'). But this time it is probably correct according to analyzer output:

curl 'localhost:9200/test/_analyze?pretty=1&text=application+server&analyzer=ta'

However, it would be great if there is an option how to avoid matching documents having partial synonym phrases only.

### Query: eap

The same output (and issues) as for the previous query.

### Query: server

Original text: `This is ApplicationServer and EAP world.`
Highlighted text: `This is ApplicationServer and EAP world.`

Original text: `Hello EAP World!`
Highlighted text: `Hello EAP World!`

**Problem**: the word `world` should not he highlighted.