elasticsearch/elasticsearch — pull request #3278 by mattweber (2013-07-01 17:20:37+00:00)
Title: Terms Lookup by Query/Filter (aka. Join Filter)

This PR adds support for generating a terms filter based on the field values
of documents matching a specified lookup query/filter. The value of the
configurable "path" field is collected from the field data cache for each
document matching the lookup query/filter and is then used to filter the main
query. This is can also be called a join filter.

This PR abstracts the TermsLookup functionality in order to support multiple
lookup methods. The existing functionality is moved into FieldTermsLookup and
the new query based lookup is in QueryTermsLookup. All existing caching
functionality works with the new query based lookup for increased performance.

During testing of I found that one of the performance bottlenecks was
generating the Lucene TermsFilter on large sets of terms (probably since
it sorts the terms). I have created a FieldDataTermsFilter that uses the
field data cache to lookup value of the field being filtered and compare it to
the set of gathered terms. This significantly increased performance at the
cost of higher memory usage. Currently a TermsFilter is used when the number
of filtering terms is less than 1024 and the FieldDataTermsFilter is used
for everything else. This should eventually be configurable or we need to
perform some test to find the optimal value.

Examples:

Replicate a has_child query by joining on the child's "pid" field to the
parent's "id" field for each child that has the tag "something".
```
curl -XPOST 'http://localhost:9200/parentIndex/_search' -d '{
"query": {
"constant_score": {
"filter": {
"terms": {
"id": {
"index": "childIndex",
"type": "childType",
"path": "pid",
"query": {
"term": {
"tag": "something"
}
}
}
}
}
}
}
}'
```

Lookup companies that offer products or services mentioning elasticsearch.
Notice that products and services are kept in their own indices.
```
curl -XPOST 'http://localhost:9200/companies/_search' -d '{
"query": {
"constant_score": {
"filter": {
"terms": {
"company_id": {
"indices": ["products", "services"],
"path": "company_id",
"filter": {
"term": {
"description": "elasticsearch"
}
}
}
}
}
}
}
}'
```