Issue 1:
Background

The most important thing about specifying analyzers is the analyzer used at index time needs to be basically the same as the analyzer used at query time. If completely different analyzers were to be used, you would either produce terms that could never be found at query time, or query for terms that could never exist in the index. The API for specifying analyzers on fields does allow to set index and query analyzers separately. This is to allow things like synonyms where you may want to only add synonyms at index time (yields cheaper queries later), or just add them within the query (more flexible since synonyms can be changed dynamically).

_analyzer today

Today we have multiple ways to specify which analyzer will be used on text fields. At indexing time, the order to check is as follows:

analyzer for the field

_analyzer (proposed to remove here)

type level default analyzer (will be removed in #8874)

index level default analyzer

_analyzer is a special field in the document which specifies the name of an analyzer to use as the default for that document. This means that the same field for one document can use a completely different analyzer than another document. The typical use case for this is working with documents in many languages, where each document contains a field specifying which language its main data is in (e.g. subject and body fields). Then at query time, either a single query is used with a “magic” analyzer over that field, or a conjunction of queries that use every analyzer the data may have been indexed with.

Problems with _analyzer

The typical use case for _analyzer has many problems:

If using the single analyzer at query time approach, the “magic” analyzer is never good enough. It cannot possibly cover all the terms that may have been produced by each languages' analyzer, so some terms are never matchable. For example, “die” in German would be a stop word, while the same word in English is simply a regular word. Either the magic analyzer removes “die” (in which case English documents about dying can not be found) or it includes it, and German documents that contained it can not match (since the indexing process removed that term).

Scoring will be skewed. Text relevance models rely on term statistics to weigh the importance of a term for a given document versus the importance of that term for the entire index. Because some words may analyze to the same term in different languages, the frequencies of terms can be skewed, which will distort where documents matching these words appear in query results.

Mappings code is already complicated, and this feature further complicates following the logic of where Analyzers are set in code.

Having multiple ways to set the analyzer used at index or query time is also confusing on users, as they have to decide which way is “better”.

Proposal

I propose to remove _analyzer, and the associated “analyzer” setting of the match query. Removing this, along with the type level default in #8874 will simplify specifying analyzers considerably. There would be no loss of end functionality, since better results can be achieved with multiple fields.

Alternatives to _analyzer

One alternative to dealing with multiple languages is to use n-grams. While this can itself be tricky to deal with, it is worth mentioning.

An alternative that is more directly in line with the current uses of _analyzer is having one field per language. This requires slight modifications to client code for indexing (copy data into the field for the appropriate language, instead of specifying the language in a field) and querying (query the appropriate language field, instead of selecting an analyzer for that language. This will produce better results since documents from other languages cannot accidentally appear in the results, and relevance results should be as expected for that language.

Issue 2:
ES with Multi-Cluster Search Support

It'd be nice to be able to query across multiple clusters and get their aggregated results.

Our initial motivation is to view Kibana results across multiple clusters. See: Enhancement: Allow conntections to multiple ES backends from a single Kibana instance.

However, since we also use/query ES directly, a proxy would work far better than changing Kibana.

In the discussion above, it was decided (by Shay) that the place to do it would be at the ES level.

The following is a proposal to achieve it at the ES level. This is some very early planning and I've decided to post it early to make sure I'm not duplicating work or am on a totally wrong path (which is totally possible). Any feedback/comments are appreciated.

Proposal

The general plan is to make a query only node (termed 'search load balancer' in elasticsearch.yml) and have a list of cluster names that we want to query.

During a search, the node will query from each of the shards in each of the clusters and aggregate the results.

Details

Make a query only node

In elasticsearch.yml

node.master: false

node.data: false

I'm hoping this means that we can isolate code changes to only the search portion.

Accommodate multiple clusters

Have ZenDiscover reach out to all the listed clusters to get their state.

ClusterService will contain a map of ClusterName -> ClusterState.

To maintain the interface, clusterService.state() will just return the first cluster.

We can have another interface that will allow us to get the the cluster map.

MulticastZenPing will have to listen for changes in the listed clusters and update the corresponding cluster state.

Searching across multiple clusters

I've only looked at TransportSearchTypeAction and TransportSearchQueryThenFetchAction so far.

In the BaseAsyncAction, we'll need to get all the relevant shards across each cluster and query them (sendExecuteFirstPhase).

We change expectedSuccessfulOps and expectedTotalOps to the multi-cluster counts so that in onFirstPhaseResult we know when to move on (innerMoveToSecondPhase).

In moveToSecondPhase, we again use the metadata from each cluster and do the actual fetch of the documents.

Finally in innerFinishHim, we merge all the results with the SearchPhaseController and return the response via the normally.

Thanks!