Issue 1:
The following looks like a bug to me (but maybe I misunderstand the intent of the two filters in question and how they are meant to interact).

Elasticsearch version
2.2

Plugins installed
https://github.com/francesconero/elasticsearch-concatenate-token-filter/tree/2.2.0

JVM version
java version "1.8.0_51", Java(TM) SE Runtime Environment (build 1.8.0_51-b16), Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)

OS version
Mac OSX 10.10.5

The problem:

I am trying to use the dictionary_decompounder filter to extract sub-words and at the same time stop other words from being stemmed/decompounded using the keyword_marker filter - the dictionary_decompounder ignores tokens marked as keywords and de-compounds them.

The expectation is that the tokens marked as keywords are not touched by the dictionary_decompounder filter (in the same way as stemmer filters do not touch them).

Additionally I notice that the dictionary_decompounder will generate duplicate tokens when the input token is equal to a matching sub-word ("keuken" in the example below) - this looks like it is not intended but it is easily worked around using a unique filter (so, for me, it's not so much of a problem)

Steps to reproduce**:

1. create a test index:

N.B. the control_* analyzers are included to show that stemming does not occur on keywords and also that the word "messing" is [incorrectly] stemmed to "mes"

{
"analysis" : {
"analyzer" : {
"control_stem" : {
"type" : "custom",
"tokenizer" : "standard",
"filter" : ["kp_snow"]
},

"control_kwstem" : {
"type" : "custom",
"tokenizer" : "standard",
"filter" : ["nl_keywords", "kp_snow"]
},

"test" : {
"type" : "custom",
"tokenizer" : "standard",
"filter" : ["nl_keywords", "nl_dehyphenate", "kp_snow"]
}
},

"filter" : {
"kp_snow" : {
"type" : "snowball",
"language" : "Kp"
},

"nl_keywords" : {
"type" : "keyword_marker",
"keywords" : [
"messing"
]
},

"nl_dehyphenate": {
"type" : "dictionary_decompounder",
"min_word_size" : 6,
"min_subword_size" : 3,
"max_subword_size" : 15,
"word_list" : [
"keuken",
"mes"
]
}
}
}
}

2. Perform an analyzer query

Assuming the created index is called "decomp_test" the following request illustrate expected/actual analyzer results:

http://localhost:9200/decomp_test/_analyze?analyzer=test

Actual Tokens
Expected Tokens

messing, mes, keukenme, keuk, mes, keuk, keuk
messing, keukenme, keuk, mes, keuk

Issue 2:
A potentially useful processor is one that can generate one or more "fingerprints" from an incoming document. This could aid in finding duplicates, detecting plagarism, or clustering similar documents together.

I think there are two realms of fingerprinting: content fingerprinting and structural fingerprinting.

Content Fingerprinting

Hashes the content of fields to generate a fingerprint-per-field, and optionally, a fingerprint that represents all the fields. Could use simple hashing, or perhaps something more sophisticated like MinHash, SimHash, Winnowing or Probabilistic Fingerprinting.

The API could look something like:

{
"fingerprint": {
"type": "content",
"fields": ["foo", "bar"],
"hash": "minhash",
"hash_all": true
}
}

E.g. specify the type of fingerprinting we want to do (content), a list of fields to hash, the style of hashing and if we should also hash all the hashes together. The output would then be the document + new fingerprint fields:

{
"foo": "...",
"bar": "...",
"fingerprint_foo": 1283891,
"fingerprint_bar": 8372038,
"fingerprint_all": 3817273
}

Structural Fingerprinting

The other mode of fingerprinting could be structural in nature (this is the one I'm more interested in, tbh). Instead of fingerprinting the content of fields, we are actually fingerprinting the structure of the document itself. Essentially, we would recursively parse the JSON and hash the keys at each level in the JSON tree. These hashes then become a fingerprint for the structure of the document.

Importantly, this type of fingerprinting ignores the leaf values...we just want to fingerprint the JSON keys themselves.

{
"fingerprint": {
"type": "structure",
"root": ["foo"],
"recursive": true,
"hash": "murmur3",
"hash_all": true
}
}

root: defines where to start recursing, in case you only care about a portion of the document. Could be omitted or set to "*" to process the entire document

recursive if you want the processor to fingerprint all the layers. False if you just want the top-level of keys hashed.

hash: murmur, minhash, etc

hash_all: if all the hashes should be hashed together to build a final fingerprint

And the new document:

{
"foo": {
"bar": {
"baz": "buzz"
},
"beep": {
"boop": "bop"
}
},
"fingerprint_level1": 001734,
"fingerprint_level2": 992727,
"fingerprint_level3": 110293,
"fingerprint_all": 235240
}

Instead of a fingerprint-per-field, we now have one per "level".

I can think of a number of objections to both of these, but this should at least kick off the discussion :)

/cc @martijnvg