Issue 1:
If I PUT this template definition to http://localhost:9200/_template/my_new_template it works, but notice the field name "some_field " actually has a trailing whitespace in the String (I think that should be trimmed).

{
"template": "some_index_*",
"mappings": {
"some_mapping": {
"properties": {
"some_field ": {
"type": "double"
}
}
}
}
}

Issue 2:
In #6736 I started trying to define specs for valid IDs, index names, field names etc, to avoid problems such as conflicts created by having an ID called _mapping.

I think this is the wrong approach - a significant number of users will find that they have used identifiers which are no longer illegal. Instead, we should identify where conflicts are possible and implement escaping to allow the unambiguous avoidance of conflict.

Possible conflicts can arise in:

directory or file names (eg index "C:/")

URL paths (eg an ID called _mapping or _create)

query string (eg a routing value containing a comma)

field paths (eg a field containing a .)

scripting (eg a fieldname called _id)

We should provide a simple consistent escaping mechanism with minimal restrictions on allowable characters.

Directory or file names

Today we use the actual index name to create a directory on the filesystem. Instead, the original index name should be stored in the index metadata, and the directory name should be escaped as printable ASCII only (eg percent encoding?).

There are more restrictions that need to be applied to filesystem names, most of which are already enforced today:

must be lowercase

must NOT start with an underscore, a + or a -

index names starting with '.' are reserved

. and .. are forbidden, though they can occur in the name

\, /, *, ?, ", <, >, |, ,, are forbidden

illegal windows filenames are forbidden: CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these names followed immediately by an extension; for example, NUL.txt is not recommended.

The same should apply to snapshot names.

Filename length should be checked after escaping, not before.

URL paths and query strings

Unicode characters and / are already handled by standard URI encoding. The only special characters are:

a leading underscore _

a comma ,

These can be escaped automatically by prepending \ (so we would need to escape literal \'s as well).

The * is used as a wildcard for various URL parameters, eg index and type names. We cannot automatically escape the wildcard (because usually it should be a plain wildcard). Similarly, the user can't escape it as my_literal_\*, because the \ would itself be escaped as: my_literal_\\*, meaning that the wildcard would still function as a wildcard.

The * should not be allowed in any identifiers which support wildcards (eg index, type, node names etc).

Index wildcards can use a leading + or - to include/exclude patterns, but these characters should already be excluded by the index naming rules above.

Field paths and Scripting

A field containing a . can be escaped as foo\.bar (and a field containing \ as foo\\bar).

Fields beginning with _ can clash with metadata fields like _id or _timestamp, both in queries, aggregations, sorting, and scripting.

There are four options:

Forbid fieldnames beginning with an underscore - this is too stringent and will impact existing data

Reserve existing metadata field names - this makes it difficult to add more

Allow \ escaping to distinguish the metadata _id field from the body \_id field

Add a single namespace for metadata fields, eg _root._id or _doc._id (_meta is already taken unfortunately)

Also see #5558