logstash/logstash — pull request #895 by mpasternacki (2013-12-23 04:31:37+00:00)
Title: Config refactor, syntax tweaks (not backwards compatible)

After unsuccessfully trying to escape a code snippet for the Ruby filter, I have looked into the configuration implementation. The implementation looked back. We've had an interesting weekend.

I have modified syntax to implement proper string escaping (it was not possible to escape some strings in current implementation). The double-quoted string escaping is NOT backwards compatible, as it did not seem possible to implement proper escaping without modifying current behaviour. After changes, double-quoted string is parsed by the JSON module, and single-quoted string is interpreted as POSIX shell single-quoted string (i.e. no escaping, no single quote inside the string possible).

I have expanded syntax to make it consistent with JSON: accept exponential `1.0e3` number syntax, accept `:` as alternative key/value separator, allowed (optional) commas between key/value pairs, added `null`, `true`, and `false` literals. After that, most of the syntax (in particular, plugin attributes & hashes) is a superset of JSON, which makes it much easier to automatically generate configuration snippets. I have added parsing fixtures from `json.gem`'s test suite to ensure that JSON support is complete.

I also propose a refactor that simplifies the code, makes the code and syntax more consistent, and also implements some of the suggestions left in the comments. The code is more concise (`grammar.treetop` and `config_ast.rb` have together lost ~128 lines of code, even though the changes introduce new comments).

As the change is not backwards compatible, I suppose it needs to wait until at least a minor version release (assuming that the string parsing rules not allowing to express some strings are considered a bug). Please let me know if it needs any changes that would be needed to support the transition (automatic translation of old configuration, maybe?). I will update the documentation once the shape of the refactor has been decided.