textmate/textmate — pull request #1276 by vmg (2014-11-12 23:19:53+00:00)
Title: [RFC] Add syntax to include grammar by resetting their base

Allan,

Here's a small proposal to fix a (non-critical) issue we found while deploying TextMate grammars to production.

As you obviously know (since you designed the format, haha), `include` rules in grammars can use the `$self` and `$base` magic variables to recursively include themselves or the base grammar at the root of the parse tree. This is a crucial feature to parse many programming language that have some kind of recursion in their syntax.

Grammars like C (`source.c`), however, routinely include `$base` instead of `$self` for their recursion rules. This is because the C grammar is included from other languages (like `source.cpp` or `source.objc`) to provide basic syntactic parsing, and when including itself recursively, we want the base grammar to be included again (or else chunks of `source.cpp`would be parsed as `source.c`, as they would be missing the C++ rules).

The bug, which I believe is not trivial to fix, arises on languages that include a grammar like `source.c` not to extend their syntax, but to parse a chunk of code as a different language.

Two obvious examples of this are `source.lua` and `source.ruby`, which include `source.c` to highlight an external block of C declarations or a heredoc with C code, respectively.

The result looks like this:

![screen shot 2014-11-13 at 12 08 00 am](https://cloud.githubusercontent.com/assets/42793/5020401/2ce51712-6ac9-11e4-82b9-f2f67e79fbaf.png)

In this case, when `source.lua` includes `source.c`, and as soon as C does a recursive include (when parsing the inside of a struct definition), the `$base` rule is obviously Lua, so all the C parsing breaks. We're now parsing Lua inside C inside of Lua. This is not what we want!

So, how can we work around this? I propose the following small change in syntax: In an include rule, a scope name followed by two hashes (`##`) resets the base grammar for the inclusion.

Examples:

- `source.ruby#`
Include `source.ruby` with the current base as base.
Equivalent to `source.ruby`.

- 'source.ruby##`
Include `source.ruby` with `source.ruby` as base

- `source.ruby#regexp`
Include the rule `regexp` from `source.ruby`, using the current
base as base.

- `source.ruby##regexp`
Include the rule `regexp` from `source.ruby`, but use
`source.ruby` as base.

This change will allow any languages that need to require a sub-language in an isolated way to do so. I believe this is the least intrusive way to fix this issue.

@sorbits: Do you think this is reasonable, or can you come up with a more elegant way to fix the issue? I'm all ears and eager for your feedback. I'd love to get this fixed! :)

Cheers,
vmg