ajaxorg/ace — pull request #930 by gjtorikian (2012-09-11 05:19:24+00:00)
Title: Make Syntax Highlighter Generation Over 9,000 Times More Efficient

Hold on to your hats, kids, because I am about to _blow them off_.

Frustrated with the frequent requests for new syntax highlighter in Ace, I developed a tool to transform _.tmlanguage_ files into Ace syntax highlighter modes, along the same lines as _tool/tmtheme.js_ does. It takes one argument--the _.tmlanguage_ file--and creates two files: one for the language mode, and one for the set of highlight rules. Developers will still need to add the code into _kitchen_sink.html_ and _demo.js_.

With this method, I was able to convert the Jade syntax highligher in a little under an hour. Yeah, it's that awesome. I bundled the Jade highlighter into this PR. I can't seem to find out how to generate the tests, though.

If that were the only things that changed--Lord, I wouldn't have to write all of this down. But alas, as with many things, once I went down the rabbit hole of providing one simple tool, the changes kept coming....

## Notes and Limitations in the TMLanguage Converter Tool

It is an understatement to say that the tool is imperfect. Probably, this will never be able to be fully autogenerated. In _.tmlanguage_ files, for instance, one sees the use of lookbehinds/negative lookbehinds, which JavaScript regexps simply do not have. There's a list of other non-determinable items, too:

* Deciding which state to transition to
While the tool does create new states correctly, it labels them with generic terms like `state_2`, `state_10` _e.t.c.
* Extending modes
Many modes say something like `include source.c`, to mean, "add all the rules in C highlighting." That syntax does not make sense to Ace or this tool
* Rule preference order
I could not determine whether the ordering of the content in a _.tmlanguage_ file has precedences
* Gathering keywords
Most likely, the user will need to take keywords and run them through `createKeywordMapper()`

Syntax highlighters and themes generated by the tools have their UUID now added to the beginning. I thought this would be a good way to distinguish between updates to the files.

## Limitations in Ace's Mode Highlighting

To be perfectly honest, the Jade highlighting is about 85% of the way there. What's missing? Jade is extremely sensitive to whitespacing. Coloring of items depends often on the arrangement of higher parents. For example, with comments:

```jade
// block comment start
comment
commment
comment
not a comment
```

The idea is that the starting comment is indented four spaces. As long as subsequent lines exceed that number, they are considered a block comment. The last line, also being of four spaces, does not get matched.

I could not figure out how to do this in Ace. Basically, when I transition to the `next` state, I need to remember the initial whitespace capture. So:

```json
{
"token" : function(space, text) {
return "punctuation.section.comment";
},
"regex" : "^((\\s*)\/\/)(?:\\s*$)",
"next": "comment_block"
}

...
"comment_block": [
{
"token": function(text) {
return "text";
},
"regex": "^(\\1\\S|$)",
"captures": "1",
"next": "start"
},
{
"token": "comment.block.jade",
"merge" : true,
"regex" : ".+"
}
}
```

Where `captures: 1` is a completely fake thing I created to represent the `\1` capture from the beginning.

## Updates to Themes & Theme Tool?

The fun doesn't stop. I noticed that some of the Jade syntax highlighting was not working. This is because we were missing a few _.tmtheme_ variables in the theme generator. I added these directly into the CSS files but I felt dirty. When I regenerated the files from the _.tmtheme_, it overwrote and erased settings for `ace_indent-guide`, as well as some other classes (like gutter and print-margin).

Should we change the theme tool to stop writing over existing files ? Either we move everything into the tool, or stop using it for existing themes. I guess what would be best would be to migrate to a hybrid approach--if a rule exists, ignore it, don't overwrite the file--otherwise, add the new rules in.

I would also like to stop using somewhat invented Eclipse, GitHub, Chrome_ _e.t..c._ themes and move to using actual existing _.tmtheme_ files for these.

## Updates to C Syntax Highlighting

Believe it or not, this PR was even larger. In my zeal for using the new tool, I started to provide Ace syntax highlighting for Objective-C. However, the task was taking me too long, so I opted to drop it out of this branch in favor of another location (for another time).

While working on Obj-C, however, I noticed that our C syntax highlighting is really missing a lot of stuff. I left those updates in this PR. You can see an impact of the changes here: http://imgur.com/a/QUEgT