madflow/flow-netbeans-markdown — pull request #33 by stengerh (2013-11-17 21:19:09+00:00)
Title: New lexer for syntax-highlighting

I have experimented with a new lexer which is based on a depth-first, left-to-right traversal of the PegDown AST. The lexer has the following properties:

* It does not generate empty tokens.
* The total length of the generated tokens equals the length of the input.
* It generates an empty token sequence for an empty input.
* It is able to handle nested styles, e.g. emphasis in headers, bold links, etc. Each peace of Markdown source is hihglighted according to innermost applicable style.

The last one is a change from the current behaviour but could be easily changed if desired by the users. Please see the example below which illustrates the highlighting in the editor.

Markdown source:
>

### Header with [link][example]
a *b **c** d* e

HTML as rendered by PegDown:
>

Header with link

> a b c d e

Highlighting with old lexer:
>

## Header with [link][example]
a *b **c** d* e

Highlighting with new lexer:
>

### Header with [link][example]
a *b **c** d* e

[example]: http://example.com