Shopify/liquid — pull request #235 by trishume (2013-07-26 17:52:22+00:00)
Title: Add a Real Parser. Closes #229 and closes #225.

@fw42 @RichardBlair @boourns @tobi
This massive pull request contains a brand new parser and lexer for variable tags and if statements that can be easily extended to work with all other tags. It uses proper parsing so it is both easier to understand and less fraught with quirks and bugs. It gives real syntax errors that tell the user what they did wrong.

The parser and lexer are made as utilities that can easily be used in other tags.

This fixes many problems with the liquid parser including closing #229 and #225. It makes the variable tag sane and
tells users what they did wrong instead of silently doing whatever the heck it wants.

#### Backwards Compatibility
I also added a system for determining the parsing mode. By default it is set to a compatibility mode where if a syntax error occurs it will add an error message and then fall back on the old parser instead of crashing. It can also be set to use either only the old parser or only the new parser.

Shopify could be configured to use the warning mode so that existing templates would continue working but it would show syntax warnings in the theme editor. In Liquid 2.6 the warning mode should be the default but in Liquid 3.0 the strict mode could be the default.

#### Tests
I added many tests for both the lexer and parser classes as well as fixing tests to expect syntax errors. I kept the tests for the lax parser and added a helper that sets the parsing mode to lax. Interestingly, there were even tests
that asserted that certain bugs like #225 existed.

#### Internals
Two new classes were added `Liquid::Parser` and `Liquid::Lexer`. The lexer is a hand-written one using `StringScanner` and the parser is a hand-written LL(k) recursive descent parser.

Tags can use the parser class as a utility to do their own parsing. Here is an example of using the parser class:
```ruby
p = Parser.new(markup)
# Could be just filters with no input
@name = p.look(:pipe) ? '' : p.expression
while p.consume?(:pipe)
filtername = p.consume(:id)
filterargs = p.consume?(:colon) ? parse_filterargs(p) : []
@filters << [filtername, filterargs]
end
p.consume(:end_of_string)
```

#### Examples
Here are some examples of what it fixes:
```
{{ variable.: }}
old => ""
new => Liquid::SyntaxError: Expected id but found

{% assign foo = 1 + 2 %}{{ foo }}
old => "1"
new => Liquid::SyntaxError: Unexpected character +

{% if true && false %} wrong {% endif %}
old => "wrong"
new => Liquid::SyntaxError: Unexpected character &

{{---E(R[\'\"\'\"\'(+=EH%*^(@#^%$)?||?\eE,PUZE:::~~~~``}}
old => ""
new => Liquid::SyntaxError: Unexpected character (.
```

#### Benchmarks

I added a new benchmarking command `rake benchmark:lax` which runs the benchmarks with the old parser.

Old parser (`rake benchmark:lax`):
```
Rehearsal ------------------------------------------------
parse: 3.840000 0.050000 3.890000 ( 4.026889)
parse & run: 8.110000 0.050000 8.160000 ( 8.154465)
-------------------------------------- total: 12.050000sec

user system total real
parse: 3.410000 0.020000 3.430000 ( 3.436315)
parse & run: 7.950000 0.040000 7.990000 ( 8.140029)
```

New parser (`rake benchmark:run`):
```
Rehearsal ------------------------------------------------
parse: 5.560000 0.040000 5.600000 ( 5.745558)
parse & run: 10.800000 0.040000 10.840000 ( 10.930402)
-------------------------------------- total: 16.440000sec

user system total real
parse: 5.450000 0.020000 5.470000 ( 5.575437)
parse & run: 10.630000 0.040000 10.670000 ( 10.783157)
```

The new parser is a bit slower. According to basic profiling the performance issues are mostly in the lexer, which I could try writing in Ragel but I think the added complexity of Ragel outweighs the performance benefits.