Shopify/liquid-c — pull request #16 by pushrax (2014-11-04 18:38:50+00:00)
Title: BlockBody parsing

Unfortunately did most of this before realizing @dylanahsmith had already written something very very similar. At least it taught me a lot about this part of Liquid. Relies on https://github.com/Shopify/liquid/pull/479

Current performance:

```
ruby -----------------------------------
parse: 37.9 (±18.5%) i/s - 1083 in 30.035048s
parse & run: 20.4 (±14.7%) i/s - 582 in 30.056232s

master ---------------------------------
parse: 68.0 (±19.1%) i/s - 1938 in 30.055672s
parse & run: 27.8 (±18.0%) i/s - 798 in 30.038484s

block ----------------------------------
parse: 94.2 (±18.1%) i/s - 2691 in 29.976884s
parse & run: 34.8 (±14.3%) i/s - 1017 in 30.018563s
```

This doesn't support line numbers, since the C tokenizer currently doesn't. My current thoughts on line numbers:

- Most of the time we don't need to know line numbers, and when we do we only need it for a small ratio of tokens.
- The C tokenizer can easily determine the line number of a pointer into its source.
- When the C tokenizer is in use, every token is produced by it.

So, rather than store the line number on the token, if we can make the token (ruby string) be a pointer into the original source string, we can pass it back into C to determine its line number. This could be implemented through a wrapper method in Ruby Liquid that just returns `token.line_number`, which Liquid-C would patch to use the fast method.

@Shopify/liquid