Shopify/liquid — pull request #328 by dylanahsmith (2014-03-25 23:21:19+00:00)
Title: Implement tokenization in a C extension.

@arthurnn, @csfrancis for review
cc @tobi, @Shopify/liquid

## Problem

A significant amount of CPU time can be spent rendering liquid, and it is difficult to optimize liquid further using ruby.

## Solution

Add a liquid C extension that initially just does tokenization, and pass a tokenizer object to tags instead of an array of strings. Tags were only calling `shift` on the tokens array, so only this method needed to get implemented.

The ruby implementation is being kept for jruby compatibility.

Although this already improves the benchmarks, the more significant improvement will come from implementing Liquid::Block#parse in C using this tokenizer directly, which was omitted from this pull request for simplicity.

## Benchmark

before
```
Rehearsal ------------------------------------------------
parse: 4.190000 0.040000 4.230000 ( 4.223385)
parse & run: 9.130000 0.040000 9.170000 ( 9.179921)
-------------------------------------- total: 13.400000sec

user system total real
parse: 4.110000 0.020000 4.130000 ( 4.124358)
parse & run: 9.050000 0.060000 9.110000 ( 9.103650)
```

after
```
Rehearsal ------------------------------------------------
parse: 3.850000 0.040000 3.890000 ( 3.881036)
parse & run: 8.860000 0.030000 8.890000 ( 8.898069)
-------------------------------------- total: 12.780000sec

user system total real
parse: 3.760000 0.020000 3.780000 ( 3.785345)
parse & run: 8.960000 0.050000 9.010000 ( 9.015109)
```