Shopify/liquid-c — pull request #12 by dylanahsmith (2014-10-09 07:40:45+00:00)
Title: Liquid variable parsing that replicates behaviour of the regexes

cc @pushrax

This is a proof of concept variable lax parsing written in C which attempts to replicate the behaviour of liquid's regexes exactly. There are a lot of changes in liquid-c's variable parsing code that were needed to actually handle all the possible edge cases, perhaps assuming some of those edge cases don't happen in practice or not noticing all of them.

I have included unit tests with contrived examples that exercise some of these edge cases. I have run these against ruby liquid to confirm that these are quirks of the lax parser. It would be great if we could remove some of these edge cases by logging when we encounter them to remove the ones that aren't seen in production.

There are certainly inefficiencies in this code to better replicate the behaviour of the separate regexes ruby liquid uses to e.g. find the end of filters, find the filter name, and then find the arguments. Additionally, this is not based on https://github.com/Shopify/liquid/pull/443, so there is still more parsing done during variable rendering.

There are enough code paths here that I don't fully trust the code at the moment and I think implementing strict parsing instead is a better step forward. Mostly I am opening this pull request to show what is needed to handle the lax variable parsing edge cases in C.