mikel/mail — pull request #490 by bpot (2013-01-09 02:14:53+00:00)
Title: Replace Treetop parser with a Ragel based parser

This pull request replaces the Treetop based parser with a [Ragel](http://www.complang.org/ragel/)-based parser. This change is primarily to improve the performance of message processing. Compared to the Treetop parser the Ragel version is ~7.5x faster.

The new parser exhibits the same behavior as the current parser except for a couple case where the Treetop parser was incorrectly handling fields. I've submitted PRs to fix both issues: [PR #487](https://github.com/mikel/mail/pull/487) [PR #481](https://github.com/mikel/mail/pull/481). A couple specs related to these issues are marked pending in the current PR. Assuming these other two PRs are merged I will rebase and remove the pending lines from those specs.

The change in parsers necessitates removing a [public interface](https://github.com/mikel/mail/blob/85fd50a60c839eca1aa59ce0238b513efaf5d8e4/lib/mail/fields/content_transfer_encoding_field.rb#L26-L30). It is already deprecated but still may require a major version bump.

I know this is a massive change, let me know if there is anything I can do to make it more digestible.

# Benchmark

Parsing a set of 1000 emails from the enron data set:

```
Mail-2.5.3: 24.78s (40.35 emails/second)
Mail-2.5.3 w/Ragel Parser: 3.245290s (308.6 emails/second)
```

# Parser layout

```
lib/mail/parsers/
address_lists_parser.rb # Build data Structs for the Elements (lib/mail/elements)
content_disposition_parser.rb # by interpreting actions emitted by the state machine modules.
...
ragel/
common.rl # Main grammar definition
ruby/
machines/
address_lists_machine.rb # Ragel state machines which emit events that
content_disposition_machine.rb # are consumed by the higher level parsers
...
```

# Further Work

I am also working on a native parser (based on the same ragel grammar) that will improve performance even further. It uses an FFI interface to a custom shared module so that gains can be shared with Rubinius and JRuby. This is one of the advantages of using a Ragel-based parser and why there is a strict separation between the state machine modules and the classes that interpret the actions.