Use closures instead of eval/exec for parsing callbacks

Use closures instead because they:
 * are parsed before execution: showing syntax errors ASAP
 * allow code to remain in Python source files:
  - allows tracebacks access to the specific source line that went wrong
 * easier to get right
  - editing is easier due to syntax highlighting
  - testing is easier because syntax errors are caught earlier

The only reason for exec()ing generated code was to:
 A) control the docstring of the callback
 B) parameterize the token/rule name

A) is now accomplished by assigning to .__doc__. B) is accomplished by
closing over the `name` or `rule_name` variables respectively.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Execute code SECTIONs in grammar defintion with source info set

Ensure that both the source filename and line numbers are available and
correct. This permits tracebacks flowing through that code to be
displayed correctly.

Line number information is now preserved for SECTION tokens and finally
attached to the AST.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Update tutorial.md
Merge pull request #11 from muggenhor/closures-instead-of-exec

Use closures instead of exec and attach location info when we *do* exec()

Doubles parsing performance for pypy!
Merge pull request #12 from TheChuckster/patch-1

Updated tutorial.md
Add testcase to verify non expansion of '?list: item+' rules if more than one item matches

When using the EXPAND1 (?) operator on a rule you don't expect it to
expand if it contains more than a single item, regardless of how it came
to contain more than a single item. I.e. if the ?list rule uses a EXPAND
(@) @item rule: the list's items (children) shouldn't expand after the
list had already been expanded.

I.e. in the case of >= 2 matches these rules are expected to be
equivalent:

?list: item+;
?list: item | item item+ ;

This testcase checks for that, but currently that is *not* the case.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Expand children *before* EXPAND1 ?rules

This ensures that EXPAND1 ?rules are never replaced by *multiple*
grand-children of that rule (caused by children expanding to their
children).

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Merge pull request #13 from muggenhor/dont-expand1-lists-with-multiple-items

BUGFIX: Don't expand1 rules (e.g: ?rule) with multiple children
Test whether empty EXPAND1 lists behave as expected

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Prevent `IndexError`s when building empty EXPAND/EXPAND1 lists

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Add test cases for the expected structure of FLATTEN rules

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Add test-case for resursive FLATTENing rules

Specifically verifies that #rules marked for FLATTENing don't produce
unboundedly deep trees, causing stack overflows when PLY debugging is
enabled.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Flatten recursive #rules in-place to significantly reduce tree-depth

Same solution as 8d4860b5bd (that was for in-place expansion of @rules
and ?rules).

In short: without this change recursive #rules marked for FLATTENing can
cause ASTs with unbounded depth. Due to a recursive `repr()`
implementation this can cause stack overflows when running with
`debug=True`. This commit prevents this problem.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
refactor parser-rule actions to use same definition for all rules

I.e. use the same action when matching a parser-rule with flags (?#@) as
without. Due to previous changes the rule for EXPAND/EXPAND1/FLATTEN was
already suitable to be used for the non-flag case. As a result the only
changes are indentation and comments along with the removal of the
non-flag-only action.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Merge tag 'bugfixes' of git://github.com/muggenhor/plyplus

AST: add test cases, fix bugs, refactor

 * More thoroughly document expected AST in form of test cases
 * Fix a bug causing `IndexError`s on empty EXPAND/EXPAND1 [@?]rules
 * Prevent unbounded tree-depth on FLATTEN #rules
 * Merge parser-rule-action code to be the same for all rules
Add testcase: creation of Grammar objects from non-standard file objects

Verify that a Grammar can be created from file-like objects other than
Python's standard 'file' object. In this particular case: the standard
StringIO classes.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Don't assume that every file-like object has a 'name' attribute

When creating `Grammar` instance do not assume that just because we've
received a file-like object containing the grammar definition it'll also
have a 'name' attribute. Instead *always* try to get the 'name'
attribute, falling back on failure too using "<string>" as replacement
value.

Additionally verify that not just non-file-like objects but file-like
objects' read return value as well are of string type.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Stay closer to regular pickle protocol

Pickle by default pickles obj.__dict__, __getstate__ overrules that.
Rather than returning something completely incompatible with __dict__,
return a shallow copy of __dict__ with all contained weak references
removed. This gets rid of the 'parent' attribute in a way compatible to
the default __setstate__ implementation and as such removes the need to
provide a non-default implementation for that.

Additionally move this pickle-protocol behaviour to a mixin-class to
allow easy reuse where necessary.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Allow pickling of `parent` attribute on `STree`s

From the pickle documentation:
> Object sharing happens when there are multiple references to the same
> object in different places in the object hierarchy being serialized.
> pickle stores such objects only once, and ensures that all other
> references point to the master copy.

I.e. the circular parent references *can* be serialized by converting
the weak references to strong references while serializing and
converting back to weak references while deserializing.

This change actually does that. As a result pickling a subtree with
parents computed will actually end up storing the entire tree while upon
deserialization all parents of the subtree's root will end up getting
destroyed immediately.

Signed-off-by: Giel van Schijndel <me@mortis.eu>
Merge branch 'pickling'