Repository: lark-parser/lark
Pull request #680 (2020-08-28 19:31:38)

Make Trees reconstructable from reprs
This request makes the strings returned from `repr(some_token_or_tree)` valid python code by changing the repr output to include quotes around a tree's 'data' or a token's 'type'.

### For example:
Let's take a possible parsed version of a binary-op for some language.
(shown with `.pretty()` for layout)
```
binop_expr
    expr
        var_expr    a
    binop
        add
    expr
        var_expr    b
```

Without the proposed changes the `repr` of the above Tree would look like:
```
Tree(binop_expr, [Tree(expr, [Tree(var_expr, [Token(NAME, 'a')])]), Tree(binop, [Tree(add, [])]), Tree(expr, [Tree(var_expr, [Token(NAME, 'b')])])])])
```

The change proposed would alter this to look like:
```
Tree('binop_expr', [Tree('expr', [Tree('var_expr', [Token('NAME', 'a')])]), Tree('binop', [Tree('add', [])]), Tree('expr', [Tree('var_expr', [Token('NAME', 'b')])])])])
```
now `Tree('binop_expr',` and `Token('NAME',` now have quotes around the data and type.


This change has is beneficial in two ways:
1. Primarily, It will allow trees to be stored as text in files without having to pickle the Tree Object. 
2. This change brings the `__repr__` methods of Token and Tree into compliance with what is suggested for the repr function. 

The repr spec"
https://docs.python.org/3.8/reference/datamodel.html?highlight=__repr__#object.__repr__


Patch:
diff --git a/lark/lexer.py b/lark/lexer.py
index 6eb3eca..8fc9e4b 100644
--- a/lark/lexer.py
+++ b/lark/lexer.py
@@ -142,7 +142,7 @@ class Token(Str):
         return (self.__class__, (self.type, self.value, self.pos_in_stream, self.line, self.column, ))
 
     def __repr__(self):
-        return 'Token(%s, %r)' % (self.type, self.value)
+        return 'Token(%r, %r)' % (self.type, self.value)
 
     def __deepcopy__(self, memo):
         return Token(self.type, self.value, self.pos_in_stream, self.line, self.column)
diff --git a/lark/tree.py b/lark/tree.py
index 45346c6..0b7114b 100644
--- a/lark/tree.py
+++ b/lark/tree.py
@@ -39,7 +39,7 @@ class Tree(object):
         return self._meta
 
     def __repr__(self):
-        return 'Tree(%s, %s)' % (self.data, self.children)
+        return 'Tree(%r, %r)' % (self.data, self.children)
 
     def _pretty_label(self):
         return self.data
@@ -207,4 +207,3 @@ def pydot__tree_to_graph(tree, rankdir="LR", **kwargs):
 
     _to_pydot(tree)
     return graph
-   


Test patch:
diff --git a/tests/test_trees.py b/tests/test_trees.py
index d759cf1..cca92f9 100644
--- a/tests/test_trees.py
+++ b/tests/test_trees.py
@@ -7,6 +7,7 @@ import pickle
 import functools
 
 from lark.tree import Tree
+from lark.lexer import Token
 from lark.visitors import Visitor, Visitor_Recursive, Transformer, Interpreter, visit_children_decor, v_args, Discard
 
 
@@ -22,6 +23,9 @@ class TestTrees(TestCase):
         data = pickle.dumps(s)
         assert pickle.loads(data) == s
 
+    def test_repr_runnable(self):
+        assert self.tree1 == eval(repr(self.tree1))
+
     def test_iter_subtrees(self):
         expected = [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z'),
                     Tree('a', [Tree('b', 'x'), Tree('c', 'y'), Tree('d', 'z')])]
@@ -232,4 +236,3 @@ class TestTrees(TestCase):
 
 if __name__ == '__main__':
     unittest.main()
-