Repository: neogeny/TatSu

Issue:
Left recursion bug when not at top level?
I have the following grammar that purports to parse declarations of the form: `<type> <name>`. The `type` is either an identifier, or an array of `type`, so `int x` and `int[] x` and `int[][] x` would be valid declarations.

```py
import tatsu

grammar = r"""
identifier = /\w+/ ;

type = (vector_type | leaf_type) ;
vector_type = base:type '[]' ;
leaf_type = id:identifier ;

decl = type:type name:identifier ;
"""

print(tatsu.parse(grammar, 'int x', start='decl'))
```

Yet when I try to parse `int x`, I get:

```
tatsu.exceptions.FailedToken: (1:1) expecting '[]' :

^
vector_type
type
decl
```

`int[] x` works fine, and so does `int` if parsed at the top level (`start='type'`). I'm using TatSu 4.3.0.

Discussion:

Patch:
diff --git a/README.md b/README.md
index 8aa1d6e..cd6ced6 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 [![license](https://img.shields.io/badge/license-BSD-blue.svg)](https://raw.githubusercontent.com/neogeny/tatsu/master/LICENSE.txt) [![pyversions](https://img.shields.io/pypi/pyversions/tatsu.svg)](https://pypi.python.org/pypi/tatsu) [![fury](https://badge.fury.io/py/tatsu.svg)](https://badge.fury.io/py/tatsu) [![circleci](https://circleci.com/gh/neogeny/TatSu.svg?style=shield)](https://circleci.com/gh/neogeny/TatSu) [![travis](https://secure.travis-ci.org/neogeny/TatSu.svg)](http://travis-ci.org/neogeny/TatSu) [![docs](https://readthedocs.org/projects/tatsu/badge/?version=stable)](http://tatsu.readthedocs.io/en/stable/) [![landscape](https://landscape.io/github/apalala/TatSu/master/landscape.png)](https://landscape.io/github/apalala/TatSu/master)
 
-> \*At least for the people who send me mail about a new language that they\'re designing, the general advice is: do it to learn about how to write a compiler. Don\'t have any expectations that anyone will use it, unless you hook up with some sort of organization in a position to push it hard. It\'s a lottery, and some can buy a lot of the tickets. There are plenty of beautiful languages (more beautiful than C) that didn\'t catch on. But someone does win the lottery, and doing a language at least teaches you something.\*
+> *At least for the people who send me mail about a new language that they\'re designing, the general advice is: do it to learn about how to write a compiler. Don\'t have any expectations that anyone will use it, unless you hook up with some sort of organization in a position to push it hard. It\'s a lottery, and some can buy a lot of the tickets. There are plenty of beautiful languages (more beautiful than C) that didn\'t catch on. But someone does win the lottery, and doing a language at least teaches you something.*
 >
 > [Dennis Ritchie](http://en.wikipedia.org/wiki/Dennis_Ritchie) (1941-2011) Creator of the [C](http://en.wikipedia.org/wiki/C_language) programming language and of [Unix](http://en.wikipedia.org/wiki/Unix)
 
@@ -16,14 +16,14 @@
 Installation
 ------------
 
-``` {.sourceCode .bash}
+``` {.bash}
 $ pip install TatSu
 ```
 
 Using the Tool
 --------------
 
-竜 **TatSu** can be used as a library, much like [Python](http://python.org)'s `re`, by embedding grammars as strings and generating grammar models instead of generating [Python](http://python.org) code.
+竜 **TatSu** can be used as a library, much like [Python](http://python.org)\'s `re`, by embedding grammars as strings and generating grammar models instead of generating [Python](http://python.org) code.
 
 -   `tatsu.compile(grammar, name=None, **kwargs)`
 
@@ -44,7 +44,7 @@ Using the Tool
 
 This is an example of how to use 竜 **TatSu** as a library:
 
-``` {.sourceCode .python}
+``` {.python}
 GRAMMAR = '''
     @@grammar::CALC
 
@@ -99,7 +99,7 @@ if __name__ == '__main__':
 
 This is the output:
 
-``` {.sourceCode .console}
+``` {.console}
 # PPRINT
 [ '3',
   '+',
@@ -128,7 +128,7 @@ This is the output:
 License
 -------
 
-You may use 竜 **TatSu** under the terms of the [BSD](http://en.wikipedia.org/wiki/BSD_licenses#2-clause_license_.28.22Simplified_BSD_License.22_or_.22FreeBSD_License.22.29)-style license described in the enclosed [LICENSE.txt](LICENSE.txt) file. \*If your project requires different licensing\* please [email](mailto:apalala@gmail.com).
+You may use 竜 **TatSu** under the terms of the [BSD](http://en.wikipedia.org/wiki/BSD_licenses#2-clause_license_.28.22Simplified_BSD_License.22_or_.22FreeBSD_License.22.29)-style license described in the enclosed [LICENSE.txt](LICENSE.txt) file. *If your project requires different licensing* please [email](mailto:apalala@gmail.com).
 
 Documentation
 -------------
diff --git a/grammar/tatsu.ebnf b/grammar/tatsu.ebnf
index 3ea4896..f017615 100644
--- a/grammar/tatsu.ebnf
+++ b/grammar/tatsu.ebnf
@@ -25,9 +25,13 @@ directive
     =
     '@@' !'keyword' ~
     (
-            name:('comments' | 'eol_comments' | 'whitespace') ~
+            name:('comments' | 'eol_comments') ~
             ~
             '::' ~ value:regex
+        |
+            name:('whitespace') ~
+            ~
+            '::' ~ value:(regex|'None'|'False'|`None`)
         |
             name:('nameguard' | 'ignorecase' | 'left_recursion' | 'parseinfo')
             ~
@@ -40,7 +44,7 @@ directive
             name:('namechars')
             ~
             '::' ~ value:string
-    )
+    ) ~
     ;
 
 
@@ -452,8 +456,8 @@ regexes
 
 regex
     =
-    | '/' ~ @:?"([^/\\]|\\/|\\.)+" '/' ~
-    | '?/' ~ @:?"(.|\n)+?(?=/\?)" ?"/\?+" ~
+    | '/' ~ @:?"([^/\\]|\\/|\\.)*" '/' ~
+    | '?/' ~ @:?"(.|\n)*?(?=/\?)" ?"/\?+" ~
     | '?' @:STRING
     ;
 
diff --git a/tatsu/bootstrap.py b/tatsu/bootstrap.py
index 5fcba66..88103ff 100644
--- a/tatsu/bootstrap.py
+++ b/tatsu/bootstrap.py
@@ -17,7 +17,8 @@ import sys
 
 from tatsu.buffering import Buffer
 from tatsu.parsing import Parser
-from tatsu.parsing import tatsumasu
+from tatsu.parsing import tatsumasu, leftrec, nomemo
+from tatsu.parsing import leftrec, nomemo  # noqa
 from tatsu.util import re, generic_main  # noqa
 
 
@@ -120,8 +121,6 @@ class EBNFBootstrapParser(Parser):
                                 self._token('comments')
                             with self._option():
                                 self._token('eol_comments')
-                            with self._option():
-                                self._token('whitespace')
                             self._error('no available options')
                     self.name_last_node('name')
                     self._cut()
@@ -130,6 +129,26 @@ class EBNFBootstrapParser(Parser):
                     self._cut()
                     self._regex_()
                     self.name_last_node('value')
+                with self._option():
+                    with self._group():
+                        self._token('whitespace')
+                    self.name_last_node('name')
+                    self._cut()
+                    self._cut()
+                    self._token('::')
+                    self._cut()
+                    with self._group():
+                        with self._choice():
+                            with self._option():
+                                self._regex_()
+                            with self._option():
+                                self._token('None')
+                            with self._option():
+                                self._token('False')
+                            with self._option():
+                                self._constant('None')
+                            self._error('no available options')
+                    self.name_last_node('value')
                 with self._option():
                     with self._group():
                         with self._choice():
@@ -174,6 +193,7 @@ class EBNFBootstrapParser(Parser):
                     self._string_()
                     self.name_last_node('value')
                 self._error('no available options')
+        self._cut()
         self.ast._define(
             ['name', 'value'],
             []
@@ -910,14 +930,14 @@ class EBNFBootstrapParser(Parser):
             with self._option():
                 self._token('/')
                 self._cut()
-                self._pattern('([^/\\\\]|\\\\/|\\\\.)+')
+                self._pattern('([^/\\\\]|\\\\/|\\\\.)*')
                 self.name_last_node('@')
                 self._token('/')
                 self._cut()
             with self._option():
                 self._token('?/')
                 self._cut()
-                self._pattern('(.|\\n)+?(?=/\\?)')
+                self._pattern('(.|\\n)*?(?=/\\?)')
                 self.name_last_node('@')
                 self._pattern('/\\?+')
                 self._cut()
diff --git a/tatsu/buffering.py b/tatsu/buffering.py
index abd303c..4ab6cf7 100644
--- a/tatsu/buffering.py
+++ b/tatsu/buffering.py
@@ -306,12 +306,9 @@ class Buffer(object):
 
         if is_match:
             self.move(len(token))
-            if not self.nameguard:
-                return token
-
             partial_match = (
-                token.isalnum() and
-                token[0].isalpha() and
+                self.nameguard and
+                all(self.is_name_char(t) for t in token) and
                 self.is_name_char(self.current())
             )
             if not partial_match:
@@ -343,9 +340,7 @@ class Buffer(object):
         if pos is None:
             pos = self._pos
 
-        if pos >= len(self._line_cache):
-            return LineInfo(self.filename, self.linecount, 0, self._len, self._len, '')
-
+        pos = min(pos, len(self._line_cache) - 2)
         start, line, length = self._line_cache[pos]
         end = start + length
         col = pos - start
diff --git a/tatsu/codegen/python.py b/tatsu/codegen/python.py
index 13bbf13..f02c55b 100644
--- a/tatsu/codegen/python.py
+++ b/tatsu/codegen/python.py
@@ -495,6 +495,7 @@ class Grammar(Base):
                 from tatsu.buffering import Buffer
                 from tatsu.parsing import Parser
                 from tatsu.parsing import tatsumasu, leftrec, nomemo
+                from tatsu.parsing import leftrec, nomemo  # noqa
                 from tatsu.util import re, generic_main  # noqa
 
 
diff --git a/tatsu/contexts.py b/tatsu/contexts.py
index 0fb9375..be58778 100644
--- a/tatsu/contexts.py
+++ b/tatsu/contexts.py
@@ -71,7 +71,8 @@ def nomemo(impl):
     impl.is_memoizable = False
     return impl
 
-class closure(list):  # noqa
+
+class closure(list):
     pass
 
 
diff --git a/tatsu/grammars.py b/tatsu/grammars.py
index 8843994..aef1e77 100644
--- a/tatsu/grammars.py
+++ b/tatsu/grammars.py
@@ -296,7 +296,7 @@ class Token(Model):
         return ctx._token(self.token)
 
     def _first(self, k, f):
-        return set([(self.token,)])
+        return {(self.token,)}
 
     def _to_str(self, lean=False):
         return urepr(self.token)
@@ -333,7 +333,7 @@ class Pattern(Model):
         return ctx._pattern(self.pattern)
 
     def _first(self, k, f):
-        return set([(self.pattern,)])
+        return {(self.pattern,)}
 
     def _to_str(self, lean=False):
         parts = []
@@ -710,7 +710,7 @@ class OverrideList(NamedList):
 
 class Special(Model):
     def _first(self, k, f):
-        return set([(self.value,)])
+        return {(self.value,)}
 
     def _to_str(self, lean=False):
         return '?%s?' % self.value
@@ -780,6 +780,7 @@ class Rule(Decorator):
 
     def parse(self, ctx):
         result = self._parse_rhs(ctx, self.exp)
+        self._add_defined_attributes(result)
         if self.is_name:
             ctx._check_name()
         return result
@@ -787,14 +788,16 @@ class Rule(Decorator):
     def _parse_rhs(self, ctx, exp):
         ruleinfo = RuleInfo(self.name, exp.parse, self.is_leftrec, self.is_memoizable, self.params, self.kwparams)
         result = ctx._call(ruleinfo)
-        if isinstance(result, AST):
-            defines = compress_seq(self.defines())
-            result._define(
-                [d for d, l in defines if not l],
-                [d for d, l in defines if l]
-            )
         return result
 
+    def _add_defined_attributes(self, ast):
+        defines = compress_seq(self.defines())
+        if not isinstance(ast, (AST, Node)):
+            return
+        for d, l in defines:
+            if not hasattr(ast, d):
+                setattr(ast, d, [] if l else None)
+
     def _first(self, k, f):
         if self._first_set:
             return self._first_set
diff --git a/tatsu/parser_semantics.py b/tatsu/parser_semantics.py
index 3d8daf8..f0a16ea 100644
--- a/tatsu/parser_semantics.py
+++ b/tatsu/parser_semantics.py
@@ -121,6 +121,10 @@ class EBNFGrammarSemantics(ModelBuilderSemantics):
     def grammar(self, ast, *args):
         directives = OrderedDict((d.name, d.value) for d in ast.directives)
         keywords = set(ast.keywords or [])
+
+        if directives.get('whitespace') in ('None', 'False'):
+            directives['whitespace'] = ''
+
         return grammars.Grammar(
             self.grammar_name,
             list(self.rules.values()),


Test patch:
diff --git a/test/buffering_test.py b/test/buffering_test.py
index 732d5cb..802ccf9 100644
--- a/test/buffering_test.py
+++ b/test/buffering_test.py
@@ -11,6 +11,7 @@ import random
 import unittest
 from codecs import open
 
+from tatsu import parse
 from tatsu.buffering import Buffer
 from tatsu.util import ustr
 
@@ -80,8 +81,8 @@ class BufferingTests(unittest.TestCase):
                 start = n + 1
         text_len = len(self.text)
         info = self.buf.line_info(1 + text_len)
-        self.assertEqual(info.line, 1 + len(self.text.splitlines()))
-        self.assertEqual(info.start, text_len)
+        self.assertEqual(info.line, len(self.text.splitlines()) - 1)
+        self.assertEqual(info.end, text_len)
 
     def test_linecount(self):
         b = Buffer('')
@@ -93,6 +94,18 @@ class BufferingTests(unittest.TestCase):
         b = Buffer('\n')
         self.assertEqual(2, b.linecount)
 
+    @unittest.skip('not valid')
+    def test_namechars(self):
+        grammar = '''
+            @@namechars :: '-'
+            start =
+                "key" ~ ";"  |
+                "key-word" ~ ";" |
+                "key-word-extra" ~ ";"
+                ;
+        '''
+        self.assertEquals(['key-word-extra', ';'], parse(grammar, 'key-word-extra;'))
+
 
 def suite():
     return unittest.TestLoader().loadTestsFromTestCase(BufferingTests)
diff --git a/test/grammar/directive_test.py b/test/grammar/directive_test.py
index b91e86e..530def0 100644
--- a/test/grammar/directive_test.py
+++ b/test/grammar/directive_test.py
@@ -4,6 +4,7 @@ from __future__ import absolute_import, division, print_function, unicode_litera
 import unittest
 
 import tatsu
+from tatsu.exceptions import FailedParse
 from tatsu.util import trim
 from tatsu.codegen import codegen
 
@@ -23,6 +24,31 @@ class DirectiveTests(unittest.TestCase):
         code = codegen(model)
         compile('test.py', code, EXEC)
 
+    def test_whitespace_none_directive(self):
+        grammars = [
+            '''
+                @@whitespace :: None
+                @@nameguard :: False
+
+                test = {'x'}+ $;
+            ''',
+            '''
+                @@whitespace :: False
+                @@nameguard :: False
+
+                test = {'x'}+ $;
+            ''',
+        ]
+        for grammar in grammars:
+            model = tatsu.compile(grammar, "test")
+            self.assertEquals(['x', 'x'], model.parse('xx', trace=True))
+            try:
+                self.assertEquals(['x', 'x'], model.parse('x x', trace=True))
+            except FailedParse:
+                pass
+            else:
+                self.fail('parsed through non-whitespace')
+
     def test_eol_comments_re_directive(self):
         grammar = '''
             @@eol_comments :: /#.*?$/
diff --git a/test/grammar/left_recursion_test.py b/test/grammar/left_recursion_test.py
index 30962c3..2d8bd25 100644
--- a/test/grammar/left_recursion_test.py
+++ b/test/grammar/left_recursion_test.py
@@ -4,7 +4,7 @@ from __future__ import absolute_import, division, print_function, unicode_litera
 import unittest
 
 from tatsu.exceptions import FailedParse
-from tatsu.tool import compile
+from tatsu.tool import compile, parse
 
 
 class LeftRecursionTests(unittest.TestCase):
@@ -476,3 +476,18 @@ class LeftRecursionTests(unittest.TestCase):
 
         ast = model.parse('a(b, c)', start='expr', trace=trace, colorize=True)
         self.assertEqual(['a', '(', ['b', 'c'], ')'], ast)
+
+    def test_not_at_top_level(self):
+        grammar = r"""
+            identifier = /\w+/ ;
+
+            type = (vector_type | leaf_type) ;
+            vector_type = base:type '[]' ;
+            leaf_type = id:identifier ;
+
+            decl = type:type name:identifier ;
+        """
+        self.assertEquals(
+            {'type': {'id': 'int'}, 'name': 'x'},
+            parse(grammar, 'int x', start='decl').asjson()
+        )
diff --git a/test/grammar/semantics_test.py b/test/grammar/semantics_test.py
index 7ee837f..040336f 100644
--- a/test/grammar/semantics_test.py
+++ b/test/grammar/semantics_test.py
@@ -91,3 +91,19 @@ class SemanticsTests(unittest.TestCase):
         self.assertTrue(hasattr(ast, "b"))
 
         self.assertTrue(issubclass(D, (A, B, C)))
+
+    def test_optional_attributes(self):
+        grammar = r"""
+            foo::Foo = left:identifier [ ':' right:identifier ] $ ;
+            identifier = /\w+/ ;
+        """
+
+        grammar = compile(grammar)
+
+        a = grammar.parse('foo : bar', semantics=ModelBuilderSemantics())
+        assert a.left == 'foo'
+        assert a.right == 'bar'
+
+        b = grammar.parse('foo', semantics=ModelBuilderSemantics())
+        self.assertEquals(b.left, 'foo')
+        self.assertIsNone(b.right)