Repository: neogeny/TatSu

Issue:
Surprising whitespace handling
Given this example
```python
import tatsu

parser = tatsu.compile(r'''
statement = 'SELECT' 'FROM' table $ ;
table = name:id ;
id = /[a-z]+/ ;
''')

string = 'SELECT FROM   foo'
value = parser.parse(string, parseinfo=True)

table = value[2]
assert table['name'] == 'foo'

parseinfo = table['parseinfo']
print(parseinfo.tokenizer.text)
print(f'{parseinfo.pos * " "}^')
```
I find whitespace handling a surprising. The whitespace between `FROM` and the table name is not skipped over before matching the `table` rule. This results in correct parsing but the `parseinfo` for the `table` rule. I would have expected whitespace to be skipped before attempting to match the `table` rule.

Discussion:

Patch:
diff --git a/README.rst b/README.rst
index eb57f92..683c25c 100644
--- a/README.rst
+++ b/README.rst
@@ -32,12 +32,10 @@ input, much like the `re`_ module does with regular expressions, or it can gener
 |TatSu| supports `left-recursive`_  rules in PEG_ grammars using the
 algorithm_ by *Laurent* and *Mens*. The generated AST_ has the expected left associativity.
 
-Starting with version 5.9.0 |TatSu| requires Python 3.11 or later.
-While no code in |TatSu| yet depends on new language or standard library features,
+|TatSu| requires a maintained version of Python (3.11 at the moment).  While no code
+in |TatSu| yet depends on new language or standard library features,
 the authors don't want to be constrained by Python version comaptibility consideration
 when developing features that will be part future releases.
-Therefore, to simplify version pinning for users of the library,
-they decided to proactively bump the Python minimum required version to 3.10.
 
 |TatSu| releases in the 5.7 series closely track releases in the 5.8 series
 while maintaining compatibility with Python 3.8 and later.
diff --git a/tatsu/contexts.py b/tatsu/contexts.py
index 06458c4..f5d0f96 100644
--- a/tatsu/contexts.py
+++ b/tatsu/contexts.py
@@ -569,6 +569,7 @@ class ParseContext:
 
     @property
     def memokey(self):
+        self.tokenizer.eat_whitespace()
         return MemoKey(self._pos, self.rule, self.substate)
 
     def _memoize(self, key, memo):
diff --git a/tatsu/tokenizing.py b/tatsu/tokenizing.py
index 918f082..bc7077f 100644
--- a/tatsu/tokenizing.py
+++ b/tatsu/tokenizing.py
@@ -40,6 +40,9 @@ class Tokenizer:
     def token(self):
         return self.current
 
+    def eat_whitespace(self):
+        raise NotImplementedError
+
     def next(self):
         raise NotImplementedError
 


Test patch:
diff --git a/test/parsing_test.py b/test/parsing_test.py
index 70b79e3..5c8a5a9 100644
--- a/test/parsing_test.py
+++ b/test/parsing_test.py
@@ -1,9 +1,10 @@
 # -*- coding: utf-8 -*-
+import json
 import unittest
 import tempfile
 
 import tatsu
-from tatsu.util import trim, eval_escapes
+from tatsu.util import trim, eval_escapes, asjson
 from tatsu.grammars import EBNFBuffer
 
 
@@ -115,6 +116,20 @@ class ParsingTests(unittest.TestCase):
         model = tatsu.compile(grammar=grammar)
         model.parse('4 + 5')
 
+    def test_skip_whitespace(self):
+        grammar = '''
+            statement = 'FOO' subject $ ;
+            subject = name:id ;
+            id = /[a-z]+/ ;
+        '''
+        model = tatsu.compile(grammar=grammar)
+        ast = model.parse('FOO' + ' ' * 3 + 'bar', parseinfo=True)
+        print(json.dumps(asjson(ast), indent=2))
+        subject = ast[1]
+        assert subject['name'] == 'bar'
+        parseinfo = subject['parseinfo']
+        assert parseinfo.pos == parseinfo.tokenizer.text.index('bar')
+
 
 def suite():
     return unittest.TestLoader().loadTestsFromTestCase(ParsingTests)