Repository: hylang/hy

Issue:
The lexer hits the maximum recursion depth given a file with too many comment lines
     $ yes ';' | head -n 500 >/tmp/foo.hy
     $ hy /tmp/foo.hy
     Traceback (most recent call last):
       File "/home/hippo/Desktop/hyenv/bin/hy", line 11, in <module>
         load_entry_point('hy', 'console_scripts', 'hy')()
       File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 344, in hy_main
         sys.exit(cmdline_handler("hy", sys.argv))
       File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 332, in cmdline_handler
         return run_file(options.args[0])
       File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 211, in run_file
         pretty_error(import_file_to_module, "__main__", filename)
       File "/home/hippo/Desktop/hyenv/hy/hy/cmdline.py", line 184, in pretty_error
         return func(*args, **kw)
       File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 95, in import_file_to_module
         _ast = import_file_to_ast(fpath, module_name)
       File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 53, in import_file_to_ast
         return hy_compile(import_file_to_hst(fpath), module_name)
       File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 41, in import_file_to_hst
         return import_buffer_to_hst(f.read())
       File "/home/hippo/Desktop/hyenv/hy/hy/importer.py", line 34, in import_buffer_to_hst
         return tokenize(buf + "\n")
       File "/home/hippo/Desktop/hyenv/hy/hy/lex/__init__.py", line 17, in tokenize
         return parser.parse(lexer.lex(buf))
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/parser.py", line 32, in parse
         lookahead = next(tokenizer)
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 56, in __next__
         return self.next()
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 41, in next
         return self.next()
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 41, in next
         return self.next()
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 41, in next
         return self.next()
       [Previous line repeated 976 more times]
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexer.py", line 38, in next
         match = rule.matches(self.s, self.idx)
       File "/home/hippo/Desktop/hyenv/lib/python3.6/site-packages/rply/lexergenerator.py", line 33, in matches
         return Match(*m.span(0)) if m is not None else None
     RecursionError: maximum recursion depth exceeded



Discussion:
This feels awfully...familiar? What version of rply are you using?
0.7.4, the latest on PyPI. Can you reproduce this?
This is a bug in rply (alex/rply#52) that is fixed in master but not yet in any release.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -30,7 +30,7 @@ def run(self):
                         "." + filename[:-len(".hy")])
         install.run(self)
 
-install_requires = ['rply>=0.7.0', 'astor>=0.5', 'clint>=0.4']
+install_requires = ['rply>=0.7.5', 'astor>=0.5', 'clint>=0.4']
 if os.name == 'nt':
     install_requires.append('pyreadline>=2.1')
 


Test patch:
diff --git a/tests/compilers/test_ast.py b/tests/compilers/test_ast.py
--- a/tests/compilers/test_ast.py
+++ b/tests/compilers/test_ast.py
@@ -588,3 +588,8 @@ def test_setv_builtins():
         (defn get [self] 42))
       (defn if* [self] 0))
     """)
+
+
+def test_lots_of_comment_lines():
+    # https://github.com/hylang/hy/issues/1313
+    can_compile(1000 * ";\n")